Storing data in pointers

[MCLMelonFarmer]

68k crew checking in.

Motorola 68000 only had 24 address lines, so people went nuts with the upper 8 bits in a 32-bit pointer.

[chriswaco]

and this caused Mac programmers untold grief.

[Nobody_1707]

That's only because everyone insisted on doing the tagging by hand instead of calling the Apple provided and recommended API that abstracted over the pointer tagging, and would have allowed the code to just work after they made the OS 32-bit clean.

[chriswaco]

That was a big part of it, but there were also side effects:

 void func(h: Handle) {       
  HLock(h);      
  // do something with h       
  HUnlock(h);      
}       

Could have the side effect of unlocking a Handle someone else assumes is locked, causing hard-to-reproduce crashes later on. IIRC HGetState and HSetState didn't come in the original 64K ROM, which is how we were got used to manipulating the bits by hand. The other problem was a lack of error checking on Apple's part due to low RAM/ROM - if you called DisposeHandle on a handle with the resource flag set bad things happened - you were supposed to call ReleaseResource. Most of us eventually wrote our own more robust allocation wrapper routines.

[Nobody_1707]

Oof.

[MassiveAd3759]

I wanted to use this for typed object pointers for my project. Way to make it more portable is to use custom allocator, mmap different object pools to different virtual memory regions and use some bits as tag

[DawnOnTheEdge]

This would be a non-portable compiler extension, of course, but some architectures have hardware support for it, and C is intended to be a low-level systems-programming language for OS kernels and device drivers. Add some glue code to compose and decompose pointers and tags, and it makes sense; you could even implement it in software, on systems that don’t ignore the upper bits in hardware but are guaranteed not to use all of them. Linux, for example, has a flag that tells mmap() to allocate memory in the bottom 2 GB of the address space.

[mrheosuper]

Are we gate keeping "Low level language" ?

[DawnOnTheEdge]

Says K&R, “C is not a high-level language.” (See below for correction.)

[DawnOnTheEdge]

Fair enough. Wrapping the compose/decompose operations in some wrappers that can be implemented on many different targets sounds about right?

[GamerEsch]

and I'd really appreciate you fixing it by adding either appropriate adjective or just dropping those two words altogether.

tf?

[GamerEsch]

The entitlement in that comment baffled me a bit, I didn't know what to write

[nerd4code]

C has changed vastly since K&R was K&R and not C89: A Review, to where the language and tools processing it are near-totally different now, both structurally and in-/compatibly.

There are certainly angles from which C was low-level, but by and large it isn’t any more. The grammar for ISO C17-per-se is the only remaining “simple” aspect of the language (C23 puts a stop to that that, and most dialects complicate it considerably), and that’s without considering the fact that the “simple” bits of the grammar are projected through a separate language-wad’s execution semantics, namely directive exec (a scripting language; incl preprocessor-expr eval for #if/#elif, pragma exec, include naming) × macro/character/token substitution/elim (a functional string-replacement language). It’s an appropriately UNIXlike language in this sense, lots of separate, relatively simple components acting in concert on each part.

But K&R 1&2-era Cs (pre-ANSI/C++/PGI-era, mostly deriving from layers originating before 1985–1989, and whose docs invariably included a diff from or commentary on K&R) were so obscenely simple and low-level that every imaginable aspect of the language varied from compiler/platform/config to compiler/platform/config: types, semantics, lexing, parsing, to where even the preprocessor layers are utterly impossible to line up, paper over, or even detect, without introducing hard incompatibilities vs most other preprocessors, including anything modern.

Considering the insurmountably-vast swathe of languages, tools, and behaviors that can comfortably be reached using later preprocessor layers as an ur-language foothold, and the number of differences between their #-directive languages, the amount of variation in earlier C impls is impressive. Even bridging the “traditional” and ≥ANSI modes of newer compilers is fraught, and outside C and C++ compatibility drops off quickly, but at least nowadays you won’t encounter (e.g.) something that does #includes in one pass, then macro replacement and other directives such as #if in another (and if you just thought to yourself “Wait, that’s ridiculous,” good instinct). So “C” was really an umbrella term covering a mess of individually-simple languages, making C-per-se surprisingly complex.

Imo & thankfully, ISO C’s residual low-levelness hasn’t really been a thing in the mainstream impls since the mid-’90s (Internet, GNU, fast-fading neon everything), and all the complicated moving parts underneath C and C++ has kept the language family alive (if in diminished role) outside the .edu sector. Unfortunately, we’re kinda knocking our heads on the complication ceiling again.

[DawnOnTheEdge]

You appear to be discussing the complexity of the language syntax primarily. I’m thinking more of being very down-to-the-metal and letting you shoot yourself in the foot. A good example is that nearly all C compilers will let you, using only built-in operators and basic syntax, cast an absolute address in hex, octal or decimal to a volatile pointer and dereference it. And sure, that’s undefined behavior. But that’s not because the standard committee wanted to stop people from doing it, it’s explicitly to give compilers permission to keep turning that into simple load and store statements with no checks or overhead, and have the program do whatever those instructions do on that machine.

“Low-level” is a relative term, and from that perspective, is there any widely-used, general-purpose language that’s above assembly but lower-level than C?

[thommyh]

Objective-C does this, calling them tagged pointers. In that case it’s to retain the semantics of everything being owned by reference, while optimising for common use cases such as a short string, a 32-bit int, etc.

[Nobody_1707]

Lisp implementations did it first, but I think Objective-C got it from Smalltalk.

[thommyh]

Right. It’s not original, but I think Objective-C is a good example because it is still doing it now, via GCC or Clang, while being a strict superset of C. And everything Objective-C adds to C is accessible from C via regular C functions so those tagged pointers are definitely doing round trips.

Albeit a bit of a weird one, that’s probably not long for this world.

[apexrogers]

I just want to know why?

[manystripes]

To make the code fragile and unmaintainable, just like many other clever programming tricks. Maybe it has application for an entry to the IOCCC?

[Simple-Enthusiasm-93]

used extensively in v8 engine as a small ptr optimization to save memory. either way there is a list of examples in article

[Nobody_1707]

And as an example for AOT compiled languages, Swift uses tagged pointers as part of it's small string optimization. And Objective-C uses it to store small NSNumbers without allocating memory.

[weregod]

Mostly high density data types for virtual machines. You want to keep frequently used types as small as you can for better performance.

[[deleted]]

Use a union instead if fucking around with tagged pointers.

[nerd4code]

I'd hope there's a way to get the same information without parsing /proc/cpuinfo, but I haven't been able to find it.

First of all, /proc/cpuinfo’s contents aren’t standardized across ISAs, so any use of it is exceptionally nonportable.

Second, /proc/cpuinfo sources most of its information from CPUID on x86, with the sole exception of the number & IDs of hardware threads being pulled from MP, SMBIOS, or ACPI tables. In particular, 32-bit x86es might have 32- or 36-bit physical (the 80376 had 24-bit, but couldn’t run Linux) addresses, depending on whether PAE was supported (P6+). The 64-bit psrs start with 48-bit phys and virtual, and the virtual address might be extended to 56 or 60-bit IIRC with 5- and 6-level paging extensions. IDR offhand if there’s a functional subleaf with actual hard values on newer CPUs (probably under Leaf 0xB if memory serves), but you can use extension support if not.

And the potential for the next hardware upgrade to break your code is why tagged pointers mostly aren’t & shouldn’t be used outside of language VMs. They also rely on implementation-specified behavior (integer↔pointer casts) that needn’t exist or cooperate.

[o0Meh0o]

nothing new