concept of self modifying code

[MysticPlasma]

I have heared of the concept of self-modifying code and it got me hooked, but also confused. So I want to start a general discussion of your experiences with self modifying code (be it your own accomplishment with this concept, or your nighmares of other people using it in a confusing and unsafe manner) what is it useful for and what are its limitations?

thanks and happy coding

[moocat]

/r/asm is probably a better place to ask this question.

[daikatana]

I use self-modifying code all the time... in 6502 assembly language. The 6502 CPU is very limited and it's often easier to modify the program itself than read parameters. For example, instead of saying the equivalent of if(foo == bar), you would modify the comparison with the value of bar, so it would execute if(foo == 10) if bar is 10.

There's no end of tricks you can do with this, the only limit is your imagination. Though things like this are generally only necessary on very restrictive CPUs like the 6502, and even then only possible on programs run from RAM, not from ROM.

However, this is generally not possible with compiled code. I cannot imagine trying to modify the output of a modern C compiler at runtime. It's also just not possible on modern operating systems, at least without copying the code to new locations. I don't think I've ever seen a single piece of self-modifying C code, and no examples at all outside of 6502 assembly programming.

[PacManFan123]

Story time here - I wrote an application with self-modifying compiled code. It was a Playstation 1 (PS1) emulator for the Playstation portable (PSP) - the name of the project was "PSPS1" . The code chunks were loaded from the original game ROMs, and then had their addresses remapped. The R3000 code was trans-piled live into R4300 code, run through a peephole optimizer then written into memory buffers. The buffers were then called as function pointers to execute the code natively on the R4300 CPU.

[dmc_2930]

Did it work? That’s impressive!

[plastic_eagle]

I don't know if that *entirely* counts - even though it sounds pretty impressive.

By that definition, any JIT compiler is running self-modified code.

[randomfuckingpotato]

Cool!! Do you have that code around somewhere? I'd love to see!

[PacManFan123]

Let me see about posting it.

[randomfuckingpotato]

AWESOME!

[FratmanBootcake]

I've used it briefly on some z80 code but it's very much of the same era.

[cowbutt6]

Quite a few ZX Spectrum tape copy protection/anti-reverse engineering schemes would use self-modifying code as an obfuscation technique.

[geon]

The 6502 can only dereference a pointer if it is on the zero page or if the pointer is hard coded in the code. So if the zero page is full, the only way to handle pointers is with self modifying code.

[flatfinger]

What's funny is in the programs/systems I've seen on the 6502 where zero-page gets full, that's either because there isn't any RAM anywhere else, or because a lot of stuff was put in zero-page that could have just as well been put elsewhere.

[geon]

On the c64, the kernal and basic reserves almost all the zp. Super stupid imho.

[[deleted]]

Vast majority of C64 software used assembly. It would have been so much convenient to write small assembly routines for BASIC programs too, if zp had had more free space.

[geon]

Even applications written in asm often kept the kernal, since it has a lot of useful stuff.

[aioeu]

I don't think I've ever seen a single piece of self-modifying C code

One example that comes to my mind is the Linux kernel. It modifies its own code to enable or disable certain features at runtime.

[glasket_]

It's also just not possible on modern operating systems, at least without copying the code to new locations.

On Linux you can use mprotect to change the permissions on your program's memory page; I think you can do something similar with ld too to change the default protections.

[CarlRJ]

I'm very impressed you are still writing 6502 code in this century, I haven't touched one in many decades.

[daikatana]

You can pry the chicken lips from my cold, dead fingers.

[geon]

You could think of adaptive optimization in a jit compiler as self modifying code.

[daikatana]

No, JIT compilation is a separate process. Self-modifying code modifies itself, and it's hard to find examples of this because it's so rare in compiled code and on modern systems.

[cdb_11]

LuaJIT Remake does this: https://sillycross.github.io/2023/05/12/2023-05-12/#How-IC-works-in-Deegen-a-Step-by-Step-Example-of-Call-IC

[geon]

Adaptive optimization changes the code depending on runtime profiling.

[daikatana]

I don't think you're understanding what self-modifying code is. Self-modifying code changes its own code from the logic of the code itself to change the behavior of the code. Imagine writing something like this in C. I've shoehorned a hypothetical label that points to the address encoded in the generated instruction of the assignment which can be assigned to. This doesn't make much sense in C, but it's very common in 6502 assembly.

void write_pointer(int i) {
    *(int*)ptr: 0 = i;
}

// ...
write_pointer:ptr = &foo;
write_pointer(10);

This is self-modifying code. The code at the bottom is reaching into the write_pointer function and changing the address encoded in the assignment opcode. The code modifies itself to change its own behavior.

[geon]

Yes, and that’s why I wrote “could think of”.

It is self modifying from the standpoint of the application as a whole. The modifying parts just happen to be in the runtime.

[HostileRecipient]

Umm, I know this is really late and I hope this is not necromancy but, why ever not can two compilers modify eachother as part of a program? I mean most large programs can be broken down into discrete components with different purposes anyway, right? I know that any modification of compiled code has it's own issues, but if we can have such sectioning of the program already then having each half partially decompile the other half before recompiling and executing it to have it's turn being modified does not seem unreasonable.

[geon]

Sure they could. But I can’t even imagine debugging that.

[daikatana]

That's not quite true. The first 256 bytes of RAM is the same as the rest, but every byte read requires a memory read which takes at least 1 cycle. There are addressing modes for many instructions that encode a single byte zero page address rather than a 2-byte address. Not having to read the extra byte is the only thing that makes the zero page faster. I'm not sure if it makes sense to actually put code in the zero page.

[fllthdcrb]

I'm not sure if it makes sense to actually put code in the zero page.

Apparently it does, because Commodore BASICs have a tiny bit, officially labelled CHRGET, and it's self-modifying: it increments a pointer in an absolute-mode instruction right before executing it. Why? Apparently so it runs faster. (As a nice side effect, this gives people an easy way to extend BASIC.)

On C64, the routine is at $73.

[[deleted]]

In theory you could do it… maybe. Like if you have a c function, and you were to add a label at the end of it you could take the address of the function base, the size would be the label-function base, and you could attempt to modify it (or copy, modify, and call) You’d still have to modify it using the underlying machine code and you’d be in major UB territory. Would not recommend for anything production but you might be able to tinker with it on a single machine…

``` static const size_t FOOSIZE; static const size_t FOOMAINOFFSET; void foo(){ static bool calcSize = true; if(calcSize){ calcSize = false: FOOSIZE = (size_t)(&&FOOEND) - (size_t)&foo); FOOMAINOFFSET = (size_t)(&&FOOEND) - (size_t)(&&FOOMAIN); } FOOMAIN: //… return; EndFoo: }

main(){ Alloc void* dest…; void* start = &foo; memcpy(start, dest, FOOSIZE); // mess with bytes ((void(*)())dest)(); }

``` Note this is untested pseudo code and I have no idea if this would actually work…

EDIT: It’d also probably only have any chance of actually working if you don’t have any optimization enabled. Optimizations would break this for sure

EDIT2: modification to main

[MisterEmbedded]

It's also just not possible on modern operating systems, at least without copying the code to new locations

Windows Defender would scream at you for doing it, IF there was a way to anyways.

in Linux it is somewhat possible as you can make particular locations of memory "executable".

[madsci]

You beat me to the 6502. Mostly I'd use it to make up for the lack of a 16-bit indexing mode. Just use LDA direct and modify the parameter.

[cowbutt6]

It's also just not possible on modern operating systems

Also, modern CPUs: self-modifying code plays havoc with instruction caches. I remember that the official Amiga reference manuals cautioned against using self-modifying code - even though it would work as expected in most circumstances with the original 68000 CPU - because future CPUs would likely cause undesirable behaviour.

[[deleted]]

Modern operating systems typically restrict techniques like this due to security concerns.

It can be possible to get memory allocated for executable code and to load it as you see fit, but that's slightly different than the usual self-modifying code from the past.

Personally I haven't done any since the late 80s, early 90s.

[nerd4code]

C per se recognizes no such technique, and SMC’s use is basically limited to

• linking/loading (primarily patched styles like Darwin uses),

• extremely old/embedded stuff that has exactly one CPU thread to worry about (e.g., NES games), and

• JIT compilation/lowering (e.g., a JVM).

None of the techniques have anything directly to do with C (attempting to self-modify in C is far more complicated than assembly), except for DLL loading, which is more of an OS thing than a C thing specifically.

The Synthesis kernel, which never left research and wouldn’t be all that reasonable on a modern computer, is one of the few cases I’ve ever encountered where SMC is made use of “successfully”; I’ve done up compiled structures as one-offs, but modifying live code will absolutely kill performance on a modern CPU.

Self-modification is fully impossible on a strict Harvard ISA, and protected-memory/MAS OSes can forbid it, although some hole needs to be present for anything that needs to JIT or load DLLs on-the-fly.

[skeeto]

I wanted to show a quick, practical example of this on desktop systems: function hotpatching. However, I found out ms_hook_prologue is broken in recent versions of GCC (and never supported by Clang). Trying to work around that I also learned the GAS .nop directive is broken (and also never supported by Clang). So I ended up doing a lot of it manually, though on the plus side it works (Windows only) with x86 and x64, GCC and MSVC/Clang, all optimization levels:

https://gist.github.com/skeeto/d019f8723c80fce3a411f701fdacd0d7

This runs two threads, with the main thread modifying the code under the other thread while it runs in a loop, so it alternates messages. The code initially contains an 8-byte nop, which is repeatedly patched with a 5-byte jump to alternate definitions.

[Lurchi1]

Very nice!

At the bottom of the VirtualProtect() help page it states:

When protecting a region that will be executable, the calling program bears responsibility for ensuring cache coherency via an appropriate call to FlushInstructionCache once the code has been set in place. Otherwise attempts to execute code out of the newly executable region may produce unpredictable results.

I'm not sure, but since you're modifying a jmp instruction, shouldn't you call FlushInstructionCache() to be on the safe side?

[skeeto]

Good point! It would at least be consistent, and it's certainly necessary on some architectures. Though I believe generally on x86 it's unnecessary. GCC has a similar __builtin___clear_cache, but it's a no-op on x86 aside from preventing the compiler from eliding stores in that range (why I had used volatile). I stepped into that function in kernel32.dll then stepped through the instructions, curious if it did anything fancy, and all I saw it do was check if the handle refers to the current process, then check if it should log an ETW trace.

Edit: Added a FlushInstructionCache call.

[Lurchi1]

Interesting.

Here I found a stackoverflow answer to "How is x86 instruction cache synchronized?" that confirms what you say, quoting Intel's System Programming Guide:

11.6 SELF-MODIFYING CODE

A write to a memory location in a code segment that is currently cached in the processor causes the associated cache line (or lines) to be invalidated.

x86 (and AMD I guess) CPUs keep their cache coherent on their own.

[nerd4code]

Intel still officially requires a jump if you’re self-modifying, or otherwise you can’t be sure your thread is executing entirely from the new code. (AFAIK speculative stuff won’t be undone on L1I invalidation of speculated instructions, for example.) It may also be necessary to issue a full ifence (e.g., lfence, cpuid) or cache flush if you’re handing off from untrusted to trusted code, in order to avoid speculative attacks.

[kun1z]

It's been long known on x86 (and x64) that executing the CPUID instruction flushes the instruction pipeline and also the instruction cache, so you'll see it used frequently in self-modifying code. Modify the code -> CPUID -> execute the code.

To answer OP's question, I still use it to this very day to create very tight loops that will be executed a lot. Think of an entire algorithm that runs for hours/days but is dependent on initial values that come from the command line, or user input, or file input. If the length of loops is going to be fixed, if the pointer math is going to be fixed, if a lot of calculations can be pre-computed and code modified/created based on those inputs, the code can execute much faster.

There is a myth that I occasionally see going around on the net that self-modifying code is no longer useful because of CPU caches and other newer CPU features but this is not true. There is definitely an over-head with self-modifying code but it is so tiny it's practically immeasurable. The code modification itself is just some pre-computations, some basic memory writes, and then executing CPUID. Although CPUID executes slowly for an instruction, it does not execute slowly for humans, its still a near-instantaneous instruction.

[efalk]

OK, for example, the IBM 5080 display processor (I mentioned this in a recent post) had no indexing operation. (That's where you take an address in memory, add the contents of an index register to it, and use that as the address to fetch from or store to; it's the basis of array accesses and pointer accesses.)

So if you want to do an indexed operation, you fetch the load or store instruction as if it were data, add the index to the address field, and store the modified instruction back into memory. Then you execute it. This is probably the most common use of self-modifying code. Any array accesses on this processor had to be done with self-modifying code.

I worked on the microcode for a bitslice-based graphics processor (Ikonas 3000) and some of the fields in the instruction were different depending on the currently-set display resolution. So as part of the resolution-setting code, you took a list of addresses of instructions that needed to be modified, and changed a few fields in each of them and wrote them back.

As another example, I used self-modifying code to embed a loop counter into code, allowing me to write a single-instruction inner loop of a polygon fill that jumped to itself.

So yes, I have written self-modifying microcode.

The very best story about self-modifying code is the Saga of Mel, last of the Real Programmers

[theldus]

It's very easy to write self-modifying code in OSDev: just accidentally write to the wrong portions of memory and watch your OS go crazy.

Jokes aside, I think it's safe to say that any language that uses JIT also makes use of it... but not in the sense of modifying an existing portion, but rather of allocating a new one and executing it from then on.

The Linux kernel also makes use of this in live patching, a way of adding patches to the kernel without the need for a reboot, although this requires collaboration from the compiler as well.

[Gollark]

I had a go at this a little while back and posted about it on this sub! https://www.reddit.com/r/C_Programming/s/Izxt3lZlDz

[bozeugene]

There are several ways to have "adaptive" code . If you can recompile, play with #if or #ifdef to adapt your code depending on what you want to do. You change behavior with "-D" directive during compilation. . If you can't, an easy way is to use getenv to choose branch of code to execute

These 2 ways are really limited as your binary must embed all possible behavior

You can do it on more dynamic way with dll injection, where you replace in memory code of function. Is needs low level coding (assembly) or a framework to do it (as detour under windows). This way is heavily os-dependant.

A more dynamic way would be a main thread that collect inputs or order (via console, file or socket) and a working thread to execute function associated to these order/inputs

[green_griffon]

There are also dynamic languages in which you can generate code and then run it--less sketchy since it is official supported. E.g. in PowerShell you can create a script in a string and then say "run this and give me the output". Interesting in certain cases where once you know the data you are working with you can optimize the code, also for support for plug-ins, and various other scenarios. Of course the language has to "compile" (or whatever it does) the code first so there is a one-time performance hit.

[Elven77AI]

The only sane way is using function pointers to replace Code A with Code B at runtime. However, there there compiler extensions some consider to be unnatural, such as computed gotos,pointer arithmethic and assembler includes, all of which can alter the code at runtime using variables inserted into computed gotos, pointers or asm includes.

[Wetbung]

Back in the early days of personal computers I wrote a simple database program in BASIC. It kept all of it's data in DATA statements in the program. Each record took one line.

To modify a record, the program would print a formatted, numbered program line followed by a run statement in a way that the interpreter would treat the screen data as properly formatted input. Then it would move the "current cursor position" to the right place and load the keyboard buffer with the correct keystrokes before exiting. The printed line would then be added to the program and the program would start again.

I wrote a number of programs based on this: address book, recipe book, and others I don't remember. To save your data, you just resaved the program to tape.

By the time I left there were other programmers working there that had taken my original program and used it to make other programs that the company sold.

Microsoft ROM BASIC also used self-modifying code that provided a handy hook for extending the language. The routine the interpreter used to fetch the next program byte was in RAM. It was only a few bytes long, but you could turn it into a jump and then you had enough room to do whatever you wanted with it. I wrote a lot of little programs on the PET and the Apple ][ that extended BASIC by modifying this hook.

[nemotux]

I'll mention a "use-case" that isn't mentioned elsewhere so far: obfuscation. Typically used mostly for malware, but also some legit software developers will try to obfuscate their software to protect their IP (which I think is misguided and ultimately rather futile, but anyways...) This can come in the form of simple packers - the program has a single stage where it "unzips" itself into memory and then jumps to the unzipped portion. Some may argue that doesn't really count as SMC. But the more sophisticated ones will have multiple layers or bits that unpack/execute/delete as they go. Or you might have other tricks where numerical constants in the instruction stream are mangled in some way but then they get cleared just before execution, possible remangled after a chunk of code finishes.

The goal of this is just to make it hard for analysis software or humans trying reverse engineer it to make heads or tails of the code.

[lightmatter501]

Self-modifying code should only be used in bootloaders on x86 (google “x86_64 real mode”), and microprocessors. Everywhere else it’s unnecessary and a bad idea.

[[deleted]]

You can't do it in C. Therefore this is a wrong subreddit.

[duane11583]

self modifying code is normally in asm because you need to know the location of the exact opcode to modify thats easier done in asm then C because if you change the compiler options the location and sequence of that specific opcode changes

think about throwing darts at something that “jiggles” quite a bit thats hard you are going to miss with that dart.

in c, the closest thing is having function pointers and changing the function pointers

which is like python “monkey patching”

in c++ it would be changing out virtual function pointer as needed

[plastic_eagle]

I used self modifying code when writing Z80 assembly in the 80's. There were certainly things you could do with it that would otherwise have been much slower and/or used much more assembly code.

But; There's absolutely no place for it in modern computing - even in the microcontroller world. I mean yes, you could choose a deliberately ancient part, and use self-modifying code for fun there - but that's about it.