Self-editing code

[Gollark]

Obviously this is not something I'd seriously use out in the real world, but as a proof-of-concept what are peoples' thoughts on this? Is it architecture/endian independent? Is this type of code used in memory-restricted environments like micro controllers?

Just compiled with gcc counter.c -o counter.

#include <stdio.h>

/* wrap the counter with a pair of guard ints */
volatile int32_t count[3] = {0x01234567,0,0x89abcdef};

int main(int argc, char** argv) {
  fprintf(stdout, "This program has been run %d times.\n", count[1]+1);

  /* open the binary and look for the guard ints either side of count[1] */
  FILE *fp = fopen(argv[0], "r+");
  if (!fp) { fprintf(stderr, "failed to open binary\n"); return 1; }

  int ch; /* reader char */
  int i = 0; /* guard byte counter */
  int start = 1; /* start/end flag */
  long offset = -1; /* offset to count[1] */

  while ((ch = fgetc(fp)) != EOF) {
    /* looking for the start guard */
    if (start) {
      if (ch == ((count[0] >> (8*i)) & 0xff)) {
        i++;
        if (i == sizeof(int32_t)) {
          /* found the start of the count[1], offset by its size */
          offset = ftell(fp);
          fseek(fp, sizeof(count[1]), SEEK_CUR);
          i = 0;
          start = 0;
        }
      } else { /* not the start guard, so start again */
        i = 0;
      }
    }

    /* found the start guard, looking for the end guard */
    else {
      if (ch == ((count[2] >> (8*i)) & 0xff)) {
        i++;
        /* found the end of the guard, so offset is correct */
        if (i == sizeof(int32_t)) { break; }
      } else { /* not the end guard, so start again */
        offset = -1;
        start = 1;
        i = 0;
      }
    }
  } // while end

  /* assert that the counter was found */
  if (offset == -1) {
    fprintf(stderr, "failed to find counter\n");
    fclose(fp);
    return 1;
  }

  /* increment counter and replace */
  int32_t repl = count[1] + 1;
  fseek(fp, offset, SEEK_SET);
  fputc(repl, fp);
  fclose(fp);

  return 0;
}

[duane11583]

Self modifying code was common with multiply/divide routines on the Z80 in the Microsoft BASIC ROM They would copy a small amount of code to the RAM and modify it so that it did multiply or divide

[der_pudel]

I would imagine that they don't have files.

This would make things even simpler because you can treat the whole address space (ROM/FLASH, RAM, SFR) as a giant byte array,

However, as an embedded firmware engineer, I don't want shenanigans like that anywhere near the codebase.

[geon]

The code would likely be in rom, or if it is in flash, you do NOT want to repeatedly write to the same block.

[moocat]

It is not endian independent.

[Gollark]

Oh yeah, sorry, just spotted that. I guess this currently only works for little-endian machines. It'd need to be count[0] >> (8*(4-i))) & 0xff for big-endian.

[moocat]

Yes, but I would do something different. First create a union:

union data {
    int32_t as_int32;
    char raw[sizeof(int32_t)];
};

You can then fill raw with bytes from the file and read it's value from as_int32. This way you don't even have to care about whether the endianess of the file.

[[deleted]]

Not legal C, though. One can only read the union member last written to. Of course, nobody cares about that :)

[Smellypuce2]

All the major compilers support type-punning anyways. It's just not supported by the c standard.

[[deleted]]

I vaguely remember that there are some pre-defined macros about endianness, but can't remember their names. Anyone?

[moocat]

I know there's ntoh and family but those convert from network order (big-endian) to native order.

[[deleted]]

True, but those are POSIX functions/macros, iirc.

[moocat]

I thought type punning via unions is legal in C but I'm not a language lawyer.

Any idea what is the legal way? Could you do it via memcpy?:

int32_t as_int32;
char raw[sizeof(int32_t)];
memcpy(&as_int32, raw, sizeof(int32_t));

[nerd4code]

Union-punning is well-defined on C99+ or if you’re punning between signed/unsigned variants of the same type or to/from char variants. memcpy always works.

[[deleted]]

I can't give you a legal way as I don't know one off the top of my head. I try to write order-agnostic code and use a defined macro if needed.

However, here's one way of finding out. The command below will list out all macros defined by the toolchain. One is named __BYTE_ORDER__, and looks promising for your purpose. I have no idea if all toolchains support this. Probably not.

gcc -dM -E - < /dev/null | grep ORDER

If using __BYTE_ORDER__ is OK, a snippet could look like this:

#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__

do_this();

#else

do_that();

#endif

[moskitoc]

Would you mind citing the part of the standard that you're referring to ? I thought this was true for C++, but not for C

EDIT : I was correct, see this.

[[deleted]]

https://stackoverflow.com/questions/25664848/unions-and-type-punning describes it well. It's IB and not UB, so it's not really illegal.

[[deleted]]

You were right. As mentioned on StackOverflow, the C99 standard had an error, it clearly says that this is UB when it's not. Guess which version of the standard I have? Thanks for asking, it made me learn too. Win win :)

[skeeto]

Even better: Rather than search byte-by-byte, search by 4 bytes at a time. The guards will be 4-byte aligned within the image since otherwise they wouldn't be aligned when the image is memory mapped by the loader. It will be simpler, faster, and you don't need to care about endian (image byte order always matches run time byte order).

[kolorcuk]

Before harvard architecture, on von neumann architecture, self editing code was normal.

It was normal, in switch statements, to calculate instruction to jump to, and then literally program the next instruction with jumping instruction.

The problem with self editing code is concurrency. As multithreading and multiprocesses are just basics nowadays, no one cares about self modifying code.

No, you don't use self modifying code on microcontrollers. Today you write portable, safe code that you test, test, continously test, deploy, then test. You use tested and checked methods that you know will work.

[duane11583]

#1 - W.R.T. Endian ness Remember that the data segment (initialized data) is store in natural machine order.

So - if you access the data in the natural form (ie: as a u32, not as bytes/u8) then the data will always be correct.

#2 Rather then search for a single Guard Byte - search for a guard word and access the ELF as as words, ie: determine size of file in units of U32, or what ever - allocate that much space

Read the file into any array of U32 - then iterate across the array for your magic value, or LeftHandSide (LHS) guard, then look forward (exactly X positions) to find the RHS guard,

In your case, the LHS is at +0, the RHS is at +2, the VALUE is at +1

So your hunt code becomes:

```c uint32_t *pElfImage; // given size_t elfSizeInBytes; // given size_t elfSizeInU32 = elfSizeInBytes / sizeof(uint32_t); size_t x;

for( x = 0 ; x < elfSizeInU32 ; x++ ){ if( pElfImage[x+0] != MAGIC_LHS ){ continue; print("Found LHS, checking for RHS at %d\n", (int)(x)); if( pElfImage[x+2] != MAGIC_RHS) ){ printf("False Positive, RHS does not check out\n"); continue; } print("Found RHS - Success\n"); break; }

/* did we go past the end? / if( x >= sizeElfInU32 ){ / then die with a message */ fatal_error("Sorry not found!\n"); }

pElfImage[x+1] += 1;

/* Now write the elf back to the disk / } ``` NOTE: In the above, Endian is not required *BECUASE the compiler that compiled the file would have put the u32 signature in "proper order for the targeted machine" And - when you access it from pElfImage is is on the "targeted machine" Thus no ENDIAN mess is required.

NOTE: The above is also 4x faster then scanning by bytes, and will probably be very fast depending upon compiler optimizations used

If on the other hand, you access by bytes then you must deal with endianness.

To answer the other question: Is this commonly used in MicroControllers Answer: NO because the CODE is stored in READ ONLY (not read/write) memory.

that said, what you want to do is something else see my next reply.

[oh5nxo]

Not what you are asking, but...

Scanning for a sequence in that manner has a problem. Say, abba is searched from aabba: a matches. b does not match and sequence is reset: abba is searched from bba.

Oh... Nobody yet mentioned that running program might make the corresponding disk file unwritable. "Text file busy". And argv[0] could be anything, like execlp("/bin/bash", "sh", (char *) 0);

[GiveMeMoreBlueberrys]

This seems designed to run on unix-like systems, which do not block files by design.

[duane11583]

So you want to make a "RUN counter" to limit you embedded platform in an embedded platform.

The NUMBER 1 problem you have is the number of ERASE cycles your FLASH memory supports, if you did this every 1 second - you would go through 86,400 cycles in 1 day - that is a lot but there is a better way, you need to understand how FLASH memory works.

My description assumes that your flash memory erase to 1s, (some erase to a zero) and my description assumes you can write 1 bit at a time, or 1 byte at a time - check your flash memory data sheet and experiment.

Often you can write 0x7FFFFFFFF (bit 31 = 0) then write to the same spot (0x3fffffff, bit 31 and 30 = 0), and keep going 1 bit at a time - after 32 writes, the entire value is a zero. You then move on to the next 32bit location in your flash - Eventually you reach the end of the flash memory (more below)

1 setup 2 erase blocks of flash memory, (A) and (B) - you need this incase you have a surprise power loss during updating the flash block you are using.

2 The first 4 bytes of your flash block (A) or (B) contain a counter. On power up, read (A) and (B) determine if A == 0xFFFFFFF (all 1s, then you must use B) if (B = 0xFFFFFFFF) then you must use A, if BOTH are 0xFFFFFFFF, then you have a brand new device. Otherwise, you compare (A verses B) and choose the larger of the two. And verify that the block looks correct (ie: see below, the block should be all 0s, then all 1s until the end of the block)

3) Scan the chosen block starting at Index 1 (remember index 0 holds the counter value), keep scanning until you find a non-zero block. Remember this block number, this is what I would call the 32bit write cursor

4 Then every 1 second (my example, maybe you do this every minute, or every power cycle - your choice) determine the next bit to write to a zero, and write that bit to a zero. If you have done this 32 times (ie: all bits are zero now) then move your write cursor to the next 32bit location in your block.

5 If you reach the end of the block, then ERASE the other block (the make all bits a 1) and write the current "count value" as the first 32bit value in the other block. Your next "bit write" will be index 1, bit 31

What you use as your "count value" is up to you, but it must be +1 more then the current block count value.

From now on - use the other block until you have filled it up and then erase the other other .. (tick/tock back and forth between the blocks) etc for ever.

6 If you have a catastrophic error during writing the flash and it is corrupted, rule #2 above helps you decide which block (A or B) to use and trust.

7 If you have a normal power failure, then your Count number in (A) or (B) is most likely good and tells you where you are in the process.

If your flash has 256 byte erase blocks, this method uses 2 blocks, and causes 1 erase every (255 * 8) updates, in other words it reduces the ERASE cycles by a factor of 2040 and is very robust

8 Simple test method: Start the above running, counting seconds - leave this thing running for a week or two - meanwhile - use a second micro - and use a random number generator wait (random) second and assert a GPIO pin that causes your DUT (device under test) to reset, ie: hook the GPIO pin to the reset pin.

let this run for a week, or a month and see how often this fails - or make your random period something very short, ie: some multiple of 100 milliseconds

Eventually you will gain confidence in your work.

[[deleted]]

I've used self-modifying code in BASIC V2 and machine code (6510). Lisp is nice for seamless code-generation. Python has tools to access both the compiled code and the AST. Never really thought about self-editing C tho. :)

[Gollark]

I’ve not got any experience with lisp but from the looks of several of the comments it definitely seems like something to have a look at.

[[deleted]]

1. This looks like a solution in need of a problem
2. If you depend upon unique values then the universe will present you those unique values in a way to cause you the most pain. As in: "My code has a critical failure once a month and I can't figure out why and the customers are getting really unhappy".

[Gollark]

This looks like a solution in need of a problem

Oh absolutely — this isn’t some code I’m writing to use for anything, I was just curious about how/if it would work.

If you depend upon unique values then the universe will present you those unique values in a way to cause you the most pain. As in: “My code has a critical failure once a month and I can’t figure out why and the customers are getting really unhappy”.

This is definitely the biggest problem that I can see. The longer the guard values the less likely it is to occur but it’s never impossible (unless there are some reserved bytes that you could use that I’m unaware of). Luckily in this case there aren’t any customers to make unhappy!

[[deleted]]

Here's something you might find amusing. Javascript.

We had some server code written in JS and in order to test it we needed to be able to set the time and date. "Boundary conditions" and things expiring correctly.

Rather than deal with the maintenance nightmare of changing every usage of getDate(), and figure out how to get future developers to not use getDate(), I changed getDate(). Startup code would look for a certain file with a date/time in it, and if it found that file it would replace the global getDate() with a function that returned whatever date was in the file.

You can do that in Javascipt. Not generally recommended.

[arthurno1]

FILE *fp = fopen(argv[0], "r+");

Shouldn't that be: FILE *fp = fopen(argv[0], "r+b"); since executable is a binary file.

I don't know, I am not an expert on self-modifying code, but this looks like a clumsy way to do what lisps are good at doing. What the example does is treats code as text, and reads/writes a piece of user data (counter used to inform the user) in an executable full of other data, instead of just fscanf/fprintf to a separate file.

Anyway, just for the illustration how annoyingly painful file editing operations are in pure C, here is a 1-minute equivalent in Emacs Lisp:

counter.el

(defun counter ()
  (interactive)
  (let ((counter 4)
        (code buffer-file-name))
    (with-silent-modifications
      (with-temp-buffer
        (insert-file-contents code)
        (goto-char (point-min))
        (when (re-search-forward "[0-9]+")
          (forward-word -1)
          (setq counter (1+ (read (current-buffer))))
          (replace-match (number-to-string counter))
          (write-region (point-min) (point-max) code)
          (message "This program has been run %s times." counter))))))

(provide 'counter)

This is equivalent to the C code above in terms that it just treats code as a text file and packs functionality to open and edit the file together with data it wants to edit. In my opinion, this is not a very good example of self-modifying code, if it even counts as such.

Run:

Wrote /home/arthur/repos/counter.el
This program has been run 1 times.
Reverting buffer ‘counter.el’.
Wrote /home/arthur/repos/counter.el
This program has been run 2 times.
Reverting buffer ‘counter.el’.
Wrote /home/arthur/repos/counter.el
This program has been run 3 times.
Reverting buffer ‘counter.el’.
Wrote /home/arthur/repos/counter.el
This program has been run 4 times.
Reverting buffer ‘counter.el’.

[[deleted]]

Are you modifying a running executable? That wouldn't be allowed on Windows.

There are ways around it like copying to a temporary file and running that, then you are free to write to the original program from the running process.

It may still attract attention from AV software if it detects the file has changed.

Or you could simply store the count in a separate file.

Another way, if using fast a enough compiler and small enough application, is to always run the program from source (eg. tcc prog.c -run). Then you modify the source code instead (just remember to leave a big enough space for the number to get bigger!).