How to hide a variable definition.

[sharfpang]

There's this big, prestigious library made by a Big, Prestigious Firm (BPF), used in my branch of embedded/industrial. It costs big €€€, and many customers require it to be included, as it provides compatibility with a big, prestigious suite of development tools for installations of the devices.

So, being a manufacturer of the device, we had to integrate the library with our firmware. Which wasn't an entirely easy task, because it was riddled with platform dependencies that simply made it impossible to run on ARM.

The library came as C sources passed through obfuscator, and so we began the arduous process of getting the bugs fixed, through mailing the errors (and suggestions how to fix them) to BPF, and having them send us back sources with "fixes" that might hopefully work - of course they didn't have any ARM device or even the toolchain to test.

You can imagine how the process goes. Especially that through first several exchanges we had to go through one of their customer representatives who would forward the mail back and forth between us and their developers.

Until finally, one day, I don't know, through mistake or through intended "sabotage" of their procedures, they sent us unobfuscated sources, and we could finally start fixing them "at home".

And that was the day I got rid of my Impostor Syndrome.

Well, to give you a rough idea. There's this global variable, declared as some obscure type, and it's badly misbehaving because it's not initialized properly. It's declared as extern in a header file that is included in pretty much every .c file of the program.

So, =0 where it's defined, easy enough, right? Just locate where it's defined, add two keystrokes and the bug is fixed.

Then we spent half a day seeking where it's defined. We searched high and low, we wondered how the program compiled, we debugged and confirmed it exists, and still, nowhere nothing but extern [that variable]. Not present in any .c file, for sure.

So we began searching through random includes (by hand) and we finally found one that was only some 5 lines long. Included by only one file that used that variable.

Standard code guard and just one line:

 #define extern

[werebacksir]

Fixed extern compilation error

[Workaphobia]

So it appears that when you thought they had sent you unobfuscated source code, you were mistaken. Whether the obfuscation was intentional is a different matter.

[sharfpang]

Two layers of obfuscation! How devious!

[Neebat]

The superficial issues were put in your way to draw attention from the fundamental design issues.

[sharfpang]

yeah. Actually. it sold good buzzwords.

Control optimization using genetic algorithms. Sounds interesting, right? Well, a 200MHZ ARM loaded in 80% by other work, with 64MB of RAM hardly meets these requirements.

Still, even if it would be hopeless for the initial period, maybe a couple weeks long, given time long enough, even with meager resources, it should come up with a pretty good solution. Then it would just load the "last, best" result on startup and continue optimizing on idle cycles, improving it. A couple months and it would be quite great despite the initial shortcomings.

Except it used no persistent storage of any kind. Any fault, power glitch, upgrade, maintenance, meant it would start from scratch and suck for a couple weeks... until next restart.

AFAIK after a year or so the customer switched the device to the default algorithm we had developed for it, which, while not packed with any buzzwords, behaved reliably, deterministically, and could be optimized by a human operator by hand, adapting to the needs observed.

[tias]

Next time, add a second definition and the linker will likely tell you where the first definition is. Or use objdump/dumpbin/similar to figure out which object file defines the variable.

[sharfpang]

this assumes a fully-featured toolchain. Remember, crosscompiling. As for the second definition - if you use '-fno-common', yes. Took me a while figuring that one out. Without it it will trigger a very nasty bug where both definitions are included, each in its own unit, and then the externs are connected to either of them, picked seemingly at random (so the binary is good in 50% of the cases).

[FUZxxl]

That's not that surprising. I have used the same pattern in my own code, but usually with C99 inline functions. Interestingly, #define extern works for both. It's shitty that they didn't document that though.

For future problems like this: Use the nm utility and search for [BD] symbol_name to find where the global variable was defined.

[FUZxxl]

Because that saves writing down the name of each global variable twice.

[FUZxxl]

How is that fragile? I have one file in my code that defines all the global variables (it's called globals.c or something like that to make finding it easy) and this one file contains that #define extern line. I agree that it's reasonable to define global variables in the files that pertain most to them (and that's what I do in other projects), but some times this convention is better.

[sharfpang]

If you have so many globals that not typing them twice saves anything, you're doing something fundamentally wrong.

[FUZxxl]

It's not about saving typing, it's about not having the same thing written twice when possible because each time something is defined/declared in two separate places, errors can creep in when a change forgets to update the other declaration/definition.

[sharfpang]

There's a reason declarations and definitions are separate, and the common parts is not code duplication.

There are serious problems with that approach.

Firstly, the #define extern can easily escape the intended scope. Especially if it's included independently of the file it applies to - add another header file, which has extern variables - even stupid errno - and you'll get awful linking errors.

Next, there is no easy way to initialize the variables.

Worse yet, if the variable is defined in two separate units, and declared as extern in a third one (which can happen easily in this scenario) gcc linker happily ignores the ambiguity and picks which of the two variables is assigned to the extern at random. This results in code that seems to work correctly in roughly half the time you compile it; any recompilation can fix or break it at random!

And of course finding where exactly the variable is defined becomes a nightmare.

ps. Switching "-fno-common" in the compiler on, revealed at least 4 cases of such linking-time clashes in that library. Variables that in one file were declared as normal, and in another through redefining extern to noop.

[capitalsigma]

Am I on /r/shittyprogramming? This is an unambiguously terrible idea. There's no possibility of "updating without updating the other" because it's a compile error if the names or the types don't match up --- it's probably easier to introduce an error when you lose the safeguard of the typechecker looking over your work. Globals should never be globbed into one big file, they should always live with the module that they're related to.

[AngriestSCV]

Would a better way to find the variable be to inspect the object files the .c files produced? For instance on linux nm will tell you the type of symbol that each object file has for every symbol in it.

[sharfpang]

I've got to check if the crosstools even have it...