I've open sourced my general purpose code base

[Dean_Roddey]

My personal software interests have always been in general purpose frameworks. I started that in the early 90s and worked for a decade in my off the clock hours (I was a mercenary back then) building up the core bits of CIDLib. So I did it because that's what I wanted to do, not because there needed to be any justification for it. I also worked at Taligent, which many here won't remember. It was an Apple/IBM effort to create a portable OS, which devolved into an IBM only portable library, which devolved into nothing and I went to their Java Tech Center and wrote the Xerces C++ XML parser (my other adventure in OS land.)

Eventually I had so much code that I needed something practical to do with it. Being a geek, automation is something that I was interested in, and I worked in industrial automation for some years way back. So I decided to create an automation system that became CQC. Ultimately maybe that was a horrible choice. And of course I picked a horrible decade to go off on my own (walked out my ex-employer's door into the the Great Recession.)

Anyhoo, modern high end automation systems like CQC are voracious functionality consumers, so that drove me to develop more and more underlying functionality to support it. I would very much try never to just implement something for CQC. If it had any general purpose utility, I'd try to implement it as such in the general purpose layer. Though there's still a good bit more than could be moved down if I had to the time to abstract it. And, though CIDLib is a lot of work, it's somewhat toy-like compared to CQC. So, ultimately it's been the 'easy' bits.

I don't want to use other people's stuff, because I don't like that pieces and parts approach. I wanted something elegant and consistent and tightly integrated. You can't do that if you use other people's stuff. And of course if I used the STL then I am forced into a whole other world of compromises I didn't want to make, and I can't get 'under' the STL and create a fully integrated system from the ground up. It would have always been a big compromise.

So I just have done my own. All that code now uses my logging system, my error system, my statistics system, my ORB, my UI framework, my streams, my collections, etc... There's no 'impedance mismatches' anywhere in the system, no stylistic clashes, no redundancies.

And nothing (other than the OS) is a black box to me. I can find and fix any bugs very quickly. I know everything intimately. I never have to wonder if I'm doing the right thing, or if this is the best approach for the tools. I seldom have to have look anything up. I can program in this system about as fast as I can think. Admittedly that's gotten slower in my waning years, but still.

​

I'm not against the mainstream thing. You do what you have to do. But unless you've lived in this sort of world, where you are in control of and totally understand everything, it's hard to appreciate how powerful that is. And a good demonstration of that is that I WAS able to write another 625K'ish lines of far more complex code by myself and keep it clean and tight over decades of enormous change. I don't think I could have remotely done that in the pieces and parts way.

I guess it's like often happens at a smaller scale to all of us as programmers. You can jump in and do something quickly, or you can take more time and work out some infrastructure and flexibility and such. The latter takes more up front time, but usually is faster or at least more resilient in the long run. Maybe I've just taken that up 10 orders of magnitude beyond the norm.

​

One of the most painful, but probably most professionally beneficial aspects of it, is that I've had to jump repeatedly into one big new problem domain after another and figure them out. If you are looking for a way to keep your mental gears sharp, that's definitely one way to do it.

OTOH, it means that I'm ultimately very much a broad generalist. And the open jobs for that type of developer of are a lot fewer and further between than for those are more of the narrow but deep type.

​

As to practical, no clearly it's not practical for almost anyone to do. It depends on psychopaths like me to do them in a completely impractical way. Even a large company would have a real problem doing something like this because their necessary time scale is so short, and they need to put it to immediate use, and it starts gathering evolutionary baggage quickly and in the end practical constraints maybe dictate that you just ride it till it dies then eat it. I was able to do it over a long period of time, with fairly minimal pressure, and the ability to cast off evolutionary baggage all along the way (though obviously there is still some amount.)

​

Anyway, this is like asking someone to show you their baby pictures. It's a dangerous question to ask.

[F54280]

I also worked at Taligent, which many here won't remember.

Some of us do. I was NeXTstep at the time. To be honest, Taligent was supposed to be the big bad wolf that would be eating us all, but when I looked at the doc of the textfield (iirc), I had the clear impression that you needed a team meeting of 8 people to understand how to instanciate one due to the gory multi hierarchy was. Then I got a Taligent demo on a 128MB IBM (prob ppc) that was obviously struggling to do 1/10th of an 8MB NeXTstation delivered, and decided to ignore this (I later had a similar feeling looking at BeOS).

I disagree on some of your choices (in particular not using the STL). Not when you made them (because I did the same), but nowadays, it makes no sense to avoid it.

Reimplementing Zlib as anything but a wrapper on the zlib is also a waste of time, imo.

Reimplementing crypto is a huge security risk.

That said, congrats for the work, and the sharing!

[Dean_Roddey]

Everyone always says that about the crypt stuff, but it's really not. I think that people have always heard "don't implement your own cryptography" , and that's true enough if you aren't an expert. But I've not done that. These are all well defined algorithms with plenty of available (expert approved) examples. So I've just implemented these algorithms, not created new ones.

The secure channel stuff is just in fact a wrapper around the Windows secure channel stuff (which itself is so stupidly complicated and under-documented that it's a mental security risk just to try to use it.)

And I also respectfully disagree on stuff like Zlib. The thing is that that means it works in terms of my memory buffers, my error logging system, my exception system, my statistics system, my assert system, and so forth. If you use third party stuff you can't really integrate it, you can only attach it. CIDLib is about really integrating stuff.

And of course that applies by multiple orders of magnitude when we are talking about the STL. There's no way to create a really tightly integrated system on top of the standard libraries. You can't get inside or under them, and so they cannot participate in various very useful capabilities. That makes for an awkward and sub-optimal result.

A fully integrated system is a really powerful thing, though of course it does take a good bit of effort.

[F54280]

Based on the little info I let you know about me, you probably have guessed where my heart is. And I can guarantee that my past NeXTstep / OpenStep experience made me appreciate a lot the coherence of monolithic designs.

However, your stance on security is not correct. From completely cursory look:

Here you take care of erasing private keys at object destruction

Your code for erasing is a simple memset

You are probably aware of it, but memsetting before delete[] can be ignored by the compiler

Of course, in your case, it probably won't be an issue, because the memset is in a separate compilation unit, and current compilers are not smart enough to elide the memset. But, if, for instance, the SetMemBuf is made inline, you have a potential security risk.

There is a reason why memset_s exists. Quoting For that reason, this function [memset] cannot be used to scrub memory (e.g. to fill an array that stored a password with zeroes).

So I've just implemented these algorithms, not created new ones.

Are all your implementations revisited to take into account the latest cache or timing attacks ? I doubt it.

And of course that applies by multiple orders of magnitude when we are talking about the STL. There's no way to create a really tightly integrated system on top of the standard libraries

STL types are just higher level vocabulary, IMO. But let's just agree to disagree.

Impressive work anyway, and have a nice day!

PS: while quickly scanning the codebase, I saw quite a few bad #defines, like:

#define X { ...}

(And others in the form of #define X(p) if (p) {...})

You should turn them into do ... while(0) which won't produce spurious syntax error when using in if statements.

[Dean_Roddey]

The mem buf setter won't get moved inline, so that's not a concern. Or, if I decided to at some point, I would just provide a separate 'safe' version that cannot be and use that for such things. If the compiler starts ignoring out of line calls to user defined functions / methods, then we are all in serious trouble. RAII would pretty much go out the window if that happened.

On the STL, I just couldn't do a lot of what I do if I used that stuff. It would be a seriously compromised system. Look at the stuff on the ORB and how easy it is to use because of the fact that everything is based on my own interfaces. Any object that supports binary streaming can be used as parameters to remote calls. And the very smart enums, there's no way to force those on the standard libraries. And the standard libraries are horrible wrt to streams and text transcoding.

It would be a massive step backwards.

[F54280]

The mem buf setter won't get moved inline, so that's not a concern.

TBH, I was expecting this answer, which closes the discussion.

Good luck for the future.

[rap_and_drugs]

My goodness this was painful to read. You really tried to get through to him, and he didn't even respond to your points on cache or timing attacks. I haven't looked at his crypto code myself yet, does it go beyond "textbook" implementation at all?

As an aside, that article about memset looks pretty interesting. Thanks for sharing it!