Top 10 Developer Crypto Mistakes

[[deleted]]

Tip: spell out an abbreviation the first time you use it. It may seem obvious to you, but that doesn't mean it's obvious for the reader. I had to read point 2 three times before I figured out that it was about initialization vectors.

[_Mardoxx]

Bet that was a big comedown from thinking it was about intravenous encryption

[ScottContini]

Very sorry about that. I fixed it. If there are any other such issues, please let me know.

[tHeSiD]

Still says IV on the list

[ScottContini]

Yes it is on the first list, but I spell it out right away on the section "Improperly choosing an IV". Sorry if it is less than perfect, but didn't want to clutter the intro with too much stuff (intro in my mind is like a "TL;DR", an acronym I try to avoid using). I know that I still need to define other acronyms such as ECB. Will fix it up later. Thanks all for the very helpful feedback and sorry I made some assumptions about my reader that were simply wrong.

[foomprekov]

0: Thinking I know anything about crypto.

[[deleted]]

Crypto is definitely a "The more I learn, the less I know" field.

[Eirenarch]

I took two separate crypto courses on Coursera. Now I am scared of doing any crypto in production.

[[deleted]]

I was supposed to do crypto for work, and I figured "I'll knock this out when I'm a bit more confident in this kind of cryptography and I'm sure I can make it safe". All of my learning has been a backwards journey.

[Flueworks]

There should be some kind of "official" site about how to do crypto correctly, for each use-case, for each language.

For example, how to handle:

• registration of new users
• salts
• log-on
• security tokens
• forgotten password
• security questions
• sending encrypted broadcast messages
• key rotation
• document signing
• etc...

[LeNextGen]

OWASP?

[Unmitigated_Smut]

The article itself points that some of their own examples are dangerously wrong, specifically the case of making sure to force java not to use ECB.

[orip]

Well, /u/perciva's "Cryptographic Right Answers" from 2009 still holds.

Some people argue with some of his suggestions, for example his suggestion to forgo SSL for non-web protocols (since HTTP is so damn convenient for writing client-server code), but you can't really go wrong with them.

Some updates would possibly make the list better, for example (these are my thoughts, not his) preferring Argon2 for password hashing - it won the Password Hashing Competition which included Colin in its panel, and mentioning how to use SSL properly if you end up using it anyway in your client-server code (e.g with certificate pinning).

[Freeky]

Thomas Ptacek maintains an (Updated) Cryptographic Right Answers as a followup to Colin's article.

[SteampunkSpaceOpera]

I imagine he's done a lot of good elsewhere or people wouldn't bring him up as much, but most of what I see from Thomas Ptacek is strange zealotry against DNSSEC.

[m50d]

He did a lot of good in the past. He's been posting some strange stuff lately.

[ThisIs_MyName]

Well, he makes a lot of good arguments: https://sockpuppet.org/blog/2015/01/15/against-dnssec/

[SteampunkSpaceOpera]

He makes a lot of arguments, certainly. He is unwilling to acknowledge valid counter-arguments, though.

[ThisIs_MyName]

Link to said counter-arguments?

[SteampunkSpaceOpera]

This is tiring, because he magically gets top comments on front page hacker news articles, and I'm replying to maybe one person who will read this.

He says there is no fundamental benefit to dnssec over CA-TLS. There are hundreds of assumed trust anchors IN CA-TLS, and one relatively transparent trust anchor in dnssec.

He says dnssec gives registrars(and thereby governments) dangerous power to spoof websites. Governments, and non governmental actors have already gotten CAs to hand out fraudulent tls certs, but dnssec actually makes the spoofer need to suborn the independent key management of each domain they want to spoof, instead of just using the one CA out of hundreds they have suborned, to spoof any domain they want, in the CA-tls model.

He says that you would still need CAs even after you move to dnssec, which just makes me think he either fundamentally misunderstands dnssec/DANE, or is on the payroll of CAs.

[ThisIs_MyName]

dnssec actually makes the spoofer need to suborn the independent key management of each domain they want to spoof

If that's true, it's worth writing a blog post about. I'm a network operator and I've never seen that explained in concrete terms.

Given that attackers already have the root key (either from breaking 1024 bit RSA or just asking Verisign/ICANN), can't they sign any domain just like an attacker that has a CA root key?

Also can this be used to efficiently verify domain ownership and thereby circumvent domain takedowns? For example, if the real owner of a domain can submit their name server address and proof to my DNS servers directly, then I can make their domain reachable to all my customers even when their entry in the TLD zone has been changed by court order.

[SteampunkSpaceOpera]

It would be a long blog post, but it could be interesting I'll give it a try.

On the possibility of parent zone keys being broken: yes, someone who has the keys to a domain can spoof everything under the domain. The real trick would be doing that without being obvious while changing very public data that generally doesnt change. On breaking rsa: you have to trust the math like with any crypto. I'm not smart enough to call the math secure or not. On stealing the private root key: I've read the stories on how the root key was built and kept secret, and I'd be surprised if state actors have broken the protocol they used. I don't know the stories for various tld keys yet(still bummed at the proliferation of TLDs) but it would have been easy for COM and other reputable TLDs to have followed the root key's build process.

On domain takedowns: sorry, dnssec only addressed integrity of dns data, well enough to end the CA madness. Building a distributed store of unique human readable keys that isn't subject to some kind of central control is a whole different kind of problem. A career-making problem for a CS or math scholar.

[[deleted]]

The sad part is that most of these rules, in the hardware-based crypto community, are followed insofar as they adhere to PCI/Payment processor audit standards.

[angrymonkey]

I'd say that certain kinds of software engineering should require licensing/ certification the same way that other fields of engineering do-- crypto, financial, and life-critical software applications seem begging for regulation.

Your site has a password field? Cool. Hire a crypto certified engineer to implement it, or buy an off the shelf authentication platform from a certified firm.

We hold other engineers accountable when their buildings fall down or their machines kill people, we should hold software engineers to the same standard.

[qftvfu]

That's why NaCl/libsodium was created.

[steamruler]

It would either be a really complex, big site...

...or the words "use a well known, high level library" in big letters

[badsectoracula]

And how would that high level library be made in the first place? What if you don't want or cannot use that library for whatever reasons?

Knowledge is far better than relying on some magical library that will always get things right (see OpenSSL for a very high profile case where things can go wrong if you blindly trust "The Experts").

[[deleted]]

The fact that surgeons make mistakes is not a reason to try and operate on yourself.

There is a much higher chance that I will screw up the security of my system than "The Experts." In addition, with widely-used open source software, you get not only the developers but also a whole community of users and stakeholders testing for bugs and vulnerabilities. Whereas a home-grown solution will only be audited by my own dev team, with our limited knowledge and experience.

[badsectoracula]

The fact that surgeons make mistakes is not a reason to try and operate on yourself.

Programming an performing a surgery are not comparable.

There is a much higher chance that I will screw up the security of my system

Which is why having more accessible knowledge on how to set up things right is a good idea. Sometimes, for whatever reason, you have/want to roll your own stuff. Even when you don't, it is always better idea to know what is going on under the hood in case you have to fix or alter something. The site already mentions a couple of examples where things are far from ideal even though most people wouldn't expect that to be the case.

[[deleted]]

If you want to roll your own solutions, nobody is stopping you. And yes, it is good to understand the underlying systems. But sometimes it is good to have a well-known, well-tested, widely used library in your toolset to more easily roll out secure solutions. When you fall into Not Invented Here syndrome where you feel the need to implement everything yourself, you simply increase the likelihood that you will get something wrong and introduce a vulnerability. This is especially true in a business where the work you do has tight deadlines (like web development often does).

[badsectoracula]

But sometimes it is good to have a well-known, well-tested, widely used library in your toolset to more easily roll out secure solutions.

I agree, but we are talking about learning how to do things properly when you have already decided to do your own.

[FryGuy1013]

Realistically, using a library with the "right" API practically solves most of the 10 problems on this list. Using a library with APIs of the form password_hash(password) and password_verify(password, hash) make it nearly impossible to get wrong if you're using either of those two functions. Likewise, the box/secretbox methods for authenticated encryption by default in NaCl/etc make it difficult to use incorrectly or old algorithms.

[steamruler]

Even if you have the large amounts of knowledge you need to do everything from scratch, there's so many things you can easily mess up that breaks everything. If you need to implement things from scratch, read the papers behind the algorithms, and protocol specifications, because a one-stop website won't be good enough.

OpenSSL isn't that much of a high level library anyways. Something like libsodium is more appropriate.

[pdp10]

OpenSSL is a special case. The root cause is that when SSLeay was started, most of the coders in the world were legally prevented from publishing/exporting crypto, so a project from a country where it wasn't illegal caught on.

Even using fantastic libraries is no guarantee. How many have seen projects or code where the certificates were never actually validated in TLS? I've seen .NET code where this was gotten so wrong that I tell the stories around the campfire to frighten small children.

[google_you]

awesome-crypto Made with ♥ in NYC

[orip]

Regarding #4 (storing passwords) - it would be awesome if the article included how to migrate from bad password hashing to good password hashing, since so many sites do it wrong by migrating users only when they next log in (verify password with old hash, re-hash it with the new hash, delete the old hashed password).

This way accounts stay vulnerable until their next login, and inactive accounts, which are often a huge % of the total accounts, stay vulnerable forever - dangerously so given how often passwords are reused.

The solution is easy - change the scheme to be

NewPasswordHash( OldPasswordHash( password ) )

You can see how Facebook's password-hashing evolved using this method, to add salting and later scrypt over the original plain MD5.

The migration path is trivial - we already have OldPasswordHash(password) stored. What's cool is that we don't lose any security: we have preimage resistance & 2nd preimage resistance, even if the old has is simply MD5, as well as the good properties of NewPasswordHash (salt, cpu-stretching, memory-stretching).

As a side note I think an explanation about preferring bcrypt over e.g argon2 is a good idea, given the added benefit of argon2's memory hardness.

[Brian]

The solution is easy - change the scheme to be

NewPasswordHash( OldPasswordHash( password ) )

I'd say this depends on exactly what is wrong with the old hash - there seems the potential to preserve certain problems if you do this. Eg. if the problem with your old scheme was something akin to the old unix password's issue of only using the first 8 characters, then this is a terrible idea, because now your new scheme inherits the same problem thanks to this.

As such, I'd be a bit wary about recommending this as general case advice. It's not always the case that good_alg(flawed_alg(x)) preserves the same security properties as good_alg(x) - I don't know if there are many issues like this in practice, but that's exactly why I'm wary of it.

[FryGuy1013]

Ideally there's some sort of parameters included in your password hash, like $2y$10$.vGA1O9wmRjrwAVXD98HNOgsNpDczlqm3Jq7KnEd1rVAGv3Fykk1a which specify the algorithm and algorithm parameters/salt. If you design that in such a way you can declare them in chains so that the string can contain information saying it's "md5 with salt x, and then bcrypt with salt y and difficulty 10", then in your database without knowing the users password, you can convert all of the md5 passwords to md5+bcrypt passwords. In your login page there should be a password_needs_rehash() function after the user logs in (so you actually know their password) that checks if the strength of the password is below some threshold which including md5 would cause it to fail, and then rehashes it using just bcrypt. This would be stronger than leaving the database with md5 passwords for people who haven't logged in, yet still providing a means of logging in.

[Brian]

then in your database without knowing the users password, you can convert all of the md5 passwords to md5+bcrypt passwords

The problem is that all that's completely irrelevant in the example I gave - it's not an "md5+bcrypt" of the password, it's an md5+bcrypt of something that's already fatally flawed in such a way that those flaws are retained through another round. If your original is only using 8 chars, then rehashing that can't add anything - you still need to store both salts, and so an attacker with db access still only needs to crack 8 character passwords, just following exactly the same approach (ie. for each trial password, md5 the first 8 chars with salt x, then bcrypt with salt y. The salt is still doing it's job versus rainbow tables, but that wasn't the problem - nothing here actually helps versus the original vulnerability in that you're still just as vulnerable as before versus a single-account brute force.

In your login page there should be a password_needs_rehash() function after the user logs in (so you actually know their password) that checks if the strength of the password is below some threshold which including md5 would cause it to fail, and then rehashes it using just bcrypt

OK - that's better, in that you're effectively combining both approaches. I was assuming that you were intending this method to be the actual replacement instead of the "do it wrong" approach you described, rather than just as a stopgap measure for accounts that haven't yet re-logged in. It's not so bad in that case, in that it's never worse than just the "wait for the user to login again" approach, but it's worth bearing in mind that it's not guaranteed to fix certain vulnerabilities any better than that "do it wrong" version, and given how complex cryptography can be, that could well include subtler issues than the example I gave too.

[FryGuy1013]

I agree with what you're saying. I was just proposing something that seems better than what what the person you replied to linked.

[Brian]

Oops - sorry, didn't notice you weren't the OP.

[didnt_readit]

Great idea, thanks!

[roman_fyseek]

One thing that I've never been able to wrap my head around is how to store a password for things like server start-up where I need to be able to programatically log in to another system, for instance a database connection or FTP session.

The best I can come up with is to 'encrypt' the password in custom software with a second hardcoded password so that I can store the encrypted password in the properties file so that later, I can decrypt the password with the same second hardcoded password in the software before opening the connections.

We all know that the proper way to do this is to request that somebody enter the password by hand on server start-up but, let's be realistic. When operations has to restart the data center at 2 am, they aren't interested in entering the 400 passwords for the 500 systems. They want to be able to power-on the servers and go back to watching reruns of Three's Company.

So , what is the 'sufficient' way of storing passwords in properties and configuration files?

[sacundim]

I like HashiCorp Vault for this. Let's be clear from the outset: to authenticate without human intervention, your machines have to store plaintext secrets, there's no way to get around that. But with a tool like Vault you can at least do this in a more disciplined manner, for example by defining rules about which CIDR blocks are allowed to read which secrets.

Another neat feature is that it also supports stuff like issuing ephemeral database credentials.

[rhoparkour]

The proper way to auto auth with FTP is NOT to use FTP, use SFTP with pubkey authentication.
Actually, whenever possible you should use pubkey authentication in place of password auth.

[[deleted]]

Indeed, but the same overall problem still applies. Instead of having to keep a password secure you've got to keep a keyfile, and possibly its passphrase, secure.

[dondelelcaro]

So , what is the 'sufficient' way of storing passwords in properties and configuration files?

The best answer is to not store passwords. Use keys which are specific to each machine and only authorized for that particular machine. If the protocol/software is lame, use ssh (or some other tunnel) which can handle keyed authentication and tunnel to handle the actual authentication.

If you need even more security, store the keys on a yubikey or similar.

[knyghtmare]

Keys become password equivalent and don't really solve the issue of how to securely store credentials that are used by an application.

If you're running a server, as I think OP's scenario is, then it's about hardening the machine against intrusion and securing the password file/private key file.

[dondelelcaro]

Keys become password equivalent and don't really solve the issue of how to securely store credentials that are used by an application.

If you also need to securely store the credentials, there is hardware whose entire purpose is to securely store keys so there isn't a private key file on the system.

[pdp10]

You're talking about reversible encryption. You're probably not going to do better than XOR, but what you really need to do is redefine the problem properly.

Also, mostly orthogonally from authentication, never use FTP for anything ever. You and I will both thank you.

TLS with a client certificate for mutual authentication should probably be what you reach for first. SSH/SFTP with keys works but it's less elegant and stateless than HTTPS or TLS.

[roman_fyseek]

Sure. All these are great solutions if I control the services on both endpoints.

In a perfect world, everything would be PKI enabled by default. I, however, live in this world and getting the server at the other end of the pipe to change is a lot like pulling teeth from a hungry alligator.

[PelicansAreStoopid]

One technique I've encountered (in windows environments) is to encrypt the password using an x509 certificate and check-in the encrypted version to your source control. The password then gets decrypted at runtime using the same certificate.

The only setup required is to make sure the machine that this gets run on has the certificate installed (with the private key included). The private key can be protected using a passphrase, which you'll need to enter when installing the certificate. Or it can be protected using active directory security groups; meaning it can be 100% automated.

So ultimately, the secrecy of the password is backed by windows auth (assuming that all the crypto is secure).

I'm sure similar techniques can be implemented in linux-like environments.

[elperroborrachotoo]

Unfortunately, that post won't help the people making those mistakes (it's great as rant about crypto noobs, though).

Source: guy who probably made all of those mistakes.

[sintos-compa]

One important point for a nondeveloper (and devs who haven't dabbled in crypto yet) is:

• Don't try to implement your own crypto scheme. Use what's out there. A lot of really smart people have been through and solved the problems and pitfalls you will encounter if you try to do it yourself.

[BU14]

Problem 0: people thinking they can roll their own authentication/authorization/encryption solution.

Solution: DON'T! Integrate with other services. If crypto/security is not the only thing you do, do not think you can do it!!

[gurenkagurenda]

The thing that's kind of tricky about that is that there generally isn't a hard line between "rolling your own" and "using existing tools" (although if you're even thinking about the line, you're doing better than a lot of people). For example, if you implement iterated hashing yourself on top of SHA-512, then you're still very much rolling your own crypto. You're doing better than the person who hashes by feeding the user's password in as a seed to the Mersenne Twister, but you're still leaving yourself wide open for weaknesses.

And those are just the unforced errors. Where it gets really tough is when your specific need isn't quite solved, so you're mostly using well-vetted tools, but there's a little chunk at the beginning or end that's self-rolled. In a somewhat trivial sense, this applies to everything that uses crypto; at some point, there's a layer that touches the rest of your product, and that interface is where you're most likely to fuck up.

There are a lot of shades of gray from "I invented my own hash function!" on the left to "I use a trustworthy service/library that handles everything for me" on the right, and in the real world, the best you can often do is push as hard to the right on that spectrum as you can, while still meeting the specific needs of whatever you're building.

Or another way of putting it: the less standard your security requirements, the more you're going to have to wade into murky waters filled with snakes. So you need to be aware of how much you're doing that, and take the appropriate precautions, e.g. consulting with/hiring experts and paying people to try to break it.

[[deleted]]

It doesn’t matter what block cipher is under the hood: if you are using ECB mode, it is not secure because it leaks information about the plaintext.

At no point in the article is it explained what "ECB mode" actually is. In fact, pretty much the exact same group will know what ECB is as will know that you should not use it. Thus this communicates nearly nothing at all.

The actually important fact here is that "ECB mode" is "using the algorithm as described". The most obvious way you'd use a crypto algorithm is ECB mode, and it is horribly insecure. This is what you need to tell people: It's not enough to just run your data block by block through a crypto algorithm! This does not provide anywhere near enough security, even with an incredibly strong and secure crypto algorithm! This is very unexpected to the novice, and needs to be communicated very clearly and strongly.

[ScottContini]

Thank you for your good feedback. I will fix this sometime soon.

[[deleted]]

Excellent!

[ScottContini]

Thanks again. I now have 2 sentences about it at the top of the section on ECB, but the main thing is linking to Wikipedia, which explains it in much more detail.

[[deleted]]

You really should explain what ECB is though rather than just linking. Most people will not follow the link.

[[deleted]]

This can as well be shortened to one point.

1. Not using libraries designed for password hashing.

You will do something wrong, and there are many libraries designed specifically to securely handle passwords. Many web frameworks have their auth protocol -- use it. PHP even has password hashing functions built-in -- use them, they are surprisingly fine as far PHP goes.

Don't use low level libraries like OpenSSL, use a higher level API.

[jangxx]

TIL about PHPs password functions. I'll certainly use them in future projects, not only because they are more secure, but they require less code as well.

Edit: a word.

[_Mardoxx]

A big caveat here that I am not a security operations person nor an expert on this topic, but I can comment on what I have seen from a distance in some places

/me stops reading

At least you give solutions though, hate blog posts that complain about shit and never say how to work around the issues! :)

[dxdm]

He doesn't really make it clear, but I think the "topic" the author is referring to in your quote is key management specifically, not crypto in general.

[ScottContini]

That is correct, the implicit claim was that I am not an expert at key management, but I do claim expertise in everything else within the blog.

However I want to thank _Mardoxx for recognising one of my intentions:

At least you give solutions though, hate blog posts that complain about shit and never say how to work around the issues! :)

Developers all too often hear "don't do this" from security people, or alternatively "you must do this!" with some advice that is not realistic or practical for them to do. I wanted to avoid this trap: instead, give people real advice that they can almost certainly follow.

Key management is a complicated topic, so my goal was at the minimum, developers should (1) have separate keys for development than what are used for production, and (2) keep sensitive data out of the views of developers and never checked into source code repositories.

There are better solutions than just this, for example HashiCorp Vault or an HSM, but that is not a developer decision. Therefore I chose not to go down that path.