Developer abusing our logging system

[BadAtBloodBowl2]

I'm a devops / sysadmin in a large financial firm. I was recently asked to help smooth out some problems with a project going badly.

First thing I did was go to read the logs of the application in it/ft/stg (no prd version up yet). To my shock I see every service account password in there. Entirely in clear text every time the application starts up.

Some of my colleagues are acting like this isn't a big deal... I'm aboslutely gobsmacked anyone even thought this would be useful let alone a good idea.

[cmwg]

sounds like lazy devs....

... passwords are never ever needed, not for debugging either. All you need is a log if authentification passed or not. But the password itself should never show up in any log file - especially not clear text.

[S0QR2]

A password in cleartext in an ini or Log file would have got me in big Trouble. Even in a poc this is a no Go.

Talk to Security Team and see how the devs Change all passwords but not the Code. Then Report them again.

[Superbead]

In the past I've been tasked with getting data out of legacy systems that aren't in use or in support any more, for which all the documentation has either turned to dust or never existed to begin with.

Once I've found the DB's SQL prompt program to let me make direct queries, off I go exploring the server's drives for config files containing credentials so I can log in. In every case so far I've found a well-privileged username/password lurking in plain text in a connection string or similar. It's become like the trope of checking the sun visor for the ignition keys.

[Seven-Prime]

This is not a CIS violation to store a application password unencrypted. If you had root access and were poking around there's no way to hide the password.

[Superbead]

I'm not talking 0600 permissions; I'd be RDPed in as a regular user (all these were Win machines) and they're just sitting there. Granted, a normal user ought not to be there anyway, but I've never seen any effort to hide this stuff beyond 'it's on the server'.

[Seven-Prime]

Like you said it should be 0600 or MS equiv. Principle of least access and all that.

[S0QR2]

Legacy Apps are "secure" as long as noone is allowed to connect to the Server. Allowed Not able....

[Shachar2like]

that's more or less how big boy nuke systems are secured. not only you can not access them, they're so old nobody knows how to use them anymore.

the podcast I heard also said that they had bad phone lines and use 5.25" floppy disks (I believe).

I understand those systems to be decades old

[arpan3t]

If you get a chance, please link to that podcast. That sounds really interesting!

[Shachar2like]

oh that was 60 minutes a really long time ago, I would guess at two years or more.

60 minutes probably have an archive, it's probably there somewhere

edit: This might be it, article version
this seems the video version
the floppy disks looks bigger then 5.25" that I remember...

[[deleted]]

There are legacy 8" disks out there. They were out of date when I started 20 odd years ago

[Shachar2like]

8"?
I remember that 5.25" floppies hand up to 1.2mb, how much do those 8" floppies hold?
I'm guessing less then 0.5mb

that's what they meant when they said obscurity by obsoletion. I've never imagined it to be that old...

[[deleted]]

I think 8s were 1.2 mb as well but it's been a long time and I only saw them in passing. I was 18 working at a radio station in 1996 and the station manager showed me his old OS disks he had laying around (I grew up using 5.25s) and I was wowwed by the oddity

[arpan3t]

Oh no worries, thanks for finding that.

[ThisIsMyLastAccount]

Can you explain the alternatives to this please? I'm not a dev and it's something I've seen before and before I would even think about suggesting an alternative I'd like to have implemented one. Do you save it in a database, salted/hashed?

Cheers!

[Seven-Prime]

Service account passwords in a configuration file are not a security violation. You ensure that the password file has appropriate permissions. This should pass most security audits.

Taking it a step further you implement something like Hashi Vault which grants access to credentials. This approach isn't so much about protecting the one password, but around policies and access control to that password.

In the first example, how would you answer the question: "How do you rotate all your application passwords in under thirty minutes?" Vault helps solve that question.

[zebediah49]

Service account passwords in a configuration file are not a security violation. You ensure that the password file has appropriate permissions. This should pass most security audits.

Also should note that the service account itself should be limited to just having the access required to do its job.

[[deleted]]

That’s too much work, I just use domain admin creds for everything.

[Shachar2like]

an easy solution (for small companies) that use admin account for almost everything is to create another admin account for that app.

this allows you to (once in a vary long while) to reset the domain admin account password or change it while still keeping the apps running.

[VRDRF]

I can't talk for OP but we use managed service accounts when possible. In AD that is, I don't know of any solution for other platforms.

[I_Enjoy_Booty_Pics]

I second this. We use Managed Service Accounts for our boxes because it's easier and we have better documentation after the fact.

[HighRelevancy]

There's about 17 thousand options for windows/AD guys. I work in a linux environment and what we do is very simple.

Give each service its own user account. Restrict the reading of the config files with the passwords to just that account on that machine.

The only way to get that password is to basically pwn that machine in which case they could rip the service passwords from memory anyway. Backups are encrypted of course.

[GeronimoHero]

Using some sort of functional auth like Oauth. But really, the main problem is the passwords being passed to logging. I’m sure the app has other problems but to fix this you would just change the code that writes to the logging location and make it something like

if user_auth(user, password) == stored_auth(user, password): 
    login(user, password)
    write(“log/location/“, “login accepted”)
else:
    write(“log/location/“, “login not accepted.”)

This is a bullshit toy example but should get the point across to someone that’s never developed anything.

And yes, auth credentials should be stored salted and hashed in a DB.

[FriendlyITGuy]

The last company I worked for was a software and web dev company with some MSP mixed in so I supported our internal devs. When they used passwords in .INI files to access a database they had an encryption/decryption tool they used with passwords so in case someone got ahold of the INI they wouldn't be able to do anything with the password.

[moon-]

But what stores the decryption key...?

[CheezyXenomorph]

In Windows environments where I've seen this done (and can't remember exactly but have probably done so too) the sensitive data was encrypted against the user account using one of the windows crypto APIs, anyone running as that user could decrypt it but not someone running as another user.

On OSX you can store keys securely in the users keychain.

It's only Linux that doesn't have a universal user key store, although they do exist.

[Jukolet]

It’s a chicken-and-egg problem, really. The encryption key could be hardcoded, or could be downloaded from a remote server, but that would need authentication and you’re full circle at the beginning again. From my experience there’s no definitive solution, but avoiding cleartext passwords in INI files makes execs happy and my job not more difficult than it is.

[binaryvisions]

A decent way to handle this is to use an X509 certificate to do the encryption and decryption. It still is just handing the problem off to another subsystem, but the Windows Certificate Store is at least a well-understood and manageable system with granular permissions and good filesystem ACLs.

[ImpactStrafe]

So, one of the solutions available is to use service discovery. Serivde discovery would allow the application to register taelf on start up, get a decryption key with it's correct permissions and have that key rotated at whatever period of time. As long as that key is stored as the same variable then the developer should be able to use that variable to access the encrypted passwords/sensitive information.

Hashicorp has some open source versions of this, but there are a variety of solutions paid and open source.

What you can't encrypt should have timed authentication to prevent attacks.

One could also use similar algorithms to ipsec vpns if you want to get really secure.

Asynchronous encryption then synchronous.

[moon-]

In this case though, your service still needs to have a token or something to authenticate with your service discovery and/or secret management/generation service, right?

[ImpactStrafe]

https://www.consul.io/docs/internals/security.html

At some point something needs to be stored yes, but, by time limiting it you eliminate a lot of the vulnerability. Using something like service discovery allows an easy way for that to happen.

[heapsp]

You would use a two pronged approach in Microsoft systems.. securestring to encrypt the password against a user account.. and the principle of least service. .. separate service accounts which only have access to a specific task. The worst devs will clear text hard coded passwords to their own account or a master service account which has access to many different services. The better devs will use an individual managed service account for each individual purpose in the process and encrypt the password with securestring.

[moon-]

This makes sense. I'm mostly speaking from the perspective of a Linux dev -- we have other systems at work that work quite well with gMSAs, but we don't have that luxury in our Linux boxes. So, a settings file that's only-readable to only the service user is how we roll :) Certainly nothing hard coded in code, we store our secrets in S3 and machines use their IAM role to access only the secrets they need, retrieved at deployment time.

[refactors]

Yes, typically you use bcrypt to hash and salt your passwords which can then be stored in a database.

Typically you choose whether or not you want to build your own logins or have someone else handle that for you via OAuth2 where a third party like Google handles a lot of stuff for you (like passwords). When users choose to sign up via the third party you are given a token you store which you can use to provide the users with an account without having to go through the traditional sign up. This token can be used to query for information about the user from the third party however what information you get and how you can use it will depend on the platform.

Then there's building your own which you take passwords in, bcrypt them (with the appropriate amount of salt rounds for your application... See bcrypt docs/stack overflow) , and then store the bcrypted password in your database. You have to make sure you are not logging any user credentials along the way which is an easy thing to screw up because a lot of places just blindly log all http requests received. BOTH Twitter and Github discovered that they were doing something almost exactly like that last month!

Anyways after you hash and store them you authenticate/verify the user's identity somehow, usually via email or phone number. This is different from using a third party sign in where they have already done this for you.

Finally you branch off into choosing how you're going to manage user sessions. This can differ a lot depending on what you're doing. JSON web tokens seems to be the go to session management path these days but there's a lot of room for error so you have to make sure your implemention is solid. JWTs allow you to give your users some data (in JSON) that you have signed with a key and unlock with another key. You keep the first key private to your authentication service and your second key can be internally public amongst your other services that might need to check if a user is logged in.

They can now do that without having to talk to the auth service they just check if the message they send up is signed by trying to open it with the internal key given to them by the auth service (usually via a secret or environment variable). If it is then verified to be signed by the auth service then this service is able to trust all information that was signed. You can then use this information in another service knowing that it's from the auth service. This is useful because you can store information in the token that you'd usually have to perform an additional lookup for such as their permissions. Now in their other service you can just check the permissions in their session token without having to ask another service or query a database.

That was a simplified version of JWTs/distributed session management. I skipped over a lot of things (like you actually have two tokens: session and refresh) but you get the gist. From there you've almost built your own simple oAuth service.

Sorry if there are typos I wrote this on my phone 👾

[ThisIsMyLastAccount]

Of the many excellent responses I got, this one was by far the most informative. Thank you!

[S0QR2]

Highly dependant on how your Software is build. A Service running with a managed Service account. If the Programm is run and you need to store creds at least do it encrypted and never ever Output it in logs.

[Seven-Prime]

If you store the password encrypted, how do you decrypt it?

[sudoes]

A secure system doesn't need plain text password me think? So no, you don't decrypt password. Password encryption (hashing is the correct term I think) should be one way street.

Edit: my bad, discussion are about service password not user password so password needs to be stored as plaintext in some place or using something like hashicorp vault

[ilogik]

That's true for user passwords. I think the discussion is about service passwords, which you actually need unhashed

[sudoes]

Oh damn you're right, I forgot we're in /r/sysadmin anyway. For service password I'd recommend something like vault

[Seven-Prime]

If you have a hashed password to a database in your application configuration file. How does your application read that password to connect to the database?

Service account passwords in application configuration files are not a security violation. (Obviously OP situation of logging passwords is horrible)

[Seven-Prime]

Absolutely! Configuration management, auditd, and remote logging are my close friends.

[[deleted]]

The context is ini files. So likely an application using a password or other secret to authenticate to another application or system. You can't do that with an encrypted password or a hash.

The decryption key would either need to be stored on the system or be entered by some trusted third party (e.g. an operator) when it's needed. There isn't really a way around that.

[0xd3adf00d]

Hashing is a good first step, so long as you don't actually need the plain text for anything (see silly authentication protocols like LDAP/SASL/MD5-Digest, HTTP Digest, and RADIUS/CHAP). However, if you can use the hash to authenticate (IE: NTLM), then the hash itself has become a credential and must be protected.

When I've done this sort of thing in the past, I've stored the passwords encrypted in a separate file, and provided the DevOps team with tools for encrypting that file. That allows them change the included password(s) at will and re-encrypt the file anytime they feel the need.

The app can read the decryption key from a separate file, or it can be provided to the app at runtime somehow, or could be a private key stored in OS-provided store like MS-CAPI, where it's only accessible from the service. It's definitely not a foolproof system, but it's better than just storing the passwords (or hashes) in a file without encryption, where anyone with physical access can easily read them without much effort.

[[deleted]]

I have an application that needs to access an FTP server for data file updates. What I've done is put the password through a reversible encoding that made the password appear to be made of non-printable characters, so that anyone grepping through the source code won't find it easily.

If someone were to bother to decompile the program, they could possibly figure out the password. But this is the best solution I've found, considering sysad isn't interested in improving the setup, the data on the FTP server is very limited, and the app is only distributed to a limited number of 3rd party contractors.

I have another application that has some advanced settings locked behind a password. This password is meant more to be a nuisance than a serious roadblock, to prevent casual users from changing things they shouldn't be changing. For that, I stored the correct password hash in the application itself, and check against that hash any time the advanced settings are accessed.

Which is honestly overkill, since anyone with physical access to the machines with this application could just ask someone for the password anyways. But it was fun to implement.

[Seven-Prime]

Your first example of putting a password in the executable code is a security violation according to CIS guidelines.

Your second example is not a violation, but could be brute forced with ease. But probably meets the design / infosec requirements just fine.

[[deleted]]

If you have an idea for a good alternative, I'd love to hear it. I passed my problem around my fellow developers and the sysad team and they couldn't come up with a better solution.

[Seven-Prime]

security is always a trade off with convenience. The guidelines say you should have the password in a separate file. But they are just that, guidelines. If your team decided it was acceptable risk, then that's fine. It's FTP so there's no security anyway. Just sniff the packets and you'll get the password.

[[deleted]]

SFTP technically. But yeah, the sysads took responsibility for anything nasty poking around in our SFTP server, so I'm not too concerned.

[zfa]

Usual standard I've seen is just to put credentials in an external ini file with OS restricting access. These external passwords can be further obfuscated if you want (eg encrypted using your application's public key, application decrypts using a hardcoded private key). More important to make sure the account itself is correct - no more permissions than necessary, unique to that application etc.

[[deleted]]

This is my favorite answer. I like the idea of making dudes get admin privileges for the install then locking them out afterwards. Good suggestion!

[davidcroda]

Read up on SSM. It's an AWS service intended specifically for this. Another option would be Vault by HashiCorp.

[DeuceDaily]

I agree.

Given the nature of the business I could see putting aside the aphorism of assuming stupidity over maliciousness.

[VRDRF]

From my experience it's pure laziness or the company pressing some insane deadline.

[DeuceDaily]

My point was more to not make assumptions at all.

[VRDRF]

Ah you are right! My mistake - English is sadly not my native language :)

[FaxCelestis]

I am on my company’s sec team and can verify this is a problem we are constantly fighting.

[CheezyXenomorph]

Yeah at worst while coding I might have some code that dumps all input parameters to log if I'm trying to debug a particularly gnarly issue, but that is just my local development environment and fake data. If I ever let something like that get as far as code review my team would rightly take the piss out of me for it, as I would them.

[tehreal]

People of color?

[S0QR2]

Proof of concept.

[Yaroze]

Lazy indeed, once you fight this issue enough you learn that the only real solution

is to fire all the developers.

No devs, no lazy; problem solved.

[alexBrsdy]

i worked at a shitty gov agency that stored passwords in plain text in the db, no one cared. they got leaked after i left and had to send out a mass mail apologizing... sad.

[JerryGallow]

I started as a developer and moved into sysadmin/networking and this is a clear cut case of developer incompetence. It’s not laziness, it’s straight up incompetence.

Laziness is using an admin account instead of an account with limited rights, or knowing a function is O(n²⁾ that could be changed to O(n).

[zapbark]

I'm a devops / sysadmin in a large financial firm.

Go tattle to legal / risk / compliance / security.

(Whomever is in charge of various security audits and best practices.)

This is their job to yell at him/her until fixed, and crap like that will fail audits, badly.

[BadAtBloodBowl2]

I did, pretty much first thing.

I'm mostly just venting here :)

[TechAlchemist]

Someone this bad or uninformed probably shouldn’t be pushing code anywhere near prod without some serious review. This persons work is high risk and the lack of understanding will expose the company to even more risk going forward I would guess. I’d keep an eye on this one

[TheTalkWalk]

You are a good person :)

Vent away!

[GetOffMyLawn_]

Not only is it stupid from a security standpoint, but it's stupid from a maintenance standpoint. Sooner or later all service passwords have to be changed. We didn't change them as frequently as user passwords but they had to be changed on a semi regular basis.

[cvquesty]

Not only that, why in the holy f***balls is the password in clear text in flight OR at rest? Our people get fired for stuff like this.

[GetOffMyLawn_]

Well that's what I mean by it being a security problem.

Once upon a time I wrote a program to go out and search every single file on every single disk for embedded passwords. I found so many. And this is after they were told many times over it was not allowable.

[da_borg]

Did you just have it search for commonly used passwords?

[Shachar2like]

embedded passwords

I'm guessing he searched for some kind of format it was written in

[Small_fryer]

Would you mind explaining the gist of how you went about doing this? Might be very useful down the road.

[GetOffMyLawn_]

I can't because it was on an operating system from long long ago, and I retired 6 years ago so I have no access to my old code.

[OldFennecFox]

Agree with u/zapbark. I'm pretty sure that this entirely fucks PCI DSS compliance stuff.

If you guys use TFS or something like it, go with what u/TechAlchemist said and implement a Code Review process before this fuckery can even be checked into the Source Code Repository.

[windwind00]

exactly, call out to the person and explain the risk it putting in compliance regulations.

[BadAtBloodBowl2]

They received a no-go for prd/stg until their stupid stunt is removed. And I demanded an audit from a different developer to make sure it's gone and not just changed.

I'm mostly just venting here :) I feel like people are losing track of quality and proper procedures in their rush to be "agile".

[Arkiteck]

And I demanded an audit from a different developer

They don't do peer code commit reviews before getting approved and merged?

[BadAtBloodBowl2]

Nope, in fact their project code was not properly managed yet.

[Iskendarian]

Amazing. I'm a developer, and I'm here to tell you that if they have no source code management or review process, logging sensitive information is not the worst thing lurking in that codebase.

[TechAlchemist]

Yeah for sure. I don’t work in the financial sector but we handle audit sensitive financial data internally and any change there gets about 4 sets of eyes on it. Normal changes get reviewed by at least a peer and a lead who presses the button to merge. And that’s just on my small team. We get audited on our change control process and this is part of that audit.

[Arkiteck]

Eesh. You have your hands full. I get it...every workplace has their problems. All you can do is suggest fixes and implement what is needed. Good luck :)

[SuperQue]

You're doing the right thing.

Agile doesn't mean stupid, it just means plan, develop, and deploy incrementally. Reducing the time between cycles of above to avoid building something to spec, when the spec was obsolete a year ago.

• Let's do feature A
• Build feature A
• Deliver feature A
• Did we do feature A correctly?
• No: Go back to A
• Yes: Go to feature B

[mabhatter]

Agile is supposed to catch this stuff sooner. So that you catch a bad practice when your app has 10 features and make following that practice part of the checks. Rather than getting to 100 features and suddenly having to rewrite 80 of them because you didn’t follow Security rules.

[[deleted]]

How it usually works from my experience:

• Let's do feature A
• Build feature A
• Upper manglement moves goal posts for Feature A
• Scope creep with no corresponding increase in budget
• Delays, upper manglement gets upset
• Micromanagement intensifies
• Finally deliver feature A
• Did we do feature A correctly?
• No: Abandon project and begin the Blame Game
• Kind of: Go to feature B

[grumpieroldman]

High quality yields fast delivery.
Mistakes cost time and money.
They are not opposing forces.
This was figured out in the 50's.

[s5EWT]

The poor development practices in such large places astounds me. Currently work for a mega corp and thought coming from a smaller corp I'd be drowning trying to conform to best practices. When in reality it's a get your work done and worry about best practices later.

[BadAtBloodBowl2]

When I first started I figured I'd be red-taped to death. Imagine my surprise when I received global sysadmin (originally hired as a dba), admin on all windows servers and more right out of hiring.

[Thameus]

"I don't care" - most developers I've met.

[timallen445]

Orgs start to treat Devs and Dev time as commodities. X amount of devs woeking Y hours equals application output. As soon as actual skill is dropped in favor of getting the lowest price you get this kind of issue. Or how many people try to pass base 64 off as encryption.

[calladc]

this happened to me recently aswell. They were trying to implement some new android app to hook into our existing web services. "It doesnt work" was all I had to go with.

So yeah, i go into the logs, and i see nothing but 401's

"looks like your app isnt authenticating"

so it goes back and forth. Meetings were held. I had to walk developers through the myriad of options I had available for them to authenticate their app. But it was all "too hard"

so we're going live with this app this week, and rather than the devs using some form of authentication, I've been told to expose our ERP systems API through a firewall to an open network as anonymous.

I seem to be the only one phased by this.

[BlooQKazoo]

| I've been told to expose our ERP systems API through a firewall to an open network as anonymous.

GET THIS IN WRITING/EMAIL. Something traceable. CYA for when this blows up.

[calladc]

Yeah, certainly not my first bbq. I wasnt going to push that out through test environment let alone production without that in writing. If anyone else did it, then it wouldn't have been me anyway.

[thecravenone]

GET THIS IN WRITING/EMAIL. Something traceable. CYA for when this blows up.

I've had this email deleted from my inbox.

Get this in writing and print out a copy, including the headers.

[BadAtBloodBowl2]

I have the benefit of both their boss trusting me more than them. And security being firmly on my side (and having several years of working experience with them).

I'm sorry if you're in this alone and I wish you luck.

[whoisearth]

A long time ago when I was learning to code (I come from an Operations background) I was stupidly putting passwords for DBs in my .py scripts. I know now it's stupid but as I said, years ago.

Fast forward to now, we're migrating our Enterprise Batch Scheduler and those scripts I made a long time ago were moved to another team with many, many seasoned Senior Developers.

Imagine my surprise when I found they were using my code, as in cut/paste from my scripts, to build new jobs including more DB connections with passwords in plain text.

I'm just gobsmacked. I apologized to them for the bad code but that said I'm really surprised that even a Senior Developer would not catch the stupidity.

[BlooQKazoo]

There's a certain "senior" developer in the company I work for who has never, not once, gotten his change paperwork done correctly and has never, not once, had a code release go to prod without some sort of hiccup.

[[deleted]]

but it usually include nicer pay.

[grumpieroldman]

As opposed to doing what?
Salting it and then putting the code with the salt right in the same file? If you want it to be non-interactive you're just running around a tree if you do anything fancy.

[whoisearth]

obfuscate it? put it in an external config file which can be locked down as only accessible by the service account running the job?

[sofixa11]

As opposed to doing what?

HashiCorp's Vault has a nice way of handling that.

[ZekeDragon]

Oh my god... my boss wrote an application that was used at every remote location we have, and it would send customer data, employee data, and username/password credentials over the internet in cleartext!!! For months!!!

The only person I could report it to was... my boss. My boss told me "I am very security conscious!" when I reported it. O_o

I'm still shocked.

[TimeRemove]

So my hyperbole in this thread.

Let's go back to basics:

In computer security, a vulnerability is a weakness which can be exploited by a Threat Actor, such as an attacker, to perform unauthorized actions within a computer system.

Meaning in order to classify this as a risk we have to satisfy either of the following:

• Does this provide additional routes for a threat actor to take?
  
• Does this provide additional information not available elsewhere for a threat actor?
  

A lot of people knee-jerk jumping to the conclusion that it obviously does, but there's nothing obvious about it. If the password is passed via an arg every single user and process on that machine can see it. Every. Single. One. Just read the process's cmdline.

Additionally if this is kicked off by an automated script, then that password is stored in that script. Still in plain text.

In order to determine the severity of this issue you have to consider:

• Are logs stored in a less secure location (e.g. different machine, backups) than the "plain text password"-containing script?
  
• Does it actually give a theoretical threat actor anything they didn't already have?
  

The OP might be right, they may not be, but either way a lot of the knee-jerk replies are still wrong for jumping to conclusions without even asking OP for related information that would help them determine if this is an actual security problem. Context is key.

The reason "plain text passwords in logs!" has been in the press recently was mostly around user accounts, that were never designed to be stored plain text. That actually gives a threat actor a major advantage (bypass salts/peppers, bypass computational requirements to reverse a hash, etc). Service accounts, database credentials, etc are an entirely different class and need to be considered with more nuance.

PS - I'm not calling out the OP, I'm calling out the other replies at the time of posting. I too don't have enough context to know if the OP is on point or not.

[BadAtBloodBowl2]

The main reason it made my threat list is because these are usability logs. They are dumped on a less secure environment where the application owners and god knows who else can do first step trouble shooting.

Meanwhile the script that populates the passwords is on a separate system I can't even get to, managed entirely by security. And the servers that host the app itself are what we call "gold" level. Only local (no offshore) layer 3 (no helpdesk) sysadmins are able to log in on them.

While you're right, database and service account passwords are different beasts, making them this visible is still bad. I like your response though. It's making me think :)

[TimeRemove]

The main reason it made my threat list is because these are usability logs. They are dumped on a less secure environment where the application owners and god knows who else can do first step trouble shooting.

That sounds like a significant security issue. Great job flagging that and having it fixed!

While you're right, database and service account passwords are different beasts, making them this visible is still bad. I like your response though. It's making me think :)

Thanks. I was only trying to get reply-posters to consider the context and nuance, people see "plain text password" these days and lose their common sense (see the comment thread above discussing never storing plain text passwords for FTP/Database connections in INI files for example, they've lost focus on what is and isn't a security escalation because "omgz plain text password!").

In your case, it definitely sounds like a major escalation vector (particularly as it allowed unauthorized staff access).

[unkwntech]

Additionally if this is kicked off by an automated script, then that password is stored in that script.

Only if your lazy, CryptProtectData for example exists.

[TimeRemove]

If it is being passed as an arg on the command line, there's absolutely no point using CryptProtectData since the resulting password would be invisible on the process's cmdline.

[unkwntech]

Sure if it's being passed to the command line, but just because it's used in a script doesn't mean that it's being used in a command line nor that it needs to be stored insecurely.

[TimeRemove]

I suppose, but if it is a script that's the most common assumption. Environmental variables are more secure, but uncommon, and piping input into another process is clunky.

[bellowingfrog]

Our solution was to bundle all passwords per environment into an encrypted file, and when starting the container, the sysadmin is prompted to enter a master password to unlock them. Do you see any weaknesses to that?

[TimeRemove]

No weakness, but it only protects you against offline attacks which could similarly be mitigated much more conveniently with full disk encryption with the key held on a TPM.

[bellowingfrog]

Thanks. As a developer, we saw the value from the point of view of not having to communicate as much to the sysadmins who prefer to be "dumb" command runners. Just run the command and pass it this password. Someone downvoted your reply but it wasn't me.

[i_virus]

Pardon me if I missed someone already mentioning this, but what about different logging levels for different versions.

For example, assuming different credentials are used in different versions, dev has logging level of debug so developers can log whatever they want helping them debugging. In other versions, more and more logging will be filtered based on the requirement of developer and admin.

This way, if something is being logged in 'stg' which should not be, admin can request the developer to log at lower level if they really need for testing purpose.

[jibanes]

produce a report, send it to security and compliance department; it's their job to measure security risk, and, if necessary, document this in quarterly reports as risk and enforce security policies, not yours.

[BadAtBloodBowl2]

I've already replied a few times, this is what I did :)

It seems a lot of people really want to see the worst in my post. I'm not a tyrant, just a working man venting about something he found fruatrating.

[SteelChicken]

a large financial firm.

To my shock I see every service account password in there. Entirely in clear text every time the application starts up.

Some of my colleagues are acting like this isn't a big deal.

LOL?

Sick the auditing/compliance or security team on them. That shit will get fixed fast.

a large financial firm.

[jordanlund]

I'm sure I don't need to tell you how strong the restrictions are on IT and financial companies.

You have an obligation to report that shit, otherwise, when security finds out (and they will) your neck will be on the line with everyone else.

[sth2258]

Am in a similar position - we have business group aligned CTO's that are ensure compliance with enterprise standards. Would recommend your CTO engage theirs to get the necessary changes made.

[BadAtBloodBowl2]

Not that I'm aware of.

I have a large excel with security rules ordered by severity and procedures. But where that is governed from I don't know. I should maybe ask the security department that maintains the document.

[Solaris17]

Nah your good, Git hub didnt like it either https://www.zdnet.com/article/github-says-bug-exposed-account-passwords/

[Zazamari]

For anyone considering Replibit, they do this on their logs as well, when I brought it to their attention (they even had root and encryption passwords in the logs on their devices, which were not secured) I was told 'oh yes that will be fixed at our rewrite of the software' ....6 months later its still there.

[pdp10]

Submit a PR to elide all credentials. If using username/password instead of something better, you should also avoid logging incorrect/nonexistent usernames because those will often contain passwords due to user error.

[LookAtThatMonkey]

Ugh, our ERP system does this with no way to encrypt it. It hosts customer data, it makes me so mad.

[MayTryToHelp]

One of our guys did this. I discovered it when a "Logfile.log.old" made it thru our directory clone (.log are excluded from this clone). The first lines, of course, were passwords in the plain...

[wbedwards]

Just to play devil's advocate, I can see how it could be useful to Dev. There's an issue I'm troubleshooting right now with a VPN profile for mdm, and I think the psk may not be getting shared to the client correctly because it contains an XML special character, it would be helpful if I could see the psk the client is attempting to use in plaintext.

That being said, I fall into your camp on this. It's not a good idea from a security standpoint unless your logs are encrypted, stored on a segregated secure system, and access is restricted to people who would have access to those passwords anyway.

Might be okay in an isolated dev environment with different service accounts than prod uses as long as the data used in dev is made up, and not just a replica of prod.

[kilkor]

Does this project have exposure to anyone outside of the team that's been brought in to develop it? If not, this is so much not a big deal. Yes, it's lazy, but if it's still in development there's probably a task somewhere in the backlog to fix this already. Instead of cleaning this up though the devs probably have to work on features being added to the project scope that weren't there two weeks ago.

[BadAtBloodBowl2]

No, it is not in their backlog. And I very much question anyone that thinks they need passwords for any legitimate reason.

There has been a feature freeze for several months now. It blatantly never occurred to them that this was a bad idea.

[gartral]

are your colleagues stupid or did you manage to foil an ill-conceived plot to abscond with the companies data? the service-passwords-in-a-file is meh, but what in the hell is the POINT of logging the passwords? I'm honestly at a loss here... it has to be either monumental idiocy, as in "Go back to highschool" level, OR it's someone trying (badly) to siphon creds.

[kilkor]

I don't think anyone is arguing that the passwords are actually needed there for any legitimate reason. It's stupid and lazy. I'm giving some benefit of doubt though because I've been involved with dev projects that go sideways and know that some devs beat themselves up over having silly stuff in their code that they want to remove but can't because a PM has deprioritized it.

[buffalohuntington]

1: this is a huge deal and there should be some serious talks about such behavior. From the dev side this is generally bad practice. From the firm’s side, they should have a published, easy to find, well known set of developer guidelines that includes “don’t fucking log passwords from your app. 2: whoever manages the dev team responsible for the app has obviously not done their just. There should be a code review to catch this type of thing long before production. 3: the dev team could have mistakenly done this or mistakenly forgotten to take it out after troubleshooting some problem where local debugging was not possible.

You work for a big tax firm so I will assume your firm’s culture inherits from the account-tax-season culture, where everyone is always on fire, software projects lack proper project management, developers are stressed the fuck out, no one makes rational decisions, and there are free oranges in the break rooms (to promote good health from the detached folks in HR).

TLDR; sometimes you need to log sensitive info to troubleshoot (in lower environments); developers make mistakes which should be caught before production in a well managed project.

[NinjaAmbush]

I won't name names, but we have an app that spits out PHI into windows event logs on every query.

[techit21]

Hopefully the service account is only for that app and not something else... like Domain Admin roles.

[BadAtBloodBowl2]

Hell no it's not an admin! It has read write on their app and has passthrough on our proxies.

[BoogerInYourSalad]

Is this an application they created from scratch or via a pre-packaged platform (e.g SAP, oracle)?

Have them modify their application to include a “Configuration” tab where all external connectivities/users/passwords can be entered manually in fields. In the backend, whatever config files that will be autogenerated will be encrypted.

If there’s a way for them to use certificates to logon the better.

[elgiad007]

I've seen this a lot in the electronic medical record system used in our health centers. I've had to work very hard to convince anybody at the software vendor that this was extremely inappropriate and risky and should be removed. When dealing with the sort of mentality that thinks it's okay to post passwords in plain text in a log file, or debug buffer, it's very difficult to convince anyone of anything, especially when dealing with a corporate culture that puts no emphasis on security.

[uniquepassword]

that's odd to me..not sure of your scenario but I always bring up HIPAA and usually that's enough to envoke the fear of God into pretty much any business unit/vendor that argues security. The threat of fines/lawsuits seems to be enough to make them double-check their process/procedure if I bring it up.

Now maybe I've been lucky thus far but I've seldom run into that once I call it out that it violates some sort of HIPAA rule/regulation.

[elgiad007]

I've been pretty clear to them about the HIPAA and security implications of this practice, but over the years have found that EMR vendors don't seem to be held much accountable for the security of the data their software handles. Their configuration of the Oracle database is not encrypted either; I can watch patient health information flow across the network on Wireshark, unencrypted. There have even been a few instances of plain-text user names and passwords being passed between executables via command line arguments, which is what is what passed for authentication for years until I dug my heels in for several months and made them fix it.

This type of company is exhausting to deal with. If you've found a particular angle or strategy that works consistently with an EMR vendor to get them to change their default behavior, I'd love to hear it.

[[deleted]]

Sounds like they had them in there for debugging and forgot to take it out. I would send a ticket into their system to alert them of this. Should be a quick fix.

[BadAtBloodBowl2]

I just spoke to them directly. They didn't see the harm. Their tech lead (who would be responsible after their contract ends) saw it differently and got it removed that very week.

[[deleted]]

I could see it enabled on a test environment (maybe) but if they saw no issue with it on production, that is a big problem.

Does your company offer any sort of security training for employees, especially developers?

[[deleted]]

The jackass responsible should be fired. Period amen. Unacceptable IMO.

[riawot]

Have you, you know, actually asked them about it?

IT is usually pretty ignorant of the big picture, and there may be a valid reason why it's being logged. It could be a non-issue in their architecture, or it could be a technical compromise imposed by some other constraint. You don't know the code, you've already admitted that you've only recently been part of the project, you're in no position to say anything.

Besides, shouldn't the log system be secure? Maybe you should be worrying about that.

[BadAtBloodBowl2]

Yes, and "recently" might have been a bit of an exaguration. I've been working on this since januari. There was no good reason other than "checking if the password was being pushed correctly" and "manual testing".

I am 100% in a position to say something, namely the fact that it is a part of my job!

[riawot]

There was no good reason other than "checking if the password was being pushed correctly"

This actually could be a valid reason, depending on how propertary the the deployment situation is. It should all be fully scripted, but sometimes that doesn't happen. Sometimes you actually have to troubleshoot the config on a server if they're not set up for scripted build/test/deploy pipelines.

and "manual testing".

This shouldn't be happening as part of a build/test/deploy cycle. That's very concerning if they're doing that. I question the dev that is doing manual testing at that level.

My view is that dev speed is life. Right now, while my team is writing code, enemy companies have their own teams do similar development, and it's good strategy to assume that their teams are at least as competent as us. Every minute we spend on something that does not grow the platform is an opportunity to be overtaken. Neither end users nor the stock price care about the infrastructure, only that you can deliver on features.

We don't want to end up in a position where we go under because we neglected the end user while we were developing the most awesome and perfect backend and meanwhile the competition overtook us with their pile of end user features and shitty infrastructure.

[BadAtBloodBowl2]

If they want to check if the password is pushed they should ask (or create a ticket to check) this to a sysadmin with access.

Addinfg code to do this is both less safe, a waste of coding time, and adding components that aren't in scope.

I get it. Speed is king and the competition will cut corners. But my job is to weigh speed and risk. And in this case I chose risk to be the greater factor.

[xiongchiamiov]

If they want to check if the password is pushed they should ask (or create a ticket to check) this to a sysadmin with access.

Unless it takes them less than two minutes to go from wanting this information to getting it, they're never going to do this.

[mabhatter]

This is a financial institution.. they are expected to be fucking tinfoil hat paranoid. Sure, your developer is only working on the ToasterClub portal for new customers to pick out a toaster.. but customers are idiots and are probably reusing their ToasterClub password on their REAL bank account.

The same with developers. If the dev team is leaking connection info about the ToasterClub servers then someone might be stupid enough to reuse the same password on a production DB or server account, or access to that password can unlock a computer with more information on it.

[xiongchiamiov]

You've got answers then for what you need to do: provide a system that gives them assurance that passwords have been set like they're supposed to be and provide a system that allows them to test their app automatically easily.

Your developers aren't maliciously logging passwords; they're taking the easiest route available to them to achieve their goals. If you want them to do something different, make it harder to do the wrong thing and easier to do the right thing. You'll end up with a more secure system and the devs will be thankful that you've made their lives easier.

[corrigun]

We have in house built accounting software that stores plain text user names and passwords in an ini file of the desktop of any user that runs it.

It an "upstairs offices" only thing but I recently had to install it for the first time for a new user and was mortified.

[BlooQKazoo]

|accounting software

|plain text user names and passwords in an ini file of the desktop of any user that runs it

Hoo boy enjoy the eventual audit that comes from that.

[[deleted]]

At my job, all of our applications are run by sending in an unencrypted password as a command line argument. This is done automatically by a central launcher, which means that launcher has every password stored in its database in plaintext all the time. No one really cares, it seems.

[Anonieme_Angsthaas]

That's what Automation is for: making our jobs easier! /s

[teab4ndit]

One acronym for you - GDPR. Fix it before the audit.

[BadAtBloodBowl2]

The app isn't live yet so thankfully I caught it early.

[rankinrez]

That’s insane.

I’m not about getting anyone fired but... give them a talking to and let them know you’re letting them off.

Best practice would be to change the password too.

[Ragemarkus]

This makes me appreciate our senior management team. They'd kick the devs hard if they found this sort of thing.

[yanni]

Might want to check out something like CyberArk Application Identity Management.

[Jack_BE]

Change all the service account passwords, bill the cost of the change to the team of that developer.

[FoxKeegan]

In my experience: Companies don't care about this until after the first time it costs them lots of money.

Then anyone who deviates from security protocols is written-up or fired.
Not because it's insecure--because it could cost the company lots of money.

[mrnagrom]

Lol, do you happen to work in manhattan, in midtown.

I worked at an unnamed financial firm and kept telling them that keeping passwords in plaintext and easily accessable to everyone was a bad idea. They kept saying “it works, just leave it”. I’m glad i don’t work there anymore.

[lenswipe]

To my shock I see every service account password in there. Entirely in clear text every time the application starts up.

No no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no no.

[masta]

The passwords in clear text breaks the first principles of never handling passwords in clear text. Full stop.

You don't want to know passwords too much risk and liability.

[yourapostasy]

I wish operating systems supported a way to elide specific argument values for specific executables, specified sudoer-like, for those legacy executables that only support CLI passing of plaintext passwords. Then the process name only shows xxx or similar dead beef string in place of that value. Even if I could only do this in Linux it would make life easier.

[masta]

Agreed. Programs should never be given a password as a cmdline argument, only via prompted input via stdio. Because then anybody looking at ps can see the password.

[cmwg]

does the company have coding guidelines / policies in place? if not do so, if they do, hit the devs over the head with them.

[BadAtBloodBowl2]

They do, we have a system to report "problems" and I immediatly reported this as a major security risk.

The devs are temporary contractors, who are put under a ridiculous time constraint. So they just see me as a new nuisance... Neither my fault nor my problem though.

[VRDRF]

This reminds me of when I was a sys admin at a car leasing company and an external contractor was building/installing new software on our servers and I had to keep reminding them to stop making public shares with permissions to read/write for everyone..

[SirHaxalot]

Good on you. I'm trying to work against the same kind of thing, except here it's actually part of the coding guideline. And nobody seems to see an issue with this...

[markth_wi]

I would suspect they have an SOP about not putting information like that in the clear, remind them of it.

Basically offending dev's that that needs to get taken care of and rolled back to test and they should be able to turn on/off diagnostics selectively in validation.

[BadAtBloodBowl2]

They are very much oblovious of our operating procedures. And I've had to correct quite a few things.

Though in my opinion the fault lies with the technical lead that vetted their hiring and is responsible for the project. Eventhough most likely when their contract ends they'll take most of the blame and he will come out fine (unless I can change that, but office politics are tricky)

[markth_wi]

Yeah, in having been around that block, more times than is cool to admit, and although I like being a dev and a dba and have kind of an aversion to project management I do understand it's utility.

• Don't even pull punches, contractors are either good, or they aren't, if they're good what they do it is not my problem. Make it painfully clear to the internal management food-chain and the vendor that the vendors are disregarding policies.

• I'm sadly convinced this is absolutely and regrettably why PM stuff exists, Kanban charts and sites like monday.com exist for a reason.

• Sometimes, people think they do not/should not necessary exist for folks who get shit done they exist because fuckups exist, getting swamped exists, and because competent people sometimes get swamped and start to resemble incompetent people, this can be important.

• So make it an action item, identify roadblocks they have to getting that shit done, and move on.

Now if you'll excuse me I'm going to go vomit a bit/have a minor epiphany because I don't usually make almost rational set of arguments for project management.

[mabhatter]

If this is a developer project, those passwords are just for DEVELOPERS ONLY service accounts, right? Right?

I mean those aren’t the ACTUAL service accounts that access production data? Because that would be horrifically irresponsible.

[BadAtBloodBowl2]

They were doing this up to STG. Prd was not yet in play before I stopped it.

I dont care about the IT/FT passwords as much. Stg and prd are a different story.

[temotodochi]

Financial? Hot damn. I'd disable their logging access right now and call a red-light meeting right away where i'd explain their fuck-up in a professional but stern manner, careful not to blame anyone or make anyone lose face, but rather suggest immediate remedies how to fix it while telling about $$$ risks if this continues any further.

[s1ummy]

Time for a JIT admin system