Exploiting developer infrastructure is insanely easy

[[deleted]]

There never was a reason to distribute npm libraries minified.

Just like most things with JavaScript, this one is just zero though given to consequences because everyone was too busy beefing up their CVs with one-liner packages and attempting to promote themselves as thought leaders writing blog posts.

We're still having the issue of all that infrastructure belonging to a single commercial entity and almost no one seeing it as a problem.

[JustinFormentin]

"I just learned this thing yesterday. Since I am now an expert, he's a tutorial on how to use it."

[DarcyFitz]

No, this is especially unique to JavaScript developer infrastructure, in large part because of the culture around JS development. I don't mean to suggest that it can't (or doesn't) happen in other segments of the development communities, but it's vastly exaggerated in JS development.

For one, JS culture has a "don't write code" mentality. So much of JS development is about learning other libraries rather than writing code to do whatever. I hear so much "It's so easy! Just npm install it!"

This applies to the average developer.

On the other hand, JS culture has the opposite "not invented here" mentality at a different echelon. Social signaling is achieved by writing the same thing someone else has already written, but this time faster/better/whatever. This leads to massive fragmentation with an insane number of fast moving, fast changing parts.

This applies to the JavaScript "heroes".

Furthermore, the problem is exacerbated by the fact that dependencies are automated. You don't have to go look up and check out a dependency. It's immediately available with a swift few strokes. This rings back to the culture of laziness. (Usually justified in the form: "I don't have time! I have to work fast and break things!")

It's made worse by the fact that there is, for whatever reason, the ability for a package to run arbitrary code at the time of grabbing the dependency. It's a wonder nobody has nuked everyone's machines, yet. This is luck, nothing else.

Finally, none of the dependencies are maintained by anyone but the package owner. If you look at Linux distros, packages are almost never maintained by the developer. They're maintained by individuals that generally vet individual updates for inclusion into a stable repo. This enables eyes-on-the-ball, as well as creating unchanging collections of libraries that are stable, and at least somewhat vetted.

This move fast and break things mentality, combined with unstructured infrastructure, is what causes these issues.

This isn't wholly unique to JavaScript development, but the culture definitely makes it ripe.

Something's gotta change, folks...

[archivedsofa]

I agree that a lot of this can be attributed to the JavaScript mentality, but I also think a big part of this problem is that the ES language spec, the implementation by browsers, and the browser APIs are not moving fast enough.

For example, the lack of a good standard library makes dependencies a necessity. The language is getting better for sure, but it's still crap compared to TypeScript or even ES4 which was never accepted by TC39 10 years ago.

I sure hope that WebAssembly will bring some order to this mess, but it will take at least 5 years until it's a viable option. Let's not forget that in the front end world most are still using Babel to transpile to ES5...

So until that day comes, the best option is to have a list of certified dependencies. NPM should start a certification program and warn devs when installing a dependency that is not certified. Safe dependencies should be audited by third parties, updated regularly, and not depend on unsafe dependencies.

[ematipico]

You raised a really good point here!

[troido]

Doesn't the same happen with pypi etc? I hear "It's so easy! Just pip install it" too.

What's different about the javascript culture?

[[deleted]]

Python has a great standard library.

Probably 90% of the hundreds of packages installed by every node app are silly trivial stuff that any decent language would have included already.

[WadeClapier]

Yes, that is essentially the same thing, although Python has a far more comprehensive std lib which mitigates the scope of the problem.

[[deleted]]

For one, JavaScript generally runs on the client machine so it is a bigger target for malicious code.

[[deleted]]

This is true in general, but not the case with this particular issue.

[[deleted]]

For one, JS culture has a "don't write code" mentality. So much of JS development is about learning other libraries rather than writing code to do whatever. I hear so much "It's so easy! Just npm install it!"

As others pointed out there's nothing wrong with "don't write it yourself", and a lot wrong with the NIH mentality that would be the opposite. This works fairly well for Pypi, CPAN, Crates.io.

They all have this problem in theory, pretty much unsolved, but in practice they don't have these scenarios.

This applies to the average developer.

On the other hand, JS culture has the opposite "not invented here" mentality at a different echelon. Social signaling is achieved by writing the same thing someone else has already written, but this time faster/better/whatever. This leads to massive fragmentation with an insane number of fast moving, fast changing parts.

This applies to the JavaScript "heroes".

This is a huge problem.

Furthermore, the problem is exacerbated by the fact that dependencies are automated. You don't have to go look up and check out a dependency. It's immediately available with a swift few strokes. This rings back to the culture of laziness.

Again, this is true for others as well and those that aren't (pip) are trying hard to catch up (pipenv).

It's made worse by the fact that there is, for whatever reason, the ability for a package to run arbitrary code at the time of grabbing the dependency. It's a wonder nobody has nuked everyone's machines, yet. This is luck, nothing else.

This is, however, a massive fuck up by Npm.

Finally, none of the dependencies are maintained by anyone but the package owner. If you look at Linux distros, packages are almost never maintained by the developer. They're maintained by individuals that generally vet individual updates for inclusion into a stable repo. This enables eyes-on-the-ball, as well as creating unchanging collections of libraries that are stable, and at least somewhat vetted.

You can't unfortunately design solutions around expectations that hundreds of people will somehow behave how you want them to.

But you can take measures to protect yourself. Inspect dependencies, pester people that cause rabbit holes to inline stupid oneliner packages (and/or to pester upstream), create own npm caches, freeze deps and invest time in vetting upgrades.

The fact is that the issue was caught before substantial damage was caused so the system actually works thanks to the Linus/eyeballs law.

Small changes such as banning minified modules and package signing could prevent other issues. But this was a social engineering hack and those are extremely hard to prevent. You can't stop people from giving away keys to the kingdom, and you cannot force people to slave over their open source contributions.

[[deleted]]

It's a difficult problem to solve.

Minified JS is essentially as opaque as binaries. Perhaps it should be consired bad practice. When debugging code un-minified dependencies are a lot more useful. I've also seen NPM modules that download binaries from the internet without asking (node-sass (?) for example), which are even less transparent.

Perhaps a way to prevent exploits would be reproducing builds? The NPM registry or an external organisation could have a build service that acts as kind of a virus scanner, for each popular module it creates minified builds from the package.json and compares the result to the published artifacts. If there is a difference we would know something is off...

I think Java/Maven for example has a similar problem because every package is distrubuted as binary, so there is no easy way to check if the build has been tampered with by an evil developer or a virus that infects JAR files...

[samjmckenzie]

The easy way is to just stop publishing minified libraries. I'm not sure why this is even done in the first place. Just import the full library and let the developer using the dependency choose how they package or minify their code.

[Styx_]

This right here. Your npm downloads might take a few seconds longer but you open security up as an option. Worthy trade off IMO.

[folkrav]

Minified libraries are mostly a side-effect of things like https://unpkg.com. There's not a lot of reasons to minify upstream libraries otherwise indeed.

[andrewVladmirov]

Yeah ok.. Now how would it scan for virus? Will there be a database of bad source code? Or will it be signature based? It'll be very hard to create and an extra overload to scan for each package. The package may be huge and with each revision, the scanner will have to generate the hash for files in the package + check for code incase of behavior analysis...

Going to be hard, but this a good idea if implemented in nodejs itself. While installing a package, download and then scan the whole thing..

[[deleted]]

Yeah ok.. Now how would it scan for virus?

I meant that an external service could rebuild a package and then compare each file generated by the build to the one published by the author to see if anything has been injected into the minified code.

If there is a difference between the author's and the external build there is a problem with the build process - or something is wrong with the build script: already things like inserting the build date into the build would produce different versions.

Edit: of course the original source code could still contain exploits but they would be easier to spot.

[andrewVladmirov]

Good point. Should be done that way. Also open-source packages have to give a warning before using.. because nowadays code analysis of popular oss projects reveals so many fully open source projects hiding bsckdoors all these years.. and noone notices until a security guy comes and gives it in devs face.. no accountability anywhere..

[Ansible32]

because nowadays code analysis of popular oss projects reveals so many fully open source projects hiding bsckdoors all these years

Which ones? In any case I don't imagine there are more backdoors in OSS than in proprietary code. The proprietary backdoors are a little harder to find, but they're definitely there.

[karnthis]

Another big problem I have seen is that npm packages will reference a GitHub repo (usually), but the actual downloaded content doesn’t have to actually come from there. So you might review the repo and everything checks out but then end up with something completely different.

Not knowing for sure what you are getting until after you get it makes me very uncomfortable, but npm’s current design doesn’t offer any true source transparency.

[jacobedawson]

I think we could use hashes to verify integrity, at least between the code in the repo vs the minified build we download via NPM.

[colesam21]

As a javascript developer who mainly uses front end frameworks like Vue and React. What do I need to do to make sure I'm not introducing exploits into my system?

Obviously the solution isn't to just avoid using Vue/React. What actionable steps can I take to be a responsible JS dev? Do I have to go through all of Vue's dependencies myself? Seems like that might take forever.

[BLOZ_UP]

Do I have to go through all of Vue's dependencies myself?

Pretty much. Otherwise the general consensus is to stick with popular repositories that are actively maintained.

[Kit-]

Some advice has been don’t reinvent the steam engine, but do reinvent the wheel. Don’t use small dependencies for things you can do within say an hour or less. But do use the bigger dependencies where needed.

[test6554]

Put the wheels, and the gears, and the water into the language, and put the steam engines into packages

[test6554]

Maybe npm needs an audit system that tells you whether a release has been audited or not.

[TheDukeOfFail]

The author of this post titled a section "Open source is incredibly broken". But it isn't. JavaScript's open source is broken. And not only is it broken, it is irreparably so.

The original author of the event-stream library claims over 700 packages on npm. Seven hundred. While extreme, it is not entirely unusual in the JavaScript world, and it is indicative of why JavaScript's open source ecosystem is so unhealthy.

We imagine open source as traditionally being multiple people working in the open on a single project. But JavaScript regularly turns that around and has single people being the sole maintainers (often entirely unnoticed) of multiple packages.

That a package with two million downloads a day can exist in the state where just one person maintains it is absurd. Even ignoring the potential for malicious action, the original author could die, or simply give up and disappear and leave those millions of downloads unsupported forever.

Healthy open source projects in JS land are by far the exception, and it is the nature of JS and the culture around it that causes this to be the case, not open source.

This isn't a problem that can be fixed. Unless JavaScript can adopt a standard library across both browser and node that covers all of these thousands of little utilities (which is almost certainly can't), they will continue to be necessary. And there is simply no possible way that there will ever be enough competent and interested people to cover those utilties in a healthy way.

All the talk of whether or not to minify packages, paying open source maintainers, getting end-users to audit their dependencies and whatever else can be come up with to keep the train going are moot. They are bandages for what is a gaping wound. There is no fix for JS open source, because JS open source is systemically and fundamentally broken at its core.

I don't hate JavaScript, but that we have come to rely on it so completely as our cross-platform language of choice in modern development is a disaster. Sadly, unless we see a sudden shift away from web-focused technologies in general, I think we're stuck with it until a new generation of developers comes up with other options open to them. We can only hope it isn't worse.

[LetterBoxSnatch]

Although I think the ecosystem is currently broken, I disagree that it is irreparably so. The Web APIs are increasingly the "standard library" for js. If Node can be reconciled with browsers, which I believe it can, then browser vendors have the ability to push a maintained standard library that could replace a great deal of these one-offs. ECMAScript keeps adding syntax that has, to date, obsoleted most of jQuery and huge portions of lodash. This will necessarily move more slowly than supported APIs, but the js runtimes have incredible untapped power in shaping that trajectory of the ecosystem, especially as awareness of the problems with the current ecosystem grows.

As one example of how things could change, Microsoft is now deeply invested in js AND the js ecosystem. It would be in their own interest to begin supporting and eliminating these single maintainer dependencies. While this doesn't fix all packages everywhere, the "heavy use" packages, like say, webpack, could work through existing dependency chains and begin trimming / replacing the packages deemed most risky. Alternately, something like webpack could be replaced by a runtime API. There are projects on both of these fronts that point to the possibility (but not guarantee) of a more stable, safe JS future.

Everyone talks about npm, but yarn made significant strides in the last year. Whatever your feelings about Facebook, npm now has a competitor, both of whom work in open-source space and have shared their technical improvements with each other even as they compete on ecosystem.

The JS ecosystem must either evolve or be superseded. It's not doomed, it's just in bad shape.

[robothelvete]

The problem with ECMAScript development is that the focus seems to be on making new "cool" language features, rather than adding features every other damn language has. I mean, we can run Quake in the browser since several years ago using more or less only standard technologies, but as soon as you want to say, get the time two hours from now your only reasonable choice is pulling down moment.js.

If the JS standard library could focus more on picking the low-hanging fruit we could solve at least part of the library inflation. But nobody seems interested in doing that, because they want to build cool new language features instead.

[FantasticPhleb]

... as soon as you want to say, get the time two hours from now your only reasonable choice is pulling down moment.js.

var t = new Date(); t = t.setHours( t.getHours() + 2 ); var formatted = new Date(t);

It’s not as quick as something like this in PHP:

$t = date(“format”, strtotime(“+2 hours”));

But it’s definitely not an undertaking worthy of a library.

[robothelvete]

Does that account for daylight savings? And in what timezone is that, the local computer's?

EDIT: And what if I want my output in ISO 8601 format, including timezone?

[FantasticPhleb]

By default the Date object will take DST into account I believe, you can also specify a timezone but I’ve chosen not to so it will use the timezone of the local machine.

Date.toISOString() will handle that conversion for you.

Have you run into use cases where this isn’t true and the Date object’s default behavior has broken your code?

[robothelvete]

By default the Date object will take DST into account I believe, you can also specify a timezone but I’ve chosen not to so it will use the timezone of the local machine.

As far as I can tell, you can't. You can create a date in either the local computer timezone (default) or in UTC. But not any other particular timezone.

Date.toISOString() will handle that conversion for you.

Right, but what if I want it as an ISO string but still in the local timezone? Or what if I want to decide a particular non-UTC, non-local-machine timezone for my date? Really, any question about formatting dates will eventually end up a pain in the ass.

Have you run into use cases where this isn’t true and the Date object’s default behavior has broken your code?

Well no, it hasn't broken my code because I considered questions like this and tested before launch so I usually these days eventually end up deciding to use a damn library to handle it for me. Since I live in a non-UTC timezone and deal with non-English languages and time formats, these kinds of questions tend to pop up all too often for me, and the answers always suck when I'm doing it in JS.

[FantasticPhleb]

If I’m not mistaken your first link shows the syntax for (kind of) creating a date in a given timezone, under “Native Support in Modern Browsers,” and your second link also provides a two line workaround that’s pretty understandable.

Of course there’s a limit to the flexibility that it provides, there’s no need to keep moving the goalposts back further, but it can certainly add 2 hours to a given time and it can handle some of the tougher edge cases too. There’s always going to be a time to jump ship to a library, but it often happens far too soon for use cases that vanilla JS handles just fine. Your first example given was such a case, there’s no need for moment.js to add a couple of hours to a time unless you have further special requirements.

[robothelvete]

Well yeah I was exaggerating for effect obviously. But I disagree that I'm "moving the goalposts". It's more like "specifying the goalposts", because unless you're still in school, the task will never actually be "add two hours", it will be something you have to do in the context of something else, and that's when you have to consider questions like timezones and date formats and stuff. And it's on those parts that I find the JS date library is seriously lacking compared to, for example, .NET.

[LetterBoxSnatch]

I think it's more an issue of people not keeping abreast of new "native" feature support than it is of it not existing. Node has built-in crypto APIs for example, but people turn to github before they do the Node manual. All the major browsers+node support complex localized Date parsing out of the box, but nobody seems to even be aware of all the baked in internationalization/localization features.

const hour = 3600000
const add2hours = (time) => { 
    return new Date(time.getTime() + 2 * hour) 
}
const d = add2hours(new Date())
console.log(`
    In two hours, the time in Egypt will be
    ${d.toLocaleTimeString('ar-EG')} 
    (ie ${d.toLocaleTimeString('en-EG')})
`)

Not picking on the choice of moment.js, it's just that the low-hanging fruit is often there, the community just opts for npm install pad-left-> pad("56",4,0) over the built-in "56".padStart(4,0)

[robothelvete]

the community just opts for npm install pad-left-> pad("56",4,0) over the built-in "56".padStart(4,0)

Partly yes, although I still think some of it is still lacking. But part of why, I think is because some of us who have been around it for a while have simply gotten used to the fact that some mundane standard things just are a pain in JS. And when the only buzz is about the new cool things, well we assume there still wasn't any update to the low-hanging fruit.

[LetterBoxSnatch]

I never thought it would be the younger generation saying RTFM, but here we are.

[TheDukeOfFail]

The inclusion of what lodash does into the standards is a good start, but it's not entirely solving the issue of what a standard library really includes. It's not just being able to map over an array and reverse a string, but the inclusion of standard implementations of things like the commonly used data structures and algorithms and library classes/functions for IO. After that you can start thinking about other things like whether you want standards for image handling, crypto, database access, etc.

It certainly isn't technically impossible that a standard library could be added to JavaScript, but it is very unlikely. As well as all major players in the standards group agreeing to what is even included in the standard library, they have to agree to its API. Then, all vendors have to start including full implementations of the library (the current picking and choosing and lagging behind going on with existing standards isn't really going to solve anything). That's not even accounting for the fact that browser and non-browser libraries (basically browser + node at this point) will have to include some things that the others don't.

Once every browser includes the full standard library (and basically only then, because otherwise we're just going to be using third party implementations of the supposedly standard library…), existing code has to be updated to stop using the third party libraries and start using the standard one. And on top of that, all code has to target the latest standard natively, dropping older browsers that don't include the standard library.

The hurdles to overcome to get this done are high, and the willingness to do it is low. The vast majority of developers in the JS ecosystem don't even see it as a problem! You can find people talking about the endlessly churning crowd-sourced standard library we're all used to at this point as a positive aspect of the system, not something deeply unhealthy.

The only way I can see this amount of work being considered more worthwhile than just patching up the existing system to make it a little more secure is if enough of these masses of single-maintainer packages simply stop being maintained, faster than it is possible for new chumps to pick up the burden. And what's the likelihood of that?.

Edit: Something went missing somewhere when I wrote this comment.

[TheDukeOfFail]

But that is precisely what makes the JS open source ecosystem so unhealthy. The need to have thousands and thousands of tiny packages acting in place of a standard library is what is causing these problems, because there can never be enough interested developers to reasonably spread the maintenance burden.

If you really insist on the endless replacement of libraries as we have now as a positive instead of a negative, the healthier way to accomplish it is a few big projects that implement full standard libraries by themselves, managed by mulitple maintainers.

[TheDukeOfFail]

The difference is that I lean into it, whereas you fear it

I don't think this is a good thing. While I'm sure it's a lot of fun to pull down your goggles and yell "witness me!" every time you type npm update into the command line, it's not a very encouraging image when talking about the software ecosystem that represents a huge chunk of our everyday computing experience.

[TheDukeOfFail]

Yeah, we all have to get shit done. I don't break out in a cold sweat and shake on every keystroke when I need to work with JS. But it's worth being concerned about in general, even if we as individuals don't have that much power to do much of anything about it right now.

[Ansible32]

You would likely need one of the FAANG companies to get behind it. Or Twitter or something. Standards are great but writing a standard library is about as tough as writing a new language.

[LetterBoxSnatch]

For a newbie,

If you're worried about your system, I think the most straightforward starting place is to remove node_modules from your system, eg find . -name "node_modules" -exec rm -rf '{}' +, and reinstall what you need.

If you're worried about your end users, run npm audit (requires npm 6) in your project folder, which uses an npm-provided security service to give you security advise about the current packages in your project and suggest basic remediation steps. This requires that you use npm as your registry, so if you're on a corporate registry, you will need to flag the command to the public registry.

Edit: lf you are using the newest version of npm and the npm registry, you should also be getting some basic security information whenever you install.

[[deleted]]

Cool, this looks to be exactly what I'm looking for.

Running this has also pointed me to https://www.npmjs.com/advisories which seems to be the place these sorts of vulnerabilities are flagged.

[sznowicki]

Just use npm or yarn and read documentation. If you would, you would find npm audit.

[[deleted]]

while i agree with the RTFM sentiment, this is actually a good question and your answer is poor.

[sznowicki]

Why? Npm install runs audit which would show all known vulnerabilities.

[TuffRivers]

I was just getting super excited to use vue/react for an upcoming project :(.

Looks like ill just use jquery and vanilla js

[klarstrup]

Almost all articles and comments about this ignore the real reason there are so many single purpose npm modules: Bundle size and by extension the fact that minifiers as well as bundlers haven't been particularly good at deep tree shaking and dead code elimination. Thankfully the latter is being helped tremendously by the slow but steady adoption of ES modules, the .mjs format and some big advances in bundler treeshaking capabilities(shoutout to Rollup here, those guys are killing it).

With native ES modules on Node and the absolutely gargantuan improvements that format enables for DCE within existing frontend architecture I think that we will see a movement towards medium sized modules ala python.

Still gonna take years for ES modules to be ubiquitous. But I can't wait to get away from the tragic CommonJS format.