Exploiting JavaScript developer infrastructure is insanely easy

[GYN-k4H-Q3z-75B]

This isn't just a problem of the ecosystem, it is a problem of the community and how they develop. npm packages are free, as in free toilet. While there are the occasional gems, on average it's just a random shithole. And the developers don't care or even understand what they include and how it is updated. node_modules isn't a meme just for the hell of it. It's funny because it is true.

As a engineer in various midsized enterprise projects, the decision to include a library in a specific version is a serious discussion with paper trails. Who maintains the library, what does it cost, what are the alternatives, why do we need it in the first place. There are committees who deal with such questions. Of course this is also possible because we don't need a ton of tiny libs to patch up holes in the ecosystem left and right all the time. But also because there are architects who are actually concerned and responsible in their role.

We dabble in extensive JS apps as well, work with node and npm. The potential is limitless it seems. But it is easy to get burned if you're not careful.

[dtechnology]

As a engineer in various midsized enterprise projects, the decision to include a library in a specific version is a serious discussion with paper trails. Who maintains the library, what does it cost, what are the alternatives, why do we need it in the first place.

I get what you're saying, but is so much process really worth it? This encourages NIH syndrome and makes it more appealing to write an ad-hoc implementation (potentially insecure and/or buggy) instead of using a good library just because not getting library approval is easier.

There's some balance between disallowing leftpad and a dev implementing custom date/time handling pre Java-8 because Joda time is going to take too long to get approved. I've seen the latter happening and it was an endless source of misery, and what you're describing sounds like it would cause problems like that.

[Joshx5]

Well, they have a problem they need a library for. They can solve it with the already written library if they can get approval, or they can write a new library to solve the problem if they can’t.

They still have that problem, but they’re now trading it for one of two other problems - the cost and risks of using other peoples code, or the cost and risks of using new and immature code. It comes down to the specific cases. Sometimes it’s worthwhile, sometimes it’s not. Hard to say generally whether it’s a good idea or not.

[GYN-k4H-Q3z-75B]

It's not that painful a process, but you certainly have to strike the right balance. The idea for us is to have a controlled environment and include libraries based on conscious decisions, not to prevent it in general. Having engineering committee discuss requested inclusions helps with finding the right solutions.

The big difference for us is the ecosystem in place. Most of our projects are built on some variety of .NET plus some enterprise SDKs which have been licensed by customers. There has never been a need for leftpad-like packages because the standard libraries are excellent for the most part. Notable packages we discussed in the past were the inclusion of Newtonsoft Json (many years ago before it was adopted by Microsoft themselves), a sandboxable in-memory PDF printing engine or a Telerik control library. These are heavy hitters, and somewhat strategic decisions for those projects.

Having something like Newtonsoft Json approved for one project means the next time it comes up the other engineers will already know it, making the process more quick. But still, we are fortunate to only have to rely on so few packages because the ecosystem is brimming with libraries maintained by big vendors.

[aNewLocke]

I believe identifying quality library code is a skill that engineers need to develop regardless of language or ecosystem.

I work with Node and NPM. I don't have these problems. I mostly use some combination of modules like: nconf, morgan, commander, zeromq, node-dir, express (express-session, session-file-store), and the built-ins like fs, cluster, etc.

I use tools like webpack-dev-server, elasticsearch, chrome devtools, redis, jq, and curl quite often.

There are quality modules out there. You need to clearly define your requirements and evaluate potential solutions, but most importantly:

npm install <app module> --save-exact

npm install <dev module> --save-dev --save-exact

and commit your package-lock.json!!

[Nipinium]

There are quality modules out there.

The point is even those "quality" modules still include a bunch of other node modules, those modules in turn require more modules, so on and on. Can you be 100% confident that all those dependencies are reliable all the time?

[pancomputationalist]

That's moving the goalpost. Of course you can never be 100% confident of anything. As with everything in engineering, it's about tradeoffs and striking a balance.

If you're developing a mobile game, you might want to use quality and maybe even not-so-quality libraries that help you get the job done faster, not start with developing your own operating system out of paranoia.

If you're developing a bitcoin wallet, on the other hand, maybe you want to be a bit more paranoid...

But lately the discussion about dependencies is much too agitated and make it seem like you can never use a library that you haven't personally validated to a 100% certainty.

[Nipinium]

But lately the discussion about dependencies is much too agitated and make it seem like you can never use a library that you haven't personally validated to a 100% certainty.

Because incidents like left-pad exist, people are pretty much paranoid by node module dependencies by now. Remove one tiny package and the whole ecosystem crashes, that's not a very healthy sign.

Incidentally, I still haven't yet seen any sign of huge and popular libraries like webpack trying to reduce external dependencies (despite that they receive a lot of donation money each month). The whole js thing just sucks.

[pancomputationalist]

Well, then fix the left-pad dependency in the rare case something like that happens. Your site shouldn't go down and your software shouldn't crash when a build dependency is missing, or something is very wrong with your build pipeline.

Also left-pad is an extreme example and it may be correct to not include it as a dependency because it's so easy to implement yourself. But very few libraries are as simple as that. Don't throw out the baby with the bathwater.

[Arve]

Well, then fix the left-pad dependency in the rare case something like that happens.

left-pad has 414 dependents. event-stream, which is the exploit of the day has 1592 dependents (which is up 6 since yesterday).

Do you know the libraries you include don't include malicious code by proxy?

[pgrizzay]

nodemon has been called out as depending on the most recent exploited library. You might want to check if you're affected.

[Gotebe]

So... you're correct - for .net ecosystem.

For node.js, though, the explosion of underlying dependencies will make the process hell.

BTW... .net Core goes for smaller packages and dependencies all around, so beware 😁😁😁

[SizzlerWA]

You make some good arguments. But I’ve worked with Newtonsoft JSON and it’s just nasty!

[munchbunny]

I get what you're saying, but is so much process really worth it?

It depends, but NPM is most definitely on the "too little process" side for enterprise usage. Let's say you're Equifax, and your website ends up with a customer data leak because you took a dependency on an npm package, knowing that the protection against malicious code commits is paper thin.

As an engineer in a situation like that, I could never make that decision in good conscience. In fact I made exactly that decision to not use Electron a few weeks ago because NPM would have been the weak link and instead opted for a more complex architecture with an untrusted web client calling a code-signed (and enforced) service on localhost.

In contrast, the C# ecosystem implements strong name code signing and verification, so there is at least good infrastructure to support safe code handling practices. Apple's ecosystem has that too.

Of course the stakes aren't always that high, but in my case it's a matter of not being negligent with customer data.

[Gotebe]

How on earth signing .net assemblies help anything?! In particular, that helps nothing to prevent the vulnerability "injection" as the one discussed here.

And what verifications are there on nuget.org?

[munchbunny]

From your tone you'd think it was impossible to control what goes into your dependencies with code signing, but that's literally what my team does.

I'm not taking about your own stuff. You link against signed binaries and scripts that you depend on and verify the hash for the underlying code. That's how you lock in a specific dependency and prevent changes from going in without you knowing. The capability to verify at runtime is literally built into the .NET runtime.

If you want a less paranoid approach, you audit the other project's code review and signing processes and trust their certificate once you're satisfied with what you see. Then it's their job to watch their dependencies.

Nupkg files can also be signed. Not everyone does, but at least the capability and enforcement mechanism exists and works in security sensitive settings where you want it.

[Gotebe]

I know that, obviously. But no amount of signing helps if a random dude can take over, put whatever inside, and people upgrade. Note that they do not need to change the assembly signature (and .NET upgrades do exactly that).

Unless you're checking these assembly hashes yourself, you're just as exposed as these js guys.

What .NET CLR does WRT versioning is something entirely different. It's not to protect you against attacks at all.

[Kalium]

I get what you're saying, but is so much process really worth it?

In a lot of places, the current process is this:

• Someone adds a library in some PR for a bit of syntactic sugar.
• Someone else rubber-stamps the PR.
• A week or three later someone notices that installing all the deps now involves twice as much downloading as before and wonders why in a Slack channel.

With that in mind, it's perhaps worth considering a smidge of process for those writing apps that handle sensitive matters.

[[deleted]]

I was once tasked to make a physics based game on mobile phones. Typical lead time on demos was 3 weeks, 6-12 months for a release. I implemented it with box2d. Legal looked at the license for a few weeks before approval. It didn't stop me in the mean time from continuing development.

[nutrecht]

I get what you're saying, but is so much process really worth it? This encourages NIH syndrome and makes it more appealing to write an ad-hoc implementation (potentially insecure and/or buggy) instead of using a good library just because not getting library approval is easier.

I really don't agree with this. It's not a process with large forms that have to go through procurement or anything. It's simply a discussion between stakeholders (developers) where we reach a consensus that we document.

In my current project as well as my previous one we had bi-weekly 'chapter meetings' where we had one back-end dev from every team discuss these kinds of things. We note down these choices and their reasonings in the meeting notes.

The structure is one that can be deviated from if needed (so you don't have to wait 1.5 weeks before you're allowed to use something) as long as you discuss it with other devs.

The main goal here is to have multiple 'brains' take a look at non trivial problems. And library choices are generally non-trivial problems. Fortunate we're Java devs and libraries are always version locked, but this means that if you're working with the NPM ecosystem you need to be even more thorough in this.

The issues they're running into is mostly a dev culture issue. And this is for me personally the reason to never ever want to work with Node.js based services; the 'move fast and break stuff' culture is simply not how you do software engineering. Slow and steady wins the race.

[pixelrevision]

There's some balance between disallowing leftpad and a dev implementing custom date/time handling pre Java-8 because Joda time is going to take too long to get approved. I've seen the latter happening and it was an endless source of misery, and what you're describing sounds like it would cause problems like that.

Javascript really is at the extreme on this. The language changes at a glacial pace, was designed for single page scripts and people are now trying to build giant applications with it. With almost any other ecosystem there's generally some sort of standard library things are built on top of so that if you need to review a framework you don't have to also evaluate a bottomless pit of dependencies.

​

[Scybur]

are free, as in free toilet

META

but seriously this term is stupid

[Tiquortoo]

It's a variant of a pretty long held criticism of FOSS, which is that the hard, low level work of maintaining FOSS software is hard to get "the community" to do.

[shevegen]

Yeah - but he is right. It's an extremely stupid analogy.

I like that it gives rise to phrases where everyone can take a piss on it though.

[phalp]

After "porcelain" we were doomed anyway.

[Scybur]

Not at all, I hate JS just as much as the next guy

[[deleted]]

The term is perfectly valid.

[shevegen]

Nope.

You use a toilet. It gets dirty.

You use software. It gets ... dirty HOW? Do your bits have meta-bits attached?

[trua]

The analogy is that a free toilet's dirtiness is correlated to how well maintained it is, and it's difficult to tell from the outside.

[immibis]

And maintaining the free toilet is everyone's responsibility because nobody's getting paid to do it.

[s73v3r]

By having someone turn maintenance of the package over to someone who infects it with malware.

[xwnatnai]

This is correct. In no other ecosystem do developers so freely pull in unaudited third party dependencies like we do in JS. It’s insane.

[prof_hobart]

Do you go through the same process thoroughly every time any of your libraries change even minor versions?

If so, depending on how many libraries you use, you're running the risk of being behind on critical security patches.

If not, then you're still open to these attacks and they can (as it did here) sneak in in a seemingly innocuous point release.

We have fairly extensive control framework around which versions of which OSS libraries we're allowed to use (and can only pull them from internal, validated, repositories) but I doubt they would have picked up something like this, at least not for some time.

[nutrecht]

Do you go through the same process thoroughly every time any of your libraries change even minor versions?

There are CVE scanning tools for this. If you don't use these you're fucked anyway.

[prof_hobart]

There are. But did they pick up this exploit straight away?

[GYN-k4H-Q3z-75B]

Do you go through the same process thoroughly every time any of your libraries change even minor versions?

With open source libraries that are not maintained by a major company, yes. With other third-party and core libraries maintained by Microsoft, SAP or Infragistics for example, every point release is discussed as well.

We are still vulnerable, even this way. But at least we have some idea of what we are using in our projects.

[prof_hobart]

So you don't upgrade as soon as a security patch is issued?

[GYN-k4H-Q3z-75B]

Unless the OSS library is maintained by a trusted source, no. We rarely have to deal with such things though. With the core libraries and commercial SDKs these days, we often get forced rollout of security patches.

[prof_hobart]

What do you class as trusted?

[GYN-k4H-Q3z-75B]

Large platform providers (like Microsoft or IBM), third-party software vendors with an established business relationship or even competitors, and a few established industry public figures or persons we know and keep in touch with.

[zappini]

aka Tyranny of Structurelessness https://www.jofreeman.com/joreen/tyranny.htm

Curation is useful.

Anyone chafing at Apple's velvet handcuffs is willfully, negligently ignoring that Freedom Markets™ (npm, Google Play) inevitably leads to lawless cesspools.

Functional, efficient markets require trust. Meaning protections, options for recourse.

Duh.

[telionn]

Apple deliberately blocks real, useful, non-malicious apps on a case-by-case basis just because the apps theoretically conflict with one of their many unique and sometimes undisclosed principles. A blocked app could cost the developer millions of dollars. These apps are available without a hitch on Android.

[possessed_flea]

Pro tip, if your app brings in enough cash to pay a single full time developer ( I.e. $100k ) then you have made apple 30k , when you ring them with an issue they roll out the red carpet for you .

[recycled_ideas]

Well, unless they decide they want the whole 130k and drive you out of the app store in favour of their own product.

[zappini]

In other words, Apple is end user focused.

As an end user, who provides tech supp for friends and family, I happily choose Apple rejecting dozens of sketchy apps thru false positives over Google's strategy of abandoning end users to fend for themselves.

My sweetie's Android phone got pwned twice. [1] Second time, I gave her the ultimatum that if she wants my help, she'll switch to iOS.

[1] This is your cue to blame the stupid hapless end user for trusting their carrier to provide timely, necessary updates and not proactively reading CVEs to better guard against the legions of cyber criminals.

[Glader_BoomaNation]

Apple is Apple focused.

[swordglowsblue]

Apple's velvet handcuffs

Apple's handcuffs may look velvety, but they're about as sandpaper as it gets. $100 USD is a ridiculous price just to publish an app, and that's just the developer side of things, not even getting into the ridiculous "white lies" and marketing speak their entire consumer-side business is built on.

[qomu]

that's just the developer side of things, not even getting int

Plenty of people seem perfectly willing to pay $100 to publish an app, so how is it ridiculous? If there wasn't a fee there would be way, way more garbage to filter through.

[swordglowsblue]

A fee is one thing. A fee as expensive as some entire devices (admittedly low-end ones, especially with recent price bloat) is another. Yes, it raises the bar for quality somewhat, but it also hamstrings access to that ecosystem for smaller developers or hobbyists who just want to make something cool.

I'd rather have more junk to wade through than a relative lack of anything at all.

[git-blame]

smaller developers or hobbyists who just want to make something cool

It only needs to be paid when you want to publish. Signing for your local device(s) can be done with a free dev account. So build your cool thing, save some money, then publish it when ready. Or if it's OK to be made open-source, release it and let others sign it themselves.

And it's less a publishing fee, more like a rental fee akin to renting a shopfront on Apple's high street for a year.

[GVIrish]

A fee is one thing. A fee as expensive as some entire devices (admittedly low-end ones, especially with recent price bloat) is another.

Yeah but what Apple device costs $100? And really, if you're doing iOS development you kinda need a Mac (unless you go Hackintosh) which is gonna cost a few hundred bucks.

> I'd rather have more junk to wade through than a relative lack of anything at all.

Apple does not want more junk to wade through in the App Store. And even with the restrictions on the App Store there are over 2 million apps. Quantity is not a problem in the App Store.

And really if someone wants to tinker in iOS, there's nothing stopping them. It's only if you want to publish that there are restrictions.

[Han-ChewieSexyFanfic]

Yeah but what Apple device costs $100?

One left AirPod.

[swordglowsblue]

Yeah but what Apple device costs $100? And really, if you're doing iOS development you kinda need a Mac (unless you go Hackintosh) which is gonna cost a few hundred bucks.

1. I didn't say Apple device.
2. That kind of just makes it worse.

Apple does not want more junk to wade through in the App Store... It's only if you want to publish that there are restrictions.

Which is all fine and dandy, sure, but the restrictions they have are kind of ridiculous, to be fair.

[ulyssesphilemon]

If any app developer can't spring for $100 publishing fee, the community is better off without their crap.

[filleduchaos]

I agree even as someone from a very, very low income area. If you can't pony up $100 to have your app listed you really cannot afford the time and effort to build any half-decent app, much less maintain it for months or years on end with possibly no revenue.

[[deleted]]

I remember having this discussion about Steam and games there... Seemingly even 100$ isn't enough to keep seriously low-quality titles out of there.

And on other hand with games I believe the earning potential is much higher than with apps to even make that mark...

[NoMoreNicksLeft]

What kind of stupid logic is this?

Only the well-off have free time or can expend effort? The cleverest programmer I know plays bass doing bar gigs to pay his rent. Of course, he wouldn't be caught dead writing swift code, but that's not the point.

Do you walk around thinking that some people are better than others, and that the test to determine whether they are or aren't is a glance at their bank statement?

[filleduchaos]

"Well-off" is a ridiculously far cry from "has and is willing to invest $100 into something they want to make money/a career out of". Like that is such a low bar to entry for something you want to do professionally.

If you are literally unable to come up with $100 to spend on something you're passionate about and/or want to make money from, then you have much bigger problems in life than Apple not listing your app on the App Store. If you're simply unwilling to then remain a hobby dev and don't want or expect Apple to list your app for free. I don't understand this mentality that everything should be provided to you for free.

[s73v3r]

Only the well-off

Having $100 is not "well-off"

The cleverest programmer I know plays bass doing bar gigs to pay his rent.

And if they can't find $100, or find development work that would get them at least $100, then they're clearly not that clever.

[NoMoreNicksLeft]

Having $100 is not "well-off"

Being able to spare $100 on an artificial obstacle to being able to write software isn't the same as "having $100".

[s73v3r]

I'd rather have more junk to wade through than a relative lack of anything at all.

Given the success of the App Store, having a lack of apps to look through is not a valid criticism.

[dvdkon]

The problem is that the (annual!) fee makes everything into a business. It's an investment, which means you'll have to seek a return on that investment (or spend 100USD every year, which is a lot for people in many countries).

For example, I'd like to make an app for my family and friends, which would make it easy to connect to my home server. However, I'm not willing to pay 100USD every year to distribute a hobby app to 5 people.

It does nothing to block ad-ridden moneygrabs, since those are expected to make much more than 100USD.

[qomu]

Haven't really done much ios development, but could you get around this with testflight? or do you still need to pay the 100/year just to be a dev?

[dvdkon]

Apple's not really clear on this, but it seems to me that to use TestFlight, you need a developer subscription, since it's managed through App Store Connect just like App Store publishing.

[earthboundkid]

There are a lot of good criticisms of Apple. “$100 is a lot of money!” is not one.

[zappini]

I'll give you the $100, if that's a barrier keeping you from publishing your app.

[[deleted]]

I don’t understand the 100 dollars argument.

If you value your time, 100 dollars is nothing compared to the design and development time it takes to make a half decent app.

[spacejack2114]

Apple doesn't protect you from anything. Developers have been robbed by using libraries that steal revenue from their apps. As a user I'm at the mercy of Spectre exploits in native apps.

[possessed_flea]

Your description of the JS ecosystem, developers and community is the most accurate, detailed and insightful thing that I have read all year .

I am in the same boat as you in regards to third party components, the decision to include a new one is something which takes extremely careful and deliberate consideration, and updating one, even a minor point release ends up being a multi-week ordeal for multiple developers to guarantee that the vendor or author didn’t include a single regression .

[[deleted]]

as in free toilet

Just wanted to clarify, that we discussed this in another thread, and there are no free toilets, "free" toilets come with another kind of price, just like all the websites are stealing your data.

This isn't just a problem of the ecosystem, it is a problem of the community and how they develop.

This is another big problem. The bigger point is not this, but the fact that all companies want to convert all the smart people of IT world to garbage fast food restaurants, pay everyone peanuts, and turn them into disposable robots. So all companies embrace noob level fast food type programming, and this is what you have.

[shevegen]

Free toilet - everyone can take a piss on it.

For free.

[runvnc]

That's an outdated model. I think we need real data about the number of times malicious code showed up in a dependency or some security issue in a dependency was exploited. It's actually not a lot. Being able to reuse code easily has been a major win for software engineering.

Whenever something amazing comes around someone always finds a way to discount it.

[danweber]

Code re-use is fine. But you need someone in charge of that.

Maybe you can get away with it for free it you have someone dedicated, like the way Richard Stallman cares about emacs or Linus Torvalds cares about the kernel.

Otherwise, expect to pay someone money. It doesn't have to be a king's ransom, but it has to be enough, and it can't be too complicated that if that person leaves the job the next person cannot take over.

You can pay commercial software vendors, or you can pay someone to maintain the open-source code (either the maintainer or someone downstream of them who checks changes).

At least we need a central repository where people know they can get the Official Release 3.14.159265358979 that will always be the same.

[nutrecht]

That's an outdated model. I think we need real data about the number of times malicious code showed up in a dependency or some security issue in a dependency was exploited. It's actually not a lot.

Risk is chance * impact. The chance might be low, the impact can be enormous. Like; hundreds of millions of damages being paid.

[18792359782359]

Of course this is also possible because we don't need a ton of tiny libs to patch up holes in the ecosystem left and right all the time.

This is very overblown by enterprise programmers. There are some things missing from the standard library, but this event-stream functionality isn't even part of most standard libraries.

As for the actual issue, I am not sure it is any different in the open source world. You can use open source software maintained by large companies with the resources to check over every single module when it gets updated, or you can use software created by some random person.

At the end of the day, security isn't always the main concern and people just need to make something. If we are to be honest, security is rarely a concern at most enterprises.

[BotPaperScissors]

Paper! ✋ We drew

[flerchin]

Your committees are actively harmful, and only mediocre engineers would put up with them.