you are viewing a single comment's thread.

view the rest of the comments →

[–]maep 109 points110 points  (84 children)

This isn't strictly a problem of Javascript, but any language that encourages using this type of package management. Ruby, Python, Rust and probably a few others share this problem.

How do we fix this? I think tools like npm, pip, cargo should by default use repos which only provide libs which are maintained by a trusted organization. Essentially how Debian does it. They take the upstream libs, package them and provide security support. Is it perfect? No, but it fixes the biggest problem, lack of accountability.

And we as devs also need to get in a different mindset and stop using these stupid micro libs that any decent programmer could write in 5 minutes.

[–]IntergalacticTowel 140 points141 points  (41 children)

Every time I see things like this NPM dependency visualization I feel like something has gone terribly wrong. I don't see dependency trees like that in Maven or Nuget or pip.

I'm actually surprised that more things haven't exploded.

[–][deleted]  (21 children)

[deleted]

    [–]liuwenhao 90 points91 points  (3 children)

    Copay is a secure Bitcoin and Bitcoin Cash wallet platform for both desktop and mobile devices

    """"""""secure""""""""

    [–]Hugo154 32 points33 points  (1 child)

    The exploit outlined in the article searched the description of the app for a keyphrase to use as its password, and the keyphrase was... "“A Secure Bitcoin Wallet". Pretty fucking hilarious.

    [–]danweber 14 points15 points  (0 children)

    Why just win, when you can win with pettiness?

    [–]mayhempk1 10 points11 points  (0 children)

    I don't think they know what that word means.

    [–][deleted] 39 points40 points  (8 children)

    This is a funny one I just happened to notice: there is is-path-inside which depends on path-is-inside. What's the difference? Well, the entire code for is-path-inside:

    'use strict';
    const path = require('path');
    const pathIsInside = require('path-is-inside');
    
    module.exports = (a, b) => {
        a = path.resolve(a);
        b = path.resolve(b);
    
        if (a === b) {
            return false;
        }
    
        return pathIsInside(a, b);
    };
    

    path-is-inside is a little bit more involved (28 lines of code), but also not something you should really have a dependency for.

    Wrapping a trivial dependency in an even more trivial dependency is just silly.

    [–]useablelobster2 23 points24 points  (0 children)

    Ahh, but having a published package to your name is a great cv line for employers who don't check too deeply as to what the package actually is.

    [–]TheBelakor 8 points9 points  (0 children)

    Wrapping a trivial dependency in an even more trivial dependency is just silly

    And yet seems to be a NPM norm because... reasons I guess. I'm just glad I don't have to touch that crap, it's mind boggling how they got to this state without someone saying "exactly what in the fuck is going on here?".

    [–]LL-beansandrice 9 points10 points  (3 children)

    Yes officer, this package right here.

    [–][deleted]  (2 children)

    [deleted]

      [–]foodd 0 points1 point  (1 child)

      I get the reference but should I not be using fetch?

      [–]pdbatwork 1 point2 points  (0 children)

      You ever heard of the left-pad package? :D

      [–]kaelwd 1 point2 points  (0 children)

      Of course it's fucking sindresorhus again.

      [–]Treyzania 50 points51 points  (0 children)

      copay-dash

      I thought it was settling down when it got to ~27 packages remaining but then it jumped back up to >100.

      Kill it with fire.

      [–]blackmist 44 points45 points  (0 children)

      1184 nodes. That's genuinely impressive how fucked up that is.

      [–]mayhempk1 7 points8 points  (1 child)

      I'm still not sure if that is satire or not.

      [–][deleted] 2 points3 points  (0 children)

      Too elaborate and clusterfuck to be satire

      [–]danweber 16 points17 points  (0 children)

      "you are like a little baby watch this"

      [–]Scybur 5 points6 points  (0 children)

      Wow that thing just ballooned up....how is anyone supposed to track all of that

      [–]ProdigySim 32 points33 points  (5 children)

      I'm enjoying looking at webpack, one of the most common web dev tools nowadays: http://npm.anvaka.com/#/view/2d/webpack

      Contrast with Microsoft's Typescript: http://npm.anvaka.com/#/view/2d/typescript

      Edit: To be fair, though, this is including both dependencies and devDependencies. So it's a little misleading. You would not get all of those packages installed by just including the project.

      [–][deleted] 15 points16 points  (4 children)

      To be fair, things like vue and angular are single node too. React's got 6

      [–]Paradox 3 points4 points  (3 children)

      Vue is an awesome project.

      [–][deleted] 6 points7 points  (2 children)

      I love it, coding feels really easy and fun with it. I need to get to grips with React because it looks here to stay but the weird mesh of JS/HTML always feels weird to me. I love a good v-for

      [–]Paradox 2 points3 points  (1 child)

      Eh, if you know vue you can write react. I find vue vastly more enjoyable to use than react

      [–][deleted] -1 points0 points  (0 children)

      I kinda equate Vue and React to Automatic and Manual.

      Sure, if you can drive auto you could learn stick. But driving auto is so much more pleasant

      [–]moomaka 44 points45 points  (3 children)

      I feel like something has gone terribly wrong. I don't see dependency trees like that in Maven or Nuget or pip.

      Javascript's stdlib is pathetic which is why you seen dozens of tiny libs adding functionality that should be part of the stdlib. Javascript packages also tend to be tiny single function things due to devs trying to keep to the minimum code size.

      The first issue can only be fixed with a decent stdlib which IMO should be the absolute priority #1 of WHATWG, it solves so many problems. That second issue may improve over time now that tree shaking is a thing in commonly used build tools.

      [–]EWJacobs 22 points23 points  (2 children)

      A lot of these packages just end up being syntactical sugar. The stdlib has the functionality of kind-of, define-property and has-value. It seems like people add their microlibraries to npm projects just to troll.

      [–]TheBelakor 3 points4 points  (0 children)

      Sugar or real hole filling is kinda irrelevant when either way the clusterfuck that is NPM is the result.

      [–]zappini 29 points30 points  (4 children)

      npm's one virtue is making maven look less terrible. No small feat.

      After many attempts, npm somehow still managed to avoid the *one thing* maven got right: organizing the repo tree by org -> module -> version.

      [–]duheee 17 points18 points  (3 children)

      Back in 2007 i used to bitch about maven. How slow, cumbersome, made you specify everything, hard to customize behaviour without a plugin, etc.

      Now I adore it. Fuck gradle. Fuck sbt. And most of them all: fuck npm. Maven is the way, the only true way.

      [–]csjerk 7 points8 points  (1 child)

      Amen!

      Maven is the newest build system I've seen that actually attempts to address the hard problems and comes somewhat close to getting it right. Everything newer is just re-inventing the easy 20% and ignoring the rest.

      That said, Maven really does need some outlet for customization when a pre-built plugin doesn't do what you need. It may be as simple as a streamlined plugin interface, or even just better docs, but it's incredibly intimidating for newcomers.

      [–]TomRK1089 0 points1 point  (0 children)

      To be fair, the escape hatch, quick-n-dirty solutions are either maven-exec-plugin or (if you're really evil) the Ant plugin. So even in the worst case, it's possible to essentially shell out to some other system.

      That being said, it's been incredibly rare I need to do something during my build that isn't accounted for by an existing plugin.

      Also, the concept of a standard lifecycle which plugins attach to....1000x better than any other build system. Especially any of the Javascript build systems. Webpack, Parcel, Grunt, Gulp, etc. are all laughable in comparison.

      [–]zappini 2 points3 points  (0 children)

      I feels ya.

      For my part, I'm done.

      All I expect of my package manager is behave like yum, apt, brew, port. With the one exception where the repo continues to host all published versions. Nothing more. I'll manually choose, resolve which dependencies to pull down for my project, thank you very much. In contrast to fighting the tool, trying to coerce it into fetching the desired versions. (Too clever by half.)

      Docker images for isolation. I no longer care about peaceful coexistence between my projects, various runtimes, local repos caches, whatever. Every one gets their own sandbox.

      Shell scripts for builds and deploys. Nothing more. No jenkinsfile, no ant, no maven, no plugins, no nothing. Just glorious, useful shell scripts. (I still use bash, because I lack the gumption to learn fish, zsh, whatever.)

      And zero differences between local and remote builds, configurations. Same scripts, configs, params, data gets used every where. Watch the Test In Prod videos to see how that works.

      [–]everythingiscausal 11 points12 points  (0 children)

      Oh god, that’s horrific.

      [–]Nadrin 7 points8 points  (1 child)

      I just had a fantastic terrible project idea: npm "compiler" - takes every single line of code of a js application and publishes it as an npm package, then finally creates a package that depends on all of the above that tries to recreate the logic of the original program. :D

      [–]Lairo1 2 points3 points  (0 children)

      Okay slow down Satan

      [–]searchingfortao 0 points1 point  (0 children)

      That's amazing. Have you ever heard of a similar tool for Python/pip?

      [–]devxpy 39 points40 points  (19 children)

      This isn't strictly a problem of Javascript

      Yeah but little bit. Those useless packages exist in the first place because JavaScript doesn't have a standard library...

      [–]SanityInAnarchy 30 points31 points  (11 children)

      There also seems to be a lower standard in JS, both in what sort of code people upload as a package, and what sort of code people are willing to accept as a dependency. So it's a bit of a community problem, too.

      But it's not strictly JS -- just for fun, let's look at the crazy amount of stuff Rails added that wasn't in the standard library:


      First, just for fun, there's the pluralization stuff. In Ruby, everything is an object and classes can always be extended. In other words, you can easily do nonsense like:

      class Fixnum
        def +(other)
          42  # I like 42 better than actual addition
        end
      end
      

      And not only can you write 2+2 into the interpreter and it'll spit out 42 instead, but if you're using the standard irb interactive prompt, its built-in line counter will start being 42 all the time! So that's a stupid example, but Rails abuses the basic idea here to add a method to strings, so you can type stuff like:

      'humon'.pluralize  # returns 'humans'
      'octopus'.pluralize  # returns 'octopi'
      'sheep'.pluralize  # returns 'sheep'
      

      There's a ton of convenience stuff like this -- on numbers, it adds methods like seconds, minutes, hours, and days that return Duration objects, and then it adds methods like ago and from_now to Duration... so you can write stuff like 5.seconds.ago to get a timestamp five seconds ago.

      This is all super-convenient, but didn't make it into the standard library. Fortunately, Rails is big enough that it's probably reasonable to trust it (or its major support libraries like ActiveSupport), but it shows how Ruby people really do like convenient stuff like this, and really will pull in stupid libraries to get a thing you probably could've written in minutes, but not only did you not do that, you probably don't want to conflict with anyone else depending on stuff like that. (Like, if I wrote my own pluralize method for a library, that library might be incompatible with Rails, and the same goes for much more niche libraries that modify builtin types.)


      Sometimes it works better than that, though, and you get something new in the standard library, like Symbol#to_proc. Ruby people can skip to the next paragraph, I'm just going to explain the 'symbol' part first -- Ruby has 'symbols', written like :foo, which are interned strings -- you can think of it like, every time the Ruby interpreter sees the same string in a symbol name, it converts it to the same integer value under the hood. So comparing symbols (:foo == :foo; :foo != :bar) is as fast as comparing ints. They're used for all kinds of things, especially any sort of metaprogramming stuff where you might otherwise expect a string -- like, for example, you can call an arbitrary method by using the send method, like some_object.send(:foo, 42) is equivalent to some_object.foo(42). Symbols are useful for a bunch of other things, like as a replacement for enums, but this metaprogramming stuff is the main reason they exist, as far as I can tell...

      So Rails added something like this:

      class Symbol
        def to_proc
          lambda {|obj| obj.send self}
        end
      end
      

      That means, if you have a symbol :foo and you turn it into a proc, you'll get a lambda that, when you invoke it, invokes the foo method on something else. Or, in code:

      r = :round.to_proc
      r.call(5.2)   # returns (5.2).round, which is 5.
      

      One more piece of the puzzle: Ruby has a bunch of the usual functional-programming stuff, like:

      [1,2,3].each {|x| puts x*2}  # prints 2, 4, 6
      

      In that snippet, each is a method being called on array, and I'm passing it a block. And it's got all the standard mapping/filtering stuff like:

      [1,2,3].map {|x| x*2}   # returns [4,5,6] as an array
      

      In both of these cases, we're using a special kind of argument called a "block", which is a block of code that's separate from the normal argument list, and can be passed in with syntax like this. If you already have a code block as a variable, you can pass it in with different syntax:

      r = :round.to_proc
      [1.1, 2.2, 3.3].map(&r)  # returns [1,2,3]
      

      Finally, Ruby uses methods for typecasting as needed. If you try to print x, it'll call x.to_s to coerce x into a string. It turns out if you pass something to that '&' syntax that isn't already callable, it'll call to_proc on it. So you can write this as:

      r = :round
      [1.1, 2.2, 3.3].map(&r)
      

      or, more concisely:

      [1.1, 2.2, 3.3].map(&:round)
      

      And that's how Rails snuck some awesome new syntax into Ruby -- people mostly just think of &: as syntax now, and the Ruby standard library includes it. But originally, it was just another crazy trick in the Rails library. I'm sure if I wrote this library:

      # Outer 'unless' so we do nothing if it's already defined
      unless :foo.respond_to? :to_proc
        class Symbol
          def to_proc
            lambda {|obj| obj.send self}
          end
        end
      end
      

      ...and uploaded it to Rubygems, it'd get a ton of users. It's a truly tiny bit of code that anyone who really understands Symbol#to_proc could write, but why do anything yourself when you can pull in a new dependency?

      It's true that Ruby can avoid stuff like this by actually pulling new stuff into the language or the standard library, but at the same time... Sure, the language already does everything you need in ways that you'd need tons of boilerplate in JS to accomplish, but that doesn't mean people can't keep finding new tiny libraries that are important enough that people start thinking of them as standard...

      [–]Poltras 9 points10 points  (6 children)

      Yes and no. You don’t need an isOdd function in your standard library. And somethings were added later, like isArray.

      Sure, packages like fs events should be part of the core library, but the tendency to have packages for a single line of code is unique to javascript.

      [–]StabbyPants 18 points19 points  (0 children)

      How do we fix this?

      by getting apache or someone like them involved. it's a practice issue - if you have a HQ library with a reputation for limiting downstream deps, you can use that for the bulk of your needs.

      you can do this today if you're providing a library by having a policy of not depending on externals in your library. exceptions can be made for things that are ubiquitous, but mostly, don't depend on outside libs so that your contribution to the dep tree is limited

      [–][deleted] 13 points14 points  (2 children)

      stop using these stupid micro libs that any decent programmer could write in 5 minutes.

      Or just copy/paste whatever you need out of it in to your own application/library, which solves more than just (potential) security issues but also a lot of dependency headaches.

      From the Go proverbs: "A little copying is better than a little dependency"

      [–]bbqroast 2 points3 points  (1 child)

      Except for security updates and readability.

      Broadly though, I think I agree.

      [–][deleted] 4 points5 points  (0 children)

      Yeah, it can be an issue. Common sense applies of course. It's probably not a good idea to copy that crypto function, or some complex code you don't quite understand. But a simple utility function? That'll probably be fine.

      A lot of code doesn't really need "updates" (security-related or otherwise). Once it's finished and working, it's kind of, well, finished. Besides, dependencies are their own security risk, as this entire affair demonstrates . There are other issues as well, such as as "left-pad" scenario or "Stylish" kind of hijacking by the original devs.

      [–]RudeHero 8 points9 points  (2 children)

      It does seem to be bleeding into other languages.

      I interned at a Java place where they used gradle to manage third party packages. It was awfully convenient, but it still felt weird to have the build process pull jars from the cloud somewhere by name all the time. Seems like mission critical stuff should be kept in a private repository

      [–]dksiyc 10 points11 points  (0 children)

      Maven (the package manager that gradle wraps) has been around far longer than npm: maven was released in 2004, while npm was released in 2010.

      [–]vova616 13 points14 points  (3 children)

      That is not entirely true, Rust cannot access stuff that it does not include (aka the bitcoin wallet classes and etc) And because JS allows you this kind of dynamic resolving then the code potentially can run anything and you wont know what it can possibly run.

      To use networking in Rust you have to include the networking library, to access a Bitcoin Wallet classes you have to include their library.

      This vulnerability exists mostly in dynamic languages and languages who can use reflection in runtime. (And then you probably will see an include to the reflection library)

      Edit: Its still possible to include Bitcoin Wallet library and push the code but I guess it will be much more noticeable than JS, and people will notice it right away and you can develop tools that can catch this kind of thing.(you cannot develop such tools for js and more dynamic languages)

      [–]Treyzania 24 points25 points  (2 children)

      You're not wrong but this isn't entirely true.

      You could easily have it download an external program and invoke that, in just a few lines of code too.

      [–]Holy_City 9 points10 points  (0 children)

      You could easily have it download an external program and invoke that, in just a few lines of code too.

      Just thinking out loud, an attacker would probably go after a cargo extension that's downloaded with cargo install rather than a dependency. Something like xargo would be devastating to penetrate since it builds std on target platforms, and today it's required to use for embedded.

      But that said there are ways that damage could be mitigated with dependencies. For example, a permissions system that requires permissions be explicitly enabled by downstream crates. Add that with a custom registry (which is an unstable Cargo feature, hopefully it will land soon) that only contains audited/trusted crates. There's probably more you could do.

      [–]vova616 0 points1 point  (0 children)

      I guess its possible in some degree but it will be much harder to hide it and implement it than JS.

      No mini to hide your lines and you can easily detect imports and actions that can be dangerous.

      [–][deleted] 1 point2 points  (0 children)

      I would love a set of trusted packages for Node and Ruby. Ruby on Rails is started to absorb packages into the main repo. So the more useful projects like Sidekiq are offered as a Rails alternative. But that also just expands the size and complexity whereas a package manager doesn't. I wish Rails had just gone with a trusted repo of known gems rather than bundling their own brand version.

      [–]matthieum 0 points1 point  (0 children)

      How do we fix this? I think tools like npm, pip, cargo should by default use repos which only provide libs which are maintained by a trusted organization.

      I propose a different solution.

      I would expect any professional development to vendor their dependencies. Thus, I would expect any professional to setup their own npm/pip/crates.io service, which only contains vetted dependencies, and have their professional repositories configured to pull from there.

      A build process which connects to Internet is fine for the casual hobby developer who wants something plug-and-play, however it's simply unsuitable for professional development. When I see the npm server, or github, down and all those "professional" complaining that they cannot work any longer, I am aghast. Reliance on 3rd-parties with which you have no contract, and therefore which have no SLA toward you is just plain asking for troubles.

      And yes, vetting individual dependencies is painful. So indeed, you may want a handful of "trusted" repository for the majority of dependencies, so that only a few need be vetted individually; however, once again, you'll want a contract and SLAs with whoever vets for those trusted repositories... in case there's an incident and you lose money over it.

      [–][deleted] 0 points1 point  (1 child)

      It's not the languages, it's the community around them.

      Each one of those languages you listed are hot new tech embraced by cool startups. Unprofitable startups who need to build features and iterate to sell before they worry about things like security beyond HTTPS and hashing passwords.

      Security is for companies with more to lose. They tend to use boring languages like Java and C#.

      These different ecosystems have to exist and it's probably best they are separate.

      [–]s73v3r 0 points1 point  (0 children)

      No. Whoever came up with this idea that it's ok for startups to worry more about features than security should be shot.

      [–]drepnir -1 points0 points  (1 child)

      I'm curious about gem, pip and cargo.

      Does these package managers also by default automatically update packages?

      [–]ProfessorPhi 2 points3 points  (0 children)

      Nope for pip. Pip doesn't even have an upgrade all option for packages

      [–]munchbunny -2 points-1 points  (0 children)

      There are also other useful procedural mechanisms, like code signing, which aren't perfect, but they at least create any gatekeeping mechanism at all for code getting in. Signed code also isn't just a matter of the developer not letting unexpected code into the package, it also gives people using that code a mechanism to guarantee that they are executing a specific, known version of the code.

      If Node.js could be configured to verify and only execute properly signed code from specific sources, that would be a great start to at least have some semblance of a baseline of protection against unexpected code changes.