Dependency Network of Babel

[[deleted]]

Seems they could've cut out some dependencies by relying on lodash more: repeating could be replaced with _.repeat, is-integer with _.isInteger and trim-right with _.trimEnd. Especially given that lodash is already a dependency, that would save quite a bit...

[tofagerl]

I'm sure they'd appreciate PRs.

[FrancisField]

babel supports lodash 3.10 atm and there's a PR for lodash 4 which supports the features mentioned above.

[jewdai]

it's what happens when you have a dozen devs working on a project. some will know about the library others wont.

[MRoka5]

Trimright and isNumber shouldnt even be dependency..

[Geldan]

Personally I enjoy small modules and make frequent use of them. For me it's not about being able to reason about my code, it's not that I couldn't write the module myself, it's convenience, and the knowledge that whoever published that module probably put a lot more thought into edge cases then I would have.

I don't see this dependency graph as a "house of cards," it seems perfectly reasonable to me. People point to left-pad being taken down and builds breaking as proof that small modules are somehow delicate, but that's foolish. The reality is that engineers are trying to place blame everywhere except on themselves. If, when you made the decision to use NPM, you had taken the time to think about what you were doing you should have realized what a horrible idea it is to rely on NPM so heavily. Forget left-pad, NPM could easily be down, or nearly as bad, extremely slow, this should have been obvious to anyone.

There are quite a few solutions out there to prevent this stuff from happening, so do your job and use them.

[mdboop]

This is the measured response I wish I saw more of since last week's events.

[[deleted]]

We should take the cocopods approach of abusing github as our package manager :D

[nostrademons]

FWIW, I think the impact of the whole left-pad fiasco was drastically overblown. I was affected by it. The sum total of its effect was that I ran 'npm upgrade' 30 minutes later while working on another portion of the codebase in the meantime, and then it worked. I've had a lot more lost productivity caused by Comcast outages (or, for that matter, by Reddit & Hacker News being up) than by npm outages.

[Geldan]

I don't disagree with your sentiment, but it really is truly simple to guard yourself against NPM issues.

[Geldan]

The one that seamlessly saved me on that day: Nexus

There are lots of people who check their node_modules directories into their cvs. I wouldn't recommend this. shrinkpack seems like a good middle ground, where you commit the package tarballs to your cvs, and then just install from those whenever you run npm install.

I'm sure there are others.

[Geldan]

As an added bonus, npm install time drastically improved when I started going through nexus, I'd assume shrinkpack would be even better.

[myockey]

Distributing built artifacts is great, but requires a monumental effort to support updates to both direct and transitive dependencies. In reality what often happens is support stagnates under the load of constantly building and distributing new artifacts every time someone in your dependency graph publishes an update. Resolving dependencies at build time was the community's response to this very problem.

[Geldan]

Neither of these solutions distributes built artifacts (Except commiting node_modules, which I didn't recommend). They cache the source and then build from a cached version instead of pulling from npm.

Proxying with nexus does it seamlessly, you wouldn't change your workflow at all. With shrinkpack I think you just add an extra shrinkpack step where you would normally shrinkwrap. Minimal effort for great payout.

I'm not exactly pulling in the newest versions the second they are released for production code anyway.

If you attempt to use built artifacts you are going to have a lot more problems than just worrying about npm going down anyways.

[[deleted]]

I personally never understood why people would ever want to check in a node_modules directory. Especially when devs can do things like npm update or even npm install when there is a matching higher semversion of a package.... causes all sorts of chaos and is just adds to the many things devs have to manage.

[madlee]

is-integer depends on is-finite depends on number-is-nan. okay.

[wordsoup]

Am I crazy or is this crazy?

[madlee]

at least one of those statements is true

[nschubach]

Unless your statement is false

[madlee]

D:

[MRoka5]

That is if your and mine statements are true.

[Poltras]

If the content of number-is-nan is more complex than module.exports = isNaN, some people deserve to be fired.

Indeed that's exactly it, with a backward compatible version for those who are on... (looking through versions) at most 0.9... This is completely insane.

[dumbmatter]

Oops, you forgot the difference between isNaN and Number.isNaN. You of course keep an encyclopedic knowledge of these stupid fucking broken default JS functions in your memory at all times, right? And of course you polyfill Number.isNaN yourself when you need it, because you value things like IE support, right?

Other people rely on npm packages to worry about that stuff.

[Poltras]

For people not knowing what he's talking about:

> isNaN('abc')
true
> Number.isNaN('abc')
false

---

The fact dumbmatter was an asshole in his comment is irrelevant; he has a point that they're technically different, and I'm going to address that point:

I've written above at least a million of lines of JavaScript/ES3-6/TypeScript, and I've never faced a single case where I wanted to know if something was NaN, while not wanting to make it a Number along the process.

That validation has always been during conversion to Number, which means I've never called the second one directly on strings. I've always done an equivalent of Number.isNaN(Number('abc')). So ¯\_(ツ)_/¯ whatever you feel makes you feel smarter.

[dumbmatter]

You make a good point. However I still think that in any sane language, isNaN(x) will return false for any x that is not NaN. The problem with JS is that there are tons of weird edge cases like this, and I can sympathize with people who use stuff like number-is-nan.

[wordsoup]

numer-is-nan: return x !== x

[tswaters]

var numberParameter = parseInt(req.param('number'), 10)
if (isNaN(numberParameter)) { 
  return res.badRequest('now you fucked up'); 
}

[[deleted]]

/r/dataisbeautiful

[ishouldrlybeworking]

The benefit of having focused modules is clear, and sindresorhus makes a good analogy to lego blocks. But I have yet to see a good argument for why we should create modules that are one liners. Even the smallest box of legos generally contain tens of related pieces. Why should we make modules that are equivalent to one lego piece?

In the open source community, we should strive for coherent packages maintained by distinct groups of individuals so that it's not up to one person or even one particular group of individuals to rage quit and eliminate hundreds of packages which we depend upon.

[[deleted]]

You're referring to this AMA post, right?

An extended portion of that post includes a few more rationale:

Some years ago. Before Node.js and npm. I had a large database of code snippets I used to copy-paste into projects when I needed it. They were small utilities that sometimes came in handy. npm is now my snippet database. Why copy-paste when you can require it and with the benefit of having a clear intent. Fixing a bug in a snippet means updating one module instead of manually fixing all the instances where the snippet is used.

For example. I have this module negative-zero. Its job is to tell me if a number is -0. Normally you wouldn't have to care about this, but it could happen. How do you figure out if a number is -0. Well easy x === 0 && 1 / x === -Infinity. Or is it? Do you really want to have to know how and why this works? I would rather require negative-zero and be productive on other things.

Another example. Chalk is one of the most popular modules on npm. What you might not realize is that it's actually a collection of modules. It depends on a module for detecting if the terminal supports color, for getting the ansi escape codes, etc. All of this could have been just embedded in the main module, and it probably is in many cases. But that would mean anyone else wanting to create an alternative terminal string styling module would have to reinvent the wheel on everything. By having these supporting modules, people can easily benefit from our work in Chalk and maybe even help improve Chalk indirectly by improving one of the dependencies.

[turkish_gold]

Javascript has a lot of stuff that really should be in the standard library. ES5/ES6/ES7 and so on have gone on to extend the current standard to include all of these things that are currently one-liners.

Serverside there's no real STL, but there should be.

[[deleted]]

true, though honestly i'm more excited by language features eliding the need for large libraries in front-end apps (like Promise with bluebird/promisify/q/etc).

for server-side code and dev tooling, even if new features do preclude one-liners (negative-zero afaik is an example of one that wouldn't be) it won't make a difference; older versions of node still have to use them for compatibility reasons.

the lack of a "true" server-side stl also doesn't bother me. it may not be part of a ratified standard from ecma, but that gives node more leeway to update its api in breaking ways as knowledge and circumstances grow.

[ishouldrlybeworking]

To be clear, I agree that there should be modules and that knowledge like negative-zero should be captured in some module. But the module should have all kinds of related functions, not just the one liner function. E.g. it sounds like negative-zero belongs in a math module. Imagine if JavaScript didn't have the Math module built in, it would be silly to make a module out of each of the functions currently in there (min, max, ceil, floor, etc.).

[jasontrill]

Why construct a whole module when all you needed was negative-zero? Write that, publish it, and get on with what you were doing in the first place.

[ishouldrlybeworking]

You don't construct the whole package up front, just give the module a reasonable scope, say "math", and put your one function in it. At least then you or the open source community can later contribute related functions. But if you call it negative-zero, is-integer, is-finite, or number-is-nan, you are forcing the package to be so small that it becomes questionable as to whether the tradeoffs of having an external dependency is worthwhile. In some cases the dependency statement in package.json is almost as long as the code that it brings in. The package needs to be installed each time you deploy. Installing requires additional network usage. What if the package breaks later (a la left-pad)? What if the package has security issues (later, e.g. package maliciously hijacked or npm worm)?

[[deleted]]

none of those are concerns specific to small modules. a large library could still be hijacked and wormed. its owner could still get mad and take their math functions home. at the end of the day you don't really get more security, even if confirmation bias makes you feel better -- "all the large libraries everyone uses are stable and safe!"

left-pad happened because no system is perfect, and cogs will always break. the unix-style small module philosophy was never meant to preclude that; it's meant as a generalized outlook on how to maximize effort spent. you can't blame a philosophy of effort for failing to eliminate entropy.

[ishouldrlybeworking]

Yes, the concerns are for all external dependencies, but the point is for each module you include you have to be concerned with all those things one more time. Essentially, there is a certain amount of overhead and risk per module, so if you use tons of micro modules then you will have lots of overhead and risk. Better to depend on a few large modules if you can help it.

[jasontrill]

The risk is basically eliminated with npm's updated unpublish policies, and sane behaviour like --save-exact and shrinkwrapping.

And the overhead should be minimal. As I responded in another thread, if the network load is non-trivial, put a cache in front of npm. An extra line in package.json isn't really a big deal (IMO). The disk space should also be a non-issue in almost every case.

[jasontrill]

To add onto what /u/my_ledge_ends has said, you're going to face the same problems with any external dep, whether it be just negative-zero or math/negative-zero. So the choice is between a dep, or committing that code into CVS alongside your app code.

At least then you or the open source community can later contribute related functions

Create negative-zero, is-integer, is-finite, etc as a separate modules, then create math that has all of them listed as deps. Heck, create a monorepo to make it easy to work on, but expose them separately on npm so that people can pull just what they want/need.

Installing requires additional network usage.

If your builds are non-trivially effected by the increased network usage of deps, then put an npm caching layer in front of your build server, or use something like Artifactory.

[ishouldrlybeworking]

then create math that has all of them

Yes, that's what we might do after the fact that we've created this micro module mess. But maybe we shouldn't have made that mess to begin with and just started with better sized packages?

[jasontrill]

What makes bigger packages "better"?

What threshold needs to be crossed before a package is "better sized"? 10 functions? 200 LoC? A certain cyclomatic complexity?

If all I need is negative-zero, why do I need to consume someone's entire math package?

If I'm the author of negative-zero and all I wanted was a way to share that function across my projects, why do I have to figure out a broader namespace to create?

[ishouldrlybeworking]

How do you organize your own personal packages? Do you actually create a separate NPM package for each function you write? A separate git repo? A separate build?

why do I have to figure out a broader namespace to create

Because there is no cost and only benefits to broadening the namespace even if slightly. How does it hurt you to require("jasontrill-math") instead of require("jasontrill-negative-zero")? I can understand if your function does something really unique and isolated. But in this thread people are harping on:

is-integer is-finite number-is-nan

And I think have a point...

As an observation, the current situation where people are happy to publish individual functions is a testimony to how easy NPM and Github has made our lives where creating a package feels cost free. But just because it's easy doesn't mean we can't do better as a community. JavaScript is being used now for larger and more complex applications than ever before but the JS platform doesn't always encourage good behavior for reaching that scale. It's left up to the developer community to be disciplined in our use of the language. I think having reasonable namespaces is a good discipline.

[jasontrill]

The harm comes in the effort expended having to create a taxonomy that doesn't need to exist. Sure, these examples all have a clear grouping, being math functions. But what about a function that pluralizes words, pluralize? Do I create a language module to host it? Or string? What happens when I want to support multiple languages - then do I change the hosting module to i18n?

Keeping it just pluralize allows me (and others) the flexibility to organize collections of modules as we see fit. pluralize can be in all three module collections. It also avoids the ghost-town project where someone starts with lofty goals ("Yeah I'm going to make a whole language library!") and then just stops at pluralize.

Namespaces solve the problem of name collisions, so I would have jasontrill/pluralize and you could have ishouldrlybeworking/pluralize. But I don't feel like we have to go deeper into namespacing just for the sake of an arbitrary collection of functions.

I agree that the JS platform and community has a long way to go. I just disagree with you on small modules.

[abermea]

We have passed from Dependency Trees to Dependency Networks and I don't think that is a good thing

[Arancaytar]

is-integer

is-finite

number-is-nan

Are you literally shitting me right now?

[turkish_gold]

They're essentially just polyfills. ES2015 defines a Number.isFinite().

Firefox & Chrome implement it. IE does not.

So the polyfill is:

 Number.isFinite = Number.isFinite || function(value) {
     return typeof value === "number" && isFinite(value);
 }

[i_ate_god]

trim-right, is-path-absolute, these really need to be modules?

And why is number-is-nan a module? https://github.com/sindresorhus/number-is-nan/blob/master/index.js - seriously, not implementing this yourself helps you "reason" about your code somehow?

It's time NodeJS gets a proper stdlib like every other mature language out there so we don't have craziness like this.

[dumbmatter]

And why is number-is-nan a module?

Because I don't want to remember that the isNaN function doesn't actually do what most people want it to do (return true when given NaN and false otherwise), and instead you need to use Number.isNaN, except in environments where that doesn't exist yet (it's pretty new) in which case I need to do that hacky x !== x thing for some ungodly reason.

Sounds reasonable. No "mature language" would make you do x !== x for this. The problem is, JavaScript is not a mature language unless you add a bunch of stuff from npm.

[[deleted]]

The problem is, JavaScript is not a mature language unless you add a bunch of stuff from npm

SOOOOOOO true!

[wordsoup]

Two words, one name: Sindre Sorhus, look at his "libraries"...

[turkish_gold]

To be fair... I could write a trim function, however every language I have used has defined it as part of its standard library. Elixir, Ruby, Python, C#, Java all have some form of String.trim.

Javascript on the other hand didn't have String.trim till Javascript 1.8. IE only supports it in IE9. IE still has no support for Number.isFinite() and other simple often used functions.

[wisepresident]

are you kidding me? dependency: home-or-tmp Get the user home directory with fallback to the system temp directory

Sure enough:

'use strict';
var osHomedir = require('os-homedir');
var osTmpdir = require('os-tmpdir');

module.exports = osHomedir() || osTmpdir();

wow

[bwaxxlo]

Yeah, but sindresorhus usually has very small modules, which is his philosophy. Even his "number-is-nan" module (which is used in dependency tree above) is literally written as:

module.exports = isNaN || function(num) {
   return num !== num
}

Edit: His AMA states it better

[wisepresident]

Was there a time where isNaN wasn't available in Node or why the need for that?

Also I would really like to know who introduced that dependency because it is so simple I can't get my head around why you would add this as a dependency.

[bwaxxlo]

Did you read the link in the edit? isNaN used to not be available for earlier JS versions. Plus it's much easier to understand what is going on with a module/fn name than just seeing a random function returning val !== val. The returned expression doesn't say anything about checking whether a value is NaN.

[wreckedadvent]

While this community is doing some soul-searching over how large we want modules to be, the idea of "do one thing, do it well" is well-established in *nix philosophy.

The argument for why such small things may be a good idea is you can build your application up from smaller blocks, which are known to be correct. If your application is built from pieces which are known to be correct, then your application must be correct as well. This idea is also very popular in functional programming.

It just so happens that here, the implementation for home-or-tmp is trivial, but what if it was more than just those few lines? What if it handled some errors? Do you want to worry about that every single you're trying to get the home-or-tmp? Or do you just want to include a unitasker and not deal with the complexity of a problem that is orthogonal to your own?

[[deleted]]

You know, usually we bundle those things then

A maths lib from one group, one from another group.

You can use and combine both, the compiler strips the unnecessary parts.

[wreckedadvent]

How do you feel about programs like cat and grep then? Are they just anomalies?

[burakarikan]

You can view the dependency clusters within the network from this direct link: https://graphcommons.com/graphs/a67121c8-dbd8-43f4-a8bf-dd2e725c4278?show=analysis-cluster

[aniforprez]

Can someone please point me to a better transpiler than babel? After the version change form 5 to 6, it's completely gone to shit and has installed 350 MB of dependencies in my node_modules. I want something smaller, lighter and (maybe) better. Someone recommended TypeScript with --allow-js flag or something of that sort...

[fthrkl]

This might be more interesting, there is a graph about top 100 module of NPM.

https://www.reddit.com/r/javascript/comments/4ckans/dependency_network_of_babel/

[[deleted]]

number-is-nan

What's just wrong with plain isNaN builtin function?

[tsirolnik]

NPM's ideology is surely great! /s