Gauge – Low maintenance test automation! Gauge is free and open source test automation framework that takes the pain out of acceptance testing.
DigitalOcean – DigitalOcean is simplicity at scale. Whether your business is running one virtual machine or ten thousand, DigitalOcean gets out of your way so your team can build, deploy, and scale faster and more efficiently. New accounts get $100 in credit to use in your first 60 days.
Algolia – Our search partner. Algolia’s full suite search APIs enable teams to develop unique search and discovery experiences across all platforms and devices. We’re using Algolia to power our site search here at Changelog.com. Get started for free and learn more at algolia.com.
Click here to listen along while you enjoy the transcript. 🎧
Okay, Kball here. I’m at Node + JS Interactive one more time. I’m here with Nick Nisi…
…and we are talking to Laurie Voss, co-founder and COO of npm.
How’s it going, Laurie?
Pretty good, it’s a great conference so far.
We’re excited to have you with us. You gave a talk yesterday afternoon… Do you wanna tell us a little bit about it?
The two sources of data that we used were basic registry stats, so we can tell if people are downloading more stuff or less stuff on a per-package basis, and we also ran a really big survey of 16,000 npm users, and asked them directly “So what are you doing?”
That’s a really interesting point. So are there surprising examples of things that people actually are all using?
[03:53] There are tons of surprising examples of things that everybody is using. I think the one that’s been getting the most gasps of surprise was the TypeScript stat, which is we asked people “Are you using TypeScript?”, and 46% of people said yes. 46% of npm users is well over four million people, so that’s a surprising usage space for TypeScript.
We were not expecting the result, and as a result, the question was kind of vague. Using TypeScript could mean that you are using some modules that are written in TypeScript, it could mean that you are writing TypeScript, it could mean that someone in your team is writing TypeScript and you have to use it, but you hate it. So when we run the survey again, which will be in a month or two from now, we’re gonna be asking that question in a lot more detail, but at the moment, 46% is a lot; you can’t ignore that. That’s not just the Microsoft crowd, that’s a whole bunch of people.
That is shocking.
That is really neat. One of the things I’ve been seeing is there’s a lot of cross-talk across frameworks, in a way that there didn’t necessarily use to be… Angular, React, Vue, Dojo - they’ve all kind of consolidated down to a bunch of best practices, and then they have kind of unique takes on those.
React Router, which is the most popular router framework for React applications - it is in fact Ember’s router. [laughter]
Yeah, which is great for the web, that they’re actually – essentially, lots of different experiments going on, but then when something proves out, it gets quickly cross-pollinated across all these different things.
Yeah. That is interesting. I’m curious, for the survey that you did, of npm users, how do you control for predilection to respond to surveys? For things like enterprisy folks versus not, and other factors that may play in there.
Our website gets 10 million hits every 90 days, and we ran the survey for 90 days, so we’re pretty sure that 10 million people saw the survey. I’m a data nerd, so I wrote a 5,000-word blog post about the methodology and all the things that are wrong with the methodology. The ultimate answer is you can’t, but the saving grace there is the survey is definitely biased towards people who would answer surveys, but there’s no reason to believe that people who are biased towards answering surveys would be particularly in one camp or another; there’s no reason why React users are more likely to answer a survey than an angular user. I cannot think of a reason that an enterprise user would be more likely to answer a survey than a user who’s in full-time education.
There’s definitely some bias there, but the bias is universally, across all groups, towards people who have some free time this afternoon.
The survey is optionally nominated. One of the things that we wanted to do when we were doing the survey is share the data with everybody, but unfortunately, as a result of not strictly saying this should be anonymous, people put a ton of really important personal information into their survey responses, so it made it very hard to put out into the world. We had to put out aggregate results only, because people kept typing their email addresses and snippets of code into our survey boxes… We can’t give that away, that’s not safe.
What does the survey tell you about the language itself?
What about the language? It tells us all sorts of things about–
Does it tell you anything about – I probably took it, but I can’t remember actually taking it, but… Does it tell you about people maybe using newer features of the language, or…? It probably tells you that, mixed with data about what’s actually being downloaded from npm; it probably tells you what people are using in terms of like Babel, and other things… Does it glean information about specific features of the language that are very popular, or anything like that?
And there’s all these web developers, there’s these great libraries - we should really make them available to web developers”, and it turns out npm didn’t have to do anything; everybody just figured it out on their own, and Webpack and Babel came along, and now everybody who’s doing web development is using npm to do it… But there’s a fundamental point of friction there, which is that all of these libraries were written in CommonJS, which is the module format for Node. And CommonJS does not work in browsers, it was not designed for browsers; even if browsers supported it tomorrow, it would be a bad idea, because it’s synchronous. It’s not designed to work in a browser environment at all.
I think a lot of it is because the new features come out, and they’re very helpful; a lot of it is syntactic sugar, but it just makes us more expressive with what we’re trying to write… but when it comes out and it gets finalized, and even if you’re waiting until stage four and things that are for sure going to be in the next version of the language, on the web at least, a lot of us still have to support all the browsers, or enterprise versions of more modern browsers that won’t have those features for maybe another year or two.
We have more data than anybody else about what people are using; we should be able to indicate to authors and users, you know, “Only 2% of users can’t use this feature. It’s safe. Or if 2% is too big for you, it’s not safe.” We can give ambient information, and without having to do any kind of hard and fast “You can’t publish that feature anymore” stuff, which is totally contrary to our ethos; we can just nudge people towards “You can use this now, it’s safe. We promise. We’ve got the data.”
Without getting too deep into a debate about the module syntax, do you see that as a way to smooth things out going forward for the web, as maybe more ES module support comes to modules that are published to npm? I mean, it’s not even out yet, but as that goes forward…
My ideal state– I don’t think we’ll ever get rid of transpilation in general… a) Because people always want features that aren’t quite there yet, and b) transpilation often provides efficiency. Bundling is an efficiency gain that is over and above just being able to use it at all. Bundling provides efficiency gains that are always going to be true. It’s always going to be faster that way, because the web is the web.
Yeah. I’m not sure that… I think folks have looked at this – once you adopt a build step, it’s extremely rare that you’re gonna go back, and there are real benefits to having that level of machine processing, and optimization, and various other things in there… But it puts a barrier in place. And if we can just get rid of the barrier piece and say “This is how you go from dev mode to pro mode…”
Right. The history of technology in general is always like “Whatever is the easiest to get started with wins over and above any other features that it’s got”, right? That’s how PHP became the dominant language of the web for a very long time. People were like “Oh, this thing is clown shoes, and it’s not for serious programmers”, and I’m like “Yes, but I can get somebody started on a PHP application in 10 minutes, and I cannot do that with Java”, and therefore PHP became a gigantically installed language.
You raise an interesting point when you’re talking about Vue, and how more and more people are starting to take Vue instead of React, and things like that… What do you see as the lifecycle of a project like React? I’m actually doing some research using your API right now, because npm has phenomenal data on this. I’m looking at what does Vue’s growth path look like relative to React’s growth path. It’s several years later, in a lot of different ways…
[20:12] My favorite example of this is Backbone, which at one point was 1% of all registry downloads, and now it’s an afterthought. Most people don’t think about Backbone, but backbone still gets 250,000 a day, because people are still maintaining a lot of stuff in Backbone.
Well, the WordPress REST API stuff is still using Backbone collections and models… Even in Gutenberg, which is coming out in React, the stuff dealing with data structures is Backbone.
Right. These things are surprisingly long-lived… Although there are a couple of frameworks which didn’t die, and instead they sort of transcended, and my favorite example is jQuery. Everybody’s like “Oh, jQuery is old news. Nobody uses jQuery anymore.” That’s not true. Everybody uses jQuery. jQuery’s primary contribution to web technology was that “the DOM API is a pain in the ass, let’s use CSS selectors to select DOM elements instead”, and that is built into browsers now. The thing that they invented is part of the web now. It’s absolutely bedrock; everybody uses jQuery every day. It merged into the walls and you don’t even know that it’s there anymore.
I kicked off a really interesting thread on Twitter a couple of months ago, where I asked people who have actually built things with Web Components and like it, “Tell me about your experience with Web Components.” I was very clear. I was like, “If you think Web Components are bad and you’re not using them for some reason, I am not interested in hearing about why. Only people who actually use Web Components in production, for something, tell me what you liked about them”, and it kicked off this really interesting sort of mini-conference on Twitter, where a whole bunch of people who are using Web Components talked about their experiences.
What I came away with was even the people who are most enthusiastic about Web Components recognize that they are - I don’t know if fatally, but certainly currently flawed. They liked how fast they were, because they’re native, so they’re very fast… They liked that they had a good backwards compatibility story, because if the browser doesn’t know what a tag means, it just ignores the tag and renders this stuff inside of it, which is a great fallback.
Yeah. And JSX, like syntax, almost did make it into the language, in ES4.
E4X was something that was proposed.
Right, yeah. I don’t know if you ever used that.
I actually had to use that in production. It was a nightmare. It was all of the joy of a DOM API, with a whole bunch of angle brackets for no reason. [laughter]
That seems like a lot of React right now though, too… [laughter]
I have non-data-backed strong feelings about templating languages…
Basically, I believe in the maxim that “All templating languages slowly accumulate features until they are PHP.” [laughter]
Fair enough. [laughter] I like that description. You’re just gonna reinvent PHP… That’s great.
The thing about having been a web developer for 22 years is that I spend an awful lot of time being Old Man Yells At Clouds now. [laughter] I’ve seen us do this specific thing wrong three or four times, so I’m…
I’m done, yeah.
Fool me three times, I’m done. [laughter] I’m not doing it again.
I feel you, for sure. That’s great.
[28:14] Fair point. Yeah, I’m not sure. I think we get a little bit of that as we evolve the language, too. Or creating syntactic sugar for patterns that recur over and over again, to make the way that we’re writing code more expressive for that.
Yeah, context in this has long been confusing, and each iteration just makes it more confusing, because there are more ways to do it, and each have their own subtle nuances.
Right, right. Eventually, you end up with a Perl obfuscation contest, where like “Is it line noise or is it Perl?” [laughter]
I heard a description of most Perl as “It’s a write-only language.”
Right. That’s the problem with sufficient expressiveness - you can express anything, and no one can read it anymore.
There’s a lot of other nuances to the language now too, with async programming, and especially async/await. That’s a really great syntax, and I love using it, but it can be very easy to shoot yourself in the foot with that as well, and not realize that you need to return a promise from this, or return a value in a specific way, and that could just get swallowed up if it’s not – it’s a very subtle bug, that can go unnoticed easily.
Yeah. I mean, it’s a lot better than using actual promises, and the observed reality is that it’s a lot better than callback hell was…
So I’ll take subtle bugs over giant, glaring bugs any day.
I honestly think using actual promises, once again, in the learning stage, is really helpful. I would rather somebody do promises first, and then go to async/await, than try to jump them straight to async/await.
This is one of the problems I have with web technology in general, especially when – you know, I talk at bootcamps all the time, and people are like “So how did you get started in web development?” I was like, “Well, I got started at the same time as web development.” I learned web technologies by having them accumulate around me. I don’t know how I would start learning web development if I had to start today. It’s big, and it’s confusing, and the only reason that I know it is because me and it have been around roughly the same amount of time. [laughter]
It’s really hard to learn, and that’s one of the reasons I’m like “I don’t know…” In particular, I get angry when people say “Oh, people should learn the fundamentals.” I’m like, “I have no idea what the fundamentals mean anymore.” People used to complain at me because I was using PHP, and PHP parsed the HTTP headers for you, and they were like “You need to know the fundamentals of the web. You should be parsing your own HTTP headers, dammit.” I’m like, “Nooo, I shouldn’t…! That would be a terrible idea.”
There’s people who are like, “Oh, if you don’t understand pointer math, you don’t really understand programming.” I’m like, “No one has used a pointer since 1995, my friend.” [laughter] Not unless they really wanted to.
Yeah, I think getting to where you can build something that is cool - and that threshold has changed over time; it’s like “What is something cool?” But the faster you can get somebody there, the more likely they are to actually stick, and keep doing development.
Right. That’s the joy of the web as a platform, right? It’s like, you start programming, and the same day you’ve got a thing. You’ve got buttons you can click, and stuff happens, and a line appears on the screen… And that doesn’t happen with any other programming language. Programming languages that are not the web, you’re like “Well, I worked all day, and if you run this command, it spits out the phrase Hello, world.” Right? But on your first day as a web developer, you’re like “I built a whole app. It’s done. It’s sending emails to people. People are getting genuine value out of this.” The web has so much more power, and it’s so much more satisfying as a platform…
…which is one of the reasons that I describe myself as a web supremacist. I think it’s great, I think it’s awesome, I think it’s better than any of the other platforms. I’m here for all of the platforms, but this is mine.
[36:01] Yeah… I think the fundamentals debate is an interesting one, because I similarly – I’ll push back at folks who say “You’ve gotta do that first”, but I also do think it is important that you keep in mind that there may be fundamentals that it’s worth learning about. Don’t learn React and say “I’m done.” This is not an industry to be in if you don’t wanna keep learning… Because stuff is gonna continue to change, but there’s also layers upon layers – like, the number of abstractions you have at the web is mind-boggling. How many places in human endeavors can you go down 10, 12, 15 levels of abstraction, at each one its own field of study?
Right. The chip designers are like “Everything is a level of abstraction to what’s happening in the silicon”, and we ignore that completely, until “Whoops!”, it turns out that we made a performance enhancement in 1998 and it turns out it made all computers after 1998 vulnerable to timing attacks, and now we’re screwed. That was a real case of the abstraction leak coming up to bite us; we were like, “Oh, well, it’s in the silicon. There’s no way we can patch this…”
But you make a good point, which is if all you know is the frameworks, and frameworks have a lifetime of five years, then your risk becoming like a ColdFusion developer; you only ever knew ColdFusion. And there are still lucrative jobs to be had writing ColdFusion, but only about 50 of them, and… [laughter] Like, that’s fine, but you probably should have switched to something bigger at some point in your career. That’s a real risk for people who get too deep on one framework.
Yeah. The one nice thing with the level of convergence we’re seeing in front-end frameworks is if you get really good at React, picking up Vue is gonna be pretty straightforward… And vice-versa - if you get really good at Vue, picking up React is gonna be pretty straightforward.
Yeah, that’s because there’s really good ideas that are coming out of those, and just being implemented in subtly different ways, but…
Yeah, the convergence of the frameworks is definitely an indicator that this has gone beyond a framework choice to a thing that we should transcend.
Are there any other things on the horizon that you see, of - okay, this is an area that maybe isn’t as visible as the massive web frameworks of React and Vue and Angular and all those things, but where there is some level of convergence or accelerated activity going on?
I think 2019 might not be the year that you put GraphQL in production, but it’s the year that you see somebody else put GraphQL into production, and you end up having to learn it as a result. The growth of Apollo, which is the most popular client library on npm for GraphQL stuff - the growth of Apollo is staggering. It looks like early React. And there’s some real benefits there to people who are starting an API from scratch. I don’t wanna have to use an ORM - it’s a nightmare - but I would like to be able to make complex queries of a data model without having to write a zillion REST endpoints. GraphQL has an answer for that. I think everybody should dip their toe in that water this year.
I’ve seen some interesting things if you have an existing REST API - you can essentially wrap it with GraphQL, and get started dipping your toes in without having to get rid of your old API, or anything like that.
Going back to the data from the survey a little bit - has that transformed npm, the tool, in any way?
[39:40] Absolutely. I think one of the strongest points of data that you can get is competition, and that brings up Yarn. Yarn was Facebook running into a problem before anybody else did. Facebook were writing bigger and more complicated web applications than anybody else, and as a result they were one of the first - they weren’t THE first, but they were one of the first people to run into a problem with semver. When you had 20 modules in a tree and you were using npm’s package.json and you had semver-compatible updates being pulled in automatically, that was fine; the chances of somebody having accidentally released a breaking change disguised as a security release were pretty low. But the average web app these days has 1,000 modules in it. Some of them have 2,000 modules in it, and the chances of nobody in that tree having messed up even once are nil.
Facebook was having every build be broken all the time, so they were like “No, this is untenable. We are locking everything down to minor versions”, and that was literally yarn’s major innovation. I don’t wanna say their only innovation, but it was the big thing that Yarn was doing. It was like “We will lock everything by default.” We’d been considering for a long time, and the adoption of Yarn showed us that we were overdue on that one. We took too long to do that.
In npm 5 we put package-lock in. Package-lock it turns out also has some strong performance benefits, simply because you don’t have to be checking the server to see if there’s anything newer, because you know that it’s package-lock. You just download the stuff that’s in package-lock. Using a lockfile automatically made npm twice as fast, and npm 6 is 20 times faster than npm 4 was…
…because this – it’s a Discourse group called Package.Community, which is a domain, because this is 2018… [laughter] The Yarn developers are in there, the npm developers are in there, pnpm - all of the other alternative package managers are in there, and everybody started sharing code with each other about like “How do we make these things faster?” and they all got faster. The result is they’re all now basically the same speed, because all of the low-hanging fruit is gone. We did everything that we could, and now all of the package managers are about the same speed.
That’s another really nice example of the cross-sharing of information that we’re getting now.
You’ve been in the industry even longer than I have, I’ve been around a while, but it just feels like we’re getting a lot better at the fundamental underpinnings of “How do we make this stuff better?”
I don’t know, I think tech has always had that in its DNA in a way that no other industry has. When a car company comes up with a new way of making their car more fuel-efficient or anything, they don’t go to CarCon and have a huge keynote address about “You too can make your cars faster!” They’re like, “No, we’re keeping this under our hats. We’re gonna patent it, and no one gets to use it forever.” But we do the opposite - as soon as we figure out how to make something cool, we tell everybody exactly how to do it. We’re like, “And you too should build your web app better!”
That’s great, it’s one of my favorite things about tech - everybody just by default shares what they know. It’s one of the things that keeps me motivated about tech. I think we’re getting faster at it, but I think we’ve always been good at it.
Awesome. Anything else you wanna talk about?
Not off the top of my head, no. I think it’s been a super-fun conversation. We’ve hit all of the points I wanted to hit, I think.
Yeah, this has been great.
Yeah, we really appreciate you sitting down with us.
Thank you so much.
Thank you for having me. It’s been a great conversation.
Our transcripts are open source on GitHub. Improvements are welcome. 💚