Notes & Links
The topics covered in today's show:
- Project(s) of the Week: Lesser known JS standards — Abstract Blob Store, Abstract Chunk Store
- Web Assembly is ready.
- What kind of impact will it have on the ecosystem?
- Brendan is worried vendors will mess this up
- Cartoon intro to Web Assembly
- Getting Started with Web Assembly in Node.js
Project(s) of the Week
Our transcripts are open source on GitHub. Improvements are welcome. 💚
Hi, everybody. It's me.
And Rachel White, say hello...
Web Assembly... Alex, tell me what this is.
If this all really happens and people start adopting it, what kind of developer would it affect the most for adoption?
Oh, that's pretty rad. I like that it's giving people more options to create more things. I saw that one of the other things that we have to talk about is getting started with Web Assembly in Node.js... I guess we could just talk about that next. Aside from getting started with it in Node.js, are there any other options that either of you know of for somebody that wanted to take a deeper dive into this and try it out?
Well, even in Node it's actually behind the flag right now. The first thing in this article about getting started it's like, "Run Node with this crazy flag that exposes Web Assembly." I think it's still a bit of a moving target in terms of the implementations that exist right now... Alex, do you know if Chrome and Firefox are doing it from under a flag, or if you have to run them in a special mode to play with it, or what?
I don't actually know... I've never run any Web Assembly; maybe that makes me a bad spokesperson for it, but...
My guess is that you have to run it with a flag. Usually, Chrome doesn't turn on non-default features in V8.
...so you probably have to run it with some kind of flag to really play with it today. One of the announcements that they've just put out on their list is that they feel that the current spec is ready for all the browsers to implement and actually expose. So that' like a sign from the spec community that they think that it's stable enough and now we need people to start exposing it to the world. But that means that it's probably not there quite for people to play with.
Yeah. It's a pretty weird thing... Google has tried this on its own before with NaCl and PNaCl and all that stuff, just like sandboxing C code, or whatever. This is different than that because there is a compile target and stuff like that, but kind of the platforms that currently use things like this are pretty different than the web, so a lot of the challenges that were around, like browsers don't all run the same version - they're versionless, essentially; the web just updates as it goes, so everything old has to continue to work for a very long time, and then you can only add new stuff, and the APIs need to be open, everyone needs to be able to implement them and they need to work cross-platform... So there are a lot of challenges.
I'd be very interested to see the benefit over -- I'm sure they have some benchmarks, I haven't seen... But just like "Here's an asm compiled thing... Web Assembly vs Assembly.js" - how much do the primitives and stuff that are added to this stack help? And I don't know what the fallback story is... Do you compile to asm and asm.js? And then if you don't have Web Assembly, it kind of falls back to asm.js? Does that make sense?
[00:08:23.14] Yeah, I think that we should back up a little bit and explain asm.js a little bit more.
Right. But there are a lot of V8 people that disagree with that pretty heavily.
Right, so there are V8 people that disagree with that approach, but what they essentially did was that they made their interpreter really fast for all of the use cases, including that very small subset.
Right, they just didn't need the comment; they were just like, "Why don't we just always make those fast and detect them?"
Then also I dug up the flag... It's only in Chrome Canary, not the other things, and flags enable Web Assembly in Firefox Nightly - it's in the About/Config - and then there are only preview versions of Microsoft Edge, and Safari has their "We think we'll support it in the future" type status page, but no one's ever seen it.
We are so far away from that, though...
Yeah, for a hot second.
Right. Well, I mean... Yeah, I struggle to even see a future where that's viable for an alternative... Like if you want to run Ruby on top of Web Assembly. I definitely see it for, like you were saying, WebGL and Canvas, and people who want these lower-level abstractions for, say, doing math. They're probably gonna get a lot out of the optimizations, because they can work at that really low level and they don't need a lot of higher-order dynamic optimized.
They're just jealous.
While you two were going super in-depth as to what is going on, I was reading more of Lin Clark's awesome cartoon intro to what makes Web Assembly Web Assembly, and what is it... If anybody else doesn't know what the hell it is - like me - you should go read it, because it's really good. And it also makes comparisons in a really easy to understand way. I think that diagrams are great, and Lin explains it very accessibly.
[00:16:22.29] Yeah. I haven't had a chance to really dive into it yet, but there's a module spec in Web Assembly as well, so I'm gonna have to dive into that and see what that means.
Yeah. I mean, it's definitely a lot simpler than the ES6 module spec. But because it's so much simpler, I actually have a harder time figuring out how it fits into the spec.
It sounds very early on in the... It sounds like the first course you take at Stanford for programming language.
It runs everywhere.
Yeah, that's great. I can tell you as someone who has a computer science degree, who did Java in his first two classes, that I spent 50% of my time understanding Java - which is fine... If you're learning Java, you should try to understand their primitives, and all that stuff... And then the other 50% of the time trying to run Java and set up my environment to the point where I could run Java.
Yeah. I mean, the tooling we were given in my class had some of this built into the editor we were supposed to use, but the ease of use of dev tools isn't amazing. There's still plenty of people I see just like "alert()" debugging but the ease of use of dev tools is certainly easier than running a strace against some native program. The tooling has accidentally become much accessible, so I think you're absolutely right that debugging could be a skill that accidentally benefits from this.
And even to get to debugging, you have to render a program. And running your program in Java -- ugh! It's like 20 minutes to get the VM spun up etc. It's not really made for that quick turnaround time. I think between Node.js and Python there's not a huge difference. I mean, there is a big difference in start time if we're talking about microservices, but for development and workflow, they both run relatively quickly compared to Java. And this is why Java developers have these giant IDEs that are sort of like trying to run their code while they're writing it, so that they don't have to try to run it on a command line and see if it failed or not.
Back when I was in school, I'd have a problem that I hit a bug, and you'd search for the bug - this was true of PHP back then too, but you know, I'm an old man now... But you'd search for the bug and you wouldn't find someone solving it, you'd find a web page that's their contact form returns that error as the page, or whatever... [laughter] It had been indexed, but it was actually just an instance of the error occurring, not a solution to your problem. [laughter]
[00:24:19.07] Unrelated to that, one thing I'd be interested in is how much it matters that people start with a dynamic, non-typed language. I feel like I'm in no way a purist when it comes to functional programming or typed languages... I pretty much thing that you can JIT your way out of all of those problems a lot of the times, and that typing is often overhead that I don't want or need. But the fact that I was forced to do types means that I had that option, if I come across a use case for it... So I wonder if they should do TypeScript or Flow, at least for some of their projects, like "Alright, run flow on this one and type all of your different things."
I'm actually reading deeper into this, and they're revising multiple of their CS courses. One of the that they offer is Computer Science and Social Good, which is kind of rad... They let the students see practical applications of what they'll be making, so they can get a better insight into what they'll experience once they leave school and actually start a development career, which is really great.
It's California. [laughter]
Oh, come on...
Yeah... I mean, I'm not hugely surprised of this move just because I've heard of... Feross Aboukhadijeh who does WebTorrent - he went to Stanford and he's been back to do random stuff with students there. I know that Guillermo Rausch who created Socket.io has done additional summer courses there, helped them with teaching and stuff like that.
Yeah. To come back to your typing point, though... [laughs]
Well, when you learn a dynamic language, you learn about types... You just don't learn about static typing. You know what a string is, you know what an array is, you know that they're different, and then you also have to learn these coercion semantics, right?
Yeah, but you don't learn about function overloading or pattern matching on arguments... There's a whole world around types that I think is worthwhile to learn, even if you don't agree that you want to be doing that for your types of programs. I think types catch a lot of errors that I don't have, or that bubble up anyways. If I send the wrong type to most of my functions, they will throw an error; they'll just do it in runtime. And as long as you test them, most of the times you can catch those if you have good coverage, so types have dubious value there.
At Stripe we use flow against a huge portion of our React codebases, and we get very free, good documentation out of that. "Here's a component, here are all the props, here are the types of the props, here's a little generated component builder", to where it will give you the dropdown text boxes to fill in the props and then you can preview exactly what things will look like that are running in the code.
I think types enable some very beautiful things outside of just build time checking of enforcement uptypes. So whatever... People should learn types if they're getting a CS degree, regardless of whether they...
[laughs] I think that you should kind of explain a little bit more about flow type in particular, but also the differences with typescript and traditional typing because it is a very new approach to typing. I don't think that I've seen the flow type thing in any other language or any other toolchain before. I actually really like this approach to both learning how types work and where they can be used, and also this very kind of iterative/additive value mode, rather than kind of an all-in or all-out mode.
One major difference is that TypeScript allows you to fully -- actually I think Flow allows that, as well... They both allow you to fully externally type things, so you can actually just reference code from a separate type file, and say like "This function over there is this", so you actually don't even have to markup your code any differently. But there are compilers where if you wanna say, "Here's a function, and the first argument is list of people, and that is the type array where the elements are an array of strings", and then the next element is a boolean, whether you want them to do something.
[00:32:04.20] So you can have these little type definitions inside your code, and then whenever you call a function somewhere else in your code and you send a string to the place where it expects a boolean, before you ever run your code, these little checkers can NPM-run Flow and it'll check to make sure that everywhere that you're using an API, you're sticking to the types...
The fundamental property of the web, that everything can run on current browsers and anyone kind of agrees to that, we end up in a situation where a lot of things are optional and could be done to parts of code bases... So different teams at Stripe use Flow to different degrees and to different levels of requiredness.
One thing that Flow and TypeScript add to your experience though is generally better editor/IDE environments - TypeScript especially, because Microsoft makes TypeScript, Facebook mostly makes Flow... So the TypeScript bindings into their Visual Studio Code editor are very strong, because Microsoft has had years and years of experience writing strongly-typed editor things to where you can refactor every location of a call to some API, or jump to different areas or automatically generate things and get code hints and all sorts of things, because of the types; the Flow bindings in Atom are pretty good as well, but probably not as polished as the TypeScript ones.
Yeah, if you look through Visual Studio Code, you basically come to these typed definitions for not just all of Node Core, but for most common NPM modules, like Request or Express, and stuff like that. They define the whole API there, so that you can get all kinds of crazy, nice editor stuff.
Yeah, there's an open source project - Flow-typed, or something like that... There's ones for each of the things. So there are open source libraries that don't use the typed languages, but whenever you integrate a third-party library, you want the types for that. So all they do is they maintain a third-party type definitions for popular libraries. Some the actual team does, the Node bindings and some of the most popular modules, but there's actually an open source thing where you can submit the types for a library that you don't run and just say, "Hey, these might be helpful to someone." At Stripe, since we use Flow and we use some third-party things, we can also pull in someone else's third party type definitions of that, and then whenever we use that library, we get all of the niceties from it.
There was a lot of arguments for a long time about adding types to the language, and we've pretty much given up on that.
Yeah, I think that's dead because of these.
Well, when ES4 died, those died.
I personally like types much better for documentation and people-related benefits, like IDEs and stuff like that, much more than I like it for safety and speed. It seems like everytime we think something about safety and speed is true with types, someone on the V8 team shows us that we're wrong. If what I just said is incorrect, please don't send me hate mail, but...
[laughs] It's true, except sometimes it's the Firefox team; sometimes it's also the SpiderMonkey people. Okay, I think we're about ready to have another break now. Right after the break, when we come back, we're gonna get into the featured project of the week and some of our picks. We'll be right back.
Today I have two projects; one is called Abstract Blob Store, I believe... I lost my notes. [laughs] Yeah, Abstract Chunk Store. These are by Max Ogden and Mathias Buus, and they're building out a bunch of stuff...
You're talking about storage, and you lost the information?
[laughs] Yes, exactly.
It's eventually consistent, okay?
[laughs] I can throw database jokes all day. So the idea here is that --
Let's catch that idea for now.
Alright... [laughs] Okay. Anyway, you're just way throwing me off today... I should have known not to go toe-to-toe with puns with Alex, but that was a huge mistake. Anyway, so we have these two libraries, Abstract Blob Store and Abstract Chunk Store by Mathias Buus and Max Ogden. They work on the Dat Project, so they're doing a lot of open science, open data stuff, and they're storing a lot of stuff all the time.
They're in the Dat Project?
Yeah, they both work on the Dat Project.
Explain that to me...?
Dat is a small tool -- I mean, I guess it's kind of growing in terms of ecosystem, but it's a toolchain for open scientists to share data, and to manipulate data and then share those manipulations. You can think of it like Git, but for data and for open science. To be honest, Max has been working on a way to get people to share data and their manipulations of data since 2008-2009, and continued to try to build stuff that was higher-level, and eventually I think figured out that what was really missing -- you know, he wanted GitHub for data, but you need Git first before you can have GitHub, so Dat is basically Git for data.
[00:40:16.10] Yeah, it wants to make people collaborate with sharing data more, too. I think they have a Knight Foundation grant too, so they do a bunch of cool stuff.
They actually have a bunch of grants, yeah. You can go to the RFC podcast on the Changelog network and there's a podcast on Request For Commits with Max where he talks about the grants and how to get them and how to grant-fund open source.
Wow, that's awesome. Max actually was clean-shaven when he started on the Dat Project and he said once people finally adopted it he would shave, and look where we are.
He's not shaved.
Yes, he's got a big beard. [laughter]
Anyway, so to come back to these lesser-known Node Standards... These are really cool, really powerful stuff that developers can work with, because if you just wanna store chunks of data somewhere, you could just pick up a library for S3 for the exact kind of storage mechanism that you want, but if you wanna future-proof your code a little bit or if you wanna expose a module to do something (to do this behavior) and you don't want them to necessarily require it to run in something like S3, you can use Abstract Blob Store - or Abstract Chunk Store, depending on your kind of use case. Then the actual underlying storage mechanism is completely abstract and people can throw in their own.
This is a really good way to build out an ecosystem of good modules that are storing data without locking them into a particular vendor or a local file system, even.
How does it compare to something we already have, like IndexDB, or something like that? The primitives are different?
It's a little bit lower level than IndexDB... Although I don't think that's the right term. It's doing a lot less than IndexDB, because it's not doing any kind of sorting, it's not actually indexing anything.
Is it persistent?
Yes. Well, yeah, you assume that Abstract Blob Store is persistent. But there is a set and a get, and when you set something, you assume that you'll be able to get it later.
How async is it?
Some of these are actually in memory as well, so they don't persist indefinitely. Some of them don't.
But I think that there actually is a good corollary here with IndexDB. In the Node.js ecosystem and also now in the browser with stuff like PouchDB, these tools aren't built to IndexDB, they're actually built to what's called abstract LevelDOWN or LevelUP. So the whole LevelDB ecosystem built these kinds of abstract standards really early on, so that if you relied on level-up for your set get and for all the stuff that you do with IndexDB or with LevelDB - if you relied on LevelUP, you could actually swap out the underlying level up implementation.
So people wrote some in the memory, and people wrote them to work in the browser, and people wrote them to work on top of local storage, and SQLite... So eventually, PouchDB actually moved over to LevelUP...
I was gonna ask...
Yeah, so they could take advantage of all those underlying data stores. Cool stuff!
Have there been any cool demos, or anything like that? Or is it just mostly early days...?
I think these standards end up getting buried in the things that people are actually building, right? There's some really cool IPFS demos, and IPFS uses Abstract Blob Store internally, and Abstract stuff. In fact, there's an IPFS Abstract Blob Store that they expose to everybody else too, so you can use that as an upper-level storage mechanism. But even the underlying storage mechanisms where they store their internals also uses this abstract store. So those are some good examples.
[00:44:11.17] I think that the biggest demo of all this stuff is probably the Dat Project and the stuff that they're building.
Cool. I think they'd get more traction if they didn't name it Abstract Blob Store.
I'm thinking like UltraStore... That's pretty good. [laughter] Honestly...
Yeah... The thing that really gets traction is the thing that's built right on top of this. LevelUP got a ton of traction, and because of that traction, a lot of people implemented abstract LevelDOWN stores, even though it was called Abstract LevelDOWN.
Well, LevelDOWN in itself is actually a pretty clever name, right?
Anyway... That was quick.
That was good. Now we can really spend I think a lot of time on our individual picks, so why don't we get into our individual picks for the week?
I'll start, since I think Rachel might be having some connectivity issues. Mine isn't super long, since I've already kind of talked about it... Mine's gonna be the flow-typed - it's a repo that lets people commit to, and it has the Flow type definitions for maybe a-hundred-something open source projects: Backbone, Bluebird promises, Request, Chalk, Chi... All these different things that you probably use.
This allows you to immediately jump into Flow in your code, even though you might not have done any of it on your own. So you can literally just pull in Flow, not type anything of yours, and then use an editor that can handle this stuff - Atom Plus... There's some Flow integrations in Atom that are pretty good. And then instantly, once you start typing stuff around the express bindings that you pulled in or the request bindings, you'll start getting function argument completion and all that stuff, without you doing anything.
This project alone could get you better editor experience for things that you know the least, because, like, there's something you didn't write, and then maybe it'll save you some time, like looking up documentation. I assume Lodash is in here; let me check to make sure... Yeah, Lodash 3.x-4 is in here. So if you, like me, can't ever remember all the different things in Lodash, it'll be a really great thing to just add this one thing and then instantly you have type definitions for every single Lodash function everytime you try to write one. I think that's super nifty.
Also, a fun fact that I forgot to mention earlier is that Flow, the actual thing that runs to check your code and find the different types and find the bugs in your type is written in OCaml, which is my vote for what Stanford should use as their default language that they teach. [laughter]
One of the guys on my team found a bug in the invitation of Flow, and he wanted to fix it, so he had to learn how to run OCaml and get an environment set up and submit a patch that way, and I just thought it was very funny, because I didn't know people liked OCaml until very recently.
[laughs] Rarely people that are really into typing. I'm looking at the Request definition in here, and it's really funny because so many of the Request functions take dynamic arguments... They'll take different types and then do different [unintelligible 00:48:04.24]
Right, so there's all of these "declare any", "declare any" types... It's pretty funny.
At least you would still get the name in your autocomplete stuff. So even though it wouldn't give you like, "This must be an array", but it'll still give you useful things, even just with any-s, so it's not the end of the world.
I'm here. You go first, though...
Okay, I'm gonna bring up Offline Camp, actually. There's this great little community; it's called Offline First, but they're really handling a lot more than just offline web use cases. It's a lot of the people from the Hoodie community, and PouchDB and stuff like that... But they're really digging into not just offline, but also with Fuzzy internet and what you need to deal with peer-to-peer web... There's a lot of overlap in all these use cases.
There's this new community; the organizers of the community have just been phenomenal. They've been doing a great, great job organizing, and documentation, and getting people involved, and that's included these offline camp events which are really small, really intimate, and you kind of go off and stay in some house or mansion that they rented somewhere... Because of that, it's a really limited number of people. This one that's happening in Berlin in late-April, early-May is only 30 people, but I highly recommend it. I think there's an application process because there's so few spots, but I greatly encourage anybody who's getting into offline or peer-to-peer or anything like that to apply and get involved in the community.
Does that bump up against any of the JSConf EU stuff? I don't know when those dates.
Yeah, it's right before JSConf EU.
Okay. Do it all.
Well, not right before... It's like the weekend before.
But Berlin is pretty fun.
Also during Eurovision... Oh, but you'll all be offline, so you won't be able to know who wins...
No, I mean, there is internet at the camp. [laughs]
If you need something to do in that week in Berlin, there's this really good Vietnamese noodle place called Monsieur Vuong that I would suggest you go to. That's my actual project of the week.
They throw star anise in their pho, and it's really good. That's a good spot. There's also an amazing dumpling place that has this dish called Stripes of Beef, which are just these thin slices of beef with -- I don't know what they're doing with some kind of... It's like Sichuan pepper and a few other things, and some chili oil. But it's one of the best dishes you'll ever have.
Yeah... I actually think that's a translation error; it's actually tripes of beef, and those are intestines. [laughter]
I think they meant strips of beef, and they put an 'e' in there because they actually did mess up the translation a little bit...
I actually don't remember anything special that I ate when I was in Berlin last time, so I'll have to listen to your advice this time.
Eat Döner Kebab at four in the morning.
Yeah, I guess you didn't hang out with me enough if you didn't have any amazing food, because that's pretty much all I do there.
I've never been in Berlin while you were there, so...
Right... There we go. Are all three of us gonna be at JSConf EU, actually?
Yeah, I'll be there.
Did you say yes or no, Alex?
I have a human child to take care of now... It might be another year before I get out there.
Okay, so you're bringing the child with you, is what you're saying.
Yeah, that's the plan.
Okay, awesome. Also, JSConf EU I guess is a good pick; it's a great conference. Awesome kind of tent pole event. Rachel?
[00:51:53.15] Yes. Okay, I do have a pick. If you're interested in data vizualization with D3 and other really cool stuff, there's two women... One lives in San Francisco, the other lives in Amsterdam, and they have this project called Data Sketches, where each month they are taking different topics and experimenting with data viz through exploration of how to show information based off of each of those topics. I saw one of them speak in January - Shirley Woo is one of them, and the other woman is Nadia Bremer (I hope I'm saying those names right).
It's just really interesting to see all the different ways that you can take data and have it be informational. They have seven months so far, and they each have the same topic for each month, except they take it in totally different directions. One month they picked books, and what one of the women did versus what the other woman did is completely different, and it's super cool to see the differences of how they made stuff. I'll post the link in the chat.
Awesome. There have been so many good sketches and drawings of rad stuff lately. Mariko killed that again with some sketches about SHA1; those were great.
Yeah, I'm a fan of this trend.
It's definitely positive.
There's so many people that are more like visual learners, so I think it's... It's more interesting than just looking at a pie chart, or a bar graph, especially when you are utilizing D3 and you're able to make that information interactive so you can see data sets changing over time, or how certain information is relational towards other things that you have in your set. It's really awesome.
I think OpenVis Conf is coming up too, and they just exclusively deal with this kind of stuff.
Also CSV Conf is coming up. We were talking about Dat and visualizations, and that's actually a nice intersection. CSV is also a bunch of cool visualizations, like OpenVis Conf, but it's also about small data, basically.
Yeah, I think CSV Conf is 2nd to 3rd May, and OpenVis Conf is April 24th and 25th.
Sweet. On that note, we'll leave it there. Rate us on iTunes... Thank you, everybody.
Our transcripts are open source on GitHub. Improvements are welcome. 💚