Algolia – Our search partner. Algolia’s full suite search APIs enable teams to develop unique search and discovery experiences across all platforms and devices. We’re using Algolia to power our site search here at Changelog.com. Get started for free and learn more at algolia.com.
OneMonth.com – One of the best places to learn how to code…in just one month. If you’re interested in taking your career to the next level head to OneMonth.com/jsparty and get 10% off any coding course.
Fastly – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com.
Standards & Opening the Black Box
- TC39 on GitHub
- Myles Borins
- Daniel Ehrenberg
- Maggie Pint
- TC39 proposals
- The TC39 Process
- How to join ECMA
On the distribution of stakeholders
On the Boundaries of the Spec
- Amal’s talk on ASTs for Refactoring
- Babel parser
- Dojo upgrade tool (using ASTs)
- Awesome AST
Alright, Kball here. I’m here at Node+JS Interactive with my buddy Nick Nisi…
We are here with Jory Burson, the standards liaison at Bocoup, and Amal Hussein, the open platform–
Senior open web engineer.
Senior open web engineer, as well as a veteran podcaster herself… Welcome!
Hi, thanks for having us.
Thank you, hello.
Yeah, that’s a great point, and a point I hope to make in my talk tomorrow morning. We’ve kind of used the word web standards a bit as like a dirty word, in a way… There’s a lot of history behind it, it’s not exactly positive; you know, you kind of conjure up some bad vibes… But it’s really changed quite a bit, and I have to give a lot of credit to some folks on TC39 specifically, like Myles Borins, Daniel Ehrenberg, Maggie Pint… Folks who’ve done a lot of outreach and bringing people into the fold, opening up that sort of black box for how decisions get made, how people can participate, that kind of things.
I think the tides are turning a little bit, maybe perception is changing - at least we hope, because it’s actually quite a bit of fun, and we’d love to get more people involved… That’s a big theme for the TC39 folks at this conference.
Yeah. So do you wanna open that black box a little bit for our listeners?
Sure, yeah. So what can I share…? Some things that people might know - we meet about six times a year (TC39 does), but a lot of the work happens in the open on GitHub. You can go to our GitHub pages and track the proposals, track the conversation, see what people – or bring your own proposal to the table, that’s another possibility. All of that is pretty openly documented, and that’s something that we really wanna emphasize.
At the meetings themselves a lot of debate happens, a lot of discussion, presentations about different kinds of proposals, and just sort of analyzing where might there be problems, where do implementers think that there are gonna be issues, and then providing the feedback.
[00:04:06.13] It’s a process that takes a long time, and that’s something that I think can’t be overstated. A lot of folks get these ideas, they get very passionate about it and they wanna see it happen now, but we can’t work at that pace when we’re talking about changing the web. The web is a long-term platform, so we have to think about change management for it in a very thoughtful, pragmatic way.
The process does involve a lot of conversation, and getting a lot of people to think about it, debate, provide use cases, write tests… So that we know that the decisions that are being made are the right ones.
There’s a proposal process that your listeners might be familiar with, where we have different stages. There are four stages; we start down at the very bottom with “Hey, this is an idea. Wouldn’t it be cool if…?” and it can take a while to go from that stage to the final stage, stage four, which is everybody agrees that it’s the thing to do. We have multiple examples of use cases, we’ve shown that it’s not or maybe it might introduce a regression, there’s tests for it, but once it gets to that stage, it’s going to be in the next edition of the language, which is now produced on a yearly basis. So that can take a long time, and there’s a lot of stuff that happens in-between, but it’s very important.
You mentioned the GitHub, tc39/proposals. It’s something that I have bookmarked and look at very regularly. I don’t know the details of how things move, and that’s one thing I was gonna ask… You said that you meet six times a year. Do proposals move between stages only at those times, during those meetings, or…?
Yes, absolutely. They do a lot of discussion and work and championing and thinking and that kind of thing on the issues between, but it doesn’t advance until the committee meets in person, looks each other in the eyes and says “Do we all agree that this is ready for the next stage?” That’s really important, and I know that we wanna work in an asynchronous fashion and everything, but sometimes what it takes is everybody getting in the same room and saying “Yes, we’re really on the same page about this.”
“Yes, we can.”
“Yes, we can.” [laughs]
You said anyone can introduce a proposal…
But then does it take somebody from TC39 to champion that? Is that how it works?
Yes. Currently, Ecma International is the standards body that produces the specification, and members of Ecma companies can come and participate in the TCs. I do wanna make a pitch that Ecma membership is actually really affordable, and it’s a pretty easy-peasy process to join… So if participating in standards work in a formal capacity is something that you’re interested in, I’m @jorydotcom on Twitter, let me know; I’d be very happy to help you with your application, or whatever. Easy stuff.
And that’s what it takes to be there? Because we were talking about there’s all this in the open, but what Nick is saying - all the decisions are happening in this room with everyone. What does it take to be in the room where it happens?
Two things. One is that you join the member organization Ecma, and the other is that you can be an invited expert, and we’re actually really open to that as well. So especially if you’re thinking about joining, let us know, because we would be happy to invite you to come see what it looks like in person.
[00:07:58.23] I think it’s worth noting that it’s a three-day event; the meeting has grown from maybe ten people – it used to be less than ten people working on the standard, and now I think the last meeting we had like 60 people. We’ve gotta know who’s coming, basically… It could grow to the size of a small conference if we weren’t careful, but we do want to invite folks who are interested in participating to come do that as an invited guest, especially if you’re thinking about joining… Or if there’s an area that you would like to present to the committee about, we’d like to hear from you.
Yeah, I’d love to chime in here… The TC39 are kind of put on this pedestal, and to some degree rightfully so, because it’s really hard work, but it’s also really a self-selecting group of people. It’s really tedious work, and the output is lovely. We have wonderful features like async/await that are implemented in all browsers, that work… But it takes a lot to get there, and I think one of the things that really makes the TC’s decisions very cumbersome sometimes is that there’s multiple stakeholders, the first stakeholder (and the most important one) being the users of the web; the users of the web platform are very important, and we cannot break this for them… So backwards compatibility is important, making sure that implementations of new features don’t break the web, any significant portion of it anyway. So that’s one stakeholder.
Another stakeholder is developers - people that are writing and using these APIs, these primitives. So does this make sense, is this intuitive, does this fit in with the overall language design?
Another very tough stakeholder is implementers. Can V8 implement this in a way that isn’t gonna be a huge performance burden? And what does it mean if all the implementers were on board except for one? So there’s multiple stakeholders that really drive home a really hard bargain, which is why you might see a bunch of things in proposal stages, but the reality is things that actually make it into the spec for real and make it across the finish line - it’s a very small percentage of that… And rightfully so. It’s really important to think about what we put into the platform, because once it’s there, we can’t undo it. A good example of that is even just like symbols that we choose; there’s literally just a limited number of keys on a keyboard, and when we’re looking for new symbols, we have to be really careful about picking that symbol, because once we pick it, it’s gone, it’s forever taken.
So there’s a lot of really hard decisions, and it’s really humbling work. Being at Bocoup and being exposed to standards a bit more closely now, as somebody who was kind of just, you know, a standard web developer, and who’s now kind of working on the platform more closely, it’s really humbling and it’s very insightful.
Amal makes a great point - sometimes what we have to say no to is more important than what we say yes to… And the nature of it is that we’ll maybe get a dozen or more proposals for something, and maybe just one thing is ultimately what makes it through, because you can’t say yes to everything; that’s tough, and that may be part of where the negative press comes from… But it’s important.
And I’m okay with that personally, because the web is a public utility, it’s something that’s for everyone; it’s how people pay their bills, it’s how people communicate, it’s people’s only communication medium sometimes, if they’re unable to communicate physically otherwise… So we can’t break it, and we have to be careful with it.
I’d be interested to hear a little bit – Amal, you mentioned there are these different stakeholders, and there was some recent interesting discussion in the CSS world about the distribution of stakeholders and how it tends to be biased towards implementers… Is the same thing true in TC39? What is the balance of representation across different stakeholders?
That’s a great question. The nature of it is we could come up with a dozen different specifications, but none of it matters unless it gets implemented. So the power is really in the hands of the implementers. But we have quite a few folks from companies you would be surprised to hear - Airbnb, PayPal… There are a couple smaller companies like Bocoup, Tilde, for example… So I would say that [unintelligible 00:13:15.06] is another one. If I were to give it a percentage, I might say it’s 50/50. I feel like it’s 50/50. But the latter voices are still the implementers, because ultimately if they don’t put it in, it doesn’t matter…
So let’s go back a little bit and talk about the staging of things. I think there was some interesting stuff we were talking about maybe covering, in terms of when do you jump in on something, what do the different stages mean, and what happens if you start using something before it’s fully solidified? …that sort of area. So what are those stages and what do they imply for developers?
Yes. I’m a conservative person, I’m fairly risk-averse, so my advice to people is don’t implement it until it’s actually published… But generally, if somebody wants to implement something in stage four, that’s okay, that’s a safe bet. Usually, if it’s stage four, it’s not coming out; it’s going to be adopted in the final draft of the spec. It’s when you start looking at things that are younger than that, less mature ideas, that you get into the super-danger zone… Again, the first stage being “This is an idea. You might want to implement it just as a test case, a proof of concept.” By stage three we expect there to be tests, and some really solid use cases… And stage four - that’s when you’re in the safer zone.
[00:16:18.10] What’s the process for making decisions about what’s moving from stage to stage?
What happens is somebody will author a proposal, and they will bring it to the meeting; a champion brings it to the meeting and just presents. There is discussion, and at the end of that discussion the champion might ask “What do you guys think? Do you think it’s time? Can we move it to the next stage?” And the committee will say yes, or “No, we want to see more XYZ…”
Then the champion will take that feedback, or they’ll get their green light to move it to the next stage, and then they have more work to do. That all happens on the champion’s timeline, or whoever has authored the proposal; they move at the pace that they can move, that the group of people who feel like this is important can move. So it’s not a timeline-driven sort of thing; it’s about effort. And I guess it’s time-intensive in like how much time does it take to get people to the table, to get the idea seasoned, if you will.
And to get that green light, is that a majority vote? Is that a unanimous consensus? What does that look like?
We’re consensus-seeking, so a lot of the times the question is “Does anybody oppose moving this to the next stage?” and a lot of times the answer is yes.
So really anyone has a veto power, almost?
Yeah. Everybody comes to the table as peers, so anybody could say “I’m not comfortable with that”, and that’s just an opportunity to discuss the issue more acutely. The debate is very valuable, and people saying “Hm…” is not a bad thing. It’s very healthy.
You mentioned tests are part of that, and I think Amal you’re a big part of that… Can you describe what that is, what the process is around tests and how that affects a proposal of moving through the standards process?
[00:20:10.13] I’m really proud to say that my teammates, Rick Waldron… Rick is technically my boss; he’s my boss. My boss, Rick Waldron, my teammate Leo Balter, are maintainers for the project. It’s a small body of maintainers… And I’ve contributed to that project as well.
Is it a custom testing platform? It’s not like Jest, or something, right? [laughter]
It is not Jest, but it is Jest-like, in the sense that it’s a light, very thin test harness that’s used for the development purposes, but every implementation has their own runner and harness and all of that.
I think contributing is a great way to get familiar with the spec. If you really love tests – Amal loves tests, I love tests… If you really love tests, it’s a great way to learn the specification as well.
That’s really cool. I honestly did not know that there is something out there that I can contribute to.
You can contribute, and you’ll learn a lot for it… And it’s a really wonderful gift that will outlive you and all of us, because the web is here to stay, hopefully…
It also sounds like it’s a pretty low barrier way to get involved, because these things are so tiny and atomic that if you only have a little bit of time and you wanna learn about a little bit of the spec, you go, you write 3-4 tests, and you’re in the door. You’ve started onboarding and you’ve started contributing back to not just open source, but to the entire platform of the web.
Yeah. I actually wanna make a comment about other ways to get involved, beyond joining and going to these meetings, which are laborious… The test are a great way; contributing use cases, trying out some of the implementations in your own projects and providing feedback on that are all really effective data points. There’s a couple of different efforts to improve documentation, and to get more feedback from different groups like educators, for example, like “What can we be doing to make the language more accessible to newcomers?” Those are all really important things to be talking about, beyond “What new crap can we put in this thing next year?” [laughs] Which is fine, but also we have a lot of crap, so… [laughs]
That’s really good. I didn’t consider getting those perspectives. That’s a really good thing, to hear that that’s happening… More than just getting features and syntactic sugar, and making my life easier, but actually “Does it actually provide benefit beyond that?”
[00:24:02.19] I’d be curious to hear a little bit about what are the boundaries of what is and is not specified? One thing that came up in a recent JS Party that we were talking about was error messages, and the fact that the same error can have different error messages across every different browser. Where are the lines that are left up to the discretion of the implementer?
That is a great question.
Yeah, I have so many thoughts on this…
Okay, go for it.
If you’re listening to this and you’re an implementer, we wanna talk to you. [laughter]
Why are your error messages…?!
Yeah… I don’t know, Jory - is there anything else that you wanted to add to that?
No. I mean, I think you nailed it. It’s one of those things where for a lot of people on the committee there’s a desire to specify further, and then there’s a lot of pushback the other way, and in the standards process itself it’s all about that push-pull between like “Web developers want X, and implementers need to focus on Y, so what’s the harmonious balance we can reach in between?” That’s why it takes so long sometimes.
Yeah, it can take multiple committee meetings for one little thing to advance just partially, let alone – because everybody comes back with… You know, lots of really smart people breaking down a problem and finding the holes.
Yes. The most glorious bikeshed you’ve ever witnessed. [laughter]
I wanna make a community service announcement, actually, since we’re talking about standards… This is something that I know Jory is extremely passionate about as well, but please, please, please stop using proposals in production…
I primarily write TypeScript, so I kind of go by that as the – the new features, when they get added to that language, then I can start using them. Segueing into your talk a little bit - tell us a little bit about what you’re gonna be talking about at the conference.
Yeah, sure. My talk is on machine-powered refactoring, so leveraging ASTs (Abstract Syntax Trees) to basically push your legacy code and the web forward… Your legacy code, and consequently the web forward.
The idea is that the web is an evolving, moving target, and we have - as Jory so eloquently pointed out - new specifications that are being added to the language… So the language itself is shifting, the web APIs are shifting, and new things are getting added; dependencies are getting changed, versions are deprecated, and you have code that you also want to deprecate. So we have all these changes that we need to make, and we need to really get comfortable with automating that process, and doing it in a safe way, and using tools that are better than RegEx, or simple Find and Replace.
There’s nothing better than RegEx… [laughter]
Yeah, RegEx… If you say it three times, my brain will explode, so please, careful on the number of…
RegEx, RegEx, RegEx…! Halloween! Maybe a Beetlejuice will show up. [laughter]
Well, isn’t there a statement about that…? Like, any problem, you’re gonna eventually reduce it to doing RegEx? Anything dealing with text…
And then you have two problems. [laughter]
I mean, they’re so powerful up to a point, and then you can get yourself in so much worlds of pain…
Basically, ASTs are this powerful thing that helps – so when your code is basically processed by a language parser, it first gets broken down into a tree, and that tree is this predictable data structure, that then you can use to safely make changes to your code, and do an in-place regeneration… So those are the things that I’ll be exploring tomorrow.
There’s a lot of tools around that, and there’s even lots of ASTs… Do they all have similar data structures that they eventually get converted into, or do they all have their own subtle intricacies between them?
That’s a great question. There is an effort to standardize ASTs and tokens in general, but currently there’s a few different competing versions. Depending on what parser you’re using - Esprima, Babel, Acorn etc. - they’re all a little different, but as long as you’re using the same one to make your changes continuously throughout your process, you shouldn’t have any issues. It’s when you’re mixing things that you kind of get into trouble. But that’s a great question.
What are the problem domains in which someone should look at doing that, and where does the line come where you say “Okay, this is not actually easy to do from a machine. We need to get a person involved”?
Yeah, that’s a great question. If you have a change that is (I would say) queriable, and that’s a finite and scoped problem… For example, we change the API of this class, which is now used all over our codebase, to take an extra parameter; so now I wanna refactor across the codebase safely - how would I do that? That’s a good use case for an AST.
You can also leverage - in AST-based transformation - error handling, or adding in a code snippet or something like a marker for something that you wanna maybe do manually. If it’s a 90% match for your use case, but you’re like “Oh, I found something that’s funny”, that’s something that you can also do. It’s a little trick, but you can do that.
Yeah. I have a little bit of experience with this; I work on the Dojo project, and we have a CLI tool and a bunch of plugins for that… And of them is an upgrade tool. We’ve been making some drastic changes as we go, and we consolidated eight packages into one package, between versions. So one really easy transform was “Go look at all the imports, and if they match where this import was, change it to this.” That’s a really easy, finite one.
As we get more and more complex, another example is something where we have a component that still exists, but instead of it being a higher-order function that you would call, that returns a component, you just use it as a component. So I can go through and query the data AST, find out if it’s being imported, what its name is, find that token in there and then see if it’s actually being called or if it’s just being passed as an argument. From there, I can’t really refactor that, but I can output logging and say “Hey, you’re using this here. It’s changed. You should go in and fix that.”
Yeah, and that’s something that I get into, too. It’s an ability for you to query your codebase, and understand “Where does the complexity lie? Where do I have seven function calls that are being made from one function? Where do I have 17 variables being defined?” It’s a way for you to understand what are the patterns of your codebase, and then also take the opportunity to improve your custom linting rules as well.
[00:36:06.01] If you make a transform, or if you’ve deprecated some API or whatever, you can also use ASTs to extend out your linter to be a bit more custom to your team’s conventions. Or you can use it to reinforce this bug from never happening. So if you ever fix a bug, you should ask yourself, “Hey, is there a linting rule, or is there something that I can do to automate this, so that nobody else has to ever fix this bug again? If the answer is yes, then go do that.” Then the next question should be “If the answer is no, then can I write a unit test, so that our build will–” and then if the answer is “No, I can’t write a unit test”, “Okay, then can I write an integration test?” and the answer will definitely be yes.
So there’s multiple layers for stopping regressions from happening, and I’m trying to automate that and make that as binary as possible. It just takes the human emotion and human time out of it, which is really valuable. Does that make sense to you all?
Yeah, definitely. That’s not something that I had really considered. Either I haven’t written custom linting rules, but…
Super-easy, come to my talk tomorrow. You’ll learn how.
Absolutely. I imagine these talks are gonna be published online at some point…
I was just wondering that myself, and I feel like I should know that, but I don’t know that.
There are cameras in the back of the room, so…
Yeah, I think they’re all getting recorded…
So if you’re listening to this, check and see if Amal’s talk is up yet.
And they usually get them up really quick after the event.
Oh, that’s nice.
Are there any tools or references or places to start for someone who is interesting in learning more about ASTs, that you’d recommend?
My talk on YouTube… [laughter]
And my link to my slides, which maybe we can include in the show notes, or something…
For sure, yeah. Send them over.
But there’s also an awesome AST, which is kind of sparse, but I have a bunch of resources that I’ve put together when I was researching my talk, so I hope to add to that list.
If you all have time, I’d love to poke at one other subject… Bocoup is an open source consultancy, which is a model that certainly fascinates me, but I think it’s probably something a lot of our audience may be interested in as well… Can you describe what does that even mean? Open source consultancy, as compared to just an agency, or something else…
In practice today that means we’re working a lot on infrastructure tools, and primarily the test suites, but also Amal’s team… If you wanna plug Gaia really quick…
[00:40:13.23] Oh, yeah. At Bocoup, our model is we have engineers that work on our reliability standards and testing projects with our browser vendors. Then some of us also still kind of do what Bocoup was really famous for, which is we make amazing web apps; but we’ve really evolved from making amazing web apps, which we feel like the community is really there, but we’re pushing the boundaries now with computer vision and augmented reality, games interactive media… So web applications that have all of those components, that are a little bit more niche.
We’re not really working on standard React Redux apps, we’re working on really complex things that really leverage the web platform APIs heavily. Our expertise on these APIs really helps us help our customers, push the boundaries forward with what they can do with their web applications.
To your point about what’s really the difference between that kind of service versus an agency, where we try and serve is in that sort of longer-term relationship and commitment, not just to the product, but also to the technology, and helping companies understand what the lifecycle of that technology is, how to work with it in a healthy, sustainable, maintainable kind of way, versus “We need an application for our car rental company”, which has more of like a marketing and advertisement focus to it. We’re really there for the tools and the teams behind the tools.
[00:42:11.14] Got it. So you’re doing consulting work, but focused on essentially open source infrastructure. “Here’s how you’re gonna be able to use these, here’s how you plug into the web standards”, and that sort of thing.
Yeah, exactly. “Here’s how to think about where your company is, and its transition to maybe becoming more of a tech company.” That’s the thing that’s of concern right now for a lot of companies - “How do we go about our digital transformation (that buzzword), and how do we do it in a responsible way? Because we can’t just go chasing the next best framework, or whatever… That’s not really gonna get it done. How do we need to organize, how do we need to think? What are some of the things we need to learn, some of the things we need to let go of?” That kind of thing.
Last point on this, engineers at Bocoup are very unique, in that we have a strong web developer background, but we’re also really involved with the web platform… And those two worlds - it’s funny, they’re operating on the same thing; web developers make things for the web, platform engineers make the web… Right? [laughs] But they’re two different worlds, and the Venn diagram between those two is small, and Bocoup kind of interestingly fits in both of those worlds.
Yeah, we definitely have tried to do that.
Neat. Anything else you all wanna talk about?
Just how much fun I had getting to meet you two, and…
Yeah, this has been great, and I feel like I’ve learned a lot about standards, and a little bit about ASTs, and now I’m really excited; I hope I can make your talk tomorrow…
If not, I’m gonna watch that recording.
Yeah, on YouTube. Hopefully we can include it in the show notes. And thank you very much for inviting us. It’s been an absolute pleasure.
Thank you so much.
Our transcripts are open source on GitHub. Improvements are welcome. 💚