Aaron Schlesinger: Throw Node in there… [laughter] Throw everything, I guess. Throw everything in there. But what language can you pick up that doesn’t need a VM, that has high-level concurrencies/primitives built in the language, that can do networking in a couple lines of code, that’s got really good support for built-in high-level networking protocols… You know there’s not a lot of other stuff out there. And if you’re three engineers - or however many engineers it was - at Google and you’re trying to build Kubernetes, or… Shout-out to Nomad, because I saw some Nomad in the chat there… If you’re HashiCorp and you’re trying to build a distributed system abstraction layer, you kind of need those things, and you’re gonna get up and running with something like that faster with Go than probably with C++, or Java… Not sure about Node; maybe Node will get you there just as fast… Although you don’t have access to multi-core on Node, so there’s a trade-off there as well.

[00:56:08.26] So I think Go really caught on because it has most of this stuff right out of the box that you need… And it was one of the only languages that had it at the time when a lot of these things were coming up.

Jacquie Grindrod: Yeah, so I started learning it – this was actually my third attempt learning it. I’ve tried to learn it at previous jobs a few other times, and it just really didn’t feel like there was enough support or time to dive into it. This time I started learning it on a live stream. Straight from installing Go, I started live streaming, I went to the document page, and was like “I don’t even know how to run this, but here we go. Let’s get started.” And that was in preparation for my interviews at HashiCorp, because I learned that all of their products are basically in Go.

So I did all that livestreaming, and I was solving Exercism challenges on live stream, which was a bit more of a dive-in than I thought it was going to be initially, because you went straight from Hello World to “I got [unintelligible 00:04:03.08]” which had a lot of things I just wasn’t ready for, like the custom stuff… I was gonna have it open on my other screen to reference that, but…

[00:04:13.19] It was a pretty big dive, and it felt like pair-programming with basically the internet, with whoever wanted to show up, which was pretty terrifying, but it was really nice, because it ended up being a lot of people that already know Go, and who are really friendly… And they end up being really good teachers, because they kind of come by and try to guide you the right way, instead of telling you the answer, which wasn’t what I expected from that experience… So it was really nice.

David Yakobovitch: Right. If I look at an ML engineer as someone who has software experience in building applications, and a data engineer as someone who could already work with cloud systems or distributed systems… And often the bootcamps and the masters programs just don’t give enough there. That’s why you wanna look at capstones for that.

But I think the challenge is there’s so much information to cover and pack into it that data science has just become this term that encompasses the industry. And how I look at it is, simply put, what used to be big data became predictive analytics, became data science, which is now the AI industry. It’s constantly evolving… But the truth is when you look at data science roles, 60% to 80% of the work is still in the data. It’s cleaning data, it’s labeling data, it’s getting it all set up…

I featured actually just at the beginning of August on the HumAIn Podcast Mark Sears, who runs CloudFactory, which is one of the big data labeling companies between and Africa. They have 10,000 just labeling data. I think the reality that a lot of data scientists don’t know until they join the company is you’re not playing with algorithms all day. Maybe you might, but even an ML engineer - you’re gonna be working with APIs, and setting up pipelines and systems before you get to start training and testing and working with other teams to see those results.

So I definitely see a specialization occurring in the field. In fact, I’m calling now a new subfield emerging in data science, which we’re starting to see in some trend reports, called Data Science as a Service. Similar to how we saw Infrastructure as a Service, with things from like HashiCorp, Ansible and Terraform, and a lot of deployment options for the cloud, we’re gonna start seeing that (and we already are) in Data Science as a Service. We’re seeing companies like Neptune and Spell, and [unintelligible 00:19:00.20] and other ones, which have all just recently raised their series A’s, that they’re helping deploy systems.

You even see the founders of Anaconda - a couple of them branched off and launched Saturn Cloud, which is launch these systems in Dockers containers, and now do your data science. Paperspace and Brooklyn got notorious for that, and has been doing a phenomenal job partnering with companies like FastAI, and Insight Data Science Fellowship as well.

Kyle Daigle: Yeah, Jess is 1,000% cooler than I am, and so unfortunately I will leave that to her, but hopefully she can come on. Basically, with Actions there’s a couple of moving parts. You mentioned the visual editor - we make it easy for you to go in, and you can drag and drop and basically say “On a particular event” - and this is generally like a webhook event; so when a pull request is opened, when an issue is opened, when something is pushed, when a ref is deleted, all those sorts of things - I would like you to run an action.

In the visual editor it’s easy to see that you can also parallelize actions. So “I would like you to run these two actions.” Then as each action completes, it will go down the line, and so on. So you can create this interesting dependency list where you can parallelize to two separate actions - “I’d like you to run CI over here, and I would like you to check dependencies. Then if both of those pass, I want you to package this and push it up to the registry”, for example. So the visual editor makes that very clear.

[00:27:48.18] Under the hood, we use workflow files, and the visual editor ultimately goes down to that. The workflow file is ultimately a subset of HCL, the HashiCorp language - if you’re familiar with any of those products, it’s a very similar syntax; it’s not completely it, but it’s a subset. We chose that in part because we wanted to make a workflow file that was very clear to understand, that didn’t have a lot of cruft; it was human-readable, which was the primary concern… So we looked at all the obvious suspects - YAML, JSON and so on, but ultimately landed on this, because we thought it was honestly aesthetically-pleasing, which we thought would be a very important part of how easy it would be to feel these things out as human beings. So that’s the first piece here.

Now, skipping from the workflow file to an individual action, an individual action is ultimately just a Docker container. That container can do basically anything you can fit inside it. You get a single CPU, with some memory, that will take care of what your action is trying to accomplish.

If you look in the open source examples of what actions can do, usually you see the Docker file saying where we’re actually going to run this thing, what it does, so on and so forth. Any Docker container should work; there’s some limitations around exactly what we let you do with Docker, but we’re working with the community to make sure we’re making it as broad as it’s possible. Each one of those gets a couple of things for free; each action gets a GitHub token for free, and it also gets a checkout of the repository that you are working in for free. In this way we’re trying to make it very simple for you to do things like “Run these tests. Go look at these files in my repo, and then call a GitHub API and create an issue” etc.

The Actions environment is ultimately a VM where we’re running these actions on your behalf, and we’re using that VM boundary primarily for the security concerns that we have. We do some interesting things with Docker to make it a little bit more secure and so on, but ultimately, at the end of the day all of this code is going into a VM for us to run for you.

In between all this we have a couple of services that help us do this. GitHub is generally notoriously a monolithic Rails application, but in this instance we have the launch editor itself, which is the drag-and-drop and workflow file piece as its own service… Ultimately, the tool that actually receives these events and decides whether a workflow needs to be run as its own separate service, and then ultimately the tool that is running the code and sort of orchestrating and scheduling it is a separate service as well.

So that’s kind of the very quick overview of how a workflow file visually, or a workflow file-wise triggers these actions based on events, and then ultimately these actions are all run inside of a VM, which has access to the GitHub token, the repo, and then also some secrets if you wanted to put some outside tokens, and we support all that and keep it encrypted as well.

Adam Stacoviak: Including a world with big numbers. We’ve seen the headlines on Changelog.com this week in the news feed - billion-dollar valuations, hundreds of millions of dollars invested into new companies or companies that are now unicorns, HashiCorp being an open core model type company that’s just taken on a new round of gigantic funding… So there’s clearly lots of money at play here, and it’s a new world for open source every single day.

So where do we go from here? Clearly, we’ve had a great conversation, that’s led from not only Dgraph as a tech and how it applies, graph databases 101, on through how they could be used… You’re clearly super-smart; you’ve had to relicense, you’ve been through a journey… What do you suggest may be the next step? Maybe not here today, because we’re getting out of time, but what are some suggestions for you to continue this conversation in ways that are meaningful, that can get to meaningful change? Do we have a conference about it, do we do a Sustain-like unconference, or just kind of a gathering? How can this best be approached by the right people in ways that are not vicious and attacking, but in ways that are meant to actually get to change? What do you suggest?

Erik St. Martin: Yeah, and even if you look at Raft - Raft itself kind of explains how the consensus works, but you take an implementation of that, like etcd or Consul or something like that, and there’s so much more that goes into it… Especially when you talk about having a service that can stand the test of time; you need to be able to back-up the data and compact logs and all these things. There’s so much more work than just that; that’s kind of the root of it, but like you said, there’s so much more engineering and problems to solve around it.

[00:32:03.25] Moving on from performance, another topic you are very into is fuzzing. For anybody who may not be familiar with what fuzzing is, do you wanna kind of give a rundown on what fuzzing is, and then we can get into why you think everybody should be doing it?

Carlisia Pinto: So this week was the first time that I had an if error loop inside another if error loop, and I sort of wanted to report both errors. If there was an error in both loops, I wanted to wait for the second one and then report both. So I was thinking, well, there has to be a neat way to do this, and then I found this package from HashiCorp called go-multierror. I didn’t know this package before. It’s very simple, it lets you do exactly that, and I thought it was pretty neat. It just appends one error to the other, and you can have a bunch of errors just appended, just like you append to a list. I thought that was neat.

Ivan Porto Carrero: Actually, I used to work on it most of the time. I wrote it originally just to prove a point, but then it became fairly useful, and VMware started adopting it as well, so now I have somebody else working on it who is still fixing bugs; I’ve got corporate sponsorship finally, and I am personally much more interested in decentralized, distributed databases, or having – another unsolved problem that exists, but I don’t know how many people are actually confronted with it, is Gossip doesn’t work well. I’m sure you’re familiar with Serf or Consul at the very least from HashiCorp…

Carlisia Pinto: I was really excited that – afterwards I talked to my co-workers and they were also excited about the fact that Vault has a test thing that you can use… As opposed to spinning up a Vault to test your stuff against, you can just have a virtual Vault; we learned that on that talk. But then it didn’t really work out well, because when you call it, you have to import a package, they import a bunch of other packages, and if you don’t mind that, it’s okay, but… They said that’s how it is, basically, so we chose not to use it. But it’s very cool. In any case, there are a bunch of gems in that talk, for sure.

Erik St. Martin: A lot of the HashiCorp tools are magical…

Erik St. Martin: Yeah, Vault is amazing for managing secrets and keys, and rolling them. HashiCorp in general, with their open source stuff, not only do they build cool stuff, but it’s some of the best-documented stuff I’ve ever seen, too.

Travis Jeffery: I looked at that one and I looked at the Etcd as well. I felt like Hashicorp‘s just fit my brain a little bit better. It seemed a little bit simpler. The other thing is that I just like how it’s in its own repo, too. Because when I wanna look up issues for the project, I just wanna see the issues for Raft.

[00:15:55.16] With Etcd, they have their Raft library inside this huge – that’s just one small part of their huge project and huge repo on GitHub, so it’s difficult to find issues for Raft, that are Raft-specific, so that’s another thing that was annoying for me. So yeah, it was mostly that it fit my brain.

The other nice thing was that I read Consul and Nomad - I read their source code to see how they did it, so it was nice to see the same libraries. So that was another thing, it was useful to see how they did it.

Kelsey Hightower: Yeah, Flannel… Pretty much every tool in the stack - Weave, Prometheus, InfluxDB… You name it; almost everything that’s being used, even Fluentd, I believe. So it’s across the board, we seem to just land on tools with little effort just happen to have a chance that they’re written in Go. Vault from Hashicorp… It just goes on and on.

Dave Cheney: Yes, absolutely. I think it’s critical, because if companies are going to invest in the long-term – and they have started. We have Docker, all the CoreOS products, all the HashiCorp products, Kubernetes - there’s an investment. But to make that investment pay off in the future, it can’t just be just smashing out code as fast as possible. There has to be some fundamental design so we can change the code in the future, as the requirements change.

Brian Ketelsen: Yeah, I looked at Buntdb this morning when you posted it into the show notes section and it looks pretty interesting. The more interesting part for me was that they built it specifically to be a storage backend for Raft, so it meets the Raft storage interface or Hashicorp’s Raft implementation. So you can use Buntdb as a Raft store, which is kind of slick.

Erik St. Martin: Sweet. So for me, I’m gonna thank HashiCorp. Particularly I’m using their LRUcache this week, that they have available, but many times before, Brian too - Vagrant, Vault, Consul… There’s so many of their tools that are useful, so I’m just gonna blatantly say HashiCorp.

So we encourage everybody else to thank their open source projects through Twitter or any other social media. Reaching out is often just a good thing as Brian spoke to in I think episode one… Just getting that comment from people makes all the difference sometimes.

So with that said, I think we are out of time unfortunately, but it has been quite a fun episode. We definitely want to thank Bill for coming on the show with us. I know myself I’m gonna be digging through more of the stuff he’s got in the training material, because I’ve got tons of free time, too. Right? All of us?

[laughter]

Cory LaNou: I don’t think there’s ever been a time that I’ve gone into source code, like the Go raft library - that’s a pretty sophisticated piece of code that HashiCorp wrote. The first time I went in there I’m like “Oh, it’s written in Go, I should be able to understand this”, and I pretty much got in there and I’m like “Wow, maybe I don’t know anything about Go at all.” It’s just a common thing. I think we definitely should caveat… When we tell people “Hey, go read the source code. It’s out there, and you’ll understand it”, it’s definitely both the domain knowledge. It just takes a little while. Sure, I understand all the syntax, that makes sense to me, but I don’t understand what it does. That’s just a matter of spending time in there, and what I typically try to do is I stopped going to the source code first, I started going to the Go docs first. Because that gives me the overview of the API that that library’s gonna have, and that starts to give me a sense of how it’s gonna be stitched together, and then from there once I read the code it tends to make more sense. But again, that’s getting that domain knowledge, but you get it really fast from the Go docs, and you don’t get that very fast when you’re reading the code.