Ecto 2 and Phoenix Presence

Welcome back everyone, this is The Changelog and I am your host, Adam Stacoviak. This is episode 208, and today Jerod and I are talking to José Valim and Chris McCord about Ecto 2.0 and Phoenix Presence. It’s fresh off Elixir Conf Europe. We talked about our journey with Elixir and Phoenix because we’re building our new CMS using Phoenix and Elixir. We talked about Ecto 2.0, what’s happening there… Phoenix 1.2, when it’s coming out and what makes Phoenix Presence so special. At the tail end of the show we talked to Chris a little bit about some random support questions that came up along the way, so stick around for that.

Our sponsors for today’s show are Linode, Rollbar and Codeship.

Alright, we’re back, everybody. We’ve got José Valim joining us and Chris McCord. Jerod, this is a show we kind of teed up back in February, and basically back in last March when Chris first came on and he influenced us around Phoenix and Elixir; we’ve drank the Kool-aid and we got him back on, and we’re talking about some cool stuff. So what’s this show about?

That’s right… So we had a lot of listeners who’ve requested catch-up shows with past guests. We had José on, like you said, back in February, and at the end of that show you could hear us running out of time to talk about even more, and so we thought, “Well, we’ve gotta get you back on.” In the meantime Phoenix 1.0 has shipped, since we had Chris on back in March of the last year, and 1.2 is on the cusp of coming out with cool new features, so we thought let’s just have a whole party of both of them together. Thanks for joining us, guys!

Thank you.

Thanks for having me.

So we’ve been through you guys’ origin story, no need to rehash on that. If the listeners would like to hear that, check out episode #147 for Chris’ and episode #194 for José’s; we’ll link those up in the show notes.

It looks like you guys just got off of Elixir Conf Europe. Can you tell us about it?

Sure. I’ll start, Chris. So it was a really great event. It was in Berlin, and we had about 230 people. It was really great, because what was really interesting to me was to see how much the community has matured in this one year. Last year we had the Elixir Conference in Krakow and it was a smaller event, and you could say that was, you see much more a lot of people coming to the language, their first contact, and Phoenix already had some traction and people were thinking, “Oh, now I can write my applications with Phoenix, or I can use Ecto”, but they were thinking about the tool, right? They were thinking about Phoenix, or they were thinking about Ecto, and now it was really interesting at Elixir Conference Berlin because we could see that the community was… We had a lot of new people coming and they were at this stage in the adoption or they were thinking about the tools, “Oh, I can write my next project with Phoenix”, but we also saw a lot of people that at a lot of the talks they were like, “I learned this tool. I learned Phoenix, I learned Ecto, I learned Elixir”, but now we have this amazing platform with our machine for building distributed systems and solving the problems that we have today differently.

[04:00] We had more talks about distributed systems, more talks about embedded… So that was really interesting, to see how much the community could grow and mature in just a one year period.

Nice. Chris, anything to add there?

No, I think that that’s a good overview. Just like José said, we’re hearing people actually using Elixir and Phoenix-like in the large… People that work at large banks and other large, established companies that are actually getting Elixir in the door and using Phoenix internally. So it was exciting to kind of see it go from this emerging, hobbyist thing that people were excited about, to now they’ve actually pushed it into their companies and are having a big success with it.

Related to that, there was something that was also really cool. When we are on IRC, for example, Chris and I, when we are talking to people, and then Chris and I, we also talk a lot about ideas, about the future… For example what could happen in future Phoenix versions, and sometimes we see convergence. Chris and I think about some topics, and then someone pings us on IRC saying, “Hey, I have been thinking about writing my application this way”, and then we’re like, “Oh, that’s cool, because we have been discussing about it.” I’ll probably wait to talk about this when we talk about Phoenix later on. It was nice to see at the event also people there giving talks about things that Chris and I were thinking about for a while, but we were only talking between us. Then people would go and present, “Look, I’m already doing this, and it has worked in this way for us, and we got good results.” That’s also very interesting to see when it happens, and it has happened during a couple talks as well.

Very cool. Well, like Adam said during the intro, we have also drunk the Kool-aid, so to speak… José, when you were on last time, in the post-show you asked… I disclosed that we were using Elixir and Phoenix to build our next generation CMS, and you’ve asked why that was, and I gave the lamest answer of all time, which was basically “Because Chris told me to.”

[laughs]

…which is to say that when we had Chris on last March, just hearing all of his thoughts on it, why he built it the way he built it, and a lot of the things that I have experienced as a long-time Ruby and JavaScript developer, and somebody who I make most of my living building and maintaining Rails applications - I got excited about it, and Adam will attest that I get excited about almost everything that we have on… And I’m always telling to whoever it is, “I gotta try that out.”

“I gotta try that. As soon as we’re done, I’m gonna check it out.”

I’m checking that out! And then life happens, or work happens… Often times I don’t get to.

I felt Elm had him pretty good.

I still have Elm teed up. In fact, I’m looking for reasons. But with Phoenix, I actually had the opportunity to give it a shot, and I had a very small need… Basically it was for the Changelog - we have memberships, and part of membership is you get access to our private Slack room. That was all manual, so the membership would come in - we used Memberful for that, and we’d get an email… Or I don’t know if we’d get an email, Adam; we had to just go check it every once in a while…

We’d get an email, and the email would get lost in the system, and then it’d be like, “Let me add a to-do”, the to-do didn’t get done within a day or two, and then the new member who’s not getting greeted properly is saying, “Hey, what happened to the Slack room?” We just had a bad system for it.

Yeah, they email on us then we’d feel bad, and I’m thinking there’s no reason why this shouldn’t be automated. Memberful has a Webhooks API, and basically all we needed to do is take a Webhook post-call and then fire out to the Slack API and bite them into our channel. There’s nothing too tough about it, and so because I had that really small use case I could try Phoenix out. In fact, I shipped it without even learning any Elixir. I was just kind of like banging on the keyboard until things happened.

[08:05] So I just got this really fast little win, because… I mean, I could have got it in probably 20 minutes with the Sinatra app, but it took me just a couple of hours. And in the meantime I shipped it, I felt good about it, and then it wasn’t working, of course. So I went to find out why it wasn’t working, and it turned out that the Memberful webhook wouldn’t set the content type on their post, the application /json. So I was like, “Well, that’s lame”, because now basically the JSON parser was failing to parse it correctly, or Phoenix wasn’t picking it up for the right type. That required me to dig into the framework just a little bit and realize how it’s all wired together, and it also allowed me to see some stack traces. That was surprising, because I’m used to stack traces that are so long that you have no idea where you are and what’s going on, and the stack trace was like six or seven calls through the whole web stack; maybe it was more than that, but it felt like very few. And I was like, “Wow, I can actually see everything that’s going on here. This is very cool.”

What I needed to do was actually just… Because this is the only call we’re ever gonna take, and I don’t care what else happens, I can just force the content type to always be application /json, and so I just opened up the endpoint file and basically wrote my own little plug, and plugged it right into the pipeline, and everything was working and it was kind of magical.

That’s very much what Chris had been telling me about. So that was kind of like my Kool-aid moment. I didn’t dive right into it after that, but I thought “You know what, there’s something here, and I like it.” So Chris, thanks for selling me on it… And now we have some support requests. [laughter] Now we’re here with questions.

Awesome! Let me have them. [laughter]

So just to frame this conversation, we’re speaking with a certain level of… A lot of times we bring this childlike wonder to our conversations, and we’ve been criticized for that sometimes, for not having domain expertise on every topic, and to that we would say, if we had to be experts on every topic, it would be a very boring show, because we would just talk about like two or three topics. But in this case, we do have some experience, so our questions will be informed, to a certain degree. Wow, that was really long-winded. Let’s talk about Ecto.

It’s exciting, though. I mean, that’s a cool thing, how Chris influenced… I mean, just kind of rewinding back for the listeners who listen to this, not only does this show influence the people who listen, but also the people who host it. That’s interesting, to me at least.

That’s awesome. Actually, let’s bypass Ecto a little bit, because you mentioned Elm, right. And maybe Phoenix can also be a good reason for you to pick up on now, because there are a lot of people doing Elixir and they’re also interested in Elm, and I think - Chris will be able to confirm - but I think Phoenix and Now, like Using Elm on the JavaScript side is probably the most popular option today with Phoenix, maybe. I also hear a lot about Ember, and those are the two I hear the most. But a lot of people are talking about Elm, so you can see a lot of really good blog posts contemplating that goes from the beginning to the end, and there were also some integrations between Phoenix and Elm, and Phoenix channels. Do you have news on the site, Chris?

[11:33] Yeah, I have maybe just a teaser… I actually gave a keynote at Erlang Factory this year with Evan Czaplicki, the creator of Elm, and with Elm 0.17, which I think just came out last week, or very recently, there is a new websocket support, so now I want to see - and this is not a promise, but this is a kind of promise - I wanted to have an official Phoenix Elm library under the Phoenix framework organization that’s a channel’s client to Elm with the new websocket integration, because I think we can make something pretty great as far as interrupt goes. But we’re currently exploring that, so Elm is still on my to-learn list, but there’s a couple Jason steeves and sunny scog and there’s a couple of Phoenix core team members that have Elm experience that are kind of exploring what that might look like… So expect more out of that soon.

Yeah, that’s very cool. I’m still looking for a reason to check out Elm and to give a little bit of insight into the Phoenix app that we’re building for the Changelog. It’s very boring, in fact that was one of the reasons why I felt like we could tackle it in this… We have big plans long-term and we have ideas that I think the channel stuff plays into for sure, but in the meantime we’re just kind of replicating what we currently have, so that we can… The big purpose is to have multitenancy in terms of podcast support as we develop new shows, but it’s a server-side rendered content application, so we’re just using the old-school, you know, render the HTML with Elixir and go from there.

That being said, I’ve seen a lot of excitement around using it as an API for Ember and Elm applications, and I think there’s definitely some opportunities there down the road for us to check out Elm more.

Yeah. And there’s nothing wrong with server-rendered HTML. I’ll be the first to say that.

I love it, actually.

Yeah, it’s great when that’s all you need.

Yeah, absolutely.

I’m actually glad that you started using it, because we have a separate Phoenix HTML library, and we don’t get bug reports at all. And I know that people are using it, because for example you just told me that you’re using it here; other people are saying, “No APIs, just an HTML alone but because we got no bug reports for a long period of time I was like, “Damn, maybe nobody’s using this…?” [laughter] But no, it’s just that it’s actually good, it’s working without having problems. That’s a good confirmation.

I will attest to that. I’ll actually say we have support requests, I didn’t say bug requests. I actually have not hit a genuine bug in your guys’ stack yet. I’ve hit into all sorts of little issues with Brunch. If we have time at the end I would like to talk about that a little bit. That was an interesting decision, so basically to tee that up, Phoenix does not have its own asset pipeline that’s written in Elixir, integrated tightly into the framework. It uses the npm community, specifically the default is the Brunch build tool, and it just kind of like lightly couples itself to that, and you can swap it in and out. I follow along on the tracker, the Phoenix issues in the mailing list, just silently watching, and a lot of the requests that you guys get are mostly Brunch requests. In fact, sometimes I’m not sure… I have had a few issues, and I’m like, “Is this a Phoenix issue? Is this a Brunch issue?” I’m not really sure. Maybe we can talk about that later.

Let’s get to the meat of the topics here. José, when we had you on last time, we just touched on Ecto a little bit, and we’ve referenced it in this call… But to give the listeners a bit of information, this is your database connection tool - I’m not sure if you’re calling it an ORM. I know you’ve removed Ecto.Model and have Ecto.Schema, so you’re separating it quite a bit from what people who are moving from perhaps a Ruby on Rails background over to Elixir and Phoenix would think of in terms of active record or these types of other libraries that model themselves after the active record rather than either the pattern or the library active record. That’s my bad way of describing it… Why don’t you describe it better?

[16:06] No, that’s a very good introduction. So that’s one of the big features coming in Ecto 2. We would say Ecto 1 was more of a modeling tool, in the sense that you would define the schemas - at the time they were called models, so you would define this model, and then you would think, “Oh, that’s where I’m going to put my domain logic”, so you would define the functions, then you would have callbacks, and it would probably have a little bit of your domain logic as well.

We are stepping away from that, because we are starting to see a lot of the issues we saw happening elsewhere, with coupling, with callbacks… You define a callback because you want to execute something or you’re going to create something to the database, but there are some scenarios where you don’t want that callback to run, and then you have things like, “Oh, we skipped the callback, or suppressed the callback.” You start going into this old, weird lifecycle stuff, and that’s not the way we should write code. We should not write a code and then try to undo it in some places, in some ad-hoc fashion. I would prefer to write code that are small functions that I can call and compose as I call them. I don’t want to build one thing and start putting patches or hosing it; I want to have a bunch of small things and just call the functionality I need.

Ecto 2 drives a lot into this direction. We want to consider Ecto to be a tool and not what you use to model your domain. Your domain is going to be modules and functions, and Ecto can be for example considered too that it allows you to get data from the database and put it into an Elixir structure. That’s why we got rid of models and now have Ecto Schema; that’s all it does, it just allows you to get data from the database and put it into the structure, and it’s convenient because you define it once and then if you need to use it in a lot of places, you just use this Schema in different places.

But in order to show a little bit more of how you should think about it as a tool, now for example we also made these schemas… Because if you think about the database, it’s just a data source, it’s something that you can get data from. So we say, well, there are a bunch of other data sources that we have in our application. I wrote a blog post - we can include a link, I think - called Ecto that talks about a couple of new features in Ecto 2. One of them is, for example, if you have an API, that API is a data source to your application; you’re getting some data and you’re feeding it into your application that you want to parse and you want to handle it, and maybe parse other data structures the same way you would do with the database. So you can also use this schema to get for example this data from the API and validate and cast it and handle it in a bunch of different ways.

It started to look more like a collection of tools, and they work really well together, but they are not taking over what we think your application should be. Maybe it can be hard to about this, but the blog post has very good examples. For example, imagine that you wanna do a sign-up form, and then the product owner says something like, “You know what? I think we should have first name and last name”, and you’re thinking, “No, that’s a bad idea, because not everyone has a last name.” I don’t want to model my database like that, but I know that the owner is really decided on that, so you’re thinking, “I have this requirement for the UI, but I know how the data wants to look like”, and you don’t want to pollute your database with UI decisions.

[20:13] You don’t want the UI to drive your database, so you start to have a mismatch. Then you start thinking about things like this. You want the email to go to an accounts table, but you want the name to go to some other table. So you have a mismatch between what you want to present and what goes into the database, and the way we typically solve this, for example how we would solve this in Ecto 1 or in Rails is that you would add new attributes to your model, and then your model starts to be this weird thing that has a bunch of fields for this operation, and a bunch of other fields for this other operation. It starts becoming this small Frankenstein; it’s just getting you a bunch of different concerns. Then you’re thinking to just break it apart, and say “Hey, I have a schema for this, and I can handle the sign-up logic and then just get the data and put it into the database.” You can think more properly about all those different data sources and how you can handle them more directly. So that’s one of the things that are coming as a part of Ecto 2.

Yeah. I think Ecto is interesting. It’s definitely a different mindset. I’m very much coming from the active record mindset, and I’ve been an active record lover pretty much from the start; I know there’s a lot of haters… I know there’s a lot of people that like it and see its downfalls. I definitely see its downfalls, I’ve used it for many years, but one thing it does is it makes the simple things really simple.

Some of my frustrations as we started to build out that little toy - I call it toy, but that production toy, Phoenix app didn’t even have any database necessity, but as we began building the CMS, I’m starting to work with Ecto more. At first, I struggled. I’m used to being able to just hop into the console and manipulate data pretty simply in an active record style, and with Ecto there are these different components. Ecto breaks out into a repo, there is a change set idea - these are just concepts and modules, ultimately. So you have changesets, repos and queries… And you talk about composability - you’re composing out this way of manipulating data through these three things, and at first it’s difficult to know how you kind of take the pieces of Plato and munge them together to get what you want. But that started to subside and I’m starting to get it, so to speak. Can you talk through for the listeners these different components in terms of the repo, the changeset and the query?

Yeah, that’s a great question. And there is the fourth one which we were just talking about, which is the schema. They are all involved together.

So the repository is ultimately what will present your data storage. Every time you want to get something from the database, you want to write a database, you want to start a transaction, you always go for the repository. This is very important for us. I like to say that functional programming is about making the complex parts of your code explicit, and it’s very important for me for all this functionality to be in the repository because every time I call the repository I want it to be obvious, because that’s a lot of complexity if you think about what it’s doing - it’s managing connections. You need to serialize data, send that stuff to the database - that’s the TCP connection - then you need to get the data out. Then every time I talk to the database there’s a chance that there can be a bottleneck or have performance issues to your application.

[24:04] So I don’t buy this idea, for example, that all this logic should be hidden behind something like user.save, and that you should not care. Of course, you should care what is happening when you execute that thing. Putting it into an agent that is in memory and sending it to a database, that’s a whole other story, and you need to know about that. So that’s the idea of the repository.

Then we have data, which could be Elixir structures. It’s basically a key value thing, where they keys are defined beforehand, but it can also be maps, it can be anything; you can interact with the repository in different ways. So we have the repository, and then we have the Elixir Schema, which is ultimately just data, just struct and that schema which is just data.

Then we have the query, which is basically - now we have all that data in the database, and you want to slice that data in different ways and get parts of the data, so how do we do that? Then we have Ecto.Query, which is again, just Elixir data that you write the query little by little. So you’re saying, “Look for the posts table”, and then you can call a function that says “I want to get the posts that were created more than a month ago” and then “I want to get just the public posts” or you can compose a little bit, and then you are going to create this whole query thing, this whole structure, send it to the database, and the database is going to interpret that, a SQL for example if you’re using Postgres. So that’s the Query, and you mostly use it to read data from the database, to get data out.

Then we have the Changeset, which is what we use to track changes to the data. So we have the repository, where our data is, and we have the queries to get data out, but when the data comes from the database, we can put those schemes in those data structures that we’re talking about.

So we have now the data memory - how can we change it? How can we say, “Hey, I want to update the title of the post.” How can we do that? The way we do that is that we have a Changeset, and the Changeset - as the name says - contains all the changes that you wanna to do when you talk to the database. So you say, “Look, I want to update the title to this new thing”, and then you’re going to give the repository the Changeset, and then it knows how to convert that to the proper SQL and sent the command to the database. So those are the four main entities and how they act with each other.

You said something very nice at the beginning, which was you are used with the good experience, like… For example, if you’re creating a CRUD application - the simplest application that can be - it’s in the case where the data you are showing is exactly the shape of the data you want to have in the database. That case should still continue to be straightforward; you don’t want to add a lot of complexity to that.

Ecto 1, you are trying to be really like, “Oh, we have those concepts here, and you should use those concepts to do those things, because we’re trying to direct developers to the proper mindset.” But at times people are trying to do stuff, and they’re like, “Oh, this is too hard, this could be simpler. There is no reason why you put this barrier here. And there was really no reason, it’s just that we were saying, “Hey, we want you to hit this wall and then let us know what happens. Are you going to be happy that you hit the wall when you went somewhere better, or are you going to be upset that the wall is there?”

[28:07] So we were able to also take some of those walls; some were good, but some we had to take out. So Ecto 2 improves also these common cases, hey do you why is mapping to what I am having in my database? but as I said in the beginning, it also makes it clear that you are coupling those two different things. You are coupling the why to the database; if that’s what you wanna do, fine. We are not going to force you to define a bunch of different mappers, but you should have in mind that as long as we start to steer a little bit away from these that why these really map to database. We make it really easy for it to break apart, and you should break apart and start thinking about those things separately.

I think we’re hit up against our first break. More questions on Ecto for you on the other side. Specifically, I wanna talk about pre-loading, as well as a little bit more on changesets, and some other really cool things that I’ve been waiting for a database library to do, such as taking constraints that you define at a database level and allowing those to trickle all the way up into human-readable error messages without having to duplicate your work. So let’s take that break, and we’ll talk about those things and more on the other side.

Alright, we are back with José Valim and Chris McCord, talking about Ecto and Phoenix. José, before the break I mentioned pre-loading. You said Ecto 1 had a lot of hurdles in the way, barriers, and some you’ve removed, some you’ve kept. One barrier that I had quite often, and I’ve just learned to work through it and I understand the reason for it, is you won’t automatically pre-load associations on the developer’s behalf. This is something that can often lead to inefficient queries, N+1 queries and such. I feel like this is one of the barriers that you wanted to put in so that people knew exactly and had to explicitly load the data that they want for a particular use. That being said, it also can be somewhat annoying sometimes. So talk to us about pre-load.

[31:44] Yes. About pre-loading, exactly as you said, we don’t do lazy loading. You need to basically say, “Hey, I want this data.” That’s a barrier we’re not changing, because I think it’s very important. There are a bunch of decisions that lead to this. First of all, we don’t have mutability in the sense that you can just call something and then we will load the association and cache it. Elixir data structures are mutable, so we have already one issue implementation-wise. Then the other issue is exactly what you hinted - a lot of the applications I have worked on, they have N+1 query issues. Because things can be lazy loaded, automatically you basically don’t care, and then you see you’re loading a huge chunk of your database dynamically and in a non-performant way.

There are a couple more decisions related to this as well. For example - talking from the other side - we force you to think about it upfront and preload the data up front. That has a bunch of good consequences, which are also some of the reasons that lead us to this. For example, if you have to load the data upfront, then you are kind of like, “Look, I’m loading the data in the controller”, for example, because that’s where we were loading before our call interview, which means that, for example, with same things that would like your views to be pure, in a sense they should just do the data transformation. It receives a bunch of data, for example a collection from the database, and it transforms that data into HTML, and it should not have side effects; it should not write to the database, read from the database and do a bunch of crazy stuff. That makes your views really straightforward, because you’re just thinking about data transformation and the complexities in the controller. That’s a pattern we also wanted to promote, and that’s why we had this decision.

You said about the barriers - that’s one example that we try to improve a little bit more, the barrier, in the sense that having better messages when you don’t pre-load the data when you try to use it, or if you pre-load multiple times… I think in early Ecto versions if you had a post and then you called pre-load comments, and then you called pre-load comments again, it would pre-load it twice. Now we are a little bit smarter and say, “Hey, these things are already pre-loaded. Let’s just use that.” So making the functionality more convenient altogether.

One of the nice things we also did in this release is that if you pre-load all the data upfront… You’re going to say, “Hey, I have this post and I want to pre-load comments, I want to pre-load likes, I want to pre-load this, pre-load that”, so when you specify all the things you want to pre-load, now we actually pre-load them in parallel, because we know all the data you want. We have the post, so we just say, “Hey, I’m going to actually then do four queries to the database, process that data and then put it into the post.” Because we had this idea of having the whole data upfront, it allows that.

Then there are things going a little bit more into the Phoenix direction; there are things that we have been discussing for a while that we could add to Phoenix, which is if you tell the view what is the data that the view needs, instead of just going crazy and doing queries anywhere in the view, we can actually do a lot of exciting stuff. We can automatically cache views, because we know the data it depends on, and then if we can track when the data changes we know that the view in the template needs to be regenerated. It goes over this idea of having My Data and all its dependencies in one place, and not scattered throughout the view, so you can do things like automatic caching. For example, we could automatically send updates. If you say, “Hey, I have this data and when this data changes, I want to recompute this view and send to our clients using channels”, we’ll be able to do that because again, all the data dependencies, the pre-loads and so on are in one place and not scattered throughout the views, and I think the pre-loads play an important part in this whole thing.

[36:10] Let’s take a concrete example here. So we’re building a CMS for podcasts and episodes, and what not… So we have a podcast episode, and so if you think of an episode page, we’re pulling in lots of different data, and this is one of the points where… First of all, it is definitely a nice barrier in terms of as I’m writing the code I’m thinking to myself, “Wow, this is pulling in lots of different data from different places.” So when I’m pre-loading all the things I need for an episode - I pre-load the podcast that it belongs to, the hosts, the guests, the sponsors, the different channels which are like topics, and the links. So it’s pre-loading tons of different related objects or records. Are you saying that in Ecto 2 those queries will be dispatched and then brought back together, so they run in parallel? Is that what you’re saying?

Yeah, exactly. And sometimes those things are nested. You can think of it as a tree, where the pulse is the root and then sometimes you want to get the guests, and then the guest is doing more pre-loads, so that’s like one branch of the tree. Then you want to bring something else, like the likes for that episode. So you can think there is a tree, and what we pre-load in parallel are exactly those branches.

That’s very cool. The one thing I like the most about open source is when my code doesn’t have to change at all and I upgrade and it just gets faster and better… And then you do that times to the N-th degree of everybody who’s using that - it’s a beautiful thing. I love the impact that you can have when you have lots of people sharing the same codebase. Very cool.

One last thing - we’ve mentioned changesets, and I think changesets are really the gem of Ecto; I think it’s a great idea, and I think it’s well realized, this idea that often times you’re taking inputs from different places, and where do you put the information on who can do what; and in the traditional act of records style model, it all belongs to the model, so you have these callbacks or if-statements or conditional virtual attributes in all these things, and with the Changeset, you just have another changeset; so you just have this. Perhaps you have your admin changeset, and your regular user changeset, and that defines what they can and cannot change about that particular schema, which is very cool.

Also, the constraints. So talk to us about changesets and how you can take different constraints with their foreign keys or uniqueness validations from your underlying database and use those with Ecto.

This example you gave with changesets goes really well with what I said at the beginning. For example, if we take active records, you define the whole validations there in one point and maybe they are conditional, and then sometimes you want to execute the validations… As I said, you end up with a bunch of conditionals, because you add the thing and then you need to know how to undo those things. And with changesets, because it’s just a bunch of functions… I can have the admin changeset, I can have the other changeset, and if they both share a common ground, that is going to be a third function that the two changesets are going to call. There is nothing global, everything is constrained in the changeset, and you can introspect the changeset and see what is changing, what validations ran, and you can see everything. It’s very touchable; you can go and introspect there what’s happening.

One of the things that we have there is exactly the idea of constraints. So we have two things with changesets - we have validations and constraints; validations are things that you can run on the data without needing the database. I can validate the length of a string, I can see if it’s required, if it’s effectively there or not, if the user sent a new value… All those things we can validate without the database.

[40:10] But there are other things, like does this association exist if you are inserting a foreign key, or is this email unique? You cannot actually answer this question without asking the database. And if you don’t ask the database, if you implement at the application level, you are going to have duplicate data in your database.

So to solve this problem, for example, “I want this email to be unique, you actually need to go over the database”, so the idea of constraints in changesets is exactly to leverage the constraints that we have in the database. So when we create a changeset, we say “Look, if by any chance the database says that this email is duplicate because of a constraint, I want you to convert it to a nice user message.” So the constraints of the changeset are a way for us to tell the changeset, “Hey, we eventually are going to execute that in the database”, and if the database says that this is wrong, that’s how we are going to tell the user exactly what happened with this exact message. So it maps those two things: it maps your application and it maps your database. In your database you can add all the constraints that you want, and then we can still show them nicely to the user.

Yeah, very cool. Okay José, I give you one last chance on Ecto 2 new stuff. In a couple minutes give us a rundown of other cool stuff, and then we’ll switch gears and talk about Phoenix.

Alright, so we are talking about performance like parallel pre-loads, and that’s one particular case, but overall performance is better because we are now relying on something called DB connection that was made to represent the database connection, so there was a bunch of optimizations of how a connection pulling and kind of stuff works. I don’t remember exactly the numbers, but people are seeing from 50% to 80% faster in general, just queries and encoding, decoding and so on. That’s a nice thing.

I said in the beginning a little bit about the barriers, like we put some barriers because we wanted to force people to do some things. One of the barriers put in Ecto 1 was that every time you wanted to insert some data we were forcing you to use things like changesets to the database, but for insert we actually do not need to track a lot of stuff. If we want to just insert data into the database without creating a changeset, you are supposed to do that. So that’s also something we rebrought to Ecto 2, and we really built on the idea. What you can do today with Ecto 2 is you can do a repo insert and you can define your whole tree of data. You can say - going back to the show example - “Look, this is the episode that is going to have these guests, that is going to have this other information, and I want to have those comments.” Just really build a very deep Elixir structure with all the data, and then when you call repo insert we are going to traverse the whole tree, the whole data and insert it into the database. And what is really nice about this is that now… You need something like a factory system for your data because you know, it builds associations, it does this and does that. Because it’s very easy for it to just say, “Hey, this is the whole tree that I need for my task” and then the tree is really obvious; you just insert it, and Ecto is going to take care of all that for you. So that was one very nice addition. I have a blog post ready for this and it’s going to come out soon, so people can check it out for more details.

[43:53] The last feature, and I think we actually mentioned this in the last episode, was the idea of concurrent tests, even if the tests rely on the database. Elixir always had this feature where you can go to a test case, set “async: true” and then all the cases that have “async: true” run concurrently. I like to say that it’s 2016, everything you do should be using all of the cores in your machine. If you’re not doing that, you are literally just wasting time, because it’s a very easy math to do. You can never parallelize a hundred percent, but assume you can parallelize 80% of your tests and you have four cores, there’s 80% of your tests that their time could be divided by four, and you just gain a huge amount of time.

So we got this idea and we extended it with Ecto 2, so you can now run tests concurrently even if you are talking to the database. The way we do this is that every time you run a test, this test gets its own connection to the database that is inside the transaction. Then you can have a bunch of tests running and all of them are going to talk to the database with their own specific connection, and because it’s all inside a transaction, whatever a test does is not going to affect the other test. So it’s a really cool feature and we have a lot of people using it already with great success, and we have recently also integrated these with acceptance testing tools. We have two in the Elixir community, like Hound and Wallaby. You can kind of drive the testing as if you are a browser, using Titan JS or Selenium, and those tools now also work with these concurrent tasks, so it’s really great. You can have concurrent acceptance tests, and you don’t need to do much really. One line of code that you add to our test helper and then one line of code you add to the setup, and it works, so everything is faster, including your tests. That’s it, I need to breathe.

Well done, very good. I guess one request, José, if you could just get a little bit more excited about this stuff, we could all really appreciate it. [laughter] I’m just messing with you. Well, we have a lot to talk about with Phoenix… Chris, you’re still there, right?

I am still here.

Alright, Chris is still here, awesome! We’re gonna take a quick break, and Phoenix 1.2 - I guess we should probably even catch up with what’s happened in Phoenix, because we were pre-1.0 in our previous call. So we’ll catch up on Phoenix and talk about Phoenix Presence, which looks to be quite an innovative thing coming to Phoenix 1.2, on the other side of the break.

Alright, we are back with José and Chris. Chris, I guess it’s your turn as we shift gears and talk about Phoenix. We had you on in March of 2015, and I think we were pre-1.0 at that point, so we definitely wanna focus on the Presence feature that’s coming in Phoenix 1.2, but could you give us a brief recent history of Phoenix for our listeners?

Sure. I can’t believe it was over a year ago that I was on…

I know, right?

Now I’m trying to think back… So we reached 1.0 in July, so not too far after I was on. I think as far as new features since I was first on, into 1.0, I think it was just about stabilizing things. I’ll give a brief history from 1.0 to where we are now; that’s where you mentioned Phoenix Presence.

Before that, we had some performance optimizations. The whole idea with 1.0 was to get our API stable. We had some benchmarks as far as HTTP goes, and things were looking quite good, but after we released 1.0 we decided to see how our channel layer was doing performance-wise. Channels are the real-time layer in Phoenix; it wraps websockets, but you can also use long polling or other transports, but your server code remains the same.

So when we went to benchmark this, we were only able to get like 30,000 simultaneous users, which was much lower than we were hoping. It was a cool story, because initially it was like “Wow, that’s horrible”, but with a few lines of code change - I think José had the first optimization… He actually removed a little bit of code, changed a few lines, and it doubled the performance, so we got 60,000 connections. That was cool.

We repeated that a few times, where we would change a few lines, end up with a dif that it was actually less code and we would double or triple performance. In fact, our last optimization we just changed one line of code and it gave us a 10x increase in throughput.

Long story short, we will always preach that we have this great tooling for Erlang and Elixir, and that it’s really easy to get a live running look at the system. We were actually able to really put that to the test. We provisioned like 50 servers that would act as websocket clients, all sending connections to one server, because we needed to try to open like two million connections, and we were actually able to get a GUI into our server of like a live-running list of what our processes were doing, what our in-memory storages were looking like, and that’s how we optimized. It was like too easy, so we ended up with a dif that was less code to go from something that supported 30,000 connections to our channel layer that supported ultimately two million connections per server.

That was a really exciting part from after 1.0, where we were able to optimize and get like a WhatsApp scale of two million connections on a single server.

[52:02] WhatsApp was the case study that you cited in our call last year; it’s what got you excited about Erlang and Elixir, the fact that they built WhatsApp with 30 engineers or less, up to a ridiculous scale.

Yeah, that WhatsApp anecdote of like two million users per server was what got me into Erlang and Elixir in the first place.

So it had to feel pretty good when you got your channel layer to similar success.

Yeah, it was probably the most fulfilling process of this whole Phoenix open source thing, because the hype lived up to reality. I wasn’t thinking we could actually get WhatsApp-like scale, because when I read about WhatsApp, they were using FreeBSD and they forked Erlang and made some optimizations, they fine-tuned FreeBSD… So I was thinking that it was gonna be very difficult to try to replicate that kind of scale. We were doing more work; every abstraction has a cost, so having you not have to worry about the transport level, or able to send specific errors to the client.. So we were doing extra work, and it was really fulfilling to actually see that with minor changes in our initial best-effort approach with just a few tweaks was able to go to something that was able to get millions of connections. That was incredibly fulfilling to come full circle, and also it’s a great brag slide now, of showing that two million connections chart. It’s good marketing for us.

Yeah, I think we included that blog post in Changelog Weekly when you posted it, and I think that was one of our top clicks, if not our top click of the newsletter that week. I think the bragging paid off, people were interested in those results.

Awesome. And then like you said earlier about loving open source, about your code getting faster… So now if you’re using channels at Phoenix 1.0, you can upgrade to Phoenix 1.1 or 1.2 now and you’ll have something that’s like orders of magnitude faster, with changing nothing.

Awesome.

So yeah, that was the effort directly after 1.0, performance optimizations around channels, and then with 1.2, which is due out very soon, it was really all about Phoenix Presence.

Phoenix Presence started out with us wanting to solve a simple problem. What we thought was simple was we have this real-time layer and people are asking, “How do I get a list of currently connected users?” The simplest use case would be show a sidebar in chat of who’s online, or who’s in this chat room. We thought this was gonna be pretty simple to solve, and people weren’t solving it well when they deployed it to multiple servers, and this ended up being several months of work. What I thought would be a simple problem ended up being actually pretty nuanced. So I guess I can speak to Phoenix Presence and all the things we had to solve there.

Yeah.

So the issue with Presence is, one, you can be online from multiple places; if I open the app on my browser and then I’m signing in from my phone, I have to be able to distinguish those two things, because I want to show “Chris just signed online” the very first time I log into the app. But if I come in from my phone, I don’t wanna alert everyone that I’m here because I’m already there, but I might wanna show an icon that I’m now online from mobile. So you have to treat Presence as unique, even given the same user.

[55:47] And then the distributed problem is the hardest issue that almost no one solves, and that’s if you have this distributed state on the cluster, most people would just shove that into Redis or database. That works if you just assume that computers are reliable and the network’s reliable. I think most people at this point assume that nothing bad is ever gonna happen, and that’s usually the best case. The problem is if you have a net split or you have a server drop and go down, you’re gonna have orphan data in Redis or your data store; it might show a user that’s online now forever, because the server that was responsible for removing them from Redis is now dead… It caught on fire or someone tripped over the power cord.

So you end up with convoluted solutions that José and I, when we were originally planning this, we were talking through how you would implement this. Initially, we were thinking maybe we would have an adapter that could be Postgres or Redis - we were thinking in the database sense, and then you end up with just all these convoluted things of like now if a node goes down, you can have every other node periodically try to detect that one node went down and then clean up those orphan records. But if you have like a hundred servers on the cluster, now you have like a hundred servers competing to do cleanup for one other server, and it just becomes this mess and it’s not gonna scale. Not to mention you have a single point of failure once you deploy Redis, and you’re gonna have to serialize or deserialize your data in and out. So it’s got some severe side-effects, and we wanted to tackle this in a way that didn’t require the single source of truth, the single bottleneck. That’s one of the things where Presence adds CRDT, which is a conflict-free replicated data type, and that gives us the ability to have an eventually consistent list of presences that is just gonna recover from net splits or servers going down, or new servers coming up; so if a new server joins the cluster, it’s just going to request from a minimum amount of nodes all of the Presence information and it’s gonna self-heal. So you can deploy this now with no single point of failure, and it’s going to recover automatically under pretty much any scenario, whether there is like a network issue or whether you have a server just drop off forever. We’ve solved all of those hard cases that no one really gets right.

At the end of the day on the server side it’s like a couple lines of code on the server and a few lines of code on the client and you can get an active list of users, and it’s something that you don’t have to think about.

Can you give us the scope of the Presence feature for Phoenix users? You said that you have a list of active users… In terms of all that it will provide for the Phoenix user to develop their channel-based application. What is going to be there “for free”, or free for us but hard work for you all, with Phoenix 1.2?

The API is pretty simple. There’s a mixed Phoenix Presence generator that just generates you a Presence module that you can put in your supervision tree. What that’s gonna give is you can say presence-track, ‘track my user’ and you can give it some user ID and metadata. It’s like a couple lines on the server to say, “Hey, track my process and also send a list of presences down to the client”. Then the JavaScript client includes a new Presence object that handles syncing the state with the server, because you want it to be able to resolve conflicts not only on the server, on the cluster, but also on the clients. So if the client disconnects, you might have users that have come and left, and the client needs to be able to sync that state when they reconnect. So we provide just a couple functions on the client - there’s a present sync state and sync dif, so as information is replicated on the cluster, instead of getting 500 users come and go really quickly, instead of getting 500 messages on the client, you’ll get a single presence-dif event and you’ll called presence-sync-dif with a couple optional callbacks to be able to detect…

[59:48] Given that single event, if a user joined from the first time or if they joined from an additional device, you’ll be able to actually detect those cases, so you can maybe show a count of the number of devices I’m on, or show the mobile icon. Or if I’m logging off from every device, I can actually finally show “Chris left.” So we give you all those primitives, and it’s just a few lines of code that you have to write.

That’s pretty much all Presence is from the Phoenix 1.2 sense - it’s just a few lines of code on the server and the client to develop this list of active users on a given topic, whether that’s per chat room, or maybe a global active list of all users signed into the application.

It’s interesting that you mentioned the CRDT. We recently had Juan Benet of IPFS (Inter-Planetary File System) on the show, and near the end we asked him what was on his open source radar, and he had mentioned CRDT as a very interesting piece of computer science that has a lot of use cases and he thinks that it needs more exposure, because people aren’t using this data structure. At the time I was about to interject and say I think it was Phoenix Presence using CRDT, but I wasn’t sure and I also didn’t want to interrupt him. Can you talk about how you came to discover CRDT as a thing and use it for this feature?

I think you’re right, they haven’t really been put to their full potential, and that’s one of the things that excites me the most… I like to say that Phoenix is putting cutting edge CS research into practice. So we’re not just trying to say, “We can be computer science-y.” It’s exciting to me, because we’re applying this cutting edge research and we’re actually putting it into something that you can solve day-to-day. The React database, they use CRDTs, but unless you have the need for this distributed database, no one day-to-day is leveraging CRDTs, at least that I’m aware of. So I’m excited that we’re able to solve this simple use case using this really great research. The nice thing about CRDTs (Conflict-Free Replicated Data Type) is they give us consistency; we don’t have to do remote synchronization where we have a consensus protocol or we have to lock the cluster and figure out who has what. It gives us a way to be… If we can fit within constraints of the CRDT we can have much better performance and much better fault tolerance, because we can just replicate data, data can arrive out of order, it can arrive multiple times, and all of that is going to eventually commute to the same resolve. The conflicts are mathematically impossible if you fit your problem into this confined CRDT problem space.

So it has some really nice qualities in a distributed system, because you can’t really rely on the network always being reliable, and once you go to a lot of nodes, you don’t wanna have to lock the cluster to get some kind of consensus; you want that to be automatically resolved for you.

There are different kinds of CRDTs, but we knew that a particular kind of CRDT called an ORSWOT (observed-removed set without tombstones) had all the qualities that we wanted for Presence. So from that thinking, I was talking with Alexander Songe who has worked on some Elixir CRDT libraries and gave a great talk last year at Elixir Conf about CRDTs. He kind of confirmed our thoughts about, yeah, this Presence would be a perfect fit for CRDTs. So we kind of knew it was gonna be an optimal solution if we could figure it out.

[01:03:49.05] Yeah, Presence makes a lot of sense for eventual consistency, because it’s not a requirement that everything always be completely consistent, when you’re worried about who is and who is not present, as long as it eventually gets there. It makes a lot of sense.

Yeah, exactly.

So besides the talk that Chris said that we had at Elixir Conf last year, Chris’ talk at Elixir Conference now in Europe in Berlin was also really good, and if someone is finding it hard to follow only through listening, I recommend watching the talk. The video should be out soon.

Chris has some examples there. For example, he had like two nodes connected to each other, and then some users are in some node and some of the users are in the other node, and then he simulates a disconnection between the nodes, and you can see that everyone that is in one node disappears from the other node, from the screens, disappears live from the browser, for each connected client. And then as soon as he reconnects, all of the clients come back, everyone that was connected to those particular servers, because now those servers are back up and they can exchange this CRDT again between them. So then you know everyone who is back up again. That was a very good example.

Chris also showed a very good example of… It’s actually not a lot of lines of code to generate the Presence stuff; you tell in the server what you want to track, and then in the JavaScript side you just say, “Hey, every time you receive a new state, that’s how I want to change my views, my JavaScript in the browser, and every time I receive a dif, that’s how I want to change it”, and done. It’s really small to get everything working, with all those properties; no central point of failure, everything is distributed, and if nodes go down then they come back up, they are able to merge and everything just works. It’s really cool.

I think if everything works well, we are going to end up with the same problem with HTML, we don’t get bug reports, and then we don’t know if that’s because people are not using it or because it just works. But I think it just works, because we are hearing stories of people using it in production already for a while, and it’s just fine. That’s always nice to hear.

That’s the thing, yeah. I keep thinking that we’ll have bugs, but… Because CRDTs are tricky to implement; they have to be correct, there’s no ‘almost correct’ as far as CDRTs go. So I’ve released Phoenix 1.2-rc and people are using it and reporting that it works, but I don’t believe them. This took me months of work, reading research papers, having crushing self-doubt and then overcoming that, and figuring out how to parse these academic papers… And now that it’s almost released, I don’t actually believe that people are using it, because I haven’t had any show-stopping bugs. So that’s good, but also it’s… Unless someone comes and tries to mathematically verify our implementation… We’ll see how it goes, but it’s been a really exciting process, and I guess it’s kind of driven where we’ve gotta go with Phoenix beyond 1.2, which I guess we can talk about. But maybe if you wanna still stay within the realm of Presence first, or we can talk about maybe where we wanna go with what we’ve built. Because it turns out that presence is actually kind of this gold mine of untapped potential that we accidentally created.

Yeah, that was what I was gonna actually ask next, and we’re coming up against a hard stop, so let’s talk about the future a little bit. I was gonna say, is Phoenix Presence a one-off feature that y’all put a lot of work into but it kind of stands on its own, or…? To me it kind of seems like there’s building blocks that have been laid for others things, so maybe you could speak to that and the future.

[01:08:03.20] Yeah. On my very first Phoenix talk - I think it was 2014 at the very first Elixir Conf - I had a good idea where I wanted Phoenix to go, and I think I pitched it as a distributed web services framework; that was on the first or second slide. And I talked about leveraging Elixir and this distributed runtime. At the time, we were just trying to tackle the standard web use case, but I talked about the long long-term, that I wanted to have a distributed service discovery layer.

At the time I really had no idea what I was talking about, other than I knew that we had the technology, and I knew that we could solve it with Elixir, being able to just deploy a service somewhere in the cluster, that can perform some work for you, and then be able to tell it “Hey, do this thing”, and just have it work, magically.

I figured it was gonna be really far off, and even if you would ask me around 1.0 of last year, I would have told you “Yeah, I’m still interested in it, but it’s really far off.” But it turns out we’ve accidentally solved it with Presence, and kind of like pretty far into solving the solving this simple use case of showing the list of users online, we kind of realized that what we really made is a distributed process group that has some really nice qualities. It’s eventually consistent, which is nice, and then it also allows you to attach metadata about each process of joining or leaving a group, and it gives you a callback to be invoked when a process joins or leaves.

So we realized, instead of replicating users that are online, it’s exactly the same thing if we replicated what services were online. They’re both processes, so we could have instead of Chris is online, we could say “Hey, this web-crawler process is online, and it says it can do web-crawling.” And instead of listing the users that are in this Chatroom123, I could say “Give me every process on the cluster that says it can do web-crawling.” And it’s the exact same code that would apply to both cases.

So we realized that we have this service discovery layer by accident, and we’ve solved all the hard things that we would have to solve to do service discovery, and it has all of the qualities that we want as far as recovering from failure, net splits or new nodes coming online, and just having services automatically be discovered.

Where we want to go next is we wanna maybe make an API specifically around services, where we can build on top of Presence to be able to do efficient service lookup and routing, to be able to do process placement.

I wanna call a web-crawler, for example. Like something expensive, I wanna have a multiple of those deployed across the cluster. One, for failover and two, so that I can distribute that work. I would like the client to be able to say, “Automatically we’d balance” based on maybe the current work factor of each web-crawler, so the crawler can update their current number of jobs that they’re processing, and that would be in the metadata of the presence.

Then we could also do other efficient routing where we could just automatically shard based on the available web-crawlers on that cluster; that way the caller just says, “Hey, call this web-crawler service. Here’s the information” and we’ll automatically distribute that load for them.

There’s some other neat things that we can build on top of Presence, but really the planting is there today, and that’s what the most exciting part of it is for me, that we actually solved this exceptionally hard problem in our quest to show what users are online. I think it’s kind of a testament to Elixir and the platform. We used these primitives of the language and we’ve built on top of them. And given this distributed runtime and this great environment, we ended up with something far greater, really by accident.

[01:12:00.00] That’s an awesome by-product of trying to solve what is typically a simple problem, but you’ve found this greater piece to it. Since we have a hard stop for José, we’re gonna let José off the call. José, are you cool with bailing out and continuing to speak with Chris for a few more?

Yeah, definitely. I want to add one thing, my last words, so just to give an idea what we are thinking here. Imagine you’re building an application and this application is growing, and then like “Oh, jeez, I’m going to get this part of the application or this feature that we are going to do next, I want to put it elsewhere. I’m going to make it another application.” And then what you do is you start developing that separately, and then those applications talk to each other, so you need to go implement a web API for that other application, that the other application is going to call, and then you need to serialize your stuff to JSON. Then if one server cannot handle that all, you need to put an HAProxy or something like that on top of those other servers, other services. Then on your original application you need to go write the code that’s going to do the JSON deserialization, and then you need to have an HTTP client that’s going to talk to the thing, right? So you’re writing all this code and all those things, and it ends up being a lot of things.

What you end up with - you basically have a distributed system where you’re talking to other machines, but the way the distributed systems communicate with each other is just very complicated, because you’re using HTTP that’s not efficient, and then you’re using JSON, that’s not an ideal serialization format as well. So you create all those things, you need to have all those infrastructure pieces.

And here, because we have the Erlang virtual machine that runs in distributed mode, nodes can talk to each other; it already knows how to serialize data between distributed entities, so we’re gonna say that hey, you are writing your code, and then it can even be in the same project. For example, in the same application you’re like “Hey, in this project I want this to run in one node and this to run in the other node”, so when you test things locally you don’t need to start a bunch of different entities; everything can talk directly there, and then when you want to run it in production, you just say “Hey, now you run in those different machines, in different clusters”, and then you don’t need to do any of the other stuff. You don’t need to have a proxy to do the load balancing for you, because the Phoenix service system is going to take care of that. So you just say, “Hey, I want to run those things there”, and done, problem solved.

You don’t need to be writing HTTP clients, you don’t need to think about how you’re going to serialize and deserialize the data because it’s all taken care of. That’s kind of the idea we’re going at. If we can make a parallel, when we design in the Presence system and then you can have a bunch of machines in your cluster just talking to each other and you don’t need to write other dependencies, we are thinking the same here… But look, now I have those different services, running different machines, and they can talk to each other and you don’t need an HTTP client, you don’t need a proxy, you don’t need something that is going to do the service registration and management when those nodes are off. So everything’s there, we can do it because of the platform.

Wow.

Good stuff, José.

Cool, so I have to go, unfortunately, but Chris, go on, and thank you guys for having me. We will chat later.

Yeah, have a good one.

Thanks José, we appreciate it.

Thanks José, thanks for your time. We’ll talk soon.

Alright, now we’re back to Chris. That’s unusual, we’ve never had a caller drop off during a call here before. That’s what you do when you have to, and one has limited time.

Now he’s gone, we can badmouth him. [laughter]

He made some good comments about where we see service discovery going.

[01:16:01.08] Yeah, big ideas.

It really simplifies everything and gives you… Microservices are the hot movement, but it really gives you the best of both worlds, where you can develop these things just in Elixir, like you normally would, and then you deploy them out there are ‘microservices’, but you don’t have all of this munging to do, of like “Okay, now let’s talk to this API team because these things aren’t discoverable, they’re just webbing points, they have their own load balancers…” All these layers disappear, and that’s something that is really exciting to me, if we can leverage that.

Yeah, man. I’ve always loved the simplicity of HTTP from a programmer’s perspective, in terms of an interface for communications, but it always seemed like it was suboptimal when it comes to microservices. Everything JSON or HTTP, there is just a lot of stack there that you don’t necessarily need.

Yeah, especially if you already have a distributed environment. The other part of this too is because we have a distributed environment, it’s not just about making a remote procedure call. I mean, that’s part of it, but part of it too is if I want to - let’s say I’m writing a game and I have some game state and game logic, and I wanna be able to spawn these things somewhere around the cluster. So imagine a service not only can do web crawling, but maybe you have one that’s doing game state; so I wanna have a process spawn up somewhere around the cluster for me that manages a couple players game state of the game that they’re playing. That’s a long-lived process, it’s not just a remote procedure call.

What that does is I’m gonna say, “Hey, somewhere on the cluster someone spawn me a game server”, and now I’ve got a process back of that game server and I can communicate directly with that process now, just like any other Elixir code. So it’s not only like a service remote RPC call, it’s me able to do process placement on like “Hey, someone spawn me one of these things” and then I’m gonna treat it just like any other Elixir code, just like any other process. I can communicate with it directly, I can ask it later for its state, I can change its state. So it really gives you the best of the platform.

Very cool, Chris. Well, if you have a little bit of time and are open to it, I do have a list of random support questions that Jerod has, which we’re gonna maybe even ask off-air, but if we have some time, I think people will be interested in hearing your take on a few things with regard to Phoenix.

Let me have them.

So the first two are kind of combined in terms of taking Phoenix into production. Part A is “Do you suggest running it behind a proxy or not?” and related “Does Phoenix or Cowboy or the stack itself have HTTP2 support? Is there anything specific that you have to do to get that running? Or what’s the situation there?”

Yeah, so for the first part it really depends. There’s no absolute need to run Phoenix behind a proxy. In fact, I’ve heard… I don’t think it’s gonna be the normal case, but I’ve heard two different cases of Nginx actually becoming a bottleneck before Phoenix, when behind Nginx. But for the vast majority of people, well configured Nginx it’s not gonna happen, but it’s just an interesting anecdote. But as far as like a dockyard, we deploy everything by an Nginx; it’s just simpler. That’s our deploy process, and that’s how we can load balance multiple web frontends. So I think Nginx is still how I would deploy my web frontends in Phoenix, but it’s absolutely not a requirement. It really could just depend on… I would say deploy just like you would any other web stack.

[01:19:38.26] Those web frontends will happen to be clustered together with your greater Elixir cluster, but still being able to balance in front of or behind Nginx is a great option.

And for the second part, HTTP2, we’re exploring that. A Cowboy master has HTTP2 support. Cowboy 2.0 is gonna come out with HTTP2, and there’s also another library called Chatterbox, which is an Erlang HTTP2 server. So we’re currently looking at how to get HTTP2 into Plug, which is our web server abstraction that Phoenix sits on top of. It’s not there yet, we need to get into Cowboy 2.0 and this Chatterbox library and look at how they could both fit into kind of a common API under Plug.

So definitely once Cowboy 2.0 will go stable, we’ll shortly thereafter release a plug that will have HTTP2 support, and then Phoenix will just get HTTP2 on top of that.

So it’s coming, but it’s not there yet.

Okay. Talk about deployment a little bit, in terms of how you get a Phoenix application into the wild. Let’s ignore for now the platforms as a service, the Heroku Buildpacks and what not. I know there is exrm, which is the Elixir release manager, which seems to be the way suggested to move forward. I’m wondering if there is anything on top of that, similar to a Capistrano, where it’s kind of manipulating the exrm in order to do, for instance, the scp step of the application to the server, maybe database migrations, rollbacks, those kinds of things. What’s the deployment story?

Yeah, I hope the deployment story gets better… It’s not bad, it’s similar to where you were with Ruby earlier on. We have some tools, but there’s still some manual steps. For the listeners that aren’t familiar, exrm is a way to build releases. There’s two ways to deploy Elixir in Phoenix applications. One is to run the project directly, like you would run it in development. Another one is to build a release, which is a self-contained tarball of the whole Erlang VM, all of your project code, everything it needs to run; you can deploy that and run it as a self-contained entity. That gives you some nice features, like hot code upgrading.

Under the Capistrano-like case, there’s exrm mobility with tarball, but there’s this final step of okay, now to scp that onto the server, and then basically start the release and run it. Which isn’t that hard. That’s how some people deploy; they just have a bash script that just scp’s, it starts, and they’re good. But I would like to see some tooling built on top of that, because I think to give you that mixed deploy, that single task-like deploy - I think that would go a long way.

There’s a couple tools that I’ve been waiting to check out. One is called Relisa, and I think it does that for you, I just haven’t had the time to look into it. I’ll link that in the show notes.

So deploys could definitely get better… It’s not like it’s this insurmountable thing today, but I think that if we want to give people that Capistrano-like experience, because that’s just removing yet another barrier to entry to people getting this out in the world.

Yeah, absolutely. I was walking through the steps and it looked very much like, “Okay, this looks like maybe a ten-line bash script which does those for me”, but then I started thinking of Atomic, and changes, and rollbacks, and database migrations if there are any, and I thought “Hm, somebody should solve this problem. I wonder if anybody has yet.” So if you’re out there and you wanna get involved… Or maybe Relisa is it and we just don’t know yet, but there’s an opportunity to help the Phoenix community.

Yeah, and there’s also edeliver, which is what we use at dockyard. And really all edeliver is is just a bunch of bash scripts that are wrapped. We’ve had some stumbling blocks there. We’ve got it to work, but it’s not just this “Set up and you’re ready to go.” Yeah, if someone wants to get some open source street cred, that’d be a great problem to take on.

[01:24:12.09] There you go. So the last thing I have for you, I teed it up way at the beginning of the call, talking about the decision to basically bring in a third party build tool, which the default is Brunch… Which is really leveraging the npm ecosystem, and I think you were on - it was either Ruby Rogue, or the Elixir Fountain recently saying… You know, it seemed like you didn’t wanna touch the JavaScript side with a ten-foot pole, to put words in your mouth, but just staying out of that whole thing. [laughter] I know that Brunch, which seems like a really nice build tool, has some integration points with Phoenix, and there are ways to swap it out.

I was on the Phoenix Slack channel the other day and somebody mentioned that they had been replacing it with Webpack. Maybe just talk about the build tool situation, the decision you made, and how you go about changing build tools. A lot of these was people who were building Phoenix backends for JavaScript or Elm frontends; they don’t even need any part of this. But my specific support request is about Phoenix.digest, and if that’s tied specifically to Brunch, or if that would work with another build tool. But I guess to broaden it, just speak to the build tool situation in general first.

Sure. Yeah, so Phoenix.digest, just to answer that, is not tied to Brunch. I’ll touch on that in a second. So yeah, this has been the most miserable part of Phoenix… And it’s no fault of Brunch. Back story is Phoenix has really been about giving a great out-of-the-box experience, which I think is one of the most important things. And people just want the ability to compile and bundle their JavaScript and CSS. They just want it to work. Like, you put your JavaScript in a directory, you put your CSS in a folder, and it gets compiled for you when files change. And what we didn’t wanna do is write our own asset pipeline, because I didn’t wanna spend a year of my life working on that. The other side of this is, as much as… The vast majority of issues on the Phoenix issue tracker are Node and npm-related, or Brunch-related. And that’s not a problem of Brunch, it’s just I think the JavaScript community has a great tooling, but a bunch of fragile tooling. And even if we implemented our own asset pipeline in Elixir, we would still have a heart of no dependency. I’ll have to call it, that Node and JavaScript is just like an unfortunate reality in web development.

There’s no way to get away from it. If you wanna support SaaS, Es6, CoffeeScript, TypeScript - pick any tool that you use to deliver an experience on the browser, you’re gonna have a hard Node dependency; it is what it is. So even if we spent all this time writing an Elixir asset pipeline to concatenate files and polish shell scripts, you’d still need to have Node installed on the server, because we’d still be shelling out to these tools. So unless we wanted to reimplement an Es6-transpiler, SaaS, all these tools in Elixir, it’s pointless.

So instead we said, “Okay, let’s look at what the JavaScript ecosystem has”, and we investigated all of the dozen popular ones. I looked into Grunt, Gulp… From Gulp I looked into webpack, and from webpack then I checked out Brunch. And Brunch won because it was by far the simplest to use. It had this small scope as far feature set, and it was the fastest.

[01:27:56.21] A lot of these tools, like Gulp and webpack, they wanna be not only asset builders, but they wanna be task runners, test runners, they wanna run development service… They wanna do all these things, and if you’re familiar with the Node community, all these things have dependencies, and your dependencies have dependencies, and you end up with something that is like this insane dependency tree, just to concatenate JavaScript and CSS. So we liked Brunch because it was simple and fast, and it’s just JSON configuration, so you don’t have to really know how Brunch works. When you run mix phoenix.new you get a project and by default you’ll get Es6 compilation and CSS funneling, just by putting CSS and JS into a folder.

So that’s how we settled on Brunch, but we knew that it would be a point of contention, because there is like a million different tools in the JavaScript community, so what we did is we only included it by default, but there’s not coupling. So if you wanted to use webpack, it’s like a one-line change in your configuration, and then we’ll start the webpack watcher instead of the Brunch watcher. We call these things ‘watchers’ - they watch a static directory for changes, and then we shell out to them and they do whatever compiling is needed. They’ll build files into a priv/static directory, which is where our stack files live, and that’s where the digest task comes in; if you wanna digest your assets, all we say is “Your static build tool needs to build to this directory. We don’t care if it’s Brunch or webpack or Grunt”, and then we’ll digest those already bundled files.

We tried to integrate this in a way that gives you great out-of-the-box experience for the most common use cases, but if you have some other tool, you should just be able to swap it out and use what you like.

Certainly good points on Node being there for you no matter what. It’s on the frontend, you can’t get away from it… Why recreate the wheel, or redundancy, in that case making something that you don’t need?

Yeah, and there’s been a ton of misery… You know, I don’t like to…

Well, you don’t wanna spend a year of your life, like you said, doing that.

Right.

There’s better things to focus on.

Yeah. I don’t like to put down anyone else’s work, but there’s been so many times that npm install has just broken for people. You can probably sense some frustration from me, because it’s… Someone will open an issue about - there’s repeatable builds, things just break; things that have been stable on Windows would just suddenly break… We’ve had all these Windows support issues, which is interesting because it’s not… I thought that Elixir and Erlang would be tough to run on Windows, but it turns out that the biggest issue is people trying to run Node on Windows, which I thought was a solved problem.

My only hope for the JavaScript community is we can settle on tooling, instead of having so many options, and then also maybe end up with a repeatable build process that is much more stable than it is now. We’ll see how that goes…

The question is “Did Brunch depend upon left-pad?

I’m trying to think… Yes, actually it did.

Okay.

Everything depended on left-pad. [laughter]

Everything did. So therefore Phoenix depended on left-pad.

Yeah, because people started reporting issues, and it was left-pad related, which is funny.

Well Chris, it’s been fun having you. We’re near time, I think Jerod has a hard stop here in 13 minutes; I’m not sure what your timing is, but we could talk for longer… We wanted to give you a chance to sort of give some last words, like José did. So anything you wanna say in closing, we’d love to hear it.

Sure, yeah. So let’s see… Maybe just a recap of Phoenix 1.2. Phoenix 1.2, it’s a release candidate today. We have no working issues, so I think within the next week or two, by the time this airs, should be out.

[01:31:56.28] Presence is the biggest feature, we’re really excited about enabling distributed applications that you just don’t have to think about. That’s where we wanna go next, being able to give you this distributed tooling layer where you can develop an application on your laptop and then run it distributively, but the code is the same. So it’s kind of a similar theme with channels; we wanted to give you this trivial interface for real-time connections where you didn’t have to worry about how’s the client connected, what transfer are they coming over… We kind of wanna apply that same idea to distribution, where you can develop on your laptop, and then you can deploy this and you don’t wanna have to care “Is this server available locally or is it available on some computer somewhere?” or “Do I happen to have ten of these things deployed on ten computers because I want fault tolerance and scalability?” So we wanna give you that experience and take care of all those details for you.

So that’s what’s coming next, and check Phoenix 1.2 out when this airs.

Adam, before we close I’d like to give a quick shout out to Dockyard for employing Chris and allowing Chris to work on open source, I think is it full-time here on Phoenix? Or at least part-time?

Yeah, thank you for that, by the way. Yeah, my primary role is to work on Phoenix, so about three quarters of my time are spent on open source and Phoenix development. And since I’ve been there, it’s coming up on maybe six months, I’ve been almost entirely full-time on Phoenix. None of this Presence stuff would have happened without their support, so I owe them a huge thank you.

So a good way to support you supporting Phoenix would be to potentially buy services from Dockyard.

Oh, certainly.

DockYard.com/services, full project design, engineering - the full gamut.

We love companies that support open source, and we talked earlier about how the beauty of it is everybody else’s applications and projects get better by this shared effort. So companies that allow that shared effort, we all thrive based on it. So a huge shout out to them for doing that, and for all companies that are putting their hard-earned money behind open source projects as a way of sustaining the ecosystem.

That’s certainly awesome.

José’s gone, but Plataformatec, José’s company… Obviously, Phoenix wouldn’t have happened without José and Plataformatec, because they took the even crazier position of not only saying, “Okay, let’s support this web framework, but José had this crazy idea to write his own language and take a couple years off to do that, so we owe them a huge thank you as well.

Yeah, absolutely.

There’s an interesting too, we mentioned that in the earlier show, that both of your back stories… In that show with José he talked a bit about how they were bedding under early and how he was working on the side, and then they started using it and it sort of took over. So if you wanna listen to that… What episode number is it again, Jerod? Episode 194 for José’s and 147 for Chris’ shows, so go back and listen to those.

[01:35:04.09] Before I forget, I will just say my keynote from ElixirConf Europe is actually online now, so we’ll put that in the show notes, and that will take you through… I kind of pay it forward by walking through how CRDTs work to give you kind of this mental model without having to read research papers. So if you’re interested in CRDTs, that will be a good talk to watch.

I think your conversation today on that subject opened some ears for sure. It was good to have you, especially to catch back up. It’s kind of crazy, it’s been a year since you’ve been on the show, and I kind of enjoyed just sitting back, hearing all this goodness, because as Jerod’s mentioned, he’s building the CMS and it’s the future of the Changelog and what we’re doing here. So it’s great to have you back on, and José as well, to talk through the underlying technology that is building our future. To me that’s such an awesome feeling, honestly, to have that and to share that with you guys.

Yeah. I appreciate it, and you know where to find me online if you have problems, and also maybe six months or a year from now we can talk about Phoenix next in our awesome series discovery.

Only one thing I wanna mention before we close, and it’s a shame, José’s not here anymore, but you can pass the message, Chris, or he can listen to it in the produced show that goes out - we had Matz on the show (episode 202), and Matz is a fan of Elixir, so that would probably get José pretty excited. He even said it on the show, too…

That’s awesome.

And he listened to José’s show, 194. Big stuff there. Let’s close the show… So if that’s it, fellas, we can go ahead and say goodbye.

Sounds good.

Goodbye. Thanks Chris, thanks José.

Thanks for having us.