Jean Yang: Something I’m really excited about is low code with APIs. Because part of me is like “Let’s just all be really honest about what we’re doing here… Basically gluing together APIs.” So I’ve been a big Zapier fan for many years now, and I’m also a really big fan of Postman’s new low-code product called Flows. But as a programming languages person, it’s always about if your language or your builder abstractions are at a higher level of abstraction, it’s always easier to analyze what’s going on. And so from my point of view, we have to do all this stuff with API observability right now, because we have to back-engineer all the API traffic, and really piece together all of the API interactions… But if you’re just straight up using a low-code tool, that’s just right there. And so that’s something that’s really interesting and compelling to me. I think that that’s very clean from an abstraction standpoint, and it also just enables more software builders, which I think is very cool.

So to me, that cleaning up – so like right now, calling APIs from low-level code kind of feels like you’re mixing assembly with some other stuff. You’re at like a low level of abstraction. So lifting the whole abstraction layer to something that’s API-centric is very exciting to me. And then you would only need something like us for the messy stuff that you customize, or something… You know what I mean? But all the other stuff, it’s cleaner to begin with. So that’s something that’s really exciting to me.

And then there needs to be a better solution for legacy stuff. So legacy subsystems today are like toxic waste. They’re just sitting there, waiting for like a big bug or vulnerability to really cause things to spill over… And the work we’re doing is one piece of what allows people to make inroads into legacy software. I think there’s some work that Docker is doing that’s really interesting, helping people containerize legacy software… So the reason I’m excited about that is if you have legacy software that’s just kind of like sitting somewhere, running on your own systems, on a non-standard tech stack, it’s really hard to make sense of it. But the minute you virtualize it, you can start poking and prodding at it in a black box way. That support some of the stuff we’re doing, actually.

So we can only watch things… If they’re sufficiently virtual– or we could also… This is a gray area, but we could also install our agent on bare metal, etcetera, etcetera. But the minute things get containerized, things are easier. So the push to containerize and standardize infrastructure I think will help some of the legacy problem. But a lot of software tools, discussions really gloss over the fact that we have growing amounts of legacy code that are never going to be part of this future that they’re describing, and what do we do with all of that code?

Jerod Santo: According to the team behind it, Hyperswitch is “an open source financial switch, written in Rust, to make payments fast, reliable and affordable. It lets you connect with multiple payment processors and route traffic effortlessly, all with a single API integration.”

Remember when Segment took the adapter pattern and applied it to 3rd-party scripts? From what I can tell, that’s what Hyperswitch wants to do with payment processors.

It supports processors like Stripe, Braintree, Paypal… all the major credit cards including Apple Pay and Google Pay… and buy now pay later providers like Klarna and Affirm.

I’ve covered Hyperswitch here on Changelog News in the past, but I thought it was worth mentioning again now that the project seems to be maturing both in breadth and depth.

Clip from Changelog News 2022-10-17: Zed doesn’t pull any punches, and this post goes deep on the shortcomings of both Paypal and Stripe.

If that advice speaks to you, check out HyperSwitch. Written in Rust, HyperSwitch is a “payment switch that lets you connect with multiple payment processors with a single API integration. Once integrated, you can add new payment processors and route traffic effortlessly.”

If you’re already using Stripe, HyperSwitch is said to be fun, fast & easy. Those are three words not usually associated with payment processing. You can try it out in their sandbox environment or on your local system with Docker Compose or by setting up a Rust dev environment.

Jerod Santo: Speaking of developers, it appears that you’re also reaching out to developers beyond your company borders with an end-to-end integration API. Can you tell us what this is about and how you hope developers will use it?

Jerod Santo: OpenSignLabs’ mission is to democratize the e-signing process, making it accessible and straightforward for everyone. The software currently features secure signing, a user-friendly interface, audit trails, and an API for integration into other software and services. Host it yourself (React, Node.js, MongoDB) or use their cloud hosted version.

Amal Hussein: Yeah, that’s so cool. I mean, the open API spec integration is very exciting… And it’s like a no-brainer, right? That’s so cool.

Jerod Santo: Right. Yeah, that would get me to buy one. And it’s developer-friendly. So out of the box, Busy Bar comes with an Open API ready for integration into your project… So you could probably get your CI/CD involved, you can probably get your Raspberry Pi’s involved… It looks like it’s taking Golang, JavaScript, Python etc. None of your languages, except for you said you’re writing some Go now, right? Is it because the TypeScript team likes it?

Daniel Whitenack: Yeah, I kind of had a similar thought. I saw some study - I actually don’t remember which one it was off the top of my head, but it was one of those that surveyed a bunch of enterprises, or something like that… And something like 15% to 20% of enterprise companies actually had either kind of something prototyped, or some type of integration to an AI system. Maybe that’s just a simple integration with an API, or something… But if you think about that, there’s definitely kind of at that beginning I would say we’re kind of past R&D in the sense that – not that R&D is not important, and new things will be discovered, as they continually are, but in terms of the technology that has sort of dominated the news cycle in 2023, we’re moving past the heavy R&D investments, more into the “How do you scale this with software engineers, developments?” Like, if you’re a company developing these technologies, you probably are going to not win by doing R&D, you’re gonna win by creating, like you say, products, the applications, the platforms, the APIs, the systems that power this kind of growing adoption within the enterprise. So I think where you saw that kind of adoption start in 2023, that’s going to expand very rapidly in 2024, and those that keep up with that in terms of actual software and system development will likely be in a good place.

Gerhard Lazu: That’s really cool. So I’m hearing Google DNS part of the Google Cloud – Google Cloud DNS, that’s it. And Gandhi as a registrar for their DNSSEC integration through the API.

Timo Möeller: It’s really great that we are separating the labeling process and the developing process, because it will get super-complicated. I also wanted to thank Tony that he’s taking also a lot of initiative for supervising and pushing the label process… Because I have never heard of an open source labeling process. And I think this will be a mess at some point, and it will be very complicated to interact and also to supervise the quality of the labels. I think this is the great part of this change - we just have to try and we have to make it work… So this is really great, on the labeling part.

On the development part, for computer scientists of any sorts, there’s a lot of help needed. This is just a hackathon project. It’s not like a full-fledged professional industry solution, so we need a lot of help. I’m in contact actually with a data scientist also from Intel, through a contact from Tony, and she’s working on an intelligent scraper. Right now we have very manual scraping processes for each page that adjusts to the HTML structure, and we would need a more intelligent scraper that you can just point to an FAQ page and it automatically extracts questions, and also the answers. Of course, there will be errors, but this is a little bit unavoidable. Maybe we can do a review process there afterwards. So this intelligent scraping will be extremely helpful.

You also mentioned then that bringing this question-matching to other languages - this is something that is personally very important to me, because I think this will create the biggest societal impact. And there’s a lot to do, because for now we have the question-matching algorithm, with sentence transformers and BERT just implemented for English… But making this work for other languages, with multilingual language models, for example, with this cross-lingual language model open sourced by Facebook, for example. This would improve the experience a lot. This on the modeling side…

[48:10] And then also a huge help we need on getting this actually to people. After the hackathon we got contacted by a person - I don’t even know his/her name; it’s the Apache 64. This person just programmed a Telegram integration. This service has an API where it can match questions, and you can call this API and integrate it into Telegram. This bot is just working, and we also integrated the feedback mechanism to feed back the user information back into our system.

This help is really appreciated, but what I think could be important there would be maybe a WhatsApp integration, and maybe even if we extend this really to low-resource languages, where people might not have access to mobile phones with internet… Maybe have like a text message interaction. But this would be a little bit further away, I would say.

Gerhard Lazu: Yup. I have to say a lot of the questions and a lot of the things that we had to work out through the Linode API, and by using the integration which he helped build a lot of these components, we were in constant touch with him and he was a great, great Linode representative and Linode developer to help us with a lot of things… And obviously, improve things for Linode as well, and all other users.

Sid Sijbrandij: Yeah, and it turned out he solved that by integrating it closely with GitLab; it was a much better experience. It was easy to set up, it was easy to get started with, and then the suggestion came, “Let’s integrate this GitLab CI app with GitLab”, and I was not in favor in the beginning. I was like, “We already have a giant monolithic app. What are you doing? You’re making a monoRail out of this (a monoRail as in a huge rails app). This is against all the best practices, we shouldn’t do this.”

And it took a bit of time for me to come around, but our CI lead also said the same thing. We integrated it, and it’s a so much better experience. Now if you look at places like Hacker News, people say, “I’m using GitLab and I could replace not only our Git hosting tool, I could also replace our CI tool. I could also replace the Docker Registry for private containers. And not only could I replace it, but it was so much easier to set up. I’ve spent days setting up all these old products, and these products - it was a question of minutes, because it’s integrated.” You don’t have to ferry credentials around to the private container registry. No, your CI runner knows it’s the CI that’s running that project and can push the containers to that.

[39:59] So it’s a better experience, and that was counter-intuitive to me. I like the UNIX philosophy - one tool does one thing, but what I’m seeing more and more is that it is a very complex thing to make software, and that we’re using these collections of tools and if we integrate them together… Over a thousand people contributed to GitLab, so if we get all the best practices of the best people in the world and we integrate them into a tool, we just prevent a whole bunch of needless push-ups that you now have to do.

I can’t compare GitLab to the genius of Ruby on Rails, but I was greatly inspired by conventional reconfiguration and I still think there’s a lot of needless configuration in many developer pipelines; I would wanna take that away. If you wanna go fully GitLab, it’s gonna be an amazing experience. If you wanna use other tools, that’s fine too. We’re playing nice with Slack, we have great JIRA integration, we have Commit Status API for Jenkins…. We’ll play nice with other tools, but we’re gonna save you a whole bunch of time to make this idea to production happen.

Quincy Larson: Yeah, so I would still encourage people to start with web. And the reason why is about half of all developer jobs are web-focused… You hear about mobile app development, and really that is just a mobile app skin on top of a bunch of APIs that are running on the web, and stuff like that. You hear about a lot of machine learning, and things like that. And it’s true that machine learning is distinct from web development. I could talk about the big changes we’re making to the curriculum… But I would say that the skills that you learn, like data structures, algorithms - everything you learn while you’re learning to be a full stack web developer, virtually all of that is transferable. Almost everybody’s going to need to create some sort of website, or some sort of mobile app, or some sort of integration with an existing platform through an API, that serves whatever it is that they’re creating… Whether they’re creating data science insights. Most data scientists have to deal with data visualization. They have to figure out how to get what’s in their Jupyter Notebook or wherever it is they’re crunching the numbers, they need to figure out how to get that in a place where people who are making decisions based on those data can consume them, and understand them.

So web does touch pretty much everything, and that’s why I recommend starting with that. A lot of people would say “Start with systems engineering. Software systems engineering.” And learning how C works, because everything is built on top of C, and doing the classic computer science degree program work of building your own compiler, building your own operating system, building maybe your own search engine, maybe building your own LLM… Doing all those things. And then a lot of people would say you should just focus on machine learning, because that’s the future. Everybody’s going to be telling the machine in English what to do. And I’ve written a lot about this in my book, and I do believe that in the future programming will consist of talking in natural language, highly structured natural language, to a computer, the way that people on Star Trek talk to the computer and the computer builds things on the holodeck, and those things like that. But that still requires knowledge of the different layers of abstraction below. And in order to effectively get things done with technology, to some extent you do want to understand how that technology works. And I think a lot of people do have a decent understanding of how RAM works, how motherboards work, how hard drives work, how buses work… A lot of the actual computer engineering stuff that software is operating on top of. And we can certainly get a much more higher resolution understanding of that by looking at the underlying operating system kernel, and things like that.

I will tell you what the freeCodeCamp curriculum is doing… But in general, I do recommend – if you’re not sure where to start, start in web. And then work out from there. And don’t feel like “Oh, websites are – nobody uses websites anymore.” That’s just the tip of the iceberg of what web development is. Gaming… Any sort of field where you’re essentially writing software, there’s going to be some component that travels over the internet. And so a lot of those principles. So without belaboring that point any further, I will tell you where the freeCodeCamp curriculum is heading.

Gerhard Lazu: And on Hetzner - so I’m still trying to understand that integration, because you mentioned KVM. Do you talk to the Hetzner API to create anything? What does that integration look like.

Jon Calhoun: I think part of it is also thinking about the project you’re working on, like you’ve just said… Because like Mat had talked about, if you’re writing an API client and you’re trying to test that it actually does things correctly with the endpoints, one easy way to say “We probably wanna do integration tests” is if they provide an API key; there’s a good chance they’re trying to hit the actual API, but if they don’t, you can probably make the assumption in your tests that that’s not what they’re trying to do. So like you had said, you could pass the run flag; it might also be a flag that says “Provide an API key”, or… There’s also build tags which I like, but it’s a little bit different.

Johannes Schickling: Right. I mean, that directly translates to the value we are providing. It’s actually directly rooted in the cause of why we started to build Graphcool. As you’re building your own back-ends, especially REST back-ends, you find yourself a lot in integration, where you basically just want your API, and the API usually either gives you data out of a database, or it receives data and stores it in the database in some way. So you’re just building this mapping between your API and a database. This is where you put a lot of time in. It needs to evolve; it’s very error-prone, it’s not particularly exciting, but there are a lot of mistakes being made in this process. That is something we wanted to just solve, and GraphQL is the perfect foundation for that.

Based on your GraphQL schema, you’re specifying locally, or on our platform with a GraphQL schema we generate a GraphQL API for you with a managed database in the back.

Where the proprietary part is – and there’s nothing really secret about it, it’s just a lot of complexity we are abstracting away, and this is the mapping between the automatically generated GraphQL API and a scalable database.

Kelsey Hightower: Yeah, just a naming convention. Now that we’re on the topic of objects in Kubernetes, it’s very much like in Golang where there’s core types inside of the Go standard language specification. But you’re also able to crack types yourself, right? You can have a type that may be a collection of other types, and pretty much the same thing works in Kubernetes. So view it like, in my case I’ve built a tool that allows people to integrate with Let’s Encrypt. In order to do that, I could just drop something in there and just start scripting and automating a bunch of things, but in Kubernetes we also support user-defined types at runtime - we call them third party resources.

You as a user or as a vendor, you can craft a new Kubernetes object type and send it to Kubernetes through the third party extensions or third party resources, and we will automatically take your type definition in some ways and we will automatically generate the API endpoints, storage in Etcd and the integration with kubectl, which is the Kubernetes command-line tool. So now you have the ability to start creating these certificate objects.

The schema is up to you or the developer of the tool that will be observing these objects, but that will allow you to extend the system in ways that we haven’t even imagined, but still feel first-class to people that are used to using Kubernetes and the core types.

Andy Klein: And so some of the earlier places we were in from a data center point of view, they said “Okay, so here’s a rack, and here’s 25 amps. Have a good time. And oh, by the way, you can only use 80% of that.” And so you suddenly go, “I can only stack storage pods so high”, especially as the drives got bigger and started soaking up more and more electricity. And so now you go, “Well, I can put 4-terabyte drives here, but I can’t put anything with 8…” But that’s changed over time, as people actually realized, one, that these things use electricity.

So you go into a facility like that, and you say, “Okay, so do we have enough – how much electricity we got? Okay, we have plenty. Great.” For the drives today, the drives tomorrow, and so on. And then it becomes a floor layout issue; how do you optimize the space? How much air cooling do you get? Because these things will definitely produce a little bit of heat.

So you could put all the racks really, really close if you wanted to, but then you’re not getting the proper circulation, and it’s really difficult to do maintenance, and all of that. And there are a lot of really smart people out there who kind of specialize in that.

Once you decide on where you’ve got to put them, then it’s not only your storage pods, but all of the ancillary equipment, the other servers that go in. For example restore servers, or API servers. So now that we do the S3 integration, or the B2 storage piece, we had our own APIs. Now we also support the S3 API as well. They don’t work the same. So when you make an S3 call, it actually has to kind of be turned into our call on the backend, and we had to have a group of servers to do that. And so we have those kinds of servers.

And then you have just utility boxes, and monitoring systems, and so on and so forth, that all have to be built into that. So we may have an entire rack of nothing but support servers. The architecture is such that you have to know where all of the data is. And so we have servers in there, that’s their job; they know where the data is, which storage pod it is, and so on and so forth. So you go and say “Hey, I would like to have a file”, and you ask that device, assuming you’ve been authenticated etc. And it says, “Okay, you’ll find it over here. And here you go. Have a good time.”

And the same thing when you want to write something, okay? The way we write things is pretty straightforward, which is we literally connect you to a tome, actually, to a particular pod in a tome. Once you say, “Hey, I have some data and I want to write it”, and you say, “Great, here’s where you’re gonna put it.” And you’re right there. And then we have to record all of that, of course, and keep track of it for future reference, so you can get your data back.

So that whole process of laying things out - like I said, the biggest one starts with what’s your footprint, and then how many racks can you get in there, how much electricity can you support, how much cooling is there, and so on. And then of course, you just have to deal with the fact that these things are big.

So going up is really, really cool, because if we can get it – the only issue ever became one of “Does the lift guy go high enough - good old Luigi there - so that we can get them out, so we can get them back down?” What do we have to do? If I have to bring a ladder with me every time to serve a storage pod, maybe that slows me down a little bit.

Daniel Whitenack: [00:48:17.23] There’s another – I don’t know if they’re using the Stripe API under the hood, but there’s another entrant into this, which is Perplexity now offers a sort of shopping assistant with an actual experience behind it kind of built in. So you have the ability to put in like “Hey, I’m doing this project, and I’m wanting to do this and that. What are the items that I need? Help me kind of shop for those.” I think that’s kind of the vibe. And there’s a search that happens, obviously, and it’s plugged into various products… In this case, they have a merchant program, which definitely seems… So I don’t know whatever happened to kind of some of the monetization around like plugins and other things with ChatGPT, but this definitely seems like a way to kind of get your product… You know, having a wife that owns a business in the direct to consumer space, and sells products direct to consumer, there is this element of trying to figure out “Well, how do I place my product, or how does my product kind of filter up into search results when people are just searching on ChatGPT, Perplexity, whatever?” And so this does seem to be one angle on that, where you can increase chances of being a recommended product, there’s payment integrations, API, custom dashboard etc. So there’s this sort of merchant program element of the Perplexity AI-powered shopping assistant as well. Pretty interesting.

Gunnar Holwerda: I mean, I think, ultimately– so I’ve been at OpenSesame for eight years now, and we originally started with a team of six or seven. This is the standard startup growth. We had a team of six or seven, we grew to 10 or 11, and you started to run into 10 or 11 people in one sprint, one backlog, getting kind of hectic, and you’re working across different aspects. There’d be days where we’d be working in Fastly, in the VCL, and there’d be days where we’d be writing angular front-end code. And so we split into two teams at that point, to have a little bit more focus. And then when we got up to 25 people or 20 people, we saw the same need, to be able to focus on specific areas that were important to the business.

[08:11] So Tom mentioned the commerce side. We have a large legacy platform that we’re looking to improve and move away from and refactor out of, and trying to have a big group in one big team or two big teams, trying to focus on that was going to be ultimately pretty difficult, so we picked how we wanted to structure that and how we wanted to support that… And so we kind of split out into multiple different teams, one kind of focusing on our API integrations with our customers, my team, the catalog, to own the whole commerce system in terms of showing the courses, searching the courses and some of our other platform features, and then another team to own the payment side and some of the unique ways that we handle the app on the backend of how we pay our publishers and things of that sort because it’s a marketplace. And then having a DevOps team to support us in helping us have the right platform, and then a core team to help set some patterns, and they also own some legacy architecture.

So it was really to enable us to have distinct focuses - and we’ll maybe get into Team Topologies, which is a book that we all kind of read it as a leadership team and some of the engineering managers have as well. It talks a lot about cognitive load of teams, and you want to keep that as low as possible, so that people can focus on their stream-aligned work and what they’re trying to accomplish. So that was our way of whittling that down, was trying to create that cognitive load as low as possible, so that we aren’t working from Fastly to Angular all the time.

Nathan Leiby: Yeah. I was just thinking about a different part of our stack. So another big thing that Clever does is it integrates a lot of third-party systems, whether it’s web scrapers, or API integrations… Just a big part is extracting data. So another early place we started trying out Go was to build some of those data extract workers. I think that was a couple months after - I had to look at our Git commits to figure out exactly when, but… And I think that went actually through a spec process, where it’s like “Okay, we wanna build this thing”, we wrote a doc, we were like “Here’s what we wanna build.” And then at that time we had been a few months into Go, and we were pretty confident it would be a good way to build this.

We mentioned CoffeeScript, but we actually had a lot of stuff written in Python as well, to do some of those extract jobs. Not strictly a bad language for it, but in this case we were working with a well-schemad API; it was easy to think about how we’d write that data and extract logic. And I think they were really testing what is the development and collaboration process in this language. The other thing was experimental, and this was really getting it into our real dev process.

Jerod Santo: Yeah, very interesting piece of software. It's one of these things that's so freeform, that it can be – it's kind of the paradox of choice, or the thing that freezes you, like "Okay, where do I go from here?", but they provide some templates. But then that freedom also provides creativity, and it's very interesting software. Different than other things that I've used.

Speaking of spreadsheets, there's a lot of people that are using spreadsheets as their back-ends. There's Airtable, which is interesting, and then a lot of people are just building API integrations with Google Sheets, and basically providing web app front-ends to Google Sheets.

Shawn Wang: [01:09:42.13] Yeah. Time will tell. They’re trying to build one. I think the – yeah, so this is a question of business and less about tech… The moat is really user data. The more people you can get coding with this thing, and the more you can observe how people interact with these agents… Devin has a six-month headstart on everyone else on how people work with Devin-like agents. And if Devin-like agents is like the goal of this thing, then they will have the best RLHF feedback data on the planet for specifically this task. It’s the same motivation that Open AI had with ChatGPT, which is they kind of lucked into this… But now they have the longest series of chat-oriented data sources, human feedback data that you previously had to pay a lot for.

And then – so that moat is the data moat, but then actually it also becomes… You’re investing ahead of where the capabilities are. So you’re sort of saying “I will build code or write manual code to build other capabilities that I don’t have yet.” But as the capabilities grow in the fundamental model, you can just kind of swap them out for your sort of handwritten code, and be more generally capable, with the scaffold that you already built ahead of time.

So I feel like I’m being vague there, but you’ll see this in the form of the model’s ability to interact with the real world. A lot of times you’re writing integrations, you’re writing – it’ll interface with Open API. Like, screw that, man. The future is just models like surfing websites, just like anyone else would surf websites, and interacting with them exactly like you would interact with it. But right now we have to use the crutch of code, and in the future we don’t.

Peer Richelsen: Yeah, absolutely. And our core vision is to connect a billion people by 2031. We always think of what’s something we want to include, and we’ve launched, I think, 13 languages so far. And when we see something like the Web3 industry, which is still technically underserved in terms of products - like, they just recently launched like a Zoom for Web3, and there’s no Salesforce Web3, there’s no Google mail for Web3. It’s very much a new frontier, and it’s still millions of people who use cryptocurrencies today… And I think being this pioneer in this new industry is very important for us to just show that there’s more to just being a knockoff.

I’ve seen a lot of open source projects who never escape the mental model of being a Calendly alternative. I mean, eventually, we will probably get rid of that claim and be the open scheduling product. And those things, like these Web3 integrations and the developer platform and the open API - none of these things is something Calendly is doing. And so eventually, it’s not a matter of feature parity, but also rate of innovation and listening to our customers.

Jerod Santo: And so it’s even easier than writing your own API client, which I did for all the other integrations. But yeah, this one was so easy I forgot.