Beyang Liu: [00:24:24.07] Yeah, so the equivalent of a day in the life for an engineer is what does it take to take a feature from idea all the way to getting into production? It’s like one whole loop of the software development lifecycle, which we kind of model as this two loops that are connected. There’s kind of an inner loop, which is your kind of read/evaluate print loop in your editor, where you’re kind of quickly iterating on the idea, and making it come together… And then there’s the outer loop, which is sort of like the ops loop. You plan out the feature, you actually write it, you push it up, it gets reviewed, it needs to pass CI/CD, it gets rolled out, and then after it gets deployed, you’ve got to monitor and observe it to deal with any kind of production bugs or things that emerge.

So a day in the life of an enterprise developer or a developer working in the context of a very large existing codebase, it’s kind of like you have an idea for a feature - what’s the first thing you have to do? Well, the first thing you have to do is you have to go and acquire a bunch of context about the existing codebase. So in the pre-Sourcegraph world, what does that entail? It’s a bunch of grepping for stuff in your existing repository, maybe you’ve got to clone down a couple of additional repositories if you’re very intentional about it… Sometimes people just don’t bother, because they’re like “Ah, it’s too much work. I will just assume that this thing that I’m building doesn’t exist yet, so let me just go build it.” Sometimes it involves pinging your colleagues for help. So if you’re onboarding to a new part of the codebase, or you’re a new engineer, you’re going to go bug the senior engineer who was around when that code was written, who’s very short on time, and kind of crotchety because there’s like five other people you who have asked them a very similar question in the past two weeks… So that is a whole kind of tooth-pulling exercise. And that can take you, I don’t know, weeks, even months… In some of the codebases that we worked in while we were at Palantir, it’d literally take months just to orient on “Hey, this is the part of the code that actually needs modifying.” And there’s a bunch of false starts along the way, because you start writing something only to realize “Oh, you should be modifying this other piece over there.”

Okay, so that’s just getting to the point where you’re ready to start writing the code, right? Once you start writing the code, you’re in your editor, most of the code you need to write is probably boilerplate, or what we call boilerplate plus plus. It’s not exactly something you can copy and paste, but you basically want to find an existing example of using that one particular API that you need to use. And then you’d kind of turn your brain half off and just pattern-match against that example, because that’s the reference point. So provided you’ve found that example, then you can pattern-match against it. If you don’t find that example, then that’s kind of this arduous “Hey, let me discover how to properly use this API, because there’s no docs on how to use it.” And someone’s probably done this probably like a dozen times before, but I’ve never done it before, so now I have to essentially go and rediscover how to do that.

So you go through that exercise, there’s multiple rounds of kind of iterating on it, and you finally get to the point where “Okay, I’m ready to push it up.” You push it up to review… Maybe there’s multiple stakeholders that have to sign off on on the change. There’s the people you work directly with, maybe that’s part of your code review process, maybe you also made some modifications to some other shared libraries, and now other teams that own that code have to also sign off on it…

[00:27:57.29] A lot of times those other teams don’t necessarily care too much about the change that you’re trying to land, because they have their own priorities, and they’re incentivized to work on those, and not anything else that happens in the company. So who knows how long it’ll take to get all the stakeholders to sign off on that change. Or maybe you realize through the review that you get from a senior engineer that this whole approach was wrong, because you didn’t do enough search and discovery.

So it’s almost like this game of chutes and ladders. You have a potential to shoot all the way back to scratch; you just wasted a week, a month’s worth of work because he built the wrong thing. Best case, it takes a lot of time getting people to approve it, you finally get it approved, it rolls out to production, maybe it breaks something, maybe there’s a test that breaks that you didn’t catch locally, and so on and so forth… Once it gets into production, there’s a long tail of issues that could arise too, in terms of it triggered some bug, or maybe there was some security vulnerability that got introduced… Anyways, I don’t have to sketch this out, it’s already painful just describing this.

Nick Nisi: As your students are going through these and you’re encouraging them to use AI to assist with that, do you see the kind of questions that you might get as like support changing? Does it get influenced by the AI at all? Or does it make it harder or easier to reason about problems that might arise?

Jerod Santo: Yeah. I mean, all you’ve gotta do is ask yourself “What am I doing here?” and then “How do I do each thing?” It’s like “Well, we are selling ads.” We’re selling these sponsorship campaigns. So we have to be able to sell one, produce one, and then invoice somebody at the end of the day. And so you figure that out. And it’s like “Well, we’re hiring somebody.” “Okay, we’ve gotta be able to pay them.” And you figure that out. So you just kinda add these things as the needs arise.

And then there’s so – I mean, to I guess Silicon Valley and the greater tech industry’s credit, there’s a lot of good services you can build your business on nowadays, to where it’s not that hard to get started. It’s really not. You have a plethora of choices, there’s competition, prices aren’t too bad… You can start a small business relatively easy and cheap using internet-based tools.

Dimitri Mitropoulos: Okay, we are a Discord-first conference. And what we did is we set up hotline channels with every single one of the speakers, and we really encouraged them - and thankfully, none of them pushed back on that. We really encouraged them to just be in the Discord. If anyone wants to join, it’s Discord.SquiggleConf.com. But we have private hotline channels for all the speakers, and we use that extensively to communicate with them, to push them information, and also get information from them. It’s a lot more casual than sending an email. So if they have a question about something or they have a concern, we can just jump right on it. And we are very, very attentive to those.

I think that that’s helped us a lot, because - you don’t realize; like, when you’re starting out, there’s such a thing as email reputation, and you can totally hose your ability to send emails for like very non-spammy reasons. Like, we’re trying to give people information that they really need, but our email gets shut off, because we have 16 speakers, and if you send 16 of very similar emails, problems arise. So Discord solves that, because we can just send them the information that they need, and get back to them quickly, and they get back to us. It’s been really nice.

Chris Benson: I’m going to say, people need to join our Slack community, because we’ve had some really good conversations and questions and suggestions arise there… And the folks who are participating really know their stuff, and they’ve pointed me to a few things in recent months that have been really useful to have. So I’m spending more and more time taking pointers for that.

Jerod Santo: Corporate Open Source is Dead

Instead of linking to and IBM press release announcing their purchase of HashiCorp for $6.4 billion, let’s enjoy our friend Jeff Geerling’s analysis!

As free money dries up and profits slow, companies slash headcount almost as fast as community trust.

Can you smell what Jeff is cookin’?

Open source culture relies on trust. Trust that companies you and I helped build (even without being on the payroll) wouldn’t rugpull.

But time and time again, that trust is shattered.

He goes on to declare 2024 as the year that ‘corporate open source’ dies. But it ain’t all bad, not in Jeff’s eyes:

In fact, this could be a huge opportunity; what happened to the spunky startups like Ansible, HashiCorp, Elasticsearch, or Redis? They were lighting their industries on fire with great new software.

What happened to building up communities of developers, crossing cultural and economic barriers to make software that changed the world?

There are still projects doing that, but so many succumb to enterprise money, where eye-watering amounts of revenue puts profit over philosophy…

Maybe it’s time for a new open source rebellion. Maybe this time, money won’t change company culture as new projects arise from the ash heap. Maybe not, but at least we can try.

Jon Calhoun: I could definitely see that being challenging, especially if a bug or something gets introduced that all of a sudden spikes the amount of load… It essentially sounds like that writing rate you want to be constant, and constant’s easy. It’s just anything dynamic is where challenges arise. But I think that’s true for most software things, is if you can plan for things, it’s a lot easier, but if you can’t plan for them, it’s a lot harder to do.

Jon Calhoun: So I can agree with that. I suppose what I would say is, when you add metrics to it that you don’t know what’s important, are you at least envisioning scenarios where like “This could be useful if we needed to do these things?” Versus - I’ve seen cases where people just log everything, and if you ask them “What are some scenarios where this might be useful?” and they can’t tell you anything. They’re like “Oh, I just put that there, just in case.” And I feel like you get that a lot at times, where it feels like they haven’t thought out the actual problems that could arise, and instead they just shotgun-spray at the wall hoping something works.

Jerod Santo: Miroslav Nikolov knows the cost of a refactoring gone wrong, so he took some time to lay out the risks and how you can effectively address them and gives some solid advice on whether or not you should refactor. Here’s 4 pieces of advice, with caveats:

1. Refactor if things are getting too complicated, but, stop if you can’t prove it works.
2. Accompany new features with refactoring for areas you foresee to be subject to a change, but copy-pasting is ok until patterns arise.
3. Be proactive in finding new ways to ensure refactoring predictability, but be conservative about the assumption QA will find all the bugs.
4. Move business logic out of busy components, but be brave enough to keep the legacy code intact if the only argument is “this code looks wrong”.

Eric Clemmons: The PartyKit demos with server-side, or Socket RSC was pretty cool. But it just seems like React was able to say “We can better serve this need with these better abstractions, or paradigms, or whatever.” And also, it went from “Oh, React doesn’t have a router. You bring your own router”, that sort of thing. As part of that learning exercise and building more and more ambitious applications on React, it seems that it became necessary for React to say “Well, we’ve got to work with bundler integration, because you have to bundle an app today.” If you did a single page app - yeah, it made your React code base integrated, everything was collocated together, it was all component-driven, it was instant from the UI… But it also ended up yielding terrible dev experiences too, and terrible user experiences, because it started shipping 10 megabyte bundles as well. So I see the pendulum moving back and forth, and as a result, new abstractions just had to arise as part of that learning.

Eric Clemmons: It reminds me just how little we do as kind of like engineers to test for like the non-happy paths; those assumptions are baked into so much code. I mean, you can look at failure states… And it’s actually one of the beautiful things about being in the frontend space as a web developer, is that that’s a portable skill, of being able to work in the browser. You hand me a URL if something’s going wrong, and I’m going to be able to look at these really rich tools that you don’t get when you console.log in Node. I can actually expand out variables, and copy to the console, like you mentioned. It’s almost always like at the network layer, where we see that some happy path assumption isn’t true anymore. And then that’s whenever all the problems arise.

Rich Harris: Right. Right. But if you’re doing something very exploratory, and you don’t yet know what the shape of the system that you’re building is… Like, you’re kind of like uncovering the design as you write it, that’s a really bad time to be thinking about type safety. And I’ve always resisted adding types at that very, very early stage of a project that I don’t yet understand the shape of.

But the other one is the build tooling complexity that I referred to earlier. I’ve got a great example just from a few days ago… I have this codebase that is written in TypeScript, and the tests all passed fine locally. Type checking fine locally. Linting fine locally. Send it to GitHub, and the CI is failing. And it’s failing with the most inscrutable error imaginable. And it turned out from like a very long time spent digging through stack traces and something that somehow a module deep inside Babel was getting executed twice, and it was trying to update something on the exports object that had already been updated, and it was saying, “Well, this is read-only. You can’t do this.” And I couldn’t reproduce it locally. There was nothing I could do. I made sure all my versions of everything were perfectly in sync, and I just couldn’t figure it out. And I spent so long trying to sort this out, and I never would have had to if we just hadn’t been using .ts files in the first place. Because the only reason that it happened is because our Rollup configuration mentioned Babel somewhere.

So without being able to actually solve the problem, the way that I worked around it in the end was just by not using Rollup, and using ESBuild instead. And so now the build artifact is 10% larger, because Rollup generates more efficient output. That is the cost of having to deal with this bull***t. And the real solution is to just not be writing .ts files there in the first place, because then these things don’t even arise. It’s not the fault of TypeScript, but it is the fault of the enormous complex dependency chains that we’ve had to use in order to deal with all of this complexity.

Jerod Santo: It’s the only thing. Where would a competitor come from? Where would a competing project potentially arise?

Eric Clemmons: Brilliant. It reminds me just how little we do as kind of like engineers to test for like the non-happy paths; those assumptions are baked into so much code. I mean, you can look at failure states… And it’s actually one of the beautiful things about being in the frontend space as a web developer, is that that’s a portable skill, of being able to work in the browser. You hand me a URL if something’s going wrong, and I’m going to be able to look at these really rich tools that you don’t get when you console.log in Node. I can actually expand out variables, and copy to the console, like you mentioned. It’s almost always like at the network layer, where we see that some happy path assumption isn’t true anymore. And then that’s whenever all the problems arise.

Chris Benson: So the thing that I think people focus on the wrong thing on this topic - they focus on whether the existing generative models as we’re now calling them all the time are leading us into kind of artificial general intelligence, AGI. Whether it’s aware, whether it’s conscious, and whether it would have an intent to attack. And I think that that completely misses the point. I think if you want to – and I’ll take two seconds and argue both sides for a second. If you want to argue against current technology being a risk to humanity, then you’re kind of pointing and saying, “Clearly, these models are not conscious, and they are not intelligent in having a broader awareness of the world around, and having their own motivations, and such.” And so the people that are arguing that side scoff at the very nature of the fact that you could suggest that a model could threaten humanity. And within the context of that set of arguments, I think they’re absolutely right.

But there’s also another side to it, which is actually where I am kind of migrating a little bit in my own personal thinking… And that is “What if it doesn’t take AGI to be a threat to humanity?” What if the threat can arise from the fact that – for one thing, we have humans in the mix; we have humans with motivations, that create models, and have specific things that they’re trying to achieve… And so you can take the power of models, and you can shape them in certain ways to address different tasks. And so it might be, possibly, that if there is a danger to humanity - which I don’t know, I’m just speculating… But if there is, it’s shaping a bunch of models that by themselves can do one task really well, or as the generative ones, they can give you sets of things… But you combine them in ways with software and with human intent to do damage. And so that’s what I’m more concerned about, is not that the models will awake, and suddenly become conscious, and decide that they don’t like me very much, and they want to get me; I think that I’m a lot more worried about humans orchestrating a bunch of powerful tools, and maybe automating those tools in such a way that the tool keeps going. It doesn’t take constant intervention. That’s the type of thing that I would actually give a little bit of credence to in my own personal thinking in terms of - when I say that, meaning not that it’s happening, but meaning that it’s worthy of consideration. And when we talk about things like AI ethics and such, that’s where I would focus, is that there are external concerns that don’t require AGI and don’t require consciousness to achieve some really bad outcomes. What do you think of that?

Cory Doctorow: So that’s chickenization. And if chickenization sounds familiar, it’s what Uber drivers have, right? It’s what DoorDash has. It’s what – there’s a company called Arise. They hire primarily black women to do work from home call center stuff, and each of these people are misclassified as contractors. They have to start an LLC. And then because they’re a company, they have to bid on the job. They work for like Carnival cruises, and Disney, and whatever - they have to pay to be trained to be the phone operator. And then they’re listened in on constantly, and if their kids cry in the background, they are struck off; they lose the job, but they still owe the money for the training. And there’s a penalty for cancellation on their side, where if they quit their jobs, they have to pay their boss to quit their jobs. So that’s a form of chickenization as well.

Chickenization is kind of working its way up the privilege gradient from the most blue collar workers now to a more middle-class group of workers. It’s your boss’s ideal, right? All the risk is pushed on to you, all the rewards go to them. Every business owner would love to have that as their arrangement, right?

[24:18] So what’s a centaur? Well, a centaur in AI research is someone who’s machine-assisted, right? I’m a chess grandmaster, you’ve got a chess playing software program. Neither of us are as good on our own as we are together. We can beat opponents together that we couldn’t beat on our own. So that’s a centaur. But a reverse center is when the machine is in charge, and you’re at support. So say you’re an Amazon warehouse worker who’s wearing haptic gloves that go “Bzz! You need to pick that up. Bzz! You need to turn your body. Bzz! You need to drop this thing in the box. Bzz! Bzz! Bzz!”

Or say you’re an Amazon driver, right? You’ve got a facial recognition camera that’s watching your eye movements and penalizing you if you look away from the road, you’ve got a clock that’s timing your drives, that’s telling you you have to turn left, even if turning left would result in your death, and then it docks you for doing it. Machine says no - that’s a reverse centaur, right? It’s when you are the disposable meat sack for the all-important algorithm.

Mat Ryer: So here’s Jack Sparrow reading filepath walk. [in Jack Sparrow voice] “Walk walks the file tree rooted at root, mate, calling walkFn for each file or directory in the tree, including root. All errors that arise visiting files and directories are filtered by walkFn. The files are walked in lexical order, mate, which makes the output deterministic, but it means that for very large directories Walk can be inefficient. Walk does not follow symbolic links. Savvy?”

Shawn Wang: So I’ll read out this quote… So Sam Altman, CEO of OpenAI, says “I don’t think we’ll still be doing prompt engineering in five years. It’s not about figuring out how to hack the prompt by adding one magic word to the end that changes everything else. What will matter is the quality of ideas and the understanding that you want.” I think that is the prevailing view, and I think as people change models, they are understanding the importance of this.

So when Stable Diffusion 1 came out, everyone was like, “Alright, we know how to do this. I’m going to build an entire business on this” etc. And then Stable Diffusion 2 came out and everything broke. All the prompts stopped working, because they just expected a different model, and you have to increase your negative prompting, and people are like “What is negative prompting?” etc. These are all new techniques that arise out of the model, and this is going to happen again and again and again, because you’re relying on a very, very brittle foundation.

Ultimately, what we want to get people to is computers should understand what we want. And if we haven’t specified it well enough, they should be able to ask us what we want, and we should be able to tell them in some capacity, and eventually, they should produce something that we like. That is the ultimate alignment problem.

We talk about AI a lot, and you hear about this alignment problem, which is basically some amount of getting it to do what we want it to do, which is a harder problem than it sounds until you work with a programmer, and try to give them product specs and see how many different ways they can get it wrong. But yeah, this is an interesting form of the alignment problem, and it interestingly has a very strong tie with Neuralink as well, because the problem, ultimately, is the amount of bandwidth that we can transfer from our brain to an artificial brain. And right now it’s prompts. But why does it have to be prompts? It could be images. That’s why you have image-to-image in Stable Diffusion. And it could also be brain neural connections. So there’s a lot in there; I’ll give you time to pick on whatever you respond to…

Nick Nisi: And that really highlights a really important point and perk of something like Playwright… Because I’ll tell you, we have a lot of React tests in our app, and for a majority of them we’re using Jest plus React testing library, which is kind of giving us this test from the users’ perspective thing; you have a screen object, and you can get my ID and all this to find how it’s actually rendered. But it’s not rendered in a browser, it’s rendered with jsdom, and so it’s not like a real environment. It’s not often, but things do arise that are not actually happening, or happening in jsdom and not happening in an actual browser… And then when we go to debug it, which is quite often, it’s like, the only thing that we have is screen.debug, which spits out a bunch of HTML, and it’s like “Well figure it out from there…”

Chris Benson: I know that we have seen evolutions over quite a long time in OCR, in terms of that… You mentioned something though that made me curious. You were talking about like if you are using one of those early models that were pretrained, and then you didn’t get what you wanted out of it… Can you talk a little bit about what kinds of problems might arise in terms of like why weren’t they getting it out of those models, to define a little bit about the space that you’re fixing going forward?

Chris Benson: [18:01] Absolutely. I mean, I would kind of summarize that by saying that diversity matters. Diversity of experience, and diversity of the challenges of a particular culture or a group of people, and that the complexities that are arising in their experiences have to be accounted for if they want to use AI in that toolbox to address those things. So going back to your original point, which I thought made a lot of sense, was the fact that if you’re not customizing how you’re using AI and the focus of your research on the particular needs of your area, and those issues which arise from your point in a diverse world, you’ll get a substandard outcome from that.

So you can’t take something that might be a good approach in the United States and drop it into a country that has a very different culture and a very different economy and stuff; it’s not going to work well. So it takes that thoughtfulness. When I see somebody, meaning like a nation or a corporation or something like that, just kind of copying what the others are doing, it always makes me cringe a little bit, because it shows me that either they didn’t understand the need for that focus and that customization, or they simply weren’t thoughtful enough about it.

Chris Benson: It’s been a curious time lately. We’ve seen some interesting things arise… And yeah, absolutely. And people’s take on it is a little curious, too. I know we can go down that road as we get there.

Arisa Fukuzaki: Thank you, too.

Jerod Santo: I’ve been thinking about a lot of the interpersonal conflict and the problems that arise… I’ve been trying to think of like, “Well, how would you deal with this circumstance in mob programming?” But the more I think about it, there’s no new problems here. It might exacerbate existing strengths and weaknesses. Like, if you have the alpha male or the person who’s going to be bossy or egotistical, who’s regularly in your meetings, and on your team, but not in the same room with you - well, that person’s going to be in the same room with you, and so you may need to work through those things with that person quicker, or more often.

John Nunemaker: Yeah, that was pretty cool. That was actually kind of like a – I won’t say accident, too… Because I was kind of maintaining it, because everybody else had kind of moved on, and so I just kept it running, and occasionally would add things to it, and stuff; honestly, in my free time. And I remember we started getting a couple support requests here and there, people wanting to buy it… They were like, “Hey, is it for sale?” And I remember asking, and everybody’s like “I don’t know if it’s for sale. We’d have to vet the buyer.” And that kept coming back. So finally, I was like, “Alright, this isn’t gonna happen. It’s just dying right now. So if we don’t do something with it, it is gonna permanently die”, and I was like – I just like it, again, because looking back, it’s the theme or the thread that ties a bunch of that story together. So I was like, “You know what - I’ll vet the buyer. Are you guys okay if I vet the buyer?” And everyone was like “Yeah, that sounds fine.” “Everyone” being my manager, and other people, levels up. So I was like, “Alright, I’ll vet buyers.”

And then as I was thinking about that over the weekend, I remember I was like, “Why don’t I just buy it? I’m already maintaining it basically in my free time. I wouldn’t be any different, except now I would have ROI if something good happens to it. I’d have ROI.” That, again, comes back to impact. So I think on that front, that’s kind of where – so I was like, “You know what - let’s try it.” Again, I don’t think I’ve thought about it until like we’re talking about it now, but… Common thread - you never know it until you ask. So worst-case scenario, when you negotiate, they say no.

[51:57] I remember I think I offered – I don’t even remember. At that point it was losing like $1,000/month or more, just from hosting, and AWS bills, and stuff, and I was like, “Alright, I’ll give you $5,000.” So basically, this shows that I’m real, I’m genuinely offering for it, and I will take on the burden of $1,000/month, vetting the buyer, all the problems that could possibly arise from selling it, and any possible negative impact of closing it down, or any of that kind of stuff.

I sent that up the tree, and at some point they were like, “Yes.” They dropped the price to like – I don’t know if I’m allowed to talk about any of that kind of stuff, but it was like an order of magnitude lower. They were just like –

Amal Hussein: Yeah. So I went through my own journey with automation, and overcoming my fear of Bash, and the term– I would say going beyond surface-level terminal commands which most people are familiar with doing day-to-day work… And I have to say, just from my own experience, it’s a journey, and not everybody’s path is the same, you’re not solving the same problems as other people… I really love this distinction of pipeline versus source code. If you can kind of really maybe just draw the line there - write the code that does the thing that you want it to do, and then tackle the problem of how does it connect to everything else and how do you get things in and out of it as a secondary step. Because I think sometimes when you start doing things that are so brand new, and – you know, it can be daunting. It’s hard to get steam and momentum. So you can kind of build on your successes incrementally in that way. But it really is a journey, and it’s a journey that never ends; operating systems and languages, libraries, tools, new problems arise, new APIs come out, old problems creep up, security issues pop up… It’s very much a living, breathing ecosystem, even though a lot of us think of the terminal as very static. Did y’all know there’s new versions of Git? There’s features being added to Git. Linux is a living, breathing project… There’s just so much. So it’s a journey, but it’s foundational knowledge that really is like the missing manual for many web developers who started out kind of doing higher-level software design.