Go Time ā€“ Episode #253

Spooky stories to scare devs šŸ‘»

with Mat, Kris, Natalie, Johnny & Dee Kitchen

All Episodes

Mat Ryer gathers a gang of ghouls and ghosts to tell spooky developer stories! Join us to hear tales of Matā€™s $1k nightmare, Deeā€™s infinite loop of horror, Natalieā€™s haunted time as a junior dev & many, many more.

Featuring

Sponsors

Sourcegraph ā€“ Transform your code into a queryable database to create customizable visual dashboards in seconds. Sourcegraph recently launched Code Insights ā€” now you can track what really matters to you and your team in your codebase. See how other teams are using this awesome feature at about.sourcegraph.com/code-insights

FireHydrant ā€“ The reliability platform for every developer. Incidents impact everyone, not just SREs. FireHydrant gives teams the tools to maintain service catalogs, respond to incidents, communicate through status pages, and learn with retrospectives. Small teams up to 10 people can get started for free with all FireHydrant features included. No credit card required to sign up. Learn more at firehydrant.com/

Notes & Links

šŸ“ Edit Notes

Chapters

1 00:00 Opener 00:30
2 00:30 Sponsor: Sourcegraph 02:24
3 02:54 Intro 00:38
4 03:32 Welcome to Ghost Time! 01:28
5 05:00 Are you scared of horror movies? 02:21
6 07:21 Dee tells us about bluemonday 02:38
7 09:59 Mat's $1000 nightmare 02:14
8 12:13 Here's Johnny's on avoiding massive bug bills 01:20
9 13:33 Dee's infinite loop of horror 03:13
10 16:46 What is a greedy regex? 03:04
11 19:51 Natalie's haunted time as a junior dev 02:12
12 22:03 Advice for junior devs to stay out of trouble 03:20
13 25:23 Juniors should be able to make mistakes 01:29
14 27:03 Sponsor: FireHydrant 01:18
15 28:33 Campfire chill session 01:24
16 29:57 On hot CPUs and energy costs 01:10
17 31:07 Dee's support request from hell 03:02
18 34:09 Kris won't share imaginary marshmallows 00:32
19 34:41 Johnny's infinite loop of terror 04:12
20 38:53 On devs and consequences for their mistakes 03:06
21 42:05 Kris's spooky security hole 03:21
22 45:27 Dee thinks too secure is a problem 01:12
23 46:39 What's your pin number, Dee? 00:31
24 47:10 How are we gonna put this camp fire out? 00:34
25 47:43 Dee's SQL typo still haunts him 01:54
26 49:37 What *is* that sound?! 01:33
27 51:11 It's time for Unpopular Opinions! 01:02
28 52:12 Dee's unpop 01:36
29 53:48 We love you, Mat! 00:56
30 54:45 Natalie's unpop 03:37
31 58:21 We're afraid it's the end 00:33
32 59:00 Outro 00:56

Transcript

šŸ“ Edit Transcript

Changelog

Play the audio to listen along while you enjoy the transcript. šŸŽ§

Hello, and welcome to Ghost Time. Iā€™m Matt Ryer. Today weā€™re talking about tech horror stories. Iā€™m joined, as everā€¦ Johnny Boo! Johnny Boursiquot is here. Hello, Johnny.

Hello, Mattā€¦

Welcome to the spooky Go Time episode. Again, in the spirit of it ā€“ youā€™ve really got to get in the spirit of itā€¦ No? Yeahā€¦ [laughter] Weā€™re also joined here by Kris Brandow. Spooky ghostsā€¦ Hello, Kris.

Hello! Iā€™m back, again. Finally.

Yes, welcome back again. Weā€™re also joined by your friend and mine, Natalie PistunoWitch. Hello, Natalie.

Yeah, getting into the spiritā€¦ [laughter] We have a special guest joining usā€¦ Youā€™re not gonna believe this. Itā€™s spoopy Dee. Dee Kitchen. Welcome, Dee.

Thank you. Iā€™m enjoying being here. I even got the backdrop rightā€¦

Good. Well, thatā€™s a good start, because weā€™ve literally just started. I mean, really, the only way is down now, in a lot of waysā€¦ But hopefully we donā€™t go there. But we are talking about scary things today. How are you generally with scary things, Dee?

Thatā€™s my career. All of it.

[laughs] Yeah. Okay. Anyone else? Anyone scared of ghosts?

Iā€™m scared of Ghost Time.

Yeah, youā€™re scared of Ghost Time.

Iā€™m actually scared of horror movies. I donā€™t really watch them.

Oh, yeah?

Heebie-jeebies for me.

Hmā€¦

I just find them boring.

Yeahā€¦

Did you come from the industry? I remember you saying, Kris, that you see a movie, the first few minutes, and you know exactly how itā€™s gonna be laid out.

Yeahā€¦ Itā€™s the curse of having a creative writing degree and specializing in screenwritingā€¦ All movies are just kind of ruined.

Itā€™s all generics.

Thatā€™s what I said, youā€™d be happier if youā€™re just an idiot. Iā€™ve always said that.

Is this something you know from personal experience, Mat? [laughter]

Ooh, shots firedā€¦

I also donā€™t really like horror films, especially if thereā€™s any kind of contradiction in it. I canā€™t deal with that. Like, if thereā€™s an invisible thing that can grab you, first of all, itā€™s invisible, it would be blind; weā€™ve covered this. But also, if it can grab you, you can grab it, you can hurt itā€¦ Like, itā€™s not fair. Itā€™s like when the physics donā€™t apply generally; then Iā€™m just out. And I just tell everyone in the cinema, Iā€™m like, ā€œSorry, everyone. I canā€™t stay. Iā€™ve got to go because of the inconsistencies of the physics.ā€ And I just go and get some popcorn and go.

[06:10] Do you get sweet or salty popcorn?

Salty.

Do you have a choice? Or is it always salty?

No, you have a choice. [laughter] Donā€™t you have a choice? Yeahā€¦ What do you mean? Like, the police are going around saying ā€œHey, are you only having salty? What are you doing?ā€

I only discovered in my late 20s that some other countries sell popcorn that is not just salty in the cinema.

Oh, right? Yeahā€¦

And then I came to the US, and then itā€™s like not just two flavors, but 15.

Yeah, of course. Yeah. Yeah.

Thatā€™s a horror story thereā€¦

Yeah. You can choose individual bits of corn, and have different flavors, and just have as many as you want. You just program it. You do it as an app, and then it pops it on demand.

You say that, but we do have soda machines where you can choose your own flavor.

Yeah, Iā€™ve seen that.

Those freestyle Coke thingsā€¦

Has anyone come up with a good one yet? Because I imagine theyā€™re all terrible. Someoneā€™s like, ā€œYou know what? Iā€™ve accidentally pressed these three, and Iā€™ve made a brand new flavor that never existed before.ā€

Well, noā€¦ I think they make it so you canā€™t make any truly terrible-tasting ones, because that would be perhaps bad for them, soā€¦

Oh, really? Clever. How did they do that? Well, weā€™ll never knowā€¦ Well, speaking of horror stories - letā€™s get into this, shall we? Who Wants to kick us off with a spooky story? Oh, by the way, we should actually introduce Dee, because Dee, you wrote a package that I think a lot of people here will be familiar with. Can you tell us about bluemonday?

Oh, bluemondayā€¦ Itā€™s named because there was a package weā€™ll call Black Friday, which has all the best markdowns, and itā€™s a markdown package. And after youā€™ve generated markdown, markdown can include HTML, which makes it dangerous. Itā€™s probably youā€™re using this because youā€™ve got user-generated content, and you want to sanitize it. So bluemonday is named after the New Order song, but follows Black Fridayā€¦ And it basically sanitizers HTML; itā€™s the only Go package that sanitizers HTML, which is a foolish and reckless thing to attempt to take on. But thatā€™s what I did.

Amazing. And what do you like about it, and what donā€™t you like about it?

I like that it works. [laughter] Itā€™s a streaming parser, itā€™s got fixed memory, so you can use it quite comfortably in a lot of situations, and [unintelligible 00:08:20.10] I donā€™t like when people tell me thereā€™s security issues with it, and then I have to go ā€œOh, Iā€™m supposed to take this open source thing seriously?ā€ I do appreciate it. I should actually say, I do appreciate security reports. But at the same time, you never can predict when theyā€™re going to turn up, and you never know what kind of words youā€™re going to open to try and actually figure it out.

Yeah, there must be a lot of responsibility, actually, because it is a package that is used, and quite trusted.

Yeah, itā€™s used. I donā€™t know how many stars itā€™s got, but the stars donā€™t portray the amount of times itā€™s used. Like, itā€™s used in Hugo, and everyone uses Hugo. And this is the HTML sanitizer that keeps Hugo safe. And itā€™s used in so many things. Itā€™s got literally thousands of dependencies. Do I take it seriously and stressfully? No. No, I donā€™t. Iā€™ve figured that if someone is brave enough to take an open source project with an MIT license or a BSD 3-Clause, or whatever it is, and incorporate into their production software, thatā€™s on them.

Okay, fair enough. Well, I have done that, butā€¦ Good to know. I genuinely have used it though, quite a few times, soā€¦ I like it, because itā€™s like you opt into what you want to support. You explicitly say the things that you want to allow.

Yeah, thereā€™s no way of defining what makes a good HTML sanitizer. Everyoneā€™s got a different rule, depending on their use case. But the Java OWASP, Open Web Application Security thing - There sanitize it to find this really beautiful interface for sort of going, ā€œI want to allow images, but I donā€™t want to allow these images that end in .gif.ā€ And I copied their API, and then extended it for my own use. So yeah, itā€™s a really good way of doing it.

[09:58] Nice. Okay, well, Iā€™m gonna tell you about a horror story in tech of mine, that happened quite recentlyā€¦ I have this project which interacts with Twitter, and it interacts with the Twitter APIā€¦ And so it polls results and then compares them, and stuff. And thatā€™s just one of the things it does at a regular interval. And then what happened recently was something happened where the API key changed, and that request failed. And because of the way I was doing it in GCP, it meant essentially that it would retry. And because it was scheduled, it kept compounding. And this ran up a $1,000 bill for me, for yours trulyā€¦ $1,000 given, paid, goneā€¦ So thatā€™s a bit of a tech horror story. And the advice for me ā€“

Is it tax-deductible? [laughter]

Probablyā€¦

AWS famously refunds you if you get something wrong. Did GCP not do that?

I donā€™t know. Itā€™s quite recent. I havenā€™t yet tried that. Do you think I should get in touch with support and see if theyā€™ll ā€“

$1,000 would motivate me.

Yeah. There you go, $1,000. Okay, well, Iā€™ll try it, and Iā€™ll let the listeners know how we get onā€¦

It could have been worse, right?

It could have been $2000ā€¦

Or just $1,001. It would be worse, wouldnā€™t it?

What would you do, Johnny, if you saw thatā€¦?

Iā€™d call you and say ā€œHey, youā€™ve got a grand? I hear youā€™re loadedā€¦ And just wasting $1,000 here, $1,000 there, on your bugs, and stuffā€¦ā€

Honestly, when I found out about it, I wanted to just karate-chop the air. That was the kind of spooky reaction I had to it. Just like [Whew] in the air. Angry. But yeah, itā€™s a good lesson though. Like, set budgets and stuff on your things.

Do set an alarm, yeah. Budget alarms.

Yeah. Observability.

Yes, yes. And youā€™d know a thing or two about that, yeah?

Yeahā€¦ Okay, who can beat my $1,000 bill? Not $1,000 bill? Oh yeah, it was a $1,000 bill; but that makes it sound like it was one thing, doesnā€™t it? Like a single bill that had $1,000 on it; so itā€™s not that. It was just paid through a bank transfer. Okay, whoā€™s got another one?

I have one that could have cost many 1000s of dollarsā€¦

Oh, Johnnyā€¦

ā€¦if it wasnā€™t spotted. So one of the things you can do with function as a service things, like AWS Lambda, for example, is that you can trigger a Lambda when you write an object to an S3 bucket. Word of advice - do not have your Lambdas write to a bucket that they are themselves responding to. [laughter]

Ohhā€¦!

Because thatā€™s gonna give you a very nasty bill. Yeah, and you will not like what you see. So yeah, thankfully, budget alarms came to the rescueā€¦ [laughs]

Yeah, there you go. Thatā€™s the lesson there. So what happens is, an object goes in the first time, that triggers the Lambda, the Lambda then writes something into that same bucket, which then triggers another Lambda, which then writes something.

And how quickly does that get out of hand?

Very quickly. [laughs] Like, if you want to see how well a Lambda scales on your own dime, you can do that. And, yeah, itā€™ll cost you money very quickly.

Wow. Yeah. Okay. Pretty goodā€¦ But yeah, thatā€™s ā€“ the alerts came to the rescue. Nice one. Okay, anyone else got one for us?

Iā€™ve got another infinite loop oneā€¦ Are we allowed to name company names? I donā€™t know, maybe itā€™s internal and I shouldnā€™tā€¦

Yeah, I donā€™t know.

I worked for a certain company which has an orange logo that has a bit of a light shining behind it, and they man-in-the-middle the entire internetā€¦ Now, with that in mind, when I was working for said company, in their DDoS teamā€¦ We didnā€™t DDoS people; we were protecting against DDoSes.

Yeah, Iā€™ve wonderedā€¦ The DDoS team!

[unintelligible 00:13:56.02] thatā€™s the opposite of what weā€™re doing. No, we were trying to protect, and they have a systemā€¦ Theyā€™ve got all these 200 POPs (points of presence), and thousands and thousands of serversā€¦ And every single one of these is protecting some of the traffic; each machine can do like 20,000 requests per second.

[14:15] And yet, they need to be able to actually show the value back to the customer, and make these sort of decisions centrally. So you send all the logs somewhere, and theyā€™re all been sent to one data center. So what you end up with is like, if youā€™re doing globally 10 million requests per second, you get 10 million log lines per second in one place.

Niceā€¦!

A certain customer, on a certain point in time - industry and type to be non-disclosed - wrote an infinite loop in their client, and it basically spikes 8 million requests per second on top of our normal load.

Oh, wow.

And they basically broke our logging, the entire visibility. So they were effectively under attack, but now flying blind, because we couldnā€™t see anything because theyā€™d broken all the logging. That was not a good day.

Yeahā€¦ That one doesnā€™t sound fun? What happened?

We figured out which customer it was, but we couldnā€™t figure out the rest. But we asked them what theyā€™ve done, and they figured out that bit and stopped it.

Oh, wow.

They fessed up to it, they owned up to it? Somebody wrote an infinite loop?

Yes. [laughs] I think they realizedā€¦ They must have seen what was happening on their side.

So they didnā€™t pull a well ā€“ well, I will not name names, but they didnā€™t blame it on on of the interns?

Oh, weā€™ve got a certain thing where actually an intern did that. I had that intern, and heā€™s actually really good tie. Heā€™s become a full-time engineer in that team. Heā€™s really good. Learned a lesson that none of us will replicate.

Oh, yeah. I love interns. I just donā€™t like to throw them under the bus when something goes wrong with my companyā€¦ [laughs]

No, that one was interestingā€¦ Also, at said man-in-the-middle company, we had a system ā€“ the system was brilliant, right? You could send an instruction to any machine in the world in under 10 seconds, and every machine received the same instruction. And thatā€™s great when you want to say thereā€™s a new domain name, because you just have the whole world at the same time. But itā€™s really bad when you say thereā€™s a new way to stop traffic. And weā€™ve made a greedy regex. And the greedy regex was the problem. Now, frankly, the system shouldnā€™t have allowed it, but the system did allow it. And we were all at lunch, it was an all-hands lunch, and the next thing we know, we just get people running and going ā€œThe internetā€™s down.ā€ Because we used our own systems. And we lost everything internally at the same time. That was hard, too.

I feel like there are many lessons thereā€¦

There was a lot of lessons.

What weā€™re having for lunchā€¦ [laughter]

Lunch went coldā€¦

Thatā€™s scary, ainā€™t it? But hang onā€¦ So can you explain - someone that doesnā€™t know what a greedy regex isā€¦ What do you mean by that?

Yeah, a greedy regexā€¦ I mean, if you do something like .*, what youā€™re saying is match any character, any number of times. So if you .*.*, youā€™ve now exploded this any character, any number of times, followed by any character, any number of times. And what youā€™re doing is youā€™re increasing the CPU computation. Youā€™re still putting the same fixed input in, but what it can match is now ā€“ youā€™ve doubled the possibility in just that one go. And essentially, thatā€™s what happened. But some of the inputs were web pages and web traffic, so they werenā€™t small. They were quite large inputs. And under that condition, they consumed all the CPU.

So wherever this rule was applied - and we had shipped it globally to every single website, every single bit of traffic - we fried every single machine instantly. So it was about four hours for us to recover from that. And the teams I saw, they did interesting things. We were connecting directly to machines, and looking at the Prometheus on them, because we had no other observability.

Wowā€¦ And what was the impact of that? How many people were affected?

Everything was affected. We knocked out a lot. DNS, TLS, HTTP, everything. It was one of those nightmare scenarios. And you sit there as a company, you sit there and you sort of go, ā€œWhat are these meteorites?ā€ the dinosaurs went extinct by a meteorite. ā€œAs a company or product service offering, whatā€™s the meteorite thatā€™s going to hit us?ā€ At that company, we were hit by every meteorite we predicted. It survived, but still, on the days when they hit, it lays waste to everything.

[18:15] And everyone has them. The thing that youā€™ve got to realize is when youā€™re there, youā€™ve got to sympathize with ā€“ you can accidentally see another company go through thisā€¦ Theyā€™re having a bad day, and youā€™ve got to sympathize, because one of those meteorites is gonna hit you one day.

Yeah, we see the HugOps goes around often on social media, and things, of people sending their support in those difficult timesā€¦

Yeah, thereā€™s a good tradition of sending cakes to each other to sort of go ā€œThinking of youā€¦ā€

Yeah.

And try and not have your salespeople ambulance-chase.

How well was your break-glass procedures documented?

It was pretty good. We were lucky that this happened during a London lunch hour when all of the SRE team - the original SRE team - were there. Itā€™s possible for a few people to break glass; they could do so in about five minutes once we actually understood what we actually had to do. It took about 20 minutes for us to gain any visibility and to sort of understand, ā€œHey, itā€™s this feature. Go turn that off.ā€

Regular expressions, huhā€¦

Thatā€™s still hard. Theyā€™re hard for everyone. Easy to write, hard to understand what theyā€™re doing.

Why are they called that? What is regular about them?

Thatā€™s out of my domain. I donā€™t know if anyoneā€™s got the answer for thatā€¦

No, genuinelyā€¦

Iā€™ve just watched a talk from Strange Loop about regular expressions, and the speaker did go into this, and Iā€™ve completely forgotten what she said. But we can probably put that talk in the show notes. It was a really good one, about just like the history, like Ken Thompson came up; it was pretty cool. So Iā€™m like, ā€œOh, I know that dudeā€¦ā€ Yeah, but no, it has to do with like mathy things, and finite automata, and all of thatā€¦

Yeah.

When I was a junior, which is the closest to intern I was, I was working with a team lead, and when we deployed something together, we looked at itā€¦ And I forget what it was exactly, but this is a company that receives a lot of pings from the SDK of the many clients, and itā€™s all real time. And if that is not logged, then the entire transaction flow, user flow is gone forever. And we deployed something we worked on together, we worked on it for half a day, we tested it, we did all the good practices - because thatā€™s how you do with a junior, you want to show that youā€™re very thorough, and you go through all the tests, deploy, look at all the metrics, and see that it behaves as expected. And then he went to lunch, and then I stayed. Ta-daā€¦! [laughter] And then I proceeded to do something else, and then suddenly, a weird behavior started pinging Slack, all the monitoring channels, that ā€œSomethingā€™s wrong, somethingā€™s weird.ā€

And then some of the colleagues that were there tried to see where it comes from, and we couldnā€™t figure this out in 15 minutes, and then, bravely, I came to the head of the DevOps team - there was no SRE team at the time - and I said, ā€œI think itā€™s this thing that we did. Can I revert it?ā€ Nobody else from all the other senior people that was - I donā€™t have another better word than brave, but I donā€™t use the word braveā€¦ Nobody else wanted to do anything about that, because nobody was sure. And then I was like, ā€œLetā€™s do this. Letā€™s try. At worst ā€“ it cannot get much worse than thatā€, and reverting that indeed succeeded, and then we were all very happy. And then I was like, ā€œI think I know how to fix it. Can I try?ā€ and then he looked at me and said ā€œNo. Stay away.ā€ [laughter] Yeah, I to next-level brave.

Yeah. Well, what a great way to learn stuff though, isnā€™t it?

Break them, yeah.

How often does that memory come back to haunt you, Natalie?

Every time Iā€™m asked, at leastā€¦ [laughs] Every time I get to speak with other junior people. To give the good example of it, obviously, weā€™ll break something, soā€¦

Right.

ā€¦be reasonable about your expectations.

[22:01] Nice.

Iā€™m curious if you have a way ā€“ like, now that youā€™re older and wiser, and youā€™ve been through the experience, and it was a great teacherā€¦ Iā€™m wondering, do you have strategies now for doing things that are scary, that could break things? Like, do you have a strategy for tackling that now?

The thing is we did everything right at the time, right? So we did all the tests we could think of, we thought what do we expect in the logs, in the monitoring, in the dashboard, and we observed. So the only thing that I would do different is not deploy before lunch. [laughs] So you can, I guess, observe for longer. Speaking of spooky things, right?

What were you having for lunch?

I donā€™t think I had lunch that day. I mean, I was like nudging stuff, but I donā€™t think I had a lunch-lunch.

Yeah, thereā€™s another theme emerging hereā€¦ One of the main reasons to write good code is so you can just have lunchā€¦ For a good reason.

Thatā€™s another thing I would the differently; I would write good code. Yeah. [laughter]

Thatā€™s optional. If youā€™ve got good tests, you donā€™t need good code. Controversial.

So to me, what I usually tell junior members of staff is to be like, ā€œLook, we expect you to break things. Itā€™s just part of sort of maturing as an engineer. What is helpful, and even if you follow the playbooks and you do the right things and everything else, sometimes things will go wrong; whoever writes these things, how many people have touched the documentation youā€™re looking at, thereā€™s a chance that they might have overlooked something, or they took something for granted that you as a junior havenā€™t encountered yet, so you donā€™t take it for granted. So thereā€™s some steps in between. So thereā€™s some unwritten things in between the lines, that are sort of being conveyed, that you have not yet matured enough to kind of pick up on. So just document every step you take.ā€

Itā€™s much easier for the team to go back and say ā€œOkay, what ā€“ā€ Because the first thing theyā€™re going to do is ask you, ā€œWhat did you do?!ā€ Right? [laughter] So after everybody calms down, you can say, ā€œWell, these are the steps I took.ā€ Even to this day, I do this, right? If Iā€™m working with a system that I havenā€™t come across before, and I donā€™t know what the side effects are of the things that Iā€™m going to do, Iā€™ll literally, in a document somewhere, literally be copying the commands that Iā€™m issuing in the command line - Iā€™m literally gonna copy them into this doc, and Iā€™m basically Iā€™m capturing the output as I go.

Now, one could say thatā€™s sort of extremeā€¦ I mean, again, if there was a playbook for it, if there was some automation that I could just click the button, or issue the command and let it do its thingā€¦ But if I have to do this step-by-step thing, that means thereā€™s no playbook for it. That means thereā€™s no automation, thereā€™s no script, whatever it is. So Iā€™m just gonna be literally documenting what Iā€™m doing step by step by step, and if something breaks, I know exactly what broke. Or if I canā€™t tell what broke, I can ask my team and say, ā€œHey, these are the steps I was following.ā€ Right? And then nine times out of ten - maybe Iā€™m just lucky - theyā€™ll be like, ā€œOh, yeah, this thing - you should have done this command before you do thisā€, whatever, and then we find out that thereā€™s a gap in the documentation, or a gap in the process, or something like that.

Yeah. And you can update it.

Yeah. So literally, just track what youā€™re doing; that may actually end up helping you. Hey, guess what - that might even turn into a playbook or an opportunity for automation for whatever it is that youā€™re working on.

Yeah, itā€™s like step one, SSH in. Step two, check the Go version. Step three, drop all the database tables. [laughter] Spot the problem.

Itā€™s just pen testing. Itā€™s fine.

Exactly. Yeah. You shouldnā€™t be able to do that, really. If you can do thatā€¦

I think thatā€™s important though, that theyā€™re taking steps, because it helps with something elseā€¦ It helps people admit that theyā€™ve possibly done something. Who in their early career has got the courage to say, ā€œIā€™ve mucked upā€, right? ā€œI potentially have lost you money, or time.ā€ Most people are terrified; and youā€™re terrified legitimately, because youā€™ve got no experience in the industry, youā€™re brand new, youā€™re finally being paid to do something, and you think youā€™re not very good. Weā€™re long in our career, and we probably think weā€™re not very good. So early career, youā€™re crushed. And that ability to turn around and go, ā€œThat might have been me. I think I did that. I pressed this button, and then it broke.ā€ Thatā€™s tough.

[26:01] Yeah. Well, I think that speaks to like the blameless culture thatā€™s important. Itā€™s important to reach the point where your people arenā€™t punished for these mistakesā€¦ Because the last thing you want is people - like you say, they bury it, they try and hide it, or just donā€™t tell anybody, which could make the problem much worse. So yeah, I think that culture plays a big part, doesnā€™t it?

Yeah. You should always blame systems and not people. If something went wrong, itā€™s not the personā€™s fault, itā€™s why did the system allow the person to do that?

Right. So when Johnny says something thatā€™s mean to me, itā€™s not Johnny that I have to complain about. Itā€™s the system that lets Johnny get on the podcast and be horrible to me. [laughter]

Exactly. Itā€™s part of the system to be mean to you, Mat, donā€™t you know?

Yeah, it feels like it sometimesā€¦

Have we got any more horror stories? Oh, by the way, this campfireā€™s warm isnā€™t it? We can probably put an effect of a campfire over the top; letā€™s pretend weā€™re all gathered around a campfire. Oh, what do you think of the campfire, Johnny?

Sure. Yeah, yeah. [unintelligible 00:28:44.21]

Iā€™m convinced by that performance, Johnnyā€¦ Have you done actual theater? What about you, Kris? What do you think of the fire? Itā€™s cozy, isnā€™t it?

Okayā€¦

Crackly, warm fireā€¦ We donā€™t have any marshmallows, so itā€™s not as goodā€¦

Donā€™t we? Itā€™s imaginary land. Itā€™s podcast land. You can have anything you want. Check this outā€¦ Whatā€™s this? Look, look at your faceā€¦ Look, itā€™s marshmallows. Natalie, what do you think of the fire?

It shouldnā€™t be burning serversā€¦

No, it shouldnā€™t be burning servers thoughā€¦ No, this is a fire that doesnā€™t actually release any carbon. Itā€™s a good fire.

Itā€™s basically my GPU overheating. [laughter]

Itā€™s the sound of my computer

The money fireā€¦ My electric billā€¦

Itā€™s some old Intel Macs, you knowā€¦ We just turned them on, opened Slack, and now theyā€™ve made us a nice fire. Itā€™s good.

[laughs] Just have Slack and a regular expression running; itā€™ll generate enough heat to cook your marshmallow.

And those fans can definitely fly us somewhere. We could all go visit Mat in the UK.

1:Yeah. I mean, make sure you do go through proper passport control. Donā€™t just fly in at any point, because thatā€™s illegal. But yeah, otherwise do, please visit; weā€™d love to have you.

[29:57] Yeah, I remember talking about hot CPUsā€¦ The CPU Hot program that I used to have on an Amiga, and basically run it, and it made the CPU hot. And that was a program that you could have ā€“ it was on like a front of a magazine, for some reasonā€¦ ā€œWhatā€™s that doingā€¦?ā€

Someone wrote another infinite loopā€¦

Yeah, there you go. Theyā€™ve turned their horror story into a big success story, because they got on a magazine cover with a floppy disk. Soā€¦

Interesting. Now with the energy costs going up here in Europe, all the heaters are becoming more expensive, because people assume they will not have gas to heat their house. Many apartment buildings have these systems with gas. So you buy like electrical heaters to warm the place in case you might need that. And theyā€™ve become really expensive. So really, what youā€™re saying is that all you need is an old computerā€¦ Which is probably cheaper at this point.

I bet we see a spike in the downloads of Slack in that areaā€¦ [laughter]

Or that CPU Hotā€¦

How many Electron apps can I install on one machine?

Okay, has anybody got any other horror-horror stories?

Iā€™ve got moreā€¦ Iā€™ve got one which is something thatā€™s kind of triggering. I donā€™t know if anyone else has got sort of triggers from being horrified. One of my old bosses used to come to me, and if he started the sentence with, ā€œWhat do you know aboutā€¦ā€, then I knew immediately it was downhill. Itā€™s like, ā€œWhat do you know about Perl?ā€ It was like, ā€œUh-ohā€¦ Where is this going?ā€ Or ā€œWhat do you know about directory services in exchange?ā€ Itā€™s like, ā€œUmā€¦ That they exist?ā€ ā€œGreat! Youā€™ll do!ā€ and off youā€™ll be shipped to a client site. [laughter]

And I ended up at one of these clients sites, and there was a customerā€¦ And it was a big, big company. And they were basically doing a split. Merger and acquisition is the normal thing you hear about; theyā€™re doing the other thing, theyā€™re splitting in two. And they basically said to the contractor, the company I worked for, to split their Active Directory, to clone it, and then rename it to the other company name, so they had a perfect copy renamed. And I just turn up on site, I know a bit of Visual Basic, a little bit of C# at that pointā€¦ How do I approach this problem? I did not know at all.

So I get in touch with Microsoft Professional Services and go ā€œHow would you do this?ā€ And theyā€™re like, ā€œYou donā€™t. Thatā€™s impossible. Donā€™t do that. Thatā€™s reckless and foolish, and itā€™s not supported.ā€ And I was like, ā€œOkay, but Iā€™m being paid for this, and I know no better, because Iā€™m barely in my mid-20s, and Iā€™m gonna take a stab at this.ā€

And I wrote a script for the registry on one of the cloned Exchange Server machines, and I basically renamed everything. And I fired it up afterwards, and it worked.

That was my dayā€™s work done. I left. I have no idea whether that workedā€¦ [laughter] It appeared to work.

ā€œWhy is your two-week notice exactly after this project ended?ā€ ā€œOh, I donā€™t know. I donā€™t wanna know.ā€ [laughter]

Anyway, contractors, thatā€™s what you get.

Wowā€¦ That could have just worked though. But you know, I never trust code that works first time. Thatā€™s why I like a failing test beforeā€¦

The only thing that I was really, really scared about ā€“ the name of the company in the Active Directory was six characters long. And I was reasonably sure that that was a magic value. So I asked them for a new name to be the Active Directory name that was also six characters long. Thatā€™s the only thing that I think was intelligent about what I did that day. The rest of was luckā€¦

And lots of regexesā€¦ [laughs] Again.

Yeahā€¦ Good call.

Spookyā€¦ Spoopy.

Spoopyā€¦

But yeah, thatā€™s the scary questionā€¦ No oneā€™s asked me that for years. ā€œHey, what do you know aboutā€¦?ā€

Now youā€™d be like ā€œNothingā€¦ Nothing. Nothing at all. I donā€™t want to know anything about it. I know nothing.ā€

Maybe thatā€™s the thing thoughā€¦ When you see people late in their career, and youā€™re just like ā€œWhat do you know about this?ā€ and you think theyā€™re doing the ā€œIā€™ve forgotten more than youā€™ll ever learnā€, but no, theyā€™re actually going ā€œI donā€™t want to do thisā€¦ā€ [laughter]

Active Directory. Never heard of it, mate. Never heard of him. Thinks itā€™s someone manā€™s nameā€¦ Active Directory. Thatā€™s a weird name for a man. I think thatā€™s a man, just really selling that you know it, just to get the jobā€¦ Oh, just a tip there for people that want to get into that. Like I said, Iā€™m jet-lagged. And this is a spooky Halloween party specialā€¦ How are those marshmallows looking, Kris?

[34:15] Theyā€™re toasty, and brown, and delicious.

Oh, perfect. Well done. Are you gonna share, orā€¦?

Noā€¦

Nahā€¦

These are my marshmallows now.

Yeah. Although theyā€™re imaginary, butā€¦ But you know, I still want one.

We have an infinite supply. Everybody can make their own marshmallows.

Oh, yeah. Okay.

All you need is an infinite loop.

Speaking of loops, I have a scary storyā€¦ But itā€™s actually the story that I think helped me gain a new appreciation for sort of how to integrate systems, how distributed systems have sort of pitfalls, thereā€™s a trade-off for everything, and all the things that we value as best practices for dealing with integrating systems.

So I was working for an organization, a very awesome organization, a nonprofit, which basically helps students, especially in underserved communities, sort of prepare to take sort of standardized testing, and that kind of thing. So for months, leading up to a major sort of testing day - students are going to come, sit down into their classrooms, theyā€™re going to be logging in and taking an assessment to help them with the real things. And this is like a coordination across multiple schools and everything, so that the whole county is doing this thing. So weā€™re talking maybe like 3,000 to 4,000 students that are gonna all sit down and do this thing.

Wow.

And basically, Iā€™m part of the team that basically has been working on this sort of integration for months now. We had different systems talking to each other, and everything else. And development, and even in staging, everything works perfectly. [laughs] Systems can talk to each other, weā€™re sending lots of traffic, keeping an eye on things, and we are observing to the best of our ability with the tools that we have on production day, basically, which is when students actually sit down to do the thingā€¦

What we didnā€™t test against is basically having roughly 4,000 students trying to log into the system at the same time. [laughs] And because you have these systems that are talking to each other for authentication, and pulling things, whatever it is, basically, we just had a thundering herd kind of situation happening, and we didnā€™t account for that, because all of our tests, even our integration tests and everything else - they didnā€™t factor in that kind of scale. And I take responsibility for that, because I was one of the team leads, and basically, one of my questions was supposed to be, ā€œWhat is the expected number of users and clients on this system?ā€ And we touched on these things, but there were bottlenecks in the system that we should have better accounted for. And thankfully, we had enough of an understanding of what was happening, we had enough observability to be like, ā€œOh, crap, we know where the bottleneck is. We need to go do thisā€, whatever.

So within a matter of about an hour and a half or so, while students are waiting there - because they canā€™t really dismiss everybody and send everybody home; weā€™re talking like countywide 4,000+ students, all this coordination across monthsā€¦ To me, this remains the best and worst moment of my career, because Iā€™m like, ā€œHere I am, Iā€™m supposed to be serving these kids.ā€ Often, we are so far removed from the consequences of our code, right? Good or bad. Weā€™re so far removed from it. But here I am, I knew exactly what the impact that my mistake was having on these kids, right? And they already have been given a short straw in life, and here I am just making that worse, right?

So after that incident, I was never again, like, ā€œWhat do I need to do? What do I need to learn? Who do I need to talk to?ā€ Like, it was ā€“ I had to level up. And any point in my career - I canā€™t remember a single incident that has driven me to level up as much as this one, because the impact was so real. It was so in my face. Just undeniable. I thought that was scaryā€¦

[38:11] That is the $1,001 bill.

Yeah.

Listen, I would have gladly handed over $1,000 out of my pocket like to be like, ā€œLook, whatever this is, make it go away right now.ā€

Like, to the kids. Just give it to the kids.

To the kids. [laughs]

ā€œUncle Johnny has messed up again. Come and collect your $20 bills, everyoneā€¦ No college for you, but hereā€™s someā€¦ā€ Yeah, thatā€™s horrific. Yeah, but when the stakes are that high, Johnny, itā€™s likeā€¦ That is scary.

Yeah. And you feel awful. Like awful, awful, awful for being the responsible for that.

What do you think about engineers sort of having that sort of sense of consequence? Because it comes in multiple directions; you get salespeople going, ā€œIf you donā€™t do this, weā€™re gonna lose a million-dollar deal.ā€ Itā€™s not that the engineerā€™s gonna receive no million dollars round. Theyā€™re just gonna get their normal salary; theyā€™re getting the normal pay. Thatā€™s a little bit abstract. Itā€™s a ton of stress.

And likewise, when youā€™re working in incidents, someone will turn and go, ā€œItā€™s affected this air traffic control signalā€ or ā€œItā€™s affected this sort of kids, or hospital, or the traffic lights are downā€ or whatever. Kind of unhealthy, isnā€™t it? Itā€™s like, weā€™ve got to keep it abstract enough that ā€“ because I get that it changes us. Probably all of those incidents changed all of us. They were all horrifying moments. But Iā€™m also like, ā€œThatā€™s the path to burnout, sort of accepting the consequences for all of these things.ā€

Yeah. Some value in being somewhat abstracted from the consequences.

Yeah. If you consider it too much, it just weighs so heavily. Itā€™s too serious. Itā€™s too much of aā€¦ And the things you need to do to actually get out of those situations - they become even more horrifying and scary. ā€œWhat if I prolong this? What if I make it worse?ā€ And sometimes youā€™ve just got to be a bit fearless, and you canā€™t if youā€™ve got that sort of burden on you that we put on ourselves.

I feel like this is where systems can be helpful, though. I think we as an industry are pretty bad at understanding that there can be bad consequences. Itā€™s like, something terrible happens and weā€™re like, ā€œOh no, this terrible thing happenedā€, but so much of the time, that terrible thing was completely predictable, and we just didnā€™t predict it, because we thought itā€™d go fine.

Like, I worked for a lifeguard ā€“ when I was young, I worked as a lifeguard, and I remember one of the things that we always did is we trained a lot, but also did a lot to make sure that the environment was always a safe one. So we did a lot of that. Thatā€™s why lifeguards yell at people so much, and theyā€™re like, ā€œDonā€™t run! Donā€™t do these things that might result in you getting injured.ā€ I kind of feel like in software engineering we just let people do whatever, and then people slip and bash their head, and theyā€™re bleeding all over the place, and weā€™re like, ā€œOh no, how did this happen?ā€ And itā€™s like, ā€œWell, not only did we not tell people not to run, but we also left giant puddles of water on the floor, because we didnā€™t put down the proper mats to make sure that, if they are running, they can do it safelyā€¦ā€ Like, thereā€™s all these other things that you have to set up as precautions. I feel like in software engineering we just kind of donā€™t doā€¦ And part of me wonders if we donā€™t do that because there arenā€™t enough consequences flowing down to the engineers.

I feel like the number of times Iā€™ve been at companies that ā€“ Iā€™ve worked at banks, and people have been like, ā€œWell, this isnā€™t like life or death.ā€ And Iā€™m like, ā€œThis is affecting peopleā€™s money and their livelihood. What do you mean?!ā€ And like, thatā€™s always the thing that gets rolled out, is if itā€™s like, ā€œOh, well, weā€™re not doing things that could kill people.ā€ Itā€™s like, ā€œWell, but weā€™re doing things that can substantially affect peopleā€™s livesā€, and I feel like we have to take that into account. Because when we donā€™t, we do lots of immoral things, like run psychological experiments on people without their knowledge, and other terrible things, because weā€™re like, ā€œAh, whatā€™s the harm?ā€

I havenā€™t done that for weeks. I donā€™t know why youā€™re bringing it up. [laughter] Itā€™s spooky, ainā€™t it? Itā€™s a spooky show.

I have a spooky story, I thinkā€¦ It feels spooky. So Iā€™d recently joined this company, and of course, because itā€™s the modern day, theyā€™re using Kubernetes.

Classic.

And of course, theyā€™re using all the shiny things with Kubernetes, so thatā€™s using Istio. No one actually knows how any of this works. Itā€™s just like, ā€œOh, this is what weā€™re supposed to be using.ā€ So we have this big olā€™ cluster, and itā€™s running, and our DevOps people are pulling their hair, because they hate all of thisā€¦ And I start like reading through the codebase and looking at things, and Iā€™m like, ā€œOkay, these auth policies look a little funky.ā€ And then I go talk to people, and theyā€™re like, ā€œWe donā€™t really have any auth policies. Everythingā€™s just kind of open right now. Like, everything in the backend can just talk to each other. Thereā€™s no auth policies.ā€ And Iā€™m like, ā€œAre you sure? Because I see these auth policies in the codebase.ā€ And theyā€™re like, ā€œYeah, but we donā€™t think theyā€™re being used for anything.ā€ Iā€™m like, ā€œOkayā€¦ā€ So I just kind of like let it go, and go about my businessā€¦ And then I have like a few more of these conversations, and Iā€™m like, ā€œThis feels weird. But you know, all these people know more than I do.ā€

And then months and months later, someone stumbles across this one auth policy that has no labels, and no access rules, which in Istio language means that it applies to literally everything, and allows all traffic in. So this one policy had just opened our entire API, including the public API, to the entire internet, for anybody to do anything without needing any authorization. You just needed like a JSON web token that you could easily get from anywhere. And I was just like ā€“ so it caused all these problems, everybodyā€™s freaking out, and then they just rip that policy out and theyā€™re like, ā€œOkay, well, without this policy, weā€™ll be fine.ā€ But then all of those auth policies that had been sitting there, that I was like, ā€œThese look funkyā€, all those took over, and all of those were broken. So then it broke all the APIs. So then they had to put the other policy back, and then go through, and like, go find every single auth policy within Istio, and then fix all of those auth policies, and then they could finally remove that one policy that was opening everything to the world.

I think the total amount of time that the door was just open was about nine months, and to the knowledge that people had, nothing bad happenedā€¦ But yeah, it was quite horrifying.

That is still a security incident requiring disclosure, Iā€™m afraid.

I know. Iā€™m just like, ā€œIs this disclosure?ā€ [laughter]

Yeah, it was like, ā€œOhā€¦ Oh, no.ā€ It taught me a lesson that like when I see funky things, I should probably bring them up a little bit. Like, ā€œNo, no, that policy is thereā€, and that policy definitely doesnā€™t work. And some of the broken things were like some YAML white spacing thing, where itā€™s like something was tabbed in a little too far, and then people were just copying and pasting these policies, and then not testing them, which was like another thing that we had to go back and be like, ā€œPlease test the things that you put into the codebaseā€¦ Pretty please, thank youā€¦ā€

Yeah, I think that also is a bit of a lesson, is if there are bits of code and youā€™re like, ā€œNo, yeah, but that doesnā€™t do anything now. Like, that used to be doing something and now it doesnā€™t.ā€ Itā€™s like, either take it outā€¦ If itā€™s really not doing something, get rid of it.ā€

Exactly. It probably is doing somethingā€¦ And if it isnā€™t, maybe it should be, like in your case, Kris.

Spoopyā€¦

The zombie apocalypse.

Yeah.

ā€¦will be caused by Istioā€¦ [laughter]

Zombie code just haunting us.

Iā€™m never sure whatā€™s worse than those thingsā€¦ Like, the insecure environment, where it just like nothing applies, or the extremely secure environment. Iā€™ve seen somewhere, theyā€™ve been really locked down, and like everything; youā€™ve got an IP firewall rule for everything, and youā€™re like, ā€œI have total confidence.ā€ And then mysteriously, API calls between machines just didnā€™t work. And itā€™s like, ā€œWhy now?ā€

And what we realized - the debugging for this went wild; it went really low. And we were down at Wireshark and weā€™re watching whatā€™s going on, and weā€™re watching whatā€™s going on inside the kernel, but we were turning on contract connection tracking. And this is in TCP, itā€™s got little tables, state tables in there to keep track of the sort of TCP connections. But you can overflow these tables; we were turning them off, and every time we turned them on and off, we were toggling which IP firewall rules were actually matching or not.

[46:16] So we were taking existing connections and then just randomly dropping them every time we flipped these thingsā€¦ But we could never observe it, and we were just there the whole time, just going ā€œWeā€™ve lost it again. There goes the connection.ā€

And it took us weeks of just poking around, going ā€œWhatā€™s going on? I canā€™t see it. Ghost in the machine.ā€ Yeah, too secure is a problem. Honestly.

Well, in that spirit, Dee, whatā€™s your pin number? Just give us three of the numbers. Letā€™s play mastermind.

Itā€™s just like ā€“ what was it, Spaceballs? 1234ā€¦ [laughter]

Yeah. No oneā€™s gonna suspect that, I think. No oneā€™s going to try that, are they?

0000ā€¦

Does it allow you? I donā€™t think systems allow you to do that.

Why?

I donā€™t know. Iā€™m gonna try and change my pin nowā€¦ [laughter]

Oh yeah, I donā€™t knowā€¦ Maybe because itā€™s too easy. But I donā€™t know. Any other horror stories before we throw a ā€“ what do you put on fire, water? You donā€™t do that, do you? You just let it die out on its own. No, weā€™ve got to be responsible. How are we going to sort this fire out?

By throwing water in the electrical equipment?

Foam. Weā€™ll use foam.

Turn it off.

Weā€™ll use found close Slack. Yeah, close Slackā€¦

Yeah, and the fire will die down. So before we put the fire out, has anyone got any other final horror stories?

Well, you know a few of mineā€¦ Any one you want to hear?

What, do you mean in real life? [laughter]

I wrote one down, which actually I wrote in advance, about doing a SQL statement and accidentally double ā€“ putting in the semicolon after the from table. So an update. So the WHERE statement didnā€™t applyā€¦ And that was to a production systemā€¦

So what was the effect of that then? So normally, you would be updating something and specifying the WHERE, which will limit what gets changed, right?

Yeah, exactly. I tweeted this when you actually asked about it, and I think no one really appreciated what it does and how it happened. But I executed the query; I was just tidying up some debt that was leftover, and it should have been really trivialā€¦ And I practiced it, and then I copy and pasted it into the console. But after I copied and pasted the first one, for whatever reason, I fingered a semicolon, and then put in the WHERE line. But that makes it two commands. So it successfully did the update column set value equals on table, and it didnā€™t apply the WHERE. So it updated ā€“ I think was like 90 million rows, or somethingā€¦

Oh, Godā€¦

And the machine was very fast; faster than I was at finding Ctrl+Z.

Oh, noā€¦!

And thatā€™s why you work inside of transaction blocks, kids. [laughs]

Thatā€™s why thatā€™s advisable, but it was not what I was doing that day. The real mess though was actually sort of going ā€œHow can we restore this when our database backup was like 12 hours ago, and thereā€™s 12 hours of changes in other tables since then?ā€

Ouchā€¦

So youā€™re not going to sort of do anything there. So itā€™s a pull the old sort of thing, extract that table, and then go and update all the necessary rows to the right things. It takes timeā€¦

Yeah. I liked that you couldnā€™t keep up because the machine was so fast. Is that why you now insist only on running on Raspberry Piā€™s?

Intel Max. Run on Intel Max. Itā€™s the solution for everything. [laughter]

Okay, well that sound - we all heard that sound, didnā€™t we? How would you describe that sound that we just heard?

Spookyā€¦

Natalie? Spooky, yesā€¦ Natalie, how would you describe that sound weā€™ve just heard?

Did I miss the sound?

Yeah, we heard that soundā€¦ Kris, how would you describe it?

Was it the marshmallows?

I think itā€™s the sound of us closing Slack so our fireā€™s going away.

Was it marshmallows? Yeah, itā€™s kind of a spooky soundā€¦ Wasnā€™t it, Johnny? How would you describe that, Johnny?

As silent as your hairline? [laughter]

[50:13] I mean, itā€™s getting very poetic, and slightly unusual banter thereā€¦

You asked for it. I mean, you did ask for it.

Amazing. Yeah. Is it silent because itā€™s so far away? Like, itā€™s in the distanceā€¦ Thatā€™s what we mean. Like, itā€™s screaming, but you canā€™t hear itā€¦

Itā€™s too far back?

Yeah. Itā€™s just screaming ā€œWhy, dad?! Why?!ā€ You know what I mean? If you know youā€™re going to look like this, donā€™t have kids. If you know youā€™re gonna make kids that look like me, donā€™t have them. Thatā€™s my adviceā€¦ But dad didnā€™t listen.

Weā€™re all talking about recessions nowadays, but youā€™ve been in one for quite some time, right Matt? [laughter]

Oh, there we goā€¦ That was a good oneā€¦ That was a good one, yeah. Good point. Okay, wellā€¦ That sound of Johnny talking tells us itā€™s time forā€¦ Unpopular Opinions!

Okayā€¦ Whoā€™s gonna kick us off with the first popular opinion? Dee! Youā€™ve been chosen.

You scared him. He was like, ā€œWhat?! What happened?! What happened?!ā€ [laughs]

Itā€™s just sounds. I promise no oneā€™s pushing their hand around the Ouija board to make it spell out about my hairline. I donā€™t want to hear anything from the ghosts about my hairline, Johnnyā€¦ Right? So donā€™t make it do that. Everyone put your hand on it. Let it be natural, and weā€™ll see. Yeah, itā€™s floated over to Dee. Okay, Deeā€¦

It was a fixā€¦ [laughs]

ā€¦whatā€™s your unpopular opinion? It was a fix, yeah.

Mine comes from the last Go Time. One of the guests said that Go is brilliant, you donā€™t have to worry about security; it does everything for you. You donā€™t have to worry about the memory management. Everythingā€™s super cool. And I think it actually has made Go more insecure, because people are so ā€“ they put so much trust and safety in the actual sort of language, that a lot of the basics are dropping by the wayside.

For me, the number one thing people should do is sanitize inputs. And itā€™s not because I wrote bluemonday but they should do it on everything. And I just donā€™t think I see anyone doing it anywhere. Thereā€™s just great, big holes in everything, but people are still there going ā€œBut weā€™ve got memory safety.ā€ Memory safety saves you from yourself, not from others.

Hmā€¦ What do we think? Is that popular, or unpopularā€¦?

Well, it should be popularā€¦ Sanitize all your inputs?

It should be. But if you look at all of the open source stuff thatā€™s out there, very few people actually sanitize, check, validate their inputs. Theyā€™re just like ā€œIā€™ve mapped this input directly to a struct, and Iā€™m going to use it.ā€ They take their form fill, and theyā€™re on their way.

Yeah. One simple version of that is just a limit reader when youā€™re reading a body, or like of a request; you can error if itā€™s too big, and things like that. Thereā€™s bits like that. But you end up doing quite a lot of that heavylifting yourself every time.

There are libraries out there where you can add tags and say ā€œThis should be a text field. It should be no longer than this length. There should be a number, and it should be no longer than this.ā€ But far too few projects do it.

Do they use reflection? I think Iā€™ve avoided them if they ā€“ but itā€™s not a great reason to avoid it. I just tend to notā€¦

Why are you afraid of your reflection? Are you a vampire?

Well, itā€™s because I am DraCool ā€“ Yeah because you know, thatā€™s myā€¦ Russian accent there.

Because he doesnā€™t want to see that hairline, obviouslyā€¦

Ohhā€¦!

Thatā€™s why I donā€™t have mirrorsā€¦

Kris is in on itā€¦ Alright, everybody take a turn.

Yeah, Kris is in on it.

Take a stab at Mattā€¦ [laughs]

[54:10] Iā€™m like a pinata. Iā€™m like a really rubbish pinata. Imagine buying a pinata for kids and itā€™s me. [laughter] Youā€™d take it back, wouldnā€™t you? Youā€™d be like ā€œNo, weā€™ll probably go for the unicon instead, on second thoughtā€¦ I should have guessed that, reallyā€¦ā€ Okay, yeah. Fine. Thanks, Kris. Itā€™s a Halloween special, youā€™re allowed to do thatā€¦

Spooky pinata.

Yeah, exactly. Okay, and other unpopular opinions?

Natalie PistunoWitchā€¦! [wolf howl] They donā€™t know Iā€™ve just done that, so it just sounded like a sound effect in the background. Come on.

Some of the training that you should be taking occasionally throughout your career, even annually, should not be about things that are in the future, like new things, like new technologies coming and so on, but also a little bit about the back. A little bit of Assembly every now and then. It might be useful accidentally at some point in your life, because you did it, and even if not, it might ā€“ like, youā€™ll see patterns, because itā€™s all the same things, with just more and more abstractions. But itā€™s still the same things. Just seeing how itā€™s done, and how things were solved, and how problems workedā€¦ It might help you figure out the future when you rely on the past.

And itā€™s unpopular ā€“ I know, you all will not agree with me, itā€™s unpopular. I also did not do that, and donā€™t allocate time or budget for that.

Hmmā€¦ I do know what you mean. I actually have this book called, ā€œBut how do we know?ā€ I got it off the Amazon website. And basically, it talks about computing from the very bare beginnings, like literally logic gates, and then how you make a bit out of two NAND gates, just showing how the logic worksā€¦ And then building up everything in a computer like that. And it is amazing. But yeah, that was like, not something I need.

And actually, something else that occurred to me when you were saying that one is having like training or paying attention to things that you already think youā€™re good at. So not just new things that are new to you. Things that you already think, ā€œYeah, Iā€™ve got that nailed.ā€ You might be surprised; like, plenty of other things to learn. Yeah, I like that one. Weā€™ll test that one, and see if thatā€™s unpopular or not.

Even the way you did that is interesting, because you were already a couple of years a software developer, and then you look into gates, and so on, and that kind of helped you put this in place. When I started my degree, the first course I did was those logic gates, and everything was scary; calculus was scary. And then those gates - like, what? I donā€™t know, just let me pass that test and leave me alone. And if it could be the other way around, start with the programming courses, and then you go about semiconductors, and then you go about circuits, and then you speak about gates - it might have been more interesting, and actually it would make more sense. Maybe for me, at least.

I think thatā€™s the path that people take accidentally. Like, those who do bootcamps, and then they gradually, eventually, over like 5-10 years become like systems engineers and are working on like the Kernel, or TLS, or something - they donā€™t know theyā€™re doing that, but thatā€™s kind of what weā€™re doing. Theyā€™re looking back at the fundamentals. I agree itā€™s unpopular. I donā€™t know anyone who does that, but I think itā€™s genius, and we should.

We should open a university teaches that way.

I feel like thatā€™s how my career has been. I just have gotten like down the stack further and furtherā€¦

Thatā€™s interesting, yeah.

Itā€™s been fun. Iā€™ve written like a few operating system kernels, like toy kernels. It was infuriating. Modern processors are terribleā€¦ But mostly because theyā€™re so old. Like, ā€œOh, maybe I have this code from the ā€˜70s I might need to run on my Intel 12-gen chipā€, or something. You never know. Itā€™s like, ā€œNo, Intel, I donā€™t need to run code from 1980 on my new processor. Thank you.ā€

I like that too though, that people can start higher up in the stack, and can still be doing things without understanding everything. Because Iā€™ve seen it, people held themselves back because they think ā€œWell, I just donā€™t know how all of this stuff, and I need toā€“ā€ They donā€™t know that they donā€™t need to know it necessarily, which is why I say you sometimes donā€™t need to know a lot of this stuff. Just sort of get on with it, and try thingsā€¦ But it doesnā€™t work for everybody. I think thereā€™s so many different styles and things that people appreciate, and things that workā€¦ So I wouldnā€™t like to force it on everybody; as we like to say, it depends.

And Iā€™m afraid - put down your goblets of red wine. Yes, it was definitely wine, yeahā€¦ And put also away those sandwiches of miscellaneousā€¦ No, there werenā€™t any sandwiches. No, letā€™s just not do the sandwiches, but weā€™ll do the wine but, and then keep the wine, because they think itā€™s blood.

Okay, and thatā€™s all the time we have on todayā€™s Ghost Time. Thanks for joining us, everybody. See you next time, and stay spoopy!

Changelog

Our transcripts are open source on GitHub. Improvements are welcome. šŸ’š

Player art
  0:00 / 0:00