Service Weaver is a programming framework for writing, deploying, and managing distributed applications in Go. With Service Weaver, you write your application like it is a traditional, single-process Go executable that runs on your local machine. Then, you deploy it to the Cloud, and the framework breaks it down into a set of connected microservices and integrates it with the cloud provider (e.g., monitoring, tracing, logging).

MRSK deploys web apps anywhere from bare metal to cloud VMs using Docker with zero downtime. It uses the dynamic reverse-proxy Traefik to hold requests while the new application container is started and the old one is stopped. It works seamlessly across multiple hosts, using SSHKit to execute commands. It was built for Rails applications, but works with any type of web app that can be containerized with Docker.

Watch the screencast or check out the code & docs.

The rough math goes like this: We spent $3.2m on cloud in 2022. Just under a million of that was on storing 8 petabytes of files in S3, fully replicated across several regions. So that leaves ~$2.3m on everything else: app servers, cache servers, database servers, search servers, the works. That’s the part of the budget we intend to bring to zero in 2023. Then we’ll worry about exiting the 8PB from S3 in 2024.

David spoke at length on this decision when he was on Ship It! late last year.

StackQL allows you to create, modify and query the state of services and resources across all three major public cloud providers (Google, AWS and Azure) using a common, widely known DSL…SQL.

I have to admit, it does look pretty darn simple:

SELECT * FROM google.compute.instances 
WHERE zone = 'australia-southeast1-b' 
AND project = 'my-project';

TLDR: AWS is super-expensive at scale.

Of course it’s expensive to rent your computers from someone else. But it’s never presented in those terms. The cloud is sold as computing on demand, which sounds futuristic and cool, and very much not like something as mundane as “renting computers”, even though that’s mostly what it is.

But also:

It’s also about what kind of internet we want to operate in the future. It strikes me as downright tragic that this decentralized wonder of the world is now largely operating on computers owned by a handful of mega corporations. If one of the primary AWS regions go down, seemingly half the internet is offline along with it.

Methinks this will become even more of a trend in the coming months and years.

It is just not about redundancy. In this post I cover how having more AZs mean you need fewer nodes to achieve higher availability.

This lets you do so many cool things, even joining across disparate services:

select
  aws.name aws_user_name,
  slack.id as slack_user_id,
  slack.display_name as slack_name
from
  aws_iam_user as aws,
  slack_user as slack
where
  aws.name = slack.email;

+--------------------------+---------------+------------+
|     aws_user_name        | slack_user_id | slack_name |
+--------------------------+---------------+------------+
| dwight@dundermifflin.com | U2EMB8HLP     | dwight     |
| jim@dundermifflin.com    | U02HE4Z7E     | jim        |
+--------------------------+---------------+------------+

Useful for compliance, security, ops, and cost management.

As best I can tell, this provides simplified search across your infra, generates reports so you can easily audit resource usage, and lets you create/trigger jobs such as cleaning up unused resources and enforcing tag structures.

The search looks pretty powerful and you can pipe search results directly to jobs for quick processing:

search is(resource) and tags.owner==null | tag update owner "John Doe"

I won’t call SQLite’s current moment a comeback, because the most used database engine in the world doesn’t have anything to come back from. I’m going with “renaissance”, because despite its already mass adoption, there has been something of a rebirth of interest from one software sector that had previously relegated it to dev & test environments: web apps

YouTubeDrive is a Wolfram Language (aka Mathematica) package that encodes/decodes arbitrary data to/from simple RGB videos which are automatically uploaded to/downloaded from YouTube. Since YouTube imposes no limits on the total number or length of videos users can upload, this provides an effectively infinite but extremely slow form of file storage.

Filed under: ways-no-youtube-engineer-ever-imagined-people-would-use-their-software

Spacedrive helps you organize your files across many devices in one place. Here’s the motivation:

Many of us have multiple cloud accounts, drives that aren’t backed up and data at risk of loss. We depend on cloud services like Google Photos and iCloud, but are locked in with limited capacity and almost zero interoperability between services and operating systems. Photo albums shouldn’t be stuck in a device ecosystem, or harvested for advertising data. They should be OS agnostic, permanent and personally owned. Data we create is our legacy, that will long outlive us—open source technology is the only way to ensure we retain absolute control over the data that defines our lives, at unlimited scale.

A short, cogent argument why hosting your own cloud services is worth the time/effort:

Consider this, you carefully curate your playlists on Spotify but every now and then you see a certain song missing from your playlist. Same goes for videos saved in your YouTube playlists or other music/video streaming services. Then there is also the case of OTT streaming platforms where the show you were going to watch over weekend has now disappeared.

The author points to r/selfhosted and this awesome self hosted list as good resources to get started.

Rich Burroughs:

When you hear the phrase “cloud native,” is Kubernetes the first thing that comes to your mind? It is for me, and I expect I’m not alone. Kubernetes is now the second-largest open source project after Linux, and it’s the big fish in the cloud native pond. But there are many other projects in the CNCF landscape and the broader cloud native community.

So, I thought I’d list some cloud native tools that can be very useful for teams that aren’t using Kubernetes or aren’t using it for every workload. Here are 7 of them that I like a lot.

If Rich’s name rings a bell, that’s because he was just on Ship It! last week. 😉

One of the things we discussed with Paul Copplestone from Supabase was what, exactly, might a cloud native Postgres look like? Well, perhaps it will look like OrioleDB:

A new storage engine for PostgreSQL, bringing a modern approach to database capacity, capabilities and performance to the world’s most-loved database platform.

OrioleDB consists of an extension, building on the innovative table access method framework and other standard Postgres extension interfaces. By extending and enhancing the current table access methods, OrioleDB opens the door to a future of more powerful storage models that are optimized for cloud and modern hardware architectures.

I came across an interesting use case for edge compute the other day: cookie management at the edge. It probably won’t be super relevant to a ton of people, but it’s an interesting use case I wanted to share, nonetheless.

Garage is a distributed storage solution, that automatically replicates your data on several servers. Garage takes into account the geographical location of servers, and ensures that copies of your data are located at different locations when possible for maximal redundancy, a unique feature in the landscape of distributed storage systems.

It has an S3-compatible API and can be used as a storage backend for things like NextCloud, Matrix, and Mastodon. It’s being built by a non-profit in France that is “working to promote self-hosting and small-scale hosting.” Why do they do this?

self-hosting means running our own hardware at home, and providing 24/7 Internet services from there. We have many reasons for doing this. One is because this is the only way we can truly control who has access to our data. Another one is that it helps us be aware of the physical substrate of which the Internet is made: making the Internet run has an environmental cost which we want to evaluate and keep under control. The physical hardware also gives us a sense of community, calling to mind all of the people that could currently be connected and making use of our services, and reminding us of the purpose for which we are doing this.

They brand CasaOS as an operating system, but it’s really a layer on top of Linux that provides a nice UI and easy app (docker container) installation. It’s early days for the project, but we definitely need some open source options in this space, so here’s hoping they continue to progress!

Troy Hunt (of Have I been Pwned fame) has been a vocal proponent of cloud-first services for awhile. Last December, that strategy came back to bite him:

It all started with my monthly Azure bill for December which was way over what it would normally be. It only took a moment to find the problem…

He goes on to tell the tale in excruciating detail. Be careful out there, cloud natives.

If the datacenter is the computer, then the cloud is its operating system — so let’s start treating it like one and stop micro-managing thousands of individual ones.

Shawn (swyx) Wang lays out Cloudflare’s strategy to disrupt the cloud from the outside in:

While the tech industry is used to come-from-below disruption, and the software industry is increasingly grasping class-for-the-masses atomic concepts, I believe Cloudflare is writing a new playbook that is the little-guy counterpart of the embrace, extend, extinguish model used by Microsoft.

This is an article about exploring Inlets, a cloud native tunnel, with various use cases (including the Apple Silicon RC version of inlets-pro) in my home lab along with its surrounding open source ecosystem. Also some tips about using inlets-operator and MetalLB at the same time, and what could be done in a Kubernetes cluster.

Spending the time and effort to consider what happens after your cloud credits run out, including how much storage you’ll need to scale and how you plan to manage your data, pays off in the long run. Every developer wants their project to succeed.

So, how do you plan for survival and success when your free credits run out?

In 1984 John Gage of Sun Microsystems was credited as saying “The Network is the computer.” Almost four decades ago, John had a vision of distributed systems working together to be greater than the sum of their parts.

For this article, I surveyed the land of hosted IDEs and it turns out that we’ve progressed beyond running VS Code on an iPad whilst sipping a cocktail.

You can still do that, but there’s way more to it today and I’ll take you through some of use-cases and add my own thoughts. There’s also a practical guide at the end to get started with the open source VS Code browser by Coder.

Porter brings the Heroku experience to your own AWS/GCP account, while upgrading your infrastructure to Kubernetes. Get started on Porter without the overhead of DevOps and customize your infrastructure later when you need to.

For more on Porter, tune in to Go Time live on June 1st! Mat Ryer will be asking Carolyn Van Slyck all about it. If live isn’t an option… subscribe to Go Time, why don’t ya?

Namespace conflict! I mistook this Porter for that Porter which Carolyn Van Slyck works on. That Porter will be the subject of the June 1st Go Time, not this Porter. If you want us to do a show on this Porter, let us know. 😎

Marcel Klehr:

Remember Xmarks? It was great. Floccus does the same thing and even allows you to sync with whatever server you want: any Google Drive, any Nextcloud, any WebDAV server. With more backends in the works.

Floccus has extensions for Chrome (and its derivatives), Firefox, and Edge browsers.