Ryan Worl, Co-founder and CTO at WarpStream, joins us to talk about the world of Kafka and data streaming and how WarpStream redesigned the idea of Kafka to run in modern cloud environments directly on top of object storage. Last year they posted a blog titled, “Kafka is dead, long live Kafka” that hit the top of Hacker News to put WarpStream on the map. We get the backstory on Kafka and why it’s so widely used, who created it and for what purpose, and the behind the scenes on all things WarpStream.

One of the things we discussed with Paul Copplestone from Supabase was what, exactly, might a cloud native Postgres look like? Well, perhaps it will look like OrioleDB:

A new storage engine for PostgreSQL, bringing a modern approach to database capacity, capabilities and performance to the world’s most-loved database platform.

OrioleDB consists of an extension, building on the innovative table access method framework and other standard Postgres extension interfaces. By extending and enhancing the current table access methods, OrioleDB opens the door to a future of more powerful storage models that are optimized for cloud and modern hardware architectures.

Garage is a distributed storage solution, that automatically replicates your data on several servers. Garage takes into account the geographical location of servers, and ensures that copies of your data are located at different locations when possible for maximal redundancy, a unique feature in the landscape of distributed storage systems.

It has an S3-compatible API and can be used as a storage backend for things like NextCloud, Matrix, and Mastodon. It’s being built by a non-profit in France that is “working to promote self-hosting and small-scale hosting.” Why do they do this?

self-hosting means running our own hardware at home, and providing 24/7 Internet services from there. We have many reasons for doing this. One is because this is the only way we can truly control who has access to our data. Another one is that it helps us be aware of the physical substrate of which the Internet is made: making the Internet run has an environmental cost which we want to evaluate and keep under control. The physical hardware also gives us a sense of community, calling to mind all of the people that could currently be connected and making use of our services, and reminding us of the purpose for which we are doing this.

The general idea is to leverage cache-friendly ways of organizing data in structures of arrays (SoA) otherwise known “columnar” storage in database design. This, in turn allows us to iterate and filter over columns very efficiently. On top of that, this package also adds bitmap indexing to the columnar storage, allowing to build filter queries using binary and, and not, or and xor (see kelindar/bitmap with SIMD support).

Rclone acts like its source of influence, rsync, but instead syncs files and directories to many cloud storage providers. It runs on all the Linux/macOS/Windows flavors. You can even install with Homebrew on macOS.

brew install rclone

Based on what I read in the HN comments there are tons of users/uses for Rclone.

A HTTP based JSON storage. It lets you store, read & modify JSON data over HTTP APIs for FREE. Ideal for small projects, prototypes or hackathons where you don’t have to spin up your own data store.

Please don’t store anything mission critical here, but like the quote above says this could be a nice option when you just need a place to temporarily dump some data you’re working with. Simply grab a BOX_ID from the homepage and then POST away:

curl -X POST 'https://jsonbox.io/$BOX_ID' \
    -H 'content-type: application/json' \
    -d '{"name": "Schrute", "position": "Assistant (to the) Regional Manager"}'

Why cold storage? Because security:

For security purposes, Square stores a reserve of Bitcoins in an offline setting. By having these funds offline, we reduce attack surface and hence risk of theft.

Square can move the funds offline at any time, but moving them back online requires a multi-party signing ceremony. They can also embed programming logic into the cold storage modules, so that only Square-owned addresses can receive the funds. That’s defense-in-depth, right there.

Bitcoin’s latest bull run is over, but those who believe in decentralized money continue to toil away… building the future they want to exist.

Randall Degges examines the good and bad uses of Local Storage.

tldr, don’t use it to store sensitive data.

Almost every day I stumble across a new website storing sensitive user information in local storage and it bothers me to know that so many developers are opening themselves up to catastrophic security issues by doing so.

Let’s have a heart-to-heart and talk about local storage and why you should stop using it to store session data.

The New Stack with a nice rundown of what’s new/noteworthy in Node 14. The once-an-npm-package node-report is now mainlined, an experimental AsyncLocalStorage API has been added, and more.

A clever hack that is now being investigated by Google’s internal forums. How it works:

1. Google Docs take up 0 bytes of quota in your Google Drive
2. Split up binary files into Google Docs, with base64 encoded text
3. Encoded file is always larger than the original. Base64 encodes binary data to a ratio of about 4:3.
4. A single doc can store ~1 million characters. This is around 710KB of base64 encoded data.

Anurag Goel, Founder/CEO of Render, joins Adam to discuss what they’re doing to solve cloud problems for application developers. They just raised $80M they don’t even need and they’re poised to solve boring problems like object storage, and less boring things like building for the AI era.

Daniel & Chris explore the advantages of vector databases with Roie Schwaber-Cohen of Pinecone. Roie starts with a very lucid explanation of why you need a vector database in your machine learning pipeline, and then goes on to discuss Pinecone’s vector database, designed to facilitate efficient storage, retrieval, and management of vector data.

Techno Tim is back with Adam to discuss the state of homelab in 2024 and the trends happening within homelab tech. They discuss homelab environments providing a safe place for experimentation and learning, network improvement as a gateway to homelab, trends in network connection speeds, to Unifi or not, storage trends, ZFS configurations, TrueNAS, cameras, home automation, connectivity, routers, pfSense, and more.

Umm, should we make these conversations between Adam and Tim more frequent?

With Mountpoint for Amazon S3, your applications can access objects stored in Amazon S3 through file operations like open and read. Mountpoint for Amazon S3 automatically translates these operations into S3 object API calls, giving your applications access to the elastic storage and throughput of Amazon S3 through a file interface.

Early alpha. Written in Rust by the AWS team.

Sometimes all a project needs is the ability to read/write small amounts of JSON data and have it saved in some persistent storage. Imagine a simple data-model which receives infrequent updates and could be represented as JSON object. It doesn’t demand a full-blown database, but it would be neat to have a way to interact with this data and have it persist across sessions.

This is where gist-database comes in handy, by leveraging the power of the gist api you can easily create a key/value data-store for your project.

This is a perfect solution for low write / high read scenarios when serving static site content with Next.js and using Incremental Static Regeneration to keep your cached content fresh.

Welcome to 2023! A new year is the perfect time to start with a fresh perspective. Given a few bare metal hosts with fast, local storage, how would you run your workloads on them? Would you cluster them for redundancy? What operating system would you choose?

Steve Francis, CEO at Sidero Labs and Andrew Rynhard, CTO at Sidero Labs join us today to talk about running Talos Linux on bare metal.

I thouroughly enjoyed reading this. It’s interesting to think about the evolution of transport storage. To this day, the USB thumb drive has proved to be the long-term winner and something I personally use very often. Wanna install Debian or do a clean install of macOS? You’ll likely do it by creating a bootable USB thumb drive.

Before the invention of the thumb drive, computer users stored and transported their files using floppy disks. Developed by IBM in the 1960s, first 8-inch and later 5 ¼-inch and 3 ½-inch floppy disks replaced cassette tapes as the most practical portable storage media. Floppy disks were limited by their relatively small storage capacity—even double-sided, double-density disks could store only 1.44 MB of data. During the 1990s, as the size of files and software increased…

Would you be surprised to learn that the supposed inventor of the thumb drive is in jail right now in Singapore? But why?

This week we’re talking about serverless Postgres! We’re joined by Nikita Shamgunov, co-founder and CEO of Neon. With Neon, truly serverless PostgreSQL is finally here. Neon isn’t Postgres compatible…it actually is Postgres! Neon is also open source under the Apache License 2.0.

We talk about what a cloud native serverless Postgres looks like, why developers want Postgres and why of the top 5 databases only Postgres is growing (according to DB-Engines Ranking), we talk about how they separated storage and compute to offer autoscaling, branching, and bottomless storage, we also talk about their focus on DX — where they’re getting it right and where they need to improve. Neon is invite only as of the recording and release of this episode, but near the end of the show Nikita shares a few ways to get an invite and early access.

Heyang Zhou:

We decoupled SQLite’s storage layer onto FoundationDB to offer massive yet “bottomless” scalability, point-in-time reads, and the strictest level of consistency.

SQLite’s web renaissance marches on.

What’s this? Basically:

• A Raspberry Pi CM4
• WiFi, 8GB RAM
• Up to 16TB of SSD Storage
• An App Store, backups, remote access, and much more

Batch 1 is shipping now, but they’re taking pre-orders for batch 2, which ships in October. Starts at $159 and goes up from there.

User Uptrenda asked HN about their favorite data structures, sharing their own to get it started:

I’ll start: bloom filters. Lets you test if a value is definitely NOT in a list of pre-stored values (or POSSIBLY in a list - with adjustable probability that influences storage of the values.)

Good use-case: routing. Say you have a list of 1 million IPs that are black listed. A trivial algorithm would be to compare every element of the set with a given IP. The time complexity grows with the number of elements. Not so with a bloom filter! A bloom filter is one of the few data structures whose time complexity does not grow with the number of elements due to the ‘keys’ not needing to be stored (‘search’ and ‘insert’ is based on the number of hash functions.)

The responses are numerous with a lot of signal, which is increasingly rare on the orange website these days. Worth a read!

Tauri is an app framework built with Rust. You build UIs for it using virtually any frontend JavaScript framework. It has three major tenets: security, privacy, and environment. Speaking to the latter:

The apps you make are lean and performant, which reduces electricity, storage space, and general natural resource consumption. Every byte saved is a leaf on a tree that gets to grow.

This project has gotten a lot of early interest because, well, people have been waiting for an Electron alternative to emerge. Now that Tauri is 1.0 I’m guessing adoption will really start to take off.

Even more good news: we have an episode on Tauri in the hopper for ya 💪

It uses IPFS for file storage and the Ethereum Name Service for discovery and syndication. Planet is a Mac app that makes managing such a setup reasonable. It also acts like an RSS reader app, letting you subscribe to other decentralized sites running on the network.

When we had Paul Copplestone from Supabase on The Changelog I asked him what a cloud native Postgres would look like. He replied,

Decoupled compute and storage. So the idea is that you should be able to attach the compute part of it to a storage, hopefully like an infinite storage; you know, anything that is infinitely scalable. If you can do this, and in particular, if the compute can start up very fast, maybe in, say, a hundred milliseconds via some sort of HTTP response, then that’s cloud native, yeah.

Enter Neon, which sells itself as:

The multi-cloud fully managed Postgres with a generous free tier. We separated storage and compute to offer autoscaling, branching, and bottomless storage.

Behind this new business from MemSQL co-founder Nikita Shamgunov is the linked (Apache-licensed) server written in Rust.

YouTubeDrive is a Wolfram Language (aka Mathematica) package that encodes/decodes arbitrary data to/from simple RGB videos which are automatically uploaded to/downloaded from YouTube. Since YouTube imposes no limits on the total number or length of videos users can upload, this provides an effectively infinite but extremely slow form of file storage.

Filed under: ways-no-youtube-engineer-ever-imagined-people-would-use-their-software

Nice to see some efforts around standardizing MLOps. Here’s their high-level selling points:

• 📦 Docker containers without the pain
• 🤬️ No more CUDA hell
• ✅ Define the inputs and outputs for your model with standard Python
• 🎁Automatic HTTP prediction server
• 🥞 Automatic queue worker
• ☁️ Cloud storage
• 🚀 Ready for production

Max Howell, creator of Homebrew, has gone back to his notes on brew2 to apply web3 concepts to help “distribute value to open source.” He’s calling this new brew tea.

Tools like Homebrew lie beneath all development tools, assisting developers to actually get development done. We know the graph of all open source, which means we’re uniquely placed to innovate in interesting and exciting ways. This is exactly what tea will do. We’re taking our knowledge of how to make development more efficient and throwing innovations nobody has ever really considered before.

With plans to move the package registry on-chain, Max lays out the numerous benefits due to “inherent benefits of blockchain technology”:

• Packages will be immutable (no more left-pad incidents)
• Packages will always be available (we’ll use decentralized storage)
• Releases will be signed by the maintainers themselves (rather than a middleman you are told you can trust)
• Tools can be built to fundamentally verify the integrity of your app’s open source constitution
• Token can flow through the graph

Max says “token flowing is where things get really interesting,” and goes on to say “with our system people who care about the health of the open source ecosystem buy some token and stake it.”

(Thanks to Omri Gabay for sharing this first in our community Slack)

Cloudflare has a lot more to offer than merely DDoS protection and CDN services. On this episode, Jon Kuperman joins Amal & Jerod to talk through many of their cool new things like Workers, KV, Durable Objects, and R2 Storage. Thanks to listener Matt Mannucci for requesting this episode!

This is a great post byu Andy Kimball on what they mean when they say CockroachDB is now serverless:

Serverless means you don’t have to think about servers. Of course there are servers hard at work handling your application’s requests, but that’s our problem, not yours. We do all the hard work behind the scenes of allocating, configuring, and maintaining the servers. Instead of paying for servers, you pay for the requests that your application makes to the database and the storage that your data consumes.

How they got here:

Before now, a single physical CockroachDB cluster was intended for dedicated use by a single user or organization. That is called single-tenancy. Over the past several CockroachDB releases, we’ve quietly been adding multi-tenancy support…

How they scale it:

Within every Serverless cluster, there is an autoscaler component that is responsible for determining the ideal number of SQL pods that should be assigned to each tenant, whether that be one, many, or zero.

And what it all means for Cockroach’s users.

This cool open source project provides transparent client-side encryption to be used with your cloud file storage of choice. Which choices, you ask?

Works with Dropbox, Google Drive, OneDrive, MEGA, pCloud, ownCloud, Nextcloud and any other cloud storage service which synchronizes with a local directory