Databases Icon

Databases

Databases, structured data, data stores, etc.
89 Stories
All Topics

Databases simplethread.com

Relational databases aren’t dinosaurs, they’re sharks

I’ve heard way less people throwing SQL under the bus than I did back in the high-hype NoSQL days, but this article by Justin Etheredge does a good job of laying out some of the advantages and disadvantages of the RDBMS side of the fence:

The next time you hear someone describe relational databases as yesterday’s technology, or the next time you see someone assume a relational database can’t handle the needs of their unproven MVP, stop and ask them how they are going to account for the tradeoffs they’re making. Make sure they understand they aren’t skipping a dead dinosaur, they’re taking a pass on the thousands of human-years of effort that have made relational databases the sharks of the data industry.

The Changelog The Changelog #461

Fauna is rethinking the database

This week we’re talking with Evan Weaver about Fauna — the database for a new generation of applications. Fauna is a transactional database delivered as a secure and scalable cloud API with native GraphQL. It’s the first implementation of its kind based on the Calvin paper as opposed to Spanner. We cover Evan’s history leading up to Fauna, deep details on the Calvin algorithm, the CAP theorem for databases, what it means for Fauna to be temporal native, applications well suited for Fauna, and what’s to come in the near future.

Startups clickhouse.com

Introducing ClickHouse, Inc.

Alexey Milovidov, announcing the formation of a (VC funded) corporation around ClickHouse, an open source analytics DBMS:

Today I’m happy to announce ClickHouse Inc., the new home of ClickHouse. The development team has moved from Yandex and joined ClickHouse Inc. to continue building the fastest (and the greatest) analytical database management system. The company has received nearly $50M in Series A funding led by Index Ventures and Benchmark with participation by Yandex N.V. and others. I created ClickHouse, Inc. with two co-founders, Yury Izrailevsky and Aaron Katz. I will continue to lead the development of ClickHouse as Chief Technology Officer (CTO), Yury will run product and engineering, and Aaron will be CEO.

ClickHouse wasn’t always a business. It also wasn’t always open source.

Making ClickHouse open source was also not an easy decision, but now I see: doing open source is hard, but it is a big win. While it takes a tremendous effort and responsibility to maintain a popular open-source product, for us, the benefits outweigh all the costs. Since we published ClickHouse, it has been deployed in production in thousands of companies across the globe for a wide range of use cases, from agriculture to self-driving cars.

Ahmed github.com

Deploy databases and services easily for dev and testing pipelines

Peanut provides a REST API, Admin Dashboard and a command line tool to deploy and configure the commonly used services like databases, message brokers, graphing, tracing, caching tools … etc. It perfectly suited for development, manual testing, automated testing pipelines where mocking is not possible and test drives.

Under the hood, it works with the containerization runtime like docker to deploy and configure the service. Destroy the service if it is a temporary one.

Technically you can achieve the same with a bunch of yaml files or using a configuration management tool or a package manager like helm but peanut is pretty small and fun to use & should speed up your workflow!

Deploy databases and services easily for dev and testing pipelines

Databases 1729.com

The billion user table

Jon Stokes believes blockchain tech has the opportunity to take us from a world where individual corporations build their siloed users tables to a world where the entire Internet shares a single users table.

In place of a decentralized network of user data silos connected by APIs, there’s a single decentralized user data store accessible via an open protocol and a decentralized network of storage nodes. So the identity-hosting blockchain represents decentralization at the datastore implementation layer, and recentralization at the datastore access layer.

What would this produce? Jon envisions this:

Moving identity on-chain, and thereby removing the possibility of users-table-centric network effects, completely up-ends the entire landscape of API-based, access-controlled interoperability that the present Internet is built on. All of the non-technical market and political dynamics around users table size, leverage, and risk suddenly go out the window.

Databases github.com

toyDB – a distributed SQL db written in Rust

This is not a use-it-in-the-real-world kinda thing. It’s being written as a learning project, but may interest you if you want to learn about database internals. It includes:

  • Raft-based distributed consensus engine for linearizable state machine replication.
  • ACID-compliant transaction engine with MVCC-based snapshot isolation.
  • Pluggable storage engine with B+tree and log-structured backends.
  • Iterator-based query engine with heuristic optimization and time-travel support.
  • SQL interface including projections, filters, joins, aggregates, and transactions.

Databases sqlbolt.com

SQLBolt – quickly learn SQL right in your browser

This series of interactive lessons and exercises is a great place to start if you want to learn SQL. And trust me: if you don’t know SQL, you want to learn SQL. Of all the technologies and tools I’ve picked up over the course of my career, SQL has had one of the highest ROIs. It’s portable across languages/runtimes and has incredible staying power in terms of skill relevancy.

Practical AI Practical AI #139

Vector databases for machine learning

Pinecone is the first vector database for machine learning. Edo Liberty explains to Chris how vector similarity search works, and its advantages over traditional database approaches for machine learning. It enables one to search through billions of vector embeddings for similar matches, in milliseconds, and Pinecone is a managed service that puts this capability at the fingertips of machine learning practitioners.

Go github.com

A high-performance, columnar, in-memory storage engine for Go

The general idea is to leverage cache-friendly ways of organizing data in structures of arrays (SoA) otherwise known “columnar” storage in database design. This, in turn allows us to iterate and filter over columns very efficiently. On top of that, this package also adds bitmap indexing to the columnar storage, allowing to build filter queries using binary and, and not, or and xor (see kelindar/bitmap with SIMD support).

Petr Stribny stribny.name

Scaling relational SQL databases

When it comes to scaling, we might need to think about:

  • data storage, if we store more and more data and it becomes expensive or slow working with them
  • fast INSERTs and UPDATES for write-heavy workloads
  • making SELECT queries faster because of their complexity or because they need to query huge amounts of data
  • concurrency if we have many clients interacting with the database

In this article, I will present some basic ideas and starting points on scaling traditional SQL databases.

SQLite unixsheikh.com

SQLite is the only database you will ever need in most cases

SQLite is so hot right now.

Even if you start out small and later need to upscale, as long as your web application can run on the same machine as the database, which it can in 99% of the time, you can just upgrade the hardware to a beefier machine and keep business as usual.

The only time you need to consider a client-server setup is…

Jerod Santo changelog.com/posts

You might as well timestamp it

In my 15+ years of web development, there are very few things I can say are unequivocally a good idea. It almost always does depend.

Storing timestamps instead of booleans, however, is one of those things I can go out on a limb and say it doesn’t really depend all that much. You might as well timestamp it. There are plenty of times in my career when I’ve stored a boolean and later wished I’d had a timestamp. There are zero times when I’ve stored a timestamp and regretted that decision.

The Changelog The Changelog #433

Open source, not open contribution

This week we’re talking with Ben Johnson. Ben is known for his work on BoltDB, his work in open source, and as a freelance Go developer. Late January when Ben open sourced his newest project Litestream in the readme he shared how the project was open source, but not open for contribution. His reason was to protect his mental health and the long term viability of the project. On this episode we talk with Ben about what that means, his thoughts on mental health and burnout in open source, choosing a license, and the details behind Litestream - a standalone streaming replication tool for SQLite.

Founders Talk Founders Talk #75

The journey to massive scale and ultra-resilience

This week Adam talks with Spencer Kimball, CEO and Co-founder of Cockroach Labs — makers of CockroachDB an open source cloud-native distributed SQL database. Cockroach Labs recently raised $160 million dollars on a $2 billion dollar valuation. In this episode, Spencer shares his journey in open source, startups and entrepreneurship, and what they’re doing to build CockroachCloud to meet the needs of applications that require massive scale and ultra-resilience.

Lawrence Hecht The New Stack

ClickHouse has rapidly rivaled other open source databases in active contributors

Lawrence Hecht:

ClickHouse has come out of seemingly nowhere to rival Elasticsearch as the database-related open source software project with the most active contributors…

ClickHouse is column-oriented and allows for analytics reports to be generated using SQL queries in real-time. ClickHouse’s rise in popularity began in 2016, which happens to be when Apache Spark’s peak.

I first heard of ClickHouse last year when I learned that our friends at Plausible use it for their analytics backend (teamed with Postgres for relational data).

ClickHouse has rapidly rivaled other open source databases in active contributors

Databases github.com

Dolt – it's Git for data

Imagine a world where Git and MySQL got together and had a baby. They would name that baby, Dolt.

Dolt is a SQL database that you can fork, clone, branch, merge, push and pull just like a git repository. Connect to Dolt just like any MySQL database to run queries or update the data using SQL commands. Use the command line interface to import CSV files, commit your changes, push them to a remote, or merge your teammate’s changes.

All the commands you know for Git work exactly the same for Dolt. Git versions files, Dolt versions tables.

The authors also created DoltHub where you can host and share your Dolt databases.

The Changelog The Changelog #429

Community perspectives on Elastic vs AWS

This week we’re talking about the recent falling out between Elastic and AWS around the relicensing of Elasticsearch and Kibana. Like many in the community, we have been watching this very closely.

Here’s the tldr for context. On January 21st, Elastic posted a blog post sharing their concerns with Amazon/AWS misleading and confusing the community, saying “They have been doing things that we think are just NOT OK since 2015 and it has only gotten worse.” This lead them to relicense Elasticsearch and Kibana with a dual license, a proprietary license and the Sever Side Public License (SSPL). AWS responded two days later stating that they are “stepping up for a truly open source Elasticsearch,” and shared their plans to create and maintain forks of Elasticsearch and Kibana based on the latest ALv2-licensed codebases.

There’s a ton of detail and nuance beneath the surface, so we invited a handful of folks on the show to share their perspective. On today’s show you’ll hear from: Adam Jacob (co-founder and board member of Chef), Heather Meeker (open-source lawyer and the author of the SSPL license), Manish Jain (founder and CTO at Dgraph Labs), Paul Dix (co-founder and CTO at InfluxDB), VM (Vicky) Brasseur (open source & free software business strategist), and Markus Stenqvist (everyday web dev from Sweden).

Brad Fitzpatrick tailscale.com

An unlikely database migration

So the Tailscale team were using a single text file as a database (as you do) and it worked great… until it didn’t.

Even with fast NVMe drives and splitting the database into two halves (important data vs. ephemeral data that we could lose on a tmpfs), things got slower and slower. We knew the day would come. The file reached a peak size of 150MB and we were writing it as quickly as the disk I/O would let us. Ain’t that just peachy?

So, migrate to MySQL or PostgreSQL, right? Maybe SQLite?

Nope, Crawshaw had other ideas.

I won’t ruin the surprise and tell you what they went with, but I will say it’s a widely deployed system amongst cloud natives…

Chua Bok Woon github.com

sq is a code-generated, type safe query builder and struct mapper for Go

From reading through the README, this seems like a nice balance between a full-blown ORM and hand-rolling all your own SQL. For example, this point from the The mapper function is the SELECT clause. section:

In sq whatever you SELECT is automatically mapped. This means you just have to write your query, execute it and if there were no errors, the data is already in your Go variables. No iterating rows, no specifying column scan order, no error checking three times. Write your query, run it, you’re done.

Databases github.com

Graviton is like ZFS for key-value stores

Graviton Database is simple, fast, versioned, authenticated, embeddable key-value store database in pure Go… Every write is tracked, versioned and authenticated with cryptographic proofs. Additionally it is possible to take snapshots of database. Also it is possible to use simple copy,rsync commands for database backup even during live updates without any possibilities of database corruption.

Still in Alpha, but a lot of work has been done and there are features a-plenty.

0:00 / 0:00