Published on
HexDB: I Built a Database. Then I Kept Going.
An open-source database, file store, search engine, and a few other things I apparently thought should live together.
So...I built a database.
And then, as tends to happen when I get interested in something, I kept adding things to it. Search. Replication. Transactions. File storage. Queues. Streams. An admin UI. AI integration. At some point I probably should have asked whether this was getting out of hand.
That point has long since passed.
The good news is that HexDB is now officially released. It's free, open source under Apache 2.0, and available for Windows, macOS, Linux, and Docker. There's also HexDB Enterprise, a hosted version for people who want to use the database without becoming its full-time system administrator.
This is a project I've poured a ridiculous amount of engineering into, so I figured I'd explain what it actually does, why I built it, and some of the decisions behind it.
Why Build Another Database?
A perfectly reasonable question. There are already some excellent ones.
I've spent more than 25 years building software, including a lot of time with SQL databases, document stores, Elasticsearch, and distributed systems. I like these technologies. I've spoken about Elasticsearch publicly, and I've used it extensively in real systems. I have nothing against the tools we've already got.
What I do have a problem with is how many separate pieces of infrastructure we've come to accept as normal for an application.
You need somewhere to store your data. Then a search engine. Then object storage for files. Then a message queue, a scheduler, and something to manage background work. Each has its own credentials, configuration, deployment, monitoring, failure modes, and bill. And eventually somebody has to explain the whole mess on an architecture diagram.
There are very good reasons to separate these systems at scale. I've designed those architectures myself. But for a surprising number of applications, the overhead is substantial.
I wanted to see how far I could get by bringing those capabilities together into one small, fast, secure system that developers could run and understand without assembling an entire cloud platform first.
I also wanted to write a storage engine in Rust. Apparently that's what passes for a hobby around here.
What HexDB Actually Is
At its core, HexDB is a document database. It stores JSON documents in collections I call tessellations. If you've worked with MongoDB collections or relational tables, the concept will feel familiar.
Documents get time-sortable ULID identifiers. You can insert, replace, patch, delete, upsert, and bulk-write them. You can add indexes, search them, aggregate them, assign expiration times, and wrap operations across multiple tessellations in atomic transactions. Schemas are optional, but if you want them, HexDB supports versioning, validation, migrations, and rollback.
A single running HexDB server is called a hex. Several hexes working together form a lattice. Yes, I named the pieces myself. The project is called HexDB partly because of Dream in Hex and partly because of the way it protects data, which I'll get to in a moment.
You can run one hex on a laptop, a Raspberry Pi, or a server. You can also run a lattice across multiple machines for replication and failover. Each hex exposes the APIs and serves its own web-based administration console.
The installation is small, roughly 52 MB for the server, CLI, ODBC driver, and admin UI in the published benchmark build. You don't need a separate database coordinator, search cluster, message broker, or admin service just to get started.
The Six-Vertex Thing
This is probably my favorite part of the architecture, and the reason for the name.
Inside each hex, documents held in memory are encoded into six separate vertices. Four carry data and two carry parity, using Reed-Solomon error correction. Each fragment has a hash so HexDB can recognize corruption.
The practical result is that HexDB can reconstruct a document from any four good fragments. Two vertices can be lost or corrupted and the document remains readable. Background integrity checks detect and repair damaged fragments.
Files use a related idea on disk. Their encrypted content is split into four data fragments and two parity fragments, stored in six vertex directories. Those directories can even live on separate physical disks. Lose up to two of those storage vertices and the content can still be rebuilt.
That is different from replication between servers. Six vertices are part of one hex. A lattice is made of multiple hexes. I mention this because it's easy to confuse fragment-level fault tolerance with losing two whole machines, and they're very different failure scenarios.
Durability also matters. HexDB records writes in an encrypted write-ahead log and, with the normal durability setting, fsyncs them before acknowledging success. Data is later flushed into compressed, encrypted SSTables. Bloom filters, indexes, a memory cache, and compaction take care of the read and storage paths.
In other words, the six-vertex design helps recover corrupted fragments. The write-ahead log and storage engine handle durable commits. The lattice handles server-level availability. Each has its own job.
Lattices, Overseers, and What Happens When a Machine Dies
A lattice has three possible roles:
- Overseer: The hex that accepts writes, publishes changes, and runs things like schedules and stream destinations.
- Harvester: A replica that serves reads and can become Overseer if the current one disappears.
- Replicant: A standby replica that follows the Overseer but isn't eligible to lead, useful for backups and recovery.
Hexes discover one another using an authenticated shared lattice secret. New replicas synchronize a snapshot and then follow the ordered change feed. If the Overseer goes away, an eligible Harvester takes over. A recovered hex can rejoin as a replica.
By default, replication is asynchronous. For deployments that need stronger guarantees, you can require replica acknowledgements before a write is reported as successful and configure a write quorum to protect against split-brain behavior.
There's also optional slicing, which distributes 64 portions of the dataset across replicas to save their disk and memory. The Overseer still holds the complete dataset and processes all writes. HexDB scales reads and resilience by adding hexes; it isn't pretending to be an infinitely sharded, multi-writer database.
I've tried to make the tradeoffs explicit rather than burying them behind a reassuring marketing adjective.
Documents Are Only Half the Story
One of the bigger additions to HexDB is that it stores actual files. Photos, videos, music, PDFs, Office documents, Photoshop files, archives, and plenty more.
I didn't want files to be anonymous blobs with names attached. HexDB extracts metadata and makes it searchable with the same query engine used for JSON documents.
For a photograph, that might be EXIF camera information, GPS coordinates, or XMP tags. For music, it might be artist, album, bitrate, or ID3 metadata. For video, there's duration, codec, and other embedded information. For a PDF or Office document, HexDB can extract properties and readable text. Files become searchable by what they contain, instead of relying entirely on somebody remembering what they named them.
Underneath, files are broken into content-defined chunks, deduplicated using keyed BLAKE3 identifiers, compressed when worthwhile, encrypted, and stored across the six vertices. Repeated content doesn't need to be stored repeatedly.
You also get resumable uploads, ranged downloads for media playback, thumbnails, file version history, rollback, a trash, quotas, and shareable links with expirations and optional passwords. The short sharing codes even use MAR-62, an algorithm I originally developed for my GoToIt.us project. It's nice when an old project finds a new job.
And, because accepting arbitrary uploaded files is a wonderful way to invite trouble, metadata extraction runs in a restricted worker process. It has no access to the database's encryption keys or the rest of the data store.
How You Talk to It
I wanted HexDB to be useful whether you're writing an application, exploring data, or building a service that needs to react to changes.
It supports REST, GraphQL, gRPC, and read-only SQL. The SQL implementation includes joins, subqueries, common table expressions, filtering, grouping, and other familiar query features. Writes go through the document APIs or transactions.
There are client libraries for JavaScript/TypeScript, Python, and .NET, plus an Entity Framework Core provider, ODBC, and JDBC drivers. You don't have to abandon your existing language or framework to try it.
Here's a small TypeScript example using the Node driver:
import { HexDB } from "@dreaminhex/hexdb";
const db = new HexDB({
url: "http://127.0.0.1:7700",
apiKey: process.env.HEXDB_API_KEY
});
const articles = db.tessellation("articles");
await articles.insert({ title: "Hello, HexDB", published: true });
const results = await articles.query({
filter: { published: true },
sort: "-title"
});
The same data is available to GraphQL clients, SQL readers, and the admin UI, subject to the user's permissions. Field indexes, composite and unique indexes, and full-text analyzers help the query engine avoid scanning everything. There's even an index advisor that watches query patterns and recommends improvements.
The Stuff That Usually Lives in Other Services
This is where my original scope really started to wander.
HexDB has an ordered change feed, named publish/subscribe streams with consumer groups, and work queues with leases, acknowledgements, retries, delayed messages, and dead-letter handling. You can write functions in Python, JavaScript, or TypeScript, schedule them, or fire them before and after document changes.
There's a plugin system for integrating with Kafka, Amazon Kinesis, OpenTelemetry, PostgreSQL, SQL Server, S3, and other sources and destinations. You can keep the architecture simple when that's appropriate and integrate with the bigger ecosystem when it isn't.
The admin interface covers the ordinary operational work too: a live dashboard, document and file browsers, query consoles, user and role management, schemas, indexes, logs, audit events, backups, and a visualization of the lattice and its six vertices.
I spend a lot of time looking at systems that are technically capable but unnecessarily miserable to operate. I didn't want to build one of those.
Security, AI, and a Bit of Restraint
Security wasn't something I wanted to bolt on at the end. HexDB encrypts stored data with AES-256-GCM, supports TLS, authenticates communication between hexes, and includes sessions, API keys, multi-factor authentication, auditing, and custom role-based access control.
Permissions can apply to individual tessellations, to rows matching a filter, and to specific fields. File access can be limited by folder. The goal is to enforce those restrictions throughout the APIs and query paths rather than hoping every application remembers to do it.
And yes, HexDB has AI features. You can connect Claude, OpenAI, Azure OpenAI, GitHub Models, or a compatible local model. With your approval and configuration, AI can suggest file tags and descriptions, help choose video thumbnails, and provide another opinion on index recommendations. There's also an Ask interface that lets a permitted user question their data in plain English.
Every hex also exposes a built-in Model Context Protocol (MCP) server, so compatible AI agents can inspect and query the data they're authorized to see. Those MCP tools are read-only and honor the same row and field restrictions as other clients.
Most importantly, AI is optional. No external AI provider receives your data unless an administrator configures and enables that functionality. I like useful AI. I also like having a say in what gets sent to it.
How Fast Is It?
The benchmark suite ships with the source, so you can run it yourself rather than taking my word for it.
On a Windows 11 laptop with an Intel Core i9-12900H and Samsung 980 PRO NVMe SSD, the published tests measured about 31,000 bulk document writes per second, 0.18 ms median reads by ID, and 2.5 ms median individual writes, with durable WAL synchronization enabled. That's a local, single-server benchmark, not a claim that every workload or cloud instance will deliver the same numbers.
There are also tests for crash recovery, vertex repair, failover, indexing, compression, and large documents. The full benchmark report includes the methodology and the less glamorous numbers too, as it should.
HexDB Enterprise: Same Database, Somebody Else Runs the Servers
The other big announcement is HexDB Enterprise.
You can download HexDB, deploy it wherever you like, and pay nothing in license fees. That's staying. The source is available under Apache 2.0, and the hosted service uses the same open-source release.
Enterprise is for teams that would rather have us handle the infrastructure. You get your own lattice running in dedicated, customer-isolated containers, with full administrator access. We handle daily disk snapshots, upgrades, hosting, and support. You get an HTTPS address for the lattice and can point your own domain at it. You still control the users, roles, API keys, and database settings.
Plans currently start at $39 per hex per month for the Small tier. A three-hex Small lattice starts at $117 per month, before applicable taxes and regional pricing. Larger configurations are available for busier applications. One hex is fine for development; three are the sensible starting point if you want the lattice to keep operating through the loss of one hex.
I especially wanted the hosted option to avoid the usual trap where the managed edition becomes a completely different, proprietary product. Your data can be exported through the API and moved to a lattice you operate yourself. The hosted offering is a convenience, not a hostage situation.
What HexDB Isn't
I'm proud of what I've built, but it's a young database, and you should evaluate it like one.
The Overseer handles every write and keeps the full dataset, even when slicing is enabled. SQL is read-only, and some advanced SQL features aren't supported. The Entity Framework provider has limits compared with a mature relational provider. For very large, specialized workloads, dedicated products may still be the better choice.
And while the storage engine is designed for durability, asynchronous replication is the default. If an Overseer fails before its last acknowledged write reaches another hex, that write can be lost. Configure replica acknowledgements and quorum based on what your application actually requires. Keep backups, too. Replication isn't a substitute for them.
I'd much rather be specific about limitations than have somebody discover them at three in the morning.
Go Play with It
HexDB has reached its full public release, and the current code, releases, manual, drivers, and example deployments are available now. As of this article, the latest tagged release is v2.1.2.
If you're curious, start with hexdb.app. There are architecture diagrams, documentation, benchmarks, and a live playground. If you'd rather read code, the GitHub repository is open. And if you need a hosted deployment, take a look at HexDB Enterprise.
Try it. Build something. File a bug. Tell me what doesn't make sense. Contributions and genuinely useful criticism are welcome.
I started HexDB because I wanted to find out whether I could build the kind of database I'd enjoy using. It turns out I could. Now I'd like to see what other people do with it.
Doc