The problem
Most storage projects stop at a toy upload endpoint. That hides the hard parts: range requests, multipart uploads, metadata consistency, failure recovery, and durable placement of bytes.
I wanted to understand the full path from an S3 request to a chunk on disk — and make it operable inside Kubernetes instead of only on a laptop.
What I built
- An S3-compatible API with multipart uploads, range requests, and IAM-style policies.
- A metadata layer in PostgreSQL separated from the chunk store that holds the actual bytes.
- Erasure coding and replication policies for durability, with chunk-level deduplication to avoid storing identical data twice.
- A Kubernetes CSI driver and Prometheus metrics so the service can run and be observed like real infrastructure.
System shape
Decisions and tradeoffs
Metadata and bytes are different problems
PostgreSQL owns object metadata; the chunk store owns content-addressed data. Each can be reasoned about and scaled independently.
Durability is a policy, not a checkbox
Replication and erasure coding are explicit choices, not hidden implementation details.
Kubernetes-native from the start
The CSI driver turns storage into something workloads can claim, not just an API to call from a demo script.
Outcome
A working systems reference with 5 GitHub stars, built to make storage internals visible: API, metadata, placement, durability, and Kubernetes integration.
Correctness first. Metrics from day one. A storage system without observability is just a rumour that your data is still there.