With clusters provisioned, reconciled with Oxide, and reachable from outside their VPCs, storage for stateful workloads became the next layer to address. Kubernetes users request persistent storage through PersistentVolumeClaim objects and expect a Container Storage Interface (CSI) driver to create, attach, and mount the underlying volumes. Oxide had disks, but Kubernetes had no native way to manage their lifecycle.

Without an Oxide CSI driver, customers could deploy a third-party Kubernetes storage system such as Longhorn. Longhorn provides its own CSI driver and replicates data across disks attached to Kubernetes workers. However, using Longhorn meant backing its replicas with Oxide distributed disks, which already store three replicas on distinct sleds.

Layering one replicated storage system on another can create substantial write fan-out. When a three-replica Longhorn volume is backed by three-way-replicated Oxide distributed disks, one application write can fan out to as many as nine disk writes. The exact physical write amplification depends on the workload and configuration, but customers wanted to avoid that duplicated replication.

The introduction of Oxide local disks provided a way to remove the second layer of replication. Local disks have no built-in replication and remain tied to their sled, making them well suited to systems such as Longhorn that replicate data across Kubernetes nodes. Our Rancher showcase uses this approach today. It avoids stacking two replicated storage systems, though Longhorn still manages the storage lifecycle rather than a native Oxide integration.

For a native integration, my teammate Luiz wrote RFD 595 Oxide CSI Plugin. The workflow seemed straightforward on paper. When a user creates a PersistentVolumeClaim, the CSI controller creates an Oxide distributed disk. After Kubernetes schedules the pod, the controller attaches that disk to the selected Oxide instance, and the CSI node plugin formats and mounts it for the pod. If the pod is rescheduled onto another node, the controller detaches the disk and reattaches it to the new node.

Prototyping that workflow immediately exposed a blocker. Oxide requires an instance to be stopped before attaching or detaching a disk. Kubernetes, however, expects a CSI driver to attach storage to a running worker after scheduling a pod. Stopping the worker would disrupt every other workload on the node and could trigger cascading scheduling and attachment operations.

Before we can release our CSI plugin, we need to add support for disk hot-plug throughout the Oxide stack, from the hypervisor all the way up to the API. What began as a Kubernetes integration has turned into a project spanning multiple layers of the Oxide software stack.

Disk hot-plug and the Oxide CSI plugin remain under active development as of this writing. In the meantime, customers can use software such as Longhorn with Oxide local disks for dynamically provisioned persistent storage without stacking two layers of replication. When the native CSI plugin ships, customers will be able to use familiar Kubernetes storage APIs backed directly by Oxide distributed disks with replication and durability built in.