Longhorn: Implementing persistent storage for Kubernetes clusters

The Longhorn project is a new way of distributed block storage based on cloud and container deployment launched by Rancher Labs. Longhorn follows the principles of microservices, uses containers to build small independent components into distributed block storage, and uses container orchestration to coordinate these components to form an elastic distributed system.

Since the release of the Longhorn project in April 2017, there has been a great deal of interest in running Longhorn storage on Kubernetes clusters. Recently, Longhorn v0.2 was released, which supports persistent storage implementation for any Kubernetes cluster!

Why Longhorn

Today, cloud and container-based deployments are growing in scale, distributed block storage systems are becoming more complex, and the number of volumes on a single storage controller is increasing. In the early 2000s, the number of volumes on a storage controller was only a few dozen, but modern cloud environments require tens of thousands to millions of distributed block storage volumes. Storage controllers become highly complex distributed systems.

Distributed block storage is inherently simpler than other forms of distributed storage such as file systems. No matter how many volumes are in the system, each volume can only be mounted by a single host. Because of this, we envisioned, could a large block storage controller be split into multiple smaller storage controllers? To do this split, we need to ensure that these volumes are still built from a common pool of disks, and we need to have a way to orchestrate these storage controllers so that they can work together.

To take this idea to the limit, we created the Longhorn project. This is a direction we think is worth exploring, and there is only one volume on each controller, which will greatly simplify the design of storage controllers. Because the fault domain of the controller software is limited to a single volume, a controller crash will only affect one volume.

Longhorn takes advantage of the core technology of how to orchestrate large numbers of containers and virtual machines in recent years. For example, instead of building a highly complex controller that can scale to 100,000 volumes, Longhorn created 100,000 individual controllers in order to keep the storage controller simple and lightweight. We can then leverage state-of-the-art orchestration systems like Swarm, Mesos, and Kubernetes to schedule these independent controllers, sharing resources from a set of disks, working together to form a resilient distributed block storage system.

There are many other advantages of Longhorn's microservice-based design. Because each volume has its own controller, upgrading each volume's controller and replica container will not cause significant interruption of IO operations. Longhorn can create a long-running job to orchestrate the upgrade of all live volumes while ensuring that there is no disruption to ongoing system operations. To ensure that upgrades do not cause unexpected problems, Longhorn has the option to upgrade a small portion of the volume and roll back to an older version if problems arise during the upgrade process. These practices are widely used in modern microservice applications, but are less common in storage systems. We hope Longhorn can help more applications of microservices in the storage field.

Enter image description

Overview of Longhorn Features

Shared resource pool : A shared resource pool is formed from local disks or network storage installed in computing or dedicated storage hosts.
Create block storage volumes for containers and virtual machines : You can specify the size of the volume, the IOPS requirements, and the number of replicas you want to synchronize across hosts (where hosts are those hosts that provide storage resources for the volume). A replica is thin provisioned on the underlying disk or network storage.
Create a dedicated storage controller for each volume : This is probably the most distinctive feature of Longhorn compared to most existing distributed storage systems. Most existing distributed storage systems typically employ complex controller software to serve volumes ranging from hundreds to millions. Unlike Longhorn, which has only one volume per controller, Longhorn turns each volume into a microservice.
Scheduling multiple replicas across compute or storage hosts : Longhorn monitors the health of each replica, repairs problems, and regenerates replicas if necessary.
Operate storage controllers and replicas as Docker containers : For example, a volume with three replicas means four containers.
Assign multiple storage "frontends" to each volume : Common frontends include Linux kernel devices (mapped to /dev/longhorn) and iSCSI targets. Linux kernel devices are suitable for supporting Docker volumes, while iSCSI targets are more suitable for supporting QEMU/KVM and VMware volumes.
Create volume snapshots and AWS EBS-style backups : You can create up to 254 snapshots per volume, which can be backed up one by one to NFS or S3-compatible secondary storage. Only changed bytes are copied and stored during the backup operation.
Specify a schedule for periodic snapshot and backup operations : you can specify the frequency of these operations (hourly, daily, weekly, monthly, and yearly), the exact time to perform these operations (for example, every Sunday at 3:00 AM), and How many rotating snapshots and backup sets to keep.

Persistent storage supporting any Kubernetes cluster

Longhorn v0.2 supports persistent storage for any Kubernetes cluster. Once deployed on a Kubernetes cluster, Longhorn will automatically cluster all available local storage on all nodes in the Kubernetes cluster, resulting in replicated and distributed block storage. You can perform snapshot and backup operations on Longhorn volumes and replicate them synchronously to multiple nodes.

We have ported Longhorn Manager as Kubernetes Controller. All Longhorn state is stored as Custom Resource Definitions (Custom Resource Definitions, CRDs). Longhorn also does not require a separate etcd server. In addition, Longhorn Manager exposes APIs for performing Longhorn volume operations and snapshot/backup operations, which will be used during operations performed by the Longhorn UI and the Kubernetes Flexvolume driver.

Run the following command to deploy the entire Longhorn storage system on your Kubernetes cluster:

kubectl create -f https://raw.githubusercontent.com/rancher/longhorn/v0.2/deploy/longhorn.yaml

If you are using GKE, please refer here: https://github.com/rancher/longhorn/blob/master/README.md#google-kubernetes-engine

After deployment, you can find the appropriate IP by viewing the Kubernetes service in the UI: kubectl -n longhorn-system get svc

Enter image description

Now you can access the UI using 100.200.200.123 or via <node_ip>:12345.

Longhorn provides full integration of Kubernetes.

You can create a pod with volume backup via Longhorn like this:

apiVersion: v1
kind: Pod
metadata:
  name: volume-test
  namespace: default
spec:
  containers:
  - name: volume-test
    image: nginx:stable-alpine
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: voll
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: voll
    flexVolume:
      driver: "rancher.io/longhorn"
      fsType: "ext4"
      options:
        size: "2Gi"
        numberOfReplicas: "3"
        staleReplicaTimeout: "20"
        fromBackup: ""

Longhorn also supports dynamic provisioners. For example, you can define a StorageClass in Kubernetes like this:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn
provisioner: rancher.io/longhorn
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  fromBackup: ""

Then create a PVC (PersistentVolumeClaim) and use it in the Pod:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-volv-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
 resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: volume-test
  namespace: default
spec:
  containers:
  - name: volume-test
    image: nginx:stable-alpine
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: volv
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: volv
    persistentVolumeClaim:
      claimName: longhorn-volv-pvc

Everything is open source

Rancher Labs, which has always adhered to the concept of open source, launched Longhorn is still 100% open source software. You can download Longhorn on GitHub: https://github.com/rancher/longhorn

Until today, Longhorn is still in the process of continuous optimization and updating. Feel free to provide valuable comments and feedback to the Rancher team on GitHub or in the Rancher WeChat group ❤️