Disks and Volumes
Last updated
Last updated
Making persistent storage in the cluster (volumes) accessible to VMs consists of three parts. First, volumes are specified in spec.volumes
. Second, disks are added to the VM by specifying them in spec.domain.devices.disks
. Finally, a reference to the specified volume is added to the disk specification by name.
Like all other vmi devices a spec.domain.devices.disks
element has a mandatory name
, and furthermore, the disk's name
must reference the name
of a volume inside spec.volumes
.
A disk can be made accessible via four different types:
DEPRECATED
All possible configuration options are available in the .
All types, with the exception of floppy, allow you to specify the bus
attribute. The bus
attribute determines how the disk will be presented to the guest operating system. floppy disks don't support the bus
attribute: they are always attached to the fdc
bus.
A lun
disk will expose the volume as a LUN device to the VM. This allows the VM to execute arbitrary iSCSI command passthrough.
A minimal example which attaches a PersistentVolumeClaim
named mypvc
as a lun
device to the VM:
A disk
disk will expose the volume as an ordinary disk to the VM.
A minimal example which attaches a PersistentVolumeClaim
named mypvc
as a disk
device to the VM:
You can set the disk bus
type, overriding the defaults, which in turn depends on the chipset the VM is configured to use:
Note: Starting with version 0.16.0, floppy disks are deprecated and will be rejected. They will be removed from the API in a future version.
A floppy
disk will expose the volume as a floppy drive to the VM.
A minimal example which attaches a PersistentVolumeClaim
named mypvc
as a floppy
device to the VM:
A cdrom
disk will expose the volume as a cdrom drive to the VM. It is read-only by default.
A minimal example which attaches a PersistentVolumeClaim
named mypvc
as a floppy
device to the VM:
Supported volume sources are
Allows attaching cloudInitNoCloud
data-sources to the VM. If the VM contains a proper cloud-init setup, it will pick up the disk as a user-data source.
A simple example which attaches a Secret
as a cloud-init disk
datasource may look like this:
Allows attaching cloudInitConfigDrive
data-sources to the VM. If the VM contains a proper cloud-init setup, it will pick up the disk as a user-data source.
A simple example which attaches a Secret
as a cloud-init disk
datasource may look like this:
Allows connecting a PersistentVolumeClaim
to a VM disk.
Use a PersistentVolumeClaim when the VirtualMachineInstance's disk needs to persist after the VM terminates. This allows for the VM's data to remain persistent between restarts.
A PersistentVolume
can be in "filesystem" or "block" mode:
Filesystem: For KubeVirt to be able to consume the disk present on a PersistentVolume's filesystem, the disk must be named disk.img
and be placed in the root path of the filesystem. Currently the disk is also required to be in raw format. > Important: The disk.img
image file needs to be owned by the user-id 107
in order to avoid permission issues.
Note: If the
disk.img
image file has not been created manually before starting a VM then it will be created automatically with thePersistentVolumeClaim
size. Since not every storage provisioner provides volumes with the exact usable amount of space as requested (e.g. due to filesystem overhead), KubeVirt tolerates up to 10% less available space. This can be configured with thepvc-tolerate-less-space-up-to-percent
value in thekubevirt-config
ConfigMap.
A simple example which attaches a PersistentVolumeClaim
as a disk
may look like this:
DataVolumes are a way to automate importing virtual machine disks onto PVCs during the virtual machine's launch flow. Without using a DataVolume, users have to prepare a PVC with a disk image before assigning it to a VM or VMI manifest. With a DataVolume, both the PVC creation and import is automated on behalf of the user.
DataVolumes can be defined in the VM spec directly by adding the DataVolumes to the dataVolumeTemplates
list. Below is an example.
You can see the DataVolume defined in the dataVolumeTemplates section has two parts. The source and pvc
The source part declares that there is a disk image living on an http server that we want to use as a volume for this VM. The pvc part declares the spec that should be used to create the PVC that hosts the source data.
When this VM manifest is posted to the cluster, as part of the launch flow a PVC will be created using the spec provided and the source data will be automatically imported into that PVC before the VM starts. When the VM is deleted, the storage provisioned by the DataVolume will automatically be deleted as well.
For a VMI object, DataVolumes can be referenced as a volume source for the VMI. When this is done, it is expected that the referenced DataVolume exists in the cluster. The VMI will consume the DataVolume, but the DataVolume's life-cycle will not be tied to the VMI.
Below is an example of a DataVolume being referenced by a VMI. It is expected that the DataVolume alpine-datavolume was created prior to posting the VMI manifest to the cluster. It is okay to post the VMI manifest to the cluster while the DataVolume is still having data imported. KubeVirt knows not to start the VMI until all referenced DataVolumes have finished their clone and import phases.
A DataVolume is a custom resource provided by the Containerized Data Importer (CDI) project. KubeVirt integrates with CDI in order to provide users a workflow for dynamically creating PVCs and importing data into those PVCs.
Installing CDI
Pick the latest stable release and post the corresponding cdi-controller-deployment.yaml manifest to your cluster.
An ephemeral volume is a local COW (copy on write) image that uses a network volume as a read-only backing store. With an ephemeral volume, the network backing store is never mutated. Instead all writes are stored on the ephemeral image which exists on local storage. KubeVirt dynamically generates the ephemeral images associated with a VM when the VM starts, and discards the ephemeral images when the VM stops.
Ephemeral volumes are useful in any scenario where disk persistence is not desired. The COW image is discarded when VM reaches a final state (e.g., succeeded, failed).
Currently, only PersistentVolumeClaim
may be used as a backing store of the ephemeral volume.
containerDisk was originally registryDisk, please update your code when needed.
The containerDisk
feature provides the ability to store and distribute VM disks in the container image registry. containerDisks
can be assigned to VMs in the disks section of the VirtualMachineInstance spec.
No network shared storage devices are utilized by containerDisks
. The disks are pulled from the container registry and reside on the local node hosting the VMs that consume the disks.
containerDisks
are ephemeral storage devices that can be assigned to any number of active VirtualMachineInstances. This makes them an ideal tool for users who want to replicate a large number of VM workloads that do not require persistent data. containerDisks
are commonly used in conjunction with VirtualMachineInstanceReplicaSets.
containerDisks
are not a good solution for any workload that requires persistent root disks across VM restarts.
Users can inject a VirtualMachineInstance disk into a container image in a way that is consumable by the KubeVirt runtime. Disks must be placed into the /disk
directory inside the container. Raw and qcow2 formats are supported. Qcow2 is recommended in order to reduce the container image's size. containerdisks
can and should be based on scratch
. No content except the image is required.
Note: Prior to kubevirt 0.20, the containerDisk image needed to have kubevirt/container-disk-v1alpha as base image.
Note: The containerDisk needs to be readable for the user with the UID 107 (qemu).
Example: Inject a local VirtualMachineInstance disk into a container image.
Example: Inject a remote VirtualMachineInstance disk into a container image.
Example: Upload the ContainerDisk container image to a registry.
Example: Attach the ContainerDisk as an ephemeral disk to a VM.
Note that a containerDisk
is file-based and therefore cannot be attached as a lun
device to the VM.
ContainerDisk also allows to store disk images in any folder, when required. The process is the same as previous. The main difference is, that in custom location, kubevirt does not scan for any image. It is your responsibility to provide full path for the disk image. Providing image path
is optional. When no path
is provided, kubevirt searches for disk images in default location: /disk
.
Example: Build container disk image:
Create VMI with container disk pointing to the custom location:
An emptyDisk
works similar to an emptyDir
in Kubernetes. An extra sparse qcow2
disk will be allocated and it will live as long as the VM. Thus it will survive guest side VM reboots, but not a VM re-creation. The disk capacity
needs to be specified.
Example: Boot cirros with an extra emptyDisk
with a size of 2GiB
:
Ephemeral VMs very often come with read-only root images and limited tmpfs space. In many cases this is not enough to install application dependencies and provide enough disk space for the application data. While this data is not critical and thus can be lost, it is still needed for the application to function properly during its lifetime. This is where an emptyDisk
can be useful. An emptyDisk is often used and mounted somewhere in /var/lib
or /var/run
.
A hostDisk
volume type provides the ability to create or use a disk image located somewhere on a node. It works similar to a hostPath
in Kubernetes and provides two usage types:
DiskOrCreate
if a disk image does not exist at a given location then create one
Disk
a disk image must exist at a given location
Note: you need to enable the HostDisk feature gate.
Example: Create a 1Gi disk image located at /data/disk.img and attach it to a VM.
Note: Currently, ConfigMap update is not propagate into the VMI. If a ConfigMap is updated, only a pod will be aware of changes, not running VMIs.
Example: Attach the configMap
to a VM and use cloudInit
to mount the iso
disk:
Note: Currently, Secret update propagation is not supported. If a Secret is updated, only a pod will be aware of changes, not running VMIs.
Example: Attach the secret
to a VM and use cloudInit
to mount the iso
disk:
Example:
Getting a limited set of host and VM metrics is in some cases required to allow third-parties diagnosing performance issues on their appliances. One prominent example is SAP HANA.
Example:
The vm-dump-metrics
tool can be used to read the metrics:
Libvirt has the ability to use IOThreads for dedicated disk access (for supported devices). These are dedicated event loop threads that perform block I/O requests and improve scalability on SMP systems. KubeVirt exposes this libvirt feature through the ioThreadsPolicy
setting. Additionally, each Disk
device exposes a dedicatedIOThread
setting. This is a boolean that indicates the specified disk should be allocated an exclusive IOThread that will never be shared with other disks.
Currently valid policies are shared
and auto
. If ioThreadsPolicy
is omitted entirely, use of IOThreads will be disabled. However, if any disk requests a dedicated IOThread, ioThreadsPolicy
will be enabled and default to shared
.
An ioThreadsPolicy
of shared
indicates that KubeVirt should use one thread that will be shared by all disk devices. This policy stems from the fact that large numbers of IOThreads is generally not useful as additional context switching is incurred for each thread.
Disks with dedicatedIOThread
set to true
will not use the shared thread, but will instead be allocated an exclusive thread. This is generally useful if a specific Disk is expected to have heavy I/O traffic, e.g. a database spindle.
auto
IOThreads indicates that KubeVirt should use a pool of IOThreads and allocate disks to IOThreads in a round-robin fashion. The pool size is generally limited to twice the number of VCPU's allocated to the VM. This essentially attempts to dedicate disks to separate IOThreads, but only up to a reasonable limit. This would come in to play for systems with a large number of disks and a smaller number of CPU's for instance.
As a caveat to the size of the IOThread pool, disks with dedicatedIOThread
will always be guaranteed their own thread. This effectively diminishes the upper limit of the number of threads allocated to the rest of the disks. For example, a VM with 2 CPUs would normally use 4 IOThreads for all disks. However if one disk had dedicatedIOThread
set to true, then KubeVirt would only use 3 IOThreads for the shared pool.
There is always guaranteed to be at least one thread for disks that will use the shared IOThreads pool. Thus if a sufficiently large number of disks have dedicated IOThreads assigned, auto
and shared
policies would essentially result in the same layout.
When guest's vCPUs are pinned to a host's physical CPUs, it is also best to pin the IOThreads to specific CPUs to prevent these from floating between the CPUs. KubeVirt will automatically calculate and pin each IOThread to a CPU or a set of CPUs, depending on the ration between them. In case there are more IOThreads than CPUs, each IOThread will be pinned to a CPU, in a round-robin fashion. Otherwise, when there are fewer IOThreads than CPU, each IOThread will be pinned to a set of CPUs.
In this example, emptydisk and emptydisk2 both request a dedicated IOThread. vmi-shared_disk, and emptydisk 3 through 6 will all shared one IOThread.
This VM is identical to the first, except it requests auto IOThreads. emptydisk
and emptydisk2
will still be allocated individual IOThreads, but the rest of the disks will be split across 2 separate iothreads (twice the number of CPU cores is 4).
Disks will be assigned to IOThreads like this:
Block Multi-Queue is a framework for the Linux block layer that maps Device I/O queries to multiple queues. This splits I/O processing up across multiple threads, and therefor multiple CPUs. libvirt recommends that the number of queues used should match the number of CPUs allocated for optimal performance.
This feature is enabled by the BlockMultiQueue
setting under Devices
:
Note: Due to the way KubeVirt implements CPU allocation, blockMultiQueue can only be used if a specific CPU allocation is requested. If a specific number of CPUs hasn't been allocated to a VirtualMachine, KubeVirt will use all CPU's on the node on a best effort basis. In that case the amount of CPU allocation to a VM at the host level could change over time. If blockMultiQueue were to request a number of queues to match all the CPUs on a node, that could lead to over-allocation scenarios. To avoid this, KubeVirt enforces that a specific slice of CPU resources is requested in order to take advantage of this feature.
This example will enable Block Multi-Queue for the disk mypvcdisk
and allocate 4 queues (to match the number of CPUs requested).
KubeVirt supports none
and writethrough
KVM/QEMU cache modes.
none
I/O from the guest is not cached on the host. Use this option for guests with large I/O requirements. This option is generally the best choice.
writethrough
I/O from the guest is cached on the host but written through to the physical medium.
Important:
none
cache mode is set as default if the file system supports direct I/O, otherwise,writethrough
is used.Note: It is possible to force a specific cache mode, although if
none
mode has been chosen and the file system does not support direct I/O then started VMI will return an error.
Example: force writethrough
cache mode
All possible configuration options are available in the .
Block: Use a block volume for consuming raw block devices. Note: you need to enable the BlockVolume
.
In order to take advantage of the DataVolume volume source on a VM or VMI, the DataVolumes
must be enabled in the kubevirt CR. CDI must also be installed.
Go to the
Up-to-date information on supported backing stores can be found in the .
A configMap
is a reference to a in Kubernetes. An extra iso
disk will be allocated which has to be mounted on a VM. To mount the configMap
users can use cloudInit
and the disks serial number. The name
needs to be set for a reference to the created kubernetes ConfigMap
.
Note: Due to a Kubernetes CRD , you cannot control the paths within the volume where ConfigMap keys are projected.
A secret
is a reference to a in Kubernetes. An extra iso
disk will be allocated which has to be mounted on a VM. To mount the secret
users can use cloudInit
and the disks serial number. The secretName
needs to be set for a reference to the created kubernetes Secret
.
Note: Due to a Kubernetes CRD , you cannot control the paths within the volume where Secret keys are projected.
A serviceAccount
volume references a Kubernetes . A new iso
disk will be allocated with the content of the service account (namespace
, token
and ca.crt
), which needs to be mounted in the VM. For automatic mounting, see the configMap
and secret
examples above.
A downwardMetrics
volume exposes a limited set of VM and host metrics to the guest as a raw block volume. The format of the block volume is compatible with .
Note: The DownwardMetrics feature gate to use this volume. Available starting with KubeVirt v0.42.0.
To further improve the vCPUs latency, KubeVirt can allocate an additional dedicated physical CPU, exclusively for the emulator thread, to which it will be pinned. This will effectively "isolate" the emulator thread from the vCPUs of the VMI. When ioThreadsPolicy
is set to auto
IOThreads will also be "isolated" from the vCPUs and placed on the same physical CPU as the QEMU emulator thread.