> For the complete documentation index, see [llms.txt](https://kubevirtlegacy.gitbook.io/user-guide/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://kubevirtlegacy.gitbook.io/user-guide/docs/operations/unresponsive_nodes.md).

# Unresponsive nodes

KubeVirt has its own node daemon, called virt-handler. In addition to the usual k8s methods of detecting issues on nodes, the virt-handler daemon has its own heartbeat mechanism. This allows for fine-tuned error handling of VirtualMachineInstances.

## virt-handler heartbeat

`virt-handler` periodically tries to update the `kubevirt.io/schedulable` label and the `kubevirt.io/heartbeat` annotation on the node it is running on:

```
$ kubectl get nodes -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      kubevirt.io/heartbeat: 2018-11-05T09:42:25Z
    creationTimestamp: 2018-11-05T08:55:53Z
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/os: linux
      cpumanager: "false"
      kubernetes.io/hostname: node01
      kubevirt.io/schedulable: "true"
      node-role.kubernetes.io/master: ""
```

If a `VirtualMachineInstance` gets scheduled, the scheduler is only considering nodes where `kubevirt.io/schedulable` is `true`. This can be seen when looking on the corresponding pod of a `VirtualMachineInstance`:

```
$ kubectl get pods  virt-launcher-vmi-nocloud-ct6mr -o yaml
apiVersion: v1
kind: Pod
metadata:
  [...]
spec:
  [...]
  nodeName: node01
  nodeSelector:
    kubevirt.io/schedulable: "true"
  [...]
```

In case there is a communication issue or the host goes down, `virt-handler` can't update its labels and annotations any-more. Once the last `kubevirt.io/heartbeat` timestamp is older than five minutes, the KubeVirt node-controller kicks in and sets the `kubevirt.io/schedulable` label to `false`. As a consequence no more VMIs will be schedule to this node until virt-handler is connected again.

## Deleting stuck VMIs when virt-handler is unresponsive

In cases where `virt-handler` has some issues but the node is in general fine, a `VirtualMachineInstance` can be deleted as usual via `kubectl delete vmi <myvm>`. Pods of a `VirtualMachineInstance` will be told by the cluster-controllers they should shut down. As soon as the Pod is gone, the `VirtualMachineInstance` will be moved to `Failed` state, if `virt-handler` did not manage to update it's heartbeat in the meantime. If `virt-handler` could recover in the meantime, `virt-handler` will move the `VirtualMachineInstance` to failed state instead of the cluster-controllers.

## Deleting stuck VMIs when the whole node is unresponsive

If the whole node is unresponsive, deleting a `VirtualMachineInstance` via `kubectl delete vmi <myvmi>` alone will never remove the `VirtualMachineInstance`. In this case all pods on the unresponsive node need to be force-deleted: First make sure that the node is really dead. Then delete all pods on the node via a force-delete: `kubectl delete pod --force --grace-period=0 <mypod>`.

As soon as the pod disappears and the heartbeat from virt-handler timed out, the VMIs will be moved to `Failed` state. If they were already marked for deletion they will simply disappear. If not, they can be deleted and will disappear almost immediately.

## Timing considerations

It takes up to five minutes until the KubeVirt cluster components can detect that virt-handler is unhealthy. During that time-frame it is possible that new VMIs are scheduled to the affected node. If virt-handler is not capable of connecting to these pods on the node, the pods will sooner or later go to failed state. As soon as the cluster finally detects the issue, the VMIs will be set to failed by the cluster.