Troubleshooting
Paused Upgrade due to affined and unmanaged workloads
If a workload is not managed by a standard Kubernetes controller, the InfoScale operator will pause the upgrade and record the following events:
Warning UpdatePaused 23m (x89 over 41m) InfoScaleCluster
ErrorCode=10050
ErrorMsg=Resource is not managed by controller : pod test/redis
not managed statefulset,
replicaset, daemonset on <hostname>
Warning UpdatePaused 14m (x4 over 15m) InfoScaleCluster to
resume upgrade
with forced migration of such workloads, annotate InfoScaleCluster
with infoscale.veritas.com/
forceMigrate=true
If the InfoScale operator pauses due to unmanaged workloads, the administrator must handle them. This includes single pods or pods managed by other operators that do not automatically move to a different node.
-
Back up and delete these resources if they can be safely removed.
-
Annotate the InfoScaleCluster to force the upgrade if it is safe for these resources to be down. You can find the specific annotation in the event logs mentioned earlier.
For example:
oc annotate infoscaleclusters <cluster-name> infoscale.veritas.com/
forceMigrate=true
infoscalecluster.infoscale.veritas.com/<cluster-name> annotated
After the upgrade finishes using the force option, the administrator can remove the annotation from the InfoScaleCluster resource.
NFD crash in OCP upgrades from 4.19 to 4.20
The OCP upgrade was blocked because the NFD garbage collector pods were crashing. This was caused by a readiness failure, as shown in the logs below.
NAME READY STATUS RESTARTS AGE
nfd-controller-manager-658cccddf9-w9l54 1/1 Running 0 2d16h
nfd-gc-6b7549bfc4-llv5w 0/1 Running 1 (35s ago) 75s <<<
continesouly restarting
nfd-gc-fbbd48975-g2bw5 1/1 Running 0 2d16h
nfd-master-5bb757c8d4-hmdm6 0/1 Running 777 (2m33s ago) 2d16h
nfd-master-7c585f4bf-tswdp 1/1 Running 0 2d16h
nfd-worker-629wr 1/1 Running 2 (2d16h ago) 2d16h
nfd-worker-86hd9 1/1 Running 2 (2d20h ago) 2d21h
. . .
58s Normal Pulled pod/nfd-gc-6b7549bfc4-llv5w
Successfully pulled image "registry.redhat.io/openshift4/ose-node-feature-discovery-
rhel9@sha256:
5459499aedd2ebd0245d79d6dfed353c5a24c12d54bc97cbc2cecfe2489b792e" in 3.513s
(3.513s including waiting). Image size: 617600938 bytes.
32s Warning Unhealthy pod/nfd-gc-6b7549bfc4-llv5w
Liveness probe failed: Get "http://10.128.3.19:8080/healthz":
dial tcp 10.128.3.19:8080: connect: connection refused
27s Warning Unhealthy pod/nfd-gc-6b7549bfc4-llv5w
Readiness probe failed: Get "http://10.128.3.19:8080/healthz":
dial tcp 10.128.3.19:8080: connect: connection refused
27s Warning ProbeError pod/nfd-gc-6b7549bfc4-llv5w
Readiness probe error: Get "http://10.128.3.19:8080/healthz":
dial tcp <IP address>:8080: connect: connection refused...
22s Warning ProbeError pod/nfd-gc-6b7549bfc4-llv5w
Liveness probe error: Get "http://10.128.3.19:8080/healthz":
dial tcp <IP address>:8080: connect: connection refused...
22s Normal Killing pod/nfd-gc-6b7549bfc4-llv5w
Container nfd-gc failed liveness probe, will be restarted
22s Normal Pulling pod/nfd-gc-6b7549bfc4-llv5w
Pulling image "registry.redhat.io/openshift4/ose-node-feature-discovery-
rhel9@sha256:
5459499aedd2ebd0245d79d6dfed353c5a24c12d54bc97cbc2cecfe2489b792e"
18s Normal Pulled pod/nfd-gc-6b7549bfc4-llv5w
Successfully pulled image "registry.redhat.io/openshift4/ose-node-feature-discovery-
rhel9@sha256:
5459499aedd2ebd0245d79d6dfed353c5a24c12d54bc97cbc2cecfe2489b792e" in 3.457s
(3.457s including waiting). Image size: 617600938 bytes.
18s Normal Started pod/nfd-gc-6b7549bfc4-llv5w
Started container nfd-gc
18s Normal Created pod/nfd-gc-6b7549bfc4-llv5w
Created container: nfd-gc
The upgrade continued after the corresponding NFD garbage collector pod was deleted. It appears the NFD is interacting with the OCP cluster operator and blocking the upgrade. Below are the specific versions where this issue occurred:
{
"completionTime": "2026-02-06T13:02:05Z",
"image": "quay.io/openshift-release-dev/ocp-release
@sha256:91606a5f04331ed3293f71034d4f480e38645560534805fe5a821e6b64a3f203",
"startedTime": "2026-02-06T11:38:41Z",
"state": "Completed",
"verified": true,
"version": "4.20.8"
},
{
"completionTime": "2026-02-06T06:25:59Z",
"image": "quay.io/openshift-release-dev/ocp-release
@sha256:7c2001c24aa550aa228cd2d0fc0b5d9ac6656cd4267cd7c156ec758d0687758e",
"startedTime": "2026-02-06T05:03:33Z",
"state": "Completed",
"verified": true,
"version": "4.19.21"
},
Upgrades stalled: Operators not progressing after install plan approval
Since the InfoScale and License operators share the same operator group, they can sometimes cause inconsistencies. Typically, both operators create two separate Install Plans, but only one of them will list both components as "owners." When you approve the Install Plan, choose the one that shows both operator subscriptions in the preview section.
Transient node drain failures in OCP-V upgrades
You may see the following harmless error messages or alerts while upgrading the OCP-V platform.
error when evicting pods/"virt-launcher-<VMNAME>" -n "<NAMESPACE>"
(will retry after 5s):
Related details can be found in this Red Hat knowledge base.
Related articles state that these messages can be safely ignored. These pod disruption budgets allow more time for Virtual Machine resources to migrate during the upgrade.
Resource busy or Node busy error during InfoScale upgrade
Infoscale SDS upgrade can get stuck with a "Resource Busy" error if there are stale or incorrect snapshot records and tags inside the InfoScale SDS pods. When this happens, the SDS operator will repeatedly report a "node busy" status or show that a volume sync is still in progress.
If pre-flight CLI reports such snapshots delete those using below stale snapshot cleanup script, this script is part of infoscale-tools-v9.2.0.tar
Usage:
./snapshot_cleanup.sh -n \<namespace\> -p \<pod_name\> [-g \<diskgroup\>] [-d]
Options:
-n \<namespace\>: The OpenShift namespace where the pod is located (required).
-p \<pod_name\>: The name of the InfoScale SDS pod (required).
Usage:
./snapshot_cleanup.sh -n \<namespace\> -p \<pod_name\> [-g \<diskgroup\>] [-d]
Options:
-n \<namespace\>: The OpenShift namespace where the pod is located (required).
-p \<pod_name\>: The name of the InfoScale SDS pod (required).
-g \<diskgroup\>: Specific disk group to process (optional). If not specified, all disk groups will be processed.
-d: Dry-run mode. Displays the commands that would be executed without performing any actual deletions (optional).
Example:
./snapshot_cleanup.sh -n infoscale-vtas -p infoscale-sds-21432-7820e9290fa0fc26-b9phd -g vrts_kube_dg-1121
-g <diskgroup>: Specific disk group to process (optional). If not specified, all disk groups will be processed.
-d: Dry-run mode. Displays the commands that would be executed without performing any actual deletions (optional).
Example:
./snapshot_cleanup.sh -n infoscale-vtas -p infoscale-sds-21432-7820e9290fa0fc26-b9phd -g vrts_kube_dg-1121
Failed VM migrations due to timeouts - InfoScale upgrade is in a paused state
You may see these issues during an InfoScale cluster upgrade if:
-
The cluster is overloaded, or there is not enough bandwidth set for VMs to migrate.
-
A migration policy is overriding the default settings and limiting how quickly VMs can move to other nodes.
-
There are too many parallel writes, causing "dirty pages" that cannot be cleared fast enough.
-
Migrations are simply taking too long to finish.
-
In any of these cases, the InfoScale operator will wait for you to take action. You must fix the VirtualMachineMigration resource that is in an error state. Check if it has timed out, as shown in the example below.
\# oc describe vmim infoscale-fio-vm-1-114880f98092cdab -n prod
Name: infoscale-fio-vm-1-114880f98092cdab
Namespace: prod
Labels: kubevirt.io/vmi-name=fio-vm-1
Annotations: kubevirt.io/latest-observed-api-version: v1
kubevirt.io/storage-observed-api-version: v1
API Version: kubevirt.io/v1
Kind: VirtualMachineInstanceMigration
Metadata:
Creation Timestamp: 2026-03-12T13:17:59Z
Generation: 1
Resource Version: 4628614
UID: 6b39e1cc-2e8f-4d44-a644-e559319185d4
Spec:
Vmi Name: fio-vm-1
Status:
Migration State:
Abort Status: Succeeded
Completed: true
End Timestamp: 2026-03-12T13:18:15Z
Failed: true
Failure Reason: Live migration is not completed after
\#Num seconds and has been aborted <<<
Migration Configuration:
Allow Auto Converge: true
Allow Post Copy: false
. . .
\# oc describe infoscaleclusters -n infoscale-vtas kubeburner | grep Pause
Warning UpdatePaused 164m (x7 over 168m) InfoScaleCluster
migration instance infoscale-fio-vm-8-c74edd9fa2a14506 is failed in prod,
upgrade is waiting for its completion; user intervention required
Warning UpdatePaused 164m (x7 over 168m) InfoScaleCluster
migration instance infoscale-fio-vm-5-24c5edc43025ba88 is failed in prod,
upgrade is waiting for its completion; user intervention required
. . .To fix these timeouts, you can simply delete the specific VirtualMachineMigration resource. The operator will automatically try the migration again and continue with the upgrade.To prevent this from happening again, we suggest making the following configuration changes:-
Set allowAutoConverge: true in the KubeVirt resource if it is currently set to false.
-
If you see memory violation alerts in the web console showing that system memory usage is over the limit, you can increase the system-slice memory on your nodes. Note that this change will require a node rollout to take effect. Refer to this Red Hat documentation. Consult Red Hat support before making this change.
Although the previous page recommends a small amount of memory, a better general rule is to set the system reservation using this formula:systemReserved.memory = min( max(2G, 5% of totalRAM of node), 8Gi ) -
The cluster administrator should decide on this final amount based on workload surges, network bandwidth, and node management needs.
Related information