Openshift 4.9 | Container does not exist

Modified on Fri, 27 Jan, 2023 at 10:33 AM

OpenShift uses the CRI-O container runtime. OpenShift 4.9 uses CRI-O 1.22. For containerd-runc and  crio-runc container drivers, Gremlin relies on the presence of a “sandbox” container to resolve container namespaces (e.g. the network namespace shared by all containers of a pod) .


  • In CRI-O 1.22, this sandbox container (referred to as “infra” container in CRI-O documentation) is dropped after the pod is created. This is controlled by an option: drop_infra_ctr  (which is true by default)



Problem


The absence of a sandbox container prevents Gremlin from running attacks. They fail like this:


1container details : time="2022-05-11T13:07:21Z" level=error msg="container \"2584cede1cf01e77d9d9ac8f864f99f1c155268ec1095af2bbde850e73d936a2\" does not exist"


How To Reproduce


  1. install a cluster with OpenShift 4.9
  2. find a Kubernetes object not running on a master node
  3. launch the gremlin attack against object in #2


Expected: attack runs successfully


Actual: attack fails with message from above


Workaround


The problem occurs when drop_infra_ctr = true defined in the CRI-O run config. Changing this to false gets attacks working again. The attached file is an example of a machine config that will set drop_infra_ctr = false


To apply this workaround, install the machine config attached to this article: 


1oc apply -f 95-gremlin-drop_infra_ctr.yaml


Wait for the apply to go through by monitoring the machine config pool


1oc get machineconfigpools -w
2NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
3master rendered-master-4da59b029c1dc49757c63426cee6afe2 True False False 3 3 3 0 13h
4worker rendered-worker-18cfed020d41141d6b6056c61b130685 True False False 3 3 3 0 13h

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select at least one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article