Openshift 4.9 | Container does not exist

Created by Jamie Martin, Modified on Tue, 18 Oct 2022 at 04:49 PM by Jamie Martin

Background




  • OpenShift uses the CRI-O container runtime. OpenShift 4.9 uses CRI-O 1.22




  • For containerd-runc and crio-runc container drivers, Gremlin relies on the presence of a “sandbox” container to resolve container namespaces (e.g. the network namespace shared by all containers of a pod)




  • In CRI-O 1.22, this sandbox container (referred to as “infra” container in CRI-O documentation) is dropped after the pod is created. This is controlled by an option: drop_infra_ctr (which is true by default)




Problem




  • The absence of a sandbox container prevents Gremlin from running attacks. They fail like this:




1container details : time="2022-05-11T13:07:21Z" level=error msg="container \"2584cede1cf01e77d9d9ac8f864f99f1c155268ec1095af2bbde850e73d936a2\" does not exist"

Reproduce




  1. install a cluster with OpenShift 4.9




  2. find a Kubernetes object not running on a master node




  3. launch the gremlin attack against object in #2




expected: attack runs successfully


actual: attack fails with message from above


Workaround


The problem occurs when drop_infra_ctr = true defined in the CRI-O run config. Changing this to false gets attacks working again. The attached file is an example of a machine config that will set drop_infra_ctr = false.


To apply this workaround, install the machine config attached to this ticket:


1oc apply -f 95-gremlin-drop_infra_ctr.yaml

Wait for the apply to go through by monitoring the machine config pool


1oc get machineconfigpools -w
2NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
3master rendered-master-4da59b029c1dc49757c63426cee6afe2 True False False 3 3 3 0 13h
4worker rendered-worker-18cfed020d41141d6b6056c61b130685 True False False 3 3 3 0 13h

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select atleast one of the reasons

Feedback sent

We appreciate your effort and will try to fix the article