Containers missing due to unknown runtime socket file location

Background

Chao successfully registers with the control plane, but when running a kubernetes attack, it doesn't recognize any containers on your system. This means that the kubernetes attack ran will result in "error, could not resolve to any containers".

Steps to remediate

The first thing to always check is the gremlin daemon.log file. You're trying to see if there are any errors in the logs. If there aren't any specific errors that can be addressed, it's likely that our Gremlin Agent can't find the socket file for your container runtime. 

2021-11-17 01:26:52 - container driver: containerd-runc not suitable for this system: socket not available: /run/containerd/containerd.sock: No such file or directory (os error 2)

In this scenario, since we noticed that the Gremlin agent is using containerd-runc we wanted to find the correct socket file so we ran 

find / -name runc

You must also set the container driver parameter, gremlin.container.driver=containerd-runc, in order for the runtimeSocket parameter to be applied.

helm upgrade gremlin gremlin/gremlin --namespace gremlin --set gremlin.hostPID=true --set gremlin.container.driver=containerd-runc --set containerDrivers.containerd-runc.runtimeSocket=/run/k3s/containerd/containerd.sock --set gremlin.secret.managed=true --set gremlin.secret.type=secret --set gremlin.secret.teamID=******** --set gremlin.secret.teamSecret=*********** --set gremlin.secret.clusterID=*****

You can see a list of the default socket file locations that the Gremlin Agent tries to use here. For containerd-runc, the default location of the socket file is "/run/containerd/containerd.sock", but in some orchestrated environments (RKE2 for this example) the socket file is located somewhere else like "/run/k3s/containerd/containerd.sock". In order to configure the socket file location specific to your orchestrated environment you can set the value in the helm chart with this command: 

--set containerDrivers.{containerDriver-name}-runc.runtimeSocket=/path/to/file.sock

NOTE: we suffix the containerDriver name with "runc" in our helm charts!

here's a good example for RKE2:

--set containerDrivers.containerd-runc.runtimeSocket=/run/k3s/containerd/containerd.sock

If you're not using helm you can update your client.yaml file by adding a value that looks like this:

- name: containerd-sock
          hostPath:
            path:  /run/containerd/containerd.sock