Why am I seeing this message?
The Linux operating system has an Out of Memory Manager (sometimes referred to as the OOMKiller) that kills processes when it detects that the system is out of memory. When a process is reaped by the OOMKiller it receives a SIGKILL signal.
In Kubernetes, the OOMKiller terminates a process in a Pod when it violates its memory resource limit. When Gremlin’s target is killed by the OOMKiller, the Gremlin attack process is also terminated.
What hypothesis does this support?
The OOMKiller is a resilience mechanism. When the OOMKiller kills a target as the result of a Gremlin attack, this outcome confirms the hypothesis: “I expect my environment to prevent a runaway process from consuming too much memory by restarting it safely”.
Now what?
Killing the process as a result of a resource constraint is only one part of this resilience mechanism. You should make sure your application has successfully started back up.
On Kubernetes you can verify this by examining the status of the Pod itself to make sure it's in a good state post attack. Many times when Kubernetes takes a Pod offline it'll either restart it or replace it with another Pod, so our primary concern is that we're back in a healthy state.
% kubectl get pods -n $NAMESPACE $TARGET_POD
NAME READY STATUS RESTARTS AGE
examplePod-759cbc656d-85thp 3/3 Running 4 20h
Was this article helpful?
That’s Great!
Thank you for your feedback
Sorry! We couldn't be helpful
Thank you for your feedback
Feedback sent
We appreciate your effort and will try to fix the article