Verifying connectivity to the Gremlin Control Plane

Modified on Mon, 12 Aug at 1:14 PM

When running Gremlin, it's common for network devices such as load balancers and firewalls between the agent and the control plane to interfere with connectivity. Even if you can curl the API and receive a 200 response, that's not a guarantee that TLS is working as intended or that requests are being correctly proxied to the control plane.

This may present as an inability for the Gremlin Control Plane to communicate with your agents, which can result in error messages similar to:

'invalid peer certificate: unknown issuer' or 'Failed to send system info to control plane' or 'Error: could not locate Gremlin agents for supplied targets. Check that Gremlin is running on the nodes where these targets are running'.

Not configuring a proxy with the Gremlin agent does not mean that one doesn't exist. For Linux based environments, first run this command and analyze the resulting network request information (for instance, does the IP address in the request match the Gremlin API):

curl -Iv https://api.gremlin.com/v1/health

then run the following command and verify that the certificate returned is actually Gremlin's:

openssl s_client -connect api.gremlin.com:443 -showcerts

If you are in a kubernetes Alpine based container environment and the openssl or curl command are not available, at the time of this writing on the default image you are able to add it via 'apk add' with the respective command name after. For example:

apk add openssl

The availability apk depends on deployment specific characteristiss such as security posture and container image.

For Linux hosts, you can install openssl with the following commands:

#RHEL 8 and newer
sudo dnf in openssl 

#RHEL 7 and older
sudo yum install openssl 

# Debian based 
sudo apt install openssl

If you are in a Windows environment, you can use the following commands in a powershell prompt to create a x509 certificate object to inspect:

$URL = 'https://api.gremlin.com/v1/health'
$Connection = [System.Net.HttpWebRequest]::Create($URL)
$Response = $Connection.GetResponse()  
$Response.Dispose()                         
$Certificate = $Connection.ServicePoint.Certificate

You can then use 'echo' commands to view traits of the Certificate object:

PS C:\Users\Administrator> echo $Certificate.GetCertHashString()       24316153290093689D3304D8B8F25895DBC52C67
                               
PS C:\Users\Administrator> echo $Certificate.GetName()                 CN=api.gremlin.com
                                                     
PS C:\Users\Administrator> echo $Certificate.GetIssuerName()           
C=US, O=Amazon, CN=Amazon RSA 2048 M03