If a given cluster is intermittently disappearing from the Gremlin UI despite verified connectivity to the Gremlin API and there are logs in the chao Kubernetes pod indicating a 504 Gateway Timeout, it's possible that a very large update is timing out before successfully completing. This will typically occur in cases where there are very large numbers of namespaces/objects being interacted with.
At present, the workaround is to specify namespaces in the PodSpec (for most installations, this is in the corresponding Helm chart) for the chao pod. For example, this snippet would limit gremlin to looking at the specific namespaces 'default' and 'gremlin':
spec: containers: - args: - -api_url - https://api.gremlin.com/v1/kubernetes - -cert_path - /var/lib/gremlin/cert/gremlin.cert - -key_path - /var/lib/gremlin/cert/gremlin.key - -namespaces - gremlin,default
A full example:
apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "28" meta.helm.sh/release-name: gremlin meta.helm.sh/release-namespace: gremlin labels: app: chao app.kubernetes.io/instance: chao app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: chao app.kubernetes.io/version: "1.1" version: v1 name: chao namespace: gremlin spec: replicas: 1 selector: matchLabels: app.kubernetes.io/instance: chao app.kubernetes.io/name: chao app.kubernetes.io/version: "1" strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: chao app.kubernetes.io/instance: chao app.kubernetes.io/name: chao app.kubernetes.io/version: "1" version: v1 spec: containers: - args: - -api_url - https://api.gremlin.com/v1/kubernetes - -cert_path - /var/lib/gremlin/cert/gremlin.cert - -key_path - /var/lib/gremlin/cert/gremlin.key - -namespaces - gremlin,default env: - name: GREMLIN_TEAM_ID valueFrom: secretKeyRef: key: GREMLIN_TEAM_ID name: gremlin-secret - name: GREMLIN_CLUSTER_ID valueFrom: secretKeyRef: key: GREMLIN_CLUSTER_ID name: gremlin-secret image: gremlin/chao:dev-sam-18 imagePullPolicy: Never name: chao volumeMounts: - mountPath: /var/lib/gremlin/cert name: gremlin-cert readOnly: true dnsPolicy: ClusterFirst restartPolicy: Always serviceAccount: chao serviceAccountName: chao terminationGracePeriodSeconds: 30 volumes: - name: gremlin-cert secret: defaultMode: 420 secretName: gremlin-secret
Was this article helpful?
That’s Great!
Thank you for your feedback
Sorry! We couldn't be helpful
Thank you for your feedback
Feedback sent
We appreciate your effort and will try to fix the article