Gremlin issues with large numbers of objects/namespaces

Modified on Wed, 10 Jul, 2024 at 7:11 PM

If a given cluster is intermittently disappearing from the Gremlin UI despite verified connectivity to the Gremlin API and there are logs in the chao Kubernetes pod indicating a 504 Gateway Timeout, it's possible that a very large update is timing out before successfully completing. This will typically occur in cases where there are very large numbers of namespaces/objects being interacted with.

At present, the workaround is to specify namespaces in the PodSpec (for most installations, this is in the corresponding Helm chart) for the chao pod. For example, this snippet would limit gremlin to looking at the specific namespaces 'default' and 'gremlin':

spec:
        containers:
        - args:
          - -api_url
          - https://api.gremlin.com/v1/kubernetes
          - -cert_path
          - /var/lib/gremlin/cert/gremlin.cert
          - -key_path
          - /var/lib/gremlin/cert/gremlin.key
          - -namespaces
          - gremlin,default

A full example:

apiVersion: apps/v1
  kind: Deployment
  metadata:
    annotations:
      deployment.kubernetes.io/revision: "28"
      meta.helm.sh/release-name: gremlin
      meta.helm.sh/release-namespace: gremlin
    labels:
      app: chao
      app.kubernetes.io/instance: chao
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: chao
      app.kubernetes.io/version: "1.1"
      version: v1
    name: chao
    namespace: gremlin
  spec:
    replicas: 1
    selector:
      matchLabels:
        app.kubernetes.io/instance: chao
        app.kubernetes.io/name: chao
        app.kubernetes.io/version: "1"
    strategy:
      rollingUpdate:
        maxSurge: 25%
        maxUnavailable: 25%
      type: RollingUpdate
    template:
      metadata:
        labels:
          app: chao
          app.kubernetes.io/instance: chao
          app.kubernetes.io/name: chao
          app.kubernetes.io/version: "1"
          version: v1
      spec:
        containers:
        - args:
          - -api_url
          - https://api.gremlin.com/v1/kubernetes
          - -cert_path
          - /var/lib/gremlin/cert/gremlin.cert
          - -key_path
          - /var/lib/gremlin/cert/gremlin.key
          - -namespaces
          - gremlin,default
          env:
          - name: GREMLIN_TEAM_ID
            valueFrom:
              secretKeyRef:
                key: GREMLIN_TEAM_ID
                name: gremlin-secret
          - name: GREMLIN_CLUSTER_ID
            valueFrom:
              secretKeyRef:
                key: GREMLIN_CLUSTER_ID
                name: gremlin-secret
          image: gremlin/chao:dev-sam-18
          imagePullPolicy: Never
          name: chao
          volumeMounts:
          - mountPath: /var/lib/gremlin/cert
            name: gremlin-cert
            readOnly: true
        dnsPolicy: ClusterFirst
        restartPolicy: Always
        serviceAccount: chao
        serviceAccountName: chao
        terminationGracePeriodSeconds: 30
        volumes:
        - name: gremlin-cert
          secret:
            defaultMode: 420
            secretName: gremlin-secret