Google Chat: Pub/Sub Setup¶

The Pub/Sub transport is ideal for bots living in private networks (e.g., private GKE clusters) where Google Chat cannot reach the bot via HTTP.

Architecture¶

Google Chat publishes events to a Pub/Sub Topic, and the bot pulls from a Subscription. The bot is outbound-only — no Ingress or public Load Balancer is required.

1. GCP Infrastructure (Terraform)¶

Use the provided module at deploy/terraform/google-chat-bot to provision the required resources:

module "orrery_chat_bot" {
  source = "../../deploy/terraform/google-chat-bot"

  project_id          = "your-project-id"
  k8s_namespace       = "orrery"
  k8s_service_account = "orrery-chat-bot"
}

This creates: - The Pub/Sub Topic and Subscription (with retry policy and optional DLQ). - A Google Service Account (GSA) with subscriber permissions. - Workload Identity bindings for GKE. - Optional Vertex AI User binding for the Gemini provider.

Module variables¶

Workspace Add-ons / Domain Restricted Sharing If your Chat app is a Workspace Add-on, or your org enforces constraints/iam.allowedPolicyMemberDomains, the default chat-api-push@system.gserviceaccount.com will fail to bind. Copy the exact SA from the Chat API console → Configuration → Connection settings → "Service Account Email" and pass it as chat_publisher_email (typically service-<PROJECT_NUMBER>@gcp-sa-gsuiteaddons.iam.gserviceaccount.com).

2. Configure the Chat API¶

After applying Terraform, you must complete two manual steps in the Google Chat API -> Configuration tab:

1. App Authentication: Select "Service Account" and paste the gsa_email from Terraform output.
2. Connection Settings: Select Cloud Pub/Sub and paste the topic_id from Terraform output.

3. Deployment Configuration¶

Add the following to your .env (or Helm values):

# The subscription to pull from.
GOOGLE_CHAT_PUBSUB_SUBSCRIPTION=orrery-chat-events-sub

# The project hosting the subscription (if different from vertex/agent project).
GOOGLE_CHAT_PUBSUB_PROJECT=your-infra-project-id

# Performance tuning.
GOOGLE_CHAT_PUBSUB_MAX_MESSAGES=4
GOOGLE_CHAT_PUBSUB_HANDLER_TIMEOUT_SECONDS=600

# Health server (liveness /healthz, readiness /readyz).
# Readiness flips to 503 if the Pub/Sub streaming pull dies — kubelet
# will restart the pod. Default 8080 matches the Helm probe port.
GOOGLE_CHAT_PUBSUB_HEALTH_PORT=8080

# Idempotency guard (see "Idempotency" below).
#   memory   — process-local; single replica only.
#   postgres — shared via DATABASE_URL; required for >1 replica.
GOOGLE_CHAT_PUBSUB_IDEMPOTENCY_BACKEND=memory
GOOGLE_CHAT_PUBSUB_IDEMPOTENCY_TTL_SECONDS=3600

Idempotency (why a redelivery won't double-act)¶

Pub/Sub delivers at least once. The same event can be redelivered when an ack is lost to a network blip, a pod OOMs mid-callback, or a handler timeout nacks after side effects already landed. Because the Chat handler can invoke @destructive tools (restart / scale / rollback a deployment, increase Kafka partitions, silence Alertmanager…), an unguarded redelivery would run those tools twice.

The worker claims each event id before dispatching and drops duplicates (ack without re-executing). If the handler then fails, the claim is released so a legitimate redelivery still retries — you get at-most-once execution of side effects without losing failed work.

Set GOOGLE_CHAT_PUBSUB_IDEMPOTENCY_TTL_SECONDS to match the subscription's message_retention_duration (default 3600s) so a redelivery anywhere in the retention window is still short-circuited. The Helm chart refuses to render a worker with replicaCount > 1 (or autoscaling on) while the backend is memory.

Timeout alignment GOOGLE_CHAT_PUBSUB_HANDLER_TIMEOUT_SECONDS bounds a single turn; the subscriber's ack deadline is auto-extended while the callback runs, up to the subscription's message_retention_duration. Keep the handler timeout less than or equal to message_retention_duration (default 3600s) so a stuck turn is reclaimed before the message expires.

4. Run the Worker¶

The Pub/Sub worker runs as a separate process:

# Local (requires GOOGLE_APPLICATION_CREDENTIALS pointing to a SA key)
make run-chat MODE=pubsub

# In-cluster (enabled via Helm)
# pubsubWorker.enabled: true

5. Scaling the worker (multi-replica)¶

A single worker is a SPOF during incidents — Chat traffic spikes exactly when things are on fire, and one pod serializes those turns behind max_messages. To run more than one replica:

1. Switch both process-local stores to the shared backend so redeliveries are deduped — and pending approvals are visible — across pods:

   pubsubWorker:
     idempotency:
       backend: postgres     # reuses the platform DATABASE_URL
     confirmation:
       backend: postgres     # same DATABASE_URL — approval handshake shared
   

   The confirmation store matters as much as the dedup guard: with the in-memory backend, the pending raised by pod A is invisible to pod B, so the operator's approve reply (or the LLM retry that consumes the approval) lands on a pod that knows nothing about it and the flow silently dies. The Postgres store also survives worker restarts — an in-memory pending dies with the pod. The chart refuses to render a multi-replica worker while either backend is memory. Traffic is human-scale (a handful of ~1 ms row operations per confirmation), so there is no measurable latency cost.
2. Enable backlog-based autoscaling. CPU/memory are poor signals here (the worker is I/O-bound on LLM calls); scale on the subscription's undelivered message count instead:

   pubsubWorker:
     autoscaling:
       enabled: true
       minReplicas: 2
       maxReplicas: 6
       targetBacklogPerPod: 10   # scale out above ~10 unacked msgs/pod
   

Prerequisite: the backlog HPA reads an External metric (pubsub.googleapis.com|subscription|num_undelivered_messages), which requires the GKE Stackdriver custom-metrics adapter. Install it before enabling pubsubWorker.autoscaling.

Scaling lag. The adapter polls ~every 60s, so backlog scaling is not instantaneous. During known incident windows keep minReplicas >= 2 (or bump it on a schedule) rather than relying purely on reactive autoscaling.

Troubleshooting Pub/Sub¶

1. Silence in logs: Verify the subscription lives in the project specified by GOOGLE_CHAT_PUBSUB_PROJECT.
2. Permissions: Ensure the GSA has roles/pubsub.subscriber on the subscription and is registered as the "App identity" in the Chat console.
3. Heartbeat: With version 0.1.5+, the worker logs a heartbeat every 60s. If you don't see this, the process is likely stuck during initialization.
4. Duplicate actions: If a destructive action ran twice, the idempotency guard is likely on memory across multiple replicas (only possible if you bypassed the Helm guard). Switch GOOGLE_CHAT_PUBSUB_IDEMPOTENCY_BACKEND=postgres. A log line Duplicate event_id=… acking without re-executing confirms the guard is working.