GCP Debugging Scenarios

Ten failure patterns you'll actually hit running workloads on GCP — the symptom, a diagnostic flowchart, the commands that confirm the cause, and the prevention that stops it recurring. The last three focus on the database layer: Cloud SQL connection exhaustion from serverless compute, the DNS-caching gotcha that makes a completed failover look broken, and Bigtable row-key hotspotting.

1. GKE Pod Can't Access Cloud Storage / BigQuery

Symptom: Pod gets 403 Permission Denied calling GCP APIs.

flowchart TD
    classDef err fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef decision fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef verify fill:#3498db,stroke:#2471a3,color:#fff

    ERR["403 calling GCP API from pod"]:::err --> WI
    WI{"Workload Identity<br/>configured on the cluster?"}:::decision -->|No| SETUP
    WI -->|Yes| CHECK

    subgraph SETUP_GROUP["Set up Workload Identity"]
        SETUP["1. Create a GCP service account<br/>2. Bind K8s SA to GCP SA<br/>(roles/iam.workloadIdentityUser)<br/>3. Annotate the K8s SA with<br/>the GCP SA's email<br/>4. Grant the GCP SA the<br/>IAM role it actually needs"]:::fix
    end

    CHECK["Check annotation on K8s SA<br/>kubectl describe sa my-sa"]:::verify --> ROLE
    ROLE{"GCP SA has<br/>the right IAM role?"}:::decision -->|No| GRANT["Grant the role:<br/>gcloud iam bindings add<br/>--role roles/storage.objectViewer"]:::fix
    ROLE -->|Yes| TOKEN["Verify from inside the pod:<br/>curl the metadata server for<br/>/computeMetadata/v1/instance/<br/>service-accounts/default/email"]:::verify

# Step 1: Verify Workload Identity is enabled on cluster
gcloud container clusters describe my-cluster --region us-central1 \
  --format="value(workloadIdentityConfig)"

# Step 2: Check K8s SA annotation
kubectl describe serviceaccount my-app -n my-namespace
# Annotations: iam.gke.io/gcp-service-account=my-app@project.iam.gserviceaccount.com

# Step 3: Check IAM binding
gcloud iam service-accounts get-iam-policy my-app@project.iam.gserviceaccount.com
# Should show: serviceAccount:project.svc.id.goog[namespace/ksa-name] with workloadIdentityUser

# Step 4: Test from inside pod
kubectl exec -it my-pod -- curl -H "Metadata-Flavor: Google" \
  "http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email"
# Should return: my-app@project.iam.gserviceaccount.com

# Prevention: always use Workload Identity — never mount service account JSON keys

2. GKE Node Pool Scaling Not Working

Symptom: Pods stuck Pending despite Cluster Autoscaler configured, nodes not adding.

flowchart TD
    classDef err fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef decision fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff

    PEND["Pod stuck Pending"]:::err --> LOGS["kubectl -n kube-system logs<br/>-l component=cluster-autoscaler"]
    LOGS --> MSG{"What does the<br/>autoscaler log say?"}:::decision
    MSG -->|"Scale-up blocked<br/>by group minimum"| MIN["min-nodes == current node count<br/>raise the node pool's min-nodes"]:::fix
    MSG -->|"Node pool has<br/>reached max size"| MAX["max-nodes too low for demand<br/>raise the node pool's max-nodes"]:::fix
    MSG -->|"No pending pods"| NOREQ["Pod has no resource requests —<br/>autoscaler can't size a node for it<br/>set requests on every pod"]:::fix

# Check autoscaler logs
kubectl -n kube-system logs -l component=cluster-autoscaler --tail=50

# Common messages:
# "Scale-up blocked by group minimum"  → min nodes = current count
# "Node pool has reached max size"      → increase max-nodes
# "No pending pods"                     → pods have tolerations but no requests

# Check node pool limits
gcloud container node-pools describe default-pool \
  --cluster my-cluster --region us-central1 \
  --format="yaml(autoscaling)"

# Check if autoscaler is enabled
gcloud container clusters describe my-cluster --region us-central1 \
  --format="value(autoscaling.enableNodeAutoprovisioning)"

# Prevention: set explicit resource requests on ALL pods
# HPA/CA both require resource requests to function

3. BigQuery Query Costs Unexpectedly High

Symptom: Daily BigQuery bill much higher than expected.

-- Find expensive queries in last 24 hours
SELECT
  job_id,
  user_email,
  query,
  total_bytes_processed / 1e12 AS tb_scanned,
  (total_bytes_processed / 1e12) * 5 AS cost_usd,
  creation_time
FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE creation_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
  AND job_type = 'QUERY'
  AND statement_type != 'SCRIPT'
ORDER BY total_bytes_processed DESC
LIMIT 20;

-- Find tables without partitioning (most common cause)
SELECT table_name, row_count, size_bytes/1e9 AS size_gb
FROM `project.dataset`.INFORMATION_SCHEMA.TABLE_STORAGE
WHERE total_partitions = 0 AND size_bytes > 1e10  -- >10GB unpartitioned
ORDER BY size_bytes DESC;

# Fix: require partition filters on large tables
bq update --require_partition_filter project:dataset.orders

# Set billing cap per query (prevents runaway queries)
# In BigQuery console: Project → Edit → Maximum bytes billed
bq query --maximum_bytes_billed=10000000000 \   # 10GB max
  'SELECT ...'

# Prevention:
# 1. Partition all large tables by date
# 2. Set require_partition_filter=true
# 3. Grant BigQuery Job User (not Data Owner) to analysts

4. Cloud Run Service Cold Start Latency

Symptom: First request to Cloud Run takes 10+ seconds.

flowchart TD
    classDef cold fill:#e67e22,stroke:#ba6018,color:#fff
    classDef warm fill:#27ae60,stroke:#1e8449,color:#fff
    classDef req fill:#3498db,stroke:#2471a3,color:#fff

    REQ["First request arrives<br/>(no warm instance available)"]:::req --> ALLOC

    subgraph COLD["Cold start path — this is the 10+ seconds"]
        ALLOC["GCP allocates a new<br/>container instance"]:::cold --> PULL["Pull the container image"]:::cold
        PULL --> START["Start the container process"]:::cold
        START --> INIT["App initialization<br/>(connect DB, load config,<br/>warm caches)"]:::cold
    end

    INIT --> HANDLE["Handle the request"]:::req
    HANDLE --> WARM["Container stays warm —<br/>subsequent requests skip<br/>straight to Handle"]:::warm

# Check cold start frequency
gcloud logging read \
  'resource.type="cloud_run_revision" AND textPayload:"Cold start"' \
  --limit 50

# Mitigations:
# 1. Minimum instances (keep N instances warm, costs money)
gcloud run services update my-service \
  --min-instances 1 \
  --region us-central1

# 2. Reduce image size (faster pull)
# Use distroless or scratch base images

# 3. Optimize startup (lazy initialization — connect DB on first request, not at startup)

# 4. Use CPU boost (GCP gives extra CPU during startup)
gcloud run services update my-service \
  --cpu-boost

# 5. Use startup probe correctly — Cloud Run doesn't use K8s probes
# but Cloud Run waits for the container to listen on $PORT before routing

5. Spanner High Latency or Hotspot

Symptom: Spanner p99 latency spikes, or one node has much higher CPU than others.

# Check for hotspots using Key Visualizer
# GCP Console → Spanner → Instance → Key Visualizer
# Bright spots indicate hot row ranges

flowchart TD
    classDef hot fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef cool fill:#27ae60,stroke:#1e8449,color:#fff
    classDef writer fill:#3498db,stroke:#2471a3,color:#fff

    W1["New rows —<br/>sequential UUID or<br/>timestamp leading key"]:::writer -->|"100% of writes"| HOT["Split A — hot<br/>single node absorbs every insert"]:::hot
    HOT -.->|idle| HOT2["Split B"]:::cool
    HOT -.->|idle| HOT3["Split C"]:::cool

    W2["New rows —<br/>randomized UUID key"]:::writer -->|"~33% of writes"| R1["Split A"]:::cool
    W2 -->|"~33% of writes"| R2["Split B"]:::cool
    W2 -->|"~33% of writes"| R3["Split C"]:::cool

Common causes of hotspots in Spanner:

# Fix: use UUIDs generated randomly (not sequentially)
# Or use bit-reversed sequences:
# Spanner auto-shards on boundary values — random UUIDs distribute naturally

# Check Spanner metrics
gcloud monitoring read \
  'metric.type="spanner.googleapis.com/instance/cpu/utilization_by_priority"' \
  --start="2024-01-15T00:00:00Z" --end="2024-01-15T01:00:00Z"

# Prevention: design schema to avoid hotspots
# Use INTERLEAVE for parent-child relationships (not FK joins)

6. Pub/Sub Messages Piling Up (High Backlog)

Symptom: subscription/num_undelivered_messages metric growing, consumers not keeping up.

flowchart TD
    classDef err fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef decision fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff

    BACKLOG["num_undelivered_messages<br/>climbing"]:::err --> NACK{"High nack /<br/>redelivery rate in logs?"}:::decision
    NACK -->|Yes| ERRORS["Consumers are erroring —<br/>fix the processing bug;<br/>let the dead-letter topic<br/>catch true poison messages"]:::fix
    NACK -->|No| THROUGHPUT{"Consumers keeping up<br/>with the publish rate?"}:::decision
    THROUGHPUT -->|"No — too few workers"| SCALE["Scale consumers horizontally<br/>kubectl scale deployment --replicas=N"]:::fix
    THROUGHPUT -->|"No — ack deadline<br/>too short for processing time"| DEADLINE["Extend the ack deadline<br/>modify-push-config --ack-deadline"]:::fix

# Check subscription backlog
gcloud pubsub subscriptions describe my-subscription \
  --format="value(numUndeliveredMessages,oldestUnackedMessage)"

# Check if messages are being nacked (errors)
# High nack rate = consumer processing errors
gcloud logging read \
  'resource.type="pubsub_subscription" AND labels.subscription_id="my-subscription"' \
  --limit 20

# Scale up consumers
kubectl scale deployment my-consumer --replicas=10

# Tune subscription settings:
gcloud pubsub subscriptions modify-push-config my-subscription \
  --ack-deadline=60  # give consumers more time (default 10s)

# Dead letter topic: failed messages after N retries go here
gcloud pubsub subscriptions modify-dead-letter-policy my-subscription \
  --dead-letter-topic=my-dlq \
  --max-delivery-attempts=5

# Prevention:
# 1. Use push subscriptions → Pub/Sub pushes to Cloud Run (auto-scales)
# 2. Use BigQuery subscriptions → messages written directly to BQ table
# 3. Set appropriate ack deadline (longer than max processing time)

7. GCS Bucket Access Denied from External

Symptom: gsutil or SDK call returns 403 from outside GCP.

# Check bucket IAM
gcloud storage buckets get-iam-policy gs://my-bucket

# Check if object is public
gsutil acl get gs://my-bucket/my-file.txt

# Grant specific access
gcloud storage buckets add-iam-policy-binding gs://my-bucket \
  --member="serviceAccount:my-app@project.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"

# Generate signed URL for temporary public access (no IAM needed for requester)
gsutil signurl -d 1h -m GET my-service-account-key.json gs://my-bucket/file.txt

# Check if uniform bucket-level access is enabled (disables ACLs)
gcloud storage buckets describe gs://my-bucket \
  --format="value(iamConfiguration.uniformBucketLevelAccess)"
# If true: can't use object ACLs, only bucket-level IAM

8. Cloud SQL Connection Exhaustion from Cloud Run/Cloud Functions

Symptom: FATAL: sorry, too many clients already (Postgres) or Too many connections (MySQL), getting sharply worse right after a traffic spike — the same failure mode as Lambda exhausting RDS, just with Cloud Run/Cloud Functions as the culprit instead.

Cloud SQL's max_connections default isn't a flat number — it scales with the instance's memory, the same way RDS's per-instance-class cap does:

flowchart TD
    classDef math fill:#3498db,stroke:#2471a3,color:#fff
    classDef err fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef decision fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff

    SPIKE["Traffic spike —<br/>Cloud Run scales out"]:::math --> INST["Scales to max-instances<br/>(default: 100 per service)"]:::math
    INST --> POOL["Each instance opens its own<br/>DB client pool on cold start<br/>(e.g. pool max: 10)"]:::math
    POOL --> TOTAL["100 instances x 10 =<br/>1,000 connections demanded"]:::err
    TOTAL --> CAP{"Compare against<br/>Cloud SQL's real ceilings"}:::decision
    CAP -->|"db-custom-2-8192 (8GB)<br/>max_connections default"| OVER1["1,000 > 400 —<br/>too many clients already"]:::err
    CAP -->|"Cloud Run's own built-in<br/>Cloud SQL connector cap"| OVER2["1,000 > 100 connections/db —<br/>connector refuses first"]:::err
    OVER1 --> FIX
    OVER2 --> FIX["Pool centrally, not per-instance:<br/>Auth Proxy / Managed Connection Pooling,<br/>shrink per-instance pool size,<br/>or cap --max-instances"]:::fix

# Confirm connection count is climbing alongside the errors (Postgres)
gcloud monitoring time-series list \
  --filter='metric.type="cloudsql.googleapis.com/database/postgresql/num_backends"' \
  --format="table(points[].value.int64Value, points[].interval.endTime)"

# Rule out an unrelated maintenance/failover operation as the real cause
gcloud sql operations list --instance=my-instance --limit=10

# Check whether max_connections was ever overridden from the memory-based default
gcloud sql instances describe my-instance --format="value(settings.databaseFlags)"

# Confirm the effective value from inside the database
psql -h <ip> -U app -d appdb -c "SHOW max_connections;"
# db-custom-2-8192 (8GB RAM), no override --> defaults to 400

# MySQL equivalent
mysql -h <ip> -u app -p -e "SHOW VARIABLES LIKE 'max_connections'; SHOW STATUS LIKE 'Threads_connected';"

# Check what Cloud Run is actually scaled to
gcloud run services describe my-service --region us-central1 \
  --format="value(status.traffic)" 

# Fix 1: route through the Cloud SQL Auth Proxy (or Cloud Run's built-in integration),
# so the app never opens raw sockets straight to the instance
gcloud run services update my-service \
  --add-cloudsql-instances=PROJECT:REGION:my-instance

# Fix 2: shrink the per-instance pool and bound total instances
gcloud run services update my-service \
  --max-instances=20 \
  --concurrency=40

# Fix 3: turn on Cloud SQL's own server-side connection pooling
# (Enterprise Plus edition — pools on the database side, so the backend
# connection count stops scaling 1:1 with client instance count)
gcloud sql instances patch my-instance --enable-connection-pooling

# Prevention:
# 1. Always route serverless compute through the Auth Proxy, never raw sockets
# 2. Keep per-instance pool size small (2-5), not framework defaults
# 3. Set --max-instances deliberately instead of leaving it at the 100 default

9. AlloyDB/Cloud SQL Failover — the App Blames "Slow Failover," but It's a Stale DNS Cache

Symptom: An AlloyDB cluster runs a cross-region failover, the Admin API shows the promotion completed quickly, but the application keeps throwing connection timeouts for several more minutes. The on-call's first assumption is that the failover itself is slow or broken.

flowchart TD
    classDef err fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef decision fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef ok fill:#3498db,stroke:#2471a3,color:#fff

    TRIGGER["Cross-region failover triggered"]:::err --> PROMOTE["Secondary promoted to primary —<br/>completes in the normal window"]:::ok
    PROMOTE --> DNS["Cluster endpoint's DNS record<br/>updated to point at the new primary"]:::ok
    DNS --> CLIENT{"How does the app<br/>reach the database?"}:::decision
    CLIENT -->|"Auth Proxy /<br/>Language Connector"| PROXYOK["Re-resolves the current primary via<br/>the Admin API on every new connection —<br/>picks up the change immediately"]:::fix
    CLIENT -->|"Direct hostname +<br/>pooled driver"| CACHED["JDBC/Go driver or OS resolver<br/>cached the OLD IP,<br/>outliving the DNS record's TTL"]:::err
    CACHED --> DEAD["App keeps opening new sockets<br/>to the now-demoted old primary"]:::err
    DEAD --> BLAME["On-call sees timeouts and<br/>blames the failover as slow —<br/>the promotion itself already finished"]:::err
    BLAME --> FIX["Fix the client, not the database:<br/>Auth Proxy / Language Connectors,<br/>or shorten the driver/JVM's DNS cache TTL"]:::fix

Does every GCP database failover have this DNS trap? No — it depends on whether the endpoint's address actually changes:

# Step 1: verify the promotion's actual timeline before blaming it
gcloud alloydb operations list --cluster=my-cluster --region=us-central1 \
  --filter="operationType=FAILOVER" \
  --format="table(name,status,startTime,endTime)"

# Step 2: confirm the current primary and its address
gcloud alloydb instances describe my-primary --cluster=my-cluster \
  --region=us-central1 --format="value(ipAddress)"

# Step 3: check what your app's live connections are actually using
# (compare against the address from Step 2 — a mismatch confirms a stale cache)
lsof -i -a -p $(pgrep -f my-app) | grep ESTABLISHED

# Fix 1: connect through the AlloyDB Auth Proxy instead of a bare hostname —
# it re-resolves the current primary via the Admin API on every new connection
alloydb-auth-proxy "projects/my-project/locations/us-central1/clusters/my-cluster/instances/my-primary"

# Fix 2: if connecting directly, stop caching DNS past the record's real TTL
# Java: networkaddress.cache.ttl=1 in java.security (the same fix used for Aurora)
# Go: build a net.Resolver with a short-lived cache instead of relying on
#     the process-wide OS resolver cache

# Fix 3 (Cloud SQL regional HA case): test connections before handing them
# out of the pool, and cap pool maxLifetime so dead sockets get recycled fast
# HikariCP: connectionTestQuery + maxLifetime
# pgx (Go):  pool.Config().MaxConnLifetime

10. Bigtable Hot Row Key — One Node Takes All the Write Traffic

Symptom: cpu_load_hottest_node sits pinned near 100% while the cluster's average CPU is comfortably under Bigtable's recommended 70% production ceiling — write latency degrades on a subset of requests while the rest of the table serves fine.

flowchart TD
    classDef hot fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef cool fill:#27ae60,stroke:#1e8449,color:#fff
    classDef writer fill:#3498db,stroke:#2471a3,color:#fff

    W1["New rows keyed by<br/>sequential ID or raw timestamp"]:::writer -->|"100% of writes"| HOT["Tablet A — hot<br/>one node absorbs every insert"]:::hot
    HOT -.->|idle| T2["Tablet B"]:::cool
    HOT -.->|idle| T3["Tablet C"]:::cool

    W2["New rows keyed by<br/>salted or field-promoted key"]:::writer -->|"~33% of writes"| R1["Tablet A"]:::cool
    W2 -->|"~33% of writes"| R2["Tablet B"]:::cool
    W2 -->|"~33% of writes"| R3["Tablet C"]:::cool

# The classic tell: hottest-node CPU pinned high while the average is fine
gcloud monitoring time-series list \
  --filter='metric.type="bigtable.googleapis.com/cluster/cpu_load_hottest_node"' \
  --format="table(points[].value.doubleValue, points[].interval.endTime)"

gcloud monitoring time-series list \
  --filter='metric.type="bigtable.googleapis.com/cluster/cpu_load"' \
  --format="table(points[].value.doubleValue, points[].interval.endTime)"

# Open Key Visualizer to see exactly which row range is hot
# GCP Console -> Bigtable -> Instance -> Table -> Key Visualizer
# A bright stripe pinned to the high end of the key range = sequential writes

# Inspect the actual row keys hitting that range
cbt -instance=my-instance read my-table count=5
# Keys like "00001842910", "00001842911", ... confirm sequential/timestamp keys

# Prevention: never use a monotonically increasing (or decreasing) row key,
# whether it's an auto-increment ID or a raw timestamp prefix

Quick Reference: GCP Debug Commands