Kubernetes Cluster

How our GKE clusters are set up: one Autopilot cluster per environment, namespace-based tenancy, Workload Identity for credentials, and a GitOps split between Config Sync (platform config) and Cloud Deploy (application rollouts).

Cluster model

One Autopilot cluster per environment

u2i-nonprod and u2i-prod, both in europe-west1. Autopilot manages nodes for us — there are no node pools to size or patch, GKE schedules pods on right-sized nodes automatically.

Shared, not per-app

Every application runs as a namespace on the shared cluster for its environment, rather than getting a dedicated cluster. This keeps cluster count low and makes cross-cutting policy (network, RBAC, Config Sync) apply uniformly.

Private nodes, restricted control plane

Nodes have no public IPs and egress through Cloud NAT. The Kubernetes API server only accepts connections from an authorized network list.

Reaching the API server without being on the authorized network

gcloud container clusters get-credentials talks to the API server directly, so it only works from inside the authorized network (VPN, etc.). From anywhere else, use GKE's Connect Gateway instead — it proxies kubectl through a Google-managed endpoint that isn't subject to the authorized-network list: gcloud container fleet memberships get-credentials {cluster} --project {fleet-host-project}. Requires the fleet membership to already exist and your account to have the Connect Gateway IAM role; the fleet host project is separate from the cluster's own project.

Namespaces & isolation

Naming convention

{app}-dev, {app}-qa, {app}-prod for the standard lifecycle stages, and {app}-pr-{N} for ephemeral pull-request preview environments.

Namespace as the tenancy boundary

RBAC, resource quotas, and network policy are all scoped per namespace. An app team only ever gets access to its own app's namespaces, not the cluster at large.

GitOps reconciliation

Namespaces and their baseline resources (RBAC, ServiceAccounts, quotas) are declared in the u2i-tenant-infra repo and reconciled onto the cluster by Config Sync — nobody kubectl-applies namespace scaffolding by hand.

Identity: Workload Identity

No service account keys on the cluster

Every pod that needs GCP access runs under a Kubernetes ServiceAccount bound to a GCP service account via Workload Identity Federation. The pod authenticates as that GSA with no key material stored anywhere.

Per-app GSAs

Application workloads use a GSA scoped to that app (e.g. permissions to read its own Secret Manager secrets or write to its own GCS bucket) rather than a broad, shared identity.

GitOps delivery: Config Sync & Cloud Deploy

Fleet + Config Sync

Both clusters are Fleet members. Config Sync continuously reconciles cluster-wide and per-tenant configuration (namespaces, RBAC, quotas) from an OCI artifact published out of the u2i-tenant-infra repo.

Cloud Deploy renders & promotes; Config Sync applies

This tenant uses the config-sync delivery model (deployment_type: config-sync), so Cloud Deploy does NOT apply manifests to the cluster directly. Its rollout renders the app's Helm chart via Skaffold and publishes the result as a versioned OCI artifact to the config-sync registry; the per-namespace RepoSync then pulls that artifact and reconciles the app's own Deployment, Service and HTTPRoute onto the cluster. So even app workload objects carry app.kubernetes.io/managed-by=configmanagement.gke.io — not a direct Cloud Deploy apply. Cloud Deploy owns the staged promotion (dev → qa → prod) and the prod manual-approval gate; nonprod rolls out automatically.

"SUCCEEDED" in Cloud Deploy ≠ live on the cluster

Because Config Sync is the thing that actually applies, a Cloud Deploy rollout reporting SUCCEEDED only means the artifact was rendered and published. The workload isn't live until the namespace's RepoSync — scaffolded from u2i-tenant-infra (namespace + reposyncRbac) — exists and has reconciled it. If the tenant scaffolding lands after the promotion, the already-published artifact comes up the moment Config Sync can reach it.

Binary Authorization

Enforced at the Fleet level — only container images that pass the configured attestation policy can be admitted onto the cluster.

Traffic & scaling

Gateway API for ingress

Apps register a route via a shared Gateway rather than each managing its own Ingress/load balancer. The app's Helm chart declares a parentRef into the platform's Gateway.

KEDA for autoscaling

Horizontal scaling beyond plain CPU/memory (e.g. queue depth, custom metrics) is handled by KEDA ScaledObjects, templated from the shared workload chart.

Ingress: Gateways, load balancers & modes

Gateway is the abstraction, the load balancer is the engine

A Gateway (Kubernetes Gateway API) is a declaration — 'accept traffic on these listeners, route by hostname/path'. The load balancer is the machinery that actually receives and forwards packets. An app declares an HTTPRoute with a parentRef into a Gateway; the controller programs the underlying LB. So 'Gateway' = what, 'load balancer' = how.

Two modes: eg (Envoy) vs managed-lb — same API, different engine

gatewayMode (set per environment in the app's Helm values) picks the engine. eg = Envoy Gateway: an open-source proxy (kubernetes-sigs/Envoy) we install and run as pods inside the cluster, fronted by one cheap shared L4 network LB. managed-lb = a Google Cloud L7 Application Load Balancer that GKE provisions and Google operates — pricier per LB, but gives Google-edge features (Cloud Armor, Google-managed certs, global anycast). Both implement the same Gateway API, so apps switch between them by changing one value.

L4 vs L7

L4 = network load balancer (like AWS NLB): forwards TCP by IP:port, doesn't read HTTP — cheap and simple. L7 = application load balancer (like AWS ALB): understands HTTP, routes by hostname/path, terminates TLS. managed-lb is L7; the Envoy-fronting LB is L4 (the L7 smarts live in the Envoy pods).

One shared Gateway per cluster

Each cluster has a single platform-owned Gateway, cluster-gateway in the cluster-edge namespace: eg (Envoy, shared IP) in nonprod, managed-lb in prod. Every app attaches its HTTPRoute to it and contributes its own per-app TLS cert into a shared cert map (cluster-cert-map) — instead of each app running its own load balancer (~$28/mo each). Apps sharing one product family use the same per-app-cert pattern; there is no separate mechanism for 'family' vs standalone.

Shared Gateway: the trade-offs

Consolidation buys cost, and pays in shared fate

One shared Gateway per cluster is cheaper (one load balancer instead of ~$28/mo each) and more consistent — but it trades that for a shared blast radius. Think of it as one lobby with one reception desk for every tenant in the building: efficient, until the reception desk goes down. The items below are the concrete forms that shared fate takes, and when they justify deliberately splitting an app back onto its own load balancer.

Blast radius — a bad change to the shared parts hits everyone

Each app's HTTPRoute lives in its own namespace and is fairly isolated — a broken route usually only breaks that app's hostname. But the Gateway object itself, its listeners, the single IP, and the shared cert map (cluster-cert-map) are common. A bad change there — deleting the Gateway, a malformed listener, a corrupted cert map, two apps claiming the same hostname — takes down every app at once. Treat any change to cluster-gateway as a fleet-wide production change, not a per-app one.

Certificate coupling — the immutable mapRef (KNV2009)

Every app puts its TLS cert as an entry into one shared cert map. A cert-map entry's mapRef (which map it belongs to) is immutable, so you cannot move an entry between maps in place — it must be deleted and recreated. Migrating an app between cert maps, or a domain change that forces recreating an immutable Cert Manager resource, wedges the declarative sync on a KNV2009 error and needs a manual delete+recreate (we hit exactly this migrating RetroTool). Plan cert/domain changes as delete+recreate, and keep resource names stable to avoid churn.

Shared limits & noisy neighbour

One load balancer has ceilings (routes, backends, certs per map, connections) that all apps consume collectively — they creep up quietly as apps pile on. The proxy layer differs by environment: in nonprod the eg Gateway is Envoy pods running on shared cluster compute, so a traffic spike to one app can starve others (a real noisy-neighbour at the proxy); in prod, Google scales the managed load balancer, so proxy-level noise is a non-issue, but the product/quota ceilings are still shared. Cloud Armor policies can be attached per-backend, so some isolation is possible without splitting the LB.

When to deliberately split back to a dedicated LB

Consolidation is the default, but three situations justify a dedicated Gateway/LB: compliance scope (an app handling regulated data shouldn't share an edge with un-regulated apps, or it drags them into its audit boundary); separate team ownership (a shared Gateway means shared change-management — a dedicated one lets a team own its edge without coordinating); and very different traffic profiles or edge features (a high-QPS/DDoS-prone app, or one needing a conflicting TLS/Cloud Armor/CDN policy). In those cases the per-LB cost is the price of a boundary, not waste.

Secrets & storage on the cluster

External Secrets Operator (ESO)

Apps declare an ExternalSecret resource; ESO pulls the actual value from GCP Secret Manager at sync time using the pod's Workload Identity, so no secret value is ever committed to Git or held in a raw Kubernetes Secret manifest.

GCS via the GCSFuse CSI driver

Apps that need bucket-backed storage (e.g. RetroTool's uploaded board backgrounds) mount a GCS bucket directly on the pod using GKE's native gcsfuse.csi.storage.gke.io driver — annotated on the pod spec, no sidecar container involved. Cache sizes are tuned per environment in Helm values.

Config Connector for GCP resources

Buckets and other GCP resources an app needs are declared as Kubernetes-native CRDs (e.g. StorageBucket) and reconciled into real GCP resources by Config Connector, keeping infra declarations next to the workloads that use them.

Glossary

GKE Autopilot: Google's fully-managed GKE mode. Google handles node provisioning, sizing, and patching; you only declare workloads and their resource requests.
Namespace: A Kubernetes-native partition inside a cluster. Used here as the per-app, per-environment isolation boundary ({app}-dev, {app}-prod, etc.) instead of separate clusters.
Workload Identity (Federation): The mechanism that lets a Kubernetes ServiceAccount authenticate as a GCP service account without any key file — GKE issues short-lived tokens tied to the pod's identity.
Fleet: Google Cloud's grouping of GKE clusters for centrally-managed features — used here to enable Config Sync and Binary Authorization consistently across u2i-nonprod and u2i-prod.
Config Sync: A GitOps operator that continuously reconciles cluster configuration from a Git-backed source (published as an OCI artifact). Here it owns both the tenant scaffolding (namespaces, RBAC, quotas, per-namespace RepoSync) AND — via each namespace's RepoSync — the app's own workload objects (Deployment, Service, HTTPRoute) that Cloud Deploy renders and publishes. It is the component that actually applies manifests to the cluster.
Cloud Deploy: Google's managed continuous-delivery service. Here it renders each app's Helm chart (via Skaffold) and publishes the result as a versioned OCI artifact to the config-sync registry — it does not apply to the cluster directly; Config Sync does. Provides staged promotion (dev → qa → prod) with a manual approval gate on prod.
Skaffold: A CLI/config tool that renders a Helm chart (or raw manifests) with environment-specific values. Cloud Deploy calls it during the "render" phase of a rollout.
Helm chart: A packaged, templated set of Kubernetes manifests. Each app owns a chart under helm/{app-name}/, with per-environment values files.
Gateway API: The successor to Ingress for routing external traffic into a cluster. Apps attach to a shared Gateway via a parentRef instead of provisioning their own load balancer.
KEDA: Kubernetes Event-Driven Autoscaling. Scales workloads based on external metrics (queue length, custom metrics), beyond plain CPU/memory-based HPA.
External Secrets Operator (ESO): An operator that syncs values from an external secret store (GCP Secret Manager) into Kubernetes at runtime, referenced via an ExternalSecret resource — no plaintext secrets in Git.
GCSFuse CSI driver: GKE's native Container Storage Interface driver (gcsfuse.csi.storage.gke.io) for mounting a GCS bucket as a filesystem directly into a pod, without a sidecar container.
Config Connector: A Kubernetes operator that lets you declare GCP resources (e.g. a GCS bucket) as Kubernetes CRDs, reconciling them into real cloud resources — infra-as-Kubernetes-manifests instead of separate Terraform for per-app resources.
Binary Authorization: An admission control policy that only allows container images meeting a defined attestation policy to be deployed onto the cluster.