Nidhi's blog

Production Engineering • SRE • Kubernetes

Network gateway illustration

Internal Services on GKE Without a VPN: IAP + Nginx Gateway

May 5, 2026

The short version

We needed to expose internal tools - an LLM gateway, internal dashboards etc - to the team without making them public and without maintaining a VPN.

  1. An nginx Deployment in GKE acts as an internal reverse proxy
  2. Its Service is an Internal Load Balancer (private VPC IP only, never public)
  3. Google IAP sits in front for the user-facing path - Google handles auth, we handle routing
  4. One nginx config serves both the IAP path and direct internal VPC access via two domain patterns

Why IAP instead of a VPN

VPNs work. But for HTTP-only internal tools deployed in GCP accessed by a team already using Google Workspace, IAP is a better fit.

IAP (Identity-Aware Proxy) sits in front of a GCP load balancer. When a user hits the URL, Google intercepts the request, checks whether their Google account has IAM access to this backend, and either forwards or rejects. The user authenticates with their existing Google account - no extra credentials is needed and works from any network.

Access control is a single IAM binding: roles/iap.httpsResourceAccessor on the backend resource, granted to a user or group. Audit logs come for free. Revoking access is instant.

The constraint: IAP is HTTP/HTTPS only. It cannot front SSH, database connections, or arbitrary TCP. For those, you still need the VPN or a bastion. For web UIs and HTTP APIs, IAP covers everything.

The architecture

Three layers:

  1. nginx Deployment - a single nginx pod in the cluster. Its config has one server {} block per internal service. It knows how to reach each backend, either via kube-dns (in-cluster services) or a node IP and port (out-of-cluster services).

  2. Internal Load Balancer - the nginx Service is type: LoadBalancer with networking.gke.io/load-balancer-type: "Internal". GKE provisions a private IP on the VPC. No public IP is allocated. The service also exposes NodePort 30080, which the IAP ALB uses as its backend.

  3. IAP ALB - a separate HTTPS load balancer configured in GCP, pointing to the cluster nodes on port 30080. IAP is enabled on this LB. Users hit https://tool.iap.example.com, get authenticated byi Google, and the request lands on nginx.

apiVersion: v1
kind: Service
metadata:
  name: nginx-internal
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30080
  selector:
    app: nginx

The internal LB and IAP ALB are separate: the internal LB serves VPC-internal traffic on port 80, the IAP ALB serves authenticated external traffic via NodePort 30080.

The nginx config

In-cluster service (resolved via kube-dns):

server {
    listen 80;
    server_name tool.iap.example.com tool.internal.example.com;

    location / {
        resolver kube-dns.kube-system.svc.cluster.local;
        set $upstream "tool.tool-namespace.svc.cluster.local";
        proxy_pass http://$upstream:4000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Out-of-cluster service (VM or NodePort):

server {
    listen 80;
    server_name other-tool.iap.example.com other-tool.internal.example.com;

    location / {
        proxy_pass http://<node-ip>:30808;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
    }
}

The resolver trick.

If you write proxy_pass http://tool.namespace.svc.cluster.local directly, nginx resolves the hostname once at startup and caches the IP. When the upstream pod restarts and gets a new IP, nginx doesn’t know - it keeps sending to the old IP and returns 502s until nginx itself is restarted.

Using set $upstream "..." with a resolver directive changes this behaviour. nginx treats $upstream as a variable, which forces it to go through the resolver on each request. The kube-dns resolver returns the current ClusterIP, which tracks the live pod. No stale IPs, no 502s after rollouts.

Always use the set $var pattern for in-cluster upstreams. For out-of-cluster targets with a stable IP (a VM’s internal address), the direct proxy_pass is fine.

The dual-domain pattern

Each service gets two hostnames:

  • tool.iap.example.com - routes through the IAP ALB. Requires Google account authentication. Used by humans accessing the tool from a browser.
  • tool.internal.example.com - routes through the internal LB directly. No auth at the proxy layer. Used by services on the VPC calling each other.

Both hostnames resolve to the same nginx, the same server {} block, and the same upstream. The only difference is whether IAP sits in front of the request.

This means you can grant a service account direct VPC access to tool.internal.example.com without giving it an IAP IAM binding, while still requiring human users to authenticate via tool.iap.example.com. The access model maps cleanly to what each caller actually is.

When it breaks

502 from nginx after upstream pod restarts.

nginx resolved the upstream IP at startup and cached it. Use resolver kube-dns.kube-system.svc.cluster.local and set $upstream "..." - see the nginx config section above.

IAP returns 403 even though the user is logged in.

The user’s Google account doesn’t have roles/iap.httpsResourceAccessor on the backend resource. Check IAM bindings on the IAP backend in the GCP console. Groups are easier to manage than individual bindings.

Internal LB IP unreachable from a VPC service.

Firewall rules. GKE internal LBs require a firewall rule allowing traffic from the source range to the node pool on the NodePort range (30000–32767). Check for a rule matching gke-*-internal or create one explicitly.

New service added to nginx config but traffic isn’t routing.

nginx config reloads require either a pod restart or a nginx -s reload signal. If the ConfigMap was updated but the pod wasn’t restarted, the old config is still active. Add a checksum annotation to the Deployment to force rollouts on ConfigMap changes:

annotations:
  checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

WebSocket connections dropping.

Missing Upgrade and Connection headers. Add them to the location {} block:

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

Also set proxy_read_timeout and proxy_send_timeout to something longer than the default 60s if your WebSocket connections are long-lived.