How Pineapple runs from a laptop to AWS

Pineapple runs in two places.

On a developer laptop, Docker Compose starts the whole system. In production, the same parts run as services and workers in k3s on AWS.

The move from Compose to AWS raised simple questions:

• Where does each service run?
• Where do production secrets come from?
• How does a private cluster receive public traffic?
• How do we deploy without putting cloud keys in GitHub?
• What happens when one part of the deployment fails?

This post maps the system and answers those questions.

The system in one picture

The main path looks like this:

Browser or iOS app
        |
        v
Cloudflare DNS, TLS, and Tunnel
        |
        v
cloudflared inside the k3s cluster
        |
        v
Kubernetes Services
        |
        +--> public API
        +--> Centrifugo realtime service
        +--> Grafana

The API and workers use the internal services below:

API and workers
        |
        +--> PgBouncer --> PostgreSQL
        +--> Redis
        +--> S3 buckets
        +--> Centrifugo
        +--> Email, Apple, Firebase, OpenAI, Gemini, and LiveKit

The admin tools use a private path:

Laptop
  |
  +--> AWS Systems Manager port forward
          |
          +--> private Kubernetes API
                    |
                    +--> Argo CD
                    +--> Headlamp

The blog uses a different Cloudflare product. It is a static Astro site hosted by Cloudflare Pages. It does not use the application tunnel or the Kubernetes cluster.

What runs on a laptop?

Docker Compose starts the local system from server/docker-compose.yml.

The main containers are:

PostgreSQL and Redis use Docker volumes on the laptop. The database stays after a container is replaced. Redis also uses a volume in local Compose.

The API does not do every job in the request. A request can write data and add work for a worker. A worker then sends a push notification, sends an email, creates an embedding, checks an identity document, or updates a match.

Centrifugo handles realtime delivery. The API and dispatcher publish events to Centrifugo. The iOS app connects to Centrifugo over a WebSocket.

NGINX is useful in local Compose because one machine serves several hostnames. It routes requests such as these:

api.trypineapple.app       -> public API
admin.trypineapple.app     -> admin app
docs.trypineapple.app      -> API docs
river.trypineapple.app     -> River UI

The local Cloudflare Tunnel sends traffic into this local network. The tunnel has routes to local Docker service names such as web-public and web-admin.

What runs in AWS?

Terraform creates the cloud foundation. It creates the VPC, private networking, EC2 instances for k3s, S3 buckets, IAM roles, the OIDC files for AWS, the CloudFront distribution for those files, and the production Cloudflare Tunnel.

The k3s nodes are EC2 instances in the VPC. Kubernetes then runs the application workloads.

Argo CD reads the Kubernetes and Helm files from Git. It applies those files to the cluster. This makes Git the record of what should run.

The production Helm chart includes these parts:

• The public API.
• Push notification, matching, and orchestration workers.
• Optional workers for embeddings, event enrichment, email, and identity checks.
• Redis.
• PostgreSQL.
• PgBouncer.
• Centrifugo.
• Grafana and the observability services.
• Cloudflared.
• Database migration and scheduled cleanup jobs.

The production values file decides which parts are enabled. In the current values, the public API, push notifications, dispatcher, orchestrator, Redis, PostgreSQL, PgBouncer, Centrifugo, Grafana, and cleanup jobs are enabled. Some background consumers are set to zero replicas until they are needed.

PostgreSQL and PgBouncer

PostgreSQL runs as a Kubernetes StatefulSet. It uses a retained local volume on an on-demand k3s node. The current volume size is 20 GiB.

PgBouncer runs as a separate Kubernetes Deployment. The API and workers connect to PgBouncer instead of opening every connection directly to PostgreSQL.

The production DATABASE_URL therefore uses this internal name:

postgresql://<user>:<password>@pineapple-pgbouncer:6432/<database>?sslmode=disable

That name only works inside Kubernetes. It is different from the local Compose database hostname.

Redis

Redis runs as one Kubernetes pod and one internal Kubernetes Service named pineapple-redis.

The current production values set Redis persistence to false. Redis is used for fast state, matching data, and messaging. It is not the main database. Data that must survive a Redis restart belongs in PostgreSQL or another durable store.

This is an important operating detail. A single Redis pod with no disk persistence is simple, but it is not a highly available Redis design. If Pineapple needs Redis data to survive node loss, this part will need a managed or replicated Redis service later.

Centrifugo

Centrifugo runs as one Kubernetes Deployment and one internal Service named pineapple-centrifugo.

The API and dispatcher call its internal API. Clients connect through the public WebSocket URL:

wss://realtime.trypineapple.app/connection/websocket

Only the WebSocket path is sent to Centrifugo. The rest of realtime.trypineapple.app is not used for this connection.

Workers

Workers are separate container images and Kubernetes Deployments. They share the same database and secret delivery path as the API.

Examples include:

• push-notifications sends Apple push notifications.
• dispatcher sends match offers through Centrifugo or Apple push.
• orchestrator manages short-lived matching work in Redis.
• embedding-service creates search or matching embeddings.
• email-consumer sends email through Resend.
• identity-verification-consumer checks identity work.
• event-enrichment-consumer and post-call-insights-consumer process background work.
• lobby-cleanup removes old lobby state.

The chart also creates scheduled Kubernetes jobs for file-upload cleanup and payment reconciliation.

The API, workers, and scheduled jobs use the same image release version. Argo CD changes the image tag when a release is promoted.

Observability

Grafana is the user interface. Loki stores logs. Tempo stores traces. Prometheus stores metrics. Grafana Alloy collects data from the workloads.

In production, Loki and Tempo use S3 buckets in eu-west-1. Grafana stays inside the cluster and is reached through the production tunnel. It is not an open public service.

Why Cloudflare is in front

The k3s nodes are private. We did not want to open inbound ports on the EC2 nodes for the application.

Cloudflare gives us three useful things here:

1. DNS for the Pineapple hostnames.
2. TLS at the public edge.
3. A tunnel from Cloudflare to cloudflared inside the private cluster.

The tunnel connection starts inside the cluster and goes out to Cloudflare. A public request reaches Cloudflare first. Cloudflare sends it through the existing tunnel. The request then reaches a Kubernetes Service.

The production routes are:

Terraform creates the tunnel, its routes, and the DNS records. The tunnel token is stored in SSM, copied by ESO into a Kubernetes Secret, and read by the cloudflared Deployment.

We do not add NGINX in front of the production tunnel. Kubernetes Services already route traffic inside the cluster. NGINX remains in local Compose because local development needs one reverse proxy for several Docker services.

This is why Cloudflare is part of the architecture. It is the public edge and the private-cluster bridge. It is not where the API, Redis, PostgreSQL, or workers run.

Secrets: from a local file to a pod

The local .env file is for Docker Compose. It is not copied into AWS.

We create a separate production file and push approved values to AWS Systems Manager Parameter Store:

server/.env.production
        |
        v
push-production-secrets.sh
        |
        v
SSM SecureString: /pineapple/prod/*
        |
        v
External Secrets Operator
        |
        v
Kubernetes Secret
        |
        v
API and worker pods

External Secrets Operator, or ESO, is a Kubernetes controller. It reads selected SSM parameters and writes them into a Kubernetes Secret.

AWS trusts one Kubernetes service account. The trust rule names the cluster issuer, the external-secrets namespace, and the external-secrets service account. ESO cannot claim any URL and receive access. AWS checks the signed Kubernetes token against the exact issuer, audience, and service account subject.

The ESO role can read only the SSM parameter paths listed by Terraform. It cannot write or read every parameter in the account.

When a secret changes, ESO updates the Kubernetes Secret. Reloader sees the change and restarts the pods that use it. The new pod reads the new environment value when it starts.

DATABASE_URL is built during the push. It points to the production PgBouncer Service. This is why it is not known in the local .env file.

Why the OIDC endpoint exists

AWS needs a way to check that a Kubernetes token came from our cluster. k3s publishes two public files:

https://oidc.trypineapple.app/.well-known/openid-configuration
https://oidc.trypineapple.app/openid/v1/jwks

The first file describes the issuer and token endpoints. The second file contains public signing keys. It does not contain private keys.

Terraform cannot create the final AWS trust rule before k3s exists. The setup therefore has two stages:

1. Terraform creates the EC2 nodes, S3 bucket, CloudFront distribution, DNS, and certificate pieces.
2. The cluster bootstrap script reads the k3s discovery and key files.
3. The script uploads those two public files to the OIDC S3 bucket.
4. CloudFront serves the files from the private bucket.
5. DNS sends oidc.trypineapple.app to CloudFront.
6. Terraform runs again with oidc_bootstrap_ready = true.
7. Terraform creates the AWS OIDC provider and the ESO IAM role.

The OIDC bucket has versioning and server-side encryption. The Terraform state bucket is separate. Its backend uses S3 encryption and a Terraform lock file. Because that bucket was created before this Terraform workspace, its versioning and retention settings must be checked separately.

How code reaches the cluster

There are three related delivery paths.

Infrastructure

Terraform code lives under server/infra/terraform. It creates cloud resources and the production Cloudflare routes.

GitHub Actions checks Terraform on pull requests. A main-branch plan is allowed by a repository setting. An operator can start an apply from the Actions page.

GitHub does not store an AWS access key. The workflow receives a short-lived GitHub OIDC token. AWS checks the repository and branch rules, then gives the workflow a short-lived role session.

Kubernetes applications

The backend workflow checks Go code, Helm charts, Docker Compose, generated SQL, security checks, and image builds.

When a version tag is released, the release workflow builds immutable container tags and updates the GitOps image versions. Argo CD sees the Git change and syncs the Kubernetes workloads.

Terraform creates the cluster. Argo CD deploys the applications into that cluster.

The blog

The blog is a separate static Astro site. GitHub Actions builds blog/dist and uploads it to Cloudflare Pages. It does not go through the application tunnel.

What failed during the first apply?

The first production run failed in more than one place.

The AWS profile expired

The pineapple-terraform AWS profile used a credential process. aws login created a short-lived session. Terraform could use it until the session expired.

After that, the AWS provider tried the EC2 metadata service and failed because the laptop was not an EC2 instance. The fix was to log in again for local work. GitHub Actions uses AWS OIDC, so CI does not depend on a laptop session.

The Cloudflare token could edit DNS but not create a tunnel

The first token had DNS permissions. Terraform still received a 403 error when it created the Zero Trust Tunnel.

DNS permissions and Zero Trust tunnel permissions are different. The token used by Terraform needs the account permission for the tunnel resources as well as the zone permission for DNS records.

ACM returned CAA_ERROR

The OIDC hostname uses an ACM certificate in us-east-1 because CloudFront requires certificates there. ACM rejected the request because the DNS zone did not allow Amazon to issue the certificate.

Terraform now creates a CAA record for amazon.com before it waits for the certificate.

Terraform made some resources before stopping

Terraform can create resources in dependency order and stop when one resource fails. The failed run still created some S3 resources, an EC2 agent, and other state entries.

The state file records what Terraform created. It does not prove that the cluster is ready. We must check the nodes, cloud-init logs, OIDC files, Cloudflare tunnel, Argo CD, ESO, and the API after an apply.

How we check the system

We call the deployment healthy when the full path works:

1. The k3s nodes are Ready.
2. Argo CD reports the applications as Synced and Healthy.
3. ESO reports that the secrets are synced.
4. The migration job completes.
5. The API health endpoint returns 200.
6. The API can reach PostgreSQL, Redis, S3, and Centrifugo.
7. A safe SSM change reaches a new pod after Reloader restarts it.
8. The public API and realtime routes reach the right Services.
9. Argo CD and Headlamp work through the SSM port forward, but are not public.

The useful lesson is to draw the whole path before running the first apply. Each tool has a clear job:

• Terraform creates cloud resources.
• AWS stores production parameters and runs the private nodes.
• Kubernetes runs the API, data services, workers, and observability tools.
• Cloudflare provides DNS, TLS, and the tunnel to the private cluster.
• ESO moves approved secret values into Kubernetes.
• GitHub Actions checks and releases changes.
• Argo CD applies the Kubernetes state from Git.

When a deployment fails, this map tells us where to look.