How Pineapple deploys from a laptop to AWS

This guide follows one Pineapple change from a developer laptop to running pods in AWS.

It explains the parts of the system, who owns each part, and what happens when a new version is released. It also records the problems we found while building the system. The goal is a guide that a new team member can use when they need to deploy, debug, or change the platform.

The short version

Pineapple runs in two environments:

• Docker Compose runs the full system on a developer laptop.
• k3s runs the production and staging workloads on AWS EC2 nodes.

The delivery path is:

Code change
    |
    v
Pull request and CI checks
    |
    v
Merge to main
    |
    v
Create vMAJOR.MINOR.PATCH tag
    |
    +--> Build multi-platform images --> Docker Hub
    |
    +--> Update Helm values  ----------> GitOps pull request
                                              |
                                              v
                                      Merge GitOps pull request
                                              |
                                              v
                                      Argo CD watches main
                                              |
                                              v
                                      Helm renders Kubernetes objects
                                              |
                                              v
                                      k3s rolls out the new pods

The important point is that the image tag does not deploy by itself. The merged GitOps change tells Argo CD which image tag to run.

The complete architecture

Here is the system at a high level:

                         Public users and iOS clients
                                  |
                                  v
                    Cloudflare DNS, TLS, and Tunnel
                                  |
                                  v
                    cloudflared inside the k3s cluster
                                  |
              +-------------------+-------------------+
              |                   |                   |
              v                   v                   v
       public-api:8000   centrifugo:8000       grafana:3000
              |                   |
              +---------+---------+
                        |
                        v
                  Internal Services
                        |
        +---------------+----------------+
        |               |                |
        v               v                v
   PgBouncer         Redis         S3 buckets
        |
        v
   PostgreSQL

   API and workers also call Firebase, APNs, Resend, OpenAI,
   Gemini, PostHog, Sentry, and LiveKit when a feature needs them.

The public hostnames are routed through one production Cloudflare Tunnel:

The OIDC hostname is different:

oidc.trypineapple.app
        |
        v
Cloudflare DNS CNAME
        |
        v
CloudFront distribution
        |
        v
Private S3 bucket with two public OIDC files

AWS needs to read the OIDC files without reaching into the private cluster. CloudFront gives AWS a stable HTTPS URL for those files. The application traffic uses the Cloudflare Tunnel instead.

Where the code lives

The server repository has four main layers:

server/
├── cmd/                         Service entrypoints
│   ├── public-api/
│   ├── dispatcher/
│   ├── orchestrator/
│   └── ...
├── internal/
│   ├── domain/                  Business types and rules
│   ├── usecase/                 Business actions
│   ├── handler/                 HTTP and WebSocket handlers
│   └── infra/                   Postgres, Redis, S3, LiveKit, and telemetry
├── services/bernard2/           Python LiveKit voice agent
├── deploy/
│   ├── helm/                    Reusable Kubernetes chart
│   └── gitops/                  Argo applications and environment values
├── infra/
│   ├── terraform/               AWS and Cloudflare resources
│   ├── bootstrap/               EC2 and Argo setup scripts
│   └── scripts/                 SSM, OIDC, and cluster helper scripts
└── .github/workflows/           CI, infrastructure, and release workflows

An HTTP request normally follows this path:

HTTP request
    |
    v
internal/handler/http
    |
    v
internal/usecase
    |
    v
internal/domain
    |
    +--> internal/infra/postgres
    +--> internal/infra/redis
    +--> internal/infra/s3
    +--> external services

The cmd package starts a process. The handler receives the request. The use case runs the business action. The infrastructure code talks to a database or an outside service.

Workers use the same layers. They are separate processes because slow work should not hold up an HTTP request. Examples include push notifications, email, matching, embeddings, identity checks, and cleanup jobs.

The bernard2 Python service is a separate LiveKit voice agent. The backend Helm chart talks to LiveKit, but it does not install the Bernard service. That service has its own runtime and deployment path.

Local development with Docker Compose

The local system starts from server/docker-compose.yml.

Developer laptop
      |
      v
docker compose
      |
      +--> PostgreSQL
      +--> PgBouncer
      +--> Redis
      +--> API
      +--> Centrifugo
      +--> workers
      +--> River UI
      +--> Grafana, Loki, Tempo, Prometheus, Alloy
      +--> NGINX
      +--> cloudflared (when the local tunnel is enabled)

The local services use Docker network names. For example, the API can reach a local database by its Compose service name. The laptop’s .env file is for this system only.

Start the local stack:

cd server
make compose-up

The local reverse proxy has a useful job. One laptop can serve many hostnames, so NGINX routes each hostname to a Docker service:

api.trypineapple.app       -> web-public
admin.trypineapple.app     -> web-admin
docs.trypineapple.app      -> web-docs
river.trypineapple.app     -> web-riverui
grafana.trypineapple.app   -> web-grafana

The local Cloudflare Tunnel can then publish those local hostnames. The tunnel points at Docker service names such as web-public:8000.

Production does not need this NGINX layer. In production, Cloudflare sends traffic to cloudflared, and cloudflared sends it to Kubernetes Services by name. NGINX stays in Compose because it solves a local multi-hostname problem.

Local admin UIs

Argo CD and Headlamp run inside k3s. A local port forward exposes them on the laptop:

Browser
  |
  +--> local-argo.trypineapple.app
  |          |
  |          v
  |   Local Cloudflare Tunnel
  |          |
  |          v
  |   localhost:18080
  |          |
  |          v
  +--> kubectl port-forward -> Argo CD in k3s

Browser
  |
  +--> local-headlamp.trypineapple.app
             |
             v
       localhost:18081
             |
             v
       kubectl port-forward -> Headlamp in k3s

This path matters because localhost means the current machine. A browser inside another app may not share the laptop’s network namespace. When the port forward works in a terminal but a browser sees connection refused, use the local Cloudflare hostname or open the page in a browser that can reach the laptop.

Terraform creates the cloud foundation

Terraform owns the cloud resources. It does not deploy the Go services.

The main environment is under server/infra/terraform/environments/aws-dev/. The reusable k3s module is under server/infra/terraform/modules/aws-k3s-cluster/.

Terraform creates or manages:

• AWS S3 buckets for uploads, OIDC files, logs, and traces.
• The AWS security group for the k3s nodes.
• An on-demand k3s server EC2 instance.
• An optional Spot k3s agent instance.
• An encrypted EBS data volume for durable cluster data.
• Daily EBS snapshots through DLM.
• IAM roles for EC2, OIDC publication, and External Secrets Operator.
• AWS Systems Manager access for the nodes.
• The CloudFront distribution for the OIDC files.
• The ACM certificate for oidc.trypineapple.app.
• The production Cloudflare Tunnel and its DNS records.
• SSM parameters created by Terraform, such as the tunnel token and observability credentials.

The Terraform module receives a rendered copy of the bootstrap script:

module "k3s" {
  source = "../../modules/aws-k3s-cluster"

  bootstrap_script_path = abspath(
    "${path.module}/../../../bootstrap/bootstrap.sh"
  )

  k3s_oidc_issuer   = "https://${var.oidc_hostname}"
  k3s_oidc_jwks_uri = "https://${var.oidc_hostname}/openid/v1/jwks"
}

The server and agent use the same cluster token. The server starts k3s. The agent joins the server over the private VPC address.

Terraform state

Terraform state is stored in an S3 backend:

terraform {
  backend "s3" {
    bucket       = "pineapple-terraform-state-ACCOUNT_ID"
    key          = "aws-dev/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true
  }
}

State is not a normal source file. It can contain resource IDs and sensitive values. The state bucket needs tight IAM access. Encryption protects the stored object. The lock file stops two Terraform runs from changing the same state at the same time.

Bucket versioning and retention are also important for recovery. They are bucket settings, so they must be checked on the existing state bucket. A backend block alone does not turn versioning on.

Run Terraform locally with a short-lived AWS session:

cd server/infra/terraform/environments/aws-dev
terraform init
terraform plan -input=false
terraform apply -input=false

In GitHub Actions, the workflow uses GitHub OIDC to assume an AWS role. It does not store a long-lived AWS access key in GitHub.

The two-stage OIDC setup

External Secrets Operator needs AWS access. AWS must first trust tokens issued by this k3s cluster. That trust is built in two stages because the cluster must exist before it can publish its signing keys.

Stage 1
Terraform
  |
  +--> EC2 server and agent
  +--> S3 OIDC bucket
  +--> CloudFront
  +--> ACM certificate
  +--> Cloudflare DNS
  +--> OIDC publisher IAM role

Stage 2
Cluster is Ready
  |
  +--> kubectl reads k3s discovery and JWKS
  +--> publish-k3s-oidc.sh uploads two public files
  +--> CloudFront serves the files
  +--> curl verifies the public URL
  +--> Terraform runs with oidc_bootstrap_ready=true
  +--> AWS OIDC provider and ESO role are created

The public files are:

https://oidc.trypineapple.app/.well-known/openid-configuration
https://oidc.trypineapple.app/openid/v1/jwks

The discovery document names the issuer and the public JWKS URL. The JWKS contains public signing keys. It must not contain private key fields.

AWS then checks the exact issuer, audience, and Kubernetes service account subject:

Condition = {
  StringEquals = {
    "oidc.trypineapple.app:aud" = "sts.amazonaws.com"
    "oidc.trypineapple.app:sub" = "system:serviceaccount:external-secrets:external-secrets"
  }
}

ESO cannot claim an arbitrary URL. AWS uses the OIDC provider that Terraform registered, checks the signed token against the provider’s keys, and checks the exact claims above. A different URL, namespace, or service account fails the trust check.

EC2 cloud-init and k3s bootstrap

Terraform passes EC2 user data to the instance. The user data starts infra/bootstrap/bootstrap.sh.

The script is staged and safe to retry:

cloud-init user data
        |
        v
system hardening
        |
        v
format and mount EBS data disk (server only)
        |
        v
host firewall
        |
        v
install k3s server or agent
        |
        v
wait for CNI interfaces
        |
        v
check node Ready and OIDC endpoints
        |
        v
/var/lib/pineapple/bootstrap-complete

Each completed stage writes a marker under /var/lib/pineapple/bootstrap/. The full log is written to:

/var/log/pineapple-bootstrap.log

This makes a failed first run easier to inspect. The server also needs these firewall paths:

Cloud-init problems we found

Cloud-init runs during the first boot. It can race with the package manager or with the EBS device appearing. The script waits for network and retries the k3s agent join.

The main lessons were:

1. Wait for the data device. AWS may report /dev/sdf, while Linux exposes /dev/nvme1n1.
2. Wait for k3s CNI interfaces before adding the routed firewall rules. If UFW blocks Flannel, nodes can look alive while pods cannot talk.
3. Keep the on-demand server stable. Spot agents can be interrupted and must be safe to recreate.
4. Use SSM instead of opening SSH. The EC2 role includes AmazonSSMManagedInstanceCore.
5. Check the real persistent volume during image upgrades. A database image can fail when an old volume has ownership for a different runtime user. The Postgres image must repair ownership before it drops privileges.

Inspect a node through SSM:

aws ssm start-session --target INSTANCE_ID

sudo tail -f /var/log/pineapple-bootstrap.log
sudo systemctl status k3s
sudo kubectl get nodes -o wide
sudo kubectl get pods -A

Terraform can finish while cloud-init is still running. A successful terraform apply means AWS accepted the resources. It does not mean k3s, Argo CD, or the application is ready.

Cluster bootstrap: OIDC, Argo CD, and Git

After the server is Ready, an operator fetches the k3s kubeconfig through SSM. The local kubeconfig points to the private k3s API through an SSM path.

The one-time cluster bootstrap then does three things:

1. Publishes the k3s OIDC discovery and JWKS files.
2. Installs Argo CD and gives it a read-only deploy key for the server repository.
3. Applies the root Argo Application.

The root Application watches this path on main:

deploy/gitops/argocd/applications

That directory contains child Applications for:

• The production backend Helm release.
• The staging backend Helm release.
• External Secrets Operator.
• The AWS SSM ClusterSecretStore.
• Reloader.
• Headlamp.

The root Application is a small application tree:

pineapple-gitops-root
        |
        +--> pineapple-backend
        +--> pineapple-backend-staging
        +--> pineapple-external-secrets
        +--> pineapple-secret-store
        +--> pineapple-production-secrets
        +--> pineapple-staging-secrets
        +--> pineapple-reloader
        +--> pineapple-headlamp

Argo’s important settings are:

syncPolicy:
  automated:
    prune: true
    selfHeal: true

selfHeal means Argo repairs drift. prune means Argo removes objects that Git no longer declares. The Git repository is the desired state. The cluster is the result of that state.

Helm is the Kubernetes packaging layer

Helm is not the deployment controller. Helm takes values and templates and produces Kubernetes YAML. Argo CD runs Helm during a sync and applies the result.

The chart is at:

server/deploy/helm/pineapple-backend

The chart templates create:

• Deployments and Services for the API and workers.
• PostgreSQL and PgBouncer.
• Redis.
• Centrifugo.
• Cloudflared.
• Grafana and the observability stack.
• A database migration Job.
• Cleanup and payment CronJobs.

The environment values live separately:

server/deploy/gitops/environments/production/values.yaml
server/deploy/gitops/environments/staging/values.yaml

The release value is shared by the service images:

global:
  imageTag: v0.1.26
  appVersion: v0.1.26
  environment: production

services:
  public-api:
    enabled: true
    replicas: 1
  dispatcher:
    enabled: true
    replicas: 1

The chart helper uses global.imageTag when a service does not set a separate image tag. This keeps the API, workers, migration image, and cleanup jobs on one release version.

Render the chart before it reaches the cluster:

cd server
helm lint deploy/helm/pineapple-backend
helm template pineapple deploy/helm/pineapple-backend \
  -f deploy/gitops/environments/production/values.yaml

./scripts/harness/verify-k8s.sh \
  --values deploy/gitops/environments/production/values.yaml

The chart does not create an Ingress or a public LoadBalancer. Services are ClusterIP. Cloudflared is the public edge.

Database migration order

The migration Job is an Argo Sync hook with sync wave 10. The backend Application is in a later wave. The Job waits for PostgreSQL and runs the migration image before the application rollout depends on the new schema.

metadata:
  annotations:
    argocd.argoproj.io/hook: Sync
    argocd.argoproj.io/sync-wave: "10"
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded

This is why database changes must be reviewed with both the migration image and the application code. A migration that succeeds can still break an older or newer application if the contract is not safe.

Secrets: local .env, SSM, ESO, and pods

There are two different secret flows.

Local secrets

server/.env is for Docker Compose. It may contain local database and service values. It is not uploaded as one large Kubernetes secret.

Production secrets

The production file is used as an input to the SSM migration script. The script sends each approved key as an SSM SecureString:

server/.env.production
        |
        v
infra/scripts/migrate-env-to-ssm.sh
        |
        v
SSM /pineapple/prod/OPENAI_API_KEY
SSM /pineapple/prod/RESEND_API_KEY
SSM /pineapple/prod/DATABASE_URL
        |
        v
External Secrets Operator
        |
        v
Kubernetes Secret: pineapple-backend-secrets
        |
        v
API and worker environment variables

DATABASE_URL is built by the script. It uses the Kubernetes PgBouncer service name:

database_url="postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@pineapple-pgbouncer:6432/${POSTGRES_DB}?sslmode=disable"

That hostname is not used by local Compose. This is why the local .env and production input have different database connection details.

How AWS trusts ESO

The External Secrets service account has an IAM role annotation:

serviceAccount:
  name: external-secrets
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/pineapple-dev-eso

The name uses an EKS annotation because ESO supports this AWS web identity pattern, even though the cluster itself is k3s. The AWS trust policy still checks the exact k3s OIDC provider and service account subject.

The role policy is narrow:

Action   = ["ssm:GetParameter", "ssm:GetParameters"]
Resource = local.ssm_parameter_arns

ESO can read the listed parameters. It cannot write them or read every SSM value in the account.

The ExternalSecret refreshes on a schedule. Reloader watches the resulting Kubernetes Secret and restarts workloads that use it. A pod reads environment variables at startup, so the restart is what makes a changed value active.

Networking and Cloudflare

There are three network boundaries:

Internet
   |
   v
Cloudflare edge
   |
   +--> Cloudflare Tunnel --> cloudflared pod --> ClusterIP Service
   |
   +--> CloudFront --> private OIDC S3 bucket

AWS VPC
   |
   +--> k3s server: private API
   +--> k3s agent: private node
   +--> PostgreSQL, Redis, workers, and observability

Operator laptop
   |
   +--> AWS SSM Session Manager --> private k3s API

The AWS security group only allows the k3s ports from the VPC. The nodes have no public SSH rule. The Cloudflare tunnel connection starts inside the cluster and makes an outbound connection to Cloudflare.

Terraform creates the production tunnel and the DNS records:

resource "cloudflare_zero_trust_tunnel_cloudflared" "production" {
  account_id = local.cloudflare_account_id
  name       = "pineapple-dev-production"
  config_src = "cloudflare"
}

resource "cloudflare_zero_trust_tunnel_cloudflared_config" "production" {
  account_id = local.cloudflare_account_id
  tunnel_id  = cloudflare_zero_trust_tunnel_cloudflared.production.id

  config = {
    ingress = [
      {
        hostname = "api.trypineapple.app"
        service  = "http://pineapple-public-api:8000"
      },
      {
        hostname = "realtime.trypineapple.app"
        path     = "/connection/*"
        service  = "http://pineapple-centrifugo:8000"
      },
      { service = "http_status:404" }
    ]
  }
}

Terraform stores the generated tunnel token in SSM. ESO copies it into the pineapple-cloudflare-tunnel Kubernetes Secret. The Cloudflared Deployment reads that Secret.

The Cloudflare API token used by Terraform needs both kinds of access:

• Zone DNS write for the records.
• Account Zero Trust Tunnel write for the tunnel and its configuration.

DNS access alone produces a Cloudflare 403 Authentication error when Terraform tries to create a tunnel.

CI checks the code before release

The server repository has separate workflows for different jobs.

Pull request workflow

.github/workflows/backend-k8s.yml classifies the changed files and runs only the needed checks. The important checks include:

• Go tests and build.
• Go lint and vulnerability checks.
• Gosec and Gitleaks.
• Generated SQL checks.
• Docker Compose config checks.
• Compose runtime smoke tests.
• Helm rendering and kubeconform checks.
• Image build and Trivy checks when runtime files change.

This catches a bad chart before Argo ever sees it.

Terraform workflow

.github/workflows/infra.yml runs Terraform format and validation on infrastructure changes. A main-branch plan or apply uses GitHub OIDC to assume the AWS role. Apply is a manual workflow dispatch with an environment approval gate.

The separation matters:

Terraform workflow  -> AWS, Cloudflare, S3, IAM, EC2
Release workflow    -> Docker images and GitOps values
Argo CD             -> Kubernetes workloads

No workflow should run kubectl apply for the application. Argo is the application deployer.

The GitOps release flow

The release workflow is triggered by a version tag:

git tag v0.1.26
        |
        v
GitHub release workflow
        |
        +--> Check tag points at main
        +--> Build linux/amd64 and linux/arm64 images
        +--> Push images to Docker Hub
        +--> Scan images with Trivy
        +--> Verify Helm with production values
        +--> Create release/v0.1.26 branch
        +--> Change only two GitOps values files
        +--> Open a GitOps pull request

The GitOps pull request changes:

global:
  imageTag: v0.1.26
  appVersion: v0.1.26

The release helper refuses to continue when:

• The tag does not look like vMAJOR.MINOR.PATCH.
• The tag is not on the current origin/main commit.
• The checkout is dirty.
• The helper would change files outside staging and production values.

This keeps an image release separate from the approval to deploy it.

Local release path

The Makefile mirrors the release workflow. This is useful when the hosted GitHub runner is unavailable or when an operator needs to test the process locally.

cd server

git switch main
git pull --ff-only origin main
git tag -a v0.1.26 -m "Release v0.1.26"

# Show image names, Helm checks, and the exact GitOps diff.
make release-dry-run TAG=v0.1.26

# Build, push, scan, check Helm, and open the GitOps pull request.
make release TAG=v0.1.26

The local command needs Docker, Buildx, Trivy, Helm, kubeconform, yq, and GitHub CLI. It does not need kubectl, a kubeconfig, or a direct Argo connection. The release helper edits Git in a temporary worktree and pushes the release branch.

After review, merge the GitOps pull request. Argo then sees the new commit on main:

GitOps PR merged
        |
        v
Argo root Application sees main changed
        |
        v
Child backend Application runs Helm
        |
        v
Migration hook runs
        |
        v
Deployments use new Docker Hub tag
        |
        v
k3s replaces old pods and waits for readiness

There is no normal manual argocd app sync step. A manual sync can help during an incident, but it must not become the release process. Git remains the source of truth.

A practical deploy checklist

First infrastructure build

1. Set the AWS region, Cloudflare zone ID, OIDC publisher principal, and other Terraform variables in the environment tfvars file.
2. Run terraform plan and review the security group, EC2, S3, IAM, CloudFront, ACM, and tunnel changes.
3. Apply the first stage.
4. Wait for cloud-init and check /var/log/pineapple-bootstrap.log through SSM.
5. Fetch the kubeconfig through SSM.
6. Publish the OIDC files and verify both public URLs.
7. Set oidc_bootstrap_ready = true and run Terraform again.
8. Run the cluster bootstrap script to install Argo CD and apply the root Application.
9. Push the required production and staging secrets to SSM.
10. Check ESO, Argo, pods, and the public health endpoint.

New application version

1. Merge the code pull request.
2. Pull the latest main.
3. Create a version tag on the exact main commit.
4. Let GitHub Actions or make release build and scan the images.
5. Review the GitOps pull request.
6. Merge the GitOps pull request.
7. Watch Argo sync staging and production.
8. Check the migration Job and pod readiness.
9. Call the public health endpoint and test the changed feature.

Useful checks

kubectl get nodes -o wide
kubectl get pods -A
kubectl -n argocd get applications
kubectl -n external-secrets get externalsecret
kubectl -n pineapple get pods
kubectl -n pineapple get jobs

curl -i https://api.trypineapple.app/health
curl -i https://oidc.trypineapple.app/.well-known/openid-configuration
curl -i https://oidc.trypineapple.app/openid/v1/jwks

Do not run the first command against the wrong context and then assume the cluster is empty. kubectl get pods only checks the current namespace. kubectl get pods -A checks every namespace.

Problems we found and what they taught us

What each tool owns

The platform is easier to reason about when each tool has one job:

The biggest operating rule is simple: change the right source of truth. Change Terraform for cloud resources. Change Helm or GitOps values for Kubernetes resources. Change SSM through the secret migration path for production secrets. Change application code through a pull request and release tag.

Final picture

                         GitHub
                +-----------------------+
                | code and GitOps files |
                +-----------+-----------+
                            |
             +--------------+--------------+
             |                             |
             v                             v
       GitHub Actions                  Argo CD
       tests and images               watches main
             |                             |
             v                             v
          Docker Hub                 Helm + k3s
                                           |
              +----------------------------+----------------------------+
              |                            |                           |
              v                            v                           v
          API + workers              PostgreSQL + Redis          Cloudflared
              |                            |                           |
              +----------------------------+---------------------------+
                                           |
                                           v
                              Cloudflare public hostnames

Terraform creates the AWS and Cloudflare foundation below this system.
SSM and ESO deliver secrets into it.

The deployment is not one command because it has several owners. Once the boundaries are clear, the path is predictable: build an image, record the desired tag in Git, let Argo apply it, and check the real system from the edge to the database.