# Deployment Checklist

## not.bot™ Verify: Deployment Checklist

Complete the Pre-flight Checklist before you start. Each phase below assumes the previous phase is done. Open the Operations & Reference Guide alongside this document if you need context for any step.

Pre-flight decision labels (A through K) appear throughout. Fill in the value you wrote down in the Pre-flight Checklist wherever you see one.

---

## Phase 1: OpenBao

### 1.1 Install the OpenBao chart

The OpenBao image shipped in the chart (`harbor.julia.social/library/openbao-bls`, pinned by digest) already has the Chiakeys plugin baked in. No build step is required for the mainline deployment. If you need to rebuild the image from source for supply-chain validation, see Appendix A.

```bash
unzip not.bot_verify_deployment.zip
cd helm/openBao
helm dependency update
kubectl create namespace <A>
cd ../..
helm install openbao ./helm/openBao -n <A>
```

**Verify:** `kubectl get pods -n <A>` shows the openbao pod Running.

### 1.2 Initialize and unseal

```bash
kubectl port-forward svc/openbao 8200:8200 -n <A>
```

In a second terminal, point the CLI at the port-forward and initialize:

```bash
export VAULT_ADDR=http://127.0.0.1:8200
bao operator init -key-shares=1 -key-threshold=1
```

> **`VAULT_ADDR` is required.** The chart runs OpenBao with `tls_disable` set on the listener, while the `bao` CLI defaults to `https://127.0.0.1:8200`. Without the export, the first command fails against the port-forward with a TLS error such as `http: server gave HTTP response to HTTPS client`. Keep this terminal — §1.3 runs in it and inherits the export.

Save the **Unseal Key** and **Root Token** in your break-glass secret store. Losing the unseal key destroys the data in OpenBao.

```bash
bao operator unseal <UNSEAL_KEY>
read -rs -p "Enter Token: " && echo "$REPLY" | bao login -
```

> **OpenBao reseals on every pod restart.** Keep the unseal key accessible to whoever is on call. Consider OpenBao auto-unseal if your environment supports it.

### 1.3 Create the secrets engine, namespace, and scoped token

```bash
bao namespace create <D>
export VAULT_NAMESPACE=<D>
bao secrets enable -path=chiakeys bls
bao secrets enable -path=secret -version=2 kv
```

> The `secret` KV-v2 engine holds non-DID secrets that the admin service writes during normal operation — for example, the CA cert and key pair you upload when registering a Chia node in §6.2 are stored at `secret/data/blockchain-nodes/<UUID>/certificate`. The path name `secret` is fixed; the admin service is not configurable here.

Create `policy.hcl`:

```hcl
path "chiakeys/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "secret/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "secret/metadata/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
```

Apply and create the scoped token:

```bash
bao policy write <D>-policy policy.hcl
bao token create -policy=<D>-policy -orphan -period=720h
```

Save the token. You will use it in Phase 5 (admin service) and never the root token.

> **`-orphan` is required.** Without it, the new token is a child of the root token, and §1.4's `bao token revoke -self` cascades the revocation to every child — invalidating this scoped token before the admin service ever uses it. The failure surfaces later, when the admin service first writes to OpenBao, as a misleading `permission denied` on the chiakeys path. `-orphan` detaches the token so it survives the root revocation.
>
> `-period=720h` creates a renewable periodic token with a 30-day TTL. The admin service checks the token hourly and renews it automatically when fewer than 12 hours remain. A periodic token can continue to renew without reaching the fixed maximum lifetime that applies to an ordinary `-ttl` token. Alert on renewal failures and on prolonged admin-service downtime — the token only survives if something renews it inside each 30-day window.
>
> **If renewal has not happened, recovery depends on whether the token is still alive.** While it has any TTL left, authenticate as that token and renew self (do not pass the token as a positional argument, which instead asks the current client token to renew another token):
>
> ```bash
> read -rs -p "Enter scoped token: " SCOPED_TOKEN && echo
> VAULT_TOKEN="$SCOPED_TOKEN" VAULT_NAMESPACE=<D> bao token renew
> unset SCOPED_TOKEN
> ```
>
> `VAULT_NAMESPACE` matters: the token lives in the namespace you created in this section, and the admin service sends that namespace on its own renewal calls.
>
> Alternatively, use the admin service's `POST /api/admin/openbao/token/renew` endpoint. Renewal needs no special privilege — `renew-self` comes with the `default` policy the token already carries.
>
> Once the scoped token has fully expired, creating its replacement does need root, and §1.4 revokes root two steps from here. Start the port-forward from §1.2 again and leave it running:
>
> ```bash
> kubectl port-forward svc/openbao 8200:8200 -n <A>
> ```
>
> In a second terminal, point the CLI at that port-forward and clear any namespace or token inherited from earlier work. Then complete OpenBao's root-generation flow. The deployment uses a 1-of-1 unseal threshold, so you submit one unseal key:
>
> ```bash
> export VAULT_ADDR=http://127.0.0.1:8200
> unset VAULT_NAMESPACE VAULT_TOKEN
>
> bao operator generate-root -generate-otp
> bao operator generate-root -init -otp=<OTP_FROM_PREVIOUS_COMMAND>
> bao operator generate-root -nonce=<NONCE_FROM_INIT>
> bao operator generate-root \
>   -decode=<ENCODED_TOKEN_FROM_UNSEAL_STEP> \
>   -otp=<OTP_FROM_FIRST_COMMAND>
> ```
>
> The third command prompts for the unseal key and, because the threshold is one, returns the encoded token. The fourth command prints the decoded root token. Supply that root token explicitly while creating the replacement periodic token in the admin namespace:
>
> ```bash
> read -rs -p "Enter regenerated root token: " ROOT_TOKEN && echo
> VAULT_TOKEN="$ROOT_TOKEN" VAULT_NAMESPACE=<D> \
>   bao token create -policy=<D>-policy -orphan -period=720h
> ```
>
> Save the newly created scoped token, enter it at the next prompt, and confirm that it works in the admin namespace. Then revoke the regenerated root token immediately in the root namespace; the Kubernetes Secret update and rollout do not require it:
>
> ```bash
> read -rs -p "Enter newly created scoped token: " SCOPED_TOKEN && echo
> VAULT_TOKEN="$SCOPED_TOKEN" VAULT_NAMESPACE=<D> bao token lookup
> VAULT_TOKEN="$ROOT_TOKEN" bao token revoke -self
> unset ROOT_TOKEN
> ```
>
> Now update `OPENBAO_TOKEN` in the `admin-service-secrets` Secret with the newly created scoped token and restart the admin service. Clear the value from the shell once the Secret is updated:
>
> ```bash
> unset SCOPED_TOKEN
> ```
>
> Keep the unseal key retrievable — without it there is no path back from an expired token.

### 1.4 Revoke the root token

```bash
bao token revoke -self
rm -f ~/.vault-token
```

---

## Phase 2: PostgreSQL

Connect to your PostgreSQL instance as a superuser and run:

not.bot Verify uses two databases. The admin service and the signature server each create and migrate their own schema, so each owns its own database:

- **`<E1>`**, owned by `<F_USER>` — the admin service's data.
- **`<E2>`**, owned by `<G1_USER>` — the signature server's data.

The admin service also reads verification counts out of `<E2>` for the dashboard and for usage reporting. It does that as `<G2_USER>`, a read-only user, so it never holds write access to the signature server's data.

Below, `<F_USER>`/`<F>` are from Decision **F**, `<G1_USER>`/`<G1>` from Decision **G1**, and `<G2_USER>`/`<G2>` from Decision **G2**. The default names are `notbot_admin`, `notbot_signer`, and `notbot_signer_reader`; you may keep them or substitute your own.

```sql
CREATE USER <F_USER> WITH PASSWORD '<F>';
CREATE USER <G1_USER> WITH PASSWORD '<G1>';
CREATE USER <G2_USER> WITH PASSWORD '<G2>';

CREATE DATABASE <E1> OWNER <F_USER>;
CREATE DATABASE <E2> OWNER <G1_USER>;
```

Then give the admin service read-only access to the signature server's database:

```sql
\c <E2>
GRANT CONNECT ON DATABASE <E2> TO <G2_USER>;
GRANT USAGE ON SCHEMA public TO <G2_USER>;
ALTER DEFAULT PRIVILEGES FOR ROLE <G1_USER> IN SCHEMA public
  GRANT SELECT ON TABLES TO <G2_USER>;
```

> `ALTER DEFAULT PRIVILEGES` is what matters here, and it is deliberately not a one-off `GRANT SELECT ON ALL TABLES`. The signature server's tables do not exist yet — it creates them in Phase 8 — and it adds more over time as the product evolves. Default privileges apply to whatever `<G1_USER>` creates from now on, so the admin service keeps working across upgrades. A one-off grant would cover nothing today and silently miss every table added later.

No grants are needed in the other direction. The signature server only uses its own database, and the admin service owns everything in `<E1>`.

**Verify:** all three connections work.

```bash
psql -h <HOST> -U <F_USER>  -d <E1> -c "SELECT 1;"   # password F
psql -h <HOST> -U <G1_USER> -d <E2> -c "SELECT 1;"   # password G1
psql -h <HOST> -U <G2_USER> -d <E2> -c "SELECT 1;"   # password G2
```

---

## Phase 3: Keycloak

In your Keycloak admin console:

1. Create realm `admin-service`.
2. In that realm, create client `admin-service` with:
   - Client authentication: **ON**
   - Valid redirect URIs: `https://<H>/*`
   - Web origins: `https://<H>`
3. Open the client's Credentials tab and copy the client secret.
4. Create at least one user in the realm. Set **all** of: username, email, first name, last name, password. Mark **Email verified: ON** and **Temporary password: OFF**. Modern Keycloak realms require `email`, `firstName`, and `lastName` by default for any user with the `user` role; missing any of these causes the eventual login (Phase 6.1) to fail with a misleading `invalid_grant` / `Account is not fully set up` error.

Decision **H** is the admin UI hostname. The redirect URI must use the same hostname you will set in `app.baseUrl` and the admin-service ingress later.

---

## Phase 4: Chia node

The Chia node charts live in a public repo. Read `readme.md` in that repo for chart-level detail.

### 4.1 Generate the private CA

```bash
openssl genrsa -out private_ca.key 4096
openssl req -x509 -new -nodes -key private_ca.key -sha256 -days 3650 \
  -out private_ca.crt \
  -subj "/C=US/ST=Private/L=Private/O=Chia/OU=RPC/CN=Chia Private CA"
chmod 600 private_ca.key
chmod 644 private_ca.crt
```

Use the same CA and key on every Chia node. You will upload them again in Phase 6 when you register each node.

### 4.2 Build and push the node image

```bash
git clone https://github.com/GalactechsLLC/helm-chia-nodes.git
cd helm-chia-nodes
DOCKER_REPO="<YOUR_REGISTRY>/path" ./docker_build.sh
```

### 4.3 Create the environment values file

```bash
cd helm/chia-node
```

Create a new file named `values-<your-suffix>.yaml`. The suffix is yours to pick (commonly `my-mainnet`); the file does not exist in the repo — you are creating it. The README in the helm-chia-nodes repo documents the full values structure; the minimum overrides for not.bot Verify are:

```yaml
chia-blockchain:
  nameOverride: "chia-node"
  fullnameOverride: "chia-node"
  image:
    repository: "<YOUR_REGISTRY>/path"
    tag: "<YOUR_TAG>"
  chia:
    ca:
      private_ca_crt: |
        -----BEGIN CERTIFICATE-----
        ...contents of private_ca.crt...
        -----END CERTIFICATE-----
      private_ca_key: |
        -----BEGIN PRIVATE KEY-----
        ...contents of private_ca.key...
        -----END PRIVATE KEY-----
```

> **Block scalar indentation must be consistent.** Every line of the cert and key sits at the same indent under the field name. Wrong indentation produces a pod restart loop.

> **Volume size.** The chart's default `volume.size` is `500Gi`. See Operations & Reference Guide §5.6 for sizing context before changing this. Mainnet first-sync briefly co-locates a 121 GiB compressed checkpoint and the SQLite it extracts to — a smaller PVC will run out of space during extraction.

### 4.4 Create the namespace, update dependencies, diff, and deploy

The deploy scripts read `CI_NAMESPACE` from `helm/env`, which ships with `CI_NAMESPACE="infrastructure"`. Create that namespace before running the scripts. If you want a different namespace, edit the env file first and create whatever you chose instead.

> not.bot Verify operates on Chia mainnet only. The upstream `helm-chia-nodes` repo ships a separate `helm/env_testnet` file for testnet deploys — do not use it. Stick with `ENV_FILE=./env`.

```bash
kubectl create namespace infrastructure
cd ..   # back to helm/
CI_ENVIRONMENT=<your-suffix> ENV_FILE=./env ./update-chart.sh
CI_ENVIRONMENT=<your-suffix> ENV_FILE=./env ./diff.sh
CI_ENVIRONMENT=<your-suffix> ENV_FILE=./env ./upgrade.sh
```

> The upstream scripts include a `Is the Above Correct? [y/N]` confirmation prompt. If you run them over a non-interactive shell (SSH without `-t`, CI pipeline, etc.) they will hang waiting for input. Either run from an interactive terminal, or pipe `yes y` to each:
>
> ```bash
> yes y | CI_ENVIRONMENT=<your-suffix> ENV_FILE=./env ./upgrade.sh
> ```

**Verify:**

```bash
kubectl get pods -n infrastructure
kubectl logs -f <chia-node-pod> -n infrastructure
```

The pod reaches Running within a minute or two. A restart loop usually points at malformed CA YAML or a missing storage volume.

### 4.5 Wait for chain sync

The first deploy downloads a checkpoint from `torrents.chia.net` and walks forward to chain head. You cannot create the business DID or mint signature DIDs against an out-of-sync node. Continue deploying the admin service while this runs; sync only blocks the on-chain steps in Phase 6.

For redundancy, repeat steps 4.3 and 4.4 with a separate values file and release name to bring up additional nodes. Use the same private CA on each.

---

## Phase 5: Admin service

### 5.1 Extract and configure

```bash
cd helm/admin-service
```

Open `values.yaml`. Fill in these blocks. Sensitive values are supplied through a Kubernetes Secret you create in §5.2; the chart references that Secret with `*SecretKeyRef` fields and does not render plaintext production credentials into its own Secret.

```yaml
database:
  url: "jdbc:postgresql://<POSTGRES_HOST>:5432/<E1>"
  username: "<F_USER>"
  adminPasswordSecretKeyRef:
    name: "admin-service-secrets"
    key: "POSTGRES_PASSWORD"

signerDatabase:
  host: "<POSTGRES_HOST>"
  port: 5432
  database: "<E2>"
  username: "<G1_USER>"
  passwordSecretKeyRef:
    name: "admin-service-secrets"
    key: "SIGNER_DB_PASSWORD"
  readerUsername: "<G2_USER>"
  readerPasswordSecretKeyRef:
    name: "admin-service-secrets"
    key: "SIGNER_DB_READER_PASSWORD"

keycloak:
  issuerUri: "https://<KEYCLOAK_PUBLIC_HOSTNAME>/realms/admin-service"
  externalUrl: "https://<KEYCLOAK_PUBLIC_HOSTNAME>"
  clientId: "admin-service"
  clientSecretSecretKeyRef:
    name: "admin-service-secrets"
    key: "KEYCLOAK_CLIENT_SECRET"

openbao:
  enabled: true
  addr: "http://openbao.<A>.svc.cluster.local:8200"
  namespace: "<D>"
  tokenSecretKeyRef:
    name: "admin-service-secrets"
    key: "OPENBAO_TOKEN"

app:
  baseUrl: "https://<H>"
  environment: "production"
  springProfile: "prod"

juliaServer:
  host: "identity.julia.social"
  port: 443
  secure: true
  baseUrl: ""   # empty derives from host/port/secure
```

If you use a different Kubernetes Secret name or key names in §5.2, update the four `*SecretKeyRef` blocks above to match.

**TLS trust:** the admin service fetches Keycloak's JWKS over HTTPS, so the JVM must trust whatever CA signed your Keycloak's cert. Public CAs (Let's Encrypt, DigiCert, etc.) are already trusted — no action needed. **Internal CAs** (corporate PKI, self-signed for evaluation) require adding the CA's public cert to the chart's `extraCaCerts` map:

```yaml
extraCaCerts:
  my-internal-ca: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
```

The chart's init container builds a custom JVM trust store containing both the image's default cacerts and any entries you list here. Leave `extraCaCerts` empty if Keycloak's cert is from a public CA. See Appendix B for the single-VM evaluation flow.

> **Serve Keycloak at one public hostname and use it everywhere.** Both `issuerUri` and `externalUrl` should be that hostname over HTTPS (`https://keycloak.your-domain.example`, port included if it is not 443). Keycloak stamps the hostname it is started with (`KC_HOSTNAME`) into the `iss` claim of every token, and the admin service rejects a token whose `iss` does not equal `issuerUri` — so one hostname for the browser and the backend is the configuration that works. It does mean the admin pod must resolve and reach that public hostname from inside the cluster: in environments where a public hostname resolves to an address the cluster cannot hairpin to (AWS EC2 Elastic IPs are the common case), add an in-cluster DNS override so the name resolves to the Keycloak endpoint — Appendix B §B.12 has the recipe and a preflight probe that confirms both resolution and CA trust before you deploy.
>
> Confirm the configured value against the `issuer` field returned by `<issuerUri>/.well-known/openid-configuration`; the strings must match exactly.
>
> `issuerUri` must point at *your* Keycloak — Keycloak is a prerequisite you provide, not part of not.bot Verify. An internal `http://keycloak.<KEYCLOAK_NAMESPACE>.svc.cluster.local:8080/realms/admin-service` issuer only works if Keycloak also issues tokens with that exact `iss` claim (i.e. it is started with a matching `KC_HOSTNAME` and operators log in at the same URL); mixing an internal issuer with a public browser URL is the most common login failure.
>
> `externalUrl` is the URL the operator's browser is sent to during login, and must be HTTPS. If logins do fail with an issuer error, decode the JWT (e.g. jwt.io offline mode) and read its `iss` claim — that is what Keycloak is actually issuing, and what `issuerUri` has to equal.

### 5.2 Expose on the internal network

Operators reach the admin service through a `LoadBalancer` Service annotated for internal-only placement. The cloud provider provisions a private endpoint with a stable hostname; you point your internal DNS at it.

> **Single-VM / non-cloud deployments:** the cloud-LB annotations below don't apply. See Appendix B for the ingress-nginx + local CA path (set `service.type: ClusterIP` and define an Ingress).

Create the namespace and the Secret referenced by §5.1:

```bash
kubectl create namespace <B>
kubectl create secret generic admin-service-secrets -n <B> \
  --from-literal=POSTGRES_PASSWORD='<F>' \
  --from-literal=SIGNER_DB_PASSWORD='<G1>' \
  --from-literal=SIGNER_DB_READER_PASSWORD='<G2>' \
  --from-literal=KEYCLOAK_CLIENT_SECRET='<CLIENT_SECRET_FROM_PHASE_3>' \
  --from-literal=OPENBAO_TOKEN='<SCOPED_TOKEN_FROM_STEP_1.3>'
```

> The shipped admin-service image (`harbor.julia.social/library/admin-service`) is in Harbor's public `library` namespace and pulls anonymously. No `imagePullSecret` is needed for the mainline deployment. If you mirror the image into a private registry, create a docker-registry Secret in namespace **B** and set `imagePullSecret` in `values.yaml` to its name.

In `values.yaml`, set the service to `LoadBalancer` and add the internal-LB annotation for your cloud provider. Uncomment one:

```yaml
service:
  type: LoadBalancer
  annotations:
    # AWS:   service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
    # GKE:   networking.gke.io/load-balancer-type: "Internal"
    # Azure: service.beta.kubernetes.io/azure-load-balancer-internal: "true"
```

After install (step 5.3), get the LB's private hostname:

```bash
kubectl get svc admin-service -n <B>
```

Create a CNAME in your internal DNS pointing **H** at that hostname. The hostname must match `app.baseUrl` from step 5.1 and the Keycloak redirect URIs from Phase 3.

> TLS terminates at the LB. Add the certificate from Decision **J** to the `service.annotations` block above using your provider's annotation (`aws-load-balancer-ssl-cert` on AWS, managed cert resource on GCP, listener configuration on Azure). The admin service inside the pod serves plain HTTP on 8080.

### 5.3 Install and verify

```bash
cd ../..
helm install admin-service ./helm/admin-service -n <B>
```

**Verify:**

```bash
kubectl get pods -n <B> -l app.kubernetes.io/name=admin-service
kubectl logs -n <B> -l app.kubernetes.io/name=admin-service
```

Initial startup takes about 40 seconds. A pod stuck in `CrashLoopBackOff` traces back to one of the upstream dependencies; the log names which.

Browse to `https://<H>`. The Keycloak login page appears. Do not log in yet. Phase 6 covers first login.

---

## Phase 6: Business identity setup

Confirm before starting:

- At least one Chia node from Phase 4 reports synced.
- You have DNS control for the `domain` value in your `deployment-config.json`.
- You have an operator account in the Keycloak realm from Phase 3.

### 6.1 First login

Browse to `https://<H>`. Click Login. Authenticate through Keycloak with the operator account. The dashboard shows the deployment as unconfigured.

### 6.2 Register Chia nodes

Register your Chia node(s) before uploading `deployment-config.json`. The admin UI rejects the upload until at least one blockchain node is configured, because the upload triggers an on-chain transaction (business DID creation) that needs a node to broadcast through.

For each node from Phase 4, open Blockchain → Add Node and fill in:

- **Host or IP Address:** the in-cluster DNS name where the node's full_node RPC is reachable (e.g. `chia-node.infrastructure.svc.cluster.local`).
- **Port:** `8555` (Chia mainnet full_node RPC).
- **CA Certificate (PEM Format):** paste the contents of `private_ca.crt` from §4.1. Either paste the PEM block directly or use the "Upload .crt file" button.
- **Private Key (PEM Format):** paste the contents of `private_ca.key` from §4.1. This is the **CA's private key**, not a client key. The admin service uses the CA cert+key as material to mint short-lived client certificates when it calls the chia node's RPC, and to delegate the same capability to your signature servers later. Treat this file with the same care as any private CA key.

The form also shows a `Network` row (read-only display, always "Mainnet" — not.bot Verify is mainnet-only) and a `Use Secure Connection (SSL/TLS)` checkbox that is pre-checked and disabled (the chia full_node RPC is mTLS-only, so SSL is mandatory). Neither requires operator action; the CA Certificate and Private Key fields below are what you actually fill in.

> The chia full_node TLS certificate's only Subject Alternative Name is `DNS:chia.net` — a Chia design convention. The admin service handles this internally (overriding SNI to `chia.net` regardless of the Host you provide above). Generic HTTP clients connecting to chia full_node RPC would need to do the same; you do not need to configure anything for the admin service itself.

Wait until at least one node shows "synced" in the admin UI before continuing. The on-chain transaction in §6.3 requires a synced node. See Operations & Reference Guide §5.2 for what "synced" means and how long the first sync takes.

### 6.3 Upload deployment-config.json

In the admin interface, upload your `deployment-config.json`. The admin service contacts the billing server to register the deployment, using the API key from the file.

The API key is single-use: it is redeemed when registration **succeeds**. If the upload fails, the key remains valid — fix the cause and upload the same file again. Once registration succeeds, that key is spent and the file cannot be uploaded a second time.

Common upload failures and what they mean:

- **The cluster cannot reach `billingServerUrl` over HTTPS.** Fix egress, then upload again.
- **`customerId` or `organizationName` does not match Julia Social's records.** Contact `support@julia.social` — your welcome email may have been generated incorrectly.
- **"You must register at least one blockchain node before uploading deployment-config.json"** (or similar). You skipped §6.2 — go back and register a Chia node first, then re-upload.

If an upload fails and re-uploading the same file then reports that the key has already been used, stop and email `support@julia.social` for a re-issued welcome email rather than working around it.

### 6.4 Verify the business DID appeared

The §6.3 upload triggers business DID creation automatically — the admin service generates a BLS12-381 keypair in OpenBao and submits the DID transaction through a registered node. You don't click anything; the work happens in the background.

Open Business Identity in the admin UI and confirm the business DID is listed. It typically appears within a couple of minutes of the upload completing. If it's not visible after five minutes, check the admin-service logs for the on-chain transaction submission:

```bash
kubectl logs -n <B> -l app.kubernetes.io/name=admin-service --since=30m | grep -i "business.*did"
```

Common reasons the DID does not appear: the registered Chia node fell out of sync between §6.2 and §6.3 (check Blockchain → node status), OpenBao sealed or unreachable (check `kubectl get pods -n <A>`; note `bao secrets list` returns 403 with the §1.3 scoped token — it needs `sys/mounts`. Use `bao token capabilities chiakeys/business_key` instead, which should report the §1.3 policy grants, or regenerate a root token per Operations & Reference Guide §2.4), or the admin service failed to write through the Vault path (look for `permission denied` in the logs — likely a policy or KV-v2 mount issue from §1.3).

> **Do not start §6.5 until the DID is visible here.** Domain verification signs with the business DID on-chain; starting it before the DID transaction confirms (~2 minutes) fails with an unhelpful HTTP 500 in current builds.

### 6.5 Verify the domain

The admin interface displays a "Verify Domain" button. Clicking it opens a modal that walks through the DNS step. Under "Step 1: Add DNS Record" the modal shows the complete TXT record value (the `julia-did=` prefix plus the full DID) with a copy button — paste that value into a TXT record at the apex of your domain. Wait for propagation. Confirm with:

```bash
dig +short TXT <your-domain>
```

The output starts with `julia-did=`. Click Verify Domain in the admin interface.

The check itself runs upstream: Julia's identity service polls your domain's TXT records and the admin service reports its status. Expect **up to ~30 minutes** between the record becoming globally resolvable and the status flipping to verified — this is upstream polling cadence, not propagation; there is nothing to fix on your side while you wait.

If verification still fails after that: check the value is character-for-character identical (some DNS UIs add quotes or split long values), then retry.

Leave the TXT record in place. Multiple `julia-did=` TXT records (for example, one left over from an earlier deployment) are tolerated — verification matches yours among them, so there is no need to hunt down and delete old records.

### 6.6 Optional: driving Phase 6 from the command line

Every Phase 6 step has an admin REST endpoint, so the whole phase can be scripted (useful for repeatable or headless installs). Authentication is the same OAuth2 session the browser uses — Keycloak bearer tokens are **not** accepted — but the authorization-code flow works headlessly against the Keycloak login form with an ordinary HTTP client: fetch `https://<H>/oauth2/authorization/keycloak` with a cookie jar, POST the operator credentials to the returned Keycloak form action, and follow redirects back to `https://<H>` — the session cookie in the jar now authenticates `/api/admin/*` calls.

With that session (all URLs relative to `https://<H>`):

| Step | Call |
|---|---|
| §6.2 register node | `POST /api/admin/blockchain-nodes` — JSON body with host, port, CA cert + key PEMs |
| §6.3 upload config | `POST /api/admin/config/upload` — multipart, field `file` = `deployment-config.json` |
| §6.4 confirm DID | `GET /api/admin/business-details` — DID present once created |
| §6.5 start verification | `POST /api/admin/domain-verification/start` |
| §6.5 poll status | `POST /api/admin/domain-verification/check-status` |
| §7 check pool | `GET /api/admin/signature-servers` |

The TXT record value for §6.5 is `julia-did=` followed by the `did` field from `/api/admin/business-details` (the UI modal composes this for you; no API returns the composed string).

---

## Phase 7: Signature DID pool

The admin service maintains a pool of unallocated signature DIDs (target of three) in the background. DIDs are minted via Julia's identity service and recorded on chain. The pool refills automatically as signature servers consume DIDs at startup; you do not provision the pool manually.

The pool only begins populating **after domain verification (§6.5) succeeds** — it is always empty at the §6.3 checkpoint, which is expected.

**Verify:** open Signature DIDs in the admin UI and confirm at least one DID shows status **Available**. Headless, `GET /api/admin/signature-servers` returns the same list: each DID with a `status` of `Available`, `Confirming`, or `In-use`, and an `is_available` flag matching what a signature-server setup call can claim. A DID reads `Confirming` for about two minutes after minting, while its on-chain lineage confirms.

The admin service mints the pool in the background and logs each DID as it lands:

```bash
kubectl logs -n <B> -l app.kubernetes.io/name=admin-service --since=30m | grep -i "signature DID"
```

Expect `Auto-creating 3 default signature DIDs after deployment (async)`, then a `Signature DID creation complete | id=… | did=…` and a `Default signature DID 1/3 created successfully` (through `3/3`) for each one.

> **Select this pod's logs by time, not by line count.** Minting is on-chain work, and the admin service's native Chia client logs at TRACE while it runs, so the minute your pool fills can carry tens of thousands of lines. The `--since=<duration>` above matters for that reason. A `--tail=<n>` you would think generous reaches back through twenty minutes of a quiet pod but a couple of seconds of the minting burst, and the grep returns nothing. Operations & Reference Guide §8.2 covers the same point for later troubleshooting.

If you see zero DIDs more than five minutes after §6.5 completes, check the admin-service logs for Julia identity service errors (look for `POST request to Julia service endpoint: /new_signature_did` or similar). If your admin UI build still exposes an **Add Signature DID** button and the pool has not populated automatically, click it once as an interim fallback, then confirm the new DID appears as available.

---

## Phase 8: Signature servers

### 8.1 Configure

```bash
cd helm/signer-service
```

Edit `values.yaml`:

The signer needs a small bootstrap config — enough to contact the admin service at startup, after which it receives the rest of its config (DID assignment, OpenBao token, database credentials, Chia node info with TLS material) from the admin's `/admin/signatureServerSetup` response.

The signer chart contains no templates of its own — it is a thin wrapper around a `deployment` dependency chart (declared in `requirements.yaml`). The consequence for you: all runtime environment is passed as a list of env-var entries under `deployment.extraEnvs` — there are no purpose-specific keys like `adminService` or `apiKeySecret`. Leave the `image` block exactly as shipped (repository, tag, and pull settings are pinned and tested together) — it is omitted from the excerpt below for that reason; everything you add goes under `extraEnvs` and `ingress`. The package's `values-example.yaml` shows the same shape with a worked example. Configure `values.yaml`:

```yaml
deployment:
  name: julia-signer
  replicaCount: 2
  # image: — not shown; leave the shipped block untouched (see the note above)
  service:
    enabled: true
    name: http
    port: 8080
    targetPort: 8080
  extraEnvs:
    - name: ADMIN_HOSTNAME
      value: "admin-service.<B>.svc.cluster.local"
    - name: ADMIN_PORT
      value: "8080"
    - name: ADMIN_SECURE
      value: "false"
    - name: CHIA_NETWORK
      value: "Mainnet"
    - name: RUST_LOG
      value: "info"
    - name: ADMIN_API_KEY
      valueFrom:
        secretKeyRef:
          name: julia-signer-secrets
          key: API_KEY
  ingress:
    enabled: true
    ingressClassName: nginx
    pathType: Prefix
    hosts:
      - host: <I>
        paths:
          - /
    tls:
      - hosts:
          - <I>
        secretName: signer-tls
```

The signer binds `0.0.0.0:8080` by default, matching the service `targetPort` above. Leave the listener alone unless you have a reason to move it.

**Retrieve the API key value from the admin database.** The admin service generates a `businesses.api_key` row when you complete §6.3. There is no admin-UI surface for this value today, so you must query Postgres directly:

```bash
psql -h <POSTGRES_HOST> -U <F_USER> -d <E1> -c "SELECT api_key FROM businesses LIMIT 1;"
```

The value begins with `ltk_`. Create the signer Secret in namespace **C** with that value:

```bash
kubectl create secret generic julia-signer-secrets \
  --from-literal=API_KEY='<businesses.api_key value, begins with ltk_>' \
  -n <C>
```

The `valueFrom` entry in the values above reads that Secret key into `ADMIN_API_KEY`, which the signer presents to the admin service. (The Secret's key is named `API_KEY` and the env var it feeds is `ADMIN_API_KEY` — that is correct, not a leftover; keep the key name as written so the `secretKeyRef` above resolves.) This direct database query is an interim bootstrap path; a future release will move this credential to OpenBao-mediated setup so operators do not handle it.

The signer Ingress terminates TLS at the cluster edge using a pre-created Kubernetes TLS Secret. Create the Secret in the signer namespace (**C**) from Decision **K**:

```bash
kubectl create secret tls signer-tls \
  --cert=<path-to-K-cert> \
  --key=<path-to-K-key> \
  -n <C>
```

For cloud-managed certs (ACM, GCP managed cert) you have an alternative path: terminate TLS at a cloud LB above the Ingress and leave `tls: []` empty here. Decision **K**'s answer dictates which path to use.

> **Do NOT set `CHIA_FULL_NODE`, `CHIA_FULL_NODE_PORT`, `CHIA_FULL_NODE_SSL`, `PRIVATE_CA_CRT`, or `PRIVATE_CA_KEY` as overrides.** The binary populates these itself from the admin response after startup. Setting `CHIA_FULL_NODE_SSL` in particular switches the TLS path from inline-PEM-from-admin (correct) to filesystem-paths (broken — the binary fails with `No such file or directory` looking for cert files that aren't there).

### 8.2 Install

```bash
kubectl create namespace <C>
cd ../..
helm dependency update ./helm/signer-service
helm install julia-signer ./helm/signer-service -n <C>
```

The `helm dependency update` step is required: the signer chart's templates come entirely from its dependency chart (§8.1), and `helm install` fails with `found in requirements.yaml, but missing in charts/ directory` until the dependency has been fetched.

> The shipped signer image (`harbor.julia.social/library/julia_signature_server`) is in Harbor's public `library` namespace and pulls anonymously. No `imagePullSecret` is needed.

Point internal DNS for **I** at your nginx ingress controller's internal address. This ingress is internal-only, reached by your SDK adapter, the same posture as the admin LB in Phase 5. It is not exposed to the public internet.

> **Single-VM / non-cloud deployments:** use the local CA-signed cert from Appendix B §B.8 to create the `signer-tls` Secret in namespace **C** (the cert from §B.8 already covers Decision **I**'s hostname). Run the `kubectl create secret tls` command from §8.1 above, pointing `--cert` and `--key` at `/etc/verify-test/certs/server.crt` and `server.key` respectively.

### 8.3 Verify

```bash
kubectl get pods -n <C> -l app.kubernetes.io/name=julia-signer
kubectl logs -n <C> -l app.kubernetes.io/name=julia-signer
```

The startup log shows DID assignment, OpenBao connection, database connection, domain credential delegation, and honest.bot™ credential acquisition. A failed step names itself. If the admin side is not finished setting up (domain not yet verified, no assignable DID), the signer reports an explicit not-ready error and retries — that resolves on its own once §6.5 is complete and a DID is available. Common causes of a signer that does not start:

- No available DIDs in the pool (`503 "No signature DID is available to assign"`). The admin service mints to maintain the pool; if minting is failing, check that at least one Chia node is synced and OpenBao is unsealed.
- OpenBao sealed or unreachable. Unseal, check NetworkPolicies.
- Admin service unreachable from the signer namespace. Check service DNS and policies.
- Cannot reach Julia Social. Check egress.

In the admin interface, Signature DIDs → the DID this server picked up shows "Assigned" with a recent heartbeat timestamp.

---

## Phase 9: SDK integration

The SDK runs in your application backend, not in the cluster. Languages: Rust, JavaScript (Express), Python (FastAPI), Java (Spring MVC), Dart (shelf). Repo: [julia-social/julia_web_sdk](https://github.com/julia-social/julia_web_sdk).

Set three environment variables on the host running your backend:

```bash
SIGNATURE_HOSTNAME=<I>
SIGNATURE_PORT=443
SIGNATURE_API_KEY='<businesses.api_key value retrieved in §8.1>'
```

`SIGNATURE_API_KEY` is the same `businesses.api_key` value you placed in the `julia-signer-secrets` Secret in §8.1. The SDK sends it to the signature server in the `api-key` header. Do not use the one-time onboarding `apiKey` from `deployment-config.json`; that credential was consumed during §6.3 and is not the signature API key.

> `SIGNATURE_HOSTNAME` must be the signature server load balancer (Decision **I**), not the host running the SDK adapter. Pointing both at the same host loops the WebSocket proxy back on itself.

Build the adapter for your framework following the SDK readme. Configure `request_claims`, `require_site_pass`, `message_generator`, `on_success`, `on_failure`, and `expire_time`. Mount the adapter's router into your web application.

**Verify:** call `GET /signature/notbot` from your frontend. The adapter returns a request ID. Build a universal link from it, scan with the not.bot app, complete a verification. Your `on_success` callback fires with the verified response.

If that call comes back `404 Not Found`, check the key before you check the route: the signature server answers a caller it does not accept with `404` rather than an authentication error (Operations & Reference Guide §6.7).

---

## Done

Your deployment is complete. The Operations & Reference Guide covers monitoring, alerting, scaling, failure modes, and troubleshooting for each component.

---

## Appendix A: Rebuilding the OpenBao image from source

The chart's default image (`harbor.julia.social/library/openbao-bls`, pinned by digest) ships with the Chiakeys plugin (`vault-plugin-secrets-bls`) baked in. The mainline deployment uses it directly.

This appendix is for customers who need to rebuild the image from source — for example, to audit the plugin's provenance against a supply-chain review, or to mirror the image into an internal registry.

### A.1 Build environment

You need:

- Docker with buildx (`docker buildx version`)
- Outbound access to `github.com` (to clone the plugin) and `docker.io` (to pull `openbao/openbao:2.5.2` and `golang:1.22-alpine`)

You do **not** need Go or a C toolchain on your host. The Dockerfile below compiles the plugin inside an Alpine container so the resulting binary links against musl libc (matching the OpenBao runtime image) and uses a pinned Go version. Building on a glibc host (Ubuntu, Debian, RHEL) and copying the binary into the Alpine runtime image produces a binary that fails to load.

### A.2 Multi-stage Dockerfile

Create a file named `Dockerfile`:

```dockerfile
# Build stage — Alpine + pinned Go + C toolchain (cgo for blst)
FROM golang:1.22-alpine AS build
RUN apk add --no-cache git build-base
WORKDIR /src
RUN git clone https://github.com/mintgarden-io/vault-plugin-secrets-bls.git .
WORKDIR /src/cmd/vault-plugin-secrets-bls
ENV CGO_ENABLED=1
RUN go build -o /out/vault-plugin-secrets-bls

# Runtime stage — OpenBao with the plugin baked in
FROM openbao/openbao:2.5.2
COPY --from=build /out/vault-plugin-secrets-bls /openbao/plugins/
```

Build and push to your registry:

```bash
docker buildx build --platform linux/amd64 -t <YOUR_REGISTRY>/openbao-bls:<YOUR_TAG> --push .
```

Compute the plugin SHA-256 from inside the image (the chart's standalone config validates against this value):

```bash
docker run --rm --entrypoint sha256sum \
  <YOUR_REGISTRY>/openbao-bls:<YOUR_TAG> \
  /openbao/plugins/vault-plugin-secrets-bls
```

### A.3 Use your image in the chart

Edit `helm/openBao/values.yaml` to point at your image and update the plugin SHA in the standalone config:

```yaml
openbao:
  server:
    image:
      registry: <YOUR_REGISTRY>
      repository: openbao-bls
      tag: "<YOUR_TAG>"
    standalone:
      enabled: true
      config: |
        ...
        plugin "secret" "bls" {
          command     = "vault-plugin-secrets-bls"
          binary_name = "vault-plugin-secrets-bls"
          sha256sum   = "<SHA_FROM_A.2>"
        }
```

Verify with `helm template helm/openBao` before installing, then continue from §1.1.

---

## Appendix B: Single-VM evaluation deployment

The mainline Preflight and Deployment Checklist assume you already operate a Kubernetes cluster with capacity, a managed PostgreSQL, a Keycloak you log into, a private container registry, real DNS, and a working TLS story. Most evaluators do not have that on day one — they want to spin up the whole stack on a single machine to see what they're buying. This appendix is the recipe for that scenario.

This is not a production deployment. The whole point of running everything on one host is that it's cheap to throw away and rebuild. You'll lose the deployment if the host dies, you have no HA story, OpenBao reseals on every host reboot and nobody else can unseal it, and you're trusting a self-signed CA you generated yourself. None of those are appropriate for production, but all of them are fine for "does this thing work for my use case?".

When you decide to go further, swap components one at a time — PostgreSQL is the easiest first move (point an ExternalName Service at managed Postgres instead of the local container), then Keycloak (your enterprise IAM), then DNS and TLS (real records and a real CA), then the cluster itself.

### B.1 Host sizing

A single VM running every component plus its prereqs needs:

- **8 vCPU.** Chia node wants 4 vCPU with headroom (it has to keep up with chain head; a node that falls behind takes the verification flow offline until it resyncs). The admin service is light (~250m). The signature server uses 500m–1 vCPU under MPC load. OpenBao, ingress-nginx, MetalLB, kubelet, containerd, Postgres, Keycloak fill the rest. 8 vCPU leaves headroom; 4 vCPU will run but get tight under load.
- **32 GiB RAM.** Chia is the largest consumer and commonly needs several GiB during mainnet operation; leave headroom for sync, cache, and restarts. Keycloak likes 2 GiB. Postgres at this scale is 1 GiB. Admin, signer, OpenBao together are another 2–3 GiB. KIND control plane plus system overhead consumes the rest. A 16 GiB host can run a test rig, but 32 GiB leaves enough margin that one component's transient memory use does not destabilize the node.
- **600 GiB SSD.** The Chia node's 500Gi PVC dominates this, and the same disk carries the OS, container images, Postgres, and Keycloak underneath it. Inside the PVC, the chain database settles around 215 GiB after first sync and grows 5–10 GiB/month, with a transient peak near 310 GiB during checkpoint extraction. 600 GiB leaves the node its full PVC and the host room to work.

**AWS reference:** `t3.2xlarge` (8 vCPU / 32 GiB) with a 600 GiB gp3 EBS volume. For better Chia walk-forward throughput, bump gp3 IOPS to 6000 and throughput to 500 MiB/s during the first sync, then drop back to defaults afterward.

If you want to skip the Chia node entirely (and ship the deployment without on-chain operations — see Architecture & Privacy Guide for what that costs you), you can run on 4 vCPU / 16 GiB / 100 GiB. The other components are not the heavyweight ones.

### B.2 Operating system and base packages

Ubuntu 22.04 LTS is the reference. Other Debian-derived distros work the same way; RHEL-family distros need the obvious package-manager substitutions.

```bash
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg jq unzip openssl postgresql-client
```

`postgresql-client` is for your convenience — `psql` from the host to talk to the Postgres container in §B.9.

### B.3 Docker

Install Docker CE with buildx and the compose plugin:

```bash
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg]" \
     "https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io \
                        docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker "$USER"   # log out + back in to pick this up
```

### B.4 Kubernetes tooling

Install `kubectl`, `helm`, and the `helm-diff` plugin per Preflight items #1, #2, #3:

```bash
# kubectl (latest stable in the v1.32 channel)
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg]" \
     "https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /" \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list >/dev/null
sudo apt-get update
sudo apt-get install -y kubectl

# Helm 3
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# helm-diff plugin (chia-node deploy scripts need it)
helm plugin install https://github.com/databus23/helm-diff
```

Install the OpenBao CLI (Preflight #9) by downloading the matching linux-amd64 binary from the latest [openbao/openbao GitHub release](https://github.com/openbao/openbao/releases) and putting it on `PATH` as `/usr/local/bin/bao`.

### B.5 KIND cluster

KIND runs a Kubernetes cluster as a set of Docker containers on the host. For a single-VM deployment you want a single control-plane node with the host's port 80 and 443 mapped into the node — that's how ingress traffic from outside the VM reaches the cluster.

```bash
# Install KIND
curl -fsSL https://kind.sigs.k8s.io/dl/v0.30.0/kind-linux-amd64 -o /tmp/kind
sudo install -m 0755 /tmp/kind /usr/local/bin/kind
rm /tmp/kind

# Cluster config with port mappings for ingress
cat > /tmp/kind-config.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
EOF

kind create cluster --name notbot-test --config /tmp/kind-config.yaml --wait 120s
kubectl config use-context kind-notbot-test
kubectl cluster-info
```

`extraPortMappings` is what makes the cluster reachable from outside the VM. The ingress controller you install in §B.7 will bind to ports 80/443 inside the control-plane node, and KIND forwards from the host's 80/443 to those.

### B.6 MetalLB for LoadBalancer Services

KIND has no cloud LB integration. Services of type `LoadBalancer` stay in `Pending` forever unless you provide an LB controller. MetalLB is the standard in-cluster answer; it picks an IP from a pool you configure and announces it via L2 (ARP) to the cluster's network.

For this deployment, MetalLB's role is narrower than you might expect — most external traffic enters through ingress-nginx (§B.7) on the host's port 80/443 mappings, not through a LoadBalancer IP. MetalLB is here so any chart that defaults to `LoadBalancer` (the admin service does, for one) gets a working IP instead of hanging in Pending.

```bash
METALLB_BASE=https://raw.githubusercontent.com/metallb/metallb/v0.14.9
kubectl apply -f "$METALLB_BASE/config/manifests/metallb-native.yaml"
kubectl -n metallb-system wait --for=condition=Available deployment/controller --timeout=300s
kubectl -n metallb-system rollout status daemonset/speaker --timeout=300s

# Derive an IP pool from KIND's docker network so MetalLB IPs are routable inside the host
DOCKER_SUBNET=$(docker network inspect kind \
  | python3 -c 'import sys, json
cfg = json.load(sys.stdin)[0]["IPAM"]["Config"]
print(next(c["Subnet"] for c in cfg if "." in c["Subnet"]))')
PREFIX=$(echo "$DOCKER_SUBNET" | cut -d. -f1-2)
cat <<EOF | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata: { name: notbot-pool, namespace: metallb-system }
spec:
  addresses: [ "${PREFIX}.255.200-${PREFIX}.255.250" ]
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata: { name: notbot-l2adv, namespace: metallb-system }
spec:
  ipAddressPools: [ notbot-pool ]
EOF
```

### B.7 ingress-nginx for HTTPS termination

In a cloud deployment, your cloud's LB terminates TLS for you. KIND has no LB, so you terminate TLS yourself with an in-cluster ingress controller. ingress-nginx is the most widely-used; any controller with comparable feature support works.

Use the KIND-recommended manifests, which configure ingress-nginx to bind directly to the control-plane node's ports 80 and 443 via `hostPort`. Combined with the KIND `extraPortMappings` from §B.5, that means traffic to `http(s)://<your-host-public-IP>/...` lands on the ingress controller.

```bash
NGINX_BASE=https://raw.githubusercontent.com/kubernetes/ingress-nginx
NGINX_MANIFEST=controller-v1.11.3/deploy/static/provider/kind/deploy.yaml
kubectl apply -f "$NGINX_BASE/$NGINX_MANIFEST"
kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=300s
```

Once ingress-nginx is up, any Service you expose via an `Ingress` resource with a `tls:` block gets HTTPS termination using the TLS Secret you reference in that Ingress. You'll create the TLS Secret in §B.8.

**For production:** in a real cluster with a cloud LB, you don't need ingress-nginx unless your application requires its specific features. The admin-service and signer-service charts both work with a cloud LB doing TLS at L7 (or L4 with the pod doing TLS). Appendix B is the only place that talks about ingress-nginx.

### B.8 Local CA and TLS certificates

For evaluation you generate a CA and use it to issue server certs for every hostname you want to reach over HTTPS. The CA's only job is to sign one cert (or a few). You install the CA's public cert in your OS trust store and in your browser's trust store so the cert chain validates.

```bash
sudo mkdir -p /etc/verify-test/{ca,certs}
sudo chmod 700 /etc/verify-test/ca

# CA key + cert
sudo openssl genrsa -out /etc/verify-test/ca/ca.key 4096
sudo openssl req -x509 -new -nodes -key /etc/verify-test/ca/ca.key -sha256 -days 3650 \
  -out /etc/verify-test/ca/ca.crt \
  -subj "/CN=Verify Eval CA"

# Server key + multi-SAN cert
SERVER_NAMES="DNS:admin.example.com,DNS:signer.example.com,DNS:keycloak.example.com,DNS:example.com"
sudo openssl genrsa -out /etc/verify-test/certs/server.key 4096
sudo openssl req -new -key /etc/verify-test/certs/server.key \
  -out /etc/verify-test/certs/server.csr \
  -subj "/CN=example.com" \
  -addext "subjectAltName=${SERVER_NAMES}"
sudo openssl x509 -req -in /etc/verify-test/certs/server.csr \
  -CA /etc/verify-test/ca/ca.crt -CAkey /etc/verify-test/ca/ca.key -CAcreateserial \
  -out /etc/verify-test/certs/server.crt -days 365 -sha256 \
  -extfile <(printf "subjectAltName=%s" "$SERVER_NAMES")
```

Replace `example.com` with whatever domain you control. The cert covers the apex and three subdomains — admin (Decision **H**), signer (Decision **I**), and Keycloak. All four DNS names resolve to the host's public IP.

Install the CA in the host's trust store so `curl` works without `-k`:

```bash
sudo cp /etc/verify-test/ca/ca.crt /usr/local/share/ca-certificates/verify-eval-ca.crt
sudo update-ca-certificates
```

You also need to install the CA in your laptop's browser trust store. Each browser has its own trust store; consult your browser's documentation. Firefox and Chrome both let you import a CA cert as "trusted to identify websites."

> **`.app` and other HSTS preload TLDs.** Domains under `.app`, `.dev`, `.bank`, and a handful of others are in the Chromium HSTS preload list, which means the browser refuses to bypass an invalid cert warning at all — even with the CA installed, you must present a cert that validates cleanly. For evaluation, prefer a domain on a non-preloaded TLD (`.com`, `.io`, `.test`) or accept that you must get the local-CA install right before the browser will load anything.

### B.9 PostgreSQL

Run Postgres as a Docker container on the KIND network. This puts it in DNS reach of the cluster pods at the container's name, and exposes it on the host's `127.0.0.1:5432` for your own `psql` use.

```bash
POSTGRES_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=' | head -c 24)
echo "POSTGRES_PASSWORD=$POSTGRES_PASSWORD" >> ~/.verify-test-secrets

sudo mkdir -p /var/lib/postgresql-data
sudo chown 999:999 /var/lib/postgresql-data   # Postgres container's UID
sudo chmod 700 /var/lib/postgresql-data

docker run -d \
  --name postgres-verify \
  --network kind \
  --restart unless-stopped \
  -p 127.0.0.1:5432:5432 \
  -v /var/lib/postgresql-data:/var/lib/postgresql/data \
  -e POSTGRES_PASSWORD="$POSTGRES_PASSWORD" \
  postgres:16
```

Wait for it to be ready, then create the `keycloak` database:

```bash
KEYCLOAK_DB_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=' | head -c 24)
echo "KEYCLOAK_DB_PASSWORD=$KEYCLOAK_DB_PASSWORD" >> ~/.verify-test-secrets

docker exec -i -e PGPASSWORD="$POSTGRES_PASSWORD" postgres-verify \
  psql -U postgres -v ON_ERROR_STOP=1 -v kc="$KEYCLOAK_DB_PASSWORD" <<'SQL'
CREATE DATABASE keycloak;
CREATE ROLE keycloak LOGIN;
SELECT format('ALTER ROLE keycloak WITH LOGIN PASSWORD %L', :'kc')\gexec
GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak;
\c keycloak
GRANT ALL ON SCHEMA public TO keycloak;
ALTER SCHEMA public OWNER TO keycloak;
SQL
```

Expose Postgres into the cluster by DNS name with an ExternalName Service. Pods will resolve `postgres.default.svc.cluster.local` to the container:

```bash
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata: { name: postgres, namespace: default }
spec:
  type: ExternalName
  externalName: postgres-verify
  ports: [ { port: 5432, targetPort: 5432 } ]
EOF
```

The `verify` database and its users (Decisions **F** and **G**) are created later by the mainline Deployment Checklist §2.

**For production:** swap to managed Postgres (RDS, Aurora, CloudSQL, Azure Database). Update the ExternalName Service to point at its endpoint, or have your admin-service `values.yaml` reference the managed hostname directly. The schema and users from DC §2 work identically.

### B.10 Keycloak

Run Keycloak as a Docker container on the KIND network, backed by the Postgres container from §B.9.

```bash
KC_HOSTNAME="keycloak.example.com"   # match your DNS / TLS cert
KC_ADMIN_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=' | head -c 24)
echo "KC_ADMIN_PASSWORD=$KC_ADMIN_PASSWORD" >> ~/.verify-test-secrets

docker run -d \
  --name keycloak \
  --network kind \
  --restart unless-stopped \
  -p 8080:8080 \
  -e KC_DB=postgres \
  -e KC_DB_URL="jdbc:postgresql://postgres-verify:5432/keycloak" \
  -e KC_DB_USERNAME=keycloak \
  -e KC_DB_PASSWORD="$KEYCLOAK_DB_PASSWORD" \
  -e KC_HOSTNAME="https://${KC_HOSTNAME}" \
  -e KC_HOSTNAME_STRICT=false \
  -e KC_HTTP_ENABLED=true \
  -e KC_PROXY_HEADERS=xforwarded \
  -e KEYCLOAK_ADMIN=admin \
  -e KEYCLOAK_ADMIN_PASSWORD="$KC_ADMIN_PASSWORD" \
  quay.io/keycloak/keycloak:25.0.6 \
  start-dev
```

Expose Keycloak into the cluster with another ExternalName Service:

```bash
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata: { name: keycloak, namespace: default }
spec:
  type: ExternalName
  externalName: keycloak
  ports: [ { port: 8080, targetPort: 8080 } ]
EOF
```

Expose Keycloak to the browser through ingress-nginx with TLS termination from your local CA. First load the cert into a Secret in the namespace where the Ingress lives:

```bash
kubectl create secret tls verify-eval-tls \
  --cert=/etc/verify-test/certs/server.crt \
  --key=/etc/verify-test/certs/server.key \
  --namespace default
```

Then point an Ingress at Keycloak:

```bash
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: keycloak
  namespace: default
spec:
  ingressClassName: nginx
  tls:
    - hosts: [ ${KC_HOSTNAME} ]
      secretName: verify-eval-tls
  rules:
    - host: ${KC_HOSTNAME}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: keycloak
                port: { number: 8080 }
EOF
```

Browse to `https://${KC_HOSTNAME}/admin` and log in with `admin` / `$KC_ADMIN_PASSWORD`. This is the Keycloak admin console you'll use during Deployment Checklist §3 to create the `admin-service` realm.

**For production:** swap Keycloak for whatever IAM speaks OIDC. The admin service only needs an issuer URL, a client with a known secret, and a redirect URI. Any OIDC-compliant IdP (Okta, Auth0, Azure AD, Google Workspace) works.

### B.11 Local container registry (optional)

You can skip this section. The shipped charts pull from `harbor.julia.social/library/*`, which is anonymously readable; the only reason to run a local registry is to push your own images (Appendix A rebuilds) and pull them without round-tripping to the internet, or to mirror Harbor images for an air-gapped test.

```bash
docker run -d --name registry-verify --network kind --restart unless-stopped \
  -p 127.0.0.1:5000:5000 registry:2
```

Tell KIND's containerd to trust this insecure registry (HTTP, no TLS):

```bash
docker exec notbot-test-control-plane bash -c '
cat >> /etc/containerd/config.toml <<EOF

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"
EOF
mkdir -p /etc/containerd/certs.d/registry-verify:5000
cat > /etc/containerd/certs.d/registry-verify:5000/hosts.toml <<EOF
[host."http://registry-verify:5000"]
EOF
systemctl restart containerd
'
```

Pull, retag, and push from the host as `127.0.0.1:5000/...`; pull from inside the cluster as `registry-verify:5000/...`.

### B.12 DNS and routing

Point real DNS records at the VM's public IP for every hostname your local cert covers (the apex plus admin / signer / keycloak subdomains in the §B.8 example). Provider doesn't matter — anything that lets you set A records works.

In-cluster traffic to Postgres uses the ExternalName Service from §B.9 (`postgres.default.svc.cluster.local`). Keycloak is the exception: the admin service must reach it at the **same public hostname operators log in at** (§5.1 — the issuer in every token is that hostname), so the cluster does have to resolve that one name internally. The `keycloak.default.svc.cluster.local` ExternalName from §B.10 is a convenience for probes and scripts, not the address to configure as `issuerUri`.

If you want to evaluate without touching DNS, the alternative is to add entries to `/etc/hosts` on the operator's laptop pointing each hostname at the VM's public IP. That bypasses DNS but every laptop you test from needs the same hosts entries.

**AWS EC2 evaluators:** AWS doesn't NAT-loopback an instance's own Elastic IP by default. Commands from the EC2 host targeting hostnames that resolve to the EC2's own EIP (e.g. `curl https://keycloak.your-domain.example:8443/...` from the EC2 shell) hang and time out. Two fixes you'll need, one for the host and one for the cluster:

1. **Host-side: add `/etc/hosts` entries** mapping each hostname to `127.0.0.1`:
   ```bash
   sudo tee -a /etc/hosts <<EOF
   127.0.0.1 keycloak.your-domain.example
   127.0.0.1 admin.your-domain.example
   127.0.0.1 signer.your-domain.example
   EOF
   ```

2. **Cluster-side: add a CoreDNS hosts entry** for any hostname a pod needs to reach. The admin-service pod fetches Keycloak's JWKS over HTTPS at the public hostname — that lookup also resolves to the EC2 EIP and hits the same loopback wall, so the admin pod fails to log operators in.

   Get the Keycloak container's IP on the kind docker network:
   ```bash
   KC_IP=$(docker inspect keycloak --format '{{.NetworkSettings.Networks.kind.IPAddress}}')
   echo "$KC_IP"
   ```

   Patch the cluster's CoreDNS ConfigMap to add a `hosts` plugin entry, then restart the deployment. The full Corefile (replace the existing `data.Corefile` with this) is:
   ```
   .:53 {
       errors
       health { lameduck 5s }
       ready
       hosts {
          <KC_IP> keycloak.your-domain.example
          fallthrough
       }
       kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
       }
       prometheus :9153
       forward . /etc/resolv.conf { max_concurrent 1000 }
       cache 30 {
          disable success cluster.local
          disable denial cluster.local
       }
       loop
       reload
       loadbalance
   }
   ```

   Then `kubectl -n kube-system rollout restart deployment/coredns`. Verify with `kubectl run dns-test --rm -it --image=alpine:3.19 -- sh -c 'apk add bind-tools >/dev/null && nslookup keycloak.your-domain.example'` — the answer should be the kind-network IP, not the EC2's public IP.

If you skip the cluster-side fix, you'll get all the way through admin-service install and only discover the problem when an operator tries to log in (the admin pod hangs on JWKS fetch and the browser hangs on the OAuth redirect-back). Add both entries before you deploy the admin service.

**Preflight probe — run before Phase 5.** The admin pod needs two independent things to talk to Keycloak: the issuer hostname must resolve to a reachable address from inside the cluster, and the JVM must trust the CA that signed Keycloak's cert. Missing either fails silently at login time, so verify both from inside the cluster the same way the admin pod will:

```bash
kubectl create configmap oidc-probe-ca --from-file=ca.crt=/etc/verify-test/ca/ca.crt -n default
kubectl run oidc-probe -n default --rm -i --restart=Never --image=curlimages/curl:8.7.1 \
  --overrides='{"spec":{"containers":[{"name":"oidc-probe","image":"curlimages/curl:8.7.1",
    "args":["curl","-sS","--cacert","/ca/ca.crt",
      "https://keycloak.your-domain.example/realms/master/.well-known/openid-configuration"],
    "volumeMounts":[{"name":"ca","mountPath":"/ca"}]}],
    "volumes":[{"name":"ca","configMap":{"name":"oidc-probe-ca"}}]}}'
kubectl delete configmap oidc-probe-ca -n default
```

Success is a JSON document whose `issuer` field is `https://keycloak.your-domain.example/realms/master` — that one response confirms in-cluster DNS, TLS trust, and the issuer hostname in a single shot. A **timeout** is a reachability failure and has more than one possible cause: run the `nslookup` pod from the CoreDNS verification step above to split them. If the name resolves to the host's public IP, the CoreDNS override is missing or wrong; if it resolves to the expected kind-network IP, DNS is fine and the problem is on the path or at the endpoint — check that the Keycloak container is running (`docker ps`), that its port is listening, and for firewall/NetworkPolicy rules blocking the route. A **certificate error** means the CA you mounted is not the one that signed Keycloak's cert. Fix before deploying the admin service. (The probe uses the built-in `master` realm so it works before Phase 3; after you create the `admin-service` realm you can re-run it against the exact `issuerUri` from §5.1.)

### B.13 Continuing with the Deployment Checklist

At this point your single-VM rig satisfies the entire Preflight Checklist:

- Kubernetes cluster + kubectl context: KIND (§B.5)
- Helm 3, helm-diff plugin: §B.4
- PostgreSQL 14+: §B.9
- Keycloak 22+: §B.10
- Container registry: skipped (using Harbor library) or §B.11
- LoadBalancer support: MetalLB (§B.6)
- Workstation tools (openssl, bao CLI): §B.2 / §B.4
- DNS control: §B.12
- TLS certificate (Decision **J**): §B.8

Continue from the Deployment Checklist's Phase 1. Three phase-specific notes for single-VM deployments:

- **§5.1 (admin-service trust store for Keycloak's local-CA cert).** Your Keycloak is served from a cert signed by the local CA you created in §B.8. The JVM inside the admin-service pod does not trust that CA by default, so JWKS validation will fail unless you tell the chart to add the CA to the pod's trust store. In your admin-service `values.yaml` (or values overlay), set:

  ```yaml
  extraCaCerts:
    verify-eval-ca: |
      <paste the contents of /etc/verify-test/ca/ca.crt here>
  ```

  Run `cat /etc/verify-test/ca/ca.crt` on the EC2 host to retrieve the PEM block. The chart's init container will import it into the admin pod's JVM trust store at startup.

- **§5.2 (Admin service LB).** The mainline checklist annotates the admin service Service for an internal cloud LB. On KIND, set `service.type: ClusterIP` and reach the admin service through ingress-nginx — define an Ingress similar to the Keycloak one in §B.10, pointing at the admin service in your chosen namespace. TLS Secrets are namespace-scoped, so the `verify-eval-tls` Secret created in `default` (§B.10) is not visible to an Ingress in namespace **B**; recreate it there from the same local CA-signed cert before defining the Ingress:

  ```bash
  kubectl create secret tls verify-eval-tls \
    --cert=/etc/verify-test/certs/server.crt \
    --key=/etc/verify-test/certs/server.key \
    --namespace <B>
  ```

  then reference `verify-eval-tls` as the Ingress `secretName`.

- **§8 (Signature server).** Same pattern — use an Ingress for Decision **I**, not a cloud LB. Back it with the `signer-tls` Secret you already created in namespace **C** in §8.1 (not `verify-eval-tls`, which exists only in `default`).