Skip to main content

AI platform deployment

CI builds the AI service images and Helm-deploys the ai/charts/platform umbrella chart to the ai-dev namespace on Dev MicroK8s. Cloudflare Tunnel and Access front the two public hostnames.

Architecture​

  • Hostnames

    EnvPortalGateway (api)
    devai-portal-dev.insfocus.comai-api-dev.insfocus.com
    prod (reserved)ai-portal.insfocus.comai-api.insfocus.com
  • DNS — each hostname is a proxied CNAME to the insfocus_hq tunnel (<tunnel-id>.cfargotunnel.com). The tunnel's existing *.insfocus.com rule sends it to the k8s nginx over plain HTTP. No tunnel config change per hostname.

  • TLS — single-label names are covered by Cloudflare Universal SSL (*.insfocus.com).

  • Ingress — host-based, class nginx, nginx.ingress.kubernetes.io/ssl-redirect: "false".

  • Helm release — ai in namespace ai-dev. Umbrella chart ai/charts/platform with subcharts api, portal, migrator. Dev overlay: ai/charts/platform/values.dev.yaml.

  • Migrator — runs as Helm pre-install/pre-upgrade hook Jobs (controlplane, then customers) before api/portal roll. backoffLimit: 0: a failed migration fails the release.

  • Control-plane DB — Dev_InsFocus_AI_Control on the dev SQL server. Created by the migrator on first deploy.

  • Runner — deploy job runs on the self-hosted Windows runner DEV-SQL (self-hosted, Windows, X64, IIS), which has helm and kubectl.

  • Anthropic — Anthropic__ApiKey is not set in dev. The api runs on the fake provider.

Cloudflare Access​

Provisioned by ai/scripts/cloudflare-access-dev.sh (idempotent). Run with a Zero Trust Edit token:

CLOUDFLARE_API_TOKEN=<zero-trust-edit-token> ./ai/scripts/cloudflare-access-dev.sh
  • AI portal (dev) — ai-portal-dev.insfocus.com. Azure AD IdP, auto-redirect, 24h session. Policy: include email_domain: insfocus.com (same as the pr* app).
  • AI api (dev) — ai-api-dev.insfocus.com. Policy Service tokens only (non_identity, any valid service token). No browser redirect; unauthenticated calls get 403.
  • Service token — ai-dev-sdk, for clients calling the api.

The portal also validates the Access assertion itself. Set in values.dev.yaml:

CloudflareAccess__Enabled: "true"
CloudflareAccess__TeamDomain: insfocus.cloudflareaccess.com
CloudflareAccess__Audience: <AUD tag of the "AI portal (dev)" Access app>
  • Audience must be the AUD of this hostname's own Access app.
  • Chart env is a list. Helm replaces lists, so ASPNETCORE_ENVIRONMENT: Staging is restated in the overlay.
  • The api does not read Access config. Portal only.

Lifecycle​

Build and deploy (.github/workflows/ai-*.yml)​

  • Callers: AI · api, AI · portal, AI · migrator. Each calls ai-build-service-reusable.yml.
  • Triggers: PR (build only, no push), push to develop, push to releases/ai/**, workflow_dispatch with input skip_deploy.
  • Image tags: sha-<short> always (deploy pin, rollback). Push to develop also moves dev-latest.
  • Deploy job resolves the target from the ref:
    • develop → namespace ai-dev, values.dev.yaml, secret KUBECONFIG_DEV.
    • releases/ai/** → namespace ai-staging, secret KUBECONFIG_AI_STAGING.
  • Deploy moves only the just-built service to its sha- tag. Other services keep the floating channel tag.
  • Deploys are serialised by a concurrency group.
  • Preflight creates the namespace if missing and fails with a named error if insfocus-ai-db, insfocus-ai-jwt or insfocus-ai-acr is missing.
  • Path filters cover each service's code and the workflow files only. Changes under ai/charts/** do not trigger a deploy.

Bootstrap a new namespace (runbook, done for ai-dev 2026-08-22)​

  1. Create the namespace and secrets:

    kubectl --context k8s-dev create namespace ai-dev
    kubectl --context k8s-dev get secret insfocus-acr-secret -n dev -o json \
    | jq '.metadata = {name: "insfocus-ai-acr", namespace: "ai-dev"}' \
    | kubectl --context k8s-dev apply -f -
    kubectl --context k8s-dev create secret generic insfocus-ai-jwt -n ai-dev \
    --from-literal=signing-key="$(openssl rand -base64 48)"
    kubectl --context k8s-dev create secret generic insfocus-ai-db -n ai-dev \
    --from-literal=controlplane-connection='Server=<sql-host>;Database=Dev_InsFocus_AI_Control;User Id=<user>;Password=<password>;encrypt=false;TrustServerCertificate=True' \
    --from-literal=customer-base-connection='Server=<sql-host>;User Id=<user>;Password=<password>;encrypt=false;TrustServerCertificate=True'
    • insfocus-ai-acr is a copy of the dev pull secret for insfocusdev.azurecr.io.
    • customer-base-connection has no Database=. CustomerContextFactory sets the catalog per customer.
  2. Run the Cloudflare Access script.

  3. Add proxied CNAMEs for both hostnames to the tunnel.

  4. Push to develop.

  5. Run AI · api, AI · portal, AI · migrator with skip_deploy = true. This seeds the floating tag for all three images.

  6. Run any one of them with skip_deploy = false. This rolls the whole release.

After bootstrap, normal develop pushes deploy directly.

Apply a values/chart change​

gh workflow run "AI · portal" --repo InsFocusSystems/Development --ref develop -f skip_deploy=false

Decisions​

  • 2026-08-22 — Flat single-label hostnames: ai-portal-dev / ai-api-dev (dev), ai-portal / ai-api (prod). Why: Universal SSL covers one label only; portal.ai-dev.insfocus.com would need an Advanced Certificate.
  • 2026-08-22 — AI dev deploys on push to develop via Helm into ai-dev on Dev MicroK8s, using KUBECONFIG_DEV. Why: the releases/ai/** → ai-staging path had no branch, secret, environment or namespace.
  • 2026-08-22 — Route AI hostnames through the insfocus_hq Cloudflare Tunnel with proxied CNAMEs. Why: the wildcard A record reaches the HQ reverse proxy, not the cluster.
  • 2026-08-22 — Portal behind Access with Azure AD; api behind Access with service tokens only. Why: browser SSO for staff; the SDK is a non-browser client.
  • 2026-08-22 — Portal validates the Access JWT (CloudflareAccess__Enabled=true). Why: avoids a second, unrelated OTP login after Access SSO.
  • 2026-08-22 — skip_deploy dispatch input for bootstrapping. Why: the umbrella deploys all three services; the first deploy fails if sibling images lack the floating tag.

Gotchas​

  • Problem: Setup Node.js fails with Some specified paths were not resolved, unable to cache dependencies. Cause: package-lock.json is gitignored repo-wide. Fix: cache-dependency-path: front/package.json and npm install instead of npm ci (same as pr-validation.yml).
  • Problem: portal returns 404 on /. Cause: Angular outputPath pointed at ai/portal/InsFocus.Ai.Portal/wwwroot/dist (not the project's wwwroot), and Program.cs had no static-file serving. Fix: outputPath.base = ../ai/portal/wwwroot/dist; add UseDefaultFiles, UseStaticFiles, SPA fallback, and an /api/{**rest} → 404 guard.
  • Problem: new hostname returns 404 in the browser; zero matching lines in the ingress log. Cause: no DNS record, so the wildcard *.insfocus.com A record sends it to the HQ reverse proxy. Fix: proxied CNAME to the tunnel.
  • Problem: values change pushed but not live. Cause: no AI workflow filters on ai/charts/**. Fix: dispatch AI · portal with skip_deploy=false.
  • Problem (risk): with Access validation on, an Access email with no PlatformUser gets 401 unknown_user, no OTP fallback. Fix: make sure the user exists and is not Disabled before enabling. Invited users are activated on first Access login.

Debugging: does the request reach the pod?​

# 1. In-cluster, straight at the service
kubectl --context k8s-dev run c-$RANDOM --rm -i --restart=Never --image=curlimages/curl -n ai-dev \
--command -- curl -s -o /dev/null -w "%{http_code}\n" http://ai-portal.ai-dev.svc.cluster.local/
# 2. nginx by Host header
curl -s -o /dev/null -w "%{http_code}\n" -H "Host: ai-portal-dev.insfocus.com" http://<k8s-node>/
# 3. Origin bypassing Cloudflare
curl -sS -o /dev/null -D - --resolve ai-portal-dev.insfocus.com:443:<origin-ip> https://ai-portal-dev.insfocus.com/
# 4. Ingress log: it records the upstream, not the Host header
kubectl --context k8s-dev logs -n ingress <nginx-ingress-controller-pod> -f | grep ai-dev-ai-portal

Open​

  • Prod deploy (ai-prod namespace, prod kubeconfig, values.prod.yaml, Access apps, promotion trigger) — not wired yet.
  • Auto-deploy on ai/charts/** changes — being evaluated.