AI platform deployment
CI builds the AI service images and Helm-deploys the ai/charts/platform umbrella chart to the ai-dev namespace on Dev MicroK8s. Cloudflare Tunnel and Access front the two public hostnames.
Architecture
-
Hostnames
Env Portal Gateway (api) dev ai-portal-dev.insfocus.comai-api-dev.insfocus.comprod (reserved) ai-portal.insfocus.comai-api.insfocus.com -
DNS — each hostname is a proxied CNAME to the
insfocus_hqtunnel (<tunnel-id>.cfargotunnel.com). The tunnel's existing*.insfocus.comrule sends it to the k8s nginx over plain HTTP. No tunnel config change per hostname. -
TLS — single-label names are covered by Cloudflare Universal SSL (
*.insfocus.com). -
Ingress — host-based, class
nginx,nginx.ingress.kubernetes.io/ssl-redirect: "false". -
Helm release —
aiin namespaceai-dev. Umbrella chartai/charts/platformwith subchartsapi,portal,migrator. Dev overlay:ai/charts/platform/values.dev.yaml. -
Migrator — runs as Helm pre-install/pre-upgrade hook Jobs (
controlplane, thencustomers) before api/portal roll.backoffLimit: 0: a failed migration fails the release. -
Control-plane DB —
Dev_InsFocus_AI_Controlon the dev SQL server. Created by the migrator on first deploy. -
Runner — deploy job runs on the self-hosted Windows runner
DEV-SQL(self-hosted, Windows, X64, IIS), which hashelmandkubectl. -
Anthropic —
Anthropic__ApiKeyis not set in dev. The api runs on the fake provider.
Cloudflare Access
Provisioned by ai/scripts/cloudflare-access-dev.sh (idempotent). Run with a Zero Trust Edit token:
CLOUDFLARE_API_TOKEN=<zero-trust-edit-token> ./ai/scripts/cloudflare-access-dev.sh
- AI portal (dev) —
ai-portal-dev.insfocus.com. Azure AD IdP, auto-redirect, 24h session. Policy: includeemail_domain: insfocus.com(same as thepr*app). - AI api (dev) —
ai-api-dev.insfocus.com. PolicyService tokens only(non_identity, any valid service token). No browser redirect; unauthenticated calls get403. - Service token —
ai-dev-sdk, for clients calling the api.
The portal also validates the Access assertion itself. Set in values.dev.yaml:
CloudflareAccess__Enabled: "true"
CloudflareAccess__TeamDomain: insfocus.cloudflareaccess.com
CloudflareAccess__Audience: <AUD tag of the "AI portal (dev)" Access app>
- Audience must be the AUD of this hostname's own Access app.
- Chart
envis a list. Helm replaces lists, soASPNETCORE_ENVIRONMENT: Stagingis restated in the overlay. - The api does not read Access config. Portal only.
Lifecycle
Build and deploy (.github/workflows/ai-*.yml)
- Callers:
AI · api,AI · portal,AI · migrator. Each callsai-build-service-reusable.yml. - Triggers: PR (build only, no push), push to
develop, push toreleases/ai/**,workflow_dispatchwith inputskip_deploy. - Image tags:
sha-<short>always (deploy pin, rollback). Push todevelopalso movesdev-latest. - Deploy job resolves the target from the ref:
develop→ namespaceai-dev,values.dev.yaml, secretKUBECONFIG_DEV.releases/ai/**→ namespaceai-staging, secretKUBECONFIG_AI_STAGING.
- Deploy moves only the just-built service to its
sha-tag. Other services keep the floating channel tag. - Deploys are serialised by a concurrency group.
- Preflight creates the namespace if missing and fails with a named error if
insfocus-ai-db,insfocus-ai-jwtorinsfocus-ai-acris missing. - Path filters cover each service's code and the workflow files only. Changes under
ai/charts/**do not trigger a deploy.
Bootstrap a new namespace (runbook, done for ai-dev 2026-08-22)
-
Create the namespace and secrets:
kubectl --context k8s-dev create namespace ai-devkubectl --context k8s-dev get secret insfocus-acr-secret -n dev -o json \| jq '.metadata = {name: "insfocus-ai-acr", namespace: "ai-dev"}' \| kubectl --context k8s-dev apply -f -kubectl --context k8s-dev create secret generic insfocus-ai-jwt -n ai-dev \--from-literal=signing-key="$(openssl rand -base64 48)"kubectl --context k8s-dev create secret generic insfocus-ai-db -n ai-dev \--from-literal=controlplane-connection='Server=<sql-host>;Database=Dev_InsFocus_AI_Control;User Id=<user>;Password=<password>;encrypt=false;TrustServerCertificate=True' \--from-literal=customer-base-connection='Server=<sql-host>;User Id=<user>;Password=<password>;encrypt=false;TrustServerCertificate=True'insfocus-ai-acris a copy of the dev pull secret forinsfocusdev.azurecr.io.customer-base-connectionhas noDatabase=.CustomerContextFactorysets the catalog per customer.
-
Run the Cloudflare Access script.
-
Add proxied CNAMEs for both hostnames to the tunnel.
-
Push to
develop. -
Run
AI · api,AI · portal,AI · migratorwithskip_deploy = true. This seeds the floating tag for all three images. -
Run any one of them with
skip_deploy = false. This rolls the whole release.
After bootstrap, normal develop pushes deploy directly.
Apply a values/chart change
gh workflow run "AI · portal" --repo InsFocusSystems/Development --ref develop -f skip_deploy=false
Decisions
- 2026-08-22 — Flat single-label hostnames:
ai-portal-dev/ai-api-dev(dev),ai-portal/ai-api(prod). Why: Universal SSL covers one label only;portal.ai-dev.insfocus.comwould need an Advanced Certificate. - 2026-08-22 — AI dev deploys on push to
developvia Helm intoai-devon Dev MicroK8s, usingKUBECONFIG_DEV. Why: thereleases/ai/**→ai-stagingpath had no branch, secret, environment or namespace. - 2026-08-22 — Route AI hostnames through the
insfocus_hqCloudflare Tunnel with proxied CNAMEs. Why: the wildcard A record reaches the HQ reverse proxy, not the cluster. - 2026-08-22 — Portal behind Access with Azure AD; api behind Access with service tokens only. Why: browser SSO for staff; the SDK is a non-browser client.
- 2026-08-22 — Portal validates the Access JWT (
CloudflareAccess__Enabled=true). Why: avoids a second, unrelated OTP login after Access SSO. - 2026-08-22 —
skip_deploydispatch input for bootstrapping. Why: the umbrella deploys all three services; the first deploy fails if sibling images lack the floating tag.
Gotchas
- Problem:
Setup Node.jsfails withSome specified paths were not resolved, unable to cache dependencies. Cause:package-lock.jsonis gitignored repo-wide. Fix:cache-dependency-path: front/package.jsonandnpm installinstead ofnpm ci(same aspr-validation.yml). - Problem: portal returns 404 on
/. Cause: AngularoutputPathpointed atai/portal/InsFocus.Ai.Portal/wwwroot/dist(not the project's wwwroot), andProgram.cshad no static-file serving. Fix:outputPath.base=../ai/portal/wwwroot/dist; addUseDefaultFiles,UseStaticFiles, SPA fallback, and an/api/{**rest}→ 404 guard. - Problem: new hostname returns 404 in the browser; zero matching lines in the ingress log. Cause: no DNS record, so the wildcard
*.insfocus.comA record sends it to the HQ reverse proxy. Fix: proxied CNAME to the tunnel. - Problem: values change pushed but not live. Cause: no AI workflow filters on
ai/charts/**. Fix: dispatchAI · portalwithskip_deploy=false. - Problem (risk): with Access validation on, an Access email with no
PlatformUsergets401 unknown_user, no OTP fallback. Fix: make sure the user exists and is notDisabledbefore enabling.Invitedusers are activated on first Access login.
Debugging: does the request reach the pod?
# 1. In-cluster, straight at the service
kubectl --context k8s-dev run c-$RANDOM --rm -i --restart=Never --image=curlimages/curl -n ai-dev \
--command -- curl -s -o /dev/null -w "%{http_code}\n" http://ai-portal.ai-dev.svc.cluster.local/
# 2. nginx by Host header
curl -s -o /dev/null -w "%{http_code}\n" -H "Host: ai-portal-dev.insfocus.com" http://<k8s-node>/
# 3. Origin bypassing Cloudflare
curl -sS -o /dev/null -D - --resolve ai-portal-dev.insfocus.com:443:<origin-ip> https://ai-portal-dev.insfocus.com/
# 4. Ingress log: it records the upstream, not the Host header
kubectl --context k8s-dev logs -n ingress <nginx-ingress-controller-pod> -f | grep ai-dev-ai-portal
Open
- Prod deploy (
ai-prodnamespace, prod kubeconfig,values.prod.yaml, Access apps, promotion trigger) — not wired yet. - Auto-deploy on
ai/charts/**changes — being evaluated.