Skip to content
AB Test Research Designer

Runbook

Runbook

Section titled “Runbook”

Local startup

Section titled “Local startup”

Backend:

Terminal window
python -m pip install -r app/backend/requirements.txt       # runtime only
# development (tests, lint, typecheck): use requirements-dev.txt instead
python -m uvicorn app.backend.app.main:app --host 127.0.0.1 --port 8008

Frontend dev:

Terminal window
cd app/frontend
npm install
npm run dev

Docker:

Terminal window
docker compose up --build

Docker with API auth:

Terminal window
set AB_API_TOKEN=your-secret-token
docker compose up --build

Docker with split write/read tokens:

Terminal window
set AB_API_TOKEN=write-secret-token
set AB_READONLY_API_TOKEN=readonly-secret-token
docker compose up --build

Docker with signed workspace backups:

Terminal window
set AB_WORKSPACE_SIGNING_KEY=replace-with-a-long-random-secret
docker compose up --build

Docker with Slack App credentials:

Terminal window
set AB_SLACK_CLIENT_ID=your-client-id
set AB_SLACK_CLIENT_SECRET=your-client-secret
set AB_SLACK_SIGNING_SECRET=your-signing-secret
docker compose up --build

Security-hardening knobs:

Terminal window
set AB_RATE_LIMIT_ENABLED=true
set AB_RATE_LIMIT_REQUESTS=240
set AB_AUTH_FAILURE_LIMIT=20
set AB_MAX_REQUEST_BODY_BYTES=1048576
set AB_MAX_WORKSPACE_BODY_BYTES=8388608
docker compose up --build

Running behind a reverse proxy

Section titled “Running behind a reverse proxy”

The rate limiter and the auth-failure throttle bucket callers by IP address. X-Forwarded-For is client-controlled, so it is ignored by default (AB_TRUSTED_PROXY_HOPS=0) and both limits key off the direct peer. Leave it at 0 whenever the app is reachable directly — otherwise a caller can rotate the header and get a fresh bucket on every request.

When the app sits behind reverse proxies, set the hop count to how many of them append to the header (one nginx / one load balancer = 1; CDN in front of a load balancer = 2). The caller is then read as the N-th entry from the right; everything to its left is client-supplied and never trusted.

Terminal window
set AB_TRUSTED_PROXY_HOPS=1
# Optional: only honour the header when the direct peer is one of these addresses / CIDR blocks.
set AB_TRUSTED_PROXIES=10.0.0.0/8,192.0.2.7

Never set the hop count higher than the number of proxies that actually append. A caller can prepend any number of entries to the header, so an over-declared count reads one of its entries and hands it back control of its rate-limit bucket. Under-declaring is safe by comparison: it resolves to a proxy address, which merely buckets more traffic together. If in doubt, measure the chain before raising it.

Measure the chain with the diagnostics endpoint, which echoes the server-side view of the calling request. Send a marker entry and see what the infrastructure appends to its right:

Terminal window
curl -s -H "X-Forwarded-For: 203.0.113.7" https://<host>/api/v1/diagnostics | jq .network

Everything to the right of 203.0.113.7 in forwarded_for_chain was appended by a real proxy — that count is the correct AB_TRUSTED_PROXY_HOPS value, and the leftmost appended entry should be your own egress address (curl -s https://ipinfo.io/ip). After raising the hop count, repeat the call: the response must show resolved_from: "forwarded_header" and resolved_client equal to your egress address, not something you supplied in the marker.

Re-measure before changing hop counts on any reverse-proxied deployment: an over-declared hop count lets a visitor prepend entries and pick their own rate-limit bucket.

Setting AB_TRUSTED_PROXIES without a non-zero hop count is a startup error.

API-key traffic is unaffected either way — it buckets by api_key:{id}, not by address.

Postgres deployment

Section titled “Postgres deployment”

Use Postgres when the backend must support concurrent writers, multiple app instances, or database-native replication. Leave AB_DATABASE_URL empty to keep the default SQLite workflow.

Minimal env:

Terminal window
set AB_DATABASE_URL=postgresql://postgres:postgres@localhost:5432/abtest
set AB_DB_POOL_SIZE=10
python -m uvicorn app.backend.app.main:app --host 127.0.0.1 --port 8008

Notes:

  • schema bootstrapping is automatic on startup via idempotent CREATE TABLE IF NOT EXISTS
  • SQLite-specific snapshot sync stays disabled on Postgres runtimes
  • AB_DB_POOL_SIZE controls the psycopg connection pool; increase it for multi-worker deploys
  • AB_DB_PATH, AB_SQLITE_BUSY_TIMEOUT_MS, AB_SQLITE_JOURNAL_MODE, and AB_SQLITE_SYNCHRONOUS still apply only to SQLite

Legacy SQLite remote snapshot (unsupported publication path)

Section titled “Legacy SQLite remote snapshot (unsupported publication path)”

Optional HF Dataset snapshot code remains in the repository (AB_HF_* env vars and SnapshotService) and is covered by the in-repo unit suite. It is not a supported publication, demo, or production backup target (owner decision 2026-07-30) and is outside project closure. Do not use it as an operational recovery recipe; prefer signed workspace exports for portable SQLite demos and managed PostgreSQL backups for production.

Migration guide from SQLite:

Terminal window
sqlite3 D:\AB_TEST\app\backend\data\projects.sqlite3 .dump > workspace.sql
psql postgresql://postgres:postgres@localhost:5432/abtest -f workspace.sql

Demo seeding (local / container)

Section titled “Demo seeding (local / container)”

The supported demo path is the local runner:

Terminal window
python scripts/run_local.py --seed-demo

That populates the local SQLite workspace with four demo projects (checkout conversion, pricing sensitivity, onboarding completion, and feed ad click-through ratio), each with analysis/history data as applicable. Open http://127.0.0.1:8008.

When AB_SEED_DEMO_ON_STARTUP=true, the backend also runs the same idempotent startup seed after SQLite initialization (useful for Docker Compose or other self-hosted containers). The image default remains AB_SEED_DEMO_ON_STARTUP=false.

Disable seeding by setting:

Terminal window
set AB_SEED_DEMO_ON_STARTUP=false

Re-seed by starting from a fresh SQLite file and restarting the process, or by deleting demo projects through DELETE /api/v1/projects/{id} with a write-capable token and restarting with the seed flag.

Quick verification on the local runtime:

Terminal window
curl http://127.0.0.1:8008/api/v1/projects
curl http://127.0.0.1:8008/api/v1/projects/PROJECT_ID/history

Quick checks

Section titled “Quick checks”
  • health: http://127.0.0.1:8008/health
  • readiness: http://127.0.0.1:8008/readyz
  • diagnostics: http://127.0.0.1:8008/api/v1/diagnostics

If AB_API_TOKEN or AB_READONLY_API_TOKEN is enabled, send either:

  • Authorization: Bearer <token>
  • X-API-Key: <token>

Read-only tokens allow GET/HEAD/OPTIONS plus the stateless calculation endpoints (/calculate, /analyze, /results/*, exports, and similar compute-only POSTs). Anything that changes stored state still requires the write token. Setting AB_PUBLIC_DEMO=true gives anonymous visitors the same read-only scope instead of 401 (self-hosted public-demo mode). When the frontend is served, enter the token through the “API session token” field; it is stored only in the current browser session. When throttling is enabled, bursty /api/v1/* traffic and repeated bad tokens return 429 with Retry-After.

Full verification

Section titled “Full verification”

Primary Windows entrypoint:

Terminal window
cmd /c scripts\verify_all.cmd

With secure Docker compose verification:

Terminal window
cmd /c scripts\verify_all.cmd --with-docker

Non-destructive Docker verification:

Terminal window
python scripts/verify_docker_compose.py --preserve

Or through the main verify wrapper:

Terminal window
cmd /c scripts\verify_all.cmd --with-docker-preserve

Focused checks:

Terminal window
python -m pytest app/backend/tests -q
npm --prefix app/frontend run test:unit
npm --prefix app/frontend run build
npm --prefix app/frontend run test:e2e
python scripts/run_local_smoke.py --skip-build
python scripts/benchmark_backend.py --payload binary --assert-ms 100
python scripts/verify_workspace_backup.py --fixture

Signed workspace backup check:

Terminal window
set AB_WORKSPACE_SIGNING_KEY=replace-with-a-long-random-secret
python scripts/verify_workspace_backup.py --fixture

npm --prefix app/frontend run test:e2e builds the frontend if needed and runs against a temporary backend-served build on a free local port.

Workspace backup and restore

Section titled “Workspace backup and restore”

Export the full local workspace:

Terminal window
curl http://127.0.0.1:8008/api/v1/workspace/export > workspace-backup.json

Import a backup:

Terminal window
curl -X POST http://127.0.0.1:8008/api/v1/workspace/import ^
  -H "Content-Type: application/json" ^
  -d @workspace-backup.json

The backup contains:

  • saved projects
  • analysis run history
  • export events
  • saved project revisions
  • integrity counts and a SHA-256 checksum
  • optional signature_hmac_sha256 when AB_WORKSPACE_SIGNING_KEY is configured

Round-trip verification against a live DB file:

Terminal window
python scripts/verify_workspace_backup.py --db-path D:\AB_TEST\app\backend\data\projects.sqlite3

If the target runtime uses AB_WORKSPACE_SIGNING_KEY, rerun the same verification command with that env var set so signature verification is exercised, not only checksum validation.

Saved-project recovery

Section titled “Saved-project recovery”

Useful endpoints:

  • GET /api/v1/projects
  • GET /api/v1/projects/{project_id}/history
  • GET /api/v1/projects/{project_id}/revisions

Use revisions to restore an older payload into the wizard, then save to persist it as the latest version.

Multi-project comparison

Section titled “Multi-project comparison”

Useful endpoints:

  • POST /api/v1/projects/compare
  • POST /api/v1/export/comparison
  • legacy pairwise: GET /api/v1/projects/compare?base_id=...&candidate_id=... with Deprecation: true

Open a comparison dashboard for 3 saved projects:

Terminal window
curl -X POST http://127.0.0.1:8008/api/v1/projects/compare \
  -H "Content-Type: application/json" \
  -d '{"project_ids":["PROJECT_ID_1","PROJECT_ID_2","PROJECT_ID_3"]}'

Export the same selection to Markdown:

Terminal window
curl -X POST http://127.0.0.1:8008/api/v1/export/comparison \
  -H "Content-Type: application/json" \
  -d '{"project_ids":["PROJECT_ID_1","PROJECT_ID_2","PROJECT_ID_3"],"format":"markdown"}'

Export the same selection to PDF:

Terminal window
curl -X POST http://127.0.0.1:8008/api/v1/export/comparison \
  -H "Content-Type: application/json" \
  -d '{"project_ids":["PROJECT_ID_1","PROJECT_ID_2","PROJECT_ID_3"],"format":"pdf"}'

Checks:

  • project_ids must contain 2 to 5 unique saved projects
  • every selected project must already have a saved analysis snapshot
  • mixed binary and continuous selections are allowed, but the dashboard marks direct effect comparison as not meaningful
  • PDF export returns base64-encoded content in the JSON payload; decode client-side before saving to disk

Common failure modes

Section titled “Common failure modes”

Readiness returns 503:

  • check SQLite path and write access
  • check schema version and journal mode in GET /readyz
  • if AB_SERVE_FRONTEND_DIST=true, ensure app/frontend/dist/index.html exists
  • inspect GET /api/v1/diagnostics for frontend/LLM/storage details

Frontend loads but backend requests fail:

  • confirm VITE_API_BASE_URL
  • if write-token auth is enabled, confirm that the browser-session token is present and accepted by diagnostics
  • if read-only auth is enabled, verify diagnostics/docs work while mutations still reject with 403
  • if requests start returning 429, inspect Retry-After and tune AB_RATE_LIMIT_* or AB_AUTH_FAILURE_* for the target runtime
  • verify CORS env values if frontend is on another origin
  • use request_id and X-Error-Code from API failures to correlate UI errors with backend logs
  • use diagnostics runtime counters and guard settings to confirm whether failures are isolated, rate-limited, or caused by request-size policy on the current process lifetime

Workspace import fails:

  • validate JSON shape against docs/API.md
  • if the payload is legitimately large, confirm AB_MAX_WORKSPACE_BODY_BYTES is high enough for the target runtime
  • ensure referenced analysis runs and projects are consistent
  • ensure the integrity checksum still matches and that the bundle was not edited after export
  • if the runtime has AB_WORKSPACE_SIGNING_KEY, ensure the bundle still contains a valid signature_hmac_sha256

Rotating API keys

Section titled “Rotating API keys”

Prerequisites:

  • AB_ADMIN_TOKEN is configured on the backend runtime
  • you can reach GET /api/v1/keys with Authorization: Bearer <AB_ADMIN_TOKEN>

Recommended rotation flow:

  1. Create a replacement key with the same scope and any required per-key rate-limit override.
  2. Update the external consumer to use the new plaintext key.
  3. Verify traffic moved by checking GET /api/v1/audit?key_id=<new-key-id>&action=api_key_used.
  4. Revoke the previous key with POST /api/v1/keys/{key_id}/revoke.
  5. Delete the revoked key with DELETE /api/v1/keys/{key_id} once rollback is no longer needed.

Create:

Terminal window
curl -X POST http://127.0.0.1:8008/api/v1/keys \
  -H "Authorization: Bearer YOUR_AB_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"Partner write key","scope":"write"}'

Audit verification:

Terminal window
curl "http://127.0.0.1:8008/api/v1/audit?key_id=KEY_ID&action=api_key_used" \
  -H "Authorization: Bearer YOUR_WRITE_TOKEN"

api_key_used audit entries are written by a background task attached to the response, not inline with the request. For a single client this is invisible (the worker process flushes the task before sending the next response), but a concurrent observer querying the audit endpoint immediately after an authenticated request may briefly miss the entry. Re-query after ~1s if the expected entry is missing.

Revoke:

Terminal window
curl -X POST http://127.0.0.1:8008/api/v1/keys/KEY_ID/revoke \
  -H "Authorization: Bearer YOUR_AB_ADMIN_TOKEN"

Delete:

Terminal window
curl -X DELETE http://127.0.0.1:8008/api/v1/keys/KEY_ID \
  -H "Authorization: Bearer YOUR_AB_ADMIN_TOKEN"

Notes:

  • the plaintext key is returned only once in the create response; do not log or persist it outside the intended secret store
  • legacy AB_API_TOKEN and AB_READONLY_API_TOKEN continue to work during migration and can be retired separately from managed API keys

Webhook troubleshooting

Section titled “Webhook troubleshooting”

Useful endpoints:

  • GET /api/v1/webhooks
  • GET /api/v1/webhooks/{webhook_id}/deliveries?limit=50
  • POST /api/v1/webhooks/{webhook_id}/test

Common checks:

  • confirm the subscription uses https://...; plain HTTP is rejected outside AB_ENV=local localhost targets
  • verify the subscription is still enabled
  • inspect the delivery history for response_code, attempt_count, and error_message
  • if a delivery is stuck in retrying, wait for the next in-process backoff window before retrying manually
  • generic consumers must verify X-AB-Signature with the stored shared secret; Slack subscriptions do not include that header
  • repeated terminal failures move the delivery into the DB-backed dead-letter history with status=failed

Create a test subscription or re-run a probe:

Terminal window
curl -X POST http://127.0.0.1:8008/api/v1/webhooks/WEBHOOK_ID/test \
  -H "Authorization: Bearer YOUR_AB_ADMIN_TOKEN"

Inspect recent failed deliveries:

Terminal window
curl "http://127.0.0.1:8008/api/v1/webhooks/WEBHOOK_ID/deliveries?limit=50&status=failed" \
  -H "Authorization: Bearer YOUR_AB_ADMIN_TOKEN"

Slack App deployment

Section titled “Slack App deployment”

Use slack/app-manifest.yml for Slack UI import or slack manifest validate. Replace {DEPLOY_HOST} with the HTTPS backend host before importing.

Required runtime secrets:

  • AB_SLACK_CLIENT_ID
  • AB_SLACK_CLIENT_SECRET
  • AB_SLACK_SIGNING_SECRET

Useful endpoints:

  • GET /slack/install
  • GET /slack/oauth/callback
  • POST /slack/commands
  • POST /slack/interactive
  • POST /slack/events
  • GET /api/v1/slack/status

Smoke check:

Terminal window
curl http://127.0.0.1:8008/api/v1/slack/status

After installation, run /ab-test projects in Slack and verify the response lists saved projects. Slack requests are rejected when X-Slack-Signature is invalid or the timestamp is older than five minutes.

Rotate secrets from the Slack App configuration, update runtime secrets, restart the backend, delete the affected slack_installations row when replacing bot tokens, then reinstall through /slack/install.

Slack OAuth tokens at rest

Section titled “Slack OAuth tokens at rest”

slack_installations.bot_token and slack_installations.user_token are stored plaintext in SQLite (or Postgres if AB_DATABASE_URL is set). The threat model is local-first single-user use, where access to the SQLite file already implies host compromise. Specifically:

  • data/projects.sqlite3 and any custom AB_DB_PATH should sit on a filesystem that only the application user can read.
  • Public or disposable demos: do not install Slack in production-grade workspaces from the demo. Use a sandbox Slack workspace or skip the integration.
  • For a self-hosted production deployment, restrict access to the database file at the filesystem layer (e.g. chmod 600, encrypted volume, AWS KMS-backed EFS), and rotate Slack tokens promptly if compromise is suspected.
  • Token rotation: re-run /slack/install to overwrite the row; the previous token becomes inactive in Slack as well.

If you need encrypted-at-rest storage, the recommended path is to enable filesystem-level or volume-level encryption on the host rather than column-level encryption in the application — the latter requires its own KEK management and adds a recovery failure mode without changing the practical attacker model for a host with running write access.

Adding a new locale

Section titled “Adding a new locale”

The project ships with seven locales: en, ru, de, es, fr, zh, ar. Adding a new one takes a matching pair of JSON files plus registration, switcher, and verification touches.

  1. Frontend translation - copy app/frontend/src/i18n/en.json to <code>.json in the same directory and translate the strings. Current shipped locales (de, es, fr, zh, ar) all keep full leaf-key parity with en; do not ship a partial file unless review explicitly allows fallback coverage.
  2. Frontend registration - add the new code to app/frontend/src/i18n/index.ts: import the JSON, add it to resources, extend supportedLngs, and keep the fallbackLng resolver returning <primary> -> en for regional tags.
  3. Language switcher - extend SUPPORTED_LANGUAGES in app/frontend/src/App.tsx so the header switcher renders the new button. Add a label under app.language.options.<code> to every shipped locale file, not only the new one, because the switcher is visible in every active locale.
  4. RTL locales - if the locale is right-to-left, update App.tsx so it sets document.documentElement.dir = "rtl" for that code and "ltr" otherwise. Audit app/frontend/src/**/*.css for logical properties (inset-inline-*, padding-inline-*, margin-inline-*, border-inline-start-*, text-align: start/end) before shipping.
  5. Backend translation - copy app/backend/app/i18n/en.json to app/backend/app/i18n/<code>.json. Translate at least the export.markdown, export.html, warnings, and report subtrees, since those feed the /api/v1/export/* payloads and the deterministic report builder.
  6. Backend registration - extend the Language literal and SUPPORTED_LANGUAGES tuple in app/backend/app/i18n/__init__.py. The resolve_language helper already accepts any supported primary tag, so regional variants like fr-CA, zh-TW, or ar-SA fall back automatically.
  7. Tests - extend app/frontend/src/i18n.test.tsx, keep app/frontend/src/test/a11y-locales.test.tsx aligned with the current switcher shape, add app/frontend/src/test/a11y-rtl.test.tsx for RTL locales, and add backend export assertions to app/backend/tests/test_export_api.py covering the translated markdown header and primary-tag fallback.

Verification:

Terminal window
cmd /c scripts\verify_all.cmd --with-e2e

The backend bundle is small (one JSON per locale); the frontend bundle impact depends on whether locales are statically imported or lazy-loaded, so re-check the build output after adding a fully translated locale set.

Release hygiene

Section titled “Release hygiene”
  • regenerate API contracts: python scripts/generate_frontend_api_types.py
  • regenerate API docs: python scripts/generate_api_docs.py
  • run workspace roundtrip verification: python scripts/verify_workspace_backup.py --fixture
  • run full verify pipeline
  • refresh smoke screenshots if UI changed materially

Local cleanup

Section titled “Local cleanup”

python scripts/cleanup_test_artifacts.py (use --dry-run first) removes local pytest temp dirs, .coverage, the cxkm sandbox, and the mkdocs site/ build. Everything it touches is recreated on the next test/build run; no tracked content is affected.

Screenshots

Section titled “Screenshots”

Single source of truth is docs/demo/, populated by scripts/run_local_smoke.py. The mkdocs site keeps its own copies in docs-site/assets/screenshots/ because mkdocs only serves files under docs_dir. After regenerating screenshots, run python scripts/sync_doc_screenshots.py to mirror them; CI does this automatically before mkdocs gh-deploy. Files unique to docs-site (currently comparison-distribution-view.png) are left untouched.