Testing

The normal test suite uses local data and a fake translator. Once npm ci has installed the dependencies, it needs no accounts, provider keys or Docker. Tests run with Vitest and assert with its expect and assert. Start with npm run check from the repository root.

Tests sit next to their code as *.test.ts. vitest.config.ts defines three projects: unit (what npm test runs), e2e and acceptance. To focus a change, run single files:

npx vitest run packages/service/src/settings_api.test.ts
npx vitest run packages/server/src/storage/backup_files.test.ts
npx vitest run site/build.test.ts

The fake translator

The Beta 2 workflow fixture contains five source files, ten languages, nested paths and separate approved/full Crowdin exports. npx vitest run acceptance/workflow_fixture.test.ts checks its import counts, upload changes, queue order and Crowdin conversion. Its expected data is for the shared URL-driven workflow test; passing this fixture check does not prove Docker or Cloudflare execution.

npx vitest run e2e/workflow.test.ts runs the full nine-step workflow against a disposable local instance, using the built CLI, public HTTP, and Chromium. The test runner captures the LLM stub's requests and checks proofread Spanish references in every prompt. Build the CLI first with npm run build:cli. The test defines its own three-minute timeout.

For a dedicated empty instance with a deterministic LLM stub, run npm run workflow -- --url http://127.0.0.1:8000. A fresh instance needs QUASO_SETUP_KEY; an already claimed empty instance needs QUASO_ADMIN_EMAIL and QUASO_ADMIN_PASSWORD. The runner refuses an instance containing active source files. It revokes its API key and removes its browser page and temporary CLI workspace; the caller must tear down the disposable instance and its storage, which retain the fixture's account/data. This command does not configure an LLM stub or run CI. Deployment test runners must also capture/check the provider prompts and supply the normal job/publication scheduler.

packages/service/src/llm/fake.ts implements the translator port. It preserves placeholders and plural structure and gives deterministic results. Tests can model delays, failed checks and provider errors without real calls. Use it to verify retries, cancellation, budgets and races between an LLM response and a human approval. An LLM result must recheck the source and translation revision after awaiting the provider; no test should depend on an actual model's prose.

Check translation behavior

Choose tests for the behavior your change affects. The newer features have focused entry points:

Run a listed file with npx vitest run PATH, replacing PATH with its repository path. For changes to shared database behavior, also run the corresponding D1 cases with npm run cf:test. The fake provider checks how requests and results are handled; it does not establish whether a model's translations or meaning judgments are correct.

Browser tests

browser test setup starts an in-process server with in-memory SQLite and the real security headers. It builds the website when stale. Install Chromium once with npm run e2e:browser; Puppeteer starts it for each test. Tests collect console errors, failed requests and CSP violations. Use real routes where possible; intercept only behavior that is hard to cause with repeatable results. Browser tests are separate from the normal local suite because they start a browser and may need an initial download.

Screenshots in the project website use Quaso Quest's actual French source files, with English translations, in light and dark themes. Install the game's dependencies with npm ci --prefix examples/demo-game, then generate them again with node site/screenshots.ts. The script uses disposable in-memory storage and a real signed-in account. It also captures the playable game using English files exported from Quaso, without changing the example's deliberately empty tracked translation files. Reduce the size of the resulting PNGs without changing their content. Visual screenshots add to test results; they do not replace keyboard or permission tests. Include the accessibility review in release checks.

Cloudflare and upgrade fixtures

The optional Cloudflare package keeps its own Vitest configuration, which runs the tests in workerd with the Workers pool. Its service tests use D1 through the same handler and SQL adapter as the container. The old Durable Object storage service is no longer present. Docker is needed for cf:dev, not cf:test. Read Cloudflare development before changing adapters or alarms.

npm run cf:fixtures generates the local-service scenario consumed by workerd tests. Commit its diff alongside a behavior or schema change. npm run cf:fixtures -- --check fails when it is stale. For historical schema fixtures, preserve already released inputs unchanged: add the next version's fixture and expected upgrade behavior. Validate actual rows and exported files after migration, including history, roles and blue translations, and test a newer schema is rejected by an old reader. The migration steps are in the migration procedure.

Keep a database from every supported release

packages/service/testdata/upgrade/ preserves databases created with historical implementations. v1.sqlite and v3.sqlite come from Beta 1 (schema 1 at commit a290bab, schema 3 at c855ab9); Beta 2 restarted the schema, so the test checks that it refuses them and leaves them unchanged. provenance.json records their origins. These are original databases, not today's migrations run backwards. Run the upgrade check directly with:

npx vitest run packages/service/src/upgrade.test.ts

For each future supported schema:

1. Create a detached worktree at the release tag, for example git worktree add --detach /tmp/quaso-previous v1.0.0. Use a fresh scratch path and the actual tag.
2. Run that checkout's seed-dev command or dev:reset task to create its demo database. Upload a changed source file and import translations so history, uploads and activity contain data. Add representative roles, suggestions and jobs when that schema supports them. Fixtures must contain synthetic data, never real account details or provider keys.
3. Save that running version's /api/v1/export and /api/v1/status responses using its development API key. Stop the server cleanly, then copy quaso.sqlite to packages/service/testdata/upgrade/vN.sqlite. Store its responses as vN-export.json and vN-status.json, and record the source commit and scenario.
4. Extend upgrade.test.ts to copy the fixture to temporary storage, migrate it with the current service, compare surviving rows, export bytes and status, then exercise an upload, a fake LLM job and a reviewed translation. Keep existing fixture files unchanged.
5. Remove the worktree with git worktree remove /tmp/quaso-previous after retaining the artifacts, and run both the upgrade test and npm run cf:check. Review binary fixtures and their provenance together in the pull request.

Acceptance against a running instance

The acceptance/ suite is separate from npm test. It exercises the public HTTP API, the bundled CLI and Puppeteer against any reachable instance. Use a disposable development instance or a test instance: the suite creates uniquely named source files, translations and accounts. It revokes its API key and deletes temporary local files; synthetic source/history records remain on the instance. Never point it at production.

Build the CLI first. Against a running npm run dev instance, the development login is enough:

npm run build:cli
npm run acceptance -- --url http://localhost:8000

Against a claimed staging instance, set QUASO_ADMIN_EMAIL and QUASO_ADMIN_PASSWORD as environment secrets. For first setup, also set QUASO_SETUP_KEY from the deployment configuration. The suite uses configured administrator login details during setup, so later runs can sign in to the same account. Tokens may also be passed with --setup-token, but environment secrets avoid putting them on command lines.

To check the production image with local storage, use a distinct disposable Compose project:

export COMPOSE_FILE="$PWD/deploy/compose.yaml:$PWD/deploy/compose.acceptance.yaml"
export COMPOSE_PROJECT_NAME=quaso-acceptance-local
export QUASO_ACCEPTANCE_PORT=8134
docker compose build quaso
docker compose up -d --wait --wait-timeout 120 quaso
docker compose logs quaso

Copy the setup key from that disposable instance's deployment configuration, then run:

export QUASO_SETUP_KEY='the key from the disposable instance'
npm run acceptance -- --url http://127.0.0.1:8134 --compose-dir deploy
docker compose down -v --remove-orphans

Keep COMPOSE_FILE and COMPOSE_PROJECT_NAME exported: the suite's restart command must use the same stack. The override resets the operator .env, binds the port to loopback, and disables Caddy, development login and Gemini. The persistence test reports a skip without --compose-dir; the automatic translation test reports a skip without an enabled translator. Human-check challenges may skip automated volunteer scenarios; use the independent browser fixture tests for those controls. The production Compose case intentionally has no fake-provider override. The development instance exercises the fake translator instead.

.github/workflows/acceptance.yml runs the disposable Compose case on pushes to main or master and on manual requests. Its staging job runs only when these settings are present:

Configure the staging hostname, bucket, and Worker secrets using the deployment guide first. Acceptance restores the instance configuration in its clean checkout and waits for the image published by docker.yml as sha-<seven-character commit>. It uses DOCKERHUB_IMAGE when set, or <GitHub owner>/quaso, and replaces the saved image choice for that deployment. The instance configuration contains resource identifiers, not API keys or passwords. The first check prints only whether the required settings exist, without printing their values.

No GitHub Actions workflow runs on a schedule. Run the large local stress case with QUASO_SMOKE=1 npm test. The D1 performance case runs in npm run cf:test: 3,000 strings in ten languages, 30,000 translations, and 100 exported files. Upload, export, and browsing each have a 30-second budget.

To measure staging requests, run node scripts/measure.ts by hand. Add --cold-starts 3 --sleep 660 for three measurements, each after an eleven-minute idle period. Keep the container's sleep timeout below that interval. Other users or uptime checks can keep it awake; these measurements do not establish performance in every region.

Optional external and large tests

QUASO_SMOKE=1 npm test
npm run test:llm

The first enables the 50,000-string / 30-language local stress case. The second needs GEMINI_API_KEY, makes real provider requests and may use paid tokens; it evaluates the sample texts and writes results for review. Never make either a requirement for a small unrelated patch. A test-provider timing result is not evidence of translation quality. Real staging deployments, regional latency, cold starts, D1 Time Travel and native-speaker evaluation remain separate release checks.

Beta 2 workflow on the production image

Run npm run docker:smoke to build and check the image, then npm run docker:workflow for all nine workflow steps through the image's server, real browser and CLI. To test another local image, run node scripts/docker_workflow.ts IMAGE after npm run build:cli.

The runner starts disposable containers on a dedicated Docker network. A TLS fixture owns generativelanguage.googleapis.com on that network; the app trusts its temporary certificate and uses a synthetic key saved through Settings. External DNS is disabled for the app. The fixture checks that every translation prompt includes proofread Spanish. Application code and the production image have no test-provider switch. Containers, network, certificates and data are removed on success or failure. The Acceptance workflow runs this check, and release publication depends on the same check passing. Real Cloudflare execution remains separate.

Accessibility review

Quaso aims to meet the Web Content Accessibility Guidelines (WCAG) 2.2 at level AA. The source and automated tests cover specific controls and states; they do not prove conformance across every page, browser, or assistive tool. Record the tested build and any unresolved problems in a release review instead of treating this list as completed evidence.

Start with the checks relevant to your change:

npx vitest run packages/design-system
npx vitest run e2e/accessibility.test.ts e2e/editor.test.ts e2e/design_system.test.ts

These tests cover defined theme contrast pairs, translation-state colours, keyboard behavior, focus after actions, dialogs, form labels, status announcements, and narrow layouts. The contrast tests check declared colour pairs; check the actual page too, including overlays and states introduced by the change.

Then review the affected workflow by hand:

1. Use only the keyboard. Reach Skip to content, navigation, inputs, actions, and dialogs. Check that focus remains visible and returns to a useful place after closing a dialog or deleting an item.
2. Check light and dark themes, 200% text zoom, narrow screens, and long text. Confirm that controls and error messages remain readable and reachable.
3. Check right-to-left source and target text, plural inputs, and placeholders. State shapes and labels must explain status without relying on colour.
4. Enable reduced motion and check animation behavior. Do not infer that every page respects the preference from the presence of a shared stylesheet.
5. Use VoiceOver and NVDA with someone who regularly uses a screen reader. Check headings, labels, navigation, live messages, and the full translation and review tasks. Record the browser, operating system, assistive tool, and findings.

The old website guide's Sprint 7 checklist was a development record, not a current completed audit. A fresh review is still needed before making a release-wide accessibility claim.