# Test each pull request like a user with an AI tester

Turn on tested deploys: an AI tester uses each pull request's preview like a user, signs in, writes checks and holds production while one is red.

Tested deploys put an AI tester on each pull request. It opens the pull request's [branch preview](https://console.t74.sorv.dev/docs/operate/previews) in a browser, signs in, uses what the pull request changed the way a person would, and writes checks that run again on later pushes. Everything here is also an `ox tests` command ([the full list](https://console.t74.sorv.dev/docs/cli#tests)).

With tests on, each push to a pull request starts a run on its preview: an AI tester signs in, uses what changed in a browser and writes checks that run again on later pushes. The result shows on the pull request as the `ox tests` check, and Not a bug lets a held deploy go.

## Tested deploys in five steps

1. **Turn them on.** Turn on Tested deploys in Settings, Beta features, then choose On on the project's Tested deploys card ([what it needs](https://console.t74.sorv.dev/docs/operate/tests#turn-on)).
2. **Open a pull request.** The tester uses what it changed on its preview and writes checks. The result shows on the pull request as the `ox tests` check.
3. **Read a red.** A failed check says what a user would hit, with the error, what was expected and received, and a picture. Bug? Push a fix ([read a run](https://console.t74.sorv.dev/docs/operate/tests#runs)).
4. **Not a bug.** When the red is on purpose, click Not a bug. Once every red of the run is marked, the run counts as passed.
5. **Production waits.** With On, production deploys a commit only once staging passes the saved checks. A red holds it and says why. Report only, never hold runs the same tests and never holds.

## Turn tested deploys on

Tested deploys are off until you turn them on, first for your account in Settings, Beta features, then for a project. They work on every plan, Free included. Before ox lets you, the project needs:

- [Branch previews](https://console.t74.sorv.dev/docs/operate/previews#previews-on) on, since the tester uses the pull request's preview.
- A repository connected through the ox GitHub App, since runs follow pull requests.
- A [staging copy](https://console.t74.sorv.dev/docs/operate/previews#staging) that tracks the branch production deploys, where every commit bound for production is checked first.
- A server with room for the test browser and one preview in the half of its memory that previews share. ox names the size it needs when there is not.

In the console, the Tests card on the project's Settings tab lists these while tests are off, each with a link to fix it, and lets you pick On once all are done. Picking On shows what each run sends; tick I agree and save. Settings, Beta features also links each project's Tests card.

The first time, add `--accept-data-sharing`, which agrees that the AI tester reads your pull requests and previews (the [terms](https://console.t74.sorv.dev/terms#tests) list what is sent):

Turn tested deploys on:

```sh
ox settings beta tested-deploys on
ox tests shop on --accept-data-sharing
ox tests shop
```

There are three settings. **On** holds the deploy while a check is red. **Report only, never hold** (`ox tests shop report`) runs the same tests and only tells you. **Off** stops them. Turning tests off, for the project or for your account, completes the `ox tests` check a stopped run had open as neutral, so a required check does not block the merge. Turning them back on tests each open pull request again. `ox tests shop` prints the setting, the testing credit, how signed-in testing works, GitHub's status, the held commit, the queue and the last runs.

Tell the tester what your users do in one sentence or a few, up to 500 characters. It reads this as data:

Say what the app is for:

```sh
ox tests shop about "Shop owners list products, and buyers pay with a card."
```

## Let the tester sign in

Most apps keep their real pages behind a sign-in, so the tester signs in before it tests. It tries these ways in order, and `ox tests shop signin` says which one works or what to do. `ox tests shop signin check` proves it again with a run.

### 1. Test accounts from the copy's variables

Many apps make an admin from variables at start, such as `ADMIN_EMAIL` and `ADMIN_PASSWORD`. Name those variables once, and ox signs in with them once per run on each copy, so a sign-in cap never locks the tester out. You give the variable names, never a value: ox reads the values from the copy, never shows them to the AI, and refuses other text before sending it.

On the Tests card, open Test settings, fill one row per account under Test accounts and Save accounts; from the CLI:

Name a test account:

```sh
ox tests shop accounts set name=admin,role=admin,email-var=ADMIN_EMAIL,password-var=ADMIN_PASSWORD
ox tests shop accounts
```

Each account is one user, under its own name. A project has at most 4 accounts. A name is lowercase letters, digits and hyphens, starting with a letter, up to 12 characters. Use `email=ADDRESS` when the address is not secret, and `user-var` when your app signs in with a user name instead: staging then gets a made user name, not a `tester@` address. Use `user=NAME` when the user name is fixed in your app, such as an admin always named `admin`: a user name is not secret, so it needs no variable. A preview's database starts empty, so this works when your app's release or seed step makes that account from those variables. Make that step safe to rerun, finding the account by its email or user name. An admin made only on the app's first start works on staging too: a new staging copy's first deploy holds its own value for each of these variables production keeps to itself or lacks, yours or one ox makes. When staging was already running before such a variable got its value, the Tested deploys card says so and offers [Remake staging](https://console.t74.sorv.dev/docs/operate/previews#staging).

### 2. A sign-in helper command

The sign-in helper is a command that runs on your server with your app's settings and makes a test user. ox gives it `OX_USER_NAME`, `OX_USER_EMAIL`, `OX_USER_PASSWORD` and `OX_USER_ROLE`, and it must exit 0. It runs again on later runs, so it should succeed when the user is already there.

Set the sign-in helper:

```sh
ox tests shop signin-helper "python manage.py make_test_user"
ox tests shop signin-helper --clear
```

For an app that signs in with a link or a code sent by email, or only with another site, let the helper print a sign-in link your app honors as its last line, `OX_SIGNIN_URL=` followed by the link. The link must be on the copy's own host. ox opens it instead of filling in a form.

### 3. The app's own sign-up

When anyone can sign up, the tester makes its users through your app's sign-up form.

When none of these works, the run still tests the pages a visitor sees, and its summary starts with **signed-in pages not tested** and the reason.

When a check the tester wrote for the change could not reach the app because its user did not sign in during that run, the run is not a plain pass either: its summary starts with **signed-in pages not tested**, the user that failed and the fix, and the check on GitHub is neutral. The run still passes, so production is not held.

## Keep preview keys test-only

A preview starts with production's variables and the tester clicks as a user would, so its keys must be test keys. While a copy holds production's value of a credential (a provider's key, a `*_KEY` name, a remote database), its runs wait. Mark the key [production only](https://console.t74.sorv.dev/docs/operate/variables#production-only), give the copy its own value, or confirm it is a test key. The Tests card lists each such key at its top with those three actions:

Find and confirm keys:

```sh
ox tests shop keys
ox tests shop keys confirm MAIL_KEY
```

A live Stripe key, or a value that names production's address, can never be confirmed.

## Read a run, then fix it or mark it Not a bug

A run's summary says what a user would hit, what was checked and works, and what was not tested. A pass that tested none of the change or stopped early, unless the tester said nothing shows, starts with **Not tested: no check covers the change** and its GitHub check is neutral. When a change touches a worker or cron job and the tester wrote no check, the summary says **Not tested: a background job changed** and names the job, and the check on GitHub is neutral. When the change is too large to list every file, it says a job **may have changed**. That never holds production. For each red check, `ox tests shop run ID` prints the error, what was expected and received, the failing step's line and the page's text at the failure. It also says why a run ended in an error or was skipped. `ox tests shop artifacts RUN DIR` saves the failed checks' pictures.

In the console, a run's page is titled by what it tested, such as **PR #18 · fix/cart-total** or **Staging · 3f9a2c1**, links its preview and pull request, and says what it blocks. While a run waits or runs, its page updates itself and says what it is doing. Checks that did not count and parts left untested share one **Skipped** list, each with its reason, such as "unstable", "failed, cause unclear" or "already broken on main". A preview's page and staging's show their latest run.

A red check holds production at the commit that has it. When the red is a bug, push a fix. When it is not a bug, mark it Not a bug, with a reason of up to 200 characters if you want to give one. Once every red of the run is marked, the run counts as passed and the held deploy goes ahead. A check the pull request proposed then leaves the branch. ox remembers the exact result you accepted: the page, the element and the two values. After the merge, a red with that same result anywhere in the project reads "matches Not a bug" and does not hold anything; any other failure stays red. Undo it on the Tested deploys card to be asked again. The sign-in recipe is never Not a bug: a run that finds it broken proves a new one, or `ox tests shop signin check` does. ox counts each Not a bug as a wrong red when it measures how often the tester is wrong.

Mark a red check Not a bug:

```sh
ox tests shop runs --branch fix/login
ox tests shop not-a-bug 412 signup.spec.mjs --reason "The new limit is on purpose."
ox tests shop not-a-bug 412 --all
```

A held production says why on the project's overview, in its Projects row and on the Tested deploys card. The Tested deploys card opens the run that holds it, shows its red checks, links the commit for a fix and has one button, **Not a bug, deploy it**. On the overview, the New commit's button reads **Deploy anyway** and deploys the commit at once, without its tests; only auto-deploy waits. When the newest run failed, the overview also says what failed and opens that run. `ox tests` and the alert say the same with the command to run:

What a hold says:

```text
Console: Production is waiting for 3f9a2c1 because "signup works" failed on staging.
ox tests: Production is waiting for 3f9a2c1 because "signup works" failed on staging. Bug? Push a fix. Not a bug? Not a bug deploys it (ox tests shop not-a-bug 412 --all).
```

When the pull request means to change what a saved check expects, such as a button's label, the tester updates the check without asking you. Its title, description or a commit message must name each new value in words the tester quotes, the old check must still pass on staging and the new one fail there, and the new one must pass 3 times out of 3 on the preview. A run updates at most 12 checks asserting at most 5 new values, so a rename updates every check it touches, its summary opens with each update and the words it quotes, and the `ox tests` check shows neutral rather than green, so you read it before you merge.

When a preview does not build, the pull request's `ox tests` check is neutral, titled **Not tested: the preview did not build**, with why and the next step. So is a preview deploy that ox could not finish, even after deploying it again once. A commit that already passed or failed keeps that check. A build out of the previews' memory share names the share and the fix.

When staging passes the newest commit, production deploys it right away. A commit with nothing to test does not wait. When the project has no saved checks yet, or staging serves no web address, the run on staging ends Skipped, and production deploys the commit, not tested. The run on the Tested deploys card says why it was skipped.

A preview runs its branch merged into main ([what a preview runs](https://console.t74.sorv.dev/docs/operate/previews#merged-into-main)), so a branch that is behind main still runs the checks main saved. When the branch conflicts with main, those checks cannot run there: the summary says "3 saved checks not run: conflicts with main", and the `ox tests` check is neutral, never green. Resolve the conflict and push. When main saves checks after the preview was built, the summary says to push again to run them.

When staging serves a commit that lacks the commits that saved some checks, such as after the branch was rewritten, the run leaves those checks out, says how many, and holds the commit. Check again would leave them out again, so push a commit that has them, pause them, or use Deploy anyway.

While staging's tests have not ended, the commit is waiting, not failing, and the Tested deploys card shows it in progress, not red. A hold for a key that staging shares with production names the key and `ox tests shop keys`.

When a held commit has no red to judge, such as when staging failed to deploy or GitHub or the network cut it short, `ox tests shop deploy-anyway --reason R` lets it ship untested once. A commit held by a red is refused, and the refusal says to use Not a bug. A commit that staging will not deploy by itself, or that waited for staging for over 150 minutes, longer than a head run waits for a slot and runs, is held that way too, and ox alerts you. `ox tests shop pause CHECK --reason R` stops a saved check from holding deploys, and `resume CHECK` lets it hold them again.

A failure that passes in a moment, such as a server's ox agent that dropped or GitHub not answering, never holds production by itself. The run is run again once, shown as "retrying once", and production waits for it. When ox cannot read the tests' state, it tries again at the next poll, with no alert, and only when that lasts over 150 minutes does it hold the commit and alert you.

A run waits while a deploy, a restore or another job runs on its preview or staging, and starts when that job ends. A preview that does not answer within 60 seconds, by the health check its deploy uses, ends the run in error before the tester starts. While a run compares checks on staging, no other run uses staging, and when staging deploys or runs another job during the comparison, those checks end in error. When a check's user could not sign in and the sign-in works the next time, the check runs again and that decides it.

### When a run reaches a limit

A check that runs the test browser out of its 1.5 GB ends with **the test browser ran out of its 1536 MB of memory** and is not retried, so split or pause it. A replay that runs out of time names the checks that never ran and the slowest that did, so pause those.

### The check on GitHub

When the ox GitHub App may write checks (`checks: write`), each pull request's run shows on it as the `ox tests` check, and marking a check Not a bug turns it green. When a run is red, the check says that merging with it red holds production until a fix passes or someone clicks Not a bug, with a link to the run in the console. Without that permission, the Tested deploys card asks you to accept Checks: Read and write in the ox GitHub App's permissions. For a public repository or a password-protected copy, the check holds only the state, the counts, each red check's name and finding in a user's words, and ox's own reason and next step for a skipped or error run.

While a run waits, the check says what for: a free test slot, the run before it, the test tools being installed, the AI budget or the next UTC day, or, titled Waiting for the owner, a paused plan or a key that holds production's value or could not be read. When GitHub fails to take a check, ox sends it again, waiting longer each time, so the check is not left in progress. When a pull request merges, ox saves its checks when GitHub tells it, and if that word is lost, within minutes when ox next asks GitHub, up to a day after the merge.

## Record test videos

Turn on **Record test videos** in Test settings, shown while the setting is On or Report only, never hold, or run `ox tests shop record on`. After each verdict ox films one check: the first that failed, in any run, or the first proven new check of a pull request that passed, never the sign-in check. A passed staging or replay run films nothing. It films after the verdict, so a video never changes a result. A merged pull request's preview stays until its video is made. Videos stay on your server: a project keeps its newest 10, within a fifth of the server's free space, with no expiry. A video is not kept when it showed a secret ox knows, or when the page hid text in a closed shadow root, which ox cannot check. The user name or email a test account signs in with is no secret here, as almost every app shows who is signed in; its password is. Watch a video before you post it, as it shows your preview's data and test users. Play one under Recordings on the Tested deploys card, on its run's page, or with Watch on the Last run and Held rows; a run that was not filmed says why.

## Cron jobs

When a project has cron jobs, the tester can run them. For a change to a job, it runs the job on the preview, checks its exit status, then checks what the job should have changed on a page or through the app's API. A saved check that runs a job is replayed like any other, so each copy it runs on runs its own jobs.

A job runs at most 5 times in one batch of checks, one at a time, and ox waits up to 30 seconds for it to end, never past the time the check has left. A job that exits with an error is the check's answer: a check that expects success is red. A job ox could not run, because it is already running, the project is stopped or busy with a deploy, it did not end within 30 seconds or before the check ran out of time, or no job has that name, is reported as an infrastructure error and never as a red check. ox shows the tester a job's exit status and never its output. A saved check that runs a job is left out of a replay on a server whose ox agent is too old to run jobs, and the run says so.

## What it costs

AI testing is prepaid credit at cost: you pay what the model costs ox, with no markup. Credit never expires and is not refunded. Each pull request's run uses what its AI calls cost, from your account's testing credit, which you buy USD 100 at a time in Settings or with `ox plan credit`. When your credit is less than the most a run may cost, or at USD 10 of AI in a day for your account, a pull request's run only replays the saved checks; at a daily limit for all of ox, runs wait for the next day. Production deploys never wait on your credit. When it is out, saved checks still run on every pull request and on staging, and the AI writes no new checks until you add credit. A run's cost shows in cents, as "under $0.01", or as "free" for a run that used no AI. The Testing credit card lists each run with its pull request, result, cost and date. ox emails you once when your credit falls to 30% or less of what you had after your last top-up.

See and add testing credit:

```sh
ox plan usage
ox plan credit
```

## For AI coding agents

A coding agent can push, wait for the run and read what failed, all with `--json`:

Push, wait, read:

```sh
ox tests shop run --branch fix/login --commit "$(git rev-parse HEAD)" --wait --json
ox tests shop artifacts 412 ./shots
```

`run --wait` exits 0 when the run passed (Not a bug included), 1 when it failed, 5 when it ended in error, 6 when it was skipped or superseded with no run after it, 7 when the wait ended before the run did, and 8 when the plane refused. Any token can read runs. A token with deploy access can also run `again`, and every other change needs full access, so an agent that pushes and waits cannot mark its own red Not a bug.

## What is sent and kept

The checks run on your own server against the preview and the staging copy, never against production, and ox never stores a test user's password. To write and judge the checks, ox sends the pull request's text and changes, and the text of the preview's pages, to DigitalOcean's AI inference. The [privacy policy](https://console.t74.sorv.dev/privacy#tests) lists all of it, with how long each piece is kept, and [Security](https://console.t74.sorv.dev/docs/security) covers how the copies are kept apart.

Last updated 2026-10-25. The page as HTML: https://console.t74.sorv.dev/docs/operate/tests