An agent finishes a change and tells me all the tests pass. That sounds reassuring, but I still want to know which tests it ran, whether it ran them after the last edit, and whether the result holds outside its session. I use coding agents to write much of the code in my projects. Code arrives quickly, and I need a repeatable way to check it before I decide it belongs on the main branch.
Through developer programs, I’ve had access to unreleased models and the opportunity to test them before they become widely available. So I built a local benchmarking tool that runs prompts through different models and saves the data locally for analysis. A Python backend reads the results, and a React dashboard displays them. I needed accurate measurements, and I needed to keep unreleased model data out of the public repo. The results stay in a local folder covered by .gitignore, and the tests use fixtures instead of my private runs.
But sometimes I forget something, or an agent makes a change I miss in review. CI matters to me because it makes checks happen every time, without relying on either of us to remember each step.
This post walks through the checks I run and what each one catches, with a few examples. They are specific to my project, but if you use coding agents, you may want to consider a similar approach. Write down the rules that matter, then back each one with a check that fails when it’s broken.
Turning instructions into checks #
I keep a set of instructions for coding agents: support Python 3.11 or newer, use the Python standard library, rebuild the dashboard after frontend changes and never invent a measurement. The standard library, often shortened to stdlib, is the set of modules that ship inside Python with no install step, such as json, argparse and unittest. A stdlib-only rule means a project uses only those built-in modules and adds no third-party packages. Instructions like these are requests, and an agent can miss one, especially deep into a long session. CI gives each rule something behind it, a check that fails when the rule is broken.
Take Python compatibility. An agent might use itertools.batched to split a list into chunks. It is part of the standard library, so the choice looks reasonable, but it arrived in Python 3.12 and my project supports 3.11. That mistake can survive a successful local run on a newer interpreter, because the newer one has the function. The CI workflow runs the Python suite on both 3.11 and 3.15, so an import that fails on the older version produces a failed check on the pull request. Here is the failure, from running a one-line import on Python 3.11:
$ python3.11 cfg.py
Traceback (most recent call last):
File "cfg.py", line 1, in <module>
from itertools import batched
ImportError: cannot import name 'batched' from 'itertools' (unknown location)
The python3 on my PATH is 3.15 and loads the same file without complaint. That is the whole problem, because locally everything looks fine. With the matrix in CI, I can see whether a change meets the requirement without having to ask the agent which interpreter it used.
What “all tests pass” actually means #
Continuous integration runs a defined set of checks against a committed revision. In this project, GitHub Actions runs the workflow on pull requests and on pushes to main, which gives me a record of which commands ran, which code they tested and what happened. Running on pull requests means the checks start as soon as the agent opens one, so nobody has to remember to run them or take its word that it did. Running on pushes to main retests the result after a merge, which catches two changes that were each fine alone and broke each other together.
Without that record, “all tests pass” leaves room for interpretation. The agent might have run one test file, tested before its final edit or relied on a package already installed on my machine. During a long session, it is easy to lose track of which result belongs to which change. The pipeline repeats the procedure in a separate environment, starting from a fresh checkout every time. When something fails, I can give the agent the failing command and its output, then put the next revision through the same checks, which makes the result easier to reproduce and the failure easier to investigate.
What the pipeline checks #
The whole pipeline is one file, .github/workflows/test.yml, with two jobs. They start at the same time on separate Ubuntu 24.04 machines, so one push can report a Python problem and a frontend problem together.
| Check | Job | What it catches |
|---|---|---|
| Python tests on 3.11 | test | Code that needs a newer Python than the oldest version I support, and any import outside the standard library |
| Python tests on 3.15 | test | Code that works on the old interpreter and breaks on the newest one |
| Frontend tests | frontend | Broken logic in the dashboard’s formatting, filtering and comparison code, and views that render the wrong text |
| Install from the lock file | frontend | A package.json and lock file that disagree |
| Build the dashboard | frontend | A frontend change that no longer builds |
| Bundle matches source | frontend | A committed web/app.js that is out of date |
The workflow runs on every pull request and every push to main, and on nothing else. That is a fix from early on. The first version ran on every push, so a branch with an open pull request ran the tests twice, once for the push and once for the pull request. One added line restricting pushes to main stopped that.
The test job runs python3 tests/test_tool.py once for each of two Python versions. I picked 3.11, the oldest version the tool has to support, and 3.15 because it is the newest version, so it catches code that works on the old interpreter and breaks on a newer one. The tests are one file of unittest cases, so there is no pytest and no pip install. The job installs nothing, and that is what turns the stdlib rule into something that fails. A test backs it up by reading every import in the launcher, the modules in lib/ and the test file, and failing on anything outside the standard library. The launcher and the server also guard the floor themselves. On anything older than 3.11 they exit with a “needs Python 3.11 or newer” message, and tests check that guard.
The frontend job runs on Node 24. npm ci installs exactly what the lock file says and fails if package.json and the lock file disagree. npm test runs Node’s built-in test runner, so there is no Jest or Vitest. npm run build uses esbuild to bundle the React source into one file, web/app.js, and the last step compares that fresh bundle with the committed one.
Nothing on that list is exotic. The value is that all of it runs every time, in the same clean environment, with nothing left over from an agent’s session. The suites are also quick. On my machine the Python suite runs 228 tests in about ten seconds and the frontend suite runs 109 in a fraction of a second, so nobody has a reason to skip them.
What the tests are watching for #
A pipeline is only as useful as the tests it runs, and I write them around one question. What could an agent change that looks right and still does damage? An agent writes code that reads well, passes its own spot checks and sounds sure of itself. None of that tells me whether it broke something I depend on, and some breakage is hard to see in a diff, like a number quietly turning into zero or a server starting to answer requests from outside the machine. So the tests concentrate on the places where a plausible change could do the most harm.
They fall into three groups. The first guards the numbers. The Python tests run the real launcher and parsing code against fixtures, which are sample session logs and captured provider output, and they check that the results hold up when the input is messy, such as logs out of order, calls with no timing and retries. The second guards the boundary around private data. The server is meant to be read-only apart from deleting runs and to answer only on localhost, and the tests assert exactly that, including that requests from other sites are rejected. The third turns my instructions to the agent into failures: stdlib only, the Python floor and no semicolons in text users see. A rule that lives only in a document gets forgotten, and a rule that fails the build does not.
The frontend tests do the same job for the dashboard. Most of them check the pure functions that format and compare the numbers. A few render the real views to HTML in Node, with no browser, and check the text, so a change that shows unknown data as the word null fails before it ships. Each of these tests marks a place where a confident, plausible change could do damage without anyone noticing, which is why they are the ones I keep.
The mistakes that look fine #
Three of these checks catch mistakes that otherwise look fine. The first is a measurement that goes missing. If a provider’s log leaves out a measurement, the tool should preserve that absence. Turning it into zero could make a model look exceptionally fast, and a chart built on that zero would look as clean as any other. So two tests guard it from opposite sides. One checks that a call with no timing gives None for model time. The other checks that a real recorded zero stays zero.
def test_no_durations_and_no_api_ms_is_none_not_zero(self):
...
self.assertIsNone(data["model_ms"])
def test_a_real_zero_stays_zero(self):
...
self.assertEqual(data["model_ms"], 0)
The second is a bundle that drifted. The React source lives in frontend/src, but the bundle people actually run, web/app.js, is committed to the repo. If an agent edits a component and forgets to rebuild, the tests still pass and users still get the old page. So the last step of the frontend job rebuilds the bundle from source and runs this:
git diff --exit-code web/
Any difference fails the check. Here is the behavior in a throwaway repo, where the committed file said v1 and the rebuilt one says v2:
$ git diff --exit-code app.js
-bundle v1
+bundle v2
$ echo $?
1
The real bundle is much larger, but the signal is the same. A non-zero exit code turns the job red. It only works because the build is deterministic. React and esbuild are pinned to exact versions and installed from the lock file, so the same source produces the same bundle on my Mac and on the CI runner.
The third is a package nobody installed. If a change adds import requests, the Python job fails with ModuleNotFoundError, because no step ever installed it. I want the stdlib rule for two reasons. The tool should run on a bare machine with nothing installed beyond Python itself, and every dependency is code I did not write sitting next to data I cannot share. Agents also tend to reach for a package like requests by habit, so this check catches the convenient choice before it reaches main.
Money and private data #
Some providers in this tool bill real API keys, and some of what it records is output I can’t share, so the pipeline is built to touch neither. The workflow passes no secrets, so CI has nothing to bill with. The tests mock subprocess calls and CLI lookups, which means a test that exercises the Codex path never starts a real CLI. The test file also blocks .env, so importing the launcher can’t read local credentials. And CI never sees a real run, because the run folder is gitignored and the tests use fixtures. The Muse fixtures use made-up model IDs and prompts, and the captured Codex run comes with a note saying exactly what was changed in it.
Where the checks stop #
A green check means the checks I wrote passed. It does not mean the change is right. An agent can write a feature and its tests with the same wrong assumption, or turn a failing check green by loosening an assertion, adding a skip or deleting a test. The workflow file lives in the repo too, so a pull request can edit the checks that judge it. That is why I read the files that define passing first: the tests, the workflow and any new fixtures. For changes that affect the dashboard, I also open it myself, since the view tests render static HTML and not a real browser.
Fixtures deserve the closest look, because they are committed to a public repo and they are the one place a real run could slip in. There is one gap I know about. .gitignore stops accidents, but it does not stop git add -f or a real run copied into a fixture, and nothing in the pipeline checks for that yet, so it falls to review. The tool also reads log formats that belong to the providers, and my fixtures freeze a sample of each. If a vendor changes a format, the parsers can be right about the old one while every test stays green.
The pipeline needs tending too. It originally ran on ubuntu-latest, until that label moved to a newer release that no longer carried the oldest Python version in the matrix. That job then failed because the interpreter did not exist, not because any test failed. Pinning the runner to ubuntu-24.04 fixed it, and the lesson stuck: a job whose purpose is to test an old version needs a platform that does not move. The checkout and Python setup actions have also needed a bump to a newer major version to get off an older Node runtime. The workflow was written with an agent as well, and it still needed me to read it.
Running the checks yourself #
The commands match what the workflow runs, so CI should never be the first place a problem shows up.
python3 tests/test_tool.py
cd frontend && npm ci && npm test && npm run build
git diff --exit-code web/
On my Mac the python3 on my PATH is 3.15, so I also run the Python tests on a 3.11 interpreter installed through uv. That is the closest local match to the 3.11 job. Running npm run build rewrites web/app.js, so if git status shows it as modified, it goes in the commit.
What this gives me #
The agent writes the change and opens a pull request. CI runs the same checks in a clean environment every time and leaves a record of what ran. I read the result, send back anything that failed along with its output, and review the diff, starting with the files that define passing. Then I merge the change myself. Nobody in that chain grades their own work.
None of this stops an agent from being wrong. It makes the mistake visible before the change reaches main, and it does that the same way on every pull request, without anyone having to remember. If you use coding agents, the setup that suits you will look different from mine. The habit carries over: decide which mistakes would hurt, back each one with a check, and treat “all tests pass” as a claim until a clean machine confirms it.