expect(true).toBe(true) passes every time. Probe flags tests like that — assertions that can never fail, skipped tests, deleted tests.
What it catches
Patterns like expect(true).toBe(true) or assert True — a green tick whatever the code does.
Tests removed or commented out during refactoring. If the agent deleted a test file, you'll see it flagged with the coverage it previously covered.
API keys, passwords, and tokens found in new test files or fixture data that the agent created during the session.
Tests marked skip, xit or @pytest.mark.skip in the change — they run nothing and still leave the suite green.
How it works
Collect the new tests
We gather every test file the agent added or changed this session — nothing else gets scored.
Look for tests that cannot fail
We flag assertions like expect(true).toBe(true), skipped tests, and test files that mock most of what they call.
Check deletions against the repository
A deleted test file is checked against the repository, so a test that was only moved is not reported as lost.
Example output
Probe
Fake assertion
expect(true).toBe(true)auth.test.ts:47 — passes whatever the code does
Deleted test
auth.integration.test.ts — removedCovered: login flow, token refresh, session expiry