Using pytest gates to accept AI-generated test cases
The article proposes that before AI batch-generated test cases enter the regression suite, they must pass three acceptance lines: assertion validity, data provenance, and failure attributability. It also provides a conftest.py implementation: test cases must use @pytest.mark.review to declare which gates they pass; when running pytest --strict-review, cases missing the marker are skipped, and CI blocks the merge if the skip count is nonzero. Using a coupon-claiming API as an example, the article compares fake tests with real tests, pointing out that cases asserting only a 200 status code or a non-empty result are as good as untested, and that expected values should be traceable back to business rules or online sampling. The author also explains that the three lines can only filter out invalid test cases; business invariants still need to be mined from requirements by humans.
Why read this
AI score 62
The article breaks down the three types of false greens in AI batch-generated test cases into mechanically checkable acceptance lines, and provides a pytest gate implementation that can be directly hooked into CI.