Gauntlet testing¶
Every release runs a gauntlet before a single claim ships: the server killed at every step of every write, tens of millions of seeded fault schedules, six nightly fuzzers, every file ever uploaded to PyPI, and security audits by every frontier model. The checks are public — run any of them yourself.
Real clients, real server¶
The test suite doesn't check that pypiron's output looks correct. It starts the real binary and drives it over HTTP with the tools your team uses: uv, pip, poetry, pdm, pipenv, hatch, flit, and twine. A test passes only when a real client publishes, resolves, and installs a real package against the running server. No mocks stand in for the clients — mocks would only test our assumptions about them.
The core three — uv, pip, and twine — run on every pull request, against local disk, S3, and Azure. All eight run in the weekly compatibility matrix (the client-by-feature grid, with exact client versions).
Every file on PyPI¶
pypiron parses filenames, wheel tags, and package metadata. If that parsing is wrong on even a rare shape, an install breaks. So every parser runs against every file ever uploaded to PyPI — all 17,130,626 of them — and matches ground truth on each one. The check re-runs weekly in CI, so new packages can't drift out from under it.
It survives being killed¶
The one failure that matters for a package server is an index that promises a file it can't deliver — a broken install. pypiron is built so an interrupted write can never leave that state, and three suites try hard to prove otherwise:
- Kill at every step. The server is
kill -9'd at each point of every write and must come back to a consistent, installable tree every time (crash sweep). - A node dies mid-upload. Several nodes share one bucket while one is killed in the middle of an upload; after restart every node serves byte-identical indexes and every acknowledged upload still installs from every node (fleet chaos).
- A hostile upstream. When the proxy fetches from PyPI, it's fed truncated, corrupt, hash-mismatched, and hanging responses. Each surfaces as an error and leaves nothing behind — no half-written file in the cache for a later request to serve as good (upstream faults).
Simulated disasters¶
The chaos suites kill one real process at a time. Two more suites go further:
- Deterministic simulation. A whole multi-node fleet lives out disasters — crashes, partitions, storage faults, clock jumps — faster than real time in a simulator, and any failure reproduces exactly from an 8-byte seed. 22,985,591 seeded fault schedules at the time of writing; zero outstanding findings.
- Model checking. What the simulator samples, a model checker settles: every interleaving of writers, workers, crashes, and byte conflicts within its bounds is checked exhaustively, not sampled. The checker runs the server's own decision code, so the proof can't drift from the binary.
Adversarial inputs¶
The code that reads attacker- or upstream-controlled bytes — filename and wheel parsing, metadata, index rendering, range requests — is exercised by six coverage-guided fuzzers that run every night. Each one hunts for a crash or a broken invariant.
Before release, a fuzzer found a real HTML-injection bug in the index renderer — a crafted name could break out of an HTML attribute. It was fixed, and the fuzzer that caught it now guards against its return.
Supply-chain hygiene¶
pypiron guards your supply chain, so its own has to hold up. A new security advisory anywhere in the dependency tree fails the build on every pull request. And the code is security audited by all frontier models — the same models that built pypiron ran audit pass after pass until the findings came back clean. All told, over $7,000 of frontier-model compute (at API list prices) went into building and hardening pypiron.
Benchmarks you can re-run¶
The throughput numbers aren't ours to grade. The benchmark rigs are published docker-compose setups for pypiron and all five competitors — bandersnatch, pypiserver, pypicloud, devpi, and proxpi. Clone the repo and run them. Results and the head-to-head feature grid are in Compare.
What runs when¶
| When | What runs |
|---|---|
| Every pull request | Format, lint, unit tests, the core client suite (uv, pip, twine) on local disk, S3, and Azure, plus the dependency-advisory gate |
| Nightly | All six fuzzers, coverage-guided |
| Weekly | The full-PyPI corpus check and the eight-client compatibility matrix — poetry, pdm, pipenv, hatch, and flit join the core three |
| Continuously | The deterministic simulator, on the order of a hundred thousand seeded fault schedules a night |
Every flag and its PYPIRON_* env var is in
Configuration.