lcert_verify.stream

Verify a bundle without holding all of it in memory at once.

verify_bundle parses bundle.json into Python objects. That is fine until the loci arrays get large: a locus is a handful of floats, each a 24-byte Python object inside a list, so peak memory runs roughly 9x the file size. Measured, an 800,000-locus bundle is 22 MB on disk and 210 MB resident.

This module walks the document incrementally and verifies one certificate at a time, discarding each before reading the next. Peak memory becomes the file plus the largest single certificate, rather than the file plus everything.

When it helps, and when it does not. Both numbers below are the same 22 MB file with the same 800,000 loci, differing only in how they are grouped:

The second row is the honest limit: nothing can stream inside one certificate, because its loci arrays have to exist before they can be checked. If your bundles are one enormous certificate, this module buys you almost nothing, and the fix is to split the certificate — which the format has always allowed.

Three design choices, all deliberate:

It is not in the frozen verifier. _verifier.py is one stdlib file small enough to read in full, and that auditability is the package's headline property. This module calls the same checking functions the ordinary path calls; nothing about how a certificate is judged is reimplemented, only how it is reached.

It uses the standard library's own incremental decoder. An earlier version scanned bytes by hand to find value boundaries. It was correct and it was four times slower, because a 22 MB byte-at-a-time Python loop costs more than the parse it was avoiding. JSONDecoder.raw_decode does the same job in C.

It requires a trust anchor. The ordinary path also checks that bundle.json round-trips through the canonical serializer, which needs the whole document at once. Streaming cannot do that, so it does not pretend to: with no fingerprint it returns UNVERIFIED. With one, byte identity is established exactly, which is strictly stronger than the canonical check — the same argument the JavaScript implementation already makes.