cert_atlas.score

Score a verifier against the atlas.

THE METRIC, and why it has two halves.

A verifier that rejects everything catches every forgery. A verifier that accepts everything never raises a false alarm. Either is useless, and either can top a one-sided leaderboard. So a submission is scored on both and ranked on the minimum:

detection   = invalid cases correctly REJECTED / all invalid cases
precision   = valid   cases correctly ACCEPTED / all valid cases
atlas_score = min(detection, precision)

A verifier claiming soundness must score detection = 1.000. Anything less means a forgery in this atlas gets through it. Precision below 1.000 means it rejects artifacts that are genuinely fine — which in practice gets the verifier switched off, so it is a soundness problem one step removed.

Per-defect results are always reported alongside the aggregate, because which case you miss matters more than how many.

VerifierFn = typing.Callable[[str, dict], bool]

def load_index(atlas_dir) -> dict:

def score( atlas_dir, verifier: Callable[[str, dict], bool], *, families: Optional[List[str]] = None, jobs: int = 1) -> dict:

Run verifier over every case and compute the metric.

verifier returns True if it ACCEPTS the artifact. Exceptions count as a rejection, since a verifier that crashes has not accepted anything.

jobs > 1 runs cases concurrently in threads. That is worth it only for a verifier invoked as a subprocess, where the cost is interpreter startup and the GIL is released while waiting — an in-process verifier gains nothing and may lose. Results are assembled in index order regardless, so the score, the row order and the missed list do not depend on scheduling.

def command_verifier( argv_template: List[str], accept_returncode: int = 0, timeout: float = 30.0) -> Callable[[str, dict], bool]:

Adapt an external CLI to the scorer.

argv_template uses {path} as the placeholder, e.g. ["my-verifier", "--check", "{path}"]. Acceptance is signalled by exit code.

A verifier that hangs is scored as a rejection after timeout seconds rather than blocking the run forever — an answer that never arrives is not an acceptance.

def format_report(res: dict, *, show_rows: bool = True) -> str: