independent · multimodal AI · VLM evaluation · reliability
AeroSynth-Eval tests whether generated aircraft inspection images improve performance on untouched real photographs. Its real-image evidence comes from the public AGDD canopy dataset; generated images remain controls or augmentation.
Added a frozen-manifest runner comparing real-only training with equal-budget random and evaluator-selected augmentation over matched seeds. It checks source-group leakage and defect recall as well as F1. The implementation has fixture tests; a real augmentation study and independent uncertainty analysis remain pending.
The public AGDD track uses 197 training pairs and 22 untouched validation pairs. Across ten equal-budget seeds, mixed training improved mean macro F1 from 0.3881 to 0.4419, but mean crack recall fell from 0.4000 to 0.3167. That regression means the augmentation is not approved as a safety improvement.
The separate DLR track uses 3,224 real inspection images from a public MIT release. After rebuilding the split to hold out nearby capture sessions, the first 645-image test run reached 0.9777 dent recall but only 0.5969 ROC-AUC because of false alarms. It is a baseline for further work, not a maintenance claim.
The evaluator-development track now materializes 1,735 pairwise judgments from the official GenAI-Bench image-generation test set. Prompt-grouped deterministic splits contain 1,246 training, 254 validation, and 235 held-out votes. This gives the project genuine human preference labels for general generated-image quality. It does not tell us whether an aircraft defect is correctly identified; that still needs qualified aerospace reviewers.
The separate generated benchmark spans aircraft regions, anomaly types, materials, backgrounds, lighting conditions, and capture profiles. Exactly 12 scenarios are protected test cases so development decisions cannot silently consume the evaluation set.
Every scenario has a generated PNG with recorded generation specification, seed, renderer version, timestamp, and SHA-256 digest. Registry and manifest checks reject missing assets, mismatched metadata, and corpus drift before evaluation.
The repo contains a balanced development-only annotation queue and validated reviewer-response contracts. Synthetic rater fixtures test the agreement-analysis pipeline in CI but are explicitly excluded from any claim about human agreement.
The optional Apple-Silicon runner processes one development asset at a time only after validating corpus and image integrity. It binds the response to the exact asset, scenario, rubric, model, runtime, prompt, and image digest.
The project does not claim human-rater agreement, calibrated VLM accuracy, or operational inspection capability. AGDD is a small noncommercial benchmark. The next DLR aircraft-dent track has a verified MIT licence and a reported 6,000+ images, but it remains an acquisition plan until the archive, annotations, duplicates, and group-safe split are audited.
The P100 run attempted every fixed development case and raised schema success from zero to eight after the output contract was hardened. This proves the Linux/CUDA batch path and records the remaining structured-generation failures; the next evidence is independent human rating, a second VLM baseline, agreement analysis, and protected-test evaluation.