independent · search & retrieval · GenAI evaluation · ML systems
AeroRAG-X is my independent extension of the Delta Air Lines-sponsored Project HERO: an evaluation-first technical knowledge system built on public NASA NTRS material. I use it to study retrieval quality, grounding, model adaptation, distributed training, high-throughput serving, agentic tool use, system reliability, and deployment as separate measurable problems.
The retrieval pipeline combines BM25 and MiniLM dense retrieval, fuses rankings with Reciprocal Rank Fusion, reranks a bounded candidate set with a cross-encoder, and preserves document, page, URL, checksum, and evidence identity throughout the answer path. An evidence-sufficiency gate can refuse before generation when the retrieved material is not strong enough.
I started with a frozen QASPER evaluation over 281 NLP papers and 888 answerable questions. A simple within-paper TF-IDF baseline reached 76.24% any-evidence recall@10. SciFact then tested global scientific evidence retrieval and reached 89.89% evidence-document recall@10. I also imported the checksums and label distribution for the official TREC 2024 RAG release: 20,283 retrieval judgments and 2,840 citation-support judgments. Those labels are independent, but they are not aerospace-specific.
For the NASA corpus, I replaced the correct source in 200 frozen cases with a retrieved but non-relevant chunk. The provenance guard rejected all 200. That result covers source identity, not subtle semantic overstatement. A richer 50-case author-audit packet is ready, but its manual decisions are not filled in or claimed as complete.
On the frozen 32-query study, both Base + grounded RAG and LoRA + grounded RAG completed all queries with 100% measured answerability accuracy, unsupported-query refusal, claim citation coverage, citation-reference validity, source-document coverage, and structural validity under the defined benchmark.
LoRA increased formal answer decomposition while preserving those measured system-level reliability properties: 32 to 53 formal claims across the 20 answerable grounded cases.
On the later protected quality analysis, conservative expected-concept coverage increased from 38.16% to 51.32% and full answer-to-claim capture from 10.00% to 45.00%. Strict claim-to-evidence support remained broadly similar at 65.62% vs 67.92%. The repo also preserves three contradicted LoRA claims rather than hiding them.
The first bounded adaptive-retrieval policy made the protected result worse: answerability fell from 91.67% to 83.33% and unsupported refusal from 83.33% to 66.67%. A separately evaluated scope-qualifier safeguard later improved its own held-out set from 50.00% to 92.86% answerability and 40.00% to 100.00% unsupported refusal. These are kept as separate experiments rather than merged into one flattering story.
A typed tool registry now supports retrieval, source inspection, evidence-sufficiency checks, citation validation, and multi-source comparison through a dynamically routed LangGraph agent. Graph-step, tool-call, and retrieval budgets are explicit. Checkpoints support bounded human review, fault injection covers timeout and dependency failures, and Agent, Retrieval, and Inference services communicate through typed asynchronous contracts with trace and evidence identity preserved.
The repository now contains a matched single-GPU versus two-GPU PyTorch FSDP experiment for the same assistant-only Qwen workload. The trainer implements mixed precision, gradient checkpointing, deterministic distributed sampling and rank-aware seeding, full parameter/gradient/optimizer sharding, sharded checkpointing, resume, and reduced metrics for loss, throughput, step time, memory, and checkpoint behavior.
The experiment is implemented but has not been run on the required two-GPU CUDA host. Its report therefore contains a preregistered setup and pending cells, not invented scaling numbers.
vLLM, SGLang, and TensorRT-LLM are integrated behind the same structured-generation boundary as Transformers, so retrieval, reranking, evidence sufficiency, prompt hardening, and output validation remain fixed. The streaming benchmark covers concurrency 1, 8, 16, and 32 and compares normal requests with repeated moderation-policy prefixes, measuring throughput, output tokens per second, p50/p95 latency, TTFT, TPOT, failures, and GPU memory.
This is a runnable controlled study, not yet a claim that one serving engine won on a particular GPU.
The matched Qwen workload now has PyTorch FSDP, DeepSpeed ZeRO-3, and Megatron-LM tensor/sequence-parallel treatments. Each path records its checkpoint and restart boundary. FSDP remains the primary study; the additional integrations exist to compare memory, communication, conversion cost, and operational complexity under the same objective.
A controlled local comparison measured Transformers MPS float16 at 715.11 ms mean latency and MLX affine 4-bit at 278.43 ms on the same Apple-Silicon host, with artifact sizes of 1448.83 MiB and 313.10 MiB respectively. The token totals differ across runtimes, so this is a runtime/footprint benchmark rather than a model-quality comparison.
The FSDP, DeepSpeed, Megatron, vLLM, SGLang, and TensorRT-LLM GPU matrices still need executed repetitions and raw results. The GRPO work currently provides reward contracts, anti-reward-hacking tests, bounded tool environments, and Base/LoRA/GRPO ablation contracts; it does not claim a successful RL improvement yet.