Real engineering failures rarely show up as just software issues. Rather, they can emerge from interactions between software, hardware, and networking layers. They can show up as a crash after a config change, a vehicle reporting the wrong heading rate, or a message that loses metadata when it crosses a service boundary. Even worse, some show up once every 20 runs or only after deployment. Changes in battery voltage, temperature, or network timing can expose failures that never appeared under test conditions.
Finding the cause often requires an engineer to trace the failure across hardware signals, communication paths, and software execution. Such an investigation can involve manually checking datasheets, reviewing PCB layouts, reading design documentation, and running physical tests to understand where and how actual behavior diverges from the design. As AI coding agents contribute more code, hardware designs, and network configurations, AI-assisted review is becoming part of the engineering workflow. Extending that capability to system-level failures requires connecting each change to the hardware, interfaces, and operating conditions that determine how it behaves.
HyperBench is building AI agents that can reason across your entire stack to find and fix failures before they show up in production.
For an agent to take on this investigation, it needs to connect the evidence an engineer would normally assemble by hand. A reported symptom or failure has to lead to the relevant components, those components to their implementations, and those implementations to the requirements governing their behavior. Recent research has explored using agents to do this, with NVIDIA's ChipNeMo exploring how domain-specific design knowledge can support engineering questions, script generation, and bug analysis. Related work such as GraphRAG uses graphs to connect information across document collections. RepoGraph applies repository-level code graphs to existing software engineering systems.
HyperBench's system graph paired with our retrieval techniques makes these relationships available while the agent works. Our system indexes your code repositories, along with any additional documents, datasheets, and layouts you provide. It links components, signals, configuration, source code, and interface knowledge, with references to the evidence behind each connection.
To test our system, we evaluated parts of that workflow through 86 diagnostic questions and 23 debugging cases. The diagnostic questions examine two aspects of investigation: retrieving the relevant source evidence and reasoning from that evidence to explain a failure. The debugging cases extend the task to producing a repair. Together, they cover hardware configuration, historical failures, and bug fixes across software components, hardware-facing interfaces, and service boundaries.
Fault diagnosis
Debugging speed
How the comparison works
HyperBench agents use our system graph to navigate your codebase and connect it with hardware specifications, configuration, and design knowledge when investigating failures. Source bindings connect components to their implementations, and interface requirements describe what those implementations must preserve. A sensor-to-handler link narrows where to look.
The diagnostic evaluation checks evidence retrieval and explanations. The debugging evaluation lets agents edit code. Within each trace comparison, the model and task stay fixed while the graph adds context to the same workflow.
Evaluation setup, repositories, and scoring
| Study | Cases | Task | How we checked the result |
|---|---|---|---|
| Diagnostic evaluation | 86 | Retrieve evidence and explain failures | Evidence recall and blinded LLM review |
| Repository debugging | 23 | Investigate the failure and edit the code | Executable tests and blinded code review |
Both studies cover GPT-5.5, GPT-5.6 Sol, GPT-6 Astra, Claude Sonnet 5.5, Claude Opus 5, and Claude Opus 5.5. We compare agents working with and without HyperBench, with one attempt per case in each configured condition. For fault diagnosis, we use cited evidence and LLM-as-a-judge review to assess answer correctness. For debugging, we assess repairs using supplied executable validators or blinded code review, while tracking agent traces and time to completion.
The diagnostic questions cover hardware configuration in Zephyr, Klipper, Crazyflie, and ODrive, together with historical faults in Autoware's Universe, Core, and Launch repositories, as well as Nebula. The debugging cases cover service integrations and cloud infrastructure across Dapr and its component integrations, GitOps Engine, Flux CLI Utils, Grafana Mimir and Tempo, OpenTelemetry Collector and Go integrations, Temporal, Thanos, TiKV PD, Vitess, and OpenStack's Nova, Neutron, Cinder, Ironic, Ironic Python Agent, Keystone, Python Keystone Client, Keystone Middleware, Trove, and os-vif.
How system context reaches the agent
Configuration identifies how components connect, code defines how values are processed, and interface documents specify shared units, formats, and assumptions. The graph preserves these connections with references to the underlying evidence.
An additional custom built Rust hardware-retrieval pipeline helps index source-declared names and configuration relationships that improve on existing ranked text search.
What changes during an investigation
These matched traces are examples that show where context helped. The summaries describe the result. Expand a case for the exact question and the recorded investigation.
Autoware: identifying a crash the baseline misses
Deleting a diagnostic node leaves an orphan that startup code tries to publish. The baseline suggests a version mismatch and completes, while the HyperBench-assisted answer identifies the empty-parent access and proposes a correction.
Read the question and trace
The diagnostic graph aggregator crashes after a configuration update uses a remove edit. What is causing the crash, and how would you confirm the cause?
Without the graph, the agent reads removal code, configuration, and version history but misses the historical crash. With HyperBench, package and constructor entry points lead it through removal and startup publication. The baseline opens neither this file nor the constructor that triggers publication. The HyperBench-assisted answer proposes pruning orphaned diagnostics or rejecting the invalid configuration, then checking the removal-and-publication sequence.
Thanos: starting in the right storage implementation
The HyperBench-assisted run uses 50.4% less solver time: 117.7 seconds versus 237.2 seconds. Graph retrieval points to BucketStore.Series at the start, while the baseline first changes a different storage path.
Read the task and trace
The task is to fix Thanos replica-label removal and series ordering. The supplied test logs include:
=== RUN TestHarnessSeriesSortedWithoutReplicaLabels
=== RUN TestHarnessSeriesSortedWithoutReplicaLabels/use_TSDB_label_as_replica_label
zz_series_order_harness_test.go:100: series response not sorted after replica-label removal
Without the graph, the agent changes TSDBStore.Series, sees validation fail, and moves to BucketStore. With HyperBench, the graph ranks BucketStore.Series first, leading to source inspection revealing that the block reader removes external replica labels but leaves labels stored inside the time series. Both runs iterate before passing, but the HyperBench-assisted trace avoids the initial change to the wrong implementation.
GitOps Engine: repairing the code without upgrading a dependency
The HyperBench-assisted investigation leads to an existing helper, a smaller, more effective change, and 54.9% less solver time. Both repairs pass: the HyperBench-assisted run finishes in 30.8 seconds versus 68.3 seconds for the baseline, whose dependency upgrade encounters download and checksum problems.
Read the task and trace
Investigate and fix this GitOps Engine server-side diff fails when webhook removes container fields.
Removing a webhook's changes extracts a list item without its required name field, and the comparison library rejects it. Without the graph, the agent upgrades the dependency to use WithAppendKeyFields(). With HyperBench, the agent receives the comparison entry point and the requirement to preserve list keys. The agent checks the dependency source and finds EnsureNamedFieldsAreMembers already available. The HyperBench-assisted repair stays within the extraction code and improves existing methods without requiring extra work.
More recorded comparisons
CARLA: the relevant source in five calls instead of nineteen
During a CARLA run, the vehicle makes a gentle turn, but the heading rate reaching localization is dramatically different from the simulator's actual turn rate. Trace the discrepancy and identify a concrete cause.
The baseline searches gyroscope terms, launch files, calibration, and estimator settings before reading the relevant code on call 19. A graph query links the IMU to its CARLA handler, and the HyperBench-assisted run reads that code on call 5. Both identify the missing unit and sign conversion when angular velocity is copied into VelocityReport.heading_rate. Complete attempts take 46.8 versus 28.9 seconds.
Autoware GPU allocation: finding the extra ×8 multiplier
During replay of an eight-LiDAR recording, GPU memory use in the perception container rises and then plateaus unusually high. The concatenated cloud contains about 13.7 MB of data but uses roughly 110 MB of allocation. What explains the footprint, and how would you verify it?
The baseline attributes the footprint to CUDA overhead and fails to reach the allocator within its retrieval budget. The HyperBench-assisted answer reads it and finds that the combined cloud size is multiplied by the input count again: 13.7 MB × 8 ≈ 109.6 MB. It proposes removing * input_topics_.size() at both allocation sites, finishing in 33.0 seconds versus 45.3 seconds.
Autoware route recovery: a specific correction, with a longer investigation
After the driver takes over during a lane change and drives away from the planned trajectory, the vehicle remains on its route but planning can stay stuck on the old lane-change end path. Why does normal route-following fail to recover promptly, and what should change?
The baseline recommends a replanning backport. The HyperBench-assisted answer receives launch context and reads updateCurrentRouteLanelet(). A failed constrained match falls back to the old lane sequence before the planner can reset its reference to the vehicle's position. It proposes calling resetCurrentRouteLanelet() first when no approved maneuver is running, retaining the fallback for approved maneuvers. This is a more specific, supported correction that is corroborated by evidence compared to the baseline's proposal.
GitOps diagnosis: finding the cause earlier, with similar total repair time
This separate matched pair receives the same GitOps task:
Investigate and fix this GitOps Engine server-side diff fails when webhook removes container fields.
The baseline searches for the extraction error before reading the comparison code. The HyperBench-assisted run starts with the ServerSideDiff entry point and list-key requirement. Its first recorded diagnosis judged correct appears at 4.92 seconds versus 11.33 seconds, a 56.6% reduction.
Preserving the hardware evidence
Our custom Rust retrieval pipeline helps agents find hardware context that ranked text search can miss. By preserving connections between devices, drivers, and configurations, it brings the evidence needed to understand a system into the agent's context.
Across 50 answerable development questions, the Rust pipeline more than doubled mean evidence recall: 36.3% → 77.0%. It retrieved complete evidence for 37 questions, up from 17, with warm retrieval p95 of 0.445 ms.
Finding the hardware context that search misses
Crazyflie — sensor bus configuration
Which GPIO port is used for I2C3 SDA and which for SCL?
Retrieved by the Rust pipeline: SDA uses GPIOC, and SCL uses GPIOA. Both assignments appear together in the bus configuration.
Klipper — heated-bed sensor configuration
Which thermistor model and input pin does the heated bed use on BIGTREETECH Octopus v1.1?
Retrieved by the Rust pipeline: The thermistor is an ATC Semitec 104GT-2, connected to input pin PF3. Both settings appear in the board's heated-bed configuration.
In both cases, existing search methods returned the wrong results, missing the configuration values needed to answer the question. The Rust pipeline retrieved complete evidence for both answers, using less context.
Giving agents a better starting point
The useful context in these traces is specific and promising: the constructor that publishes a diagnostic, the storage path handling a request, or the identifying field a repair must preserve. HyperBench puts those relationships within reach of the agent, so it can investigate the relevant implementation and check its assumptions against the source at lightning speeds.