Acceptance Testing an Ultra-Low-Latency Trading System: Latency Distribution, P99/P99.9, PCAP Replay and Acceptance Criteria

Evaluating Latency Distribution and Extreme-Tail Performance

Average latency is a poor acceptance metric for an ultra-low-latency trading system. A path that averages a few hundred nanoseconds can still produce multi-microsecond spikes that erase edges, create risk, or violate internal service levels. Acceptance testing therefore centers on the shape of the latency distribution, the behavior of the extreme tail, realistic replay of production traffic, and explicit numeric criteria that a build must meet before it is allowed into production.

The goal is not to demonstrate that the system is fast on a quiet lab bench. It is to prove that under representative message rates, burst patterns, and failure conditions, the residual latency remains inside a pre-agreed envelope.

Credible acceptance of an ultra-low-latency trading system requires measuring the full latency distribution (especially P99 and P99.9), replaying production-like PCAP traffic, and applying written numeric criteria that cover both the body and the tail of the distribution under defined load.

Why the Average Misleads

In a high-message-rate environment, the mean is dominated by the common case. The economic and risk impact lives in the infrequent but expensive outliers, the moments when a queue builds, a cache misses, an interrupt lands, or a lock is taken. A system that reports an attractive average while hiding a long tail will look healthy in a summary dashboard and still lose priority or generate unwanted fills when the market is most active.

Acceptance therefore treats the distribution as the primary artifact. Histograms, percentile tables, and worst-window statistics replace single-number summaries.

Defining the Measurement Points

Latency must be measured between well-defined boundaries. Common wire-to-wire or tick-to-trade definitions include the moment the last bit of a market-data packet arrives at the NIC or optical tap and the moment the first bit of the corresponding order leaves the outbound interface. Internal stage latencies (parse complete, book updated, risk checked, order serialized) are recorded in parallel so that regressions can be localized.

Timestamps should be taken with hardware support or tightly synchronized software clocks. Software-only timestamps that cross kernel boundaries introduce their own jitter and can mask or invent problems. The measurement itself must not perturb the path under test.

P99, P99.9 and the Tail That Matters

P50 or P95 describe the typical case. P99 and P99.9 describe the case that occurs often enough to matter at production message rates. At one million events per second, the events beyond P99.9 still number in the thousands per second. Those events determine whether the system keeps priority, avoids adverse selection, or stays inside risk limits.

Acceptance criteria are therefore written against specific percentiles and against specific load windows. A common pattern is to require that P99 under sustained load stays below a hard threshold and that P99.9 under the worst observed burst window stays below a higher but still bounded threshold. Some teams also track the maximum or a high percentile of the worst N-second window in a session.

PCAP Replay as the Realistic Stimulus

Synthetic traffic is useful for micro-benchmarks. It rarely reproduces the burstiness, message-size mix, inter-arrival patterns, and correlative behavior of a live exchange feed. PCAP replay of production or near-production captures supplies that are realistic.

A well-engineered replay framework preserves original inter-packet timing (or scales it in a controlled way), injects the capture at line rate or at defined multiples, and allows the system under test to be measured exactly as it would be in production. Dual-feed captures enable arbitration testing; captures that include gaps or sequence errors exercise recovery logic. Replay also makes results repeatable: the same PCAP run against successive builds produces comparable distributions.

Care is required to keep the replay engine itself from becoming the bottleneck or from introducing artificial smoothness. Hardware timestamping on both the replay and the measurement points keeps the numbers trustworthy.

Writing Explicit Acceptance Criteria

Criteria should be numeric, load-conditioned, and percentile-specific. Examples of the form that survives scrutiny include:

• Under sustained production-rate replay, wire-to-wire P99 ≤ X nanoseconds and P99.9 ≤ Y nanoseconds.
• Under the worst recorded burst window from the capture set, P99.9 remains ≤ Z nanoseconds.
• No continuous gap in processing longer than T microseconds.
• Risk-check stage latency stays inside its own percentile envelope so that the pre-trade gate cannot be outrun.

Criteria are attached to a defined software and hardware configuration, a defined set of PCAPs, and a defined measurement methodology. A build that passes on a quiet synthetic test but fails the PCAP-based tail criteria is rejected.

Functional Correctness Alongside Latency

Acceptance is not only about speed. Sequence integrity, book correctness after gaps, proper dual-feed arbitration, risk-check outcomes, and recovery after disconnects must also pass. A system that is fast but occasionally emits an incorrect order or misses a risk limit is not acceptable. Latency and functional test suites therefore run on the same builds and the same replay stimuli.

Instrumentation and Avoiding Heisenberg Effects

The act of measuring can change the system. Heavy logging, synchronous telemetry, or shared timestamps that contend with the critical path inflate the very tails being measured. Acceptance builds therefore use lightweight, preferably hardware-assisted, instrumentation. Detailed diagnostics are reserved for failure investigation, not for the pass/fail run.

ShengliSoft and Hardware-Accelerated Paths

When an FPGA feed handler, risk engine, or order path is part of the system, the same disciplines apply. Wire-to-wire measurements must include the hardware stages; PCAP replay must exercise the optical or electrical inputs the FPGA actually sees; and acceptance criteria must cover the residual software path that remains after the hardware has done its work.

ShengliSoft's Extreme Feed Handler, RiskGuard, and related FPGA platforms are designed for deterministic processing of Chinese equity and futures market data and risk checks. Acceptance testing of systems that incorporate these components still requires full-distribution measurement and realistic replay so that the combined hardware-plus-software residual latency is known under load.

Governance of the Acceptance Gate

The acceptance report should be reproducible and reviewable. It records the exact build identifiers, the PCAP set and its provenance, the measurement points, the percentile results, the pass/fail status against each criterion, and any exceptions. Changes to criteria require explicit review; silent tightening or loosening of thresholds undermines trust in the gate.

A system that cannot produce a clean, percentile-based acceptance report under realistic replay is not ready for production, regardless of how attractive its average latency appears in a lab summary.

Distribution, Tail Thresholds and Realistic Replay Turn Latency Claims into Verifiable Acceptance

Acceptance testing of an ultra-low-latency trading system succeeds when it replaces single-number averages with full latency distributions, elevates P99 and P99.9 to first-class criteria, drives the system with production-like PCAP replay, and applies written numeric thresholds that must be met under defined load. Teams that institutionalize these practices gain a repeatable gate that protects both performance and risk, whether the critical path runs in software, in FPGA logic, or in a hybrid of the two.

FAQ

1. Why is average latency insufficient for acceptance?

The average hides the tail. Economic and risk outcomes are driven by the infrequent slow events that P99 and P99.9 expose.

2. What do P99 and P99.9 represent in practice?

At high message rates, they describe events that still occur thousands of times per second. Those events determine whether priority is kept and whether risk limits are respected under stress.

3. Why use PCAP replay instead of purely synthetic traffic?

PCAPs reproduce real inter-arrival times, burst structure, message-size mix, and correlative behavior that synthetic generators rarely match, making the measured distribution more representative of production.

4. What belongs in a written acceptance criterion?

A specific percentile, a numeric threshold, a defined load or PCAP set, and clear measurement boundaries (for example, wire-to-wire or stage-to-stage).

5. How should timestamps be obtained?

Prefer hardware timestamps or tightly synchronized, low-overhead software clocks. Measurement must not itself become a source of jitter on the critical path.

6. Does FPGA acceleration change the acceptance approach?

The same distribution, percentile, and replay disciplines apply. Measurements must simply include the hardware stages and the residual software path that follows them.

7. How does ShengliSoft fit into this testing model?

Systems that incorporate ShengliSoft FPGA feed handlers or risk engines still require full-distribution, PCAP-driven acceptance so that the combined residual latency under realistic load is known and gated.

8. What is the minimum evidence an acceptance report should contain?

Build identity, PCAP provenance, measurement points, percentile results (including P99 and P99.9), pass/fail against each criterion, and any exceptions or observations.

Disclaimer

This content is for informational purposes only and does not constitute investment advice. Financial markets and technology investments carry risk. Please conduct your own research before making investment or business decisions.