Introduction: "More Terrifying Than Slow Performance Is a 1-Tick Consistency Bug"
In a financial trading system, no matter how fast and fluid the UI becomes, if a displayed price or orderbook quantity deviates from the true market state by even a single tick or a single share, the engine has failed.
Our production environment presented three particularly demanding consistency challenges:
- Multi-Exchange Regime (Primary Exchange + Alternative Exchange [ATS] + Consolidated):
Traders switch seamlessly betweenPrimary Exchange,Alternative Exchange (ATS), andConsolidatedviews within a single screen—or compare them side by side. - Cross-Domain Completion Reordering (Expected Auction vs. Regular Continuous Trade):
At the exact 09:00:00 transition from opening auction to regular trading, if the worker processing the final auction packet (Expected) finishes 0.1ms after the worker processing the first continuous trade packet (Trade), the UI could remain stuck displaying an "Expected Execution" badge all day. - Race Between Initial Snapshot Request/Reply (RQ) and Real-Time Pub/Sub (SB):
During the tens of milliseconds while a newly selected symbol awaits its initial server response (RQ), real-time trade and quote packets (SB) continue streaming in. If the delayed RQ response blindly overwrites the state, it clobbers newer real-time ticks with stale prices.
In [Part 3], we examine our Deterministic State Projection Architecture that resolves all three consistency challenges without locks.
1. Per-Symbol 3-Slot Exchange Isolation & Real-Time Best Bid/Ask Synthesis
Each symbol context (SymbolContext) maintains three independent RCU snapshot slots internally:
flowchart LR
subgraph Context["SymbolContext (1 per Symbol)"]
S0["Slot 0: Consolidated Market"]
S1["Slot 1: Primary Exchange"]
S2["Slot 2: Alternative Exchange (ATS)"]
end
P1["Primary Exchange Packet"] --> S1
P1 -->|"Cross-Compare Best Bid/Ask"| S0
P2["Alternative Exchange Packet"] --> S2
P2 -->|"Cross-Compare Best Bid/Ask"| S0
P0["Consolidated Feed Packet"] --> S0
Best Bid / Best Ask Synthesis Rules
When an individual exchange packet arrives, the best ask (Ask 1) and best bid (Bid 1) in the Consolidated slot are synthesized according to strict Market Microstructure rules rather than naive overwriting:
- Consolidated Best Ask: Among valid (
> 0) Ask 1 prices across both exchanges, the lower price ($\min(\text{Ask}_{\text{Primary}}, \text{Ask}_{\text{ATS}})$) wins. If both exchanges quote the exact same Ask 1 price, their quantities are summed. - Consolidated Best Bid: Among valid (
> 0) Bid 1 prices across both exchanges, the higher price ($\max(\text{Bid}_{\text{Primary}}, \text{Bid}_{\text{ATS}})$) wins. Likewise, if both exchanges quote the exact same Bid 1 price, their quantities are summed.
Because background shard workers compute this synthesis eagerly during the Copy-on-Write (CoW) step, any UI view can switch exchange tabs and retrieve a complete, ready-to-render snapshot with a single GetSnapshot(symbol, exchangeType) call.
2. Raw Domain Image Separation & Ingress-Order Projection (StockRealtimeState)
In [Part 2], we explained how Trade, Expected, and Depth packets are processed in parallel across dedicated shard worker groups to maximize throughput.
The hazard arises at session boundaries—such as the 09:00:00 opening bell or the 15:20:00 closing auction transition:
- Packet #100: Final auction expected execution packet (
Expected, price 71,000) - Packet #101: First regular market trade packet (
Trade, price 71,200)
Suppose the Trade worker finishes Packet #101 first and updates the display state to 'Actual Trade (71,200)', but 0.1ms later the Expected worker finishes Packet #100 and overwrites the shared display fields. The screen would falsely revert to 'Expected Trade (71,000)' even after continuous trading has begun!
The Solution: "Never Overwrite Shared Display Fields Directly—Project from Raw Domain Images"
Inside each symbol state, we separated the shared display fields from per-domain raw images (TradeImage, ExpectedImage, DepthImage) paired with their monotonic ingress sequence numbers (ingressOrder):
struct RealtimeDomainState {
// 1. Independent per-domain raw images (never clobbered by other domains)
uint64_t lastTradeOrder = 0; // Ingress sequence of latest Trade packet
uint64_t lastExpectedOrder = 0; // Ingress sequence of latest Expected packet
uint64_t lastDepthOrder = 0; // Ingress sequence of latest Depth packet
TradeRawImage tradeImage; // Raw state of latest actual trade
ExpectedRawImage expectedImage; // Raw state of latest auction expected execution
DepthRawImage depthImage; // Raw state of latest 10-level orderbook depth
// 2. Deterministic State Projection
void ProjectToDisplay(SymbolSnapshot& out) const {
// Display state is determined NOT by which worker thread finished last,
// but by which packet entered the network ingress later (ingressOrder)!
if (lastExpectedOrder > lastTradeOrder) {
ApplyExpectedToDisplay(expectedImage, out);
} else if (lastTradeOrder > 0) {
ApplyTradeToDisplay(tradeImage, out);
}
// Pure 10-level orderbook depth projects independently of Trade/Expected
if (lastDepthOrder > 0) {
ApplyDepthToDisplay(depthImage, out);
}
}
};With this design:
- Even if Packet #101 (
Trade) commits first and Packet #100 (Expected) commits slightly later,lastTradeOrder(101) > lastExpectedOrder(100)holds true—soProjectToDisplay()deterministically converges on Packet #101 (Actual Trade). - Furthermore, pre-market and auction Reference Price calculations—distinguished from previous close via sign semantics (
Price - Changeon up-ticks,Price + Changeon down-ticks)—were unified into a single pure function (PriceSemantics) shared across all parsers and UI views.
3. Lossless Merge of Initial Request/Reply (RQ) and Real-Time Streams (SB)
When a user selects a new symbol on screen, the client performs two concurrent operations:
- Sends an initial master & 10-level orderbook snapshot query (Request/Reply, RQ) to the server.
- Registers a real-time trade/quote subscription (Publish/Subscribe, SB) for that symbol.
What if 3 real-time trades arrive during the 30ms round-trip while the initial RQ response is travelling across the network?
- If the arriving RQ response is blindly
memcpy'd into the screen struct, those 3 newer real-time trades are wiped out. - Conversely, if real-time ticks are rendered before the RQ arrives, static reference fields present only in the RQ payload (price limits, previous day volume, par value, etc.) remain uninitialized and corrupt the display.
We solved this via a LookupSession startOrder boundary and Selective Realtime Overlay:
sequenceDiagram
participant UI as "UI Screen (View)"
participant Engine as "MarketDataEngine"
participant Server as "Market Server"
UI->>Engine: 1. BeginLookupSession(Symbol) -> Record startOrder = #500
UI->>Server: 2. Subscribe Real-Time (SB) & Send Initial Query (RQ)
Server-->>Engine: 3. (While awaiting RQ) Real-time ticks #501, #502 arrive -> Accumulated in RCU Snapshot
Server-->>UI: 4. Initial RQ response arrives (State as of <= #500)
UI->>Engine: 5. Decode RQ endianness on background worker into base structure
UI->>Engine: 6. Call OverlayRealtimeSince(startOrder = #500)!
Note over UI,Engine: Only domains with order > #500 (#501, #502) are selectively overlaid atop the RQ base (preserving RQ-only static fields)
Production War Story: Variable-Length OrderBook Reply Packets
While validating initial orderbook RQ responses on the live network, we discovered a subtle wire protocol behavior: symbols with Liquidity Providers (LP) returned the full 1,194-byte structure, whereas regular symbols without LPs omitted the trailing 210-byte LP quantity block and arrived as 984-byte packets.
A naive check like reply.size == sizeof(FullDepthResponse) would reject valid regular-stock responses as malformed. We introduced a wire validator backed by compile-time static_assert checks for both valid sizes (WithoutLP = 984, WithLP = 1194):
template <typename WireDepthStruct>
struct DepthWireValidator {
static constexpr size_t SizeWithoutLp =
offsetof(WireDepthStruct, rsqty) + 3 * sizeof(decltype(WireDepthStruct{}.rsqty[0]));
static constexpr size_t SizeWithLp = sizeof(WireDepthStruct);
static_assert(SizeWithoutLp == 984 && SizeWithLp == 1194, "Wire depth layout mismatch");
static bool IsValidSize(int bytes) noexcept {
return bytes == static_cast<int>(SizeWithoutLp) ||
bytes == static_cast<int>(SizeWithLp);
}
static void CopySafe(WireDepthStruct& dst, const void* src, int bytes) noexcept {
std::memset(&dst, 0, sizeof(dst)); // Omitted trailing LP block is automatically zero-initialized
if (IsValidSize(bytes)) {
std::memcpy(&dst, src, static_cast<size_t>(bytes));
}
}
};
Combining compile-time layout verification (static_assert) with zero-initialized bounded copy absorbed wire-level protocol variations cleanly and safely.
Coming Up Next
Although MarketDataEngine now serves accurate, lock-free snapshots, the UI thread still faced heavy lifting in screens displaying watchlists with hundreds of symbols or real-time time-and-sales grids ingesting thousands of ticks per second.
In [Part 4], guided by the philosophy "Let the UI Thread Only Paint", we introduce ScreenWorker background pre-formatting of 28 display fields, Dirty Coalescing, and our 64MB fixed-memory Ring-Buffer VirtualGrid.