[High-Performance C++ Engine #5] 1% CPU & 1-Second Lag War Stories (Message Storms, ActiveX Batching, 60-Cell Clipping)

▲ [Figure 5] Suppressing 423/s Win32 message storms via In-Flight Latch, ActiveX COM Redraw batching, and 60-cell persistent back-buffer clipping

Introduction: "We Eliminated the Freezes, But We Aren't Done Yet"

When we analyzed the 9-minute market open benchmark logs (TO-BE) after deploying our initial architecture, the maximum UI freeze had dropped from 1,531ms → 15ms. Yet as we scrutinized the fine-grained profiler counters, three intriguing anomalies stood out:

1. Anomaly ① (Win32 Window Message Storm):
   Even though screens pulled snapshots on a 50–100ms timer, the background notification router fired 228,687 PostMessage(WM_ENGINE_NOTIFY) calls (423.5 messages/sec) across open windows over the 540-second window.
2. Anomaly ② (31ms Spikes in the Shared OrderBook Control):
   Despite migrating to the shared orderbook pipeline, single-frame rendering in SharedOrderBookControl occasionally spiked to 31ms.
3. Anomaly ③ (Perceived 1-Second Lag in the 60-Cell Multi-Quote Matrix):
   While the watchlist and orderbook windows updated at lightning speed, the 60-symbol Multi-Quote Matrix (MultiQuoteMatrix) noticeably stepped its quotes only once every ~1 second (750–1,000ms) during the market open burst.

In [Part 5], we walk through how we tracked down and resolved each of these three production anomalies.

1. Troubleshooting ①: Silencing a 423/s Message Storm with an In-Flight Latch

Root Cause Analysis

Our initial notification router enforced a per-HWND 50ms time-based throttle (kThrottleIntervalMs = 50) to prevent excessive WM_ENGINE_NOTIFY messages. But what happens when 60 to 70 child windows (HWNDs) are open simultaneously?

• Even at one message per 50ms (20/sec) per window, 70 windows $\times$ 20/sec = up to 1,400 kernel PostMessage calls per second!
• Worse still, each window's OnEngineNotify handler merely flipped a boolean flag (m_isDirty = true;), while actual rendering happened later inside OnTimer.
• In other words, before the window's timer even had a chance to consume m_isDirty, the router saw that 50ms had elapsed and fired another redundant PostMessage just to set m_isDirty = true again!

The Solution: Consumer-Acknowledged In-Flight Latch (PendingNotificationState)

Moving beyond simple time throttling, we designed an atomic In-Flight Latch that strictly suppresses any further PostMessage until the UI screen explicitly acknowledges consumption (AcknowledgeConsume):

struct PendingNotificationState {
    static constexpr DWORD kThrottleIntervalMs = 50;
    static constexpr DWORD kMaxInFlightHoldMs  = 250; // Safety fallback timeout for hidden/modal windows

    std::atomic<bool>  pending{false};
    std::atomic<bool>  inFlight{false};
    std::atomic<DWORD> lastPostTick{0};

    // Invoked by the UI screen when its timer actually consumes the snapshot!
    void AcknowledgeConsume() noexcept {
        inFlight.store(false, std::memory_order_release);
    }

    bool TryAcquirePostSlot(DWORD nowTick) noexcept {
        // 1. If the screen hasn't consumed the previous notification yet (inFlight == true),
        //    immediately suppress PostMessage until the 250ms safety fallback expires!
        if (inFlight.load(std::memory_order_acquire)) {
            const DWORD last = lastPostTick.load(std::memory_order_relaxed);
            if (static_cast<DWORD>(nowTick - last) < kMaxInFlightHoldMs) {
                return false; // Completely blocks redundant kernel messages
            }
        }
        // 2. Enforce minimum 50ms interval + acquire single post slot via CAS
        // ...
        inFlight.store(true, std::memory_order_release);
        return true;
    }
};

We paired this with two additional micro-optimizations:

• $O(\log N)$ Lookup on Subscription Registration: Replaced an $O(N_{\text{topics}} \times N_{\text{items}})$ nested scan inside SubscriptionRouter::SetAt with a direct lookup in m_mapCookieNotifyState.
• Pre-Filtering & Conditional SetEvent in DispatchToScreens: Packets from exchange feeds not subscribed by a screen are discarded before touching the worker queue lock, and the kernel event (SetEvent) is signaled only when transitioning the queue from empty to non-empty (needWake).

2. Troubleshooting ②: ActiveX OrderBook put_Redraw(flexRDNone) Batching & VI Dirty-Check

Root Cause Analysis

Many legacy trading screens rely on commercial COM/ActiveX grid controls (such as VSFlexGrid) to render orderbook tables. Each time the orderbook (SharedOrderBookControl::DrawOfferBid) refreshes, updating 10 levels of bid/ask prices, sizes, and diffs triggers 40 to 100 individual put_Cell(...) COM interface calls.

• When the grid's Redraw property remains enabled during those calls, every single put_Cell invocation triggers internal ActiveX cell-range recalculation and partial invalidation logic 40 to 100 times per tick.
• Additionally, 8 static Volatility Interruption (VI) cells (base price, upward/downward trigger prices)—which rarely change during the day—were being overwritten via put_Cell on every single depth tick without checking if their values had changed.

The Solution: RAII Redraw Batch Guard + Static VI Dirty-Check

// 1. Suspend internal ActiveX redraws (flexRDNone) upon entering DrawOfferBid,
//    and restore redraws exactly once after all put_Cell calls complete
long prevRedraw = VSFlex8N::flexRDDirect;
const bool batchRedraw = m_bookPilot.Active();
if (batchRedraw) {
    prevRedraw = m_pGrid->GetRedraw();
    if (prevRedraw != VSFlex8N::flexRDNone) {
        m_pGrid->put_Redraw(VSFlex8N::flexRDNone);
    }
}

// ... (Batch execute put_Cell for modified orderbook cells) ...

// 2. Invoke COM calls for Static VI cells ONLY when values actually differ from m_prevHogaData
const bool bViChanged = m_bForceRepaint ||
    (m_currHogaData.viBasePrice != m_prevHogaData.viBasePrice) ||
    (m_currHogaData.viUpPrice   != m_prevHogaData.viUpPrice)   ||
    (m_currHogaData.viDownPrice != m_prevHogaData.viDownPrice);
if (bViChanged) {
    UpdateStaticViCells();
}

// 3. Restore Redraw on exit (triggers a single consolidated repaint)
if (batchRedraw && prevRedraw != VSFlex8N::flexRDNone) {
    m_pGrid->put_Redraw(prevRedraw);
}

This lightweight batch guard and VI dirty-check eliminated dozens of redundant COM layout recalculations per orderbook tick.

3. Troubleshooting ③: Solving the "1-Second Quote Lag" Mystery in the 60-Cell Multi-Quote Matrix

Our most dramatic debugging investigation centered on the 60-symbol Multi-Quote Matrix (MultiQuoteMatrix). During market open testing, while other views updated instantaneously, quotes in this 60-cell matrix stepped sluggishly once every ~1 second.

Tracing the execution path revealed four independent mechanisms compounding together to produce an exact 750ms ~ 1,000ms refresh cycle!

flowchart TD
    A["Cause 1: Max 4 child windows polled per 50ms timer (60 / 4 * 50ms = 750ms queue cycle)"] --> E["Perceived Result: Quotes only update once per second!"]
    B["Cause 2: Whenever 4ms budget was used, Observe(tickUs >= 4000) self-inflicted +50ms pressure up to +700ms"] --> E
    C["Cause 3: When mouse focus was on another window, active=false (base 300ms) -> 300ms + 700ms = 1,000ms cap!"] --> E
    D["Cause 4: Empty polls reset m_lastLiveRefresh cooldown + arriving ticks didn't call WakeLive"] --> E

The 4 Compounding Causes & Their Fixes

1. Fixing the Self-Inflicted pressure Penalty Bug:
   The normal loop exit condition in FrameBudgetCoordinator::Run was if (Micros() - start >= 4000) break; (gracefully yielding once the 4ms frame budget was filled). Immediately after the loop, however, policy_.Observe(tickUs >= 4000) treated tickUs >= 4000 as an overload spike—adding a +50ms pressure penalty each turn up to +700ms! In short, whenever the coordinator did its job and used its 4ms budget, it penalized itself into a 1,000ms (1-second) refresh interval.
   Fix: Decoupled the normal loop budget boundary (4,000us) from the true single-callback stall threshold (8,000us) via Observe(tickUs >= 8000).
2. Removing the Hardcoded 4-Slot Limit + Introducing WakeLive:
   Expanded the live phase (phase == 1) slot limit from 4 to max(16, children.size()) (covering all 60 cells) and shortened the coordinator timer to 30ms. Furthermore, arriving real-time notifications (OnEngineNotify) now immediately call WakeLive(hwnd, due) so the cell refreshes on the very next 30ms tick rather than waiting for a distant scheduled due timestamp. Visible windows (!hidden) also remain active = true (100ms base cadence) even when mouse focus is on an adjacent window.
3. Fixing Empty-Poll Cooldown Resets & Unsubscribed Domain inFlight Holds:
   Polling a child window with no market changes no longer resets its m_lastLiveRefresh timestamp, preventing real quote ticks arriving 5ms later from being blocked by a false cooldown. And when a cell wakes up due to an unsubscribed domain update (e.g., program trading) on a shared snapshot, it immediately calls NotifyScreenConsumed so the inFlight latch isn't held for 250ms.
4. Persistent Back-Buffer Clipping (ExtSelectClipRgn) on Trade-Only Ticks:
   Repainting all 20 orderbook rows across 60 windows on every trade is expensive in GDI. On ticks where only last price/volume changed and quotes remained identical (tradeOnly), we partially invalidate only the top header/price/mini-candle rects (m_rectManju | m_rectCurrent | m_rectGrp) and apply ::ExtSelectClipRgn(..., RGN_AND) against the persistent offscreen bitmap (m_paintBitmap)—completely skipping the 20-row orderbook table drawing routines (DrawMedo, DrawMesu)!

Coming Up Next (Series Finale)

Every piece of the puzzle is now in place. After all of these architectural and micro-level optimizations, what did the real-world market open benchmark scorecard look like? And how did we roll out such a massive core engine replacement across a multi-decade production trading system with zero downtime and zero incidents (Dual-Path Architecture)?

In our finale, [Part 6], we present the Production Market-Open Scorecard (AS-IS vs. TO-BE Quantitative Comparison), our Zero-Downtime Dual-Path Rollout Strategy, and key engineering takeaways.