Quant Infrastructure & KDB+/q

Why infrastructure matters

Every strategy you have studied — alpha signals, backtesting, portfolio construction, machine learning — rests on one question: where does the data come from, and how is it stored and served? Infrastructure is the bridge between raw market data and a deployable strategy.

Modern markets continuously emit many types of time-stamped data: trades, quotes, order-book updates and news events. The scale of a single trading day is enormous.

How data flows: exchange to research

Almost every market-data architecture follows the same pipeline. Data is born at the exchange, cleaned by a feed handler, persisted in a database, then consumed by research and trading.

flowchart LR
  EX([Exchange]) --> FH([Feed Handler])
  FH --> DB[(Database)]
  DB --> RS([Research Team])
  DB --> TR([Traders])
  classDef s fill:#eef2ff,stroke:#4f46e5,stroke-width:1px,color:#3730a3;
  classDef d fill:#ecfdf5,stroke:#0d9488,color:#0d9488;
  class EX,FH,DB s
  class RS,TR d
      

What is an exchange?

An exchange is the venue where buyers and sellers meet — NSE, BSE, NASDAQ, NYSE. Its core output for a quant is a continuous stream of two things: trades and quotes.

What is market data?

Market data splits into the two records every downstream system is built around:

A trade record — exactly what was bought, at what price, in what size, and when.

A quote record — the best bid and ask, with the spread $= \text{ask} - \text{bid}$.

Feed handlers

Each exchange speaks its own wire format. A feed handler is the translator that sits between the raw exchange streams and the firm's internal systems. It performs four steps:

Tick data

A short slice of a tick stream interleaves event types down to the millisecond:

Tick vs OHLC

The alternative to ticks is OHLC — Open, High, Low, Close — a periodic summary over a fixed interval (e.g. one bar per minute). The trade-off is detail versus compactness.

Time-series data

Time matters because trading is about sequence and causality. Around any event — a trade, a news headline — we constantly ask three time-ordered questions:

Why KDB+ exists

KDB+ (queried with the language q) is a database built specifically for this problem. It was created because three pressures collided.

flowchart LR
  A([Massive market data]) --> P{{Problem}}
  B([Slow traditional queries]) --> P
  C([Real-time requirements]) --> P
  P --> K([KDB+ solution])
  classDef s fill:#eef2ff,stroke:#4f46e5,stroke-width:1px,color:#3730a3;
  classDef d fill:#ecfdf5,stroke:#0d9488,color:#0d9488;
  class A,B,C,P s
  class K d
      

The core idea is simple: store data efficiently, query it quickly, analyse it rapidly. The power comes from raw speed and a design that fits time-series naturally.

Columnar vs row storage

The single biggest reason KDB+ is fast on market data is that it is columnar. Traditional databases are row-based — a record's fields are stored together — while KDB+ stores each column contiguously.

flowchart TB
  subgraph ROW [Row storage - reads entire rows]
    direction LR
    R1[time, sym, price, size]
    R2[time, sym, price, size]
    R3[time, sym, price, size]
  end
  subgraph COL [Columnar storage - reads only needed columns]
    direction LR
    C1[time time time]
    C2[sym sym sym]
    C3[price price price]
    C4[size size size]
  end
  ROW --> COL
  classDef s fill:#eef2ff,stroke:#4f46e5,stroke-width:1px,color:#3730a3;
  classDef d fill:#ecfdf5,stroke:#0d9488,color:#0d9488;
  class R1,R2,R3 s
  class C1,C2,C3,C4 d
      

SQL ↔ q: the same logic, two syntaxes

If you know SQL, you already know most of q. KDB+ tables look like SQL tables — the difference is they are column-oriented and tuned for symbols and time. Symbols in q are written with a leading backtick, e.g. `AAPL; that backtick is normal q syntax for a symbol literal.

Filtering — WHERE → where

-- SQL
SELECT * FROM trades WHERE symbol='AAPL';

/ q
select from trades where sym=`AAPL

Grouping & averaging — GROUP BY → by, AVG(price) → avg price

-- SQL
SELECT symbol, AVG(price) FROM trades GROUP BY symbol;

/ q
select avg price by sym from trades

Aggregating — SUM(size) → sum size

-- SQL
SELECT symbol, SUM(size) FROM trades GROUP BY symbol;

/ q
select sum size by sym from trades

Time filtering — BETWEEN → within

-- SQL
SELECT * FROM trades WHERE time BETWEEN '09:30' AND '10:00';

/ q
select from trades where time within 09:30 10:00

VWAP and the research workflow

A canonical analytic both SQL and q solve identically is VWAP — the Volume-Weighted Average Price. It weights every trade price by the size traded, so big prints count more than small ones.

In practice VWAP is computed inside a Python + q loop. KDB stores and serves the data; Python explores, builds features and evaluates.

flowchart LR
  MD([Market Data]) --> KDB[(KDB+)]
  KDB --> PY([Python])
  PY --> FE([Features])
  FE --> SG([Signals])
  SG --> BT([Backtest])
  classDef s fill:#eef2ff,stroke:#4f46e5,stroke-width:1px,color:#3730a3;
  classDef d fill:#ecfdf5,stroke:#0d9488,color:#0d9488;
  class MD,KDB,PY s
  class FE,SG,BT d
      

That four-step loop — query historical data → build features → generate signals → evaluate performance — is the typical quant research day.

Research vs production

Key takeaways