Reinforcement Learning for Trading

Optimal Execution is a sequential decision problem: at each step you observe a state (how much inventory is left, how much time, what the book looks like) and choose an action (how much to sell now). That's a Markov decision process, and it's exactly the shape reinforcement learning was built for.

Why bother, when closed-form answers exist?

Because the closed-form answers are only as good as their assumptions. Almgren–Chriss gives you an elegant efficient frontier — conditional on linear impact and random-walk prices. Real books have nonlinear impact, transient impact that partially recovers, queue effects, and adverse selection. An RL agent learns the policy from the data directly, and doesn't need you to have guessed the functional form correctly.

The lineage

• Nevmyvaka, Feng & Kearns (2006) trained a modified Q-learning agent on millisecond equity data and beat benchmark policies using nothing but remaining inventory and time — then improved further by adding book-derived features.
• Deep Q-Networks (2015) replaced the lookup table with a neural network, making richer state spaces tractable.
• Double DQN (2016) fixed a systematic overestimation bug in DQN by separating the network that chooses an action from the network that evaluates it.
• Macrì & Lillo (2024) paired Double DQN with time-varying impact coefficients, which is the direct ancestor of the architecture in my own work.

The uncomfortable part

RL agents are stochastic. Train the same architecture on the same data with a different random seed and you get a different policy, with different performance. A single training run tells you almost nothing.

This is the methodological point I care most about: if you report one run's result, you're reporting a sample from a distribution and calling it a measurement. You have to train many times and characterize the spread — which is what the MoE study is really about.