Understanding
Gotham RL

What Are We Building?

Imagine a video game where the player is an artificial intelligence, the game world is a financial market, and the score is measured in profit and loss. That is Gotham RL in one sentence: an offline trading simulator where an AI agent learns, through millions of simulated trades, how to enter and exit positions on futures contracts.

The word "offline" is key. We are not plugging this into a live brokerage and letting it loose on real money. Instead, we replay historical market data, let the agent make decisions, simulate fills with realistic slippage, and measure whether the agent's strategy improves over time. Think of it like a flight simulator for a trading bot - all the physics of real markets, none of the financial risk during training.

The Three Pillars

Every part of the system maps to one of three responsibilities:

A 30-Second Primer on Futures

We trade two specific futures:

• NIY (Nikkei 225): Japan's flagship stock index. Tick size: 5 points, each tick worth 500 JPY. Traded on CME. Our trading window is the Tokyo session: 00:00–06:00 UTC, with a prime window at 00:30–01:30 UTC.
• NQ (Nasdaq 100): America's tech-heavy stock index. Tick size: 0.25 points, each tick worth $20. Also on CME. Our trading window is the US session: 13:30–20:00 UTC, with a prime window at 15:00–16:00 UTC.

The High-Level Flow

The Foundation

The common/ package is imported by everything else. It provides configuration, logging, and domain constants - the bedrock that every other module stands on.

Configuration: Layered YAML + Pydantic

If you've worked with Spring Boot's property resolution or Rails' environment configs, the pattern here is familiar. We layer multiple configuration sources, where higher-priority sources override lower ones:

1. Init kwargs (highest priority) - passed directly in code, mainly used in tests.
2. Environment variables: prefixed with GOTHAM_, nested with __. For example, GOTHAM_DATABASE__HOST=10.0.0.5 sets database.host.
3. Environment-specific YAML: config/{GOTHAM_ENV}.yaml. Set GOTHAM_ENV=prod to load config/prod.yaml.
4. Base YAML (lowest priority) - config/default.yaml, always loaded.

All of this is implemented with Pydantic Settings, which gives us runtime type validation for free. If someone sets GOTHAM_DATABASE__PORT=banana, the app fails immediately at startup with a clear validation error, not deep in a database connection two hours later.

@lru_cache(maxsize=1)
def get_settings(**overrides: Any) -> GothamSettings:
    """Singleton access to settings. Cached after first call."""
    return GothamSettings(**overrides)

The @lru_cache means the first call creates the settings object, and every subsequent call returns the same instance. Tests clear this cache in conftest.py so each test gets a fresh config.

Domain Constants: The Language of the System

Two frozen dataclasses define the financial instruments and their trading sessions:

@dataclass(frozen=True)
class InstrumentSpec:
    symbol: str        # "NIY" or "NQ"
    exchange: str      # "CME"
    currency: str      # "JPY" or "USD"
    tick_size: float   # 5.0 or 0.25
    point_value: float # 500.0 or 20.0
    session: SessionName

NIKKEI = InstrumentSpec("NIY", "CME", "JPY", 5.0, 500.0, SessionName.TOKYO)
NASDAQ = InstrumentSpec("NQ",  "CME", "USD", 0.25, 20.0,  SessionName.US)

Trading Sessions and Time Zones

Everything in the system uses UTC internally. The session windows define when each market is "active":

The prime trading window is where the agent actually makes decisions. It's a narrow 60-minute slot within each session that we've identified as having the best liquidity and trend behavior. Outside this window, the agent is forced to skip. This is an intentional constraint - most professional day traders focus on a specific window rather than trading all day.

Structured Logging

The logging setup uses structlog with two rendering modes: JSON for production (machine-parseable, feeds into log aggregation) and colored console output for development. Both share the same pipeline of processors - timestamp injection, log level annotation, exception formatting - only the final renderer differs. Log files rotate at 50 MB with 5 backups via RotatingFileHandler.

Timeframes

The Timeframe enum defines four granularities: M5 (5 minutes), M15 (15 minutes), H4 (4 hours), and D1 (1 day). Raw data enters as 5-minute bars. We aggregate up to higher timeframes for trend analysis - the idea being that a trend visible on a daily chart is more significant than one visible only on a 5-minute chart. The constant TIMEFRAME_MINUTES maps each to its duration: 5, 15, 240, 1440.

Getting Data In

The data/ package handles everything from raw CSV files to validated, stored candles in TimescaleDB. This is a classic ETL pipeline with financial-data-specific validation.

OHLCV Candles: The Universal Language of Price

CandleSource: A Pluggable Protocol

Different data sources implement the same Protocol:

@runtime_checkable
class CandleSource(Protocol):
    def fetch(self, instrument: str, start: date, end: date) -> pl.DataFrame:
        """Return 5m OHLCV DataFrame with columns:
        timestamp, instrument, open, high, low, close, volume, contract_month"""
        ...

We currently have two implementations: HistDataSource (reads 1-minute CSVs from the HistData website and aggregates to 5-minute bars) and CSVSource (reads pre-formatted 5-minute CSVs). The Protocol pattern means adding a new source - say, a direct Interactive Brokers feed - requires zero changes to the rest of the pipeline.

1-Minute to 5-Minute Aggregation

The HistDataSource reads raw 1-minute data and aggregates it using Polars' group_by_dynamic:

# 1m → 5m: group_by_dynamic("timestamp", every="5m", label="left")
# Aggregation rules:
#   open  = first()
#   high  = max()
#   low   = min()
#   close = last()
#   volume = sum()  (or 0 if no data)

Why 5-minute bars and not 1-minute or 15-minute? It's a balance. One-minute bars are noisy - too much random fluctuation for the agent to learn from. Fifteen-minute bars are too coarse - you miss important price movements. Five minutes is the sweet spot used by many institutional quant systems.

Data Validation

The CandleValidator catches two categories of problems:

There's also a zero-volume check during active sessions (a candle with no trades during market hours is suspicious) and a gap detector that identifies missing candles by comparing actual timestamps against expected 5-minute intervals.

The BackfillService

This is the orchestrator that ties it all together. It processes data in 30-day batches with per-batch commits for crash resilience:

1. Query MAX(timestamp) from the database to find where we left off (resume point).
2. For each 30-day batch: fetch from the CandleSource, validate, upsert into the database, commit.
3. If the process crashes halfway, restart picks up from the last committed batch. Upserts are idempotent - re-inserting an existing candle just updates it.

Why TimescaleDB?

TimescaleDB is PostgreSQL with time-series superpowers. Our candle data is inherently time-series - ordered by timestamp, frequently queried by time range, and write-heavy during backfills. Hypertables automatically partition data by time, making range queries fast without manual shard management. Continuous aggregates precompute our 15m/4h/1d rollups from 5m data, so we never aggregate at query time.

If you've worked with PostgreSQL before, you'll feel right at home. The connection is configured via Pydantic settings (DatabaseConfig: host, port 5432, name/user "gotham") and exposed as both synchronous (.url → postgresql://) and async (.async_url → postgresql+asyncpg://) connection strings. The Docker Compose setup runs TimescaleDB on PostgreSQL 18 with health checks.

Gap Detection

After backfilling, the GapDetector verifies data completeness. It generates expected 5-minute timestamps for each instrument's session window, skips weekends, and compares against actual timestamps in the database. Missing candles are reported as GapInfo objects with the expected time and gap type ("missing_candle" or "unexpected_gap"). Sessions that cross midnight (like Tokyo) are handled by adding a day to the end time when end ≤ start. This is critical for data quality - a missing candle during the agent's trading window could cause incorrect feature calculations.

Contract Rolls

Reading the Market

The features/ package transforms raw price data into meaningful signals. Think of features as the agent's senses - the eyes, ears, and intuition that let it perceive what's happening in the market.

IFVG: Inverse Fair Value Gaps

This is the core trading concept in the system. Let's build it from first principles.

A Fair Value Gap (FVG) is a three-candle pattern where price moves so aggressively that it leaves a "gap" - a price range where very little trading occurred. Imagine three candles in a row where the middle candle is exceptionally long. If the high of candle 1 doesn't overlap with the low of candle 3, there's a gap between them.

An Inverse Fair Value Gap is a regular FVG that gets "inverted" - price comes back and crosses through the gap zone from the other side. When a bullish FVG is inverted (price closes below it), that zone often becomes a resistance level. When a bearish FVG is inverted (price closes above it), it often becomes support. This is the trading edge: IFVGs mark zones where price is likely to react.

The code in features/ifvg.py detects FVGs by scanning 3-candle windows. The minimum gap size is 4 ticks (configurable). Each IFVG goes through a lifecycle:

Each IFVG also gets a quality score based on the displacement candle (the big candle that created the gap). The scoring logic is:

# IFVG Quality Scoring (features/ifvg.py)
if gap_ticks >= 8 and body_ratio > 0.70:
    quality = "high"    # score = 3
elif gap_ticks >= 6 or body_ratio > 0.60:
    quality = "medium"  # score = 2
else:
    quality = "low"     # score = 1

The intuition: a big, decisive candle with a large gap signals strong institutional activity. A high-quality IFVG with a gap of 10+ ticks and a body that fills 80% of the candle range is much more likely to act as a reliable support/resistance zone than a small, indecisive one.

Fill percentage tracks how much price has penetrated the IFVG zone. For a bullish IFVG: fill = clip(penetration / gap_size, 0, 1) where penetration = high - zone_lower. When fill hits 100%, the zone is dead. Test count increments each time price touches the zone boundary without fully filling it.

The IFVG module outputs 5 features per bar: ifvg_count_active (number of live IFVG zones), ifvg_nearest_dist (distance in ticks to the nearest zone, 0 if inside one), ifvg_best_quality (0 = none, 1–3 scale), ifvg_avg_fill_pct (how "used up" the active zones are), and ifvg_direction_bias (bullish count minus bearish count, normalized to [-1, 1]).

Trend Analysis: Multi-Timeframe Momentum

We compute trend features at three timeframes - M15, H4, and D1 - because trends have a hierarchy. A stock can be trending up on the daily chart while pulling back on the 15-minute chart. The compute_trend_features function computes EMA-20, EMA-50, their slopes, ATR-14 (a volatility measure), and a structure classification per timeframe.

Structure classification uses swing points (local highs and lows detected with a 5-bar lookback). If the last 4 swing highs are rising and the last 4 swing lows are rising, it's an UPTREND. Both falling: DOWNTREND. Otherwise: RANGING.

The final composite trend score blends all three timeframes:

trend_score = clip(
    D1_score × 0.40 +    # Daily trend matters most
    H4_score × 0.35 +    # 4-hour trend is next
    M15_score × 0.25,    # 15-min trend is tactical
    -1.0, 1.0
)
# Where: uptrend = +1.0, ranging = 0.0, downtrend = -1.0

Session Features: Time-of-Day Context

Markets behave differently at different times. The first hour of a session is typically volatile (the "opening drive"), while the middle is often choppy. Session features capture:

• minutes_since_open: how far into the session we are
• in_trading_window: whether we're in the prime 60-minute slot
• window_progress_pct: position within the window (0.0 = start, 1.0 = end)
• overnight_range: high minus low during the 2 hours before session open. A wide overnight range often signals a directional day.
• prior_session_high/low/close: yesterday's key levels, which act as magnets and barriers for today's price.

Room-to-Right: How Far Can Price Go?

Entering a trade is pointless if the price immediately runs into a wall of resistance. The Room-to-Right (RTR) module estimates how much space the price has to move before hitting obstacles.

It works by detecting liquidity pools: clusters of swing highs or swing lows that are close together in price. The algorithm greedily clusters swing points within 0.5% of each other. When 3 or more swing points cluster together, that's a liquidity pool. Pools are classified as "resistance" (mostly swing highs), "support" (mostly swing lows), or "mixed". Pools act as magnets: the more "touches" at a level, the more likely price will react there.

The RTR score starts at 100 and is decremented for each pool: penalty = 10.0 × touches / max(distance_factor, 0.1) where distance_factor = distance_to_pool / ATR. Nearby pools with many touches cause the biggest reductions. The rtr_score_long considers pools above the current price (obstacles to upside moves), while rtr_score_short considers pools below (obstacles to downside moves).

There's also an exhaustion measure: exhaustion_pct = clip(intraday_range / ADR_20, 0, 1) where ADR_20 is the average daily range over the past 20 trading days. If today's range already exceeds 70% of the 20-day average, the market may be running out of energy. When exhaustion exceeds 70%, an additional penalty is applied to both RTR scores: penalty = (exhaustion_pct - 0.70) × 100. The module also tracks obstacle_count_long and obstacle_count_short: the raw count of pools in each direction.

The Pre-Screen Gate

Not every 5-minute bar is worth evaluating for a trade. The pre-screen is a simple AND filter that all four conditions must pass:

pre_screen_passed = (
    ifvg_count_active >= 2         # At least 2 active IFVGs
    AND abs(trend_score) >= 0.3    # Clear enough directional bias
    AND max(rtr_long, rtr_short) >= 30.0  # Enough room to move
    AND in_trading_window == True   # Within our 60-min window
)

Displacement Detection

Within the trend module, there's a displacement detector that identifies aggressive candles - ones where institutions are likely driving price. A candle qualifies as a "displacement" if both conditions hold: the body fills >70% of the total range (it's decisive, not indecisive) and the range exceeds 1.5× ATR (it's large relative to recent volatility). Displacement candles are the building blocks of FVGs and trend structure changes.

The Pipeline: Putting It All Together

The run_feature_pipeline function orchestrates all six modules in sequence:

1. build_multi_timeframe(candles_5m): aggregates 5m bars to M15, H4, and D1 candles.
2. compute_trend_features(candles_5m, htf_frames): computes EMAs, slopes, structure, ATR, displacement per timeframe, then joins to 5m via join_asof with backward strategy (forward-fills the most recent higher-timeframe value).
3. compute_ifvg_features(result, tick_size): detects FVGs, tracks inversions, scores quality, outputs 5 features per bar. Also computes structural stop levels from IFVG zone boundaries for each direction.
4. compute_session_features(result, instrument): adds session-aware columns: minutes since open, window position, overnight range, prior session levels.
5. compute_rtr_features(result): detects liquidity pools, computes room-to-right scores and exhaustion metrics. Also computes structural target levels from nearest liquidity pools for each direction.
6. compute_structural_rr(result): computes risk-reward ratios from IFVG-derived stop levels and liquidity-pool-derived target levels. Falls back to ATR-based distances when structural levels are unavailable. Outputs structural_rr_long and structural_rr_short per bar.
7. apply_pre_screen(result): applies the AND filter, adds the pre_screen_passed boolean column.

The output is a Polars DataFrame with the original 5m OHLCV data plus all computed feature columns - roughly 45 additional columns (including structural stop/target levels and R:R ratios). This enriched DataFrame is saved as a Parquet file for downstream consumption by the LLM assessment generator and the RL training pipeline.

The AI Eye

The llm/ package uses Anthropic's Claude to add qualitative intelligence that's hard to capture with math alone. Think of it as a junior analyst who reviews each setup and gives a structured opinion.

Why Use an LLM at All?

Some market patterns are easy to quantify (a 50-bar moving average crossing above a 200-bar one). Others are harder: "this looks like a failed breakdown that's about to squeeze higher" or "this choppy price action suggests institutional accumulation." Experienced traders see these patterns intuitively. An LLM can encode some of that pattern recognition into features that the RL agent can learn from.

The LLM is not making trading decisions. It's providing 6 additional features to the observation vector. The RL agent decides what to do with them.

What Claude Sees

For each pre-screened bar, we build a compact prompt with these sections:

1. Compact CSV: the last 50 candles as ts,O,H,L,C,V, timestamps as HH:MM. This is the raw price action.
2. Trend Summary: structure and slope at each timeframe, plus the composite trend score.
3. IFVG Context: count of active IFVGs, nearest distance, best quality, direction bias.
4. Session Context: minutes since open, window progress, overnight range, prior session levels.
5. Room to Right: long/short RTR scores, exhaustion percentage.

Forcing Structured Output

We don't want free-text responses. We use Claude's tool use API with tool_choice={"type": "tool", "name": "submit_assessment"} to force the output into a predefined JSON schema. The LLMAssessment schema has 12 structured fields:

Encoding: Text to Numbers

Neural networks eat numbers, not strings. The encode_assessment function converts the LLM's structured output into 6 floats, all scaled to [0, 1] or [-1, 1]:

llm_confidence         = confidence                    # [0, 1]
llm_setup_type         = setup_type_index / 4.0         # [0, 1]  (5 types → 0..4)
llm_rr_estimate        = min(rr_estimate / 5.0, 1.0)    # [0, 1]
llm_regime             = regime_index / 3.0              # [0, 1]  (4 regimes → 0..3)
llm_concern_count      = min(len(concerns) / 5.0, 1.0)  # [0, 1]
llm_narrative_sentiment = narrative_sentiment             # [-1, 1]

Caching to Parquet

Calling Claude during RL training would be impossibly slow and expensive. Instead, we pre-generate all assessments once and cache them as Parquet files. The generate_assessments function processes all pre-screened bars in batches of 50, saving checkpoints along the way. It supports resume - if the process crashes, it skips already-cached timestamps. Failed API calls get a null encoding (all zeros).

Quality Checks

Two automated checks catch potential problems: confidence saturation (more than 30% of assessments at confidence = 1.0, suggesting the LLM is overconfident) and regime concentration (more than 80% of assessments sharing the same regime, suggesting lack of discrimination).

Cost Tracking

The client tracks token usage per request and computes cost using keyword-based model identification. The pricing table:

For bulk assessment generation, we use Claude Haiku to keep costs manageable. The generate_assessments function logs progress with running cost and ETA, so you can see "450/1200 assessments done (37%), $2.14 spent, ETA 18 min" in the console. The client uses exponential backoff for retries (sleep(2^attempt)) but doesn't retry on client errors like 400 or 401.

Prompt Versioning

The system prompt loaded from prompt_v1.txt (or prompt_v2.txt) defines the persona and evaluation criteria for the LLM. Version 2 added narrative_sentiment as a new field. Each cached assessment records its prompt_version, so you can retrain with v2 assessments without invalidating v1 data. The resume logic in generate_assessments filters by version when deciding what to skip.

RL for Backend Devs

This chapter explains reinforcement learning concepts from scratch. No math prerequisites - just analogies and intuition.

The Dog Training Analogy

Reinforcement learning is like training a dog. You don't show the dog a million examples of "good behavior" and "bad behavior" (that would be supervised learning). Instead, you let the dog try things, and you give it a treat when it does something good and a stern "no" when it does something bad. Over many repetitions, the dog figures out which behaviors lead to treats.

Episodes: One Game Level

An episode is one complete run of the game. In our case, one episode is one trading session - say, the Tokyo session on January 15th. The agent sees each 5-minute bar in sequence, decides whether to trade at each bar, and the episode ends when the session closes. Then we reset and start a new episode (maybe the US session on January 15th, or Tokyo on January 16th).

Episodes always end by truncation (the session ran out of time), never by termination (the agent didn't "die"). If the agent has an open position at session end, it's force-closed at the current price. This is realistic - a day trader never holds overnight. The reset logic selects episodes via a seeded random number generator (np.random.default_rng(42)) for reproducibility.

Policy: The Strategy

The agent's policy is its strategy - a function that maps observations to actions. In our case, it's a small neural network with 2 hidden layers of 64 neurons each (written in config as policy_net_arch: [64, 64]). The input is 36 numbers (the observation), and the output is a probability distribution over possible actions. At the start of training, this is essentially random. By the end, it should have learned patterns like "when the trend is strongly bullish and there's a fresh IFVG with favorable structural R:R, enter long."

Why such a small network? Trading decisions don't require the deep pattern recognition that image classification does. A [64, 64] MLP has roughly 5,000 trainable parameters - enough to learn the relationships between 36 input features and 2 output sub-decisions (entry and size), but small enough to train quickly and avoid overfitting. If the network were too large, it might memorize specific market patterns from the training period rather than learning generalizable rules.

PPO: Learning Carefully

PPO (Proximal Policy Optimization) is the specific algorithm we use to update the policy. The key idea: don't change your strategy too much in one update. If a few lucky trades make "go all-in on every trade" look great, PPO prevents the agent from swinging wildly to that extreme. It uses a "clip range" (set to 0.2) that limits how much the probability of any action can change in a single update step.

We specifically use MaskablePPO from the sb3-contrib library. The "maskable" part is crucial - it means we can tell the agent "these actions are not allowed right now" and it respects those constraints during both action selection and learning.

Key Hyperparameters

Action Masking: Guardrails

The agent can't do whatever it wants. Action masking enforces rules like "you can't enter a new trade when you're already in one", "you can't trade if you've hit the daily loss limit", and "you can't enter a direction where the structural risk-reward is unfavorable." This is implemented as a binary mask - an array of 6 booleans (one per sub-action option across the 2 dimensions) where False means "this option is blocked."

The Trading Brain

The rl/ package implements the Gymnasium environment, observation builder, action space, position simulator, and reward function. This is where all the pieces come together.

The 36 Observation Features

Every 5-minute bar, the agent sees exactly 36 numbers, all scaled to the range [-1, 1]. Why? Neural networks learn best when inputs are small and centered near zero. Here they are, grouped by source:

String-valued features get mapped to numbers: slopes become {rising: 1.0, flat: 0.0, falling: -1.0}; structures become {uptrend: 1.0, ranging: 0.0, downtrend: -1.0}. Distances are divided by ATR (Average True Range) to make them volatility-adaptive - "20 ticks away" means something very different on a calm day versus a volatile one.

The Action Space: 9 Combinations

The action space is MultiDiscrete([3, 3]): two independent sub-decisions, each with 3 options. That's 3 × 3 = 9 possible combinations. Stop-loss and take-profit are not agent decisions - they're derived from market structure (explained below).

Structure-Aware Stop-Loss and Take-Profit

Rather than letting the agent choose stop and target distances (which adds combinatorial complexity and leads to over-fitting), the system derives them from market structure:

Action Masking Rules

Certain actions are blocked based on the current state. The mask is a flat array of 6 booleans (3 for entry + 3 for size). When any of these conditions are true, long and short entry are masked (only skip is allowed):

• Already in a position: can't enter a second trade
• Daily P&L ≤ -3R: daily loss limit hit, stop trading
• Trades today ≥ 5: maximum trades per session reached
• Structural R:R below 1.5: long is blocked if structural_rr_long < 1.5; short is blocked if structural_rr_short < 1.5. This prevents the agent from entering trades where the market structure doesn't offer a favorable risk-reward ratio.

Position Simulation

When the agent enters a trade, we simulate it with 1-tick slippage (you always get a slightly worse fill than the ideal price - this is realistic). Each subsequent bar, we check: did the price hit the stop-loss? (Checked first - conservative assumption.) Did it hit the take-profit target? We track MAE (Maximum Adverse Excursion - worst drawdown during the trade) and MFE (Maximum Favorable Excursion - best unrealized profit during the trade). Commissions are realistic: $1.25 per contract for NQ, 80 JPY per contract for NIY.

The trailing stop has a special mechanism: when the trade's MFE reaches 1R (i.e., the trade has moved one risk-unit in your favor), the stop automatically moves to breakeven (entry price). This locks in a risk-free trade. The stop_moved_to_breakeven flag ensures this happens only once per position.

A completed trade produces a CompletedTrade dataclass with detailed statistics: entry/exit prices, P&L in ticks, risk in ticks, the realized R-multiple (P&L / risk, net of commission), whether it hit the target or stop, bars held, MAE, MFE, commission in the instrument's currency, and the exit reason ("stop", "target", or "session_end"). The realized_rr formula accounts for commissions by converting them to ticks: commission_ticks = commission / (size × tick_size × point_value), then realized_rr = (pnl_ticks - commission_ticks) / risk_ticks.

The Reward Function

The reward has three components:

The trade reward is the realized R-multiple of the closed trade, with a 0.3 bonus for hitting the target (encouraging the agent to let winners run rather than cutting them short). The step penalty accelerates when the daily P&L drops below -2R, punishing the agent for digging deeper into a losing day. The patience bonus is tiny (0.001) but crucial - it gives the agent a small positive reward for not trading when there's no position. Without this, the agent would see "skip" as a zero-reward action and prefer to always enter trades.

Teaching and Testing

Training, evaluation, baselines, and the criteria for deciding whether a model is good enough.

The Training Loop

Training follows the standard PPO loop: collect experience, compute advantages, update the policy network. In concrete terms:

1. The agent plays through episodes, collecting 2,048 steps of experience (observations, actions, rewards).
2. Using those 2,048 steps, PPO computes "advantages" - how much better or worse each action was compared to what was expected.
3. The policy network is updated in mini-batches of 256 steps, with the clip range preventing drastic changes.
4. Repeat from step 1 until we've reached 1,000,000 total timesteps.

1 million timesteps sounds like a lot, but each "timestep" is just one 5-minute bar. With ~12 bars per trading session and ~250 trading days per year, that's roughly 3,000 simulated sessions - or about 12 years of daily trading compressed into a few hours of compute time.

The training hyperparameters are tuned for this specific problem:

# TrainingConfig (common/config.py)
total_timesteps:   1_000_000   # Total experience to collect
learning_rate:     3e-4        # Adam optimizer step size
n_steps:           2_048       # Steps per rollout buffer
batch_size:        256         # Mini-batch size for updates
gamma:             0.99        # Discount factor (long-term focus)
clip_range:        0.2         # PPO clipping parameter
ent_coef:          0.01        # Entropy bonus for exploration
checkpoint_freq:   50_000      # Save every 50K steps
eval_freq:         50_000      # Evaluate every 50K steps
max_daily_loss_r:  3.0         # Daily loss limit (in R-multiples)
max_trades_per_session: 5      # Max trades before forced skip
policy_net_arch:   [64, 64]    # Hidden layers

Callbacks: Checkpoints and Evaluation

During training, two callbacks run automatically:

• Checkpoint callback: saves a snapshot every 50,000 steps to models/checkpoints/. If training crashes, you can resume from the latest checkpoint.
• Evaluation callback: every 50,000 steps, runs the current policy on the validation set (up to 50 episodes) with deterministic=True and logs metrics. If the Sharpe ratio beats the previous best, saves the model to models/best/best_model.zip.

Evaluation Metrics

A model must pass all three criteria to be "promoted":

Additional metrics tracked but not used as promotion gates: avg_rr (average R-multiple per trade), profit_factor (gross profit / gross loss - infinity if no losses), total_r (sum of all R-multiples across all evaluated episodes), and trades_per_session (helps spot over-trading).

The evaluation function runs model.predict(obs, deterministic=True, action_masks=masks) for up to 500 steps per episode and collects all CompletedTrade objects. Setting deterministic=True means the agent picks the most probable action (no exploration noise), which gives a cleaner picture of what the policy has actually learned versus what it stumbles into randomly.

Baseline Agents: The Minimum Bar

How do you know if your trained agent is any good? Compare it against baselines that require no learning:

LLM Ablation

To measure whether the LLM features actually help, we train a separate model with ablate_llm=True. This zeros out the first 6 observation features (the LLM-derived ones) by setting obs[:LLM_FEATURE_COUNT] = 0.0 at each step. The agent sees only the 30 numerical features from IFVG, trend, session, RTR, structural R:R, microstructure, and portfolio modules. If the full model significantly outperforms the ablated one on the same validation set, the LLM assessment adds measurable value. This is a controlled experiment - everything else stays identical, so any performance difference is attributable to the LLM features.

Time-Based Splits

We split data chronologically, never randomly:

The Full Journey

Let's trace one data point's lifecycle from raw CSV to policy update. This is where all eight chapters converge.

One Data Point, Thirteen Steps

The Model Registry

When training completes, save_model writes four files to models/{timestamp}/: the model weights (model.zip), a metadata.json (instrument, training duration, git hash), eval_metrics.json (Sharpe, drawdown, win rate, profit factor), and a config_snapshot.yaml (frozen copy of the config used for training). The best model is also saved to models/best/best_model.zip.

list_models scans the models directory and returns all saved runs sorted by timestamp, making it easy to compare experiments.

Config System in Action

Change one value in config/default.yaml and it ripples through the entire pipeline. Here are some examples of how single config changes affect the whole system:

• Set features.pre_screen_min_trend: 0.5 (up from 0.3) → fewer bars pass pre-screen → fewer LLM calls → fewer training episodes → a more selective but potentially undertrained agent.
• Set training.ent_coef: 0.05 (up from 0.01) → the agent explores more random actions during training → slower convergence but wider strategy search.
• Set features.ifvg_min_gap_ticks: 8 (up from 4) → only large FVGs are detected → fewer active IFVGs per bar → pre-screen becomes harder to pass.
• Set training.gamma: 0.95 (down from 0.99) → the agent discounts future rewards more heavily → prefers quick trades over patient setups.
• Set features.ema_fast: 10 (down from 20) → the fast EMA reacts quicker → more slope changes detected → potentially noisier trend signals.

The config snapshot saved with each model makes experiments reproducible. You can always answer "what settings produced this model?" by reading config_snapshot.yaml in the model directory.

Feature Configuration Reference

For reference, here are the key configurable thresholds that shape the feature pipeline:

# FeatureConfig defaults (common/config.py)
ifvg_min_gap_ticks:     4.0     # Minimum gap size to detect an FVG
ifvg_max_age_bars:      100     # Bars before an IFVG expires
ema_fast:               20      # Fast EMA period
ema_slow:               50      # Slow EMA period
atr_period:             14      # ATR lookback period
displacement_body_pct:  0.70    # Min body ratio for displacement candle
displacement_atr_mult:  1.5     # Min range as multiple of ATR
rtr_lookback_days:      20      # Days for liquidity pool detection
pre_screen_min_ifvgs:   2       # Min active IFVGs to pass pre-screen
pre_screen_min_trend:   0.3     # Min |trend_score| to pass pre-screen
pre_screen_min_rtr:     30.0    # Min RTR score to pass pre-screen

# Structural SL/TP defaults (rl/env.py EnvConfig)
stop_atr_fallback_mult: 1.5     # ATR multiple for stop when no IFVG zone
target_fallback_r:      2.0     # R-multiple for target when no liquidity pool
min_structural_rr:      1.5     # Minimum R:R to allow entry (action mask gate)

What's Next

The system has two modules shown with dashed borders in the architecture diagram - they exist as stubs, ready for implementation:

• Execution (execution/) - connects to Interactive Brokers via their API on port 4002. Takes the agent's action and converts it into real orders with proper position sizing, bracket orders (stop + target), and session-end flatten logic.
• Monitoring (monitoring/) - Telegram bot for real-time alerts (trade entries, exits, daily P&L summaries) and health checks (is IB connected? Is data flowing? Is the model loaded?).

The transition from offline to live trading is the next frontier. All the infrastructure is in place: the config system supports ib.readonly=False, the position simulator can be swapped for real order management, and the evaluation metrics define what "good enough" looks like. The gap between simulation and reality is always non-trivial, but the architecture was designed with this transition in mind.