# ETF Quant Intraday — Handoff

_Last updated: 2026-07-07. Branch: `etf-quant-intra`. Repo: `/home/user/fut`._

## 1. What this project is

Cross-sectional **mean-reversion** strategy on a **35-ETF** universe, **5-second bars**, **lag-1 taker** execution.
Walk-forward CatBoost MultiRMSE predicts 5 forward mid-return horizons; a portfolio optimizer turns
predictions into cash-neutral, vol-targeted weights with turnover penalties; we backtest net-of-cost P&L.

**Core pipeline:**
- **Model:** walk-forward CatBoost MultiRMSE, depth 3, seed 0, `train=4h`, `step=2h`, embargo = 2×Hmax.
  5 horizons: `15s / 30s / 1min / 5min / 30min`. Base learner + seed-stability choices are settled
  (FastGBM bag ×48 was the seed-stable contest winner; production uses single seed 0 MultiRMSE).
- **Optimizer** (`fut.ablate.build_target`): `w ∝ Σ⁻¹α`, soft factor-neutralization (`var_keep`/`rho`),
  vol-target 0.20, cash-neutral. `alpha` = simple mean of the horizon predictions.
- **Turnover control** = tcost penalties on Δw inside the optimizer (NOT rebalance-frequency — that
  option was explicitly dropped): L1 (`l1_tc·hs·|Δw|`), L2, capped-dust L1 (`lcap_tc·hs·min(|Δw|,l1_cap)`,
  solved by CCP with `ab._N_CCP` iters), reweighted-L1 (Candès–Wakin–Boyd). Plus `gate_g` (per-name
  edge>spread gate) and `cov_every` (recompute Σ every N bars) knobs.
- **Backtest:** `fut.ablate.band_walk(..., lag=1, rebar=1, k_grid=[0])`. **Only ever report lag=1**
  (lag=0 fills at mid[t], an input to the signal — not executable). Returns net_day, gross_day, sharpe,
  oos_days/oos_daily, traded_days.

## 2. The big finding (why we cleaned the data)

The old "champion" tcost config (+0.090%/day, Sharpe +2.79) was **100% a data artifact**. Root cause:
the production crawler (`polygon_etf_crawl.py`) set `mid = close` where `close` was the **last raw trade**
of a 5s bar — unfiltered. Odd-lot / off-exchange bad prints became the bar price, creating 8–12% phantom
5s "moves" that a mean-reversion book fakes profit reverting. (e.g. EPI 46.44 = a 15-share odd-lot print.)

After cleaning, the same config is net **−0.006%/day** (g/c 0.82) on the full universe. The **gross** signal
is real and survives cleaning (gross Sharpe +20–40, Mar–Jun) — there IS edge at mid — but on the full 35-name
universe it does **not clear the taker spread**. This is the central open problem.

## 3. The clean-data pipeline (DONE, committed)

Robust 5s bar builder from Polygon trades — fixes the `mid=close` bug. Committed as:
- **`src/fut/etf_bars.py`** — `bars5s_clean(df, ticker)` + `pull_trades_day(ticker, day)`.
  Price (open/high/low/close/vwap/mid) built ONLY from "good" trades = round-lot (size≥`RL=100`) survivors
  of a centered rolling-median tick-outlier filter (reject |p−med|/med > `TICK=0.01`). Illiquid-name fallback
  (`RL_MIN_FRAC=0.2`) keeps all sizes when round-lots are sparse. **Volume = ALL trades** (odd+round).
  `eff_spread = (buy_vwap − sell_vwap).clip(lower=0)` via tick-rule. Final bar-level Hampel pass on mid.
- **`src/fut/etf_recrawl.py`** — the re-crawl driver (was `clean_recrawl.py`).
- **`src/fut/refill_incomplete.py`** — per-month completeness check vs dirty reference; 2 workers;
  bracket-safe pgrep (`[e]tf_recrawl.py`). Built after a network outage silently dropped 59 ticker-days
  (per-day try/except still wrote partial files). 0 lost after refill.
- **`src/fut/panel_rebuild.py`** — memory-safe (per-ticker part files, 4 workers, concat once at end).
  eff_spread/spread_now now flow from the clean bars (self-consistent; no dirty carry needed).
- **`src/fut/etf_predict.py`** — retrains 5-target MultiRMSE on the clean panel, computes `y_fwd_mid_15s`
  on the fly, runs `rolling.run(...)` → `data/h15s_full/predictions.parquet`.

**Bandwidth:** crawl is throttled (0.04s/page, capped workers) — user requirement, keep it that way.
**Times:** report Pacific 12hr like `2:02pm` (lowercase, no tz suffix). Box clock is UTC — convert with
`TZ=America/Los_Angeles`. Never label `date -u` output as PT.

## 4. Current data state (verified 2026-07-07)

| Artifact | Rows | Roots | Range | Notes |
|---|---|---|---|---|
| `data/etf_panel.parquet` (5.28 GB) | 19.5M | 35 | **2025-12-29 → 2026-06-25** | clean panel, ~6 months only |
| `data/h15s_full/predictions.parquet` (326 MB) | 12.2M | 35 | 2025-12-29 → 2026-06-25 | 5-horizon preds on clean panel |
| `data/etf_bars5s/root=<TK>/*.parquet` | — | 35 | same | clean 5s bars |

- Clean panel also uploaded to `s3://someben-fut/derived/etf_panel.parquet` (byte-verified).
- Data is **gitignored**. Working tree is clean; no background jobs running (load ~0.07).
- **⚠️ The panel is still only ~6 months.** The long-history rebuild is the #1 prerequisite (see §6).

## 5. Results so far (clean data, all lag=1)

- **Full 35-universe, best tcost config** (l1=6, dust lcap=24/cap=0.005, cov_every=1, k=0, all-5 horizons):
  net ≈ −0.006%/day, **g/c 0.82**. Gross is positive; cost eats it.
- **cov_every:** helped on DIRTY data (60), **HURTS on clean** (use `cov_every=1`).
- **Horizons:** `all5` (mean of all five) is optimal. Fast horizons (15s/30s) carry the signal; dropping
  them (`nofast`/`slow`) is worse. Slower-only horizons are a dead end.
- **Tight-spread universe** (`fut` scratchpad `univ_clean.py`, ranked tightest→widest half-spread):
  `tight16` reaches **g/c 1.44**, net +0.011%/day — net-positive — BUT boundary-fragile (N=15 and N=17 go
  negative). That's noise, not durable edge. User then said: **revert to full universe**, don't chase it.
- Dust/capped-L1 lifts g/c from plain-L1's ~0.44 up to 0.82 on the full universe (still net-neg).

**Interpretation:** real gross alpha, insufficient to clear taker cost on the full universe at current
data length. Push gross accuracy UP (the HPO) before touching execution again.

## 6. NEXT UP — the plan (approved, not yet launched)

### 6a. Build the long clean history (PREREQUISITE — do first)
Confirmed Polygon has full-35 coverage **2024-01-11 → now** (IBIT launched 2024-01-11 caps the balanced
35-ETF start; ~2.5 yr = ~5× current data). Steps:
1. Re-crawl clean 5s bars over the full range with `etf_recrawl.py` (throttled) → `data/etf_bars5s/`.
2. `refill_incomplete.py` to guarantee 0 lost ticker-days.
3. `panel_rebuild.py` → `data/etf_panel.parquet` (full range).
4. Re-upload to S3 (multipart; scratchpad `upload_panel.py`).
5. `etf_predict.py` → regenerate `data/h15s_full/predictions.parquet`.

### 6b. Optuna TPE HPO (the main event) — decisions locked
- **Objective:** gross Sharpe (zero tcost, full universe, walk-forward). Push gross accuracy first;
  execution/tcost is a SEPARATE later stage.
- **Split:** none — full sample (user: "no split, do full sample").
- **Compute:** **Modal, NON-PREEMPTIBLE** (must finish). Modal spend needs approval ($50 cap).
- **Search space:** target horizon(s), model features, training window (these need model retrain per trial)
  × covariance window, covariance shrinkage, factor-neutralization `var_keep`/`rho` (construction-side,
  cheap — no retrain). Nest/amortize: retrain once per (features,train,horizon), sweep the cheap
  construction knobs inside. Ranges already agreed as fine.
- Existing scaffolding: `src/fut/hpo.py`, `src/fut/modal_hpo.py` (both modified in an earlier WIP — check
  `git stash`/status before reusing; they were dirty at one point).

### 6c. Execution/tcost stage (AFTER gross is maximized)
Re-run the tcost optimizer (L1/L2/dust/reweighted-L1 + gate_g) on the best gross config to see if the
improved gross now clears the spread on the full universe.

## 7. Hard constraints (do not violate — from user + memory)

- **NEVER** propose passive / maker / limit-order / liquidity-provision / market-making execution. Ruled out.
- **NEVER** dismiss fast-horizon (15s/30s) signal as "bid-ask bounce"/noise — it's just a trading horizon.
- **Only report lag=1** P&L. Never show or mention lag=0.
- Report times Pacific 12hr as `2:02pm`; box is UTC, convert.
- Never use the words "honest"/"honestly"; never use "fold" (say "step"/"window").
- **SAVE EVERYTHING** (rule #1). **Actively monitor** long jobs, relay incremental results, never
  fire-and-forget (rule #10). **User controls all launches** — plan, don't launch without go-ahead.
- Commit only when asked. Be bandwidth-conscious. Don't suggest "save less to S3" or "reserved" AWS.

## 8. Infra notes (from the cost Q&A)

- S3 egress ($0.09/GB out) is the cost driver; same-region compute→S3 is free. **Cloudflare R2** =
  zero egress, S3-compatible — recommended for the cache.
- AWS box for control+trading: **r7a.large** (2 vCPU / 16 GB, us-east-1, on-demand ~$93/mo, ~$100–110 all-in
  with EBS). **On-demand only** — no reserved.
- Heavy parallel work → **Modal** (already the plan). To let the local machine also pitch in on the HPO:
  point local + EC2 workers at a **shared Optuna study** (RDBStorage); each pulls trials independently,
  NAT-friendly (workers dial out). No cluster framework needed. Can't pool RAM across machines — only
  parallel tasks. Don't hand-roll a Dask/Ray cluster when Modal already does this.

## 9. Key files quick-reference

| File | Role |
|---|---|
| `src/fut/ablate.py` | optimizer `build_target` (gate_g, cov_every, tcosts) + backtest `band_walk` |
| `src/fut/etf_bars.py` | clean 5s bar builder from trades (`bars5s_clean`, `pull_trades_day`) |
| `src/fut/etf_recrawl.py` | clean re-crawl driver |
| `src/fut/refill_incomplete.py` | completeness check + repair dropped ticker-days |
| `src/fut/panel_rebuild.py` | rebuild panel from clean bars (memory-safe part files) |
| `src/fut/etf_predict.py` | retrain 5-horizon MultiRMSE → predictions.parquet |
| `src/fut/rolling.py` | walk-forward driver (`rolling.run`) |
| `src/fut/hpo.py`, `src/fut/modal_hpo.py` | Optuna HPO scaffolding (WIP — check dirty state) |
| `src/fut/polygon_etf_crawl.py` | OLD production crawler with the `mid=close` bug — DO NOT reuse for price; untouched by scope |
| scratchpad `univ_clean.py` | tight-universe + horizon sweep |
| scratchpad `tc_isolate.py` | tcost isolation sweep |
| scratchpad `upload_panel.py` | multipart S3 upload |

Scratchpad dir: `/tmp/claude-1001/-home-user-fut/fba1411f-f78c-4f40-bf50-02e0c542e56d/scratchpad/`
(session-specific — copy anything you want to keep into the repo).

## 10. Resume checklist

1. `cd /home/user/fut && source .venv/bin/activate` (python is `python3` inside the venv only).
2. `git status` on branch `etf-quant-intra`; check `hpo.py`/`modal_hpo.py` dirty state.
3. Kick off **§6a** (long clean history) — throttled crawl → refill → rebuild → re-upload → re-predict.
   Monitor incrementally.
4. Then build & launch the **§6b** Optuna HPO on Modal (get spend approval first).
5. Keep §7 constraints in view at all times.