MISTAKES · PUBLIC POSTMORTEMSNDA-SAFE · PUBLIC DATA · GATE-CAUGHT
What I got wrong.
A book of 9 winners with no postmortems is a book primed for selection bias.
A research book that ships 9 winners and hides the failure modes is a book primed for selection bias. Below are five real failures from the quant pipeline. what was wrong, which gate caught it, what I repaired, and what the discipline is for. NDA-safe. Public data.
A book is only as honest as its weakest postmortem. The minimum of wins and postmortems is the binding constraint.
CHRISTIAN.T.MACIONUTC+85 FAILURESNDA-SAFEOWNER-VERIFIED
- M01critical
Shipped a backtest with a 1-bar forward shift. Looked like a 5.07 Sharpe.
What was wrongWhile refactoring a momentum signal pipeline, I accidentally left a `.shift(-1)` on the signal column for two days. The "result" Sharpe was 5.07. The strategy was nonsense. every "trade" was using tomorrow's signal today.
Which gate caught itG21 to G25 (look-ahead discipline, point-in-time verification)
RepairWrote project 09 as the canonical one-line shift test. The leaked-Sharpe was 5.07; the real Sharpe after removing the shift was 0.55. The phantom edge was 88% of the apparent alpha. The test now runs as a pre-ship gate on every backtest.
LessonA 5.07 Sharpe is not a feature. A 5.07 Sharpe that survives a one-line shift test is not, in fact, surviving. it is lying. The shift test is now mandatory. There is no "we'll just be careful next time."
- M02high
In-sample Sharpe 0.91 in cross-sectional momentum. Out-of-sample Sharpe −0.03.
What was wrong18 BTC-correlated altcoins, ranked by 14-day return, long top decile, short bottom decile. IS Sharpe 0.91. The OOS Sharpe was −0.03. Bootstrap CI [−0.12, +0.18] straddles zero. cannot reject the null.
Which gate caught itG11 to G15 (locked OOS windows, pre-registration, frozen-spec evaluation)
RepairReported both numbers, side by side. The "honest decay" framing in the project writeup replaced what would have been a published-only-IS-Sharpe story. The bootstrap CI was reported with the bound, not as a one-sided significance test.
LessonCross-sectional dispersion in a tightly-correlated universe (BTC-altcoin beta ≈ 1) is too small a signal to overcome rebalance friction at 14-day cadence. Project 05 (pairs via cointegration) is the *first* gate for stat-arb in this universe; cross-sectional momentum is downstream. The right order would have caught this earlier.
- M03medium
BTC/ETH spread: looked like a textbook stat-arb. ADF said no.
What was wrongBTC and ETH are the most obvious pair in crypto. The textbook trade is to hedge, fade the spread, collect mean-reversion. The ADF test statistic was −2.11. Critical value at 5% is ≈ −2.86. Spread has a unit root; the trade is not stationary. Half-life 208 days. OOS Sharpe ≈ 0.
Which gate caught itG1 to G5 (mechanical validity), G6 to G10 (statistical nulls)
RepairDid not ship the trade. Shipped the rejection, with the ADF statistic and the half-life. The project is a published counter-example to the assumption "BTC and ETH are cointegrated."
LessonStat-arb starts with the cointegration test, not with the trade. If the spread isn't stationary, no position-sizing rule will save it. Project 05 is the pre-trade filter for projects 02 and 06.
- M04medium
Fade-funding trade IS Sharpe 1.11. OOS Sharpe −0.05. Alpha decayed post-2023.
What was wrongThe textbook perpetual-swap carry trade is to fade funding (short when funding > 0). IS Sharpe was 1.11. OOS Sharpe was −0.05. The carry was real (+11.9% annualized), but the trade that captured it was systematically arbitraged away as the participant mix shifted.
Which gate caught itG11 to G15 (locked OOS windows, 5-era stability)
RepairThe OOS window was locked before data was touched. The decay was visible in the rolling OOS-Sharpe trace. Sharpe 1.11 → 0.91 → 0.71 → 0.42 → 0.04 → −0.05 over the rolling window. The "carry is real, trade is dead" conclusion was honest.
LessonCarry signals are observable, regularly-paid, and visible to anyone with the data. which means they crowd fast. Pre-registering the OOS window before touching the data is what makes this a study and not a justification.
- M05low
First DSR pass on project 01: forgot γ₃·SR̂ term. Decoy result.
What was wrongWhen implementing the Deflated Sharpe Ratio from the Bailey & López de Prado 2014 paper, I first wrote the simpler form: DSR = (SR̂ − E[max SR]) / σ_SR̂. The paper's full form has the higher-moment correction: (1 − γ₃·SR̂ + (γ₄·SR̂² − 1)/4) under the radical. Without it, returns with non-zero skew/kurtosis inflate the denominator and DSR is overstated.
Which gate caught itG6 to G10 (parameter-prior sensitivity, Monte-Carlo coverage)
RepairRe-derived the formula from the paper's full expression. Added γ₃ and γ₄ to the calculator inputs as defaults (0, 3) with a comment that real-data estimates are recommended. The interactive DSR calculator on /methodology ships the full form.
LessonWhen the paper gives you a higher-moment correction, the higher-moment correction is the paper. The simplified form is for textbook examples. Always read the equation, never paraphrase it from a Stack Overflow answer.
What this page is not
Negative space as design.
This page is not self-flagellation, and it is not a humility performance. It is a record of the failure modes the gate stack is built to catch. If a gate didn't catch the failure, the gate gets updated; the failure gets logged. There is no "we'll be more careful next time" to there is a pre-ship test.
- I am not above shipping a 5.07 phantom Sharpe. that is exactly the failure mode the gates exist to prevent.
- I am not claiming the gate stack catches everything. there are 31 gates and they fail closed on the obvious ones; the subtle ones come later.
- I am not hiding any failures that I am aware of. if a postmortem would change your hire/no-hire, it is on this page.
- I am not interested in "fake failure" stories. these are real failures from the actual pipeline, with the actual numbers.
Want the failure log for a specific strategy?
If you have a written spec, send it. I'll send back which of G1 to G31 already apply to your stack, which ones would catch your last failure, and which I'd build.