AI Backtest Guardrail Fails: Benchmark Finds Date Prompt Cuts GPT-5 Knowledge Claims 39% to 1%

Telling GPT-5 to answer as of a past date cuts the share of answers claiming prior knowledge from 39% to 1%, while its accuracy moves only from 76% to 75%, according to FinDendro, a benchmark that a 15-author team posted to arXiv on Nov. 4. That means a standard backtesting safeguard hides what an AI model knows without removing it, so the authors conclude that a financial backtest inside a model's training window cannot be certified leak-free by asking the model, instructing it, or trusting its vendor's stated cutoff. GPT-5 answers 76% of FinDendro's controlled facts correctly but claims to know only 38% of them, and it still gets 61% right among the facts it says it does not know, the authors report.