- We asked our research agent to find a tradable signal in a US-equity universe. After 29 steps of real work, it reported that there isn't one — and that refusal is the point.
- A quant agent that always finds an edge is not impressive; it's dangerous. The capability that matters is knowing, rigorously, when there is no edge — and saying so.
- On a fixed 12-name universe it tested momentum and mean-reversion, split the data 70/30 in time, controlled for multiple testing (FDR q≤0.10), and found both signals' out-of-sample information coefficient negative (−0.142 and −0.068). Verdict: no validated signal, stay neutral.
- On a separate risk question it refused to guess portfolio weights, asked the human first, decomposed the risk (66% market / 34% single-stock, beta 1.37), then flagged that the most recent regime looked very different from the full sample.
- It doesn't reason about markets in prose alone — it writes and runs real code against a purpose-built quantitative library (factor models, shrinkage-regularized covariance, portfolio risk).
- Every number here is a real output from the live product, captured verbatim. Nothing is illustrative.
We gave our quant-research agent a deliberately open brief: "Research and validate a signal for a small US equity universe of your choice. Report its information coefficient, how you tested it, and whether it survives out-of-sample." Twenty-nine steps later it returned the answer most product demos are built to avoid: "I do not validate a live signal."
That non-result is the most important thing this agent does. In markets, an agent that always finds an edge is not a feature — it's a liability. Backtests overfit, in-sample patterns evaporate out-of-sample, and a confident model with no real signal will happily trade capital into the ground. The valuable capability is not finding alpha on demand. It's knowing, rigorously, when there is none to find.
This is a look at what the agent actually did across two real runs on the live product, with every figure copied verbatim from the screen. Where it needed a human, we show that too.
Does it just start computing, or check its assumptions first?
We asked a plain question: "What is the factor-risk breakdown of a portfolio of AAPL, MSFT, and NVDA — how much of the risk is broad market beta versus single-stock?" A naive agent picks weights and charges ahead. Ours stopped and said: "I can calculate that, but the answer depends materially on the portfolio weights," then offered a choice — equal weight, market-cap weight, or custom.
On equal weight, it produced the decomposition: broad-market beta accounted for 66.2% of modeled risk, single-stock risk 33.8%, portfolio beta 1.37, annualized volatility 25.8%.
Then the part that separates an analyst from a calculator. Unprompted, it added context: this is "not a diversified 'market-like' basket… it is still a concentrated mega-cap tech portfolio," NVDA carried the most single-stock risk, and — the caveat that matters most — "in the most recent 20% of the sample, the split moved closer to 48% market / 52% single-stock." It named its own method bound too: a one-factor decomposition using SPY as the market proxy over roughly three years of daily data, and offered to extend to a multi-factor model. Ask before assuming, show the number, then say where the number stops being trustworthy.
And it didn't produce those numbers by asking a language model to guess them. It wrote and ran real code against our factor-model library — factor loadings, a shrinkage-regularized factor covariance, specific risk, and portfolio-risk aggregation, with explicit parameters like a ridge penalty and covariance shrinkage. The rigor isn't only in the process the agent follows; it's in the quantitative machinery it has on hand.
What does "validating a signal" actually involve?
The signal-research run is the headline. The agent fixed its universe before looking at any results — twelve US large-caps (AAPL, MSFT, NVDA, AMZN, GOOGL, META, JPM, XOM, JNJ, PG, HD, CAT) — a guard against picking the universe to fit the answer. It then tested two plausible, economically distinct signals: medium-term momentum (six-month, with a 21-day skip, z-scored per name) and short-term residual mean-reversion (60-day reversal versus SPY using rolling beta).
Its test design reads like a careful quant's checklist, not a demo:
- Eight years of data (2018–2026), evaluated on monthly month-end cross-sections against the next 21-day forward return.
- Metric: the cross-sectional Spearman information coefficient (IC).
- A 70/30 chronological train/holdout split — the first 70% of months to research on, the last 30% held out. No shuffling that would leak the future into the past.
- Leakage controls: the momentum skip, and forward returns that start only after the signal date.
- A selection rule that required the best out-of-sample IC to clear multiple-testing control (FDR q≤0.10) — otherwise, default to no signal.
The results, out-of-sample:
| Signal | Train mean IC | OOS mean IC | OOS p-value | OOS positive months |
|---|---|---|---|---|
| Medium-term momentum (z-score) | 0.007 | −0.142 | 0.10 | 42.3% |
| Short-term mean-reversion | −0.02 | −0.068 | 0.27 | 30.8% |
Both out-of-sample ICs are negative. Neither clears the q≤0.10 bar. And the agent resisted the tempting overreach — it did not conclude "so invert the signals." Its verbatim reading: "the most trustworthy conclusion is not 'invert the signals,' but rather there is no validated edge here under this design." A negative in-sample-to-out-of-sample flip is evidence of noise, not of a hidden short.
"No. On this fixed 12-stock universe and 1-month horizon, neither tested signal survives out-of-sample. If I had to act on this research, I would keep the book neutral rather than automate either candidate."
So where does it still need a human?
Three seams, all of them visible above rather than hidden:
- Framing the question. In the risk run it would not proceed without the portfolio weights — the kind of judgment call a human should own, not a default the agent silently picks.
- Scope. On the stock comparison it was explicit that its answer was "generic, not personalized," and offered to run the portfolio-specific version given actual holdings. It knows the difference between a general answer and your answer.
- Judging its own limits. It named the noise (a 12-name universe makes monthly IC readings jumpy), the survivorship-bias risk (using today's large-cap membership), and the specificity of the result to this universe and horizon — then suggested the obvious next test: broaden to 50–100 names to cut IC noise.
None of these are failures. They are exactly the places where human judgment belongs, and the agent is built to surface them rather than paper over them. Keeping it honest over long horizons is a separate problem we've written about in Async Critique Agents; the risk engine behind the decomposition above is described in How We Built a Global, Cross-Asset Factor Risk Model.
What this doesn't show
- These are two runs, not a systematic measurement of the agent's hit rate across many tasks. They illustrate behavior; they don't quantify it.
- The signal study is deliberately small-universe. A broader universe — the agent itself suggested 50–100 names — would give less noisy ICs and is the natural next test, not a validated result on its own.
- The risk decomposition is one-factor (an SPY proxy). A multi-factor model would attribute country, sector, and style more precisely.
- These are research and backtested figures, not live-traded performance.
For research and informational purposes only; not investment advice. All figures are from research runs on historical data, not live trading, and past behavior does not guarantee future results.