OracleProto Open Source: Building a Strict Ruler for LLM Predictive Ability

Ask a large model, “Will Nvidia’s earnings exceed expectations next week?” or “How will this conflict end in three months?”, and you’ll most likely get a roundabout, diplomatic response. It’s not that the model is dumb—it’s that no one has ever seriously taught it how to predict, let alone evaluated how good its predictions are.

This week, an open‑source project called OracleProto surfaced on linux.do, aiming to change that. It’s not another leaderboard, but a reproducible evaluation framework specifically for the task of “prediction.” The core idea is simple: lock the model’s knowledge cutoff and force it to make truly out‑of‑sample predictions.

Existing Prediction Benchmarks Are Either Fake or Dirty

To understand OracleProto’s value, we need to understand why evaluating predictive ability is so hard.

There are mainly two industry approaches. One is the real‑time benchmark, where models make bets on events that haven’t yet happened, such as next month’s CPI or the outcome of an election. The advantage is that the data is perfectly clean—the model can’t possibly know the answer in advance. The problem is that the questions have a shelf life: once the event happens, the question expires, and the benchmark quickly loses statistical meaning. Results generated today can’t be directly compared with results from a new model three months later, because the questions have changed.

The other is historical replay, using past events as test questions. The biggest convenience here is repeatability and cross‑model comparison, but the cost is fatal: those events are almost certainly already in the model’s training data. Asking a model to “predict whether a company would go bankrupt in 2023” is really asking it to recall, not to infer. That’s not backtesting—it’s an open‑book exam.

Writing “please pretend you don’t know anything after 2024” in a prompt doesn’t really work either—for a model that has digested the entire internet, that’s basically no constraint at all. Knowledge boundaries are embedded in the weights, and prompts can’t lock them.

This is the pain point the OracleProto author keeps stressing: in predictive evaluation, time is the core variable—but every existing method leaks along the time dimension.

Treat Time as a Hard Constraint, Not a Suggestion

OracleProto’s solution is straightforward—some might say blunt—but also the strictest currently visible.

During question generation, the framework applies three layers of temporal isolation:

First, anchor the model’s knowledge cutoff date. Each model has its own cutoff—GPT‑4, Claude 3.x, Gemini, DeepSeek, Qwen, and so on. The framework tailors the sample pool for each one, keeping only events that occurred after the model’s cutoff but have since been resolved in reality. This blocks most possibilities of “open‑book” leakage.

Second, time masking. When constructing prompts, any temporal clues that could reveal that an event has already occurred are removed or rewritten—specific dates, relative expressions like “last week” or “yesterday,” and outcome‑revealing phrases that only make sense post‑event. The idea is to prevent the model from sniffing out the timeline contextually.

Third, tool time lock. If the model has browsing or search tools, the accessible time window is hard‑limited to before its knowledge cutoff. In other words, even if you let it go online, it can only look up what it already should have known. This is especially critical when evaluating agent‑style models with live connectivity.

Finally, a content‑level leakage detector catches any accidental signs that the model “already knows the outcome”—for example, directly stating results or referencing posterior information. Those samples are flagged and removed. According to the author, this multi‑layer setup keeps the leakage rate to about 1%, which is an impressively low number in benchmark design.

An analogy: it’s the same as strict out‑of‑sample backtesting in quantitative finance. You can’t train on 2020 data and then backtest on 2019 data and claim your strategy works. OracleProto brings that same style of rigor into LLM evaluation.

Nine Models Tested: Expensive ≠ Accurate

The author ran 80 questions across nine mainstream LLMs, including GPT, Claude, Gemini, DeepSeek, and Qwen—the top players. Two findings stand out.

First, the accuracy gap is actually quite small. The difference between best and worst is only 10.1 percentage points. This suggests that, on true out‑of‑sample prediction tasks, top models perform at roughly the same struggling level. It confirms a suspicion: prediction isn’t a proprietary skill of any one model—it’s a weak spot across all LLMs because none have been explicitly trained for it.

Second, cost differences are absurdly large. The cost per correct prediction differs by a factor of 82 between the most and least expensive models. That figure reflects not just token price, but the combined effect of accuracy × price × number of calls required. Some models are cheap per token but need multiple reasoning attempts; others are expensive but get it right in one shot. OracleProto introduces a cost‑effectiveness metric that’s very practical for engineering use.

For developers, the takeaway is: when choosing a model for predictive tasks, don’t just look at leaderboard ranks—look at cost‑normalized accuracy. In batch‑prediction scenarios such as sentiment forecasting, financial signals, or supply‑chain risk assessment, that 82× cost gap directly decides whether a product can scale.

If you want to replicate the tests yourself, aggregator platforms like OpenAI Hub make it easy to call GPT, Claude, Gemini, and DeepSeek under one API key—saves you from switching SDKs and simplifies controlled experiments.

The Real Value: Recycling “Expired” Prediction Data

Accuracy and cost are surface metrics. The most brilliant part of OracleProto is this: it makes old, “expired” prediction datasets valuable again.

Here’s why that matters. Over the past decade, academia and industry have accumulated many prediction datasets—Good Judgment Project’s geopolitical forecasts, various economic‑forecasting competitions, thousands of resolved Metaculus questions. They were valuable while events were pending, but once the answers became known, they lost evaluation value, since modern models likely already saw them in training.

OracleProto flips that logic. As long as an event happened after a target model’s cutoff date, it can be re‑packaged as a valid prediction question. That means:

• Every new model generation instantly unlocks a large batch of previously “expired” datasets, thanks to the advancing cutoff.
• Evaluation sets can be generated dynamically per model—no more “one benchmark until it dies” problem.
• These samples can serve not only for evaluation but also as SFT (supervised fine‑tuning) or RLHF/GRPO training data specifically tailored to enhance predictive ability.

That last point is the author’s biggest ambition. Current LLMs perform poorly on predictive tasks largely because their training data lacks structured samples that pair timestamped contexts with outcomes and reasoning chains. OracleProto provides an industrial pipeline for transforming historical data into exactly that type of training corpus. In other words, predictive ability can evolve from accidental emergence into a deliberately trained skill—that’s the long‑term promise of this framework.

The Framework Itself: More Engineered Than Expected

A quick look at the repo reveals solid engineering quality, with clear modular structure:

• Sample collection and cleaning: Pulls from public event sources and labels by time, domain, and verifiability
• Time masker: Three‑layer filter (NER + rules + LLM validation) handles explicit and implicit temporal signals
• Evaluation scheduler: Supports major closed‑source APIs and local open‑source models (via vLLM or similar inference backends)
• Leakage detector: Independent, pluggable post‑processing module
• Leaderboard: An online leaderboard site for community submissions

Datasets are mirrored on Hugging Face, the paper is on arXiv—the whole package is built for reproducibility, scalability, and collaboration, not just a one‑off demo.

For teams working on LLM evaluation or prediction‑based applications, there are several possible uses:

1. Model selection: Benchmark models to see their true predictive performance and cost profile
2. Domain adaptation: Replace with your own industry events (finance, healthcare, supply chain, geopolitics) following the same temporal‑isolation logic
3. Training data generation: Use the question‑generation and labeling modules to produce SFT/RL datasets in bulk
4. Leakage verification: Integrate the leakage‑detection module into your existing evaluation pipeline

Some Less Optimistic Notes

Of course, there are limitations.

The knowledge cutoff is itself a fuzzy concept. What model creators publicly declare as a cutoff may not match the real one in their weights—especially models that blend in newer data via RLHF or continual fine‑tuning. OracleProto relies on these declared cutoffs; if that assumption breaks, contamination leaks in. The author acknowledges this and uses content‑level leakage detection as a partial safeguard, but it’s not a cure‑all.

Also, with only 80 questions, the sample size is small for reliably separating top performers. A 10.1‑point accuracy spread with that sample size still has a wide confidence interval. Future iterations with hundreds or thousands of items—and finer domain splits (economy, politics, tech, sports)—would be welcome.

And finally, a more philosophical question: Should predictive ability even be a core metric for LLMs? The field already has MMLU, GSM8K, HumanEval, GPQA, and many more. Adding a prediction benchmark—does it make evaluation more comprehensive, or just intensify leaderboard fatigue? Hard to say yet. But at the very least, OracleProto shines light on a blind spot that no existing benchmark covers—and that contribution is real.

Closing Thoughts

In the past two years, the LLM race has focused on reasoning, coding, and long‑context processing—but “making justified judgments about uncertain futures” has remained vague. One reason is simply that no one knew how to evaluate it properly. OracleProto isn’t the final answer, but it clarifies the problem: prediction evaluation is about temporal discipline, and temporal discipline must be enforced by engineering, not by prompting.

Dragging something once seen as mysticism into a reproducible scientific paradigm—that alone is worth a star.

References

• OracleProto Open‑Source Post (linux.do) – The author’s community post describing the project, design philosophy, experimental results, and critiques of existing evaluation methods