<think>Planning markdown-preserving translation</think>

Financial AI’s Annual Exam Enters the Real-World Arena: 20,000 Participants, Data at the Tens-of-Billions Scale, and Model Evaluation No Longer Focused Solely on Leaderboards

The annual examination for financial AI is evolving from “using a few questions to test a model” into “having the entire industry run business operations together.” Recently, the annual financial AI evaluation released its interim results: more than 20,000 participants, approximately 5,000 teams, and over 30 organizations took part. The associated data has reached the tens-of-billions scale and is being made available to the industry.

This is not an ordinary leaderboard update. Over the past four years, financial foundation-model evaluation has evolved from isolated question answering and knowledge tests to complex reasoning and risk identification, and then to multimodal and agent-based tasks. Today, evaluations are beginning to simulate real financial sectors—including banking, securities, funds, insurance, trusts, and futures—to assess whether models can produce explainable, traceable, and actionable results when data is incomplete, information is contradictory, and process constraints are strict.

In other words, financial AI is finally beginning to take a “real-world practical exam.”

The Evaluation Has Scaled Up; the Real Change Lies in the Boundaries of the Problems

The financial industry has no shortage of models—or of one or two impressive demos. What it lacks is a yardstick that allows different models, institutions, and solutions to be compared from the same starting line.

Previously, financial AI evaluations generally revolved around several relatively well-defined tasks: financial knowledge question answering, financial-report information extraction, text classification, sentiment analysis, and risk-event identification. These tasks are suitable for quickly comparing a model’s knowledge coverage and language capabilities, but they remain noticeably distant from real business operations.

Actual credit approval does not simply ask a model whether “this company presents a risk.” Instead, it may provide the applicant’s occupation, income, transaction history, credit history, proof of assets, and behavioral records, and require the system to identify inconsistencies among them. Analyzing a research report is not merely a matter of generating text that sounds professional; the system must also verify figures, reconstruct the statistical basis, determine the relevant time period, and explain which materials support its conclusions. Customer service, investment advisory, compliance review, and anti-money-laundering systems must operate within permission boundaries, regulatory requirements, and human-review mechanisms.

This means that the focus of model evaluation has shifted from “answering one question correctly” to “completing part of a business process.”

The significance of this annual evaluation lies in simultaneously raising the number of participants, the scale of the data, and the complexity of the tasks to a new level. The 20,000 participants and 5,000 teams show that financial AI is no longer a closed competition among a small number of leading laboratories. The participation of more than 30 organizations indicates that the evaluation data and task design are incorporating more industry-side experience, rather than having standards defined independently by a single team.

The tens-of-billions-scale dataset deserves even more attention. Its value lies not only in its size, but also in the fact that the evaluation is beginning to approach the real complexity of financial business: long texts, multiple tables, multi-turn interactions, cross-document verification, temporal information, and multimodal materials all expose models to input conditions far beyond those of traditional benchmarks.

Of course, “open” does not mean placing all raw financial data directly on the public internet. Financial data involves privacy, trade secrets, and regulatory requirements. In practice, viable approaches typically include de-identified data, synthetic data, feature-based data, controlled evaluation environments, and publicly available task definitions and evaluation tools. The larger the dataset, the higher the governance costs. Whether the evaluation can explain its data sources, de-identification methods, annotation procedures, and authorization boundaries will determine whether the industry can trust it over the long term.

Four Years of Change: From Knowledge Exams to Business Stress Tests

Looking at the evolution over the past four years, financial AI evaluation has broadly gone through four stages.

Stage One: Testing Whether Models “Understand Finance”

Early evaluations focused on financial terminology, basic concepts, policies and regulations, financial indicators, and industry knowledge. If a model could not clearly explain concepts such as the price-to-earnings ratio, capital adequacy ratio, duration, or net interest margin, it would have difficulty participating in business discussions.

This stage addressed the knowledge threshold for models, but it also had clear limitations: a model could rely on memorization and linguistic patterns to answer questions without necessarily possessing genuine financial reasoning capabilities. In particular, when questions involved complex definitions, changes over time, and cross-validation across multiple materials, simple knowledge question answering could easily conceal flaws in the model.

Stage Two: Testing Whether Models “Can Reason”

As reasoning models emerged, evaluations began incorporating financial calculations, causal judgments, investment analysis, risk transmission, and case simulations. Models were required to extract evidence from multiple passages and then form a judgment, rather than directly outputting a seemingly definitive conclusion.

This step moved financial AI from “knowledge-base question answering” toward becoming an “analytical assistant.” However, it still had a problem: many questions were static and had clearly defined answer boundaries. Even without truly understanding the business context, a model could achieve a high score through pattern matching.

Stage Three: Testing Whether Models “Can Handle Real-World Materials”

Multimodal evaluation subsequently became a focus. Contracts, bills, bank statements, business licenses, financial reports, screenshots, and scanned documents were incorporated into testing. Models had to do more than recognize text; they also had to handle low resolution, skew, glare, occlusion, and complex layouts.

These tasks are particularly important for credit, insurance claims, and operational reviews. In credit, for example, a system may need to determine whether information is consistent across identity documents, income materials, and transaction records. An anomaly in the materials may not originate from a single field, but from the relationships among multiple fields. A genuinely useful model does not merely “see an image clearly”; it can go further and explain where the conflicts are, what they mean, and whether human intervention is required.

Stage Four: Testing Whether Models “Can Enter a Closed-Loop Business Process”

The core of the latest stage is agents and process-based tasks. Models must call retrieval systems, calculators, database queries, or rule engines to complete the entire chain, from information collection and evidence verification to risk assessment and result generation.

At this point, the evaluation metrics also change. Accuracy remains important, but it is no longer the whole picture. Institutions also care about whether:

• the model cites the correct evidence;
• it proactively refuses to answer when information is insufficient;
• it can distinguish facts, inferences, and recommendations;
• it complies with permission and regulatory boundaries;
• it can handle long contexts consistently;
• it produces hallucinations, makes unauthorized calls, or leaks sensitive information;
• it supports human review and end-to-end auditing.

From this perspective, financial AI evaluation is increasingly resembling a stress test for software systems rather than a language-proficiency exam.

Why Large-Scale Participation Matters More Than a Single Champion

The market typically focuses on “who came in first.” For financial AI, however, the more important signal from this evaluation may not be the champion model, but the expansion of the participating ecosystem.

The 5,000 teams indicate that participants are no longer limited to large-model companies. Bank technology subsidiaries, fintech companies, university teams, cloud providers, and vertical-solution vendors can all use the same set of tasks to validate their technical approaches. Some will use general-purpose foundation models, some will choose finance-specific models, and others will improve system performance through retrieval-augmented generation, rule engines, multi-model routing, and agent orchestration.

This will bring about a direct change: competition in financial AI is shifting from “whose base model is larger” to “who can combine models, data, tools, and processes more effectively.”

In many financial scenarios, the optimal solution is not necessarily a single model. A more reliable system may consist of multiple modules: the model handles understanding and generation; the retrieval system provides the latest policies and business knowledge; the calculation engine performs precise arithmetic; the rule engine enforces hard compliance constraints; and human checkpoints handle high-risk decisions. Only if evaluations can cover these integrated capabilities will they more closely reflect the technical choices enterprises actually face when making purchases.

This also explains why financial institutions have increasingly emphasized integrated training and inference, private deployment, data governance, and multicloud compatibility when selecting solutions in recent years. The model itself is merely the foundation. Whether it can be integrated into existing core systems, reduce inference costs, be audited, and be continuously updated determines whether a project can move from proof of concept to production.

The Other Side of Tens-of-Billions-Scale Data: More Data Does Not Mean a More Reliable Evaluation

Large-scale data is undoubtedly a step forward, but data volume cannot simply be equated with evaluation quality.

The first issue is data contamination. If questions, answers, or similar samples remain publicly available for a long time, models may achieve high scores by memorizing training data. In the end, the evaluation measures whether the model has “seen” the content, rather than whether it “knows how to solve” the problem. Evaluations therefore need dynamic question sets, hidden test sets, and regular update mechanisms, along with detection of contamination in model training data.

The second issue is annotation consistency. Financial questions often involve differences in definitions and standards. The same indicator may have different interpretations depending on accounting standards, time periods, and institutional types. If annotation rules are not transparent, model scores will be difficult to reproduce, let alone serve as a reference for institutional procurement and regulatory oversight.

The third issue is long-tail risk. A high average score does not mean that a system is suitable for financial business. A model may perform consistently on 99% of ordinary samples but make serious misjudgments on 1% of high-risk samples. The resulting losses may far outweigh the efficiency gains in routine scenarios. Evaluations therefore need to report separately on high-risk samples, refusal rates, false-positive rates, false-negative rates, and performance in extreme scenarios.

The fourth issue is cost and latency. Financial institutions will not necessarily purchase the “smartest” model. They must also consider the cost per call, peak concurrency, response time, deployment resources, and failure-recovery capabilities. A model that is slightly more accurate but costs several times as much and adds several seconds of latency may not be suitable for customer service or real-time risk control. Conversely, a model that is somewhat less capable but stable, inexpensive, and easy to deploy privately may have greater commercial value.

Therefore, the next stage of financial AI evaluation should move from a single overall score to multidimensional reporting, covering at least six dimensions: capability, reliability, security, explainability, cost, and engineering efficiency.

Will Evaluation Become a “Qualification Threshold” for Financial AI?

Previously, financial institutions often relied on vendor demonstrations, customized testing, and internal expert reviews when procuring foundation models. Different vendors used different data, prompts, and scoring methods, making the final results difficult to compare horizontally.

If annual evaluations can continue operating and gradually establish an open, stable, and reproducible task framework, they may become a common language for the industry. Like public benchmarks in software engineering, evaluation results may not directly determine procurement decisions, but they can help institutions quickly eliminate obviously unqualified solutions and then devote resources to deployment validation and security reviews.

However, evaluation cannot replace acceptance testing in production environments. The ultimate performance of a financial model still depends on an institution’s own data quality, knowledge-base update speed, permission system, workflow design, and human-governance capabilities. A public leaderboard can answer “How does the model perform in general?” but not “How will it perform once placed in your system?”

A more realistic trend is for evaluation to continue moving downward from the model layer to the system layer:

1. Model layer: Evaluating knowledge, reasoning, long-context, and multimodal capabilities;
2. Component layer: Evaluating retrieval, tool calling, rule coordination, and structured output;
3. Process layer: Evaluating whether the model can complete end-to-end business tasks;
4. Governance layer: Evaluating permissions, auditing, refusal behavior, risk isolation, and human takeover;
5. Operations layer: Evaluating cost, latency, stability, and continual-learning capabilities.

Whoever can achieve acceptable results across all five layers will be closer to delivering genuine financial productivity.

What This Means for Model Vendors and Developers

For model vendors, financial verticalization cannot consist merely of adding the words “financial capabilities” to marketing materials. Models need sustained investment in domain-specific data, reasoning training, fact verification, citation traceability, and safety alignment. In particular, hallucinations in the financial domain cannot be addressed with a disclaimer alone. Systems must be able to constrain their output when uncertain or hand the issue over to a human.

For developers, the new evaluation framework also provides a clearer engineering direction. Instead of repeatedly adjusting prompts, they should focus more on data and system architecture: establish versioned evaluation sets and record the impact of every model change; separate factual retrieval, numerical calculation, and permission checks from the model; set hard rules and human review for high-risk tasks; and conduct regression testing with hidden sets and adversarial samples before launch.

For financial institutions, evaluation results can serve as an entry point for solution selection, but should not be the sole basis for decisions. A more valuable approach is to combine public benchmarks with the institution’s own de-identified data to create a two-tier testing framework: “general industry capabilities + institution-specific capabilities.” Only then can a model’s performance on a public leaderboard potentially translate into real benefits in production systems.

Conclusion: Financial AI Competition Is Entering a “Verifiable” Phase

Over the past four years, the narrative around financial AI has gradually shifted from “what can a model do?” to “what can a model reliably accomplish?” The opening of data at the tens-of-billions scale for this annual evaluation, along with the participation of tens of thousands of individuals and thousands of teams, shows that the industry is addressing the need for large-scale validation.

It will not solve financial AI’s hallucination, bias, compliance, and deployment problems overnight. But it has at least brought the discussion back from demos and slogans to data, tasks, metrics, and results.

What financial institutions truly need has never been a model that merely looks powerful on a leaderboard. They need a system that knows when to answer, when to refuse, when to call a tool, and when to hand a matter over to a human in complex business environments. The value of large-scale evaluation lies precisely in shortening, as much as possible, the distance between “appears usable” and “has been validated for deployment.”

Going forward, the annual financial AI examination will not only compare who achieves the highest score, but also who is more stable, less expensive, and safer—and who can truly withstand long-term operation in a production environment.