Microsoft MAI Model Training Data Controversy: “Clean” Promise Meets Common Crawl

Just a few months after the launch of Microsoft’s MAI series models, the company has been challenged over the source of their training data.

Tech media outlet The Decoder reported yesterday (June 5) that the composition of training data disclosed in Microsoft’s MAI technical paper contradicts its previous claim that the models were “completely based on enterprise-grade, clean, and commercially licensed data.” The paper shows that MAI model training data includes not only licensed corpora but also datasets like Common Crawl, which are scraped from the public internet.

This is no small matter. When Microsoft launched the MAI-1 preview last August, AI division CEO Mustafa Suleyman stressed that it was “a foundation model trained entirely from scratch,” specifically highlighting that no distilled data from third-party models was used. This gave the impression that Microsoft was playing it by the book, with all training data legitimately licensed — unlike models built on large-scale crawling of the public web.

But the technical paper tells a different story.

The Real Data Sources in the Technical Paper

According to Microsoft’s disclosed technical documentation, the MAI model adopts a hybrid approach of “publicly available data” and “licensed human-generated data.” The former includes internet-scraped datasets like Common Crawl, while the latter comprises truly commercially licensed content.

What is Common Crawl? It’s a public web-scraping project maintained by a nonprofit organization, which has regularly crawled internet content since 2008 and made the datasets publicly available. Nearly all mainstream large models — GPT-3, LLaMA, Stable Diffusion, etc. — have used it. The problem is that whether the content in Common Crawl can be legally used has always been in dispute — it’s indeed “publicly available,” but “public” does not mean “licensed.”

Microsoft’s handling of this point in the paper is rather subtle. They claim to have used their own crawlers to collect web data while adhering to the Robots Exclusion Protocol (robots.txt) and relevant HTML meta tags. In other words: as long as a website doesn’t actively block crawling, they treat it as fair game.

The Decoder’s evaluation cuts to the chase: this logic is akin to “an unlocked door implies consent to enter.” Responsibility is shifted to content providers — if you want to protect your content, you must proactively configure robots.txt or meta tags. No configuration? Then they assume you agree.

Not the First Time, Won’t Be the Last

The issue Microsoft faces is essentially the same dilemma plaguing the entire AI industry: finding a balance between “needing massive amounts of data” and “copyright compliance.”

In March, Encyclopædia Britannica sued OpenAI for using nearly 100,000 encyclopedia articles to train ChatGPT without authorization, alleging that generated content was “almost verbatim” copies. OpenAI responded with its usual line: “based on publicly available data, compliant with fair use.” Publishers, however, aren’t buying it — if AI summaries divert users from visiting the original site, that’s not “transformative use,” but a commercial threat.

Other similar cases include writers suing Meta for training the LLaMA model on pirated ebooks, and multiple news organizations collectively suing AI companies. The defense logic from AI firms is largely uniform: fair use, transformative creation, driving innovation. But legal precedents haven’t yet established consensus, and most cases remain in litigation.

Interestingly, some media outlets have opted for cooperation rather than confrontation. In March, News Corp reached a licensing agreement with Meta worth up to $50 million annually, and British publisher Reach signed a usage-based payment deal with Amazon for its Nova AI model. This shows that a licensing model is viable — but costly and slower to negotiate, compared to simply crawling the public web.

Microsoft’s Dilemma: Self-Developed Models and OpenAI Relations

Microsoft’s launch of the MAI series models is, in itself, an effort to find a new balance in its relationship with OpenAI.

Over the past few years, Microsoft has invested more than $13 billion in OpenAI and provided critical computing power through Azure. But OpenAI has recently started relying more on CoreWeave, Google, and Oracle cloud services. Last year, Microsoft even listed OpenAI as a competitor in its annual report, alongside Amazon, Apple, Google, and Meta.

In this context, the significance of the MAI series is not only about having “self-developed capabilities” in the technical sense but also strategically about “reducing single-source dependence.” Suleyman has stated in interviews that Microsoft will in future use multiple model sources — OpenAI, open-source models, and self-developed models — with an “orchestrator” deciding which model to call and when.

However, MAI-1’s performance is not yet outstanding. On the LMArena leaderboard, it ranks 13th, behind models from Anthropic, DeepSeek, Google, Mistral, OpenAI, and xAI. Microsoft emphasizes that this is only a preview version, and performance will improve with user feedback — but catching up with top-tier models will require more time and investment.

Meanwhile, Microsoft has also launched the speech generation model MAI-Voice-1, claiming it can generate one minute of high-fidelity audio on a single GPU in under a second. It is already integrated into Copilot Daily and Podcasts. Compared to MAI-1’s awkward ranking, the speech model’s rapid deployment and efficiency demonstrate Microsoft’s execution strength in certain verticals.

How “Clean” Is Clean Data?

Back to the training data itself — Microsoft’s problem is not about using Common Crawl per se, but about ambiguous and inconsistent messaging.

Had they from the start said “we use a hybrid of licensed corpora and curated public web data,” it wouldn’t have been particularly controversial — nearly all models use such a mix. But Microsoft chose to emphasize “enterprise-grade, clean, and commercially licensed” to give the impression of steering clear of any grey areas. Once the technical paper came out, that impression collapsed.

The deeper issue is that the concept of “clean data” lacks a unified standard. Does it mean no pirated content? No harmful information? Or that all sources are legally licensed? Each company defines it differently, and regulators have yet to clearly specify.

Microsoft notes in its paper that it uses its own web crawlers and complies with robots.txt. Technically, this is a responsible approach, but legally, it remains contentious. Robots.txt is merely a “gentleman’s agreement,” not a legal instrument. Adhering to it can reduce moral risk but cannot eliminate copyright lawsuits.

In practice, the MAI model was trained on 15,000 H100 GPUs — a medium-to-small scale by today’s standards. Suleyman stresses they’ve shown “world-class performance can be trained on relatively small clusters,” which indicates that their data selection and training efficiency are quite good. But it also indirectly confirms that diverse data sources (including public web data) directly impact model performance.

The Industry Needs a New Consensus

Ultimately, AI training data copyright issues need to be resolved through legislation and industry consensus, not through each company defining its own rules.

The EU’s AI Act has already required transparency in data usage, mandating AI companies to disclose training data sources and comply with copyright law. In the U.S., several states are advancing similar laws, though no unified federal rules exist yet. Until then, the boundaries of “fair use” will be tested repeatedly in lawsuits.

For developers, this means that when choosing models, beyond performance, cost, and API compatibility, they must consider the provider’s data compliance. While using the model may not directly incur copyright liability, if a model is taken down or restricted due to copyright issues, dependent applications will also be affected.

Microsoft’s controversy is essentially part of the growing pains the whole industry is experiencing: transitioning from “whatever works” to “compliant and sustainable.” The MAI model’s technical capability is fine, but its data strategy’s transparency needs improvement. For a company that’s the world’s most valuable and claims to “responsibly advance AI,” the bar should be set higher.

---

References

• Microsoft MAI series AI model training data exposed, discrepancy in “commercially licensed only” claim - IT Home — IT Home’s coverage of the Microsoft MAI model training data controversy
• Microsoft launches two MAI-1 series self-developed models, work on next-generation MAI-2 has begun - Tencent News — Background and technical details of Microsoft’s MAI series model launch