Guides9 min read

LLM Training Data: Where Labs Get Text and Documents

Troveo Team

Troveo

If you're sourcing text for a language model, you've already used the public web. So has every other lab. The text that's left has an owner you have to ask, and most of it sits in publishers' archives and on companies' shared drives. Read on to find out the four kinds of LLM training data, why the public web isn't enough, what labs license now, what makes a document dataset useful, what to check before you license one, and where to get it.

Article banner reading LLM Training Data, on text and document data for language models

The four kinds of LLM training data

Kind of dataWhat it teaches a modelWhere it falls short
The public web and open datasetsGeneral language, facts and common knowledgeEvery lab has it, the quality is uneven, and the rights are being argued in court
Licensed published content (news, books, journals)Edited, accurate writing and specialist knowledgeIt's finished work. It shows the result and leaves out how the work got done
Expert-written dataHow a specialist reasons through a defined taskBilled by the hour, and limited by how many experts you can hire
Private business documentsHow real work gets done inside a company, from the first draft to the approved versionHas to be licensed from the company that owns it, with names and personal details masked
The four kinds of LLM training data, what each teaches a model, and where each falls short.

Most teams start in the first row. Common Crawl collects pages from the public web, Wikipedia covers reference knowledge, and arXiv holds research papers.

Why the public web isn't enough

Every lab has it. A model trained on the same pages as its competitors learns the same things, and data that every lab already holds moves no benchmarks.

Courts are asking where the copies came from. In Bartz v. Anthropic the judge ruled that training on books was fair use, and Anthropic still paid 1.5 billion dollars to settle because much of its library came from pirate sites. In July 2026, Hachette, Elsevier and Cengage sued Google over books they say it used to train Gemini. Our guide to AI training data lawsuits covers where each case stands. This isn't legal advice.

The work itself was never published. Labs are training models to do work (draft a proposal, write a runbook, review a spec). The public web has templates and advice about proposals. It doesn't have a company's 400 real ones, the ones that won and the ones that lost, with the comments in the margins.

What labs license now

Labs now pay for five kinds of text.

  • News and magazine archives. OpenAI licensed News Corp's archives and current content, and Amazon licensed news content from The New York Times.
  • Books. HarperCollins licensed select nonfiction backlist titles for model training in 2024, with each author choosing whether to take part. Bloomberg reported that the buyer was Microsoft.
  • Research papers and journals. Taylor & Francis and Wiley have both licensed academic content to AI developers.
  • Forum posts. Reddit licensed its posts and comments to Google and to OpenAI.
  • Business documents. Companies license their procedures, proposals, specs, reports and slide decks. Buyers ask for presentations as native Google Slides and PowerPoint files that show real work.

Frontier labs like OpenAI, Anthropic and Google DeepMind train on text at the largest scale, and labs like Mistral, Cohere and WRITER build language models for business use. Both groups run into the same limits on public text.

Our guide to AI content licensing deals covers how these agreements are structured. Code is its own category, and our guide to code datasets for AI covers it.

What makes a document dataset useful

A pile of finished PDFs is worth little to a training team. Six things raise the value.

  • Real work. Procedures, runbooks, proposals, specs and postmortems written by people who do the job. Templates and marketing copy add nothing a model can't already find.
  • Versions and comments. The draft, the revisions, the comment thread and the version that got sent show how work moved from an idea to a result.
  • The folder structure and the dates. They keep each document in order and in context.
  • Native files. Google Docs, Word, Slides and PowerPoint files keep the structure that a flattened PDF loses.
  • Links to other systems. A proposal that connects to the CRM record and the email thread shows whether the deal closed.
  • Years of history in a specialized field. Twelve years of engineering specs and postmortems is a different asset from three years of generic memos.

The documents that carry the most show how a company does a task. Procedures, runbooks and playbooks come first. Proposals and statements of work show how a company scopes and prices a job. Specs, project plans and postmortems show how work was planned, what went wrong and what changed. Meeting notes and decision memos show why a decision was made and who signed off.

Labs count text in tokens, so ask for token counts by file type along with the number of files. Our guide to enterprise operational data covers why buyers value company records over generic text, and our guide to training data for AI agents covers why connected records matter.

Pretraining, fine-tuning, evaluation and retrieval are four different uses

Labs do four things with licensed text, and each one is a separate right in the license.

  1. Pretraining. The model learns general language and knowledge from a large volume of text.
  2. Fine-tuning. You adapt a trained model to a field or a task with a smaller, more specific set.
  3. Evaluation. You hold the text back and test the model on material it hasn't seen. Private documents are useful here because they were never public.
  4. Retrieval at answer time. The model looks the text up when it answers a question, without having trained on it.

Decide which of the four you need before you negotiate, and get each one named in the agreement. A license can expire while a model trained under it doesn't, so ask what happens to the files when the term ends.

What to check before you license text and documents

Ask for a sample first. Deals usually start with a curated slice of the dataset in front of your research team. Then run seven checks.

  1. Does the seller own it? Work a services firm produced for a client often belongs to the client, and it stays out.
  2. Does the license name AI training? Older content licenses often cover display or distribution and say nothing about training a model.
  3. Where did each file come from? Ask for a record per file. "Publicly available" isn't an answer. Our guide to AI data provenance covers how to verify it.
  4. Have names and personal details been replaced with pseudonyms to a documented standard?
  5. Is anything in it under a confidentiality agreement, or from an HR, payroll or medical folder? That stays out.
  6. Are duplicates and near-copies out, and is there a manifest of what's included?
  7. Is the license exclusive, and which uses does it name? Our guide to exclusive vs non-exclusive data licenses covers the tradeoff.

Our guide to rights-cleared training data covers what clearing a dataset takes in every category.

Where to get LLM training data

You have four routes, and most labs use all of them.

  1. Start with open datasets. They're free and large, and every other lab has them.
  2. License published content from its publisher. News groups, book publishers and journal publishers negotiate directly, and each deal covers one catalog.
  3. Pay experts to write it. Specialists write and grade text to your brief. You control the format, and every hour is billed. Our guide to AI data collection companies covers the vendors that work to order.
  4. License business documents from the companies that wrote them. Operating companies hold years of procedures, proposals, specs and decks, and they license them selectively. Our guide for companies on what their docs are worth to AI labs covers the owner's side.

If your model reads images, audio or video alongside text, our guide to multimodal datasets covers that market.

Where Troveo fits

Troveo is a data licensing marketplace, and we've paid rights holders more than 20 million dollars. Documents and files are one of the nine categories in our business data program, alongside chat, email, CRM, support, project tools, finance systems, code and internal databases. Companies license their documents and presentations through us after we replace names and personal details with pseudonyms to a documented standard and record what's included in a manifest. You see an anonymized snapshot of what a company holds before anything is exported, and you sign one agreement with us. Tell us what your model needs to read and we'll show you what we hold.

Frequently asked questions

What is LLM training data?
The text a large language model learns from. It comes in four kinds: the public web and open datasets, licensed published content like news, books and journals, expert-written data, and private business documents licensed from the companies that wrote them.
Where does LLM training data come from?
Mostly from the public web, collected by crawling. Labs add licensed archives from publishers, text written by paid experts, and documents licensed from companies.
What are examples of open LLM datasets?
Common Crawl collects pages from the public web, Wikipedia covers reference knowledge, and arXiv holds research papers. They're free and widely used, so every lab trains on the same ones.
Why do AI labs license text when the web is free?
Because every lab already has the web, and courts keep asking where each copy came from. Licensed text comes with permission from its owner, and private documents show work the web never published.
What kinds of text do AI labs license?
News and magazine archives, books, research papers and journals, forum posts, and business documents like procedures, proposals, specs, reports and slide decks.
What makes a document dataset valuable for training?
Real work in a specialized field, with the versions, comments, folder structure and dates intact, in native file formats, and linked to the records that show how the work turned out.
What rights do you need to train a commercial model on licensed text?
A license from the owner that names each use you need (pretraining, fine-tuning, evaluation, retrieval at answer time). This isn't legal advice, and your counsel should review the agreement.
Where can AI labs license business documents for training?
From the companies that wrote them, either directly or through a marketplace that has signed them. A marketplace lets you review an anonymized snapshot first and license from many companies under one agreement.

Related articles

Back to Resources