Brave Sells Its Search Index To AI, And Your Website Is Probably In It
Brave's index is being sold to AI companies, so most sites' content may already reach AI answers even if those sites never licensed it.
On this page
Blocking AI crawlers by name no longer covers the search indexes that feed AI answers
The New York Times has sued Brave, maker of the Brave browser and Brave Search. It alleges Brave copied more than 30 billion webpages without identifying its crawler or giving site owners a way to block it, then sold that content to AI companies. Brave hadn't responded when Search Engine Journal ran its analysis this month, so these are allegations. Brave sells access to its index through the Brave Search API. Its own pages name Cohere, Mistral, Perplexity and You.com as users. The Times complaint names Anthropic, Perplexity, Mistral, xAI and Cohere as buyers, and none of them is a defendant.
Can I block Brave's crawler without blocking Google?
No. Brave says its crawler carries no distinct user agent and follows the rules set for Googlebot, so allowing Google while excluding Brave isn't possible through robots.txt. You get both or neither.
Brave's help page says its crawler "does not advertise a differentiated user agent". Its stated reason is that many sites admit Google and turn everyone else away. The only exit it offers is the noindex tag, which also removes the page from Google. The Search Engine Journal author found his own site in Brave's results, with a stored copy of a homepage description he had changed hours earlier. His robots.txt names many AI bots, yet it had no way to mention Brave. If your AI-bot policy lives in robots.txt, there is now a supplier you can't address in it.
Which AI answer engines actually draw on Brave's index?
The Times names Anthropic, Perplexity, Mistral, xAI and Cohere as customers. Brave's own site lists Cohere, Mistral, Perplexity and You.com. Evidence for Claude is older, and current use is unconfirmed.
The Claude evidence is a 2023 Brave post about Anthropic. In March 2025 Anthropic added Brave Search to its subprocessor list, as TechCrunch reported. The same week, developer Simon Willison found that Claude's cited results matched a Brave search exactly. The Search Engine Journal author says he couldn't confirm that Claude still uses Brave. Treat the list as "possible", not "confirmed", for anyone but the vendors that Brave itself names.
Does being in the index mean I'll be cited?
Not necessarily. Nothing in the coverage shows how often indexed pages surface in AI answers. The 30 billion figure appears in both the Times complaint and on Brave's own AI page.
The figure is a count of pages, defined and measured by the seller. It doesn't say how many distinct sites those pages cover, how many are duplicates, or how fresh they are. It also says nothing about how a downstream assistant ranks results from the index. Presence in an index is the entry condition for being retrieved, not evidence of citation. What the story does change is the attribution problem. If an assistant answers from a purchased index, your server logs may never show that assistant's own crawler. Counting AI visibility by named bot will undercount.
What should I ask my AI search vendor?
Ask which search index supplies its live retrieval, whether that index honours noindex and robots.txt, and how a publisher can opt out of that supplier without vanishing from Google.
Vendors that say they respect publisher controls should be able to name their upstream index. If they can't, their opt-out story stops at their own crawler. Ask the same of any content-provenance or detection tool that claims to track where your text is reused. Its coverage depends on which indexes it can see.
Frequently asked questions
Has Brave answered the lawsuit?
Not as of the Search Engine Journal piece. Everything above about copying and selling is the Times's claim.
Is Anthropic being sued?
No. The complaint names it as a customer of Brave. Only Brave is the defendant.
Will noindex get me out of Brave's index?
Brave's help page points to it as the route out. The cost is that Google drops the page as well.
Sources: Search Engine Journal, which in turn cites the New York Times complaint, Brave's help and About pages, TechCrunch and Simon Willison.
The original story
Brave sells its search index to Cohere, Mistral, and Perplexity. A 30-second site: search shows whether your pages are part of it. The post Brave Sells Its Search Index To AI, And Your Website Is Probably In It appeared first on Search Engine Journal.