Meta · Training crawler

Meta-ExternalAgent

Meta-ExternalAgent is Meta's crawler for collecting content that may be used to train its AI models, including the Llama family, and to power Meta AI products. Meta documents that it respects robots.txt, and it is controlled separately from Meta-ExternalFetcher, the agent used for user-initiated link previews and fetches. Blocking Meta-ExternalAgent excludes your content from Meta's AI training.

Last updated July 2026

User-agent token: Meta-ExternalAgent
Operator: Meta
Feeds: Meta AI / Llama training
robots.txt: Unverified

How to control Meta-ExternalAgent in robots.txt

Edit the robots.txt file at the root of your domain (for example https://example.com/robots.txt), add one of the groups below, then save and re-deploy. Remember: a named User-agent: Meta-ExternalAgent group overrides your global User-agent: * rules, so repeat any private Disallow paths inside it.

Allow Meta-ExternalAgent (recommended for AI visibility)

# Welcome Meta-ExternalAgent, but keep private areas blocked.
# A named user-agent group overrides "User-agent: *", so repeat
# your own private Disallow rules inside this group.
User-agent: Meta-ExternalAgent
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/

# Welcome Meta-ExternalAgent, but keep private areas blocked.
# A named user-agent group overrides "User-agent: *", so repeat
# your own private Disallow rules inside this group.
User-agent: Meta-ExternalAgent
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/

Block Meta-ExternalAgent

# Block Meta-ExternalAgent from the entire site.
User-agent: Meta-ExternalAgent
Disallow: /

# Block Meta-ExternalAgent from the entire site.
User-agent: Meta-ExternalAgent
Disallow: /

FAQ

Does Meta-ExternalAgent respect robots.txt?

Yes. Meta states Meta-ExternalAgent honors robots.txt, and you can control it independently from its user-initiated fetcher, Meta-ExternalFetcher.

What is the difference between Meta-ExternalAgent and Meta-ExternalFetcher?

Meta-ExternalAgent crawls for AI training and product data. Meta-ExternalFetcher retrieves specific links a user or product requests, such as for link previews.

Is your site visible to AI crawlers?

Run a free AI-visibility audit to see which AI crawlers can reach your content and how often you get cited by ChatGPT, Perplexity, Claude, and Google AI Overviews.

Run a free audit

Part of the Cite Hustle AI crawler directory. For the full framework on AI search visibility, read the GEO methodology.