ByteDance · Training crawler

Bytespider

Bytespider is ByteDance's web crawler, used to gather data for its AI products such as the Doubao assistant. It has a contested reputation: multiple reports through 2024 to 2025 described Bytespider crawling aggressively and disregarding robots.txt directives. Because its compliance has been inconsistent, treat a robots.txt block as a first step and confirm enforcement with server logs or a WAF rule if you need to guarantee it.

Last updated August 2026

What does Bytespider do?

User-agent token: Bytespider
Operator: ByteDance
Feeds: ByteDance / Doubao AI training
robots.txt: Unverified

Historically reported to crawl aggressively and disregard robots.txt; behavior has been inconsistent, so verify with server logs.

How do you control Bytespider in robots.txt?

Edit the robots.txt file at the root of your domain (for example https://example.com/robots.txt), add one of the groups below, then save and re-deploy. Remember: a named User-agent: Bytespider group overrides your global User-agent: * rules, so repeat any private Disallow paths inside it.

Allow Bytespider (recommended for AI visibility)

# Welcome Bytespider, but keep private areas blocked.
# A named user-agent group overrides "User-agent: *", so repeat
# your own private Disallow rules inside this group.
User-agent: Bytespider
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/

# Welcome Bytespider, but keep private areas blocked.
# A named user-agent group overrides "User-agent: *", so repeat
# your own private Disallow rules inside this group.
User-agent: Bytespider
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/

Block Bytespider

# Block Bytespider from the entire site.
User-agent: Bytespider
Disallow: /

# Block Bytespider from the entire site.
User-agent: Bytespider
Disallow: /

FAQ

Does Bytespider obey robots.txt?

It is unverified. ByteDance has indicated Bytespider should respect robots.txt, but independent reports have documented it ignoring those rules, so don't rely on robots.txt alone. Verify in your logs.

How do I reliably block Bytespider?

Add a robots.txt disallow as the baseline, then enforce it at the server or WAF level (by user agent or IP) if you need a hard guarantee, since its robots.txt compliance has been unreliable.

Is your site visible to AI crawlers?

Run a free technical access audit to inspect published crawler controls and raw HTML delivery. It does not measure whether an engine indexed or cited the site.

Run a free audit

Part of the Cite Hustle AI crawler directory. For the full framework on AI search visibility, read the GEO methodology.