robots.txt<\/code>, you can block these AI crawlers from accessing your website or blog.<\/p>\n\n\n\n![\"\"](\"https:\/\/yolohive.com\/wp-content\/uploads\/2024\/12\/ai-bots-1024x683.jpg\")
<\/figure>\n\n\n\nWhat is a robots.txt file?<\/h2>\n\n\n\n
The robots.txt<\/code> file is a tool that allows website owners to manage how search engine crawlers interact with their content. It gives you the power to disallow specific bots from crawling your site, ensuring greater control over your content.<\/p>\n\n\n\nThe syntax below shows how to block a single bot using a user-agent:<\/p>\n\n\n\n
user-agent: {BOT-NAME-HERE}\ndisallow: \/<\/code><\/pre><\/div>\n\n\n\nBelow shows how to allow specific bots to crawl your website using a user-agent:<\/p>\n\n\n\n
User-agent: {BOT-NAME-HERE}\nAllow: \/<\/code><\/pre><\/div>\n\n\n\nWhere to place your robots.txt file?<\/h2>\n\n\n\n
Upload the file to your website\u2019s root folder: <\/p>\n\n\n\n
https:\/\/example.com\/robots.txt\nhttps:\/\/blog.example.com\/robots.txt<\/code><\/pre><\/div>\n\n\n\nLearn More About robots.txt<\/code><\/h2>\n\n\n\nIf you\u2019re ready to take control of your website\u2019s accessibility, dive deeper into the details of robots.txt<\/code> with these helpful resources:<\/p>\n\n\n\n\n- Introduction to robots.txt<\/a><\/strong> by Google: Understand the basics of how
robots.txt<\/code> works and how to configure it effectively for your site.<\/li>\n\n\n\n- What is robots.txt? | How a robots.txt file works<\/a><\/strong> from Cloudflare: A comprehensive guide to the purpose and function of
robots.txt<\/code> in managing web crawler access.<\/li>\n<\/ul>\n\n\n\nHow to block AI crawlers bots using the robots.txt file<\/h2>\n\n\n\n
The syntax:<\/p>\n\n\n\n
user-agent: {AI-Crawlers-Bot-Name-Here}\ndisallow: \/<\/code><\/pre><\/div>\n\n\n\nBlocking Google AI (Bard and Vertex AI generative APIs)<\/h2>\n\n\n\n
Add the following two lines to your robots.txt:<\/p>\n\n\n\n
User-agent: Google-Extended\nDisallow: \/<\/code><\/pre><\/div>\n\n\n\nAdditional Information on User Agents and AI Bots<\/h3>\n\n\n\n
For more information about managing crawlers, you can review the list of user agents<\/a><\/strong> used by Google crawlers and fetchers. This can help you identify legitimate Google bots accessing your site.<\/p>\n\n\n\nHowever, it\u2019s important to note that:<\/p>\n\n\n\n
\n- Google does not provide CIDR, IP ranges, or ASN details<\/strong> for its AI bots, making it challenging to block them directly via your web server firewall.<\/li>\n\n\n\n
- As a result, using a
robots.txt<\/code> file remains one of the most effective methods to guide compliant crawlers and restrict access to your content.<\/li>\n<\/ul>\n\n\n\nFor advanced control, monitor your server logs for unusual activity and configure additional security measures, such as rate limiting or IP blocking, to complement your robots.txt<\/code> directives.<\/p>\n\n\n\nBlocking OpenAI using the robots.txt file<\/h2>\n\n\n\n
Add the following four lines to your robots.txt:<\/p>\n\n\n\n
User-agent: GPTBot\nDisallow: \/\nUser-agent: ChatGPT-User\nDisallow: \/<\/code><\/pre><\/div>\n\n\n\nOpenAI utilizes two distinct user agents for its operations: one for web crawling and another for browsing, each associated with unique CIDR and IP address ranges. Configuring firewall rules to block these requires an advanced understanding of networking concepts and root-level access to a Linux server.<\/p>\n\n\n\n
If you\u2019re not familiar with these technical aspects, such as managing CIDR ranges or configuring firewalls, it\u2019s advisable to seek the assistance of a Linux system administrator. Keep in mind that OpenAI\u2019s IP address ranges are subject to change, which can turn this process into an ongoing effort to keep up with updates\u2014a game of cat and mouse.<\/p>\n\n\n\n
1: The\u00a0ChatGPT-User is used by\u00a0plugins<\/strong>\u00a0in ChatGPT<\/h3>\n\n\n\nBelow is a list<\/a> of user agents used by OpenAI\u2019s crawlers and fetchers, along with their associated CIDR or IP address ranges. To block OpenAI\u2019s plugin AI bot, you can configure your web server firewall to restrict access from specific IP ranges, such as 23.98.142.176\/28<\/code>.<\/p>\n\n\n\nHere\u2019s an example of how to block a CIDR or IP range using the ufw<\/code> command or iptables<\/code> on your server:<\/p>\n\n\n\nUsing UFW:<\/strong><\/p>\n\n\n\nsudo ufw deny from 23.98.142.176\/28 <\/code><\/pre><\/div>\n\n\n\nUsing iptables:<\/strong><\/p>\n\n\n\nsudo iptables -A INPUT -s 23.98.142.176\/28 -j DROP <\/code><\/pre><\/div>\n\n\n\nThese commands prevent any traffic originating from the specified IP range from accessing your server. Make sure to review and update your firewall rules periodically to account for changes in OpenAI\u2019s IP ranges. If you\u2019re unfamiliar with configuring firewalls, consider enlisting the help of a Linux system administrator.<\/p>\n\n\n\n
2: The\u00a0GPTBot \u00a0is used by ChatGPT<\/h3>\n\n\n\n
Below is a list<\/a> of user agents used by OpenAI crawlers and fetchers, along with the associated CIDR or IP address ranges. You can block these ranges directly on your web server using either the ufw<\/code> command or iptables<\/code>.<\/p>\n\n\n\nHere\u2019s an example shell script to block those CIDR ranges:<\/p>\n\n\n\n
Shell Script to Block OpenAI CIDR Ranges<\/strong><\/p>\n\n\n\n#!\/bin\/bash\n# Purpose: Block OpenAI ChatGPT bot CIDR \n# Tested on: Debian and Ubuntu Linux\n# ------------------------------------------------------------------\nfile="\/tmp\/out.txt.$$"\nwget -q -O "$file" https:\/\/openai.com\/gptbot-ranges.txt 2>\/dev\/null\n \nwhile IFS= read -r cidr\ndo\n sudo ufw deny proto tcp from $cidr to any port 80\n sudo ufw deny proto tcp from $cidr to any port 443\ndone < "$file"\n[ -f "$file" ] && rm -f "$file"<\/code><\/pre><\/div>\n\n\n\nIf you\u2019re unfamiliar with firewall configuration, consult a Linux system administrator for assistance. Regularly update the script with new ranges to keep up with changes in OpenAI\u2019s IP list.<\/p>\n\n\n\n
Blocking commoncrawl (CCBot) using the robots.txt file<\/h2>\n\n\n\n
Add the following two lines to your robots.txt:<\/p>\n\n\n\n
User-agent: CCBot\nDisallow: \/<\/code><\/pre><\/div>\n\n\n\nCommon Crawl, a non-profit foundation, operates a bot called CCBot, which is widely used to collect data for training AI models. Blocking CCBot is also essential if you want to prevent your content from being utilized in this way. However, similar to Google, Common Crawl does not provide CIDR, IP address ranges, or autonomous system information (ASN) that can be used to block its bot through your web server firewall. This limitation makes it challenging to restrict their access at the network level.<\/p>\n\n\n\n
Blocking Perplexity AI using the robots.txt file<\/h2>\n\n\n\n
Another service that uses generative AI to rewrite your content is PerplexityBot. To block this bot, you can add the following rule to your robots.txt<\/code> file:<\/p>\n\n\n\nUser-agent: PerplexityBot \nDisallow: \/ <\/code><\/pre><\/div>\n\n\n\nAdditionally, PerplexityBot has published its IP address ranges<\/a>, which you can block using your Web Application Firewall (WAF) or web server firewall. This ensures an added layer of protection against unauthorized access to your content.<\/p>\n\n\n\nBlocking Anthropic AI (Claude)<\/h2>\n\n\n\n
Add the following lines to your robots.txt file:<\/p>\n\n\n\n