ChatGPT Browse with Bing - What Websites Can ChatGPT Browse?

OpenAI has re-launched ChatGPT Browse with Bing. This study looks at what websites can ChatGPT browse and which ones it is unable to browse. 

Not just what websites are blocking Browse with Bing but exploring what websites can you actually have ChatGPT browse and provide useful information from. 

Summary of Findings:

1. ChatGPT Can Not Browse 47% of the Top 100 Websites
2. There appear to be 3 reasons ChatGPT can not browse a website:
3. Being blocked by the sites Robots.txt (24 of the 47)
4. Internal limitations put on it by OpenAI
5. Technical Difficulty
6. We found 1 instance of ChatGPT not following the Robots.txt Blocking

Method:

On Sep 28, 2023 we looked at the top 100 websites and completed 2 slightly different checks to see if ChatGPT could read the contents of the webpage.

If there was any discrepancy between the first 2 checks we completed a 3rd check. 

Results can be seen here

Key Findings

Finding 1: ChatGPT Can Not Browse 47% of the Top 100 Websites

The most popular websites that ChatGPT CAN NOT browse include:

1. Facebook.com
2. Google.com
3. Instagram.com
4. Pinterest.com
5. Amazon.com
6. Linkedin.com
7. Twitter.com

The most popular websites that ChatGPT CAN browse include:

1. YouTube
2. Wikipedia
3. TikTok
4. Reddit

Finding 2: There are 3 reasons ChatGPT can not access a website. 

Reason 1 - Website Blocking ChatGPT Browse with Bing Using Robots.txt

On Aug 7 OpenAI provided documentation on how to block GPTBot. 

Since then 26% of the top 1000 websites have blocked GPTBot among other AI bots.

According to ChatGPT it is adhering to Robots.txt instructions for Browse with Bing…

Reason 2 - Browsing Restriction Placed on ChatGPT by OpenAI

OpenAI has seemingly placed browsing restrictions on a large number of sites. 

ChatGPT communicates several different reasons for the self-imposed restriction. However, caution should be used whenever asking ChatGPT about itself.

With an increasing list of ChatGPT lawsuits, there could be legal considerations that impact what sites OpenAI has decided to let ChatGPT browse.   

Reason 3 - Temporary Error Browsing Site

There seemed to be some sites that were inconsistently unable to be browsed by ChatGPT.  

This could very likely be the result of Browse with Bing being a new feature and kinks still needing to be worked out. 

Finding 3: ChatGPT Did Not Follow Robots.txt for 1 Website

Tumblr.com blocked GPTBot early on, Aug 9, 2023 in its Robots.txt file:

However, ChatGPT Browse with Bing was still able to browse certain (non-paywalled) pages of the site.

How to Activate ChatGPT Browse with Bing

You need to be a ChatGPT Plus subscriber and can activate it by going to…

• Settings & Beta
• Beta Features
• Activate “Browse with Bing”

Example Prompt to Check a Site:

“visit each of these websites and record the title of the site, if you are unable to visit the website list the reason why. Go to each website and report the results in a table. Complete all scans before responding:”

Example Result: