Reddit Blocks Search Engine Scrapers, Including Bing

In a bold move, Reddit has updated its robot.txt file to block search engines, except for Google, from crawling its site. This means that Reddit results will no longer be displayed on other search engines, including Microsoft's Bing. The move is seen as a bid by Reddit to maintain control over its valuable data and maximize profits. Reddit signed a $60 million per year data deal with Google back in February, which has seen Google referring a heap more traffic to its pages. It seems that this deal has now empowered Reddit to set a precedent on data access.

According to Reddit, this move is not specifically linked to the Google deal. Reddit has been in discussions with multiple search engines, but has been unable to reach agreements with all of them. Some search engines are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI training. Both Reddit and X (formerly Twitter) have upped the price of their API access, in order to ensure that AI projects are not profiting off of their insights. This move is aligned with Reddit's goal of implementing more controls over its data, in order to maximize its profits.

As a publicly listed entity, Reddit is looking to enhance value for its shareholders, and building its business through various means is key to its long-term viability. Reddit's data is highly valuable, as its communities cover a range of niche topics, providing human insight and answers to common web queries. This data can help to improve AI chatbots and systems, which is why Google has opted to pay Reddit for access. It seems that Reddit is now looking for similar deals with other search engines, and if they don't provide it, it's cutting them off.

This move will likely hurt Reddit traffic to some degree, by reducing referral links, but Reddit has obviously decided that such an impact is worth the risk, in order to place a higher value on its data. It'll be interesting to see if other platforms follow suit, and whether Google, and others, are forced to make data deals to maintain scraper access. The company with the most valuable data will win out in the AI race, and Reddit definitely has some of the best quality data inputs available.

As the importance of data control and access continues to grow, it's essential for businesses to maintain a strong online presence and maximize their social media reach. At The TopTierSMM, we offer a range of SMM services, including likes, followers, and comments, to help businesses boost their online visibility and stay ahead of the competition. Shop now and take your social media to the next level!