The VK company announced a ban on the robot of the famous company OpenAI, which owns the popular chatbot ChatGPT, from collecting any data from the Zen service. Journalists from the Kommersant publication said that the corresponding ban is currently set in a file belonging to the Zen service robots.txtwhich is used by various sites and platforms to regulate the rules for automatic collection of data from resources.
The press service of the VK company emphasized that the work of the GPTBot robot was indeed blocked in order to reduce the level of load that was placed on the Zen servers.
The VK company’s commentary on this matter states that the recommendation system of the Zen service is currently one of the largest in the Russian segment of the Internet. The services are heavily loaded, operate continuously, and process over 150,000 requests every second. In this regard, it was decided not to add GPTBot from OpenAI to the corresponding file in order to competently use the technical resource without creating additional load. In the Zen service, a huge number of publications are created on an ongoing basis, both in text and video format, so all resources are aimed at providing high-quality user experience, VK clarified.
The robots.txt file, as experts note, is purely advisory in nature. Therefore, from a technical point of view, nothing can prohibit the robots of any companies from ignoring the directives that are spelled out in it. For example, the Internet Archive project, which works in the field of creating copies of various web resources, announced in 2019 that it would completely ignore all installations from robots.txtbut in 2022 this decision was reversed.
The American company OpenAI uses the GPTBot robot to collect information, which is subsequently used to develop neural network products. Interestingly, Russian users currently do not have official access to these products.
Roughly similar robots are also used by the Russian company Yandex and the American corporation Google, with the help of which search engines maintain up-to-date data on various resources.
