My name is Natalia, I'm a software developer at Yandex (https://www.yandex.ru). My team is building a large database, which contains objects from real world, their attributes and different types of relationships between them.

 

Wikipedia is one of our main data sources. Until recently we've been downloading data from it in a half-manual mode, but now we want to make it automatically with our bots. It works fine for all what we need except for API pages. As robots.txt states "Friendly, low-speed bots are welcome viewing article pages, but not dynamically-generated pages please."

 

Our bot uses crawl-delay so it won't bother you more often than you allow. Moreover, we are planning to make no more than 20-30 requests per 5 minutes.

 

Is there some way to add an exception to robots.txt for our bot?

 

Thank you for your time.

 

Best regards,

Ostapuk Natalia