Problem/Motivation

The module effectively acts as a crawler when it retrieves microdata from pages. Crawlers need to be polite and only request pages from a domain with the specified crawl delay.

Proposed resolution

Keep a list of the domains as they are accessed. Get their crawl delay on the first access and maintain a 'last_access' property which allows us to check whether that number of seconds have gone by. Wait until they have.

Comments

Anonymous’s picture

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.