Closed (fixed)
Project:
Microdata Import
Version:
7.x-1.x-dev
Component:
Code
Priority:
Normal
Category:
Task
Assigned:
Unassigned
Reporter:
Created:
22 Dec 2011 at 13:52 UTC
Updated:
5 Jan 2012 at 16:50 UTC
Jump to comment: Most recent
The module effectively acts as a crawler when it retrieves microdata from pages. Crawlers need to be polite and only request pages from a domain with the specified crawl delay.
Keep a list of the domains as they are accessed. Get their crawl delay on the first access and maintain a 'last_access' property which allows us to check whether that number of seconds have gone by. Wait until they have.
Comments
Comment #1
Anonymous (not verified) commentedFixed with http://drupalcode.org/project/microdata_import.git/commit/54d66cb