This example demonstrates how to build a Drupal-native web scraper. It imports events from a single month of Stanford University's calendar by navigating to the page for each day and creating a node for each event on the day's list. Each event is then queued for scraping its details.
Note: As of 2022-01-31, this project appears to no longer be supported. If you are interested in taking this project over, or you as the project maintainer feel this message has been posted in error, please reply to https://www.drupal.org/project/projectownership/issues/3248166.
The ShrinkTheWeb module uses the ShrinkTheWeb service functionality to cache and display website screenshots (e.g. web pages thumbnails).
Visitors can report an Issue (e.g. pothole) to a Jurisdiction (i.e. a Council). Once confirmed by the submitter, notification of an Issue is emailed to the relevant jurisdicton for resolution. Issues can include a latitude and longitude, category, description and a photo. Visitors can add Updates (comments) to an Issue, and indicate that the Issue is fixed. To minimise usage hurdles, user accounts are not required. To minimise spam, confirmation by email is required for new Issues and Updates.
The site uses multiple web services:
Google Maps: for map display, with markers representing Issues. http://maps.google.com/maps/geo: address geocoding to latitude and longitude. http://geonames.org: postcode geocoding to latitude and longitude. http://koordinates.com: reverse geocode API to find Jurisdiction from from latitude and longitude.
It's a working site, but the code needs plenty of work to make it applicable to other jurisdictions.
Do not expect this code to work "out-of-the-box". Only use this code if you can work on it directly to customise for your jurisdiction.