[master] Redux: UK Geocoding

good ideas..

two quick points:

a. pairing postcodes with city/town names..in the UK. that would be useful and I would imagine it would be implemented in a simple flat comma seperated text file or an include file. is there copyright issues on the royal mail list? am guessing there is...so perhaps a bit of googling will track down (pun intended) a source that you could use.

b. until seeing that mapquest thing...I was playing with inserting a direct link to multimap.com with the Users post code in the link...which was great as it also offers a camera shot of the street as well...

the only drawback was the multimap.com site drives my anti-spybot software mental as it tries to install all sorts of weird cookies or something, so I stopped using it. that mapquest thing looks good, even if it doesn't have the camera thing.

if it's of use...I posted how I automatically inserted a link to a street map in a users profile here:

http://drupal.org/node/14466#comment-26095

Dub

I have updated the location.uk.inc file to link to Google Maps and Multimap.

The links use the geographic lat/lon co-ordinates if they are available, or, if not, postcodes, and if there is no postcode then street name and city.

I hadn't realised there was a free distribution of UK postal sectors at jibble (though the file doesn't seem to be there at present). I wonder if this could perhaps be tied into the postal area or postal district names from Wikipedia to allow lookup from postcode to town/city and also provide a lat/lon whilst it's at it? Are there any licencing issues with doing that?

Quick question - do you absolutely need to have a local copy of the PAF?

Why not screenscrape Google Maps which handily returns the lat/long pair in its page (although not via the API).

Quick and dirty example, lifted and PHP'd from this Perl example:
http://morland.theoretic.org/archives/2005/04/15/free_geocoding_thanks_t...

http://colonsay/~martin/geocode/gmap.php

Code at

http://colonsay/~martin/geocode/gmap.txt

Presumably also, one doesn't need a complete geocoded PAF stored locally in the zipcodes table. One only needs entries for where you've got nodes as a local cache.

So perhaps the answer for where no complete db is freely available:
1) On node creation and Google-supported location is provided, geocode it then:
a) If it's not already in the zipcodes table, put it there
b) else if it is and it's different, assume a geocode update (it happens - our postcode is currently geocoded wrongly by Google, but correctly as per latest version of PAF) and update the table

2) On location search:
a) If it's in the zipcodes table, use that
b) If it's not, Geocode from Google, use the result and store it in zipcodes

This is tuned for reads: making the assumption that searches will be more frequent than node adds, and that the performance hit of screenscraping Google isn't significant at node creation.

Of course, once the data is in zipcodes table, it's business as usual.

Hmm screenscraping Google Maps may be against their ToU. Yahoo! provide a very cool Geocode API, but, you guessed it, it's North America only.

It's a good idea Martin, but as you point out, it may breach their Terms of Use. Sadly I think that would almost certainly be the case.

For now I reckon postal sectors (like M14) is the way to go as they would provide a set of locations to a few miles resolution or so I think (similar to a US 5 digit zipcode?) - though I'd like to see that dataset to see their distribution.

ideally PAF files would be centralised (not local) to make maintenance easier.

if this is pie in the sky, my apologies, but:

what about clubbing together to buy a licenced PAF from the Royal Mail and storing it on a Drupal server ... ?

I sincerely doubt you could buy that for any reasonable price. I don't think Royalmail is quite that silly.

Also, if the information is on wikipedia, it is almost definitely free to use/download/etc. Look at the copyright information at the bottom of the page.

Royalmail data licences seem to be way overpriced, and simply not that useful.

All I can find on wiki is this http://en.wikipedia.org/wiki/List_of_postal_areas_in_the_United_Kingdom which appears to be very similar to the 'province list' dankarran has used in the patch file. It only covers the first 2 letters of the postcode, so does not enable a very high level of detail.

The jibble source (http://www.jibble.org/ukpostcodes/) covers the first 2 letters and first 2 numbers (eg AB12) which should narrow down locations to within a few miles (I guess). Also includes Long/Lat co-ordinates which are very useful. (not come across the x/y co-ords in metres before ...?)

It should be very easy to map the wiki list (approx 120 city names) onto the jibble file.

I'll do it manually if someone can tell me how! I'm no programmer so I don't understand what format the inc file data needs to take.

found a UK postcode file via http://www.brainstorm.co.uk/uk_post_code_search.htm
it seems freely availble : "available for free download" ! though probably best to ask ...
File is 1800kb text format (the smaller files contain less towns (rows) but same details (3 columns)

sadly they do not include lon/lat data.
It looks like the records are also incomplete (my home town is not listed).
snip eg: (drupal codefilter collapses the spaces that separate columns)

Information supplied by http://www.brainstorm.co.uk/post-codes.txt - creative solutions for mobile media.

Town                            County                          Postcode
~~~~                            ~~~~~~                          ~~~~~~~~

A'Chill                         Highland                        PH44
Ab Kettleby                     Leicestershire                  LE14
Abbas Combe                     Somerset                        BA8
Abberley                        Hereford and Worcester          WR6
...

there is also a telephone STD area code file at http://www.brainstorm.co.uk/stdcodes.csv
my hometown appears on this one ... :)

"01130","Leeds, W. Yorks"
"01131","Leeds, W. Yorks"
"01132","Drighlington, W. Yorks"
"0113250","Rawdon, W. Yorks                               [Leeds]"
"0113252","Morley, W. Yorks"
"0113255","Pudsey, W. Yorks"
"0113258","Horsforth, W. Yorks                            [Leeds]"

are these of any use?

Responses to a few things in one:

1. The Post Office doesn't have a problem with providing a location API - QAS do it already. However, it's unlikely to do so without a significant chunk of money.
2. You're right Dann, for most purposes (nearest X/X within Y distance), postcode district (AB1 etc - sector is AB1 1) is probably good enough
3. Does anyone know if there is © on the actual postcodes? Their geocoding is a list of facts, probably, so unlikely to be ©able.

4. Can we not just go Pinchot on this and JFDI? Throw the Brainstorm list (sorted by unique postcode district) at Google once and just then use that[1]?

   Taking advantage of the 'probably against' ToU we should be able to apologise afterwards without too much in the way of consequences, unless we're dumb enough to do a jibble and make the compiled list separately (and very visibly) downloadable. RoyalMail are probably not going to be as upset about district-level data as the real money is at postcode unit level for geodem etc.

[1] Caveat updates to the PAF, so prolly needs to be an annual event, but by next year, the Yahoo! GeoCode API may cover the UK. Not so much of a worry at district level anyway.

Incidentally, I have the base 'geocode from postcode and insert into zipcodes table for future user' function working... see attached

Minor niggle with this approach - postcode district data doesn't always return data for 3 char data elements (AB1 for example). Ho hum - much of it is useful, though. I may do sample full postcodes for those districts that failed and re-run.

If you do it that way, here's the problematic districts (see attachment)

Blimey, I'm a busy bee today and should have concatenated these replies.

Anyway, the fun about trying to link town names to postcodes is that there isn't a simple correlation - some towns are split between postcode districts, and all have multiple towns/villages/hamlets per district. Same story for counties. And postcode areas are *far* too unspecific to be useful (as an exercise, take a look at the coverage of PA, or IV).

So if you're using the postcode as a primary key, you're not getting enough at the district level to be absolutely sure of the town and county. As http://www.statistics.gov.uk/geography/postal_geog.asp points out, we can go to sector level (and have 11598 rows in the table), but there isn't a comprehensive list available (unless anyone's got it to hand). Even the Brainstorm district list is short - the above URL claims 3064 districts, and Brainstorm only provides 2531.

To provide an accurate, up to date table, you're either going to have to license PAF, or do an on-demand query to an online service. If you want it *free*(beer), you're screenscraping until such time as Yahoo! brings its API to the UK.

Martin,

Are you saying then that if we were to geocode those missing districts then that would give us a complete dataset? I got lost somewhere in the thread of which files you were comparing. I don't have any issues getting the lat/lon data for just a small set of distrcts but I don't think we should be scraping the sites for complete postcode geocoding...

You're right when you say about tieing postal districts to names, that's not a simple task, and perhaps unnecessary - just a nicity? I realised that the other night when trying to get a dataset of Isle of Man (IM) postal districts together. I'll be geocoding IM1-IM9 soon, and maybe even IMx x as that data is indirectly available (perhaps with a little of my local knowledge) through the post office website in the form of a map showing where the areas are.

By the way, I found that list of geocoded postal sectors that used to be on jibble.org, and thought I'd put it here for safekeeping and incase any of you guys don't have it.

Interestingly, Google doesn't return locations for AB1 (or any other, seemingly) of those other districts that appear to be missing, though they do return results that have AB1 as part of the content on their pages. I wonder if they are using this same non-complete dataset of geocoded postal sectors?

Dan

If we geocoded those missing areas we'd have a *more* complete set. But ONS suggests that there are 3064 Districts (AB1), whereas the Brainstorm dataset - like Jibble - only has 2600 or so. I've attached my dataset, which has counties (subject to the caveat that some districts cross county borders). Where the lat/long is 23.875/57.630033, this means "unavailable from Google"

You're right though, Google doesn't appear to have data for AB1 at *all*. All the postcode searches I try for there fail (eg Aberdeen Dyce airport's postcode: AB1 7DU).

You're also right in that scraping the complete postcode dataset (1.78million elements) and including it in the module download is probably both unnecessary, unfair, and wouldn't add enough benefit for the size of download required. I still think that an on-demand basis for postcode units is probably best, but with the district level data as backup. This means we don't have to distribute detailed data *at all* and each installation's dataset is tuned for their users.

So, where does this leave us? Perhaps with accepting, geocoding and storing full postcodes for nodes, but on performing Nearest X searches, cutting the data entered down to district level (which is mostly already geocoded in the db) for performance reasons. So a search would effectively be:

$user_location="tf10+7et";
$user_search_loc=explode("+", $user_location);
$user_geocode=geocode($user_search[0]); 
// select long/lat for tf10 in the zipcodes table
// returns 52.769816,-2.386649
return(nearby_nodes($user_geocode));

which returns nodes with geocoded full postcodes in the vicinity.

Hi,

I too am very eager to use Drupal to do proximity searches and to link to Google maps. Is there any news regarding this project?

Many thanks,
Jack

I am not sure if this is what you're looking for, but a number of companies offer postcode geocoding through Web Service. iShareMaps does this. It does cost to integrate it, though cheap. It is free to use on their website. See www.geocode.isharemaps.com.

Has anyone seen the free collaborative geocoder.us style http://www.freethepostcode.org/
It has both downloadable data files and geocoder lookup.
It's only got ~700 entries right now but is definately worth keeping an eye on!

As far as paid lookup services go I also found http://www.postcodeanywhere.co.uk/, which has a more flexible pricing model than http://www.geocode.isharemaps.com/ - as well as a the standard fee (which is also cheaper £160/year for street level) you can buy 'credits', 1 per geo lookup and £50 for 1,000 credits. This is more useful for smaller sites!

Here's a much nicer way of getting log/lat from a UK postcode:

http://www.streetmap.co.uk/streetmap.dll?GridConvert?name=EX331JZ&type=P...

Screen scrape and store on node creation. Seems to be always up to date too (for my test postcodes).

That URL in full:

http://www.streetmap.co.uk/streetmap.dll?GridConvert?name=EX331JZ&type=PostCode

I'm still interested in this but don't have much time available to work on it at the moment.

Having been working a lot on the OpenStreetMap project and recently a little on the Free The Postcode project, I'm very reluctant to do screen scraping of postcodes. In my eyes, the data is either going to have to come from a free source (eg Free The Postcode) or a source that has proper licencing arrangements in place. Perhaps this will be Google at some point, but the UK is one of the countries missed out of their geocoding API.

With Free The Postcode, I'd say around 50% of postal areas have coverage at present, and there's still a long way to go before all of the districts [zz9] contain the location of at least one postcode. But once the project grows some more, it would be good to tie into the geocoding facilities over there to be able to pull in approximate locations based on postcodes being submitted to Drupal.

Looking at Free The Postcode project it is going to take a bloody long time to cover everything well enough to use the data in a functioning site.

Maybe an established Drupal site could be used to collect the postcode data as its requested (via a screen scrape) and then contribute the values back to somewhere.

How is anybody going to know if the long/lat is borrowed from somewhere else? :-S

Have a look at www.geonames.org they got a great global database going and open source webservices available - do you think that could be used?

I am not 100% familiar with how this works, but from you comments I thought I would have a go, not successful but am I going down the right direction (see attached file)? Has anyone else had any success with getting this working for the UK yet?

On a related note see http://drupal.org/node/79015, I am also finding that the Province is no longer saving.

Sorry, think I missed a point earlier on, after trying to return data from goodle uk geocoding, I realised that is not available for the UK, probably said that earlier in this stream. Does anyone know if it will be at some point?

regarding to your new varable "location_geocode_uk_google_apikey":

the Gmap Module already asks for this api key: why not reuse it and its variable?
http://drupal.org/project/gmap

the variable is called "googlemap_api_key"

I suggest to replace yours with the one of the gmap.module

I've done some quick work at http://drupal.org/node/75459 to create a location.uk.inc file that uses the Geonames.org webservice to fetch rough locations for UK addresses... (download via that link)

I think geonames are using the Jibble postcode database (or similar), so search returns a location based on the first part of the postcode only. When a postcode isn't provided, the search uses the city name to location a latitude and longitude.

It's all a bit messy at the moment (it doesn't advertise itself as a geocoder properly etc..) but may do the job for people wanting quick rough UK geocoding...

I'm hoping to do some more work on it soon....

public domain UK Postcode -> lat: http://www.npemap.org.uk/data/outward1list

Is this still going?

I'm struggling with the UK province list - the muddling of counties and cities, the odd abbreviations (and their use in $location), then there's the issue of Wales, Scotland and the Channel Islands... and Ireland vs Repulbic of Ireland

Can't really believe I'm the only one!! Is there a more 'UK' friendly location.uk.inc you're all using or is there an even better way?

Funny indeed! and thanks for helping! I'm very grateful.

I'm kind of surprised I'm the first Brit to have raised this! Surely there are other Birtish location users out there?

I like location - it seems a very 'Drupal' way to do addresses (though for 'true' Drupalness reusable venues seem like a must (to me!)) but the user interface (not being able to select country, then province so the province list isn't so overwhelming) and problems I'm having categorising addresses so I can search on them are such that I'm wondering if I'll have to go to taxonomy? as there seems no relationship ability between countries and provinces (and I think ideally I'd even like one more layer province - district)

Anywhere which isn't 'true' London (or Manchester) uses Counties and they're fiercely proud of them - the Royal Mail has 'had a go' at several counties (especially Middlesex) but the inhabitants are pretty resistant to losing their counties!! 'Official' information at (also referenced by though as I say counties are really quite important - and I need to be able to produce county lists of location data.

I think you might like !! though even he seems to do down counties!

For example -
Grand Harbour Hotel, West Quay Road, Southampton, Hampshire, SO15 1AG - is a pretty typical address and this is how they use it themselves (and Hampshire is abbreviated to HANTS not HAM!!)

London's a law unto itself - I've never seen letters addressed to 'HMF' => "Hammersmith and Fulham" (and I lived in London for over 20 years!!) - they're 'London Boroughs' and it would be quite nice to have them optionally in the address but HMF would mean nothing to anyone - they just don't abbreviate - so I don't know what to do about them (but if the pretty name was displayed by default in $location it wouldn't really matter what abbreviation was in the database).

Counties are a big deal - think States! - and the smaller the county the prouder - many have their own flags! and you fiddle with their borders at your peril, ignoring them is a capital crime! It is true that a letter will get to a town and a postcode but trying to do regional event listings with just that is very hard indeed!

For example, someone in Totnes, Devon would expect to see

11 Something Street
Totnes
DEVON
PP11 1PP

Location currently produces

11 Something Street
Totnes, DEV
PP11 1PP

DEV means nothing to anyone over here!! and it gets even worse for Bedfordshire etc. - Bedfordshire would normally be BEDS or Bedfordshire - not BDF. I think I can sort the abbreviation issue by working through them and putting 'better' abbreviations in but it would still be nice to be able to have the full county.

City duplication is another problem - I wouldn't want an address to say Plymouth, Plymouth! It should be Pymouth, DEVON. I need them to put the city in the city! otherwise I can't search (again if they went in the same optional field as London boroughs (as a sort of 'near Plymouth') that would be ok). Don't large states have 'neighbourhoods' or 'districts' people would be interested in? like say all events
in Queens in New York?

Wales, Scotland and Northern Ireland like to see themselves in addresses - this makes life very tricky - especially as Province has baggage - Wales is a Province but Scotland isn't.

They'd expect their addresses to look like

Ballathie House Hotel
Kinclaven
Stanley
Perthshire
PH1 4QN

ideally from outside Scotland they'd also like to see

Ballathie House Hotel
Kinclaven
Stanley
Perthshire
PH1 4QN 
SCOTLAND
United Kingdom

Similarly

Penmaenpool
Dolgellau
Gwynedd LL40 1YB
WALES

Again - it's very important to be able to produce a list of Scottish events - I can't do that if there's no way of searching on them!

Then there are the Channel Islands - Jersey, Guernsey etc (and the Isle of Man and Isle of Wight which would expect to see themselves in an address!).

Apollo Hotel
St. Saviour's Rd
St. Helier 
Jersey 
JE2 4GJ 

Seaview Hotel
High Street
Seaview
Isle of Wight
United Kingdom
PO34 5EX

ah ... a nation built on compromise :P ... and the Republic of Ireland (AKA Eire) doesn't have any postcodes at all ... ;)

Is there any chance any of this could make it into the proper releases?

The patches seem to add geocoding support to some extent - if the Counties/Province/Kingdom issues could be resolved it would make locations useable for UK users at last!

Google are now offering UK Geocoding - so I've created a location.uk.inc at http://drupal.org/node/158101 which can use this.

(Doesn't take into account any of the discussions above...)

Tim

Hi, sorry to revive an old thread but I'm a bit confused - has support for UK postcodes been implemented within the locate module?

Ok, so I can use geocoding on my site via the google API? How do I go about using that?

Hi
Im wondering if anyone has any more info on how this is going. I have a list of UK post codes with coordinates, but its set in a different format to the US sql list that is included with the location module.

So im wondering, If I go through the list and reformat it so its the same, then upload it, will it work as a database for UK post codes?

Yes, in order to enable Drupal to search for nodes by proximity to a user-entered part-postcode (up to 4 chars) you just need to create a SQL file based on the free postcode data listed above, import it into zipcodes database, then modify location.uk.inc so that it contains the two functions "location_latlon_rough_uk" and "location_get_postalcode_data_uk". Copy them from location.us.inc (dont forget to rename the functions) and alter so that it does not mess with the postcode (look for substr, str_par). As my requirements don't require city or province names I have not put those into the SQL, these could be easily added for a more complete package.

I am attaching the sql script which inserts the FREELY available postcode to lat/long data into Drupal. For reference here are my versions of the functions:

function location_latlon_rough_uk($location = array()) {
  if (!isset($location['postal_code'])) {
    return NULL;
  }
  $result = db_query("SELECT latitude, longitude FROM {zipcodes} WHERE country = '%s' AND zip = '%s'", $location['country'], $location['postal_code']);
  if ($row = db_fetch_object($result)) {
    return array('lat' => $row->latitude, 'lon' => $row->longitude);
  }
  else {
    return NULL;
  }
}
function location_get_postalcode_data_uk($location = array()) {
  if (!isset($location['postal_code'])) {
    return NULL;
  }
  // Now we pad the thing and query.
  $res = db_query("SELECT * FROM {zipcodes} where country = '%s' AND zip = '%s'", $location['country'], $location['postal_code']);
  if ($row = db_fetch_object($res)) {
    return array('lat' => $row->latitude, 'lon' => $row->longitude, 'city' => $row->city, 'province' => $row->state, 'country' => $row->country);
  }
  else {
    return NULL;
  } 
}

Hi Andrew
Thanks for all that. Ive installed it all, and it works, albeit a little randomly. Im having a few problems with getting the right results in views and wondered if you could help, dunno if this is something I have done wrong or maybe a small bug.

For instance I tag nodes with a Postcode, so there all ready. I have one marked with the postcode TN24

But when I search and put tn24 (5 miles radius) in the search, nothing is returned. It cant find local things! Its returning other postcodes just fine, again some of them are a little out in their distance.

How precise should this be? any ideas?
Thanks

Marking master.
Changing component to Data update.

Guys,

commited on HEAD. Soon on Location 3.0

regards,

massa

If anybody using postcodeanywhere to receive Postcodes, here is javascript file which can be easily implemented for specified form.
Please change your licence code and id to the postcode fields inside of the file.

Created new project if any interested:
http://drupal.org/project/postcodeanywhere
which integrate postcodeanywhere service