Many listings are getting a 'Failed to Geocode' error and I think it is because they have a ')' at the end of the postal code, the listings that are geocoded, do not have the parentheses

Saving item 1640345 [1142.74 sec, 89.51 MB]                             [notice]
Item 437 of 500 [1142.74 sec, 89.51 MB]                                 [notice]
Failed to Geocode: 601 Bobwhite Ct , Lodi, WI 53555) [1142.85 sec,      [notice]
89.54 MB]
Saving item 1640152 [1142.88 sec, 89.57 MB]                             [notice]
Item 438 of 500 [1142.88 sec, 89.57 MB]                                 [notice]
Failed to Geocode: N10828 Rainbow Terr , Fox Lake, WI 53933) [1142.99   [notice]
sec, 89.6 MB]
Saving item 1639300 [1143.01 sec, 89.64 MB]                             [notice]
Item 439 of 500 [1143.01 sec, 89.64 MB]                                 [notice]
Failed to Geocode: 2107 Lewis St , Cross Plains, WI 53528) [1143.13     [notice]
sec, 89.67 MB]
CommentFileSizeAuthor
#11 address.png180.05 KBjday

Comments

jday’s picture

Status: Active » Closed (works as designed)

The problem might have been a daily limit on geocoding...not sure - but I've run the import script several more times now and haven't gotten any failures.

jday’s picture

Category: bug » support

ok so now I have a lot of listings that have not been geocoded and re-running the rets-import does not generate that info. I'm guessing it won't happen until something in the listing changes, is there a way to run the geocode separately from the import? like the photo processing?

jday’s picture

Category: support » bug
Status: Closed (works as designed) » Active

UPDATE
I added another field to collect during imports to see if that would trigger the geocoding for the listings missing that info, and it looks like my theory that the failures might be due to a daily limit was wrong. I continue to get the 'Failed to Geocode' errors for any listing that has a right parentheses at the end of it.

camidoo’s picture

not really sure where that right parentheses is coming from. Yea there isn't really a way to re-geocode like re-downloading the images. However, when you add a field mapping it will reset the hash to '', causing the import process to re-process that listing. So you can either add and then remove a field mapping, or open mysql and run a query to UPDATE drealty_listing SET hash = ''; or something similar to force all the listings to be reprocessed.

Which geocoder handler are you using to do the geocoding? I used to use Google, however they have a limit of 1500 per day, i have started using Yahoo! Placefinder and the results are just as accurate, and they're daily limit is 20k or something like that.

kevinquillen’s picture

This happens to me too with Innovia vendor for properties that already have lat/long coordinates saved. When I update the listing in drush, it throws that error for every listing. If I clear coordinates out and rerun, no error.

jday’s picture

I am using Yahoo placefinder and the listings that are failing have not previously been geocoded. Looking at the database, the latitude and longitude fields are NULL or 0.000000 and the zip code fields do not have the mysterious parentheses that appears in the failure messages.

UPDATE:
After adding another field and running the import again, I'm seeing that the failures happen on listings that have '0.000000' values in the lat/long fields. So I'm guessing the RETS feed is supplying those values...? and therefore sabotaging the geocoding by making the geocoding function believe the task is already done....

kevinquillen’s picture

I saw that happen too. What's the harm in overwriting the value or having them set as NULL instead of 0?

camidoo’s picture

actually, the geocoding does overwrite the values...

basically the when a listing is processed it loops through all the mappings that are defined...


  foreach($field_mappings as $mapping) {
    ....
    // $item->latitude = $rets_item['latitude'];
    $item->$mapping->field_name = $rets_item[$mapping->systemname];
    ....
  }

Once it's done doing the field mappings, then it checks to see if we are geocoding this 'class'

  if ($class->do_geocoding) {
            $street_number = isset($item->street_number) ? $item->street_number : '';
            $street_name = isset($item->street_name) ? $item->street_name : '';
            $street_suffix = isset($item->street_suffix) ? $item->street_suffix : '';

            $geoaddress = "{$street_number} {$street_name} {$street_suffix}, {$item->city}, {$item->state_or_province} {$item->postal_code}";
            // remove any double spaces
            $geoaddress = str_replace("  ", "", $geoaddress);

            if ($latlon = drealty_geocode($geoaddress, $class->geocoder_handler)) {
              if ($latlon->success) {
                $item->latitude = $latlon->lat;
                $item->longitude = $latlon->lon;
                drush_log(dt('Geocoded: @address to (@lat, @lon)', array('@address' => $geoaddress, '@lat' => $item->latitude, '@lon' => $item->longitude)));
              } else {
                drush_log(dt('Failed to Geocode: @address)', array('@address' => $geoaddress)));
              }
            } else {
              drush_log(dt('Failed to Geocode: @address)', array('@address' => $geoaddress)));
            }
  }
   $item->latitude = $latlon->lat;
   $item->longitude = $latlon->lon;

The two lines above essentially overwrite the value of anything that was pulled from the rets feed for lat and lon.

** well i figured out where the ')' was coming from this is on line 474 and 477 of drealty.daemon.php.

  drush_log(dt('Failed to Geocode: @address)', array('@address' => $geoaddress)));
  //should read
  drush_log(dt('Failed to Geocode: @address', array('@address' => $geoaddress)));

So, something is causing the geocoding to fail. Since you are using the yahoo placefinder, did you set the yahoo app id in the:

admin/config/content/geocoder

reason i ask is because, it won't complain if you don't have it set and will 'act' like it is set, but fail to geocode anything.

Let me know.

jday’s picture

Yes I have my api code from yahoo entered into the geocoder admin page. Out of 11K+ listings, 8,480 are failing and 1,914 are geocoded correctly...

I've copied some of the failed addresses into the map.yahoo.com site directly and it found most of them, two addresses it couldn't find. Maybe I'll try google and see what happens.

camidoo’s picture

I think I 'may' have figured out the problem.


            $street_number = isset($item->street_number) ? $item->street_number : '';
            $street_name = isset($item->street_name) ? $item->street_name : '';
            $street_suffix = isset($item->street_suffix) ? $item->street_suffix : '';

            $geoaddress = "{$street_number} {$street_name} {$street_suffix}, {$item->city}, {$item->state_or_province} {$item->postal_code}";

right now i'm concatenating the address out of these fields:

"{street_number} {street_name} {street_suffix}, {city}, {state} {postal_code}"

However if you've mapped:

street_dir_prefix or street_dir_suffix 

They never get put into the address, so the address should be put together like this:

"{street_number} {street_dir_prefix} {street_name} {street_dir_suffix} {street_suffix}, {city}, {state} {postal_code}"

Let me know if you've mapped either of those fields. If so i'll make the changes post haste and get them pushed up to see if it works. If not, then back to the drawing board.

jday’s picture

StatusFileSize
new180.05 KB

interesting...I do have street_dir_suffix mapped, but now I'm thinking that since the feed only has one value for direction it should be mapped to the street_dir_prefix field (attached screen shot)

camidoo’s picture

well i just pushed up a new release that should address not mapping those fields. so if they're mapped they'll now be included:

            $geoaddress = isset($item->street_number) ? $item->street_number : '';
            $geoaddress .= isset($item->street_dir_prefix) ? $item->street_dir_prefix : '';
            $geoaddress .= isset($item->street_name) ? $item->street_name : '';
            $geoaddress .= isset($item->street_dir_suffix) ? $item->street_dir_suffix : '';
            $geoaddress .= isset($item->street_suffix) ? $item->street_suffix : '';
            $geoaddress .= ", {$item->city}, {$item->state_or_province} {$item->postal_code}";

is how the address is built now.

camidoo’s picture

I'm also thinking that it might not be a bad idea to include an option for encoding based on the unparsed_address or encoding based on a concatenated address, i'll add that to the todo list.

jday’s picture

Thanks for your attention to this camidoo
I moved the mapping of the address direction to the street_dir_prefix field but looks like it's coming in before and after the street name

Item 117 of 500 [193.88 sec, 56.76 MB]                                  [notice]
Failed to Geocode: 1760Dewey Ave, Beloit, WI 53511 [193.99 sec, 56.79   [notice]
Saving item 1643728 [194 sec, 56.79 MB]                                 [notice]
Item 118 of 500 [194 sec, 56.76 MB]                                     [notice]
Failed to Geocode: 119WLawton StW, Edgerton, WI 53534 [194.11 sec,      [notice]
Saving item 1644529 [194.13 sec, 56.79 MB]                              [notice]
Item 119 of 500 [194.13 sec, 56.77 MB]                                  [notice]
Failed to Geocode: 819EMain StE, Reedsburg, WI 53959 [194.24 sec,       [notice]
camidoo’s picture

oh snap, i see the problem, thanks!

camidoo’s picture

there's a new release up there or you can grab the latest from git, via a 'git pull'.

Additionally, can you check to ensure that you don't have the field 'L_AddressDirection' mapped twice?

you can open phpmyadmin or another mysql tool and run this query:

SELECT * FROM `drealty_field_mappings` WHERE `systemname` = 'L_AddressDirection'

and see if it's there more than once.

jday’s picture

woe is me
1. I unchecked the Street Dir Suffix box, deleted the mapping
2. Checked Street Dir Prefix box, added the mapping
3. Ran the import - got the direction before and after the street name
4. Went into the database and deleted the 'street_dir_suffix' fields from the drealty_fields, drealty_field_mappings, and dreatly_listing tables
5. Ran the import
6. Error ->WD php: PDOException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'base.street_dir_suffix' in 'field list'

So I'm guessing I'll have to trash the db and start from scratch, which leads me to a new problem with the connection export/import feature for which I'll start a new issue

jday’s picture

After dumping the database and starting again, it looks like adding those fields to the $geoaddress has fixed the 'failed to geocode' errors I was getting. Camidoo is a genius.

camidoo’s picture

Status: Active » Needs review

Glad to hear that it's now working for you!

Can anyone else confirm that this is working for them?

stockliasteroid’s picture

I was having issues in my setup where the address_number and address_street were right up against each other... This was with beta18. In my case it was because the double-space replace in the geoaddress is replacing double spaces with no space at all. It seems like it should replace a double space with a single space, correct? Thus:

// remove any double spaces
$geoaddress = str_replace(" ", " ", $geoaddress);

That fixed my issue. anyway...

camidoo’s picture

Yea, i feared that may cause problems, there has to be a more elegant solution in how i'm putting together the address. I'll do some research and see what i can come up with.

kevinquillen’s picture

Seems like this should be closed? 3.x uses AddressField/Geo.

kevinquillen’s picture

Version: 7.x-2.0-beta11 » 7.x-3.0-alpha1
Status: Needs review » Fixed

Using AddressField + GeoField + Geocoder has been pretty straight in 3.x. Please re-open if this continues to occur.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.