Allow using a google API key

Duplicate of #1949954: Client-side geocoding for proximity searches, to avoid Google's new 2500 per day limit.
We need to implement client-side initiated geocoding.

Sorry, didn't read the original issue properly. Probably not a duplicate. Something else might be going on here. However I vaguely remember that Google not only imposes a limit on the number of geocoding requests per day, but also on the request rate (i.e. requests/second).

Did you try the latest dev that has this patch to limit the request rate? https://drupal.org/node/1672742#comment-6800094.

I added a daily rate check to make sure I didn't run over the daily limits. It uses variables to store the current state.

Does this help?

I'm also seeing this error. I thought I found a fix when I rebuilt permissions last week, it started working immediately. But that action is not having the same effect today. As with the other posters, there is no way I'm coming anywhere close to the Google Geocoder daily limit.

same issue .. not even close but can no longer add new locations to the map (using Geofield Map). It worked great for months and now only displays the old stuff, but it's not possible to add a new point to the map.

same...
It worked great for months and now only displays the old stuff, but it's not possible to add a new point to the map.
Any ideas?

Ran into the same problem last week and fixed it.

The problem:
The google geocoding API allows for 2,500 requests per 24 hour period. My domain sends one request once a week and even then I received now and then the OVER_QUERY_LIMIT error.
Apparently my provider (one.com), like many others do not assign a unique ip-address to your domain (www.brouwerijhetnest.be in my case) instead, one.com let many domains share the same ip-address.
The google API limits are default calculated per ip-address so if you have bad luck and are in a pool of domains with the same ip-adress and one or more of the domains in the same pool makes extensively use of the google geocoding web service you are in trouble.
Besides the limit on the number of requests per day Google applies also a request rate limit (sending requests too fast, i.e. too many requests per second), this too can be triggered when a lot of domains share the same ip-address. Both potential problems will be solved by the proposed solution.

The solution:

The google API does not need an API key, which is fine, but if you use one you get per-key instead of per-IP-address quota limits.

Step1: get an API key (key for server applications) (it is free)
https://developers.google.com/maps/documentation/geocoding/#api_key

Step 2: include the API key with the geocode request
Make a few changes in the file : \sites\all\modules\geocoder\plugins\geocoder_handler\google.inc
Usually I download it by ftp (FileZilla) from the server, edit it (notepad++) and upload it again.
In the code at line32 add your api key and in the next line change the “http” request in “https”

function geocoder_google($address, $options = array()) {
  try {
    geophp_load();

    $query = array(
      'address' => $address,
      'sensor' => 'false',
      'key' => 'AIzaSyDKSQicatPi4RUrlpmbH2m9rTwT4IyxYMI', 
    );

    $url = url("https://maps.googleapis.com/maps/api/geocode/json", array('query' => $query));

That’s it. From now on you can follow in your Google Developers Console (https://console.developers.google.com) the number of requests and where they came from, but more importantly the limits will now be calculated per API key request.

Hope this works for you too, and if it does I hope that one the maintainers of this wonderful Geocoder module will implement this solution in the module so that the google API-key can be inserted in the configuration/Geocoder settings menu as is currently the case with the Yahoo and Yandex API keys.

Re #8:
Great analysis and solution luk.stoops!
Agree with you that something along the lines of your solution would be highly desirable.
Rik

Attached is a patch that adds Google Business API support, it updates the settings form to accept the client id and signature and then append the client id and 'signs' the URL with the signature. If you add in these details the limit is raised to 100,000 calls per day, so hopefully this solves your problem!

Thanks @luk.stoops for the very useful analysis!

On our site, we have a multi-value address field. This field predictably stops working after 20 items. It seems that 20 lookups per second is the Google geocoder limit on the free accounts. Even if you have done no other lookups that day, 20 is the most you can do at one single time. However, we found that even a tiny delay can allow the geocoding to work properly, so we put a sleep statement in geocoder_widget_get_field_value() like this:

      // Around line 270:
      foreach ($source_field_values as $delta => $item) {
        if (($delta + 1) % 20 === 0) {
          sleep(1);
        }
        // ... rest of function
      }

This adds a 1 second pause every 20 geocodings.

Other alternatives to this problem could be fixing the Diffing code in geocoder_widget_get_field_value(). Right now, geocoding happens on *every* value on *every* save. If the diff values were used to look up only changed addresses, we could significantly cut down on lookups. This would make it so editors could add up to 20 addresses at a time, then edit the node and re-add additional ones. It'd also speed up the node saving process significantly. I'll file a separate patch to that effect because it should have a performance benefit and would only inadvertently fix this problem with Google geocoding.

Doh! Looks like my exact solution has basically already been implemented in #1672742: Avoiding OVER_QUERY_LIMIT due to more than 5 requests in a second in google geocode! All we need to do is update to the latest dev of the module, and now there's a setting for delay between requests to Google. This module could use a new stable release.

I'm getting this also with either ZERO_RESULTS or OVER_QUERY_LIMIT errors. Setting a delay of 1s between calls makes no difference. Will see if I can check from a different IP.

Seeing with ZERO_RESULTS. The stage site is working but not the Prod...
@pmackay, did IP make a difference for you?

I did some more experimenting and got this working. Not sure now if was due to moving network or code changes.

@pmackay, mattsmith3: ZERO_RESULTS simply means google can't figure out where your address is located, which is completely unrelated to OVER_QUERY_LIMIT errors.

When you use Geocoder to geocode an address field, are the results cached? If so, how?

If you (like me) are running on your Pantheon Dev environment, your free google geocoding limit is used up by everyone else also from that IP address.
So you get a OVER_QUERY_LIMIT almost instantly.
The fix for this is to get your own Google maps API key, but this module is currently only written to use the free/anonymous tier.

I've written a patch to add you own google maps API key.
Note, this is different from the business API support.

You enter you Google maps api key from admin/config/content/geocoder
and the request now uses https - a requirement from Google for using a key.

If you don't enter a key it continues to work as before, but you could hit the limit if you are on a shared resource.

OK, that was a bad patch. I didn't patch against the latest version of the 7.x-1.x branch.

This patch does.

Can someone confirm it works?

Changing title and requalifying as "Feature request".

Tested geocoder-google_maps_key_api-2024809-v2.patch #20 against 7.x-1.x-dev.

At the first test, where I left the API code blank, the geocoding worked well but I get every time a notice:
Notice: Undefined index: geocoder_apikey_google in geocoder_google() (regel 37 vanC:\Users\Luk\Sites\hetnest\sites\all\modules\geocoder\plugins\geocoder_handler\google.inc).
After inserting my API key the geocoding went smoothly without notices. Great!
Checked in my Google Developers Console that my API key was used in the request. OK!

Would be nice to add in the configuration/Geocoder settings menu some info about where to obtain an API key (similar as the other keys)
You can obtain a Google Geocoding API Key at https://developers.google.com/maps/documentation/geocoding/#api_key

nice job, thx

I can confirm that patch in #20 works as well (with API key entered). Didn't think to try it without, so not sure about the error mentioned in #23.

-Daniel
Apollo Data Solutions

Ok, let's address the notice so it's a useful message for the user, and it's good to go, as several people are confirming it works.

I did some extra testing,
Switched off, uninstalled and removed the geocoder module.
Installed it again and tried it without an API key. This time there was no notice as mentioned in #23.
But apparently the uninstallation process does not remove all the modules data. If I insert my API code and then switch off, uninstall and remove the module, clear the cache and later reinstall it, my API code pops up magically.

So to take a clean start I reinstalled geocoder 7x-1.2 and recovered my database from my latest backup.
Then I installed the patched geocoder-7.x-1.x-dev version and tried to geocode an address. The notice pops up:

Notice: Undefined index: geocoder_apikey_google in geocoder_google() (regel 37 vanC:\Users\Luk\Sites\hetnest\sites\all\modules\geocoder\plugins\geocoder_handler\google.inc).

Then I went to the configuration/Geocoder settings menu and without entering anything I just pressed save configuration.
This does the trick, from now on no more notices!

Ok, so replacing $geocoder_settings['geocoder_apikey_google'] != "" by empty($geocoder_settings['geocoder_apikey_google']) should do the trick for everything to be fine.

Not really, now I get:

Notice: Undefined index: geocoder_apikey_google in geocoder_google() (regel 41 van C:\Users\Luk\Sites\hetnest\sites\all\modules\geocoder\plugins\geocoder_handler\google.inc).

I'm not a PHP specialist but should it not be the reverse of "empty"?

By the way, I do not understand why you call this "a useful message for the user". To me it seems just an indication to all future users that something went wrong during the development of the new version of geocoder. Every user, except those that will use a google API, (the majority) will get this error message each time they geocode an address.

Yeah, you are totally right, it should be the reverse of "empty". With it, there should be no message any more, and everything should be right.
By "a useful message to the user", I meant something like "Your google API key is missing, please go to [url] to set it", or something like that.

OK,
Tried with: !empty($geocoder_settings['geocoder_apikey_google']) and that seems to work.
I ‘ll put the changes in a new patch v3, added info on how to get the API key and brushed up a typo.

Then I tested it from scratch. It works. No more notices, and the geocoding works fine with and without the API key.

@Simon I now understand your "a useful message to the user" and maybe that will be a good idea for the other API’s but the google geocoding API is only optional so I think there is no need to notice the user which option he is using.

The real problem with this google api is that the variable it is not properly initialized during the installation of the module (and that it is not removed during the uninstall process but that’s an other issue).
The use of the !empty() function is a nice workaround and if this is OK for the maintainers of the module they can go for it.

For anyone reading this before the patch is added to the main release, make sure your request url is https:// and not http:// (as google require this when using an API key). If you have a shared/cloud style host you can also use http://requestb.in/ to receive dummy requests to obtain your servers true external ip.

Maybe the patch has already added to the geocoder 7.x-1.x-dev release?

#30 works for me (and restored my sanity). Thanks for taking the time to contribute it!

There appears to be one last item before the patch can be committed:

The real problem with this google api is that the variable it is not properly initialized during the installation of the module (and that it is not removed during the uninstall process but that’s an other issue).
The use of the !empty() function is a nice workaround and if this is OK for the maintainers of the module they can go for it.

We really need to get the variables created & removed properly.

Patch reroll against latest dev. And allows for both methods: the free API key (Google Geocoding API Key) and business (Google Maps for Work).

Great work all in this thread.
I hope someone with commit rights can apply the latest version of the patch soon.

Excellent fix - works well for me. And the Google Developer's Console allows me to monitor the frequency of requests. This was important for me during the development of a site where we are importing around 20k nodes. Slow, but steady, and at least it's working now.
Thanks for the latest patch from derek.deraps.

This is great patch... works nicely. I got exactly this problem on shared hosting. Wouldn't have figured this out quickly without this post. Many thanks. Adding a Google Encoding API to the patched module works nicely.
Geocoder really needs another stable release btw. Current one is from August 2012... seriously?

@basvredeling, there are currently a lot of open issues that I would like fixed before releasing a new version. Last time I checked, these were all in "needs review". As I lack the time to review them, I'm counting on the community to do it, but only a few things are moving.
What you've done here (reviewing and changing status) is the perfect step towards a new release. I will now take some time to commit this patch, and be done with this issue ;-) Thank you, and feel free to do the same for other issues ;-)

I'm up for reviewing some more and if you need a hand maintaining this I might be up for co-maintainership.

Thanks for the work you've done. I cannot grant any rights on the module, I am just a "temp" maintainer, I stepped up a few months ago, but I didn't find any more time since. I'll look at what you did, but I can already say that closing issues like you did is a great way to make everything clearer for maintainers ;-) Thanks !

sure, no problem

#2024809-35: Allow using a google API key works as advertised. Thanks derek.deraps

best part is now I can see each request at https://console.developers.google.com/

one minor change I can think of is providing more helpful error messages.

Consider this error message:

Exception: Google API returned bad status.\nStatus: ZERO_RESULTS in geocoder_google() (line 120 of /var/www/sites/all/modules/contrib/geocoder/plugins/geocoder_handler/google.inc).

How would I find out which address is failing on what entity?!

Should we at least include the address in query returning the error?

For example;

from:

function geocoder_google($address, $options = array()) {
...
    if ($data->status != 'OK') {
      $args = array('@status' => $data->status);
      $msg = t('Google API returned bad status.\nStatus: @status', $args);
      throw new Exception($msg);
    }
...
}

to:

function geocoder_google($address, $options = array()) {
...
    if ($data->status != 'OK') {
      $args = array(
        '@status' => $data->status,
        '@address' => $query['address'],
      );
      $msg = t('Google API returned bad status. Status: @status Address: @address', $args);
      throw new Exception($msg);
    }
...
}

@osman I agree that a more informative error message would be useful. But it seems like a different issue than #2024809: Allow using a google API key

Patch in #35 works great for me now, thanks!

+1 for RTBC

For those in this thread who may need it, I wanted to offer another option for the API limits. I've just added a sandbox module called "Rules Throttle", which provides a simple configurable rules action, allowing an administrator to set a delay using usleep (in microseconds). In this way, you could use rules to customize your delays, on an as needed basis, for as many resources as you needed to throttle. It appears to be working for me.

As for the API key issue, thank you for everyone who contributed! I'll be sure to post back any issues, but so far so good.

Huge collective work! Thanks for all the contributions, sorry for the delay in the commit, and keep up the work in the other issues!