As of August 23 or so, ARIN has changed the format of their db, adding a new column of data, as well as changing the name of the ftp download file. As a result, our ip2country module stopped recognizing US IP addresses. I have put in a temporary fix on our site, but would like to have it validated before posting a patch- I only speak PHP at a kindergarten level!
My fix- edit the ip2country.inc as follows:
1) line 36: change "delegated-arin-latest" to read "delegated-arin-extended-latest"
2) line 84: remove the "count($parts)!=7" condition
The second change is the one up uneasy about- it removes a data check that I don't know the purpose of. but for the short term, it got us back up and running.
| Comment | File | Size | Author |
|---|---|---|---|
| #6 | patchresults.txt | 1.46 KB | Kitz |
| #5 | 2078071.patch | 3.17 KB | tr |
Comments
Comment #1
tr commentedI'll try to push a fix for this sometime today.
Comment #2
tr commentedActually, I can't reproduce this problem. On my test site, the module is able to read from all five registries without an error. What errors did you see (and where did you see those errors), and what registry(ies) were you using?
As of today, ARIN seems to be maintaining both the "delegated-arin-latest" and "delegated-arin-extended-latest" lists, but perhaps they weren't doing that when they initially rolled out the "extended" one.
Also, I don't see that the "extended" list has more columns - they look the same to me, with the exception that the "extended" list has more records. I don't know where those extra records come from.
Have you seen an explanation of the changes anywhere? If so I would like to read about what they've done / plan to do.
The check on the 7 parts is to filter out the "summary" records, which don't hold valid country<->IP data.
Comment #3
Kitz commentedThanks TR-
The problem may be hard to see- i didn't get any error message when trying to load the db- the code as written just skips the lines with the extra column (ie all the lines in ARIN's new format) without throwing an error. The only way to see the problem is to test a couple of US ip addresses. Try anything from 3.0.0.0 to 3.255.255.255 for starters. Our own IP (sorry, don't have it with me at home) also failed. As downloaded, the new format file skips so many US lines that most US ip's come up as "not assigned". This happened first with ARIN, then a week later with APNIC and RIPE .
I looked in the ip2country db, and found only a handful of lines for the US.
To fully see what was going on, I ftp'ed the new and old files (RIPE has the old formats in an archive folder) and opened them in excel to compare them, whence I was able to see the new column. From discussions I've seen, ARIN has a file under the old file name, but it doesn't have the proper data in it. I didn't go to the ARIN ftp site, so I can't say for sure if that's true.
ARIN posted a notice on August 14th about no longer supplying the old format files- if you go to arin.net and follow the link to their announcement archive, you can see the announcement: "ARIN NOW PROVIDING EXTENDED STATS ONLY".
Comment #4
tr commentedOK, thanks. Here's what I've been able to figure out:
ARIN, and only ARIN out of the 5 registries, has decided to add an additional column to the database it provides. They make a point of saying this is allowed, and indeed it IS allowed by standards document describing the file format, found at ftp://ftp.arin.net/pub/stats/arin/README (You can get this same document from any of the other registries, just use ftp.afrnic.net, ftp.apnic.net, ftp.lacnic.net, or ftp.ripe.net.)
However, ARIN has also decided to stop providing a file called "delegated-arin-latest" - this file name is REQUIRED by the same document above that allows them to add an additional column. Instead they provide a file called "delegated-arin-extended-latest", which according to the standards document is not allowed.
All ARIN had to do was create a symbolic link on their site between "delegated-arin-latest" and "delegated-arin-extended-latest" and they would have been fully compliant. I have no idea why they don't do this.
ARIN's announcement can be found at https://www.arin.net/announcements/2013/20130814.html
All the other registries still provide a file called "delegated--latest" as called for in the standard, and as expected by this module. ARIN is the authority for US IP addresses, which is why all the US numbers disappeared when the module couldn't find "delegated-arin-latest"
So the fix is two-fold:
1) Remove the check for 7 columns, which I used to distinguish data records from non-data records. Instead, I will explicitly identify and remove header records and comment records in a different manner.
2) Add a special case in the code where the ARIN data needs to be named with "-extended". I'd rather do this than rename them all "-extended", because as I said above the standards document doesn't guarantee the "-extended" name.
I'm testing the fix and a few other improvements right now, should be finished tonight.
Comment #5
tr commentedI committed a fix. Patch attached if you want to apply the fix without downloading the latest -dev.
Backport to D6 pending ...
Comment #6
Kitz commentedTR-
Thanks for the quick response. I tried the patch on our backup site, but got 3 hunk failures. Note that I'm fairly bad with patches, and am working from home without most of my software tools, so it may be me and not the patch. Nonetheless, I'm attaching the hunk failure report, and the .rej file contents, both in the same text file, if you want to follow up.
I'll be better able to test the patch on Tuesday when I'm back at work- I'll post my results then.
Comment #7
tr commented@Kitz: You must have an older version of the module. Patches are generated against the most recent -dev version. Or perhaps your download of the patch failed - the "patch unexpectedly ends in middle of line" is a little strange, as the patch is definitely good.
Anyway, I've committed the fix to both the 6.x-1.x and 7.x-1.x branches, so if you simply upgrade to the latest -dev it should work again. I'm leaving this issue open to remind myself to port the fix to the 8.x-1.x branch, which is currently broken because of all the changes that are being made in D8 these days.
I'm also working on some SimpleTest cases to ensure that the automated tests will detect the problem if something similar happens again in the future. Those should be added in the next few days.
Comment #8
tr commentedActually, port to 8.x-1.x was easy. So now it's fixed in all three branches.
Comment #9
Kitz commentedTR-
I installed the 7.1 dev version, and it appears to be working perfectly.
Thanks for the quick fix!
Comment #10
kennywyland commentedI'm trying to install this version, but it fails catastrophically when I enable the module:
Error: Call to undefined function ip2country_update_database() in
/var/www/vhosts/thehurleypro.com/sites/all/modules/ip2country/ip2country.install, line 97
Comment #11
tr commented@kennywyland: Whoops. Not a result of this patch, but of another improvement I made recently. I opened a new issue at #2085289: Latest -dev doesn't install to address your bug report.