@hpvd opened this issue on January 7th 2014

do we use the best available free geo ip database from maxmind as default?

we use by default this link to get the data base:

http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz

it has a file size of 10,6MB

on http://dev.maxmind.com/geoip/geoip2/geolite2/

you could find today the Link to a free database with a size of 14,4MB

http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz

=> is this an updated better Version with more/better/more accurate information in it? Or only a different format?

@nullnull-00 commented on January 7th 2014

This article explains some changes: [http://dev.maxmind.com/geoip/geoip2/whats-new-in-geoip2/]

@mattab commented on January 9th 2014

Thanks for the suggestion!

Note: I read this on their site: Downloadable databases for GeoIP2 are not yet available. However, when they are, they will use our new MaxMind DB format. Once the databases are released, we will provide additional details on their use on this dev site.

SO we have to wait that they make the DB available.

@hpvd commented on January 9th 2014

yes I read this too but if you uses the links in the ticket above, you can already download the files...

@anonymous-piwik-user commented on June 3rd 2014

+1.. there is a huge difference between geolite and geolite2 in terms of brute data..i downloaded both in csv and the newer one is 3x larger so 3x times more data! csv format is the accurate measure for comparison here, not compressed binaries which might work on different algorithms

hope to see this ticket solved.. in my case visitors` cities are correctly reported in less than 50 % of the cases (i use google analytics real-time map to compare, which is always accurate).. but i am sure that this is different from country to country, others might have 90% accurate results on the same old database

@g-p-g commented on July 16th 2014

Trying to use the now legacy geoip database I get "Constant GEOIP_NETSPEED_EDITION_REV1 already defined", after inserting an if statement for checking if it was already defined GeoIP (PECL) is not able to tell me the database type. The message I get is: "This GeoIP implementation has access to the following types of databases: ."

Any chance this is going to get some attention soon so we can easily use geoip2 with piwik and not even worry about the above issue?

@ak5 commented on August 14th 2014

I am also getting the "Constant GEOIP_NETSPEED_EDITION_REV1 already defined" error. Is there a fix for this?

@sim4life commented on August 22nd 2014

Downloadable databases for GeoIP2 are available now: http://dev.maxmind.com/geoip/geoip2/geolite2/

So can we have Piwik support new GeoIP2?

@piwats commented on November 8th 2014

I too get the "Constant GEOIP_NETSPEED_EDITION_REV1 already defined" Error. :(

@joostdekeijzer commented on March 30th 2015

The GeoIp2 database output is different from GeoIp1 concerning names, codes and structure. See output gist using default test IP 194.57.91.215

As you can see, it also contains (some) translations of geographic names and the contenent_code for Europe has changed from "eur" to "EU".

Below are some questions to ask ourselves, but first some examples:

The city name output is:

[city] => Array
    (
        [geoname_id] => 3033123
        [names] => Array
            (
                [de] => Besançon
                [en] => Besançon
                // etc.
                [zh-CN] => 贝桑松
            )
    )

The Region Names aren't supported the way they are in GeoIp1. There now is an subdivisions property containing an array of area's. For above IP that's as follows; index 0 being the "biggest":

[subdivisions] => Array
    (
        [0] => Array
            (
                [geoname_id] => 3017372
                [iso_code] => I
                [names] => Array
                    (
                        [de] => Franche-Comté
                        // etc.
                        [fr] => Région Franche-Comté
                        // etc.
                    )
            )

        [1] => Array
            (
                [geoname_id] => 3020989
                [iso_code] => 25
                [names] => Array
                    (
                        [de] => Doubs
                        // etc.
                    )
            )
    )

For different countries, the devisions are different and not necessarily mapping to the GeoIp1 setup.

What is the best way to handle these changes? I don't know. Some questions that I think should be answered first:

Converting the new GeoIp2 output to exisiting GeoIp1 data will be error prone, so maybe it's best to store visit-data in a new format (using the geoname_id's only)?

Should current visit-data be transformed into the GeoIp2 structure? If not, how to mix both GeoIp1 and GeoIp2 data in reports?

In https://github.com/piwik/piwik/pull/6690#issuecomment-86860922 @olavmrk suggested making a split between new installs and existing setups. Is that feasible?

Implement server support for apache and/or nginx ?

I'm looking forward to your comments...

@mattab commented on March 31st 2015

Hi @joostdekeijzer

it's nice to talk about this important new feature :-) Here are some notes:

Implementation: I think Geoip2 support in Piwik should be implemented as a location provider just like GeoIP is currently implemented. (code). The implementation detail of the geoip2 location provider will be hidden in the location provider class, via the implementation of these few methods here: https://github.com/piwik/piwik/blob/master/plugins/UserCountry/LocationProvider.php#L67-135

Admin UI: currently users can choose in the Geo location UI to use Default or GeoIP (Php) or GeoIP (Pecl) or Geoip (Apache). With Geoip2 support they could also choose for example the new Geoip 2 (Php) and maybe Geoip2 (PecL) and Geoip2 (apache) (not sure what options there are for geoip2)

Data migration: I think we don't have to worry about data migration for this task, as IMO there won't be major issue there (but maybe i'm missing something).

I hope it helps, let me know if you have any other question!

@AgentGod commented on June 18th 2015

Hello, what is the status of the implementation of Geoip2? Can we download and you use it?

Thanks.

@g-p-g commented on June 18th 2015

From my usage I haven't noticed issues anymore.

@hpvd commented on June 18th 2015

@g-p-g are you sure you are using the enhanced database of geoip version 2 ? version 1 is working fine of course...

@g-p-g commented on June 18th 2015

@hpvd I posted here about 1 year ago regarding this issue (you can see my comment way above). It's working for me using the latest Piwik release (not just the latest, but a few before the latest too). It's also possible that Piwik upgrades stopped reverting changes in place to get this working, which until a few version ago had to be redone after each update.

@AgentGod commented on June 19th 2015

@g-p-g How did you make it work. Previous files were .dar files and these new ones are .mmdb. Did you made some modifications?

@g-p-g commented on June 19th 2015

@AgentGod, @hpvd, and others sorry for the false hope. I mixed this with other things, the Piwik installation is in fact not using geoip2.

@AgentGod commented on June 19th 2015

not good news @g-p-g :) @mattab is there any ETA for implementing the option you describe in your post?

@mattab commented on June 22nd 2015

No ETA yet, we have not scheduled this improvement at this stage. It would be awesome if a developer from the community would issue a Pull request for this, but I understand it is a non trivial change.

@joostdekeijzer commented on June 22nd 2015

Hi all,

I have done some work on GeoIP2 support, but currently don't have time to continue on it. I can PR it, but it's not finished.

See: https://github.com/joostdekeijzer/piwik/tree/GeoIP2 or more specifically https://github.com/joostdekeijzer/piwik/commit/258f7a7584af7f3c9d42bb9cc9569334fb151ed7 which is the commit to add my current implementation.

Don't forget to run compuser update to install the GeoIp2 code.

Good to know The new GeoIp2 libraries don't support the area_code, so it's left empty.

What it does do - Read mmdb files if properly named and put in the /misc/ directory - Output GeoIP info, but I've only tested by calling the API, eg: /index.php?module=API&method=UserCountry.getLocationFromIP&ip=194.57.91.215&format=JSON what looks fine

Needs work on - tests (currently none) - some IPv6 addresses to test with that are "future proof" and "non IPv6-mapped IPv4" - Since the mmdb format is open, maybe create a small custom test mmdb with a static set of data? - auto-update datafiles - The current automatic updates forms are hard-coded and only for the .dat files. It should change depending on which location provider is selected? - The auto-update cron function must be updated to download and test mmdb files - speed - Needs testing - Not sure if https://github.com/joostdekeijzer/piwik/commit/258f7a7584af7f3c9d42bb9cc9569334fb151ed7#diff-663e1f37bf83d5b3c62915f859c498a7R47 is the best solution since the city and country datafiles overlap and if city is available probably country shouldn't be used. - probably a lot more! ;-)

@berters commented on September 18th 2015

+1 for this - would love to pay for a more accurate database to use with Piwik :)

@hpvd commented on September 18th 2015

@berters if you would pay for more geoip accurate data in piwik (like me!!!) please have also a look at this ticket: and vote for it by commenting it and asking for reopen... https://github.com/piwik/piwik/issues/8121

@sharifhasankhan commented on September 30th 2015

Can we use new GeoIP2-City.mmdb file downloaded from Maxmind with piwik to locate ipv4 and ipv6 ?

If anyone did it please guide with details description.

@raj45 commented on November 12th 2015

I also just bought the MaxMind City database, which only comes in the .mmdb format. Does Piwik support the .mmdb format?

EDIT: It seems like there is a .dat format available, called "GEO-133" (the .mmdb version is called "GeoIP2-City"). I will return with feedback after testing it.

EDIT #2: The GEO-133 Maxmind database works, and the paid version is a bit more precise on city level: "Unknown" has gone down from 26% to 12% for Danish cities.

@JHGitty commented on January 24th 2016

+1 we need support for this. The old legacy geoip database say I live in HongKong but I am in Germany.

@nntoan commented on May 12th 2016

+1 for this. GeoLite legacy is really out of update for few (actually a lots) countries.

@Danw33 commented on June 14th 2016

Joining the chorus of +1's here, good to see it's still an active topic with the recent label & milestone change (thanks @mattab 👍). Looking forward to seeing this implemented! 😄

To others that end up here before buying a MaxMind DB: Purchasing a GeoIP2 DB also grants you access to the legacy GeoIP DB format database too - We've just purchased the GeoIP2 ISP database, and the legacy version was available and integrated with Piwik 2.16.1 just fine. (Although as previously mentioned the legacy format is a lot smaller).

@JHGitty commented on June 15th 2016

Isn't the paid legacy version also heavily outdated?

@Danw33 commented on June 15th 2016

@JHGitty The legacy versions show as being up-to-date on MaxMind's download page, although I have no way to compare it with the GeoIP2 DB to find out.

UPDATE: I've sent off for our business IP to be listed correctly in the Organisation DB - I guess we'll find out how up-to-date they are if we show up correctly in a couple of weeks time.

@Danw33 commented on July 1st 2016

@JHGitty just got this week's GeoIP Organisation Legacy on our server and sure enough we're now showing in it - It's just as up-to-date as the new format one. :smile:

@mattab commented on July 8th 2016

just got this week's GeoIP Organisation Legacy on our server and sure enough we're now showing in it - It's just as up-to-date as the new format one. :smile:

that's great news!

@hpvd commented on July 8th 2016

thanks for sharing info with us! beside it's up to date - does it also contain same amount/details of data?

The question is also does this also apply to all the other types of geoip databases?

I think even if v1 of database is ok for some cases, there is no way around making v2 usable in Piwik...

@ummelmann commented on July 8th 2016

I would appreciate, if the new format would be used in piwik. Of course the legacy geolite-db is still maintained, but the new format has more granular data and so it can be more accurate in some case, but for me there is another reason, why the new format should be used: IPv6 - the legacy geolite has two seperated DBs for IPv4- and IPv6-entries - the geolite2 has tables for both address-types combined in one file. Additionally it seems, that the file-struture has become more effective, so I assume that the lookup will be faster with the geolite2.

For the integration of geolite2 into piwik you could use the page where you can choose the location-provider for the geolocation and add besides the GeoIP(php)-location-provider a new one called GeoIP2(php) and use the GeoIP2-php-API to parse the geoip2-database-files: https://github.com/maxmind/GeoIP2-php

@hpvd commented on July 8th 2016

seems to be one of the most discussed issues :-)

just to highlight what's the difference between v1 and v2: there was already a link to a nice summary posted 2,5 Years ago: https://github.com/piwik/piwik/issues/4487#issuecomment-48313707

@parruc commented on August 4th 2016

Adding my +1 to the chorus. Some data from the old DB is heavily outdated/wrong.

@diabl0 commented on August 24th 2016

Please take a look at https://github.com/piwik/piwik/pull/10431 pull request

@hpvd commented on September 28th 2016

@mattab: diabl0 did not only offer a pull request but now he also made a plugin for geoip2 support see https://plugins.piwik.org/GeoIP2 or https://github.com/diabl0/piwik-geoip2 => would be awesome to make Piwik benefit directly of this great work...

@sgiehl commented on September 28th 2016

@hpvd correct. We suggested him to do so. Having stuff like that in a new plugin makes much more sense than having everything in core. Maybe it would also make sense to move the old GeoIP provider to a separate plugin in marketplace, so everone can choose which one to install.

@hpvd commented on September 28th 2016

@sgiehl many thanks for explanation. If so, the step to move also geoip1 to a plugin would make absolutely sense.. Otherwise it maybe a little confusion for piwik users how to deal with this when wanting to use geoip2, because these questions arise: - will this geoip2-plugin will work flawlessly within piwik (which has already geoip1 integrated)? - will there be any conflicts between both? - how do I know which data piwik uses really? - do geoip2 plugin benefit from piwiks autofetch/-update for geoip1 databases? - if so, how to set up? - ...

@hpvd commented on September 28th 2016

beside - geoip2 plugin, - geoip1 plugin,

a third plugin in this row could be - a connection to maxmind webservice of high accurate data but on pay per use base (see https://github.com/piwik/piwik/issues/8121 )

This could maybe make use of the official webclient (php) supporting this -as a good starting point see https://github.com/maxmind/GeoIP2-php

@mattab commented on September 28th 2016

Ideally, in a few months, once we have validated everything works as expected and once @diabl0 plugin is much used, we would (maybe?) integrate the GeoIP2 plugin in core, and move the GeoIP1 stuff to a plugin on the Marketplace, and we'd keep full backward compatibility so that nobody's geo location is affected. Since we don't have time to nicely handle & test all this at the moment, we proposed to make Geoip2 on the marketplace and keep geoip1 in core, for now. Hope it makes sense!

@diabl0 commented on September 28th 2016

will this geoip2-plugin will work flawlessly within piwik (which has already geoip1 integrated)?

It was working as core (mentioned erlier PR https://github.com/piwik/piwik/pull/10431 ) and it's currently working without problems on my company instance using current 3.x-dev branch

will there be any conflicts between both?

None observed, it shows as another Location Provider

how do I know which data piwik uses really?

Check your settings - currently used location provider is marked

(on screen there are unknown values because site was accessed from intranet IP address)

You can also reattribute older data using console command:

./console usercountry:attribute

do geoip2 plugin benefit from piwiks autofetch/-update for geoip1 databases?

At this time - no, but it is planned as soon as i get access to Maxmind account from my company (and this can take some time as everything in larger corporation).

@hpvd commented on September 28th 2016

wow. Many thanks for these that fast and positive pieces of information!

regarding testing autofetch: the 2 free databases available also got updates from time to time (and needs to be refreshed) : https://dev.maxmind.com/geoip/geoip2/geolite2/

@GreenReaper commented on October 11th 2016

The performance characteristics of the new version may need to be tested against the legacy database. In the Java version of the GeoIP2 plugin there is a performance issue related to text deserialization.

Obviously Piwik is not Java, but it'd be unfortunate to see a significant regression in lookup times, especially if Piwik doesn't use most of the new data.

@s1awa commented on December 20th 2016

@diabl0 your plugin is not working for me in piwik 3.0. See https://github.com/piwik/piwik/issues/11034

@grzchr15 commented on January 8th 2017

In our case we need accurate Maxmind and Intranet IPAM information. As i donot see how to chain location providers currently with i cloned diabl0s work and added a Intranet Data loookup in front of the maxmind dbs. So if found in the own IPAM data it takes this data. If not use Maxminf https://github.com/diabl0/piwik-geoip2/issues/4 https://github.com/grzchr15/piwik-geoip2-andintra

@grzchr15 commented on January 8th 2017

Tested with Piwik 3.0.0

@AgentGod commented on January 15th 2017

I am using it with latest Piwik 3.0 and seems working fine

@simplusgroup commented on May 26th 2017

Has anyone had any luck getting the ISP and sort data to show in Piwik? We have not yet been able to install the ISP/Org database, we have access to the new version (2) and the legacy version. We've put both types of files into the misc directory but we are still seeing "Org: Unknown ISP: Unknown".

We have been able to install the lite city database.

Powered by GitHub Issue Mirror