do we use the best available free geo ip database from maxmind as default?
we use by default this link to get the data base:
it has a file size of 10,6MB
you could find today the Link to a free database with a size of 14,4MB
=> is this an updated better Version with more/better/more accurate information in it? Or only a different format?
This article explains some changes: [http://dev.maxmind.com/geoip/geoip2/whats-new-in-geoip2/]
Thanks for the suggestion!
Note: I read this on their site: Downloadable databases for GeoIP2 are not yet available. However, when they are, they will use our new MaxMind DB format. Once the databases are released, we will provide additional details on their use on this dev site.
SO we have to wait that they make the DB available.
yes I read this too but if you uses the links in the ticket above, you can already download the files...
+1.. there is a huge difference between geolite and geolite2 in terms of brute data..i downloaded both in csv and the newer one is 3x larger so 3x times more data! csv format is the accurate measure for comparison here, not compressed binaries which might work on different algorithms
hope to see this ticket solved.. in my case visitors` cities are correctly reported in less than 50 % of the cases (i use google analytics real-time map to compare, which is always accurate).. but i am sure that this is different from country to country, others might have 90% accurate results on the same old database
Trying to use the now legacy geoip database I get "Constant GEOIP_NETSPEED_EDITION_REV1 already defined", after inserting an if statement for checking if it was already defined GeoIP (PECL) is not able to tell me the database type. The message I get is: "This GeoIP implementation has access to the following types of databases: ."
Any chance this is going to get some attention soon so we can easily use geoip2 with piwik and not even worry about the above issue?
I am also getting the "Constant GEOIP_NETSPEED_EDITION_REV1 already defined" error. Is there a fix for this?
Downloadable databases for GeoIP2 are available now:
So can we have Piwik support new GeoIP2?
I too get the "Constant GEOIP_NETSPEED_EDITION_REV1 already defined" Error. :(
The GeoIp2 database output is different from GeoIp1 concerning names, codes and structure.
See output gist using default test IP
As you can see, it also contains (some) translations of geographic names and the
contenent_code for Europe has changed from "eur" to "EU".
Below are some questions to ask ourselves, but first some examples:
The city name output is:
[city] => Array ( [geoname_id] => 3033123 [names] => Array ( [de] => Besançon [en] => Besançon // etc. [zh-CN] => 贝桑松 ) )
The Region Names aren't supported the way they are in GeoIp1. There now is an
subdivisions property containing an array of area's. For above IP that's as follows; index 0 being the "biggest":
[subdivisions] => Array (  => Array ( [geoname_id] => 3017372 [iso_code] => I [names] => Array ( [de] => Franche-Comté // etc. [fr] => Région Franche-Comté // etc. ) )  => Array ( [geoname_id] => 3020989 [iso_code] => 25 [names] => Array ( [de] => Doubs // etc. ) ) )
For different countries, the devisions are different and not necessarily mapping to the GeoIp1 setup.
What is the best way to handle these changes? I don't know. Some questions that I think should be answered first:
Converting the new GeoIp2 output to exisiting GeoIp1 data will be error prone, so maybe it's best to store visit-data in a new format (using the
Should current visit-data be transformed into the GeoIp2 structure? If not, how to mix both GeoIp1 and GeoIp2 data in reports?
In https://github.com/piwik/piwik/pull/6690#issuecomment-86860922 @olavmrk suggested making a split between new installs and existing setups. Is that feasible?
I'm looking forward to your comments...
it's nice to talk about this important new feature :-) Here are some notes:
Implementation: I think Geoip2 support in Piwik should be implemented as a location provider just like GeoIP is currently implemented. (code). The implementation detail of the geoip2 location provider will be hidden in the location provider class, via the implementation of these few methods here: https://github.com/piwik/piwik/blob/master/plugins/UserCountry/LocationProvider.php#L67-135
Admin UI: currently users can choose in the Geo location UI to use
GeoIP (Php) or
GeoIP (Pecl) or
With Geoip2 support they could also choose for example the new
Geoip 2 (Php) and maybe
Geoip2 (PecL) and
Geoip2 (apache) (not sure what options there are for geoip2)
Data migration: I think we don't have to worry about data migration for this task, as IMO there won't be major issue there (but maybe i'm missing something).
I hope it helps, let me know if you have any other question!
Hello, what is the status of the implementation of Geoip2? Can we download and you use it?
From my usage I haven't noticed issues anymore.
@g-p-g are you sure you are using the enhanced database of geoip version 2 ? version 1 is working fine of course...
@hpvd I posted here about 1 year ago regarding this issue (you can see my comment way above). It's working for me using the latest Piwik release (not just the latest, but a few before the latest too). It's also possible that Piwik upgrades stopped reverting changes in place to get this working, which until a few version ago had to be redone after each update.
@g-p-g How did you make it work. Previous files were .dar files and these new ones are .mmdb. Did you made some modifications?
@AgentGod, @hpvd, and others sorry for the false hope. I mixed this with other things, the Piwik installation is in fact not using geoip2.
not good news @g-p-g :) @mattab is there any ETA for implementing the option you describe in your post?
No ETA yet, we have not scheduled this improvement at this stage. It would be awesome if a developer from the community would issue a Pull request for this, but I understand it is a non trivial change.
I have done some work on GeoIP2 support, but currently don't have time to continue on it. I can PR it, but it's not finished.
See: https://github.com/joostdekeijzer/piwik/tree/GeoIP2 or more specifically https://github.com/joostdekeijzer/piwik/commit/258f7a7584af7f3c9d42bb9cc9569334fb151ed7 which is the commit to add my current implementation.
Don't forget to run
compuser update to install the GeoIp2 code.
Good to know
The new GeoIp2 libraries don't support the
area_code, so it's left empty.
What it does do
- Read mmdb files if properly named and put in the
- Output GeoIP info, but I've only tested by calling the API, eg:
/index.php?module=API&method=UserCountry.getLocationFromIP&ip=126.96.36.199&format=JSON what looks fine
Needs work on
- tests (currently none)
- some IPv6 addresses to test with that are "future proof" and "non IPv6-mapped IPv4"
- Since the mmdb format is open, maybe create a small custom test mmdb with a static set of data?
- auto-update datafiles
- The current automatic updates forms are hard-coded and only for the .dat files. It should change depending on which location provider is selected?
- The auto-update cron function must be updated to download and test mmdb files
- Needs testing
- Not sure if https://github.com/joostdekeijzer/piwik/commit/258f7a7584af7f3c9d42bb9cc9569334fb151ed7#diff-663e1f37bf83d5b3c62915f859c498a7R47 is the best solution since the
country datafiles overlap and if
city is available probably
country shouldn't be used.
- probably a lot more! ;-)
+1 for this - would love to pay for a more accurate database to use with Piwik :)
@berters if you would pay for more geoip accurate data in piwik (like me!!!) please have also a look at this ticket: and vote for it by commenting it and asking for reopen... https://github.com/piwik/piwik/issues/8121
Can we use new GeoIP2-City.mmdb file downloaded from Maxmind with piwik to locate ipv4 and ipv6 ?
If anyone did it please guide with details description.
I also just bought the MaxMind City database, which only comes in the .mmdb format. Does Piwik support the .mmdb format?
EDIT: It seems like there is a .dat format available, called "GEO-133" (the .mmdb version is called "GeoIP2-City"). I will return with feedback after testing it.
EDIT #2: The GEO-133 Maxmind database works, and the paid version is a bit more precise on city level: "Unknown" has gone down from 26% to 12% for Danish cities.
+1 we need support for this. The old legacy geoip database say I live in HongKong but I am in Germany.
+1 for this. GeoLite legacy is really out of update for few (actually a lots) countries.
Joining the chorus of +1's here, good to see it's still an active topic with the recent label & milestone change (thanks @mattab 👍). Looking forward to seeing this implemented! 😄
To others that end up here before buying a MaxMind DB: Purchasing a GeoIP2 DB also grants you access to the legacy GeoIP DB format database too - We've just purchased the GeoIP2 ISP database, and the legacy version was available and integrated with Piwik 2.16.1 just fine. (Although as previously mentioned the legacy format is a lot smaller).
Isn't the paid legacy version also heavily outdated?
@JHGitty The legacy versions show as being up-to-date on MaxMind's download page, although I have no way to compare it with the GeoIP2 DB to find out.
UPDATE: I've sent off for our business IP to be listed correctly in the Organisation DB - I guess we'll find out how up-to-date they are if we show up correctly in a couple of weeks time.
@JHGitty just got this week's GeoIP Organisation Legacy on our server and sure enough we're now showing in it - It's just as up-to-date as the new format one. :smile:
just got this week's GeoIP Organisation Legacy on our server and sure enough we're now showing in it - It's just as up-to-date as the new format one. :smile:
that's great news!
thanks for sharing info with us! beside it's up to date - does it also contain same amount/details of data?
The question is also does this also apply to all the other types of geoip databases?
I think even if v1 of database is ok for some cases, there is no way around making v2 usable in Piwik...
I would appreciate, if the new format would be used in piwik. Of course the legacy geolite-db is still maintained, but the new format has more granular data and so it can be more accurate in some case, but for me there is another reason, why the new format should be used: IPv6 - the legacy geolite has two seperated DBs for IPv4- and IPv6-entries - the geolite2 has tables for both address-types combined in one file. Additionally it seems, that the file-struture has become more effective, so I assume that the lookup will be faster with the geolite2.
For the integration of geolite2 into piwik you could use the page where you can choose the location-provider for the geolocation and add besides the GeoIP(php)-location-provider a new one called GeoIP2(php) and use the GeoIP2-php-API to parse the geoip2-database-files: https://github.com/maxmind/GeoIP2-php
seems to be one of the most discussed issues :-)
just to highlight what's the difference between v1 and v2: there was already a link to a nice summary posted 2,5 Years ago: https://github.com/piwik/piwik/issues/4487#issuecomment-48313707
Adding my +1 to the chorus. Some data from the old DB is heavily outdated/wrong.
Please take a look at https://github.com/piwik/piwik/pull/10431 pull request
@mattab: diabl0 did not only offer a pull request but now he also made a plugin for geoip2 support see https://plugins.piwik.org/GeoIP2 or https://github.com/diabl0/piwik-geoip2 => would be awesome to make Piwik benefit directly of this great work...
@hpvd correct. We suggested him to do so. Having stuff like that in a new plugin makes much more sense than having everything in core. Maybe it would also make sense to move the old GeoIP provider to a separate plugin in marketplace, so everone can choose which one to install.
@sgiehl many thanks for explanation. If so, the step to move also geoip1 to a plugin would make absolutely sense.. Otherwise it maybe a little confusion for piwik users how to deal with this when wanting to use geoip2, because these questions arise: - will this geoip2-plugin will work flawlessly within piwik (which has already geoip1 integrated)? - will there be any conflicts between both? - how do I know which data piwik uses really? - do geoip2 plugin benefit from piwiks autofetch/-update for geoip1 databases? - if so, how to set up? - ...
beside - geoip2 plugin, - geoip1 plugin,
a third plugin in this row could be - a connection to maxmind webservice of high accurate data but on pay per use base (see https://github.com/piwik/piwik/issues/8121 )
This could maybe make use of the official webclient (php) supporting this -as a good starting point see https://github.com/maxmind/GeoIP2-php
Ideally, in a few months, once we have validated everything works as expected and once @diabl0 plugin is much used, we would (maybe?) integrate the GeoIP2 plugin in core, and move the GeoIP1 stuff to a plugin on the Marketplace, and we'd keep full backward compatibility so that nobody's geo location is affected. Since we don't have time to nicely handle & test all this at the moment, we proposed to make Geoip2 on the marketplace and keep geoip1 in core, for now. Hope it makes sense!
will this geoip2-plugin will work flawlessly within piwik (which has already geoip1 integrated)?
It was working as core (mentioned erlier PR https://github.com/piwik/piwik/pull/10431 ) and it's currently working without problems on my company instance using current 3.x-dev branch
will there be any conflicts between both?
None observed, it shows as another Location Provider
how do I know which data piwik uses really?
Check your settings - currently used location provider is marked
(on screen there are unknown values because site was accessed from intranet IP address)
You can also reattribute older data using console command:
do geoip2 plugin benefit from piwiks autofetch/-update for geoip1 databases?
At this time - no, but it is planned as soon as i get access to Maxmind account from my company (and this can take some time as everything in larger corporation).
wow. Many thanks for these that fast and positive pieces of information!
regarding testing autofetch: the 2 free databases available also got updates from time to time (and needs to be refreshed) : https://dev.maxmind.com/geoip/geoip2/geolite2/
The performance characteristics of the new version may need to be tested against the legacy database. In the Java version of the GeoIP2 plugin there is a performance issue related to text deserialization.
Obviously Piwik is not Java, but it'd be unfortunate to see a significant regression in lookup times, especially if Piwik doesn't use most of the new data.
@diabl0 your plugin is not working for me in piwik 3.0. See https://github.com/piwik/piwik/issues/11034
In our case we need accurate Maxmind and Intranet IPAM information. As i donot see how to chain location providers currently with i cloned diabl0s work and added a Intranet Data loookup in front of the maxmind dbs. So if found in the own IPAM data it takes this data. If not use Maxminf https://github.com/diabl0/piwik-geoip2/issues/4 https://github.com/grzchr15/piwik-geoip2-andintra
Tested with Piwik 3.0.0
I am using it with latest Piwik 3.0 and seems working fine
Has anyone had any luck getting the ISP and sort data to show in Piwik? We have not yet been able to install the ISP/Org database, we have access to the new version (2) and the legacy version. We've put both types of files into the misc directory but we are still seeing "Org: Unknown ISP: Unknown".
We have been able to install the lite city database.