@mattab opened this Issue on August 5th 2015 Owner

Right now, demo.piwik.org shows "This connection was reset" error. It's completely broken and not accessible. Looking at errors logs we can see continuously these errors added:

[Wed Aug 05 15:19:24 2015] [error] [mod_geoip]: Error while opening data file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Aug 05 15:19:25 2015] [notice] child pid 510957 exit signal Segmentation fault (11)
Error Opening file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat

Suggested solution:

  • when the /misc/GeoLiteCity.dat file is corrupted, Piwik / PHP process should not crash
  • can we add a test that simulates a corrupt /misc/GeoLiteCity.dat and check that Piwik tracking + UI still work?
@mattab commented on August 5th 2015 Owner

It breaks Tracker too! eg. http://demo.piwik.org/piwik.php?redirecturl=http%3A%2F%2Fplugins.piwik.org%2FCustomOptOut&c_i=click...... does not redirect.

Looking on the server, the file does not actually exist.

$ ls -la /home/piwik-demo/www/demo.piwik.org/misc/Geo*
-rw-r--r-- 1 piwik-demo piwik-demo 19237666 2012-12-21 10:57 /home/piwik-demo/www/demo.piwik.org/misc/GeoIPCity.dat.broken
-rw-r--r-- 1 piwik-demo piwik-demo  2918213 2014-11-05 18:35 /home/piwik-demo/www/demo.piwik.org/misc/GeoIPISP.dat
-rw-r--r-- 1 piwik-demo piwik-demo 32646602 2014-11-05 18:35 /home/piwik-demo/www/demo.piwik.org/misc/GeoIPOrg.dat
-rw-r--r-- 1 piwik-demo piwik-demo 19237666 2012-12-21 10:57 /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat.broken

I'm wondering why Piwik tries to load the file when it does not even exist, maybe this is the bug?

@mattab commented on August 5th 2015 Owner

FYI: to workaround this issue I had to re-download the geoip database:

$ wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
$ gunzip GeoLiteCity.dat.gz
$ ls GeoLiteCity.dat -la
-rw-rw-r-- 1 piwik-demo piwik-demo 15418580 2015-08-05 05:46 GeoLiteCity.dat
@tsteur commented on August 6th 2015 Owner

I tried to reproduce by removing the file misc/GeoIP.dat and by corrupting this file (removing lots of content everywhere, replacing it with other files etc). I cannot reproduce it neither in Tracker nor in Piwik. Also tried Default Provider, PHP provider and PECL provider. Even restarted apache etc after each change.

Do you still have that corrupt file? Tried http://demo.piwik.org/misc/GeoIPCity.dat.broken but doesn't work. Which provider is used?

@mattab commented on August 10th 2015 Owner

unfortunately we cannot reproduce the issue yet.

If you experience this issue, please comment here!

@mattab commented on September 2nd 2015 Owner

Again got the issue today. Demo.piwik.org is down, error log shows:

Error Opening file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:50 2015] [error] [mod_geoip]: Error while opening data file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:51 2015] [notice] child pid 3040200 exit signal Segmentation fault (11)
Error Opening file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:52 2015] [error] [mod_geoip]: Error while opening data file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:53 2015] [notice] child pid 3040305 exit signal Segmentation fault (11)
Error Opening file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:54 2015] [error] [mod_geoip]: Error while opening data file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:55 2015] [notice] child pid 3040425 exit signal Segmentation fault (11)
Error Opening file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:56 2015] [error] [mod_geoip]: Error while opening data file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:57 2015] [notice] child pid 3040529 exit signal Segmentation fault (11)
Error Opening file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:58 2015] [error] [mod_geoip]: Error while opening data file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:59 2015] [notice] child pid 3040641 exit signal Segmentation fault (11)

the file is that is being opened does not even exist:

piwik-demo<a class='mention' href='https://github.com/piwik'>@piwik</a>:~/www/demo.piwik.org$ ls -la /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
ls: cannot access /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat: No such file or directory

It's worrying that when this file is not even there, Piwik is broken in this way. let's investigate!

@tsteur commented on September 2nd 2015 Owner

I investigated and couldn't reproduce that time. Maybe someone else can

@sgiehl commented on September 6th 2015 Member

Do we have mod_geoip for apache on demo.piwik.org? Maybe it crashes if the configured GeoIPDBFile is not found?

@mattab commented on September 8th 2015 Owner

Do we have mod_geoip for apache on demo.piwik.org?

Yes mod_geoip for apache is activated on demo

Maybe it crashes if the configured GeoIPDBFile is not found?

Maybe indeed, if you can reproduce let us know!

@sgiehl commented on September 8th 2015 Member

Haven't installed mod_geoip yet, but I'll try to reproduce later...

@sgiehl commented on September 8th 2015 Member

I've just installed mod_geoip on my development box. Removing or corrupting does produce the " Error while opening data file" but everything is still working - no segfault.
Which version of apache/mod_geoip is installed?

@mattab commented on September 8th 2015 Owner

Apache 2.2

$ apt-cache show libapache2-mod-geoip | grep Version
Version: 1.1.8-2
@stijnstijn commented on November 30th 2015

I am having this exact issue. Running a Piwik install without problems, followed the steps in the FAQ for installing Geolocation via PECL, and now the config page for geolocation won't load and that segmentation fault shows up in the log.

I haven't really been able to find out more about this and the workaround of redownloading the database didn't work for me, but I'll be happy to do anything that may help you debug this.

Interestingly when I comment out the geoip.custom_directory directive in php.ini, everything works again and PECL geolocation works as well. The config page says the directive is empty but I suppose Piwik found the way to the database file on its own.

@raj45 commented on January 29th 2016

The GeoIPCity.dat was renamed by the system to GeoIPCity.dat.broken, and geotracking stopped working, I don't know when. My Piwik-installation worked fine, though.

I restored a fresh copy of GeoIPCity.dat to the /misc folder today, and everything seems to be working again. I am using the GeoIP (Apache) service.

@raj45 commented on February 5th 2016

I got the below error by mail, and the location no longer works. Also, GeoIPCity.dat in the /misc folder has been renamed to GeoIPCity.dat.broken ...

ERROR [2016-02-03 00:28:55] GeoIPAutoUpdater: Encountered PHP error when performing redundant tests on GeoIP loc database: 2: include(): Failed opening '/data/www/website/tmp/cache/tracker/piwikcache_Translations-en-ebd64ae248b2308a68b47baeadaf0cc549a66c38.php' for inclusion (include_path='/data/www/website/vendor/pear/pear_exception:/data/www/website/vendor/pear/console_getopt:/data/www/website/vendor/pear/pear-core-minimal/src:/data/www/website/vendor/pear/archive_tar:.:/usr/share/php:/usr/share/pear') on line 112 of /data/www/website/vendor/doctrine/cache/lib/Doctrine/Common/Cache/PhpFileCache.php.

@mattab commented on February 7th 2016 Owner

@sgiehl Maybe you have a chance to look into this issue? we also experience it on demo2, it would be awesome to address it in our LTS :+1:

@sgiehl commented on February 7th 2016 Member

I've retried to reproduce, but I'm not able to reproduce the segfault. Maybe it only occurs with a special combinations of os/php/pecl versions.

We maybe should try to reduce to possibility that those files are even missing. If I've seen that correctly, the GeoIPUpdater tries to download and update those files. If a downloaded file doesn't work it will be renambed to *.broken. Why don't we keep a backup of the file that was used before, and reuse that one if the update fails?

Powered by GitHub Issue Mirror