@orlitzky opened this Issue on October 15th 2014

This started about a week ago, I don't think anything changed here. Every night we run,

import_logs.py --enable-http-errors --enable-http-redirects --enable-bots --enable-static --recorders=6 --url=https://analytics.example.com/ <all logs from yesterday>

Now it regularly gets stuck:

Parsing log /var/log/apache2/example.com/www/access/access-2014-10-14.log...
6886 lines parsed, 6284 lines recorded, 79 records/sec (avg), 148 records/sec (current)
6886 lines parsed, 6461 lines recorded, 80 records/sec (avg), 177 records/sec (current)
6886 lines parsed, 6676 lines recorded, 82 records/sec (avg), 215 records/sec (current)
...
6886 lines parsed, 6676 lines recorded, 25 records/sec (avg), 0 records/sec (current)

and eventually fails with the following:

6886 lines parsed, 6676 lines recorded, 25 records/sec (avg), 0 records/sec (current)
Fatal error: ''
You can restart the import of "/var/log/apache2/example.com/www/access/access-2014-10-14.log" from the point it failed by specifying --skip=5 on the command line.

Is that the sixth line of the log file? If so, there's nothing weird in it.

The problem began on 2.7.0, but persists after an upgrade to 2.8.0.

@tsteur commented on October 15th 2014 Owner

Is there a chance that you can send us the log with like the first 10 lines? In case it is reproducible with only 10 lines...
If so, please send to hello at piwik.org.

@orlitzky commented on October 15th 2014

It's on its way, but some more testing suggests that the log file isn't (entirely) to blame. With six recorders,

202 lines parsed, 190 lines recorded, 94 records/sec (avg), 77 records/sec (current)
...
202 lines parsed, 190 lines recorded, 5 records/sec (avg), 0 records/sec (current)
...

(and eventual death). But if I set it to 1,2,3,4, or 5 recorders, it succeeds almost immediately.

@tsteur commented on October 16th 2014 Owner

Thx for sending the log. Unfortunately, I couldn't reproduce no matter using 1,2,3,4,5,6 or 10 recorders. Tried it several times. Even added a website to my Piwik to make sure it is tracked. Also had a look at the code and could not find anything. I am using Python 2.7.3 by the way.

That's my output http://pastebin.com/U6x8atf4

Also tried to import other logs with more entries from VisitorGenerator without any issues.

@diosmosis @mattab can you guys try to reproduce it as well?

@orlitzky commented on October 16th 2014

It certainly is perplexing. It fails with 6 and 7 recorders, but works with 8.9, and 10. Python here is 2.7.7. Strace was... not enlightening.

@mattab commented on December 1st 2014 Owner

Decreasing priority, feel free to comment if you also experience this issue with high number of recorders.

@mattab commented on March 12th 2015 Owner

Issue was moved to the new repository for Piwik Log Analytics: https://github.com/piwik/piwik-log-analytics/issues

refs #7163

This Issue was closed on March 12th 2015
Powered by GitHub Issue Mirror