This started about a week ago, I don't think anything changed here. Every night we run,
import_logs.py --enable-http-errors --enable-http-redirects --enable-bots --enable-static --recorders=6 --url=https://analytics.example.com/ <all logs from yesterday>
Now it regularly gets stuck:
Parsing log /var/log/apache2/example.com/www/access/access-2014-10-14.log... 6886 lines parsed, 6284 lines recorded, 79 records/sec (avg), 148 records/sec (current) 6886 lines parsed, 6461 lines recorded, 80 records/sec (avg), 177 records/sec (current) 6886 lines parsed, 6676 lines recorded, 82 records/sec (avg), 215 records/sec (current) ... 6886 lines parsed, 6676 lines recorded, 25 records/sec (avg), 0 records/sec (current)
and eventually fails with the following:
6886 lines parsed, 6676 lines recorded, 25 records/sec (avg), 0 records/sec (current) Fatal error: '' You can restart the import of "/var/log/apache2/example.com/www/access/access-2014-10-14.log" from the point it failed by specifying --skip=5 on the command line.
Is that the sixth line of the log file? If so, there's nothing weird in it.
The problem began on 2.7.0, but persists after an upgrade to 2.8.0.
Is there a chance that you can send us the log with like the first 10 lines? In case it is reproducible with only 10 lines... If so, please send to hello at piwik.org.
It's on its way, but some more testing suggests that the log file isn't (entirely) to blame. With six recorders,
202 lines parsed, 190 lines recorded, 94 records/sec (avg), 77 records/sec (current) ... 202 lines parsed, 190 lines recorded, 5 records/sec (avg), 0 records/sec (current) ...
(and eventual death). But if I set it to 1,2,3,4, or 5 recorders, it succeeds almost immediately.
Thx for sending the log. Unfortunately, I couldn't reproduce no matter using 1,2,3,4,5,6 or 10 recorders. Tried it several times. Even added a website to my Piwik to make sure it is tracked. Also had a look at the code and could not find anything. I am using Python 2.7.3 by the way.
That's my output http://pastebin.com/U6x8atf4
Also tried to import other logs with more entries from VisitorGenerator without any issues.
@diosmosis @mattab can you guys try to reproduce it as well?
It certainly is perplexing. It fails with 6 and 7 recorders, but works with 8.9, and 10. Python here is 2.7.7. Strace was... not enlightening.
Decreasing priority, feel free to comment if you also experience this issue with high number of recorders.
Issue was moved to the new repository for Piwik Log Analytics: https://github.com/piwik/piwik-log-analytics/issues