@anonymous-piwik-user opened this Issue on April 4th 2012

Hi Piwik-Team

I'm new to the apache and web-analytics world and i received the task to migrate from urchin 6.0 to piwik.

We use a vm machine as a piwik server with 4 cpus and 4gb ram.
After installing piwik 1.7.2 RC6 i've tried to import a 200MB apache logfile ( over 1 Millions of lines ).

The records/sec is between 20 - 40... and take over 10h to import! i've tried to change the amount of recorders to the suggested value ( recorders = cpu's -1 ) but this changed nothing :-(

i don't use any special options:
python import_logs.py --url=http://piwiksrv --show-progress --idsite=6 --config=/var/www/piwik/config/config.ini.php /root/wwwlogs/sito_test_log/sitobps_access_log --recorders=3

cut out from import process:
1011393 lines parsed, 915958 lines recorded, 26 records/sec
1011393 lines parsed, 915983 lines recorded, 25 records/sec
1011393 lines parsed, 916007 lines recorded, 24 records/sec
1011393 lines parsed, 916032 lines recorded, 25 records/sec
1011393 lines parsed, 916058 lines recorded, 26 records/sec
1011393 lines parsed, 916084 lines recorded, 26 records/sec
1011393 lines parsed, 916110 lines recorded, 26 records/sec
1011393 lines parsed, 916138 lines recorded, 28 records/sec
1011393 lines parsed, 916164 lines recorded, 26 records/sec
1011393 lines parsed, 916190 lines recorded, 26 records/sec
1011393 lines parsed, 916215 lines recorded, 25 records/sec
1011393 lines parsed, 916238 lines recorded, 23 records/sec
1011393 lines parsed, 916263 lines recorded, 25 records/sec
1011393 lines parsed, 916286 lines recorded, 23 records/sec
1011393 lines parsed, 916309 lines recorded, 23 records/sec

also during the import process the top output doesn't show any high io waits, load ...

any tips you can give me?

Thx and regards

Ivan Torretti

Keywords: log import

@mattab commented on April 4th 2012 Owner

Thanks for the feedback. If you know Python it would be great to profile and find out why it is so slow on your system. You should achieve 100 requests per second with standard server like yours.

PLease post your feedback on the official ticket for this script: #703 and we appreciate any kind of debugging you can do!

This Issue was closed on April 4th 2012
Powered by GitHub Issue Mirror