@d-bo opened this issue on July 24th 2014

Apache logs batch imported using command:

python import_logs.py --url=http://xxxxx/piwik/index.php $f --idsite=2 --enable-http-errors --enable-http-redirects --enable-static --enable-bots

Response cut:

Processing '/home/vathsven/www-logs/apache-access.100.gz' file...
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log /home/vathsven/www-logs/apache-access.100.gz...
1200 lines parsed, 400 lines recorded, 392 records/sec (avg), 400 records/sec (current)
1723 lines parsed, 1000 lines recorded, 492 records/sec (avg), 600 records/sec (current)
2400 lines parsed, 1600 lines recorded, 524 records/sec (avg), 600 records/sec (current)
2937 lines parsed, 2200 lines recorded, 541 records/sec (avg), 600 records/sec (current)
3600 lines parsed, 2800 lines recorded, 552 records/sec (avg), 600 records/sec (current)
4200 lines parsed, 3400 lines recorded, 559 records/sec (avg), 600 records/sec (current)
4600 lines parsed, 3800 lines recorded, 536 records/sec (avg), 400 records/sec (current)
5400 lines parsed, 4600 lines recorded, 567 records/sec (avg), 800 records/sec (current)
5800 lines parsed, 5000 lines recorded, 548 records/sec (avg), 400 records/sec (current)
6505 lines parsed, 5800 lines recorded, 572 records/sec (avg), 800 records/sec (current)
7106 lines parsed, 6400 lines recorded, 574 records/sec (avg), 600 records/sec (current)
7600 lines parsed, 6800 lines recorded, 559 records/sec (avg), 400 records/sec (current)
8200 lines parsed, 7400 lines recorded, 562 records/sec (avg), 600 records/sec (current)
8800 lines parsed, 8000 lines recorded, 564 records/sec (avg), 600 records/sec (current)
9600 lines parsed, 8800 lines recorded, 579 records/sec (avg), 800 records/sec (current)
10400 lines parsed, 9600 lines recorded, 592 records/sec (avg), 800 records/sec (current)
11000 lines parsed, 10200 lines recorded, 592 records/sec (avg), 600 records/sec (current)
11800 lines parsed, 11000 lines recorded, 603 records/sec (avg), 800 records/sec (current)
11828 lines parsed, 11600 lines recorded, 602 records/sec (avg), 600 records/sec (current)

Purging Piwik archives for dates: 2014-04-14

To re-process these reports with your newly imported data, execute the following command: 
$ /path/to/piwik/console core:archive --url=http://example/piwik/

Reference: http://piwik.org/docs/setup-auto-archiving/ 

Logs import summary
-------------------

    11828 requests imported successfully
    171 requests were downloads
    0 requests ignored:
        0 invalid log lines
        0 requests done by bots, search engines, ...
        0 HTTP errors
        0 HTTP redirects
        0 requests to static resources (css, js, ...)
        0 requests did not match any known site
        0 requests did not match any requested hostname

Website import summary
----------------------

    11828 requests imported to 1 sites
        1 sites already existed
        0 sites were created:

    0 distinct hostnames did not match any existing site:



Performance summary
-------------------

    Total time: 19 seconds
    Requests imported per second: 595.88 requests per second

Cannot see results on the dashboard. Nothing really imported ?

@mattab commented on July 24th 2014

To re-process these reports with your newly imported data, execute the following command: $ /path/to/piwik/console core:archive --url=http://example/piwik/

@d-bo commented on July 24th 2014

seems to be not re-processing:

[5dbf0] ---------------------------
[5dbf0] INIT
[5dbf0] Piwik is installed at: http://xxxxxxx/piwik/index.php
[5dbf0] Running Piwik 2.4.1 as Super User
[5dbf0] ---------------------------
[5dbf0] NOTES
[5dbf0] - If you execute this script at least once per hour (or more often) in a crontab, you may disable 'Browser trigger archiving' in Piwik UI > Settings > General Settings. 
[5dbf0]   See the doc at: http://piwik.org/docs/setup-auto-archiving/
[5dbf0] - Reports for today will be processed at most every 10 seconds. You can change this value in Piwik UI > Settings > General Settings.
[5dbf0] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
[5dbf0] - Will process 0 websites with new visits since 7 days 0 hours 
[5dbf0] - Will process 1 other websites because some old data reports have been invalidated (eg. using the Log Import script) , IDs: 2
[5dbf0] ---------------------------
[5dbf0] START
[5dbf0] Starting Piwik reports archiving...
[5dbf0] Archived website id = 2, period = day, 0 visits in last last52 days, 0 visits today, Time elapsed: 1.831s
[5dbf0] Archived website id = 2, period = week, 0 visits in last last104 weeks, 0 visits this week, Time elapsed: 4.966s
[5dbf0] Archived website id = 2, period = month, 0 visits in last last52 months, 0 visits this month, Time elapsed: 1.898s
[5dbf0] Archived website id = 2, period = year, 0 visits in last last7 years, 0 visits this year, Time elapsed: 0.837s
[5dbf0] Archived website id = 2, 4 API requests, Time elapsed: 9.538s [1/1 done]
[5dbf0] Done archiving!
[5dbf0] ---------------------------
[5dbf0] SUMMARY
[5dbf0] Total visits for today across archived websites: 0
[5dbf0] Archived today's reports for 1 websites
[5dbf0] Archived week/month/year for 1 websites
[5dbf0] Skipped 1 websites: no new visit since the last script execution
[5dbf0] Skipped 0 websites day archiving: existing daily reports are less than 10 seconds old
[5dbf0] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
[5dbf0] Total API requests: 4
[5dbf0] done: 1/1 100%, 0 vtoday, 1 wtoday, 1 wperiods, 4 req, 9652 ms, no error
[5dbf0] Time elapsed: 9.652s
[5dbf0] ---------------------------
[5dbf0] SCHEDULED TASKS
[5dbf0] Starting Scheduled tasks... 
[5dbf0]  No task to run
[5dbf0] done
INFO CoreConsole[2014-07-24 10:10:06] [5dbf0] ---------------------------

nothing at dashboard. apache logs are 'default' formatted and > 100Mb.

This issue was closed on July 24th 2014
Powered by GitHub Issue Mirror