@donjan opened this Issue on October 9th 2014

Trying to get direct piping of log analytics to work on an Apache 2.4.7 / Ubuntu 14.04 system.
From looking at issues https://github.com/piwik/piwik/issues/3757, https://github.com/piwik/piwik/issues/3163 and https://github.com/piwik/piwik/issues/6200 something like this in the Apache conf file should work:

CustomLog "|/usr/bin/python -u /var/www/piwik/misc/log-analytics/import_logs.py --url=http://testsrv/piwik/ - --idsite=1 --log-format-name=common_vhost" vhost_combined

But this doesn't add any data to Piwik (checked in PMA, and accordingly nothing in the web UI). Adding --output=/var/log/piwik/test.log or checking Apache's error.log doesn't provide more clues (I can provoke errors in both files by using incorrect syntax or unknown option names).

The access log is in the vhost_combined format and the following as root does work:

tail -n1 access.log | /usr/bin/python -u /var/www/piwik/misc/log-analytics/import_logs.py --url=http://testsrv/piwik/ - --idsite=1 --log-format-name=common_vhost

It correctly adds the last page hit ("1 requests imported to 1 sites"), but when omitting the last option (--log-format-name=common_vhost) it gives:

0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log (stdin)...
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Fatal error: cannot automatically determine the log format using the first 100000 lines of the log file. 
Maybe try specifying the format with the --log-format-name command line argument.

I've tried various permutations, verbosely copying from misc/log-analytics/README.md and the threads linked above (especially since https://github.com/piwik/piwik/issues/3757 concludes that something somehow does work), but can't get a clear handle on what is going wrong. It looks as if it should be working, so any hints or an up to date how-to are appreciated.

@donjan commented on October 12th 2014

@mattab
What would be an estimate for "long term"?
Log analytics is the main option for me since PHP tagging on various 3rd party web apps is cumbersome and JS tracking isn't acceptable. Log piping instead of rotation would make me switch to Piwik.

@gaumondp commented on October 14th 2014

@donjan , long term in Piwik's universe means 3 to 5 years.

Source: http://piwik.org/roadmap/

@mattab commented on October 17th 2014 Owner

@donjan if you need something faster you are welcome to sponsor developer time on it. Please get in touch at: http://piwik.org/consulting/ if you're interested

@mattab commented on March 12th 2015 Owner

Issue was moved to the new repository for Piwik Log Analytics: https://github.com/piwik/piwik-log-analytics/issues

refs #7163

This Issue was closed on March 12th 2015
Powered by GitHub Issue Mirror