@sudeshg opened this issue on August 7th 2014

Hi,

How can i create custom log format in piwik. Actually i am unable to load log file through log_analytics.py script. I think i have to create new log format.

Sample log line: xxx.xxx.xxx.x - - [21/Dec/2013:04:11:59 +0000] "GET /cds/phQoANRmI3SvEcHzQurM3vMGqI5u6iAYcEGjbwLrPZOLVbcOeys-qSavG9Fz03sZ1PAZkys48rhfHPrG4qhwZGB31Q5b4IdAgDaMYJO8inpJntM.?id=b2YxxoncjD7B-jzZLa75&expirationTime=1387858318631 HTTP/1.1" 200 6218 "-" "-" 15

Error Message: Fatal error: cannot automatically determine the log format using the first 100000 lines of the log file. Maybe try specifying the format with the --log-format-name command line argument.

Could you please send me documentation for creating custom log formats?

thanks, Sudesh

@sgiehl commented on August 7th 2014

Is that any standard log format or something you defined? If it is an standard we might consider to add it to core. Otherwise please have a look at the script itself: https://github.com/piwik/piwik/blob/master/misc/log-analytics/import_logs.py#L256 You may need to add your custom format on your local copy

@ocontant commented on August 7th 2014

I have the same issue with a what seems to be a legit common log format. Only mine had an extra space between the - - after the IP. I had to sed them out. I'm also looking to fix it in import script instead of using sed for 3To of gz file.

Do you think you could revise the regex of the common log format and take care to ignore extra join space? *like

 "  -  -  "

instead of

 " - - "

or if you don't want to put it in a review of your core, can you just give me the _COMMON_LOG_FORMAT regex or let me know how to add a new one if it is easy.



Thank you very much! And thank you for coding this very good apps :)

@mattab commented on August 9th 2014

@ocontan

Do you think you could revise the regex of the common log format and take care to ignore extra join space?

t :+1: if you would be able to provide pull request we would definitely merge it

@ocontant commented on August 10th 2014

Hi,

I tried, but my regex-fu is not enough strong. I think some of the syntax you used I’m not familiar with. My feeling is that the regex I would use to change it would not be as optimized as the one you are currently using.

If you could do it, that would be better in my opinion. Wish my regex-fu was as strong as the guys who made it in the current version. J

From: Matthieu Aubry [mailto:notifications@github.com] Sent: 9 août 2014 13:50 To: piwik/piwik Cc: Olivier Contant Subject: Re: [piwik] Custom log format for log analytics (#5947)

@ocontan

Do you think you could revise the regex of the common log format and take care to ignore extra join space?

t :+1: https://assets-cdn.github.com/images/icons/emoji/unicode/1f44d.png if you would be able to provide pull request we would definitely merge it

— Reply to this email directly or view it on GitHub https://github.com/piwik/piwik/issues/5947#issuecomment-51693615 . https://github.com/notifications/beacon/2682054__eyJzY29wZSI6Ik5ld3NpZXM6QmVhY29uIiwiZXhwaXJlcyI6MTcyMzIyNTgxMSwiZGF0YSI6eyJpZCI6MzkwNzA4MDZ9fQ==--15f2753e903f963483c498a81206817fe4ddf7e7.gif

@mattab commented on December 18th 2014

Hi @ocontant do you mind creating a new issue specifically for the "extra space" thing? we will definitely fix it. Thanks

This issue was closed on December 18th 2014
Powered by GitHub Issue Mirror