@mattab opened this Issue on January 6th 2015 Owner

The goal of this issue is to create a new FAQ that answers "How do I configure Redis to store tracking requests, sessions and/or caches, for better scalability?".

Some ideas of the answer to the question:

  • list requirements as currently listed in QueuedTracking: redis 2.8+, phpredis
  • explain how to enable the features:
    • Tracker: use QueuedTracking plugin
    • Sessions: change config file?
    • Caches: change config file?
@mattab commented on March 17th 2015 Owner

I'm removing some content from this other FAQ How do I configure Piwik Tracking for high reliability?

Content removed:


**How do I improve reliability of Piwik tracking as to not be dependant on the Mysql servers being up and running at all times?** 
Follow these steps:

*   On the Piwik tracking server(s), disable synchronous tracking in the config file: in config.ini.php, below the `[Tracker]` section, set: <pre>record_statistics=0</pre> 

*   At this stage, your web server(s) still receive the piwik.php requests but will directly return the 1*1 transparent gif beacon. These requests to piwik.php are still logged into your webserver access log files. They are not stored in your  Piwik database yet.
*   Regularly (eg. every hour or once a day), merge all these Tracking servers access log files into one file. This merged access log file will contain all requests to piwik.php including for each: the user agent, IP address, datetime, the piwik.php? URL, etc.
*   Sort this merged log file by date. 

*   To import the log file, in the Piwik server running the import, edit config.ini.php, and below the `[Tracker]` section, set: <pre>record_statistics=1</pre> 

*   Import this log file by running the [Log Analytics tool](http://piwik.org/log-analytics/). Use the option <pre>--replay-tracking</pre> This option will "replay" the piwik.php tracking requests into Piwik, effectively storing the visitors, visits, pageviews and goals data into your Mysql Database. Learn more about replaying logs [in this FAQ](http://piwik.org/faq/log-analytics-tool/faq_19221/).

*   Run the [archive pre-processing](http://piwik.org/docs/setup-auto-archiving/) after the visitors were successfully imported.

**Advantages:**

*   Tracker does not depend on the Mysql server being up. If Mysql is down, tracking requests are still collected in the server access log files. Piwik is more reliable.

**Challenges:**

*   If you have several frontends servers collecting the piwik.php tracking pixels requests, then make sure you merge all files prior to importing them with [Log Analytics](http://piwik.org/log-analytics/). 
*   After merging all server log files, sort them by date to ensure a given user's requests are always in the right order prior to import in Piwik. Sorting can be complex on very large log files.
*   Check your log rotation works well. Make sure not to replay the same log data multiple times. Piwik does not de-duplicate records and will import everything thrown at the [Tracking API](http://developer.piwik.org/api-reference/tracking-api).
*   Unless you import the log files often (eg. every minute), you will lose the [Real time](http://piwik.org/docs/real-time/) capability of Piwik. Typically, one would import logs every hour or every day.
*   Do not run the cron archiving process until after all visitors have been imported for the previous date. If you run the archive.php process too early (before visitors are imported for yesterday or past dates), then analytics data will be incomplete, and you would have [to re-process the report data](http://piwik.org/faq/how-to/faq_59/).
@mattab commented on March 17th 2015 Owner

Created FAQ: How do I configure Piwik to use Redis for better scalability?

and edited this FAQ to link to the new one: How do I configure Piwik Tracking for high reliability?

(removed content is in above comment)

@mattab commented on March 17th 2015 Owner

I chose not to document for now how to use Redis for Sessions and Caches. @tsteur maybe we should consider documenting this if it works already, thoughts?

@tsteur commented on March 17th 2015 Owner

It does work already, I'm not sure what you need?

This Issue was closed on March 17th 2015
Powered by GitHub Issue Mirror