@mattab opened this issue on March 10th 2008

Currently data can be pushed in the database using the piwik.php script, called from the piwik.js tag.

For example an HTTP request that stores a page view in Piwik looks like:


We want to improve the way Piwik logs data at tracking time. - Piwik will now create 'access logs', similar to apache access logs, containing all the REQUEST details (url, the 'piwik' cookie which depends on #409, user agent, referer url, IP, language, POST data, other $_SERVER, etc.). - Every 10s (or 30s or 1min) the 'Tracking Bulkloader' will be triggered by the Maintenance task (see #1184). - it will connect to the DB once, then read all log lines and process visits in memory, - creating flat files for the DB updates, - eventually using memcache for visits/pages/options/cookies data store, - then bulk inserts/updates visits/pages/conversions and cookies - The process would log reports2010-09-08 04:03:02 - Loaded 1405 visits, 45000 page views, 345 goals - Duration 32s - Logs since 2010-09-08 04:02:52. How do we handle timeout errors? memory errors? - Persists status of 'bulk logs loaded' in piwik_option - disconnect DB

Speed of tracking data load would be greatly improved and would compensate the performance loss resulting from the cookie loss in #409.

The log replay script would work in several modes - replay a single visit using a HTTP request containing the input data (the content of one log line from the log file). - The script API to record stats would become public and be documented. This will allow any user to record stats in Piwik from any source (mobile phones, php apps, desktop apps, etc.). - replay a set of visits given a log filename, will read the log file and load it in the DB # Use cases for tracking logs loader

This way, we can use the log replay script in the following use case - Tracker performance improvement described above: Every 10s (or 30s or 1min) the 'Tracking Bulkloader' loads all logs at once, which is much faster than connecting to the DB/updates/inserts at every page view. - There will also need to be a new 'Super user' setting - Enable tracking logs bulk loader - Record visits in DB every 10s - Enable Memcached wrapper - Performance testing replaying existing real logs make it easier to test performance changes in different releases and when doing code updates. See #2000

Notes - Maybe the tracking code has to be modified to remove the logic that 'selects' or tries to 'match a visitor md5config to a previous visit' as it can be done using the cookie store (#409) I believe. - can we use an existing code to parse apache logs in an efficient way, and which would work with several log formats automatically (including windows IIS formats)? - see also the DB schema even if it's outdated, most fields and tables are unchanged.

@robocoder commented on August 28th 2009

Attachment: Proposed plugin ActiveTrack.zip

@mattab commented on January 25th 2012
@mattab commented on January 25th 2012
@odinho commented on May 9th 2009

This would make Piwik easy to prefer over Analytics, as admins then can imports years worth of logs and data into Piwik and see the trends more easily.

I would be able to make the people at work to use Piwik instead of Google Analytics if I can import the common apache-logs into Piwik.

@robocoder commented on July 24th 2009

(In [1330]) Fixes #876 - Piwik_Tracker_Visit should not validate uninitialized $this->request in constructor; make it possible to push data (in the absence of a REST API; see #5554); fix duplicate function error messages when calling Piwik::printSqlProfilingReportTracker() more than once

@robocoder commented on July 24th 2009

Will tackle in conjunction with #762.

@robocoder commented on August 28th 2009

Proposed plugin attached. - API.php uses the Tracker.getNewVisitObject hook - ignore Tracker.php and API-alternate.php; these follow the implementation model of the (fake) visit generator are included for comparison only - @todo move translation(s) in lang/en.php into core - @todo add test of setVisitInformation() - @todo add return value to API methods and Piwik_API_Request() variant of existing tests

@robocoder commented on August 30th 2009
  • @todo investigate extending to handle reqts of TrackerSecondaryDb's replay script.
@mattab commented on August 31st 2009

really interesting! - in the API, I think there is an issue with the setVisitInformation - in a REST api because there is no state, all parameters need to be passed at each request, so we might need to add cookies/ip/timestamp in each function.. this is quite ugly but I'm not sure how else it could work, without introducing token responses, caching, etc... - getVisitObject move to the Controller instead of API?

Before putting the plugin in core, I think we should do the cleanup of hook names and come up with a consistent naming.

Please post here your updated versions - really looking forward to have a better look.

@mattab commented on January 18th 2010

vipsoft, did you try this plugin with real traffic and see if it was working? How does it deal with cookies? Would it work in the real case of a traffic replay from apache logs?

@mattab commented on February 11th 2010

set, I think this is a bug in the code. Also, I expect it doesn't deal well with cookies and therefore would leave to invalid statistics. This is an alpha/not ready for production release.

@mattab commented on May 28th 2010

See an example of a PHP function that creates a request to piwik.php



@halfdan commented on July 15th 2010

Will this module not create race conditions due to parallel writing to the logfile? In difference to Apache we don't have the file open the whole time - the file gets opened/closed every time piwik.php gets requested and writes data to it. Normal file system implementations don't care about simultaneous write, so we're gonna loose some log lines on high load piwik sites..

Even if there were a locking mechanism in the file system, it will probably screw the server load with high stat(), wait() and I/O.

@mattab commented on July 22nd 2010

moving to post 1.0 as it requires significant time / testing.

@mattab commented on January 8th 2011

halfdan, I think we will have to lock the file before writing a new line into it, then release lock. Hopefully there is a large margin before this ends up causing performance issues.

@mattab commented on January 8th 2011

I just did a quick test: with 200 concurrent requests, 1000 total requests, calling a php file acquiring a lock and appending 8k to a log file. If the lock is non blocking it fails to acquire it in approx 50% of requests and 1000 requests take ~0.15s. If the lock is blocking 1000 requests take 0.15 sec.

So the conclusion is that we are safe to write to log file as long as we lock, this is not going to be the bottleneck

I used this script with ab

    $fp = fopen("foo.txt", "a");
    if (flock($fp, LOCK_EX)) { // add | LOCK_NB to lock
        fwrite($fp, str_repeat(rand(1,9),4*1024) ."\n");
        flock($fp, LOCK_UN);
    } else {
        print "Could not get lock!\n";
@mattab commented on January 8th 2011

Random thought: to be safe implementing this change, it would be nice to research code coverage status of our tracker unit tests. I believe it could be pretty easy to cover close to 100% but we just don't know what is not tested with our current integration tests.

@mattab commented on January 13th 2011

Once we implement bulk loading, hopefully we can get rid of the Tracker/DB custom classes and use Zend all the time?

@Fil commented on January 23rd 2011

I need this (just mentioning it to follow the ticket changes :-) )

@robocoder commented on March 3rd 2011

I assume during import, there is no browser-triggered archiving or scheduled tasks.

It's been a while since I've looked at the tracker performance (the biggest issue for me was a memleak), so it would be timely to review, since Piwik 1.2 will have first party cookies.

It would be nice if the importer could also parse a custom log format (if defined), or handle the tables created by mod_log_dbd or mod_log_sql.

@robocoder commented on March 6th 2011

The gethostbyaddr() call in the Provider plugin can be a bottleneck. There are some interesting ideas in the user contributed notes for http://php.net/gethostbyaddr that we could (1) test for during installation to see which methods work and how well for a given environment, and (2) use the best method at runtime.

@mattab commented on March 7th 2011

jdhildeb, this is very interesting. We will very soon (few months) load logs in parrallel and I didn't take in consideration that this might be a bottleneck. Alternatively, we can do the DNS lookup on the piwik.php hit and log it. This might be easier and will not cause issues with DNS slowness until many requests per second. Also, the DNS server should be locally cached to the server itself (not sure how this is called again...)

@mattab commented on March 7th 2011

Kay, no this is what DNS cache server is for..

@robocoder commented on March 7th 2011

re: comment:48

Right now, gethostbyaddr() is called for a new visit. I'm opposed to moving it to piwik.php because it would mean calling gethostbyaddr() on every request. - Where PHP's gethostbyaddr() is able to resolve the IP address, it typically takes about 0.04 seconds on my box/network. - However, if PHP's gethostbyaddr() is unable to resolve the IP address, it can take a long time (e.g., 5 seconds), e.g.,

time php -r 'var_dump(gethostbyaddr(""));'
  • Moreover, it doesn't appear that the failure is cached, so it's always a "miss" -- i.e., a subsequent call to gethostbyaddr() for the same IP address will take just as long.

I've created ticket #2152 to investigate performance enhancements for gethostbyaddr().

@robocoder commented on March 7th 2011

re: comment:51

Before we add more caching to the tracker, we need to follow-up on #735 (tracker mem leak).

Caching geoip and provider information may have limited benefit because these plugins are called for a new visit (session).

@mattab commented on March 31st 2011

jdhildeb, the idea of the bulk import will be to keep using piwik.php as it is, but instead of logging in the DB, it will log a "pretty log" in a file, a bit like an apache log but enriched with all the data piwik.php receives. - Regarding the DNS lookups My idea was to have the server do the dns lookup before recording in the file, so that when we load data in bulk, we don't have to do any dns requests.

Alternatively, we could disable the DNS lookup since it is not so useful and costly.. - Also, an idea is to make the queue logging system very trivial so that other scripts could directly log to the queue from apache/piwik logs rather than go through piwik.php again - Queue system should be pluggable: file, reddis, or even mysql driver available

@mattab commented on November 3rd 2011

lorieri, please contact me at matt att piwik.org and we can discuss these things. WE are hoping to work on Scalable Piwik in the next few months and will create a brainstorm group. Email me with your traffic details and use case. Cheers

@mattab commented on January 25th 2012

From email

There was a problem registering visitors at a higher load.
Such problem has already been described in #5554.
We solved this problem by recording visitors in syslog and
processing of the log. Perhaps that can help you. 
Don't hesitate to contact me if you have any questions!

See attached file.

@mattab commented on July 29th 2012

The API to import Bulk Visits and Pages in Piwik is now implemented! See: #3134

It's now possible to import ie. 100 visits or 1000 pageviews in one POST http request

@mattab commented on December 14th 2012

The feature is implemented.

We have integration tests as well (ImportLogsTest)

See the Tracking API reference in: http://piwik.org/docs/tracking-api/reference/

in particular the section "Advanced: Bulk Tracking Requests"

@mattab commented on November 24th 2014

We're working on Reddis for storing tracker requests, see #6075

This issue was closed on December 14th 2012
Powered by GitHub Issue Mirror