Yesterday, we lost connectivity to our stats Piwik server. When we started investigating the issue, we realized that the hard drive had literally 0MB free. We were surprised, because we have a website with moderate traffic.
Actually, with further investigation, we have seen that the database was not taking more than 80MB. The problem was that a giant file was stored in the folder /var/www/tmp/climulti. There were two files, a .pid file and a .output file, and the latter was taking as much as 69GB of hard drive storage. Unfortunately, as the machine was barely usable because of the lack of free space, I've not been able to compress it, send it to an external server or open it. However, I could see the first line and it seemed to be an error log, yet not similar to those of Apache Server.
I assume some kind of infinite-loop error occurred and it managed to take all the free storage in the hard drive. However, I have no idea of what could happen. Any idea?
PS: I know it is difficult to know what could happen given that I could not provide the contents of the file. If only we knew what kind of information is saved in that folder, it would be Ok.
We were using Piwik 2.8.3, the latest version up to now.
The tmp/climulti folder holds process IDs and output for php processes that Piwik launches. This is done for concurrency, so the output will be for commands like
php /path/to/piwik/console climulti:request ?somepiwikquerystring which invokes a Piwik request on the command line. It's mainly used during archiving.
I assume you've setup a cron to do archiving? Can you show us the crontab entry? Can you also tell us more about the php executable you use to run archiving?
Any information about your php config & server could also be helpful.
and after this is solved maybe someone can provide a fix to not bottleneck our servers
To prevent such issues we could maybe add a check in Proccess::hasFinished and declare a process as finished once the file size of
.pid is > 1MB and
.output is > eg 100MB. The size should "usually" never exceed 1MB I think. I heard something similar maybe 4 months ago in a chat so this does not seem to be the only user having this problem.