./console core:archive --force-all-websites --force-date-range=2015-06-01,2015-06-25
which for some reason quits in the middle of the list of 20 sites.
Later on, run:
./console core:archive --force-all-websites --force-date-range=2015-05-01,2015-05-31
the output of which gives
Will ignore websites and help finish a previous started queue instead. IDs: 11, 12, 13, 14, 15, 16, 17, 18, 19, 20
while the rest of the output will be like
Archived website id = 15, period = year, 4 segments, 1352807 visits in years included in: 2015-05-01,2015-05-31, Time elapsed: 495.273s
I hereby conclude that the previous job was finished but with the new parameters (date range). It means that my instruction to archive May was only half honoured while the one to archive June was never actually finished.
I'm not quite sure if I understand. I think one bug might be that if someone uses
--force-all-websites we should not process a previously started queue. This might have been regressed.
Also the queue should maybe not used in general if a
--force-data-range is given since it would be not really correct. Eg the queue might have been started with a different data range.
I think one bug might be that if someone uses --force-all-websites we should not process a previously started queue. This might have been regressed.
Also the queue should maybe not used in general if a --force-data-range is given since it would be not really correct.
Good point! I think it is the right fix, to not use the shared queue when
--force-all-websites are used at the same time.
Maybe there are other parameters that when set, should not use a shared queue?
By the way there is also a related problem that whenever we run the archiver we set the archiving time at the end: https://github.com/piwik/piwik/blob/2.14.3/core/CronArchive.php#L399
This means when executing the archiver the next time we will just look for that
lastSuccesRunTime but we may have actually only processed one site at
lastSuccessRunTimestamp, all other sites have a completely different
lastSuccessRunTimestamp. This can also lead to some side effects when using multiple archivers at the same time. When using
force-siteids etc we should maybe not set the
lastSuccessRunTime, only when actually all websites were processed.
Maybe it might be worth having a look at this in 2.15 as it will be the stable version for a year @mattab
@tsteur thanks for suggestion, moving to 2.15.0 :+1:
Let me know how you see a fix for this, or we can discuss in the PR.
I'm not working on this one