@akrus opened this Issue on July 27th 2015

Hi,

I have Piwik running (latest stable version) and real-time segmented search is quite slow. I've changed this to use archived data (cron) and executed full re-archive using 'piwik/console core:archive --url=http://piwik -vv -
-force-all-periods=315576000 --force-all-websites --force-date-last-n=10000'. This took one day to complete, but as I can see on the graphs, e.g. Visits over Time, in one segmented search there is no data between Apr 2014 and Jun 2015 (so I have Mar 2014 and Jul 2015), in another - Jan-Jun 2015 is missing and so on. Why this could happen? Executing console core:archive again doesn't change anything. New days are appearing fine though.

See screenshots attached.
piwik_daily
piwik_monthly
piwik_segment_settings

@tsteur commented on July 27th 2015 Owner

but as I can see on the graphs, e.g. Visits over Time, in one segmented search there is no data between Apr 2014 and Jun 2015 (so I have Mar 2014 and Jul 2015), in another - Jan-Jun 2015 is missing and so on

I'm not sure if I understand the issue right. It is normal that two different segmented searches result in different data. It depends on the query and data whether it is normal that there is no data or not. Or do you mean when executing the same segmented search multiple times you get a different result?

Can you let us know the segments that you use?

@akrus commented on July 27th 2015

Same segmented search with "segmented reports are processed in real time (default)" has all the information available, see graph below:
piwik_realtime_daily
When I change this to use archives - the data is missing (graphs in first post) and I can't find out why as no errors appear during archiving.

@bodomic commented on July 30th 2015

That's a good question, I experience the same issue too. Realtime works fine, archived one is patially lacking data.

@tsteur commented on July 30th 2015 Owner

I have no clue how this can happen right now. Maybe someone else does? It would be probably beneficial to get access to a database or to get a dump of a database to debug this issue - if possible. If that's possible for someone, please email us at "hello at piwik.org" and leave a comment here (in case it goes into spam folder)

@quba commented on July 30th 2015 Contributor

Do you monitor archiving logs?

Such situation may happen if there were archiving errors in the past (e.g. memory limit and others). It's possible that there are incomplete archives marked as done.

There are 2 solutions to confirm:

Hope that helps.

@akrus commented on July 30th 2015

Okay, increased memory_limit to 2G, dropped data for 2015_02 and launched archiving again, let's see...

@akrus commented on July 30th 2015

Now it archived fine, how to deal with this situation without reading logs all the time?

@quba commented on July 30th 2015 Contributor

Make sure that you have memory_limit set properly (2G should be fine but it depends on your traffic).

You can also set a basic monitoring (even cron job) that will grep the log file searching for errors and send an e-mail if needed.

@akrus commented on September 3rd 2015

Alright, I dropped all tables in MySQL related to archives, re-indexed everything and now the data is there, but I have another problem - whenever I add any new segmented search (e.g. new region) and switch it to use 'pre-processed data' it shows nothing and re-indexing doesn't help, the only solution is to again clear everything and initialize it from scratch.
Doesn't look like correct behavior...

@RMastop commented on September 3rd 2015 Contributor

I have the same issue here,
The default value of process_new_segments_from = "beginning_of_time" is still in place.
However I see data not being archived.

One solution is to delete the tables, but for large sites this is not really an option.
You don't want to delete a month of archived data for 50+ sites because of one segment failing.
Invalidate the data is not the easiest solution, because you have to create a URL with comma separated dates or a bash file that calls the URL for every date you would like to invalidate. (It would help to be able to invalidate per range, and segment)

@mattab commented on September 18th 2015 Owner

whenever I add any new segmented search (e.g. new region) and switch it to use 'pre-processed data' it shows nothing and re-indexing doesn't help, the only solution is to again clear everything and initialize it from scratch.

@akrus Can you explain exactly the step to reproduce? it seems to work fine for us

I have the same issue here, The default value of process_new_segments_from = "beginning_of_time" is still in place. However I see data not being archived.

@RMastop do you still have this issue? how can we reproduce this?

@RMastop commented on December 9th 2015 Contributor

Hi @mattab,
We reproduced the issue. Will write the details about it to Piwik.pro support.

@mattab commented on April 11th 2016 Owner

Hi @RMastop @akrus Could you please try again with 2.16.1 stable? we fixed a related issue recently in https://github.com/piwik/piwik/issues/9468 https://github.com/piwik/piwik/issues/10022 which may have fixed this issue. let us know please if you are still affected or if it's fixed using 2.16.1...

@akrus commented on April 11th 2016

Upgraded now, should I execute full re-index or usual cron job should fix this?

@mattab commented on April 11th 2016 Owner

Maybe wait 24 hours and see if it fixed itself already. If some old data is still not processed, you can re-process it all with the command in: https://piwik.org/faq/log-analytics-tool/faq_17994/

If you are still seeing the issue we'll investigate more and get to the bottom of it...

(added Major tag since when data doesn't show or is not tracked properly in Piwik, we consider it a major issue.)

@RMastop commented on April 11th 2016 Contributor

Hi @mattab,

I'm sorry, I'm not allowed to connect to the instance where it happened.
I'll try to contact someone to validate.

Cheers,
Richard

@mattab commented on July 18th 2016 Owner

whenever I add any new segmented search (e.g. new region) and switch it to use 'pre-processed data' it shows nothing and re-indexing doesn't help, the only solution is to again clear everything and initialize it from scratch.

@akrus could you please create a new issue? I will mark this issue as solved for now, but if you still experience this issue in Piwik 2.16.2 or later, please open a new issue and we will investigate!

@XinfinityoO commented on September 4th 2016

I m having similar issues ..i cant see old records

@mattab commented on September 27th 2016 Owner

@XinfinityoO please create a new issue and write down as many details as possible

This Issue was closed on July 18th 2016
Powered by GitHub Issue Mirror