@mattab opened this Issue on October 7th 2015 Owner

Our goal is to create a new open source plugin that will let Super Users easily invalidate any report in Piwik. This will provide a user friendly way to make this frequent action.

Proposal

  • add a new menu entry in Administration > Invalidate reports (maybe there's a better location?)
  • the menu entry and functionality is only available to Super Users (since it can result in rather big performance issues if someone invalidates a very big long range on a large piwik DB)
  • the page displays:
    • Title Invalidate historical reports
    • Description:

You can invalidate all historical reports for a specific website, or for all websites, by clicking the button "Invalidate historical reports". When you invalidate historical reports, they will be re-processed from the raw logs the next time archiving will run. This is useful when you want to force Piwik to re-process historical data for all reports, for example when:

  • you created a new Custom Report and want the Custom Reports to be processed for all historical data
  • you created a new Funnel and want your Funnel reports to be processed for all historical data
  • you have modified the raw visitor information (for example by importing new visitor logs in the past) and want these changes to raw logs reflected in all your reports.
  • Website selector (would default on "All websites" maybe?) and segment editor.
  • a button "Invalidate all historical reports"
  • on click on the button, display a popover:

You are about to invalidate all historical reports, which will cause Piwik to re-process all historical reports the next time archiving run. Re-processing all historical reports may take a long time. Are you sure you want to invalidate all historical reports for [website $WebsiteName|All websites]?

  • if a segment was also selected, also append and for segment $segmentName
    • when no segment was selected, append and for all Segments.
    • on submit:
    • the API CoreAdminHome.invalidateArchivedReports is called with those parameters that user confirmed
    • For the date parameter, we can use $websiteCreationTime,today to invalidate all reports (method getCreationDateFor)
    • If "All websites" was selected, we can use $earliestWebsiteCreationTime,today to invalidate all reports (use method getMinMaxDateAcrossWebsites)
    • we set period to false so that all periods are invalidated
    • on successful submit, write a message such as:

Reports have been successfully invalidated. These reports will be reprocessed the next time reports archiving will run.
The CoreAdminHome.invalidateArchivedReports API returned:
$API_OUTPUT_HERE

  • ideally include a UI test for the initial page load, and for the page after successful submit of an invalidation
  • Update FAQ: https://piwik.org/faq/how-to/faq_155/ to link to this new plugin as an alternative to the API solution
  • host plugin at piwik/plugin-InvalidateReports or so

Notes

@gaumondp commented on October 8th 2015

Invalidate reports (maybe there's a better name?)

From a user POV, I would say we don't want to "invalidate" anything. We really want to "recreate reports" or re-generate", "reconstitute" reports. The action to invalidate is a side-effect hidden under the hood.

Warning popover

I know we can't estimate time or CPU needed but maybe add a generic warning for exemple if the period is more than X days if entire DB is > 1 GB ?

@RMastop commented on October 8th 2015 Contributor

I started to write a response this morning. The only reason I have used the invalidate feature, is because I needed to recreate an historic report, or archive a new segment for a certain period. It's more about the need to force archiving a segment or period. New segments are not archived from the very first date of tracking, for performance reasons. By being able to force archiving a fine grained set of data, we keep the load to a minimum and still be able to provide the requester their reports.

@mattab commented on November 10th 2017 Owner

Hi @sgiehl Updated the description/spec, could you take a look?

@sgiehl commented on November 11th 2017 Member
  • For the date parameter, we can use $websiteCreationTime,today to invalidate all reports (method getCreationDateFor)
  • If "All websites" was selected, we can use $earliestWebsiteCreationTime,today to invalidate all reports (use method getMinMaxDateAcrossWebsites)

We can do that for the first version, but we defiantly need to do that better way later. If purging old log data is active, invalidating old reports could remove data that can't be reprocessed. So maybe it would be better to check for which day log data is available and leave all older data untouched...

@mattab commented on November 12th 2017 Owner

So maybe it would be better to check for which day log data is available and leave all older data untouched...

It should already be implemented, see method removeDatesThatHaveBeenPurged and findOlderDateWithLogs, but it would be great to double check this works as expected in this case?

@sgiehl commented on November 13th 2017 Member

Plugin will soon be published: https://github.com/piwik/plugin-InvalidateReports

This Issue was closed on November 17th 2017
Powered by GitHub Issue Mirror