@mattab opened this issue on January 3rd 2008

Update: see follow up issue https://github.com/piwik/piwik/issues/5246

Most popular pages/keywords/etc. are popular every day, and it is hard to know what changed since the day before. What would be interesting would be to visualize: - Which pages significantly increased/decreased? - Which URLs links/SE keywords brought me significantly more/less traffic compared to last period?

This could help answer questions such as: - Why did the website traffic suddenly decrease? Traffic from some Keywords/Referers significantly decreased, and some pages significantly were less seen. - Similar with answering "Why did traffic suddenly increase?" Why are my Ecommerce orders growing so much, how do I know the overall positive contributors explaining our growth? # User interface updates for new reports & hints ## New report: Overall Movers and Shakers

This report would have to query in particular: page URL, page Titles, Keywords, websites, campaigns, maybe 8 reports, for current AND previous periods. Making 16 API calls should work fine and comparison would be pretty fast hopefully.

This report could be available as a widget showing the top 5 movers and shakers overall (increased keywords, pages, etc.). ## Extend existing report: Highlight Movers and Shakers

Currently reports display "Label", "Visits count" by default. There is a button to display 5 more columns and a view for Goal conversions metrics.

We could maybe improve at least the "basic" view showing only visits.

In this view we could for example display in a new column on the right, a green or red % increase, when there is a a major change (threshold set to +/- 10% for example). Otherwise leave blank. See also #291 ## Extend report: Display Movers & Shakers in all reports' footer

Below the report, or above?, we could display these rows from the report that have significantly changed since yesterday.

We could put threshold of minimum visits impacted = 20 minimum % increase or decrease = 2% for example?

So it would not display minor changes that are not interesting.

They could be displayed on top of the report

*Best countries*
(green sign) India (+50%) and Scotland (+33%)
(red sign) France (-33%) and Germany (-44%)
|---------|--------|
| Country | Visits |
|---------|--------|
| India   | 176    |
| Scotland| 54     |
| France  | 23     |
| Germany | 23     |
[....]
Next >

## Display evolution % next to all sparklines

Idea is to show the increase and decrease of Visits, Conversions, etc. basically next to each Sparkline. There is a proposal mockup at: http://issues.piwik.org/attachments/5711/sparklines.png # Implementation

To produce a report "Period VS Previous period" for each keyword, we would need to select all keywords from both periods (2 datatables), and compare both to extract significant increases/decreases.

The code would be optimized so that it returns in less than 2 seconds when calling 8 different APIs so comparing 16 datatables.

The algorithm will typically extract: - items with a growth of N % - new items (new keyword since yesterday) with at least M % - Never show items with less than P visits - By default, we should enable the display in the UI for only the very significant changes, with a high threshold of visits impacted. The feature should not bring noise if the changes are not that interesting.

New widget to show this great insights: - New smart widget to show what's changed in dashboard automatically based on last login date: #3250 # Similar tickets - compare 2 periods: #291 - show evolution for a given keyword/page: #534

@hpvd commented on March 3rd 2014
@mattab commented on May 27th 2014
@mattab commented on May 27th 2014

Attachment: insights overview.png

@mattab commented on May 27th 2014
@mattab commented on November 15th 2008

see GA hack on http://www.juiceanalytics.com/writing/keyword-trends-google-analytics-greasemonkey/

@mattab commented on September 19th 2011

To produce significant movers and shakers, the algorithm could maybe select "current month" dataset and compare it to yesterday if period=day, or to last week if period is week, for example to find out the items with an average very different from the number "this week" ?

See also relevant article: http://www.wired.com/magazine/2011/06/ff_feedbackloop/all/1

ALso interesting: http://www.juiceanalytics.com/writing/keyword-trends-google-analytics-greasemonkey/

    display_limit: This controls the maximum entries that each table will contain. This may be useful for large, sprawling sites.
    growth_tolerance: This is the percentage growth parameter. Changing it to .10, for example, will catch everything that has grown by 10%, as opposed to the default 50% and 20%, respectively.
    minimum_number_elements: This is a significance benchmark that can be used to limit what is displayed upon the screen. By default, only keywords with at least 10 elements are displayed upon the screen. Referring Sites does not have a minimum by default, but one can be set if desired.
    limit: Limit is more of an internal parameter that determines how many entries should be downloaded from Google in order to get the results that are visible here on the page. Lower the limit to increase speed. If the limit is set to a very high number, you will get the largest result set, but you will have to sit around for a while for the results to load. Since the results are downloaded ordered by volume, raising the limit from the default numbers will not actually give more significant results. You will simply get more of the smaller results, such as keywords with only 1 hit.
    look_back: This is a very important parameter. The script uses the date displayed upon your Google Analytics page to determine the full range that you want to consider in your results, but look_back determines how many of those days are used for the significance test. So, say the range you have displayed in Google is March 23  April 22 and your look_back is 7 days. The script will compare the average referrals for a given keyword from April 16-22 to the average from March 23-April 15, and will return the keyword only if the recent average is 20% higher than the rest of the time period. Thus, if you want to increase the total range of the data, change the dates on the actual webpage. Change look_back only if you want to change the period of significance.

See also #3032 Trends reports

@mattab commented on June 25th 2012

In general, this feature has much potential. Maybe other developers have better ideas to surface interesting data from the raw reports. We could write a simple plugin that would use the raw data to show insights. Other developers could then implement their own algorithm very easily by using the piwik APIs. We could even do a blog post and invite developers in a contest to write algorithms?

We could have a system that gives out several data insights profiles: - Negative/Positive Alerts when significant decreases in different categories: SEO (keywords, sites), Content (urls, downloads), Marketing (campaigns), Ecommerce (Products), Technical (Browsers, OS changes in bounce rate), Audience (returning vis, countries, mobile) - Display Possible actions for the users: - Marketing: check campaigns with visits but high bounce rate/low conversions, - Content: highlight high views ecommerce products with low conversion rates, highlight top viewed pages with higher than avg exit rate... - Technical: show significant pages with slow and/or slower than avg loading times

See also KPIs E-commerce : quels indicateurs pour votre tableau de bord for inspirations on good metrics that matter (french)

@mattab commented on December 14th 2012
  • Idea: Add new mode: color in red / orange, or green + lighter for good and bad rows. Make it a clog option. Also display, left of cog, "Enable Colors"?
  • when looking at daily reports, show best yearly items not found in daily
  • Similarly mode that compares with last month to show "New Links" or "New keywords"

see also nice ideas in http://www.kaushik.net/avinash/six-visual-digital-marketing-analytics-strategic-solutions/

@mattab commented on February 1st 2014

Consolidating milestones FTW

@tsteur commented on February 28th 2014

In 3055a6571d93272d72a5eb3201bf797577c63488: refs #5477 started to work on displaying Insights. Added a widget for the dashboard and a new Insights visualization. More to come... still early version so expect a lot of things to change but I am still happy about feedback

@mattab commented on March 2nd 2014

In 3f2916b99967e8276ce53314d28594d630101ca5: Enable Insights plugin for all beta testers in the coming weeks. Please send us your feedback about this new Magic Feature! refs #5477

@tsteur commented on March 3rd 2014

In b062cb7bf3086e635c39658106f1223dfd347ffd: refs #5477 added tests and fixed some bugs

@tsteur commented on March 3rd 2014

In 7714334dee2f1f5353183867ac6f5955f121e15f: refs #5477 do not call Insights API in integration tests

@tsteur commented on March 3rd 2014

In 2a60ff35fbf059e5a2076528950d1cec00a95ae4: refs #5477 fix tests

@tsteur commented on March 3rd 2014

In 8110e9db5c6589e7f426b2aec6a4f7ccb2e6146e: refs #5477 added possibility to place a widget multiple times (in case you want to see the same wdget with different visualizations and/or different parameters like different filters to make it easier comparable), removed some controls and save parameter changes in dashboard

@tsteur commented on March 3rd 2014

In 727da1b3cbb5a7e95be430a6cce22e9b34ce5453: refs #5477 some bugfixes and started to work on insights widget

@tsteur commented on March 3rd 2014

In 4de7581ca9032a548e619e41103aea478fe67635: refs #5477 fix tests

@hpvd commented on March 3rd 2014

coool :-)

Many thanks for your work on this!!

two tiny comments for looking inside on the table sites: - should the filter bar stay at the bottom? At the top near to the segment-editor may be also a great place which may prevent the need for scrolling up and down when working with segment editor and insight filter at the same time... - same with switching between standard and insight view: bottom or top? one idea maybe something like tabs at the top so you could easily see where you are and switch between the two views...

@hpvd commented on March 3rd 2014

using green and red as font colour for - and + values is great. Maybe one could visualized the differences even more by using some kind of bars directly in the table. Put an rough concept as attachment.

@tsteur commented on March 3rd 2014

In 856a3e613cf4d89efd0d4c8a7556a98632773277: refs #5477 reducing memory usage and improving performance

@tsteur commented on March 3rd 2014

In cbe88c1ab471c717fbee06eebe556cc97c5da6e5: refs #5477 destroy all tables but the insights tables

@tsteur commented on March 3rd 2014

In 6e022a78f6f1ac9e0b481433d6ac670dd5d5f119: refs #5477 also delete the last table

@tsteur commented on March 3rd 2014

In 62939c122a683de27ed714f3c1af2fa84700b0ef: refs #5477 do not flatten for now, brings massive performance improvement 10min > 1 sec as Referrers_getKeywords takes forever and also leads in memory exhausted when report is flattened

@tsteur commented on March 4th 2014

In 6034c0ead3016b0369e0bdcd0a8a252564aa88d7: refs #5477 fix tests

@tsteur commented on March 4th 2014

In 23186d76e8632683f6658e42c93488ac854d65d5: refs #5477 make sure text ellipsis works and label is always only one line. moved test fixture to insights plugin as demonstration for other plugin developers as it is very insights related anyway

@tsteur commented on March 4th 2014

In 8601c094bd1b0e4dbf80877ea7d00ad831a1b7ba: refs #5477 some more adjustments to the widgets and added a less file

@hpvd commented on March 4th 2014

please see: #4799 "broken after update to 2.1 final"

because disabling the inside plugin fixes it...

@tsteur commented on March 5th 2014

In c25f396c2452760b9c40c95b6889a3abdfcb317e: refs #5477 started to work on movers and shakers and lots of minor improvements

@tsteur commented on March 5th 2014

In a9e8651ec0f0e68fb3693de3a38fb1c51351d850: refs #5477 some refactorings to have better testable code, loads of tests and bugfixes

@tsteur commented on March 5th 2014

In 4bb969cea4d446c230f46a41905f8a6fff4765f7: refs #5477 added more tests and some bugfixes

@tsteur commented on March 5th 2014

In 862e532af3672352a7f7e9eb2d763bcb0803f4e2: refs #5477 removed min impact control as it might be not clear what it is anyway and user can change the limit filter if he wants to see only entries with very high impact

@tsteur commented on March 5th 2014

In 593dbc8e0e8e521e17e1c784c2d6d4debbbc510b: refs #5477 some UI tweaks

@tsteur commented on March 6th 2014

In ad9ab5d884bab393467b4b9140e3460f99b46f8b: refs #5477 more tweaks to insights and movers and shakers and more tests

@tsteur commented on March 7th 2014

In 4165176308adc982bb127bb19ff8825340573f67: refs #5477 some code improvements and tests, also some adjustments for sites having not many visits

@tsteur commented on March 7th 2014

In fc6fe7f8ad59498061886f162375a6e75df11f66: refs #5477 make sure always a change of at least 2 visits is required. Added icon license information, will move the information later

@tsteur commented on March 7th 2014

In aa404661555b104ca9eddb49084e38f2d2e6ec03: refs #5477 fix syntax error

@tsteur commented on March 10th 2014

In d210180a4609f8f89107e10fe8a4eafe577eaa24: refs #5477 ui tweaks and translations

@tsteur commented on March 10th 2014

In 81c3786bd049add8e170e1c53da32676b56c1d23: refs #5477 moving translations to general en.json

@tsteur commented on March 10th 2014

In 78e96146da2bb59a6a3b7a55e542be0ab353ec9c: refs #5477 average filter is no longer needed

@tsteur commented on March 10th 2014

In 3710bb49dc11df86f40a87c65cd1038b3bbff2db: refs #5477 some more code tweaks and documentation

@tsteur commented on March 10th 2014

In a4f6fc581725b21efaf079adf9fe360830247b8d: refs #5477 moved legalnotice info for insights icon

@tsteur commented on March 10th 2014

In e96399a840d942f8aabb387159ff9dc5d5ce4c45: refs #5477 added missing translations

@tsteur commented on March 10th 2014

In 2e1cb7bf6c14919a4b0f6e37e90fd72823e9c07b: refs #5477 make sure all column in overview widget have same width

@mattab commented on April 16th 2014

In 9af9ae3fe1030607ccee86100089c849a8c479e6: Refs #5477 New good looking icon for data insights for Morpheus theme!

@mattab commented on May 27th 2014

First version of this feature is now available! - Current movers and shakers screenshot: [[http://issues.piwik.org/attachments/5477/Movers%20and%20shakers.png]] - Current Insights overview widget screenshot: http://issues.piwik.org/attachments/5477/insights%20overview.png - On the Referrers > Websites widget, click on the Light bulb "Insight" icon, here is what it looks like: http://issues.piwik.org/attachments/5477/Insights%20view%20of%20a%20report.png

Well done Thomas for this work, it is a significant feature in terms of usefulness and potential of "insights". we are committed to improving it in the future, so please, report your feedback!

Leave us your feedback, ideas, suggestions in this ticket: #5246 Data Insights, Movers and shakers: improvements and suggestions

#5246

This issue was closed on May 27th 2014
Powered by GitHub Issue Mirror