@mattab opened this Issue on October 30th 2015 Owner

The goal of this issue is to discuss how to make Custom Variables more powerful and also easy to use. Custom Variables are the best way to assign metadata / properties to a given user (visit) and/or action (pageview, event, etc.).

Notes:

  • Google Analytics has renamed Custom Variables into Custom Dimensions and Metrics, which makes more sense to users.
    • they removed the need to set the custom variable name in the JavaScript
    • now one only needs to choose a slot integer eg.
ga('send', 'pageview', {
  'dimension15':  'My Custom Dimension'
});
  • In our case @ Piwik
    • we could do the same (remove the need to set a Custom dimension name): setCustomDimension(15, value); (this requires defining Custom Dimension name + scope in a new User Interface > Create new Custom Dimension)
    • or maybe we could instead remove the need to set a slot index:
      setCustomDimension('My Dimension name', value, scope = "page");

Notes about implementation:

  • After discussing quickly internally, we have figured out that to properly implement Custom Dimensions we will have to implement it from "scratch" and we could not re-use Custom Variables feature. Custom dimensions and custom metrics would be stored in new datatable table fields (not re-using custom_var_k|v*)
  • we could maintain the Custom Variable feature for a while, but would maybe need to remove it at some point.
@mattab commented on October 30th 2015 Owner

Advantages of Custom dimensions over Custom Variables:

  • Easier to track: in JavaScript no need to set the scope + integer slot
  • One defines the name of a custom dimension in the UI. So one can change the dimension name at once across all dates
  • Custom Dimensions are easier to segment: there is a new segment automatically available for each custom dimension eg. segment=dimension15@=xyz
  • Custom dimensions are displayed as Submenu entries
    • Custom dimension of scope "visit" are displayed under "Visitors" menu
    • Custom dimension of scope "page" are each displayed under "Actions" menu
  • Less storage needed is needed for Custom Dimensions: we only store dimension {Value}VS we store custom variables {Name + Value}

EDIT: Moved to https://github.com/piwik/plugin-CustomDimensions/issues/6

@RMastop commented on October 30th 2015 Contributor

+1

@tsteur commented on November 2nd 2015 Owner

Questions:

  1. Should it be possible to delete custom dimensions? If one deletes a custom variable, we'd either have to only mark it as deleted so a user can undo it at any time or if the slot is supposed to be reusable afterwards we'd need to delete all data from existing logged requests for that dimension (and in theory even the archives as they would contain data for a dimension index that is now used for something else). Ideally we would not actually allow to delete it and only mark it as deleted if needed. The slot would never be available again for this website and there's a chance one is running out of custom dimensions quickly. New dimensions require schema changes in the database which can be slow. This is probably also why Google has an active flag.
  2. I'd set max length for a custom dimension to value 255 for now. Max length custom variable is currently 200. Maybe we should allow more characters for custom dimensions?
  3. In theory we could allow to configure to have eg only 5 custom dimensions in scope visit but 10 in scope page but I presume we want to keep it the same? Allowing different numbers of custom dimension might lower the needed schema changes. Eg if one wants more custom dimensions in scope visit it will be much faster to only change schema of log_visit table.
  4. API.getSegmentsMetadata is problematic as the dimensions will depend on the idSite. This means there will be different segments for each website. API.getSegmentsMetadata accepts an array of idSites. As different sites contain different custom dimension segments it's hard clear what to run in this case. There's an issue to fix this in 3.0 https://github.com/piwik/piwik/issues/7834 and to never allow multiple idSites but for 2.X it is a bit problematic. I presume we won't add the custom dimension plugin to core in 2.15.X so it might be okay if the API behaves a bit weird here until 3.0 is released. If multiple idSites are given, should we add no segment or all segments? I'd personally just add no segment in this case as it can be misleading when returning all segments and returning the same segment multiple times under different names.

FYI:

  • During tracking, if someone sets a value for a dimension that is not configured yet, we won't save the value and only ignore it. This is because we don't know for which scope the dimension is supposed to be etc. (scope page|visit will be configured server site)
@mattab commented on November 2nd 2015 Owner

Should it be possible to delete custom dimensions?

Let's do like Google and use an Active flag, which makes it clear that the "dimension slot" is booked by this dimension. The dimensions cannot be deleted, only renamed.

Max length custom variable is currently 200. Maybe we should allow more characters for custom dimensions?

Sounds good to use limit of 255

In theory we could allow to configure to have eg only 5 custom dimensions in scope visit but 10 in scope page but I presume we want to keep it the same?

It'd be better to allow different number of slots for "visit" and "page" scope.

When one needs to add a "visit" scope custom dimension, it wouldn't ALTER the large log_link_visit_action table and save much time.

There's an issue to fix this in 3.0 #7834 and to never allow multiple idSites but for 2.X it is a bit problematic.

Could we already fix #7834 which would solve the API.getSegmentsMetadata issue?

Notes:

  • we will need to communicate to author of SDKs (iOS, Android, C#, Java, etc.) that there is a new Tracker API available, so that they implement the new setters
@tsteur commented on November 2nd 2015 Owner

Should we go with setCustomDimension(15, value); or set('dimension15', value) or setCustomDimension('dimension15', value);?

As one probably has to use dimension15 when using a segment, it might be worth to go with set('dimension15', value)

@mattab commented on November 2nd 2015 Owner

As one probably has to use dimension15 when using a segment, it might be worth to go with set('dimension15', value)

Because we have the same issue already with custom variable (segment names which include the slot eg. customVariablePage15 but users call setCustomVariable(15, ...))... it would be inconsistent to use set('dimension15', ...... setCustomDimension(15, would be consistent with our existing JS API.

having set( name, ... would make piwik.js behave like analytics.js. IMO this would be nice to have as it simplifies the JS APi, and would make writing SDKs quite easier.

Maybe you could create a separate issue for this feature request? and maybe we could add this set( name already in 2.15.1 if not too much work...

@tsteur commented on November 2nd 2015 Owner

I don't really understand, so we should go with setCustomDimension(15, value) and create an issue to add set(name, value) later?

@mattab commented on November 2nd 2015 Owner

+1, this is what I suggest to stay consistent

@tsteur commented on November 3rd 2015 Owner

Another question: The custom dimension id:

By default you can specify up to 5 custom dimensions per website per scope. Is it ok to have custom dimenion ids across all sites? Say you use 50 sites, 5 custom dimensions in 2 scopes each, then dimension ids are from 1-500 (eg setCustomDimension, 500, value) or should they always start from 1-5 per site? (which I presume but takes bit more work)

@mattab commented on November 3rd 2015 Owner

or should they always start from 1-5 per site? (which I presume but takes bit more work)

:+1: for starting at 1 for each website, it should also nicely map to the custom dimension DB field indexing this way?

@tsteur commented on November 4th 2015 Owner

OK will do start at 1 for each website but they won't map the custom dimension db field index as one dimension can be configured for different scopes.

Another "problem": Custom Dimension will be a plugin on the marketplace for now eg

  • To be able to release updates quickly as there might be some bugs and might have some missing but needed features
  • Will require schema changes for pretty much all log tables when installing the plugin (unless we do not preinstall any dimensions but wouldn't be so nice)

If it will be a plugin on the marketplace we'll have the tracker method setCustomDimension in core but the actual logic in a plugin... so we kinda need some way of letting plugins extend the tracker? Or would it be okay to have a method in core unless we don't document it in default tracker page?

@tsteur commented on November 4th 2015 Owner

@mattab also wondering if it makes sense to use log_action for custom dimensions in page scope?

@mattab commented on November 5th 2015 Owner

Or would it be okay to have a method in core unless we don't document it in default tracker page?

+1 let's add method in piwik.js in core for the purpose of the plugin. in a later Piwik we will allow plugins to extend piwik.js but not needed at the moment.

also wondering if it makes sense to use log_action for custom dimensions in page scope?

No better KISS, and write /duplicate the values in each row which has a custom dimension set.

@mattab commented on November 5th 2015 Owner

Thoughts:

  • In the visitor log, we need to show "Custom dimensions" set for each visitor and each page view.
@tsteur commented on November 5th 2015 Owner

#6111 is a lot of effort, not sure about that, need to see later... I already added the dimensions to the Live.getLastVisitDetails API output.

@mrjoops commented on November 5th 2015

On a side note, as Piwik follows semver, it doesn't look like a good idea to implement it in a patch version (namely 2.15.1).

@tsteur commented on November 5th 2015 Owner

Absolutely true. We'd have to release a 2.16 in that case. We're developing it in a new plugin at https://github.com/piwik/plugin-CustomDimensions

@tsteur commented on November 25th 2015 Owner

A first version is available on the Marketplace: http://plugins.piwik.org/CustomDimensions (requires latest Piwik 2.15.1-b3 beta version)

@mattab commented on November 25th 2015 Owner

:tada: && :rocket:

@tsteur commented on December 3rd 2015 Owner

I wonder if we can close the issue as we have other issues in the Custom Dimensions repository eg re documentation

@mattab commented on December 7th 2015 Owner

Added:

Updated:

  • Custom Variables guide linking to Custom Dimensions FAQ + added section When not to use Custom Variables? // As of Piwik 2.15.1, it is recommended to use the new Custom Dimensions feature instead of Custom Variables.

The plugin is now officially released!

@mattab commented on December 7th 2015 Owner
@mattab commented on February 1st 2016 Owner

FYI: installed Custom Dimension plugin on demo.piwik.org and setup a custom dimension on idSite=1 called Changelog entry so now this plugin is setup on both demo and demo2

This Issue was closed on December 7th 2015
Powered by GitHub Issue Mirror