@mattab opened this Issue on March 17th 2015 Owner

The goal of this issue is to decide on a process to ensure that the Piwik Product team has a workflow that will ensure that the CI builds across our 50+ repositories are green on every monday.

Goals:

  • Decide who is responsible to fix the builds on Monday
    1. maybe someone from the team will volunteer to "own this"
    2. or maybe a person rotating each week or month?

We could test something for a few weeks or months (eg. test the solution i. ) and if that does not well, change it to another process...

Your feedback most welcome!

@mattab commented on April 2nd 2015 Owner

one feedback is that fixing the tests every week is quite time consuming - maybe we could schedule to fix the builds of all plugins only once a month, for example ~ two weeks before the release

@mattab commented on April 7th 2015 Owner

Goals

  • Detect when a plugin is broken / its behavior is changed by a change in the stable Piwik platform
  • Detect when a plugin is broken when running with the minimum required piwik version (as set in its plugin.json file)
  • fixing the tests every week could be time consuming (we'd like to find a safe way to have to fix builds less often while meeting our high quality requirements)

Assumptions

  • we ensure our plugins don't call non-public APIs, (requires: #7569 and #7633)
  • we ensure our plugins don't call deprecated code. (requires: #7569 and #7633)

Summary: our plugins should never break or get broken by a newer Piwik version.

Proposal Plugin CI jobs setup

  • (1) Plugin CI job running against minimum required Piwik would fail build when the CI job fails
    • this is current behavior, it allows us to detect when a plugin is broken when running with the minimum required piwik
  • (2) Create a new plugin CI job running against latest stable Piwik that would fail the build if the job fails
    • it would allow us to detect when a plugin is broken / its behavior is changed by a change in the stable Piwik platform
  • (3) mark the Plugin CI jobs against Piwik platform master branch as Allowed failure.
    • eg. if bugs are introduced in Piwik master, or API changes, the plugin CI would not fail, but product developer could look at the CI job to see any failing test against master
  • (4) we still refresh all plugin CI jobs once a week on Sunday
    • this is current behavior

Summary

  • By making those proposed changes, we would meet our Goals above,
  • we would also save time and effort as the builds would not fail every week but may only fail after a stable version of Piwik has been released (or when the plugin starts to break against the minimum required piwik).
    • (this may cause Plugin bugs when after a platform stable release, the plugin build fails due to a bug in platform stable release, but this should be a rare case considering that our plugins will only call public APIs and non-deprecated code...)
    • (if we start to have such problems, then we could solve the problems by for example introducing a new routine. Imagine 1 week before stable release, a Product team member or release manager would want to check that the latest beta/RC version works well with our plugins. He would do this by looking at the plugin CI jobs that are allowed failures and would manually check if the failures are due to a cosmetic change or if any failure is due to a regression in core platform.)

what do you think?

@Tobias-Conrad commented on April 7th 2015

may also check external plugins wp-piwik and woocommerce tracking.
e.g. they show data on dashboard which may is broken after update piwik.

@diosmosis commented on May 20th 2015 Member

@mattab I think this issue is a lot simpler now. PRO builds test against specific Piwik versions only, correct? In this case we only have to maintain the open source plugins that run tests against master.

@mattab commented on May 20th 2015 Owner

I think this issue is a lot simpler now. PRO builds test against specific Piwik versions only, correct?

I believe that one suggestion by @mgazdzik and team is to run the PRO builds against the latest Piwik stable version only. @diosmosis I don't know if it's implemented yet though, or whether it's consistently implemented (eg. all premium plugins run builds against latest stable only, while open source plugins run builds against latest stable + master).

Maybe we need to confirm this is what we want to do, and then ensure it's consistently applied across all plugins, then hopefully we will be in a better position with regards to keeping builds green?

@mattab commented on June 24th 2015 Owner

we are making some progress

  • we are making improvements to CI Status dashboard from time to time in https://github.com/piwik/ci-status/issues
  • builds-artifacts now shows a nice and more usable UI: http://builds-artifacts.piwik.org/
  • our other ongoing project is to think about, and investigate, remaining UI screenshot tests that fail at random #6693
  • @diosmosis is working on making builds green across dozens of plugin repositories, especially after recent DI change and improvements to tests framework in core platform. Epic work :+1:
@mattab commented on April 1st 2016 Owner
This Issue was closed on April 1st 2016
Powered by GitHub Issue Mirror