The goal of this issue is to solve one common occurrence of failing CI builds.
In particular the goal is to update the auto-generator for travis.yml file so that
- when a plugin.json is set to require a Piwik core version that is newer than the current stable version (eg. a recent beta) then the travis.yml will be generated with the job
TEST_AGAINST_CORE=latest_stable set as
- when a plugin.json requires piwik core older than the lates stable, then do not mark the job as allow_failures.
See example in this build: https://travis-ci.org/piwik/plugin-CustomAlerts/builds/45933709 - The build fails against the latest stable because it uses new APIs which didn't exist then.
Follows up #6544 and #6542
I think the latest_stable build should be always an allowed failure, or the dev who adds new features should explicitly make it an allowed failure when making the build green.
The point of testing against latest_stable is to detect when a plugin is currently broken with the latest stable. Seeing red build will help us detect the cases where this plugin's plugin.json does not yet indicate that the plugin requires a more recent Piwik core version. if we can make it automated then we should otherwise we'll forget etc.
Changing a specific entry in matrix/env during generation is very non-trivial (requires correctly & completely parsing travis YAML, detecting build via environment variables + taking into account exclude, correctly outputting formatted YAML w/ preserved comments + other parts). What's in the ticket description may not be possible w/o a lot of work.
Ok makes sense. So if we forget about the issue description, maybe there's something we could do.
For example would it make more sense to rename
latest_stable job to be instead
minimum_required ? it would run the tests against the minimum required version tag that is specified in the plugin.json, at all times. then we wouldnt need to change the travis.yml generation, and still we would have red build when the minimum required version is not set properly. thoughts?
I think that's a great idea! Should do better than testing against latest stable too.