@mattab opened this issue on January 6th 2015

The goal of this issue is to solve one common occurrence of failing CI builds.

In particular the goal is to update the auto-generator for travis.yml file so that - when a plugin.json is set to require a Piwik core version that is newer than the current stable version (eg. a recent beta) then the travis.yml will be generated with the job TEST_AGAINST_CORE=latest_stable set as allow_failures. - when a plugin.json requires piwik core older than the lates stable, then do not mark the job as allow_failures.

See example in this build: https://travis-ci.org/piwik/plugin-CustomAlerts/builds/45933709 - The build fails against the latest stable because it uses new APIs which didn't exist then.

Follows up #6544 and #6542

@diosmosis commented on January 6th 2015

I think the latest_stable build should be always an allowed failure, or the dev who adds new features should explicitly make it an allowed failure when making the build green.

@mattab commented on January 6th 2015

The point of testing against latest_stable is to detect when a plugin is currently broken with the latest stable. Seeing red build will help us detect the cases where this plugin's plugin.json does not yet indicate that the plugin requires a more recent Piwik core version. if we can make it automated then we should otherwise we'll forget etc.

@diosmosis commented on January 6th 2015

Changing a specific entry in matrix/env during generation is very non-trivial (requires correctly & completely parsing travis YAML, detecting build via environment variables + taking into account exclude, correctly outputting formatted YAML w/ preserved comments + other parts). What's in the ticket description may not be possible w/o a lot of work.

@mattab commented on January 6th 2015

Ok makes sense. So if we forget about the issue description, maybe there's something we could do.

For example would it make more sense to rename latest_stable job to be instead minimum_required ? it would run the tests against the minimum required version tag that is specified in the plugin.json, at all times. then we wouldnt need to change the travis.yml generation, and still we would have red build when the minimum required version is not set properly. thoughts?

@diosmosis commented on January 6th 2015

I think that's a great idea! Should do better than testing against latest stable too.

This issue was closed on January 7th 2015
Powered by GitHub Issue Mirror