@mnapoli opened this Issue on October 13th 2014 Member

Usually one would run the tests simply on the local server before committing it. Actual problem is our tests are very slow. For most developers it can take up to an hour for the tests to finish. Running the UI tests might take even longer. This is why we usually commit often to let Travis run the tests and to see whether we break anything.

As said we currently run our tests on Travis but it can take up to hours to get feedback there. Especially when there are many commits at the same time and resulting the jobs are queued. Even waiting 10 minutes for the tests to start on Travis is not acceptable as it means we have to switch to another task until I get a result (10 min waiting time + up to 20min execution time).

Following #6412 (Getting build feedback from CI is too slow) a possible solution would be to:

  • Use Travis CI to run the full test suite on PHP 5.3 only
  • Run the full test suite on all the other PHP versions + code coverage outside of Travis, in an Amazon EC2 instance for example
    Currently, the Travis build takes 36 minutes (see https://ci-status.com) plus the queue time. In theory, that time would be divided by 4, so a little less than 10 minutes. That would also mean much less queue time.

This reduces the waiting time a bit but as there are many developers working on this projects we will often still have to wait for a test to start etc. Travis does NOT scale as we cannot have more concurrent jobs.

The idea to solve this is to have a console command that starts the tests immediately on an AWS instance. This means no waiting time and possibly the tests execute faster as on travis. It would be nice to have the following features:

  • Run tests against a specific git hash, branch, tag
  • Apply a patch on top of the checked out version so one does not have to commit to test anything
  • Screenshot diffs for UI tests

We have evaluated for instance JoliCI but it doesn't support the same features as Travis does. For instance there is no database and there are no notifications. Otherwise it is a cool tool!

Update:
We have a new command tests:run-aws that launches an instances based on a custom AMI. There the tests are executed immediately so no waiting time! As expected the execution time is much faster compared to Travis:
Integration Tests: 3:40 minutes on Travis, 1:40 on AWS
System Tests: 16 minutes on Travis, under 8 minutes on AWS
UI Tests: 21 minutes on Travis, 15 minutes on AWS

On top we save the waiting time for Travis to configure the virtual machine which saves as 1-3 minutes.

Later we could have different AMIs for different PHP versions etc.

To further improve test response we could at some point to start multiple phpunits and execute tests in parallel. Maybe a project that does this already exists. Anyway, we would have to use multiple databases etc which is not supported by Piwik yet.

@diosmosis commented on October 13th 2014 Member

Any thoughts about using drone: https://github.com/drone/drone & http://drone.io ?

I don't know what it will take to get it to work w/ .travis.yml files.

@mattab commented on October 13th 2014 Owner

drone.io does not look cheap as getting 8 concurrent builds already costs 400 USD...

@diosmosis commented on October 13th 2014 Member

Its open source I think, see the GitHub link.

@mnapoli commented on October 28th 2014 Member

Maybe we should close this issue? JoliCI can't run our travis.yml anyway because it's too complex.

@tsteur commented on October 28th 2014 Owner

I'm still working on it. Will update issue later

@tsteur commented on October 29th 2014 Owner

Please note the following UI tests always fail on AWS as the screenshots contain for instance a path which is different to the one on travis:

Installation_system_check
UIIntegrationTest_admin_manage_tracking_code
UIIntegrationTest_admin_plugins Expected    
UIIntegrationTest_admin_security_info   
UIIntegrationTest_admin_visitor_generator
UIIntegrationTest_db_connect_error
Updater_main

Either we update the puppet at some point to have another user travis and same file structure (lot of work) or we fix the screenshot tests somehow to not print paths or we just leave it like this for a while

@diosmosis commented on October 30th 2014 Member

We can manually remove the paths from screenshot tests by grepping through the HTML and looking for PIWIK_INCLUDE_PATH and replacing w/ '/'. Should be enough I think.

@tsteur commented on October 30th 2014 Owner

Good idea. I'll create an issue for this.

@tsteur commented on October 30th 2014 Owner

I am going to close this for now. All future changes can be handled in new issues.

@mattab commented on October 30th 2014 Owner

Great work guys!

This Issue was closed on October 30th 2014
Powered by GitHub Issue Mirror