Usually one would run the tests simply on the local server before committing it. Actual problem is our tests are very slow. For most developers it can take up to an hour for the tests to finish. Running the UI tests might take even longer. This is why we usually commit often to let Travis run the tests and to see whether we break anything.
As said we currently run our tests on Travis but it can take up to hours to get feedback there. Especially when there are many commits at the same time and resulting the jobs are queued. Even waiting 10 minutes for the tests to start on Travis is not acceptable as it means we have to switch to another task until I get a result (10 min waiting time + up to 20min execution time).
Following #6412 (Getting build feedback from CI is too slow) a possible solution would be to:
This reduces the waiting time a bit but as there are many developers working on this projects we will often still have to wait for a test to start etc. Travis does NOT scale as we cannot have more concurrent jobs.
The idea to solve this is to have a console command that starts the tests immediately on an AWS instance. This means no waiting time and possibly the tests execute faster as on travis. It would be nice to have the following features:
We have evaluated for instance JoliCI but it doesn't support the same features as Travis does. For instance there is no database and there are no notifications. Otherwise it is a cool tool!
We have a new command
tests:run-aws that launches an instances based on a custom AMI. There the tests are executed immediately so no waiting time! As expected the execution time is much faster compared to Travis:
Integration Tests: 3:40 minutes on Travis, 1:40 on AWS
System Tests: 16 minutes on Travis, under 8 minutes on AWS
UI Tests: 21 minutes on Travis, 15 minutes on AWS
On top we save the waiting time for Travis to configure the virtual machine which saves as 1-3 minutes.
Later we could have different AMIs for different PHP versions etc.
To further improve test response we could at some point to start multiple phpunits and execute tests in parallel. Maybe a project that does this already exists. Anyway, we would have to use multiple databases etc which is not supported by Piwik yet.
drone.io does not look cheap as getting 8 concurrent builds already costs 400 USD...
Its open source I think, see the GitHub link.
Maybe we should close this issue? JoliCI can't run our travis.yml anyway because it's too complex.
I'm still working on it. Will update issue later
Please note the following UI tests always fail on AWS as the screenshots contain for instance a path which is different to the one on travis:
Installation_system_check UIIntegrationTest_admin_manage_tracking_code UIIntegrationTest_admin_plugins Expected UIIntegrationTest_admin_security_info UIIntegrationTest_admin_visitor_generator UIIntegrationTest_db_connect_error Updater_main
Either we update the puppet at some point to have another user travis and same file structure (lot of work) or we fix the screenshot tests somehow to not print paths or we just leave it like this for a while
We can manually remove the paths from screenshot tests by grepping through the HTML and looking for PIWIK_INCLUDE_PATH and replacing w/ '/'. Should be enough I think.
Good idea. I'll create an issue for this.
I am going to close this for now. All future changes can be handled in new issues.
Great work guys!