The goal of this issue is to run the Screenshot test suites of 200 screenshots tests, so that they run on PHP 5.3 and 5.5.
Recently we had an issue on PHP 5.3 that could have been discovered by a UI test if they were running on 5.3. Currently they only run on 5.5 though... so we missed it.
+1 for that. Maybe we could run the tests on other php versions some kind of time triggered (e.g. every 3-6 hours), so they won't run after each commit. Guess that should be enough to identify such errors.
I think the UI tests are there to test UI stuff (JS, CSS, HTML) so testing many different PHP versions shouldn't matter so much. Ideally, all the other stuff is covered by unit / integration / system tests. Unfortunately, this is not 100% the case eg for some Controller code. Would personally run them only on PHP 5.3. If needed also on the latest meaning 5.6. Nothing else.
In an effort to keep jobs small, maybe we only run UI tests on 5.3.3 then!
maybe we only run UI tests on 5.3.3 then
Shall we still run them on 5.3.3? Currently they are running on 5.5. Not sure what the latest thoughts on this are...
:+1: to run on lowest PHP version 5.3.3!
Just FYI: It seems to make the UI tests slower from 18-20 minutes to 25-28 minutes but think it is still good to use PHP 5.3.3 to run the tests. It is worth it as we will catch more errors this way.