@mattab opened this Issue on October 10th 2014 Owner

The goal of this issue is to run the Screenshot test suites of 200 screenshots tests, so that they run on PHP 5.3 and 5.5.

Recently we had an issue on PHP 5.3 that could have been discovered by a UI test if they were running on 5.3. Currently they only run on 5.5 though... so we missed it.

Proposed tasks

  • Modify travis.yml so that UI tests run on 5.3 and 5.5 (possibly also 5.4 ?)
  • Modify core + pro Plugins UI tests travis so they run on 5.3 and 5.5.
@sgiehl commented on October 10th 2014 Member

+1 for that. Maybe we could run the tests on other php versions some kind of time triggered (e.g. every 3-6 hours), so they won't run after each commit. Guess that should be enough to identify such errors.

@tsteur commented on October 12th 2014 Owner

I think the UI tests are there to test UI stuff (JS, CSS, HTML) so testing many different PHP versions shouldn't matter so much. Ideally, all the other stuff is covered by unit / integration / system tests. Unfortunately, this is not 100% the case eg for some Controller code. Would personally run them only on PHP 5.3. If needed also on the latest meaning 5.6. Nothing else.

@mattab commented on October 17th 2014 Owner

In an effort to keep jobs small, maybe we only run UI tests on 5.3.3 then!

@mnapoli commented on October 19th 2014 Member

maybe we only run UI tests on 5.3.3 then

+1

@tsteur commented on November 10th 2014 Owner

Shall we still run them on 5.3.3? Currently they are running on 5.5. Not sure what the latest thoughts on this are...

@mattab commented on November 10th 2014 Owner

:+1: to run on lowest PHP version 5.3.3!

@tsteur commented on November 10th 2014 Owner

Just FYI: It seems to make the UI tests slower from 18-20 minutes to 25-28 minutes but think it is still good to use PHP 5.3.3 to run the tests. It is worth it as we will catch more errors this way.

This Issue was closed on November 10th 2014
Powered by GitHub Issue Mirror