@tsteur opened this issue on June 12th 2015
  • If there is a failure we do now output JS errors, not loaded resources, ... immediately within the tests. Even if the UI tests do not finish within 50 minutes we still get an idea of what is going on. It prints the result at the bottom of the test again but as it is only for failing tests shouldn't matter so much. Code wise we could have copied the mocha spec reporter and adjusted this one but can still do it later if needed. There's no coloured output for this yet so hope that's okay. It should help us to troubleshoot things.
  • I noticed we detect whether there are any ajax requests running and capture a screenshot only if all ajax requests and images are loaded. We did not check for JS resources and neither for AngularJS requests (there are quite a few of them) which can lead to random test results. I added a check for the AngularJS requests. As the detection of loaded images and requests can be buggy I thought it would be nice to use PhantomJS's events onResourceRequested, onResourceReceived, ... This way we would not have to take care of this in core and we can easily print a list of not finished requests at the end of the test. Unfortunately we have some synchronous requests and for those requests no "response" event will be triggered meaning we cannot detected that the sync request is finished. Therefore I do now check whether either phantomjs detected that all resources are loaded or our core logic. Once we refactored https://github.com/piwik/piwik/issues/8020 we can trust PhanomJS. This will allow us to detect more accurately when a page is loaded or stopped being active etc.
  • While working on this I noticed the Menu tests were wrong. There was a test for Admin Menu but it was actually User Menu tests so I changed it and added a test.

An example can be seen here : - https://travis-ci.org/piwik/piwik/jobs/66323509#L728 - https://travis-ci.org/piwik/piwik/jobs/66323509#L781 - Cannot find an example currently where it fails to load a page within 2 minutes. There we will now see a list of still loading resources

The tests that fail seem to also fail on master?!?

@diosmosis commented on June 13th 2015

Once we refactored #8020 we can trust PhanomJS. This will allow us to detect more accurately when a page is loaded or stopped being active etc.

Is it possible to detect the synchronous request and display a warning in phantomjs? Or even fail the test? Would perhaps help in finding all the synchronous requests.

@diosmosis commented on June 13th 2015

Did you try and reduce the default wait time? W/ the new wait detection logic, maybe it could be lowered.

@mnapoli commented on June 15th 2015

I haven't reviewed the code diff but the change seems useful (e.g. the example builds you linked to).

I added a check for the AngularJS requests

:+1:

As the detection of loaded images and requests can be buggy I thought it would be nice to use PhantomJS's events onResourceRequested, onResourceReceived

Are "images not fully loaded" a problem sometimes? Sounds good in theory but I wonder whether it's worth the effort?

@diosmosis commented on June 15th 2015

Not sure what the travis issue is, but when I try to run the ActionsDataTable UI tests locally I get two timeouts. After I finish looking at other PRs, I'll look into this more closely.

@tsteur commented on June 15th 2015

Are "images not fully loaded" a problem sometimes? Sounds good in theory but I wonder whether it's worth the effort?

Yes, we will be able to detect much more accurately whether all resources are loaded. Eg we currently do not detect whether *.js, font files are loaded etc. Also the current logic goes throw every DOM node all the time and checks whether there are images defined via 'backgroundImage', 'listStyleImage', 'borderImage', 'borderCornerImage', 'cursor' and checks whether they are loaded or not. It can easily happen that we don't detect all images this way and not accurately. This way random build failures can happen. Also we will be able to run tests faster at some point by detecting better when the page finished being active. Also if we forget to reset some ajax requests in core some pages might not finish loading. Using PhantomJS it will just work (in theory g)

@tsteur commented on June 15th 2015

Is it possible to detect the synchronous request and display a warning in phantomjs? Or even fail the test? Would perhaps help in finding all the synchronous requests.

I don't think we can detect them, I had a quick look. It will be quite a bit of work to refactor them as we will now have to use callbacks everywhere a lot of code needs to be changed. We can just search for send(true) in the code base and should find them

@tsteur commented on June 15th 2015

Did you try and reduce the default wait time? W/ the new wait detection logic, maybe it could be lowered.

We should test in a separate branch after all tests are green. It's currently hard to tell since many UI tests fail randomly. It would be awesome if we could lower it

This issue was closed on June 16th 2015
Powered by GitHub Issue Mirror