@diosmosis opened this Issue on February 22nd 2014 Member

Converting UI tests to JavaScript will confer many benefits including:

  • Clearer test output (& no travis timing issues).
  • Faster tests (we can setup just once for all tests & don't have to execute phantomjs multiple times).
  • Clearer test code (via Jasmine's BDD style output).

Steps to complete:

  • Move all of test environment setup to Fixture class. (done in uiunittests branch)
  • Extend mocha & chai to handle screenshot compare use-case.
  • Create JS script that will run a PHP program to setup the database and run each UI test suite.
  • Modify existing UI tests.
  • Modify travis config to run via phantomjs.

Reason for using Jasmine: phantomjs uses Jasmine internally for its tests so we know it'll work. mocha + chai can be used as an alternative.

CC-ing tsteur as per matt's request.

Update: Using mocha + chai is actually easier to get running and easier to extend.

@tsteur commented on February 23rd 2014 Owner

+1 for MochaJS see #4691

@diosmosis commented on February 24th 2014 Member

In 71dee21d4988ae03cad321b0d9f1fd735151884b: Refs #4739, adding mocha + chai to tests/lib dir.

@diosmosis commented on February 24th 2014 Member

In 39c8961b418e151d13b8652049c257186df3962d: Refs #4739, remove unneeded persist_fixture_data.

@diosmosis commented on February 24th 2014 Member

In f4e39c9a87e38ee17f1b2ddaf0136543e42a47cc: Refs #4739, #4189, remove persist_fixture_data config option code, placed OmniFixture into a namespace, removed PHP screenshot testcase classes and added fixture for use w/ JavaScript screenshot tests.

@diosmosis commented on February 24th 2014 Member

In f2faa23854874b25cb0ac54656b7b3d90a1dd0b7: Refs #4739, make sure recursive directory links are created even when --persist-fixture-data is used.

@tsteur commented on February 24th 2014 Owner

@capedfuzz it would be cool if we could reuse some parts of this to write JavaScript unit tests for all the angular code

BTW: probably not really useful but just mention it... I built something similar a year ago to run tests using PhantomJS + Jasmine 2.0 but would now use MochaJS. Reckon there is not really anything new for you but just wanted to mention it. There is a testrunner https://github.com/tsteur/titanium-alloy-jasmine-testing/blob/master/tests.js and a loader https://github.com/tsteur/titanium-alloy-jasmine-testing/blob/master/specs/jasmine/utils/loader.js who finds all the test files in a specs directory: https://github.com/piwik/piwik-mobile-2/tree/master/specs

@diosmosis commented on February 24th 2014 Member

@tsteur If the tests for the angularjs code will run in a webpage, then you won't need any code I've written. The screenshot tests run directly in phantomjs, not in a webpage.

@diosmosis commented on February 25th 2014 Member

In bba1420dd5effbfde43f592007a5566d7a808697: Refs #4739, allow JavaScript screenshot tests to override Piwik behavior by storing values in tmp/ file, add hook so plugins can setup their own test environment, reset user password on test fixture setup.

@diosmosis commented on February 27th 2014 Member

In bf92e42b541553f4bc585c6a210e2c6632aa4d61: Refs #4739, add console command to run UI tests.

@diosmosis commented on February 27th 2014 Member

In 35586dfaf52c795cfc4aa07891364f7bcabbe7fd: Refs #4739, pass groups to new RunUITests console command.

@diosmosis commented on February 27th 2014 Member

In 8ba7f01679981cd9375acb091e638e437bd15a67: Refs #4739, move screenshot testing library code to core repository (from UI repository).

@diosmosis commented on February 27th 2014 Member

In 4ad2b987467fd57c0ef5a0bbaa8c7f66845d69e1: Refs #4739, make sure UI screenshots are stored in plugin directories if tests are for plugins, cleaned up path output in console and add .gitignore to DBStats plugin.

@diosmosis commented on February 27th 2014 Member

In 8d72a9fc7f1b8792f67f80dbb86bd55f681369b8: Refs #4739, fixing bug in processed screenshot dir path creation.

@diosmosis commented on March 4th 2014 Member

In ba6a60ac294e8663687ef67cb04a060346f7d809: Refs #4739, add information about new UI tests to screenshot testing README.

@diosmosis commented on March 4th 2014 Member

In f073d1056bd17ee0f6940b9e571a7e5c890d4b64: Refs #4739, modify SyncUITestScreenshots command to work with new screenshot testing code and display location hint in diffviewer output if expected file is stored within plugin and not the UI tests repo.

@diosmosis commented on March 4th 2014 Member

In 28a2f9b15cd146ad6bf5921e351767a6d9bd9392: Refs #4739, add travis only command line options to UI test runner: --store-in-ui-tests-repo, --use-github-expected.

@diosmosis commented on March 4th 2014 Member

@tsteur FYI, I did some thinking and I think the screenshot testing code could be used for angularjs tests too. For example:


describe("Something", function () {
    var webpage,
        url = config.piwikUrl + "?module=CoreHome&action=blankPage"; // page loads all JavaScript but no contents

    before(function (done) {
        webpage = require('webpage').create();
        webpage.load(url, done);
    });

    after(function () {
        webpage.close();
    });

    it("...", function () {
        var result = webpage.evaluate(function () {
            // do test here ...
        });

        expect(result).to.be.equal('whatever');
    });
});

I'm not sure how much extra work it will be to get it to work or if there are any big benefits, but I think it's possible.

@diosmosis commented on March 4th 2014 Member

In bb5e2b37ae5f04f78ec3eae636d70e38bb34f89f: Refs #4739, remove old capture.js file, add section about screenshot testing config to README and remove unused constant from Config.php.

@diosmosis commented on March 6th 2014 Member

In f44b9529a1cc8961782be8b240d9ab77d309789a: Refs #4739, move segment setup code to UITestFixture and make sure it is run on every test run.

@diosmosis commented on March 6th 2014 Member

In 9c9a91fc883b61bd3cd3330e23f9a64f29d2e1fc: Refs #4739, remove zeitgeist query parameter handling in TestingEnvironment.

@diosmosis commented on March 6th 2014 Member

In c9f13e1cc9c8348df8a3f4679116c8e85df2455e: Refs #4739, add dragDrop utility function to screenshot test page renderer and capture alert()'s during screenshot tests.

@diosmosis commented on March 6th 2014 Member

In d789162805fda622eb46c717b4d8c846d5b4b3ad: Refs #4739, display page logs when a screenshot test times out.

@diosmosis commented on March 6th 2014 Member

In 0a983024f0845a4f29e6c8bbc492858b7968b39e: Refs #4739, in the screenshot test diffviewer, display the location hint using the processed file path if the expected file does not exist yet.

@diosmosis commented on March 6th 2014 Member

In a43cae6db06688d289eea9a8474dae563f66bc13: Refs #4739, pass along errors to callback in testEnvironment Piwik API/Controller calls.

@diosmosis commented on March 6th 2014 Member

In 202671abdf55b4bfaa6d04af8cfebb5990da9b8c: Refs #4739, only pad widgets in screenshot tests when the element is a direct child of a .widget element.

@mattab commented on March 11th 2014 Owner

awesome work Benaka there! is this ticket closed?

@mattab commented on March 20th 2014 Owner

In e4b1c9c738d8ee9db0bea59a79c4c9373168ccd0: Refs #4739 Run the UI tests build as early as possible so we don't have to wait 30 minutes to see screenshots

This Issue was closed on April 1st 2014
Powered by GitHub Issue Mirror