@mattab opened this Issue on June 19th 2009 Owner

We use bamboo for continuous integration: https://bamboo.openx.org/

currently it runs the webtest that tests the installation process + reset password feature.
it should also run the unit tests and fail when at least one test doesn't pass. there is some magic involved to rewrite simpletests results in XML to be parsed by the tool.

The tool has been failing lately and we would like to setup our own Build integration software on 2 servers we have available for the task.

@robocoder commented on January 18th 2010 Contributor

bamboo.openx.org server appears to be no longer available; should we close this or look into setting up our own (eg using xinc)?

@mattab commented on January 18th 2010 Owner

vipsoft, we should consider running our own bamboo, or maybe Hudson as it seems better. We could have 1 or 2 boxes by our sponsor Free to support it - if you want to have a try I can give you the server details

@mattab commented on January 18th 2010 Owner

I looked at xing, interesting to build this in php! the project hasn't seen an update for a while though.. I used Hudson in the past, and people were very happy with it; !http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson

@robocoder commented on January 18th 2010 Contributor

I thought we could use xinc to avoid using a java server, but since webtest is java-based, we might as well use bamboo or hudson.

@mattab commented on February 11th 2010 Owner

Note: the current Bamboo is at http://bamboo.openx.org:8085/browse/PIWIK-TRUNK but seems to fail due to some errors. We would like to setup our own integration build server at eg. tests.piwik.org

@mattab commented on May 9th 2010 Owner

I asked atlassian and they offered a free license of Bamboo. However, it might be better for us to pay for hosted continuous integration server (bamboo or hudson). If anyone knows such a hosted service, let us know!

@robocoder commented on May 17th 2010 Contributor

System requirements are:

  • multiple PHP CGI versions (e.g., 5.1.3, 5.2.0, 5.2.6, 5.2-SVN, 5.3.1, 5.3-SVN)
  • Tomcat (or other Java server) to support bamboo or hudson; or Apache if we go with xinc
  • Java runtime, ant, JDK for WebTest
  • MySQL (and in the future, other databases)

Is Free still offering a box (or two)? Maybe we can set this up ourselves.

@mattab commented on May 17th 2010 Owner

Anthon,
Yes the servers are still available. I can give you the details by email if you'd like to give it a try.

@robocoder commented on May 31st 2010 Contributor

Should test Piwik with:

  • safe_mode disabled and no open_basedir restriction
  • safe_mode enabled
  • open_basedir restriction
  • safe_mode enabled and open_basedir restriction

To test the one click update process, a script could download and install the latest release (latest.zip). Then instead of api.piwik.org/1.0/getLatestVersion, it could check api.piwik.org/1.0/getLatestSvnRevision, and update to the latest svn snapshot, e.g., !http://dev.piwik.org/trac/changeset/{latest revision}/trunk?old_path=%2F&format=zip. (Unlike the build script, Trac packages it in the "trunk" folder instead of "piwik".) It gets a little trickier because the piwik_option table stores the version number in UpdateCheck_LatestVersion, not the SVN version.

@robocoder commented on May 31st 2010 Contributor

It would be nice if there was some support for tinderbox-like scripts to collect build results from remote build servers because it would be desirable to have builds continuously tested on Windows IIS and XAMPP.

@robocoder commented on June 3rd 2010 Contributor

Should also include testing with E_STRICT because some environments don't permit the error_reporting level to be overridden by .htaccess or scripts.

@robocoder commented on June 3rd 2010 Contributor

(In [2269]) refs #818 - unit tests with both pdo_mysql and mysqli adapters

@robocoder commented on June 3rd 2010 Contributor

(In [2270]) refs #818 - fix typo in target

@robocoder commented on June 3rd 2010 Contributor

(In [2271]) refs #818 - bamboo.openx.org doesn't have mysqli extension ... comment out for now

@robocoder commented on June 5th 2010 Contributor

Replying to vipsoft:

It would be nice if there was some support for tinderbox-like scripts to collect build results from remote build servers because it would be desirable to have builds continuously tested on Windows IIS and XAMPP.

Cool. Hudson and Bamboo support remote agents, or on the cloud (Amazon EC2) -- what Atlassian calls "elastic agents". Neither provide Amazon Machine Images for Windows, nor support for Microsoft Azure.

@robocoder commented on June 13th 2010 Contributor

Minor issue with Hudson on Jetty: http://issues.hudson-ci.org/browse/HUDSON-6760

@robocoder commented on June 18th 2010 Contributor

(In [2318]) refs #818 - build directory missing a folder

@robocoder commented on June 18th 2010 Contributor

Note to self: It appears there's a bug in HtmlUnit 2.7 that causes WebTest R_1804 to fail the web test after sign in. Sizzle.filter (in jquery.js 1.4.2) is throwing an exception (Syntax error, unrecognized expression: ul li) for this selector in superfish-modified.js, "li:has(ul li)".

@mattab commented on June 30th 2010 Owner

Anthon, I have a few questions

  • Will Hudson run on Mysql, mysqli all tests?
  • Is there a way to trigger the JS tests for piwik.js, as well as unit tests and webtests?
  • Would it be better if we migrate to PhpUnit if we want to have the unit tests ran easily (right now Bamboo doesn't parse Simpletest output)

I'm working on acceptance tests that will test Tracker with known data sets, and then call all API methods and Widgets, and check the XML/ HTML output. I hope to finish this in the next few days, which would give great coverage to Piwik testing. Do you know when Hudson service could be available, any chance before end of Month/1.0 release?

Thanks

@robocoder commented on June 30th 2010 Contributor

Replying to matt:

Anthon, I have a few questions

  • Will Hudson run on Mysql, mysqli all tests?

Eventually. I commented out the mysqli unit test because bamboo.openx.org (or rather its elastic agent) doesn't have the mysqli extension installed.

  • Is there a way to trigger the JS tests for piwik.js, as well as unit tests and webtests?

We could use webtest (via HtmlUnit browser simulation). Need to investigate further.

  • Would it be better if we migrate to PhpUnit if we want to have the unit tests ran easily (right now Bamboo doesn't parse Simpletest output)

I might be able to wrap it in a webtest testcase.

I'm working on acceptance tests that will test Tracker with known data sets, and then call all API methods and Widgets, and check the XML/ HTML output. I hope to finish this in the next few days, which would give great coverage to Piwik testing. Do you know when Hudson service could be available, any chance before end of Month/1.0 release?

As of last nite, I have it running locally but the webtest is failing on the second-to-last step. I tried using Webtest 3.0 and the same version of webtest as on bamboo.openx.org (r 1783) but both choked, so I'm now running the latest snapshot and trying to locate the source of the failure in HtmlUnit. Right now, it's not looking hopeful for end-of-the-month.

@robocoder commented on June 30th 2010 Contributor

Re: piwik.js. It looks like we can use Selenium for native browser testing. (There's a Hudson plugin for this.)

@robocoder commented on July 1st 2010 Contributor

(In [2414]) refs #818 - phpdoc 1.4.3 expects this to be an absolute path

@robocoder commented on July 1st 2010 Contributor

Looks like #660 works around the problem in HtmlUnit.

I'm going to start moving my build setup over to dev6.

@mattab commented on July 1st 2010 Owner

Awesome.

The continuous build server running Unit tests as well as webtests will be a life saver!! (not to note the possibility of running mysqli and later postgresql...)

@robocoder commented on July 2nd 2010 Contributor

Another option for native browser testing of piwik.js is to run our own Testswarm instance. http://wiki.github.com/jeresig/testswarm/

@robocoder commented on July 6th 2010 Contributor

It looks like bamboo.openx.org is down, so there's greater urgency to get this setup on dev6.

In r2429, Matt indicates that Config.test.php fails when run standalone. We should definitely consider iterating through the unit tests, in addition to all_tests.php, as this can catch some side-effects (due to globals).

@mattab commented on July 6th 2010 Owner

Fixed the config.test.php in r2433

@robocoder commented on July 7th 2010 Contributor

(In [2445]) refs #818 - add webtest wrapper around unit tests

@robocoder commented on July 11th 2010 Contributor

Hudson is now running at http://qa.piwik.org:8080/ with some the unit tests are failing. (See other tickets.)

I'll open a new ticket for improvements.

@halfdan commented on July 11th 2010 Member

Nice work! The feeds are broken though:

http://qa.piwik.org:8080/hudson/rssFailed

The items links to iproxy.ca:8080.

@robocoder commented on July 11th 2010 Contributor

Thanks. Hopefully that's the last config migration change.

@robocoder commented on July 12th 2010 Contributor

(In [2471]) refs #818 - set-up database_tests section for unit testing via webtest wrapper

@mattab commented on July 12th 2010 Owner

(In [2472]) Refs #1446 Refs #818

  • Fixing unit tests on local box by ignoring ExampleAPI.getPiwikVersion which would fail at every new version

Note: on my box, when I run all_tests.php, I don't see the green bar at the bottom of the page anymore. It simply displays the Time and Memory delta. I didn't see any commit that coudl have broken this so I'm a bit confused as to why the green/red bar doesn't display anymore?

@mattab commented on July 12th 2010 Owner

Reopened as it seems unit tests are not being executed?
See for example output in: http://qa.piwik.org:8080/hudson/job/Piwik/93/console

Also few other questions

  • can we access webtests artifacts? this is important to see what webtests failed, and see what the output was like.
  • can we access integration tests artifacts (eg. in tests/integration/processed/ ?). This is important in case an integration test fails on Hudson but not in local box.
  • are you planning to use Piwik_QA twitter to post build results?

thanks!

@mattab commented on July 12th 2010 Owner

(In [2474]) Refs #1125

  • fixing integration tests
  • fixing bug when no visit (called function on non object)
    Refs #818: all_tests.php is now green on my box
@mattab commented on July 12th 2010 Owner

(In [2478]) Refs #818
When tests fail with a fatal error, it will now display as expected in the test page. This was the cause for the green/red bar not displaying (a fatal error was thrown in the UserSettings API called from the Integration tests, and was hidden).

@robocoder commented on July 13th 2010 Contributor

(In [2485]) Refs #818 - upcase words in installation steps; use "JavaScript" consistently; fixes WebTest (installation)

Artifacts can be access through the "Workspace" link, once you're logged in, e.g.,

(There's a configuration option to archive artifacts somewhere...)

Yes, the plan is to use Piwik_QA to tweet build results.

@halfdan commented on July 13th 2010 Member

Is it possible to make Artifacts public? I can't login and would very much like to see the current build status and possible errors.

http://trac-hacks.org/wiki/HudsonTracPlugin would be nice for trac to see the build history from hudson in the trac timeline.

@robocoder commented on July 13th 2010 Contributor

re: public artifacts are possible, but we'll have to review the artifacts to ensure no passwords and internal paths are disclosed.

re: hudson trac plugin. Good idea. We don't currently use the Bamboo Trac plugin that's currently installed.

@robocoder commented on July 13th 2010 Contributor

I've rebuilt php with --enable-mbstring, so the mb_convert_encoding calls work (requires for .csv and .tsv output).

The webtest wrapper on the unit tests is currently failing because the output isn't proper html, eg text before the <html> and after the </html>.

http://qa.piwik.org:8080/hudson/job/Piwik/ws/default/tests/results/002_UnitTests/001_response_invoke.xml/*view*/

@robocoder commented on July 13th 2010 Contributor

(In [2488]) refs #818 - clean up output

@robocoder commented on July 13th 2010 Contributor

(In [2489]) refs #818

@robocoder commented on July 13th 2010 Contributor

Current status of our Hudson server:

  • php 5.2.13 and pdo_mysql
  • unit tests via php-cli (excludes tests/integration) passes
  • webtest install passes
  • webtest unit tests (includes tests/integration) passes
  • phpdocs generated
@mattab commented on July 13th 2010 Owner

Are the phpdocs artifacts available for download? It would be cool to link the http://piwik.org/docs/tracking-api/ doc page to the PiwikTracker API page (to the doc from latest successful build maybe?).

@robocoder commented on July 14th 2010 Contributor

I made some configuration changes to the job, but they're not having any effect. It appears that webtest results are treated as build artifacts, and artifacts are not published if the build fails.

I'll try scripting a workaround.

@robocoder commented on July 15th 2010 Contributor
@robocoder commented on July 15th 2010 Contributor

In [2502], exclude integration tests if REQUEST_UR or HTTP_HOST not defined; this means we can now fully test with php-cli if these CGI environment variables are defined.

@robocoder commented on July 15th 2010 Contributor

(In [2503]) refs #818 - PATH_INFO and SCRIPT_NAME are also accepted

@mattab commented on July 15th 2010 Owner

Great for artifacts. Should the URL be prefixed by the job ID? (or do we only keep artifacts for last build?)

@mattab commented on July 15th 2010 Owner

When you say, "if generated", are the latest successful build doc pages always going to be there (ie. failing builds don't delete old docs). Can I safely link to http://qa.piwik.org/phpdocs/Piwik/PiwikTracker.html ?

@robocoder commented on July 15th 2010 Contributor

Webtest results are only for the latest build.

Yes, should be safe to link to the docs. It's only overwritten if the build succeeds.

@mattab commented on July 15th 2010 Owner

I linked the page from the bottom of: http://piwik.org/docs/tracking-api/

This Issue was closed on July 15th 2010
Powered by GitHub Issue Mirror