PCF Testing

This will be the first of a few posts using Sikuli for automated testing of Miro and Miro Video Converter, mainly because I am in love with the Sikuli Project, have been learning tons as I work on this, and hope others will want to use the test suites to contribute to PCF testing, and perhaps help create some.

Background

I first heard about the Sikuli Project from one of our developers in Jan 2010, right after we had purchased a 1 year license for Eggplant.   It was with a heavy-heart that we had decided to go with single node license of the expensive and proprietary tool, but we needed something.

The early releases of Sikuli were cool, but were missing what I considered a critical feature, text recognition.  The 0.10.2 release in August 2010 showed some promise and we decided that we would not renew the Eggplant license at the end of 2010, but instead commit to reworking the automated tests to use Sikuli.

The Sikuli-X release series introduces a ton of new features and improvements including text recognition and way to run setup the tests as python unittest suites, making the dream of an easily maintainable, cross-platform automation test suite a reality.

Some Advantages of using Sikuli over other tools

As I continue to work on creating the Miro test suites, I am hugely grateful to the team that makes Sikuli, I think they have done a fine job with the application, documentation and support.   When I compare it to my previous experiences with using Eggplant,  it’s wins all comparisons.  For example:

1. Licensing and Sharing: Sikuli Script and Sikuli IDE are both released under the MIT License. The app and our tests can be used by anyone who wants to try them. With Eggplant I was the only person able to use the license, locked to a specific machine and os. If my internet connection went off during a test run, the license checker would fail and my tests would crash out.  An extra hassle that nobody really needs.

2. Language:  While they provide a GUI for screen capture, I think the real advantage lies in the command-line tool.  With that, you can create tests in a real language. For both the Miro automation project and other PCF web projects that use selenium-rc, I’ve been learning the basics of python.  With the tests set up as python unittests, it’s pretty simple to create easily maintainable tests from re-usable parts.  At the same time learning something that’s useful beyond this particular task.

3. Text Recognition coolness:  Since Miro uses native widgets and testing miro requires adding a lot of different feeds and finding items, solid text recognition is the most important feature to me. Creating a screenshot for each feed, for each os, would be a nightmare.  And having to define text styles for each size, font, style, os as you do with Eggplant is equally tedious.  Sikuli uses the opencv library, and in general it just works.  There are a few instances where some fonts or sizes are harder to find, but it was pretty simple to work around those cases for the miro tests.

4. Support and Documentation:  When I ran into some issues with RC1 of Sikuli-X – I filed a bug and got some immediate feedback.  The bug itself was a blocker for me, and the developer actually provided me with a special build to work around the issue until RC2 was released. I love that!  The documentation is clear, readable and accessible.

The Miro Test Suites

The test suites for Sikuli testing assume that you are using Sikuli X 1.0 (rc2 or later) and are designed around the instructions from the How to Run Sikuli from the Command line page, section Running sikuli-script.jar from Command line. All the tests are available in a git-hub repository under the PCF-qa project.  And documentation in on the PCF wiki, Miro Test Automation page.

Right now, there is a quick-test run that takes care of about general build checkout to make sure all the parts and pieces are in order, and takes about 30 minutes to run.  If you choose to run all the tests, it takes about 3 hours for the tests to complete.

I’m still actively working on adding tests to improve coverage, but it should be possible to run tests on Windows, Ubuntu and OS X (10.6 or 10.7) and get some meaningful results.

Summary

This post is mainly my way to say Thank You! to the Sikuli team, as I think their tool is fabulous.  Anyone who wants to try out the Miro tests is welcome to give it a shot.  This has been a great learning experience for me so far  – so if you see places where the tests can be improved, please let me know or send a patch.

On Saturday at about the same time the lightening talks were going on at the Open Video Conference, I came home and had my own lightening dialog.

It struck about 400m from my house, tearing up the road.

It traveled through the ground and found the phone lines, blowing off the front cover and frying the wires on all the boxes along the road.

When it got to my house, it scorched the wall and box where it connects to the house phone lines.

It passed along the line, to the router, taking that out for good.

It even went from the router and damaged the ethernet port of the pc that was connected.

Lesson learned: unplug when the lightening comes.

One aspect of testing Miro has just gotten a lot more fun!

Yesterday Paul released and update to the Miro Guide. A portion of the testing of the guide updates was accomplished using a few automated test suites driven by Selenium Core.  We chose to use selenium core because we wanted to integrate with our existing test case management tool, litmus as well as allow everyone to be able to participate in the testing.

This testing is for the miro test channel guide, which is where the development takes place before updating to the live guide.  It would be great to get more volunteers and a greater variety of browsers tested.  If you just want to give it a spin, you can open the test suites directly in your browser of choice:

• FrontPageNav: checks the basic links and layout of the guides main page.
• Register User Actions:logs into a test account verifies some of the actions of a logged in user.
• Submit Feed: error checking of the submit feed form.

There are a few more test suites, and if you want to participate in guide testing, the best thing to do is to go to Litmus select the testchannelguide2.parti… test run, enter your configuration, select the selenium (automated) test group and follow the very simple instructions.  Testing via litmus will allow us to track the tests run, errors and browser used.

It would be great to get some feedback from others familiar with using selenium.  If you would like to get involved in the test creation process, here is a page with much more details about miro guide testing with selenium.

So you’ve found a bug, or maybe you have an enhancement request.  You’ve taken the time to fill out the form, provide useful detailed information, log files… click Submit.  You check your email the next day and find that all your work has been returned with the simple statement:  Duplicate.

Arrgh.

You can avoid all that with a few simple search tips – which will save yourself some effort and help us track and reproduce existing bugs.

If you are running the regression test cases and you encounter what you think is a bug, the easiest way to look for a duplicate is to first check those that are associated with the test case.  Along the right side of each test case title, there is a view button.

Clicking that button will open a table containing the history of that particular test case and contain links to any bugs reported.  Clicking on the comment will display brief information the tester entered about the error.  Selecting the link in the right column will open the bug in bugzilla.

If you use firefox for your browser, then the searching is super easy once you add bugzilla to your list of search engines.  Go to the PCF instance of bugzilla.  Then, right-click in the search engine pulldown list and select Add Bugzilla.  After that – you can select bugzilla as your engine, enter some descriptive words and it will perform a quick search of only open bugs.   Usually a quick browsing this list of bugs and enhancement requests is all it takes to find the duplicate.

And lastly, what about when you get an automatic crash reporter notification that look like this:

For the 2.0 release, we have added a new diagnostics tool.  From the Miro Help menu, select Diagnostics.  This will open a window that displays essential information about Miro on your system.  Click the Show button for the Log file location.

If you browse through the log file, you will see some the exception information.  A crash report always starts with  —– CRASH REPORT (DANGER CAN HAPPEN) —–, after that we provide revision and system information.  Below that is the Exception information:

If you copy some of the unique words from the last line of the exception information, and drop it in the bugzilla search – you should be able to tell if that particular crash has been reported.  Reporting those crashes with log information is an extra step you can take that is very helpful.

What to do if your bug is already in bugzilla?  Please, add a comment to the bug with your operating system, build information and any extra information that can help the developers reproduce it.

I think many people are hesitant to jump into testing because everyone knows that the nightly builds are  by definition, unstable, and people already have their channels set up just as they like them.

There are two ways you can preserve and restore your data prior to starting a test run, OPML export, or by copying off your database.

Export Channels OPML

This option is built into miro and can be accomplished as easily as going to the Channels menu and selecting Export Channels (OPML)… . This will create a list of your channels and channel folders that you save off to your system. You can then import the list at any time using the Channels menu and selecting Import Channels (OPML)… .

Using this, will only save off your channel list. If you delete your database, and start over with a clean system, Importing the channels via OPML will not preserve the downloaded, watched or unwatched status of videos in each channel.

Database Backup and Restore

Currently database backup is still a manual process and the location varies with your operating system.

• Windows XP: C:\\Documents and Settings\(you)\Application Data\Participatory Culture Foundation \Miro\Support\sqlitedb
• Vista: C:\\Users\(you)\AppData\Roaming\Participatory Culture Foundation\Miro\Support\sqlitedb
• OS X: /Users/(you)/Library/Application Support/Miro/sqlitedb
• GTKX11: ~yourhome/.miro/sqlitedb

Just copy your sqlitedb file to a safe location on your system.  If something goes wrong during testing – you can do a fresh install, copy your database back to the original location, restart Miro and be back up and running.

* Note:  If you migrated your movie storage – restoring the old db would cause it to lose track of those items.

I hope these two options make you feel a little more comfortable about diving in to some of the testing builds.  If anyone out there wants to help make the backup / restore process a little less manual – reply to this post.