Using Sikuli to Automate Miro Testing
Posted by Janet
This will be the first of a few posts using Sikuli for automated testing of Miro and Miro Video Converter, mainly because I am in love with the Sikuli Project, have been learning tons as I work on this, and hope others will want to use the test suites to contribute to PCF testing, and perhaps help create some.
Background
I first heard about the Sikuli Project from one of our developers in Jan 2010, right after we had purchased a 1 year license for Eggplant. It was with a heavy-heart that we had decided to go with single node license of the expensive and proprietary tool, but we needed something.
The early releases of Sikuli were cool, but were missing what I considered a critical feature, text recognition. The 0.10.2 release in August 2010 showed some promise and we decided that we would not renew the Eggplant license at the end of 2010, but instead commit to reworking the automated tests to use Sikuli.
The Sikuli-X release series introduces a ton of new features and improvements including text recognition and way to run setup the tests as python unittest suites, making the dream of an easily maintainable, cross-platform automation test suite a reality.
Some Advantages of using Sikuli over other tools
As I continue to work on creating the Miro test suites, I am hugely grateful to the team that makes Sikuli, I think they have done a fine job with the application, documentation and support. When I compare it to my previous experiences with using Eggplant, it’s wins all comparisons. For example:
1. Licensing and Sharing: Sikuli Script and Sikuli IDE are both released under the MIT License. The app and our tests can be used by anyone who wants to try them. With Eggplant I was the only person able to use the license, locked to a specific machine and os. If my internet connection went off during a test run, the license checker would fail and my tests would crash out. An extra hassle that nobody really needs.
2. Language: While they provide a GUI for screen capture, I think the real advantage lies in the command-line tool. With that, you can create tests in a real language. For both the Miro automation project and other PCF web projects that use selenium-rc, I’ve been learning the basics of python. With the tests set up as python unittests, it’s pretty simple to create easily maintainable tests from re-usable parts. At the same time learning something that’s useful beyond this particular task.
3. Text Recognition coolness: Since Miro uses native widgets and testing miro requires adding a lot of different feeds and finding items, solid text recognition is the most important feature to me. Creating a screenshot for each feed, for each os, would be a nightmare. And having to define text styles for each size, font, style, os as you do with Eggplant is equally tedious. Sikuli uses the opencv library, and in general it just works. There are a few instances where some fonts or sizes are harder to find, but it was pretty simple to work around those cases for the miro tests.
4. Support and Documentation: When I ran into some issues with RC1 of Sikuli-X – I filed a bug and got some immediate feedback. The bug itself was a blocker for me, and the developer actually provided me with a special build to work around the issue until RC2 was released. I love that! The documentation is clear, readable and accessible.
The Miro Test Suites
The test suites for Sikuli testing assume that you are using Sikuli X 1.0 (rc2 or later) and are designed around the instructions from the How to Run Sikuli from the Command line page, section Running sikuli-script.jar from Command line. All the tests are available in a git-hub repository under the PCF-qa project. And documentation in on the PCF wiki, Miro Test Automation page.
Right now, there is a quick-test run that takes care of about general build checkout to make sure all the parts and pieces are in order, and takes about 30 minutes to run. If you choose to run all the tests, it takes about 3 hours for the tests to complete.
I’m still actively working on adding tests to improve coverage, but it should be possible to run tests on Windows, Ubuntu and OS X (10.6 or 10.7) and get some meaningful results.
Summary
This post is mainly my way to say Thank You! to the Sikuli team, as I think their tool is fabulous. Anyone who wants to try out the Miro tests is welcome to give it a shot. This has been a great learning experience for me so far – so if you see places where the tests can be improved, please let me know or send a patch.