PCF Testing

Help us test Miro 5, Release Candidate builds

You can download them here:

• Windows (5.0-rc1)
• OS X (5.0-rc1)
• Source (5.0-rc1)

To run the regression tests

1. Go to litmus.pculture.org and sign in or create an account if you don’t already have one.
2. Select the recommended test run for Miro 5.0 Beta and RC Testing
3. Select your OS configuration and start running tests from the groups that are listed.
4. For each test that you run, please mark it pass / fail / unclear. Don’t worry if you can’t run all the tests.
5. If you encounter an error, please file a bug and provide the bug number in the results and a quick description in the comments.

Bugzilla information (Fixed / Open / Create)

• Miro 5.0 Fixed bugs
• Miro 5.0 – all bugs open and closed
• Submit a new bug

Extra info

• Please read Good Bug Reports for bug guidelines
• If you are new to testing you can check out our getting started guide
• If your time is limited, doing the 10 short tests in the Quicktest will be immensely helpful.
• Track the testing progress

This will be the first of a few posts using Sikuli for automated testing of Miro and Miro Video Converter, mainly because I am in love with the Sikuli Project, have been learning tons as I work on this, and hope others will want to use the test suites to contribute to PCF testing, and perhaps help create some.

Background

I first heard about the Sikuli Project from one of our developers in Jan 2010, right after we had purchased a 1 year license for Eggplant.   It was with a heavy-heart that we had decided to go with single node license of the expensive and proprietary tool, but we needed something.

The early releases of Sikuli were cool, but were missing what I considered a critical feature, text recognition.  The 0.10.2 release in August 2010 showed some promise and we decided that we would not renew the Eggplant license at the end of 2010, but instead commit to reworking the automated tests to use Sikuli.

The Sikuli-X release series introduces a ton of new features and improvements including text recognition and way to run setup the tests as python unittest suites, making the dream of an easily maintainable, cross-platform automation test suite a reality.

Some Advantages of using Sikuli over other tools

As I continue to work on creating the Miro test suites, I am hugely grateful to the team that makes Sikuli, I think they have done a fine job with the application, documentation and support.   When I compare it to my previous experiences with using Eggplant,  it’s wins all comparisons.  For example:

1. Licensing and Sharing: Sikuli Script and Sikuli IDE are both released under the MIT License. The app and our tests can be used by anyone who wants to try them. With Eggplant I was the only person able to use the license, locked to a specific machine and os. If my internet connection went off during a test run, the license checker would fail and my tests would crash out.  An extra hassle that nobody really needs.

2. Language:  While they provide a GUI for screen capture, I think the real advantage lies in the command-line tool.  With that, you can create tests in a real language. For both the Miro automation project and other PCF web projects that use selenium-rc, I’ve been learning the basics of python.  With the tests set up as python unittests, it’s pretty simple to create easily maintainable tests from re-usable parts.  At the same time learning something that’s useful beyond this particular task.

3. Text Recognition coolness:  Since Miro uses native widgets and testing miro requires adding a lot of different feeds and finding items, solid text recognition is the most important feature to me. Creating a screenshot for each feed, for each os, would be a nightmare.  And having to define text styles for each size, font, style, os as you do with Eggplant is equally tedious.  Sikuli uses the opencv library, and in general it just works.  There are a few instances where some fonts or sizes are harder to find, but it was pretty simple to work around those cases for the miro tests.

4. Support and Documentation:  When I ran into some issues with RC1 of Sikuli-X – I filed a bug and got some immediate feedback.  The bug itself was a blocker for me, and the developer actually provided me with a special build to work around the issue until RC2 was released. I love that!  The documentation is clear, readable and accessible.

The Miro Test Suites

The test suites for Sikuli testing assume that you are using Sikuli X 1.0 (rc2 or later) and are designed around the instructions from the How to Run Sikuli from the Command line page, section Running sikuli-script.jar from Command line. All the tests are available in a git-hub repository under the PCF-qa project.  And documentation in on the PCF wiki, Miro Test Automation page.

Right now, there is a quick-test run that takes care of about general build checkout to make sure all the parts and pieces are in order, and takes about 30 minutes to run.  If you choose to run all the tests, it takes about 3 hours for the tests to complete.

I’m still actively working on adding tests to improve coverage, but it should be possible to run tests on Windows, Ubuntu and OS X (10.6 or 10.7) and get some meaningful results.

Summary

This post is mainly my way to say Thank You! to the Sikuli team, as I think their tool is fabulous.  Anyone who wants to try out the Miro tests is welcome to give it a shot.  This has been a great learning experience for me so far  – so if you see places where the tests can be improved, please let me know or send a patch.

Miro 4, Release Candidate 2 builds are available, and if you would like to contribute, here is how you can help:

1. Download the build for your operating system: windows, os x, or source

2. Connect your Android or USB device, and import and sync your files.

3. Play all your music. Make dynamic playlists and shuffle through.

4. Try out the home-sharing feature. Stream or download music and videos on your local network.

5. Tell us what happens, comment here, or file a bug.

6. Help update the translations.

For more in-depth testing, here are the regression testing details:

Read the rest of this entry »

This weekend we put out a Beta build for Miro 4.  There are some amazing new features and bug fixes!  Please download a copy and help us test it.

os x | windows | source

Have an Android or Blackberry device? Help us add support for your device by filling out the form at http://bit.ly/mirodevicehelp.

Rock out your tunes with our sweet UI

• Share with other Miro Libraries on your local network
• Sync devices and copy media
• More ways to make playlists
• View the playback and download history
• Improved searching and organization of your music
• Easier ways to add music and video to your collection
• UI Navigation with the keyboard
• Better performance

It’s cleaner, clearer, simpler and faster!!! 

Please help us test.

This is a beta build and we know there are some more bugs to fix, help us find the ones we don’t know about yet.

Bugzilla information (Fixed / Open / Create)

• Miro 4.0 Fixed bugs
• Miro 4.0 – all bugs open and closed
• Submit a new bug

Litmus Testing

1. Go to litmus.pculture.org and sign in or create an account if you don’t already have one.
2. Select the recommended test run for Miro 4.0 Beta and RC Testing
3. Select your OS configuration and start running tests from the groups that are listed.
4. For each test that you run, please mark it pass / fail / unclear. Don’t worry if you can’t run all the tests.
5. If you encounter an error, please file a bug and provide the bug number in the results and a quick description in the comments.

Extra info

• Please post a bug for any issues you find, it’s the fastest way for it to get attention.
• Please read Good Bug Reports for bug guidelines
• If you are new to testing you can check out our getting started guide
• If your time is limited, doing the 10 short tests in the Quicktest will be immensely helpful.
• Track the testing progress

* Updated links for Miro 3.0 – Release Candidate 3

Miro 3.0 Release Candidate 3 is now available on the nightly builds page for testing:

• windows,
• mac,
• source.

Here’s a quick rundown of some of the new features and how they work.

Subtitles

Whether embedded or external, subtitles display be enabled and disabled during playback. The feature is implemented for all operating systems and more technical details are available.

To display embedded or sidecar subtitle files:

1. Start playback on a video file.
2. Click the Playback -> Subtitles menu, or the Subtitles icon on the playbar.
3. Available languages or tracks are displayed for selection.
4. Choose the file and subtitles will display.

To choose a subtitle file from a location on your system:

1. Start playback on a video file.
2. Click the Playback -> Subtitles menu, or the Subtitles icon on the playbar.
3. Choose the Select a Subtitles Files… option and browse to the subtitle file.
4. While you are looking for the file, playback will pause, and resume when the selection is made.

There is also a global preference available to automatically display or disable subtitle files.

Edit Cataloged Items

Users are now provided with greater control over how items are displayed in the Library. This is useful to change titles, provide better descriptions or change the audio or video classification. To edit an item:

1. Select the item
2. From the File (Video) menu, select Edit Item, or right-click and select Edit Item from the context menu.
3. In the Edit item dialog, you can modify title, description, and whether it should be treated as video, audio or other for playback purposes.
4. The full path to the item on your system is displayed in the dialog.
5. Lastly, files can be dragged and dropped between the Video, Audio and Other sections of the library. This will automatically change it’s setting for playback purposes.

Play file with an external player

This long requested feature, useful for advanced users and people who prefer an additional playback option was implemented by a contributor. Special thanks go to Jason Woofenden. To use an external player:

1. Right-click on an item, and select Play Externally from the context menu.
2. The item will be opened with the system default player for that file type.

There is also a global preference available under the Playback settings to always use an external player.

The developers have fixed a TON of BUGs for a 2.5.3 release and it is time to get some serious testing done on the posted release candidate build.

Changes include some fixes to some unknown errors, updates to vlc and libtorrent, changes for some os x 10.6 specific issues, and some other file naming problems on windows.

Thanks to a contribution for Jason, we have also added the ability to play any item in an external player via the right-click context menu. This is the initial implementation of a long-requested feature.

Because there are so many changes that affect all part of Miro, we really need to get to broad testing on this.  Please download the nightly build for your os: windows, os x, source.

There is a 2.5.3 rc test run in litmus.  The focus areas test group consists of tests for the areas that have been changed the most.  Please give us some of your time and run few test cases.

The first release candidate build for Miro 2.5 has been posted:

• windows
• os x
• source

I’ve been testing them on windows and os x, and they are looking very solid.  It would be great if a few more people could run some of the regression tests, as the lightening destroyed my machine that had linux and windows XP.  Also – I don’t have a machine with os x 10.5 (only 10.4.11).

Also – for windows users, it appears that the update to vlc 1.0 has fixed bug #11300 a pretty annoying problem where windows would report an error on shutdown.

Finally, please report any issues on bugzilla, get satisfaction, or just comment here.

We are about to start looking for more help testing the nightly builds for 2.5-svn and I wanted to give a few more details about things that have changed and areas to focus on with the testing.

The most obvious change is the reorganization of the Library.  Instead of Library, New and Individual Downloads, you will now be presented with Video, Audio, Other and Downloading.

The video and audio main view has filters for All, Unwatched and Non-feed items to make it easier to find your downloaded shows.

There are also significant changes to the audio playback ui.  Audio playback no longer displays a blank window.  Instead the interface remains active and the progress bar on the chrome displays the playing file.

Speed and the database changes

Both Ben and Will and Nicholas have mentioned this in past posts.  Ben redid the way the database is loaded and stored.  He’s given us some solid startup speed improvements.  He has also mostly given up on pickles, so in general the items are plain and readable.  If you are a db junkie, you can finally see inside to the data guts of miro.

Because of the big database changes, we have finally started backing up the old database prior to any upgrades.  This should make you feel a bit more secure about testing nightly builds because it’s a lot easier to go back without data loss.

The Database Backup page has been updated to give instructions on where and how to find the database and return to an earlier version if necessary.

One aspect of testing Miro has just gotten a lot more fun!

Yesterday Paul released and update to the Miro Guide. A portion of the testing of the guide updates was accomplished using a few automated test suites driven by Selenium Core.  We chose to use selenium core because we wanted to integrate with our existing test case management tool, litmus as well as allow everyone to be able to participate in the testing.

This testing is for the miro test channel guide, which is where the development takes place before updating to the live guide.  It would be great to get more volunteers and a greater variety of browsers tested.  If you just want to give it a spin, you can open the test suites directly in your browser of choice:

• FrontPageNav: checks the basic links and layout of the guides main page.
• Register User Actions:logs into a test account verifies some of the actions of a logged in user.
• Submit Feed: error checking of the submit feed form.

There are a few more test suites, and if you want to participate in guide testing, the best thing to do is to go to Litmus select the testchannelguide2.parti… test run, enter your configuration, select the selenium (automated) test group and follow the very simple instructions.  Testing via litmus will allow us to track the tests run, errors and browser used.

It would be great to get some feedback from others familiar with using selenium.  If you would like to get involved in the test creation process, here is a page with much more details about miro guide testing with selenium.