PCF Testing

Help us test Miro 5, Release Candidate builds

You can download them here:

• Windows (5.0-rc1)
• OS X (5.0-rc1)
• Source (5.0-rc1)

To run the regression tests

1. Go to litmus.pculture.org and sign in or create an account if you don’t already have one.
2. Select the recommended test run for Miro 5.0 Beta and RC Testing
3. Select your OS configuration and start running tests from the groups that are listed.
4. For each test that you run, please mark it pass / fail / unclear. Don’t worry if you can’t run all the tests.
5. If you encounter an error, please file a bug and provide the bug number in the results and a quick description in the comments.

Bugzilla information (Fixed / Open / Create)

• Miro 5.0 Fixed bugs
• Miro 5.0 – all bugs open and closed
• Submit a new bug

Extra info

• Please read Good Bug Reports for bug guidelines
• If you are new to testing you can check out our getting started guide
• If your time is limited, doing the 10 short tests in the Quicktest will be immensely helpful.
• Track the testing progress

This will be the first of a few posts using Sikuli for automated testing of Miro and Miro Video Converter, mainly because I am in love with the Sikuli Project, have been learning tons as I work on this, and hope others will want to use the test suites to contribute to PCF testing, and perhaps help create some.

Background

I first heard about the Sikuli Project from one of our developers in Jan 2010, right after we had purchased a 1 year license for Eggplant.   It was with a heavy-heart that we had decided to go with single node license of the expensive and proprietary tool, but we needed something.

The early releases of Sikuli were cool, but were missing what I considered a critical feature, text recognition.  The 0.10.2 release in August 2010 showed some promise and we decided that we would not renew the Eggplant license at the end of 2010, but instead commit to reworking the automated tests to use Sikuli.

The Sikuli-X release series introduces a ton of new features and improvements including text recognition and way to run setup the tests as python unittest suites, making the dream of an easily maintainable, cross-platform automation test suite a reality.

Some Advantages of using Sikuli over other tools

As I continue to work on creating the Miro test suites, I am hugely grateful to the team that makes Sikuli, I think they have done a fine job with the application, documentation and support.   When I compare it to my previous experiences with using Eggplant,  it’s wins all comparisons.  For example:

1. Licensing and Sharing: Sikuli Script and Sikuli IDE are both released under the MIT License. The app and our tests can be used by anyone who wants to try them. With Eggplant I was the only person able to use the license, locked to a specific machine and os. If my internet connection went off during a test run, the license checker would fail and my tests would crash out.  An extra hassle that nobody really needs.

2. Language:  While they provide a GUI for screen capture, I think the real advantage lies in the command-line tool.  With that, you can create tests in a real language. For both the Miro automation project and other PCF web projects that use selenium-rc, I’ve been learning the basics of python.  With the tests set up as python unittests, it’s pretty simple to create easily maintainable tests from re-usable parts.  At the same time learning something that’s useful beyond this particular task.

3. Text Recognition coolness:  Since Miro uses native widgets and testing miro requires adding a lot of different feeds and finding items, solid text recognition is the most important feature to me. Creating a screenshot for each feed, for each os, would be a nightmare.  And having to define text styles for each size, font, style, os as you do with Eggplant is equally tedious.  Sikuli uses the opencv library, and in general it just works.  There are a few instances where some fonts or sizes are harder to find, but it was pretty simple to work around those cases for the miro tests.

4. Support and Documentation:  When I ran into some issues with RC1 of Sikuli-X – I filed a bug and got some immediate feedback.  The bug itself was a blocker for me, and the developer actually provided me with a special build to work around the issue until RC2 was released. I love that!  The documentation is clear, readable and accessible.

The Miro Test Suites

The test suites for Sikuli testing assume that you are using Sikuli X 1.0 (rc2 or later) and are designed around the instructions from the How to Run Sikuli from the Command line page, section Running sikuli-script.jar from Command line. All the tests are available in a git-hub repository under the PCF-qa project.  And documentation in on the PCF wiki, Miro Test Automation page.

Right now, there is a quick-test run that takes care of about general build checkout to make sure all the parts and pieces are in order, and takes about 30 minutes to run.  If you choose to run all the tests, it takes about 3 hours for the tests to complete.

I’m still actively working on adding tests to improve coverage, but it should be possible to run tests on Windows, Ubuntu and OS X (10.6 or 10.7) and get some meaningful results.

Summary

This post is mainly my way to say Thank You! to the Sikuli team, as I think their tool is fabulous.  Anyone who wants to try out the Miro tests is welcome to give it a shot.  This has been a great learning experience for me so far  – so if you see places where the tests can be improved, please let me know or send a patch.

We have posted RC1 builds for Miro 4.0.2 on the nightly builds page.  You can download builds for:

• windows
• osx
• for source builds or an update to the ubuntu testing ppa – please harass bendk on irc #miro-hackers

This is a bug-fix release that contains some performance improvements and a few subtle enhancements.   You can see the list in the draft release notes.

In general there are a number of fixes and performance tweaks related to the feed updating and audio playback when there are a large number of feeds and watched folders.

We’ve also fixed the issue of the .miro file getting written to connected drives on windows and have fixed some database issues.

Any help you can provide with testing these builds is greatly appreciated.  For how you can help… Read the rest of this entry »

Miro 4, Release Candidate 2 builds are available, and if you would like to contribute, here is how you can help:

1. Download the build for your operating system: windows, os x, or source

2. Connect your Android or USB device, and import and sync your files.

3. Play all your music. Make dynamic playlists and shuffle through.

4. Try out the home-sharing feature. Stream or download music and videos on your local network.

5. Tell us what happens, comment here, or file a bug.

6. Help update the translations.

For more in-depth testing, here are the regression testing details:

Read the rest of this entry »

This weekend we put out a Beta build for Miro 4.  There are some amazing new features and bug fixes!  Please download a copy and help us test it.

os x | windows | source

Have an Android or Blackberry device? Help us add support for your device by filling out the form at http://bit.ly/mirodevicehelp.

Rock out your tunes with our sweet UI

• Share with other Miro Libraries on your local network
• Sync devices and copy media
• More ways to make playlists
• View the playback and download history
• Improved searching and organization of your music
• Easier ways to add music and video to your collection
• UI Navigation with the keyboard
• Better performance

It’s cleaner, clearer, simpler and faster!!! 

Please help us test.

This is a beta build and we know there are some more bugs to fix, help us find the ones we don’t know about yet.

Bugzilla information (Fixed / Open / Create)

• Miro 4.0 Fixed bugs
• Miro 4.0 – all bugs open and closed
• Submit a new bug

Litmus Testing

1. Go to litmus.pculture.org and sign in or create an account if you don’t already have one.
2. Select the recommended test run for Miro 4.0 Beta and RC Testing
3. Select your OS configuration and start running tests from the groups that are listed.
4. For each test that you run, please mark it pass / fail / unclear. Don’t worry if you can’t run all the tests.
5. If you encounter an error, please file a bug and provide the bug number in the results and a quick description in the comments.

Extra info

• Please post a bug for any issues you find, it’s the fastest way for it to get attention.
• Please read Good Bug Reports for bug guidelines
• If you are new to testing you can check out our getting started guide
• If your time is limited, doing the 10 short tests in the Quicktest will be immensely helpful.
• Track the testing progress

Miro 3.5 RC builds are available

• OS X (3.5-rc2)
• Windows (3.5-rc2)
• Source (3.5-rc2)

What’s New

Miro 3.5 brings some key enhancements and bug fixes:

• Video Conversions (think MVC with batch processing)
• Fixed Proxy support
• Preference to specify font and encoding for subtitles
• Preference to select display language
• Better management of queued downloads

View the Release Notes for complete change information.

Bugzilla information (Fixed / Open / Create)

• Miro 3.5 Fixed bugs
• Miro 3.5 – all bugs open and closed
• Submit a new bug

Litmus Testing

1. Go to litmus.pculture.org and sign in or create an account if you don’t already have one.
2. Select the recommended test run for Miro 3.5 Release Candidate Testing
3. Select your OS configuration and start running tests from the groups that are listed.
4. For each test that you run, please mark it pass / fail / unclear. Don’t worry if you can’t run all the tests.
5. If you encounter an error, please file a bug and provide the bug number in the results and a quick description in the comments.

Extra info

• Please post a bug for any issues you find, it’s the fastest way for it to get attention.
• Here is a nice clear guide to submitting bugs
• If you are new to testing you can check out our getting started guide
• If your time is limited, doing the 10 short tests in the Quicktest will be immensely helpful.
• Track the testing progress

* Updated links for Miro 3.0 – Release Candidate 3

Miro 3.0 Release Candidate 3 is now available on the nightly builds page for testing:

• windows,
• mac,
• source.

Here’s a quick rundown of some of the new features and how they work.

Subtitles

Whether embedded or external, subtitles display be enabled and disabled during playback. The feature is implemented for all operating systems and more technical details are available.

To display embedded or sidecar subtitle files:

1. Start playback on a video file.
2. Click the Playback -> Subtitles menu, or the Subtitles icon on the playbar.
3. Available languages or tracks are displayed for selection.
4. Choose the file and subtitles will display.

To choose a subtitle file from a location on your system:

1. Start playback on a video file.
2. Click the Playback -> Subtitles menu, or the Subtitles icon on the playbar.
3. Choose the Select a Subtitles Files… option and browse to the subtitle file.
4. While you are looking for the file, playback will pause, and resume when the selection is made.

There is also a global preference available to automatically display or disable subtitle files.

Edit Cataloged Items

Users are now provided with greater control over how items are displayed in the Library. This is useful to change titles, provide better descriptions or change the audio or video classification. To edit an item:

1. Select the item
2. From the File (Video) menu, select Edit Item, or right-click and select Edit Item from the context menu.
3. In the Edit item dialog, you can modify title, description, and whether it should be treated as video, audio or other for playback purposes.
4. The full path to the item on your system is displayed in the dialog.
5. Lastly, files can be dragged and dropped between the Video, Audio and Other sections of the library. This will automatically change it’s setting for playback purposes.

Play file with an external player

This long requested feature, useful for advanced users and people who prefer an additional playback option was implemented by a contributor. Special thanks go to Jason Woofenden. To use an external player:

1. Right-click on an item, and select Play Externally from the context menu.
2. The item will be opened with the system default player for that file type.

There is also a global preference available under the Playback settings to always use an external player.

If so, you will most certainly want to take a look at Release Candidate 1 for Miro 3.0.  Miro-3.0-rc1 is now available for testing on Windows, Mac and Linux.

We have fixed about a zillion bugs and have implemented some great features such as:

• Support For Subtitles; embedded, co-located or browse toanywhere on your system.
• Item Metadata Editing; change the title, description, or audio / video specification.
• Play External Settings; global preference for all files, and a context menu for single items.

There are major speed improvements for navigating the UI, adding files to miro, and saving large search feeds.  We’ve improved shortcuts for audio playback, made some visual improvements and much, much, more.

This release also include updates to vlc (windows) and libtorrent (all os).

We are looking for help testing the release candidate, and hope you will download it and report back to us any of your findings.

To let us know how it goes, you can:

• comment here
• report a bug
• post in the forum
• email feedback@pculture.org

If you’d like to help more, please take a look at the regression tests or see the getting started guide.

As always, all help is greatly appreciated.

Update

Miro 2.5.4-rc2 has now been posted.  Please take the time to download and check it out.

windows: http://pculture.org/nightlies/Miro-2.5.4-rc2.exe

os x: http://pculture.org/nightlies/Miro-2.5.4-rc2.dmg

source: http://pculture.org/nightlies/Miro-2.5.4-rc2.tar.gz

Read the rest of this entry »

The developers have fixed a TON of BUGs for a 2.5.3 release and it is time to get some serious testing done on the posted release candidate build.

Changes include some fixes to some unknown errors, updates to vlc and libtorrent, changes for some os x 10.6 specific issues, and some other file naming problems on windows.

Thanks to a contribution for Jason, we have also added the ability to play any item in an external player via the right-click context menu. This is the initial implementation of a long-requested feature.

Because there are so many changes that affect all part of Miro, we really need to get to broad testing on this.  Please download the nightly build for your os: windows, os x, source.

There is a 2.5.3 rc test run in litmus.  The focus areas test group consists of tests for the areas that have been changed the most.  Please give us some of your time and run few test cases.