Paul Swartz's PCF Devblog

We just rolled out a couple new features onto the Guide.

1. Featured Queue: This is mostly relevant for moderators. Previously, to feature a channel on the front page, moderators had to manually unfeature an older channel before featuring a new one. Now, when a channel is featured, it goes into a queue with all the other channels featured by moderators. Then, once a day, the oldest featured channel is removed from the front page (but still visible on the Featured page) and replaced by an entry from the queue. It picks the channel round-robin style so that all the moderators get channels on the front page.
2. Adult Flagging: This is an oft-requested featured. Now moderators will be able to flag channels as Adult and they’ll be filtered (by default) from all views. If you want to see them, then you can visit the Adult category and agree to see them.

Matt moved the database onto its own server, which along with some new caching code has sped the site up a lot. Hopefully it’ll be able to handle the spike when Miro 1.0 comes out.

One thing that I’ve been thinking a lot about (esp. now that Miro is nearing 1.0) is an API for the Guide. I’d love for the Guide to be more directly useful from Miro (think rating channels, getting recommendations, etc), but also more useful for other projects. After all, the Guide is a great resource for Internet TV channels, regardless of the software ones uses.
So I’m posing a question to the developers/users out there: what would you like to see in a Guide API? What would you like to be able to do/access? And how would you like to be able to access it?

On Wednesday, I spent the day finishing up the Landing page. It didn’t end up quite like Dean and I wanted due to some issues with the video, but I think it’s still good for the new users to get an introduction to Miro.

Thursday and Friday were spent working on the new featured channel queue. It’s not live yet, but it should make the featured channel section more useful for everyone. Now when a super-moderator features a channel, it goes into a queue. The featured section on the front page is updated once per day, pulling a channel from the queue round-robin style so that everyone gets a chance to see their channel on the front page.

This week I’ve been working on making the Guide faster in preparation for the eventual 1.0 release of Miro. The database has been getting more and more use and so I’m trying to add more caching to keep the guide going fast.

That’s what I’ll be working on for the rest of this week, along with some visual tweaks. If I’ve got time, I’ll be working on #9161, adding a category for Adult and a category for Sexuality channels.

I’ve been reading some of the architecture articles on High Scalability (YouTube, Twitter, Amazon, Flickr) and while the descriptions of hardware aren’t particularly useful to me (Miro Guide is one, about to be two, servers; not that big), but a lot of the tips and lessons learned I think may be applicable.

• Keep it Simple: I was a victim of this earlier, trying to make a system that was too complicated.
• Don’t make the database the central bottleneck: We’re running into this problem now, where sometimes a simple insert will time out because the database is being hit so much.
• Denormalize: I haven’t done this, nor do I have plans to in the near future, but it’s a least something to think about.
• Avoid complex joins: Nor this, but it may be becoming a problem. The next item may help with this
• Cache everything: We don’t do this very well, and it’s one of my goals. We just added a Squid server, so giving some hints to that about what’s changed will be good. Also, using memcached a bit more.
• Make your website an open service by creating an API. After the 1.0 release, this is one of the big features I want to work on. I think it’ll make Miro Guide a lot more useful both inside and outside of Miro and allow some refactoring of the actual guide,
• Measure, measure, measure: I starting doing this a couple weeks ago. Now I get an daily e-mail saying what parts of the Guide are running slowly. It’s both a good way to see what needs work, but also how well the caching is working.
• Abstraction: again, something that I’d like to do in conjunction with the API.

This wasn’t a particularly productive week for me. Mostly I was working on the landing page, fixing some small issues and doing a new layout for the channel subscription section. I did a lot of work trying to make the test cases for guide work again, and they’re almost there. I’ve just got a couple hard-to-diagnose bugs. I also added a couple new tests for the newly-approved channel queuing feature. I’ve also made a little progress on the featured queue, but mostly that’s waiting on the test cases passing again so I can add tests for that before I start in.
Today I was working on an extension to the Guide’s error reporting mechanism. Currently, errors are logged to a file which I check periodically, but mostly I’m notified by Dean who’s notified by users. Now those errors are mailed to me, along with a bit of context so that Dean doesn’t have to play messenger.
Next week, I’ll be working on the landing page, a quick redesign of the error page to make it friendlier, the tests, and then the featured channel queue.

Most of what I’ve done this week was for the new landing page. A quick summary: it’s a three-step page: 1) A video introducing the user to Miro 2) A way to get some channels they may enjoy 3) some FAQs. Embedding the video was difficult, but I think I got it working on Windows and Mac. I don’t think it’ll work for GTK, so I’m not trying anymore.
The FAQ section is working perfectly, and I’ll be finishing up with the subscription section this week.

I also got two smaller features onto the Guide. First, there’s a new notification section at the top of pages. This will let us announce things in an unobtrusive way, and also remind users of things they’re supposed to do. Second, there’s a new queuing feature for the new channels. Instead of being dropped onto the front page and new channels pages all at once when moderators approve them, they’re added one per hour. This gives them more time on the front page and therefore more exposure.

This last week was less productive than I would have liked. I’ve been fighting off a cold, for the most part unsuccessfully. However, I have managed to get some work done on the Guide. (Features may not be on https://www.miroguide.com/ yet, but they will be soon)

• Supermoderators can now send e-mails to the owners of featured channels through the guide
• Statistics for channel owners: on the channels by user page, each channel has some simple statistics on how many people have subscribed today/this month and what rank that channel is at.
• Fixed a bug with the rating stars flashing the non-jQuery version, and a bug on Safari where the stars would be all yellow even without a rating

This week, I’ll be working on:

• Queuing for newly approved channels, so that they don’t appear in the guide all at once
• A system for displaying notifications at the top of the page, like the message that are currently there for announcing ratings and reminding users to approve their accounts
• Some small frontend speedups (enabling gzip compression, CSS sprites)
• The big feature: recommendations based on ratings.

We rolled out ratings a while ago, but we recently made a big push for them with a Top Rated page and a blog post. Now I’m throwing my hat in the ring.

Currently, there are about 950 ratings (100 of them are “Not Interested”). I’d like to start working on personalized recommendations, but I’d like to have some more data. A lot more data. I’d like about 2000 ratings so that I’ve got a good range for adding this new feature. If you all could rate some more channels, I’d really appreciate it. Also, you’ll appreciate it when the guide can tell you about new channels you’ll like.

The last week was mostly a bug-fixing week: I think I got through at least 10 bugs on Bugzilla for the Guide. I added the Top Rated channels page, and changed the layout for that and the popular channels page to show the rating.
I did a little bit of caching work, and the site is feeling a lot faster. Last week, the worst page had an average rendering time of 20 seconds. Now, the worst rendering time is 3 seconds, and the average time is 0.6 seconds. I’m still working on it with BDK to make it a little simpler on our end, but not at the expense of the caching being effective.
This week I’ll be working on adding some features: releasing approved channels slowly to avoid overloading the “New Channels” page, the ability to download individual videos from channels, and some simple statistics for channel owners.

This week I worked on fixing some display issues on Firefox and Safari. I also updated the donate bar when we broken $50k. I also finished up a couple scripts which check un-approved users. Old users were grandparented in, but new users have to give a valid e-mail address and confirm it with a code.

The rest of my time has been spent doing caching things. There’s some new code that’ll be going live in the Guide hopefully later tonight which will hopefully make things a little better. Then Ben and I will be discussing what we think the best way to make the guide keep going faster is. I’ve been doing some logging, and I’m getting some good data. Currently the slowest pages overall are the 1st channel submit step, the channel details pages and the subscription-hit pages. The slowest pages to render are the front page, the languages and popular pages, and the channel submit pages. Searching is also slow, but on average it takes about 1 fewer second to render than the popular page.