indexing items by ID rather than URL

Yesterday I made Forumzilla index items in its datastore by ID rather than URL. I did it to fix bug 12132, which causes Forumzilla to ignore multiple items with the same URL in feeds like the wiki.mozilla.org Recent Changes feed.

This is a pretty significant change to be taking on the stable 0.5 branch, but I'm doing so because I think it's the necessary minimal fix for important feeds. Since not all feeds provide unique IDs for their items, I generate them if necessary by taking the SHA-1 digest of each item's URL, date, title, and description.

To get SHA-1 digest functionality, I converted Paul Johnston's JavaScript implementation of the SHA-1 algorithm to a JavaScript XPCOM component which implements the nsISHA1Service interface. I might have used his implementation of the MD5 algorithm instead, but its BSD license comes with an advertising clause (I think because he's reusing some code that includes such a clause), while his SHA-1 implementation's BSD license contains no such clause.

To prevent Forumzilla from redownloading already downloaded items, I wrote some code that checks to see if new items are actually old items that were previously indexed by URL. If so, I just convert the old record to a new, ID-indexed one and don't redownload the item.

With the changes, the aforementioned wiki feed now works, and my regular subscriptions all seem to work, too, but the changes will need more testing to make sure I haven't regressed any feeds in the process, so I built an 0.5 branch development package you can use to test the changes. I've tested it in Thunderbird 1.0, Thunderbird 1.5 (release candidates), and Mozilla 1.7 on Linux. Give it a try in your own favorite compatible mail client, and let me know how it works for you.

handling mailing lists, newsgroups, and feeds

Users participate in discussions and receive announcements via a variety of delivery methods, including mailing lists, newsgroups, and feeds. Although these methods are all technologically different, users interact with them in roughly the same ways. They subscribe to and unsubscribe from a source of discussion/announcements, collect its messages into folders, sometimes save those messages indefinitely, and other times periodically delete old messages.

Users don't care about the technology providing communication, they care about the communication itself. So the UI of a messaging client like Thunderbird shouldn't arbitrarily differentiate between delivery methods but rather focus on what users actually want to do with the messages they receive. Thunderbird's UI should include the following features:

• ways of subscribing to sources (in this one regard, the UI may vary noticably between methods, since users tend to stumble upon them in different ways, and it may be difficult to hide some details of the subscription process);
• when a user subscribes to a source, the client automatically creates a collection (a.k.a. "folder") in which to store the source's messages;
• global and per-source options (with sensible defaults) for receiving messages in the Inbox, in a collection, or in both places;
  
• readily-available contextual options (f.e. toolbar buttons, context menu items, buttons embedded into the message display, etc.) for managing sources (i.e. unsubscribing from them, changing their settings);
• when a user receives a message from a discussion/announcement to which he isn't subscribed, readily-available contextual options for subscribing to it;
• global and per-source options for deleting messages after a certain amount of time, a certain number of messages, or a certain amount of storage space;
• a way to browse the list of all subscriptions and manage them;
  
• a distinction between turning off message reception (either temporarily or permanently) and removing the source and its messages completely (not sure which should be called "unsubscribing"; probably the former);

Note that unsubscribing from a mailing list is technically very different from unsubscribing from a feed, since mailing list subscriptions are stored on a list server. But from the user's perspective, it's irrelevant where or how the subscription is stored, since either way she wants the same thing: to stop receiving the messages. Users shouldn't have to learn two different UIs for doing the same thing just because the technology behind their behavior differs, just like they shouldn't have to learn different ways of accelerating depending on whether their car has a carburetor or fuel injection; Thunderbird should handle those details under the hood.

For feeds, that means no longer polling the feed. For mailing lists, it means interacting with the server to cancel the subscription. And even though there's no standard way to unsubscribe from a mailing list, most lists are run by one of a small set of list server packages which are identifiable from the headers they add to messages, so it's possible to implement a few package-specific handlers that can automate unsubscribing from 99% of lists.