2011-08-11

Why the Add-on SDK Doesn't "Land in mozilla-central"

Various Mozillians sometimes suggest that the Add-on SDK should "land in mozilla-central" and wonder why it doesn't. Here's why.


The Add-on SDK depends on features of Firefox (and Gecko), and the SDK's development process synchronizes its release schedule with Firefox's. Nevertheless, the SDK isn't a component of Firefox, it's a distinct product with its own codebase, development process, and release schedule.

Mozilla makes multiple products that interact with Firefox (addons.mozilla.org, a.k.a. AMO, is another), and distinct product development efforts should generally utilize separate code repositories, to avoid contention between the projects regarding tree management, the stages of the software development lifecycle (i.e. when which branch is in alpha, beta, etc.), and the schedules for merging between branches.

There can be exceptions to that principle, for products that share a bunch of code, use the same development process, and have the same release schedule (cf. the Firefoxes for desktop and mobile). But the SDK is not one of those exceptions.


It shares no code with Firefox. Its process utilizes one fewer branch and six fewer weeks of development than the Firefox development process, to minimize the burden of branch management and stabilization build testing on its much smaller development team and testing community. And it merges its branches and ships its releases two weeks before Firefox, to give AMO and addon developers time to update addons for each new version of the browser.

Living in its own repository makes it possible for the SDK to have these differences in its process, and it also makes it possible for us to change the process in the future, for example to move up the branch/release dates one week, if we discover that AMO and addon developers would benefit from three weeks of lead time; or to ship twice as frequently, if we determine that doing so would get APIs for new Firefox features into developers' hands faster.

Finally, the Jetpack project has a vibrant community of contributors (including both organization staff and volunteers) who strongly prefer contributing via Git and GitHub, because they find it easier, more efficient, and more enjoyable, and for whom working in mozilla-central would mean taking too great a hit on their productivity, passion, and participation.

Mozilla Labs innovates not only on features and user experience but also on development process and tools, and while Jetpack didn't lead the way to GitHub, we were a fast follower once early experiments validated its benefits. And our experience since then has only confirmed our decision, as GitHub has proven to be a fantastic tool for branch management, code review/integration, and other software development tasks.

Other Mozillians agree: there are now almost two hundred members and over one hundred repositories (not counting forks) in the Mozilla organization on GitHub, with major initiatives like Open Web Apps and BrowserID being hosted there, not to mention all the Mozilla projects in user repositories, including Rust and Zamboni.


Even if we don't make mozilla-central the canonical repository for SDK development, however, we could still periodically drop a copy of the SDK source against which Firefox changes should be tested into mozilla-central. And doing so would theoretically make it easier for Firefox developers to run SDK tests when they discover that a Firefox change breaks the SDK, because they wouldn't have to get the SDK first.

But the benefit to Firefox developers is minimal. Currently, we periodically drop a reference to the SDK revision against which Firefox changes should be tested, and developers have to do the following to initiate testing:

  wget -i testing/jetpack/jetpack-location.txt -O addon-sdk.tar.bz2
  tar xjf addon-sdk.tar.bz2
  cd addon-sdk-[revision]
  source bin/activate
  cfx testall --binary path/to/Firefox/build

We can simplify this to:

  testing/jetpack/clone
  cd addon-sdk
  source bin/activate
  cfx testall --binary path/to/Firefox/build

Whereas if we dropped the source instead of just a reference to it, it would instead be the only slightly simpler:

  cd testing/jetpack/addon-sdk
  source bin/activate
  cfx testall --binary path/to/Firefox/build

Either of which can be abstracted to a single make target.

But if we were to drop source instead of a reference thereto, the drops would be larger and riskier changes. And test automation would still need to be updated to support Git (or at least continue to use brittle Git -> Mercurial mirroring), in order to run tests on SDK changes, which periodic source drops do not address.


Now, this doesn't mean that no SDK code will ever land in mozilla-central.

Various folks have discussed integrating parts of the SDK into core Firefoxincluding stable API implementations, the module loader, and possibly the bootstrapperto reduce the size of addon packages, improve addon startup times, and decrease addon memory consumption. I have written a very preliminary draft of a feature page describing this work, although I do not think it is a high priority at the moment, relative to the other priorities identified in the Jetpack roadmap.

And Dietrich Ayala recently suggested integrating the SDK into core Firefox for use by core features, by which he presumably also means the API implementations/module loader/bootstrapper rather than the command-line tool for testing and packaging addons.

Nevertheless, I am (and, I suspect, the whole Jetpack team is) even open to discussing integration of the command-line tool (or its replacement by a graphical equivalent), merging together the two products, and erasing the distinction between them, just as Firefox ships with core features for web development.  We've even drafted a feature page for converting the SDK into an addon, which is a big step in that direction.

But until that happens, farther on up the road, the SDK is its own product that we develop with its own process and ship on its own schedule. And it has good reason to live in its own repository, and a Git one at that, as do the many (and growing number of) other Mozilla projects using similar processes and tools, which our community-wide development, collaboration, and testing infrastructure must evolve to accommodate.

6 comments:

Robert said...

Currently, Jetpack addons have to be repackaged with a new SDK version to be compatible with each Firefox release, right?

Isn't that a burden we should be trying to avoid?

Pavel Cvrcek said...

Robert, I think myk brings better answer for you but as a localizer of AMO I saw option for repact add-ons based on Add-on SDK automatically. So if you host add-on on Mozilla Add-ons it should solve this problem. Not sure if this option is live on production server.

KWierso said...

Addons on AMO have two methods of compatibility bumping:
AMO's automated maxversion bumping for addons it determines to be compatible, and automatic repacking of the SDK when new versions come out.

matej said...

@Robert yes, some of my Add-on SDK were repackaged for upgraded versions of Firefox automatically.

@Myk Why not git.mozilla.org?

myk said...

@matej: I would actually very much like to host the canonical repository for the Add-on SDK on a Git server hosted by Mozilla (i.e. git.mozilla.org), as I describe in bug 528360, comment 19.

myk said...

@Robert: yes, that's a burden we should be trying to avoid, and it's a reason to consider landing API implementations in core Firefox.