CouchDB Continuous Integration

This page explains Apache CouchDB’s Continuous Integration setup at http://ci.couchdb.org:8888/ (this is currently a private setup, all committers can request access. We will open this up gradually)

Authors note: THIS IS A WORK IN PROGRESS, everything might be wrong, all is up for grabs, please help, love, Jan.

Goals

The overarching goal here is to allow us to ship more software faster and with more confidence. In particular:

• Ensure that release branches are always in a shippable state.
• Ensure that release branches always build and test with their minimal dependency versions.
• Ensure that feature and bugfix branches build and test.
• Ensure that feature and bugfix branches can merge into their intended upstream release branch.
• Create platform / system binaries for supported OSs for easy testing of new features or for verifying bugs.
• Ensure that all branches work with all supported combinations of upstream dependency versions (see dependency Matrix below)
• All of the above should be tested on all supported operating systems and versions. See Operating System Matrix below
• Give developers and testers in-depth information on build failures.

Non Goals

• Ponies.

Dependency Matrix

CouchDB 1.4.x:

• Erlang R13B4, Erlang R15B03-1

• Spidermonkey 1.7.0, 1.8.5
• ICU (TBD)

CouchDB 1.3.x:

• Erlang R13B4, Erlang R15B03-1

• Spidermonkey 1.7.0, 1.8.5
• ICU (TBD)

CouchDB 1.2.x:

• Erlang R12B5, Erlang R15B03-1

• Spidermonkey 1.7.0, 1.8.5
• ICU (TBD)

CouchDB 1.1.x:

• Erlang R13B2, Erlang R14B04

• Spidermonkey 1.7.0, 1.8.5
• ICU (TBD)

(not strictly supported, but what the heck) CouchDB 1.0.x:

• Erlang R12B5, Erlang R14B04

• Spidermonkey 1.7.0
• ICU (TBD)

Operating System Matrix

Linux:

• Ubuntu: 10.04 (Desktop 32/64 / Server 64)
• Ubuntu: 12.04 (Desktop 32/64 / Server 64)
• Ubuntu; 12.10 (Desktop 32/64 / Server 64)
• Centos 6.3 (32/64)
• Centos 5.2 (32/64)
• Debian
• Gentoo
• FreeBSD 9.0, 8.3
• Mac OS X: 10.8.2, 10.7.5, 10.6.5 (32/64)
• Windows: 7, 8 (32/64).
• OmniOS/Solaris

Notes

Since we can’t build against one of multiple installed JS engines, we need a build slave for each tested JS engine.

For multiple Erlang versions, we use “Matrix” builds.

Roadmap

Set up all necessary build slaves. From the above, these are missing today:

• Debian
• Gentoo
• Mac OS X 10.6.5
• Windows 7
• OmniOS/Solaris
• FreebSD 8.3
• others.

Add integration jobs for feature and bug fix branches.

Open up access to the public.

Expand test configuration range (we currently only test one spidermonkey per host os)

Migrate to ci.apache.org.

How can I help?

All this is to get a CouchDB QA team going. So far the developers and users do QA, but no dedicated team takes on the task of caring about QA, creating testing infrastructure, etc. This could be you!

Since this is a non-existent team, you can do anything end everything. You can help shape the

The Build Job

Each one of these varies by build slave (i.e. Operating System and configuration), but what they all do is this:

• Check out the latest version of CouchDB for its branch. Each Build Job only builds a single branch.
• Clean up the current workspace with git clean -fdx
• [Set environment variables]
• ./bootstrap
• ./configure [$ARGS]
• make distcheck

At this point, we have an apache-couchdb-$VERSION_INFO.tar.gz that is packaged up like a release.tar.gz, its sources have been build successfully and all JS and Erlang tests have been run (with 1.2.0 and earlier, only the Erlang tests were run). THESE ARE NOT OFFICIAL APACHE RELEASES.

• ./configure --prefix=/tmp/$ERLANG_VERSION
• make install
• tar czf /tmp/$ERLANG_VERSION

Now we created a platform-dependent binary version that unpacks into /tmp that users and devs can run on their systems, given they have the same dependencies in the same places, e.g. to test out new features or verify bug fixes. THESE ARE NOT OFFICIAL APACHE RELEASES.

• Normalize .tar.gz filenames
• Upload .tar.gz files to S3.

This is so we can distribute the files easily. We might want to move that to ASF infrastructure as well.

• Notify #couchdb-ci on irc.freenode.net

This is so devs get real time info on builds.

FaQ

What is CI / Continuous Integration?

CI or Continuous Integration is the idea that with ongoing development of a piece of software all sorts of automated tests are run alongside. For CouchDB that means that we set up software that tries to build and test release branches and feature branches while they are being developed on in multiple configurations (e.g. various Erlang versions) on different operating systems.

For more info see https://en.wikipedia.org/wiki/Continuous_integration

Why isn’t this using http://ci.apache.org/?

The goal eventually is running this on official infrastructure. To get there, we needed team members to figure out what needs to be done. To be able to learn all the details, we started with a private installation of Jenkins. We hope to migrate this to ci.apache.org in due time.

Why isn’t this using Travis CI?

We’d like multi-OS support, Travis doesn’t provide that (yet).