Part of ASF Build Services, the ASF runs a Jenkins continuous-integration server at https://builds.apache.org, with email@example.com mailing-list (see archive) and BUILDS JIRA project for issue tracking.
The Jenkins setup consists of a master server running on an Ubuntu machine (crius) and several build slave servers, including six Ubuntu slaves (ubuntu1 to ubuntu6= minerva, vesta, hemera, janus, juno, quirinus, being transitioned on mid-july 2014 to ubuntu-1 to ubuntu-6= ubuntu-1, ubuntu-2, ubuntu-3, ubuntu-4, ubuntu-5, ubuntu-6), FreeBSD, Solaris 10 x86 and Windows slaves. There are also some slaves dedicated to specific projects (Hadoop, Lucene, Tapestry)
Any ASF project can run build and test tasks for their projects from this server. PMC chairs can grant access to Jenkins to any committer (see below for instructions) to get an administrative account on Jenkins, in order to create the Jenkins jobs representing your projects.
Those of you in the Incubator, there are many PMC Chairs belonging to the Incubator PMC, including but not limited to the Incubator PMC Chair. So if a mentor can't do it, ask around. If no answer within a reasonable time frame, then by all means open an INFRA ticket, mentioning your earlier attempts to get a PMC Chair to do it.
Your users will then be able to view results of those builds on the web.
FAQ For PMCs
- How do I get an account
- How do I report a bug / contact the maintainers?
- Getting started
- What should I do with the build and test tools that my project needs?
- Are there recommended or mandatory Jenkins settings for ASF projects?
- How do I allow Jenkins to mail to my project's "dev" list?
- How do I setup pre-commit Builds (similar to Hadoop's)
- Where do I find more information on Jenkins?
- Who maintains the Jenkins servers?
FAQ For Administrators
- How do I restart Jenkins?
- How do I restart a Jenkins Unix Slave?
- How do I restart a Jenkins Windows Slave?
- How can I see what tasks are open in JIRA?
- How do I install a new version of Jenkins?
- How do I install a new Jenkins plugin?
- Setting up a Ubuntu slave
- Where do I find more information on administering our Jenkins installation?
- What do Administrators do?
FAQ For PMCs
How do I get an account
Jenkins uses the Apache LDAP servers for authentication. To give a committer access to Jenkins, the committer must be made a member of the hudson-jobadmin group. This is done using the modify_appgroups.pl script on minotaur.apache.org. Ask your PMC chair to execute this command. For example:
modify_appgroups.pl hudson-jobadmin --add=<Apache username>
To list current members of that group use
If you need shell access to any of the slaves, you need to open an issue as described below. This should normally only be needed if your builds require custom tooling that you need to install.
How do I report a bug / contact the maintainers?
When your account request has been processed, you will receive an e-mail containing your login details.
What should I do with the build and test tools that my project needs?
Various version of the following tools are supplied in /home/hudson/tools:
- forrest ($HOME/tools/forrest/apache-forrest-0.8 on hudson-solaris, minerva and vesta)
- erlang ($HOME/tools/erlang/latest on hudson-solaris, minerva and vesta)
- litmus ($HOME/tools/litmus/latest on hudson-solaris, minerva and vesta)
Are there recommended or mandatory Jenkins settings for ASF projects?
When configuring your job, these settings are a good idea:
Use one of these locks for long running builds (> 1 hour):
- Long-running jobs on minerva.apache.org
- Long-running jobs on vesta.apache.org
- Long-running jobs on Ubuntu (ie. both vesta and minerva)
Activate the Build Timeout Plugin and set a realistic timeout value.
Note that this is forced by the Admins by running a script on an irregularly basis which set it to the double of the estimated duration with a minimum of 10 minutes.
Repository browser (Recommended)
Use ViewSVN, URL http://svn.apache.org/viewvc/
- This allows working links to the change notifications in SVN.
Tie this project to a node (Recommended)
- For Ubuntu 12.04, tie your job to one of:
For legacy Ubuntu 10.04 use ubuntu2
- For Solaris 10, tie your job to:
- For Windows, tie your job to:
- For OS X, tie your job to:
If your build is not system-dependent (most simple Java builds, etc.) you can leave your build untied to maximize the number of executors available for the build. Otherwise please use the generic system labels Ubuntu, Solaris and Windows unless your build needs special tooling that has only been installed on one of the build servers.
How do I allow Jenkins to mail to my project's "dev" list?
Send an email to FOO-allow-subscribe-jenkins=builds.apache.org@TLP.apache.org. Replace "FOO" with the name of your dev list, e.g. "dev", and "TLP.apache.org" with the name of your TLP. Note that you should send this request from a moderator email address.
On the job's Config page, tick "E-mail Notification", and type FOO@TLP.apache.org in the Recipients field. Tick 'Send e-mail for every unstable build'.
Mails will now be sent every time a build starts failing, or when a failing series of builds starts working again.
How do I setup pre-commit Builds (similar to Hadoop's)
See PreCommitBuilds for more info.
Where do I find more information on Jenkins?
Who maintains the Jenkins servers?
Currently, the administrators are:
Gavin McDonald <gmcdonald /at/ apache.org>
Jake Farrell <jfarrell /at/ apache.org>
Olivier Lamy <olamy /at/ apache.org>
Andrew Bayer <abayer /at/ apache.org>
Giridharan Kesavan <gkesavan /at/ apache.org>: Hadoop slaves
FAQ For Administrators
How do I restart Jenkins?
Log into builds.apache.org via SSH. The following command can be used to clean up all pending Jenkins processes and start from a fresh state:
sudo /etc/init.d/hudson restart
Note that given the number of historical builds that Jenkins must load into memory, it may take up to 10 minutes for the Jenkins webapp to become functional after a restart.
Restarting the Jenkins master will automatically restart every Unix slaves, but not Windows ones: restarting the Windows slaves has to be done manually.
How do I restart a Jenkins Unix Slave?
Visit https://builds.apache.org/computer/ (logging in if necessary), click the slave's name, and hit Disconnect. Give a reason and confirm you want to disconnect the slave. Then ssh to the slave box, kill all processes owned by the "hudson" or "jenkins" uid:
sudo pkill -15 -u hudson ; sleep 5 sudo pkill -9 -u hudson
and, back on the web UI, hit Connect to restart the slave.
How do I restart a Jenkins Windows Slave?
Visit https://builds.apache.org/computer/ (logging in if necessary), click the slave's name, and hit Disconnect. Give a reason and confirm you want to disconnect the slave. Then... TBD
How can I see what tasks are open in JIRA?
This JIRA search will display the current list of tasks for the 'Jenkins' component of the Infrastructure project (that's us). You may want to subscribe to that Filter; hit 'Save' to save it as a filter, then 'Manage' and 'Subscribe'.
How do I install a new version of Jenkins?
SSH to builds.apache.org and run /home/hudson/upgrade-to-latest.sh to update to the latest Jenkins version. The script takes care of downloading Jenkins and performing the necessary installation procedures. Then send a notice on build@ to let people know of the upgrade.
How do I install a new Jenkins plugin?
Jenkins plugins extend the functionality of Jenkins but also might risk some stability on the build cluster, so be careful. Plugins are used to collect and present data for Findbugs and Checkstyle, to name a few. These plugins are packaged as .hpi files and are available from http://jenkins-ci-org/. The plugin file goes in ~hudson/hudson/plugins/. To install or upgrade a plugin, follow these steps:
- Download the new plugin file
- Move the old plugin file to a backup location incase you need to rollback
Copy in the new plugin file into the plugins directory
- Shutdown and startup Jenkins
Note that the plugin directory must be owned by the hudsonadmin user, not the hudson user. The start script should take care of this for you.
Setting up a Ubuntu slave
- Create user
- adduser jenkins
- Set up SSH access by softlinking ssh pub key and add jenkins to sshusers
- rsync over tools from some other Ubuntu slave
- Install packages
- sudo apt-get install erlang-dev erlang-eunit g++-multilib gcc-multilib ia32-libs rubygems libmono-system-web2.0-cil libboost-test1.40-dev libevent-dev libbit-vector-perl php5-cli erlang-base mono-gmcs libmono-dev librspec-ruby libboost-dev flex bison pkg-config ruby-dev python-dev php5-dev libglib2.0-dev cabal-install ghc6 rake python-twisted libghc6-binary-dev libghc6-network-dev libghc6-http-dev
- Softlink /home/hudson to /home/jenkins for tools locations to work
- Add ulimit -n 40000 to /etc/init.d/ssh and restart ssh. If sshd is running with the -D flag, you will first need to stop it using sudo service ssh stop.
- Copy over ~/.m2/settings.xml from some other Ubuntu slave
Where do I find more information on administering our Jenkins installation?
SVN: https://svn.apache.org/repos/infra/infrastructure/hudson (obsolete)
Jenkins state monitoring: hourly builds.a.o check (obsolete?)
What do Administrators do?
- Jenkins upgrades and restarts
- Add plugins
- Kill stuck builds
- Set policy for Jenkins usage
- Watch changes to this wiki page
The Jenkins zone is backed up nightly to /zonestorage/backup , which in turn is backed up to another ASF machine periodically. /export/home/hudson, user home directories, and /opt are backed up. Note however that historical build data and artifacts are not backed up.