VmgumpConfig - Gump Wiki

Debian

Upgrade to sid:

#deb file:///cdrom/ sarge main

deb http://linux.csua.berkeley.edu/debian/ sid main
deb-src http://linux.csua.berkeley.edu/debian/ sid main
#deb http://linux.csua.berkeley.edu/debian/ testing main
#deb-src http://linux.csua.berkeley.edu/debian/ testing main

deb http://security.debian.org/ testing/updates main

Install the following via apt-get install:

required: apache2 cvs libxp-dev libxt6 libxtst6 python python2.3 subversion xvfb mysql-server python2.3-dev python2.3-mysqldb python2.4-dev unzip libxml2 libxml2-utils xsltproc libtool curl ntp ntpdate java-6-sun
convenience: lynx sudo vim bzip2 mutt nano mysql-client rsync
for building projects: autoconf automake

Create gump user and group and stuff

useradd -d /home/gump -s /bin/bash gump
addgroup gump
adduser gump gump
adduser gump staff
mkdir /home/gump
chown -Rf gump:gump /home/gump
mkdir /x1/gump
ln -s /x1/gump /srv/gump
chown gump:gump /x1/gump
echo 'general@gump.apache.org' > ~gump/.forward
echo 'pmc@gump.apache.org' > ~root/.forward

Set up mysql

Secure the root account (http://dev.mysql.com/doc/mysql/en/default-privileges.html):

shell> mysql -u root
mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('newpwd');
mysql> SET PASSWORD FOR 'root'@'vmgump' = PASSWORD('newpwd');

Create a gump database and user

mysql> create database gump_public;
mysql> GRANT ALL PRIVILEGES ON gump_public.* to 'gump'@'localhost' identified by 'passwd';
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.01 sec)

set up tables

cd /srv/gump/public/gump/mysql
mysql -u gump -p gump_public < gump.sql 
# enter password here...

Other prereqs

We manage most of these in /opt:

mkdir /x1/opt
mkdir /x1/opt/__versions__
rm -r /opt
ln -s /x1/opt /opt

Apache Maven:
- Download Maven to /tmp (select .tar.gz format)
- cd /opt/__versions__
- tar -xzf /tmp/maven-xxx.tar.gz
- create a symbolic link /opt/maven pointing to the installed version
- Note: MAVEN_HOME will be exported and added to the PATH in step 5
rdflib
- Download Rdflib to /tmp (select .tgz format)
- tar -zxf /tmp/rdflib-xxx.tgz
- cd rdflib-xxx
- (as root) python2.3 setup.py install
- (as root) python2.4 setup.py install
ntpdate
- point /etc/init.d/ntpdate to ntp.unitedlayer.com
- run sudo ntpdate -v ntp.unitedlayer.com
mvn Repository Proxy
- check out http://svn.apache.org/repos/asf/gump/mvnrepo/tags/0.2/
- run ant zip - Ant 1.7.0 or better is required
- cd /opt/__versions__
- unzip /tmp/repoproxy.zip
- mv lib mvnrepoproxy-0.2
- cd ..
- ln -s __versions__/mvnrepoproxy-0.2 mvnrepoproxy

Optional prereqs

Mono:

install Mono

cd /tmp
wget ftp://www.go-mono.com/archive/1.2.3.1/linux-installer/1/mono-1.2.3.1_1-installer.bin
md5sum mono-1.2.3.1_1-installer.bin # check value!
chmod +x mono-1.2.3.1_1-installer.bin
sudo -i
./mono-1.2.3.1_1-installer.bin --mode text # install to /opt/__versions__
ln -s /opt/__versions__/mono-1.2.3.1 /opt/mono

Note: /opt/mono will be added to the PATH in step 5, the same is true for /opt/mono/lib/pkgconfig and PKG_CONFIG_PATH

TODO: figure out which packages of the below on brutus is needed

lrwxrwxrwx   1 root root    35 Oct  6  2004 ant -> /opt/__versions__/apache-ant-1.6.2/
lrwxrwxrwx   1 root root    31 Jan  7 11:57 cocoon -> /opt/__versions__/cocoon-trunk/
lrwxrwxrwx   1 root root    32 Jan  7 11:57 forrest -> /opt/__versions__/forrest-trunk/
lrwxrwxrwx   1 root root    29 Jan 24 23:47 jdk1.3 -> /opt/__versions__/jdk1.3.1_14
lrwxrwxrwx   1 root root    27 Oct  6  2004 jdk1.5 -> /opt/__versions__/jdk1.5.0/
lrwxrwxrwx   1 root root    31 Oct  6  2004 jetty -> /opt/__versions__/jetty-4.2.21/
lrwxrwxrwx   1 root root    23 Oct 12  2004 kaffe -> /opt/__versions__/kaffe
lrwxrwxrwx   1 root root    29 Oct  6  2004 loader -> /opt/__versions__/loader-1.0/
lrwxrwxrwx   1 root root    49 Oct  6  2004 mysql -> /opt/__versions__/mysql-connector-java-3.0.15-ga/
lrwxrwxrwx   1 root root    31 Oct  6  2004 rdflib -> /opt/__versions__/rdflib-2.0.3/
lrwxrwxrwx   1 root root    30 Oct  6  2004 servlet-api -> /opt/__versions__/servlet-2.4/
lrwxrwxrwx   1 root root    32 Oct  9  2004 webware -> /opt/__versions__/Webware-0.8.1/

Gump Setup

Set up /srv/gump/[flavour]/..., the first flavour being public:

set CVSROOT per Gump Getting Started
mkdir -p /srv/gump/public
cd /srv/gump/public
svn checkout https://svn.apache.org/repos/asf/gump/live/ gump (note we're not using the "trunk" branch!)

create/edit ./metadata/vmgump.xml (to match `hostname`.xml and supplying the database password)

<workspace name="vmgump-public"
        basedir="/srv/gump/public/workspace"
        jardir="/srv/gump/public/jars"
        pkgdir="/srv/gump/packages"
        logdir="/srv/gump/public/results"
        logurl="http://vmgump.apache.org/gump/public"
        version="0.4">

        <property name="build.sysclasspath" value="only"/>
        <sysproperty name="java.awt.headless" value="true"/>

        <profile href="profile/gump.xml"/>

        <database database="gump_public" user="gump" passwd="password" />

        <!-- additional background threads, over main thread -->
        <threads updaters="0" builders="0" />
</workspace>

sync over packages from brutus.apache.org:/srv/gump/packages [shared, not under 'flavour'].
sync over brutus.apache.org:~gump/.cvspass (saves typing in a whole bunch of 'public' cvs passwords)
TODO: sync over brutus.apache.org:~gump/.maven
install LICENSE and rt.jar of a J2SDK 1.3 (I used a local copy that I have downloaded ages ago) into /srv/gump/packages/java-runtime-1.3

create/edit /srv/gump/public/gump/cron/local-env-vmgump.sh:

export CC="ccache gcc"
export CXX="ccache g++"

export JAVA_HOME=/usr/lib/jvm/java-6-sun
export CLASSPATH=$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

export MAVEN_HOME=/opt/maven
export PATH=$PATH:$MAVEN_HOME/bin

export M2_HOME=/opt/maven2
export PATH=/opt/maven2/bin:$PATH

export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/opt/mono/lib/pkgconfig
export PATH=$PATH:/opt/mono/bin

export MVN_PROXY_HOME=/opt/mvnrepoproxy

create/edit /home/gump/.bash_profile:

umask 002
. /srv/gump/public/gump/cron/local-env-vmgump.sh

set up cron for user "gump":

#NOT "OFFICIAL" Public - The official Gump run, the one that sends email (uses JDK 1.4 and the 'live' branch)
#0 0 * * *       cd /srv/gump/public/gump/cron; /bin/bash gump.sh all --official

# Public - these are subruns of public that don't send email but update the web site
#0 18 * * *    cd /srv/gump/public/gump/cron; /bin/bash gump.sh all
0 0,6,12,18 * * *    cd /srv/gump/public/gump/cron; /bin/bash gump.sh all

#DISABLED JDK 1.5 - This Gump runs on JDK 1.5
#0 6 * * *       cd /srv/gump/jdk15/gump/cron; /bin/bash gump.sh all

#DISABLED Test - This Gump runs on JDK 1.4 but uses the latest-and-greatest gump code out of 'trunk'
#0 12 * * *      cd /srv/gump/test/gump/cron; /bin/bash gump.sh all

#DISABLED Gump3 - This is the new Gump...
#0 12 * * *      cd /srv/gump/gump3/Gump3/; /bin/bash gump run --databasename=gump3

#DISABLED Kaffe - This Gump runs on top of Kaffe and uses the 'live' branch
#0 3,9,15,21 * * *      cd /srv/gump/kaffe/gump/cron; /bin/bash gump.sh all; /usr/bin/killall -9w kaffe-bin; rm -f /home/gump/workspaces/kaffe/workspace/ant/src/main/org/apache/tools/ThisIsALink

# Clean up older artifacts
0 0 * * * /usr/bin/find /srv/gump/*/jars -type f -ctime +6 | /usr/bin/xargs -r /bin/rm

#TODO Update the local environments and workspace information
#55 * * * * /home/gump/update-envs.sh

#TODO Checkout (from CVS) and update blog
#1,16,31,46 * * * * /home/gump/update-blog.sh

#TODO Update website from svn
#2,17,32,47 * * * * /home/gump/update-site.sh

#Clean up after POI...
0 0 * * * /bin/rm -f /tmp/*.xls

configure /etc/apache2/sites-available/vmgump.apache.org somewhat like this: {{{NameVirtualHost *

ServerAdmin pmc@gump.apache.org ServerName vmgump.apache.org
DocumentRoot /var/www/vmgump.apache.org
<Directory />
- Options FollowSymLinks AllowOverride None
</Directory> <Directory /var/www/vmgump.apache.org>
- Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all
</Directory>
ErrorLog /var/log/apache2/vmgump.apache.org.error.log # Possible values include: debug, info, notice, warn, error, crit, # alert, emerg.
LogLevel warn
CustomLog /var/log/apache2/vmgump.apache.org.access.log combined ServerSignature On Alias /gump/public/ /srv/gump/public/results/ Alias /gump/public-jars/ /srv/gump/public/jars/
<Directory /srv/gump/>
- HeaderName /disclaimer.html Options Indexes MultiViews FollowSymLinks AllowOverride None Order allow,deny Allow from all
</Directory>
<Location /gump/public/workspace_defn.html>
- Order deny,allow Deny from all
</Location>
ProxyRequests Off <Proxy *>
- Order deny,allow Allow from all
</Proxy> ProxyPass /gump3/ http://localhost:8080/ ProxyPassReverse /gump3/ http://localhost:8080/

</VirtualHost>}}}

mkdir /var/www/vmgump.apache.org && chown gump:gump /var/www/gump.apache.org
a2ensite vmgump.apache.org && a2enmod proxy && a2dissite default
/etc/init.d/apache2 reload
PLEASE DON'T edit gumprun.py to disable all actors except the xdoc one, since one of the others seems to be
- causing weird issues.

gump3 setup

useradd -d /home/gump3 -s /bin/bash gump3
addgroup gump3
adduser gump3 gump3
adduser gump3 staff
mkdir /home/gump3
chown -Rf gump3:gump3 /home/gump3
passwd gump3
$EDITOR ~gump3/.passwd # save password
chmod 600 ~gump3/.passwd
chown gump3:gump3 ~gump3/.passwd
su - gump3 # do the below as "gump3" user in /home/gump3...
svn co https://svn.apache.org/repos/asf/gump/branches/Gump3
echo 'general@gump.apache.org' > ~gump3/.forward

disable auth caching for svn, editing ~gump3/.subversion/config so that

### Section for authentication and authorization customizations.
[auth]
### Set store-passwords to 'no' to avoid storing passwords in the
### auth/ area of your config directory.  It defaults to 'yes'.
### Note that this option only prevents saving of *new* passwords;
### it doesn't invalidate existing passwords.  (To do that, remove
### the cache files by hand as described in the Subversion book.)
store-passwords = no
### Set store-auth-creds to 'no' to avoid storing any subversion
### credentials in the auth/ area of your config directory.
### It defaults to 'yes'.  Note that this option only prevents
### saving of *new* credentials;  it doesn't invalidate existing
### caches.  (To do that, remove the cache files by hand.)
store-auth-creds = no

[VmgumpConfig/Gump3BashConfig]
install gump3 prereqs

exit # fromp gump3, back into su mode
apt-get install python2.4 python2.4-mysqldb python-pmock
# no python2.4-pmock...
cp /usr/lib/python2.3/site-packages/pmock.py /usr/lib/python2.4/site-packages
python2.4 -OO -c "from pmock import *; import pmock; import sys; sys.exit()"

test it

su - gump3
GUMP_TEST_NO_MYSQL=yes gump test

Done!

Get to business...

su into gump, and run some tests: {{{cd /srv/gump/public/gump/python

export PYTHONPATH=pwd python gump/check.py -w ../brutus.xml all --debug }}}

make sure we trust svn.apache.org
cd /srv/gump/public/gump; svn up (hit 'p' to store cert)
now kick of an "actual run" manually:

cd /srv/gump/public/gump/cron; /bin/bash gump.sh all

Gump Farm Layout Details

Current structure is:

File System:

    /srv/gump -- root
    /srv/gump/packages -- shared packages

    /srv/gump/{flavour} -- e.g. public or jdk15 or test or ...
    /srv/gump/{flavour}/gump -- Installation of Gump
    /srv/gump/{flavour}/workspace -- working area 
    /srv/gump/{flavour}/results -- WWW site
    /srv/gump/{flavour}/jars -- Artifact Repository

    /opt -- prereqs not installed using apt
    /var/www/vmgump.apache.org -- website

HTTP:

    http://vmgump.apache.org/gump/{flavour} ->
        /srv/gump/{flavour}/results -- WWW site

    http://vmgump.apache.org/gump/{flavour}-jars/ ->
        /srv/gump/{flavour}/jars -- Artifact Repository

Currently deployed flavours:

    public -- against SVN /live, does e-mail notification, automatic runs.

    test -- against SVN /trunk, manual runs. Often cleaned out to save disk space.