Eyebrowse

Eyebrowse is the software that implements the Apache Software Foundation's mailing list archives. This page documents procedures for those members of the infrastructure team who help to maintain the archives.

This document is in a very early stage, and is intended only for internal reference. If you are having problems with an ASF mailing list archive, please inform the project members, who will contact the infrastructure team. If you are looking for general information regarding eyebrowse, please visit the eyebrowse web site and subscribe to their lists.

Task Lists -- specific admin requests and related instructions

Move James Mailing Lists
DONE Add Incubator Lists
DONE Add Apache Lists
DONE Add Maven Lists
DONE Add XML Lists
DONE Add HTTPd Lists
DONE Add ["ApacheCON" lists]
Move Cocoon Mailing Lists
DONE Add new Jakarta Lists
Current Service Requests
Make Service Request
Service Requests - (need to be reposted to Bugzilla)

Q&A:

<B>STOP!</B> Read the List Setup instructions first.
Why isn't some list archiving?
- There are a number of possible reasons:
  - A failure to delete a lockfile is usually at the cause of stopped indexing. Usually deleting that lockfile and then kicking off a re-index somehow fixes the problem.
  - Sometimes the indices stored in MySQL are incorrect, or the active flag of a list's mbox files isn't set for any of its data files.
  - The .qmail file for that list on nagoya may be missing or wrong.
What is Eyebrowse written in?
- Java. There is a Perl front-end, but the COMMAND_LINE_BRANCH in the tigris.org CVS has a Java front-end that would use Jakarta Commons Commandline. It needs to be integrated into the main Eyebrowse trunk.
How do I repair an index?
- There are actually two indices.
  - RDBMS (MySQL, in this case) index of the messages stored within the mbox-format file(s) for a given list. This meta data consists mostly of header and offset information.
  - Jakarta Lucene index of actual message content for optimized searching of message bodies.
- Both of these indices should be updated periodically (on nagoya via a cron job running as unix user "dlr"). Updates of both are affected by the presence of lock files (which are sometimes stale).
How do we repair an index when, for example, it has indexed Feb, parts of March, nothing from April, and then indexed May?
- You have to re-index each mbox file starting from the last known good state. This is involves manual modifications to the SQL database, and is likely to seriously piss off anyone who has linked to specific message, as both their message sequence (eb_Message.msgNo) and interal primary key (eb_Message.msgId) will be changed.
- There is a policy issue as well as a technical one. Do we favor archival completeness over persistence of bookmarks?
Where do I get the initial mbox files to start populating?
- For each top level project, /www/tlp.apache.org/mail/list has the mbox files in compressed format (except the current month).

TODO Here are two strategic initiatives which simplify maintenance of archives.apache.org:

Testing and deployment of the simplistic scheduler recently added to CVS HEAD. With this, we could eliminate the use of lock files.
Support for list renames. We got a lot of these a while back, and to preserve msgId's I moved data around on the file system and in the database by hand (a very error-prone process).
- See the James task list for a scheme in SQL that might help.