Differences between revisions 164 and 165
Revision 164 as of 2013-12-09 23:51:35
Size: 18850
Editor: SteveRowe
Comment: fix edit conflict
Revision 165 as of 2013-12-09 23:54:21
Size: 18880
Editor: SteveRowe
Comment: rearrange Version.LUCENE_XY upgrade text
Deletions are marked like this. Additions are marked like this.
Line 41: Line 41:
    * Find/replace LUCENE_XY -> LUCENE_X(Y+1) across all of Lucene and Solr, but do NOT change usages under the lucene/analysis/ that allow version-specific behavior. These should be findable with the following near instances of these constants: {{{:Post-Release-Update-Version.LUCENE_XY:}}}     * Find/replace LUCENE_XY -> LUCENE_X(Y+1) across all of Lucene and Solr. These should be findable with the following commented-out token near instances of these constants: {{{:Post-Release-Update-Version.LUCENE_XY:}}}.
        * Do NOT change usages under the lucene/analysis/ that allow version-specific behavior.

This page is to help a Lucene/Solr committer create a new release (you need committer rights for some of the steps to create an official release). It does not reflect official release policy - many of the items may be optional, or may be modified as necessary.

See the ASF Releases Policy page.

Release Process

Prior to every major or minor release (i.e. all except bugfix-only point releases) a feature freeze phase takes place for about 1-2 weeks. At the beginning of the feature freeze the trunk is branched and no commits other than serious bug fixes, documentation or build updates are permitted. This period of time should be used for extensive testing, documentation improvements and for cleaning up old JIRA issues.


  1. (Optional) Edit your gnupg/gpg.conf file and add a default-key value with your Code Signing Key (as a HEX value) otherwise you may need to specify -Dgpg.key=XXXXXX later for signing. Make sure it is your 4K-bit key and not a 1K-bit key (esp. for ASF old timers). See http://www.apache.org/dev/release-signing.html for more information.

  2. You may need to setup people.apache.org as a known host for ssh if you haven't already. This happens when you log in manually and follow the prompts.


  1. On dev@lucene.apache.org, decide on

    1. which JIRA issues shall be committed before a release is made; set the appropriate "Fix Version" in JIRA for these issues
    2. the date for branching the trunk and the start of the feature freeze phase
    3. the length of the feature freeze phase
    4. a tentative release date
  2. Put up (and/or fix up existing) release notes, at http://wiki.apache.org/lucene-java/ReleaseNoteXY and http://wiki.apache.org/solr/ReleaseNoteXY and ask on dev@ for input

Branching & Feature Freeze

  1. Run ant precommit to run a bunch of sanity & quality checks, then fix any problems that are found.

  2. For the first release in a minor release series - i.e. X.Y.0 - create a minor release branch off the current major version branch, e.g. for minor release 4.2:
    svn copy https://svn.apache.org/repos/asf/lucene/dev/branches/branch_4x \
    https://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_4_2 -m "Starting Lucene/Solr 4.2 branch."
  3. After creating a new release branch, update the version in the base release branch (e.g. branches/branch_4x):

    • In solr/CHANGES.txt and lucene/CHANGES.txt, add X.Y+1 release sections

    • In top-level build.xml

      • Update property version to X.Y+1-SNAPSHOT

      • Update property fakeReleaseVersion to X.Y+1

    • In lucene/common-build.xml:

      • Update property dev.version.base to X.Y+1-SNAPSHOT

      • Update property tests.luceneMatchVersion to X.Y+1-SNAPSHOT

    • In lucene/core/src/java/org/apache/lucene/util/Constants.java, update LUCENE_MAIN_VERSION to X.Y+1

    • In lucene/core/src/java/org/apache/lucene/util/Version.java, add new enum constant LUCENE_X(Y+1)

    • Find/replace LUCENE_XY -> LUCENE_X(Y+1) across all of Lucene and Solr. These should be findable with the following commented-out token near instances of these constants: :Post-Release-Update-Version.LUCENE_XY:.

      • Do NOT change usages under the lucene/analysis/ that allow version-specific behavior.
    • In dev-tools/maven/pom.xml.template:

      • Update base.specification.version to X.Y+1.0

      • Update tests.luceneMatchVersion to X.Y+1.0

    • In dev-tools/idea/.idea/workspace.xml:

      • Update each mention of lucene.version to X.Y+1-SNAPSHOT

      • Update each mention of tests.luceneMatchVersion to X.Y+1-SNAPSHOT

  4. If the X.Y default index format is different from the X.Y-1 default index format, add a X.Y-1 index to lucene/dev/branches/branch_4x/lucene/core/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java - see the comments there for instructions.

  5. If it is a new major version, also update IndexFormatTooOldException to have the correct minimum version string.

  6. Send a note to dev@ to inform the committers that the branch has been created and the feature freeze phase has started. Include Do's and Don'ts for the feature freeze phase:
    • No new features may be committed to the branch.
    • Documentation patches, build patches and serious bug fixes may be committed to the branch. However, you should submit all patches you want to commit to Jira first to give others the chance to review and possibly vote against the patch. Keep in mind that it is our main intention to keep the branch as stable as possible.

    • All patches that are intended for the branch should first be committed to trunk, merged into the minor release branch, and then into the current release branch.
    • Normal trunk and minor release branch development may continue as usual. However, if you plan to commit a big change to the trunk while the branch feature freeze is in effect, think twice: can't the addition wait a couple more days? Merges of bug fixes into the branch may become more difficult.
    • Only Jira issues with Fix version "X.Y" and priority "Blocker" will delay a release candidate build.

Release Workflow

  1. Go to the JIRA "Manage Versions" Administration pages (https://issues.apache.org/jira/plugins/servlet/project-config/LUCENE/versions and https://issues.apache.org/jira/plugins/servlet/project-config/SOLR/versions) and add a new (unreleased) version for the next release on the trunk (for a major release) or branch (for a minor release).

  2. Check out the branch with: svn co https://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_X_Y

  3. Build the code and javadocs, and run the unit tests: ant clean javadocs test. Make sure that you are actually using the minimum compiler version supported for the release. For example, 4.x releases are on Java6 so make sure that you use Java6 for the release workflow.
  4. Examine the results. Did it build without errors? Were there Javadoc warnings? Did the tests succeed? Does the demo application work correctly? Does Test2BTerms pass?
  5. Remove lucene/benchmark/{work,temp}/ if present

  6. Build as defined below in the "Building the Release Artifacts" section.
  7. If during the feature freeze phase bug fixes are committed to the X.Y branch then build another release candidate and announce on dev@ and java-user@ that everyone should use the new RC for testing.
  8. Call a release vote on dev@ and cc general@lucene.apache.org .

Building the Release artifacts

If after the last day of the feature freeze phase no blocking issues are in JIRA with "Fix Version" X.Y then it's time to build the release artifacts

  1. Option 1: use dev-tools/scripts/buildAndPushRelease.py to build a release candidate, and optionally push the results to people.apache.org, and optionally run dev-tools/scripts/smokeTestRelease.py on it. Read the instructions in comments at the top of buildAndPushRelease.py.

  2. Option 2: manually build a release candidate:
    1. Verify that svnversion reports a single revision with no modified changes

    2. Remove lucene/benchmark/{work,temp}/ if present

    3. Build the release artifacts: ant -Dversion=X.Y.Z [-Dgpg.exe=/path/to/gpg -Dgpg.key=YourKeyID -Dgpg.passphrase=YourPassPhrase] prepare-release (Items in [] may be optional)

    4. Sanity check the overall size of each artifact. EG, compare to the last release.
    5. Make sure that for each release file an md5 checksum file exists.
    6. ant -Dversion=X.Y.Z copy-to-stage to move the artifacts up to the staging area (see the build target for all the options for securly copying the artifacts up)

    Example for 3.1:
    •      PROMPT$ ANT_OPTS="-Xmx256m -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=128M" ant -Dversion=3.1.0 -Dgpg.key=FE045966 prepare-release
           # Inspect the artifacts
           PROMPT$ ant -Dversion=3.1.0 -Drc=rc3 -Dscp.user=gsingers copy-to-stage

Testing the Release artifacts

  1. There is a script in SVN to do automated checks on a release candidate, e.g.

python3.2 -u dev-tools/scripts/smokeTestRelease.py http://people.apache.org/~whoever/4.4.0rc8-rev5551212 5551212 4.4.0 tmp


Once three PMC members have voted for a release, it may be published.

  1. Tag the release from the same revision from which the passing release candidate's was built:
    svn copy -r1492800 https://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_X_Y \
    https://svn.apache.org/repos/asf/lucene/dev/tags/lucene_solr_X_Y_Z -m "Lucene/Solr release X.Y.Z"
  2. If this is a point release, copy the CHANGES.txt entry for this point release back to the trunk's CHANGES.txt. Remove any duplicate entries from trunk's CHANGES.txt (ie, each issue should appear only once, in the earliest point-release that contains the fix, on the assumption that all future releases also contain the fix).
  3. Add the new releases to svnpubsub:
    1. Check out the Lucene PMC svnpubsub release area:
             svn co https://dist.apache.org/repos/dist/release/lucene/ dist/release/lucene # this might take some time: ~ 3 beers
    2. Make a directory for the new release under lucene/java/ and lucene/solr/.

    3. Copy the successfully voted RC to dist/release/lucene/java/ and dist/release/lucene/solr/, excluding the maven/ directories.

    4. Commit the release candidate. # this might take some time: ~ 2 beers
  4. Publish Maven Artifacts

  5. Wait 24 hours to give the mirrors a chance to get the new release. There is a script that will continually check the number and percentage of mirrors (and Maven Central) that have the release: dev-tools/scripts/poll-mirrors.pl.

TODO: optimize this checkout to use no-parents or whatever the svn option is to not require the downloading of old releases

Pushing website changes & Announcing

Website += javadocs

The problem: Lucene's and Solr's voluminous per-release javadocs break the standard CMS process for the website (i.e., committer commits to the source tree; the CMS buildbot generates the site, then commits to the staging tree; committer reviews and then publishes to the production tree), because dynamic website updates, currently scheduled at 21 minutes after the hour on the hour, interrupt the extremely long commit times for javadocs being staged by buildbot, resulting in failed commits, caused by conflicts with the dynamic updates: by the time the buildbot-triggered commit has finished, its svn tree has been rendered stale.

The solution: skip committing javadocs to the source tree, then staging, then publishing, and instead commit javadocs directly to the production tree. Ordinarily this would be problematic, because the CMS wants to keep the production tree in sync with the staging tree, so anything it finds in the production tree that's not in the staging tree gets nuked. However, the CMS has a built-in mechanism to allow exceptions to the keep-production-in-sync-with-staging rule: extpaths.txt.

extpaths.txt lists paths in the production tree, relative to the project website's root directory, that are allowed to be out of sync with the staging tree.

For more info, see the following sections in the Apache CMS Reference:

Update extpaths.txt

  1. svn co --depth=immediates https://svn.apache.org/repos/asf/lucene/cms/trunk/content website-source

  2. cd website-source

  3. Add Lucene javadocs dir: echo core/X_Y_Z >> extpaths.txt

  4. Add Solr javadocs dir: echo solr/X_Y_Z >> extpaths.txt

  5. svn commit -m "Update CMS production sync exceptions for X_Y_Z javadocs" extpaths.txt

  6. cd ..

  7. rm -rf website-source

Push javadocs to the CMS production tree

Don't be alarmed that you don't see your extpaths.txt changes yet in this production checkout: after you do this section you will then be able to publish the lucene website from staging and extpaths.txt will update in this repository automatically.

  1. Ensure your refridgerator has at least 4 beers.
  2. svn co https://svn.apache.org/repos/infra/websites/production/lucene/content/ website-production

  3. untar the release.tgz to create releaseX_Y_Z/
  4. cp -r releaseX_Y_Z/lucene/docs website-production/core/X_Y_Z

  5. svn add core/X_Y_Z

  6. svn commit # this might take some time: ~ 3 beers

  7. cp -r releaseX_Y_Z/solr/docs website-production/solr/X_Y_Z

  8. svn add solr/X_Y_Z

  9. svn commit # this might take some time: ~ 1 beer

  10. Confirm you can browse to these directories manually, and especially that solr javadocs link back to lucene's correctly.

For the 'cp' commands above, See also HowToGenerateNiceJavadocs if you optionally feel like making prettier javadocs. But don't worry about this if it seems confusing, someone else could always volunteer to do it later at any time.

Update redirect to latest Javadoc

We make it possible to link to latest javadoc by providing redirect links for e.g. http://lucene.apache.org/solr/api/solr-core/ which will auto redirect to whatever is the latest released version, i.e. http://lucene.apache.org/solr/4_3_0/solr-core/ for 4.3.0. This is handled in .htaccess:

  1. Goto CMS root http://lucene.apache.org/

  2. Click on the CMS bookmarklet

  3. Scroll down to the file .htaccess and click [Edit]

  4. Locate the three lines starting with RedirectMatch temp /core|solr/api/... and change the version url component to match X_Y_Z above. Click Submit

  5. Click Commit, fill in a message and commit your change. No need to publish site yet - that will be done in next chapter.

NOTE As an alternative to CMS online editing, you can checkout the site from SVN and edit/commit .htaccess that way.

Update the rest of the website

NOTE: don't do this until you are ready to go to production. Every hour at :21 (e.g. 8:21, 9:21, ...) buildbot will commit all content from staging to production.

  1. Once mirrors are ready, use the bookmarklet or whatever to do the minor changes and news blurbs and stuff.
  2. update the core & solr doap.rdf files to reflect the new versions using the Apache CMS

  3. Publish the site, e.g. by visiting http://lucene.staging.apache.org, invoking the CMS bookmarklet, then clicking the publish link.

  4. Wait for these changes to appear on Apache's main webserver (http://lucene.apache.org) before doing the next steps (see http://www.apache.org/dev/project-site.html for details on how the site is mirrored to Apache's main web servers). Once they appear, verify all links are correct in your changes!

Announce the release

  1. Announce the release on general@lucene.apache.org , dev@lucene.apache.org , java-user@lucene.apache.org and announce@apache.org mailing lists. A draft should be sent to the development list first to clarify any possible issues in the announcement. Make sure the announcement has a brief blurb at the top saying what Lucene is (it will save you several snarky comments in reply.) Mails to the announce list should be sent from an @apache.org email address and contain a signature. Release announcements can be shared/edited on the wiki at http://wiki.apache.org/lucene-java/ReleaseNoteXY and http://wiki.apache.org/solr/ReleaseNoteXY, or in https://svn.apache.org/repos/asf/lucene/dev/misc/announcements.

  2. Add the new version to Wikipedia (english and maybe your own language)
  3. Add the new release to https://freecode.com/ (formerly freshmeat.com)


Update JIRA

  1. Go to the JIRA "Manage Versions" Administration pages (https://issues.apache.org/jira/plugins/servlet/project-config/LUCENE/versions and https://issues.apache.org/jira/plugins/servlet/project-config/SOLR/versions), click on the release date field for the version you just released, put in the release date, and then click the "Update" button.

  2. Go to JIRA search in both Solr and Lucene and find all issues that were fixed in the release you just made, whose Status is Resolved, and do a bulk change to close all of these issues. Uncheck the box that says "send an email for these changes".
  3. Do another JIRA search in both Solr and Lucene to find all issues with Unresolved Resolution and fixVersion of the release you just made, and do a bulk change to the fixVersion to be both the trunk version and the next version on the branch you just released from. Uncheck the box that says "send an email for these changes".

Don't mirror old releases

Shortly after new releases are first mirrored, they are copied to the archives. Only the latest point release from each active branch should be kept under the Lucene PMC svnpubsub area dist/releases/lucene/. Older releases can be safely deleted from dist/releases/lucene/, since these releases are already backed up in the archives.

svn remove old releases, including X.Y-1, from dist/releases/lucene/java/ and dist/releases/lucene/solr/, then svn commit.

Update WIKI

The Sorl WIKI has a page for every version which is often linked to from WIKI pages to indicate differences between versions, example: http://wiki.apache.org/solr/Solr4.3. Do the following:

  1. Update the page for the released version with release date and link to release statement
  2. Create a new placeholder page for the "next" version, if it does not exist

ReleaseTodo (last edited 2017-10-17 18:44:09 by SteveRowe)