This page is prepared for Hadoop Core committers. You need committer rights to create a new Hadoop Core release.
These instructions have been updated to use dev-support/bin/create-release. Earlier versions of this document are at HowToReleaseWithSvnAndAnt and HowToReleasePostMavenization and HowToReleasePreDSBCR. For releasing from the 2.6.x or the 2.7.x line, you'll need to consult HowToReleasePreDSBCR to find applicable steps.
Applicable Hadoop versions: 2.8.0 and above; 3.0.0 and above.
READ ALL OF THESE INSTRUCTIONS THOROUGHLY BEFORE PROCEEDING!
Start with this query:
project in (HADOOP, HDFS, YARN, MAPREDUCE) AND "Target Version/s" = 3.0.0-beta1 and statusCategory != Done |
Filter this list down until it's only issues with a Target Version of just "3.0.0-beta1". My query ended up looking like:
project in (HADOOP, HDFS, YARN, MAPREDUCE) AND "Target Version/s" = 3.0.0-beta1 and "Target Versions/" not in (2.9.0, 2.8.3, 2.8.2) AND statusCategory != Done |
Check the query for the next most common set of target versions and again filter it down:
project in (HADOOP, HDFS, YARN, MAPREDUCE) AND "Target Version/s" = 3.0.0-beta1 and "Target Version/s" = 2.9.0 and statusCategory != Done project in (HADOOP, HDFS, YARN, MAPREDUCE) AND "Target Version/s" = 3.0.0-beta1 and "Target Version/s" = 2.9.0 and "Target Version/s" not in (2.8.2, 2.8.3) and statusCategory != Done |
Return to the original query. If there aren't too many, update the remaining straggler issues by hand (faster than doing the bulk edits):
project in (HADOOP, HDFS, YARN, MAPREDUCE) AND "Target Version/s" = 3.0.0-beta1 and statusCategory != Done |
To deploy artifacts to the Apache Maven repository create ~/.m2/settings.xml
:
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd"> <servers> <server> <id>apache.staging.https</id> <username>Apache username</username> <password>Apache password</password> </server> </servers> </settings> |
When releasing Hadoop X.Y.Z, the following branching changes are required. Note that a release can match more than one of the following if-conditions. For a major release, one needs to make the changes for minor and point releases as well. Similarly, a new minor release is also a new point release.
Update the version on trunk to (X+1).0.0-SNAPSHOT
mvn versions:set -DnewVersion=(X+1).0.0-SNAPSHOT |
hadoop.version
in the root pom.xml
file to the same value; validate with a clean build.Commit the version change to trunk.
git commit -a -m "Preparing for (X+1).0.0 development" |
Update the version on branch-X to X.(Y+1).0-SNAPSHOT
mvn versions:set -DnewVersion=X.(Y+1).0-SNAPSHOT |
hadoop.version
in the root pom.xml
file to the same value; validate with a clean build.Commit the version change to branch-X.
git commit -a -m "Preparing for X.(Y+1).0 development" |
Update the version on branch-X.Y to X.Y.(Z+1)-SNAPSHOT
mvn versions:set -DnewVersion=X.Y.(Z+1)-SNAPSHOT |
hadoop.version
in the root pom.xml
file to the same value; validate with a clean build.Commit the version change to branch-X.Y.
git commit -a -m "Preparing for X.Y.(Z+1) development" |
hadoop-project/src/site/markdown/index.md.vm
to reflect the right versions, new features and big improvements.Update the version on branch-X.Y.Z TO X.Y.Z
mvn versions:set -DnewVersion=X.Y.Z |
Note: Please also also update the hadoop.version property in the root pom.xml (see HADOOP-15369) and for releases off branches earlier than 3.3.1, the hadoop.assemblies.version in hadoop-project/pom.xml
mvn versions:set-property -Dproperty=hadoop.version -DnewVersion=X.Y.Z mvn versions:set-property -Dproperty=hadoop.assemblies.version -DnewVersion=X.Y.Z |
(The hadoop.assemblies.version update isn't needed on recent releases; since HADOOP-17663 the command to set it will not touch any files, and so completely harmless)
Now, for any branches in {trunk, branch-X, branch-X.Y, branch-X.Y.Z} that have changed, push them to the remote repo taking care of any conflicts.
git push <remote> <branch> |
These steps need to be performed to create the _N_th RC for X.Y.Z, where N starts from 0.
<release-year>
in hadoop-project/pom.xml
). If not, create a JIRA to update the property value to the right year, and propagate the fix from trunk to all necessary branches. Consider the voting time needed before publishing, it's better to use the year of (current time + voting time) here, to be consistent with the publishing time.For the Apache release, a machine capable of running Docker- and Internet- capable, build the release candidate with create-release
. Unless the --logdir
is given, logs will be in the patchprocess/
directory. Artifacts will be in the target/artifacts NOTE: This will take quite a while, since it will download and build the entire source tree, including documentation and native components, from scratch to avoid maven repository caching issues hiding issues with the source release.
For x86 architecture CPU, we can use the following command:
dev-support/bin/create-release --asfrelease --docker --dockercache |
Note:
if we encounter a 401 permission issue, it may be due to the local Maven configuration not being read. We can try the following command.
dev-support/bin/create-release --asfrelease --docker --dockercache --mvnargs="-Duser.home=<mvn setting.xml path>" |
For Arm architecture CPU, we can use the following command:
dev-support/bin/create-release --docker --dockercache --mvnargs="-Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false" --deploy --native --sign |
Note:
While it should fail create-release
if there are issues, doublecheck the rat log to find and fix any potential licensing issues.
grep 'Rat check' patchprocess/mvn_apache_rat.log |
export version=X.Y.Z-RCN
Tag the release candidate:
git tag -s release-$version -m "Release candidate - $version" |
(This is for branch-2.10 only. mvn deploy
is invoked by the dev-support/bin/create-release script of 3.0.0 and above. see HADOOP-15058.)
Deploy the maven artifacts, on your personal computer. Please be sure you have completed the prerequisite step of preparing the settings.xml
file before the deployment. You might want to do this in private and clear your history file as your gpg-passphrase is in clear text.
mvn deploy -Psign,dist -DskipTests |
Copy release files to a public place and ensure they are readable. Note that home.apache.org
only supports SFTP, so this may be easier with a graphical SFTP client like Nautilus, Konqueror, etc.
sftp home.apache.org > cd public_html > mkdir hadoop-${version} > put -r /home/hadoop/hadoop-${version} .... > bye |
Staging
Repositories" from the left navigation pane, select the check-box against the specific hadoop repository, and close
the release. At the time of deploying the maven artifacts, if there are different hadoop items of this release candidate at “Staging Repositories”, drop the stale RC first.In 5 days if the release vote passes, the release may be published.
export version=X.Y.Z
Tag the release. Do it from the release branch and push the created tag to the remote repository:
git tag -s rel/release-${version} -m "Hadoop ${version} release" git push origin rel/release-${version} |
Check out the corresponding svn repo if need be
svn co https://dist.apache.org/repos/dist/release/hadoop/common/ hadoop-dist |
Commit the changes (it requires a PMC privilege)
svn ci -m "Publishing the bits for release ${version}" |
svn move
.Copy and commit the CHANGELOG.md and RELEASENOTES.md:
cp target/artifacts/RELEASENOTES.md hadoop-common-project/hadoop-common/src/site/markdown/release/${version}/RELEASENOTES.${version}.md cp target/artifacts/CHANGELOG.md hadoop-common-project/hadoop-common/src/site/markdown/release/${version}/CHANGELOG.${version}.md |
Copy the jdiff xml files for this version to their appropriate directory.
cp hadoop-common-project/hadoop-common/target/site/jdiff/xml/Apache_Hadoop_Common_${version}.xml hadoop-common-project/hadoop-common/dev-support/jdiff cp hadoop-hdfs-project/hadoop-hdfs/target/site/jdiff/xml/Apache_Hadoop_HDFS_${version}.xml hadoop-hdfs-project/hadoop-hdfs/dev-support/jdiff find hadoop-yarn-project -name "Apache_Hadoop_YARN_*_${version}.xml" | xargs -I{} cp {} hadoop-yarn-project/hadoop-yarn/dev-support/jdiff find hadoop-mapreduce-project -name "Apache_Hadoop_MapReduce_*_${version}.xml" | xargs -I{} cp {} hadoop-mapreduce-project/dev-support/jdiff |
Update hadoop-project-dist/pom.xml
<jdiff.stable.api>X.Y.Z</jdiff.stable.api> |
Release
Checkout the website if you haven't already
git clone https://gitbox.apache.org/repos/asf/hadoop-site.git -b asf-site |
Create the new release announcement (Set environment variable version first. export VERSION=X.Y.Z).
cat << EOF > src/release/${VERSION}.md --- title: Release ${VERSION} available date: 202X-XX-XX linked: true --- <!--- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> This is the first stable release of Apache Hadoop TODO line. It contains TODO bug fixes, improvements and enhancements since TODO. Users are encouraged to read the [overview of major changes][1] since TODO. For details of TODO bug fixes, improvements, and other enhancements since the previous TODO release, please check [release notes][2] and [changelog][3]. [1]: /docs/r${VERSION}/index.html [2]: http://hadoop.apache.org/docs/r${VERSION}/hadoop-project-dist/hadoop-common/release/${VERSION}/RELEASENOTES.${VERSION}.html [3]: http://hadoop.apache.org/docs/r${VERSION}/hadoop-project-dist/hadoop-common/release/${VERSION}/CHANGELOG.${VERSION}.html EOF |
linked: true
line from the previous release file, eg. from src/release/3.0.0.md. Docs/downloads of the releases with linked:true
will be linked from the menu.add the docs and update the content/docs/current
link, by doing the following:
cd content/docs tar xvf /path/to/hadoop-${version}-site.tar.gz # Update current2, current, stable and stable2 as needed. # For example rm current2 current ln -s r${version} current2 ln -s current2 current |
hugo serve && firefox http://localhost:1313
Regenerate the site, review it, then commit it per the instructions in HowToCommit. (The generated HTML files also should be committed. Both src and the rendered site are in the same repo.)
hugo git add . git commit git push |
Docker images containing release binaries are accessible at DockerHub. A docker image is automatically built by pushing a commit to one of the two special branches: docker-hadoop-2 and docker-hadoop-3. An example is HADOOP-18681. Please contact Ayush Saxena and Wei-Chiu Chuang who have admin privilege to the Apache Hadoop DockerHub repo and can add additional tags to the docker image produced.
The docker images can be downloaded using docker command:
docker pull apache/hadoop:3.3.5 |
The hadoop-thirdparty repository shades a number of thirdparty dependencies that are used by Hadoop. They are shaded to avoid classpath conflict with downstream applications. The Hadoop project may occasionally need to update and release hadoop-thirdparty artifacts. The steps are largely similar to the above. See the How To Release Hadoop-Thirdparty wiki for details.