Building/Testing Hadoop under IDEA
Hadoop builds with Ant, but you can set it up to work under IDEA for testing and some iterative development. This does not take away the need to run Ant; you just run it side by side.
Before you begin
- Be on Java6. That's where Hadoop is going.
- Have the build work on the command line.
- Create and edit a 'build.properties' file; add the following line
The above line is required. The default location, 'build/webapps', will not work. StatusHttpServer locates 'webapps' via classloader.getResource("webapps") and it must be in classpath.
- Do a command line build first, to set up all the output dirs.
Creating a new module
Create a new IDEA module for Hadoop.
build/src conf src/ant src/contrib/streaming/src/java src/core src/examples src/hdfs src/mapred src/native/src src/tools
- Even if you are not working on an area, adding it to the source tree makes refactoring and class/method use queries more reliable.
Everything under build/ goes away on a clean build, and needs to be picked up again by resynchronizing IDEA (if it is not automatic)
By default, the webapp metadata goes into build/webapps, which is not the right place to be picked up by the IDE. Moving it under build/resources/ is needed to place it somewhere manageable.
build/src is required for compiled jsp files. Unfortunately, there is no separated ant task to regenerate them. The best is running ant command line.
conf is required for hadoop-default.xml to be copied to build/classes. Configuration will load hadoop-default.xml as a resource via classloader.
test source directories
- Exclude stuff under there that you do not need.
Like 'Source folders', everything under build/ goes away on a clean build. You can re-create it by running 'generate-test-records' ant task.
Set these to the full path of where Hadoop's Ant build sticks things, such as :
Test output: /home/user/hadoop-core/build/test/classes
everything in the lib/ directory.
If you have global libraries set up for Log4J, JUnit and Jetty (for example), use them and omit the versions in lib/. Do keep in sync with library versions, especially that of Jetty.
For tests, you need to add everything in the src/test/lib/ directory.
Setting up a test run
To run JUnit tests under the IDE, create a new test configuration pointing to the chosen tests.
- Use the classpath and JDK of the Hadoop module.
- Select the package, class or method containing the chosen tests
- VM Parameters: you must set up the logging directory
JRockit users: consider editing conf/log4j.properties to
log4j.appender.console.layout.ConversionPattern=%-4r %-5p %c %x - %m%n
This may seem odd, but it eliminated deadlocks in the logging.