The NetUI tests consist of two parts – standalone JUnit tests and Test Recorder (TR) tests. Both types of test are based on JUnit. The standalone JUnit tests are just that; TestCase subclasses that test the NetUI APIs outside of a running server. The Test Recorder tests are tests that run against a running Tomcat instance in Beehive.

Both the standalone JUnit and Test Recorder tests have two broad categorizations – DRTs and BVTs. The DRT (Developer Regression Test) suite *must* pass at 100% before any checkins can be submitted into the $BEEHIVE_HOME/netui tree. The BVT (Build Verification Tests) suite consists of all of tests to run. Most of us tend to run the BVTs before we checkin, though they don't always pass at 100%; for example, today we've got 21 TR tests that fail out 404 total. Generally, additional BVTs shouldn't break on checkins either. <g>

Currently, all test suites produce JUnit test results in the usual format using the <junitreport> Ant task. These results are produced in a subdirectory of $BEEHIVE_HOME/netui/build. The specific locations for the JUnit and TR tests are below.

Contents

NetUI Server Tests

In addition to the API-level JUnit tests, NetUI uses a piece of home-grown software called the Test Recorder (TR) to run tests against a live server – Tomcat is the server in the context of Beehive. The DRT suite currently runs from this directory

   $BEEHIVE_HOME/netui/test/webapps/drt/

against a webapp in a subdirectory of the above called "coreWeb" using the Test Recorder configuration / tests stored subdirectories testRecorder/config and testRecorder/tests respectively. There is also a build.xml file in the above directory that controls the Tomcat server, coreWeb build, and running test suites. This directory is assumed to be the working directory for the rest of the Ant commands in this section.

In order to run a set of TR tests, the following steps occur:

build the coreWeb webapp
start Tomcat
deploy the coreWeb webapp
run a test suite (drt, bvt, data binding, etc), which produces an HTML test result report
stop Tomcat

Currently in Beehive, the instance of Tomcat that is used lives in:

 $BEEHIVE_HOME/installed/jakarta-tomcat-5.0.25/...

When the DRTs run as part of an "ant clean deploy drt" from either $BEEHIVE_HOME/ or $BEEHIVE_HOME/netui, this process is done automatically meaning that the build takes care of building the webapp, starting / stopping Tomcat, and running the tests. The result of a DRT run can be found here:

   $BEEHIVE_HOME/netui/build/testRecorder/junit/drt/html/junit-noframes.html

In addition to an automatic DRT run, the same steps can be run manually from the $BEEHIVE_HOME/netui/test/webapps/drt directory. This is very useful when developing tests or just learning about NetUI as a normal web browser can be used to walk the tests manually. To start the Tomcat server, you'll need two shells both with a Beehive development environment (see $BEEHIVE_HOME/BUILDING.txt) in the directory $BEEHIVE_HOME/netui/test/webapps/drt.

In the first shell, start Tomcat:

   ant start

In the second shell, build the coreWeb web application:

   ant build

Now, the "coreWeb" web application is ready to be installed on the running Tomcat server. The steps needed to deploy to Tomcat can vary depending on the state of Tomcat. There are three relevant targets:

deploy – deploys the webapp for the first time. This will fail if the webapp is already running.
undeploy – undeploys a running webapp from Tomcat. This will fail if the webapp is not running or did not start successfully
redeploy – redeploy a successfully running webapp. This will fail if the webapp did not start successfully

These targets can be run in combination to make sure that a webapp deploys correctly to Tomcat; for example, if the "coreWeb" webapp is already deployed, it needs to be redeployed or undeployed and then deployed. Feedback for each step is available on the console of shell two. Also, it is possible for a webapp to fail deployment, so just be sure to read the console output messages to make sure that deployment was in fact successful. To deploy "coreWeb" initially, run in shell two:

   ant deploy

To redeploy this webapp, run ant redeploy or ant undeploy deploy. The latter is usually a safe command as it will remove the webapp if it doesn't exist and then deploy it. Note, it's acceptable for some of these targets to fail – if "coreWeb" isn't running on Tomcat and ant undeploy is run, it will fail like this:

Buildfile: build.xml
undeploy:
undeploy:
[tomcatundeploy] FAIL - No context exists for path /coreWeb
BUILD FAILED
C:\dev\apache\beehive\netui\test\webapps\drt\build.xml:147: Following error occurred while executing this line
C:\dev\apache\beehive\test\ant\runTomcat.xml:31: FAIL - No context exists for path /coreWeb

To deploy it, just run "ant deploy" which succeeds with output like this:

Buildfile: build.xml
deploy:
deploy:
    [echo] deploy webapp from file://c:\dev\apache\beehive/netui/test/webapps/drt/coreWeb with context path coreWeb
[tomcatdeploy] OK - Deployed application at context path /coreWeb
BUILD SUCCESSFUL
Total time: 13 seconds

Once the coreWeb webapp is deployed successfully, it is accessible using the URL

   http://localhost:8080/coreWeb

which should bring up a test suite index page with the Beehive logo and a list of tests and test categories. When running a particular test, the yellow bar at the bottom of each page is the Test Recorder.

To run the DRT test suite against a running server, run this command in shell two:

   ant drt.running

which puts the test results here:

   $BEEHIVE_HOME/netui/build/testRecorder/junit/drt/html/junit-noframes.html

To run the BVT test suite from the same directory, run this command in shell two:

   ant bvt.running

which puts the test results here:

   $BEEHIVE_HOME/netui/build/testRecorder/junit/bvt/html/junit-noframes.html

For more information, take a look through the $BEEHIVE_HOME/netui/test/webapps/drt/build.xml file and look at the targets and how they call into the webapp build and start / stop Tomcat targets in build files that live in:

   $BEEHIVE_HOME/test/ant

To stop Tomcat, run ant stop in shell two.

When diagnosing a test failure or problem running the webapp, there are several places to look to see if errors have occurred:

the Tomcat console
the shell two console for errors reported when test recorder tests fail
the Tomcat log files, which are stored by date in $BEEHIVE_HOME/external/jakarta-tomcat-5.0.25/logs
in addition, NetUI can log error messages into the Tomcat console, and soon into a netui.log file

When reporting errors into the beehive-dev mailing list, having any failures / errors reported in the above helps when diagnosing a problem. Also, because we have error / failure test cases, it is often acceptable for NetUI tests to throw exceptions or errors in the course of regular execution. While seeing a stack trace can be alarming, lots of tests should do this. <g> So, just be sure to check the test result output as the source of record for the success of a particular test or suite.

Overview of the Test Recorder

The Test Recorder tests comprise the bulk of the NetUI tests suite. The TR is a piece of software that NetUI develops which consists of client and server pieces. It simulates the interaction of a web browser with a running server. The TR runs in two modes – "record" and "playback". Developers use the TR in "record" mode to record a set of user interactions through a normal web browser (like Firefox) against a running, Test Recorder enabled web application. The result is a "recorded" test which is stored in an XML file; examples of these can be found in $BEEHIVE_HOME/netui/test/webapps/drt/testRecorder/tests. When the DRT or BVT suites run, the client half of the TR uses these "record" XML files to "playback" a set of user interactions with the server. This is done by reading request parameters out of the "record" file and building requests to send to the server; then, the response stream is captured which results in a "playback" file. The "record" and "playback" results are compared; if there are no differences between the two, the tests are considered a pass. Otherwise, the test is a failure. Currently, this diff is done line-by-line and in the future could be done as an XML or HTML diff.

The TR is a very strict and accurate way for changes in the NetUI runtime to be verified. Because the response stream of a rendered JSP is differenced against an expected result, even simple changes can break tests, which can be good because it helps NetUI developers understand the impact of changes to the code base. Thus, we these tests help to maintain NetUI stability.

The TR is configured on a per-webapp basis using files like those used to configure the DRT's coreWeb webapp:

   $BEEHIVE_HOME/netui/test/webapps/drt/testRecorder/config

The TR uses its own Ant file to run a test suite against a server:

   $BEEHIVE_HOME/netui/test/ant/testRecorder.xml

Certainly, more documentation will follow on the TR and how to record / update test results.

JUnit API Tests

The source for these tests lives in $BEEHIVE_HOME/netui/test/src/junitTests/.... The TestCases which run as part of the DRTs are specified in $BEEHIVE_HOME/netui/test/ant/junitDrt.properties. The JUnit DRTs can be run like this:

   cd $BEEHIVE_HOME/netui/test/ant
   ant drt

This will build all of the test code and run the JUnit tests specified in the above file. We use a simple <netui-junit> Ant task that takes the .properties file above and filters all of the classes in the junitTests.jar file to produce the set of TestCases to run as DRTs. The Ant to drive the JUnit DRTs lives in $BEEHIVE_HOME/netui/test/ant/junit.xml. The JUnit test results can be found here:

   $BEEHIVE_HOME/netui/build/drt.testResults/html/junit-noframes.html

Page tree

NetUI Testing

NetUI Server Tests

Overview of the Test Recorder

JUnit API Tests