How to use the render kit testing framework


<a name="Description">1.</a> Description

The render kit testing framework lets Trinidad developers write unit tests for renderers. It cannot test that your renderers are truly "correct": that requires visual and behavioral inspection. It is not intended to test the component-level processing of our components; that is the responsibility of other unit tests. What it can do is test the following:

  1. Verify that output does not change from checkin to checkin when not intended.
  2. Verify that when output does change, that it changes only in precisely the expected way. (It is crucially important to note that we are not in any way guaranteeing that our HTML will not vary from release to release, sometimes massively so; the testing framework is designed to catch unintentional changes. Application developers, therefore, cannot rely on the stability of our HTML to perform regression testing.)

  3. Verify that attributes that are supposed to have an effect on the output are not ignored
  4. Conversely, verify that default values for attributes have no effect on the output
  5. Verify that the proper UIComponent is passed to startElement() (necessary both for visual editing and for PPR to function)
  6. Verify that exactly the same calls are made to ExternalContext and ViewHandler APIs for encoding URLs, even where those APIs often are no-ops at runtime. (These calls are never no-ops in the testing framework. Note that this cannot test that you're making the right calls - just that changes to renderer code doesn't add or remove calls unexpectedly.)

  7. Verify some common output errors don't happen: unbalanced output, duplicate attributes

These set of tests let us catch two common sources of bugs: PPR failures related to a failure to passing the UIComponent to startElement() directly, and many, many attributes that are exposed on our tags yet not actually rendered in any way! (Javascript attributes and shortDesc appear to be the worst offenders).

In addition, by not requiring the JSP engine to be present, the testcases avoid pulling in unnecessary dependencies and testing "too much". The testing framework finally gives us a chance to make core changes in the framework, and even entirely rewrite renderers, with a high degree of confidence that our results are entirely as intended.

Test scripts are stored as simple XML files in trinidad-impl/src/test/resources/org/apache/myfaces/trinidadinternal/renderkit/testScripts. The scripts are designed to be extremely simple, nearly trivial, to write, to ensure that we have coverage as wide as possible.

The desired output of the testcases is stored as golden files in trinidad-impl/src/test/resources/org/apache/myfaces/trinidadinternal/renderkit/golden. These files are source-controlled. The output of the script is always compared against this golden file. Any diffs are considered a test failure, as are failures to meet with the other verifications listed above.

The output of these golden files is output in a special manner to make diffing easy. All attributes are listed on separate lines, and attributes are always sorted alphabetically. This means that we are not testing the order in which attributes are written, which we shouldn't be testing, since order doesn't mean anything. (Note that these files are written out in XML, not HTML, and this means that boolean attributes show up as disabled="disabled", not just disabled.)

1.1 Tested combinations

We currently test a fairly large set of skin/agent combinations. However, we only actually verify all the assertions listed above for the minimal skin on Gecko; this is to avoid generating a large number of errors. For all other combinations, we're only checking that the visualization has not changed from one checkin to the next.

The full list of combinations:

This is missing some combinations: we aren't testing any multibyte languages (these generate very, very few diffs for our output), nor are we testing Palm, or our "printable" output mode, but it does give us a very wide coverage of all critical output paths.

1.2 When you get a failure or an error

When you get a failure or an error, you must take the following steps:

If it is an error (duplicate attributes, tests that should diff but don't, etc.), you must:

If it is a test failure caused by a golden file diff, you must:

<a name="HowTo">2.</a> How To

The tests are run by the org.apache.myfaces.trinidadinternal.renderkit.CoreRenderKitTest test case, in the trinidad-impl project (NOT from the top level):

  mvn -Dtest=CoreRenderKitTest test

To add a testcase, add a new .xml file into the trinidad-impl/src/test/resources/org/apache/myfaces/trinidadinternal/renderkit/testScripts directory. (Syntax of those testcases discussed below.) When a diff fails, or if there is no golden file at all, its output will be written into the trinidad-impl/target/test-failures directory. After diffing the "test-failures" and "golden" directories and determining that all diffs are expected, move the failures into the "golden" directory and be sure to check in the golden file along with the test script (or any fixes).

Other command-line switches are:

2.1. Testcase syntax

Just to get started, here's an example:

<test xmlns=""
  <base-component tr:type="org.apache.myfaces.trinidad.CoreOutputText" value="Text"/>
  <attribute-test name="styleClass" value="OraErrorText"/>

This is the basic structure all test scripts follow.

  1. A root <test> element

  2. A base-component element defining the "base" component
  3. A series of test elements defining extra tests to apply on top of the component

2.1.1 Building components

The type of the component is defined by tr:type, which must match a componentType value. (NOTE: javax.faces componentTypes are not supported!)

Attributes can be set directly as attributes on the element.

A limited subset of EL can be used in attributes:

To add a child, just add a <component> element (NOT <base-component>) inside your component.

To make that child a facet, set the "tr:facet" attribute.

An example of most of this:

  <base-component tr:type="org.apache.myfaces.trinidad.CoreTable" value="#{simpleList}" var="row" rows="5">
    <component tr:type="org.apache.myfaces.trinidad.CoreColumn">
      <component tr:facet="header" tr:type="org.apache.myfaces.trinidad.CoreOutputText" value="StringHeader"/>
      <component tr:type="org.apache.myfaces.trinidad.CoreOutputText" value="#{row.string}"/>
    <component tr:type="org.apache.myfaces.trinidad.CoreColumn">
      <component tr:facet="header" tr:type="org.apache.myfaces.trinidad.CoreOutputText" value="IntHeader"/>
      <component tr:type="org.apache.myfaces.trinidad.CoreOutputText" value="#{}"/>

2.1.2 attribute-test

<attribute-test> supports three attributes: "name", "value", and "matchesBase". Type-conversion is supported; EL is not. If "value" is ommitted, <attribute-test> will assume its a string, and will use "test-[name]" as the value. e.g., <attribute-test name="onclick"> tests onclick with the value test-onclick. This is usually fine because with most strings, you just care that the value is used, not how its used (since that's just about impossible to test anyway.)

If "matchesBase" is set to true, then the test is required to be unchanged from the base test. This is useful for testing default values.

An attribute-test can contain other attribute-test elements, in which case the test will consist of setting all the attributes; this is important for attributes that have no effect unless others are also set (e.g., bandingInterval is ignored if banding is not set).

2.1.3 js-test

<js-test> is a convenience element that is like adding <attribute-test> for every attribute beginning with "on" (e.g., onclick, ondblclick, etc.)

2.1.4 enum-test

For enumerated types, <enum-test> lets you test all values of the enumeration with one element. An example from panelGroup:

  <enum-test name="type" default="default"/>

We've specified "default" here only because, for some reason, panelGroup is not currently defining a default value for this attribute (which is basically a bug in panelGroup).

2.1.5 boolean-test

<boolean-test> lets you test both true and false values for a boolean property. Of course, it verifies that the "default" produces no changes, and that the non-default does. However, it adds an additional feature. You can put additional tests inside of <boolean-test>, and it'll run those tests with both values of the boolean property also set. This is very useful for testing boolean properties that have widespread effects on our components. For example, "simple" on our input components massively changes output, so here's a snippet of a test for inputText:

<test ...>
  <base-component tr:type="org.apache.myfaces.trinidad.CoreInputText" value="Text"/>
  <boolean-test name="simple">
    <attribute-test name="onclick"/>
    <attribute-test name="styleClass" value="OraErrorText"/>
    <attribute-test name="fooBar" matchesBase="true"/>
    <boolean-test name="disabled"/>
    <boolean-test name="readOnly"/>

<boolean-test> does not support matchesBase! It supports "default", and the default has to match the base.

2.1.6 Other tests

If you'd like to add another test, please do! Add code to TestScript to define the new test type, add code to TestScriptParser to define how to parse it, and document it here.

2.2 What not to test

There's been a lot of confusion about what should or should not be tested.

Tests should not try to test every property of a component. Specifically, they should not test:

<a name="Gotchas">3.</a> Gotchas

Renderers that generate randomized output will break the renderkit testing framework; randomized output must be avoided.

Also, if a Renderer defaults some property differently from run to run - for example, chooseDate defaults its value to the current date - then you'd better set that property on base-component.

"id" is automatically set to "mainId" on the base component, and that ID is reused for every test. However, all tests render from inside of the same UIViewRoot in one pass. Therefore, any sort of auto-incrementing output will make all tests differ from the base, whether they should or not, which can both hide problems and make other non-issues look like problems.

<a name="Issues">3.</a> Issues

We currently surround all components inside a CoreDocument and CoreForm (both with hardcoded IDs), and ignore the output generated by those two tags. This does mean that we can't actually test CoreForm itself or CoreDocument; we should enhance the framework to detect at least these two scenarios.

The component hierarchy is currently fixed for all tests in a single test script, and there's no way to set attributes on anything but the root component or add children or facets. More test scripts would help.


Trinidad_RenderKit_test_framework (last edited 2009-09-20 23:02:03 by localhost)