Here I try to summarize what we have discussed about Harmony testing configurations (TestNG) in harmony-dev, and list the problems which we shall solve and make common agreements in community.

Please feel free to edit this page. - Richard Liang

How to handle bootclasspath test

We have agreed that bootclasspath tests and classpath tests should be put into different directories

Reference: How to handle bootclasspath tests

Grouping Harmony Test

We may define Harmony test groups according the following dimensions:

Operating System: os.<os id>

• os.<os id> - group of tests which are designed for one specific operating system. A test may be in more than one of the groups. e.g., @Test(groups={"os.win", "os.linux"})
• @Test - default group of tests which pass on any operating system.
  
  

Architecture: arch.<arch id>

• arch.<arch id> - group of tests which are designed for one specific architecture. A test may be in more than one of the groups. e.g., @Test(groups={"arch.IA32", "arch.IA64"})
• @Test - default group of tests which pass on any architecture.
  

Test Type: type.impl

• type.impl - groups of tests which are designed for Harmony-specific implementation
• @Test - group of tests which are tests for APIs in the Java Specification. We may call it type.api group, but to make the annotations simple we omit "type.api"
  
  

Test Level: level.regression, level.integration, level.stress, etc. (Levels of Test refer to the increase in complexity as moving through test cycle.)

• level.regression - group of tests which are regression for some bugs. (We have no clear requirement to exclude/include regression test, so this group may not be defined.)
• level.integration - group of tests which aim to verify the interaction between modules
• level.stress - group of tests which aim to verify performance
• @Test - group of unit test
  

*Test State*: state.broken, state.broken.\[os id\].\[arch id\], state.broken.<vm>.\[vm id\]

• state.broken - group of tests which fail on every platform, because of bugs of tests or implementation. We need to fix the bugs of tests or implementation to make them pass.

• *_state.broken.\[os id\].\[arch id\]_* - group of tests which fail on one specific platform. "os id" and "arch id" may be omitted. For example, _state.broken.win_ means the test fails on windows with any architecture

• *_state.broken.<vm>.\[vm id\]_* - group of tests which fail on one specific vm \\

Environment: env.display

• env.display - group of tests which need a display to run. For example, ATW and Swing tests.
• @Test - group of tests which have no particular requirement on testing environment.
  

VM: vm.<vm id>

• vm.<vm id> - group of tests which are designed for on specific vm
• @Test - group of tests which pass on all vm.