org.apache.qpid.junit.extensions
Class TKTestRunner

java.lang.Object
  extended by junit.runner.BaseTestRunner
      extended by junit.textui.TestRunner
          extended by org.apache.qpid.junit.extensions.TestRunnerImprovedErrorHandling
              extended by org.apache.qpid.junit.extensions.TKTestRunner
All Implemented Interfaces:
junit.framework.TestListener

public class TKTestRunner
extends TestRunnerImprovedErrorHandling

TKTestRunner extends TestRunner with the ability to run tests multiple times, to execute a test simultaneously using many threads, to put a delay between test runs and adds support for tests that take integer parameters that can be 'stepped' through on multiple test runs. These features can be accessed by using this class as an entry point and passing command line arguments to specify which features to use:

 -w ms       The number of milliseconds between invocations of test cases.
 -c pattern  The number of tests to run concurrently.
 -r num      The number of times to repeat each test.
 -d duration The length of time to run the tests for.
 -t name     The name of the test case to execute.
 -s pattern  The size parameter to run tests with.
 -o dir      The name of the directory to output test timings to.
 --csv       Output test results in CSV format.
 --xml       Output test results in XML format.
 

This command line may also have trailing 'name=value' parameters added to it. All of these values are added to the test context properties and passed to the test, which can access them by name.

The pattern arguments are of the form [lowest(: ...)(: highest)](:sample=s)(:exp), where round brackets enclose optional values. Using this pattern form it is possible to specify a single value, a range of values divided into s samples, a range of values divided into s samples but distributed exponentially, or a fixed set of samples.

The duration arguments are of the form (dD)(hH)(mM)(sS), where round brackets enclose optional values. At least one of the optional values must be present.

When specifying optional test parameters on the command line, in 'name=value' format, it is also possible to use the format 'name=[value1:value2:value3:...]', to specify multiple values for a parameter. All permutations of all parameters with multiple values will be created and tested. If the values are numerical, it is also possible to use the sequence generation patterns instead of fully specifying all of the values.

Here are some examples:

 -c [10:20:30:40:50] 
Runs the test with 10,20,...,50 threads.
 -s [1:100]:samples=10 
Runs the test with ten different size parameters evenly spaced between 1 and 100.
 -s [1:1000000]:samples=10:exp 
Runs the test with ten different size parameters exponentially spaced between 1 and 1000000.
 -r 10 
Runs each test ten times.
 -d 10H 
Runs the test repeatedly for 10 hours.
 -d 1M, -r 10 
Runs the test repeatedly for 1 minute but only takes a timing sample every 10 test runs.
 -r 10, -c [1:5:10:50], -s [100:1000:10000] 
Runs 12 test cycles (4 concurrency samples * 3 size sample), with 10 repeats each. In total the test will be run 199 times (3 + 15 + 30 + 150)
 cache=true 
Passes the 'cache' parameter with value 'true' to the test.
 cache=[true:false] 
Runs the test with the 'cache' parameter set to 'true' and 'false'.
 cacheSize=[1000:1000000],samples=4,exp 
Runs the test with the 'cache' parameter set to a series of exponentially increasing sizes.

CRC Card
Responsibilities Collaborations
Create the test configuration specified by the command line parameters.

Todo:
Verify that the output directory exists or can be created., Verify that the specific named test case to execute exists., Drop the delay parameter as it is being replaced by throttling., Completely replace the test ui test runner, instead of having TKTestRunner inherit from it, its just not good code to extend.

Nested Class Summary
protected static interface TKTestRunner.TestDecoratorFactory
          TestDecoratorFactory is a factory for creating test decorators from tests.
 
Field Summary
protected  boolean csvResults
          Flag that indicates the CSV results listener should be used to output results.
protected  String currentTestClassName
          Holds the name of the class of the test currently being run.
protected  List<TKTestRunner.TestDecoratorFactory> decoratorFactories
          Holds a list of factories for instantiating optional user specified test decorators.
protected  int delay
          Delay in ms to wait between two test cases.
protected  Long duration
          The length of time to run the tests for.
protected  int[] params
          The parameter values to pass to parameterized tests.
protected  Integer repetitions
          Number of times to rerun the test.
protected  String reportDir
          Directory to output XML reports into, if specified.
protected  TKTestResult result
          Holds the test results object, which is reponsible for instrumenting tests/threads to record results.
protected  String testCaseName
          Name of the single test case to execute.
protected  String testClassName
          Name of the test class.
protected  String testRunName
          Name of the test run.
protected  int[] threads
          Number of threads running the tests.
protected static DateFormat TIME_STAMP_FORMAT
          Used for generating the timestamp when naming output files.
protected  boolean xmlResults
          Flag that indiciates the XML results listener should be used to output results.
 
Fields inherited from class junit.textui.TestRunner
EXCEPTION_EXIT, FAILURE_EXIT, SUCCESS_EXIT
 
Fields inherited from class junit.runner.BaseTestRunner
SUITE_METHODNAME
 
Constructor Summary
TKTestRunner(Integer repetitions, Long duration, int[] threads, int delay, int[] params, String testCaseName, String reportDir, String runName, boolean csvResults, boolean xmlResults, List<TKTestRunner.TestDecoratorFactory> decoratorFactories)
          Constructs a TKTestRunner using System.out for all the output.
 
Method Summary
protected  WrappedSuiteTestDecorator applyOptionalUserDecorators(WrappedSuiteTestDecorator targetTest)
          If there were any user specified test decorators on the command line, this method instantiates them and wraps the test in them, from inner-most to outer-most in the order in which the decorators were supplied on the command line.
protected  junit.framework.TestResult createTestResult()
          Creates the TestResult object to be used for test runs.
protected  WrappedSuiteTestDecorator decorateTests(junit.framework.Test test)
          Applies test decorators to the tests for parameterization, duration, scaling and repetition.
 junit.framework.TestResult doRun(junit.framework.Test test, boolean wait)
          Runs a test or suite of tests, using the super class implemenation.
static void main(String[] args)
          The entry point for the toolkit test runner.
protected static List<TKTestRunner.TestDecoratorFactory> parseDecorators(String decorators)
          Parses a list of test decorators, in the form "class.name[:class.name]*", and creates factories for those TestDecorator classes , and returns a list of the factories.
protected  void registerShutdownHook(ShutdownHookable hookable)
          Registers the shutdown hook of a ShutdownHookable.
protected  junit.framework.TestResult start(String testClassName)
          Initializes the test runner with the provided command line arguments and and starts the test run.
 
Methods inherited from class org.apache.qpid.junit.extensions.TestRunnerImprovedErrorHandling
start
 
Methods inherited from class junit.textui.TestRunner
doRun, getLoader, pause, run, run, runAndWait, runFailed, runSingleMethod, setPrinter, testEnded, testFailed, testStarted
 
Methods inherited from class junit.runner.BaseTestRunner
addError, addFailure, clearStatus, elapsedTimeAsString, endTest, extractClassName, getFilteredTrace, getFilteredTrace, getPreference, getPreference, getPreferences, getTest, inMac, inVAJava, loadSuiteClass, processArguments, savePreferences, setLoading, setPreference, setPreferences, showStackRaw, startTest, truncate, useReloadingTestSuiteLoader
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

TIME_STAMP_FORMAT

protected static final DateFormat TIME_STAMP_FORMAT
Used for generating the timestamp when naming output files.


repetitions

protected Integer repetitions
Number of times to rerun the test.


duration

protected Long duration
The length of time to run the tests for.


threads

protected int[] threads
Number of threads running the tests.


delay

protected int delay
Delay in ms to wait between two test cases.


params

protected int[] params
The parameter values to pass to parameterized tests.


testCaseName

protected String testCaseName
Name of the single test case to execute.


testClassName

protected String testClassName
Name of the test class.


testRunName

protected String testRunName
Name of the test run.


reportDir

protected String reportDir
Directory to output XML reports into, if specified.


csvResults

protected boolean csvResults
Flag that indicates the CSV results listener should be used to output results.


xmlResults

protected boolean xmlResults
Flag that indiciates the XML results listener should be used to output results.


currentTestClassName

protected String currentTestClassName
Holds the name of the class of the test currently being run. Ideally passed into the createTestResult() method, but as the signature is already fixed for this, the current value gets pushed here as a member variable.


result

protected TKTestResult result
Holds the test results object, which is reponsible for instrumenting tests/threads to record results.


decoratorFactories

protected List<TKTestRunner.TestDecoratorFactory> decoratorFactories
Holds a list of factories for instantiating optional user specified test decorators.

Constructor Detail

TKTestRunner

public TKTestRunner(Integer repetitions,
                    Long duration,
                    int[] threads,
                    int delay,
                    int[] params,
                    String testCaseName,
                    String reportDir,
                    String runName,
                    boolean csvResults,
                    boolean xmlResults,
                    List<TKTestRunner.TestDecoratorFactory> decoratorFactories)
Constructs a TKTestRunner using System.out for all the output.

Parameters:
repetitions - The number of times to repeat the test, or test batch size.
duration - The length of time to run the tests for. -1 means no duration has been set.
threads - The concurrency levels to ramp up to.
delay - A delay in milliseconds between test runs.
params - The sets of 'size' parameters to pass to test.
testCaseName - The name of the test case to run.
reportDir - The directory to output the test results to.
runName - The name of the test run; used to name the output file.
csvResults - true if the CSV results listener should be attached.
xmlResults - true if the XML results listener should be attached.
decoratorFactories - List of factories for user specified decorators.
Method Detail

main

public static void main(String[] args)
The entry point for the toolkit test runner.

Parameters:
args - The command line arguments.

parseDecorators

protected static List<TKTestRunner.TestDecoratorFactory> parseDecorators(String decorators)
Parses a list of test decorators, in the form "class.name[:class.name]*", and creates factories for those TestDecorator classes , and returns a list of the factories. This list of factories will be in the same order as specified in the string. The factories can be used to succesively wrap tests in layers of decorators, as decorators themselves implement the 'Test' interface.

If the string fails to parse, or if any of the decorators specified in it are cannot be loaded, or are not TestDecorators, a runtime exception with a suitable error message will be thrown. The factories themselves throw runtimes if the constructor method calls on the decorators fail.

Parameters:
decorators - The decorators list to be parsed.
Returns:
A list of instantiated decorators.

doRun

public junit.framework.TestResult doRun(junit.framework.Test test,
                                        boolean wait)
Runs a test or suite of tests, using the super class implemenation. This method wraps the test to be run in any test decorators needed to add in the configured toolkits enhanced junit functionality.

Overrides:
doRun in class junit.textui.TestRunner
Parameters:
test - The test to run.
wait - Undocumented. Nothing in the JUnit javadocs to say what this is for.
Returns:
The results of the test run.

decorateTests

protected WrappedSuiteTestDecorator decorateTests(junit.framework.Test test)
Applies test decorators to the tests for parameterization, duration, scaling and repetition.

Parameters:
test - The test to decorat.
Returns:
The decorated test.

applyOptionalUserDecorators

protected WrappedSuiteTestDecorator applyOptionalUserDecorators(WrappedSuiteTestDecorator targetTest)
If there were any user specified test decorators on the command line, this method instantiates them and wraps the test in them, from inner-most to outer-most in the order in which the decorators were supplied on the command line.

Parameters:
targetTest - The test to wrap.
Returns:
A wrapped test.

createTestResult

protected junit.framework.TestResult createTestResult()
Creates the TestResult object to be used for test runs. See TKTestResult for more information and the enhanced test result class that this uses.

Overrides:
createTestResult in class junit.textui.TestRunner
Returns:
An instance of the enhanced test result object, TKTestResult.

registerShutdownHook

protected void registerShutdownHook(ShutdownHookable hookable)
Registers the shutdown hook of a ShutdownHookable.

Parameters:
hookable - The hookable to register.

start

protected junit.framework.TestResult start(String testClassName)
                                    throws Exception
Initializes the test runner with the provided command line arguments and and starts the test run.

Parameters:
testClassName - The fully qualified name of the test class to run.
Returns:
The test results.
Throws:
Exception - Any exceptions from running the tests are allowed to fall through.


Licensed to the Apache Software Foundation