Opentest core infrastructure provides the following components:
- A Test Management System
- A Test Master Controller
- An Execution Engine interface
The top two components above are common across all users of Opentest, regardless of the type of product that they are testing. While the last component, the execution engine interface, is used by Opentesters to implement their own execution engines that suit the specific needs of the products they are testing. This idea of separating things that change from things that don't change is a common theme in software design patterns.
The Opentest architecture has 3 layers:
- TMS: A Test Management System that stores all the artifacts related to testing such as test cases, test plans, test results, etc. The current Opentest implementation uses Testlink as the Test Management System.
- TMC: A Test Master Controller which functions as the brain of the system and takes care of dispatching test request to the appropriate Service Provider.
- Service Providers: Provides critical services required to execute a test.
The TMC has the following sub-components:
- Dispatcher: Receives all the test job requests from the TMS and interacts with the Resource Manager to find the appropriate service provider to execute the next step.
- Resource Manager: Keeps track of all the services providers available in the system, their types and capabilities and select the best candidate for a given request.
- Monitor: Provides information about service providers status.
- TMS Writer: Save test results into TMS.
There are two main types of services providers in a typical test run:
- BEEs: Build Execution Engines. Build or retrieve the software that is going to be tested.
- TEEs: Test Execution Engines. Run the steps to test the software provided by the BEE. The TEEs are the entities that carry most of the actual test job. There are different types of TEEs suited to carry different types of tests. Each test case identifies the TEE type required to run the test. Of course, the same TEE type could be used to run many different test cases.
A typical test case flow run by a TEE includes the following steps:
- Power-cycling the Device Under Test (DUT) and loading the software to be tested on it.
- Booting the DUT.
- Configuring the DUT.
- Configuring any test equipment that might be part of the test.
- Connecting the DUT to the test equipment.
- Executing one or more validation steps involving the DUT and/or the test equipment.
- Determining the test outcome (Pass or Fail)
- Creating results file and logs.
The components on each layer run on machines running STAF. In other words, every machine that is part of Opentest must have STAF installed on it and must be running the STAF daemon process called STAFProc. The components in the system communicate by sending STAF request and exchanging XML files.
The following figure shows the main components described above.
Test Plan data
As shown in the previous section, the Opentest system is very flexible as it allows running tests using different Build and Test Execution Engines. But as it is typically the case, flexibility also brings complexity, so there must be a place in the system to configure or decide which execution engine type to use, which device under test type to use and other parameters, as you can guess that place is the Test Plan that it is exported from the Test Management System (Testlink).
The Test Plan must contain at least the following information:
- List of test cases to run
- Hardware Under Test or Device Under Test (DUT) type.
- Software Under Test and BEE type required to build it.
- Each Test Case in the test plan identifies:
- The TEE type
- TEE Logic or script to load and run on the TEE identified above
- Test Parameters
- Special DUT requirements and test equipment. For instance, the test case might specify that the DUT must have USB ISO capabilities or some specific test equipment is required to run the test.
All communication between the components in the system is done through STAF, therefore, all interfaces for communication between components are defined as a set of STAF requests. The following sections describe the interfaces defined for the OpenTest components.
The interaction between TMS and TMC should provide the means to support the following functionality:
- Start execution of a test session.
- Query the status of a test session being executed.
- Allow the system to cancel a running test session.
To support these features the Dispatcher subcomponent of the TMC implements the following STAF requests which are used by the TMS:
TESTSESSION START <test data path>
This request allows the system to start a test session execution, where <test data path> is the path of the file containing the test session information in XML format, the structure of the file is defined by the TMS's export operation. This request must return a test session handle that can be used to query the status or cancel a test session.
TESTSESSION STATUS <test session handle>
This request allows the system to query the status of a test session execution, where <test session handle> is the handle returned by the TESTSESSION START request when the test session was started.
TESTSESSION END <test session handle>
This request allows the system to cancel a test session execution, where <test session handle> is the handle returned by the TESTSESSION START request when the test session was started.
The framework must be able to build or access the software assets that are being tested, to do so the BEEs in the framework implement the following requests which are used by the Dispatcher subcomponent of the TMC.
BUILD ASSET <AssetType> VERSION <VersionInfo>
This request allows the TMC to ask the BEE to BUILD the asset specified by <AssetType> based on the version information provided in <VersionInfo>. Where <AssetType> is the software asset required by the test session (typical values for this field are kernel, fs, etc); and <VersionInfo> is a string that includes all the information necessary to build the asset specified in <AssetType>, the values of <VersionInfo> depends on the type of BEE for example: for a BEE of type Arago the <AssetType> can be a build string that includes the arago recipe, for a BEE of type file the <AssetType> can be UNC path.
Upon successful completion of this request the BEE must return a STAFResult whose RC equals STAFResult.Ok and whose STAFMapClass contains an "id" element that uniquely identifies the asset just built, and a "path" element that points to the path where the asset built is located. The "id" element of the result allows the framework to implement a cache of software assets that are index by this "id".
GET BUILDID ASSET <AssetType> VERSION <VersionInfo>
This request allows the Dispatcher component of the TMC to query a BEE for the "id" of the software asset specified by <AssetType> which was built/retrieved based on <VersionInfo>.
Upon successful completion of this request the BEE must return a STAFResult whose RC equals STAFResult.Ok and whose STAFMapClass contains an "id" element that can uniquely identify the asset, an "asset" element whose value should be equal to <AssetType>, and a "version" whose value should be equal to <VersionInfo>The "id" element of the result allows the framework to implement a cache of software assets that are index by this "id".
The framework must be able to run the tests on the software assets specified in the build description. To do so the Dispatcher component of the TMC assigns the test to one or more of the TEEs registered in the framework, in order for the Dispatcher to assign the tests to the TEEs, the TEEs in the framework implement the following requests:
RUN <XML test data path>
This request allows the Dispatcher component of the TMC to assign one or more tests to the a TEE, where <XML test data path> if the path where the test XML test data is located. The XML format of the file complies with the TMS export function, it is the responsibility of the TEE to transform this file to the appropiate format that it needs to run the tests.
Upon successful completion of this request the TEE must return a STAFResult whose RC equals STAFResult.Ok and whose STAFMapClass contains a "resultsfile" element that points to the path of an XML file containing the results of the tests executed by the TEE. The structure of the XML file must comply with the TMS import result function.
This request allows the Dispatcher check if the TEE is alive before assigning a test to it.
Upon successful completion of this request the TEE must return a STAFResult whose RC equals STAFResult.Ok.
Sample End-to-End Test flow
The figure below shows the main steps executed during a typical end-to-end test execution.
- User creates a test plan using TestLink web interface if one does not exist already. The test plan is a collection of test cases, so it is assumed that the test cases exists.
- User request execution of a test plan. The test plan contains the list of test cases to run, the Device Under Test to run the tests on, and the software to load on the device under test.
- TestLink exports the test plan data to an XML file.
- TestLink sends a STAF request to the Dispatcher to run the test plan contained in the exported XML file.
- The Dispatcher looks into the test plan data and finds out the type of BEE required, it then asks the resource manager to allocate a BEE of the right type for him.
- The Dispatcher sends a STAF request to the BEE to retrieve or build the software assets that are going to be used in the test.
- The BEE sends a STAF message to the Dispatcher when it finishes building the software asset.
- The Dispatcher looks into the test plan data and finds out the type of TEE required, it then asks the resource manager to allocate a TEE of the right type for him.
- The Dispatcher sends a STAF request to the allocated TEE to run the test.
- The TEE might optionally request services from another Service provider but it must allocate the required service provider type via the Resource Manager.
- The TEE sends a STAF message to the Dispatcher when it completes the test execution.
- The Dispatcher sends a STAF request to the TMS DB writer to save the test results in the TMS.
- The Dispatcher optionally sends an e-mail to notify the test completion.
- The TMS DB Writer service saves results into the TMS.
- Users optionally generate test reports.
- Test Architecture
- Software Assets flow or How software is built and move to the target
- Hardware Assets flow or How the Resource Manager works
- Dispatcher data flow
- Build Execution Engines (BEE) Interface
- Test Execution Engines (TEE) Interface
- Logging Strategy
- TestLink Custom Fields
- TI's TestLink Keywords conventions
- Adding a TEE
- TI's Assets naming conventions