OpentestArchitecture

From AragoWiki

Jump to: navigation, search

Back to OpenTest's Home


Contents

Overview

Opentest core infrastructure provides the following components:

  1. A Test Management System
  2. A Test Master Controller
  3. An Execution Engine interface

The top two components above are common across all users of Opentest, regardless of the type of product that they are testing. While the last component, the execution engine interface, is used by Opentesters to implement their own execution engines that suit the specific needs of the products they are testing. This idea of separating things that change from things that don't change is a common theme in software design patterns.


Architecture Change

As of version 16.08 the architecture has changed in order to improve the scalability of the system that was being limited by the TMC running out of system resources.

Unfortunately the changes implemented since version 16.08 do not allow backward compatibility with components released with olders versions of the framework.

This page now include informations for both architectures in order to highlight the difference between the two architectures.

Why the new Architecture

Once Opentest has been installed and the amount of users gradually increase the number of test requests the framework receives will increase accordingly. This together with the implementation of features like allowing each test case in a test session to be allocated individually have increase the number of sockets used per test session request by the Test Master Controller (TMC), as a result, some failures in the TMC has been observed that required some components to change starting with version 16.08. The next to sections describe the communication between the layers in both the old and new architecture, we will use this information to highlight the improvement provided by the new architecture

Events/Messages v16.05 and older

File:Event_graph_v16.05.jpg

In this version when the user clicks to execute a Test session the following communication occur between the layers:

  1. TESTSESSION START: A message along with and xml file containing the test information is sent from the TMS to the DISPATCHER in the TMC
  2. REQUEST BEE: Base on the information contained in the xml the software assets information is parsed and the DISPATCHER sends a REQUEST BEE message to the RESMGR for each BEEs required to fetch the assets
  3. BEE GRANTED?: The DISPATCHER peridically checks if the BEE was granted
  4. BEE: The RESMGR grants the BEE(s) to the Dispatcher
  5. GET BUILDID: Once the BEE is granted the DISPATCHER sends a GET BUILID message for each asset the BEE can fetch
  6. BUILDID REC?: The DISPATCHER periodically checks if the BEE has returned the BUILDID for the asset
  7. BUILDID: The BEE returns the BUILDID for the asset to the DISPATCHER
  8. GET ASSET:Using the BUILDID the DISPATCHER checks if the assets is already in the cache, if not the DISPATCHER requests the BEE to fetch the software asset.
  9. ASSET RECEIVED?: The DISPATCHER periodically checks if the BEE has fetched the asset
  10. ASSET: Once the BEE has fetched the asset ii provides it to the DISPATCHER
  11. RELEASE BEE: The DISPATCHER tells the RESMGR the BEE is free to service more requests
  12. REQUEST TEE: The DISPATCHER requests each TEE required to run the tests defined in the XML file
  13. TEE GRANTED?: The DISPATCHER periodically checks to see if a TEE was granted
  14. TEE: Once the TEE is idle, the RESMGR grants the TEE to the DISPATCHER
  15. RUN TEST: The DISPATCHER requests the TEE to run the tests
  16. COPY ASSET: The DISPATCHER transfer the assets to be tested to the TEE
  17. TEST ENDED?: The Dispatcher periodically checks if the TEE has finished running the tests
  18. TEST RESULTS: The TEE has finished running the tests and provides the XML result file to the dispatcher
  19. RELEASE TEE: The DISPATCHER tells the RESMGR the TEE is free to service more requests
  20. SAVE RESULTS: Based on the information in the XML results file the DISPATCHER send a message to the TLWRITER to update the results in the TMS
  21. SAVE RESULT: The TLWriter updates the results in the TMS

Events/Messages v16.08 and Newer

File:Event_graph_v16.08.jpg

In this version the RESMGR and DISPATCHER have been combined into a single service and the TLWriter has been moved to the TEEs and write the results to the TMS directly from the TEE. When the user clicks to execute a Test session the following communication occur between the layers:

  1. TESTSESSION START: A message along with and xml file containing the test information is sent from the TMS to the DISPATCHER in the TMC
  2. GET BUILDID: Based on the information contained in the xml the software assets information is parsed, the DISPATCHER finds an IDLE BEE and sends a GET BUILID message for each asset the BEE can fetch
  3. BUILDID REC?: The DISPATCHER periodically checks if the BEE has returned the BUILDID for the asset
  4. BUILDID: The BEE returns the BUILDID for the asset to the DISPATCHER and the DISPATCHER releases the BEE
  5. RUN TEST w BUILDID: The DISPATCHER finds IDLE TEE(s) and requests each TEE to run the tests based on an XML test cases file and the BUILDID information it has obtained
  6. REQUEST BEE: Base on the information contained in the xml the software assets information is parsed and the TEE sends a REQUEST BEE message to the DISPATCHER for each BEEs required to fetch the assets
  7. BEE GRANTED?: The DISPATCHER peridically checks if the BEE was granted
  8. BEE: The DISPATCHER grants the BEE(s) to the TEE
  9. GET ASSET:Using the BUILDID the TEE checks if the assets is already in the cache, if not the TEE requests the BEE to fetch the software asset.
  10. ASSET RECEIVED?: The TEE periodically checks if the BEE has fetched the asset
  11. ASSET: Once the BEE has fetched the asset it provides it to the TEE and the TEE starts running the tests
  12. RELEASE BEE: The TEE notifies the DISPATCHER the BEE is available to service other requests
  13. WRITE RESULTS: After The TEE has finished running it send a message to the TLWRITER to update the results
  14. SAVE RESULTS: The TLWRITER saves the results in the TMS
  15. TEST DONE: The TEE notifies the DISPATCHER it has finished running the test and is available to service other requests

New Architecture Improvements

When comparing the previous section with the steps describe here it can be seen that by combining the RESMGR and DISPATCHER all resource allocation required by the DISPATCHER is handle internally and does not require the use of a socket, also, moving the TLWriter together with the TEE and updating the results means that the DISPATCHER does not have to use sockets to update results and problems in DISPATCHER-TEE communication does not mean the results are lost anymore as long as the TEE is able to communicate with the TMS. All these improvements provided by the new architecture translate into a more scalable and robust framework

New Architecture (v16.08 and newer)

System Layers

The Opentest architecture has 3 layers:

  1. TMS: A Test Management System that stores all the artifacts related to testing such as test cases, test plans, test results, etc. The current Opentest implementation uses Testlink as the Test Management System.
  2. TMC: A Test Master Controller which functions as the brain of the system and takes care of dispatching test request to the appropriate Service Provider.
  3. Service Providers: Provides critical services required to execute a test.

The TMC has the following sub-components:

  • Dispatcher: Receives all the test job requests from the TMS and keeps track of all the services providers available in the system, their types and capabilities and selects the best candidate for a given request.

There are two main types of services providers in a typical test run:

  • BEEs: Build Execution Engines. Build or retrieve the software that is going to be tested.
  • TEEs: Test Execution Engines. Run the steps to test the software provided by the BEE and uses a TMS Writer to save test results into TMS. The TEEs are the entities that carry most of the actual test job. There are different types of TEEs suited to carry different types of tests. Each test case identifies the TEE type required to run the test. Of course, the same TEE type could be used to run many different test cases.

A typical test case flow run by a TEE includes the following steps:

  1. Power-cycling the Device Under Test (DUT) and loading the software to be tested on it.
  2. Booting the DUT.
  3. Configuring the DUT.
  4. Configuring any test equipment that might be part of the test.
  5. Connecting the DUT to the test equipment.
  6. Executing one or more validation steps involving the DUT and/or the test equipment.
  7. Determining the test outcome (Pass or Fail)
  8. Creating results file and logs.
  9. Saving results in the TMS

The components on each layer run on machines running STAF. In other words, every machine that is part of Opentest must have STAF installed on it and must be running the STAF daemon process called STAFProc. The components in the system communicate by sending STAF request and exchanging XML files.

The following figure shows the main components described above.

File:MainComponents_v16.08.jpg

Test Plan data

As shown in the previous section, the Opentest system is very flexible as it allows running tests using different Build and Test Execution Engines. But as it is typically the case, flexibility also brings complexity, so there must be a place in the system to configure or decide which execution engine type to use, which device under test type to use and other parameters, as you can guess that place is the Test Plan that it is exported from the Test Management System (Testlink).

The Test Plan must contain at least the following information:

  1. List of test cases to run
  2. Hardware Under Test or Device Under Test (DUT) type.
  3. Software Under Test and BEE type required to build it.
  4. Each Test Case in the test plan identifies:
    1. The TEE type
    2. TEE Logic or script to load and run on the TEE identified above
    3. Test Parameters
    4. Special DUT requirements and test equipment. For instance, the test case might specify that the DUT must have USB ISO capabilities or some specific test equipment is required to run the test.

Interfaces

All communication between the components in the system is done through STAF, therefore, all interfaces for communication between components are defined as a set of STAF requests. The following sections describe the interfaces defined for the OpenTest components.

TMS-TMC

The interaction between TMS and TMC should provide the means to support the following functionality:

  • Start execution of a test session.
  • Query the status of a test session being executed.
  • Allow the system to cancel a running test session.

To support these features the Dispatcher subcomponent of the TMC implements the following STAF requests which are used by the TMS:

TESTSESSION START <test data path> [SEPREQS] [TIMEOUT <Number><s|m|h|d|w>]

This request allows the system to start a test session execution, where:

<test data path> is the path of the file containing the test session information in XML format, the structure of the file is defined by the TMS's export operation
SEPREQS option used to signal that each test case in the XML file should be handle as a separtate test request to a TEE
TIMEOUT option used to specify how long the DISPATCHER should wait for the tests to be allocated before cancelling a request takes <number> and s for seconds, m for minutes, h for hours, d for days and w for weeks. 

This request must return a test session handle that can be used to query the status or cancel a test session.

TESTSESSION STATUS <test session handle>

This request allows the system to query the status of a test session execution, where <test session handle> is the handle returned by the TESTSESSION START request when the test session was started.

TESTSESSION END <test session handle>

This request allows the system to cancel a test session execution, where <test session handle> is the handle returned by the TESTSESSION START request when the test session was started.

TMC-BEE

The framework must be able to build or access the software assets that are being tested, to do so the BEEs in the framework implement the following requests which are used by the Dispatcher subcomponent of the TMC.

Upon successful completion of this request the BEE must return a STAFResult whose RC equals STAFResult.Ok and whose STAFMapClass contains an "id" element that uniquely identifies the asset just built, and a "path" element that points to the path where the asset built is located. The "id" element of the result allows the framework to implement a cache of software assets that are index by this "id".

GET BUILDID ASSET <AssetType> VERSION <VersionInfo>

This request allows the Dispatcher component of the TMC to query a BEE for the "id" of the software asset specified by <AssetType> which was built/retrieved based on <VersionInfo>.

Upon successful completion of this request the BEE must return a STAFResult whose RC equals STAFResult.Ok and whose STAFMapClass contains an "id" element that can uniquely identify the asset, an "asset" element whose value should be equal to <AssetType>, and a "version" whose value should be equal to <VersionInfo>The "id" element of the result allows the framework to implement a cache of software assets that are index by this "id".

TMC-TEE

The framework must be able to run the tests on the software assets specified in the build description. To do so the Dispatcher component of the TMC assigns the test to one or more of the TEEs registered in the framework, in order for the Dispatcher to assign the tests to the TEEs:

The TEEs in the framework implement the following requests:
RUN <XML test data path> SHANDLE <session handle> REQID <id> [MD5INFO <assetname>=<md5sum/buildid> ... <assetname>=<md5sum/buildid>]

This request allows the Dispatcher component of the TMC to assign one or more tests to the a TEE, where:

 <XML test data path> is the path where the test XML test data is located. The XML format of the file complies with the TMS export function, it is the responsibility of the TEE to transform this file to the appropiate format that it needs to run the tests.
 <session handle> is a handle provided by the requester to associate this test request with a test session
 <id> is a unique id associated with this request. This value together with the SHANDLE uniquely identify a test request within a test session
 <assetname=buildid> Optional parameters containing the buildid information of one or more asset. The syntax of the values in this field are <asset name>=<buildid info> 

Upon successful completion of this request the TEE must return a STAFResult whose RC equals STAFResult.Ok if the request was succesful or an error code if there were any problems processing the request

PING

This request allows the Dispatcher check if the TEE is alive before assigning a test to it.

Upon successful completion of this request the TEE must return a STAFResult whose RC equals STAFResult.Ok.

The TMC implements the following request:
TESTDONE SHANDLE <session handle> REQID <id> RC <return code> RESULT <result text>

This request is available to the TEEs to notify the Dispatcher that is has finished processing the requests and it is available to service a new request, where:

 <session handle> is the handle used by the Dispatcher during the RUN request issued to the TEE
 <id> is the id value used by the dispatcher during the RUN request. This value together with the SHANDLE uniquely identify a test request within a test session
 <return code> is the STAF rc value representing the result of the staf calls made while running the tests. STAFResult.Ok means that all STAF requests were succesful
 <result text> additional information associated with the value of <return code>

Old Architecture (v16.05 and older)

System Layers

The Opentest architecture has 3 layers:

  1. TMS: A Test Management System that stores all the artifacts related to testing such as test cases, test plans, test results, etc. The current Opentest implementation uses Testlink as the Test Management System.
  2. TMC: A Test Master Controller which functions as the brain of the system and takes care of dispatching test request to the appropriate Service Provider.
  3. Service Providers: Provides critical services required to execute a test.

The TMC has the following sub-components:

  • Dispatcher: Receives all the test job requests from the TMS and interacts with the Resource Manager to find the appropriate service provider to execute the next step.
  • Resource Manager: Keeps track of all the services providers available in the system, their types and capabilities and select the best candidate for a given request.
  • Monitor: Provides information about service providers status.
  • TMS Writer: Save test results into TMS.

There are two main types of services providers in a typical test run:

  • BEEs: Build Execution Engines. Build or retrieve the software that is going to be tested.
  • TEEs: Test Execution Engines. Run the steps to test the software provided by the BEE. The TEEs are the entities that carry most of the actual test job. There are different types of TEEs suited to carry different types of tests. Each test case identifies the TEE type required to run the test. Of course, the same TEE type could be used to run many different test cases.

A typical test case flow run by a TEE includes the following steps:

  1. Power-cycling the Device Under Test (DUT) and loading the software to be tested on it.
  2. Booting the DUT.
  3. Configuring the DUT.
  4. Configuring any test equipment that might be part of the test.
  5. Connecting the DUT to the test equipment.
  6. Executing one or more validation steps involving the DUT and/or the test equipment.
  7. Determining the test outcome (Pass or Fail)
  8. Creating results file and logs.

The components on each layer run on machines running STAF. In other words, every machine that is part of Opentest must have STAF installed on it and must be running the STAF daemon process called STAFProc. The components in the system communicate by sending STAF request and exchanging XML files.

The following figure shows the main components described above.

File:MainComponents.jpg

Test Plan data

As shown in the previous section, the Opentest system is very flexible as it allows running tests using different Build and Test Execution Engines. But as it is typically the case, flexibility also brings complexity, so there must be a place in the system to configure or decide which execution engine type to use, which device under test type to use and other parameters, as you can guess that place is the Test Plan that it is exported from the Test Management System (Testlink).

The Test Plan must contain at least the following information:

  1. List of test cases to run
  2. Hardware Under Test or Device Under Test (DUT) type.
  3. Software Under Test and BEE type required to build it.
  4. Each Test Case in the test plan identifies:
    1. The TEE type
    2. TEE Logic or script to load and run on the TEE identified above
    3. Test Parameters
    4. Special DUT requirements and test equipment. For instance, the test case might specify that the DUT must have USB ISO capabilities or some specific test equipment is required to run the test.

Interfaces

All communication between the components in the system is done through STAF, therefore, all interfaces for communication between components are defined as a set of STAF requests. The following sections describe the interfaces defined for the OpenTest components.

TMS-TMC

The interaction between TMS and TMC should provide the means to support the following functionality:

  • Start execution of a test session.
  • Query the status of a test session being executed.
  • Allow the system to cancel a running test session.

To support these features the Dispatcher subcomponent of the TMC implements the following STAF requests which are used by the TMS:

TESTSESSION START <test data path>

This request allows the system to start a test session execution, where <test data path> is the path of the file containing the test session information in XML format, the structure of the file is defined by the TMS's export operation. This request must return a test session handle that can be used to query the status or cancel a test session.

TESTSESSION STATUS <test session handle>

This request allows the system to query the status of a test session execution, where <test session handle> is the handle returned by the TESTSESSION START request when the test session was started.

TESTSESSION END <test session handle>

This request allows the system to cancel a test session execution, where <test session handle> is the handle returned by the TESTSESSION START request when the test session was started.

TMC-BEE

The framework must be able to build or access the software assets that are being tested, to do so the BEEs in the framework implement the following requests which are used by the Dispatcher subcomponent of the TMC.

BUILD ASSET <AssetType> VERSION <VersionInfo>

This request allows the TMC to ask the BEE to BUILD the asset specified by <AssetType> based on the version information provided in <VersionInfo>. Where <AssetType> is the software asset required by the test session (typical values for this field are kernel, fs, etc); and <VersionInfo> is a string that includes all the information necessary to build the asset specified in <AssetType>, the values of <VersionInfo> depends on the type of BEE for example: for a BEE of type Arago the <AssetType> can be a build string that includes the arago recipe, for a BEE of type file the <AssetType> can be UNC path.

Upon successful completion of this request the BEE must return a STAFResult whose RC equals STAFResult.Ok and whose STAFMapClass contains an "id" element that uniquely identifies the asset just built, and a "path" element that points to the path where the asset built is located. The "id" element of the result allows the framework to implement a cache of software assets that are index by this "id".

GET BUILDID ASSET <AssetType> VERSION <VersionInfo>

This request allows the Dispatcher component of the TMC to query a BEE for the "id" of the software asset specified by <AssetType> which was built/retrieved based on <VersionInfo>.

Upon successful completion of this request the BEE must return a STAFResult whose RC equals STAFResult.Ok and whose STAFMapClass contains an "id" element that can uniquely identify the asset, an "asset" element whose value should be equal to <AssetType>, and a "version" whose value should be equal to <VersionInfo>The "id" element of the result allows the framework to implement a cache of software assets that are index by this "id".

TMC-TEE

The framework must be able to run the tests on the software assets specified in the build description. To do so the Dispatcher component of the TMC assigns the test to one or more of the TEEs registered in the framework, in order for the Dispatcher to assign the tests to the TEEs, the TEEs in the framework implement the following requests:

RUN <XML test data path>

This request allows the Dispatcher component of the TMC to assign one or more tests to the a TEE, where <XML test data path> is the path where the test XML test data is located. The XML format of the file complies with the TMS export function, it is the responsibility of the TEE to transform this file to the appropiate format that it needs to run the tests.

Upon successful completion of this request the TEE must return a STAFResult whose RC equals STAFResult.Ok and whose STAFMapClass contains a "resultsfile" element that points to the path of an XML file containing the results of the tests executed by the TEE. The structure of the XML file must comply with the TMS import result function.

PING

This request allows the Dispatcher check if the TEE is alive before assigning a test to it.

Upon successful completion of this request the TEE must return a STAFResult whose RC equals STAFResult.Ok.

Sample End-to-End Test flow

The figure below shows the main steps executed during a typical end-to-end test execution.

File:EndToEndFLow.jpg

Steps Description:

  1. User creates a test plan using TestLink web interface if one does not exist already. The test plan is a collection of test cases, so it is assumed that the test cases exists.
  2. User request execution of a test plan. The test plan contains the list of test cases to run, the Device Under Test to run the tests on, and the software to load on the device under test.
  3. TestLink exports the test plan data to an XML file.
  4. TestLink sends a STAF request to the Dispatcher to run the test plan contained in the exported XML file.
  5. The Dispatcher looks into the test plan data and finds out the type of BEE required, it then asks the resource manager to allocate a BEE of the right type for him.
  6. The Dispatcher sends a STAF request to the BEE to retrieve or build the software assets that are going to be used in the test.
  7. The BEE sends a STAF message to the Dispatcher when it finishes building the software asset.
  8. The Dispatcher looks into the test plan data and finds out the type of TEE required, it then asks the resource manager to allocate a TEE of the right type for him.
  9. The Dispatcher sends a STAF request to the allocated TEE to run the test.
  10. The TEE might optionally request services from another Service provider but it must allocate the required service provider type via the Resource Manager.
  11. The TEE sends a STAF message to the Dispatcher when it completes the test execution.
  12. The Dispatcher sends a STAF request to the TMS DB writer to save the test results in the TMS.
  13. The Dispatcher optionally sends an e-mail to notify the test completion.
  14. The TMS DB Writer service saves results into the TMS.
  15. Users optionally generate test reports.

Deep Dive

  1. Test Architecture*
  2. Software Assets flow or How software is built and move to the target*
  3. Hardware Assets flow or How the Resource Manager works*
  4. Dispatcher data flow*
  5. Build Execution Engines (BEE) Interface*
  6. Test Execution Engines (TEE) Interface*
  7. Logging Strategy*
  8. TestLink Custom Fields
  9. TI's TestLink Keywords conventions
  10. Adding a TEE*
  11. TI's Assets naming conventions

* Some of the information in this sections has been deprecated since Opentest version 16.08

Personal tools