From AragoWiki

Revision as of 13:20, 6 September 2013 by Chernandez (Talk | contribs)
(diff) ← Older revision | Current revision (diff) | Newer revision → (diff)
Jump to: navigation, search


What's linux-devtest

It is a python script to run automated tests on Linux using Opentest framework.

How to install it

Download latest opentest_installer*.tar.gz from untar it, run it and then:

  • If you plan to run tests using TI's board farm
Select options 2 (STAF) and 8 (Command Line Tools)
  • If you plan to run tests on your own desktop
Select options  2 (STAF), 4 (TMC), 5 (VATF), 6 (TEE) , 7 (BEE) and 8 (Command Line Tools) 

NOTE: Depending on your linux distribution, you may also need to install python module 'argparse'.

There are many files that contain setup configuration. The installer will preset most values during the installation process, however the most important configuration parameters are documented here:

  • Test Master Controller (TMC) IP address
 Defines Test Master Controller to send test requests to.
 Location: config/site_info.xml <tmc_machine> xml element
 Default: (which is TI Germantown TMC server address)
  • Local FTP Server URL
 Defines ftp server directory url where local files (i.e. kernel, dtb, etc.) should be copy to.
 This URL should be accessible from TMC defined above.
 Location: config/site_info.xml <local_ftp_server_url> xml element

Configuration parameters applicable to installation case 2) only (running tests on your own desktop):

  • VATF TEE's bench file
 Describes the equipment (DUT and test) available in the VATF Test Execution Engine
 and its properties
 Location: <Opentest root>/bench/
  • NFS export root directory
 Root directory where NFS tar balls (if provided) can be untared and exported to DUT
 Location: VATF bench file

How to use it allows users to load a specified evm type with specified software images and then run a set of tests.
The tool supports 3 different types of test executions as described in the following sections:

Executing ad-hoc commands -s <commands or script>

Use this option to run any arbitrary commands or even a shell script that can reside in your host pc.

Setting EVM type ... -hw omap5-evm,linux

The -hw syntax is evm,capabilities where capabilities is underscore-separated list of strings

Setting SW images ... -p /my/path/MLO -b /my/path/uboot -k /my/path/uImage -d /my/path/dtb -n

All SW images are optional. If you don't specify them, the test script assumes that by default they are available in the MMC card. It is possible to tweak the default behavior by defining arguments using --advance-params option. For more information see advanced-params

Forcing execution on a specific Test Execution Engine (TEE)

linux-devtest users have different options regarding which TEE host is used to run a test.
If you don't know or you are not sure about what a TEE is, please see Opentest Architecture overview
The different options to select a TEE are:

1) Let Opentest use any TEE that is capable of running the test request.
In this case, Opentest will select a capable idle TEE, and if all capable TEEs are busy, it will put the request in the shorter queue.
This is the default behavior when one does not pass -a or -f (more about these is coming below) parameters

2) Use only TEEs that belong to a given farm. The farm are collection of boards dedicated to test automation purposes and located on the same site.
At the moment, we have farms at TI Germantown (called tigt_farm) and at TI Dallas (called tid_farm).
Users may want to choose a farm close to them for cases where software images are large or when user want the board to use a NFS server hosted by the users PC. To request test execution on a given farm use -f option as shown below: ... -f tigt_farm

3) Use a specific TEE. For instance, users who owns TEEs may prefer to run the tests on their TEEs. To force execution on a particular TEE use the -a option. The assignment string pass to -a is "<TEE> on <Machine>", where TEE can be either just the TEE type or the type@id and Machine is the IP address. For example:

./ ... -a "vatf on" 

Executing ad-hoc Test Examples

1) Running arbitrary commands

./ -hw omap5-evm,linux -a "vatf on" --advanced-params var_use_default_env=yes -s "echo 'hello world'; echo 'goodbye world'"

2) Running shell script from host machine

./ -hw omap5-evm,linux -a "vatf on" --advanced-params var_use_default_env=yes -s /home/a0850405/tmp/

Executing LTP-DDT Tests -t <LTP-DDT testplan name>

Use this option to run your own LTP-DDT test plans.

Setting EVM type ... -hw beaglebone

The -hw syntax in this case just identifies the platform name.

Setting SW images

See setting SW images

Creating your own LTP-DDT testplan

1) Update default tests. This will clone ltp-ddt project if required.

./ -U

2) Create your own test plan

cp tests/ tests/

3) Modify tests_to_run list in tests/ to suit your needs by copying entries from tests_available list into it.
Each entry in tests_available lists points to a ltp-ddt test scenario (aka test suite). The syntax is:
<test scenario name>:<setup requirements>:<test cases filter>

Executing LTP-DDT Tests Examples

1) Running mytests testplan using specified kernel, devicetree blob and network filesystem. Execution is forced to tee vatf@2 on machine

./ beaglebone -n -a "vatf@2 on" -k /mnt/gtautoftp/tmp/carlos/nightly-images/ti-linux-3.8-rc5/uImage-beaglebone.bin -d /mnt/gtautoftp/tmp/carlos/nightly-images/ti-linux-3.8-rc5/uImage-beaglebone.dtb -w -t mytests

2) Running default ltp-ddt tests (i.e. no -t | -T | -s options) on am335x-evm

./ -hw am335x-evm -a "vatf@4 on"  -n

Executing Testlink Testplans -T <Testlink testplan>

Use this option to run Testlink testplans.

Setting EVM type ... -hw beaglebone

The -hw syntax in this case just identifies the platform name.

Setting SW images

See setting SW images

Executing Testlink Tests Examples

1) Running Testlink lcpd_functional_sanity_tests that it is part of linux_psp2 project

./ -hw am335x-evm -n -a "vatf@4 on"  -k /mnt/gtautoftp/tmp/carlos/nightly-images/ti-linux-3.8-rc5/uImage-beaglebone.bin -d /mnt/gtautoftp/tmp/carlos/nightly-images/ti-linux-3.8-rc5/uImage-beaglebone.dtb -T linux_psp2:lcpd_functional_sanity_tests

See arguments list for more information about how to get the list of projects and testplans available in Testlink

Detailed arguments list

usage: [-h] [-hw HW] [-p PRI_BOOT] [-b SEC_BOOT] [-k KERNEL]
                       [-m KERNEL_MODULES] [-d DTB] [-r RAMFS] [-n NFS]
                       [-s SCRIPT | -t TESTS | -T TESTPLAN | --list-projects | --list-testplans LIST_TESTPLANS | --list-builds LIST_BUILDS]
                       [-w] [-a ASSIGNED_TO_TEE] [-c]
                       [--advanced-params ADVANCED_PARAMS] [-R REPORT] [-U]
                       [-f FARM]
Run ltp-ddt tests using Opentest
optional arguments:
 -h, --help            show this help message and exit
 -hw HW                DUT type and optional capabilities. i.e.
 -p PRI_BOOT, --pri-boot PRI_BOOT
                       Primary bootloader. Can be local file or http/ftp url
 -b SEC_BOOT, --sec-boot SEC_BOOT
                       Secondary bootloader (i.e. u-boot). Can be local file
                       or http/ftp url
 -k KERNEL, --kernel KERNEL
                       kernel image to load on DUT. Can be local file or
                       http/ftp url
                       kernel modules to install in the FS. Can be local file
                       or http/ftp url
 -d DTB, --dtb DTB     device tree blob to pass to kernel. Can be local file
                       or http/ftp url
 -r RAMFS, --ramfs RAMFS
                       RAMFS image to load on DUT (i.e. rootfs.ext2.gz). Can
                       be local file or http/ftp url
 -n NFS, --nfs NFS     Use NFS. Value can be local file, http/ftp url or nfs
                       url (i.e. server:/path/to/nfs/root). Recent filesystem
                       tar ball is always available at
 -s SCRIPT, --script SCRIPT
                       Shell test script to run in the DUT
 -t TESTS, --tests TESTS
                       File with list of ltp-ddt tests to run
                       Run Testlink named testplan on named testproject
                       optionally using named build.
 --list-projects       Get list of Testlink test projects
 --list-testplans LIST_TESTPLANS
                       Get list of Testlink test plans for named project ID.
                       Value must be valid testlink project ID. Used --list-
                       projects to get valid project IDs
 --list-builds LIST_BUILDS
                       Get list of Testlink test builds for named testplan
                       ID. Value must be valid testlink testplan ID. Used
                       --list-testplans to get valid testplan IDs
 -w, --distribute-workload
                       Allow running test on multiple DUTs in parallel
 -a ASSIGNED_TO_TEE, --assigned-to-tee ASSIGNED_TO_TEE
                       Force tests to run on this Test Execution Engine
 -c, --force-test-scripts-clone
                       Clone latest TEE test scripts
 --advanced-params ADVANCED_PARAMS
                       extra tweaks for advanced users. Separate multiple
                       values with ~
 -R REPORT, --report REPORT
                       Get Testlink testplan results in this format
 -U, --update-default-tests
                       Get latest version of test cases from ltp-ddt
 -f FARM, --farm FARM  Force test to run in a board at this farm (tigt_farm,
                       tid_farm, tii_farm). To run on any farm then just use


  • var_use_default_env=1 means Boot using default environment (i.e. env default -a -f; boot )
  • var_use_default_env=2 means Do not touch uboot env, just power cycle and let it go

how to create new tests (ltp-ddt)

The easier way to create new ltp-ddt tests is to run tests using nfs and then to build/install your local ltp-ddt into this filesystem.
One way to implement this strategy is to follow this approach:
1) Get recent arago-test-image filesystem installed on your PC by running hello world (default) test

./linux-devtest -hw omap5-evm -t default -n 

2) Make sure ltp-ddt sources are up to date (in case previous step did not clone ltp-ddt)

./ -U

3) Build Kernel, modules and headers and install them in the filesystem

cd <your kernel source root>
make ARCH=arm CROSS_COMPILE=arm-arago-linux-gnueabi- uImage modules
sudo make INSTALL_MOD_PATH=<nfs root> modules_install 
sudo make INSTALL_HDR_PATH=<nfs root>/usr headers_install

4) Build and install latest ltp-ddt sources

cd <ltp-ddt root dir>
# Change cross-compiler, kernel include and kernel usr include and platform accordingly in the command below
# KERNEL_INC should point to you kernel sources include directory
# KERNEL_USR_INC should point to the filesystem/usr/include directory
make CROSS_COMPILE=arm-arago-linux-gnueabi- KERNEL_INC=<kernel sources root>/include KERNEL_USR_INC=<nfs root>/usr/include SKIP_IDCHECK=1 PLATFORM=omap5-evm
# Finally install ltp-ddt in the filesystem
sudo make DESTDIR=<nfs root>/opt/myltp SKIP_IDCHECK=1 PLATFORM=omap5-evm install

how to contribute tests back to ltp-ddt

After Setting up your ltp-ddt sources you can create patches against ltp-ddt and submit your contributions to
You may want to check the README-DDT file in the ltp-ddt directory for more information about how to create new tests.

More information

  • Check linux-devtest's README file
  • Check README-DDT file in ltp-ddt directory
  • Ask for help at
Personal tools