OpentestHowTo

From AragoWiki

(Difference between revisions)
Jump to: navigation, search
(How-To Start STAF)
 
(19 intermediate revisions not shown)
Line 1: Line 1:
= How-To Install Opentest Client tools =
= How-To Install Opentest Client tools =
* Find a Ubuntu system (10.04 or later should be fine)
* Find a Ubuntu system (10.04 or later should be fine)
-
* Download and untar latest opentest_installer*.tar.gz from http://arago-project.org/files/releases/opentest/  
+
* Download and untar latest opentest_installer*.tar.gz (recommended 13.09 or later) from http://arago-project.org/files/releases/opentest/  
-
* Run the installer (e.g. ./install_opentest.sh) and Select options 2 (STAF) and 8 (Command Line Tools)
+
* Run the installer (e.g. ./install_opentest.sh) and select '''option 8''' (Command Line Tools).  This option will also install the STAF utility.  For installation details see the list below:
** The installer will install Java if required
** The installer will install Java if required
** Then it will install STAF (you can leave all the default install values)
** Then it will install STAF (you can leave all the default install values)
-
** Finally it will install opentest_client components. You can leave default values for most options, however subnets must be entered. If you are in the US we recommend entering at least following 3 subnets: 158.218.*.* 10.218.*.* 128.247.*.*
+
** Then it will install opentest_client components. You can leave default values for most options, however subnets must be entered. If you are in the US we recommend entering at least following 3 subnets (separated by spaces): 158.218.*.* 10.218.*.* 128.247.*.*
 +
** When prompted for the '''Tools installation menu''' select option 1 to ''Install All'' tools.
 +
** When prompted where to install the files you can enter '''.''' to install in the current directory.  You should see this directory in the next prompt.
 +
* Finally you can select '''option 9''' to quit.
= How-To Start STAF =
= How-To Start STAF =
Line 25: Line 28:
  fi
  fi
  export PATH LD_LIBRARY_PATH CLASSPATH STAFCONVDIR STAF_INSTANCE_NAME
  export PATH LD_LIBRARY_PATH CLASSPATH STAFCONVDIR STAF_INSTANCE_NAME
 +
* Source the /usr/local/staf/STAFEnv.sh script to properly initialize your environment for STAF (assuming that STAF was installed in the default location).
 +
*: '''. /usr/local/staf/STAFEnv.sh'''
* Finally run /usr/local/staf/startSTAFProc.sh to start STAF. Cat nohup.out file to check correct initialization
* Finally run /usr/local/staf/startSTAFProc.sh to start STAF. Cat nohup.out file to check correct initialization
  $ ./startSTAFProc.sh  
  $ ./startSTAFProc.sh  
Line 32: Line 37:
  Machine nickname : charliebrown
  Machine nickname : charliebrown
  Startup time    : 20130905-18:06:02
  Startup time    : 20130905-18:06:02
 +
* You should now be able to run the ''staf local help'' command and verify your STAF installation.
 +
 +
= How-To easily Run a boot test on multiple boards =
 +
* Make sure you have [[#How-To_Install_Opentest_Client_tools | already install Opentest Client tools]]
 +
* Also verify that [[#How-To_Start_STAF | STAF is running ]]
 +
* Go to the location where you installed Opentest client and modify :kernel, :dtb and :modules files path in CI_sample_boottest.rb file
 +
* Run following command
 +
./CI.rb -m CI_sample_boottest.rb --machine-adapter-type none --input-adapter-type none --build-name chtest --do-not-publish-results
 +
:{{SpImportant|--build-name should not have spaces. You probably want to use a unique ID (e.g. your employee ID). One can control the set of boards to test by removing entries from @machines list at the bottom of CI_sample_boottest.rb}}
 +
= How-To Run your own tests on the boards Farm =
= How-To Run your own tests on the boards Farm =
* Make sure you have [[#How-To_Install_Opentest_Client_tools | already install Opentest Client tools]]
* Make sure you have [[#How-To_Install_Opentest_Client_tools | already install Opentest Client tools]]
* Also verify that [[#How-To_Start_STAF | STAF is running ]]
* Also verify that [[#How-To_Start_STAF | STAF is running ]]
-
* Run linux-devtest.py -s from the location where you installed Opentest client tools
+
* Run linux-devtest.py -s from the location where you installed Opentest client tools.  You will find this script in the '''client''' directory.
   # Simple test to boot and echo 'hello world' upon booting.
   # Simple test to boot and echo 'hello world' upon booting.
-
  ./linux-devtest.py '''-s "echo 'hello world'"''' ...
+
  ./linux-devtest.py '''-s "echo 'hello world'"''' <options>
 +
:{{SpImportant|The command above will not run by default without some options being passed. At a minimum you should specify the board farm to use, but likely you will also add your kernel image to test}}
* Please check [[#linux-devtest.py_arguments_help | linux-devtest.py help]] for more details
* Please check [[#linux-devtest.py_arguments_help | linux-devtest.py help]] for more details
-
 
+
= How-To Run LTP-DDT tests on the boards Farm =
-
= How-To Run ltp-ddt tests on the boards Farm =
+
* Make sure you have [[#How-To_Install_Opentest_Client_tools | already install Opentest Client tools]]
* Make sure you have [[#How-To_Install_Opentest_Client_tools | already install Opentest Client tools]]
* Also verify that [[#How-To_Start_STAF | STAF is running ]]
* Also verify that [[#How-To_Start_STAF | STAF is running ]]
* Run linux-devtest.py -t from the location where you installed Opentest client tools
* Run linux-devtest.py -t from the location where you installed Opentest client tools
-
  ./linux-devtest.py '''-t <LTP-DDT testplan name>''' ...
+
  ./linux-devtest.py '''-t <LTP-DDT testplan name>''' <options>
 +
:{{SpImportant|The command above will not run by default without some options being passed. At a minimum you should specify the board farm to use, but likely you will also add your kernel image to test}}
* Please check [[#linux-devtest.py_arguments_help | linux-devtest.py help]] for more details
* Please check [[#linux-devtest.py_arguments_help | linux-devtest.py help]] for more details
 +
 +
== Creating your own LTP-DDT testplan ==
 +
* Update default tests. This will clone ltp-ddt project if required.
 +
./linux-devtest.py -U
 +
* Create your own test plan
 +
cp tests/default.py tests/mytests.py
 +
* Modify tests_to_run list in tests/mytests.py to suit your needs by copying entries from tests_available list into it. <br>
 +
Each entry in tests_available lists points to a ltp-ddt test scenario (aka test suite). The syntax is: <br>
 +
<test scenario name>:<setup requirements>:<test cases filter>
 +
 +
== How to create new LTP-DDT tests ==
 +
:{{SpImportant| This step is more involved. It requires cloning/modifying/building/installing ltp-ddt in the rootfs }}
 +
* Export NFS filesystem in your host. <br>
 +
LTP-DDT expects certain test binaries to be installed in the filesystem. If you use a tisdk filesystem then all the required test tools have been already installed in the filesystem. One can get a recent tisdk rootfs image from http://lcpd.dal.design.ti.com/core-sdk/nightly/deploy/images/ <br>
 +
Select appropriate tisdk-rootfs-image-<MACHINE>*.tar.gz
 +
 +
* Make sure ltp-ddt sources are up to date. This step will clone latest ltp-ddt project into ltp-ddt directory
 +
./linux-devtest.py -U
 +
 +
* Build Kernel, modules and headers and install them in the filesystem
 +
cd <your kernel source root>
 +
make ARCH=arm CROSS_COMPILE=... zImage modules
 +
sudo make INSTALL_MOD_PATH=<nfs root> modules_install
 +
sudo make INSTALL_HDR_PATH=<nfs root>/usr headers_install
 +
:{{SpImportant| Make sure you use recommended toolchain. See [[Setting_Up_Build_Environment#Cross-compile_toolchain | toolchain installation instructions]] }}
 +
 +
* Build and install latest LTP-DDT sources
 +
cd <ltp-ddt root dir>.  # This is the directory created by Step 2 above
 +
# Change cross-compiler, kernel include and kernel usr include and platform accordingly in the command below
 +
# KERNEL_INC should point to you kernel sources include directory
 +
# KERNEL_USR_INC should point to the filesystem/usr/include directory
 +
make CROSS_COMPILE=arm-linux-gnueabihf- KERNEL_INC=<kernel sources root>/include KERNEL_USR_INC=<nfs root>/usr/include SKIP_IDCHECK=1 PLATFORM=omap5-evm
 +
# Finally install ltp-ddt in the filesystem
 +
sudo make DESTDIR=<nfs root>/opt/myltp SKIP_IDCHECK=1 PLATFORM=omap5-evm install
 +
<br> <br>
 +
 +
== how to contribute tests back to ltp-ddt ==
 +
Please submit your LTP-DDT patches to opentest@arago-project.org. <br>
 +
You may want to check the README-DDT file in the ltp-ddt directory for more information about how to create new tests.
 +
<br>
= How-To Run Testlink tests plans on the boards Farm =
= How-To Run Testlink tests plans on the boards Farm =
 +
* Use this option to run a Testlink test plan. If you use this option results will be stored in the Testlink server.
* Make sure you have [[#How-To_Install_Opentest_Client_tools | already install Opentest Client tools]]
* Make sure you have [[#How-To_Install_Opentest_Client_tools | already install Opentest Client tools]]
* Also verify that [[#How-To_Start_STAF | STAF is running ]]
* Also verify that [[#How-To_Start_STAF | STAF is running ]]
* Run linux-devtest.py -T from the location where you installed Opentest client tools
* Run linux-devtest.py -T from the location where you installed Opentest client tools
-
  ./linux-devtest.py '''-T <Testlink testplan>''' ...
+
  ./linux-devtest.py '''-T <Testlink testplan>''' <options>
 +
:{{SpImportant|The command above will not run by default without some options being passed. At a minimum you should specify the board farm to use, but likely you will also add your kernel image to test}}
 +
 
 +
The '''Testlink testplan''' name syntax is: ''<project name>:<testplan name>''
 +
 
 +
To get the list of available projects, use following command:
 +
./linux-devtest.py --list-projects
 +
 
 +
To get the list of available testplans on a project, use ''--list-testplans <project ID>'' where project ID can be obtained from ''--list-projects'' command. For example:
 +
./linux-devtest.py --list-testplans 1140921
 +
'''The above command will return the list of testplans for LCPD linux. Project linux_psp2(id=1140921) hosts LCPD's linux test plans.''' <br>
 +
 
 +
Please note that ''--list-testplans'' uses project '''ID''', while ''linux-devtest.py -T'' uses project and testplans '''names'''. <br>
 +
 
* Please check [[#linux-devtest.py_arguments_help | linux-devtest.py help]] for more details
* Please check [[#linux-devtest.py_arguments_help | linux-devtest.py help]] for more details
Line 80: Line 150:
== -u, --user-bins argument ==
== -u, --user-bins argument ==
-
Executable or tarball of executables that should be installed on FS
+
Executable or tarball of executables that should be installed on FS.
This is useful for cases where user wants to run a binary that is not available in the default filesystem
This is useful for cases where user wants to run a binary that is not available in the default filesystem
The value provided can be either a local file or a public http/ftp url
The value provided can be either a local file or a public http/ftp url
Line 107: Line 177:
== -T, --testplan argument ==
== -T, --testplan argument ==
-
Use this option to run a Testlink test plan. If you use this option results will be stored in the Testlink server.
+
* Use this option to run a Testlink test plan. If you use this option results will be stored in the Testlink server.
-
The Testlink Testplan name syntax is: <project name>:<testplan name>
+
The Testlink Testplan name syntax is: <project name>:<testplan name>. For example:
  ./linux-devtest.py -T linux_psp2:ch_sandbox
  ./linux-devtest.py -T linux_psp2:ch_sandbox
-
This command will return the list of projects
+
* To get the list of available projects, use following command:
  ./linux-devtest.py --list-projects
  ./linux-devtest.py --list-projects
-
and this one will return the testplans available for linux_psp2 project. linux_psp2 is the project name for most Linux-related work
+
* To get the list of available testplans on a project, use ''--list-testplans <project ID>'' where project ID can be obtained from ''--list-projects'' command. For example:
  ./linux-devtest.py --list-testplans 1140921
  ./linux-devtest.py --list-testplans 1140921
-
+
'''The above command will return the list of testplans for LCPD linux. linux_psp2 is the project name that host LCPD's linux test plans.''' <br>
 +
 
 +
Please note that ''--list-testplans'' uses project '''ID''', while ''linux-devtest.py -T'' uses project and testplans '''names'''.
 +
 
== -f, --farm argument ==
== -f, --farm argument ==
forces usage of a board in a farm.
forces usage of a board in a farm.
-
 
+
-f tigt_farm
-
== -c, --force-test-scripts-clone argument ==
+
-f tid_farm
-
Makes sure that latest version of Opentest's scripts are used by forcing a pull of test scripts project at the Test Execution Engine.
+
== --advanced-params ==
== --advanced-params ==
advanced-params are variables that advanced users can set to tweak execution.
advanced-params are variables that advanced users can set to tweak execution.
At the moment the only advanced-params supported by linux-devtest.py is ''var_use_default_env''.
At the moment the only advanced-params supported by linux-devtest.py is ''var_use_default_env''.
-
Usually linux-devtest.py will try to boot the DUT using provided software images (i.e. kernel, filesystem, etc.), but adavanced users may change that default behavior.
+
Usually linux-devtest.py will try to boot the DUT using provided software images (i.e. kernel, filesystem, etc.), but advanced users may change that default behavior.
* var_use_default_env=1 means Boot using default environment (i.e. env default -a -f; boot )
* var_use_default_env=1 means Boot using default environment (i.e. env default -a -f; boot )
-
* var_use_default_env=2 means Do not touch uboot env, just power cycle and let it go  
+
* var_use_default_env=2 means Do not touch uboot env, just power cycle and let it go
= Errors =
= Errors =
* If you get ''ImportError: No module named argparse'' while running linux-devtest.py then you need to install python-argparse
* If you get ''ImportError: No module named argparse'' while running linux-devtest.py then you need to install python-argparse
  $ sudo apt-get install python-argparse
  $ sudo apt-get install python-argparse

Current revision as of 21:24, 17 October 2013

Contents

How-To Install Opentest Client tools

  • Find a Ubuntu system (10.04 or later should be fine)
  • Download and untar latest opentest_installer*.tar.gz (recommended 13.09 or later) from http://arago-project.org/files/releases/opentest/
  • Run the installer (e.g. ./install_opentest.sh) and select option 8 (Command Line Tools). This option will also install the STAF utility. For installation details see the list below:
    • The installer will install Java if required
    • Then it will install STAF (you can leave all the default install values)
    • Then it will install opentest_client components. You can leave default values for most options, however subnets must be entered. If you are in the US we recommend entering at least following 3 subnets (separated by spaces): 158.218.*.* 10.218.*.* 128.247.*.*
    • When prompted for the Tools installation menu select option 1 to Install All tools.
    • When prompted where to install the files you can enter . to install in the current directory. You should see this directory in the next prompt.
  • Finally you can select option 9 to quit.

How-To Start STAF

  • Check if STAF is running
$ staf local help
Usage: STAF [-verbose] <Endpoint | LOCAL> <Service> <Request>
  • If you don't get an error while running staf local help command, then STAF is running and you are good to go. However if you do get an error then you need to open /usr/local/staf/STAFEnv.sh and verify that STAF_INSTANCE_NAME is set to a value as shown below
if [ $# = 0 ]
then
   STAF_INSTANCE_NAME=STAF
else
   if [ $1 != "start" ]
   then
       STAF_INSTANCE_NAME=$1
   else
       # Ignore "start" STAF instance name
       STAF_INSTANCE_NAME=STAF
   fi
fi
export PATH LD_LIBRARY_PATH CLASSPATH STAFCONVDIR STAF_INSTANCE_NAME
  • Source the /usr/local/staf/STAFEnv.sh script to properly initialize your environment for STAF (assuming that STAF was installed in the default location).
    . /usr/local/staf/STAFEnv.sh
  • Finally run /usr/local/staf/startSTAFProc.sh to start STAF. Cat nohup.out file to check correct initialization
$ ./startSTAFProc.sh 
$ nohup: appending output to `/home/charliebrown/nohup.out'
$ cat /home/charliebrown/nohup.out
Machine          : charliebrown
Machine nickname : charliebrown
Startup time     : 20130905-18:06:02
  • You should now be able to run the staf local help command and verify your STAF installation.

How-To easily Run a boot test on multiple boards

./CI.rb -m CI_sample_boottest.rb --machine-adapter-type none --input-adapter-type none --build-name chtest --do-not-publish-results
IMPORTANT

--build-name should not have spaces. You probably want to use a unique ID (e.g. your employee ID). One can control the set of boards to test by removing entries from @machines list at the bottom of CI_sample_boottest.rb


How-To Run your own tests on the boards Farm

 # Simple test to boot and echo 'hello world' upon booting.
./linux-devtest.py -s "echo 'hello world'" <options>
IMPORTANT

The command above will not run by default without some options being passed. At a minimum you should specify the board farm to use, but likely you will also add your kernel image to test

How-To Run LTP-DDT tests on the boards Farm

./linux-devtest.py -t <LTP-DDT testplan name> <options>
IMPORTANT

The command above will not run by default without some options being passed. At a minimum you should specify the board farm to use, but likely you will also add your kernel image to test

Creating your own LTP-DDT testplan

  • Update default tests. This will clone ltp-ddt project if required.
./linux-devtest.py -U
  • Create your own test plan
cp tests/default.py tests/mytests.py
  • Modify tests_to_run list in tests/mytests.py to suit your needs by copying entries from tests_available list into it.

Each entry in tests_available lists points to a ltp-ddt test scenario (aka test suite). The syntax is:
<test scenario name>:<setup requirements>:<test cases filter>

How to create new LTP-DDT tests

IMPORTANT

This step is more involved. It requires cloning/modifying/building/installing ltp-ddt in the rootfs
  • Export NFS filesystem in your host.

LTP-DDT expects certain test binaries to be installed in the filesystem. If you use a tisdk filesystem then all the required test tools have been already installed in the filesystem. One can get a recent tisdk rootfs image from http://lcpd.dal.design.ti.com/core-sdk/nightly/deploy/images/
Select appropriate tisdk-rootfs-image-<MACHINE>*.tar.gz

  • Make sure ltp-ddt sources are up to date. This step will clone latest ltp-ddt project into ltp-ddt directory
./linux-devtest.py -U
  • Build Kernel, modules and headers and install them in the filesystem
cd <your kernel source root>
make ARCH=arm CROSS_COMPILE=... zImage modules
sudo make INSTALL_MOD_PATH=<nfs root> modules_install 
sudo make INSTALL_HDR_PATH=<nfs root>/usr headers_install
IMPORTANT

Make sure you use recommended toolchain. See toolchain installation instructions
  • Build and install latest LTP-DDT sources
cd <ltp-ddt root dir>.  # This is the directory created by Step 2 above 
# Change cross-compiler, kernel include and kernel usr include and platform accordingly in the command below
# KERNEL_INC should point to you kernel sources include directory
# KERNEL_USR_INC should point to the filesystem/usr/include directory
make CROSS_COMPILE=arm-linux-gnueabihf- KERNEL_INC=<kernel sources root>/include KERNEL_USR_INC=<nfs root>/usr/include SKIP_IDCHECK=1 PLATFORM=omap5-evm
# Finally install ltp-ddt in the filesystem
sudo make DESTDIR=<nfs root>/opt/myltp SKIP_IDCHECK=1 PLATFORM=omap5-evm install



how to contribute tests back to ltp-ddt

Please submit your LTP-DDT patches to opentest@arago-project.org.
You may want to check the README-DDT file in the ltp-ddt directory for more information about how to create new tests.

How-To Run Testlink tests plans on the boards Farm

  • Use this option to run a Testlink test plan. If you use this option results will be stored in the Testlink server.
  • Make sure you have already install Opentest Client tools
  • Also verify that STAF is running
  • Run linux-devtest.py -T from the location where you installed Opentest client tools
./linux-devtest.py -T <Testlink testplan> <options>
IMPORTANT

The command above will not run by default without some options being passed. At a minimum you should specify the board farm to use, but likely you will also add your kernel image to test

The Testlink testplan name syntax is: <project name>:<testplan name>

To get the list of available projects, use following command:

./linux-devtest.py --list-projects

To get the list of available testplans on a project, use --list-testplans <project ID> where project ID can be obtained from --list-projects command. For example:

./linux-devtest.py --list-testplans 1140921

The above command will return the list of testplans for LCPD linux. Project linux_psp2(id=1140921) hosts LCPD's linux test plans.

Please note that --list-testplans uses project ID, while linux-devtest.py -T uses project and testplans names.

linux-devtest.py arguments help

command help

# use -h flag to show command help
./linux-devtest.py -h

-hw argument

The -hw argument selects the type of hardware (aka DUT) to run the tests on. It is also possible to specify capabilities or requirements that selected DUT must have. For instance, capabilities could be used to ask for a board that has an usb mass storage device attached to it. The argument syntax is -hw evm,capabilities where capabilities is an optional underscore-separated list of strings Valid evm names are the same used by OE/Yocto: am335x-evm, omap5-evm, beaglebone, dra7xx-evm, etc. For valid capabilities see Capabilities cheat sheet

linux-devtest.py ... -hw omap5-evm,linux

-p, -b, -k, -m, -d, -r, -n Software arguments

Set software images (i.e. bootloader, kernel, filesystem, etc.) to used for the test. It is not mandatory to provide all of them. For example if bootloader (-p and -b) are not provided then the system will try to boot the DUT using whatever bootloader image is already installed on it. Most software image values can take either a local path or a http/ftp url. Use ./linux-devtest.py -h for more details


-u, --user-bins argument

Executable or tarball of executables that should be installed on FS. This is useful for cases where user wants to run a binary that is not available in the default filesystem The value provided can be either a local file or a public http/ftp url

linux-devtest.py ... -u /home/tmp/myapp

-s, --script argument

Use this option to run any arbitrary commands or a shell script in the DUT. The syntax is either a string of semicolon-separated commands (e.g. "echo 'hello'; echo 'bye'") or a path to a shell script. In either case the test will pass if the return value of the commands or script is zero.

linux-devtest.py ... -s /home/tmp/mytest.sh

-t, --tests argument

Use this option to run your own ltp-ddt test plan. Please note that -s, -t and -T are mutually exclusive options

 linux-devtest.py ... -t mytests

To create your own ltp-ddt test plan follow this steps:

  • Update default tests. This will clone ltp-ddt project if required.
./linux-devtest.py -U
  • Create a copy of default test plan
cp tests/default.py tests/mytests.py
  • Modify tests_to_run list in tests/mytests.py to suit your needs by copying entries from tests_available list into it.

Each entry in tests_available lists points to a ltp-ddt test scenario (aka test suite). The syntax is:
<test scenario name>:<setup requirements>:<test cases filter>


-T, --testplan argument

  • Use this option to run a Testlink test plan. If you use this option results will be stored in the Testlink server.

The Testlink Testplan name syntax is: <project name>:<testplan name>. For example:

./linux-devtest.py -T linux_psp2:ch_sandbox
  • To get the list of available projects, use following command:
./linux-devtest.py --list-projects
  • To get the list of available testplans on a project, use --list-testplans <project ID> where project ID can be obtained from --list-projects command. For example:
./linux-devtest.py --list-testplans 1140921

The above command will return the list of testplans for LCPD linux. linux_psp2 is the project name that host LCPD's linux test plans.

Please note that --list-testplans uses project ID, while linux-devtest.py -T uses project and testplans names.

-f, --farm argument

forces usage of a board in a farm.

-f tigt_farm
-f tid_farm

--advanced-params

advanced-params are variables that advanced users can set to tweak execution. At the moment the only advanced-params supported by linux-devtest.py is var_use_default_env. Usually linux-devtest.py will try to boot the DUT using provided software images (i.e. kernel, filesystem, etc.), but advanced users may change that default behavior.

  • var_use_default_env=1 means Boot using default environment (i.e. env default -a -f; boot )
  • var_use_default_env=2 means Do not touch uboot env, just power cycle and let it go

Errors

  • If you get ImportError: No module named argparse while running linux-devtest.py then you need to install python-argparse
$ sudo apt-get install python-argparse
Personal tools