Gecko [ fernandoquevedodop.com ]

Name	Size	Permission
fhs	[ DIR ]	drwxr-xr-x
menu-policy.html	[ DIR ]	drwxr-xr-x
perl-policy.html	[ DIR ]	drwxr-xr-x
policy.html	[ DIR ]	drwxr-xr-x
README.css	481 B	-rw-r--r--
README.html	18 KB	-rw-r--r--
README.txt.gz	6.21 KB	-rw-r--r--
autopkgtest.html	14.02 KB	-rw-r--r--
autopkgtest.txt.gz	4.78 KB	-rw-r--r--
changelog.gz	2.78 KB	-rw-r--r--
copyright	4.86 KB	-rw-r--r--
copyright-format-1.0.html	60.34 KB	-rw-r--r--
copyright-format-1.0.txt.gz	10.29 KB	-rw-r--r--
debconf_specification.html	28.64 KB	-rw-r--r--
debconf_specification.txt.gz	7.35 KB	-rw-r--r--
menu-policy-1.html	22.57 KB	-rw-r--r--
menu-policy.txt.gz	4.38 KB	-rw-r--r--
perl-policy-1.html	34.77 KB	-rw-r--r--
perl-policy.txt.gz	5.21 KB	-rw-r--r--
policy.epub	384.31 KB	-rw-r--r--
policy.pdf.gz	830.31 KB	-rw-r--r--
policy.txt.gz	142.23 KB	-rw-r--r--
upgrading-checklist.txt.gz	22.22 KB	-rw-r--r--
virtual-package-names-list.yam...	4.56 KB	-rw-r--r--

Code Editor : autopkgtest.html

<h1 id="autopkgtest-definingtestsfordebianpackages">Autopkgtest - Defining tests for Debian packages</h1>

This document describes how the autopkgtest tester core (the program
<code>adt-run</code>) interprets and executes tests found in Debian source
packages.

<h2 id="overview">Overview</h2>

The source package provides a test metadata file <code>debian/tests/control</code>.
This is a file containing zero or more RFC822-style stanzas, along these
lines:

<pre><code>Tests: fred bill bongo
Restrictions: needs-root breaks-testbed
</code></pre>

This example defines three tests, called <code>fred</code>, <code>bill</code> and <code>bongo</code>. The
tests will be performed by executing <code>debian/tests/fred</code>,
<code>debian/tests/bill</code>, etc. Each test program should, on success, exit
with status 0 and print nothing to stderr; if a test exits nonzero, or
prints to stderr, it is considered to have failed.

The cwd of each test is guaranteed to be the root of the source package,
which will have been unpacked but not built. However note that the
tests must test the installed version of the program. Tests may not
modify the source tree (and may not have write access to it).

If the file to be executed has no execute bits set, <code>chmod a+x</code> is
applied to it (this means that tests can be added in patches without the
need for additional chmod; contrast this with debian/rules).

During execution of the test, the environment variable <code>$AUTOPKGTEST_TMP</code> will
point to a directory for the execution of this particular test, which starts
empty and will be deleted afterwards (so there is no need for the test to
clean up files left there).

If tests want to create artifacts which are useful to attach to test
results, such as additional log files or screenshots, they can put them
into the directory specified by the <code>$ADT_ARTIFACTS</code> environment
variable. When using the <code>--output-dir</code> option, they will be copied into
<code>outputdir/artifacts/</code>.

Tests must declare all applicable Restrictions - see below.

<h2 id="controlfields">Control fields</h2>

The fields which may appear in debian/tests/control stanzas are:

<ul>
<li>Tests: name-of-test [name-of-another-test ...]

This field names the tests which are defined by this stanza, and map to
executables/scripts in the test directory. All of the other fields in the
same stanza apply to all of the named tests. Either this field or
<code>Test-Command:</code> must be present.

Test names are separated by whitespace and should contain only characters
which are legal in package names. It is permitted, but not encouraged, to use
upper-case characters as well.</li>
<li>Test-Command: shell command

If your test only contains a shell command or two, or you want to re-use an
existing upstream test executable and just need to wrap it with some command
like <code>dbus-run-session</code> or <code>env</code>, you can use this field to specify the shell
command directly. It will be run under <code>bash -e</code>. This is mutually exclusive
with the <code>Tests:</code> field.</li>
<li>Restrictions: restriction-name [another-restriction-name ...]

Declares some restrictions or problems with the tests defined in this
stanza. Depending on the test environment capabilities, user requests, and so
on, restrictions can cause tests to be skipped or can cause the test to be
run in a different manner. Tests which declare unknown restrictions will be
skipped. See below for the defined restrictions.</li>
<li>Features: feature-name [another-feature-name ...]

Declares some additional capabilities or good properties of the tests defined
in this stanza. Any unknown features declared will be completely ignored. See
below for the defined features.</li>
<li>Depends: dpkg dependency field syntax

Declares that the specified packages must be installed for the test to go
ahead. This supports all features of dpkg dependencies (see
<a href="https://www.debian.org/doc/debian-policy/ch-relationships.html">https://www.debian.org/doc/debian-policy/ch-relationships.html</a>), plus the
following extensions:

<code>@</code> stands for the package(s) generated by the source package containing the
tests; each dependency (strictly, or-clause, which may contain <code>|</code>s but not
commas) containing <code>@</code> is replicated once for each such binary package, with
the binary package name substituted for each <code>@</code> (but normally <code>@</code> should
occur only once and without a version restriction).

<code>@builddeps@</code> will be replaced by the package's <code>Build-Depends:</code>,
<code>Build-Depends-Indep:</code>, and <code>build-essential</code>. This is useful if you have
many build dependencies which are only necessary for running the test suite
and you don't want to replicate them in the test <code>Depends:</code>. However, please
use this sparingly, as this can easily lead to missing binary package
dependencies being overlooked if they get pulled in via build dependencies.

If no Depends field is present, <code>Depends: @</code> is assumed. Note that the source
tree's Build-Dependencies are not necessarily installed, and if you specify
any Depends, no binary packages from the source are installed unless
explicitly requested.</li>
<li>Tests-Directory: path

Replaces the path segment <code>debian/tests</code> in the filenames of the test
programs with <code>path</code>. I. e., the tests are run by executing
<code>built/source/tree/path/testname</code>. <code>path</code> must be a relative path and is
interpreted starting from the root of the built source tree.

This allows tests to live outside the debian/ metadata area, so that they can
more palatably be shared with non-Debian distributions.</li>
<li>Classes: class-1, class-2, ...

Most package tests should work in a minimal environment and are usually not
hardware specific. However, some packages like the kernel, X.org, or graphics
drivers should be tested on particular hardware, and also run on a set of
different platforms rather than just a single virtual testbeds.

This field can specify a list of abstract class names such as "desktop" or
"graphics-driver". Consumers of autopkgtest can then map these class names to
particular machines/platforms/policies. Unknown class names should be
ignored.

This is purely an informational field for autopkgtest itself and will be
ignored.</li>
</ul>

Any unknown fields will cause the whole stanza to be skipped.

<h2 id="definedrestrictions">Defined restrictions</h2>

<ul>
<li>rw-build-tree

The test(s) needs write access to the built source tree (so it may need to be
copied first). Even with this restriction, the test is not allowed to make
any change to the built source tree which (i) isn't cleaned up by
debian/rules clean, (ii) affects the future results of any test, or (iii)
affects binary packages produced by the build tree in the future.</li>
<li>breaks-testbed

The test, when run, is liable to break the testbed system. This includes
causing data loss, causing services that the machine is running to
malfunction, or permanently disabling services; it does not include causing
services on the machine to temporarily fail.

When this restriction is present the test will usually be skipped unless the
testbed's virtualisation arrangements are sufficiently powerful, or
alternatively if the user explicitly requests.</li>
<li>needs-root

The test script must be run as root.</li>
<li>build-needed

The tests need to be run from a built source tree. The test runner will build
the source tree (honouring the source package's build dependencies), before
running the tests. However, the tests are not entitled to assume that the
source package's build dependencies will be installed when the test is run.

Please use this considerately, as for large builds it unnecessarily builds
the entire project when you only need a tiny subset (like the tests/
subdirectory). It is often possible to run <code>make -C tests</code> instead, or copy
the test code to <code>$AUTOPKGTEST_TMP</code> and build it there with some custom
commands. This cuts down the load on the Continuous Integration servers and
also makes tests more robust as it prevents accidentally running them
against the built source tree instead of the installed packages.</li>
<li>allow-stderr

Output to stderr is not considered a failure. This is useful for tests which
write e. g. lots of logging to stderr.</li>
<li>isolation-container

The test wants to start services or open network TCP ports. This commonly
fails in a simple chroot/schroot, so tests need to be run in their own
container (e. g. adt-virt-lxc) or their own machine/VM (e. g. adt-virt-qemu
or adt-virt-null). When running the test in a virtualization server which
does not provide this (like adt-virt-schroot) it will be skipped.</li>
<li>isolation-machine

The test wants to interact with the kernel, reboot the machine, or other
things which fail in a simple schroot and even a container. Those tests need
to be run in their own machine/VM (e. g. adt-virt-qemu or
adt-virt-null). When running the test in a virtualization server which does
not provide this it will be skipped.</li>
<li>needs-recommends

Enable installation of recommended packages in apt for the test
dependencies. This does not affect build dependencies.</li>
</ul>

<h2 id="definedfeatures">Defined features</h2>

There are no currently defined Features.

<h2 id="sourcepackageheader">Source package header</h2>

To allow test execution environments to discover packages which provide
tests, their source packages should have a <code>Testsuite:</code> header
containing <code>autopkgtest</code> (which is currently the only defined value).
Multiple values get comma separated, as usual in control files.

This tag is added automatically by dpkg-source version 1.17.11 or later.
For earlier Debian/Ubuntu releases you need to set it manually in
debian/control by adding

<pre><code>XS-Testsuite: autopkgtest
</code></pre>

in the <code>Source:</code> paragraph.

<h2 id="implicittestcontrolfileforknownpackagetypes">Implicit test control file for known package types</h2>

There are groups of similarly-structured packages for which the contents
of <code>debian/tests/control</code> would be mostly identical. For those packages,
if <code>debian/tests/control</code> is absent, an implicit control file is
assumed. Those packages do not have to provide anything else, although
they should still include an appropriate source package header
(<code>XS-Testsuite</code>) so that they can be discovered in the archive.

<h3 id="rubypackages">Ruby packages</h3>

The source package must contain at least one of the following files:

<ul>
<li><code>debian/ruby-test-files.yaml</code></li>
<li><code>debian/ruby-tests.rb</code></li>
<li><code>debian/ruby-tests.rake</code></li>
</ul>

Implied control file: :

<pre><code>Test-Command: gem2deb-test-runner --autopkgtest 2&gt;&amp;1
Depends: @, @builddeps@, gem2deb-test-runner
</code></pre>

Note: <code>gem2deb</code> will be filtered out of the <code>Depends:</code> field, as it is
not needed to run the tests for installed packages.

Packages should declare <code>Testsuite: autopkgtest-pkg-ruby</code> in
<code>debian/control</code>.

<h3 id="perlpackages">Perl packages</h3>

The source package must contain a <code>t/</code> directory and at least one of the
following files:

<ul>
<li><code>Makefile.PL</code></li>
<li><code>Build.PL</code></li>
</ul>

Implied control file: :

<pre><code>Test-Command: /usr/share/pkg-perl-autopkgtest/runner build-deps
Depends: @, @builddeps@, pkg-perl-autopkgtest

Test-Command: /usr/share/pkg-perl-autopkgtest/runner runtime-deps
Depends: @, pkg-perl-autopkgtest
</code></pre>

Packages should declare <code>Testsuite: autopkgtest-pkg-perl</code> in
<code>debian/control</code>.

<h2 id="rebootduringatest">Reboot during a test</h2>

Some testbeds support rebooting; for those, the testbed will have an
<code>autopkgtest-reboot</code> command which tests can call to cause a reboot.
Do not use <code>reboot</code> and similar commands directly! They will cause
testbeds like <code>null</code> or <code>schroot</code> to reboot the entire host, and even
for <code>qemu</code> it will just cause the test to fail as there is no state
keeping to resume a test at the right position after reboot.

The particular steps for a rebooting tests are:

<ul>
<li>The test calls <code>autopkgtest-reboot my_mark</code> with a "mark"
identifier. <code>autopkgtest-reboot</code> will cause the test to terminate
(with <code>SIGKILL</code>).</li>
<li><code>adt-run</code> backs up the current state of the test source tree and any
<code>$ADT_ARTIFACTS</code> that were created so far, reboots the testbed, and
restores the test source tree and artifacts.</li>
<li>The test gets run again, this time with a new environment variable
<code>$ADT_REBOOT_MARK</code> containing the argument to <code>autopkgtest-reboot</code>,
e. g. <code>my_mark</code>.</li>
<li>The test needs to check <code>$ADT_REBOOT_MARK</code> and jump to the
appropriate point. A nonexisting variable means "start from the
beginning".</li>
</ul>

This example test will reboot the testbed two times in between:

<pre><code>#!/bin/sh -e
case "$ADT_REBOOT_MARK" in
 "") echo "test beginning"; autopkgtest-reboot mark1 ;;
 mark1) echo "test in mark1"; autopkgtest-reboot mark2 ;;
 mark2) echo "test in mark2" ;;
esac
echo "test end"
</code></pre>

<hr />

Debian Policy 4.6.0.1, 2021-08-18

${this.title}

Code Editor : autopkgtest.html