Merge tag 'linux-kselftest-kunit-fixes-5.14-rc1' of...

	style

	faq

	tips

	running_tips

What is KUnit?

==============

What is a .kunitconfig?

=======================

It's just a defconfig that kunit_tool looks for in the base directory.

kunit_tool uses it to generate a .config as you might expect. In addition, it

verifies that the generated .config contains the CONFIG options in the

.kunitconfig; the reason it does this is so that it is easy to be sure that a

CONFIG that enables a test actually ends up in the .config.

It's just a defconfig that kunit_tool looks for in the build directory

(``.kunit`` by default).  kunit_tool uses it to generate a .config as you might

expect. In addition, it verifies that the generated .config contains the CONFIG

options in the .kunitconfig; the reason it does this is so that it is easy to

be sure that a CONFIG that enables a test actually ends up in the .config.

How do I use kunit_tool?

========================

It's also possible to pass a separate .kunitconfig fragment to kunit_tool,

which is useful if you have several different groups of tests you wish

to run independently, or if you want to use pre-defined test configs for

certain subsystems.

Getting Started with kunit_tool

===============================

If a kunitconfig is present at the root directory, all you have to do is:

	This command will work even without a .kunitconfig file: if no

	.kunitconfig is present, a default one will be used instead.

If you wish to use a different .kunitconfig file (such as one provided for

testing a particular subsystem), you can pass it as an option.

.. code-block:: bash

	./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4/.kunitconfig

For a list of all the flags supported by kunit_tool, you can run:

.. code-block:: bash

	./tools/testing/kunit/kunit.py run --help

Configuring, Building, and Running Tests

========================================

It's also possible to run just parts of the KUnit build process independently,

which is useful if you want to make manual changes to part of the process.

A .config can be generated from a .kunitconfig by using the ``config`` argument

when running kunit_tool:

.. code-block:: bash

	./tools/testing/kunit/kunit.py config

Similarly, if you just want to build a KUnit kernel from the current .config,

you can use the ``build`` argument:

.. code-block:: bash

	./tools/testing/kunit/kunit.py build

And, if you already have a built UML kernel with built-in KUnit tests, you can

run the kernel and display the test results with the ``exec`` argument:

.. code-block:: bash

	./tools/testing/kunit/kunit.py exec

The ``run`` command which is discussed above is equivalent to running all three

of these in sequence.

All of these commands accept a number of optional command-line arguments. The

``--help`` flag will give a complete list of these, or keep reading this page

for a guide to some of the more useful ones.

Parsing Test Results

====================

KUnit tests output their results in TAP (Test Anything Protocol) format.

kunit_tool will, when running tests, parse this output and print a summary

which is much more pleasant to read. If you wish to look at the raw test

results in TAP format, you can pass the ``--raw_output`` argument.

.. code-block:: bash

	./tools/testing/kunit/kunit.py run --raw_output

.. note::

	The raw output from test runs may contain other, non-KUnit kernel log

	lines.

If you have KUnit results in their raw TAP format, you can parse them and print

the human-readable summary with the ``parse`` command for kunit_tool. This

accepts a filename for an argument, or will read from standard input.

.. code-block:: bash

	# Reading from a file

	./tools/testing/kunit/kunit.py parse /var/log/dmesg

	# Reading from stdin

	dmesg | ./tools/testing/kunit/kunit.py parse

This is very useful if you wish to run tests in a configuration not supported

by kunit_tool (such as on real hardware, or an unsupported architecture).

Filtering Tests

===============

It's possible to run only a subset of the tests built into a kernel by passing

a filter to the ``exec`` or ``run`` commands. For example, if you only wanted

to run KUnit resource tests, you could use:

.. code-block:: bash

	./tools/testing/kunit/kunit.py run 'kunit-resource*'

This uses the standard glob format for wildcards.

Running Tests on QEMU

=====================

kunit_tool supports running tests on QEMU as well as via UML (as mentioned

elsewhere). The default way of running tests on QEMU requires two flags:

``--arch``

	Selects a collection of configs (Kconfig as well as QEMU configs

	options, etc) that allow KUnit tests to be run on the specified

	architecture in a minimal way; this is usually not much slower than

	using UML. The architecture argument is the same as the name of the

	option passed to the ``ARCH`` variable used by Kbuild. Not all

	architectures are currently supported by this flag, but can be handled

	by the ``--qemu_config`` discussed later. If ``um`` is passed (or this

	this flag is ignored) the tests will run via UML. Non-UML architectures,

	e.g. i386, x86_64, arm, um, etc. Non-UML run on QEMU.

``--cross_compile``

	Specifies the use of a toolchain by Kbuild. The argument passed here is

	the same passed to the ``CROSS_COMPILE`` variable used by Kbuild. As a

	reminder this will be the prefix for the toolchain binaries such as gcc

	for example ``sparc64-linux-gnu-`` if you have the sparc toolchain

	installed on your system, or

	``$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux-``

	if you have downloaded the microblaze toolchain from the 0-day website

	to a directory in your home directory called ``toolchains``.

In many cases it is likely that you may want to run an architecture which is

not supported by the ``--arch`` flag, or you may want to just run KUnit tests

on QEMU using a non-default configuration. For this use case, you can write

your own QemuConfig. These QemuConfigs are written in Python. They must have an

import line ``from ..qemu_config import QemuArchParams`` at the top of the file

and the file must contain a variable called ``QEMU_ARCH`` that has an instance

of ``QemuArchParams`` assigned to it. An example can be seen in

``tools/testing/kunit/qemu_configs/x86_64.py``.

Once you have a QemuConfig you can pass it into kunit_tool using the

``--qemu_config`` flag; when used this flag replaces the ``--arch`` flag. If we

were to do this with the ``x86_64.py`` example from above, the invocation would

look something like this:

.. code-block:: bash

	./tools/testing/kunit/kunit.py run \

		--timeout=60 \

		--jobs=12 \

		--qemu_config=./tools/testing/kunit/qemu_configs/x86_64.py

Other Useful Options

====================

kunit_tool has a number of other command-line arguments which can be useful

when adapting it to fit your environment or needs.

Some of the more useful ones are:

``--help``

	Lists all of the available options. Note that different commands

	(``config``, ``build``, ``run``, etc) will have different supported

	options. Place ``--help`` before the command to list common options,

	and after the command for options specific to that command.

``--build_dir``

	Specifies the build directory that kunit_tool will use. This is where

	the .kunitconfig file is located, as well as where the .config and

	compiled kernel will be placed. Defaults to ``.kunit``.

``--make_options``

	Specifies additional options to pass to ``make`` when compiling a

	kernel (with the ``build`` or ``run`` commands). For example, to enable

	compiler warnings, you can pass ``--make_options W=1``.

``--alltests``

        Builds a UML kernel with all config options enabled using ``make

        allyesconfig``. This allows you to run as many tests as is possible,

        but is very slow and prone to breakage as new options are added or

        modified. In most cases, enabling all tests which have satisfied

        dependencies by adding ``CONFIG_KUNIT_ALL_TESTS=1`` to your

        .kunitconfig is preferable.

There are several other options (and new ones are often added), so do check

``--help`` if you're looking for something not mentioned here.

.. SPDX-License-Identifier: GPL-2.0

============================

Tips For Running KUnit Tests

============================

Using ``kunit.py run`` ("kunit tool")

=====================================

Running from any directory

--------------------------

It can be handy to create a bash function like:

.. code-block:: bash

	function run_kunit() {

	  ( cd "$(git rev-parse --show-toplevel)" && ./tools/testing/kunit/kunit.py run $@ )

	}

.. note::

	Early versions of ``kunit.py`` (before 5.6) didn't work unless run from

	the kernel root, hence the use of a subshell and ``cd``.

Running a subset of tests

-------------------------

``kunit.py run`` accepts an optional glob argument to filter tests. Currently

this only matches against suite names, but this may change in the future.

Say that we wanted to run the sysctl tests, we could do so via:

.. code-block:: bash

	$ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' > .kunit/.kunitconfig

	$ ./tools/testing/kunit/kunit.py run 'sysctl*'

We're paying the cost of building more tests than we need this way, but it's

easier than fiddling with ``.kunitconfig`` files or commenting out

``kunit_suite``'s.

However, if we wanted to define a set of tests in a less ad hoc way, the next

tip is useful.

Defining a set of tests

-----------------------

``kunit.py run`` (along with ``build``, and ``config``) supports a

``--kunitconfig`` flag. So if you have a set of tests that you want to run on a

regular basis (especially if they have other dependencies), you can create a

specific ``.kunitconfig`` for them.

E.g. kunit has one for its tests:

.. code-block:: bash

	$ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit/.kunitconfig

Alternatively, if you're following the convention of naming your

file ``.kunitconfig``, you can just pass in the dir, e.g.

.. code-block:: bash

	$ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit

.. note::

	This is a relatively new feature (5.12+) so we don't have any

	conventions yet about on what files should be checked in versus just

	kept around locally. It's up to you and your maintainer to decide if a

	config is useful enough to submit (and therefore have to maintain).

.. note::

	Having ``.kunitconfig`` fragments in a parent and child directory is

	iffy. There's discussion about adding an "import" statement in these

	files to make it possible to have a top-level config run tests from all

	child directories. But that would mean ``.kunitconfig`` files are no

	longer just simple .config fragments.

	One alternative would be to have kunit tool recursively combine configs

	automagically, but tests could theoretically depend on incompatible

	options, so handling that would be tricky.

Generating code coverage reports under UML

------------------------------------------

.. note::

	TODO(brendanhiggins@google.com): There are various issues with UML and

	versions of gcc 7 and up. You're likely to run into missing ``.gcda``

	files or compile errors. We know one `faulty GCC commit

	<https://github.com/gcc-mirror/gcc/commit/8c9434c2f9358b8b8bad2c1990edf10a21645f9d>`_

	but not how we'd go about getting this fixed. The compile errors still

	need some investigation.

.. note::

	TODO(brendanhiggins@google.com): for recent versions of Linux

	(5.10-5.12, maybe earlier), there's a bug with gcov counters not being

	flushed in UML. This translates to very low (<1%) reported coverage. This is

	related to the above issue and can be worked around by replacing the

	one call to ``uml_abort()`` (it's in ``os_dump_core()``) with a plain

	``exit()``.

This is different from the "normal" way of getting coverage information that is

documented in Documentation/dev-tools/gcov.rst.

Instead of enabling ``CONFIG_GCOV_KERNEL=y``, we can set these options:

.. code-block:: none

	CONFIG_DEBUG_KERNEL=y

	CONFIG_DEBUG_INFO=y

	CONFIG_GCOV=y

Putting it together into a copy-pastable sequence of commands:

.. code-block:: bash

	# Append coverage options to the current config

	$ echo -e "CONFIG_DEBUG_KERNEL=y\nCONFIG_DEBUG_INFO=y\nCONFIG_GCOV=y" >> .kunit/.kunitconfig

	$ ./tools/testing/kunit/kunit.py run

	# Extract the coverage information from the build dir (.kunit/)

	$ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/

	# From here on, it's the same process as with CONFIG_GCOV_KERNEL=y

	# E.g. can generate an HTML report in a tmp dir like so:

	$ genhtml -o /tmp/coverage_html coverage.info

If your installed version of gcc doesn't work, you can tweak the steps:

.. code-block:: bash

	$ ./tools/testing/kunit/kunit.py run --make_options=CC=/usr/bin/gcc-6

	$ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/ --gcov-tool=/usr/bin/gcov-6

Running tests manually

======================

Running tests without using ``kunit.py run`` is also an important use case.

Currently it's your only option if you want to test on architectures other than

UML.

As running the tests under UML is fairly straightforward (configure and compile

the kernel, run the ``./linux`` binary), this section will focus on testing

non-UML architectures.

Running built-in tests

----------------------

When setting tests to ``=y``, the tests will run as part of boot and print

results to dmesg in TAP format. So you just need to add your tests to your

``.config``, build and boot your kernel as normal.

So if we compiled our kernel with:

.. code-block:: none

	CONFIG_KUNIT=y

	CONFIG_KUNIT_EXAMPLE_TEST=y

Then we'd see output like this in dmesg signaling the test ran and passed:

.. code-block:: none

	TAP version 14

	1..1

	    # Subtest: example

	    1..1

	    # example_simple_test: initializing

	    ok 1 - example_simple_test

	ok 1 - example

Running tests as modules

------------------------

Depending on the tests, you can build them as loadable modules.

For example, we'd change the config options from before to

.. code-block:: none

	CONFIG_KUNIT=y

	CONFIG_KUNIT_EXAMPLE_TEST=m

Then after booting into our kernel, we can run the test via

.. code-block:: none

	$ modprobe kunit-example-test

This will then cause it to print TAP output to stdout.

.. note::

	The ``modprobe`` will *not* have a non-zero exit code if any test

	failed (as of 5.13). But ``kunit.py parse`` would, see below.

.. note::

	You can set ``CONFIG_KUNIT=m`` as well, however, some features will not

	work and thus some tests might break. Ideally tests would specify they

	depend on ``KUNIT=y`` in their ``Kconfig``'s, but this is an edge case

	most test authors won't think about.

	As of 5.13, the only difference is that ``current->kunit_test`` will

	not exist.

Pretty-printing results

-----------------------

You can use ``kunit.py parse`` to parse dmesg for test output and print out

results in the same familiar format that ``kunit.py run`` does.

.. code-block:: bash

	$ ./tools/testing/kunit/kunit.py parse /var/log/dmesg

Retrieving per suite results

----------------------------

Regardless of how you're running your tests, you can enable

``CONFIG_KUNIT_DEBUGFS`` to expose per-suite TAP-formatted results:

.. code-block:: none

	CONFIG_KUNIT=y

	CONFIG_KUNIT_EXAMPLE_TEST=m

	CONFIG_KUNIT_DEBUGFS=y

The results for each suite will be exposed under

``/sys/kernel/debug/kunit/<suite>/results``.

So using our example config:

.. code-block:: bash

	$ modprobe kunit-example-test > /dev/null

	$ cat /sys/kernel/debug/kunit/example/results

	... <TAP output> ...

	# After removing the module, the corresponding files will go away

	$ modprobe -r kunit-example-test

	$ cat /sys/kernel/debug/kunit/example/results

	/sys/kernel/debug/kunit/example/results: No such file or directory

Generating code coverage reports

--------------------------------

See Documentation/dev-tools/gcov.rst for details on how to do this.

The only vaguely KUnit-specific advice here is that you probably want to build

your tests as modules. That way you can isolate the coverage from tests from

other code executed during boot, e.g.

.. code-block:: bash

	# Reset coverage counters before running the test.

	$ echo 0 > /sys/kernel/debug/gcov/reset

	$ modprobe kunit-example-test

.. code-block:: bash

	cd $PATH_TO_LINUX_REPO

	cp arch/um/configs/kunit_defconfig .kunitconfig

	cp tools/testing/kunit/configs/default.config .kunitconfig

You can then add any other Kconfig options you wish, e.g.:

==========

*   Check out the Documentation/dev-tools/kunit/tips.rst page for tips on

    writing idiomatic KUnit tests.

*   Check out the :doc:`running_tips` page for tips on

    how to make running KUnit tests easier.

*   Optional: see the :doc:`usage` page for a more

    in-depth explanation of KUnit.

.. code-block:: c

	/* Note: the cast is to satisfy overly strict type-checking. */

	#define TEST_SHA1(in, want) \

		sha1sum(in, out); \

		KUNIT_EXPECT_STREQ_MSG(test, (char *)out, want, "sha1sum(%s)", in);

		KUNIT_EXPECT_STREQ_MSG(test, out, want, "sha1sum(%s)", in);

	char out[40];

	TEST_SHA1("hello world",  "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed");

	};

	for (i = 0; i < ARRAY_SIZE(cases); ++i) {

		sha1sum(cases[i].str, out);

		KUNIT_EXPECT_STREQ_MSG(test, (char *)out, cases[i].sha1,

		KUNIT_EXPECT_STREQ_MSG(test, out, cases[i].sha1,

		                      "sha1sum(%s)", cases[i].str);

	}

		struct sha1_test_case *test_param = (struct sha1_test_case *)(test->param_value);

		sha1sum(test_param->str, out);

		KUNIT_EXPECT_STREQ_MSG(test, (char *)out, test_param->sha1,

		KUNIT_EXPECT_STREQ_MSG(test, out, test_param->sha1,

				      "sha1sum(%s)", test_param->str);

	}

None of these are reasons not to run your KUnit tests on real hardware; they are

only things to be aware of when doing so.

The biggest impediment will likely be that certain KUnit features and

infrastructure may not support your target environment. For example, at this

time the KUnit Wrapper (``tools/testing/kunit/kunit.py``) does not work outside

of UML. Unfortunately, there is no way around this. Using UML (or even just a

particular architecture) allows us to make a lot of assumptions that make it

possible to do things which might otherwise be impossible.

Nevertheless, all core KUnit framework features are fully supported on all

architectures, and using them is straightforward: all you need to do is to take

your kunitconfig, your Kconfig options for the tests you would like to run, and

merge them into whatever config your are using for your platform. That's it!

Currently, the KUnit Wrapper (``tools/testing/kunit/kunit.py``) (aka

kunit_tool) only fully supports running tests inside of UML and QEMU; however,

this is only due to our own time limitations as humans working on KUnit. It is

entirely possible to support other emulators and even actual hardware, but for

now QEMU and UML is what is fully supported within the KUnit Wrapper. Again, to

be clear, this is just the Wrapper. The actualy KUnit tests and the KUnit

library they are written in is fully architecture agnostic and can be used in

virtually any setup, you just won't have the benefit of typing a single command

out of the box and having everything magically work perfectly.

Again, all core KUnit framework features are fully supported on all

architectures, and using them is straightforward: Most popular architectures

are supported directly in the KUnit Wrapper via QEMU. Currently, supported

architectures on QEMU include:

*   i386

*   x86_64

*   arm

*   arm64

*   alpha

*   powerpc

*   riscv

*   s390

*   sparc

In order to run KUnit tests on one of these architectures via QEMU with the

KUnit wrapper, all you need to do is specify the flags ``--arch`` and

``--cross_compile`` when invoking the KUnit Wrapper. For example, we could run

the default KUnit tests on ARM in the following manner (assuming we have an ARM

toolchain installed):

.. code-block:: bash

	tools/testing/kunit/kunit.py run --timeout=60 --jobs=12 --arch=arm --cross_compile=arm-linux-gnueabihf-

Alternatively, if you want to run your tests on real hardware or in some other

emulation environment, all you need to do is to take your kunitconfig, your

Kconfig options for the tests you would like to run, and merge them into

whatever config your are using for your platform. That's it!

For example, let's say you have the following kunitconfig: