docs: Added details on TSan to testing.rst

* ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test

  failure" section.

Thread Sanitizer

================

Thread Sanitizer (TSan) is a tool which can detect data races.  QEMU supports

building and testing with this tool.

For more information on TSan:

https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual

Thread Sanitizer in Docker

---------------------------

TSan is currently supported in the ubuntu2004 docker.

The test-tsan test will build using TSan and then run make check.

.. code::

  make docker-test-tsan@ubuntu2004

TSan warnings under docker are placed in files located at build/tsan/.

We recommend using DEBUG=1 to allow launching the test from inside the docker,

and to allow review of the warnings generated by TSan.

Building and Testing with TSan

------------------------------

It is possible to build and test with TSan, with a few additional steps.

These steps are normally done automatically in the docker.

There is a one time patch needed in clang-9 or clang-10 at this time:

.. code::

  sed -i 's/^const/static const/g' \

      /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h

To configure the build for TSan:

.. code::

  ../configure --enable-tsan --cc=clang-10 --cxx=clang++-10 \

               --disable-werror --extra-cflags="-O0"

The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment

variable.

More information on the TSAN_OPTIONS can be found here:

https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags

For example:

.. code::

  export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \

                      detect_deadlocks=false history_size=7 exitcode=0 \

                      log_path=<build path>/tsan/tsan_warning

The above exitcode=0 has TSan continue without error if any warnings are found.

This allows for running the test and then checking the warnings afterwards.

If you want TSan to stop and exit with error on warnings, use exitcode=66.

TSan Suppressions

-----------------

Keep in mind that for any data race warning, although there might be a data race

detected by TSan, there might be no actual bug here.  TSan provides several

different mechanisms for suppressing warnings.  In general it is recommended

to fix the code if possible to eliminate the data race rather than suppress

the warning.

A few important files for suppressing warnings are:

tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime.

The comment on each supression will typically indicate why we are

suppressing it.  More information on the file format can be found here:

https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions

tests/tsan/blacklist.tsan - Has TSan warnings we wish to disable

at compile time for test or debug.

Add flags to configure to enable:

"--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/blacklist.tsan"

More information on the file format can be found here under "Blacklist Format":

https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags

TSan Annotations

----------------

include/qemu/tsan.h defines annotations.  See this file for more descriptions

of the annotations themselves.  Annotations can be used to suppress

TSan warnings or give TSan more information so that it can detect proper

relationships between accesses of data.

Annotation examples can be found here:

https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/

Good files to start with are: annotate_happens_before.cpp and ignore_race.cpp

The full set of annotations can be found here:

https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp

VM testing

==========