Commit cd20358b authored May 24, 2024 by Matthieu Baerts (NGI0) Committed by Gang Yan Dec 09, 2024

selftests: mptcp: lib: support flaky subtests

mainline inclusion
from mainline-v6.10-rc2
commit 5597613fb3cf0e36d26cfd8fb2a63196da249333
category: feature
bugzilla: https://gitee.com/openeuler/kernel/issues/I9VYQ9
CVE: NA

Reference: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?id=5597613fb3cf0e36d26cfd8fb2a63196da249333



--------------------------------

Some subtests can be unstable, failing once every X runs. Fixing them
can take time: there could be an issue in the kernel or in the subtest,
and it is then important to do a proper analysis, not to hide real bugs.

To avoid creating noises on the different CIs, it is important to have a
simple way to mark subtests as flaky, and ignore the errors. This is
what this patch introduces: subtests can be marked as flaky by setting
MPTCP_LIB_SUBTEST_FLAKY env var to 1, e.g.

  MPTCP_LIB_SUBTEST_FLAKY=1 <run flaky subtest>

The subtest will be executed, and errors (if any) will be ignored. It is
still good to run these subtests, as it exercises code, and the results
can still be useful for the on-going investigations.

Note that the MPTCP CI will continue to track these flaky subtests by
setting SELFTESTS_MPTCP_LIB_OVERRIDE_FLAKY env var to 1, and a ticket
has to be created before marking subtests as flaky.

Reviewed-by: Mat Martineau <martineau@kernel.org>
Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
Link: https://lore.kernel.org/r/20240524-upstream-net-20240524-selftests-mptcp-flaky-v1-1-a352362f3f8e@kernel.org


Signed-off-by: Jakub Kicinski <kuba@kernel.org>

parent 4080d0ad

tools/testing/selftests/net/mptcp/mptcp_lib.sh

+28 −2

Original line number	Diff line number	Diff line
		@@ -21,6 +21,7 @@ declare -rx MPTCP_LIB_AF_INET6=10

		MPTCP_LIB_SUBTESTS=()
		MPTCP_LIB_SUBTESTS_DUPLICATED=0
		MPTCP_LIB_SUBTEST_FLAKY=0
		MPTCP_LIB_TEST_COUNTER=0
		MPTCP_LIB_TEST_FORMAT="%02u %-50s"
		MPTCP_LIB_IP_MPTCP=0
		@@ -41,6 +42,16 @@ else
		readonly MPTCP_LIB_COLOR_RESET=
		fi

		# SELFTESTS_MPTCP_LIB_OVERRIDE_FLAKY env var can be set not to ignore errors
		# from subtests marked as flaky
		mptcp_lib_override_flaky() {
		[ "${SELFTESTS_MPTCP_LIB_OVERRIDE_FLAKY:-}" = 1 ]
		}

		mptcp_lib_subtest_is_flaky() {
		[ "${MPTCP_LIB_SUBTEST_FLAKY}" = 1 ] && ! mptcp_lib_override_flaky
		}

		# $1: color, $2: text
		mptcp_lib_print_color() {
		echo -e "${MPTCP_LIB_START_PRINT:-}${*}${MPTCP_LIB_COLOR_RESET}"
		@@ -72,7 +83,16 @@ mptcp_lib_pr_skip() {
		}

		mptcp_lib_pr_fail() {
		mptcp_lib_print_err "[FAIL]${1:+ ${*}}"
		local title cmt

		if mptcp_lib_subtest_is_flaky; then
		title="IGNO"
		cmt=" (flaky)"
		else
		title="FAIL"
		fi

		mptcp_lib_print_err "[${title}]${cmt}${1:+ ${*}}"
		}

		mptcp_lib_pr_info() {
		@@ -208,7 +228,13 @@ mptcp_lib_result_pass() {

		# $1: test name
		mptcp_lib_result_fail() {
		if mptcp_lib_subtest_is_flaky; then
		# It might sound better to use 'not ok # TODO' or 'ok # SKIP',
		# but some CIs don't understand 'TODO' and treat SKIP as errors.
		__mptcp_lib_result_add "ok" "${1} # IGNORE Flaky"
		else
		__mptcp_lib_result_add "not ok" "${1}"
		fi
		}

		# $1: test name