Commit 7f117cd3 authored by Matthieu Baerts's avatar Matthieu Baerts Committed by David S. Miller
Browse files

selftests: mptcp: join: format subtests results in TAP 

The current selftests infrastructure formats the results in TAP 13. This
version doesn't support subtests and only the end result of each
selftest is taken into account. It means that a single issue in a
subtest of a selftest containing multiple subtests forces the whole
selftest to be marked as failed. It also means that subtests results are
not tracked by CIs executing selftests.

MPTCP selftests run hundreds of various subtests. It is then important
to track each of them and not one result per selftest.

It is particularly interesting to do that when validating stable kernels
with the last version of the test suite: tests might fail because a
feature is not supported but the test didn't skip that part. In this
case, if subtests are not tracked, the whole selftest will be marked as
failed making the other subtests useless because their results are
ignored.

This patch formats subtests results in TAP in mptcp_join.sh selftest.

In this selftest and before starting each subtest, the 'reset' function
is called. We can then check if the previous test has passed, failed or
has been skipped from there.

Link: https://github.com/multipath-tcp/mptcp_net-next/issues/368


Acked-by: default avatarPaolo Abeni <pabeni@redhat.com>
Signed-off-by: default avatarMatthieu Baerts <matthieu.baerts@tessares.net>
Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
parent d85555ac

tools/testing/selftests/net/mptcp/mptcp_join.sh

+32 −5
Original line number Diff line number Diff line
@@ -39,7 +39,9 @@ evts_ns1="" 
evts_ns2=""

evts_ns1_pid=0

evts_ns2_pid=0

stats_dumped=0

last_test_failed=0

last_test_skipped=0

last_test_ignored=1



declare -A all_tests

declare -a only_tests_ids
@@ -101,7 +103,6 @@ init_partial() 
		fi

	done



	stats_dumped=0

	check_invert=0

	validate_checksum=$checksum
@@ -216,6 +217,8 @@ mark_as_skipped() 


	print_title "[ skip ] ${msg}"

	printf "\n"



	last_test_skipped=1

}



# $@: condition
@@ -248,14 +251,32 @@ skip_test() 
	return 0

}



append_prev_results()

{

	if [ ${last_test_failed} -eq 1 ]; then

		mptcp_lib_result_fail "${TEST_NAME}"

	elif [ ${last_test_skipped} -eq 1 ]; then

		mptcp_lib_result_skip "${TEST_NAME}"

	elif [ ${last_test_ignored} -ne 1 ]; then

		mptcp_lib_result_pass "${TEST_NAME}"

	fi



	last_test_failed=0

	last_test_skipped=0

	last_test_ignored=0

}



# $1: test name

reset()

{

	append_prev_results



	TEST_NAME="${1}"



	TEST_COUNT=$((TEST_COUNT+1))



	if skip_test; then

		last_test_ignored=1

		return 1

	fi
@@ -442,10 +463,13 @@ reset_with_tcp_filter() 
fail_test()

{

	ret=1

	failed_tests[${TEST_COUNT}]="${TEST_NAME}"



	[ "${stats_dumped}" = 0 ] && dump_stats

	stats_dumped=1

	# just in case a test is marked twice as failed

	if [ ${last_test_failed} -eq 0 ]; then

		failed_tests[${TEST_COUNT}]="${TEST_NAME}"

		dump_stats

		last_test_failed=1

	fi

}



get_failed_tests_ids()
@@ -3599,4 +3623,7 @@ if [ ${ret} -ne 0 ]; then 
	echo

fi



append_prev_results

mptcp_lib_result_print_all_tap



exit $ret