test-math-iscanonical.cc: Replace bool with int

Fix GCC 7 compilation error:

test-math-iscanonical.cc: In function ‘void check_type()’:
test-math-iscanonical.cc:33:11: error: use of an operand of type ‘bool’ in ‘operator++’ is deprecated [-Werror=deprecated]
     errors++;
           ^~

Since not all non-zero error counts are errors, return errors != 0
instead.

	* math/test-math-iscanonical.cc (error): Replace bool with int.
	(do_test): Return errors != 0.

(cherry picked from commit cdd4155d and
 commit 758f1bfa)