malloc: Fix warnings in tests with GCC 9

This is a partial backport of test changes in commit
9bf8e29c ("malloc: make malloc fail
with requests larger than PTRDIFF_MAX (BZ#23741)"), without the
actual functionality changes.