Correct range checking in mallopt/mxfast/tcache [BZ #25194]

do_set_tcache_max, do_set_mxfast:
Fix two instances of comparing "size_t < 0"
Both cases have upper limit, so the "negative value" case
is already handled via overflow semantics.

do_set_tcache_max, do_set_tcache_count:
Fix return value on error.  Note: currently not used.

mallopt:
pass return value of helper functions to user.  Behavior should
only be actually changed for mxfast, where we restore the old
(pre-tunables) behavior.

Reviewed-by: Carlos O'Donell <carlos@redhat.com>