rsa_pss_compute_saltlen(): Avoid integer overflows and check MD and RSA sizes

Fixes Coverity 1604651

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
(Merged from https://github.com/openssl/openssl/pull/25085)

(cherry picked from commit 217e215e)