linux-user/mmap.c: handle invalid len maps correctly

I've slightly re-organised the check to more closely match the
sequence that the kernel uses in do_mmap(). We check for both the zero
case (EINVAL) and the overflow length case (ENOMEM).

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: umarcor <1783362@bugs.launchpad.net>
Reviewed-by: Laurent Vivier <laurent@vivier.eu>
Message-Id: <20180730134321.19898-2-alex.bennee@linaro.org>
Signed-off-by: Laurent Vivier <laurent@vivier.eu>