fpu: add mechanism to check for invalid long double formats

All operations that take a floatx80 as an operand need to have their
inputs checked for malformed encodings. In all of these cases, use the
function floatx80_invalid_encoding to perform the check. If an invalid
operand is found, raise an invalid operation exception, and then return
either NaN (for fp-typed results) or the integer indefinite value (the
minimum representable signed integer value, for int-typed results).

For the non-quiet comparison operations, this touches adjacent code in
order to pass style checks.

Signed-off-by: Andrew Dutcher <andrew@andrewdutcher.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-id: 1471392895-17324-1-git-send-email-andrew@andrewdutcher.com
[PMM: changed "1 << 63" to "1ULL << 63" to fix compile errors]
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>