Bluetooth: Fix storing negative values as unsigned char

This is mostly cleanup. There is only one caller and it just checks for
non-zero return values. Still "ret" should be int because we want to return
-EINVAL on errors.

Signed-off-by: Dan Carpenter <error27@gmail.com>
Signed-off-by: Marcel Holtmann <marcel@holtmann.org>