drm/vmwgfx: Fix an incorrect OOM return value

At the same time, make error paths return early for clarity.

Signed-off-by: Thomas Hellstrom <thellstrom@vmware.com>
Reported-by: Dan Carpenter <dan.carpenter@oracle.com>
Reviewed-by: jakob Bornecrantz <jakob@vmware.com>
Cc: <stable@vger.kernel.org>