drm/vmwgfx: Add a check to handle host message failure

Discovered by static code analysis tool.  If for some reason communication
with the host fails more than preset number of retries, return an error
instead of return garbage.

Signed-off-by: Sinclair Yeh <syeh@vmware.com>
Reviewed-by: Charmaine Lee <charmainel@vmware.com>
Reported-by: Dan Carpenter <dan.carpenter@oracle.com>