drm/vmwgfx: Properly check display/scanout surface size

The scanout surface size is the smaller of max texture size and
max STDU size.

Signed-off-by: Sinclair Yeh <syeh@vmware.com>
Reviewed-by: Thomas Hellstrom <thellstrom@vmware.com>