drm/mgag200: fix G200ER pll picking algorithm

The original code was misported from the X driver,

a) an int went to unsigned int, breaking the downward counting testm code
b) the port did the vco/computed clock bits completely wrong.

This fixes an infinite loop on modprobe on some Dell servers with the G200ER
chipset variant.

Found in internal testing.

Cc: stable@vger.kernel.org
Signed-off-by: Dave Airlie <airlied@redhat.com>