drm/amd/display: Reduce HDMI pixel encoding if max clock is exceeded

For high-res (8K) or HFR (4K120) displays, using uncompressed pixel
formats like YCbCr444 would exceed the bandwidth of HDMI 2.0, so the
"interesting" modes would be disabled, leaving only low-res or low
framerate modes.

This change lowers the pixel encoding to 4:2:2 or 4:2:0 if the max TMDS
clock is exceeded. Verified that 8K30 and 4K120 are now available and
working with a Samsung Q900R over an HDMI 2.0b link from a Radeon 5700.

Reviewed-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Thomas Anderson <thomasanderson@google.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>