drm/nouveau/disp: Support more modes by checking with lower bpc

stable inclusion
from stable-v5.10.178
commit c1bbe926a1c75a81df58cdb8b68fb213068ca6f1
category: bugfix
bugzilla: https://gitee.com/openeuler/kernel/issues/I8D927

Reference: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=c1bbe926a1c75a81df58cdb8b68fb213068ca6f1



--------------------------------

commit 7f67aa09 upstream.

This allows us to advertise more modes especially on HDR displays.

Fixes using 4K@60 modes on my TV and main display both using a HDMI to DP
adapter. Also fixes similar issues for users running into this.

Cc: stable@vger.kernel.org # 5.10+
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230330223938.4025569-1-kherbst@redhat.com


Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: sanglipeng <sanglipeng1@jd.com>