drm/radeon: improve encoder picking functions (v2)

For MST we need to be able to pick front end encoders
separate from backend, but only for MST, so we need to
make the encoder picking interface smarter.

v2: agd5f: squash in:
drm/radeon: release digital encoder before asking for new one
Reported-by: Dieter Nützel <Dieter@nuetzel-hh.de>

Signed-off-by: Dave Airlie <airlied@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>