drm/arm/malidp: Validate rotations for compressed/uncompressed framebuffers for each layer

Add support for compressed framebuffers that are described using
the framebuffer's modifier field. Mali DP uses the rotation memory for
the decompressor of the format, so we need to check for space when
the modifiers are present.

Signed-off-by: Ayan Kumar Halder <ayan.halder@arm.com>
Reviewed-by: Brian Starkey <brian.starkey@arm.com>
[re-worded commit, rebased, cleaned up duplicated checks for
 RGB888 and BGR888 and removed additional parameter for
 rotmem_required function hook]
Signed-off-by: Liviu Dudau <liviu.dudau@arm.com>