simplefb: fix unmapping fb during destruction

Unfortunately, fbdev does not create its own "struct device" for
framebuffers. Instead, it attaches to the device of the parent layer. This
has the side-effect that devm_* managed resources are not cleaned up on
framebuffer-destruction but rather during destruction of the
parent-device. In case of fbdev this might be too late, though.
remove_conflicting_framebuffer() may remove fbdev devices but keep the
parent device as it is.

Therefore, we now use plain ioremap() and unmap the framebuffer in the
fb_destroy() callback. Note that we must not free the device here as this
might race with the parent-device removal. Instead, we rely on
unregister_framebuffer() as barrier and we're safe.

Reported-by: Tom Gundersen <teg@jklm.no>
Signed-off-by: David Herrmann <dh.herrmann@gmail.com>
Acked-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ti.com>