arm: Remove vital devices last

Update announce_and_cleanup() to remove all devices, with the vital ones
being removed last.

This is an extra patch on top of the recent RFC:

   http://patchwork.ozlabs.org/project/uboot/list/?series=223280



Signed-off-by: Simon Glass <sjg@chromium.org>