usb: musb: only remove host/udc if it has been added

musb_shutdown() removes always USB host and device.
musb_init_controller() adds host and device depending on port_mode. If
port mode is set to HOST then the removal of UDC leads only to:
|(NULL device *): gadget not registered.
and nothing else happens. If port mode is set to DEVICE and we remove
the host then we oops in usb_remove_hcd().
This patch ensures that we only remove host in OTG/host mode and device
only in OTG/device mode to avoid any trouble.

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Felipe Balbi <balbi@ti.com>