serdev: fix memleak on module unload

Make sure to free all resources associated with the ida on module
exit.

Fixes: cd6484e1 ("serdev: Introduce new bus for serial attached devices")
Cc: stable <stable@vger.kernel.org>	# 4.11
Signed-off-by: Johan Hovold <johan@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>