serial: core: Don't drop port_mutex in serial_core_remove_one_port

Commit 84a9582f ("serial: core: Start managing serial controllers to
enable runtime PM") required the caller to hold port_mutex rather than
taking it locally. However the mutex_unlock() call wasn't removed
causing the mutex to be dropped unexpectly. Remove the call to
mutex_unlock() (and fix up the early return) to restore correct
behaviour.

Fixes: 84a9582f ("serial: core: Start managing serial controllers to enable runtime PM")
Signed-off-by: Steven Price <steven.price@arm.com>
Reviewed-by: Tony Lindgren <tony@atomide.com>
Link: https://lore.kernel.org/r/20230601105548.29965-1-steven.price@arm.com


Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>