serial: core: don't check port twice in a row

There is no need to check port for NULL in uart_port_deref() since we call it
only when port is defined.

There are few places that violate this. Fix them here as well.

Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>