serial: sc16is7xx: unconditionally clear line bit in sc16is7xx_remove()

There is no need to check for previous port registration in
sc16is7xx_remove() because if sc16is7xx_probe() succeeded, we are
guaranteed to have successfully registered both ports. We can thus
unconditionally clear the line allocation bit in sc16is7xx_lines.

Signed-off-by: Hugo Villeneuve <hvilleneuve@dimonoff.com>
Reviewed-by: Andy Shevchenko <andy@kernel.org>
Link: https://lore.kernel.org/r/20240409154253.3043822-3-hugo@hugovil.com
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>