serdev: ttyport: add missing open() error handling

Add missing error handling for tty-driver open() which may fail (e.g. if
resource allocation fails or if a port is being disconnected).

Note that close() must be called also in case of failed open() and that
the operation sanity check is amended to catch buggy drivers.

Signed-off-by: Johan Hovold <johan@kernel.org>
Acked-by: Rob Herring <robh@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>