serdev: only match serdev devices

Only serdev devices (a.k.a. clients or slaves) are bound to drivers so
bail out early from match() in case the device is not a serdev device
(i.e. if it's a serdev controller).

Signed-off-by: Johan Hovold <johan@kernel.org>
Reviewed-by: Sebastian Reichel <sebastian.reichel@collabora.co.uk>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>