vhost-user: disable chardev handlers on close

This otherwise causes a use-after-free if network backend cleanup
is performed before character device cleanup.

Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>