drm/sysfs: protect sysfs removal code against being run twice.

a step towards correct hot unplug for USB devices, we need to
remove the userspace facing bits at the unplug time for correct
udev operation.

Signed-off-by: Dave Airlie <airlied@redhat.com>