IB/hfi: Properly set permissions for user device files

Some of the device files are required to be user accessible for PSM while
most should remain accessible only by root.

Add a parameter to hfi1_cdev_init which controls if the user should have access
to this device which places it in a different class with the appropriate
devnode callback.

In addition set the devnode call back for the existing class to be a bit more
explicit for those permissions.

Finally remove the unnecessary null check before class_destroy

Tested-by: Donald Dutile <ddutile@redhat.com>
Signed-off-by: Haralanov, Mitko <(mitko.haralanov@intel.com)>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
Signed-off-by: Doug Ledford <dledford@redhat.com>