mac80211: fix error handling in ieee80211_register_hw()

Fix to return a negative error code -ENOMEM from the kmemdup
error handling case instead of 0.

Fixes: 09b4a4fa ("mac80211: introduce capability flags for VHT EXT NSS support")
Signed-off-by: Wei Yongjun <weiyongjun1@huawei.com>
Signed-off-by: Johannes Berg <johannes.berg@intel.com>