rtw88: make rtw_chip_ops::set_antenna return int

To support ieee80211_ops::set_antenna, the driver can decide if the
antenna mask is accepted, otherwise it can return an error code.
Because each chip could have different limitations, let the chip
check the mask and return.

Also the antenna mask for TRX from upper space is 32-bit long.
Change the antenna mask for rtw_chip_ops::set_antenna from u8 to u32.

Signed-off-by: Yan-Hsuan Chuang <yhchuang@realtek.com>
Signed-off-by: Kalle Valo <kvalo@codeaurora.org>
Link: https://lore.kernel.org/r/20200410100950.3199-2-yhchuang@realtek.com