netfilter: nf_tables: reject invalid set policy

Report -EINVAL in case userspace provides a unsupported set backend
policy.

Fixes: c50b960c ("netfilter: nf_tables: implement proper set selection")
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>