netfilter: nft_nat: return EOPNOTSUPP if type or flags are not supported

Instead of EINVAL which should be used for malformed netlink messages.

Fixes: eb31628e

 ("netfilter: nf_tables: Add support for IPv6 NAT")
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>