netfilter: nft_payload: report ERANGE for too long offset and length

Instead of offset and length are truncation to u8, report ERANGE.

Fixes: 96518518

 ("netfilter: add nftables")
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>