netfilter: xtables: untangle spaghetti if clauses in checkentry

As I'm changing the return values soon, I want to have a clear visual
path.

Signed-off-by: Jan Engelhardt <jengelh@medozas.de>