core: when parsing resource limits, be more careful with types and corner cases

Let's not convert RLIM_INFINITY to "unsigned long long" and then back to
rlim_t, but let's leave it in the right type right-away.

Parse resource limits as 64 bit in all cases, as according to the man
page that's what libc does anyway.

Make sure setting a resource limit to (uint64_t) -1 results in a parsing
error, and isn't implicitly converted to RLIM_INFINITY.