core: set fs.file-max sysctl to LONG_MAX rather than ULONG_MAX

Since kernel 5.2 the kernel thankfully returns proper errors when we
write a value out of range to the sysctl. Which however breaks writing
ULONG_MAX to request the maximum value. Hence let's write the new
maximum value instead, LONG_MAX.

/cc @brauner

Fixes: #12803
(cherry picked from commit 6e2f7894)