Make unit_name_to_instance() return UnitNameFlags

The function returns non-negative UnitNameFlags on success, and negative
errno on error. In the past we kept the return type as int because of those
negative return values. But nowadays _UNIT_NAME_INVALID == -EINVAL. And if
we tried to actually return something that doesn't fit in the return type,
the compiler would throw an error. By changing to the "real" return type,
we allow the debugger to use symbolic representation for the variables.

(cherry picked from commit 73ce91a0)