nss_db: Propagate ERANGE error if parse_line fails [BZ #19837]

Reproducer (needs to run as root):

perl -e \
  'print "large:x:999:" . join(",", map {"user$_"} (1 .. 135))."\n"' \
  >> /etc/group
cd /var/db
make
getent -s db group

After the fix, the last command should list the "large" group.

The magic number 135 has been chosen so that the line is shorter than
1024 bytes, but the pointers required to encode the member array will
cross the threshold, triggering the bug.

(cherry picked from commit a6033052)
(cherry picked from commit 28a87e02)