Turn more asserts into proper recoverable error paths.

Reported-by:  <syzbot+246b47da27f8e7e7d6fb@syzkaller.appspotmail.com>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>