libertas: fix return value when processing invalid packet

Nothing actually uses the return value yet, but we might as well
make it correct, like process_rxed_802_11_packet() does for the
same case.  Also ensure that if monitor mode is enabled (and
thus process_rxed_802_11_packet() is called) that the debugging
enter/leave functions are balanced.

Signed-off-by: Dan Williams <dcbw@redhat.com>
Signed-off-by: John W. Linville <linville@tuxdriver.com>