staging: rtl8712: return -ENOMEM instead of -1

After the commit 91d435fe replaced _malloc with kmalloc, smatch picked up a
couple of new warnings. This fixes the warning:

warn: returning -1 instead of -ENOMEM is sloppy

Signed-off-by: Vitaly Osipov <vitaly.osipov@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>