USB: ir-usb: fix double free

If the user specifies a custom bulk buffer size we get a double free at
port release.

Cc: stable <stable@kernel.org>
Signed-off-by: Johan Hovold <jhovold@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>