phy-sun4i-usb: Fix irq free conditions to match request conditions

commit 5cf700ac ("phy: phy-sun4i-usb: Fix optional gpios failing
probe")
changed the condition under which irqs are requested, but omitted matching
changes to sun4i_usb_phy_remove(). This commit fixes this.

Fixes: 5cf700ac

 ("phy: phy-sun4i-usb: Fix optional gpios failing probe")
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com>