gpio: grgpio: Avoid potential NULL pointer dereference

irqmap is optional property, so priv->domain can be NULL if !irqmap.
Thus add NULL test for priv->domain before calling irq_domain_remove()
to prevent NULL pointer dereference.

Signed-off-by: Axel Lin <axel.lin@ingics.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>