regmap: irq: Remove domain on exit

irqdomain now supports removal of domains on exit so we can properly clean
up on deletion of a regmap irqchip.

Signed-off-by: Mark Brown <broonie@linaro.org>