gpio: gpio-xilinx: Add remove function

Added remove function support.

Signed-off-by: Srinivas Neeli <srinivas.neeli@xilinx.com>
Acked-by: Michal Simek <michal.simek@xilinx.com>
Link: https://lore.kernel.org/r/1605201148-4508-7-git-send-email-srinivas.neeli@xilinx.com


[dropped pm disable call]
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>