power: gemini-poweroff: Avoid more spurious poweroffs

Even after the previous fix I have experienced more spurious
poweroffs on the gemini SoC. After this fix it finally seems
to go away.

Fixes: f7a388d6 ("power: reset: Add a driver for the Gemini poweroff")
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Sebastian Reichel <sebastian.reichel@collabora.co.uk>