pinctrl: bcm2835: Only return non-GPIOs to inputs

Allowing GPIO state to persist allows the use of gpioset to control
GPIO levels without having to use the --mode=wait feature.

Signed-off-by: Phil Elwell <phil@raspberrypi.com>