pinctrl-bcm2835: don't call pinctrl_gpio_direction()

Set the direction directly without calling pinctrl_gpio_direction().
This avoids the mutex_lock() calls in that function, which would
invalid the can_sleep = false.

Signed-off-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Reviewed-by: Florian Fainelli <f.fainelli@gmail.com>
Link: https://lore.kernel.org/r/20211206131648.1521868-3-hverkuil-cisco@xs4all.nl
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>