buildman: Reinstate removal of temp output dir in tests

This was dropped my mistake. Reinstate it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: d829f121 ("bulidman: Add support for a simple build")