drm/ttm: fix handling in ttm_bo_add_mem_to_lru

We should not add the BO to the swap LRU when the new mem is fixed and
the TTM object about to be destroyed.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Kevin Wang <kevin1.wang@amd.com>
Link: https://patchwork.freedesktop.org/patch/335246/