NFS: Don't set NFS_INO_INVALID_XATTR if there is no xattr cache

[ Upstream commit 848fdd62 ]

Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>