bt: use size_t type for length parameters instead of int

The length parameter values are not negative, thus use an unsigned
type 'size_t' for them. Many routines pass 'len' values to memcpy(3)
calls. If it was negative, it could lead to memory corruption issues.
Add check to avoid it.

Reported-by: Arash TC <tohidi.arash@gmail.com>
Signed-off-by: Prasad J Pandit <pjp@fedoraproject.org>