X.509: fix error return value on the failed path

When memory allocation fails, an appropriate return value
should be set.

Fixes: 21552563 ("X.509: support OSCCA SM2-with-SM3 certificate verification")
Signed-off-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>