SHA384 in-kernel -- problems found by IPsec

[Darren J Moffat]

Dan McDonald wrote:
> Hello!
> 
> You should be able to try this at home without BFUing to special bits, folks.
> 
> Consider the addition of SHA-2 (256, 384, and 512 varieties) support to
> IPsec.  It's very simple to enable, just utter (as root) on both machines:
> 
> 	ipsecalgs -a -P 2 -k 256 -b 16 \
> 		-n hmac-sha256,sha256,sha-256,hmac-sha-256 \
> 		-N 5 -m CKM_SHA256_HMAC_GENERAL -s
> 
> and repeat replacing: s/384/256/ s/24/16/ for sha384 and s/512/256/ s/32/16/
> for sha512.  Every ipsecalgs(1M) command enables a SHA-2 HMAC hash per the
> latest RFC (i.e. cut the SHA2 hash in half for use on the wire).
> 
> Now if you manually-key IPsec to use sha256 or sha512, life is good.
> 
> If you use sha384, though, you encounter problems where AH (or ESP auth) will
> fail UNLESS BOTH MACHINES ARE SPARC!  This means x86 vs. x86, sparc vs. x86
> or x86 vs. sparc will fail.

That reeks of an endinanness problem!

> I suspect it's a kernel crypto problem, but I'll be continuing to look at
> possible IPsec issues.  If anyone in kernel crypto land can confirm/deny a
> possible x86-specific problem on the inbound/verification side of
> HMAC-SHA384, I'd sure appreciate it.

While Tony is correct that we test CKM_SHA384_HMAC on SPARC and x86 
against NIST test vectors it appears we don't have a test in the ef 
suite for CKM_SHA384_HMAC_GENERAL.

Unfortunately the HMAC code isn't shared between user and kernel (open 
bug) so testing in userland for comparison won't help.  The HMAC code 
for the kernel is in sha2_mod.c.

I find it interesting that CKM_SHA512_HMAC_GENERAL is working but 
CKM_SHA384_HMAC_GENERAL is not.

I've had a quick look over sha2_mod.c to see how we do things 
differently from SHA384 vs SHA512 and I don't see anything obvious that 
would lead the behaviour you saw.

I think we need a small standalone test case developed (probably by 
someone in the crypto team).


-- 
Darren J Moffat