SHA384 in-kernel -- problems found by IPsec

[Anthony Scarpino]

Dan McDonald wrote:
> Hello!
> 
> You should be able to try this at home without BFUing to special bits, folks.
> 
> Consider the addition of SHA-2 (256, 384, and 512 varieties) support to
> IPsec.  It's very simple to enable, just utter (as root) on both machines:
> 
> 	ipsecalgs -a -P 2 -k 256 -b 16 \
> 		-n hmac-sha256,sha256,sha-256,hmac-sha-256 \
> 		-N 5 -m CKM_SHA256_HMAC_GENERAL -s
> 
> and repeat replacing: s/384/256/ s/24/16/ for sha384 and s/512/256/ s/32/16/
> for sha512.  Every ipsecalgs(1M) command enables a SHA-2 HMAC hash per the
> latest RFC (i.e. cut the SHA2 hash in half for use on the wire).
> 
> Now if you manually-key IPsec to use sha256 or sha512, life is good.
> 
> If you use sha384, though, you encounter problems where AH (or ESP auth) will
> fail UNLESS BOTH MACHINES ARE SPARC!  This means x86 vs. x86, sparc vs. x86
> or x86 vs. sparc will fail.
> 
> I suspect it's a kernel crypto problem, but I'll be continuing to look at
> possible IPsec issues.  If anyone in kernel crypto land can confirm/deny a
> possible x86-specific problem on the inbound/verification side of
> HMAC-SHA384, I'd sure appreciate it.
> 
> I'm not ready to file a bug per se yet until I eliminate IPsec, but the
> failures seem to point to a crypto problem.
> 
> Thanks,
> Dan

All these are testing in the kernel by the crypto framework test suite. 
  The test expects both the sparc and x86 results to match, otherwise it 
would fail, and both have been passing fine..

Tony