SHA384 in-kernel -- problems found by IPsec

[Dan McDonald]

On Tue, Feb 12, 2008 at 10:29:15AM +0000, Darren J Moffat wrote:
> Unfortunately the HMAC code isn't shared between user and kernel (open bug) 
> so testing in userland for comparison won't help.  The HMAC code for the 
> kernel is in sha2_mod.c.

In in.iked (sorry non-Sun folks - closed source), we don't use EF's HMAC code
in user-land, just the raw hashes.  (in.iked has its own HMAC layer.)

> I find it interesting that CKM_SHA512_HMAC_GENERAL is working but 
> CKM_SHA384_HMAC_GENERAL is not.
>
> I've had a quick look over sha2_mod.c to see how we do things differently 
> from SHA384 vs SHA512 and I don't see anything obvious that would lead the 
> behaviour you saw.

Here's a snippet in sha2_mac_atomic() that disturbs me just for its comment:

	/*
	 * Do an SHA2 update on the outer context, feeding the inner
	 * digest as data.
	 *
	 * Make sure that SHA384 is handled special because
	 * it cannot feed a 60-byte inner hash to the outer
	 */
	if (mechanism->cm_type == SHA384_HMAC_MECH_INFO_TYPE ||
	    mechanism->cm_type == SHA384_HMAC_GEN_MECH_INFO_TYPE)
		SHA2Update(&sha2_hmac_ctx.hc_ocontext, digest,
		    SHA384_DIGEST_LENGTH);
	else
		SHA2Update(&sha2_hmac_ctx.hc_ocontext, digest, sha_digest_len);

SHA384's hash is 48-bytes, not 60.  Also, I notice that the internal variable
sha_digest_len is set to 64 for SHA384, which doesn't seem right even though
there's a blocksize vs. keysize disconnect.  RFC 4634 (intro to SHA-2 for
IETF-types) seems to not disconnect digest_len like this.

I should go interop against other implementations again, but most of them
still do the pre-RFC-4868 truncation to 12-bytes.  But that's an
IPsec-specific problem.

Dan