Fix Serpent-AVX2 and Camellia-AVX2 counter modes
authorJussi Kivilinna <jussi.kivilinna@iki.fi>
Sat, 9 Nov 2013 20:39:19 +0000 (22:39 +0200)
committerJussi Kivilinna <jussi.kivilinna@iki.fi>
Sat, 9 Nov 2013 20:39:19 +0000 (22:39 +0200)
commitdf29831d008e32faf74091d080a415731418d158
tree384cc216ede33e8aacb2363de307b171c36ed1d3
parent51501b638546665163bbb85a14308fdb99211a28
Fix Serpent-AVX2 and Camellia-AVX2 counter modes

* cipher/camellia-aesni-avx2-amd64.S
(_gcry_camellia_aesni_avx2_ctr_enc): Byte-swap before checking for
overflow handling.
* cipher/camellia-glue.c (selftest_ctr_128, selftest_cfb_128)
(selftest_cbc_128): Add 16 to nblocks.
* cipher/cipher-selftest.c (_gcry_selftest_helper_ctr): Add test with
non-overflowing IV and modify overflow IV to detect broken endianness
handling.
* cipher/serpent-avx2-amd64.S (_gcry_serpent_avx2_ctr_enc): Byte-swap
before checking for overflow handling; Fix crazy-mixed-endian IV
construction to big-endian.
* cipher/serpent.c (selftest_ctr_128, selftest_cfb_128)
(selftest_cbc_128): Add 8 to nblocks.
--

The selftest for CTR was setting counter-IV to all '0xff' except last byte.
This had the effect that even with broken endianness handling Serpent-AVX2 and
Camellia-AVX2 passed the tests.

Patch corrects the CTR selftest and fixes the broken implementations.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
cipher/camellia-aesni-avx2-amd64.S
cipher/camellia-glue.c
cipher/cipher-selftest.c
cipher/serpent-avx2-amd64.S
cipher/serpent.c