----- Forwarded message from Saso Kiselkov ----- Date: Mon, 07 Oct 2013 00:47:52 +0100 From: Saso Kiselkov To: illumos-zfs Subject: [zfs] [Review] 4185 New hash algorithm support Message-ID: <5251F6A8.2040305@gmail.com> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 Reply-To: zfs@lists.illumos.org Please review what frankly has become a bit of a large-ish feature: http://cr.illumos.org/~webrev/skiselkov/new_hashes/ This webrev implements new hash algorithms for ZFS with much improved performance. There are three algorithms included: * SHA-512/256: truncated version of SHA-512 per FIPS 180-4. Uses all existing code from the sha2 module (with new H(0) consts), but the native 64-bit arithmetic used in SHA-512 provides a ~50% performance boost relative to SHA-256 on 64-bit hardware. * Skein-512: 80% faster than SHA-256 in optimized C implementation, and a very high security margin (Skein was a finalist in SHA-3). Also includes a KCF SW provider. * Edon-R-512: 350% faster than SHA-256 in optimized C implementation. Security margin lower than Skein. To address any security concerns associated with using new algorithms this patch also implements salted checksum support. We store a random 256-bit secret key (the salt) in the MOS and use it to pre-seed the hash algorithms (Skein and Edon-R use this, SHA-512/256 is just a straight hash). Any attacker thus cannot simply mount a collision attack on the algorithm, since they can't completely control the input. ATM I didn't implement support for booting off of pools with salted checksums, thus root pools are limited to sha256 and the new sha512 (new GRUB stage2 needed for sha512 support, obviously). It's possible, but I see fairly little reason in doing so (is anybody running dedup on their rpool and running into hash performance limitations?). For performance and correctness testing I've included a set of tiny test suites in usr/src/common/crypto/{edonr/skein/sha2}/test. Simply cd into the test subdirectory in a shell with your build environment set up and type 'make'. See attached file for an example of the output. To those who will inevitably ask about Keccak/SHA-3, here are my reasons why I didn't integrate it (in decreasing order of significance): * Keccak's software performance is worse than SHA-2. * There's no hardware support for Keccak and there likely never will be, as SHA-3 != Keccak. * SHA-3 has yet to be standardized, and there are even some questions around that, be they tinfoil hat material or otherwise: http://tiny.cc/schneier_sha-3 * To be sure, at some point in the future there will be HW support for SHA-3 (which may or may not be faster than Edon-R in SW), but seeing as how HW vendors are easily backdoored, I'm not convinced we should put a lot of trust in their work: http://tiny.cc/hw_trojans_becker13 Cheers, -- Saso root@illumos-build:...gate.git/usr/src/common/crypto# for TEST in edonr skein sha2; do ( cd $TEST/test ; make ); done Building 32-bit test... Running 32-bit test... Running algorithm correctness tests: Edon-R-224 Message: test_msg0 Result: OK Edon-R-224 Message: test_msg1 Result: OK Edon-R-256 Message: test_msg0 Result: OK Edon-R-256 Message: test_msg1 Result: OK Edon-R-384 Message: test_msg0 Result: OK Edon-R-384 Message: test_msg2 Result: OK Edon-R-512 Message: test_msg0 Result: OK Edon-R-512 Message: test_msg2 Result: OK Running performance tests (hashing 1024 MiB of data): Edon-R-256 2604591 us (6.05 CPB) Edon-R-512 4227055 us (9.81 CPB) Building 64-bit test... Running 64-bit test... Running algorithm correctness tests: Edon-R-224 Message: test_msg0 Result: OK Edon-R-224 Message: test_msg1 Result: OK Edon-R-256 Message: test_msg0 Result: OK Edon-R-256 Message: test_msg1 Result: OK Edon-R-384 Message: test_msg0 Result: OK Edon-R-384 Message: test_msg2 Result: OK Edon-R-512 Message: test_msg0 Result: OK Edon-R-512 Message: test_msg2 Result: OK Running performance tests (hashing 1024 MiB of data): Edon-R-256 2002762 us (4.65 CPB) Edon-R-512 1006284 us (2.34 CPB) Building 32-bit test... Running 32-bit test... Running algorithm correctness tests: Skein_256/256 Message: test_msg0 Result: OK Skein_256/256 Message: test_msg1 Result: OK Skein_256/256 Message: test_msg2 Result: OK Skein_512/512 Message: test_msg0 Result: OK Skein_512/512 Message: test_msg2 Result: OK Skein_512/512 Message: test_msg3 Result: OK Skein1024/1024 Message: test_msg0 Result: OK Skein1024/1024 Message: test_msg3 Result: OK Skein1024/1024 Message: test_msg4 Result: OK Running performance tests (hashing 1024 MiB of data): Skein_256/256 14110264 us (32.76 CPB) Skein_512/512 12465191 us (28.94 CPB) Skein1024/1024 16864123 us (39.15 CPB) Building 64-bit test... Running 64-bit test... Running algorithm correctness tests: Skein_256/256 Message: test_msg0 Result: OK Skein_256/256 Message: test_msg1 Result: OK Skein_256/256 Message: test_msg2 Result: OK Skein_512/512 Message: test_msg0 Result: OK Skein_512/512 Message: test_msg2 Result: OK Skein_512/512 Message: test_msg3 Result: OK Skein1024/1024 Message: test_msg0 Result: OK Skein1024/1024 Message: test_msg3 Result: OK Skein1024/1024 Message: test_msg4 Result: OK Running performance tests (hashing 1024 MiB of data): Skein_256/256 3328342 us (7.73 CPB) Skein_512/512 2549537 us (5.92 CPB) Skein1024/1024 3547695 us (8.24 CPB) Building 32-bit test... Running 32-bit test... Running algorithm correctness tests: SHA256 Message: test_msg0 Result: OK SHA256 Message: test_msg1 Result: OK SHA384 Message: test_msg0 Result: OK SHA384 Message: test_msg2 Result: OK SHA512 Message: test_msg0 Result: OK SHA512 Message: test_msg2 Result: OK SHA512_224 Message: test_msg0 Result: OK SHA512_224 Message: test_msg2 Result: OK SHA512_256 Message: test_msg0 Result: OK SHA512_256 Message: test_msg2 Result: OK Running performance tests (hashing 1024 MiB of data): SHA256 6745601 us (15.66 CPB) SHA512 19033518 us (44.19 CPB) Building 64-bit test... Running 64-bit test... Running algorithm correctness tests: SHA256 Message: test_msg0 Result: OK SHA256 Message: test_msg1 Result: OK SHA384 Message: test_msg0 Result: OK SHA384 Message: test_msg2 Result: OK SHA512 Message: test_msg0 Result: OK SHA512 Message: test_msg2 Result: OK SHA512_224 Message: test_msg0 Result: OK SHA512_224 Message: test_msg2 Result: OK SHA512_256 Message: test_msg0 Result: OK SHA512_256 Message: test_msg2 Result: OK Running performance tests (hashing 1024 MiB of data): SHA256 4551774 us (10.57 CPB) SHA512 3029591 us (7.03 CPB) ------------------------------------------- illumos-zfs Archives: https://www.listbox.com/member/archive/182191/=now RSS Feed: https://www.listbox.com/member/archive/rss/182191/22842876-6fe17e6f Modify Your Subscription: https://www.listbox.com/member/?member_id=22842876&id_secret=22842876-a25d3366 Powered by Listbox: http://www.listbox.com ----- End forwarded message ----- -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://ativel.com http://postbiota.org AC894EC5: 38A5 5F46 A4FF 59B8 336B 47EE F46E 3489 AC89 4EC5