pharpend/Vanillae
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5e107a50440d04bc5f1e97ff121cd6fac5c5d525
Vanillae/docs/kek/README.md
T
pharpend 5e107a5044 PIT FALLIN
2023-04-01 02:04:13 -06:00

632 lines
22 KiB
Markdown
Raw Blame History

# Keccak, SHA-3, and SHAKE-N algorithms explained, with code in Erlang
![I can't make a diagram for this one, fellas.[^sponge]](./sponge.jpg)
Keccak is a hashing algorithm used for the SHA-3 standard. [The NIST
standard][nist-standard] is semi-readable math clownery. Hashing algorithms by
nature have to be complicated and somewhat obfuscated, so this is kind of par
for the course.
I found [this lecture][german-lecture] and the [accompanying
notes][german-lecture-notes] indispensable when writing kek.
At the end of the day, all Keccak does is take in some input bits, process them
in a deterministic way, and hand you output bits.
The goal of this document is just to explain what the processing steps are as
straightforwardly as possible.
Keccak is the "general case", and then SHA-3 and SHAKE-128 and so on are
special cases of Keccak.
## tldr
- [Erlang code (clear)](https://github.com/pharpend/kek/blob/8a8a655a80c26ae32763cc25f1e0df8ab0653c82/kek.erl)
- [Erlang code (fast)](https://github.com/pharpend/kek/blob/8a8a655a80c26ae32763cc25f1e0df8ab0653c82/kek_fast.erl)
- [erlang-sha3 library (uses fast version of kek)](https://github.com/zxq9/erlang-sha3/blob/63193654e3c05d8031300ffcd52092f75e8b5c2f/src/sha3.erl#L85-L112)
### References
1. [Helpful lecture][german-lecture]
2. [Notes for that lecture][german-lecture-notes]
3. [NIST standard][nist-standard] (btw: the double bar notation means "concatenate")
4. [SHA-3 Wikipedia](https://en.wikipedia.org/wiki/SHA-3)
### Pitfalls
#### Pitfall: "fast keccak" versus "clear keccak"
The main reference is the "clear" Erlang code. The "fast" version is the
"clear" version with optimizations applied. The fast version was written by
Hans Svensson.
In particular, fast keccak is arity 4, and clear keccak is arity 3.
There are a couple of trivial differences in code structure, so when you're
reading code, you need to keep in mind which version you're looking at. When I
have a code sample, I will link the GitHub permalink in a comment where the
code originates from, so there shouldn't be any confusion.
```erlang
%% From "clear keccak"
%% https://github.com/pharpend/kek/blob/8a8a655a80c26ae32763cc25f1e0df8ab0653c82/kek.erl#L155-L159
-spec keccak(Capacity, Message, OutputBitLength) -> Digest
when Capacity :: pos_integer(),
Message :: bitstring(),
OutputBitLength :: pos_integer(),
Digest :: bitstring().
%% From "fast Keccak"
%% https://github.com/pharpend/kek/blob/8a8a655a80c26ae32763cc25f1e0df8ab0653c82/kek_fast.erl#L142-L147
-spec keccak(Capacity, Message, Delimiter, OutputBitLength) -> Digest
when Capacity :: pos_integer(),
Message :: bitstring(),
Delimiter :: bitstring(),
OutputBitLength :: pos_integer(),
Digest :: bitstring().
```
The reason for this is one of Hans's optimizations was speeding up the manner
in which padding was applied, and that required factoring it out into an
argument.
#### Pitfall: "NIST SHA-3" versus "Keccak SHA-3"
There are two versions of "SHA-3". There's the "NIST version" and the "keccak
version". The only difference is that NIST adds two bits (yes bits, not bytes)
of padding
```erlang
%% From the erlang-sha3 lib, which uses the fast version
%% - ignore the capacity and what not for now, will be explained in a minute
%% - OutputBitLength is the N in SHA3-N. So it would be 512 for SHA3-512, 256
%% for SHA3-256, etc.
%% - Message is the input bitstring to be hashed
%% https://github.com/zxq9/erlang-sha3/blob/63193654e3c05d8031300ffcd52092f75e8b5c2f/src/sha3.erl#L107-L112
kek(OutputBitLength, Message, keccak) ->
Capacity = 2 * OutputBitLength,
keccak(Capacity, Message, <<>>, OutputBitLength);
%% ^ keccak sha3 adds no padding bits
kek(OutputBitLength, Message, nist) ->
Capacity = 2 * OutputBitLength,
keccak(Capacity, Message, <<2#01:2>>, OutputBitLength).
%% ^ nist sha3 concats these padding bits to the end of Message
```
The reason this problem exists is the "keccak version" was used in production
code before the NIST standard was published. So now there are two versions
of "SHA-3" out there.
#### Pitfall: SHAKE-N versus SHA3-N
Note that the padding and sha3-is-keccak-but-its-not pitfall only applies to
the SHA-3 fixed-length hashing functions. These produce a fixed-length (say,
512 bit) hash.
The SHAKE-N algorithms are sort of "arbitrary length SHA-3". So instead of
producing 512 bits, you tell it how long the output length should be. The
SHAKE-N algorithms have their own padding rule, and I don't believe they're
affected by the padding thing.
```erlang
%% From the "clear" version
%% The clear version was written against the NIST standard
%% https://github.com/pharpend/kek/blob/8a8a655a80c26ae32763cc25f1e0df8ab0653c82/kek.erl#L86-L89
sha3(OutputBitLength, Message) ->
Capacity = 2*OutputBitLength,
%% SHA-3 NIST padding is used in the clear version
ShaMessage = <<Message/bitstring, (2#01):2>>,
keccak(Capacity, ShaMessage, OutputBitLength).
%% https://github.com/pharpend/kek/blob/8a8a655a80c26ae32763cc25f1e0df8ab0653c82/kek.erl#L131-L134
shake(ShakeNumber, Message, OutputBitLength) ->
Capacity = 2*ShakeNumber,
%% SHAKE has its own padding rule
ShakeMessage = <<Message/bitstring, (2#1111):4>>,
keccak(Capacity, ShakeMessage, OutputBitLength).
```
#### Pitfall: Greek letter steps require two copies of the sponge to compute
This is the pitfall I ran into, and it stalled this project for months. I
couldn't figure out what the issue was, and the project was only saved because
Hans took the time to figure out the issue and fix it. The nature of hashing
algorithms is that they are all-or-nothing and are hard to decompose. Usually
when your code doesn't work, you can do some trial and error to figure out
which part of the code is wrong. Not so with hashing.
This algorithm has state, which is a 1600-bit bit array called the "sponge".
And that state gets updated a lot. The updating procedure involves a lot of
`xor`ing this bit against that bit, and computing the parity of this string of
bits, etc.
A lot of it involves crawling down the bit array one bit at a time, and xoring
the current bit against certain bits from **the original bit array**, and then
keeping a separate copy that has the modifications.
The mistake I was making was as follows. Let's say that the update to bit 55
requires xoring it against bit 30. It generally requires xoring against the
**un-updated** bit 30. I didn't catch that detail, and was xoring against the
adulterated bit 30.
There's simply no way to find this mistake from trial and error. It's too
subtle.
You can see the commit where Hans fixed my mistake
[here](https://github.com/pharpend/kek/commit/7d67c40e6e1280f4abd4fce9122a71034ebcc142).
## SHA-s and SHAKE-s
These are the "porcelain" functions that we show to the outside world. These
are just rewrites in front of Keccak. Another way to think about it is that
Keccak has tons of settings, and each of these "algorithms" are just different
settings presets.
```erlang
%% From: https://github.com/pharpend/kek/blob/8a8a655a80c26ae32763cc25f1e0df8ab0653c82/kek.erl#L26-L134
-spec sha3_224(Message) -> Digest
when Message :: bitstring(),
Digest :: <<_:224>>.
%% @doc
%% SHA-3 with an output bit length of 224 bits.
%% @end
sha3_224(Message) ->
sha3(224, Message).
-spec sha3_256(Message) -> Digest
when Message :: bitstring(),
Digest :: <<_:256>>.
%% @doc
%% SHA-3 with an output bit length of 256 bits.
%% @end
sha3_256(Message) ->
sha3(256, Message).
-spec sha3_384(Message) -> Digest
when Message :: bitstring(),
Digest :: <<_:384>>.
%% @doc
%% SHA-3 with an output bit length of 384 bits.
%% @end
sha3_384(Message) ->
sha3(384, Message).
-spec sha3_512(Message) -> Digest
when Message :: bitstring(),
Digest :: <<_:512>>.
%% @doc
%% SHA-3 with an output bit length of 512 bits.
%% @end
sha3_512(Message) ->
sha3(512, Message).
-spec sha3(OutputBitLength, Message) -> Digest
when OutputBitLength :: pos_integer(),
Message :: bitstring(),
Digest :: bitstring().
%% @doc
%% SHA-3 with an arbitrary output bit length.
%%
%% This means Keccak with Capacity = 2*OutputBitLength. Additionally, SHA3
%% concatenates the bits 01 onto the end of the input, before sending the
%% Message to keccak/3.
%% @end
sha3(OutputBitLength, Message) ->
Capacity = 2*OutputBitLength,
ShaMessage = <<Message/bitstring, (2#01):2>>,
keccak(Capacity, ShaMessage, OutputBitLength).
-spec shake128(Message, OutputBitLength) -> Digest
when Message :: bitstring(),
OutputBitLength :: pos_integer(),
Digest :: bitstring().
%% @doc
%% This is the SHAKE variable-length hash with Capacity 256 = 2*128 bits.
%% @end
shake128(Message, OutputBitLength) ->
shake(128, Message, OutputBitLength).
-spec shake256(Message, OutputBitLength) -> Digest
when Message :: bitstring(),
OutputBitLength :: pos_integer(),
Digest :: bitstring().
%% @doc
%% This is the SHAKE variable-length hash with Capacity 512 = 2*256 bits.
%% @end
shake256(Message, OutputBitLength) ->
shake(256, Message, OutputBitLength).
-spec shake(ShakeNumber, Message, OutputBitLength) -> Digest
when ShakeNumber :: pos_integer(),
Message :: bitstring(),
OutputBitLength :: pos_integer(),
Digest :: bitstring().
%% @doc
%% This is the SHAKE variable-length hash with Capacity 512 = 2*ShakeNumber bits.
%%
%% This concatenates the bitstring 1111 onto the end of the Message before
%% sending the message to keccak/3.
%% @end
shake(ShakeNumber, Message, OutputBitLength) ->
Capacity = 2*ShakeNumber,
ShakeMessage = <<Message/bitstring, (2#1111):4>>,
keccak(Capacity, ShakeMessage, OutputBitLength).
```
## Outer Keccak
Keccak uses this "sponge" metaphor. If you think of the bits as water, the
state type of Keccak is somewhat analagous to a sponge. In this analogy, the
Keccak algorithm consists of "absorbing" the bits into the sponge, and then
"squeezing" the bits out.
Both the absorption and squeezing phases invoke "inner keccak", which is where
all the real bit-churning happens. (These are the Greek letter steps in the
standard).
`inner_keccak/1` is a 1-arity function, and the input is just the sponge. It
scrambles the bits and does some `xor`ing and what not. The operation of
applying `inner_keccak/1` to a sponge to get a new sponge is called
**kekking**. This is vocabulary I invented and have found useful (you will not
find this in the official docs).
Just pay attention to the flow for now.
- We take in our input (`Message`).
- Add some padding bits to get `PaddedMessage`.
- `InitialSponge` is a 1600-bit long array of `0` bits (the "dry sponge").
- We make the dry `InitialSponge` wet by `absorb/4`ing the `PaddedMessage`, to create `WetSponge`
- We `squeeze/3` the `WetSponge` out to get the `ResultBits`
```erlang
%% https://github.com/pharpend/kek/blob/8a8a655a80c26ae32763cc25f1e0df8ab0653c82/kek.erl#L155-L172
-spec keccak(Capacity, Message, OutputBitLength) -> Digest
when Capacity :: pos_integer(),
Message :: bitstring(),
OutputBitLength :: pos_integer(),
Digest :: bitstring().
%% @doc
%% Note: this is Keccak 1600, the only one used in practice
%%
%% Capacity must be strictly less than 1600
%% @end
keccak(Capacity = _c, Message, OutputBitLength) ->
BitRate = 1600 - Capacity,
PaddedMessage = pad(Message, BitRate),
InitialSponge = <<0:1600>>,
WetSponge = absorb(PaddedMessage, BitRate, Capacity, InitialSponge),
ResultBits = squeeze(WetSponge, OutputBitLength, BitRate),
ResultBits.
```
The padding part is kind of dumb. `absorb/4` and `squeeze/3` both call
`inner_keccak/1`, which like I said is where all the real bit-churning happens.
### Outer Keccak: Padding
The absorption phase is "chunked", meaning
1. it consumes a chunk of the input bits
2. updates the state (the "sponge")
3. if there are no more input bits to consume, absorption is done
4. else, go to step (1)
The size of this chunk is called the `BitRate`. In order for this to work, the
bit length of the input has to be a multiple of `BitRate`. The padding step
exists to make sure that is true.
The padding rule is `/10*1/`, as in the regex. So for instance, suppose our bit
rate was `10`.
```
input (bytes) : 00000111 10101100 01001111
input (grouped into 10) : 0000011110 1011000100 1111______
padded input : 0000011110 1011000100 1111100001
```
The rules are
1. There will *always* be at least two bits of padding added to the end of the
bitstring
2. The first padding bit will be `1`
3. The last padding bit will be `1`
4. *If* there are intermediate padding bits, they will all be `0`
5. The total number of padding bits added is however many makes the length of
the resulting bitstring an integer multiple of the `BitRate`
With that in mind:
```erlang
%% https://github.com/pharpend/kek/blob/8a8a655a80c26ae32763cc25f1e0df8ab0653c82/kek.erl#L176-L234
-spec pad(Message, BitRate) -> NewMessage
when Message :: bitstring(),
BitRate :: pos_integer(),
NewMessage :: bitstring().
%% @private
%% padding
%% divide the message into r-bit blocks
%%
%% the message ends with 1000...0001
%%
%% sha3 calls this /10*1/ as in the regex
%%
%% Reference: https://en.wikipedia.org/wiki/SHA-3#Padding
%% @end
% note: rem will always return a positive integer because both bit_size
% case when the message bit length is evenly divisible by the bit rate
% in this case we add a whole new r-word
pad(Message, BitRate = _r) when (bit_size(Message) rem BitRate) =:= 0 ->
% Suppose the BitRate was 8 and we had 0 bits left
% Input:
% Bits: <<>>
% Idx1: 12345678
% Result:
% Bits: 10000001
% Idx1: 12345678
% In this case we add a new r-word
NewRWord = <<1:1, 0:(BitRate - 2), 1:1>>,
NewMessage = <<Message/bitstring, NewRWord/bitstring>>,
NewMessage;
% this is the retarded case, when the bit length of the message is exactly one
% bit less than dividing the BitRate
pad(Message, BitRate = _r) when (bit_size(Message) rem BitRate) =:= (BitRate - 1) ->
% Suppose the BitRate was 8 and we had 7 bits left
% Input:
% Bits: ABCDEFG
% Idx1: 12345678
% Result:
% Bits: ABCDEFG1 00000001
% Idx1: 12345678 12345678
% in this case, we add a 1, (r-1) zeros, and a trailing 1
NewRWord = <<1:1, 0:(BitRate - 1), 1:1>>,
NewMessage = <<Message/bitstring, NewRWord/bitstring>>,
NewMessage;
% this is the general case, where there are at least 2 bits left in order to
% fill out the r-word
pad(Message, BitRate = _r) ->
% Suppose the BitRate was 8 and we had 3 bits left
% Input:
% Bits: ABC
% Idx1: 12345678
% Result:
% Bits: ABC10001
% Idx1: 12345678
NumberOfMessageBitsInTheLastRWord = bit_size(Message) rem BitRate,
NumberOfNewBitsNeeded = BitRate - NumberOfMessageBitsInTheLastRWord,
NumberOfNewZerosNeeded = NumberOfNewBitsNeeded - 2,
NewMessage = <<Message/bitstring, 1:1, 0:NumberOfNewZerosNeeded, 1:1>>,
NewMessage.
```
### Outer Keccak: Absorption phase
As I mentioned above, absorption is "chunked", and the chunk size is the bit
rate. Absorption comes after padding. So in our absorb procedure, we can
*assume* the length of the `PaddedMessage` is an integer multiple of `BitRate`.
The procedure is
1. Consume `BitRate` bits off the input `PaddedMessage` and put these bits
into `ThisRWord`
2. Bitwise xor `ThisRWord` against the `Sponge`
Let's suppose `BitRate = 10` as before
```
PaddedMessage : 0000011110 1011000100 1111100001
ThisRWord : 0000011110
Sponge : 1111110000 0100011010 1001111010 ... (1600 bits)
AugRWord : 0000011110 0000000000 0000000000 ... (1600 bits)
InnerKekInput : 1111101110 0100011010 1001111010 ... (1600 bits, result of xoring the two previous lines)
```
3. Take the freshly xored sponge and kek it
4. Repeat until you run out of `PaddedMessage` bits.
```erlang
-spec absorb(PaddedMessage, BitRate, Capacity, SpongeAcc) -> WetSponge
when PaddedMessage :: bitstring(),
BitRate :: pos_integer(),
Capacity :: pos_integer(),
SpongeAcc :: <<_:1600>>,
WetSponge :: <<_:1600>>.
%% @private
%% Assumptions:
%% 1. BitRate + Capacity = 1600,
%% 2. BitRate divides the PaddedMessage length (i.e. already have done padding)
%% @end
% can pull off r bits from the start of the message
absorb(PaddedMessageBits, BitRate = _r, Capacity = _c, Sponge) when BitRate =< bit_size(PaddedMessageBits) ->
<<ThisRWord:BitRate, Rest/bitstring>> = PaddedMessageBits,
% we bitwise xor the sponge against the r word followed by a bunch of 0s
<<SpongeInt:1600>> = Sponge,
<<Foo:1600>> = <<ThisRWord:BitRate, 0:Capacity>>,
FInputInt = SpongeInt bxor Foo,
FInputBits = <<FInputInt:1600>>,
NewSponge = inner_keccak(FInputBits),
absorb(Rest, BitRate, Capacity, NewSponge);
% empty string, return the sponge
absorb(<<>>, _r, _c, FinalSponge) ->
FinalSponge.
```
### Outer Keccak: Squeezing phase
Again, the procedure is to "absorb" bits into the algorithm's state (the
"sponge"), and then "squeeze" them out.
The way we squeeze them out is the opposite of the absorption procedure
1. Grab `BitRate` bits off the front of the `WetSponge`
2. Concatenate them to our `ResultBits`
3. If we have enough `ResultBits`, return them
4. Otherwise, re-kek the `WetSponge` and try again.
There is some commented out code here which I'm going to leave in. The reason
is it highlights that this procedure is trivial in the case of the SHA3-N
algorithms. For instance, SHA3-512 has a result length of 512 bits, and the
sponge is always 1600 bits long. So it is a 1-step procedure.
For instance:
```
OutputBitLength = 512
Capacity = 1024 = 2*OutputBitLength
BitRate = 576 = 1600 - Capacity
```
So in the SHA3-N cases, our "squeezing" phase really just amounts to grabbing N
bits off the front of the sponge.
The squeezing phase only requires the re-kekking in the case of the SHAKE-N
algorithms where the `OutputBitLength > BitRate`. Even then, it only requires
iterative re-kekking if the user picks a very long `OutputBitLength` (remember
this is a free parameter in SHAKE-N).
```erlang
%% https://github.com/pharpend/kek/blob/8a8a655a80c26ae32763cc25f1e0df8ab0653c82/kek.erl#L266-L300
-spec squeeze(WetSponge, OutputBitLength, BitRate) -> ResultBits
when WetSponge :: <<_:1600>>,
OutputBitLength :: pos_integer(),
BitRate :: pos_integer(),
ResultBits :: bitstring().
%% @private
%% squeeze the output bits out of the sponge
%% @end
%%% % simple case: bit length is less than (or equal to) the sponge size, just grab
%%% % the first ones
%%% % this is the case for the shas
%%% squeeze(<<ResultBits:OutputBitLength, _Rest/bitstring>>, OutputBitLength, _BitRate) ->
%%% <<ResultBits:OutputBitLength>>;
% general case: output bit length is greater than the sponge size, construct
% accumulatively
% this is the case for the variable-length encodings
squeeze(WetSponge, OutputBitLength, BitRate) ->
InitOutputAcc = <<>>,
really_squeeze(WetSponge, OutputBitLength, BitRate, InitOutputAcc).
% terminal case: we have enough bits in the output, return those
really_squeeze(_WetSponge, OutputBitLength, _BitRate, FinalAccBits) when OutputBitLength =< bit_size(FinalAccBits) ->
<<ResultBits:OutputBitLength, _/bitstring>> = FinalAccBits,
<<ResultBits:OutputBitLength>>;
% general case: need moar bits
% in this case
% - we grab the first r bits of the sponge, add them to the accumulator
% - re-kek the sponge
% - try again
really_squeeze(WetSponge, OutputBitLength, BitRate, ResultAcc)->
<<ThisRWord:BitRate, _/bitstring>> = WetSponge,
NewResultAcc = <<ResultAcc/bitstring, ThisRWord:BitRate>>,
NewWetSponge = inner_keccak(WetSponge),
really_squeeze(NewWetSponge, OutputBitLength, BitRate, NewResultAcc).
```
## Inner Keccak: the kek operation
This is the `f` function that you see in all the documentation. In inner
keccak, the 1600-bit sponge is now thought of as a 5x5x64 3-dimensional array.
Go re-read the [Greek letter pitfall
section](#pitfall-greek-letter-steps-require-two-copies-of-the-sponge-to-compute)
before continuing.
![NIST standard, page 11](./spongecoords.png)
- The input is the 1600-bit sponge.
- The output is a new 1600-bit sponge.
- The sponge is sent through 24 "rounds".
- Each round consists of the 5 greek letter steps.
- Each Greek letter which is a fairly straightforward transformation to the
input bit array.
- The Greek letter steps are called theta, rho, pi, chi, and iota.
- The iota step depends on which round we are in (indexed `0..23`)
```erlang
%% https://github.com/pharpend/kek/blob/8a8a655a80c26ae32763cc25f1e0df8ab0653c82/kek.erl#L324-L368
-spec inner_keccak(Sponge) -> NewSponge
when Sponge :: <<_:1600>>,
NewSponge :: <<_:1600>>.
%% @private
%% the "inner keccak" function, or the 'f' function
%% a bunch of bit bullshit
%% @end
inner_keccak(Sponge) ->
rounds(Sponge, 24).
-spec rounds(Sponge, NumRoundsLeft) -> ResultSponge
when Sponge :: <<_:1600>>,
NumRoundsLeft :: non_neg_integer(),
ResultSponge :: <<_:1600>>.
%% @private
%% do however many rounds
%% @end
% no rounds left
rounds(FinalSponge, 0) ->
FinalSponge;
rounds(Sponge, NumRoundsLeft) ->
% NRoundsLeft = 24
% idx0 = 0
% NRoundsLeft = 1
% idx0 = 23
RoundIdx0 = 24 - NumRoundsLeft,
NewSponge = rnd(RoundIdx0, Sponge),
NewNumRoundsLeft = NumRoundsLeft - 1,
rounds(NewSponge, NewNumRoundsLeft).
-spec rnd(RoundIdx0, Sponge) -> NewSponge
when RoundIdx0 :: 0..23,
Sponge :: <<_:1600>>,
NewSponge :: <<_:1600>>.
%% @private
%% do a single round
%% @private
rnd(RoundIdx0, Sponge) ->
iota(RoundIdx0, chi(pi(rho(theta(Sponge))))).
```
[german-lecture]: https://www.youtube.com/watch?v=JWskjzgiIa4
[german-lecture-notes]: https://www.crypto-textbook.com/download/Understanding-Cryptography-Keccak.pdf
[nist-standard]: https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.202.pdf
[^sponge]: Source for photo: https://www.flickr.com/photos/30478819@N08/46410395345
Reference in New Issue View Git Blame Copy Permalink