pharpend/Vanillae
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

remove all the prose from base N docs

Browse Source 
Reason is that the prose sucked. Too verbose. Needs to be simpler.
This commit is contained in:
pharpend
2022-10-11 21:13:52 -06:00
parent e5709eaf65
commit af12b061c4
1 changed files with 0 additions and 167 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/baseN/README.md
-167
View File
@@ -76,170 +76,3 @@ b64_dec([W, X, Y, Z | Rest], Acc) ->
NewAcc = <<Acc/binary, NW:6, NX:6, NY:6, NZ:6>>,
b64_dec(Rest, NewAcc).
```
## Introduction
This document explains the Base58 and Base64 notations, and the algorithms
for working with them. I wrote this document because I had a fair bit of
difficulty working this out for myself, even with a strong math background.
I couldn't find any resource on the internet explaining all of this simply.
Code examples are given in Erlang and TypeScript. These are the two most
common languages used within the Aeternity project, and both happen to be
languges that make these tasks easy. This document assumes you are familiar
with either/both languages. Even if that's not true, Erlang is a very simple
and clean language, so the code should be pretty self-explanatory if you read
it.
Base64 is kind of annoying but it's pretty straightforward to code. My
initial assumption was that Base58 was in some way "the same" algorithm but
with `n = 58` instead of `n = 64`. When I went to look up the spec, I found
this ([source](https://digitalbazaar.github.io/base58-spec/))
> ### 3. The Base58 Encoding Algorithm
>
> To encode an array of bytes to a Base58 encoded value, run the following
> algorithm. All mathematical operations MUST be performed using integer
> arithmetic. Start by initializing a `zero_counter` to zero (`0x0`), an
> `encoding_flag` to zero (`0x0`), a `b58_bytes` array, a `b58_encoding`
> array, and a `carry` value to zero (`0x0`). For each byte in the array of
> bytes and while `carry` does not equal zero (`0x0`) after the first
> iteration:
>
> 1. If `encoding_flag` is not set, and if the byte is a zero (`0x0`),
> increment the value of `zero_counter`. If the value is not zero `(0x0)`,
> set `encoding_flag` to true `(0x1)`.
> 2. If `encoding_flag` is set, multiply the current byte value by 256 and add
> it to `carry`.
> 3. Set the corresponding byte value in `b58_bytes` to the value of `carry`
> modulus 58.
> 4. Set `carry` to the value of `carry` divided by 58.
>
> Once the `b58_bytes` array has been constructed, generate the final
> `b58_encoding` using the following algorithm. Set the first `zero_counter`
> bytes in `b58_encoding` to `1`. Then, for every byte in `b58_array`, map the
> byte value using the Base58 alphabet in the previous section to its
> corresponding character in `b58_encoding`. Return `b58_encoding` as the
> Base58 representation of the input array of bytes.
I personally have no idea what that does. I found a YouTube video that
explained the Base58 algorithm in a way that made a lot more sense.
([source](https://youtu.be/GedV3S9X89c)). The video gave a clear enough
explanation of the Base58 algorithm that I could _figure out_ what is going on
and why it makes sense. I was able to relate what I was seeing in the video to
background context I happen to have from mathematics. But the video didn't
provide that context.
I want this document to explain what both algorithms do, why they make sense,
how they are different, and why they _have_ to be different. All with code
examples and sufficient mathematical context.
Let's get started.
Any data stored in a computer is represented as an integer. For the purposes of
this discussion, we're going to assume all integers are non-negative (greater
than or equal to 0). The discussion below can easily be modified to accomodate
negative integers. This would add a small amount of annoying complexity in
exchange for no gain in conceptual clarity. Nothing we are doing requires
dealing with negative numbers.
The problem we are interested in is _how do we represent really big integers in
plain text?_.
The first point I want you to take away is that **these are two totally
different solutions**. Do not be fooled by the name. It is **NOT** the
case that these are two instances of the same "Base N"
algorithm, just one is `N = 64` and one is `N = 58`. **These are two totally
different approaches to solving the same problem.**
More precisely, the underlying mathematics behind the two notations is very
similar, but the algorithms for producing them are very different. More detail
later.
Like I said, any given piece of data is---from the perspective of your
computer---just a very big integer. The difference between the two algorithms
is, roughly:
1. The Base64 algorithm thinks of that integer as a "stream of digits"
2. The Base58 algorithm thinks of that integer as a "pure integer," kind of
the way math thinks of an integer: the integer _itself_ is a different
thing than the way the integer is _represented_.
Base64 encoding/decoding involves a straightforward translation back and forth
from the machine representation of integers, without thinking too much (or at
all) about the math involved.
Base58 encoding/decoding requires thinking about the integer from a more mathy
point of view. That weird arcane algorithm above is what happens when you try
to phrase the mathematics in terms of the machine representation of
really big integers.
We're going to focus on the mathy point of view and then circle back to the
weird arcane algorithm later on.
There is actually a good reason we don't use decimal notation for really big
integers: it's extremely wasteful.
I'll explain the following in more detail in a later section. Roll with me. To
any piece of data there is associated a quantity called **information**. The
_unit_ of information is the _bit_, in the same sense that the unit of length
is the meter.
1. There are 256 distinct bytes. A single byte (machine digit)
contains exactly 8 bits $8 = \log_2 256$ of information.
2. There are 10 distinct decimal ("Base10") symbols. A single decimal digit
contains approximately 3.32 bits (`3.32 ~ log2(10)`) of information.
3. There are 64 distinct Base64 symbols. A single Base64 digit contains
exactly $6$ bits (`6 = log2(64)`) of information.
4. There are 58 distinct Base58 symbols. A single Base58 digit contains
approximately 5.86 bits (`5.86 ~ log2(58)`) of information.
What this means is, in base64 notation, each symbol consumes 6 bits of
information, roughly twice the rate of decimal notation (~3.32 bits per
symbol). What this means in practice is that a number written in Base64
notation is about half as long as a number written in decimal notation.
For instance, the number `K = 90 682 877 680 429`
1. requires 14 digits (count them!) in decimal notation
$$
\frac{(\log_2 K) \text{ bits}}
{(\log_2 10) \text{ bits per symbol}}
\approx
\frac{46.37 \text{ bits}}
{ 3.37 \text{ bits per symbol}}
\approx 13.96 \text{ symbols}
$$
2. requires 8 digits in Base64 notation (`UnnAthst`)
$$
\frac{(\log_2 K) \text{ bits}}
{(\log_2 64) \text{ bits per symbol}}
\approx
\frac{46.37 \text{ bits}}
{ 6 \text{ bits per symbol}}
\approx 7.73 \text{ symbols}
$$
3. requires 8 digits in Base58 notation (`i55xNZNt`)
$$
\frac{(\log_2 K) \text{ bits}}
{(\log_2 58) \text{ bits per symbol}}
\approx
\frac{46.37 \text{ bits}}
{ 5.86 \text{ bits per symbol}}
\approx 7.91 \text{ symbols}
$$
As you can see, the difference in space complexity between Base58 and Base64 is
pretty small, but the difference between Base10 is pretty large. Base58 has
the same alphabet (set of symbols) as Base64, minus a handful that can cause
readability or manual input issues. For instance, the Base64 alphabet contains
both the symbol `0` (numeral zero) and `O` (uppercase letter `o`). The Base58
alphabet contains neither.