[ot][spam][personal] Behavioral/Rambling Brownian Motion towards Subgoals
gmkarl at gmail.com
Mon Jul 19 05:33:18 PDT 2021
Regarding backing up, it will be slow to upload data from a home network to
filecoin. I'm thinking on setting up some way of securing it with a hash
before it uploads. I wonder if there existing tools to make a merkle tree
out of a binary blob.
I think I will make my own, or use libgame's.
I'm thinking of streaming data from a disk, hashing every 32GiB as they
come in. This could be done with a script.
We'd have a format for the hash output. It could either be a single
updating document, or a tree.
This would be a hashed but unsigned document. But maybe I could sign it
anyway so as to quickly verify it with familiar tools.
Alternatively, the various sum tools have a verify mode. They use a
The goal is to reduce the time window during which viruses can mutate data.
I may simply be paranoid and confused. In fact, I am usually these two
Thinking on a chunk of data coming in. Hash the data and give it a name.
That seems reasonable. Then we can make verification files of those files,
I suppose ...
Seems more clear and useful.
This is similar to git-annex ...
Here's a quick way to make a secure document hash:
for sum in /usr/bin/*sum; do $sum --tag $document; done 2> /dev/null
Now, considering filecoin, it would be nice if we also had the deal
The fields of interest include:
- numeric deal id
- payload cid
- piece cid?
- miner address?
- sender address?
- deal cid? hard to find or use this with the interface, seems most
- block hash?
Deal information can be found with `lotus client get-deal`
It seems reasonable to provide the DealID and Proposal Label from client
So. We'll get say a new block of data or something.
If the data is in small blocks, we would want to add this information to
some hash file. Say the DealID.
I guess the deal CID would make more sense in a hash file =S maybe I can
ease myself by using some xxxx looking label. The ProposalCid seems to be
the same as the deal CID, unsure.
I could also put a deal id into the filename, but then the old hashes don't
Thinking a little of a document containing nested documents. It could hash
itself, and append further data, and append a hash of above data ... then
we could add deal ids farther down and still have the same canonical
I looked into using gpg --clearsign, but I don't like how the encoding of
the pgp data makes it hard to debug corruption. The hash is not readily
I'm considering ...
Considering normal tools for breaking a file based on boundaries. Nested
I'm wanting it to be easy to break out the right parts of the files to
verify things, when I am confused. I'd like to be able to verify only some
of the files and hashes, when I am on a system with limited tools. This
means extracting the document's own verification document out, to verify it
Now thinking it is overcomplicated.
Basic goal: easily check that the data I uploaded is the same as what I
read from the drive, at the time I read it. So one hash, and then a way to
check that hash. We download data from the network, maybe years from now,
and check the hash.
I guess it's an okay solution.
Considering using a directory of files.
Then as more documents are added, a hash document can extend in length, or
a tarball can be used.
Considering hashing only certain lines of files. Some way to generate
subfiles from longer files. Generate all the subfiles, and you can check
We could have a filename part that indicates it is lines from another
file. Or greps, even.
While I'm making this confused writing, my system is generate 64GiB of
random data to check if filecoin automatically breaks pieces larger than
its maximum piece size. I should have used zeros, but it's mostly limited
by write speed I expect. It's hit 38GiB so I'll test it.
I told lotus to import the data while it was being generated, hoping it
will import truncated data rather than failing. It's silently spinning the
Regarding a quick hash document, let's see.
Import data in chunks to a folder. Hash data into a new file. Hash all
hashes into a new file.
Maybe that works. Using new files, the system is more clear. How does
We roughly verify files in reverse order to their creation, and we verify
only verifiable files.
So it's possible a system could be checked with something like
check $(ls *.checkable | tac) | grep -v missing
where check runs through /usr/bin/*sum or such
muscle convulsions associated with plans and decisions
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 7716 bytes
Desc: not available
More information about the cypherpunks