I'm including the entire quote of my previous message below, because david was replying to everything I said but only including the leftmost email, didn't reply when I asked about it, and I don't know why that is. More text following at bottom. On Thu, Jul 1, 2021, 3:30 AM Karl <gmkarl@gmail.com> wrote:
oramfs looks pretty cool and is similar to things I'm working on.
---
other yammer:
Regarding filesystems, I'm struggling to find a way to preserve data across corrupt systems damaging it and harddrives that break frequently. This is very hard for me to solve.
This list still makes no sense to me, no pgp keys, to me people say things that make little sense (spam filtering out of nowhere?). We need to get onto better channels. I wonder what people are using for discourse with more cryptographic integrity nowadays.
I proposed to the lsl project (used for neuroscience research) that they encrypt and authenticate their biosignal streams. I wasn't sure what system to suggest and suggested hypercore because it offers some small proof of creation after the fact They were expecting TLS of course, which I worry around because it doesn't say anything about archival integrity after decryption. Hypercore wasn't really a good suggestion because it is written in nodejs and lsl is in c++ :-/
Seems go and rust are the future. I looked up go.sum : dependencies, although retrieved from github over the network (scary way to make an ecosystem) are hashed via sha256 in a way that can be upgraded (reliable, trustworthy). Inspiring. There are multiple facilities in the go dependency system, for pulling from offline mirrors instead of github, but they aren't that easy to find. Haven't checked if the commit id of dependencies is used in the hash, or the worktree checkout, or what.
Haven't checked rust's cargo to see what their approach is. When picking a language, I want to make sure there are ways to quickly check where data corruption arises. I always use submodules and subtrees for my dependencies in older languages.
It's cool that go's approach pressures people to mirror github to dev offline. That could accomplish a lot in the world, although is likely a little limited to things written in go.
After writing the above I looked into rust a little. Rust stores its cargo.io package index in a single git repository with history. Each package's source bundle is hashed with sha256, although it does not look like the format provides for easily upgrading that algorithm. It is very inspiring that the entire package index can be downloaded and used offline to checksum one's dependencies, as a single repository with history. The format is described a little in https://doc.rust-lang.org/cargo/reference/registries.html . Additionally, executable projects made with rust include sha256sums of their dependencies in their Cargo.lock file, I believe with the intent of providing for deterministic rebuild, unsure. oramfs's dependency checksums can be seen at https://github.com/kudelskisecurity/oramfs/blob/main/Cargo.lock . Library projects that don't build executables do not include the Cargo.lock file, and hence are a little less trustworthy when shared. The format of the hash is the same as in the index and doesn't provide for smoothly adopting a new digest algorithm like go does. I'm curious if go has something like rust's single git package index repository, cause that's pretty nice. Of course git isn't to be trusted for binary files until it adopts newhash, these are ascii hashes not binary data, although technically that means scrubbing the repo to verify that holds which nobody would remember to do. Git will adopt newhash eventually.