an idea rises in me, of instead uploading data with higher value. i'm guessing that since i am a lone nerd, my high value data is too low value for it to be forthright to claim that it has high value, although it certainly does for me. so, low value data to start with. just guesses. i would indeed rather upload data i value more. kind of a neurotic approach to the task.

another idea is uploading this mailing list history! sounds a little helpful. the gitlake backend breaks it all into 100k blocks to keep it free, so it wouldn't be navigable without a frontend to stitch it together.

i often get distracted by upgrading out of date packages, when i log into a server. i've opened two panes on this one, so i can do that while also doing other things.

i remember now the name of the folder I'm working with

no. no processes are running, nor are any panes in that folder, although i did discover other ongoing tasks, briefly. i'll restart some uploading processes. i don't recall what errors happen, but i do recall it takes a bit for them to happen. i think one of the errors might be upload processes just hanging, doing something.

it was successful to use --debug and pipe stderr to a file. this is good news for me. it will help things i've been doing.

right now it's trying to resume upload of a file chunk that is 34359738368 bytes large (32 MiB). i've identified i have information in me around tuning this upload. i'm uncertain whether or not I have applied the information yet, or what it is. it relates to chunk and subchunk sizes. information: these files are chunked into 32MiB parts. this greatly slows the upload by making metadata files very very large. i have written code that provides for uploading parts that are arbitrarily sized, by subchunking them. this code is being used already in the 32MiB uploads. no, that's not accurate. that's about something different. well, it was helpful to think about a bit. the resuming is going slowly

the log has timestamps that may be helpful in identifying what is making it so slow. but is suspect it is slow because it is checking for the presence of many chunks. rising the chunk size could speed that. 1645 ET the things it is checking for are named like BLAKE2B256-s34359738368-S67108864-C206 and i can visually see that it is pausing within the region where it would check the network for presence, so that is likely what is slow. the names indicate that the remote is configured for a very small chunksize. this is the most helpful area to increase the speed, as this does relate to the metadata size too. the name says the chunks are 67108864 bytes large, probably 64MiB .

this 32GiB file of data i've generated, has 14GiB uploaded already.

in order to check the chunksize, i opened a worktree for the git-annex branch. git-annex keeps its metadata in a dedicated branch, a security/integrity problem i have run into. anyway, this branch is too big to check out at a reasonable speed. i haven't actually finished a checkout. a lighter-weight way to check the contents of the branch would be to use git commands, maybe something like git ls-tree or git cat-file .

nooooope it didn't actually check out on to git commands while that git process tries to remember arithmetic in the face of too many chunks to comprehend

anyway, i guess i'll do more processes with the split logging, and see if i can find a problem. looking at speed and reliability problems sounds helpful.

the last logs are dated march 25th and march 26th

i have nascent scripts on this machine for automating things. i'm not using them. they are still configured to upload to w3 instead of arweave. i would like to upload to both, at some point. (or any two permanent services).

the script looks designed to detect and report failures. i think a second interim script would make sense here.

$! gets the process id of a background process in bash

to not cause errors for myself later, it would help if this script detected termination and didn't leave something ongoing in the background i think that's done with signal handlers

i've started the script, in a further cludgy way i'm curious what the present issues with uploading to web3.storage are

there may also be a bug around downloading of things, which could be important if they are not being uploaded properly. so: - turn off chunking entirely, as found from the remote.log line - look to see if there is a downloading issue - pending afterward: review uploading of large chunks to see if they are resumable. there may also be an inefficiency around the system used to do the upload, it's cache may not be designed for many many files. unsure.

`man git-annex-enableremote`: This command can also be used to modify the configuration of an existing special remote, by specifying new values for parameters that are usually set when using initremote. (However, some settings such as the as the encryption scheme cannot be changed once a special remote has been created.)

the chunk parameter has been set to the empty string and i am restarting the processes uploading low value data

it seems to like chunk=0

the log indicates it is attempting to perform uploads, and it is also iterating through 64MiB chunks that have already been uploaded. guess i'd better see whether it is uploading in chunks or not.

the system has become temporarily nonresponsive during path completion. i was investigating what a process was doing, and tried to tab-complete a temporary file it has open

this kind of stuff can really confuse me. the file doesn't exist but is open. that can happen on linux.

it's getting ECONNABORTED sending a POST request with body possibly taken from such a tmp file

strangely, in the log, many lines are labeled based on the subprocess id that generates them, which you'd think would be usable to uninterleave them, but many of the debug lines are split across many lines, which is strange since their content is relayed based on a line-based protocol where a newline terminates the data it looks intentional though as the extra lines have timestamps

i guess i'll filter one of the failing subprocesses manually

the error output is confusing and too verbose i haven't finished parsing it but it looks like it gives sufficient information to exactly reproduce the post request using curl, for debugging or reporting to a sysadmin if the tmpfile it's reading from has been destroyed, i'd have to access it based on the open process's handle or something

the shell is nonresponsive, which confuses me. it might be network lag.

the host is behaving like the error it reported. my attempt to connect to it is now hanging. i thought i had paused the upload process, so there shouldn't have been new developments, but i could be mistaken. often when my systems do this it is due to thrashing: too much memory allocation

similarly typing things, characters do not show up in the line buffer

i guess i'll do something else now.

On 5/6/22, punk <punks@tfwno.gf> wrote:

hey FUCKING IDIOT, stop sending utter trash to the list.

thank you for your log