2022-10-06 1319+0000 I’m back for a bit after some things. I feel like looking at the code some more. I have a couple hours to my appointments, and am ready enough to go. 1349 I’ve gotten through the attention output. Torch has an existing function for calculating cosine similarity. Cosine similarity is simply the normalized dot product of the vectors: i.e. the cosine of the n-dimensional angle between them. I’ve pushed my draft to git. Next is organisation, testing, and then often a bit of recoding when I find my mistakes. Noting I still have time before my appointments. 1430 I’ve quickly organised the draft a little and tested it imports and constructs without parse failures. Pushed to git. I’m spinning out a little. My appointments are soon. Next is relearning to use this library to test my added algorithm. 1436 Since I copied an existing algorithm almost exactly, I can run all its tests on my copied code. The tests also include an example of using it, which seems the simplest way to learn this. I’m running into pytest crashing weirdly after updating a dependency. 1457 looks like i need to enable the [dev] extra on mosaicml composer to pull in the test dependencies. the tests run now! i have 5 mistakes to resolve. 1503 i pushed a change to git. the vanilla tests are segfaulting on my system associated with tensorflow. my algorithm doesn’t use tensorflow so i drill down into just it. the tests over the algorithm all pass — but having looked at them i know they don’t verify its output, just its structure. next is to try to use the code to do something simple, and compare a result to without the change. my appt is at 1600; i’m in the habit of going a little early to protect against missing it from weird states of mind i can have.

2022-10-07 0935+0000 notes while away: - causal shape exception seems like a sequence length issue elsewhere, sounds fun - exponential decay may need to be updated every epoch - epochs should have multiple batches 1203 it actually looks like using a causal mask here would mean improving upon the research in the paper, or at least understanding the math more in-depth. without already having a working example of anything at all, it would make sense to not use one at first, which kind of means classification or regression data rather than sequential generation data 1229 i glanced through this gpt implementation and it seems to always have the causal mask enabled. i quickly copied the core changes in the gpt algorithm over to bert, untested, pushed to git. excited to switch to bert. it was hard to make the decision. 1425 popping on between stuff to note that causal masking can likely be done simplistically by changing the binding sum to an accumulation. this might broadcast the unbinding, slowing the kernel, but not changing the overall complexity, could be wrong 1636 i’ve patched cumsum into the gpt variant for causal masking, and gotten my test to run. loss actually is reducing steadily on the first run O_O kind of dissociated and pushed to git quickly 1727 in my tiny test (2M tokens of binary addition), i see slightly better results w/out HRR, and faster time w/ HRR. i’m using the cumsum hack and an incredibly tiny gpt model with only 4 params. (i dropped the params to see if the approaches would get more different results; they didn’t) i’m not seeing order of magnitude differences with this tiny setup. 1755 i added layer and separated out the data gen overhead from the actual model passes and there is actually a 60% training speedup with comparable loss, 8 params and the cumsum slowdown. my interest has waned some. i’d like to port t5 since seq2seq models are supposed to perform much better, and org my code, … 1812 https://github.com/mosaicml/composer/issues/1602 editor’s note for visitors not experienced with machine learning: models can have billions of parameters and operate on very complex data, so my different results with only a few params and unimaginably trivial data would not be considered relevant