[ot][spam][crazy] adapters for semibalanced trees?

Undiscussed Past Horrific Abuse, One Victim Of Many gmkarl at gmail.com
Sun Jul 24 14:59:45 PDT 2022


I bundled up some inputs (poorly), and extended the tokenizers to process symbols found in code, and continued with the same adapter even though those things had changed.

The loss isn't dropping very fast anymore. There's also a bug I'm running into, where the code I did not write is just hanging and needs a reboot.

I've been considering blaming the tokenization changes, which might be resolved by enabling fine-tuning for the embedding layer of the model. This would increase performance, too. Not sure whether the systems support that.

It could also be the bundling of the input. And if there is a mistake in data generation, that could do it too. It could also be changing the data format on it midway.

Before I started messing with it, the loss had hit .6 or .7 and seemed reasonable for it to drop more. Been hanging at 1.4 now. I feel like it's the new tokens, or how I did them, but I don't know for sure.

Anyway I'm having trouble guiding my body etc around it this afternoon, so I'm likely to just let it keep chugging.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/html
Size: 1107 bytes
Desc: not available
URL: <https://lists.cpunks.org/pipermail/cypherpunks/attachments/20220724/847dc75c/attachment.txt>


More information about the cypherpunks mailing list