[ot][spam][crazy][log] transformer?

Undescribed Horrific Abuse, One Victim & Survivor of Many gmkarl at gmail.com
Thu Oct 6 05:07:48 PDT 2022


2022-10-06T08:29+00:00 Last night I started looking into the “alibi”
algorithm within https://github.com/mosaicml/composer , which is a
plugin-based way to mutate the component of transformers that the
milets paper discusses. I’ve forked it to
https://github.com/xloem/composer , presently into a placeholder
branch named “wip”, and have started laying a little space to put
holographic representations in. I’m referring to the source of
https://github.com/MahmudulAlam/Holographic-Reduced-Representations
for simple and normative ways to implement the core function. It's
quite easy to see the effort as one of simple boilerplate, with these
resources open next to each other.
This morning part of my mind is badly misbehaving, and that’s scary
for me.  I think some of me is terrified about use of machine learning
for oppression. But I think part of me still sees this is very useful.
I have two appointments today that are physically difficult for me to
prepare for.
09:28 I’ve put hrr utility functions into the gpt2 attention
replacement file and changed its attn matmul to be a call to hrr
binding (about 1/3rd of the mutations needed) and pushed to git. I’m
unsure whether to use 2d or 1d calls, but 2d seems a little more
logical, although the paper sums along the sequence dimension so
likely only 1d is needed. I’m also unsure yet how I will test my
implementation: part of the reason for factoring the hrr calls out.
Next step is to learn how gpt tracks the attention mask so as to make
it work for hrr.
0942 I’ve realised to use 1d binding rather than 2d so that token
masks apply correctly. The distraction of this brief incorrectness is
confusing me enough somehow that I’m taking a break to stabilise some.
1204 I’ve started into the second half where the attention output is
produced, and pushed to git. I’m handling a lot of spasmodic thinking,
very hard to stabilise small concepts around the code in my working
memory.  I’m thinking I’ll step away to prepare for my day some.


More information about the cypherpunks mailing list