26 Apr
2025
26 Apr
'25
6:50 p.m.
so http://github.com/karl3wm/httptransformer now has a resuming class i can load llama 405b and process 0.00002% of it on my tiny system, then reboot the system and start right off again at 0.00002% and process up to 0.00004% ! :D
you can try this with `python test_nettensor.py` but I _do not recommend doing it unless you are spending the time working on it_ because it is incredibly inefficient. i do expect it to correctly complete prompts if let run for weeks.
it doesn't do any paralleliz-- i tried it on google colab, but it actually ran at about the same speed because of the synchronous streaming nature --
so there's some interest in focusing on speeding up the network portion, the biggest bottleneck