karl3＠writeme.com wrote:
karl3＠writeme.com wrote:
{{{{{{there was some energy around making a network-based inference engine, maybe by modifying deepseek.cpp (don't quite recall why not staying in python, some concern arose) task got weak, found cinatra as a benchmark leader for c++ web engines (although pico.v was the top! (surprised all c++ http engines were beaten by java O_O very curious about this, wondering if it's a high-end test system) never heard of V language but is interesting it won a leaderboard) inhibition ended up discovering concern somewaht like ... on this 4GB ram system it might take 15-33GB of network transfor for each forward pass of the model ... [multi-token passes ^^ karl3＠writeme.com wrote: the concern resonates with difficulty making the implementation, and some form of inhibition or concern around using python. notably, i've made offloading python hooks a lot and they never last due to the underlying interfaces changing (although those interfaces have stabilized much more now as hf made accelerator as their official implementation) (also i think the issue is more severe dissociative associations than the interface, if one considers the possibility of personal maintenance and use rather than usability for others). don't immediately recall the python concern seem to be off task or taking break, but it would make sense to do disk caching too. there is also option of quantizing. basically, LLMs and AI in general place r&d effort between the user and ease, smallness, cheapness, power, etc
I poked at python again. The existing implementations of the 8 bit quantization used by the model all require an NVidia GPU which I do not presently have. It is fun to imagine making it work, maybe I can upcast it to float32 or something >)