[ot][spam][crazy] coping spam while trying to make code to prompt a model

Undescribed Horrific Abuse, One Victim & Survivor of Many gmkarl at gmail.com
Sat Jan 7 00:16:15 PST 2023


i think it was because i exhausted gpu ram in that torch session

this fits on a 2GB gpu (and gives poor resullts, which is expected):

import transformers, torch
bloomz = transformers.pipeline('text-generation', 'bigscience/bloomz-560m')
bloomz.model.to(torch.bfloat16)
bloomz.model.to('cuda')
bloomz.device = bloomz.model.device

There are some code around for finding the largest model that fits in
available ram.
Some of mine is at
https://github.com/xloem/test_matrix_bot/blob/main/module_rwkv.py . It
uses this:
MEMORY_BOUND = min(torch.cuda.mem_get_info()[0],
psutil.virtual_memory().available // 2)
You then multiply the parameter count of each model by the datatype
and compare to the memory bound.


More information about the cypherpunks mailing list