Re: [ot][spam][crazy] coping spam while trying to make code to prompt a model

i think it was because i exhausted gpu ram in that torch session this fits on a 2GB gpu (and gives poor resullts, which is expected): import transformers, torch bloomz = transformers.pipeline('text-generation', 'bigscience/bloomz-560m') bloomz.model.to(torch.bfloat16) bloomz.model.to('cuda') bloomz.device = bloomz.model.device There are some code around for finding the largest model that fits in available ram. Some of mine is at https://github.com/xloem/test_matrix_bot/blob/main/module_rwkv.py . It uses this: MEMORY_BOUND = min(torch.cuda.mem_get_info()[0], psutil.virtual_memory().available // 2) You then multiply the parameter count of each model by the datatype and compare to the memory bound.