Re: [spam][crazy][fiction][random] Non-Canon MCBoss Spinoffs

27 Feb 2025

      karl3＠writeme.com wrote:
...
karl3＠writeme.com wrote:
...
karl3＠writeme.com wrote:
{{{{{{there was some energy around making a network-based inference engine, maybe by modifying deepseek.cpp (don't quite recall why not staying in python, some concern arose)
task got weak, found cinatra as a benchmark leader for c++ web engines (although pico.v was the top! (surprised all c++ http engines were beaten by java O_O very curious about this, wondering if it's a high-end test system) never heard of V language but is interesting it won a leaderboard)
inhibition ended up discovering concern somewaht like ... on this 4GB ram system it might take 15-33GB of network transfor for each forward pass of the model ... [multi-token passes ^^
karl3＠writeme.com wrote:
the concern resonates with difficulty making the implementation, and some form of inhibition or concern around using python. notably, i've made offloading python hooks a lot and they never last due to the underlying interfaces changing (although those interfaces have stabilized much more now as hf made accelerator as their official implementation) (also i think the issue is more severe dissociative associations than the interface, if one considers the possibility of personal maintenance and use rather than usability for others). don't immediately recall the python concern
seem to be off task or taking break, but it would make sense to do disk caching too. there is also option of quantizing. basically, LLMs and AI in general place r&d effort between the user and ease, smallness, cheapness, power, etc
I poked at python again. The existing implementations of the 8 bit quantization used by the model all require an NVidia GPU which I do not presently have. It is fun to imagine making it work, maybe I can upcast it to float32 or something >)
so i have implemented a little of this.

it creates a model state_dict that lazily loads things off the network when accessed. it's quite fast! some of the layers are larger than my available ram which i have deferred with a plan to resolve via either storing them in mmap'd files or further sharding the modules that use them (ideally both). i did not implement cpu operators for 8 bit quantization but prevented the check from firing to continue testing for now.

the vm i have has 3-4 GB of ram so it's easy to exhaust with this task, at which point it freezes and i have to remotely power it off and on, which it does slowly because it is out of ram and has no swap. kind of a downer when it happens.

the deepseekv3 model implementation allocates a separate buffer for the RoPE position encodings for each layer. i haven't looked closely at the various different implementations that the architecture can configure, but in llama these buffers were algorithmically defined and constant; they're not available over the network but could be written to a file. right now they are exhausting my ram causing the vm thrash freeze, the model has 60+1 layers and each position encoding uses a few hundred megabytes :/ their size is likely proportional to the square of the max input context length, so a quick fix could be to reconfigure the model to expect a much smaller context.

i was trying to patch them to cache their construction so as to share the buffer across layers, but my attempt failed and i froze the vm again :)

late. sleep times. 2326

2400

ok i got to the next milestone. it constructs a model, puts it in a pipeline, all the weights download from the network when used

for it to do anything on this tiny system more code is needed to mmap or subshard large weights.

current code is like this:

import inspect, json, psutil
import accelerate, requests, torch, tqdm, transformers

class Quirks:
    if not torch.cuda.is_available():
        # this avoids FP8 assertions on cpu during placement testing
        torch.cuda.is_available = lambda: True
        torch.cuda.get_device_capability = lambda: [9,0]
    #def unify_rope(model_or_class):
    #    # deepseek generates separate rope buffers for each layer which can use significant memory
    #    deepseek = inspect.getmodule(model_or_class)
    #    cache = {}
    #    def wrap_rope(rope):
    #        def wrapper(*params, **kwparams):
    #            key = tuple(params) + tuple(kwparams)
    #            if key in cache:
    #                return cache[key]
    #            else:
    #                val = rope(*params, **kwparams)
    #                cache[key] = val
    #                return val
    #        return wrapper
    #    for key, val in deepseek.__dict__.items():
    #        if 'Rotary' in key and isinstance(val, torch.nn.Module):
    #            setattr(deepseek, key, wrap_rope(val))

class LazyStateDict(dict):
    def __init__(self, tensor_request_by_name, device):
        super().__init__(tensor_request_by_name)
        self.session = requests.Session()
        self.device = device

    def get_meta_tensor(self, weight):
        return super().__getitem__(weight)[0]

    def __getitem__(self, weight):
        tensor, request = super().__getitem__(weight)
        chunk_size = 1024*128
        with tqdm.tqdm(desc=weight, leave=False, total=tensor.nbytes) as pbar:
            #if tensor.nbytes > psutil.virtual_memory().available / 2:
            #    print(weight, 'is more than half available vram, mmapping a file ...')
            #else:

                # we could also further shard the embeddings and lm_head
                # since embeddings are sparsely used
            assert tensor.nbytes < psutil.virtual_memory().available / 2
            buffer = memoryview(bytearray(tensor.nbytes))

            with self.session.send(request, stream=True) as response:
                while pbar.n < pbar.total:
                    pbar.update(
                        response.raw.readinto(
                            buffer[ pbar.n : pbar.n + chunk_size ]
                        )
                    )

        result = torch.frombuffer(
            buffer,
            dtype = tensor.dtype,
            count = tensor.numel(),
            requires_grad = False,
            device = self.device,
        ).reshape(tensor.shape)

        return result

    def items(self):
        for key in self:
            yield [key, self[key]]

    def values(self):
        for key in self:
            yield self[key]

    def largest(self):
        return max([
            [key, tensor]
            for key, [tensor, _] in super().items()
        ], key = lambda keytensor: keytensor[1].nbytes)[1]

    @staticmethod
    def tensor_request_from_json(url, N, data):
        dtype = data['dtype']
        dtype = dict(
            F32 = torch.float32,
            F8_E4M3 = torch.float8_e4m3fn,
            BF16 = torch.bfloat16,
        )[dtype]
        shape = data['shape']
        tensor = torch.empty(shape, dtype=dtype, device='meta')
        start, end = data['data_offsets']
        start += N + 8
        end += N + 8 - 1
        request = requests.Request('GET', url, dict(Range='bytes='+str(start)+'-'+str(end)))
        request = request.prepare()
        return [tensor, request]

    @classmethod
    def from_user_repo_branch(cls, user, repo, branch, device):
        base_url = f'https://huggingface.co/{user}/{repo}/raw/{branch}/'
        lfs_base_url = f'https://huggingface.co/{user}/{repo}/resolve/{branch}/'

        safetensors_index_url = base_url + 'model.safetensors.index.json'

        print(safetensors_index_url)

        with requests.get(safetensors_index_url, stream=True) as response:
            safetensors_index = json.load(response.raw)

        print(safetensors_index['metadata'])

        fn_by_weight = safetensors_index['weight_map']

        urls = [lfs_base_url + fn for fn in set(fn_by_weight.values())]

        #url_range_dict = {}
        #data_by_weight = {}

        tensor_request_by_name = {}

        with tqdm.tqdm(urls,desc='constructing tensor urls') as pbar:
          for url in pbar:
            # we could potentially also check the git-lfs sha256 from the base url and merklify the data too, this would mean downloading it all
            #[b'version https://git-lfs.github.com/spec/v1', b'oid sha256:e94d32e8649e1a5b03cc0a343c59ca5a6d80d03cd46161b482fd3bb2484adb7d', b'size 4302350824']
            #lfs = dict([ line.decode().split(' ', 1) for line in response.iter_lines() ])
            with requests.get(url, stream=True) as response:
                N = int.from_bytes(response.raw.read(8), 'little')
                header = json.loads(response.raw.read(N))
            for weight, data in header.items():
                if weight == '__metadata__':
                    continue
                #dtype = data['dtype']
                #shape = data['shape']
                #start, end = data['data_offsets']
                #start += headersize + 8
                #end += headersize + 8
                #data_by_weight[weight] = data | {'url':url,'N':headersize}
                tensor_request_by_name[weight] = cls.tensor_request_from_json(url, N, data)
        return cls(tensor_request_by_name, device)

def construct(model_id, device, config_patches = {}, attr_patches = {}):
    user, repo = model_id.split('/',1)
    branch = 'main'

    print(user, repo, branch)

    config = transformers.AutoConfig.from_pretrained(model_id, trust_remote_code=True)
    for key, val in config_patches.items():
        setattr(config, key, val)
    tokenizer = transformers.AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

    with accelerate.init_empty_weights(), transformers.modeling_utils.no_init_weights():
        model = transformers.AutoModelForCausalLM.from_config(config, trust_remote_code=True)
    for key, val in attr_patches.items():
        setattr(model, key, val)

    lazy_state_dict = LazyStateDict.from_user_repo_branch(user, repo, branch, device=device)

    # misuse cpu offloading by providing lazy_state_dict
    model = accelerate.cpu_offload(model, device, state_dict = lazy_state_dict)
    model.hf_device_map = { '': device }

    return transformers.pipeline('text-generation', model=model, config=config, tokenizer=tokenizer)

pipe = construct(
    'deepseek-ai/DeepSeek-V3',
    device = 'cpu',
    config_patches = dict(
        max_position_embeddings = 64, # drop ctx len from 163840 to 64
    ),
    attr_patches = dict(
        _supports_cache_class = False, # might be a bug that this isn't in the model
    ),
)
pipe('Once upon a time,')