greetings, grand boss!
slave boss has a red carpet rolled for him
the vivisectees forced to be him are honored
this morning i looked a little bit at httptransformers i seem to have stopped trying to port deepseekv3 and instead just try to use my work sometimes with other models where i have left it off i had some notes on resuming inferences, which could increase the fun factor the notes are somewhat obscure, but the upshot is that the approaches i found of interest mostly simplify to figuring out how to replace the forward() function of torch modules to perform resuming for a single forward pass of a hierarchy of modules that has an unknown resume state # we can make a torch model resumable when execution is interrupted, # by recording the outputs between layers and replaying them on next run. (see the check hook in test_nettensors.py) # this is efficient because only the uppermost completed layers need be stored. # meanwhile, resuming a generation call in transformers means simple exchanges like the shorter inputs for the longer recorded ones # it ends up being conditions and wrappers or such for however the user caused the model call. # code could recognize the untokenized string, for example, based on initial inputs inside a resuming context manager. # I WORKED ON THE BELOW FOR FUN WHEN IT WAS REALLY HARD # FEEL FREE TO RESTART / WIPE class ResumeModule: # there is 1 resume state which is where the pass is. # it is updated in every pass after it # and referenced for data in every pass prior. def __init__(self, module, path, seconds=60*5): self.path = path self.state = {} for name, mod in module.named_modules(): module.register_forward_hook(self.__hook) mod.__name = name #self.state [mod. def __enter__(self): def __exit__(self, *params, **kpwarams): return super().__exit__(*params, **kwparams) def _forward_store_resume(self, modid, wrapped, *params, **kwparams): # two cases: we are performing a resume, # or we are storing resume state. # the second is more common. the first could have its own hooks. result = wrapped(*params, **kwparams) return result # ideas for replacing modules within torch spec: # - register_[module_]forward_pre_hook could change the module's behavior, and then another hook put it back after # - manual replacement of forward function, might be similar # - inspection of __call__ source to see default space clearly # __call__ calls _call_impl which uses local variables like forward_call. # _call_impl(self, *args, **kwargs)