karl3＠writeme.com wrote:
karl3＠writeme.com wrote:
karl3＠writeme.com wrote: there's some interest in 'downloading only top k items' [this involves looking at the layer algebra [and coming up with ways to identify low-contributing values. [[we have solved this before possibly/optionally including preprocessing to categorize things top k is more fun! it seems niftier to make ummmmmmmmm so we've got some input logits. these are probably getting multiplied by a _huge_ matrix. we could technically do a naiveish approach of discarding the parts that are multiplied by values near zero. (we could actually consider that each dot product has large values and small values, and skip all values that are smaller than a percentage of the largest values.) - this works much better if we find a way to clump the mask based on locality :/ since http servers like to send regions of bytes not sparse masks - this is really cool if we make like a bayesian or error-labeled datatype, so instead of 3.4 it's more like 3.4+-0.31 this would give much more useful information at the end but yeah it seems interesting to just try the mask! involves some simple torch kernel algebra there's a small space here where one can get the _same exact output_ by predicting that some products would be smaller than the precision of the sum ... this might at least need information on the magnitude of the weights unsure ... ... but there are likely heuristics one could apply here that would be accurate because of the rote nature of the training process and possibly a lack of useful+accurate information one would expect from an overtiny number multiplied by an overlarge one ...
that's kind of more in line with the intent of httptransformer and llm_logits, to be able to work on things like that on your cellphone, but i didn't make llm_logits for this model
ummm i guess i'll look a little at matmul
97 number_passes = math.ceil(weight.mem_usage_frac()) 98 if number_passes == 1: 99 product = torch.matmul(input, weight.fetch(progress=name, validate_usage=False).T) 100 else: 101 rows_at_once = math.ceil(weight.shape[0] / number_passes) 102 -> product = torch.cat([ 103 torch.matmul( 104 input, 105 weight[offset : offset+rows_at_once].fetch(progress=f'row{offset}-{offset+rows_at_once}/{weight.shape[0]}', validate_usage=False).T 106 ) 107 for offset in tqdm.tqdm(range( (Pdb) p input.shape torch.Size([1, 6, 16384]) (Pdb) p input.abs().max(dim=-1) torch.return_types.max( values=tensor([[1.5359, 0.2287, 0.1609, 0.1848, 0.1869, 0.2321]], dtype=torch.float64), indices=tensor([[ 6303, 6303, 14427, 14427, 14427, 205]])) (Pdb) p input.abs().min(dim=-1) torch.return_types.min( values=tensor([[1.0807e-08, 1.3109e-07, 3.4837e-07, 7.3625e-08, 1.0437e-06, 7.8622e-08]], dtype=torch.float64), indices=tensor([[ 78, 6285, 10787, 5347, 10964, 15229]])) (Pdb) p input.dtype torch.float64 so yes it's a float64 but 1e-6 is still a lot smaller than 0.1, aren't you curious what would happen if we only multiplied the largest 16 values? or the largest 1024?