skunks get personal armor with government money [joan of arc asked for accurate spirit information on AI influence/lawyer
---
i'm still working on rep/dict.py !
for a little smidge i didn't have wifi so i made a very simple file-based backend and just now got it working
here's my current crash
File "/home/karl3/projects/rep/rep/dict.py", line 392, in <module> doc.update([[val,val]]) File "/home/karl3/projects/rep/rep/dict.py", line 347, in update super().update(keyhashitems()) File "/home/karl3/projects/rep/rep/dict.py", line 225, in update self.array[:] = IterableWithLength(content_generator(), capacity) ~~~~~~~~~~^^^ File "/home/karl3/projects/rep/rep/array.py", line 56, in __setitem__ self.doc[start * sz : stop * sz] = data ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/karl3/projects/rep/rep/rep.py", line 167, in __setitem__ piece = prefix + data[:] + suffix[:suffixoff] ~~~~^^^ File "/home/karl3/projects/rep/rep/rep.py", line 220, in __getitem__ buf += next(self.iteration) ^^^^^^^^^^^^^^^^^^^^ File "/home/karl3/projects/rep/rep/dict.py", line 218, in content_generator assert superidx * expansion + subidx == int.from_bytes(dbg_keyhash[:hashbytes], 'big') >> hashshift ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError
[stares blankly at crash] i wonder what all this funny text means!
so what's going on here. it's trying to validate this chunk of updates. the code is intended to check things are correct. a chunk of updates is a sequence of keyvals here, which have a keyhash that has a superidx, subidx, and newidx. the code is trying to verify that the newidx is correct by calculating it two ways. 0835
(Pdb) p [superidx, expansion, subidx], [hashbytes, dbg_keyhash[:hashbytes], hashshift, int.from_bytes(dbg_keyhash[:hashbytes], 'big') >> hashshift] ([0, 8, 1], [1, b'\x0c', 5, 0])
the fact that the regenerated newidx is 0 shows that the superidx of 0 is correct. 0837 (Pdb) p newidx2subidx(int.from_bytes(dbg_keyhash[:hashbytes], 'big')
hashshift) 0
the subidx is 0 too. so that shows that this value was likely placed in the wrong index of update_chunk for its content, and i can move an assertion to the previous block ...
so the superidx is 0 that means it's engaging the first item of the original array. the subidx is 8, the hashshift is 5, and the expected final index is 0.
it's another problem with my bitwise arithmetic
(Pdb) list 202 197 assert superidx == newidx >> (hashbits - self._hashbits) 198 #subidx = (newidx >> hashshift) & expansionmask 199 subidx = newidx & expansionmask 200 assert superidx * expansion + subidx == newidx# >> hashshift 201 return subidx 202 for superidx, item in enumerate(tqdm.tqdm(self.array if self._capacity else [self._sentinel], desc='growing sentinel hashtable', leave=False)): 203 update_chunk = [self._sentinel] * expansion 204 if item != self._sentinel: 205 keyhash = self._key(item) 206 newidx = int.from_bytes(keyhash[:hashbytes], 'big') >> hashshift 207 update_chunk[newidx2subidx(newidx)] = item (Pdb) list 213 208 dbg_additions = [] 209 while next_superidx == superidx: 210 item = next_item 211 newidx = next_newidx 212 dbg_additions.append([next_newidx, next_keyhash, next_item]) 213 update_chunk[newidx2subidx(newidx)] = item
somewhere before line 213 here which is where this item is placed. if newidx2subidx() has output that wholly depends on its input, then newidx here must mismatch a shifted mask of item's key. 0839 ... :D
214 next_newidx, next_keyhash, next_item = updates.pop() if len(updates) else [1<<hashbits,None,None] 215 next_superidx = next_newidx >> (hashbits - self._hashbits) 216 for subidx, item in enumerate(update_chunk): 217 dbg_keyhash = self._key(item) 218 -> assert superidx * expansion + subidx == int.from_bytes(dbg_keyhash[:hashbytes], 'big') >> hashshift
not sure if i posted that i separated wholeidx into byteidx and newidx because i was sometimes assuming it was downshifted and othertimes not. [i could have referenced the old code to be consistent, that might have been clearer, but it is a logical computer-checked system anyway
(Pdb) p item b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' (Pdb) p update_chunk [b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', bytearray(b'\xd6\x00\x00\x00\x00\x00\x00\x00\xd8\x00\x00\x00\x00\x00\x00\x00'), b'\xd6\x00\x00\x00\x00\x00\x00\x00\xd8\x00\x00\x00\x00\x00\x00\x00', b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00']
i'm guessing all those 16-long sequences of zeros are bugs in the file backend i tried to add :D
oh no that's the new algorithm where it preallocates a length of sentinels. hmm so expansion is 8, there's identical data here in 0-based slot 2 and 3, that's strange; and one's a bytearray meaning they came from different places