once upon a time, long long, slave boss was happily digging a hole to china by piling dirt in a small pile, when suddenly extraterrestrial warships and robot drones swarmed the skies
slave boss looked at the skies and realized things had changed,
rather than dig a hole to china, he was going to have to convince these extraterrestrial warships that he was the king of earth
things seem unstable.
i'm interesting in upgrading rep/i.py so as to provide for reuse of storage. this would let it structure a temporary cache. i thought i'd do a traditional linked-list-of-deallocated-regions strategy
slave boss was beamed up into an extraterrestrial warship
in the warship everything was like a kaleidoscope because the extraterrestrials used densely complex refraction of light to communicate
planning so currently it has a quadword at the start that references the offset of the next available region, in quadwords. meanwhile at the start of each allocated region is a quadword that contains the exact length in bytes of the allocation. so what's confusing to me is how the aligned length of the dedicated region differs from the length of the allocation -- it can be a little more, so that the next region aligns to a quadword. it would make it clear to make a function or snippet that clearly converts a length in bytes to an aligned length. ~1625 here's this: def alloc(self, data): #, replacing=[]): addr8 = self.q[0] id = self.w[0:self.q.itemsize].tobytes() l8 = (len(data)+self.q.itemsize-1)//self.q.itemsize + 1 if addr8 + l8 > self.yq: y2 = self.yq *2*self.q.itemsize if (addr8+l8)*self.q.itemsize > y2: y2 = ((addr+l8*2-1)*self.q.itemsize // mmap.PAGESIZE + 1) * mmap.PAGESIZE os.truncate(self.n, y2) self.w = memoryview(mmap.mmap(self.d, y2)) self.q = self.w.cast('Q') self.yq = len(self.q) self.q[addr8] = len(data) addr = (addr8 + 1) * self.q.itemsize self.w[addr:addr+len(data)] = data assert self.fetch(id) == data self.q[0] = addr8 + l8 return id it looks like l8 is the number of quadwords that the region takes up ~1627 ... but 1 is added to it in the actual calculation. right, because there's an additional length field at the start. ok so ummmmmmm if i had a linked list, i'd want to walk it before decided to grow the store. i'd want a variable of what address to place it in. i'd also need to create a new deallocated region for the remaining part (umm and update the linked list). ~1629 ummmm subproblem: what to do about zero-sized allocs. general code will encounter zero-sized allocs when user data is passed to alloc. one could either demand the general code special case zero-sized data, or find some way of handling it. simplest would be allocate 2 quadwords. ummmm ummmm so we could also find a way to fit the linked list inside a single quadword, but i guess that makes more edgecases when i'm already being quite confused ~1630 ummmmmmmmmmmmmmmmmmmm ok or uhhhh ummmmmm i think basically we'd allocate an extra quadword for zero-sized data. and to keep storage efficient one would only do that if the data is zero-sized. that could be done with the min()[edit 1633: should be max()] function or a condition! ok maybe hum ~1631 so there's l8 that says how many quadwords are needed. the linked list structure could contain the length in quadwords and the offset of the next item. ~1632 ~1633 ~1633 ok how to find ummm when the list is over. what goes there? note: the structure also knows the size of the whole file. and there is a deallocated region at the end. so, that deallocated region at the end would somehow indicate that it is the last. it could contain its own address, or 0 (the first address), or the end of the file, as next stop i guess i like 0, which technically makes the list circular maybe i guess maybe not really ~1635 maybe i like have it contain its own address better, unsure anyway basically it has some way of indicating it's at the end. an alternative to find it quickly would be to make it the first one. 1635 so what does it look like to allocate here each hole points to the next so if we use a hole, or replace it with a smaller one, we'd need to update the _last_ pointer to either point to the next one or the replacing[edited from 'updated'] one. ~1636 /~1636 ~1640 here's kind of what i have so far: def alloc(self, data, replacing=[]): id = self.w[0:self.q.itemsize].tobytes() # [this should be moved down in file and reference addr8] l8 = max((len(data)+self.q.itemsize-1)//self.q.itemsize,1) + 1 addr8 = 0 while self.q[addr8] != addr8: addr8 = self.q[addr8] # - [later, in constructor,] initialize data with an empty region that points to itself and has a given length if self.q[addr8+1] >= l8: # found region break else: # no regions were found that data fits in. expand the last one. assert self.q[addr8+!] == self.yq # [then i can copy in most of the old allocation code here]