swapfiles without root
i was 'asked' to generalize my memmapping idea to work for every process on a system. this became relevant for me when doing a large build on an android phone. after brief websearch, my approach was to fork microsoft's 'mimalloc' userspace allocator to provide for memmapping files on use. my fork is in the 'vmem' branch of https://github.com/xloem/mimalloc . it is a quick hack and may abort when subject to weird fork races etc. it also may let you run arbitrarily large processes on phones and raspberry pis. it makes one swapfile per process,and only cleans the swapfile up if the process terminates normally (atext() call). somebody with more systems experience than me could probably speed it up and stabilise it without much work. # to install: git clone https://github.com/xloem/mimalloc cd mimalloc git checkout vmem mkdir build cd build cmake .. make install # to enable: export LD_PRELOAD=/path/to/mimalloc.so VMEM_PREFIX=/path/to/swapfile # now processes run in this shell use per-process swapfiles
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 hello Karl! my replies below, as per usual :) ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, January 4, 2022 12:41 AM, k <gmkarl@gmail.com> wrote:
i was 'asked' to generalize my memmapping idea to work for every process on a system. this became relevant for me when doing a large build on an android phone.
*grin* i remember running full gnuradio toolchain builds on an ARM gumstick. gotta pull out all the stops!
after brief websearch, my approach was to fork microsoft's 'mimalloc' userspace allocator to provide for memmapping files on use. my fork is in the 'vmem' branch of https://github.com/xloem/mimalloc . it is a quick hack and may abort when subject to weird fork races etc. it also may let you run arbitrarily large processes on phones and raspberry pis.
it makes one swapfile per process,and only cleans the swapfile up if the process terminates normally (atext() call).
somebody with more systems experience than me could probably speed it up and stabilise it without much work....
i'll take a look later this week when i have some time. i'm curious to add compression to the file backed memory, as flash i/o is horribly slow, and compression may result in a net gain despite adding CPU load. (new embedded devices have more cores anyway, so this might not be bad) fun stuff, keep it up Karl! best regards, enjoy some nature too :) -----BEGIN PGP SIGNATURE----- iNUEAREKAH0WIQRBwSuMMH1+IZiqV4FlqEfnwrk4DAUCYdR0el8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0NDFD MTJCOEMzMDdEN0UyMTk4QUE1NzgxNjVBODQ3RTdDMkI5MzgwQwAKCRBlqEfnwrk4 DJRkAP48YCOKzaCV5ceEXlwwzLH8cDLjUMgF8fnhUKNKEBeopQEAorJOqWZS7dri Wlo8H5mkPQQf8E3uSDX1sQuRt1kDXQk= =OAiS -----END PGP SIGNATURE-----
hi thank you, it's just a quick hack :) the current big bug is that mem is not copied, rather shared, on fork(), which means some is likely getting a little corrupt. [this may or may not change.] On Tue, Jan 4, 2022, 11:24 AM coderman <coderman@protonmail.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
*grin*
i remember running full gnuradio toolchain builds on an ARM gumstick. gotta pull out all the stops!
yeah. did you just use lots of swap?
somebody with more systems experience than me could probably speed it
up and stabilise it without much work....
i'll take a look later this week when i have some time. i'm curious to add compression to the file backed memory, as flash i/o is horribly slow, and compression may result in a net gain despite adding CPU load. (new embedded devices have more cores anyway, so this might not be bad)
mimalloc intercepts malloc() and free() the vmem branch opens a file and calls mmap() when a page is needed if you like this project, after seeing how simple my changes are you might reimplement them better. fun stuff, keep it up Karl!
best regards, enjoy some nature too :)
:) hope you are well !
-----BEGIN PGP SIGNATURE-----
iNUEAREKAH0WIQRBwSuMMH1+IZiqV4FlqEfnwrk4DAUCYdR0el8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0NDFD MTJCOEMzMDdEN0UyMTk4QUE1NzgxNjVBODQ3RTdDMkI5MzgwQwAKCRBlqEfnwrk4 DJRkAP48YCOKzaCV5ceEXlwwzLH8cDLjUMgF8fnhUKNKEBeopQEAorJOqWZS7dri Wlo8H5mkPQQf8E3uSDX1sQuRt1kDXQk= =OAiS -----END PGP SIGNATURE-----
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, January 4, 2022 5:44 PM, k <gmkarl@gmail.com> wrote: ... [ re: gnuradio builds ] did you just use lots of swap? yes; although i don't remember if it made a difference to use a swapfile or a swap partition... i also had to put a cooler on the device, so that it didn't overheat and die :0 good luck!
there are also still two unmerged branches with attempts to address the forking concern, haven't engaged them further. fixed a bus error. commit 848710ed307c7547f9c822081c4d274ad3c0536c (HEAD -> vmem, xloem/vmem) Date: Tue Feb 8 19:18:14 2022 -0500 fixed bus error for >2GB allocation caused by lack of off_t offset in mi_unix_mmapx
participants (3)
-
coderman -
k -
Undiscussed Horrific Abuse, One Victim of Many