Anyone here able to answer this annoying buffer issue on bulk copies? ----- Forwarded message from Zenaan Harkness <zenaan@freedbms.net> ----- From: Zenaan Harkness <zenaan@freedbms.net> To: debian-user@lists.debian.org Date: Wed, 8 Jul 2020 17:52:06 +1000 Subject: minimizing buffer size (i.e. page cache) for bulk copy/rsync -- Re: Swappiness in Buster On Wed, Jul 08, 2020 at 08:26:18AM +0200, Martin Reissner wrote:
For the sake of completeness, I seem to have solved it after some more research and it turned out to be systemd, as following bug that was reported for Centos7 seems to be applying to Debian Buster as well:
https://github.com/systemd/systemd/issues/9276
Luckily the workaround mentioned in:
https://github.com/systemd/systemd/issues/9276#issuecomment-442514543
applies here as well and by setting this to the desired swappiness value and rebooting the system so far it seems to work as before and swapping out is only done if it can't be avoided.
Your thread brings to mind an "$AGE_OF_LINUX_KERNEL + 1 year" years old Linux kernel bona fydee, bulk copy bug bear: Doing a bulk copy e.g. using cp or rsync, to copy GiBs far greater than available RAM from one disk to another, results in ... <sinister musak please> Total Annihilation of Desktop Latency <sinister laugh> This happens to this day even on the latest Linux kernels on Sid such as 5.7.0-1-amd64 to pick an entirely random example... and has been literally bugging me since the beginnings of my Debian journey ~20 years ago. Note that in this extremely humble user's experience, even ionice --class Idle ... is not even able to quell the Memory Beast's all consuming hunger games for all possible RAM and then some, in it's insatiable appetite for all the page cache in... <more sinister musak please> The Entire Universe <sinister laugh> What needs to happen ™©® What is needed is an option somewhere, strictly abided by, where "For the following command" only say 1MiB of buffer and Page Cache, in TOTAL, is able to be used by that command ever, until its completion. In this way my 16GiBs of RAM will __finally__ be left to its rightful owners Firefox, mutt, and my XOrg's cursor, since to me as an extremely humble desktop user, these are the only things in the world that I ever care about, and I esPECIALLY want to NOT care that a 1,700 GiB movie disk backup takes an extra 5 or 10 % longer than the NINE hours it already takes across its USB 2 connection! Seriously, is there a way to stop bulk copies from eternally flushing my $Desktop's cached pages down the drain minute after blessful minute, hour after gratitude filled hour? The heavens will truly be pleased .. ----- End forwarded message -----