minimizing buffer size (i.e. page cache) for bulk copy/rsync -- Re: Swappiness in Buster

[Zenaan Harkness]

Anyone here able to answer this annoying buffer issue on bulk copies?



----- Forwarded message from Zenaan Harkness <zenaan at freedbms.net> -----

From: Zenaan Harkness <zenaan at freedbms.net>
To: debian-user at lists.debian.org
Date: Wed, 8 Jul 2020 17:52:06 +1000
Subject: minimizing buffer size (i.e. page cache) for bulk copy/rsync -- Re:
 Swappiness in Buster

On Wed, Jul 08, 2020 at 08:26:18AM +0200, Martin Reissner wrote:
> For the sake of completeness, I seem to have solved it after some more
> research and it turned out to be systemd, as following bug that was
> reported for Centos7 seems to be applying to Debian Buster as well:
> 
> https://github.com/systemd/systemd/issues/9276
> 
> Luckily the workaround mentioned in:
> 
> https://github.com/systemd/systemd/issues/9276#issuecomment-442514543
> 
> applies here as well and by setting this to the desired swappiness value
> and rebooting the system so far it seems to work as before and swapping
> out is only done if it can't be avoided.


Your thread brings to mind an "$AGE_OF_LINUX_KERNEL + 1 year" years old
Linux kernel bona fydee, bulk copy bug bear:

Doing a bulk copy e.g. using cp or rsync, to copy GiBs far greater than
available RAM from one disk to another, results in ...

  <sinister musak please>
    Total Annihilation of Desktop Latency
   <sinister laugh>

This happens to this day even on the latest Linux kernels on Sid such as
5.7.0-1-amd64 to pick an entirely random example... and has been
literally bugging me since the beginnings of my Debian journey ~20 years
ago.

Note that in this extremely humble user's experience, even

    ionice --class Idle ...

is not even able to quell the Memory Beast's all consuming hunger games
for all possible RAM and then some, in it's insatiable appetite for all
the page cache in...

  <more sinister musak please>
     The  Entire  Universe
   <sinister laugh>


What needs to happen ™©®

What is needed is an option somewhere, strictly abided by, where "For
the following command" only say 1MiB of buffer and Page Cache, in TOTAL,
is able to be used by that command ever, until its completion.

In this way my 16GiBs of RAM will __finally__ be left to its rightful
owners Firefox, mutt, and my XOrg's cursor, since to me as an extremely
humble desktop user, these are the only things in the world that I ever
care about, and I esPECIALLY want to NOT care that a 1,700 GiB movie
disk backup takes an extra 5 or 10 % longer than the NINE hours it
already takes across its USB 2 connection!

Seriously, is there a way to stop bulk copies from eternally flushing my
$Desktop's cached pages down the drain minute after blessful minute,
hour after gratitude filled hour?

The heavens will truly be pleased ..

----- End forwarded message -----