software - multiple version installs (any distro developers here?)
Any distro developers here? I just want to float an idea with a couple folks before throwing it out publicly - I'm sure this has been thunked before, but I'd like to see the libre unix software ecosystem move forward on this. Firstly, witness the glory of /opt : $ tree /opt -L 3 /opt ├── bin │ ├── eclipse -> ../e/eclipse/eclipse │ ├── libreoffice -> ../l/libreoffice/program/soffice │ ├── lo -> libreoffice │ ├── pg -> ../p/pg/pg # hypothetical │ ├── soffice -> libreoffice │ └── vuze -> ../v/vuze/latest/vuze/vuze ├── e │ ├── eclipse -> eclipse-4.6 │ ├── eclipse-4.6 │ │ ├── artifacts.xml │ │ ├── configuration │ │ ├── eclipse │ │ ├── icon.xpm │ │ ├── plugins │ │ └── readme │ └── eclipse-themes ├── l │ ├── libreoffice -> libreoffice5.2 │ ├── libreoffice4.8 │ │ ├── CREDITS.fodt │ │ ├── help │ │ └── program │ └── libreoffice5.2 │ ├── CREDITS.fodt │ ├── help │ ├── LICENSE │ ├── LICENSE.html │ ├── presets │ ├── program │ └── share ├── p │ └── pg -> postgresql └── v └── vuze ├── 5600 └── latest -> 5600 So we see this particular /opt/bin is a (admittedly small in my personal example) symlink farm, pointing out to various versions of various installed software. Also, it is structured like a Debian style package pool, with directories {a..z} to contain the various (manually in this case) installed software. With just a smattering of /etc/.. based meta files, .deb meta files and tool awareness, a distro could install all software in this form - say into /usr/a/.. /usr/b/.. etc., and /usr/lib, /usr/bin and /usr/share become symlink farms. The primary benefits I see are: - simple multiple versions install of same software/ library/ etc - different software can easily depend on different versions of other software/ library/ etc, since everything can easily be parallel installed - existing Unixy standards can be more easily maintained (/bin /usr /lib) with symlink farms - end user can much more easily maintain their own custom /opt/bin to override version preferences rather than learning arcane per distro package incantations Main problems: - it's a partially new install/ FHS system (though a multi-year transition shouldn't be particularly painful afaict) - really needs to become part of FHS, and therefore cross-distro - therefore needs buy-in from >1 distro, or at least a non-intrusive transition plan/ capability Does anyone foresee any other particular problems with this concept?
Might take a look at homebrew on OSX ... it's similar to your idea , everything is installed into its own little packaged directory and then symlinked back to /usr/local/{bin,sbin}/.... It beats macports and fink all to fuck. John On August 16, 2016 1:20:39 AM EDT, Zenaan Harkness <zen@freedbms.net> wrote:
Any distro developers here? I just want to float an idea with a couple folks before throwing it out publicly - I'm sure this has been thunked before, but I'd like to see the libre unix software ecosystem move forward on this.
Firstly, witness the glory of /opt :
$ tree /opt -L 3 /opt ├── bin │ ├── eclipse -> ../e/eclipse/eclipse │ ├── libreoffice -> ../l/libreoffice/program/soffice │ ├── lo -> libreoffice │ ├── pg -> ../p/pg/pg # hypothetical │ ├── soffice -> libreoffice │ └── vuze -> ../v/vuze/latest/vuze/vuze ├── e │ ├── eclipse -> eclipse-4.6 │ ├── eclipse-4.6 │ │ ├── artifacts.xml │ │ ├── configuration │ │ ├── eclipse │ │ ├── icon.xpm │ │ ├── plugins │ │ └── readme │ └── eclipse-themes ├── l │ ├── libreoffice -> libreoffice5.2 │ ├── libreoffice4.8 │ │ ├── CREDITS.fodt │ │ ├── help │ │ └── program │ └── libreoffice5.2 │ ├── CREDITS.fodt │ ├── help │ ├── LICENSE │ ├── LICENSE.html │ ├── presets │ ├── program │ └── share ├── p │ └── pg -> postgresql └── v └── vuze ├── 5600 └── latest -> 5600
So we see this particular /opt/bin is a (admittedly small in my personal example) symlink farm, pointing out to various versions of various installed software.
Also, it is structured like a Debian style package pool, with directories {a..z} to contain the various (manually in this case) installed software.
With just a smattering of /etc/.. based meta files, .deb meta files and tool awareness, a distro could install all software in this form - say into /usr/a/.. /usr/b/.. etc., and /usr/lib, /usr/bin and /usr/share become symlink farms.
The primary benefits I see are: - simple multiple versions install of same software/ library/ etc - different software can easily depend on different versions of other software/ library/ etc, since everything can easily be parallel installed - existing Unixy standards can be more easily maintained (/bin /usr /lib) with symlink farms - end user can much more easily maintain their own custom /opt/bin to override version preferences rather than learning arcane per distro package incantations
Main problems: - it's a partially new install/ FHS system (though a multi-year transition shouldn't be particularly painful afaict) - really needs to become part of FHS, and therefore cross-distro - therefore needs buy-in from >1 distro, or at least a non-intrusive transition plan/ capability
Does anyone foresee any other particular problems with this concept?
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Tue, Aug 16, 2016 at 12:48:42PM -0400, John wrote:
Might take a look at homebrew on OSX ... it's similar to your idea , everything is installed into its own little packaged directory and then symlinked back to /usr/local/{bin,sbin}/....
Good, so someone's tried it.
It beats macports and fink all to fuck.
An encouraging data point. In your experience, why is it such a better FHS layout?
On 8/16/16, Zenaan Harkness <zen@freedbms.net> wrote:
[symlinks, versioning and stuff]
I'm sure this has been thunked before
Many times, over many decades, by many people... and been done. Those users and shops have their entire /usr/local or /opt (bin sbin lib include share doc etc and so on) as nothing but script managed symlink trees as means of tagging the active default version. And keep the multiple program-M.m.r versions on disk therein, which are manually callable as needed for new testing and legacy use. With scripts to build, install, and symlink it all. Dependencies, namespace collisions, runtime issues with ld rpath and friends, and the whims of program authors... all tend to buzzkill many efforts there. Github might have something for you. There are also variant symlinks: https://www.dragonflybsd.org/cgi/web-man?command=varsym§ion=1 http://leaf.dragonflybsd.org/cgi/web-man?command=varsym.conf§ion=5 http://leaf.dragonflybsd.org/cgi/web-man?command=varsym§ion=2 https://wiki.freebsd.org/200808DevSummit?action=AttachFile&do=view&target=variant-symlinks-for-freebsd.pdf https://github.com/drq883/varsym https://lwn.net/Articles/680705/ https://github.com/onslauth/varsymfs The "FHS" of a base OS, be it Linux or BSD, typically exclusively reserve /usr/local for their default package manager, and reserve /opt for site use. Some sites prefer /opt first in PATH once it's compiled and installed by competant system admin. Single letter trees for package installs serve no purpose and are anti human. There's no performance need to turn the fs into a binary tree (todays kernels and fs can handle flat dirs many times the needs of any installed package set). Nor need to turn the fs into a database, that's what package managers are for.
I'd like to see the libre unix software ecosystem move forward on this.
Nobody wants such a symlink mess in the base system, ever. And things like pivot root, snapshots, portable processes, rolling release, vm, jail, layered fs, and patchable kernel are all happening. Leaving just handling /usr/local... Most people usually don't have need for multiple versions, just a current version that works. Even packaging just that is hard and a lot of work by a lot of people. Thus port / package tools naturally remain rather simple. Sure there are issues with limited accomodations for versioning / site in package managers. So see /opt above. Or roll your own distro, and packager and reach wide adoption with it.
arcane per distro package incantations
Until then, excepting some tool and port sharing among Linux and BSD's, things are exactly this, and wishful thunking. If troubled on Linux, at least look at the BSD's... their hier's are cleaner and enforced, partly from not going the "distro" model of random thunkers.
On Tue, Aug 16, 2016 at 08:45:16PM -0400, grarpamp wrote:
On 8/16/16, Zenaan Harkness <zen@freedbms.net> wrote:
[symlinks, versioning and stuff]
I'm sure this has been thunked before
Many times, over many decades, by many people... and been done.
Those users and shops have their entire /usr/local or /opt (bin sbin lib include share doc etc and so on) as nothing but script managed symlink trees as means of tagging the active default version. And keep the multiple program-M.m.r versions on disk therein, which are manually callable as needed for new testing and legacy use. With scripts to build, install, and symlink it all.
Awesome! That's exactly what I want. Except I want Debian to do all that for me.
Dependencies, namespace collisions, runtime issues with ld rpath and friends, and the whims of program authors... all tend to buzzkill many efforts there.
A major distro heading that way (e.g. Debian) might encourage developers to increase the configurability of their own software perhaps? Gotta start the improvement somewhere...
Github might have something for you.
I want to establish, re doing this "at the distro level": - whether it's a better way - are their any insurmountable runtime issues And therefore if it all makes sense, to see if there'd be any buy in with developers of a particular distro. Is it generally a better way, say for the large mainstream desktop and server focused distros? In what scenarios is installation direct to /usr/[bin|lib|share|...] actually a distinctly better way?
There are also variant symlinks: https://www.dragonflybsd.org/cgi/web-man?command=varsym§ion=1 http://leaf.dragonflybsd.org/cgi/web-man?command=varsym.conf§ion=5 http://leaf.dragonflybsd.org/cgi/web-man?command=varsym§ion=2 https://wiki.freebsd.org/200808DevSummit?action=AttachFile&do=view&target=variant-symlinks-for-freebsd.pdf https://github.com/drq883/varsym https://lwn.net/Articles/680705/ https://github.com/onslauth/varsymfs
The "FHS" of a base OS, be it Linux or BSD, typically exclusively reserve /usr/local for their default package manager, and reserve
Ahah! Yes so we need a new default packaging/ installation format to be supported by RPM and DPKG, to support the better way, e.g.: /usr/{a..z} has the package install files, and /usr/[bin|lib|share|..] are symlink farms. How hard can it be to support this more rational approach?
/opt for site use. Some sites prefer /opt first in PATH once it's compiled and installed by competant system admin.
These fundamentals have no reason to change. The "/opt" example was simply how I am locally using /opt for custom software installs beyond what debian provides. Also convenient to use .../{a..z} for my offline "setups" (.tar.gz downloads), git clones, documentation (books, URLs I save) etc - each of these {a..z} hierarchies has been inspired primarily by: - Debian's package pools - Daniel Berstein's "universal program namespace" intention which never really took off.
Single letter trees for package installs serve no purpose and are anti human.
Disagree - not an absolute. Check this:
With just a smattering of /etc/.. based meta files, .deb meta files and tool awareness, Debian could install all software in this form - say into /usr/a/.. /usr/b/.. etc. ⋅ There are over 48000 packages in debian. What's just a smattering times 48000? ⋅ Hah! Check this out for anyone on a Debian system: ⋅ /var/lib/dpkg/info/ ⋅ In fact, you might prefer one of the following :) ⋅ ls -l /var/lib/dpkg/info/ | head -n 30 ls /var/lib/dpkg/info/ | wc ⋅ to get a feel for the existing situation in Debian. ⋅ Up to 6 (count 'em: 1, 2, 3, 4, 5 and ... 6 YEAH), six files, get
On Aug 15, 2016 9:15 PM, "Zenaan Harkness" <zen@freedbms.net> wrote: this, per package! ⋅ I find that quite amazing. ⋅ That means that on a test install where all ~48K packages are installed, this one directory would contain roughly 250K files. ⋅ Quite impressive really...
So look into /var/lib/dpkg/info - that's heading for 9K files on my system - and this is a relatively fresh install (<12months)! That's not human friendly. Debian's package pools are definitely more human friendly (says the almighty humble I). But sure, a flat hierarchy may be more human friendly, still with each package in its own /usr/program-v0.7.2/... directory, and still having the usual symlink farms in /usr/bin and /usr/lib etc. It's probably a bike shed issue - my preference is, given the number of packages that get installed in "an average desktop installation", to have an {a..z} layer, as in /usr/{a..z}/program-v0.7.2/...
There's no performance need to turn the fs into a binary tree (todays kernels and fs can handle flat dirs many times the needs of any installed package set). Nor need to turn the fs into a database, that's what package managers are for.
In agreement on this. The point is just multiple versions parallel installs, that's all. This should make a lot of things easier in various ways - Debian "backports" for example, not only custom end-user installs, besides the obvious developer test installs - think: $ git checkout tag-v-3.7.14 $ ./configure DEST=/usr/m/my-program-v-3.7.14 $ make $ make install $ make symlink-as-primary Wouldn't you like it to be "GNU/Linux, the hegemonic OS"?
I'd like to see the libre unix software ecosystem move forward on this.
Nobody wants such a symlink mess in the base system, ever.
Have you never been frustrated fighting the base package system with multiple versions of something you want installed? Tried to install a backports only to end up in a dependency hell where some backport lib must also be installed, and this stuffs up other packages from your base install? Given my experience, your "nobody wants" is too broad by at least 1 :)
And things like pivot root, snapshots, portable processes, rolling release, vm, jail, layered fs, and patchable kernel are all happening.
Leaving just handling /usr/local...
Symlink farms to actual per-version install folders would make "alt 'environments' almost trivial to implement with nothing but a change in PATH and one or two other env vars.
Most people usually don't have need for multiple versions, just a current version that works. Even packaging just that is hard and a lot of work by a lot of people. Thus port / package tools naturally remain rather simple.
Sure there are issues with limited accomodations for versioning / site in package managers. So see /opt above. Or roll your own distro, and packager and reach wide adoption with it.
:) I suspect could be possible to have as a simple plugin/set of rules, for dpkg, which would not conflict with the existing layout, just supplement for those who want a "better backports installation ability". That's the itch that gets me on this one - failed attempts to install backports packages - along with some successful attempts too - but my experience is inconsistent, and it is evident the process is more complex than it needs to be.
arcane per distro package incantations
Until then, excepting some tool and port sharing among Linux and BSD's, things are exactly this, and wishful thunking.
I still think the different distros would not share, in general, the binary packages due to compile flags and preferred versions. Multi version program installs would apply to any distro.
If troubled on Linux, at least look at the BSD's... their hier's are cleaner and enforced, partly from not going the "distro" model of random thunkers.
Yes, a bucket list item for me - my attempt at Red Hat doesn't bode well though - we are creatures of habit. Thanks heaps for the feedback.
On Wed, Aug 17, 2016 at 01:30:17PM +1000, Zenaan Harkness wrote:
A major distro heading that way (e.g. Debian) might encourage developers to increase the configurability of their own software perhaps?
Developers != distro|package maintainers, and free software should be portable and not distro specific. So, developers wouldn't have anything to do here, only maintainers.
Ahah! Yes so we need a new default packaging/ installation format to be supported by RPM and DPKG, to support the better way, e.g.:
So look into /var/lib/dpkg/info - that's heading for 9K files on my system - and this is a relatively fresh install (<12months)!
That's not human friendly.
Because humans are not the intended audience for this stuff. Use dpkg -l [| less or the like].
The point is just multiple versions parallel installs, that's all.
There are already solutions for this, e.g. look at PC-BSDs packages. Or use a container. Or compile yourself and set --prefix accordingly.
Given my experience, your "nobody wants" is too broad by at least 1 :)
You're free to start a new project. Maybe others will join, maybe not. best, Tom
On Aug 17, 2016, at 3:12 AM, Tom <tom@vondein.org> wrote:
On Wed, Aug 17, 2016 at 01:30:17PM +1000, Zenaan Harkness wrote: A major distro heading that way (e.g. Debian) might encourage developers to increase the configurability of their own software perhaps?
Developers != distro|package maintainers, and free software should be portable and not distro specific. So, developers wouldn't have anything to do here, only maintainers.
Ahah! Yes so we need a new default packaging/ installation format to be supported by RPM and DPKG, to support the better way, e.g.:
So look into /var/lib/dpkg/info - that's heading for 9K files on my system - and this is a relatively fresh install (<12months)!
That's not human friendly.
Because humans are not the intended audience for this stuff. Use dpkg -l [| less or the like].
The point is just multiple versions parallel installs, that's all.
There are already solutions for this, e.g. look at PC-BSDs packages. Or use a container. Or compile yourself and
Docker might be a little big for every binary in the system ;). I tend to agree that the base system is ok without being a bunch of symlinks, but doing something like this for /usr/local does work great on OSX with homebrew (which is written in ruby btw) John
On Wed, Aug 17, 2016 at 3:38 AM John Newman <jnn@synfin.org> wrote:
On Aug 17, 2016, at 3:12 AM, Tom <tom@vondein.org> wrote:
On Wed, Aug 17, 2016 at 01:30:17PM +1000, Zenaan Harkness wrote: A major distro heading that way (e.g. Debian) might encourage developers to increase the configurability of their own software perhaps?
Developers != distro|package maintainers, and free software should be portable and not distro specific. So, developers wouldn't have anything to do here, only maintainers.
Ahah! Yes so we need a new default packaging/ installation format to be supported by RPM and DPKG, to support the better way, e.g.:
So look into /var/lib/dpkg/info - that's heading for 9K files on my system - and this is a relatively fresh install (<12months)!
That's not human friendly.
Because humans are not the intended audience for this stuff. Use dpkg -l [| less or the like].
The point is just multiple versions parallel installs, that's all.
There are already solutions for this, e.g. look at PC-BSDs packages. Or use a container. Or compile yourself and
Docker might be a little big for every binary in the system ;). I tend to agree that the base system is ok without being a bunch of symlinks, but doing something like this for /usr/local does work great on OSX with homebrew (which is written in ruby btw)
+1 for Homebrew on Macs, but then you're using a Mac ;-) There's also Nix and its GNU NIH copy Guix, which can be installed in your homedir or /usr/local on most Unix-like systems. They both have the added advantage of having any number of separate "profiles" (i.e. symlink farms) for building packages, doing day-to-day stuff, etc. And since they modify the system monotonically, you can always roll back packages (but not any non-versioned state) semi-atomically, take snapshots (which you could do with ZFS anyway), etc. "Docker for everything" is not a totally off the wall idea. Not specifically docker but namespaces & cgroups for making software that's used to Unix-like discretionary access controls and make it act more like object-capability software. You'd need a UI or shell or something for expressing what capabilities to pass along, for example something that interprets annotated filenames, converts them to filenames in the process's own namespace, and handles mounting the files into the process's namespace. Of course, then you're probably just reinventing SELinux or something, though I do think there's potential there to make the user interface a little less obtuse than SELinux policies.
On August 23, 2016 5:56:39 PM EDT, Sean Lynch <seanl@literati.org> wrote:
On Wed, Aug 17, 2016 at 3:38 AM John Newman <jnn@synfin.org> wrote:
On Aug 17, 2016, at 3:12 AM, Tom <tom@vondein.org> wrote:
On Wed, Aug 17, 2016 at 01:30:17PM +1000, Zenaan Harkness wrote: A major distro heading that way (e.g. Debian) might encourage
to increase the configurability of their own software perhaps?
Developers != distro|package maintainers, and free software should be portable and not distro specific. So, developers wouldn't have anything to do here, only maintainers.
Ahah! Yes so we need a new default packaging/ installation format to be supported by RPM and DPKG, to support the better way, e.g.:
So look into /var/lib/dpkg/info - that's heading for 9K files on my system - and this is a relatively fresh install (<12months)!
That's not human friendly.
Because humans are not the intended audience for this stuff. Use dpkg -l [| less or the like].
The point is just multiple versions parallel installs, that's all.
There are already solutions for this, e.g. look at PC-BSDs
developers packages.
Or use a container. Or compile yourself and
Docker might be a little big for every binary in the system ;). I tend to agree that the base system is ok without being a bunch of symlinks, but doing something like this for /usr/local does work great on OSX with homebrew (which is written in ruby btw)
+1 for Homebrew on Macs, but then you're using a Mac ;-)
I just got dual display functionality working properly on my hackintosh. For years I ran only Linux or FreeBSD at home but I'm afraid I was rather seduced by the combination of Unix and good UI / apps that Mac OSX brought...and when combined with the low price point of a hackintosh (and the big fuck you to capitalism), it's hard to resist... Of course I still use a Linux desktop at work, primary hosted server is freebsd, etc. ..
There's also Nix and its GNU NIH copy Guix, which can be installed in your homedir or /usr/local on most Unix-like systems. They both have the added advantage of having any number of separate "profiles" (i.e. symlink farms) for building packages, doing day-to-day stuff, etc. And since they modify the system monotonically, you can always roll back packages (but not any non-versioned state) semi-atomically, take snapshots (which you could do with ZFS anyway), etc.
Somehow I've never played with this particular piece of software... Will check out.
"Docker for everything" is not a totally off the wall idea. Not specifically docker but namespaces & cgroups for making software that's used to Unix-like discretionary access controls and make it act more like object-capability software. You'd need a UI or shell or something for expressing what capabilities to pass along, for example something that interprets annotated filenames, converts them to filenames in the process's own namespace, and handles mounting the files into the process's namespace. Of course, then you're probably just reinventing SELinux or something, though I do think there's potential there to make the user interface a little less obtuse than SELinux policies.
I don't need /bin/ls and /bin/cp and /bin/[insert simple base util here] wrapped in a container and never found SELinux worth the hassle, but I suppose some people might want that level of headache... ;). John -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Tue, Aug 23, 2016 at 5:33 PM John <jnn@synfin.org> wrote:
On August 23, 2016 5:56:39 PM EDT, Sean Lynch <seanl@literati.org> wrote: ...
"Docker for everything" is not a totally off the wall idea. Not specifically docker but namespaces & cgroups for making software that's used to Unix-like discretionary access controls and make it act more like object-capability software. You'd need a UI or shell or something for expressing what capabilities to pass along, for example something that interprets annotated filenames, converts them to filenames in the process's own namespace, and handles mounting the files into the process's namespace. Of course, then you're probably just reinventing SELinux or something, though I do think there's potential there to make the user interface a little less obtuse than SELinux policies.
I don't need /bin/ls and /bin/cp and /bin/[insert simple base util here] wrapped in a container and never found SELinux worth the hassle, but I suppose some people might want that level of headache... ;).
Sure, the base utilities are *probably* secure, but only because they're mature at this point. But /bin/ls and /bin/cp have access to everything you do, and they an allocate arbitrary amounts of RAM, open network sockets, delete all your files, DoS the whole machine. Not too hard to avoid some of that with ulimit et al, but where do you draw the line at what's a "base utility"? tar? rar? mawk? And what happens the next time someone puts a "bug fix" into ls that causes a security regression? /bin/ls only needs access to your current directory or the directories you type on the command line, and to the inodes of any of the files therein. And then only read access. I suspect that wouldn't be too hard to add as annotations to the arguments themselves, similar to how we have wildcards that are interpreted by the shell now. Automatic named pipes are probably a better analogy. Now I feel like I need to go code up a prototype...
If troubled on Linux, at least look at the BSD's... their hier's are cleaner and enforced, partly from not going the "distro" model of random thunkers.
Yes, a bucket list item for me - my attempt at Red Hat doesn't bode well though - we are creatures of habit.
Thanks heaps for the feedback.
You haven't had the full pleasure of running a UNIX system until you've used FreeBSD :). I prefer it for my own servers every time.... but of course I'm running against the grain with that preference, the day job being a pure redhat shop... I couldn't say whether FreeBSD has as good an end user desktop experience, although I suspect it should, assuming your hardware supported. I think nvidia has an X11 driver for FreeBSD in addition to Linux... I've actually been pretty impressed with redhat 7 and the default gnome 3, after installing the nvidia drivers (I'm not an open source nazi, they work better than nouveau...) John
participants (6)
-
grarpamp -
John -
John Newman
-
Sean Lynch -
Tom -
Zenaan Harkness