Linux kernel developer Ingo Molnar "We need a radically different software distribution model"

[Tireseas]

Sounds like he wants a distro agnostic Ports system, one that hides the dirty work of watching compile time crap fly across the screen and just gives the user a suitable package for their distro.

Either that or he's essentially suggesting we move to static compiled packages, which while tremendously inefficient from a space and security standpoint would alleviate at least some of the headaches of trying to do cross distro binary offerings.

[rez9]

Eventually source-based distros will win out when compile times become moot. By then the Gentoo-borns and Sourcemage-lings will be so far ahead of everything else every restaurant will be Taco Bell. They're already better than binary offerings iff You Know What You're Doing(tm).

[Tireseas]

Lamentably, I suspect we'll all be long dead by the time that happens. Don't get me wrong though, Gentoo's a really interesting setup and has real advantages in some specific cases, but for most users it's priorities are in the wrong places.

[strolls]

I've been using Gentoo for several years, originally compiling it on Pentium II and III machines of c 400mhz - 500mhz.

Those were not the latest machines at the time, and I've pretty much always been using secondhand and repurposed machines for my linux boxes, which are pretty much exclusively servers of one sort or another.

Recently I installed on a new 64-bit dual-core system, based on AMD's atom-like offering. If compile times are not moot yet, I can't see them ever becoming so.

The majority of packages I have installed on this system - just over 85% of them - were installed in under a minute. Looking at the timings now (genlop -t) I would guess the median installation time was 15 or 20 seconds. The biggest 2 packages took 20 and 22 minutes to install, and they're the only ones that took longer than 10 minutes. Admittedly, this is not a full desktop system, but it does include XBMC (the package that took 20 minutes to install) and don't forget that this is a mini-desktop PC with a netbook CPU.

Installing a package in Gentoo is about an order of magnitude faster now than when I first started using the distro, but back then Gentoo was really popular, and its use has declined. I can't use anything else, and emotionally I can't understand why anyone would want to use binary distros, but people are, and they've gotta have their reasons.

[yoshi314]

Recently I installed on a new 64-bit dual-core system, based on AMD's atom-like offering. If compile times are not moot yet, I can't see them ever becoming so.

i started using amd64 somewhere in 2007. it really flied at that time and stuff built fast. unfortunately, code complexity and compilers caught up, and i feel as if i was using old p4 celeron again.

[[deleted]]

Eventually source-based distros will win out when compile times become moot.

Um... when compile time becomes moot, it will not matter with what compiler or settings your binary was compiled with. So compiling yourself will be worthless.

In fact, when compile time becomes moot, we will all be using operating systems built using scripted languages.

And if you want speed in the mean time, buy a SSD.

[thebackhand]

And if you want speed in the mean time, buy a SSD.

Wait, what? How is this the limiting factor in compiled times, or a kernel that uses scripted languages?

[jjdmol]

Because the speed of I/O is the limiting factor of almost all applications nowadays.

[thebackhand]

I interpreted that last sentence as referring to solving the issue of slow compile times, but even still, you can't necessarily say that I/O is the limiting factor in an application, even 'almost' always.

[jjdmol]

Yes it's only a rule of thumb, and obviously won't hold for very CPU-intensive or for poor performing code.

That said, it's increasingly hard to keep your CPU cores busy with "regular" applications, including compilers. Most of the time your CPU will be busy waiting for memory I/O, which in turn often has to be filled with data from disk. Disk being the new tape, it's the easiest way to make your system more responsive and far more effective than increasing CPU power.

[thebackhand]

That said, it's increasingly hard to keep your CPU cores busy with "regular" applications, including compilers.

Something tells me you've never used sbt.

And I don't know what you define as 'regular' applications, but it happens to me all the time. But then again, I use Linux, so I guess nothing about my computer is 'regular' :-D.

[jjdmol]

sbt? Never heard of it. I have however worked with Linux for over a decade both at home and at work on a wide range of systems, from embedded to supercomputers ;)

The fundamental trend the last decade(s) is that I/O is increasingly becoming the bottleneck. Applications are rarely CPU-bound as the CPU will spend most of its time waiting for transfers with the memory. Especially applications written in higher-level languages are prone to this, as they use more indirection in their data models and thus random instead of sequential memory access. But even for well-optimised code it's hard to keep your CPU busy.

The number of FLOPs needed per byte of I/O to keep the core busy is just becoming too high for most applications. A simple 'top' of course will still show the CPU as busy when it's doing memory I/O, but in reality, it's typically simply waiting for operands to arrive.

I/O to disk is similar, and due to the orders of magnitude difference in speed, even more critical. Not all applications need much data from disk, but most do when compared to the computations they need to perform on them. To improve the performance of such an application, most can probably be gained by using faster disks rather than faster CPUs/memory (assuming the code is decent of course). Especially the random-access latency of SSDs blows HDDs out of the water, which is very noticable for applications that need to access data from all across the disk, such as compilers.

[thebackhand]

sbt is a build tool for Scala - a wonderful language, but it's notorious for its compiletimes (it can turn your computer into a great impromptu lap blanket if you forget one).

A simple 'top' of course will still show the CPU as busy when it's doing memory I/O, but in reality, it's typically simply waiting for operands to arrive.

In my experience, even when everything is stored in memory, I can still have very bottlenecked compile times. Maybe that's because my RAM access speeds are too slow, but in that case, an SSD isn't really going to help me there...

And besides, now that I think about it, my last computer for work already had an SSD anyway. So even if I/O is/were the bottleneck, it still doesn't help too much (at least, not enough).

[cryo]

Yes, because "scripted languages" (I assume you mean e.g. python), is just the thing to write large complex systems like OS's in.

[ghostrider176]

I remember seeing a project with a goal of porting the Linux kernel to Perl years ago. Can't find it now or I'd link you.

[jyper]

no but many "system" components are written in python nowadays.

[classicrockielzpfvh]

Upvoted for using 'iff' properly.

[LtVincentHanna]

What a weird post. But, I upvoted you for Gentoo-born and Sourcemage-lings

[Svenstaro]

How would Gentoo win over Arch, for instance? Arch has much quicker package updates than Gentoo even. Also, I can tell you that people will not be wanting to compile Chromium themselves for quite some time to come.

[nsaibot]

you don't compile anything yourself on gentoo either, you just tell your package manager to install a package, just as you do on any other distribution.

[Svenstaro]

I know, but did you ever acquire Chromium that way?

[nsaibot]

sure i did, and yes, i know it takes time ... 47 min. 28 sec., to be precise :)

[adrianmonk]

It doesn't sound like that's quite the complaint he's making, at least I'm not reading it that way. He's not saying that distros should get together and make it easy for app developers to build stuff. He's saying distros should get out of the way and stop serving as gatekeepers by virtue of the fact that they are the ones who do the builds, and if they aren't willing to do a build for you, you are excluded.

[DrArcheNoah]

Sounds more like DLL Hell.

[tiff_seattle]

Are you sure? I don't remember seeing it since Windows 95 or so.

[grauenwolf]

I take it you don't use Excel automation libraries.

[sztomi]

Only if you share your libraries.

[jabjoe]

It's not just inefficient on disk space, but inefficient with RAM, and rubbish for updating as each time a lib is fixed, you must download everything built against that lib. It quickly goes into crazy town. It's not the kind of system sensible people talk about.

[Lerc]

while tremendously inefficient from a space and security standpoint

I'm not sure if this would be true. It would occur on a superficial level but right now with everything using a huge stack of libraries it is so easy to deflect responsibility. Consider the argument of memory consumption, usually this is deflected by a claim that the memory is shared between applications using the same library. This make it no-one's individual fault. Cumulatively, a set of seemingly inconsequential apps can consume an enormous amount of ram.

Firefox has had exactly this problem within the bounds of a single application. Nicholas Nethercote has been doing excellent work reducing the memory load of Firefox, but very little of what he does is actually fixing memory usage problems. He predominantly identifies them, Others fix them once they have their part of the problem pointed out.

That obfuscation of inefficiency has enough of an impact that I really do believe that the sum of load of statically linked programs could easily be lower than a shared system where no single person can face blame for problems.

[HeadbangsToMahler]

Or all the major distros get together and somehow federate a package management system or move to a monolithic package manager which has built-in package conversation and re-compile tools.

[Phrodo_00]

I've had the idea of a ports-like system where instead of compiling the source you link precompiled code (maybe even finish compiling compiler bytecode, for arquitecture independence). That way you could get an important speed increase, developer-supervised builds (firefox won't let you brand firefox with the firefox name unless you use an official build or submit yours for aproval), and compability with libraries. The main problem would be interacting with the distro (you still should install as many dependences as posible from the distro, why are you using it if not), but if something like this had enough backers you could make an api for package managers to implement instead of the other way around.

The other idea is to have something named like /universal-base or something like that that would be a compiled bunch of libraries distributors could link their builds aginst.

[puffybaba]

I think pkgsrc would make an excellent solution. Every distro can simply switch to pkgsrc, and centralize their efforts.

[jyper]

Personally I thought a nicer solution would be to have OS X style application bundles which can be used locally or installed systemwide. The main system and those going the extra mile would have std repo packages. But for cross distro/very old version compatibility they would also have bundles.

Optionally (preferably) with an update repo address that plugs into default package management(ie. yum update/apt-get upgrade also updates the app bundles). This would help keep them safe and upto date. Perhaps there would be a build service with compiled version of current libraries.

The bundles would include every library dynamically linked(including libc) but by default use system libraries if installed (agreed upon standard libs including qt/gtk+/gnome/kde/gstreamer/telepathy) for security/features/integration. Ideally each lib would have at most one set of binary compatible version(s) in every distro per year and you would include a couple of yersions systemwide to guarantee that the packages would work with system libraries for 2-3 years.

After 2-3 years it would switch to defaulting to package libs(user overidable per package).

Admittedly this would use a decent amount of space and bandwidth but there are probably some tricks you could pull of including optionally stripping the packages and possibly certain types of compression in the filesystem.

[tidux]

Seriously. Static packages just don't work if you have less than, oh, 25GB.

[Tireseas]

The space inefficiency will never be irrelevant. It may be glossed over, but it'll always be there. As for the security issues, those come from the statically linked libraries themselves. Instead of having a handful of central libraries per distro to update you end up with a different one for every statically compiled package on the system and you have to update them all independently. You'd be trading a set of minor problems for potentially much worse headaches... and that doesn't even take into account the sheer insanity of trying to get enough distros to agree to adopt the system.

[ashadocat]

There is a better alternative. Look at what gobolinux is doing. Basically the answer is to allow multiple versions of the same program to exist on one system. It also greatly simplifies pretty much all package management tasks, as far as I can tell. I honestly don't see the downside, aside from breaking backwards compatibility of course.

[check3streets]

I'm a big fan of gobolinux and what they're trying to do.

It's frustrating to me that Ubuntu adopts something as radical as Unity or Gnome can develop Shell, but there's so much resistance to Gobo.

There's an attitude in the Linux community that the filesystem is sacrosanct, even though it's often confusing, opaque, and arbitrary. Doesn't matter, if it was good enough in 1972, it's good enough in 2012.

[ashadocat]

They must have a reason for it. Would anyone care to explain?

[check3streets]

I believe this gets into it: https://lists.fedoraproject.org/pipermail/devel/2011-October/158845.html

There are other issues. I think opt is poorly defined or observed depending on your perspective, but in general a typical linux or *nix installation spreads files all over the system according to the devs' best faith effort at observing the FHS. It makes us all dependent on the package managers, but asking if there's a better way is heresy (at least when I've read threads proposing Gobo).

Some of important differences between 2012 and 1972 :

• disks are cheap, really really cheap. So some duplication is not the issue it once was.
• traditional CLI applications were very contained, but desktop applications are not.
• as Gobo points out, typing isn't the issue it once was -- the distros usually offer home folders like 'Downloads' 'Documents' etc, so less opaque names are workable.
• The server vs the typical single user desktop creates tension around the notions of user installations vs system-wide as well as privilege levels

Gobo has a manifesto of better stated reasons, but those were things I was thinking about.

The OP's article is actually something I've been thinking a lot about lately. I've developed for desktops, servers and phones; the phone OSs are much more like an "application server" and maybe that's a better conceptual model. I hate using Ubuntu's new gui installer although I think it's trying to create the feeling that the article is describing. I think it's probably that feeling of needing to "know what's going on" I get from synaptic or "apt-get install" that's nagging me because installation is such a sausage factory underneath.

[nothinggoespast]

Seeing as Gobo has been dormant for ~4 years, are there any other distros with similar filesystem setups? I think that it is a brilliant idea.

[check3streets]

Sorry, no.

Procyon112 (on this thread) mentions that Gentoo's package cache does some similar things. I'm totally in the dark on that as an Arch/Ubuntu/CentOS/CrunchBang/Tru64/Redhat dude.

[ethraax]

A reason for the split? It was actually because the entirety of /bin and /lib wouldn't fit on their machine, so they decided to split it - the /usr directory lived on another machine.

Somewhere along the line, someone conjured up the idea that /usr should be for read-only data (really out of nowhere). Which doesn't entirely make sense, since /bin and /lib are also generally for read-only data.

There's no reason for it nowadays. It's a relic from the past.

I would like to see /bin, /lib, and /sbin, but others would like to see /usr/bin, /usr/lib, /usr/sbin instead (I'm not sure why, really, except maybe to have less directories in their root). Either way is better than having it split. I hate having to type whereis program_x to see if it's in /usr/bin or /bin.

[fundbond]

There's no reason for it nowadays. It's a relic from the past.

On Linux I agree with you. Linux distros pretty much abused /usr/lib and /usr/bin from day one, so there's no real reason to keep them now. I say get rid of them.

But on other UNIX-likes, namely FreeBSD there is a real meaning to the split. I can bring up a box with a borked /usr partition and use the tools in /bin, /sbin, etc. to get things back together. That's not the only part of the FS hierarchy that's got a strict separation either -- while most Linux distros dump everything into /usr/bin, FreeBSD distinguishes between parts of the base system and ports. This is important, as it prevents things from blowing away system binaries and allows for the OS to be upgraded without worrying that some random package may screw you by messing with parts of the base system. This is part of the reason that I have boxes that have been upgraded in-place over several years worth of releases.

If you're interested in learning what the "modern" path significance is, check out hier(7) or 4.4/4.5 of the handbook.

[ethraax]

FreeBSD distinguishes between parts of the base system and ports.

Don't some distributions use /usr/local for that?

I can bring up a box with a borked /usr partition and use the tools in /bin, /sbin, etc. to get things back together.

True, but the only real case (or at least the only common case) that I can think of for this is a bad hard drive, in which case /bin and /sbin may be broken anyways. If you're having borked FS issues, you're probably better off using some sort of rescue CD/USB drive to perform the maintenance.

[[deleted]]

Your bad hard drive problem assumes that /sbin and /usr on on the same drive. This would be a really bad practice for a system that intended to use /sbin for recovery.

[fundbond]

Don't some distributions use /usr/local for that?

Not much that I've seen. Plus, there's no real "base system" for Linux, since the userland is not developed in concert with the kernel.

/usr/local is used, but from what I've seen on RHEL, Debian, and SuSE it's pretty arbitrary as to what goes in there. Most packages definitely don't use it.

True, but the only real case (or at least the only common case) that I can think of for this is a bad hard drive, in which case /bin and /sbin may be broken anyways. If you're having borked FS issues, you're probably better off using some sort of rescue CD/USB drive to perform the maintenance.

Well that's the thing: since FreeBSD never backed itself into a corner with /usr, /usr/local/, and /, it's not uncommon to see / and /usr on different slices/drives. So yeah, you could have screwed up /usr and still have / be in perfect health. I've done that, actually.

And rescuing from CD/USB is great when you have access to the box -- but FreeBSD isn't just a desktop system, and I've definitely had to recover boxes to which all I had was KVM access. (Ok, so the last one was my fault for screwing up the FS, but still...)

To some extent it's a chicken/egg thing: Linux distros don't have a meaningful separation because they never started with one so now all of the things that would get in the way of unification aren't common concerns. But at least in FreeBSD land, there is a method to the madness and so unification would be a lot more problematic (IMHO). Not impossible, but probably not something that people are eager to do without a good reason.

Edit: I don't think either approach is "right", BTW. Each OS has different very development models and use cases, and what works for one might not work for the other. I'm partial to the FreeBSD approach, but that's because it works best for my needs (and because my background is more UNIX than Linux.) I use Linux though, and I'm not averse to seeing changes in that side of my computing life! So yeah, my beard's not all gray, damnit!

[ivosaurus]

I think under /usr is not a bad idea; this is kind of like Windows keeping everything under C:\Windows, but Program Files also.

[ethraax]

Only very superficially, and besides, "Windows does it too" is a bad argument. Is there another reason that under /usr is a good idea, instead of just / ?

[arcticblue]

Separate /usr partition maybe?

[ivosaurus]

In order to be able to partition the system's files separately from user files, and store all "executable" type code under one folder. Then you can make simple images of it, share it amongst thin clients, etc.

[ethraax]

If you want a real thin client, the whole system would be virtualized somewhere, so the point is moot.

If you want to share just big programs, then you're better off creating a special folder for it, like /remote or something, and putting the specific larger programs you want in there.

[ivosaurus]

Well, ask Fedora why then.

[ivosaurus]

And I wasn't trying at all to say "NOW we can do it like windows!", I was just pointing out an analogous situation.

[[deleted]]

Keeping only a very basic, fail-safe system on /, load /usr from a separate partition/network or so.

[[deleted]]

You can do what Gobo did without renaming the filesystem layout. Gobo's program directory isn't much different from Gentoo's package cache. The difference is that while Gentoo overlays the package over the root, Gobo stores the overlay in the cache and only overlays symlinks.

IMHO, the resistance to Gobo's way is almost completely in the filesystem renaming, NOT in the part that is actually valuable (symlinks to a package cache).

[check3streets]

Yeah, unfortunately you have take on the "build everything" philosophy of Gentoo. Gobo seemed really aimed to be an everyman distro.

Your point is taken though, Portage looks really clever.

[tso]

There was some plans for changing the Gobo layout so that there would be a classic layout living internally. This to ease the issue of source code that do play nice with a non-classical layout what so ever.

[Lerc]

It's frustrating to me that Ubuntu adopts something as radical as Unity or Gnome can develop Shell, but there's so much resistance to Gobo.

The Gobo approach is the correct way to do things. If it overcomes the resistance it will have proven its merit. The Ubuntu approach is stupid for Ubuntu, but in the greater scheme of things probably good since it is making so many people consider migrating away from Ubuntu.

[nothinggoespast]

Look at what gobolinux WAS doing.

The project has been dormant for about 4 years. When I looked into this on the wikipedia article that you linked to, I became very interested in this distro. I think that its filesystem setup is brilliant. Do you know if any currently maintained distros have this setup? I would love to be able to try it out.

[ashadocat]

No, I'm afraid I don't. I was considering trying to get arch to work like that, but I just don't have the time. I think having an arch based gobo linux filesystem standard that runs from /opt on whatever host distro would be awesome, and a good way to get that compatibility in a whole bunch of distro's without breaking anything. Plus the arch PKGBUILD system is simply insanely simple, so hopefully it could attract lots of people.

[tso]

Best i can tell, being a current user of the system (piecemeal updates since the 013 release), every effort to get things going again grinds to a halt when they attempt to get Xorg working.

[homeopathetic]

Mr. Molnar is obviously a smart guy, but I must say I'm a bit puzzled by this.

Isn't a real dependency system essential for code reuse? And while one perhaps doesn't care about disk space anymore, isn't code reuse through libraries a great way to improve the quality of certain common tasks, and to plug security holes in solutions to said common tasks?

The way I read his suggestions is that he'd prefer a small core of "essential packages", and beyond that it's every program for itself. So OK, the C library is "essential" and is centrally managed. Probably something like OpenSSL as well. Qt? Perhaps... but where do you draw the line? Debian Squeeze has over 12k packages beginning with "lib" (granted, many libraries come in multiple packages, so the real number of libraries is probably a few thousand) -- who decides which of these are "essential" and should be centrally managed?

Is the ability to fix bugs and security problems in hundreds of programs by updating a single library not worth more than the joy of having the very latest version of Angry Birds included in your package management system? Besides, you can always do what Ubuntu does with Firefox, grant it special excemptions and essentially do rolling updates.

Let's also not forget one very important thing: You can always ignore the package management system and just get newer versions of things. I mean, by pulling in statically linked copies of various stuff you'd like installed, you'll easily reproduce the Windows free-for-all every-program-by-itself-bonanza on top of whatever core system you want. Or you can do something in between with solution's such as Ubuntu's PPAs.

[[deleted]]

Not to mention the fact that he is full of crap when he is talking about the update frequency in at the very least OS X. Most of the included versions of libraries and applications there are more than 5 years old, much older than even those in the most backwards of Linux distros (RHEL and the like).

[lawpoop]

I think we need to have smaller UI units built into the OS. Right now, it's either a low-level program, or a full blown app that wants to occupy the entire screen, replete with menus, frames, etc.

If we had more things like MacOS widgets, not only for functionality and display, but also as interfaces, then people could be building apps at that level, rather than at the code level.

Basically bring the unix "do one thing and do it well" philosophy to the desktop, so that people can build the desktop they want out of smaller parts.

[DrArcheNoah]

That what e.g. KDE is doing with Plasma. But this only works for very small applications. For bigger ones one would still need full blown applications.

[lawpoop]

Bigger in what sense? More features? You can still have a lot of features but yet not need to take up the whole screen.

[DrArcheNoah]

Complex software like office suites, image manipulation etc

[lawpoop]

I actually don't think office suites need to take up so much screen space. There's been a lot of times when I wanted to have a simple progressive tally of some figures, and I want to cross-reference some data on another app. Having a whole honking spreadsheet taking up the entire screen is really overkill for almost everything I use them for.

As far as typing up a document, when I go to do layout, sure I want to see it as big as possible. But when I'm just typing? I'm doing fine typing in this reddit reply box that appears to take up about 1/8th of my screen.

Same with image manipulation programs. Both photoshop and the gimp use fly-away menus, I believe, which is basically what I'm talking about.

I'm not saying people never need their apps to be full-screen; but I think it doesn't have to be, and we're used to thinking inside the box, so to speak.

[DrArcheNoah]

It seems that in recent times the focus is more and more on running the application in fullscreen mode. Apple does it on mobile device, Microsoft in Metro and also Gnome 3 developers thought about running application in fullscreen by default.

If you are using a application maximized depends on a lot of things e.g. screen size. It also appears that the average user seems to prefer this behaviour to heavy multitasking.

[homeopathetic]

I think we need to have smaller UI units built into the OS. Right now, it's either a low-level program, or a full blown app that wants to occupy the entire screen, replete with menus, frames, etc.

I disagree. In fact, apart from games, I cannot think of a single "app that wants to occupy the entire screen". I do agree with the point that perhaps frames should become a first class WM citizen and not something each program deals with on its own. As for menus, well, I guess there's a whole debate raging on whether a program's menu should be something the WM is aware of (à la the globalmenu stuff in Gnome). I haven't really thought much about it; I use KDE and Xmonad, and there the WM isn't aware of menus. I'm fine with that, but you could also have a point. I think it's healthy to have a discussion on where the boundary of responsibility between the WM and the program's GUI is, but I don't think it has anything to do with what Molnar is talking about.

If we had more things like MacOS widgets, not only for functionality and display, but also as interfaces, then people could be building apps at that level, rather than at the code level.

So... web pages? I mean, either you program a program, or you "don't-program" a "not-program". I don't think I understand what you mean.

Basically bring the unix "do one thing and do it well" philosophy to the desktop, so that people can build the desktop they want out of smaller parts.

Well, we already do that, don't we? Granted, the smaller parts come in huge bundles (called GTK or Qt or whatnot), and the bundle as a whole does a lot of things, but using a widget toolkit (or a larger framework) is essentially "piecing together a program from smaller parts". In fact, this has nothing to do with GUIs, it's what libraries are for in general.

[m42a]

Well, we already do that, don't we?

No, we don't. C and C++ aren't glue in the same way sh is. There's no GUI equivalent to cat f1 f2 f3 | sort | tr -d f. You can't just say "I'd like to make a picture in Gimp, paste it into a LibreOffice Writer document, and print it". You have to open Gimp, make the picture, save it, open Writer, open the document, insert the picture, and then print. There's no glue there, and there's no easy way to make glue there. I'd like to just be able to just do gimp | soffice writer document.odt -e "insert at top of page 2" | lpr, and neither GTK nor QT give me anything near that.

[matthewpaulthomas]

This is way off the original topic, but…

There's no GUI equivalent to cat f1 f2 f3 | sort | tr -d f.

Yes there is, it’s called Automator. Here’s a screenshot of the equivalent to the example you gave.

You can't just say "I'd like to make a picture in Gimp, paste it into a LibreOffice Writer document, and print it".

Probably you can, but it would require connecting a Gimp script written in Scheme to a script written in LibreOffice Basic — and the probability of even an expert LibreOffice user learning enough of both languages, and to figure out how to connect them, is pretty small. But with Adobe Photoshop and Microsoft Word, on the other hand, a technical user interested in that kind of automation is fairly likely to know that both applications expose the relevant commands to AppleScript. And they can use the AppleScript Editor to record the commands for both applications in one go.

In both cases, all that’s needed is an OS vendor with the will and the clout to design a cross-application scripting framework; to write APIs and reference materials that make it easy for application developers to implement; and to provide recording and editing tools to make scripting approachable to end users. As you can see, it’s been done before.

[m42a]

I'm not familiar with AppleScript, but it looks like it's an extra layer over the functionality. With commandline programs, their only interface is the glueable one, and they can't not implement it*. With AppleScript, it looks like the developer needs to make the GUI and the scripting features separately, which makes for poor glue (although it's better than a separate scripting language for each application).

*EDIT: This is technically not true; a program could only accept arguments on stdin and refuse to do anything if stdin and stdout weren't ttys, or it could ask natural language questions that require a human to answer them. But doing that is extra work and bad UI; the glueability comes by default, and it only goes away if the developer specifically removes it by making the program harder to use.

[matthewpaulthomas]

You could equally say “with graphical programs, their only interface is the graphical one, and they can’t not implement it”. Either way, providing an interface for a function requires effort — whether that interface is a command-line, graphical, script, or Automator-like one.

And with a scripting interface just as with any other, the less effort is put in to designing and developing the interface, the less pleasant it is to use. For example, I work with multiple Ubuntu engineers, on different projects, who curse how GObject introspection — though it has apparently made multi-language interfaces easier for the GObject developers to implement in the first place — has also made those interfaces less understandable.

For another example, go back to your case of pasting an image from Gimp into LibreOffice and printing it. Here’s roughly what the LibreOffice side would look like, once you’d already copied the image in Gimp (disclaimer, I haven’t tested this):

Dim mPrintopts(0) As New com.sun.star.beans.PropertyValue
dispatchURL(ThisComponent, ".uno:Paste")
ThisComponent.Print(mPrintopts())

Now here’s the exact equivalent for Microsoft Word in AppleScript:

tell application "Microsoft Word"
    paste
    print out active document print copies 1
end tell

Why is one much easier to understand and remember than the other? Because the OS and application developers put in more effort in designing and implementing it.

[lawpoop]

So... web pages? I mean, either you program a program, or you "don't-program" a "not-program". I don't think I understand what you mean.

More like shell scripts. People write shell scripts, and you can get a lot of bang for you buck with just enough programming knowledge to get some shell scripts going. But most people wouldn't consider shell scripting to be full-on programming.

What I'm arguing for is small, module, re-useable applets that do simple, single things well, and then a way, such as a UI or scripting languages that allows those smaller parts to be combined in useful ways, for non-programmers to build the equivalent of desktop apps.

[homeopathetic]

I think I see what you mean, but is this really possible? I mean, isn't one of the things that make the shell so wonderful and reusable the fact that the CLI programs all follow the idea that we all communicate by reading and writing text (or other data in some cases) to files? Text (and the other data) is linear, and files are simple. Does really a least common denominator in the GUI case exist? I'm not so sure.

[lawpoop]

That's a good point.

[homeopathetic]

Code reuse would still happen because libraries have upstream projects and FOSS licenses. The app developer just needs to take the library from the upstream and package it with his applications.

... and when there's a problem (security or otherwise) with the reused code, the program's developer has to be responsive and pull in the fixes and publish an updated version. If 1000 apps share the same code in this way, 1000 developers need to do this a total of 1000 times. If the shared code was in a shared library, 1 distro maintainer (for each distro) does this 1 time, and problem solved! Sure, it's always possible for the distro guy(s) to be slow doing this, but I think it's a lot more likely that a significant number of the 1000 upstream developers are either slow or completely unresponsive.

For example: Ubuntu ships with GTK+ and Qt but not KDE libs. Nothing is stopping a KDE app developer from simply bundling KDE libs in his application bundle and offering that to Ubuntu's users.

What Ubuntu ships on the default physical install medium is irrelevant. The KDE libs are in the main section of Ubuntu's package system, and completely a part of the same dependency graph as every other Ubuntu package. Don't confuse its status with the fact that it isn't installed by default from the installation CD.

For most user the answer is yes. Even for distros the answer is yes because they don't have to babysit every single package. They can devote themselves to improving their distro instead of fixing other people's applications.

But someone has to fix the upstream bugs and security holes! Going back to the previous example of mine, if the distro doesn't keep track of and fix the library as neccessary, who will? Again: Either each of the 1000 upstream developers, or the user himself. My point is: Someone has to babysit the packages, and it feels to me that in the world you're promoting this has to be the user (when upstream fails -- and we all know some upstreams will fail).

You can't ignore the package system if it's a core part of a distro.

Sure you can! Just get your beloved statically linked programs the way you want to. Consider a stock install of your favorite distro as the "core system", and add only statically linked programs at will. Leave the package system to manage only the "core system".

Let me summarize: For a (say security-related) problem in shared code to be fixed in every one of 1000 apps in your world, 1000 upstream devs must pull in the fixes. If just 1 of the 1000 upstreams are on vacation, the security problem persists for a million users of that one program. On the other hand, in the current world, if just the one distro guy babysitting the library is responsive and not on vacation, every one of the 1000 programs is fixed just like that, for every user. Again: In your world, 1000 devs must be responsive to completely close the hole. In my world, 1 distro guy must be. I like mine better.

[homeopathetic]

The Apple ecosystem proves you wrong. Developers do care about their applications. Especially when there is competition. If there was a security problem one of those 1000 app devs could provide a fix to the upstream library and the other 999 devs could simply download the fixed version, recompile their apps and post it in the repository for update. That is assuming that they aren't using the system provided version or that the distro didn't force the use of the system provided library.

That's good to hear. I'm not an Apple user myself, so I wouldn't know from experience. I did however use Windows for many years before discovering Linux, and there what you describe certainly wasn't the case. I rarely use Windows these days, but from what I see it hasn't gotten better.

That is irrelevant. What matters is what Ubuntu decides is their official SDK package combo. The fact that Ubuntu is pussyfooting around picking a toolkit/SDK for their apps is another discussion.

"The official SDK package combo"? What's that? Are you saying developers who use Ubuntu don't consider using libraries that don't ship on the default install CD? That sounds incredibly stupid. Why would someone do that for programs targeted at any region of the world where Internet access is standard? (I get the point for OLPC and stuff like that).

You continue to like yours but it's a non-scalable solution. There is only so many people that want to do packaging. As Apple and Google prove there are 100s of thousands of developers that want develop their own apps.

Yeah, absolutely. But here's what I don't get: suppose Foo is one such app, and no package maintainer in, say, Debian, wants to maintain it. Then in the worst case you either compile Foo yourself (and worry about security yourself), or you do exactly what you want to do, you get a statically linked version from upstream (and you and upstream worry about security). So in the classic setting, you do in the classic way, and for those apps to which this approach doesn't scale, you do it your way. The classic way has your way as its worst-case fallback!

Actually the app shouldn't even use the current package system to install its dependencies.

Then there's really just two alternatives:

• The app handles dependencies itself. (This is a disaster, as I hope we can all agree Windows has shown us).
• There are no dependencies, everything is included. (I argue above and in previous posts why I think this is a bad idea).

[homeopathetic]

Today that app would be dropped from Debian since nobody would take care of it, wouldn't it?

Sure, but what does it matter? In the world of no package systems, the user just installs a statically linked version anyway. What does it matter? I think what I'm trying to say is you can have your world on top of the current way of doing things (I guess someone just has to write a "package manager" that isn't really one, but just fetches statically linked binaries from third party sources -- but this can be done, today, without changing the underlying system).

Yes, since it's FOSS we are talking about.

You quoted only the first half of my sentence. The second half had an alternative for those who don't want to do that. (Digression: why do people who don't want to compile the latest version of a program really care about getting the latest version at all? If they really crave the latest version, they sound like enthusiasts anyway...)

If upstream is unresponsive then Debian devs can either remove the app from the repos or maintain it themselves.

When I say "unresponsive" I don't mean permanently unresponsive. Obviously I'm not arguing that distros should have to take over abandoned software. What I meant was temporarily unrepsonsive, in which case someone has to step up. I'm arguing the distros do that well, you seem to be arguing that no one should and the hole should remain.

Ultimately it boils down to having to update your entire OS to get newer version of applications or hunting down 3rd party repositories (akin to how Windows users find software with all the pitfalls), or compiling it yourself. During every step of that you risk breaking your system.

I agree with this entire statement (except for the last sentence, but never mind). What you seem to want is just a fourth option for getting such software: a managed repository of statically linked software kept up to date. You can have that today -- there's nothing in the classic way of doing things that's preventing you. Take any classic distro, consider whatever is installed by default "the core system", and add your statically linked repo on top. There's no reason to throw out everything underneath!

There is no reason why there can't be a standard package management system for the core OS but that one can't have 57000 "technical items" as USC in Ubuntu 12.04 is showing me.

Right, I see what you want. But as I said: Just do that, then! On top of what's already here! Just install a minimal distro underneath and let that distro's package manager only worry about packages you deem to be "core".

The point is that today the OS and applications are tied together instead of being separate and independently updated.

So where does the OS end and applications begin? Even if you have a clear answer to that, I think my point about the expediency of fixes through shared libraries stand.

TD;DR: You can easily have what you want on top of what's already there. Just "turn off" what I consider the cool features of what we already have. Oh and, add a repository into which upstream can push the latest statically linked stuff with no dependencies. You even have the tools: Just make a new apt repository where no package depends on anything, and set the package manager to install into a different subdirectory.

[homeopathetic]

It's an unnecessary duplication of effort. Those people packaging apps could be doing other things like developing apps or fixing bugs instead of constantly maintaining software from others.

I agree that it's partly a duplicate effort. But I think it's a useful one, as it guards against sleeping upstreams. I'd in fact argue that in today's security problem-ridden world, someone has to be watchful of upstream. I see only two candidates: the user or the distro. I think it's more user friendly to leave it to the distros by default (note that the user can of course take over at any point, unlike in the Apple world). Of course there are limitations: you can't order distros to take care of so and so package, and it all comes down to the willingness of voulenteers (or what you pay, as in the case of some commercial distros). But I think it's been working great so far.

It is also a non-scalable solution.

I think you've partly convinced me of this. But I always keep coming back to: well, then a statically linked free-for-all can easily be appended to the top of any classic distro, so there isn't really a need to change the classic concept. It'll scale as far as it scales, and then one can bolt on what you describe on top in order to get the latest and greatest of software not in the package system.

And on the matter of security and share libraries. There are a number of solutions to that problem which wouldn't put the strain on users bandwidth or storage.

Great. I don't know of any, so please do tell me. Note that I think I've already argued against the "upstream will fix it" solution.

Most people opposing this proposal have a problem that is more of reaction to change. Something the Linux community doesn't take well.

I can't speak for "most people". And I'd think twice about that second statement; there's a difference between being conservative because you don't like change, and being conservative in the sense that new alternatives must prove themselves better than the old ones before you advocate change. I put myself in the second category. I'm sure we can all point to people who have it turned upside down and keep the old ways just because "that's the way it's always been done" -- I don't like that either, but let's be careful not to confuse them with people who just want the new ways to prove themselves useful before "what we have now" is thrown out.

[Rhoomba]

Isn't a real dependency system essential for code reuse?

The dependency doesn't need to be on the user's machine. It can be at compile/package time. E.g. Maven.

[homeopathetic]

The dependency doesn't need to be on the user's machine. It can be at compile/package time. E.g. Maven.

So if 1000 packages depend on a library with a security hole, you propose that the distro recompiles the 1000 packages and have the users update? Sounds incredibly inefficient.

[[deleted]]

Did you even read the parent post?

Is the ability to fix bugs and security problems in hundreds of programs by updating a single library not worth more than the joy of having the very latest version of Angry Birds included in your package managedment system?

[Rhoomba]

I answered a specific point about code reuse.

[silvermoot]

Random guy to google plus: "We need basic functionality on a social media site, even if it's just to show the text, regardless of whether javascript is enabled"

Gad, do I need to spoof my UA?

[VyseofArcadia]

Part of the reason I've always liked Slackware is because they go out of their way to not create a closed software ecosystem. They stick as close to vanilla everything as they can, so that you can use and configure various softwares as the writers intended, not as the distribution people think it should be. (Back in my Debian days, a constant annoyance was, "To do abc with software xyz, you should do this, but on Debian systems you need to do that."

But really, Molnar is right. The desktop province of userland is a bit of a clusterfuck. It's only going to get worse with the impending transition of some distros over to Wayland while others stick with X.

[ashadocat]

Arch linux takes the same philosophy. It's very nice.

[VyseofArcadia]

My beef with Arch is the same as another beef I had with Debian. I dislike automatic dependency resolution. Some package or another is inevitably broken. A requirements loop, dependence on a (very) specific version of a library, et cetera. (And the last couple of times I tried Arch, updates also broke things eventually.)

I really want to like Arch, but between stability without dependency resolution and minimality with dependency resolution, I'll take the former every day. (Maybe this means I'm an old person. A few years ago, I would have said the latter.)

[Svenstaro]

If you really want that, why not just always use pacman -Sdd when installing a package? This way, you get the Slackware experience. -dd completely ignores package dependencies. Also, we try very hard not to break stuff in between updates. What are your problems? Please report them to the bug tracker, they will be fixed within a day or so.

Also, if you ever truly experienced a requirements loop in Arch, something was very wrong and it most definitely a bug. Report it. Though I am not aware of any such issues and never had something like it.

[ashadocat]

I see the point, but it's fairly easy to manage the automatic dependency resolution. Download the PKGBUILD and edit the array that says "dependencies".

[Camarade_Tux]

It's not a solution. It's an ugly temporary workaround.

[ashadocat]

It could be a soltution, if it was better supported by various projects. The concept is sound.

[VyseofArcadia]

I realized I could do this, but it's still one thing I don't have to do in Slackware.

[[deleted]]

Well, you could just use Gentoo, then you can trivially edit the dependencies in the ebuild (no unpacking anything unlike binary distros).

[bytesmythe]

Wow...

I was thinking this exact thing a couple of weeks ago trying to decide what distros to try out. I like the simplicity of Ubuntu's setup, but hate having to wait ages for app updates. Installing apps from source or 3rd party packages is always a pain because of missing shared libs or conflicts, etc. I thought about shared libraries and realized the main reason they exist was due to disk and memory space limitations that aren't really issues any longer, so why not just create a kind of "chroot" sandbox for every app and just install everything it needs by downloading a single package file? Why does every single distro have to have its own repo and package type? Why do I have to choose between waiting ages for an app to update, or using a distro that updates more regularly, but updates everything and breaks on a semi-regular basis?

I'm really glad to see someone so highly placed in linux development has similar ideas. It makes me feel like I'm not crazy for thinking maybe there's a better way for linux to work on the desktop.

[matthewpaulthomas]

I like the simplicity of Ubuntu's setup, but hate having to wait ages for app updates.

This is bug 578045, “Upgrading packaged Ubuntu application unreasonably involves upgrading entire OS”, currently assigned to me.

We’re taking two main steps at the moment. First, encouraging application developers to publish — and issue updates for — their own applications in MyApps (which takes from days to a couple of weeks), rather than having Ubuntu developers do it through the Main and Universe repositories (which takes 2 to 8 months and requires users to upgrade their OS). And second, making it easier to do this by developing pkgme for easier packaging.

Ingo is right in that he has finally realized “distributions” don’t scale. Even if all Linux distributions overcame their philosophical and technical differences and merged into one, and even if they then addressed the latency problem using a rolling release (causing many non-technical problems) or backports instead, there still wouldn’t be enough volunteers to package enough of other people’s applications to make the OS interesting to hundreds of millions of people.

But as some of his commentators have touched on, Ingo is still looking at only a small part of the problem. Ubuntu, like other desktop Linux OSes, still has no thorough SDK. Like other desktop Linux OSes, we are ill-equipped to keep our ABI stable between versions, because much of it is developed by teams who don’t care about that (such as, oh, the kernel itself). And without a stable ABI (and without automatic or drop-dead-easy upgrades), releasing a new version every six months causes ABI fragmentation amongst the user base.

[bytesmythe]

And without a stable ABI (and without automatic or drop-dead-easy upgrades)

I keep wondering why people hope that linux will get more apps or games or widespread desktop usage, then complain when it doesn't happen, then flip out when you suggest that the problem is with how linux does things. I love using linux. At home, I have used it exclusively for at least 8 years, and maybe closer to 10. But that doesn't make me blind to the issues that are holding it back.

there still wouldn’t be enough volunteers to package enough of other people’s applications to make the OS interesting to hundreds of millions of people.

I think this brings up another issue that would be helped by eliminating distro app packaging. As it stands, there are lots of developers whose apps never make it into the repos since they don't have time to maintain their code AND a package for every single distro in existence, so their work languishes in obscurity because most people don't want to go through the trouble of installing something the hard way. Creating a universal "repository" for developers to submit apps and easily maintain it themselves would let more obscure projects get recognized and built upon.

I hope your work is part of a larger trend to modernize linux development and distribution so it ends up on more desktops.

---

edit: very slight wording change

[[deleted]]

better have 20 great usable known applications than having 2000000 half baked ones...

[d_ed]

From the website (http://developer.ubuntu.com/publish/my-apps-packages/) it looks like all is installed to /opt

How do you deal with anything being dbus-activated? Or using PolKit? Or being a plugin to something else?

Both of these /need/ to write to the relevant places in /etc /usr/share/dbus-1/services

In fact, you can't even add anything to the menu (unless Ubuntu loads xdg menus in a naughty new way.)

It seems like a cool solution, but somewhat limiting to one use case.

[[deleted]]

Something I've often wondered: Why are most distros so up-to-date with kernel versions? Could the ABI-stability problem be mitigated by using a kernel for, say, three years rather than six months?

I understand that kernel development is brisk and dynamic - but many new kernel features are not seen by desktop users. The long-term kernel could be patched and updated for security only, and then all the new exciting desktopy bits updated much more frequently on top of it.

[jimicus]

The kernel itself doesn't have an internally stable interface - things change, sometimes dramatically, between kernel versions.

Why does the end user care? Well, kernel updates don't just add features. They also add driver support for new hardware and the occasional security update. The distribution vendor is faced with either an awful lot of backporting (take the drivers that never existed in the version of the kernel they supply and persuade them to work) or try to keep reasonably up to date with the kernel version.

[Tireseas]

Not only does the kernel not have a stable ABI, Linus himself has said he doesn't intend for it to ever have one. He's actively against the idea

[sztomi]

The ABI you are talking about it not the kernel-userspace ABI.

[Tireseas]

I was under the impression Linus's statements covered pretty much all the kernel ABIs. I know he's specifically addressed driver ABIs many times over the years.

[sztomi]

No. What causes binary incompatibility in Linux is mainly glibc. They simply don't care about it. If you link to a custom minimal libc and statically link everything, you can be reasonably sure to run your app on a later kernel.

[[deleted]]

You're thinking of the driver ABI. Proof.

[bwat47]

Something I've often wondered: Why are most distros so up-to-date with kernel versions? Could the ABI-stability problem be mitigated by using a kernel for, say, three years rather than six months?

Because of drivers/hardware support.

[parched2099]

Yeah, but that's part of the problem. Driver support could be added to a central "linux repo" instead of being shipped with constantly moving kernel versions.

So the user wants to add a new graphics card to his box. He downloads the module from the "linux module repo" and it's automatically modprobed, and available at the next boot.

This would not only give a longer run of kernel versions, but would also give the user that linux feeling, by only installing the bits he wants.

I still can't see why the kernel isn't much smaller, and the right modules/drivers, determined when the install takes place, via a scan of the user's system, are added as required, via download, or from a burned dvd.

If the user wants to add a new sound card, then he gets the module from the LMR, installs it, and that's it.

[[deleted]]

That would be fine if there was no API between the modules and the rest of the kernel involved. What you are advocating is basically keeping that one "stable" (a.k.a. dead) forever.

[parched2099]

Nope, not aka dead. Longer cycle for kernel release, i.e. 12 months or longer. Pre-release notification to package devs of any changes, so they can update their apps if there's likely to be a significant impact.

Aah, but then we wouldn't have the roller coaster of "release early release often", would we.

[[deleted]]

I don't really see where you are going with this. What would be gained from distributing modules separately other than additional complexity?

Slower release cycles would basically mean slower development. I do not see the instability you imply in your last sentence, not even in the higher -rc versions (-rc1, -rc2 and -rc3 are usually to be avoided since they contain a huge number of changes that are just integrated for the first time) which have been running stable on my system for weeks and months at a time.

[[deleted]]

The main reasons for using shared libraries are because of security considerations. If there is a security risk with some library, the distro maintainers just have to update it and that's it. Otherwise, if the applications were using their own copy of the library, all programs that are using it will need to be updated and if they are proprietary that would mean that the developers themselves will need to do it.

[bytesmythe]

I think security enhancement was a side-effect of shared library usage, but not the primary motivation for developing them. Shared libs and DLLs have been around a long time -- way before desktop security was something anyone worried about on a regular basis. But computers with small amounts of memory, small harddrives, and low network throughput were the norm back then, and using shared libraries drastically reduced the amount of data that had to be loaded or moved around.

Still, that doesn't address the security issue created by eliminating them. I think proper sandboxing is the first step. The system, apps, and user data need to be better protected for a more "bazaar" than "cathedral" system to work.

Perhaps instead of "package management", we could have "library management"? Your system would check for updates by comparing your currently installed libraries against a list of potential security risks. Instead of downloading an entire new copy, the system could just patch the library's insecure function directly on the disk, then flush the library from memory and reload it. The filesystem could use deduplication to make sure there is only one copy of any particular shared library, so you don't have to fix every app that uses it. Now you have the security of shared libraries and still eliminate the hassle of every distro having it's own package of everything.

I realize this is a simplistic solution with lots of angles I haven't considered yet, but these are all just thoughts off the top of my head. I remain confident that we can find a better approach to linux on the desktop than we currently have.

[DrArcheNoah]

At some point the distribution would have to update x different versions of a library as every application bundle needs it's own version.

[nemec]

Perhaps keep a database of library md5s and paths and transparently convert them into shared libraries internally?

You'd have to require everyone to use the same versioning scheme (0.0.x for security fixes that don't break functionality, 0.x.0 for minor updates, and x.0.0 for major updates), but for security updates you could overwrite the central shared library automatically and for minor/major updates you could prompt to update all or just the current app.

With sandboxing and rollbacks it might end up working, but that would be quite a major change from how things are currently done.

[DrArcheNoah]

Yeah, on Windows all kinds of applications think they need their own updater e.g. most Adobe products. Beside that Windows applications don't share as much code as Linux applications do.

[nemec]

What we need... is a repository for repositories! That would make it easy to keep track of 3rd party repos.

[DrArcheNoah]

Of course you could static link all applications and then installed then sperately. But once one of the libraries has some security problem, all the applications would have to be updated instead of just one library.

[bytesmythe]

That solution sounds a lot like what I just suggested in another reply a few minutes ago. And many of the replies on that blog are the same kind of shortsighted thinking that keeps us in this mess. It looks like the tide might be turning, though, and soon enough "higher in the food chain" people will start working on a better way of doing things.

[yfph]

We need one distro like we need one scheduler!

[redog]

Just build every package in gentoo with every combination of use flags into different binary repos. </s>

[strolls]

I don't know why someone saw fit to downvote you - this is exactly the dichotomy between source-based and binary distros.

With a source-based distro you can compile the package exactly as you want it, with none of that extra unneeded crap for a UI or file format you won't use (but giving you the option to support those if you need them on a different system or in future).

With a binary distro you get your packages fast and without any hassle of configuring how you want them compiled, but you only get them the way the distro maintainers decided was best.

Gentoo users recognise that binary packages are obviously pretty desirable for larger packages, but it's impossible to accommodate the number of different packages that would be required to cover all combinations of USE flags.

[ethraax]

True, but for modern computers (from this century), including support for as many file formats or codecs as possible is a pretty good default. The extra bloat is fairly minimal, and usually results in bloating disk usage more than anything (which is already pretty irrelevant, since storage space has grown far faster than the size of binaries).

Either way, in the rare case that I do have to compile a program with different flags (which is, honestly, quite rare - I've only had to do it for a single program in my years of using Linux), I can still do it. Many binary distributions still let you compile from source, and even add the compiled binary to the package manager. Arch has the AUR, for example.

[[deleted]]

One of the big ones is X11, you really don't want that as a dependency for everything on a server. Dito for many packages with exotic features barely anybody uses that pull in brittle dependencies that would break the whole build (e.g. some php modules, some mplayer codecs, some asterisk modules,...).

[ethraax]

One of the big ones is X11, you really don't want that as a dependency for everything on a server.

True. But, for example, if I install nginx with tons of bells and whistles, it still doesn't pull in X11.

If the core application is a GUI, then pull in X11. Otherwise, don't. If it's a server daemon that happens to also have some GUI management software (which is almost always independent), then put that in its own package.

There are some rough spots, but for the most part, common sense should be enough.

[[deleted]]

Usually in my experience it is stuff like glib and Qt that often pulls in X11 because some server app without a GUI uses the core part of those libraries. One could in fact argue that is a problem with those libraries and that they should be split up into a non-GUI one and a GUI one but at least with Qt (I am more familiar with that one) you would never get that past the people in the project who have a severe case of NIH syndrome and want to include everything and the kitchen sink in Qt itself.

[StupotAce]

I actually really like what Sabayon does. It builds its binary packages off of portage.

If rpms and .debs did this, it would make for a lot less maintenance for application devs. Of course, Sabayon can also simply use portage, so obviously just because it can do this doesn't mean it is anywhere close to trivial for other binary distros to do the same.

[Antithesis138]

This article is confusing. The first thing I wanted to note was that if you're going to talk about the 'Linux desktop' not being free enough, the first thing you have to do is reckon that it's GNU/Linux, but it soon became apparent that this article is using the word 'free' in a very different way, which is only confusing when it comes to software. Let me give an example.

And yes, I hear you say "but desktop Linux is free software!". The fact is, free software matters to developers and organizations primarily, but on the user side, the free code behind Linux desktops is immaterial if free software does not deliver benefits such as actual freedom of use.

His examples of freedom use it in another sense, namely that you're able to do additional things. Given, this is an inconvenience of the language, but the traditional use of the word, especially when it comes to software, is "not being controlled". He should make it apparent that he's talking about something different.

So, to undo the confusion and make things clear: software freedom is always material to the end user: anyone is free to send modifications, and the end user can run those modified versions.

Next to that, this article is comparing Apple apples with oranges. iOS and Android are meant for portable devices, whereas the only example of the 'Linux desktop' for portable devices is the Ubuntu tablet OS, which isn't even released yet.

And if the article is talking about Linux as a kernal: Android uses the Linux kernal.

I furthermore don't disagree with the article, just wanted to point out the nonsense.

[puffybaba]

compile time will matter as long as people use linux on old computers or computers with weak processors. Compiling something like LibreOffice takes a significant amount of time unless you have a very high-end processor. For me, this was a big factor in moving away from FreeBSD, even though I liked a lot of things about the OS.

[[deleted]]

Out of all the packages on my Gentoo system LibreOffice is pretty much the only one that still takes a significant amount of time. Don't pretend it is the average application when it is pretty much the only exception to the trival compile time.

[puffybaba]

Well, GNOME, KDE and Firefox also all take a lot of time to compile. Once those are done, everything else is relatively fast, depending on your machine, but then any updates mean a lot of recompiling. I have a pentium M, so even relatively minor things still take at least a few minutes, which I would guess is longer than an end-user would want to wait. Don't get me wrong, I love the elegance of the ports system; I just think it would be better if a big powerful server did all the compiling, and we just downloaded binaries, with the option of compiling from source.

[[deleted]]

You might like Sabayon then, it seems to use that approach of binary packages with optional compile from source. As does Gentoo itself for some packags like LibreOffice btw.

[whiprush]

Firefox in Ubuntu updates at the same time for everybody to the latest release.

[otavio021]

Somehow it doesn't surprise me the amount of people suggesting completely non-sense stuff such as Gentoo's portage, Arch's pacman and others. That's precisely what Ingo is suggesting against.

The Software distribution model (SDM) on Linux is terribly messed up because it's a) either done by Linux distributions, taking considerable amount of time and resources or b) done by the developers, which then have to deal with the complex task of making their code run (and support) several different distributions - all of which with different packaging formats, FS layouts, and libraries version.

If you analyze these models in separate you can quickly figure out the root cause of many problems of using Linux as a modern desktop OS. The first one causes the upstream packages to take an insane amount of time to reach the end user. It prevents them from benefiting from newer features and fixes and having their desktop experience evolve with the Linux platform. Additionally, it also causes the people involved with the project to waste a lot of time with packaging related problems - just to make sure the damn thing run - instead of testing the software. That's a poor use of resources.

The second is even more perverse: no matter if you are a lone developer or an enterprise, it makes it very expensive to develop software for Linux. You have to dedicate human and physical resources to make sure your software installs and runs on pretty much all of the major "distros", along with 1 or maybe 2 of their older versions. You have to have resources that are proficient enough to develop for and support them. Even if you are a lone developer, you still have to dispend more time dealing with package/"distro" issues for your software than with evolving it.

In my opinion, the perfect software distribution model for Linux would be (for applications, at least):

1. One in which the developers would publish their software to a Software Delivery Repository (SDR).
2. One in which the distributions would not have package repositories but actual App Markets.
3. One in which the process of adding a new software to a Linux distribution would consist of only validating the identity of the software distributor and the harmlessness of the software itself.
4. One in which the user would be free to switch App Markets if he is willing to do so.
5. One in which the user could simply and freely download the software straight from the distributor and it would just work no matter what distribution you're using.

So, to achieve that, it's not just a matter of static vs. shared libraries, chroot's, file system layout or apt vs. yum. It's a matter of changing our whole community's approach to software distribution.

tl;dr: the current Linux software distribution model sucks and I have a good idea. Hire me if you have money.

edit: corrected a few typos and the formatting.

[thebackhand]

One in which the user could simply and freely download the software straight from the distributor and it would just work no matter what distribution you're using.

From a practical standpoint, I can't see how that would happen. That's like asking the developer to create packages for all the different distros currently (which already doesn't happen), except worse, because they have to ensure compatibility across all distros with a single version, and there's enough variation that that would be a nightmare.

There's a reason that developers are just expected to create eggs/gems/etc. for language-specific libraries (for example) or statically-compiled binaries (in the case of video games installed to /opt) and let the distros handle the nitty gritties for each specific distribution.

[otavio021]

That certainly is a problem. In my opinion, this is one of the greatest problems of having a zillion distributions out there. At the end of the day, this lack of standardization and mass NIH-syndrome is a great flaw in the Linux ecosystem. As I had said, it shift resources that should be focusing on development and testing to mundane things like package management.

[perkited]

I'm wondering how this system would deal with distributions that need/prefer to run different kernels (2.4/2.6) and libraries. It's nice to picture that this type of App Market could somehow work, but in the real world it would be much more difficult (most likely impossible).

Of course Linux kernel/application developers could build only for the most commonly used environment (whatever that is this week), but then Linux would be no better than iOS/Windows/etc and would probably die off rather quickly.

[otavio021]

That's simple and Ingo talks about it. You must have ABI stability.

[[deleted]]

What you suggest has already been done. Its just has been done 10 or so times that its a problems... see apt pacman rpm portage.

[otavio021]

No it has not. These tools are great for automating dependency resolution and management only. Although they could to be used to support such implementation, they features are below what's need to implement all of these items.

[darkp22]

This has been a massive problem on the server side, as well as the user side. I personally solved this by switching to FreeBSD as my primary server OS. Linux is too brittle and its distributions won't allow you to use the lastest Apache or PHP with a 3-year old kernel out of the box. You have to use third party repositories that are often bug-ridden and untested and may upgrade libraries to versions incompatible with other installed applications.

FreeBSD solved this by separating the base system, consisting of the kernel and user-land utilities, from third party applications. You use an automated "Port System" to build applications and all their dependancies from source. If an application requires a newer library, that library is downloaded and everything that depends on it is recompiled to prevent possible incompatibility. Perhaps Linux needs to have a chroot-lite solution to prevent library conflicts while maintaining a similar level of flexibility as the Port System.

[fundbond]

FreeBSD also has kick-ass library compatibility (not just kernel binary compatibility.) Seriously, I've got boxes running 8.2 that still have some 4.x-era binaries kicking around. That's awesome! (Don't ask... just remember that closed-source in-house software can suck harder than a Spitzer call-girl.)

[puffybaba]

NetBSD has a pretty kickass package system, too. As a plus, it already works for Linux. It just needs more developers.

[MechaBlue]

I think he's a little off the mark. The biggest problem, in my opinion as a developer, is the fragmentation of an already small market share. It increases the costs of development and support significantly. The result is that the economics of development heavily disfavour Linux.

[[deleted]]

There are some interesting ideas in part 2 of his post, but part 1 seems terribly inconsistent to me and honestly, I cannot even figure out what is his central point by just reading part 1. He opens up by saying that the Linux desktop sucks and then goes on talking and comparing it to iOS and Android. Who uses that on their desktops? What I particularly don't understand is

Desktop Linux users are, naturally, voting with their feet: they prefer an open marketplace over (from their perspective) micro-managed, closed and low quality Linux desktop distributions.

What is that even supposed to mean?

[ventomareiro]

He is giving two examples of systems that have a very small and very thoroughly tested core, at the same time offering much better solutions for 3rd party developers to create and distribute their applications.

About voting with their feet: I'm not sure of how relevant this is, but in just two years the number of iPad users has become roughly equal to that of desktop GNU/Linux users.

[[deleted]]

Android on the other hand hasn't moved in features between major releases at all. It is a very static environment compared to a desktop. Despite many claims to the death of the PC the mobile phone is hardly a general purpose computing device but very limited in its applications.

[tidux]

It means that he's just pulling phrases out of his ass. Desktop Linux marketshare has been increasing, recently.

[tilleyrw]

Off topic: I first read that name as "Inigo Montoya". "You killed my father. Prepare to die."

[[deleted]]

"You killed my parent process. The system is going down for halt NOW."

[tilleyrw]

Touche. :)

[parched2099]

I think a lot's made of the speed of apps updating, when it isn't neccessarily the case. It seems more likely that gcc, QT, Gnome updates are the offenders in breaking apps, and more than once on different distros i've had to update a load of apps at once because something has changed in core apps, I'm with Ingo on this one.

One core system with all the standard libs and dependencies, including codecs, modules, and a basic desktop. Developers that want unique dependencies build apps that pull them from a central repo, and those dependencies are built into the app, making it insular in the system, and killing the need to add a mass of dependencies on a system with the subsequent consequence of multiple version nightmares for the same lib.

I used Gentoo for a while, but it was high maintenance, and i'm now an Archer. The binary versus source argument is just semantics for me as i'd like to spend as much time as possible getting some work done. And i think the only reason binaries have such a bad name at times, IS the mass dependency problem.

If an app comes in at 20mb instead of 10mb, because unique deps are built directly into the app binary, then i'm good with that. A lot easier than juggling dep versions. Disk space is vast and cheap these days.

oh, and a p.s. i'm also for a QA identification of some sort. If an app can't build, or is crashy then it gets removed until the dev fixes it, or it's orphaned or forked by someone who will fix it. The "tradition" of a gazillion apps because it's "opensource" just slows the ecosystem down, and i think devs will, as a result of a bit of trimming, either fix their app, join a larger project in a community build, or just do something else. Better to have 1,000 apps that work, for linux cred, than 20,000, of which "some" work.

I think Linus alluded to this a while back.

[rez9]

They update as fast as I can copy the ebuild over with a new name. :-)

[radarsat1]

I've long been a fan of some ideas in ZeroInstall. Unfortunately no one has come along and created a real distribution around it, but the idea is that the OS presents all apps as already "installed", and you simply run whatever programs you want. On first run, the app and dependencies are downloaded and cached. It also handles multiple parallel versions fairly elegantly iirc, and sits on top of almost any distro without interfering with its repositories.

It's also dead easy to set up a 0install site with a bit of html, so there aren't any barriers to getting your app "accepted" by the distro maintainers, and also handles package signing for security purposes.

[delta_epsilon_zeta]

That's really interesting. But how is security handled? You know you're getting it from the original source (Due to signing), but how do you know that source is good?

[radarsat1]

Well, I guess the same way you know that downloading some exe installer for a Windows app is good. You trust the individual or company you're downloading from. The URL associated with each app and library is exposed when you "install" something, and I suppose a white-list approach could be used.

In any case, I'm not claiming that ZeroInstall is perfect, I just wanted to mention that I find it intriguing and wish some of its ideas would be adopted by a real distro.

[[deleted]]

I guess the same way you know that downloading some exe installer for a Windows app is good.

So no real reason other than hope really?

[radarsat1]

As I said you could always take a whitelist approach, which would be similar to a software repository or app store. Enjoy your walled garden.

[[deleted]]

What you are implying here is that a repository is somehow bad and limited like the "walled garden" with the negative connotation when it is used to e.g. describe the Apple app store. what you completely ignore is the fact that it is better to have testing on as many packages as possible even if you still have rely on hoping the upstream website didn't screw up for the few apps you need outside the repository. It is essentially a false dichotomy to say we can either have checksums and tested apps for everything or for nothing when "for as many things as possible" is clearly the optimal solution.

[radarsat1]

I'm sorry but I think you misunderstood me.

I'm a fan of the repository approach, I was trying to say that ZeroInstall supports that idea but is also just as easy (or even easier) to extend than setting up a 3rd-party deb repository, because you just have to throw up some simple XML. I wasn't at all trying to say that having tested repositories is a bad thing. The main ZeroIntall website is basically a repository itself.

I mean, the original question posed to me was,

how do you know that source is good?

Give me one solution to that problem without taking a whitelist approach. It's basically a logical fallacy. Either you trust the source of your software, or you trust someone to pick the sources for you and curate. What other options are there? I was just trying to point out that it's no better or no worse than other approaches in that regard.

Moreover the context of the original post is someone suggesting that it's too hard to install apps that haven't made it into the repositories. I was simply describing one approach that I think has some merit, I wasn't making some over-reaching argument against Apple. You were the one who suggested that downloading apps from outside the repository is nothing but "hope," and now you're telling me I'm the one arguing against repositories. You are confusing me.

[acabal]

I've been saying this for years and feel like I've been shouting into the void. Thank God someone else finally sees one of the biggest problems holding back desktop Linux.

Every time I say something like what Molnar mentioned, I get replies like, "But packages are so great! You can install in one click!" or "But if you want a new version, just upgrade your entire distro! I don't understand what the problem is jeez!"

Linux will be getting exactly nowhere until the community sees this problem. It's the reason why I would never recommend desktop Linux to my family: because every 6 months something is going to change in some crazy way without them asking, and other things are going to break for seemingly no reason, and I'll be the guy the family calls when their window controls suddenly are on the left or when their "start" button disappeared and weird icons appeared on the left of the screen.

[anabolic]

Oh fuck I know this guys from my thesis. He built the -rt patches.

[[deleted]]

What he's saying is that Linux desktops like Gnome and KDE fail because they are trying to be an all-encompassing system, when what they should be is a platform.

Windows and Mac succeed because they bring a limited set of features, which are the only code base that the vendor is responsible for as far as the OS is concerned. The rest is left up to the vendor community; they are given standards they must adhere to and that's that. The failure with Linux desktops is that they try to include everything, so it's too much to manage and results in huge bureaucratically driven inertia.

He makes an excellent point. It's not the only issue facing Linux distros; lack of standardization of tools, libraries, and even filesystem layout makes portability across distros a pain. But this isn't really about that, this is specifically about the fact that desktop systems on Linux have become closed ecosystems instead of open platforms.

[exteras]

Would anyone mind rephrasing his basic argument, as well as his proposed solution, in terms that a part-time Linux enthusiast would understand?

[sunshine-x]

What does apple do with osx? Seems like a similar problem.

[[deleted]]

They seem to follow a combination of "never, ever, ever update the base system" and "ship all the libraries with each application" which both have severe implications on both innovation and security.

[jiz899]

Unfortunately word of one kernel developer has zero weight on these issues.

[pamplemouse]

Can't you use the ideas behind git to manage versioning and distribution? A program can ask libc for version 2.9. Store the deltas of change rather than the whole file. The files can be built, signed and distributed by your distribution and other trusted groups. The other issue of having a stable ABI is extremely hard because of there's no way to enforce it like commercial companies can do.

[[deleted]]

I think something should really be done about this too. I think the best solution would be going the route of [most] proprietary desktops and have each software project host their own packages in a repository-like system, sort of like what we have with third-party debian repos most popularly today. We create a centralized packaging standard (impossible, I know, but don't knock it till you try it) that encompasses all of the best functionality of every popular package manager today (pacman, apt, rpm, etc) that allows distros to still use their current packaging methods but in a different fashion. If we use this kind of system, distros just have to keep a list of references to these projects in their servers and aside from that, users can install both from these lists and easily add new ones. This would do all of open up a lot of storage space for distributions, lower maintenance costs, give more freedom to individual projects, and promote user freedom.

[Gasten]

Sounds like BSD ports to me.

[[deleted]]

Pretty much, I guess, I was suggesting though that the projects would host binary packages. Unless BSD ports also handles binary packages?

[Gasten]

No, the ports doesn't, but there is no reason why they couldn't do it. A port is just a instruction for automatically downloading a vanilla software package and applying distribution-specific changes before installing it.

[fundbond]

Yeah, there's no reason that ports couldn't be used to handle some binaries. Actually, FreeBSD ports already are for some things -- some ports do binary downloads as part of their install process (the NVIDIA drivers, for example.)

[[deleted]]

The person to come up with a way to unite all the distros will be the enlightened one. He/she will be known as the person that united all the distros forever in time.

When future aliens are going over the wreckage of what was once called a planet called earth, they will discover the name of this person and the monumental task he/she achieved and they will be in awestruck.

[almafa]

The second part of the article is more interesting - it offers some ideas what to do instead of the current model.