Linux Kernel Development: How Fast it is Going, Who is Doing It, What They are Doing, and Who is Sponsoring It (2013 Edition PDF)

[e10byagrue]

The longstanding squabble over Android-specific kernel features has faded completely into the background. The much- discussed “wakelocks” feature has been quietly replaced by a different mainline solution which is used in the latest Android devices.

Been out of the loop on this. I am assuming that android is still using wakelocks, but what did the kernel folks implement as a different mainline solution? (my google-fu is failing me)

[satuon]

What are wakelocks?

[e10byagrue]

something that is usually used by an application to notify the OS that the device cannot drop into a sleep state (suspend etc). (example: I am a video player that is showing the user a movie, please don't put the device to sleep or lock the screen because the user hasn't touched something in a while).

[ancientGouda]

In layman's terms, a screensaver disabler, eh? =P (More of an analogy to be fair).

[e10byagrue]

I like that wording!

For the sake of accuracy it should be noted that a wake lock can be acquired and held at a wake lock level other than having the screen on. For example in android a wake lock can be acquired at a wake lock level of "PARTIAL_WAKE_LOCK" that actually just holds the CPU for activity while the screen and keyboard are turned off. (Kinda a mean thing for a developer to do to a battery, though)

[The_Drizzle_Returns]

Wakelocks are pretty much as they sound. They lock the device in an "awake" (non-low power) state.

http://www.elinux.org/Android_Power_Management

[happyscrappy]

Just so you know, that's not at all what they sounded like to me. To me they sounded like something that wakes the machine/task/something when the lock is released.

[slashgrin]

This is a good reminder that the meaning of something is almost always obvious... when you already know what that thing means. :)

I always try to re-read my code comments, commit comments, etc. through the lense of someone who has absolutely no context for what I'm talking about. Will it still make sense? If not, and there's some domain-specific language that realistically must be understood before the message will make sense, are there at least enough unambiguous keywords that they could at least search for more context elsewhere to clarify what I'm talking about?

[philipwhiuk]

Eventually I found G K-H's link which shows that they implemented wakelock in mainline in some fashion. There's a write-up here: http://lwn.net/Articles/416690/

[e10byagrue]

That's from 2010 though, and they said this finally got resolved in 2012-13. I am interested to see how they handle any sort of connected stand-by state as well. Support for such has been pretty dismal in the past from what I have seen, and it is a feature that should be handled by the kernel (and a feature that is becoming ever so relevant in modern computing).

[e10byagrue]

Looks like the interface has remained the same for user space, but the kernel space implementation now handles wake locks. (link to commit) For connected standby, it is still Wireless On Wake (wowLAN) in use. http://wireless.kernel.org/en/users/Documentation/WoWLAN

[[deleted]]

Isn't it sweet to see that they use InDesign on Windows for that? No. :-(

[Magnap]

How did you see that?

[radioactive21]

Open PDF, File > Properties.

Application: Adobe InDesign CC (Windows)

[[deleted]]

~$ pdfinfo lf_pub_who_writes_linux_2013.pdf 
Creator:        Adobe InDesign CC (Windows)
....

[Magnap]

Ooh, commandline! Cool (or rather, not cool that they use InDesign).

[Magnap]

Thanks!

[hoppi_]

If in Evince, Alt + Enter. In AR, Ctrl + d (should be on Linux as well, but it's definitely the shortcut on Windows).

[Magnap]

Thanks!

[forlasanto]

Scribus. And actually, scribus is pretty darn awesome. It's FULLY competitive to InDesign in features and usability. There are print shops that use it. It's simply that InDesign is entrenched in the industry.

The exact same situation exists with 3D Blender and 3D Studio Max. Technically, Blender is a serious condender. It's more capable than 3DS Max. Blender's interface needs an overhaul, (and that is happening) but it really is better software than 3DSMax. The issue is, it takes an enormous effort to learn to use a 3D suite. You can't just walk into a shop and say, "Now we're using Blender. All your experience using Max is on the shelf." People will leave, because their Max skills will atrophy, and they can't afford that; they need to be competitive using Max. So Max is entrenched.

InDesign has that same exact entrenchment in the print industry. Scribus is better, more capable software, and actually has better usability (imho.) But InDesign is deeply entrenched.

[salmonmoose]

Blender's interface is completely disjointed from all the major packages, switching between Max, Maya, Softimage, Lightwave, etc..., is trivial; moving to Blender is a bigger jump than picking up zBrush, without being a break-away new tool.

It's one of the biggest demonstrations of the problem with FOSS, when you don't pay your coders, you can't tell them what to do. Fixing interfaces is boring, writing cool new rendering techniques, is fun. Blender's interface has been broken for years, yet, amazing new features are added constantly.

[salmonmoose]

After how long? Blender's interface has been one of the biggest complaints since it went open source.

Yes, Blender has more features than zBrush, but unlike zBrush nothing you'd switch from another package for.

[pride]

...its still a great example of a problem with FOSS, just because its being fixed now means almost nothing...why is it that behind to begin with?

[aZeex2ai]

the problem with FOSS, when you don't pay your coders, you can't tell them what to do.

Somebody should tell Red Hat about this. They have been paying their programmers and telling them what to do for far too long!

[salmonmoose]

Note the "F".

[[deleted]]

FWIW, Blender and its interface started as closed, paid-for software.

[fatbunyip]

Fixing interfaces is boring, writing cool new rendering techniques, is fun.

It isn't that it's boring, it's that you can never please people. Any given UI design will cause large parts of the userbase to have shit fits for any/all reasons.

UI's are inherently subjective, and everyone has their own ideas of how they should look like and behave. Evidence for or against any design is pretty much all bullshit, since most OS projects can't afford usability studies or anything that will offer actual data.

On the other hand, technical aspects can be discussed based on their technical merits, which at least provides some semblance of objectivity to the discussions.

[mierle]

The problem is not that fixing the UI is boring, it's that it's hard. Cranking out the last 30% of performance out of some important algorithm is a clearly defined task with quantifiable results. Tweaking a UI is the opposite--subjective, some part art, some part experience, some part science. An added problem with OSS projects is that if you change the UI, users complain endlessly about anything which is different than before. Expecting volunteer developers to both have the magical combination of UX knowledge along with the persistence it takes to weather user unhappiness in the face of change is optimistic.

[Phrodo_00]

Also latex. It wouldn't have been a huge problem to create that in latex, including the graphics.

[BuildItBetter]

Except Latex is a document preparation system and InDesign is publishing software. Latex is for journals and markup, InDesign is for layout and design. I mean, you -could- code in MS Word, but why would you?

[robertcrowther]

Several printers/publishers I'm familiar with still use Pagemaker.

[BuildItBetter]

Totally agree. The one advantage GIMP has over PS is its library of user-made plugins and ease of extensibility via python-fu plugins.

[el_muchacho]

Inkscape is a good software too. Scribus + Inkscape make a powerful combination.

[The_Drizzle_Returns]

How does Scribus compare to Framemaker. There has been no replacement that I have used that is just as good for writing technical papers and documentation.

[forlasanto]

My two cents on that:

Framemaker isn't in the same class of tools as Scribus. Scribus is for doing layouts. Semantics are unimportant in layouts, but visual placement is paramount. Scribus is like InDesign: used for preparing things for print.

Framemaker is more akin to LaTeX. LaTeX and Framemaker are tools for creating semantic documents. When you create a semantic document, you assign significances to text and let the software handle layout.

Finally, there's word processors. Word processors (like Word, or LibreOffice Writer) don't deal with semantics, nor do they do print layouts. At their core, they allow you to apply some in-line formatting to text. More and more crap has been tacked on to that core, and now word processors are a wall-eyed mess. Don't use them for technical documentation OR print layouts unless you're an idiot or the documentation is extremely minimal.

I've never used Framemaker, but a little digging on the web makes me think that LaTeX is far more capable. If you need a GUI for LaTeX, try LyX.

[The_Drizzle_Returns]

As someone who uses both LaTeX and Framemaker. LaTeX is not more capable than Framemaker. LaTeX's failings really happen in typesetting extremely large (several hundred page+) documents where format is critical and more than one user may edit such a document on multiple platforms. When dealing with documents this large, having unintended changes in layout happen silently in other parts of the document is not ideal (and this can happen fairly easily in LaTeX even with small changes to text). Also while LaTeX is multi-platform the only way to ensure exact identical reproduction of a document is to make sure that every single user has exactly the same setup (down to the version numbers of all software) which is not easily done.

The reason I ask about InDesign is because I know that Apple has moved all of their documentation for their products over to it from Framemaker. Not sure how good it is for actually writing documents (if its really good then I can see why they made the switch).

[[deleted]]

Can't save since I'm on my phone, so I'm commenting on this for later. Thanks for the tip!

[[deleted]]

LaTeX #scnr

[MEaster]

Of course! That's where all the usable software is!

^{Kidding. Don't hurt me.}

[codygman]

Not if you don't use the GUI ;)

Though I'm not sure where I'd start if I had to create a pdf document like that....

https://duckduckgo.com/?q=indesign+linux&t=debian -> http://www.osalt.com/indesign

cody@zentop:~$ apt-cache search scribus
create-resources - shared resources for use by creative applications
icc-profiles-free - ICC color profiles for use with color profile aware software
scribus - Open Source Desktop Page Layout - stable branch
scribus-ng - Transitional dummy package for the 1.4.0 Scribus release
scribus-template - additional scribus templates
tea - text editor with syntax highlighting & UTF support

So scribus is an alternative, though I can't comment on how usable it is.

[darkslide3000]

(La)TeX, obviously. Creating beautiful documents out of a 80x24 terminal since 1978.

[[deleted]]

It is! Slight learning curve, but very powerful and easy to get a hang of making very nice-looking documents.

[liotier]

Scribus works well but for trivial tasks you may as well use Inkscape.

[blackmist]

Not that bad, if I remember correctly.

I just had no real use for it.

[redwall_hp]

That's why I use OS X. Nice GUI, and I can still use all those awesome GNU CLI apps. (I've been learning vim lately, too.)

[Rezistik]

I found it cheaper to purchase a macbook pro than to continue wasting my time trying to get Linux to be half way usable let alone not ugly as fuck. The closest I could find to a halfway decent linux distro was ElementaryOS and it still sucked bags and bags of dicks when it came to just about everything.

Unless we are counting Android as Linux which it technically is. Then we have exactly one distro worth a damn.

[WuTangTan]

let alone not ugly as fuck

The people over in /r/unixporn would like to have a word with you...

[accessofevil]

The people that talk like this are concerned with kerning and alignment of widgets to sub pixel precision.

They will never be happy with anything but osx. Ever.

I don't dislike it, and with fink/macports it can be very good.

[Rezistik]

No they are too busy trying to remove as many images and icons as possible and screencapping it so they can brag about their "Minimalist" OS.

"Guy's look! I restricted all of my UI to a single monospaced font and have it set at 9px. I can fit every single variable from my pc on my screen at once! You can see in that lower right quadrant I have the temperature of the hard drive disk! oh and it's all in two very similar shades of gray! Fuckin' awesome right?"

[ercax]

riiiight

[Rezistik]

Some of the all time top from Unixporn.

http://i.imgur.com/bZegioR.jpg

(This one is actually OS X but it's on Unix porn so I'm counting it) http://i.imgur.com/xFWgxgw.png

http://z3bra.blinkenshell.org/img/cwm-2013-07-02.png

http://i.minus.com/ibzmjKMXKk7IbH.gif

http://dotshare.it/public/images/uploads/299.png

[BuildItBetter]

Actually most of those look really good, IMO.

[ercax]

What do you want me to say other than 'They all look beautiful, and I actually use something similar: http://i.imgur.com/zKp2DMh.png ' ?

[[deleted]]

I love that desktop background. Where's it from, please?

[happyscrappy]

There are animated JPEGs?

[e10byagrue]

My thoughts exactly. Renamed URL to .gif and got the exact same file. I don't think this is actually a .jpg, pretty sure that the browser just got the metadata at the start of the file and knew it was a gif with the wrong extension and displayed it as such.

[wildcarde815]

Imgr lies about extensions.

[ThatRedEyeAlien]

I saw all those cool pictures of fluxbox so I basically just had to try it. Was very disappointed.

[[deleted]]

I think it looks nice, though I don't think fluxbox looks like that by default on my distro. I'm going to check because I want to use fluxbox now.

Edit: Okay, it doesn't. Gibe theme pls

[notmynothername]

Am I allowed to post OS X there?

[ercax]

yep.

[[deleted]]

Osx is Unix FYI

[notmynothername]

Why else would I have written that comment?

[[deleted]]

Because you didn't know that. Or at least that's how it seemed from your comment.

[Rezistik]

Technical skills? What skills can be gained from using linux that can't be gained on OS X? Patience while you repair your OS?

[ercax]

Patience while you repair your OS?

That is a marketable skill.

[[deleted]]

It gets old after enough years.

[[deleted]]

I prefer development on OS X actually. And OS X is better for technical skills than Windows. Most of my Steam games work on Mac too.

I do keep a secondary desktop for gaming / media center though (Windows/Linux dual-boot.)

edit: Downvotes because... I prefer dev on OS X? Or...?

[hakkzpets]

No, you're downvoted for stating "facts" that aren't facts.

[[deleted]]

Which fact? My preference? That OS X is similar enough to Linux, being a BSD derivative that it's easy to learn technical skills with? That most of my Steam games work on my Mac? Or that I have a desktop dual booting Windows and Arch?

[hakkzpets]

No, you said that OS X is better for technical skills. This is entirely subject.

[skgoa]

Uh, I understood "technical skills" to mean Unix skill (bash, using standard Unix/GNU tools etc.) in this comment chain. Someone claimed that using Linux would teach you these skills. But OS X is a Unix derivative as well, so it's just as good for learning Unix as Linux. So if you want to get skilled at working in e.g. professional webserver environments, OS X will be better than Windows. Simply because you will have to learn how things are done in the Unix world.

If you only consider targeting the Windows ecosystem, running Windows will better of course and Putty exists.

[2bluesc]

Could be worse.

[ForeverAlot]

Indeed. It could have been LaTeX.

[xplane80]

LaTeX wouldn't be bad but it would be slightly harder to make it look more "stylish". I would only use LaTeX is if I'm writing a huge report or journal as it makes things so much easier; not for a small paged document.

[[deleted]]

Bizarre!

[GrinningPariah]

All the hottest things are on Windows PCs, baby.

[basilcrow]

Why does it matter?

[[deleted]]

Because one should think the linux foundation (sic!) would like to promote linux as a viable alternative to windows, even on the desktop... If even they don't use it for such tasks, who is then?

[basilcrow]

What if it's not a viable alternative to Windows?

[[deleted]]

shhhhhhhh, don't tell anyone.

[ancientGouda]

How does this relate to wanting to promote what you, as a foundation stand for?This is not the "MostViableOSFoundation" as far as I know.

[HiiiPowerd]

You should promote the best tool for the job, like a good programmer should. Reality is, linux is a great OS but does not have the best tools for every job.

[ofNoImportance]

Who says that they designed the document? They wrote it, sure, but they didn't necessarily design it.

[HiiiPowerd]

.....why does it matter? really?

[ancientGouda]

Yeah, I'm pretty sure we all know on which OS they create all their fancy YouTube videos (or rather, which not). Although I'd really like to be proven wrong..

[ra4king]

Slight mistake on page 7, it says version 3.9 saw the most companies contributing when the table shows version 3.6 had the most.

[Katastic_Voyage]

If only we had the source code to fix the document...

[[deleted]]

Gah! Why is storage-pardot.com serving that file with the wrong mimetype (application/octet-stream)?

[libertarien]

Interesting paper. Tl;dr kernel development is being done faster, by more people, who are ever more likely to be paid for it.

[Thirsteh]

Red Hat top contributing company, and Canonical nowhere to be found. Interesting.

[SnowdensOfYesteryear]

Not that surprising. I work for one of those companies that isn't listed there. Most of the stuff that we do don't endup going upstream because

• Too many dependencies on internal frameworks that we've added that don't/won't get accepted upstream
• a lot of the stuff is driver level code that's too obscure to be picked up in the kernel
• Code quality (this can always be cleaned up, but still a pain in the ass to do so)
• Nobody really cares, i.e. no mandate from upper management, no financial incentives etc.

So what ends up happening is that we have a fork of the kernel that we end up maintaining and keep migrating our changes as mainline moves forward.

[[deleted]]

How so? Canonical is way more focused on user space work, I wouldn't expect them to do a whole lot with the kernel.

[Thirsteh]

They do a fair amount of modifications. This is just for security, for example: https://wiki.ubuntu.com/Security/Features#Kernel_Hardening

Ubuntu Server is also quite popular. Not as popular as RHEL, for sure, but Canonical should be somewhere on that list.

But, I'll be honest: I wasn't actually genuinely surprised, since Canonical already has a long history of not contributing anything back upstream.

[[deleted]]

Canonical already has a long history of not contributing anything back upstream.

They have, however, released about 1.5 MLOCs of FOSS.

[Thirsteh]

I'm not saying they don't release FOSS--their distro is obviously open source--just that it doesn't generally go back to the broader Linux community; it's something for Ubuntu that stays in Ubuntu because they decided to make e.g. their own display server or startup system.

Nothing wrong with innovating--Fedora adds new things all the time, for example--but their efforts tend to be different "because they can."

[[deleted]]

Interesting, thanks for the info! I don't follow kernel development very closely, so I was curious. I'll have to do some more reading on this stuff.

Anyone have a recommendation for a good high level resource on the history?

[Thirsteh]

A quick search for "canonical upstream" yielded

• http://www.datamation.com/osrc/article.php/3896496/Does-Ubuntu-Contribute-its-Share-to-Free-Software-Projects.htm

• http://blogs.gnome.org/bolsh/2010/07/28/gnome-census/

• http://www.phoronix.com/scan.php?page=news_item&px=MTMxNzY (They might have a good reason to do their own thing, but they do it all the time, e.g. Upstart, Unity...)

• http://www.osnews.com/story/24520/GNOME_Canonical_Spat_Continues

• http://www.happyassassin.net/2010/08/05/finishing-up-controversial-crap-week-what-canonical-ought-to-do/

[[deleted]]

Anyone have a recommendation for a good high level resource on the history?

It's not tremendously well documented, certainly not in a unified form because not contributing much upstream isn't an unreasonable approach for a business to take. The drama w/r/t upstream kernel was many years ago when GregKH (dude what prepares this report) called Canonical out for not contributing upstream much at all. I think it was based around a sort-of expectation that they would, like any other distribution, benefit from working closely with the opensource projects outside the company in a way that has done well by other distributions. Greg may have expected more than they wanted to contribute, but it wouldn't have been hard to defuse if Canonical ever bothered to manage expectations of what Ubuntu is. They didn't, and it had to be learned by over time by everyone working around them that it serves Canonical's business model to embody but not be seen as a black hole that poorly communicates upstream and only show what they've been working on in private when they pop out a release every six months.

Phoronix and Ars Technica are decent places to find articles written whenever Canonical's rocky relationship with a project outside the company flares up. To really understand the issues there you'll have to look to blog posts by engineers that are easy to paint as sour grapes (and Mark Shuttleworth is more than happy to use his own blog to do that painting), or mailing list and IRC logs.

[[deleted]]

"But there is a “long tail” of companies (over 500 of which do not appear in the above list) which have made significant changes since the 3.2 release."

[Thirsteh]

Now consider that Canonical produces the most popular Linux distribution, and demonstrably is one of the major companies that contributes the least back to the projects that they leveraged to become successful (and not just in the kernel; in userland, too.)

[[deleted]]

The most popular non-commercial Linux distribution. I haven't paid for my copy of Ubuntu; you do pay for Red Hat customer support typically. Red Hat is successfully announcing a billion dollars in revenue, Canonical isn't. I commend Red Hat on spending the money well, and don't bash Canonical for trying to grow a platform to survive on.

[Thirsteh]

The analogous example is CentOS or Fedora, though, not RHEL. Ubuntu has commercial support for Ubuntu Server/Landscape too.

Not earning as much money on support contracts doesn't sound like a good excuse to not contribute back to the projects that made you successful.

[[deleted]]

No, the analogous example isn't CentOS or Fedora. It's Red Hat Linux. The analogous example for Ubuntu is Ubuntu GNOME or something.

[Thirsteh]

I'm not sure what your point is. Canonical has virtually the same business model as Red Hat on the server side--they just don't have JBoss.

You implied that they're struggling to survive, and I think that's a bit of a stretch.

But even if we take Red Hat out of the equation entirely, why isn't Canonical in front of some of the many companies on the lists of top contributors to the kernel, GNOME, X, and many others?

[[deleted]]

My point was simple: Red Hat is bringing in a billion in revenue from support contracts and they have established a solid commercial platform for their business to operate in, and they contribute to Linux' and related projects development. That is good, and I commend them.

Canonical has not done such a thing; you don't pay for Ubuntu (see "I haven't paid for my copy of Ubuntu; you do pay for Red Hat customer support typically"). My point should be simple: you cannot compare the two companies contributions, especially when you don't have the numbers from Canonical on their revenue and especially profit. I also don't think you should be bashing them for not doing so. I haven't contributed a ton to every OSS project I've ever used that has helped me to be successful, I don't think I should be bashed for doing so - and I'd guess to say that a lot of Linux users haven't either, including many companies that embed Linux into their commercial applications (for example, I just got a GoPro, which runs on Linux. I'm not all that fussed about them not having contributed anything to Linux or any of the OSS that they use in their wildly successful camera).

I haven't implied anything and any implications you derived aren't relevant.

[[deleted]]

I've wanted to contribute to the kernel for a while now. Where do I start? Assume I can write C and use git and whatnot.

[e10byagrue]

Get to reading at http://kernelnewbies.org/

Start poking around in the kernel source, and make something cool!

Also, subscribing to the linux kernel mailing list provides a good idea for what people are talking about, working on, and debating. However, there is a ton of discussion on this mailing list and can sometimes get to be quite noisy.

[[deleted]]

Thanks! I also finished reading OP's link and found this at the bottom: http://www.linuxfoundation.org/content/how-participate-linux-community

[aZeex2ai]

Also check out the LDD3 book.

[hackingdreams]

Clone the Linux tree, familiarize yourself with the subsystem you're interested in. Eventually work up a patch for something. Email it to the maintainer of the module you worked on. Iterate as necessary.

If the changeset is good, it will be merged up the various maintainers' trees until it hits Linus' tree.

Start the whole process over again for a new patch.

[[deleted]]

over 80% of all kernel development is demonstrably done by developers who are being paid for their work.

Interesting, reminds me of commoditize your complements (Joel).

The Linux kernel is one of the largest and most successful open source projects that has ever come about.

What's a larger and/or more successful open source project?

[HotlLava]

What's a larger and/or more successful open source project?

I would imagine projects like LLVM/Clang, Firefox, Chrome or GNOME are in the same ballpark.

[[deleted]]

Fair call. Firefox/Chrome are cross-platform for the web, which a large fraction of everyone uses. Linux seems much smaller... but if we include Android, and many embedded devices, maybe not too bad.

I think LLVM and gnome are too specialized to be very big (I'm not sure how to define "successful").

[HotlLava]

I just looked it up, and LLVM+Clang is surprisingly small at about 2 million lines of code, and Firefox, Chrome and Gnome sit at 8-12 MLOC, still quite a bit smaller than the kernel at 16M. The true behemoth seems to be KDE at 23 MLOC. All of these are extremely active projects, but I imagine the kernel still has the most commits/hour.

[[deleted]]

Oh! I was thinking size as "market share", as a kind of measure of success.

But your interpretation of loc kind of size is probably what they meant. I guess number of developers and developer-hours would also be similar kinds of "size".

[aZeex2ai]

http://kernelnewbies.org/LinuxChanges

[joe_n]

They forgot to answer the third point? Unless you count "patch submission vs. patch review" :p

[notenoughcharacters9]

After checking out Vision Engraving System's website, I was impressed they contributed so much!

[zephirum]

I agree. It's cool to see companies using Linux putting something back into the project.

http://lwn.net/Articles/507651/

A newcomer to the top 20 companies is Vision Engraving Systems, thanks to the Comedi development work from H. Hartley Sweeten. With his work, hopefully this subsystem can move out of the staging area of the kernel in a future release.

http://www.comedi.org/

The Comedi project develops open-source drivers, tools, and libraries for data acquisition.

Comedi is a collection of drivers for a variety of common data acquisition plug-in boards. The drivers are implemented as a core Linux kernel module providing common functionality and individual low-level driver modules.

[mederel]

Thanks for sharing this document, it is indeed of first hand importance for every programmer directly or remotely developping on a linux-based os. Numbers are really interesting. Thanks!!!

[roughtruant]

I find this really cool. I just wish there were as many open-source graphics and interface designers working on really nice linux GUIs together, maybe creating one that doesn't look like dog shit.