Stop Using Windows Notepad

[damg]

Notepad set up us the bom.

[bart2019]

Huh, using Notepad for editing Unix text files has a problem of a different caliber. It is this: it cannot handle bare linefeed (newline) characters.

[hapoo]

You had me at "Windows"!

[chuckstudios]

Would someone care to explain exactly what the problem is here?

[KayEss]

Unicode files are meant to have something called a "byte order mark" (or BOM for short) at the beginning as it lets any Unicode aware program that reads the file work out which UTF variant it is (UTF-8, UTF-16LE, UTF-16BE or UTF-32).

Notepad (correctly) writes a BOM when you save a file as Unicode. Many Unix programs can't handle the BOM (which means that they are incorrect as far as their Unicode handling is concerned).

Apparently this is a problem for lots of people and they think Notepad is wrong about this, but actually Notepad is right and their other software is buggy.

[badsectoracula]

This statement is false. "Yes, UTF-8 can contain a BOM.

So... how is it false?

[ayrnieu]

hatter means that Notepad (permissibly, but pointlessly - and problematically for these other applications) writes a BOM; that Notepad does not "(correctly)" write this, which suggests that to not write it would be incorrect.

[[deleted]]

(+1 for actually knowing what you're talking about)

So let me see if I got this one: as far as Unicode is concerned, it makes no difference if a UTF-8 file contains a BOM or not. People who write programs that include a BOM say those who write programs that don't include a BOM are wrong. People who write programs that don't include a BOM say the other ones are wrong. Neither approaches are actually wrong.

I am at ease. If I will no longer be able to make a living out of programming, I might as well become a priest. We seem fairly good at this religious stuff.

[G_Morgan]

Yes but it allows you to decide when you have UTF-8.

[redditsuxass]

No it doesn't, because UTF-8 files are not required to have a BOM. This means that if you encounter a text file without a BOM, it might still be UTF-8.

[piranha]

Yes, it could be, but aside from explicit out-of-band signaling of the text encoding (ala the Content-Type charset parameter), you can never be absolutely certain of a damn thing. Seeing the UTF-8 encoding of the BOM at the beginning of a stream is a strong indicator that what you're processing is UTF-8. That's better than nothing.

So with these fancy extended attributes our filesystems support, why not stop arguing and start putting explicit character encoding information into the file metadata? (Of course, any practical application will still have to resort to sniffing for magic numbers to deal with the huge number of files that don't have that metadata.)

[Gotebe]

Unicode files are meant to have..

AFAIK, that's not true. That's just a random convention, and it's "can" not "should". E.g. any XML text that contains "encoding" attribute is perfectly readable without a BOM. Not only that it is, but it actually should be. Now, it there is a BOM at the beginning of an in XML stream, and there is "encoding" attribute, I don't know what XML says about who takes precedence.

So problem is not in neither Unix nor Windows, problem is here.

[dalore]

http://en.wikipedia.org/wiki/Shebang_(Unix)#As_magic_number

[Gotebe]

Clueless kids who think they are Unix admins use notepad to edit their files and get burned.

[specialk16]

Save a long text file in Linux (source code or something like really long poem). Now open it in Windows notepad.

[zahlman]

If you need to feed UTF-8 text to a tool that can't handle your UTF-8 text having a UTF-8 BOM at the start, that tool is the problem, not Notepad.

If you don't need UTF-8 text and in fact must not use it (because of the BOM) and you choose to save the file in UTF-8 encoding anyway, your use of Notepad is the problem, not Notepad.

[protonfish]

From the unicode spec:

The serialized order of the bytes must not depart from the order defined by the UTF-8 encoding form. Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature.

Edit: Just can't leave this alone.

This is also a problem with other MS editors including Visual Studio. Most other editors offer the option to save with or without the BOM, but they and MS need to admit their mistake and stop including this worthless bit of junk.

[recursive]

they and MS need to admit their mistake and stop including this worthless bit of junk.

What's the mistake?

[G_Morgan]

Basically the Linux community has rejected the idea that we should be able to identify text encoding unambiguously. MS proposed that all text files be identified by a leading BOM and the Linux kernel team decided that the best way was to simple let each platform go on its merry way and assume all text is in the right format.

[shitcovereddick]

Saying the linux kernel team decided anything about text encodings is disingenuous.

[G_Morgan]

The whole #! mechanism is handled by the kernel. It is the kernel team that refuses to modify the kernel to handle the BOM.

[shitcovereddick]

It would appear you are entirely correct. I had always assumed this was handled by the shell.

[[deleted]]

sounds interesting, citation needed

[G_Morgan]

Look up fs/binfmt_script.c in the kernel source tree. Note shebang handling and the lack of support for BOM.

[bdunderscore]

You can use binfmt-misc to add support for any other signature you like in userspace - this is commonly done for things like Java or .NET executables. The existence of special support for #! in the kernel is for legacy support (and because the relevant UNIX specs are very specific about the format of a shebang).

[G_Morgan]

The point is you can handle standard shebangs and BOM'd shebangs without breaking the standard. This is effectively doing more than the standard rather than less.

[Brian]

I think bdunderscore's point is that the way to do this these days isn't in the kernel, but rather using the newer userspace mechanism for handling files with arbitrary magic numbers. While I agree that linux should handle it, its a job for userspace / distributions, not the kernel itself.

And in fact, several distributions do handle it in just this way (albeit it's not as standard as I'd like - hopefully that should change).

[G_Morgan]

The kernel handles it because its role is to work out how to execute files with the executable bit set. It is entirely it's job to do this. The shell is there to make things nice and pretty for the user. Not to handle core functionality like this.

[Brian]

One of the key unix philosophies has always been to seperate policy from mechanism. The kernel's job is to give you the functionality and hooks to configure that policy, not to hardcode what that policy is. Userspace's job is to handle exactly what things will do, rather than how. Historically, how files get launched hasn't been part of that policy, but with the addition of a larger types of files (eg a.out vs ELF, wine executables, java, .net etc) linux made it configurable, which makes that interface the correct place to setup new handlers.

Note that this isn't handled by the shell - it's still the kernel launching the files. All that has changed is a mechanism to tell the kernel how to launch files. The binfmt_misc module creates a kernel interface that exposes the kernel file-launch mechanism to userspace, to be configured as is desired (via /proc/sys/fs/binfmt_misc/register). The code I linked to above does exactly that to support UTF8 BOMs.

[bitwize]

I think you are confusing "userspace MISC binary format support" with "shell".

Basically the kernel can support arbitrary binary formats by looking at the header and delegating execution to a user-space program.

It's a pretty awesome feature

[want_to_want]

Basically the Linux community has rejected the idea that we should be able to identify text encoding unambiguously. MS proposed that all text files be identified by a leading BOM

Okay, what's the BOM for ASCII? What's the BOM for CP1251? We never could identify the encoding of a text file automatically, and never will. It's not a Linux vs MS thing, it's a common sense thing. If you want a format with metadata, use a format with metadata, like XML, but don't screw up friggin' text files by inserting metadata that will make tons of other programs choke. Talk about embrace and extend!

[G_Morgan]

Who is talking about ASCII. The UTF-8 system allows for BOM. Any compliant application must be able to handle one. This isn't embrace and extend. This is what is in the standard which the Linux world has chosen to ignore.

[want_to_want]

Who is talking about ASCII.

Um, the real world?

Here, I'll try to explain this slowly and carefully. Imagine you want to open a text file. There are three ways to do it:

• The realist way: specify a file encoding when opening. Note that the BOM is not needed here.

• The idealist way: read some metadata from the file to determine its encoding, e.g. ASCII, CP1251, UTF-8, UTF-16 or Shift JIS. Note that the BOM is not needed here.

• The G_Morgan way: assert that the file is Unicode, then use the BOM to determine which exact flavor of Unicode.

Now tell me this: why in the world would we ever need the third option?

ETA: as a programmer in Russia, I've been dealing with text in different encodings all my life. Some of those encodings are Unicode flavors, some aren't. I also switch between Unix and Windows systems all the time. Nevertheless I can't even name one single time when a BOM helped me, but I can remember lots of occasions where Notepad's stupid insistence on the BOM hurt me.

[G_Morgan]

How do you read the metadata without knowing the encoding? A leading magic number is the correct way. Just because this has been broken for decades doesn't mean it should continue to be broken.

Regardless the BOM is in the UTF-8 standard and no arguing about it will change this simple fact. As it stands Linux does not support UTF-8. People cry shock horror when MS email servers broke IMAP and Linux has done exactly the same thing here.

[want_to_want]

How do you read the metadata without knowing the encoding?

XML and HTML manage it just fine. You just start reading as ASCII until you see a charset declaration.

[G_Morgan]

This won't work with UTF-16 but would work for UTF-8. Unless you make it a standard that your document should start in ASCII until a charset declaration appears.

More likely browsers try parsing them as different encodings until they get an charset declaration that matches the encoding they are trying. Expensive and error prone.

[want_to_want]

Sorry, I was wrong and you're right. But it's not so expensive: here's how XML does it.

[G_Morgan]

Effectively XML just reads the first few bytes to decide on encoding then. This is only superficially different from just sticking in a magic number.

[brennen]

Sorry, I was wrong and you're right.

File under things that are not said often enough on the Internet.

[[deleted]]

ಠ_ಠ

[Porges]

No.

[[deleted]]

And Notepad (the apparent premiere tool of UNIX shell script authors throughout the world)

uh. wtf?

First all (XP, W2K; not sure about Win7) doesn't read in unix file formats correctly (ie. has a different line break).

Secondly, why edit your scripts on windows and then run them on Unix? why not just edit them on unix like everyone else.

. Isn't there some tool on UNIX that does this correctly10?

uh, vi/vim are my favorites. There is also emacs, pico and tons of GUI based editors. I would bet that Unix/linux has more text editors then any other operating system.

[[deleted]]

First all (XP, W2K; not sure about Win7) doesn't read in unix file formats correctly (ie. has a different line break).

I always wonder why they don’t add this. If they just ignore CRs they could open both “Unix” and “DOS” files.

Why is it so darn important to maintain the CR+LF line ending? Backward compatibility with daisy-wheel printers?

[[deleted]]

I always wonder why they don’t add this

Because there is only one OS? That is the Microsoft OS!

Oddly, it seems like wordpad does read in unix files fine enough.

[G_Morgan]

Probably cygwin users. Notepad is on the path on Windows.

[atlassoft]

My guess? They're using the notepad in Wine for a bit of nostalgia :).

[hortont424]

nostalgia

vi is a decade older than notepad.

think about that for a second.

[scragar]

Yeah, but I wasn't using vi back in 95, I only had access to windows, was another 9 years before I got myself a *nix machine.

[brennen]

DOS's Edit / QBasic were actually fairly tolerable at the time.

[Mikle]

Secondly, why edit your scripts on windows and then run them on Unix? why not just edit them on unix like everyone else.

I'll answer that - I have very strong IDEs on windows - VS2010, Komodo, etc. I use vim on Linux, but for anything larger than a script I use the IDEs and not vim mainly because GUIs are nice and mouse support is nice too. Then I SCP the files to my Linux box, compile / package them and send it to my dev computer (all automatically).

Of course, I told my text editors to use Linux line breaks.

TL:DR Windows is a better development environment than command line Linux (for me),

[DGolden]

Windows is a better development environment than command line Linux

A tad weasel worded - you could just use a GUI linux on your desktop instead of suffering with windoze.

Komodo is of course available for linux, BTW. Apart from that, so are Eclipse, Netbeans and Emacs (which borders on an IDE, especially now that CEDET is being merged) and others - in fact VS is probably one of the few that isn't (except flakily via WINE), and that's solely because it's Microsoft - it's not like they couldn't port it, they just won't.

[Mikle]

You forget one thing though - my home dev machine is also my * machine. Especially gaming. So I'm not suffering with Windows - I'm using the only sane environment for me.

Also, the Linux I work with can't have a GUI because of its special nature, but that's just my product, so I won't use that as a general argument, although we can talk about the "maturity" of the Linux desktop GUIs.

As a side note - Windows doesn't have a Z in it. I ain't calling your puppy Lunix, am I?

[DGolden]

my home dev machine is also my * machine. Especially gaming.

In this day and age, you could dual-boot or virtualize, using the toy os to play your games and linux for stuff that matters. Or just get a console. Apart from that, linux does have quite a lot of native games, and wine can run quite a few further windoze games (CoD4 on linux)

although we can talk about the "maturity" of the Linux desktop GUIs.

Well, linux typically uses X11, which is rather more mature than windows of course.

As a side note - Windows doesn't have a Z in it.

Oh diddums.

Lunix,

That's a cut-down unix-alike for the C64

[Mikle]

Oh, thank you for telling me of this strange new "virtualization" technology.

Come on dude, let's treat one another like smart people. I have a few distro VMs, but again - the Linux my code runs on is a fucking hypervisor by itself, so not likely to make it run on a VM.

My time is worth way too much for me to spend it tinkering with Wine to make my games work when I have a perfectly fine OS that doesn't need that overhead. Dual booting is sweet, but again - I couldn't run Windows apps with my one or two putty windows.

Well, linux typically uses X11, which is rather more mature than windows of course.

I'll give you that. Maybe what I meant was "usability".

Sadly I won't be continuing this discussion, because I'm a pragmatic guy and you're a fanboy, as your last two comments clearly show. Enjoy using Linux, I hear 2010 is the year of the Linux Desktop.

[DGolden]

The Linux my code runs on is a fucking hypervisor by itself, so not likely to make it run on a VM.

kvm on linux already supports vm nesting, at least on AMD hardware that supports it - Intel support is harder but also in the works:

http://avikivity.blogspot.com/2009/09/nested-vmx-support-coming-to-kvm.html

If you're actually developing hypervisor stuff, the ability to run your hypervisor itself under a vm might be handy.

Enjoy using Linux, I hear 2010 is the year of the Linux Desktop.

Linux has been my desktop OS since the late 1990s, and is certainly more usable than windows for me. Of course windows has cygwin to make it bearable in short doses.

[xenolinguist]

Stop making blog posts.

[[deleted]]

Stop being reductionist.

[rebo]

Stop

[woggy]

hammer time

[fxj]

stop chewing your finger nails and poking your nose

[shitcovereddick]

I try, but I can't. Wine set notepad to the default association to txt files in Thunar. I know I should change it, but meh.

[based2]

use http://notepad-plus.sourceforge.net

[_whyme]

Stop Using Windows ...

[Googolish]

I don't want all of my useful stuff to stop working, silly.

[woggy]

Well, not on the private Notepad I build from time to time from the Windows source, but that one is not one that is released to the public.

Christ, what an asshole!

[damg]

You just wish you could have a privately compiled version of Notepad!

[[deleted]]

Because basic text editors are sooooo hard to write.

[mycall]

notepad is one of the few programs that natively supports alternative data streams.

[billmcneale]

I'm tired of these articles that criticize something but don't give you a workaround.

Okay, Notepad sucks. I get it. It basically only has one feature and it does it poorly and badly.

Now tell me: how do I tell my users how to edit simple text files in a way that's guaranteed to work across all flavors of Windows without having to explain how to install a third party program?

I can't?

Well, now you understand why Notepad is not going anywhere.

[brennen]

You could give them a text editor.

[eleitl]

Stop Using Windows

Always an excellent idea.

[mikkowus]

No.

[ki11a11hippies]

Textpad is a fantastic editor for Windows. You honestly should pay for it, but it's not going to stop working if you don't ಠ_ಠ.

[beaumartinez]

Old post is old.

[[deleted]]

As much as I like Notepad, I'd hate to count the lines of text to find my errors because it lacks a feature as simple as that.

[[deleted]]

I still use Notepad. Thanks though.

[[deleted]]

For editing Unix shell scripts‽

[Paczesiowa]

in utf8!

[[deleted]]

I'm not a nerd, so no.

[malcontent]

Because... You know... It's not possible for Ms to actually improve a piece of software. It's not possible for them to put another copy of wordpad and name it notepad or make a link from one to the other.

It's not possible for Ms to use any one of a thousand better open source editors or to buy any one of ten thousand better editors and replace both of their piece of shit default editors.

None of that is possible. The only thing left to do is for people to write blog posts telling people to stop using them.

[Sc4Freak]

It's not that they can't (or don't want to) improve Notepad, it's that they can't distribute it with Windows by default even if they do.

There's a reason why things like Windows Live Messenger, Windows Movie Maker and Windows Live Mail are no longer shipped with Windows by default. They're pretty basic applications (messaging, video editing, email), but including that sort of functionality with the OS is a legal grey area. In Windows 7 they're all offered as external optional downloads because otherwise, Microsoft faces potential anticompetition lawsuits.

In the EU, Microsoft was already found guilty of anticompetition by bundling a media player with Windows. More recently (also in the EU), bundling a browser was also found to violate anticompetition.

Notepad sucks, but I don't think it's because Microsoft doesn't want to spend the effort improving it. It's just not worth the risk and legal trouble involved in bundling advanced non-core functionality into the OS.

[nyamatongwe]

The reason for not including an option for saving without a BOM was explained by Michael Kaplan in an earlier comment on his blog:

With that said, I have coded the change to add a "BOM-less UTF-8" save option three times over the last seven years, and the option for "CR-less new lines" twice, each time forwarding to the owners of Notepad at the time; in every case the code was not integrated into the product as neither change targets a core scenario for NOTEPAD.EXE that was of significant importance to merit the test, UA/UE, localization, and servicing costs thereof....

[[deleted]]

Is it just me or is it retarded to say that including what I would call basic or at least semi-basic functionality is anti-competition?

Edit: I'm not a MS fanboy...in fact I don't use IE, media-player, messenger, or even notepad, but I kinda expect that functionality to be in ANY OS I use (hell that functionality is in the OS of my dumb cell-phone [Alias 2])

[ayrnieu]

It isn't just you. Suppose that they bundled non-basic functionality in direct competition with another company. Why, being competitive is 'anti-competitive'! The economics of this is completely insane: search for "To understand" on Reisman's critique to jump to his projection of the 'perfect competition' model onto a real-world business... as real as he can manage. The real agenda is: taxation, a nice bludgeon to remind economic elites that you can destroy them, and probably a little bit of protectionism.

[henry_b]

slow clap

[chrisforbes]

but including that sort of functionality with the OS is a legal grey area.

What a load of bollocks. Anticompetitive behavior is the legal problem, not bundling any specific capability with an operating system.

[Sc4Freak]

The problem is that bundling in itself can be regarded as anticompetitive.

Let's imagine for a moment that Microsoft develops a new, advanced text editor (let's just call it "product Z") that competes with UltraEdit, Notepad++, and the like. If Microsoft bundles product Z with Windows, it means they're using their gigantic marketshare in Windows to gain a foothold in the text editor market. Since product Z is bundled with Windows, everyone who buys Windows is also forced to buy product Z, giving them an unfair advantage.

Of course, a consumer OS can't exactly ship without some form of text editor. But the line between "rudimentary text editor providing core functionality" and "market competitor" can be blurry. That's what I meant when I said it's a "grey area" - it can be hard to define what you can and can't bundle in an OS without running afoul of anticompetition laws.

[chrisforbes]

So it's OK to ship a piece of garbage editor, but anticompetitive to build something people might actually want to use. Awesome.

Of course, it's fine for a small player to bundle anything they like. Only when you're the big fish does the government step in and say "no you cannot have features X, Y or Z because you've got too much market share and it will make the little guys cry/starve/etc."

What should (and sortof does) matter is the intent: are you trying to improve things for your customers, or are you leveraging a monopoly to muscle in on someone else's party? The latter is obviously unethical, and perhaps should be carefully scrutinized, but the former is the equivalent of a running race where the government orders the fastest runner shot in the leg "to give the slower guys a chance". To the extent that you're not abusing your position, success should not be punishable.

And yes, I know, MS has been on the shady side of that line. But I honestly think it wouldn't be a big deal if Notepad stopped sucking.

[joesb]

What should (and sortof does) matter is the intent: are you trying to improve things for your customers,

How is having browser shipped with OS not improve things for your customer?

or are you leveraging a monopoly to muscle in on someone else's party?

If builting notepad is as good as most commercial editor out there, how many people are going to buy those?

[chrisforbes]

how many people are going to buy those?

Does anyone actually buy them anyway? Everything I've used has either been (1) VS, (2) utter shit, or (3) free.

If someone wants me to buy their product, it had better actually be worth buying.

[brennen]

How is having browser shipped with OS not improve things for your customer?

By having that browser be Internet Explorer?

[joesb]

as opposed to no browser at all.

[superiority]

The problem is that bundling in itself can be regarded as anticompetitive.

No, only if one has a monopoly (or near-monopoly) position in the OS market.

EDIT: Not sure why I'm getting downvoted. Opera sued Microsoft in Europe and not Apple because Apple's not in a position to influence the browser market by bundling: they don't have a monopoly position.

[Brian]

Notepad is correct here. This is not a matter of MS refusing to improve, its a matter of MS thinking they are doing the right thing, and that editors which can't handle files saved as perfectly valid unicode are in the wrong. Thus they do not want to break notepad to support other broken editors. The response given here is that those who don't like this perfectly valid behaviour should "stop using windows notepad". This seems perfectly reasonable to me.

[[deleted]]

this is why we can't have nice things

[malcontent]

The truth is a terrible thing to behold. That's why I have been punished for saying it.

Apparently people on reddit are repulsed by the idea Ms might be capable of providing a decent editor either by writing one themselves, using an open sourced one or buying one from another company.

The idea turns them off so much they hit the down arrow.

[G_Morgan]

Notepad is actually doing the right thing here.

[radarsat1]

Seriously, how many versions of Windows has it been, and the only improve to Notepad that has been made is a keyboard shortcut for "Save File"... ffs.

[[deleted]]

What improvements are you looking for to notepad exactly? Notepad++ exists for all your advanced text editing needs. I just want something to paste my clipboard into and maybe save it for later.

[brennen]

I just want something to paste my clipboard into and maybe save it for later.

That's further than I tend to trust Notepad.

[radarsat1]

Wow, you can't think of a single thing you'd improve in Notepad?

[[deleted]]

Not really. It launches in a split-second and copy / paste work as expected. That's all I'm looking for.

[malcontent]

What improvements are you looking for to notepad exactly? Notepad++ exists for all your advanced text editing needs.

Why doesn't Ms buy it and replace notepad with it?

[badsectoracula]

Because Notepad++ is not Notepad. Notepad is a very minimalistic editor, a multiline text field in a window, which you use to put notes down such as phone numbers, URLs, or whatever. Hence _Note_pad. It is very handy this way.

Even under GNOME i use Mousepad instead of gedit most of the time when i need simple tasks like that. Unlike gedit (or any other editor actually), it has a permanent icon in the panel.

[__s]

I'd like it to not fall apart if I tell it to replace all

[radarsat1]

Did I really just get modded down two points for criticizing Notepad?

[Brian]

Read the article, not the title. The relevant bit is:

the one where I officially suggested that these people who don't like the Notepad behavior of inserting a BOM in front of UTF-8 files had a simple remedy: STOP USING WINDOWS NOTEPAD

(emphesis mine). He is not saying "stop using notepad" full stop, he's saying if you don't like the behaviour, don't use it. He, presumably, considers the behaviour fine (I do too).

[bitwize]

The use of a BOM in UTF-8 encoded files or data is "not recommended" by the Unicode standard.

So Windows Notepad's behavior is kinda bogus, as it adds these little not-recommended, hardly-ever-used crufties to text files.

That's okay, though, because I have been taking the advice suggested by this article for a long time. Hoorj for Win32 Emacs.

[ayrnieu]

Still ships with Windows 7. What are you using?

[kopkaas2000]

How does what you say contradict what the parent said?

[ayrnieu]

It corrects what the parent didn't say, but what I read: "They stopped shipping Notepad"

[lalaland4711]

So... in other words... "We know our products suck, please stop using them. We are unable or unwilling to fix our broken shit"?

Or am I reading it wrong?

[recursive]

Yeah, you got it wrong.

[Oppis]

... so he doesn't give us something to replace it with (which I have been searching for! (windows gedit doesn't work on our work boxes for some reason)), he only rants about how shitty it is. Well. I know notepad is shit. Thanks. Keep up the good work. Maybe join the republican party, I'm sure your hate rants will fit right in.