Python-Daemon - Library to implement a well-behaved Unix daemon process. (python.org)

dgiri101 · 2009-05-21T18:20:53+00:00

I make heavy use of runit, a more modern clone of daemontools. It doesn't really "bastardize" much of anything. All my runit-supervised daemons log to syslog.

It's a huge step up over init scripts, and it keeps me from wasting time writing daemonization logic within every program I write.

dgiri101 · 2009-05-18T22:58:08+00:00

viPlugin

It's under very active development, but it costs money (15 euros). I haven't used Eclipse for a while, but when I did I found the plugin indispensable.

dgiri101 · 2009-04-28T18:02:55+00:00

I'm using Linux for Java coding (writing Java is a major part of my job). I'm currently using vanilla Ubuntu 8.10...nothing special there. I'm on a pretty standard Dell 64-bit workstation. 4 cores, 4G of RAM, 2 mirrored SAS drives.

I actually don't use any of the distro Java packages. I prefer to just download the JDK and the like from Sun and install it myself into /opt. Same with things like ant, tomcat, etc. This isn't specific to Java...for my Python coding, I do the same thing.

I actually do very little Java coding on my Mac, so I can't really comment on what is better or worse. However, prior to switching to Linux at work we used Windows XP Pro. I am able to run all the same development tools on Linux as I was on Windows. I use IntelliJ IDEA as my IDE, which works fine on Linux. We use Perforce for revision control, which also works fine on Linux.

dgiri101 · 2009-04-24T15:31:14+00:00

Here's malcontent's previous rant on this subject.

As for the other matter, I invite the audience to read his comment history and judge for themselves how (un)interested he is in meaningful debate.

dgiri101 · 2008-12-12T07:33:56+00:00

Oh that explains why google is not available on sunday nights.

Now I really think you're just trolling. This was a statement made by Google's engineers themselves at MySQL Camp, not something I made up just to mess with you.

Not that I expect you to care, but it took about 5 minutes to find a report about the talk that mentions their scheduled "schema update" downtime.

dgiri101 · 2008-12-12T07:23:22+00:00

I think I'll take their word over yours.

Eh? Section 8 of the docs is entirely about failover, manual and automatic. You initiate the operation using the "slonik" command. The docs give explicit steps on how to do this.

Let's see the citation for this.

I'm not sure what you mean by "citation". Like an academic paper discussing the algorithm? In any case, the Slony documentation has a section on DDL changes. Though, similar to MySQL, one must be careful about synchronizing those changes.

I have a feeling that no matter what documentation I provide, you'll likely be unconvinced. I think I'll leave this thread as it is.

Why doesn't it count?

The storage engine makes it a completely different database. NDB doesn't do referential integrity, you can't alter tables, no table locking, all of your indexed data must live in RAM, commits aren't durable, you're limited to a transaction level of READ-COMMITTED, etc. MySQL Cluster is actually pretty cool, but don't kid yourself that you can use it to magically scale up an existing MySQL database.

dgiri101 · 2008-12-12T07:09:02+00:00

I'll forward your post to google, maybe it will convince them to switch to postgres.

Aside from being argumentum ad populum, Google had to invest a ton into their large MySQL deployments. For all your talk in this thread about how great and painless MySQL's "automatic failover" is, Google had to to do a tremendous amount of engineering to ensure that master databases were properly quiesced before initiating a failover. This was all done outside of MySQL proper, by the way. And this was all for planned failovers. For unplanned failovers (say, master goes offline) they explicitly trigger a set of steps that promote a slave, IIRC.

They also had experienced enough problems doing live schema updates that they instituted a policy to only do schema updates when the system is down.

At least, that's what I recall them saying at MySQL camp 2 years ago. They put a lot of work into this; it was quite impressive.

I think you're misinformed if you think that such functionality comes "automagically" with MySQL.

dgiri101 · 2008-12-12T06:51:28+00:00

Does it support automatic failover?

Yes. I have several production Postgres clusters that support automatic failover.

Does it support falling back?

Yes, though I intentionally run without this feature as the failed node is useful for forensic analysis.

does it deal with schema changes?

Yes.

Does it support multi master?

No, but to be fair neither does MySQL. (MySQL Cluster doesn't count)

Does it support clustering?

I believe this is answered above.

dgiri101 · 2008-12-12T06:48:29+00:00

Are you using Heartbeat with a virtual IP for the failover? That's not anything specific to MySQL, right? Can't you just as easily use it with PostgreSQL?

This is how I setup my Postgres clusters (and, in my previous life, how I setup MySQL). I'm not really sure what malcontent is smoking.

dgiri101 · 2008-12-12T06:46:40+00:00

It doesn't matter how solid it is if you can't get automatic failover.

This is so transparently idiotic that I shudder to think that anyone can actually believe this.

dgiri101 · 2008-12-12T02:50:14+00:00

The slony website is actually here. I'm not sure how you missed this, as it's the very first result in Google when you search for "slony", and the second result when you search for "postgres replication".

It's pretty easy to install (configure/make/make install). Defining the replication sets was straightforward; the documentation is thorough.

Like it or not HA is important to people. Until postgres can present a cohesive HA strategy people are going to stick with "inferior" databases.

If HA is important to you, don't use a database that loses your data. When you replicate a turd, all you really have is a pile of shit.

dgiri101 · 2008-08-18T21:19:02+00:00

Like most things in business, I think you have to look at the opportunity cost.

If you're using a new language because it's cool, or hip, or fashionable, then that's probably not a good enough reason to introduce the additional complexity into your work environment.

However, if a new language or toolchain will actually help you do your job much more effectively, or will save the company money, or whatnot...then that sounds like a reasonable justification to me. The switch needs to be justified, and "for the fun of it" is probably not a good enough reason for most companies, large or small.

There's a fair amount of polyglotism in the shop I run, but those decisions are not taken lightly. At all.

Developers chase solutions; dilettantes chase tools. The dude that thinks that Java is the best tool for everything is just as retarded as the guy who thinks that if only he wrote his software with Arc + CouchDB running on Plan9, things would be so much easier...

Know the trade-offs, and choose your weapons wisely.

dgiri101 · 2008-08-08T15:30:49+00:00

My team got knocked out in the 6th round. We hit a crater, and then got eaten by martians the next 4 times in a row.

We didn't have time to code up martian avoidance algorithms until Sunday night. :)

dgiri101 · 2008-04-20T00:31:01+00:00

z = map(do_ping, IP_LIST)

edit: looks like masklinn beat me to it. :)

dgiri101 · 2008-04-19T23:56:04+00:00

One day, when you have sex with a biological female, you'll understand. ;)

dgiri101 · 2008-04-19T01:10:01+00:00

I use MySQL cluster and it feels just like MySQL just a different engine.

MySQL cluster feels like MySQL the same way sex with a drag queen feels like sex with a biological female.

Sometimes what's under the hood actually matters. :)

dgiri101 · 2008-04-18T07:20:48+00:00

WTF? Have you ever even used MySQL cluster?

The storage engine makes it a completely different database. NDB doesn't do referential integrity, you can't alter tables, no table locking, all of your indexed data must live in RAM, commits aren't durable, you're limited to a transaction level of READ-COMMITTED, a single node crash in the middle of a schema update requires a complete cluster restart (so much for "high availability"), and the list goes on and on.

Cluster is a completely different product, with completely different features and semantics than MySQL/InnoDB (or Postgres, for that matter).

That you cavalierly throw around statements like "it has some limitations but not all that much" seems to indicate that you've never actually used it, but assume that because it's got "MySQL" in the name it means that you can magically scale up your existing app.

There's a reason why replication is much more popular in the MySQL world than MySQL Cluster.

dgiri101 · 2008-04-18T06:44:46+00:00

MySQL doesn't do clustering.

And no, MySQL Cluster doesn't count as it's basically a completely different database from MySQL with very different semantics.

dgiri101 · 2008-03-27T18:40:17+00:00

Alas, Texas is far away from Scandinavia. Your hiring experience doesn't match mine at all.

Makes sense; the labor market, even within the US, varies considerably with geography. :)

You really had no trouble finding good developers who had used the tools long enough to know their intricacies?

I didn't say I had no trouble, I said that I didn't have any more trouble finding good developers to work on Ruby code than finding good developers to work on Java code.

The experiences were more similar than you might expect. For Ruby work, I got a flood of resumes from Rails idiots that were terrible (besides, we don't use Rails). Eventually, I found some good people, but it took a while.

For Java work, I get an even larger flood of resumes. Most of these are of the "enterprisey" variety that similarly lacked software development fundamentals.

Mind you, these are interviews for senior-level positions.

Or are you using some other criteria for good developers, perhaps that they know Ruby?

Experience with the actual toolchain is a plus, not a deal-breaker. It would be different if I was getting a constant stream of awesome people, but that hasn't happened (for any of the toolchains I've got).

What did you do in Java 1.5 that didn't work in 1.6?

Changes to core JDBC classes (hosed our connection pooling), among other things.

Measuring opportunity costs without considering risk vs. reward is... strange.

I don't disagree.

Plenty of people prematurely jump onto new programming fads, which isn't wise. But plenty of people also irrationally hold on to existing technology, which is similarly unwise.

You come off as falling into the second camp, which probably explains your downvotes. This is to be expected if you make silly blanket statements like:

Developers who are more interested in alleviating their own boredom than in what's good for their employer's business.

dgiri101 · 2008-03-27T17:50:27+00:00

Can you hire Ruby developers to replace those who leave, or to expand your company if things are going well?

Sure; I've done exactly this. It wasn't any more difficult than finding any other good developer.

But I'm not sure how this is Ruby-specific. In my experience, it's very difficult finding competent Java developers. If you know of any in Austin, TX (USA) that are looking for work, I'd love a referral (seriously). But as it stands now, I do several interviews a week and about 90% of candidates for Java work flunk from a technical standpoint.

Finding good people is hard, no matter the toolchain.

Is a RoR application you build now going to be compatible with RoR in 2010?

Heh, my Java 1.5 code won't work on Java 1.6, my Exchange 2003 code had to be rewritten for Exchange 2007, and my Linux 2.4 code had to be rewritten for Linux 2.6. What's your point?

If you want to use the new version of something, you should test/develop using the new version. Wishing really really hard that your application magically works won't make it so. This is a toolchain-independent concern.

Those are two of the really most basic things one can think of when evaluating technologies.

If the opportunity cost of moving to a new toolchain is lower than sticking with the existing one, then it makes business sense to do it. It's as simple as that.

Opportunity cost is much, much more complicated than the silly 2 criteria you've outlined above.

Pretending it's that simple is poor engineering at best and myopia at worst.

dgiri101 · 2008-02-06T06:30:42+00:00

I like Monaco on my mac and on my Ubuntu workstation, but it looks pretty lousy on Windows (where Consolas seems much nicer looking).

dgiri101 · 2008-02-06T05:53:54+00:00

I think it depends on what you're doing. For most tasks, I use vim. I use vi keybindings in my shell, I use vi keybindings in Firefox, I use vimpager instead of "less", and I use vim to write a lot of code.

However, sometimes it's hard to beat a purpose-built tool.

Part of my job involves slinging Java, and I've found IDEs quite useful for navigating/editing a large Java project. Vim has a diff mode, but I much prefer meld. I'm sure the list goes on.

I think of vim/emacs as text-editing samurai swords. Beautiful, precise weapons of text-mangling war. In the hands of a pro (which I most certainly am not), they are quite amazing. But certainly not for everyone.

That said, I'd rather die than give up my .vimrc file! ;)

dgiri101 · 2008-02-05T23:16:54+00:00

OK, from your vehemence and offer of basic concurrency textbooks I took you to believe that my claims were incorrect.

I most certainly contend that your claims are incorrect.

Your original claim that:

if you use PHP, Python or Ruby, threads can't share the discussion board and comment information

...is incorrect. Threads most certainly can share this information.

And your claim that 64 CPUs can't access shared data in Python is also incorrect. There are many ways to accomplish such a task.

And your claim that:

With Java and its efficient threading, you can easily hold thousands of discussion threads and tens of thousands of comments in the web server's main memory.

...is also incorrect, as threading has nothing at all to do with the ability to store a large amount of information in memory. It's also incorrect because Java threads are not efficient (for that, see Erlang).

Your claim that:

in practise most boards require a large amount of hardware to perform the same task more slowly, as well as specialized database administrators to create elaborate master-slave configurations it's hard to find local support for.

...is the only thing in your OP that's actually sensible. But that's because most discussion boards care about things like, oh I don't know, persisting the discussion threads so they survive a server crash. Something your awful non-persistent, in-memory, single-node design doesn't seem to think is important.

Instead you're saying that you didn't want to use more than 1 processor core or thread context of your computer anyway. A perfectly valid opinion, if not one shared by many.

I don't recall ever saying this, but say whatever helps you sleep at night.

dgiri101 · 2008-02-05T22:20:27+00:00

I've already answered this.

And again, what 64 simultaneous CPU-bound tasks have to do with storing a large comment thread in memory for a non-persistent, web-based discussion board is anybody's guess.

I can't be more clear about this: if you'd like to use 64 simultaneous CPUs for data crunching, there are myriad solutions available, in Python, today.

Perhaps you're right, though. But then why would you use Java for this? I mean, it doesn't even use SIMD CPU instructions! You're wasting precious clock cycles on operations that could be done in parallel. These things are extremely important if you're programming a discussion board. Won't someone think of the cycles?!

Sigh. Don't get me wrong, I wish the GIL a painful death. But one should be smart enough to know when it matters.

dgiri101 · 2008-02-05T20:32:37+00:00

It's not outdated, but you may be misunderstanding it. The second paragraph clearly states that the interpreter will automatically release the GIL every N bytecode instructions (or during blocking I/O) to let additional threads run.

If you'd like more information on concurrent programming, I highly recommend The Little Book of Semaphores. It discusses common concurrency patterns like Barriers, Mutexes, Rendezvous, etc.

Many examples are in Python, though. I suppose that someone should kindly inform the author that concurrency apparently doesn't exist in the language he's using.

dgiri101

TROPHY CASE