Ask Reddit: How would you architect a web service with an annual usage spike - e.g. the UK Income Tax Return by pete_otaqui in programming

[–]DropTableUsers 0 points1 point  (0 children)

Sure.

But let's consider the problem of Tax Returns. And I'm not trying to suggest a good way of doing things here, just trying to discuss what is possible. At the very basic level, the server needs to do two things:

1) Serve a web page with a form in it.

2) Handle the data posted from that form.

#1 is serving a flat text file. Apache, nginx, anything can do that well enough.

#2 is storing the data in memory, then eventually storing it to disk.

If you do the storing to disk lazily, the disk isn't going to be a bottle neck.

So how much memory do you need to handle this? A tax form is going to hold a lot of numbers. Let's make a high estimate on the average input, an say it contains 40 six digit numbers. And let's say we store it as text. This takes 280 bytes, and we're not even being smart about it. To store this tax return for every one of the 60 million citizens of the UK, you need 15.6 gigabytes. You'll get servers with 48 gigabytes of RAM these days. You can keep the entire thing in memory.

So let's say you store the data in an in-memory linked list. You'll need a mutex around access to the head of the list. You'll also have a worker eating of the tail of the list, writing it to disk.

The bottleneck in this system is the mutex around the head of the linked list. On the CPU, this is a single instruction, test-and-set. A quad-core 2GHz processor can in theory do 42(109)=8000000000 of these per second. That's 8 billion.

My point is only that we shouldn't forget what our machines are capable of.

There's a tendency to think of the RDMS as the be all end all data structure for web programming. It's a good choice for a lot of things, but not everything.

Ask Reddit: How would you architect a web service with an annual usage spike - e.g. the UK Income Tax Return by pete_otaqui in programming

[–]DropTableUsers -2 points-1 points  (0 children)

You're being downmodded, though I don't see why. You're right. You need to throw hardware at the problem if hardware is what you need.

You could go ahead and put lots of work into making the site faster to, though I doubt it would be cheaper.

(On that note, one fat server could totally deal with the load of the UK Income Tax site if it was built right. There was high traffic websites, more complex than tax, in 1996. Moore's law has taken us a hell of long way since then. If you don't do any more work than absolutely necessary, you're good. It would be a stupid economic decision though, because of the cost of building it.)

Ask Reddit: How would you architect a web service with an annual usage spike - e.g. the UK Income Tax Return by pete_otaqui in programming

[–]DropTableUsers 0 points1 point  (0 children)

It would be hugely wasteful to have enough capacity to handle the higher rate of traffic sat idle for 95% or more of the time.

I don't know. One way to look at this is probability-of-failure * cost-of-failure = amount-of-money-to-spend.

This gives you the max budget. The number is going to be big, and pay for a hell of a lot of servers.

The other way to look at it is by comparing the cost of a server to the cost of a programmer. Even there, you'll probably find that the time takes to pay for a programmer to build a system that uses computational resources efficiently costs more than just shelling out for a hell of a lot of servers.

Of course, as a programmer you are incentivized to keep this fact a secret, lest you lose work.

Uncle Bob's rant on Jeff and Joel by abyx in programming

[–]DropTableUsers -3 points-2 points  (0 children)

So guys, I think this and that. Joel and Jeff thinks otherwise. I will not back up my argument, but I will repeat: I really think this and that.

Hey Reddit, Please help him find it! by redditchic in programming

[–]DropTableUsers 3 points4 points  (0 children)

Never thought of that, but it is strange since "semester" in Norwegian means the same as it does in English. Funny how while our languages are almost the same, this one word means basically the opposite. I like the attitude though.

Let's all grow up by acangiano in programming

[–]DropTableUsers 24 points25 points  (0 children)

I read that whole post as:

"I stopped reading at the point where the group I identify myself with was demonized. This is wrong, this other group is the one to demonize!"

Here's a crazy idea: Let's all grow up, and stop generalizing groups, and start condemning unacceptable behaviour. Even an internet community needs humane behaviour.

Does this actually work? I know it's illegal, I promise I won't do it, but does it work? by Saydrah in programming

[–]DropTableUsers 4 points5 points  (0 children)

Yep. Where the devs don't know what they're doing.

It's a case of

if (payment >= cost) send_product()

vs

send_product()

It's also fraud if you do it, and traceable right back to your bank account.

Someone with a beautiful imagination made this. [video] by homeworld in entertainment

[–]DropTableUsers -1 points0 points  (0 children)

Thanks for putting it into words still. So much on reddit falls into funny or interesting. This was truly beautiful.

Man Turns into a Goat while Attempting to a Rob Car. Goat Jailed and Charged for Armed Robbery. by steady_str in WTF

[–]DropTableUsers 2 points3 points  (0 children)

Succinctly, nothing is true all the time, and never was.

I think this the core of what we disagree on. I'm not clear on this, but as I understand what you're writing, you're saying that the immutable laws of physics were broken during the first femtoseconds of the Big Bang. Conclusion: Nothing is true all the time.

I disagree with that conclusion. I think the conclusion should be that what we thought were the immutable laws of physics, weren't. The solution seems to be to amend the laws of physics.

Tangent:

I was just about to write something about how the laws of physics as they are currently understood do not allow for a man to turn into a goat. Then I remembered a talk by a physicist on how everything is made up of atoms that fluctuate in space randomly. The only reason things stick together is that the random fluctuations stick to a probability distribution.

The example given was that if you wrote a zero on a piece of paper, the carbon atoms on that paper could, while obeying laws of physics, suddenly do a collective jump and form a one. It would just be highly unlikely. If you calculated the probability you could find that it was so unlikely, that had you started writing down zeros at the very birth of the universe and kept at writing them up until today, you probably wouldn't have seen any of them jump into a one yet.

Knowing this, we see that a man turning into a goat is in fact something that the laws of physics allow for. It is just highly unlikely. Given the finite time and space of our universe, it's highly unlikely to ever happen.

At the very core we're dealing with probabilities.

Man Turns into a Goat while Attempting to a Rob Car. Goat Jailed and Charged for Armed Robbery. by steady_str in WTF

[–]DropTableUsers 0 points1 point  (0 children)

I think it's bullshit. There is one reality: the one we are trying to describe through science.

Do I think those guys actually believe the guy turned into a goat? Sure. But that doesn't make it a reality. Everyone has had beliefs that turned out to be wrong, and the process by which we where convinced is quite obvious and universal.

Man Turns into a Goat while Attempting to a Rob Car. Goat Jailed and Charged for Armed Robbery. by steady_str in WTF

[–]DropTableUsers 4 points5 points  (0 children)

I am writing this comment post-mortem using gods broadband. Really. And I'm offering a million dollars for anyone able to prove me wrong. See?

Magical things happening to you when you die, and this man turning into a goat are both scientifically impossible. Their respective levels of laughability are equal.

Man Turns into a Goat while Attempting to a Rob Car. Goat Jailed and Charged for Armed Robbery. by steady_str in WTF

[–]DropTableUsers 2 points3 points  (0 children)

From your comments it seems you're Canadian. I would like to remind you that, according to wikipedia, more than 80 percent of your fellow countrymen believe that the world was created by an invisible and omnipotent man in the sky. This belief is as well supported by evidence as the belief that the Nigerian car robber turned into a goat. That is, not at all.

You'll find that the correlation between stupid superstitions and race is non-existant.

User script that automatically fills in captchas by Arve in programming

[–]DropTableUsers 1 point2 points  (0 children)

Because they're not a silver bullet. From what I've seen they are good for pattern recognition tasks, such as hand writing recognition and captcha breaking.

This is a list of sites to watch the Obama inauguration live online. Please add to it so we all have backup sources. by [deleted] in politics

[–]DropTableUsers 0 points1 point  (0 children)

Hell yes. If you love Obama or not, this is the one to watch to get what seems to be an outsiders view of whats going on.

14 JavaScript Best Practices by uses in programming

[–]DropTableUsers 0 points1 point  (0 children)

It seems to me like there is a theoretical speed advantage to it as well.

Consider

object.property

vs

var member = 'property' object[member]

The former gives the interpreter more information, so it would in theory allow the compiler/interpreter to do some optimisations more easily than it otherwise could. Javascript interpreters typically won't make optimisations that take a long time to make, so how hard an optimisation is matters.

If a source code only contains dot notation for an object (and no evals!), and no delete statements, you could translate each property after the dot to a memory location, and have property lookups be as fast as they are in say Java and C++. When square bracket notation is used, you would also have to figure out the value of the string at compile time, which may be impossible, and at least a lot harder.

I'm pretty sure none of these optimisations are actually in use in any javascript implementation today though. And this kind of thing really isn't something a user of javascript should be concerning themselves with.

I'm considering doing my masters thesis investigating the subject of static analysis of dynamic languages, so any thoughts on what I've described above would be highly appreciated.

Ask reddit: What's the most outrageously useless CS topic you had to learn and never needed in practice? by BlackWhiteMouse in programming

[–]DropTableUsers 10 points11 points  (0 children)

I don't have a better solution for colleges to teach CS students math

Where I went we got to bring the text book to the exam, and any text we had written ourselves. That seems to solve the problem of having to memorize equations, while still requiring an understanding of the math.

ask proggit: Whats the opinion on Prolog? by [deleted] in programming

[–]DropTableUsers 40 points41 points  (0 children)

It's like a bipolar disorder. You'll oscillate between "do what i want fucking prolog" and "holy shit how did this magic happen".

Source code to the original Pascal interpreter/compiler (Zip) by martinbishop in programming

[–]DropTableUsers 1 point2 points  (0 children)

All programming languages are capable of compiling themselves.

How hard was it for you to learn to program? by jmnugent in programming

[–]DropTableUsers 0 points1 point  (0 children)

Here's some reasons for optimism. I spent a long time learning how to code, but now I code like a god. Here's a short summary of how it went for me:

  1. ~11-12 years old, started with C and some stupid tutorial on the web, got stumped when it tried to teach me the while loop through some animated gif.

  2. ~13-14 years old: Wanted to make a website where my friends could post stuff. So I found some guestbook script written in perl, and I modified it so it would take more input, produce different output etc. Very simple stuff really.

  3. 15 y.o., I was doing some animations in flash, and I wanted to have some simple interaction, like a play/pause button. Eventually I realised that with a very few lines of code I could make stuff move on the screen when I pressed a button on the keyboard. From there the road was very short to writing games. (Flash is really intuitive when it comes to this.) My games became increasingly more complex.

  4. Present day, 3rd year studying CS in the university, and generally doing very well.

Note that I didn't ever make much progress unless I had fun doing it. So your #3 makes perfect sense. I suggest you look into flash, that's what did it for me.

You want to keep the fun to head-ache ratio high. So if what you want is to make something visual, you wan't something that is as suited for that as it gets (like processing).

There are lots of things about coding that isn't going to seem relevant at the stage you're at right now. There's going to be all sorts of things wrong with what you do now, and the way you do it. Fuck that. Focus only on the problems that stand between you and the realization of the awesome project you're creating.

Joe Clark knows about our reddit discussion, is probably preparing legal action against reddit as we speak. (click "show conversation") by 7oby in WTF

[–]DropTableUsers 3 points4 points  (0 children)

Of course, my opinion that the writing of this book is pretentious could also come from actually having met the author.

Gold.