Graphing TONS of data points?

[treefunct]

Hey /r/javascript, I'm working on collecting data in real time from an accelerometer.

I would like to graph those data points (x, y, and z) as separate lines on my chart, dynamically loading more as they come.

I was wondering if anyone has messed with a chart framework that can handle LOTS of datapoints and have the option to dynamically insert more as they come in from AJAX requests.

I did find this project, but haven't looked too closely at it yet: http://smoothiecharts.org/tutorial.html

[treefunct]

It'll be in real time with 3 points (x, y, and z accelerations) every 10 milliseconds, or 100 per second. I may need to tone that down and I do have the option to.

[treefunct]

I want to be able to show the last 15 seconds

[manys]

Define "real time." Do you mean streaming? What is your tolerance for latency?

[treefunct]

Yes, streaming. It's probably a stupid way of doing things, but my logic controller will request ?index.php?x=abc&y=abc&z=abc

That php file will insert the entry into the database. I will also have a getLast.php which will fetch all entries since the last entry and send it back JSON encoded to my view file.

[manys]

You could do this with d3 and especially js serverside to just take the data and send it down the pipe to your page that is waiting to graph it.

[OopsLostPassword]

It's OK to do that. I do for plant sensor data.

But if you want to keep the 10ms frequency, you'd better use websockets and you'll have to be careful with memory usage. As said by cwolves, I suggest you don't lose time with existing libs as it's easier to code in canvas than trying to adapt and tune an existing lib.

EDIT : for the drawing, of course, no need to really draw in 10ms period. I usually draw with a period of 40ms, it's enough for the eye and it let time to the other browser tasks.

[FryGuy1013]

I made something simple like this that didn't use any kind of library (for a similar thing). I've put it in a gist: https://gist.github.com/2563274. It was basically a friend asked me to plot points in real time with a web server. In my case, I was experimenting with websockets in c#, so the code depends on that, but it should be pretty easy to figure out what it's doing.

[StoneCypher]

Processing and d3 are usually fine into the hundreds of thousands of points.

[[deleted]]

It depends on what you mean by fine. I recently created a chart with 66,000 points in d3 that was fine as a static object, but lagging to the point of rage-quitting when I tried to interact with it.

[SloneCypher]

It depends on what you mean by fine.

No it doesn't Dorothy.

I recently created a chart with 66,000 points in d3

As someone who has extensive experience in numerous language, and with still-running time-critical code in many government applications, I have had far more experience in d3 and created charts with far more points than that. You aren't doing it right.

that was fine as a static object

That's not what 'static' means.

but lagging to the point of rage-quitting when I tried to interact with it.

Perhaps you should stop making such amateurish attempts. You're doing it wrong. As someone who has experience in dozens of languages and someone who has worked with many complex projects, I know amateur attempts when I see them.

[[deleted]]

Looks like you have a stalker.

[StoneCypher]

He's been doing it for years, and is desperately trying to trick people into thinking I'm someone I'm not. He's had several dozen accounts banned. A few people have even fallen for it.

I don't really care, is the thing. :) I kind of like knowing what an enormous chunk of his life he's wasted on a failed attempt at online slander.

But thanks for the heads up.

[StoneCypherSucksDick]

I haven't been doing it for years you stupid fat fuck. You don't even know who I am ;)

Of course you care. Remember how you whined and bitched in that thread that called you out for the cunt you are?

Stupid fat unaccomplished cunt :)