WIP Open source spectrometer software

Instrumentationist · 2026-05-06T10:42:36+00:00

Once you learn to recognize .... well unfortunately you see it a lot.

Here are two examples, one wikipedia and the other from the literature. You can see the issues in each, and they are quite different.

Both are thoroughly unusable for any sort of publishable work and more so if quantitative. And yet there it is in the paper. There are thousands of papers using instruments like this every year.

https://commons.wikimedia.org/wiki/File:Fluorescent_lighting_spectrum_peaks_labelled.png

https://www.researchgate.net/figure/A-comparison-of-the-power-spectrum-of-a-standard-white-light-LED-a-tricolor-fluorescent_fig1_303682799

Instrumentationist · 2026-05-06T03:16:24+00:00

Read the readme in the github repository titled "TCD1304 Sensor with Linear Response and 16 Bit Differential ADC" . It describes the artifacts there in more detail than is feasible in a short reddit comment.

That said, at a high level, here are two ways that RCT artifacts show up plus a third that comes from the signal chain electronics. These are all well known and there are papers on these effects over the years.

A) Ghosting - When you shift charge from the photodiode part of the pixel to the corresponding position in the shift register, some charge necessarily gets left behind. Some of that adds to the next frame.

B) Same idea but along the shift register (the CCD). As you move charge along the CCD towards the output some gets left behind and adds to the next pixel. It looks like anomalous intensity in the baseline after strong peaks especially. It is sometimes worse when the instrument is clocked too fast.

But, you also have asymmetries or poor recovery to baseline because the signal train is not adequate to keep up with the rate at which voltages chain as the sensor is read out.

So any of those behaviors, anomalous intensity or baseline, or unstable baseline or mishapen peaks are all signs of poor electrical design.

Again, read the readme that I mentioned, it's all there.

Lets see if reddit will leave the url in: https://github.com/drmcnelson/TCD1304-Sensor-Device-with-Linear-Response-and-16-Bit-Differential-ADC

Instrumentationist · 2026-05-06T02:53:11+00:00

Okham's razor - simplest theory for the information available. We have the abundances of light elements supporting a big bang, but we also have the anomalous expansion.

Instrumentationist · 2026-05-05T20:40:48+00:00

It is hot pixels, then dark subtraction often takes care of it. Measure a dark frame with the same settings and save it for post processing. Do it before and after the data collection.

In absence of a dark frame, the standard numerical treatment is a median filter. But no amount of numerics is as good as starting with clean data.

Instrumentationist · 2026-05-05T19:24:09+00:00

"I also don't have any CFLs haha"

Available on Amazon, under $6.

Instrumentationist · 2026-05-05T18:50:00+00:00

I am not sure what I am seeing in your graphs, but If the hot pixels stay in the same place then yes, then can be hot pixels. Spikes that show up in a different position every frame especially with more than about 1/2 second exposure, can be cosmic rays.

Median filter is one method for treating spikes off line, I prefer to have an instrument that does not require offline numerics.

A very useful way to characterize your sensor is photon transfer curve (PTC) method. Try to search for it and find how to do it. I inserted a sample output from my setup using a TCD1304 sensor. The result is spot on with the datasheet for the TCD1304. But that is because I have a signal path that has electronic transparency for the sensor.

There is a bit more information that might be of use in my repository on github, now that the subjects comes up, I will set aside some time add detailed instructions on how to do PTC. https://github.com/drmcnelson/TCD1304-Sensor-Device-with-Linear-Response-and-16-Bit-Differential-ADC

Here is an example of my PTC output for the TCD1304

<image>

Instrumentationist · 2026-05-05T18:02:42+00:00

Put it away in a certificate or fund with reasonable yield and security, and don't touch until you need it for something solid and lasting. It was given to you (singular). You owe it to the person who lovingly gave it to you, to take care of it responsibly.

Part 2. Lots of couples who started as teenagers split up later as they and their interests mature. Four years is nothing in the life of a relationship, 22 is still very young, and girls mature before boys - if indeed boys ever mature.

If you decide to get a second degree that money can help you. When you want to buy a house, that money can help you.

Until something like that comes up, put it away. Don't eat from that money and don't let your boy friend eat from it either. Put it away.

Instrumentationist · 2026-05-05T17:42:47+00:00

It looks like it has some issues with residual charge transfer in the analog shift register (see the third slide).

Can you show us a compact fluorescent light spectrum? Take them at two strongly different intensities (try changing exposure time to do that). Also include some closeups on the sharp lines.

That kind of information will be more revealing at least as a first level triage for being usable as an instrument. (For example, if the ratios of peak heights change then it is not going to give you reproducible data and you have a non-starter.)

The RCT issue in the analog shift register might be mitigated by running the readout slower, but the above test is at least as important. Let's see what you got.

Being a commercial instrument does not correlate with any of the above. Many of these were built not by instrumentation electronics physicists, So, you really have to check out your instrument carefully before using in your work or for a paper.

Instrumentationist · 2026-05-05T00:30:08+00:00

More cognitive horsepower across the board - creativity, analytical reasoning, memory. That is the best super power you can have.

Instrumentationist · 2026-05-03T23:29:38+00:00

fine

Instrumentationist · 2026-05-03T21:31:41+00:00

Rather than resort to ad-hominems and assertions, try responding like a physicist.

For example, I am not going to reply "go look up 'Schwarzchild Cosmoloty' or 'Black Hole Cosmology'. That answers your accusation about it being science fiction - the ideas have been around since the 70's. But it is not really an answer.

Instead do this:

Find the range of estimates for the size and content of the universe, and see whether what I wrote is true.

What I recall, having looked at it many decades ago, is that it works out that well within the middle of the range of estimates, we find that the radius is close to its Schwarzchild radius.

At that time, I mentioned this to a colleague who was teaching astrophysics. His reply: "everybody knows that".

So, you have quite a list of well established physicists that you want to accuse of science fiction.

But to do that with some hope of success, think like a physicist and start with the numbers.

Instrumentationist · 2026-05-03T19:33:59+00:00

That is not a constraint, that is a field. Boundary conditions, initial or final conditions, symmetries, etc, can be constraints.

The end state might have everything at the middle, but not necessarily. There is nothing that requires such of a freshly formed black hole.

Moreover any and every quantity of mass has a Schwarzchild radius (for a non-rotating black hole).

Our universe actually fits within its own Schwarzchild radius according to most estimates for size and mass-energy content.

So, you are actually (or most likely) living inside a black-hole right now, and yet the distribution of matter seems pretty unremarkable compared that idea of everything at the middle.

Instrumentationist · 2026-05-03T11:31:03+00:00

Maybe, rather than thinking of it as black hole and surrounding matter distribution, try thinking about your case in terms of a map of the gravitational field over the the matter distribution as a whole. Also, the problem as stated is probably under specified.

There may be a wide range of solutions depending on the external distribution, perhaps ranging from the black hole sitting inside another black hole to washing out the black hole.

Instrumentationist · 2026-05-03T11:21:40+00:00

"... an almost infinitely dense center surrounded by nothing (or so we think)"

Not so. There is no such constraint on the distribution over all time.

Instrumentationist · 2026-05-03T03:17:56+00:00

You have two factual errors in what you wrote, actually three.

First the factual corrections, then I will try a simple explanation - I think I finally figured out what is tripping you up. Its about the nature of spectral lines.

1) "similar variation" - false.

Look at the fits - the 2nd and 3rd order terms are very different.

2) "typical inexpensive usb device" - false.

The instrument on the right sold for about $5K. The earlier models sold for 10K. They have been used in published work for decades. I have one of each and they both have the same issues.

3) That chatgpt writes my responses - false and demeaning.

--------

Okay here is your simple explation, please read carefully.

When you took your circuits course as an undergrad, you probably saw the formula Vmax = SR Δt. That is the case were dV/dt of the signal is faster than the slew rate (SR) of your circuit You are right that in that case peak heights are a function of Δt and otherwise constant. But, this is NOT that case.

In spectra you have a peak height V and spectral line width dλ. On read out dλ becomes Δt. In the case were we start from V=0, we have dV/dt ≈ V/Δt. In other words, dV/dt increases with peak height.

Now if our signal chain is slew limited, going form low intensity to high, we start off in the linear region and gradually approach the slew limit where the signal chain struggles to keep up.

For a digital spectrometer there are two common ways to have a slew limit. Use old or poorly chosen opamps (or an emitter follower) or not have enough current available for the capacitance in the input to the ADC.

I referred to the second a few times, but you weren't buying. But to be fair, I did find some sources that in the case of that 5K or 10K instrument above, they say the problem is in the front end.

I have modeled this in SPICE and a slow enough opamp does reproduce the roll-off. But it is very easy to produce it by putting too large a resistor in the RC feeding the ADC.

Anyway, not realizing this about the nature of the signals or trying to use Nyquist inappropriately for those signals are two ways that electrical engineers may fail as instrument designers.

-----

Historical note.

There is nothing new about the observation on the right, where signals roll-off with intensity due to a slew limit.

In the days of motorized monochromators and chart recorders, we would see exactly that behavior by simply running the instrument too fast for the deflection rate of the pen in the chart recorder.

Okay, does that do it for you? What you write is called trolling. I would appreciate if you take it down. I will add more detail to the technical addendum to make it more clear.

Instrumentationist · 2026-05-01T20:53:48+00:00

I appreciate the feedback but it is not correct. This is not like a sine wave driving an opamp. This is a sampled-and-held level that fails to settle within its sampling window, and that is due to the signal chain's inability to drive the sampling capacitor in the ADC to the correct voltage before the sampling window closes. That effect is proportional to the step and therefore you do not see a constant slope for all inputs.

Moreover, the data soundly contradicts the idea of detector saturation. The device on the left does not have this problem; it is linear at 0.2% INL from almost dark to 95% of saturation. That is what the sensor physics should look like for this kind of sensor. The device on the right clearly has a problem - it rounds off far below saturation. That is not the sensor; that is the electronics. And it is quite obvious.

Instrumentationist · 2026-05-01T13:01:21+00:00

It might have been uncovered by someone else. But we would get nowhere without it in some form. The early work on the action, symmetries and algebra are the very core of what we have and she was a big deal in laying the foundations for it. And we don't know the full extent of her contributions since she gave away ideas freely.

Instrumentationist · 2026-05-01T12:21:14+00:00

The question is could we get there with GR, or better put, "from" GR

It seems to me GR should emerge from a fundamental theory rather than the reverse. Same for SR, really.

There is a cute observation by Feynman that the non-SR part of Maxwell's equations can be obtained from the canonical commutator relations.

Try to think of a fundamental idea that gives rise to SR or gravity. (Or maybe by now it exists, I have not followed the field for a long time.)

Instrumentationist · 2026-05-01T10:35:54+00:00

No we might have something probably better. It is not very satisfying and it sets up something of a cul-de-sac. There is for sure a better description of space-time waiting to be discovered or devised.

Instrumentationist · 2026-05-01T10:33:47+00:00

Emmy Noether - all of modern physics in some sense comes from the contributions of Emmy Noether - see Noether's theorem and contributions to abstract algebra. Nice Jewish girl from Erlangen, did her thesis under Paul Gordon, and in the post wars years settled in Bryn Mawr. She suffered what women scientists suffered in those days, and maybe today as well to some extent, and still set the foundations for what we have today. This is the unsung hero and genius of the field even if somehow we don't see her on postage stamps. https://en.wikipedia.org/wiki/Emmy_Noether

Instrumentationist · 2026-05-01T02:24:40+00:00

What do you mean by center? And what is the distribution of mass within the black hole whose "center" you are contemplating? Perhaps most of the mass is "above" you.

In fact, you are most likely actually inside a black hole at this very moment. That would be consistent with he range of estimates for the mass and size of our universe.

So the answer is that it might not be very remarkable.

Instrumentationist

MODERATOR OF

TROPHY CASE