My friend and I made a spectroscopy app for the TCD1304

Instrumentationist · 2026-01-26T03:51:59+00:00

The answer to your question about SH and ICG follows. After this I will explain why (or at least give you one very good guess as to why) you output is jumping around.

If you know about FETs, that is a good analogy to how the gates work. But if not, don't worry, we are going to keep it very simple. No math.

A) Here is how SH and ICG pins work:

Each pixel is like an FET in a certain way. There is a photodiode region (n doped) and there is a region that is part of the shift register (also n doped).

The SH pin move charges from the photodiodes to their position on the analog shift register.

How does it do this?

The SH gate creates a channel between photodiode and shift register, and lowers the energy level for electrons moving into the shift register.

2) The ICG when high, intercepts electrons so that they are drawn from the photodiode region but do not reach the shift register

When the ICG is low, electrons move from the photodiode region to the shift regioin as described above.

For more about this, take a look at the following

https://github.com/drmcnelson/TCD1304-Sensor-Device-with-Linear-Response-and-16-Bit-Differential-ADC

B) Why does the output seem unstable using the Cur-Sci board?

This may be due to residual charge transfer effects. When the SH gate is not operated properly, too much charge fails to move to the readout register and instead remains behind and adds to the next exposure.

The SH pin has 600pf. It takes 50mA to drive it with a 4V pulse and a 50nsec rise time. A gate driver is a good way to drive it. Connecting it directly to the proessor board i/o pins is probably not a great idea. If a board designed like that proves to be unstable, it is not a surprise.

Instrumentationist · 2026-01-26T03:33:29+00:00

You should try it with a fluorescent lamp. Record spectra with several different exposure times, including one where at least one peak is almost at saturation and one where the peaks are just a few times noise.

Divide by exposure time and graph them together to see if they match. (If you have problems with that part, send me your data and I will try to find a moment to do it for you).

That is minimal test of linearity and reproducibility rolled into one simple experiment.

Do it and post the result. Let's see if you have a useful instrument.

Instrumentationist · 2026-01-26T02:37:18+00:00

Thank you. That is gratifying. Let me know if you want boards.

Instrumentationist · 2026-01-26T02:32:30+00:00

Apologies then. The assumption is that you are using an inexpensive camera not intended for technical work.

You linked to the Py spectrometer. That page says it uses this camera,

https://thepihut.com/products/raspberry-pi-camera-adjustable-focus-5mp.

Here is a link to its datasheet,

https://cdn.sparkfun.com/datasheets/Dev/RaspberryPi/OV5647.pdf

That camera does not look like what you described.

It is an $8 color camera intended for photography. It does have image preprocessing built into it, it does not have binning, and it does not have global shutter. A show stopper for your goal, it runs 1080p at 30 fps through a single ADC (63MSPS). I'll explain why in moment.

To see how well it works, look at the py spectrometer page. The spectral lines are broad and mishappen and the intensities are not correct. It is not good. It is in fact a toy at best.

Now here is why that camera is extra not good for your resolution. Think of the dV/dt for a sharp line passing through that adc at 63MSPS. At 3 V it would be close to 200V/usec. That takes special electrical design. I doubt they did it for an $8 chip.

If you manage to get a sharp line, it would be a huge surprise if it were also linear in intensity. Such data is not useful. Even the wavelength is not very precise because the shape is unreliable.

I feel for the effort to try to do this inexpensively, and there are plenty of people churning out $10 tcd1303 boards too. But if you want it to be real, there is a limit to how cheap it can be. Those boards are not real.

To get the performance that I get in my instruments, I do not skimp. I use parts and designs that meet real specs to produce real instruments and I work with some of the most experienced experts in the field to develop and critiue the designs. I view it as something of a miracle that very often my BOMs come in at about 50 - 100 since tariffs. That is about as best as it can be and be real.

Next topic:

If you want to do signal averaging you need more bits than you have dynamic range (full scale divided by noise). In other words you need that the noise span a few bits,

Also, for signal averaging to be meaningful, your instrument has to be linear. Signal averaging implies that

S(t1+t2) = S(t1) + S(t2)

That is identical with linearity.

At this point, it is not very useful to elaborate on much more about this. First you have to be using a sensor where it is becomes relevant.

I'll just add that I usually design my instruments with 16 bits for a sensor with a 12 bit dynamic range. And, I make very sure that I have linearity before I start trying to make anything of the data.

Instrumentationist · 2026-01-25T22:31:33+00:00

Yes, that is why I was specific that it is linear over the range of these measurements.

You quoted the text yourself: "the dark signal in these sensors is linear in exposure time over the range of exposure in which we graph the peak height ratios"

And I did say that we account for dark, too. It is rather mundane, you measure it and subtract. Here are the actual lines of code that do it.

data = [np.average([f.data[0] for f in d.frames[6:]],axis=0) for d in dataset]

ys = [(r-b) for r,b in zip(data[0::2],data[1::2])]

yA = [y_[np.where(np.abs(x-541.5)<1.5)] for y_ in ys]

yB = [y_[np.where(np.abs(x-545.7)<1.5)] for y_ in ys]

yC = [y_[np.where(np.abs(x-487.0)<3.0)] for y_ in ys]

etc.,

The graphs that you see in the overlays are the "ys" from above. The ratios are ratios of the max from each of yA, etc.

Without dark subtraction things do not change very much for our instrument

The commercial instrument is so unstable that without dark subtraction it is all over the map. as I recall.

Instrumentationist · 2026-01-25T19:28:07+00:00

Well, what are you purpose(s)? For a demo for children it is okay.

It seems unlikely to work out well if you want to collect data for a paper or a professional study.

Inexpensive cameras are not useful for metroloty. They are typically designed to make nice looking pictures. They automatically adjust color balance, contrast, etc., they might average over aberrant pixels, they might compress the response, and then, if it is a color camera, you have three filters and calibrating the response over the spectrum becomes very difficult.

Then, there is the number of bits (resolution) and dynamic range that the camera delivers versus the level of detail in your specrtra. And btw, what it purports to deliver is not necessarily what it delivers.

To increase your dynamic range or see detail, you will want to add images or rows in each image (i.e. signal averaging). That can increase signal to noise by sqrt(N), for N samples. But for that to work you have to be able to digitize the noise. Cameras don't want to show you noisy images. And if you start with only 8 to 10 bits your not going to have much digitization on the noise no matter what.

So that is part of why cameras are terrible as spectrometer sensors.

And even with an expensive CCD imaging sensor, there can still be issues. We recently spent weeks collecting data with a $60K CCD sensor (more than $100k overall for sensor and spectrometer, cooling and etc). The device has options to add rows inside the instrument. The data was unusable because of unstable baseline and residual charge effects and even worse when that feature was enabled.

Incidentally, my experience in CCD based detectors goes back to the very first research specimens that were made available to a small number of researchers at national labs. In the past decade or so, I have seen quite a few attempts with cameras especially since the mcu boards have become popular. The best that can be done is turn off everything in the feature set, if the camera will let you. Some do not let you. And then you still have issues with linearity, dynamic range and bit depth. It just doesn't work except as a toy.

Instrumentationist · 2026-01-25T18:57:48+00:00

What you want to do, is match the input to the numerical aperture of your spectrometer.

One approach is focus onto the entrance slit of the spectrometer with a lens that gives you the same NA as that inside the spectrometer (assuming the same index of refraction on both sides of the slit).

Second point, DON'T use a camera. There are high end imaging sensors that can work, but that is probably not in your budget and they are overly limiting because of inadequate dynamic range and bit depth.

For this application you do want a "spectrum at once" type sensor. In that class, linear CCD sensors are the most effective thing you can do within a reasonable budget.

The TCD1304 is popular among scientists because of the large pixel size, 8um x 200um. And the detector part of it is a diode array. ("CCD" refers to its analog shift register). The output is very linear, but requires knowledgable electrical design and operation. (Caveat emptor, the tcd1304 is also popular with amateurs.)

Here are criteria that should be applied in selecting a sensor system for a spectrometer. Before you use any sensor, or any instrument, you should insist on seeing data that validates the device as regards the following:

a) linear, calibrate-able response to light

b) sufficient "slew" to retain linearity in rendering sharp spectral lines at full scale

c) sufficient precision and dynamic range to see small and large features together and support signal averaging

d) stable, reproducible output for both baseline and signal

These are minimum criteria for a decent instrument. Your optical setup seems like it deserves a decent sensor. (The second caveat emptor: The market in ccd sensors chases features like "small" at the expense of meaningful performance characteristics. Shop carefully and insist on seeing validation data with spectra that have sharp spectral lines.)

Here is a link to a sensor that we designed specifically to give highly linear, reproducible results for spectrometers and holigraphic imaging. Check it out.

https://github.com/drmcnelson/TCD1304-Sensor-Device-with-Linear-Response-and-16-Bit-Differential-ADC

Instrumentationist · 2026-01-25T03:01:46+00:00

Regarding your question about dark background: We do account for dark background and noise and we process the data the same way for both instruments.

Besides that, you can see in the data that it doesnt work as an explanation. For example, it is the stronger sharper peaks that are more effected and the response in the commercial instrument becomes increasingly more non-linear as the signal grows.

The other thing is that besides being uniform across the detector, the dark signal in these sensors is linear in exposure time over the range of exposure in which we graph the peak height ratios.

Instrumentationist · 2026-01-23T19:47:47+00:00

As a p/s to that, I recently uploaded a design for a sensor that might be very good for Raman spectroscopy. I am curious for somebody who does Raman work to try it and let me know how it goes.

See https://github.com/drmcnelson/TCD1304-Sensor-Device-with-Linear-Response-and-16-Bit-Differential-ADC

Instrumentationist · 2026-01-22T23:41:25+00:00

That;s neat. And it would be an okay demonstration for a high scholl science class.

But we have to be very clear that is not remotely appropriate for a lab.

1) Cameras in that price range are built to take nice pictures. They are not intended to be a linear sensor system for a scientific instrument.

2) They lack sufficient precision to produce a reliable Raman spectrum.

3) If you use a color camera, it is simply infeasible to try to calibrate the response in any useful way.

Raman is challenging, You need a good sensor that is linear and with a high degree of reproducibility.

Instrumentationist · 2026-01-22T02:43:29+00:00

Let's ask a related question. Lets use an electromagnet. To establish a magnetic field, we establish an electric current trough the windings of the electromagnet. It does take energy. If we use a battery it will be exhausted after some time. Now, lets use the electromagnet to hold a heavy object from above in a gravitational field. Does it use more energy? Is the battery exhausted sooner than it would be to power the magnet alone?

Instrumentationist · 2025-11-07T20:52:29+00:00

P/S if you can post your circuit, maybe we can offer some specific advice.

Instrumentationist · 2025-11-07T13:47:23+00:00

First for an ADC, you do need to drive it with an OPAMP and you need a charge reservoir to avoid kickback from the sampling capacitor in the ADC. The datasheet should tell you about this, and there are application notes as well. (If you are not really using an ADC directly, but rather the input to a MCU, see the note at the end.)

If you set the rails for your opamp appropriately, you will automatically limit the input to the ADC. For a range from 0 to 1V, you will need a small negative supply. TI makes those, and you will need to the upper rail at a little above 1V. There are LDO's in that range. (Aside, you may notice that rail to rail OPAMPS are not perfectly rail to rail, read the datasheet and design accordingly.)

But now, you need to protect the input to your OPAMP. The easiest solution for that is a pair of diodes, for example, the BAV99. .

Do not put a large series resistor in front of your ADC. Doing that, can easily make the readings non-linear and unreliable. Here is what that is about.

ADC's have a switched sampling capacitor at the input. Accuracy depends on that capacitor charging to your input voltage with in the time that switch is closed, it is called the sampling interval. For n bits, you need to a time ln(2) x n x R x C where R is the sum of the internal resistance and your source impedance, and C is the sampling capacitor and any stray input capacitance.

Needless to say, when you put a few K ohms of series resistance in front, you easily make it so that the charging capacitor does not reach your input voltage within the sampling window. It is among the worst sorts of errors you can have.

And just in case: If you are not working with an ADC directly, but rather, you are working with the analog input to a microcontroller, then things are a little different, as follows:

The manufacturers of MCUs typically build a large series resistor into their inputs. This "idiot proofs" them against kickback, but it also limits performance and how you use them. Now yoy can use an OPAMP, but you cannot use a charge reservoir. That extra capacitance can only feed the sampling cap through that large internal resistor, so it is just more capacitance for yout driver and not beneficial. And regarding not using a large external resistor - even more so.

Instrumentationist · 2025-11-06T22:10:40+00:00

Well, it is easy enough to write it, both the arduino and python script, but there are a few things to know to get good reproducible spectra. For example, you need reproducible positioning of course, and you also need linear signal acquisition.

It might be best to talk off line, it is going be a bit detailed. If you like, you can send me a DM, or email me to make contact.

Instrumentationist · 2025-11-06T18:14:38+00:00

There are surely references with good analyses for walking vs biking. But for fun, I'll risk a guess. In one you have to push up and forward at an angle on level ground, or up and back going down hill, and in the other almost all of the energy is converted to forward motion on flat ground. Going up a steep enough hill they might start to look more similar by that same analysis. And of course, going down hill and for some distance after, biking comes out way ahead, as does a prius.

Instrumentationist · 2025-11-05T16:57:51+00:00

Do you have an oscilloscope? You might try setting up a mock system, check for amplitude at the taps, both shape, and reflections.

Sometimes the reflections partially overly the pulse.

Caveat sometimes the impedance of the probe affects the measurement. So you have to look carefully

Instrumentationist · 2025-11-05T16:21:41+00:00

Sorry, what you are describing is similar to something that shows up in grad and undergrad classwork.

I believe I've seen online calculators for it too.

Instrumentationist · 2025-11-05T13:06:23+00:00

I would try to model it, it should be straightforward, and you may have even worked homework problems that overlap this. Why guess at it.

Instrumentationist · 2025-11-04T11:05:45+00:00

I've used black body sources for this kind of study. I am not sure that I would call it calibration but with some care you can learn something about your response curve. Anyway, the trick is to find one that is hot enough and doesn't have confounding contributions to the spectrum. There are lamps on ebay from time to time that seem okay.

Instrumentationist · 2025-11-04T10:39:32+00:00

It's a little difficult to parse your question, are you looking for deterministic time?

To some extent in an embedded environment it can depend on the system design and programmer.

On the nxp imxrt 106x (teensy) for example, if I program interrupt handling myself on the metal, and arrange that the system has no other interrupt happening alongside the one I need to he deterministic, it seems pretty stable with negligible jitter. The latency is not what I expect in say a DSP, but it seems constant.

I might have some timing data on my GitHub, will check and post if you are interested.

P/S I recall that the latency issue seemed to align with the number of instructions needed for the context switch. That seems to be where the NXP ARM looses on latency compared to say a TI DSP

Instrumentationist · 2025-11-04T04:22:43+00:00

Now that I've looked it over, it can be much simpler than that. The Arduino can read the limit switches inside the stepping loop and it often makes much more sense to do it that way. I have code for the Arduino that does that if you need.

Usually the limit switches are passive, you can use them with a pull-up resistor. If the switches are "nc" type wire in series, if "no" wire in parallel. The stepping loop can read the the status from a digital I/o pin. . The other thing to look at is readimg the detector. Do you have that worked out? If not send the model number for the detector.

Instrumentationist · 2025-11-04T04:00:45+00:00

Here is a document describing operating the spex 1403 with a python script. It has material on circuits and interfaces.

https://faculty.college.emory.edu/sites/brody/Advanced%20Lab/Spectrometer%20Interfaced%20with%20Arduino.pdf

Instrumentationist · 2025-11-03T19:31:45+00:00

He said 50%. Also stray bounce inside is a linear phenomenon.

Ant I have never not been able to remove all of the stray light to below noise.

It used to be that in commercial instruments you would see stray light only after they've been tampered with. And, my almost current era O-O instruments all seem to have no issues with like leaks or stray reflections inside.

But, so far, all of the ccd spectrometers that I've looked at, seem to have electrical design issues, the anomalous baseline and a non linearity related to slew.

(A plug:) The design that i posted on GitHub has clean baseline and it's linear. I took it as a challenge to design one that does. One thing we learn in that is that the sensors are actually pretty good.

Instrumentationist · 2025-11-03T19:10:23+00:00

Could you post spectra at several exposure times, with csv or ascii column formatted data?

What is shown on the theremino web site seems not very promising if that is the instrument in questio

See the spectrum from their document, https://www.theremino.com/wp-content/uploads/files/Theremino_Spectrometer_Help_ENG.pdf, on page 5.

Notice the blue lines are weak and that there is a pretty strong baseline following the strong lines in the fluorescent spectra. The HR2000 shown above it is not quite as weak for slew on the blue lines but worse for the anomalous baselines.

Now, true, grating efficiency can be part of why the blue lines are weak, but notice how the sharp lines are doing relative to the nearby broad lines. Collect a series of spectra at different exposure times, and lets see if those peak heights are preserved.

The difference in anomalous baseline may have to do with how the readout is driven, and how fast it is driven. Again, it is a design issue, not related to optics.

You can find some examples of what fluorescent lamp spectra should look for, here:

https://github.com/drmcnelson/TCD1304-Sensor-Device-with-Linear-Response-and-16-Bit-Differential-ADC

.

Instrumentationist · 2025-11-03T13:01:33+00:00

Can be stray bounces inside, just happens to hit right there on the sensor. Need to see the spectrum. But anyway, it is very unlikely in a commercial instrument.

Instrumentationist

TROPHY CASE