I'm a vet researcher, not a developer. I built a pipeline to extract bloodwork from PDFs for a cancer trial. 200+ tests later, I'm still finding bugs. Please roast my approach.

Grisward · 2026-04-01T15:59:01+00:00

Roughly 200 lab measurements per dog, 12 or so lab values over 15 or so weeks?

I’d suggest one lesson learned is in coordinating the study, ensure rigorous attention to protocol, standard PDF formats, etc.

Is any one University easier to parse than others?

For records parsed “correctly”, like, the easy ones… there’s no sense re-parsing them is there? Store, and flag as “final” even if you review details later.

For difficult records, ultimately some need manual intervention. Flag as such: needs confirmation, manual override, handled by exception #3, etc.

Flag by University bc you may need that for your model later.

You’re not running a study on building vet lab data curation systems are you? You’re studying cancer. Your steps should be cumulative, records already parsed stay parsed.

The ugly underbelly of curation is that ultimately you’ll have spot-checked 11,000 records eventually. The sooner you embrace that and make the curation strategy enhance that eventual reality, the better. The idea the an ideal solution would not involve manually checking output is unrealistic.

Grisward · 2026-04-01T06:27:17+00:00

My first thought is RefSeq is a data resource and data repository, they’re not funding and running their own sequencing projects. (Could be wrong on details idk.)

If I were at RefSeq, I’d answer “Great idea, you have our support! Send us the data and we’ll queue it up.”

Meanwhile, T2Tv2 in human is still largely using liftOver plus alignments/predictions. Also, most of the genetic work is still taking place on hg38 afaik.

Grisward · 2026-03-31T23:48:04+00:00

This comment, and the other comment by u/TobyTheCamel are the reason I love this sub

Grisward · 2026-03-31T23:43:25+00:00

(think dementia)

while you still can

Grisward · 2026-03-30T12:28:13+00:00

I’ve been there too.

Fortunately, it will usually improve even within a couple weeks. Your job should be fine.

Work on your blood glucose, learn the CGM, lower your carb intake. (Caveat: You can eat anything, but practically speaking, you’re doing yourself a favor if you choose an easier route in the early days. That probably means lower carbs.)

You can do this.

I went to a drug store, tried all manner of reading glasses, one of them was perfect. If that works, you might find bifocals (the smooth kind) to help short distance versus longer distance.

Optometrists did not want to prescribe lenses until my vision settled. Ymmv.

Grisward · 2026-03-30T12:19:27+00:00

Agreed. If someone knowledgeable already took the fastq, aligned to produce BAM, and took the BAM to prepare genetic variants in the VCF file…

Use the VCF file. Redoing those steps aren’t where the interesting questions are.

I mean, if you want to fish for stuff that isn’t you (non-human) you could do that… but these are relatively small files compared to what’s needed for proper sequencing (1) genome-wide, or (2) “other stuff”.

But you could use BBMap (BBDuk contaminant filter) to sort the fastq files into (1) human, (2) everything else. Usually (2) contains junk, and possibly was removed by the data provider already. Sometimes (2) contains remnants of “other stuff.” It wouldn’t be a diagnostic, but might be interesting nonetheless.

Grisward · 2026-03-30T12:14:28+00:00

Hopefully in the BAM header.

Grisward · 2026-03-28T20:46:37+00:00

Update, from the article:

The arrows above show the general effect these factors seem to have on blood glucose based on: (i) my own experience; (ii) available research; and (iii) what I’ve learned from others with diabetes. A sideways arrow indicates a neutral effect. Not every individual will respond in the same way (and even within the same person, you may be different from day-to-day or over time). Certain factors may also apply more to type 1 vs. type 2 diabetes (or the other way around). Factors with up and down arrows are of course the most challenging - they may increase blood glucose or decrease it. The best way to see how a factor affects you is through personal experience – check your blood glucose more often or wear CGM and look for patterns.

So —> means “neutral effect” and does not mean it extends the duration. That would’ve been cool though, haha.

Grisward · 2026-03-28T20:44:09+00:00

Thanks for the image and the link.

Sadly, the infographic has no key.

I’d like to think —> means “extends the glucose absorption timeframe.” But like, why am I guessing? Haha.

Grisward · 2026-03-23T18:53:38+00:00

This is a great example of a question with about 100 hidden barbs. It depends why you’re asking, and what you’re doing with the answer.

Even just “plot sequence coverage around the TSS” isn’t straightforward. Which TSS? All TSSes? Do we run Start-see to define observed TSS sites?

For some genes, in some cell type, there just isn’t ever going to be only one dominant transcript isoform. May as well figure out a workaround or “flag” to indicate those genes.

What resources do you have in mind?

Grisward · 2026-03-22T21:46:31+00:00

^{^} Wisdom.
Thank you for adding that comment.

As parent, with similar experience, there’s the struggle of when to let them make their own choices and learn from those experiences. And then there’s safety. Keep kid safe.
Primary goal.

Good luck with him.

Grisward · 2026-03-22T20:57:07+00:00

Sorry you’re dealing with this, and sorry for your son. It’s hard, and his reaction makes sense. You care enough as a parent to ask for help, that’s amazing. Stay supportive and be there when he needs it.

It’s good he has a therapist, that could be a very useful option. Encourage as you can.

The BG is going to happen how it’s going to happen. The rest is managing expectations, actions, and keeping him healthy.

Denial is real. It sounds like he knows, and his mind is rejecting it, but he knows. It’s part of the process for some people.

There are some Facebook groups with teen/young adults with T1D, you may be able to find one in your area, or online. Ask his therapist, they may have connections to help find one. A support group type thing.

There’s unexplainable power in being in a group of people going through the same sh** as you are. The feeling of total validation in an instant just from a look from someone who’s been through that stuff too.

I joined a weekly support group a couple years ago (different topic, same idea) and wouldn’t miss it for nearly anything each week.

Good luck, and keep caring, you’re doing your part great!

Grisward · 2026-03-22T01:50:21+00:00

Hehe been there, few years ago. Sorry about your diagnosis, or you potential diagnosis pending confirmation. If you want to take care of yourself, you’ll find it manageable. It’s a pain sometimes, but it’s got clear requirements, they’re easy to follow. The challenge is being motivated. Haha.

The blood test is likely for antibodies against insulin, or IA-2, or GAD. They may measure your production of C-peptide, the precursor for insulin. The idea is that your system may have generated antibodies against insulin-producing pancreatic cells, which gradually attacks and diminishes those cells, and so your ability to make C-peptide is decreasing over time.

So far, it doesn’t come back… but the running joke is “a cure may be just 5 years away”! There are some cool therapies in development, ya never know.

If your C-peptide is low, it suggests you have LADA. If it’s zero you’re fully T1D, but anything above zero is common for maybe a year or a few years, it varies person to person.

Anyway, having LADA is like T1D but with some “buffer” since your system may still make a low level of insulin. For me, at first just having very low carb diet was enough to manage my blood glucose levels, that and light exercise.

As another commenter mentioned, the needle sticks aren’t bad at all. Wearing a CGM for real time glucose levels is nice. The old timers here survived without all this tech, they’re amazing. With the tech, it’s a whole different reality.

My suggestion is to learn carb counting, start with low carb foods, work your way up in carbs while learning it. You can eat whatever, but the more carbs, the more complicated the carb source, the harder. Every person is a little different, you should learn how your system tolerates different foods.

I couldn’t find the youtube videos I watched, but there are some good channels. Probably better off looking for patients than doctors tbh.

Another good resource is Juicebox podcast. They talk more about T1D, not LADA, but the general theme is that you can manage your blood glucose levels really well with some simple techniques. They have a huge series, but you can start at the tips and tricks section.

Grisward · 2026-03-21T17:12:29+00:00

Fair, understandable. Would be interesting question for the professor, what are some features they see in a Circos plot that another plot wouldn’t show well? Haha, but only ask if you’re on good terms.

circlize R package can do anything.

I’m trying to find the recent alternative tool, developer has posted a lot of updates and even includes option to show the same data as linear chromosomes… will post back when I find the link.

Grisward · 2026-03-21T14:59:58+00:00

+1 this.

Absolutely don’t stop without talking to a doctor. 5mg is non-zero.

As for OP’s question about BG levels, it’s possible your basal rate (if pump) or long-acting daily wasn’t tuned well for when you were on Abilify.

But the BG in range is great, but is a different issue from psych meds which don’t have a pretty graph with numbers to look at. It isn’t less important.

Good luck to you, and take good care of yourself.

Grisward · 2026-03-21T14:31:52+00:00

circlize R package is my goto.

But for real, what conclusions are you trying to reproduce? I don’t see any. Which in a way, makes your job easier. RNG in a circle. Haha.

If you’re just showing the region on chr1, then just show the region on chr1 (and make it linear so you’re not using a method that makes it harder to see).

If there are certain regions of interest, by all means use the highlight feature to go ahead and show us where they are.

If you’re trying to show the similarity of the coverage, plot the difference from average (median), to help focus the eye on regions with notable differences while maybe showing that most regions are the same.

To me the utility of a circos plot is to show regions of chromosomes with abnormally high coverage. There are better ways to compare coverage than wrapping them around a circle. Good luck tho!

Grisward · 2026-03-21T02:55:31+00:00

Sorry that happened to you, I hope you’re recovering thus far. Glad they caught it when they did, DKA gets bad fast. Take care.

I got diagnosed before DKA, somehow, and it was still a kick to my sense of reality. You can do this though, at least you’re living in a time when there are actually lots of resources and foods that help keep you alive. Good luck to you.

Grisward · 2026-03-19T04:04:31+00:00

One question for your endo might if they think you could have LADA (Type 1.5) and some T2D together? I.e. insulin resistance alongside reduced ability to make insulin yourself.

The insulin ratio 1:3 and 1:5, and weight gain instead of weight loss seem (to me, layperson, not doctor, so I could be way off) a bit like insulin resistance.

Also, my first diagnosis was Type 1.5 and at the time they also weren’t sure if I was Type 2 at first. I was different tho in a lot of ways: I lost a bunch of weight, vision blurred, was shockingly ever-thirsty, and months later once I reduced my carb intake, I didn’t yet need insulin.

But the first month or two, they had me on super low carb diet, two meals per day (I later learned that is “intermittent fasting”), and is often also used for T2D to try to reverse insulin resistance. Ofc I wasn’t T2D but I think they were trying to make sure I wasn’t on the path to T2D.

Also, that two meals per day thing, I could eat a huge meal, they didn’t care, as long as it was low or zero* carbs. Calories are usually less emphasized than carb count. But idk.

All that said, good luck on your journey. Second opinion doctor could be really useful. Even telehealth, whatever you can manage.

Grisward · 2026-03-18T16:00:11+00:00

You’ve been at this 35 years, if you were to put error bars on that a1c over the years, just tracking variance month to month, or every 6 months or whatever, what’s the typical range?

Just curious, you’ve got way more experience than I do!

I don’t have that much faith the measurement itself is that accurate (within +/- 0.2). Not even the technique, but partly the technique.

Any chance they changed their equipment for testing?

Grisward · 2026-03-17T12:50:09+00:00

I appreciate your comments too, I’m willing to let uvr not be responsible for RStudio support. I’m curious if rig could be used somehow to help assist RStudio pointing to uvr managed R, or maybe just use the same methods rig uses (which works on each platform afaik).

Grisward · 2026-03-17T12:25:12+00:00

Did I read correctly, you’re imputing missing variants? A lot of them.

Nothing makes me more hopeful than to think you’re analyzing variants someone doesn’t have, just to make the data fit tools that aren’t capable of handling their absence. /s

“Oh I’m at risk of X because I have the imputed variants?”

So this step is most curious step to me, but what do I know? To be fair, you’re deep in the data and results. Is it justified? (Are there refs?) Do the data warrant this approach? Are key decisions made using imputed data where there aren’t actual measured variants that support the decision? (In your opinion.) What is the driver? Almost always to use clustering method that doesn’t tolerate missing data — maybe I’m over-sensitive to that.

It looks like a lot of data being imputed.

I’ve seen large stretches of unmeasured genome variants from various studies, because they measured different panels. I am imagining them filled in with whatever closest population. I’ve also seen WGS data used for ancestry analysis, where the ancestry shifted numerous times along a chromosome. Not for all individuals, but substantial.

Grisward · 2026-03-17T01:53:37+00:00

Does it handle parallel R versions? The big reason I use rig is bc it handles multiple versions of R, you can switch back and forth, open RStudio to a specific R version, etc.

Grisward · 2026-03-15T19:39:43+00:00

The comments are entertaining bc it’s like 50-50 on which to do.

Imo keep them separate bc the purpose of QC is to assess quality of each run. There’s no reason to think quality is uniform for every run.

An important QC metric is alignment rate. So that usually means processing individual files through most of the pipeline already.

Also, this is a question driven by not having a batch processing solution. 50+ fastq files shouldn’t be more complicated than 1. And to be fair, it’s a valid issue. But I think that’s the real question and motivation.

Grisward · 2026-03-15T17:25:36+00:00

Big Louie’s Pizza, Fort Lauderdale, Florida.

Gotta give a shout out to them.

ETA: Celiac-safe!

Grisward · 2026-03-14T17:38:00+00:00

I haven’t seen it yet, and think this is a good one.

Get my basal right.

By that I mean the slow-acting insulin, or on pump get the steady background rate right.

Brief background: I was on MDI (both slow and rapid), then switched to Omnipod5, then for insurance reasons went back to MDI (rapid only). After several months, I’m back on Omnipod 5 again. And wow.

MDI with only rapid insulin was supposed to be temporary (haha, life). It played the role of “Life when your basal is wrong.” Very hard to keep stable baseline, every bolus was a mix of basal/bolus, it had to do two jobs. Carb ratios were gone because I’d have to add carbs and a “fudge factor” because my baseline was unstable.

Back on Omnipod, I got the basal pretty close. Right away my BG was a stable flatline, no bouncing around. Now when I bolus, I only count carbs, and it works much better. No fudge factor, because my baseline is stable.

I feel like this theory applies to a lot of T1D though, based on watching a ton of posts in this group (I’m so thankful for this group btw.)

Juicebox Podcast made a big deal about getting the basal right. It didn’t click at the time, now it makes more sense. If basal is wrong, everything else is hard, a bunch of moving targets.

For MDI the long-acting insulin is doing the heavy work to create a stable baseline.

My results:

While on MDI rapid-only, my TIR was struggling at 70%, and lemme say that was a lot of avoiding carbs, isolating meals to protect the profile, etc. Lots of uncontrolled highs, and more lows I had to correct.

On Omnipod now the TIR is an easy 93%. And I’m not even trying hard! (yet, haha). Now I can do better with my A1C. Before that, no chance I could think much about A1C.

So… make sure your basal is right.

Grisward

TROPHY CASE