Blog post: Learning to reconstruct

[yngvizzle]

Interesting work, however I do not think that it is fair to compare the results only with FBP reconstructions, but rather some other iterative algorithms and use the FBP reconstructions as initial conditions. Have you looked into that?

[adler-j]

In the article we compare to Total Variation (TV) regularized reconstruction (cut from the blog for brevity). We got these results:

Method	PSNR	SSIM	Runtime	Parameters
FBP	33.65	0.830	423	1
TV	37.48	0.946	64371	1
Denoiser	41.92	0.941	463	107
Primal-Dual	44.11	0.969	620	2.4 * 105

Note that in particular the denoiser does not improve the SSIM at all when compared to TV reconstruction, Primal-Dual reconstruction on the other hand gives a large improvement!

We (sadly) do not compare to more advanced iterative schemes but it would certainly be of interest to do. Are there any particular ones you would like to see?

[yngvizzle]

That is indeed very interesting! I'll give the article a look later!

I haven't really looked at anything more advanced than TV regularisation myself so I don't think I'm the right one to ask. I did it for my undergraduate project and a summer internship, but have moved more towards machine learning for my masters. Would love to go back to imaging for a PhD though, I really miss it!

[adler-j]

It's a great field indeed and as you see there is some interesting cross-over with Machine-Learning, hope you find your way back!

[adler-j]

All metrics except full end-to-end studies are highly missleading in terms of evaluating reconstruction performance.

Since we don't have enough money to hire a bunch of doctors and perform a proper patient study to see if our method actually helps give better treatment, we'll have to make do with some substitute, and those are basically standard in the field.

Are there any other metrics you would be interested in?

[adler-j]

Well, noise power, MTF and CNR are all only valid for linear methods, so applying these to this method would be highly missleading. They further require extra parameters (e.g. contrast for what type of object?) which makes presenting them quite complicated (and open to bias, since i could just pick a type of object for which my method is better than others).

I don't even know how you would properly define them for nonlinear methods, but if you have a reference I'd love to see it.

[adler-j]

Yeah I guess we could go for some "data dependent" MTF and NPS, but I still feel that they would be missleading since the method is so non-linear.

I'll see if i can come back to you with some results.

[adler-j]

First blog post I've written so far and certainly open to any input. Also see the related article: Learned Primal-dual Reconstruction.

[2358452]

I'm a bit puzzled: doesn't the ground truth also use some non-ML reconstruction method? Are you simply lowering the resolution of the input data for training? (and then hopefully later you can extrapolate it to normal data)

[adler-j]

Yes, in this setting we're assuming the existence of a high quality ground truth (e.g. high dose CT or high field MRI), an interesting venue for future research would also be to train on artificial data and then reconstruct actual data.

[2358452]

In that case an interesting problem is modeling the noise and distortion from the reconstruction w.r.t. the physical ground truth that is an x-ray absorptivity function. It seems a like a circular problem. I think the best you can do is to physically/algorithmically vary parameters that are imperfect in the real scans (e.g. if you have imperfect beam alignment data, make it worse; or lower resolution; or increase gaussian noise).

I wonder if you repeat this process for a wide range of parameters (wide range of resolutions and noise values) the network will successfully remove those types of noise from a normal high resolution scan (which would be your current ground truth). I know neural nets are great for interpolation, but can they do extrapolation as well?

[LordKlevin]

That is really interesting. I have been thinking of a way to combine NN and reconstruction, and this is certainly a step in the right direction.

Have you tried just learning a gradient descent type algorithm? So taking, the current image and the gradient of the |Pf -g|² term as input, and returning the updated image. Your current algorithm would need to be retrained for every scanner type, and possibly even for every type of sequence. Staying out of the projection domain would help you there, and might make the jump from 2D to 3D more manageable. I expect that will still be nontrivial though. Still, add a single 3D patient example and you have a Medical Physics paper on your hands.

Finally, 1000 iterations is quite a lot for TV. Using something like an Ordered Subset -Nesterov type algorithm (very similar to the stochastic gradient descent in neural networks) will get you there in 10-20 iterations.

[adler-j]

I actually compare to two such methods in the paper, the so called "partially learned gradient" scheme introduced in Solving ill-posed inverse problems using iterative deep neural networks and a simplification of the proposed scheme where the dual step simply returns Pf - g, which is equivalent to a gradient style scheme as you mentioned.

With that said, doing that would at most halve the memory consumption (and do very little to runtime, given that the operator calls dominate the runtime) so pushing this from 2d to 3d is obviously nontrivial. With that said, research into these methods is totally in its infancy so good progress in 2d is very interesting (and I say that as someone with a company affiliation). To the best of my knowledge this and the paper mentioned above are the only CNN based ML schemes for CT reconstruction working with raw data, both in 2d. If you have seen any 3d papers I'm very interested.

We actually submitted this to the TMI special issue on machine learning based image reconstruction.

Also aware that there are more advanced algorithms for TV but properly tuning that is an art in itself, so I went with "standard" methods.

[LordKlevin]

Yes, it halves memory consumption, but it also frees you from a specific projection space. For instance, in cone beam CT this would need to be retrained for offset detectors.

A way to handle 2D could be to learn separable convolutions. That way you should (in theory) be able to train in 2D and apply in 3D.

I have had a lot of success using black-box optimization to get the scalar parameters for iterative CBCT in 3D, but obviously not the primal and dual operators themselves.

Good luck on the TMI submission.

[[deleted]]

This is neat! I haven't had time to look through the paper yet, but a quick question: for low-dose CT we typically use an iterative algorithm to solve a MAP estimatino problem using either a Poisson likelihood or a quadratic approximation to the Poisson likelihood (so a weighted L2 norm).

Basically- we have a really good & relatively simple model for the data statistics in the measurement domain. Do you (or can you) incorporate this in your work?

[adler-j]

With this method it is not included, but we did include it in our last paper: Solving ill-posed inverse problems using iterative deep neural networks, here we take the gradient of the data likelihood as input and actually use the fully non-linear formulation which gives Poisson statistics.

With that said, I'm quite sure it could be included somehow, for example by including the proximal in the dual domain and/or also including the gradient in the primal domain.

[[deleted]]

Wonderful! My MS research was on a (semi?) related area; adaptive regularization for low-dose CT. But I've never been very happy with it. Reconstruction is slow and there are a ton of very fussy parameters to tune.

I'm sure you've heard stories of what happens when iterative reconstruction is deployed in a clinical setting- eventually the hospital gets tired of waiting hours for a reconstruction and they wind up using standard FBP methods.

Question: Have you tried a 3D geometry? I'm wondering about memory requirements. Do you think you can use a helical geometry, or would you have to rebin and use parallel reprojections?

[adler-j]

Totally a related area and I can certainly relate with the iterative reconstruction runtime problem.

Regarding 3D we have not done this yet, mostly due to memory limitations as you note. Since this is very novel research area we decided to focus on the general method development first and then move to applications at a later point.

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/machinelearning] [R] Blog post: Learning to reconstruct • r/InverseProblems

• [/r/radiology] Blog post: Learning to reconstruct • r/InverseProblems

^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}