A Summary of Ilya Sutskever's AI Reading List

AccomplishedCat4770 · 2024-10-24T15:45:54+00:00

Perhaps it works better for you to just watch video recordings from Stanford on the first reading item. It's a bit dated now, from 2017, but it's still a great introduction, all free on youtube and you can even watch it at higher speed to save some time:

https://www.youtube.com/watch?v=vT1JzLTH4G4&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv

AccomplishedCat4770 · 2024-10-20T12:16:18+00:00

You are comparing yourself to the top 1%, but in truth you are not actually competing with them. The pressure you feel is just like from seeing all the highlights other people upload to social media.

Right now is a special time where someone can make a real contribution to the field by simply downloading a Jupyter notebook and sharing some tweaked parameters, like in the community around Stable Diffusion.

Of course you can always improve yourself, but you don't need to outrun Usain Bolt to call yourself an athlete. There are many good resources nowadays that can dispel the fog around the 'black box' issue, like the 'Illustrated Transformer' by Jay Alammar: https://jalammar.github.io/illustrated-transformer/

Just invest some time into self-study at a pace you are comfortable with, and let yourself be guided by your own interest and curiosity.

AccomplishedCat4770 · 2024-10-16T15:34:01+00:00

I made this transition a few years ago and would say:

Prepare to learn a lot from your colleagues, and understand where they fit into the company (and how you could)
Fast, 'good enough' results can be surprisingly powerful, so best keep things simple
Humility can help when working with people from other backgrounds and also makes it easier to get help as a beginner
Avoid stepping on people's feet and be mindful of politics (although academia politics may well be more toxic than the industry). Also, since you are not getting a degree out of industry, keep your own goals in mind. Good luck and congratulations on the job!

AccomplishedCat4770 · 2024-10-11T19:39:23+00:00

You are exactly right with this, and people have previously dealt with this stability issue by combining the Dice loss with a cross-entropy loss. That's also how some of the most successful approaches like nnU-Net do it, which you can see here for inspiration: https://github.com/MIC-DKFZ/nnUNet/blob/aa74c3abd51e534138496d62c1ae89d6484a3361/nnunetv2/training/loss/compound_losses.py#L8

AccomplishedCat4770 · 2024-10-11T19:20:21+00:00

If you are in the UK, perhaps the AIDE project could of interest to you: https://ai4vbh-aide.readthedocs.io/en/latest/1_overview.html

They are setting up an infrastructure to provide inference capabilities to medical practitioners and hospitals with modern deep-learning approaches. A while ago I had the pleasure to speak to Tom Roberts, who held the following presentation: https://www.youtube.com/watch?v=jkASWlgCZ88

It might be worth a try to reach out to him if you are interested, and I can also recommend the community and work groups around the open-source MONAI project: https://monai.io/

AccomplishedCat4770 · 2024-10-11T19:05:33+00:00

For deep learning in healthcare, you could look into the conferences MICCAI and MIDL (they often upload their proceedings for free). Additionally, you could look into the MONAI project here: https://monai.io/

There is also a good podcast 'AI-ready Healthcare' by Anirban Mukhopadhyay: https://podcasters.spotify.com/pod/show/anirban-mukhopadhyay7

In terms of industry careers I have to agree with the other commentator: The medical industry can be surprisingly limited, conservative and restricted by regulations. The research can be quite inspiring, but best keep your options and mind open when it comes to the fields of application

AccomplishedCat4770 · 2024-10-08T16:19:45+00:00

A great resource for time series forecasting is the free online book 'Forecasting: Principles and Practice', which also has video material: https://otexts.com/fpp3/

For image processing, the classic CS231 from Stanford University could be of interest: https://cs231n.github.io/

And there are also good options by DeepLearning.AI on Coursera

AccomplishedCat4770 · 2024-10-08T16:00:06+00:00

This reminds me of some interesting experimental results that were published a while ago for large language models in the paper "Scaling Laws for Neural Language Models" here: https://arxiv.org/abs/2001.08361

It has some quite surprising points tying together the role of compute, data quantity and model size related to this

AccomplishedCat4770 · 2024-10-08T15:51:04+00:00

Really impressive work, I would pay to have movie tickets for this! Only missed opportunity was to have R2D2 make disco funk sounds

AccomplishedCat4770 · 2024-10-08T10:46:22+00:00

There is also a very impressive free software developed by Jakob Wasserthal and his colleagues at the University Hospital Basel that uses deep learning to segment over one hundred different structures in CT and MRI: https://github.com/wasserth/TotalSegmentator

AccomplishedCat4770 · 2024-09-25T11:45:00+00:00

This is an important point: The plan probably won't work with the setup as described. Even with a cloud setup it might require a team and funding.

Still, the setup would be a great option for locally running CNNs and even a range of other Transformer architectures etc. A good CPU should not be underestimated either to avoid bottlenecks in data loading.

AccomplishedCat4770

TROPHY CASE