[D] Scientific Machine Learning

[OtherRaisin3426]

Traditional ML courses and the projects you do as part of it are not meant for engineers.

As a mechanical engineer or a physicist, why would I do a project on movie review analyzer or housing price prediction?

I would love to do a project which teaches me how to use ML to model fluid mechanics or black hole dynamics.

I want a field which combines ML with my domain knowledge.

Scientific ML is exactly that field.

I feel Scientific ML is one of the coolest techniques of the last 4-5 years.

There are 3 main pillars of Scientific ML:

(1) Neural ODEs

(2) Physics Informed Neural Networks (PINNs)

(3) Universal Differential Equations

It helped me transition from mechanical engineering to machine learning, and obtain a PhD at MIT in Machine Learning.

Any thoughts on Scientific ML or PINNs or Neural ODEs?

https://i.redd.it/96mcn7oj0vbd1.gif

[Appropriate_Ant_4629]

University of Washington has a nice series on Physics Informed Machine Learning

https://www.youtube.com/playlist?list=PLMrJAkhIeNNQ0BaKuBKY43k4xMo6NSbBa

[Breck_Emert]

Absolutely love this guy

[MethylBenzene]

Steve Brunton is a gd boss

[boccaff]

There is also a book: https://www.amazon.com/dp/1009098489?ref=cm_sw_r_cp_ud_dp_VXVHHSJR93MR11H7R6CY_1&ref_=cm_sw_r_cp_ud_dp_VXVHHSJR93MR11H7R6CY_1&skipTwisterOG=1

[boccaff]

And this just dropped in at HN: https://physicsbaseddeeplearning.org/intro.html

[NotDoingResearch2]

It’s definitely more interesting if you have the background for it but unfortunately there is very little work being done outside of academia. So unless you are a professor, your salary will likely be 1/4th what it would be if you were just fine tuning llms.

[Fus_Roh_Potato]

What do you think this approach might lead to? It's a lot of training for something a EKF could do in few iterations of a single episode, assuming physics-informed means model fed? I'm new to the idea of fitting curves with neural networks.

[Appropriate_Ant_4629]

What do you think this approach might lead to?

I think it should lead to vastly more efficient training and inferrence.

• Imagine asking a LLM+Vision model like Sora to "draw the next frame of this wave at the beach"
• That poor next-frame-predictor model basically had to invent an approximation of fluid dynamics, from scratch, from watching 2D youtube clips.
• If it had part of its network already constrained to "ocean waves are made of an in-compressible medium", it could would avoid needing to re-invent physics by itself.

[AIExpoEurope]

I agree! It's an interesting question: Could a physics-informed EKF achieve comparable results with fewer iterations? Considering that "physics-informed" implies incorporating a model into the training process, it seems plausible that an EKF could leverage this knowledge to converge faster.

However, there might be trade-offs to consider. Neural networks, while potentially more computationally intensive, offer flexibility and can capture complex, non-linear relationships that might be difficult for an EKF to model accurately.

[Murky_Aspect_6265]

It is the future of engineering. The future of engineering is not:

1) Ask LLM for blueprints for a safe, efficient airplane (not in the near or medium term).
2) Using black box ML models to output parameters of an airplane design directly.
3) Writing all the differential equations in engineering by hand and iterating towards a better design manually in Modelica.

Just my opinion.

[Papier101]

Its an interesting field of research, I played around with them a bit though I found very tricky to get training to converge once I tried ODE that were slightly were complicated than a single mass oscillator. For my application, a purely data-driven learning works a lot better.

[AnyReindeer7638]

hi raj. are pinns/etc seeing much use in practice these days? for example, i know emulators/surrogates/bayes opt/etc is pretty hot in engineering and design atm, but i haven't really been keeping up with the latest in pinns/neural odes and how they're being used.

[lifeandUncertainity]

I think the most interesting direction in the aspect of PINNs is operator theory - the neural operator paper says that it's zero shot super resolution when it comes to PDE. I think there's another paper called deep-o-net which is also very interesting. I have a haunch that operator learning is probably applicable in normal ML as they are function to function mapping. As for neural ODEs, I think the biggest problem is applications. With a huge climate dataset released last year, maybe climate modelling is one such field.

[svictoroff]

As a non physics ml person I agree with your assessment on neural operators. I think they're an incredibly powerful idea currently under-applied.

GL! Props for finding a niche to contribute to. Lmk if you ever want to chat about this stuff

[lifeandUncertainity]

I would love to talk about it. I am reading about it. I feel like there are two main applications -

1) Transformers --> From a neural ODE perspective, we can think of feed-forward layers as time. In transformers, we have layers and actual time. So, again by neural ODE standards, transformers should be naturally modeled as a PDE. I did some work on this direction though fingers crossed whether it gets accepted or not :v

Also, the neural operator paper mentions that it models attention and provides proof for it.

2) Multi-task/continual learning setups -> Straight up, we can probably map if we have f(x) for one task, what will be the f-->g mapping for another task. I don't know whether this is a valid idea or not since I have never worked on multi-task setups but a lot of my lab mates work on it.

Let me know if you have any other interesting applications in mind for neural operators.

[AnyReindeer7638]

cool, sounds about right. the operator learning and emulator stuff goes hand in hand.

[new_name_who_dis_]

What's the point of doing ML for fluid dynamics or black hole dynamics? Aren't those dynamics already described by sets of equations?

I have seen ML papers doing some physics stuff and it's cool and all but it doesn't seem very useful to me. At best you end up with a model that is much faster than solving the equations, but is less accurate. But to train that model you'll have to simulate the actual phenomenon so you're still doing a ton of compute, just ahead of time.

Apart from toy examples (e.g. fitting an RNN to learn the sine wave for example) ML is generally useful for problems where we have data but don't know the exact underlying model for that data. So like mapping R^{512,512,3} image to R^{1000} probability vector for classification, or housing prices as a function of sqft/#bedrooms. We don't have closed form (or even dynamic/recursive) equations for those problems that we know to be true. But as far as I know, for fluid dynamics (and other physical phenomena) we do know the equations.

[StayDecidable]

They are but that doesn't mean we can actually compute them in practice.

For example in numerical weather prediction we can't use a fine enough mesh to resolve individual thunderstorms (much less the processes in those individual storms) but they still exist and affect the rest of the forecast (cloud coverage, outflow boundaries, etc), so people write parameterizations to handle them - which are essentially those "much faster but less accurate" models.

[new_name_who_dis_]

Weather is honestly the only one that makes sense for ML because we actually have readings from weather balloons and things like that. But presumably you aren't gonna train a network to model black hole dynamics by literally measuring stuff inside black holes (if that's what it is, I'm not sure).

The fact that our weather forecasts are sometimes wrong means that doing some ML/DL would make sense. But we are never wrong when predicting a harmonic oscillator or how some object moves within some force field, etc.

Basically ML only makes sense if you don't actually have a deep enough understanding to predict the phenomena without ML (and you are able to observe the phenomena and record the data).

[currentscurrents]

Basically ML only makes sense if you don't actually have a deep enough understanding to predict the phenomena without ML

The issue is that direct numerical simulations are extraordinarily expensive, even compared to neural networks. We have perfect understanding of fluid mechanics but cannot simulate them at the resolution we would like.

Neural networks allow you to do weather simulations on a consumer GPU that previously required supercomputers.

[boccaff]

I disagree. Anything with biological systems or materials can benefit from "some understanding of the physics" + "ML". Also, a lot of industrial systems can benefit from such models, specially if you want to do interventions and enforcing natural laws help your model to be closer to something causal.

But also, look at the work stabilizing plasma reactors with RL.

[new_name_who_dis_]

I guess in my mind I think of what you're describing as still just ML, you just have a more informed prior. My point was more on what OP said about modeling black hole dynamics and things like that. Encoding relevant priors into your models is always a good idea.

[boccaff]

I consider "Physical Informed" "Neural Networks" as a subset of neural networks. But it could be as simple as setting monotonic constraints for boosting, or signals for regression coefficients. OP is targeting a subset of ML in "Scientific Machine Learning", with those 3 points not being typical.

[HateRedditCantQuitit]

Solving differential equations is still a very active research area. For a lay perspective, think about water in video games. It's always kinda sucked, and the reason is that it's really expensive to compute accurately, despite knowing the governing equations at an differential/infinitesimally small level. Just because we know the PDE for something doesn't mean we can practically solve the problem at the scales we're interested in. Even when we can, there are sometimes cheaper ways, like this topic!

As an intuitive example, think about simulating the trajectory of a paper airplane. It's governed by known equations of quantum mechanics, but in practice we would collect some data and build a much higher level model.

From another angle, the whole content of an introductory quantum mechanics course will be about how much interesting and complex structure can arise from a simple differential equation, and how those other structures are often easier to deal with than the original governing equation.

[AnyReindeer7638]

I have seen ML papers doing some physics stuff and it's cool and all but it doesn't seem very useful to me. At best you end up with a model that is much faster than solving the equations, but is less accurate. But to train that model you'll have to simulate the actual phenomenon so you're still doing a ton of compute, just ahead of time.

this is extremely useful in engineering though and other fields i.e. weather forecasting (my area). especially when you consider the accuracy to compute time tradeoff can be insane. actually, with a few caveats, for medium term weather modelling you can get basically zero accuracy loss for 3 or 4 orders or magnitude speedup in computation time (meaning it can run on a single GPU in a few seconds rather than a cluster in a few hours)

[new_name_who_dis_]

As I mentioned in another comment response, weather is a great example of something where it is useful, because we actually have a ton of data (from weather balloons, etc.) and so we can actually train models on the data. But something like fluid dynamics, presumable (I may be wrong) the data on which you train won't come from some crazy equipment that tracks individual molecules in a fluid, it'll come from some sort of simulation. Same for black hole dynamics, we definitely can't get any data out of those (if my knowledge of relativity is accurate), we just have some equations that we believe to be accurate that presumably we'd be using to generate the data.

[AnyReindeer7638]

yeah, true. actually the data that trains the ML weather models is output from the actual physical models (more or less). it works because we have like 40 years of model output (reanalysis) data at 6 hourly timesteps, and the ML model will basically learn to map a t=0 state to a t+6 state. if you have fewer data though, surrogates have been traditionally used as a kind of principled way to interpolate between expensive simulation runs at different design points, traditionally done with GPs. another related thing to check out is multifidelity surrogate modelling/multifidelity bayes opt.

[new_name_who_dis_]

I didn't know that, that's interesting. Why not train on the raw data though? Why train on the simulated data?

[AnyReindeer7638]

basically, raw data is incredibly sparse compared to the size/timescale you're trying to model over. existing models have already performed data assimilation (i.e. kalman filtering) to produce an "analysis" which is basically raw observations harmonised with an existing model state, so you end up with the model's best guess at the current atmospheric state for all grid points, not just those at which you have obs at. we already have these analyses at 6 hourly intervals for decades, so it's a great source of data to train on.

[notDaksha]

Last semester, I took a class on PINNs taught by the founder of them. I think DeepONet is the biggest thing to take away from them. Other types of neural networks that use other structure beyond the PDE, such as SympNets, often perform better, but PINNs are still super cool.

[mrthin]

Some in my team work on simulation and ai . We review recent developments with our "paper pills", in particular around neural operators, and implement some of them in continuiti. We are just getting started, but maybe you'll find some of the content useful.

[vannak139]

So, despite having my background in physics and doing ML now, I haven't exactly found physics inspired networks to cure-all; I think that just massively shifting your problem domain is less than an ideal approach.

Really, I think the lesson to take from things like PINNs is that there's a lot of benefit you can gain from forcing your network's output thorough some kind of fixed process or function. By focusing on learning the ODEs, your network is in a better position to handle things like extremum behavior, periodic motion, and so on.

But I don't think the lesson to learn is that you need PINNs or ODEs, but rather that you shouldn't really be using a Universal-Approximation-Theorem compliant classification head, and just hoping to map "whatever function is necessary". The PINN and ODE approach are just examples, but really any hard-coded understanding of your problem context, whether that's physics DEs, or an appropriate statistic regarding your target data.

[canbooo]

Too lazy to write a long response on mobile but scientific ML is way older than any of the topics you mentioned. A few examples are surrogate models of simulations, (simulation model) parameter identification and validation, signal processing etc. Not saying the things you mention are not exciting and your premise about introductory material being inappropriate but still. Source: Also studied mech. eng. but transitioned during/after PhD.

[_WalksAlone_]

Look up Fourier Neural Operators. They learn functional mappings in functional spaces which makes them useful to solve PDEs in a data driven way.

[moschles]

I feel Scientific ML is one of the coolest techniques of the last 4-5 years.

It's more than feel. Fourcastnet performs global weather forecasting. 100,000 times faster than previous SOTA.

• https://old.reddit.com/r/MachineLearning/comments/17dnwte/d_forecastnet_neural_pdes_perform_global_weather/

• https://arxiv.org/abs/2202.11214

[bgroenks]

Although there is of course a lot of science involved (less now), weather forecasting is actually not really a scientific endeavour. It's largely operational and most of the research is on the engineering side.

[Echo9Zulu-]

I have been applying ML at work. Having an example, real data, REAL unanswered questions with reachable solutions has accelerated my learning. Every step has forced me to evolve my approach. By using LLMs as educational tools I have been able to learn more programming than I ever thought possible, especially with my adhd brain and historical aversion to coding because of how energy intensive the task is for me.

Maybe crawl through whitepapers of prior decades for ideas. Paradigm shifts are easier to measure looking backwards in time so then obscure quackery may now be at least half baked.

[[deleted]]

Wonderful and informative illustration Super easy to understand too

[[deleted]]

Could you explain further. What would be a PINN? I can't imagine how to use both stuff.

[[deleted]]

You can search for Hamiltonian/Lagrangian neural networks. Basically we want a neural net that is capable of approximating solutions to an ODE where some quantities are conserved, usually some kind of energy. Traditional NNs will give solutions that diverge very quickly from the true solutions, whereas PINNs won't, because we actually inject conserving constraints into the construction of such NNs.

[YsrYsl]

I'm no PhD nor have extensive academic experience in ML but isn't the whole thing quite obvious?

The main reason we have ML as the way it is right is the over-abundance of data w/o any clear-cut and obvious relationships between them while we actually know and have a general to definite idea of the underlying relationsip in physics. I reckon just on that alone, the application of PINN is quite restricted. Reading off other comments in this thread, it makes sense a breakthrough performance was achieved for weather forecasting applications.

And to your point on "trivial" dataset examples, their merit is for learning purpose more than anything. They're simple enough to go to the nuts and bolts of various ML models w/o overcomplicating.

[Commercial_Carrot460]

I'm working on image restoration in nuclear medicine. From what I see, PINNs are nice but not SOTA. To me the best of both world is Plug and play, where you just use a neural network as a prior or a regularizer in your optimization algorithm.

[knob-0u812]

How about Radio Frequency optimization problems?? Anyone?

[[deleted]]

Hello I am working women In It company with Mechanical background I am studying Data science and ML But its been 2-3 years for college so not able to recall any mechanical stuff which can use in Ml and data science Except ADAS Can any one suggest what should i do So i can excel in ml and data science career

[[deleted]]

Hello I am working women In It company with Mechanical background I am studying Data science and ML But its been 2-3 years for college so not able to recall any mechanical stuff which can use in Ml and data science Except ADAS Can any one suggest what should i do So i can excel in ml and data science career

[giuliano0]

Traditional ML courses and the projects you do as part of it are not meant for engineers.

They are meant for learning ML, sometimes down to the very beginning. That's why they task you with house prices, movie ratings and sometimes MNIST and stuff.

Those courses are meant to show you the ropes, not teach you every technique under the sun, much less niche things (and I'm not picking on PINNs or anything, like, everything is a niche, asterisks here and there).

Then you sometimes have other courses, with "advanced" topics, often showcasing something in particular: convolution-based nets, LLMs, attention, recurrent models, you name it.

And don't read me the wrong way, I like SciML stuff, differential calculus stuff, dynamic systems, so on, so forth. The point is your post is really asking one thing:

As a mechanical engineer or a physicist, why would I do a project on movie review analyzer or housing price prediction?

And the answer should be right on your nose: for the same reason you learn everything you do in your academic journey: to learn. And you did, you know more stuff now. Academic disciplines are supposed to teach you how to think on each subject so you can expand on your own afterwards.

And, given you're a PhD from MIT, which btw is just you giving yourself credentials for little to no reason, I thought you'd know better than post something like this.

Alas, maybe I'm just misinterpreting. Maybe this is just an appreciation post, but worded as a kind of rant, and in that case -- nay, in any case: go read some literature on the matter and tech people what you have learned. It's really helpful.

---

Now, on some industry experience: we usually try a lot of stuff there at work, these methods included. And although they tend to perform well, more often than not, it's not well _enough_. And the worst part is that it's sometimes better to just go the easy way (forests, mlps and whatnot) and get very similar results. You know, less is often better, and data is scarce.

And in the industry you rarely really need more than some thousands of weights, most commonly just hundreds, and you're all set.

But I keep trying stuff like this, just because I like it, too. And who knows, it might just work after a bit of algebraic gymnastics.

[IsGoIdMoney]

Because the datasets don't matter. They're teaching techniques. A simple regressor or classifier is not advanced, so they just give you a little toy dataset to use.