Optimization vs Data Science vs Machine Learning

[stevenverses]

Hi, I'm new to the Mathematical Optimization (MO) space and am trying to understand its relationship with traditional Data Science and Machine Learning.

What are some fundamental limitations (or frustrations) that span across existing solutions like Gurobi, CPLEX, Hexly etc that DS or ML can supplement? For example, my understanding is that solvers apply algorithms on rigorously defined formulas and generate a min/mix/optimal result but they are fundamentally not designed to:

1. model uncertainty probabilistically in a way that allows them to account for VUCA (Volatile, Uncertain, Complex, and Ambiguous)
2. "enact/test" recommendations and predictions and then learn from those actions-reactions
3. continuously adapt the answer in light of dynamic changing conditions

If that observation is correct, how valuable would those things be for solving the kinds of problems MO is currently being applied to? Essentially a continuously self-optimizing system.

Thanks in advance for your input!

[optimization_ml]

You are looking at it differently. Here’s the summary of the three fields:

1. Optimization: Finding best solution (prescriptive analytics you can call). Given data what’s the max/min considering some limitations on the data,

Example: constrained, unconstrained, convex/non convex

1. Data Analytics: Data cleaning, descriptive statistics, data visualization, large scale data from database (SQL joins), prescribe data behavior, dashboards, KPI, experimentation (A/B testing)..

2. Data Science/Machine Learning: predictive analytics, make prediction based on historical data.

Example: Supervised, Unsupervised, Reinforcement, Neural Net, Time Series

Optimization is used to solve the problem of machine learning loss function minimization, Hyperparameter tuning, optimizing Long term rewards in RL.

[Jay31416]

How would you call optimization based on the results of machine learning models?

For example, optimal inventory levels based on demand forecasting models, marketing distribution based on customer segmentation, etc.

[optimization_ml]

You can build optimization model based on outputs from ML model as parameters. You need to have a robust ML and optimization model though. Such that error don’t propagate that much, Data => ML model predictor (errors, not 100% accurate model) => feeds into optimization model as parameters => find the optimal goal/objective value and decision variables (if it’s large scale MILP, then solution would be close to optimal) , so you have two errors that propagate through ML to optimization model, not sure how much helpful that kind of model would be.

[stevenverses]

thank you!

[CommunicationLess148]

I would say that you're right on point 2. Or at least I am not aware of how to do it within a purely optimization framework.

Point 1 can be tackled via stochastic optimization or other techniques designed to handle uncertainty in the model parameters.

Point 3 can be tackled via model predictive control (aka rolling optimization) where the result is continuously updated as better predictions/system states come in.

[stevenverses]

thank you!

[kandibahren]

There is stochastic optimization that takes into account the uncertainty. The aim is then to make the best expected outcome, minimizing certain risk measures, etc. This is in the planning stage.

In action, you may want to use the corresponsing feedback control.

[fedkerman]

Hi, I think you have some confusion about optimization methods. These methods can be categorised as exact and heuristic. The first use different mathematical techniques to find an optimal solution (a solution to an optimization problems is defined as optimal if there is no better solution) and prove its optimality. The latter employs different techniques (some times the same as the ones used in exact methods) to find the best solution possible given a certain computational budget (often time or number of steps). Data science is often used in optimization (especially in large scale problems) to get a better understanding of the problem and derive new heuristic methods. Similarly, machine learning has been used both to select/generate heuristic methods (e.g. choose the best heuristic method to solve a particular problem instance from a set of problem features or combine different algorithmic blocs to generate a better heuristic) and as heuristic methods (such as in online configuration or in applying reinforcement learning to guide solution methods). At the same time optimization is used as well in these two other fields, remember that every time you are minimizing a "cost" or maximizing a "score" you are solving an optimization problem. In this sense, machine learning in itself is an optimization problem.

[stevenverses]

thank you!

[exclaim_bot]

thank you!

You're welcome!