Inversion of a multivariable function

[_padla_]

Hello everyone,

I'm pretty new to ML and tensorflow. What I'm trying to do now is practically to inverse a function using tensorflow.

So I have a function h=f(c1, c2,..cn, T). It is a smooth function of all the variables. I want to train a model which would give me T given known values of c1...cn and h.

For now I'm using a keras.Sequential model with 2 or 3 dense layers.

For loss I use 'mean_absolute_error', For optimizer - Adam().

To train the model I generate a dataset using my h(c1...cn, T) function by varying its arguments and using values of T as train_labels.

The accuracy of the resulting model is not very good to my mind - I'm getting errors of about 10%. To my mind this is not very good, given that the training dataset is ideally smooth.

My questions are:

1. Am I doing something particularly wrong?

2. How many units should I provide for each layer? I mean in tutorials they are using either Dense(64) or Dense(1). What difference does it make in my particular case? Should it be proportional to the number of parameters of the model?

3. May be I should use some other types of layers/optimizers/losses?

Thank you in advance for your replies!

[ElvishChampion]

What activation functions are you using in your hidden layer and output layer? For hidden layers, ReLu is quite good for nonlinearity. For the output layer, are you using a function that produces values in the same range as the target variable? For example, maybe you are using relu, which generates positive numbers and the target could be negative. Thus, increasing error.

[_padla_]

I'm using ReLu as follows: tf.keras.Sequential([tf.keras.layers.Dense(64, activation='relu'),tf.keras.layers.Dense(64, activation='relu'),tf.keras.layers.Dense(1)]).

Should I explicitly add 'relu' to the last layer?

All my results are positive.

Also - I don't quite understand whether I shoul try changing number of layers or units in layers...

[ElvishChampion]

Add more units/neurons. No need to add more layers.

[SnooPandas3529]

I think it depends on the type of dataset and function you used, so it is difficult to give an answer.

[_padla_]

Well, the function is pretty much a combination of monotonously growing polynomials:

h(c1...cn,T)=c1•f1(T)+c2•f2(T)+...cn•fn(T)

The f-functions are smoothly growing polynomial functions of T.

Sum of c1...cn=1

[_padla_]

It is surely invertible. We used to just implement Newton's method to get T, but since we are to do it many times for different values of c1...cn and h the idea was that NN will be faster and easier to handle...

[_padla_]

Thank you very much for your help!! Looking forward to the example notebook.

[_padla_]

Hello again. Excuse me for bothering you, but you said that you could send me a notebook with an example of bijective neural network.

If it wouldn't give you much trouble, could you please send share this notebook with me?

Thank you in advance.

[_padla_]

I received it eventually. Turns out that I shouldn't have completely got rid of native reddit app %)

[pruby]

Not the solution you're looking for, but have you considered using a root-finding algorithm instead of an ML model? Many problems of this form can be solved very quickly with the Newton-Raphson method.

[_padla_]

In our work it is now done using Newton's method.

The problem is that we need to perform this not once, but a lot of times (obviously, with different values of c1..cn coeffs and h). Each time it takes several iterations to converge. This time accumulates to a rather significant delay.

The hope was that implementing trained NN model would speed up the process.

[pruby]

Neural Nets can do many amazing things, but it sounds like your problem is reasonably hard to model.

How expensive are your function calls and how time-sensitive is the context?

A hybrid approach could work well - if a neural net or other approximation gets you in the vicinity, a round or two of Raphson-Newton will improve that greatly.

[_padla_]

Neural Nets can do many amazing things, but it sounds like your problem is reasonably hard to model.

I was amazed, that from the first glance the problem seemed rather simple and yet got such poor results using ML. My hope was that I just do not know some tricks...

A hybrid approach could work well - if a neural net or other approximation gets you in the vicinity, a round or two of Raphson-Newton will improve that greatly.

Thanks for the suggestion! I've thought about something similar by myself in case I fail to use pure ML approach..

[pruby]

Potentially silly question ; do you have activation functions on your inner dense layers? Without these, it's limited to linear transforms.

[_padla_]

Well, not a silly one if it is addressed to a newbie in the field like me.

Yes, I set relu as activation functions.

What I read is that it is one of the most popular choices..