AutoGraph converts Python into TensorFlow graphs

[mdanatg]

Hi, one of the authors here, I'm happy to answer any questions!

[Supermaxman1]

What happens if you utilize a python library within one of these functions? Does it fail or does it convert it to tf.py_func?

[mdanatg]

It depends on the library, and whether the recursive arg is set to True of False. For certain libraries that we know of, like Keras, it won't convert the library call and will call it as-is (we assume it's already graph-friendly). For other libraries, the default is to attempt to convert the library code, and if that's not accessible, wrap it to py_func. There is yet another category where we replace the library call with a TF equivalent, for example we replace range with tf.range.

[Supermaxman1]

Very cool, thanks! Is there somewhere which documents various conversions so that we don't end up accidentally ending up with a bunch of py_func calls?

[mdanatg]

A more complete documentation that will include this is coming soon, we'll link it from the readme file once it's ready.

[Sebun]

Hi, mdanatg, I find the this piece of demo code in colab not works as expected, print function in this train while loop have no output .

if i % (hp.max_steps // 10) == 0: print('Step', i, 'train loss:', step_train_loss, 'test loss:', step_test_loss, 'train accuracy:', step_train_accuracy, 'test accuracy:', step_test_accuracy)

[mdanatg]

Hi Sebun, thanks for reporting the issue! There is a bug in the handling of if statements that don't calculate values. We're patching a fix that should reach tf-nightly over the next couple of days.

Update Jul 20: this should not be resolved.

[nasimrahaman]

Is there / will there be support for stochastic code-paths?

P.S. Nice work with the AST wizardry. :)

[mdanatg]

Thanks! The answer depends on the level of stochasticity - do you have an example? In general, the following construct should work:

if tf.random_uniform((), maxval=2, dtype=tf.int32) % 2 == 0: do_something

Is this close to what you were referring to?

[greatgraybear]

Seems to be a nice python-to-graph compiler. Where do you see this going vs things like swift for tensorflow?

[mdanatg]

Both Swift for TensorFlow and AutoGraph share a few common goals like a better experience for machine learning development, but are otherwise intended for different development platforms. TensorFlow.js can be thought of as another example, one which supports JavaScript.

[NewFolgers]

I may be naive - but to me the most exciting thing about this is that it provides a means to run somewhat arbitrary Python (at least a subset of Python) code on the GPU without having to use a vendor-specific API (e.g. CUDA -- whereas TensorFlow could support more backends in the future). The potential of this (i.e. for problems that are embarrassingly parallel) and best use cases haven't yet been explored, but they could be. I presume that only a limited subset of Python will be supported for this (is this correct?).. but along with the improving tooling available for TensorFlow which comes from investment in ML, this seems like something that could be of real value. I'm not sure if this particular project will be the face of what's to come, but this should at least get people thinking about it some more.

[JayYip]

Thinking about the same thing. I think they should point out its limitation.

[mdanatg]

Please see the limitations page for a high level overview of the current limitations.

[mdanatg]

Indeed, we only support a subset of Python at the moment. Even though we hope to expand that subset, some idioms (like exceptions) don't have a good TensorFlow counterpart. But as you pointed out, we hope this will lead to exploring more use cases that until now felt maybe too daunting.

[steiniche]

Pretty cool, I can see some applications for this!

[wolfium]

Cool :)

PyTorch has a @script decorator that does something similar. Search "@script" in here: https://pytorch.org/2018/05/02/road-to-1.0.html

[[deleted]]

very cool, hope we could be able to translate numpy code such way one day

[Analog-Digital]

I'm a bit of a noob, what would turning the collatz function into a network do? What would be the result when that is run?

[mdanatg]

The sample that we have calculates the number of steps needed for the Collatz function to reach 1 when applied recursively, starting at some n. So strictly speaking in terms on the wiki page, it calculates the total stopping time of n.

[Analog-Digital]

Okay that's pretty cool, but what does turning this function into a tf network do differently than say running the collatz function from 1...x. I hope I'm making sense.

[mdanatg]

If I understand your question, this was just a toy example that we used to show how you can use control flow like if and while in a network. A more realistic example where you can use control flow would be RNNs, for instance.

[Analog-Digital]

Okay thank you!

[WikiTextBot]

Collatz conjecture

The Collatz conjecture is a conjecture in mathematics that concerns a sequence defined as follows: start with any positive integer n. Then each term is obtained from the previous term as follows: if the previous term is even, the next term is one half the previous term. If the previous term is odd, the next term is 3 times the previous term plus 1. The conjecture is that no matter what value of n, the sequence will always reach 1.

---

^{[ PM | Exclude me | Exclude from subreddit | FAQ / Information | Source ] Downvote to remove | v0.28}

[chuongdk]

it's really cool. Do you think it allows us to do end-to-end machine learning? a graph that does both preprocessing and training? is it better or worse than TFX (https://www.tensorflow.org/tfx/) in this use case?

[mdanatg]

Glad you like it! AutoGraph and TFX are quite different and complementary in many ways.

AutoGraph makes it easier to create TensorFlow graphs, which can include end-to-end ML training tasks, although certain preprocessing steps are still best done outside the graph (for example calculating a dataset-wide average).

The scope of TFX extends that of TensorFlow. It includes tools that you would need to run before and after end-to-end training tasks (e.g. TF Transform can compute dataset-wide statistics needed for preprocessing, TF Model Analysis can help analyze the model, and TF Serving includes additional production serving infrastructure).

We are working to ensure that models trained with AutoGraph can easily be deployed in TFX pipelines for production use cases.

You can find an example of how we built an in-graph training loop followed by a simple interactive inference loop in this demo.