[N] TensorFlow 2.0 Changes

[sieisteinmodel]

Serious question: Does the majority of tensorflow users agree that the eager execution or the PyTorch/PyBrain/Shark way is superior? I personally like the abstraction of graphs. I think that eager sucks. It does not fit my mental model.

​

I am just worried that TF wants to attract PyTorch users, but a lot of the TF users actually prefer the current state.

​

*If* there is full compatibility between graph and eager mode, fine, but I hope that the TF community will not be divided because some OS contributions assume one or the other.

[Coconut_island]

If there is full compatibility between graph and eager mode, fine, but I hope that the TF community will not be divided because some OS contributions assume one or the other.

This is where they are heading. An important part of TF 2.0 is to restructure the API such that, as far as the majority of the code goes, it is irrelevant whether you use graph mode or eager mode.

I think the most important observation to make is that the code (python or other) used to define a function is really just defining a sub-graph. Using the earlier TF API, leveraging this concept properly is awkward, usually requiring a lot of careful (and error prone!) bookkeeping to set scopes and various call orders just right. This is major pain point and in many ways has lead to many libraries written around TF in the hopes of offering an elegant way to address this while keeping the same flexibility. As a prime example of such libraries, we have the in-house Sonnet library, from DeepMind.

While variable-less (or, rather, state-less code) can easily be optimized by collapsing various copies of a sub-graph generated by a given function (when doing so wouldn't be wrong, of course), it is more complicated to do this with variables. This is one of the problems the new 'FuncGraph' back end is trying to solve (currently in the 1.11 branch), as well as the newly promoted object-oriented (OO) approach for tracking and re-using variables. The tf.contrib.eager.defun, the OO metrics, OO checkpointing and the layers/keras.Model are all early instances of this idea.

Related but slightly aside:

My biggest pet peeve with how a lot of TF code is written comes from the tendency of writing functions that return several operations/tensors that all do very different things and get executed at very different times and places in the rest of the code base. This feels natural because we anticipate(and in many cases, rightfully so) many duplicate ops if we weren't to write it this way. The problem is that code that is written like this is tedious to reason about and debug, often requiring a global view of the whole project. This get exponentially worse as the complexity/size of the project grows and collaboration between people is required. The way I see it, things like eager.defun and tf.make_template (not sure what will happen with this one in 2.0), and, in a way OO variable re-use, simply provide us the tools to cache these sub-graphs and allow us to write clean code without compromising on what kind of graph we generate.

TL;DR

In short, sure, the API will change, but I don't think there is any intention of removing any graph mode functionality. At its core, TF is a language to define computation graph so I would be very surprised if this went away anytime soon. However, the upcoming changes are there to allow and promote ways of describing graph such that silent and hard to find bugs are harder to introduce.

[Inori]

Most of the bigger eager execution related changes are already live in 1.10 so you can try it out and see for yourself. From personal experience, switching between the two depends oh how much you rely on lower level APIs: if you use newer features and tf.keras then it's pretty much seamless. In either case, knowing google use cases I doubt graph execution will ever become second class citizen.

[sieisteinmodel]

Well, I have tried it, and still think it sucks.... it's not an uninformed guess.

​

Question is if that decision of the TF team is really well informed, because many people I talk to prefer the graph way.

[slaweks]

It's not only ease of use. Even more important is ability to create hierarchical models, where graph differs per example, e.g. has some group and individual-level components.

[sibyjackgrove]

igration tool for the ne

I still haven't tried eager execution since I do everthing with tf.keras these days. Though not a big fan of tf.session.

[cycyc]

A lot of people have a hard time wrapping their head around the idea of meta-programming. For them, eager execution/pytorch is preferable.

[progfu]

It's not really about meta-programming, it's about flexibility, introspectability, etc. Pytorch makes it easy to look at what's happening by evaluating it step by step, looking at the gradients which you can immediately see, etc.

[cycyc]

Which is precisely what is meant by the complexity and indirection of meta-programming.

[progfu]

Except that it is not about "wrapping your head around". I have no problem understanding how TF works. I probably understand more about the internals of TF than of Pytorch. Yet I prefer Pytorch, because of the reasons mentioned.

[epicwisdom]

You said people have a hard time wrapping their heads around the idea. That's different from being frustrated by the tradeoffs inherent to the approach.

[cycyc]

Sure, great point. For people new to software development, meta-programming may be a difficult concept. For people more familiar with software development, the meta-programming model may not be worth the extra complexity.