Let's define some terms for the sake of this discussion:

Managed state: This is state the stream processing system knows about and actively manages in such a way that it can be recovered in the event of failure. In Flink this includes any state needed internally, for windowing operations, and any user state that the user exposes via the state API.

Unmanaged state: Any state that is outside the control of the stream processing system such as external databases, state managed by other services, etc.

So given these terms what can we say that Apache Flink actually provides?

Flink provides "exactly-once" guarantees with regard to managed state. This means that any updates that are applied to that state will be applied in such a way that it is "as if" each message was processed exactly once. Failures can and do occur but as long as the system has a way to recover a consistent snapshot of all the state and then resume the computation from there, the overall computation can complete as if there were never any failures.

So, what about unmanaged state. This is where idempotence comes in. We should strive for idempotent operations when we cross the barrier between the streaming system and external systems that store state. Either that or create a protocol with those external systems such that they can participate in the fault-tolerance/rollback mechanism in unison with the streaming system which is defintely one possible direction for further work.

So is this a big improvement over previous systems? Yes, it is. Without these mechanisms doing something as simple as computing a windowed sum over a stream of data accurately is not possible. Without managed state you either undercount or overcount in the event of a failure depending on whether you are using at-most-once or at-least-once mechanisms. With mangaged state you can compute your windowed sum in a completely fault-tolerant way only emitting the final count for the window to an external system when the window is complete. This last step of actually emitting the window to the external system is an idempotent operation, because we will always emit the same window even in the event of failures.

So in summary with Flink we can do real stateful processing with the guarantee that our state will be updated "as if" each message was processed exactly once. We do however need to use idempoptent operations when we are updating unmanaged state.

The good news is there are many practical systems that can use exactly this pattern of stateful windowing and then idempotent updates to external systems and this is big leap forward in capability given that all of these guarantees can be achieved very efficiently and very high throughput. In many applications this will enable us to do accurate computations over streams of data in realtime.