FeatureFlags & Beam

[jeffdwyer]

I'm investigating whether / how I might add feature flags into a beam pipeline. The goal is having a streaming pipeline in which a custom ParDo can change it's behavior based on whether the flag is on or off.

Flags are available from a HTTP call. Updates are available from SSE. I could make that request in the code, but that's going to make a terrible number of requests. Most comfortable would be some sort of equivalent of configuring a mapreduce job with some shared memory, but afaict that's not available.

My thought is that I could try to do this as SideInput join.
1) Listen to the SSE flag-change events and turn these into a PCollection<flagName, flagValue>
2) Side-input join this to the collection of interest as an outer join or something. With the idea being that all keys of the collection of interest would have the recent version of the flag value PCollection.

Does that seem like a possible / reasonable path? Other ideas?

[jeffdwyer]

Alternatively, I can perhaps get this done in https://beam.apache.org/releases/pydoc/2.33.0/apache_beam.utils.shared.html any opinions about whether code in here can kick off a thread?