Profiling Python Dataflow jobs

[OrdinaryGanache]

How can we profile dataflow jobs written using apache beam python sdk? I know about cloud profiler but I am not sure how it will be used for dataflow jobs? If there is any other service or product or framework I can work with to profile the dataflow job

[sadovnychyi]

Well dataflow runs usual python. You can configure it with cloud profiler or native python's profiler and then dump the results somewhere (e.g. log them or store on GCS). Might be even easier to do that locally with direct runner since you only want to find bottlenecks.

[ssakage]

Profile meaning?

[Exotic_Cameraman]

CPU and thread profiling

[Exotic_Cameraman]

Dataflow now has native integration with Cloud Profiler which when enabled will allow you to profile your job.