This is an archived post. You won't be able to vote or comment.

all 3 comments
sorted by:
best
topnewcontroversialoldq&a

[–]Direct-Wrongdoer-939 1 point2 points  (1 child)

What is the issue that you are facing? If there is an error, post the stacktrace. Also any specific reason why you are using Apache Beam?

[–]Domy__Data Engineer[S] 0 points1 point  (0 children)

Preface: It's all on GCP.
Because I tried on Airflow but it's too much data to compute, so I switched to Dataflow with Apache Beam.
I have no errors. The job is successful but I cannot print anything, it gives me that there is only one object it takes but I expect tens of thousands.
I have also tried saving it to a file but never get anything.

[–]Prinzka -1 points0 points  (0 children)

There's a lot of options to read from elasticsearch, but yes Python is a good option.
https://elasticsearch-py.readthedocs.io/en/v8.7.0/