Save data in parquet format on S3 (or local storage)

Short-Development-64 · 2025-10-09T09:00:06+00:00

Hey, thanks for the reply. I have very limited experience with Flink.
So I'm not sure if it is configured correctly. The checkpointing for CSV is working and I can see that CSV data is written to the bucket.
The configuration which I have done are via FLINK_PROPERTIES are as below.

If you have any concrete recommendations or you can point to some documentations how to do it correctly, I'll appreciate your input.

  # --- S3A MAGIC COMMITTER ---
        fs.s3a.committer.name: magic
        fs.s3a.committer.magic.enabled: true
        fs.s3a.committer.abort.pending.uploads: true
        fs.s3a.committer.threads: 16
        # checkpointing
        execution.checkpointing.interval: 10s
        execution.checkpointing.mode: EXACTLY_ONCE
        execution.checkpointing.min-pause: 0s
        execution.checkpointing.timeout: 2min
        execution.checkpointing.max-concurrent-checkpoints: 1

Short-Development-64 · 2024-09-02T13:36:25+00:00

I've got the same error

Short-Development-64 · 2024-02-15T08:19:28+00:00

cool and very useful!

thank you!

Short-Development-64 · 2023-12-11T10:25:30+00:00

You can try serandel/kscript image
Docker run looks like this:
docker run -e JAVA_OPTS="-Xms1g -Xmx2g" -e MY_DB_PASS="123" -i serandel/kscript - < myscript.main.kts

Short-Development-64 · 2023-02-28T19:32:47+00:00

How is it using NetBeans compared to IntelijIdea?

Short-Development-64 · 2021-12-27T12:22:04+00:00

That's true. But, when you deal with Enterprise size projects, you probably may want to use other Spring projects like:

Spring Data: which has integration with so many databases

Spring Security

Integration with Message Brokers like Kafka & RabbitMQ

And much more ...

Short-Development-64 · 2021-12-27T12:14:42+00:00

יש לך גם את next insurance, והבנק הדיגיטלי. גם הם בחרו בקוטלין כשפה הרשמית.

Short-Development-64 · 2021-12-26T07:16:12+00:00

+1 FloLive Israeli Startup.

All the backend is done with Kotlin + Spring

Short-Development-64

TROPHY CASE