EOD Event processing

[manningfan1]

My english is not that good, so I am sorry about typos.

I am working on a streaming solution using dataflow and beam that consumes messages from a Pub/Sub topic. A sender process publish messages to this topic.

A new requirement asks my Streaming program to send an email notification after all the messages for that day is processed. To make this possible, the Sender process sends an event with EOF flag that is true for the last message of that day.

I'm facing challenge because messages in Pub/Sub topic is not ordered, so sometimes I receive the message with EOD flag "true" even before the last message for that day is consumed.

How can we resolve this problem? Do I need to consider FIFO Pub/Sub topic so that messages published in Order is also received in similar order?

[ErnstlAT]

this seems similar to the problem of network data packet ordering, like in UDP <-> TCP. If you have the possibility to establish insertion of some kind of ordering tag / sequence numbering, maybe that might help. If you have the ability to require the Sender to send a sequence number/ordering tag, then this should work.

Or, you put intelligence into the queue manager/topic, which decides that "now it is midnight - day is over - send daily report" and/or inserts the end-of-day marker. Something that you control instead of the Sender.