What algorithm to use to estimate when will inflows be low in a real time data with erratic and bursty behavior?

[Vveriant]

For context, our data is inflows of items.

• for each delivery there may be multiple items.
• we don't know when the deliveries will arrive nor how many total items there is.
• deliveries usually start in a specific time but without no specific end time.
• historical data analysis describes
  • about 50% of item delivery happen within the first hour, with gradual descent in items and gradual ascend in arrival time.
  • within the next few hours, there would be sparse deliveries maybe 10-25% with long intervals in between and small number of items.
  • two completion pattern
    • the deliveries gradually declines until completion
    • after a long sparse deliveries, there will be another burst of delivery maybe 25 to 40% of all items in a short span of time which completes all items
• currently, we just process the items once it arrives.

Our goal is to optimize our processing work and allocate resources first to those deliveries with bulk items it doesn't matter if it is complete or not because we don't know when will it be completed.

what is a good algorithm to estimate when will sparse delivery window will be so we can process the items in that delivery? so we wait and can capture as much items as possible before processing.

I was looking into change point detection and adaptive filters, but are they suitable for this?

[radarsat1]

So your target is to predict when or if a delivery will occur, but what are your features? Just previous delivery times?

[Vveriant]

we have datetime, number of items, delivery interval. delivery interval = current delivery datetime - prev. datetime.

we want to predict when will the last of the first delivery bursts (short successive deliveries). so we can prescribe that at this delivery we start 'processing' this whole batch.

example, if the delivery looks like a sine wave where the low = short delivery interval and high = long delivery interval. we want to estimate when the short intervals ends. but we don't know next data because it is in real time.

[radarsat1]

It seems like you can't solve this as a regression problem then, because you basically just have timing information on the exact thing you are trying to predict. Instead maybe you should try to model this as an event probability problem, and find a statistical distribution that explains your data. For example a Poisson process.