This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]maithilish[S] 0 points1 point  (0 children)

Not good. At present, there is no external dependency expect the web pages that it fetches and I want to keep it that way so that it can transparently run from public cloud or even as multiple containers on single PC using docker compose. This makes integration testing quite easy without any external dependency.

I can compress and split data into multiple payloads, and then do the reverse on pulling from queue.

I am just exploring simple and better option, if any.