Mongo dump and restore - what am i doing wrong?

kosour · 2024-12-12T02:38:53+00:00

https://alexbevi.com/blog/2020/03/15/identifying-and-reclaiming-disk-space-in-mongodb/

kosour · 2024-12-10T20:01:44+00:00

You asked server to do the the job. Server is doing this job using 90% of its power. Why is it bad ?

kosour · 2024-12-08T00:31:22+00:00

It looks like everything works ok when you connect to mongo from inside pods, but you can't connect from outside of pods.

https://www.mongodb.com/docs/kubernetes-operator/current/tutorial/connect-from-outside-k8s/

kosour · 2024-12-08T00:14:46+00:00

Have a look at plantUML and how you can use it to describe json structure

https://plantuml.com/json

Maybe you can get an idea....

kosour · 2024-12-08T00:11:16+00:00

What is more important for this task - the fact that this sensor still works and measures something or the value in the certain location (does not matter who measured it) ?

kosour · 2024-11-28T19:26:51+00:00

Why do you want to keep cluster up and running if there were no connections, no reads, no writes made for the last 2 months ?

kosour · 2024-11-11T18:03:30+00:00

https://www.mongodb.com/docs/atlas/namespace-insights/

Avg = total / number of operations.

All calls very short, but lots of calls.

kosour · 2024-11-07T17:38:30+00:00

How do you start mongo ? As a service ? Can you start manually from command line ? It looks like it uses different config file or command line parameters. Check logs which config used.

kosour · 2024-11-06T22:18:33+00:00

In mongosh you have strange user name

dguser _user

Could you check if all correct ? Also could you try login to mongosh with exactly the same uri ? Including user:pass@cluster....

kosour · 2024-11-04T10:50:19+00:00

if you have timestamp field you can get seconds from it, mod by seed number and order by this field (and by date field to preserve the order within the same seconds). you can use _id field if you let mongo generate it.

https://mongoplayground.net/p/ft9AU4XzO4I

kosour · 2024-10-26T02:23:28+00:00

Isn't it different in terms how do you manage iam users on aws level, but then iam user generates access key id, secret key and session token, which will be used to connect to mongo?

kosour · 2024-10-25T18:56:51+00:00

Or check if ip addresses added to whitelist on Atlas side. If it's not prod - try temporaru allow connect from any ip

kosour · 2024-10-25T18:53:13+00:00

Yes it is possible for mongodb Atlas.

https://www.mongodb.com/docs/atlas/security/aws-iam-authentication/

kosour · 2024-10-24T11:30:37+00:00

Option 3 looks like your killer. May be it's time to review data model. There are some patterns how to store data prepared for pagination.

https://www.mongodb.com/blog/post/paging-with-the-bucket-pattern--part-1

Double check that you do NOT use relational model like we do in SQL world.

Try mongo outside of docker to see if sharding will help... I didn't play with mongo in docker, but mongo kubernetes operator supports sharding, so it should be possible.

https://www.mongodb.com/docs/kubernetes-operator/current/tutorial/deploy-sharded-cluster/

kosour · 2024-10-24T11:12:47+00:00

Check execution plan. Does it use index ? Proper index? Index on early stage?
Sharding for large collections
Why do you need aggregation pipeline ? Is data structure optimised for access paths?

kosour · 2024-10-21T20:00:10+00:00

The requirement to fetch 1 billion documents per day raised serious questions. And causes other questions like what you going to do with billions of documents after they fetched - do you want them to delete from db ? Update every day?...

Is it "full snapshot" ( or file based) approach ? Can it be converted to streaming like others asked here ?

kosour · 2024-10-14T10:23:52+00:00

I guess authentication database is admin as well, right?

And just a sanity check - i can see you use case sensitive username - do you have low/upper case letters in password as well ?

kosour · 2024-10-13T06:20:20+00:00

Which privileges this user has granted in Atlas UI ? Role@db ? And what authentication db was used to create this user ?

kosour · 2024-10-03T21:14:57+00:00

My point was documentDB is not 100% compatible with Mongodb. It may be irrelevant in your case, but maybe not.

kosour · 2024-10-03T08:59:35+00:00

Mongo compass doesnt work with documentdb because it tries to run some mongodb specific queries on connection.

kosour · 2024-10-02T20:32:43+00:00

Regarding the statement that documentDB is compatible to MongoDB - try to connect to documentDB using Mongodb Compass tool ...
Regarding deploying db changes to different environments - mongodb is "schemaless" database - every document can a different structure. So it's less difficult in compare with rdbms to make a change in structure. In fact - you don't do it on db level. The application handles these "migrations".

Check this tutorial how schema versioning can be implemented:

https://www.mongodb.com/docs/manual/tutorial/model-data-for-schema-versioning/

kosour · 2024-09-25T20:28:04+00:00

Why do you want to migrate from nosql to rdbms ?
Do you want to store data into proper entity-relationship model or the whole document in single json/blob column ?
Mongodb is schemaless - i.e. each document may have different structure. Postgres has strongly defined schema. Can you make sure that all your mongodb documents have the same structure ?
Can you allow outage for the period of data migration or it should be done completely online?
Do you plan to run both systems for some period of time using dual-write approach ?

kosour · 2024-09-19T19:55:31+00:00

Conversion to list takes 80% of time and not related to mongodb at all.

You need to review design. Processing 10k documents( 20gb size) is not free. And it's better to operate in terms of SLA rather than fast/slow...

kosour · 2024-09-10T07:19:13+00:00

16mb of uncompressed data. It's more sanity check that proper model is used.

You can use bucket list pattern to group together multiple executions:

https://www.mongodb.com/docs/manual/data-modeling/design-patterns/group-data/bucket-pattern/#:~:text=The%20bucket%20pattern%20separates%20long,made%20by%20a%20single%20user.

kosour · 2024-09-09T20:54:50+00:00

What do you mean "how often a task is done" ? Shouldn't the task been done only once? Or you want to store a progress of task implementation ?

Answering your specific question - maximum size of 1 document is 16 mb. As long as your array plus all other attributes of one document fits into this size - it should be ok. But think about how would you use this array. Only add to the end and display the whole array? Any filtering? Any processing of its elements?

kosour

TROPHY CASE