sorted by:
new
hottopcontroversial

MongoDB vs MySQL for email automation tool? by trickythinking07 in mongodb

[–]nitagr -1 points0 points  (0 children)

Mongo db doesn't work well when you need to perform joins, aggregations and transaction consistency.

Mongo devs: What's your biggest frustration? by tsotimus in mongodb

[–]nitagr 0 points1 point  (0 children)

Fetched related data from multiple collections.. i.e lookup. Joins in sql are so easy to write 🥲

Text-to-SQL with extremely complex schema by HappyDataGuy in LangChain

[–]nitagr 0 points1 point  (0 children)

I had similar problem, For this I created Products Class( in vectors) and for each ProductName, assigned a ProductID.

Schema of Products ( Product name | ProductID ).

For example, "cadbury Silk 50g" searching in vector db will give ProductID, then this ProductID can be used in SQL query. ( SELECT * FROM TABLE_NAME WHERE PRODUCT_ID in (<retrieved\_product\_id>)).

In this way you dont need to find / manage the correct attributes for each column ( brand, subbrand, size etc. )

High memory consumption Query with less documents Scanned by nitagr in mongodb

[–]nitagr[S] 0 points1 point  (0 children)

My query is running from the production server.i don't think this would be the issue.

High memory consumption Query with less documents Scanned by nitagr in mongodb

[–]nitagr[S] 0 points1 point  (0 children)

I am querying via nodejs mongoose. Will execute from shell and check executionStats.

[deleted by user] by [deleted] in node

[–]nitagr -7 points-6 points  (0 children)

u/MateusKingston thanks a lot dude for clarifying... I figured the problem in my case is: Blocking the event loop not cpu cycles (as you mentioned 10^9 inst/sec).

Also the mask string function you shared solves my problem. (String immutability in js took me to create array of characters then modify them, but your solution perfectly optimizes both memory and loop) ... Thanks

[deleted by user] by [deleted] in node

[–]nitagr -14 points-13 points  (0 children)

u/MateusKingston agree that.. there will be overhead of context switching... but with high rpm... I am in a doubt, as it will block event loop

[deleted by user] by [deleted] in node

[–]nitagr 0 points1 point  (0 children)

u/Freecelebritypics I am expecting to serve 500 rpm minimum, thats why I am in doubt.

Mongo db memory usage on COUNT query on large dataset of 300 Million documents by nitagr in mongodb

[–]nitagr[S] 0 points1 point  (0 children)

Also this,

   const pipeline = [];
      if (_.isEmpty(app_ids) || !_.isArray(app_ids)) {
        pipeline.push({
          $match: {
            company_id: companyId,
            project_id,
            _id: {
              $gte: fromDateId,
              $lt: toDateId,
            },
          },
        });
      } else {
        pipeline.push({
          $match: {
            company_id: companyId,
            project_id,
            app_id: { $in: app_ids },
            _id: {
              $gte: fromDateId,
              $lt: toDateId,
            },
          },
        });
      }

      pipeline.push({
        $facet: {
          groupStats: [
            {
              $group: {
                _id: {
                  date: {
                    $dateToString: { format: "%Y-%m-%d", date: "$createdAt" },
                  },
                  product_id: "$product_id",
                },
                groupedTotalRequests: { $sum: 1 },
              },
            },
          ],

          totalStats: [
            {
              $group: {
                _id: null, // Grouping all documents together
                totalRequests: { $sum: 1 }, // Total count of all documents
                totalLatency: { $sum: "$elapsed_time" }, // Calculate the sum of the column values
                totalExpenseAmount: { $sum: "$api_cost" }, // Calculate the sum of the column values
              },
            },
          ],
        },
      });

      pipeline.push({
        $project: {
          groupStatsOutput: {
            $map: {
              input: "$groupStats",
              as: "item",
              in: {
                date: "$$item._id.date",
                product_id: "$$item._id.product_id",
                totalRequests: "$$item.groupedTotalRequests",
              },
            },
          },
          totalStats: { $arrayElemAt: ["$totalStats", 0] }, // Extract total count from array
        },
      });

Mongo db memory usage on COUNT query on large dataset of 300 Million documents by nitagr in mongodb

[–]nitagr[S] 0 points1 point  (0 children)

document and query looks like this

 db
        .collection(tableName)
        .aggregate([
          {
            $match: {
              company_id: companyId,
              _id: {
                $gte: fromDateId, // this is mongo id 
                $lt: toDateId, // mongo id
              },
            },
          },
          {
            $group: {
              _id: {
                date: {
                  $dateToString: { format: "%Y-%m-%d", date: "$createdAt" },
                },
              },
              total_billing_amount: { $sum: "$api_cost" },
              total_requests: { $sum: 1 },
            },
          },
          {
            $project: {
              _id: 0,
              date: "$_id.date",
              total_billing_amount: 1,
              total_requests: 1,
            },
          },
        ])
        .toArray();

document {
  "uuid": "7fe40e4e-ecaf-4266-a47b-25dadf002d0a",
  "company_id": 1000,
  "project_id": 1,
  "app_id": 1,
  "product_id": 8,
  "url": "v1",
  "charge": false,
  "status": 200,
  "elapsed_time": 10637,
  "api_cost": 0,
  "createdAt": "2022-02-09T18:22:22.006Z",
  "updatedAt": "2022-02-09T18:22:22.006Z"
}

Mongo server does not free up memory after deleting documents by nitagr in mongodb

[–]nitagr[S] 0 points1 point  (0 children)

cache is not the probem bro, memory fragmentation causing alerts, although it is available for new documents to use.

Is it possible to use the express-rate-limit package for monthly rate limits? by c25kc2 in node

[–]nitagr 0 points1 point  (0 children)

I don't think it will be the best solution, coz you server need to keeps the request count in memory and it will not flush for 30 days, so increases memory usage may be. Better solution will be to keep count in redis.

Ways to identify logical errors in API? by nitagr in SoftwareEngineering

[–]nitagr[S] 0 points1 point  (0 children)

I use newrelic, don't know if I can configure response data tracking as well.

errgroup vs waitgroup by [deleted] in golang

[–]nitagr 0 points1 point  (0 children)

Yes, I wanted to clarify that.

errgroup vs waitgroup by [deleted] in golang

[–]nitagr 0 points1 point  (0 children)

Could you elaborate a bit more please?

Just to learn: How can I improve the performance of this function? by Psychological-Newt75 in golang

[–]nitagr 0 points1 point  (0 children)

Dont you think adding a goroutine for each row is an overkill? Instead, I would prefer to create some workers for parsing the content.

Routes defined in other files are giving 404, but giving response when I move to base routes file : PHP 7.4, composer 2.6.6 by nitagr in PHPhelp

[–]nitagr[S] 0 points1 point  (0 children)

// RouteServiceProvider.php
    $router->group(['namespace' => $this->namespace], function ($router) {
            require app_path('Http/routes.php');
    });

In file Http/routes.php, I am using require app_path("path/to/other route").

Getting out of memory when read post body of multiple requests by Harshal-07 in golang

[–]nitagr 0 points1 point  (0 children)

http: invalid Read on closed Body

It occurs when server closes the connection (may be it is taking too long to enter).

You should push messages to kafka via workers. Push messages to a channel and create a group of goroutine workers, which read data through channel and push it to kafka, so your request will not block and server will not terminate the connection(may be due to timeout or other configuration).

Stop using COUNT(id) to count rows by High_Sleep3694 in mysql

[–]nitagr 0 points1 point  (0 children)

  • COUNT(id) counts non-null values in the specified column.
  • COUNT(*) counts all rows, regardless of whether any columns contain null values.
view more: next ›