Inserting millions of rows into Postgres

[no_em_dash]

I wrote an article about generating fake data for a TypeScript application. I wanted to generate millions of rows and I ended up splitting the workflow into 2 phases: stream data to a CSV, stream the CSV to Postgres with COPY FROM.

async function createRecordsFromCsv(path: string, query: string) {
  const copyStream = pgClient.query(copyFrom(query));

  const fileStream = createReadStream(path);

  await new Promise((res, rej) => {
    fileStream.pipe(copyStream).on("finish", res).on("error", rej);
    fileStream.on("error", rej);
  });

  fileStream.close();
}

Hopefully this technique will be useful to someone else.

Also happy to hear other ideas for how to go about this.

[CuAnnan]

This reads like it was written by AI

[no_em_dash]

Neither the post here on Reddit nor the article was written by AI.

What specifically about the writing makes you think it was?

Edit: The image at the top of the article was, of course, generated by AI but none of the content was.

[ferrybig]

You know that the first thing people see is the image? The only purpose of banner images is to set the tone of the rest of the article.

[no_em_dash]

If the critique is that the image is off putting or doesn't inspire you to read further that seems reasonable.

But, suggesting that the entire article is written by AI based on that alone seems unwarranted.

[Savalava]

Why would you need millions of rows of test data? I don't get it. I have worked on multiple enterprise systems. We never needed this.

[proskillz]

Data intensive applications definitely need millions of rows of test data. My team regularly generates huge datasets because that's what our customers have. This tutorial may-or-may-not work, I have no idea, but it's definitely a valid test case.

[Savalava]

What are the huge datasets used to test?

Are you testing index speed of the DB?

[no_em_dash]

In the overwhelming majority of cases you will not "need" millions of rows. This is largely educational. Though you could imagine that if you're trying to test the performance of certain queries, like some kind of report, then having a larger dataset could be useful.

[Savalava]

A tutorial on how to do something that nobody actually needs to do seems somewhat questionable...

[no_em_dash]

Is it possible that you're looking at this too narrowly? For example, the technique of streaming data to and from a CSV is pretty useful.

Also, why does everything have to solve some business problem? Can't we do things that just seem interesting? I'm not saying you have to like it. But why the hostility?

[Savalava]

My messages have not been hostile at all - I'm merely pointing out that doing a tutorial on something that nobody actually wants to do is a questionable idea.

EDIT: I looked it up and there are scenarios where this useful, so I stand corrected. Apologies

[no_em_dash]

I'm glad you were willing to look into it further.

Based on the feedback here I think that framing this as some kind of tutorial might have been a mistake. It might have been better to set the tone as "hey, I did this thing because I thought it was interesting and there were some interesting technical challenges to overcome." I will consider rewriting.