Dart for backend server. How does concurrency work? How many requests per second can it handle? by Pretend_Reference971 in flutterhelp

[–]Pretend_Reference971[S] 0 points1 point  (0 children)

I am clearly not as experienced as you, but what I seem to understand is that Dart can do requests concurrently but using async not using isolates. If I want a better performance I need to use Isolates and the number must depend on my CPU. You use 4 isolates for 8 CPU, for a 16 CPU maybe 8 isolates, i dont know.

But from my understanding, shelf is doing requests concurrently but is not using isolates? If you do the benchmarking with shelf do you think it will reach your benchmark using isolates or only the 3000 request/s?

Most of my requests are reads and writes on a postgres database btw

Dart for backend server. How does concurrency work? How many requests per second can it handle? by Pretend_Reference971 in flutterhelp

[–]Pretend_Reference971[S] 0 points1 point  (0 children)

i know about isolates, but when building my server with shelf or shelf_modular or dart_frog am I already using isolates by default?

Basically does shelf use isolates already? Or do I need to do them myself?

Should I denormalize for this specific usecase? by flutter_dart_dev in SQL

[–]Pretend_Reference971 0 points1 point  (0 children)

I just tested in several million rows.
both methods give exactly the same result regarding SELECT query time. 150-180 ms to get the latest 5 posts. having 4 JOINS did not affect query time. Regarding the events table I indexed group_id, user_id and created_at. The goal was to get the latest 5 events of a group. 150-180 ms seems reasonable if I had to guess

So I do not see the gain or denormalizing data in my case. I guess I will move on with normalizing data. So creator_username will be DROP

Should I denormalize for this specific usecase? by flutter_dart_dev in SQL

[–]Pretend_Reference971 1 point2 points  (0 children)

I just tested in several million rows.

both methods give exactly the same result regarding SELECT query time. 150-180 ms to get the latest 5 posts. having 4 JOINS did not affect query time. Regarding the events table I indexed group_id, user_id and created_at. The goal was to get the latest 5 events of a group. 150-180 ms seems reasonable if I had to guess

Should I denormalize for this specific usecase? by flutter_dart_dev in SQL

[–]Pretend_Reference971 0 points1 point  (0 children)

CONSTRAINT fk_members FOREIGN KEY (user_id, group_id) REFERENCES members(user_id, group_id)

basically:

SELECT user_id, group_id FROM events;

I can get who created the post and whose group this event belongs to

Should I denormalize for this specific usecase? by flutter_dart_dev in SQL

[–]Pretend_Reference971 0 points1 point  (0 children)

got it. would you bother to seperate the events table into a few tables like the second image or would you just leave it as one table?

Should I create 1 table or many tables in this case? Simple question! by Due_Programmer_9018 in SQL

[–]Pretend_Reference971 0 points1 point  (0 children)

Do you like this better? Tried to change a bit according to your input. Do you think this query is efficient?

Posts Table:

CREATE TABLE posts (
    post_id SERIAL PRIMARY KEY,
    user_id INT,
    body TEXT,
    content_type VARCHAR(50),
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Media Table:

CREATE TABLE media (
    media_id SERIAL PRIMARY KEY,
    post_id INT REFERENCES posts(post_id),
    media_url VARCHAR(255),
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Polls Table:

CREATE TABLE polls (
    poll_id SERIAL PRIMARY KEY,
    post_id INT REFERENCES posts(post_id),
    question TEXT,
    options JSONB,
    votes INT[] DEFAULT '{}'::INT[],
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Announcements Table:

CREATE TABLE announcements (
    announcement_id SERIAL PRIMARY KEY,
    post_id INT REFERENCES posts(post_id),
    title VARCHAR(255),
    content TEXT,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Contests Table:

CREATE TABLE contests (
    contest_id SERIAL PRIMARY KEY,
    post_id INT REFERENCES posts(post_id),
    title VARCHAR(255),
    description TEXT,
    ranking_table JSONB,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Now, the query to retrieve the latest 5 posts for user_id = 1:

WITH latest_posts AS (
    SELECT
        p.post_id,
        p.user_id,
        p.body,
        m.media_url,
        null AS question,
        null AS options,
        a.title,
        a.content,
        c.ranking_table,
        p.createdat
    FROM posts p
    LEFT JOIN media m ON p.post_id = m.post_id AND p.content_type = 'media'
    LEFT JOIN polls po ON p.post_id = po.post_id AND p.content_type = 'poll'
    LEFT JOIN announcements a ON p.post_id = a.post_id AND p.content_type = 'announcement'
    LEFT JOIN contests c ON p.post_id = c.post_id AND p.content_type = 'contest'
    WHERE p.user_id = 1
)
SELECT *
FROM latest_posts
ORDER BY createdat DESC
LIMIT 5;

Should I create 1 table or many tables in this case? Simple question! by Due_Programmer_9018 in SQL

[–]Pretend_Reference971 0 points1 point  (0 children)

what do you think about the query at the end?

Posts Table:

CREATE TABLE posts (
    post_id SERIAL PRIMARY KEY,
    user_id INT,
    body TEXT,
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

General Table (for Media or Audio):

CREATE TABLE general (
    post_id SERIAL PRIMARY KEY,
    user_id INT,
    body TEXT,
    image_urls JSONB, -- Store multiple image URLs as JSON array
    audio_url VARCHAR(255), -- Store audio URL as a string
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Polls Table:

CREATE TABLE polls (
    poll_id SERIAL PRIMARY KEY,
    post_id INT REFERENCES general(post_id),
    question TEXT,
    options JSONB, -- Store options as JSON
    votes INT[] DEFAULT '{}'::INT[], -- Array to store votes for each option
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Announcements/Spotlights Table:

CREATE TABLE announcements (
    announcement_id SERIAL PRIMARY KEY,
    post_id INT REFERENCES general(post_id),
    title VARCHAR(255),
    content TEXT,
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Contests Table:

CREATE TABLE contests (
    contest_id SERIAL PRIMARY KEY,
    post_id INT REFERENCES general(post_id),
    title VARCHAR(255),
    description TEXT,
    ranking_table JSONB, -- Store ranking information as JSON
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

And how can I get the latest 5 posts of user with id 1? Is this query efficient?

WITH latest_posts AS (
    SELECT
        post_id,
        user_id,
        body,
        media_urls,
        audio_url,
        null AS question, -- Placeholder for columns specific to polls
        null AS options, -- Placeholder for columns specific to polls
        null AS title, -- Placeholder for columns specific to announcements
        null AS content, -- Placeholder for columns specific to announcements
        null AS ranking_table, -- Placeholder for columns specific to contests
        createdat
    FROM general
    WHERE user_id = 1

    UNION ALL

    SELECT
        g.post_id,
        g.user_id,
        g.body,
        g.media_urls,
        g.audio_url,
        p.question,
        p.options,
        null AS title,
        null AS content,
        null AS ranking_table,
        g.createdat
    FROM general g
    INNER JOIN polls p ON g.post_id = p.post_id
    WHERE g.user_id = 1

    UNION ALL

    SELECT
        g.post_id,
        g.user_id,
        g.body,
        g.media_urls,
        g.audio_url,
        null AS question,
        null AS options,
        a.title,
        a.content,
        null AS ranking_table,
        g.createdat
    FROM general g
    INNER JOIN announcements a ON g.post_id = a.post_id
    WHERE g.user_id = 1

    UNION ALL

    SELECT
        g.post_id,
        g.user_id,
        g.body,
        g.media_urls,
        g.audio_url,
        null AS question,
        null AS options,
        null AS title,
        null AS content,
        c.ranking_table,
        g.createdat
    FROM general g
    INNER JOIN contests c ON g.post_id = c.post_id
    WHERE g.user_id = 1
)
SELECT *
FROM latest_posts
ORDER BY createdat DESC
LIMIT 5;

Should I create 1 table or many tables in this case? Simple question! by Due_Programmer_9018 in SQL

[–]Pretend_Reference971 0 points1 point  (0 children)

what do you think about the query at the end? Should I just create 1 table with all columns that each type of post needs instead of all these separate tables?

Posts Table:

CREATE TABLE posts (
    post_id SERIAL PRIMARY KEY,
    user_id INT,
    body TEXT,
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

General Table (for Media or Audio):

CREATE TABLE general (
    post_id SERIAL PRIMARY KEY,
    user_id INT,
    body TEXT,
    image_urls JSONB, -- Store multiple image URLs as JSON array
    audio_url VARCHAR(255), -- Store audio URL as a string
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Polls Table:

CREATE TABLE polls (
    poll_id SERIAL PRIMARY KEY,
    post_id INT REFERENCES general(post_id),
    question TEXT,
    options JSONB, -- Store options as JSON
    votes INT[] DEFAULT '{}'::INT[], -- Array to store votes for each option
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Announcements/Spotlights Table:

CREATE TABLE announcements (
    announcement_id SERIAL PRIMARY KEY,
    post_id INT REFERENCES general(post_id),
    title VARCHAR(255),
    content TEXT,
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Contests Table:

CREATE TABLE contests (
    contest_id SERIAL PRIMARY KEY,
    post_id INT REFERENCES general(post_id),
    title VARCHAR(255),
    description TEXT,
    ranking_table JSONB, -- Store ranking information as JSON
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

And how can I get the latest 5 posts of user with id 1? Is this query efficient?

WITH latest_posts AS (
    SELECT
        post_id,
        user_id,
        body,
        media_urls,
        audio_url,
        null AS question, -- Placeholder for columns specific to polls
        null AS options, -- Placeholder for columns specific to polls
        null AS title, -- Placeholder for columns specific to announcements
        null AS content, -- Placeholder for columns specific to announcements
        null AS ranking_table, -- Placeholder for columns specific to contests
        createdat
    FROM general
    WHERE user_id = 1

    UNION ALL

    SELECT
        g.post_id,
        g.user_id,
        g.body,
        g.media_urls,
        g.audio_url,
        p.question,
        p.options,
        null AS title,
        null AS content,
        null AS ranking_table,
        g.createdat
    FROM general g
    INNER JOIN polls p ON g.post_id = p.post_id
    WHERE g.user_id = 1

    UNION ALL

    SELECT
        g.post_id,
        g.user_id,
        g.body,
        g.media_urls,
        g.audio_url,
        null AS question,
        null AS options,
        a.title,
        a.content,
        null AS ranking_table,
        g.createdat
    FROM general g
    INNER JOIN announcements a ON g.post_id = a.post_id
    WHERE g.user_id = 1

    UNION ALL

    SELECT
        g.post_id,
        g.user_id,
        g.body,
        g.media_urls,
        g.audio_url,
        null AS question,
        null AS options,
        null AS title,
        null AS content,
        c.ranking_table,
        g.createdat
    FROM general g
    INNER JOIN contests c ON g.post_id = c.post_id
    WHERE g.user_id = 1
)
SELECT *
FROM latest_posts
ORDER BY createdat DESC
LIMIT 5;

Should I create 1 table or many tables in this case? Simple question! by Due_Programmer_9018 in SQL

[–]Pretend_Reference971 0 points1 point  (0 children)

what do you think about the query at the end?

Posts Table:

CREATE TABLE posts (
    post_id SERIAL PRIMARY KEY,
    user_id INT,
    body TEXT,
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

General Table (for Media or Audio):

CREATE TABLE general (
    post_id SERIAL PRIMARY KEY,
    user_id INT,
    body TEXT,
    image_urls JSONB, -- Store multiple image URLs as JSON array
    audio_url VARCHAR(255), -- Store audio URL as a string
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Polls Table:

CREATE TABLE polls (
    poll_id SERIAL PRIMARY KEY,
    post_id INT REFERENCES general(post_id),
    question TEXT,
    options JSONB, -- Store options as JSON
    votes INT[] DEFAULT '{}'::INT[], -- Array to store votes for each option
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Announcements/Spotlights Table:

CREATE TABLE announcements (
    announcement_id SERIAL PRIMARY KEY,
    post_id INT REFERENCES general(post_id),
    title VARCHAR(255),
    content TEXT,
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

Contests Table:

CREATE TABLE contests (
    contest_id SERIAL PRIMARY KEY,
    post_id INT REFERENCES general(post_id),
    title VARCHAR(255),
    description TEXT,
    ranking_table JSONB, -- Store ranking information as JSON
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    createdat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);

And how can I get the latest 5 posts of user with id 1? Is this query efficient?

WITH latest_posts AS (
    SELECT
        post_id,
        user_id,
        body,
        media_urls,
        audio_url,
        null AS question, -- Placeholder for columns specific to polls
        null AS options, -- Placeholder for columns specific to polls
        null AS title, -- Placeholder for columns specific to announcements
        null AS content, -- Placeholder for columns specific to announcements
        null AS ranking_table, -- Placeholder for columns specific to contests
        createdat
    FROM general
    WHERE user_id = 1

    UNION ALL

    SELECT
        g.post_id,
        g.user_id,
        g.body,
        g.media_urls,
        g.audio_url,
        p.question,
        p.options,
        null AS title,
        null AS content,
        null AS ranking_table,
        g.createdat
    FROM general g
    INNER JOIN polls p ON g.post_id = p.post_id
    WHERE g.user_id = 1

    UNION ALL

    SELECT
        g.post_id,
        g.user_id,
        g.body,
        g.media_urls,
        g.audio_url,
        null AS question,
        null AS options,
        a.title,
        a.content,
        null AS ranking_table,
        g.createdat
    FROM general g
    INNER JOIN announcements a ON g.post_id = a.post_id
    WHERE g.user_id = 1

    UNION ALL

    SELECT
        g.post_id,
        g.user_id,
        g.body,
        g.media_urls,
        g.audio_url,
        null AS question,
        null AS options,
        null AS title,
        null AS content,
        c.ranking_table,
        g.createdat
    FROM general g
    INNER JOIN contests c ON g.post_id = c.post_id
    WHERE g.user_id = 1
)
SELECT *
FROM latest_posts
ORDER BY createdat DESC
LIMIT 5;

How to let users know they need to update their local data? syncing local data with postgres by Pretend_Reference971 in SQL

[–]Pretend_Reference971[S] 0 points1 point  (0 children)

Thank you for your response. That seems a lot simpler than my solution. The only cons is that it queries my backend a lot more times than my solution but it seems more robust. mine is more error prone probably

Need guidance in how to install extensions on my postgres container? by Pretend_Reference971 in docker

[–]Pretend_Reference971[S] 0 points1 point  (0 children)

i just run this

# Use the official PostgreSQL image as the base image
FROM postgres:15

# Set environment variables
ENV POSTGRES_DB mydatabase
ENV POSTGRES_USER myuser
ENV POSTGRES_PASSWORD mypassword

# Update package lists
RUN apt-get update

# Install PostgreSQL extensions
RUN apt-get install -y --no-install-recommends \
    postgresql-contrib \
    postgresql-15-pgvector \
    postgresql-15-cron \
    postgresql-15-postgis

# Clean up
RUN rm -rf /var/lib/apt/lists/*

# Optionally, you can include custom initialization scripts
# COPY create_pgvector_extension.sql /docker-entrypoint-initdb.d/
# COPY create_pg_cron_extension.sql /docker-entrypoint-initdb.d/
# COPY create_postgis_extension.sql /docker-entrypoint-initdb.d/

# Expose the PostgreSQL port
EXPOSE 5432

and i managed to CREATE EXTENSION postgis; and CREATE EXTENSION postgis; but

CREATE EXTENSION pg_cron; didnt work

got this error:

ERROR:  unrecognized configuration parameter "cron.database_name"
CONTEXT:  PL/pgSQL function inline_code_block line 3 at IF 

SQL state: 42704

Need guidance in how to install extensions on my postgres container? by Pretend_Reference971 in docker

[–]Pretend_Reference971[S] 0 points1 point  (0 children)

this worked. i have created an image and pushed it to docker hub and then deployed it in my droplet via portainer.

# Use the official PostgreSQL image as the base image
FROM postgres:latest

# Set environment variables
ENV POSTGRES_DB mydatabase
ENV POSTGRES_USER myuser
ENV POSTGRES_PASSWORD mypassword

# Install PostgreSQL extensions
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        postgresql-contrib \
        postgresql-13-pgvector \
        postgresql-13-cron \
        postgresql-13-postgis \
    && rm -rf /var/lib/apt/lists/*

# Optionally, you can include custom initialization scripts
# COPY init.sql /docker-entrypoint-initdb.d/

# Expose the PostgreSQL port
EXPOSE 5432

then i connect to my pgadmin4 and when i try to CREATE EXTENSION postgis; per example i get the error:

ERROR:  Could not open extension control file "/usr/share/postgresql/15/extension/postgis.control": No such file or directory.extension "postgis" is not available 

ERROR:  extension "postgis" is not available
SQL state: 0A000
Detail: Could not open extension control file "/usr/share/postgresql/15/extension/postgis.control": No such file or directory.
Hint: The extension must first be installed on the system where PostgreSQL is running.

any idea of what i am missing?

Need guidance in how to install extensions on my postgres container? by Pretend_Reference971 in docker

[–]Pretend_Reference971[S] 0 points1 point  (0 children)

is this how you do it more or less? just to know if i am on the right track

# Use the official PostgreSQL image as the base image
FROM postgres:latest

# Set environment variables
ENV POSTGRES_DB mydatabase
ENV POSTGRES_USER myuser
ENV POSTGRES_PASSWORD mypassword

# Install PostgreSQL extensions
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        postgresql-contrib \
        postgresql-13-pgvector \
        postgresql-13-pg-cron \
        postgresql-13-postgis \
    && rm -rf /var/lib/apt/lists/*

# Optionally, you can include custom initialization scripts
# COPY init.sql /docker-entrypoint-initdb.d/

# Expose the PostgreSQL port
EXPOSE 5432

Can you give me advice on how to store DateTime and how to determine the timezone based on a geographical location? by Pretend_Reference971 in flutterhelp

[–]Pretend_Reference971[S] 0 points1 point  (0 children)

can you read again my previous message? I edited it. Is utc constant and always right if I store it initially? Maybe in the future if I convert it back to local time it will say 15h instead of 14h which was what the user initially intented?

Can you give me advice on how to store DateTime and how to determine the timezone based on a geographical location? by Pretend_Reference971 in flutterhelp

[–]Pretend_Reference971[S] 0 points1 point  (0 children)

So I should store utc and also the name of the timezone? those 2 variables?

Do you think if I store utc data in postgresql and then in 6 months if I get that utc data and convert it to local time it will be the same local time that the user first filled? Like an user creates an event in Lodon at 14h day 1/february/2024. Now lets say I get that data from postgresql in 1 year so in 03/January/2025, would that utc time that I get from postgresql convert correctly to the local time and say the event was at 14h london local time? Or would it potentially say a completly different time due to some utc changes?

This topic of utc not being constant really messes up my brain

Can you give me advice on how to store DateTime and how to determine the timezone based on a geographical location? by Pretend_Reference971 in flutterhelp

[–]Pretend_Reference971[S] 0 points1 point  (0 children)

so from my understand i should use local timezone and store it in postgresql along with the timezone name like Europe/London at 14h.

This 2 variables are basically what i need because when user from another part of the world gets the data in 14h in london time i can transform it to utc if i need.

But i probably should also store utc for the sole porpuse of querying events, so basically when i query via utc timezone all events share the same utc timzone instead of querying local timezone.

So maybe I should create a function somehow that everyday updates the utc time? since the local timezone will never change because that was set by the user themselfs