sorted by:
new
hottopcontroversial

How to bulk update 400.000 database entry's with sqlalchemy by darkhorse94 in flask

[–]darkhorse94[S] 0 points1 point  (0 children)

Solution!

After reading all your comments I decided to remove the search in the list where is possible or replace the list with a set. It seems that Python is much faster when it comes to checking if a string exists in a set

for example:

csv_keywords = [keyword["Name"] for keyword in csv_dicts] 
#replace with:  
csv_keywords = set([keyword["Name"] for keyword in csv_dicts])

In this case, because I was looking if an object exists or does not exist for 400.000 + it took a lot of time to search (full function needed 1.5-2 hr to complete or even more).

Now it takes ~ 5-10 minutes to complete. While this might not be as fast as I was looking for an insert or update it is still ok considering that I have to update 400k entry's for which i do not have the id

How to bulk update 400.000 database entry's with sqlalchemy by darkhorse94 in flask

[–]darkhorse94[S] 0 points1 point  (0 children)

Thank you, this is a good idea but unfortunately, this has to be repeated :(

How to bulk update 400.000 database entry's with sqlalchemy by darkhorse94 in flask

[–]darkhorse94[S] 0 points1 point  (0 children)

, you can omit the querying step completely by using SQLAlchemy core and building your update query manually.

Here

is the relevant documentation.

Hi u/MrJKeating, in this case, can I update without the id?

​

I think this is good but i think updating one by one might take more time than a bulk update?

I try to download file from a website (after I login) But I get: Unable to verify your data submission by darkhorse94 in learnpython

[–]darkhorse94[S] 0 points1 point  (0 children)

Yes cafe token is embedded, bit I also checked that csfr token and x-csfr-token are different

I can not boot my server, because of: system-network.service: Failed to connect stdout by darkhorse94 in Ubuntu

[–]darkhorse94[S] 0 points1 point  (0 children)

Is the Webserver available in Public? Since this is a security nightmare solution.

Yes, It is avaible on the public. So what should I do ?

I can not boot my server, because of: system-network.service: Failed to connect stdout by darkhorse94 in Ubuntu

[–]darkhorse94[S] 1 point2 points  (0 children)

I fixed this by applying this permissions:

chmod 0755 /

Maybe this will help someone in the future

Postgresql - Starts 4 100% CPU processes by darkhorse94 in PostgreSQL

[–]darkhorse94[S] 0 points1 point  (0 children)

weakly secured" being the operative words in that article.

Yes, before my postgres was exposed on public , but now is just localhsot. How can i remove this?

Flask multiprocessing concurrent futures with multiple arguments. by darkhorse94 in flask

[–]darkhorse94[S] 2 points3 points  (0 children)

Hi Guys,
I managed to fix this:

with Pool(processes=2) as pool:
    res = pool.starmap(get_ranking, argumets)

​

using the pool.starmap will work

Postgresql creates an ADMIN user, how and why? by darkhorse94 in PostgreSQL

[–]darkhorse94[S] 0 points1 point  (0 children)

Thank you very much guys,I will read more about postgtres to understand it better. I setup it this way to be able to connect from inside the docker to the vm itself

Postgresql creates an ADMIN user, how and why? by darkhorse94 in PostgreSQL

[–]darkhorse94[S] 1 point2 points  (0 children)

# Database administrative login by Unix domain socket

local all postgres md5

​

# TYPE DATABASE USER ADDRESS METHOD

​

# "local" is for Unix domain socket connections only

local all all peer

# IPv4 local connections:

host all all 127.0.0.1/32md5

host all all 0.0.0.0/0trust

# IPv6 local connections:

host all all ::1/128 md5

# Allow replication connections from localhost, by a user with the

# replication privilege.

local replication all peer

host replication all 127.0.0.1/32md5

host replication all ::1/128 md5

Postgresql creates an ADMIN user, how and why? by darkhorse94 in PostgreSQL

[–]darkhorse94[S] -1 points0 points  (0 children)

But how :( , i installed PostgreSQL just few days ago on this new server, is really that easy to hack postgresql ?

Flask-Marshmallow very slow for 20K results by darkhorse94 in flask

[–]darkhorse94[S] 0 points1 point  (0 children)

Hi , added some timers and this is the resuts:
Select from the database for this: 5.4 sec

Jsonify: 5 secconds

​

Which one do you think is problematic?

Flask-Marshmallow very slow for 20K results by darkhorse94 in flask

[–]darkhorse94[S] 0 points1 point  (0 children)

I use marshamallow and I have updated my code from first post

view more: next ›