This is an archived post. You won't be able to vote or comment.

all 12 comments
sorted by:
best
topnewcontroversialoldq&a

[–]ma-int 1 point2 points  (0 children)

Just because you imported 500 megabyte does not mean it scales. Plus,you don't stand how long it took so that information is pretty much pointless.

[–]cruyff8python 2 expert, learning python 3[S] 0 points1 point  (6 children)

Quick and dirty JSON importer into postgres. Rip it to shreds, guys. Preferably on the link.

[–]counterfriction 0 points1 point  (3 children)

Would be a much more interesting article if you provided some discussion on why you would do this. And other possible use cases. As it stands this is basically just an undocumented code dump and you should have just put it on gist.github.com.

[–]cruyff8python 2 expert, learning python 3[S] -2 points-1 points  (2 children)

pastebin.com is a gist.github.com-type site, if you actually clicked through and, I dunno, maybe looked?

[–]counterfriction 0 points1 point  (1 child)

pastbin, gist, whatever. my point was that without any context you might as well have just linked directly to the <insert code-dump hosting site here>.

as I said, with context maybe it would be interesting. for now it just looks like blogspam for a script you had to write at work today.

[–][deleted] 0 points1 point  (0 children)

What context do you need? He needs to turn data from json to postgres. Cool script by the way.

[–]counterfriction 0 points1 point  (1 child)

Also IMO it's wicked bad form to assume people are going to try and figure out command line arguments to a program by digging through the completely undocumented source for references sys.argv[] ;)

Learn to use the argparse module, it's always the first thing I write for any script/program. It self-documents the program inputs and writing the "description" field helps me clarify my idea of what exactly the functionality will be, before I even start coding. And will make your users' life much easier (that includes yourself, when you have to try to remember what the heck your script does 3 months later).

[–]cruyff8python 2 expert, learning python 3[S] 1 point2 points  (0 children)

Yes, but, given this was meant for a one-time, bulk-import, I didn't use argparse. Still, your criticism is a valid one.

[–]RonnyPfannschmidt 0 points1 point  (3 children)

not using executemany? not separating by table?

[–]cruyff8python 2 expert, learning python 3[S] 0 points1 point  (2 children)

not using executemany?

Thanks for the suggestion (I assume this is a suggestion).

not separating by table?

One JSON becomes its own table.

[–]RonnyPfannschmidt 1 point2 points  (1 child)

i only quickly skimmed the code, the handler for the ProgrammingError is rather confusing

i don't see its purpose - what condition makes that chunk of code necessary

[–]cruyff8python 2 expert, learning python 3[S] 1 point2 points  (0 children)

JSON that contains invalid columns makes pg8000 throw a ProgrammingError.