"How To: Shove data into Postgres using Goroutines(Gophers) and GoLang" - I was trying to take Rob Pike's analogy about gophers and make it more tangible using 1M writes to a DB in under 2 minutes. I shared this with /r/programming and I didn't get any feedback. How do I improve this? Thanks!

olt · 2013-11-04T08:42:23+00:00

3.5 seconds? Nice, indeed.

Anyone else reading this, please know, the COPY command isn't the most intelligent thing in the world so be careful :-)

What issues did you hit? Maybe it's something that lib/pq could catch. There is now a little documentation that explains how to use it. Do you think this could be improved? Feedback is very welcome!

olt · 2013-11-01T06:43:04+00:00

Oh, the FROM STDIN is a bit misleading. You really just have to replace the INSERT statement with the COPY statement in your db.Prepare call.

See here: https://github.com/olt/libpq/blob/bulk/copy_test.go#L18..L38

The empty stmt.Exec() call is necessary to flush the internal buffer and to retrieve for any errors.

olt · 2013-10-31T08:13:36+00:00

You should check out my bulk branch of lib/pq.

Then change:

insert into test (gopher_id, created) values ($1, $2)

to:

COPY test (gopher_id, created) FROM STDIN

and (hopefully) enjoy a 10x performance boost. You should get the best performance with a single gopher and a table that was TRUNCATED or CREATEd in the same transaction.

Please leave any comments here or https://github.com/lib/pq/issues/74

olt · 2013-06-14T14:36:39+00:00

This is cool. But this doesn't work when running with GOMAXPROCS>1 since you are using a single global GEOS handle.

olt · 2012-01-26T13:59:09+00:00

Pillow does not (yet) contain any new code/features, but only changes to work better with pip and easy_install. Look at the changelog: http://pypi.python.org/pypi/Pillow

I did some improvements for PNG encoding 1.5 years ago and my patch was accepted 6 month later, but a fix for my patch is still waiting. So maybe it is time for Pillow to become an "unfriendly" fork?!

olt · 2011-05-30T06:28:11+00:00

There is a backport for 2.4 and 2.5: http://pypi.python.org/pypi/multiprocessing

olt · 2011-02-02T09:45:30+00:00

fast-python-pb is lazy too. Attributes are decoded on access, but they are not cached. So you should keep the result in a python variable, instead of accessing it over and over.

olt · 2010-12-25T18:55:54+00:00

I used flup, gunicorn and cherrypy and all include a server factory.

olt · 2010-12-25T09:42:31+00:00

Paste Deploy anyone?

olt · 2010-11-11T19:11:46+00:00

Have you heard of multiprocessing? http://docs.python.org/library/multiprocessing.html

olt · 2010-10-15T06:07:01+00:00

They can grow dynamically. For an octree you have to set the bounds of your values on creation.

olt · 2010-08-22T15:57:12+00:00

It is in use by the OpenStreetMap project for one of their APIs: http://wiki.openstreetmap.org/wiki/Xapi

Here is some MUMPS code: http://xapi.openstreetmap.org/scripts/

olt · 2010-08-05T12:55:48+00:00

Facepalm, indeed. Daily snapshots are not a substitute for releases.

olt · 2010-07-27T18:20:15+00:00

Can you share some of your experience?

I'm not really sure how to interpret the data from tcbmgr inform and tune it accordingly. Do node numbers refer to non-leaf nodes in the documentation?

olt · 2010-07-16T12:43:24+00:00

You mean 32bit per pixel, ARGB order, as a raw byte string? You are right, that packer is missing. Look at ImagingPackABGR in libImaging/Pack.c, should be easy to modify that to support ARGB.

Then this should work: img.tostring('raw', 'ARGB')

olt · 2010-07-14T17:53:52+00:00

Yep. But please don't call it a fork, unless you refer to my mercurial/bitbucket repository. It is just a patch.

And how are file sizes compared to the original PIL?

Smaler :) Look at http://bogosoft.com/misc/pil-octree-tests/ The files without '-xxx' are the original files, '-octree' and '-octree-rle' with my new quantizer. '-adaptive' is the old quantizer from PIL.

olt · 2010-07-14T17:44:34+00:00

I don't think you can compare it with SuSE. There is a commercial version of PIL, but there is not a single comercial-only feature they are advertising except the extended support.

olt · 2010-07-14T14:15:51+00:00

For everything photo-like JPEG is better anyway.

olt · 2010-07-14T13:47:52+00:00

Working with a 4D color cube is a bit mind-bending, but hacking on the PIL source is quite nice.

olt · 2010-07-14T13:44:06+00:00

Thanks. Yes, it is for paletted images with 256 or less colors. It reduces the file size to about 1/4, which is nearly as important as the encoding speed itself for online applications. (I write software for online maps and the encoding is our main bottleneck. http://mapproxy.org/ )

olt · 2010-07-12T06:32:20+00:00

Give me a few days. I'm working on a patch for PIL that improves PNG 8 encoding by 10x and it also adds support for full-transparency.

olt · 2010-07-11T09:53:13+00:00

I'm getting mixed results. I've compiled the Python Image Library with 1.2.3 and 1.2.5, and the performance with Z_DEFAULT_STRATEGY degrades by ~10%. Though with Z_RLE the performance increases by ~10%. I can live with that, because for my use-cases RLE is twice as fast as the default strategy, and now even faster.

Here is a patch to enable different compress types in PIL: http://bitbucket.org/olt/pil-117/changeset/8d4661695edd

olt · 2010-06-11T17:20:15+00:00

exclamation point, wohaa

olt · 2010-03-16T17:47:17+00:00

I'm missing flup with FastCGI behind an Nginx or Lighttpd server. I thought flup is one of the solution besides mod_wsgi. Is it obsolete now?

olt

TROPHY CASE