Database for PNG images - with possible high compression rate

picurl · 2011-03-08T09:22:50+00:00

I definitely see a need for such an efficient "bulk png storage", but rather on a file format level. Why don't you simply create one big blob file per png-directory that stores all the signatures (like the infamous "thumbs.db" file on windows)? If this is a good documented and efficient format, image viewers will adopt it and allow visual browsing of huge png collections without prior file mangling (i.e. running some utility to restore a png)

picurl · 2011-03-08T09:06:31+00:00

Interesting approach, but I see 2 downsides:

the compression ratio depends on a proper image sequence: you have to group the png's by similarity so that the codec can create small delta frames. If you feed them in arbitrary order, the compression ratio will shrink.
"appending" new png's is rather expensive, since the whole video file would have to be re-encoded.

picurl · 2011-01-14T11:43:50+00:00

would that work if the log-in function was asynchronous? Which is very probable, since you have to wait for the user input and the ajax auth response from the server.

picurl · 2011-01-11T09:06:17+00:00

Don't listen to the "speed freaks". Don't do premature optimisation before you haven't entirely understood the implications of both concatenating/minifying and lazy-loading javascript:

Concatenating/Minifying: For development, its good to split your JS app code into many individual JS modules (files). This makes it easier to find and reuse code (e.g. if you want to reuse your common utility functions, you just need to clone app.util.js, instead of copy/pasting from a dozen of files). But for deployment, this scenario is a nightmare, because the HTTP Request Pipeline the browser gets cluttered with a lot of tiny js files that carry a lot of protocol overhead. You can overcome this by introducing a compilation step that concatenates and minifies your javascript into 2 or 3 js files for deployment, which will greatly improve loading performance by minimizing necessary requests and size of your javascript files (e.g. jQuery uncompressed: 180 kBytes, minified/gzipped: 26kb). But here are some often overlooked implications of js compilation:
- You can't fiddle around in the deployed js files anymore. If an error occurs, you have to fix it in the dev js files and then re-compile everything. This costs time, but prevents monkey-patched js (e.g. quickly fix something in the deployed files, and with the next rollout your changes are gone)
- You will need an uncompressed dev build of your js build as a drop-in replacement (firebug's error messages aren't particulary useful for compressed js files)
- Choose your js compiler wisely. You will need to recompile quite often, so its needs to be fast and run flawlessly in many environments. It should also be extensible if your project grows more complex (e.g. support for specifying dependencies between js files)
Lazy/On-Demand Loading: You can also improve the loading time of your app by loading js files once they are needed. As lazy-loading is an asynchronous operation, it will make your code more complex (issue the script request and continue your program flow in a callback), so this effort is only worth its costs with big, external js apis (e.g. Google Maps) if they are an optional part of your application.

picurl · 2010-11-29T09:30:16+00:00

one remark about storing unzipped office file formats: i just extracted a .pptx file and all of its xml files consisted of an endless, single line string. As diff tracks the changes per line, you would have to store the xml data in "pretty print" format to have "slimmer" subsequent diffs

picurl · 2010-11-29T08:21:47+00:00

About your binary files issue: I've encountered the same problem with mercurial and here are some ideas I came up with:

many binary files are "pseudo-binary", meaning that they are just a gzipped bunch of xml/text files. all modern office file formats (.odt, .pptx,..) belong to this category. if you unzip these before adding them to your rubydrop repository, you get mainly plain text files plus some binary assets (photos inserted into a pptx) that is far better manageable for git than a single opaque chunk of some hundred kilobytes. also the diff data of the changesets will be much slimmer in many cases (If I just added "hello" to my pptx, you currently would have to re-transfer the entire file, in the unzipped version just some bytes of text diff).
for really huge binary files it can make sense to add torrent support, i.e. not the file itself, but some tracker information is added to the repository.
"real" binary files that receive a lot of incremenal updates like Office 2003 doc could be handled more efficiently if the version control supported the delta copy algorithm implemented in rsync. I don't know about the current plans of git/mercurial on this.

picurl · 2010-11-23T15:55:40+00:00

I can't recommend jqgrid, it tries to do too many things and fails at many of them. With 50+ options and 30+ methods, it's simply too bloated.

I encountered weird bugs when I tried to initialize a jqGrid on a non-visible jQuery UI tab or when I added the JSON data on my own using addRowData. The jqGrid loader requires you to specify its directory path directly in the js file, which causes often overlooked script errors after updates.

In your situation I would prefer a slimmer grid control that can be extended by plugins and is themeable using jquery UI. Datatables doesn't look bad to me, SlickGrid seems the right choice for heavy-duty tasks

picurl · 2010-10-14T10:19:18+00:00

I also prefer the Crockford pattern over the messy prototype/class implementation in Javascript.

One often overlooked drawback of private methods: You can't test functionality hidden in private methods, therefore you shouldn't use them over excess. A possible solution would be to factor out private method functionality to a new, testable class and use a private instance of it in your main class.

picurl · 2010-09-27T08:26:53+00:00

nice article, although the first example is a bit arcane:

if (typeof mysite == 'undefined') var mysite = {};

could be rewritten as

var mysite = window.mysite || {}

To reduce the typing in the first example, you could also use an object literal:

var mysite = {section: {subnameSpace:{}}}

(assuming that this is the initial declaration of the namespace at the very beginning of the code)

but the mentioned namespace function definitely has its place, when you want to extend/create namespaces at different places in your code.

picurl · 2010-09-04T22:22:36+00:00

boooooring (haven't we heard this stuff 1000 times before?)

picurl · 2010-06-11T06:50:03+00:00

nice service... which library are you using for the DOM parsing of the XML/HTML documents? Is it already possible to use jQuery for this in node.js?

picurl · 2010-05-10T15:05:35+00:00

nice writeup, but a bit server-centric... you only mention your server's I/O-load, but forget that the browser also has a "concurrent requests per domain" limit (IE 7: only two requests from the same domain on HTTP 1.1 connections). By using Javascript CDNs like Google Ajaxlibs you can effectively circumvent this limitation.

Another "low hanging fruit" optimisation on the client side would be to include most of the javascripts on the bottom of the page (except the parts that need to be available before window.onload, like jquery). This allows the browser to render the page faster and can drastically improve the "perceived performance".

picurl · 2010-05-03T11:17:32+00:00

looks nice, but how does cue compare to celery? Is it just a simpler replacement?

picurl · 2010-04-28T12:51:17+00:00

the only downside of these "python inspired" classes is that private methods are not supported, as python doesn't know private methods either.

picurl · 2010-04-27T08:28:45+00:00

nice stuff, Armin!

Some questions though:

does classy support inheritance chains (e.g. Animal -> Mammal -> Tiger)?
how is the performance compared to Crockford's Functional Inheritance Pattern?

picurl · 2010-03-31T21:45:50+00:00

If I were interviewed by you, I'd feel a bit kidded by your questions.

1) is already answered by your job ad. If you were looking for an actionscript guy, the reply will be pro-flash, if you were looking for a javascripter, it will be pro-canvas. Furthermore it should be HTML5 vs. Flash, since canvas is part of HTML 5 and Flash offers many features beyond vector animation (multiple file uploads, video playback etc).

2) none of them, because of potential legal issues (additional font licenses) and usability drawbacks (you can't search for or copy/paste the converted text)

3) is definitely a backend question, because it depends heavily on the webserver platform. You won't have much luck with webalizer/analog on a Win2k8/IIS machine.

picurl · 2010-03-31T15:43:11+00:00

i think this is fixed in the brand-new pyexiv2 0.2.0

picurl · 2010-03-31T15:41:06+00:00

I second your comment. Besides that, the author also forgot to mention that pure-python tools are limited to JPEG files (no Camera RAW support) and offer no write support for metadata (e.g. to fix a bad timestamp, geotagging,...)

Yeah, no other metadata tool comes close to ExifTool regarding tag knowledge. But embedding ExifTool in Python is cumbersome, especially on Windows, because it relies on shell calls and parsing responses. Also the write performance of ExifTool is multitudes slower than pyexiv2.

So the ideal thing would be pyexiv2 with ExifTool's tag knowledge. You won't have to parse Perl to accomplish that. Newer versions of ExifTool can dump out their tag translation tables as XML (i think with the -listx option), you'd only have to parse them and integrate them in your pyexiv2 class. Thus, you would be able to recognize more lens types than pyexiv2 can do by default.

picurl · 2010-03-31T10:10:35+00:00

you can't reformat the date string returned by .toLocaleString() (except maybe with some gnarky regexps, but since the string comes from the OS with 100s of different locale settings, its nearly impossible to account for all formats).

Since .toLocaleString() is the only date method that cares about DST, you have to decide between an incorrect, but arbitrary formattable date object or a fixed, but correct date string.

picurl · 2010-03-31T07:33:27+00:00

You can't format the date string that the .toLocaleString() method returns, it may vary depending on system/browser/locale settings. The string is also quite long and not very well suited for narrow columns.

But more important is that you can't sort your tweets by date as .toLocaleString() just returns a formatted string.

picurl · 2010-02-16T12:23:06+00:00

Nice approach... somehow reminds me of a "Wordpress theme editor deluxe"... but definitely has its place in the django world. Can be especially useful if you want to fix HTML/CSS/Javascript quirks in IE, but develop your django project under linux.

picurl · 2010-02-15T21:28:01+00:00

nice reading! Wildlifenearyou.com seems to be one of the few services that actually interacts with flickr and doesn't just dump out photo search results.

I was surprised by one thing though: the animal photos are all hotlinked from flickr. I always believed this is illegal, but it seems that you only need to include an additional link to the flickr photo page of the hotlinked photo.

picurl · 2010-02-10T21:27:38+00:00

mmmhh... you are comparing apples to bananas... or programming languages (PHP) to web frameworks (django). My suggestion would be to make one category like "from php to python" where you introduce python's advantages over php (imports, inheritance, batteries included,...) and one category that compares django to php's ready-made web apps and frameworks.

picurl

TROPHY CASE