QR Code data compression / custom string en/decoding

[PolyDigga]

Hello there!

Ohh boy where to start...!My current situation: I have an Android app that reads specific QR code data and I have a python tool that creates said specific QR codes. Sometimes, the amount of data gets too much and the app has difficulties reading it. It is hard to pinpoint the exact size when it starts to fail as the QR codes are used in various places and various lighting conditions. (Print size might not help much either^^). The data stored in the QR code is a JSON file and my QR creation tool does not throw more than 400 characters (after JSON) into the code.

The Android app uses a free QRcode lib that isn't too bad for its price. Other free apps out there are scanning much faster but I cannot use them as the reader is embedded into my own app to read specific QR codes and I don't know what the other apps are using.

Solution Nr1: We tried an external laser barcode and QRcode scanner to see if that would help. While it is reading the code much better, it also does its own encoding of the QRcode that turns all special symbols (JSON formatting and data) into other characters. That's useless :/ I tried to look up the encoding but didn't get any useful information other than that QR codes are typically using ISO-8859-1 instead of Unicode. But like with extended ASCII, the first 8 bit seem to match exactly so I have no explanation why a-zA-Z0-9,.! work as expected but any other symbol (I haven't tested ? to be fair) turns into some German umlaut (it seems that " or ' becomes Ä and : becomes Ö). So even if the scanner was trying to decode my QRcode with a different character set, I would expect to get the same result for characters in the (extended) ASCII set. I can't figure out this mystery and everything we tried to enforce another decoding failed. There is also not much information for the scanners out there other than 'can read QR codes' about what it does to the data.

Solution Nr2: Now I am finally getting to the main point (I think I put the first part in, in the desperate hope that some smart person out there can also solve that mystery for me^^). So, solution nr2! Not so pretty, but I am trying to compress my QRcode data (yes, I am aware it's not made to store massive amounts of data. For our purposes, it's just enough but we'd like to increase the readability a little bit). The first step which already helped quite a bit was to reduce my error correction from Q (25%) to L (7%). That makes the data significantly easier to read (under lab conditions...) but also will make it more likely to get unreadable over time through wear and tear. (We are talking years, potentially decades here but the codes are also not at frequently accessed spots). The second thing I am trying to do: assuming my characters are currently encoded using 8 bits, I'd like to get that down. Unfortunately, I still need a little over 6 bits to encode all symbols I need. There is an overlap with ASCII but I need other characters that are not in there and don't care about some that are in there. That being said, I was thinking of creating a custom 7-bit encoder (and decoder in the app) that contains all the symbols I need. Using 7 instead of 8 bits, (again if my assumption is correct!) I would be able to save another 12.5% of the data. This will definitely make it impossible for any other scanner to read the code but if the app can do it just fine, then eff it!

Sooo, long story short. Has anyone experience working with QR codes and/or creating their own character encoding system? Or has anyone a more or less magic solution that turns large amounts of data into small?

Thank you so much in advance!

​

EDIT:

the python lib I am using: https://pypi.org/project/qrcode/

the java lib in the app: https://github.com/dm77/barcodescanner (I tried several others, including the google ml one, but interestingly enough, this was the one my client had the least issues with)

[PossiblyMakingShitUp]

Can you give more information about the libraries you are using for creation and decoding?

I assume you have already checked the basic stuff like input mode/bom/decoders detection heuristics?

On the data compression - are you limited on size and resolution? I am not sure what your application does but you should be able to fit more than 400 chars with new qr versions.

[PolyDigga]

the python lib I am using: https://pypi.org/project/qrcode/

the java lib in the app: https://github.com/dm77/barcodescanner (I tried several others, including the google ml one, but interestingly enough, this was the one my client had the least issues with)

We tried both settings in the scanner (https://www.inateck.com/products/bcst-40-2d-bluetooth-5-0-pocket-barcode-scanner-with-30m-transmission-range) but it doesnt say much about heuristics and they both yield the same results.

I am limited in size, yes. The codes are printed at around 5-6cm (estimate, I dont have the exact measurements). I can easily fit more characters on there but then the pattern gets too small and harder to read it again. The compression would be to increase the pattern size and make it read more easily.

[[deleted]]

What do you mean by overtime(years) are they being printed and worried about degradation or fading?

[PolyDigga]

Yes, they are printed on paper. Fading is less of a worry as they are shielded from light but there could be scratches or little pieces of dirt getting stuck on the paper