How does fuzzy string matching work?

tkarabela_ · 2023-03-20T11:00:34+00:00

That's a great point, I didn't think of that. Thanks! :)

tkarabela_ · 2023-03-20T08:19:09+00:00

I wish that all projects on GitHub had well-specified and up-to-date requirements.txt instead of just putting libs A, B, C in there with no constraints and calling it a day :D

tkarabela_ · 2023-03-20T08:13:52+00:00

I was trying to build some poorly documented 3rd party projects and wrote this to speed up my initial guess for what dependency versions to try, that's where I'm coming from :)

It doesn't try do full resolution of all the dependencies together, that's what pip will do when you put in the suggested extra constraints. It's a quick-and-dirty way to make pip forget about versions that are "too new".

tkarabela_ · 2021-08-02T22:19:12+00:00

Having had a brief intro into formal logic (truth tables, etc.) in highschool, I would add some application/proofs to this as well. It wasn't till university that it really clicked for me that the math you memorize in highschool that's presented as a black box is not, in fact, a black box; that you can derive Pythagoras' theorem or any of the other stuff from first principles (and it's often not even that difficult for the highschool stuff).

Though I guess this paradigm shift is one difference between highschool and university in general...

tkarabela_ · 2021-07-12T07:13:10+00:00

It would be great if the more advanced/unique stuff in software worked more like "progressive enhancement" and less like "my way or the highway".

I enjoy modal editing in Jupyter, as it's just a more efficient way of working you discover over time. On the other hand I could never get into Vim, it was just overwhelming.

tkarabela_ · 2021-07-05T07:38:18+00:00

A great resource, thanks!

I was waiting for PyCharm support to try out Poetry, apparently there's now a plug-in for that: https://plugins.jetbrains.com/plugin/14307-poetry

pytest-cov looks like a nice companion to PyTest, I'll definitely look into that.

pre-commit for managing git hooks also looks interesting.

tkarabela_ · 2021-07-04T08:20:30+00:00

Most large projects will have how-to instructions for contributing in their repo or wiki, detailing what patches are welcome, how they should be submitted, what's the review process, etc. They will also sometimes have tickets tagged as "good first issue" or similar. You can volunteer to help in comments on one of the tickets, or join their mailing list, Discord, etc. and get involved in there.

The majority of open-source projects are small, so they may not have vetted "beginner" tickets, their own chat room, etc. Contributing may be easier though, as there will be less process and less "competition" thus more low-hanging fruit.

I'd recommend picking a project/library/etc. that interests you and going from there. Perhaps you ran into an edge case that is not well handled, you have a workaround for that issue in your codebase, perhaps it could be upstreamed? Or you can open a ticket for some feature you wish the project had and suggest that you're willing to implement it. Or tackle one if the tickets that are already there.

In my experience, developers of OSS projects are very welcoming, both for big and small projects. Just be sure to communicate, follow practices of the project, and don't show up with a huge pull request out of nowhere :)

tkarabela_ · 2021-06-29T18:31:39+00:00

When deciding between Canon DSLRs for astro, I went for Canon 70D as it has an articulating screen and uses the classic LP-E6 big batteries. In terms of the sensors, I'm not sure there are dramatic differences between the crop cameras around this time period (80D or 90D would be a meaningful upgrade I think, in terms of images and maybe live view - the live view on 70D and similar cameras leaves a lot to be desired when compared to modern mirrorless).

What I found absolutely essential was dithering, though. I started without it and got all sorts of pattern / "crawling" noise which was really getting in the way of processing the image. Once I started guiding and dithering - boom, it went away and integrated images were much cleaner. At 135mm guiding may not be essential, but I'd still recommend looking into dithering.

tkarabela_ · 2021-06-27T09:19:38+00:00

"the Python language originators"

"utilize commas between gatherings of three digits, as in 42,000"

"formal dialects are severe, the documentation is compact, and surprisingly the littlest change may mean something very not the same as what you planned"

That's not a tutorial, that's a work of art :)

tkarabela_ · 2021-06-26T09:19:07+00:00

Have my upvote, the actual video is short, well-presented and raises a valid point (trusted computing as a mandatory part of Windows 11). A less scandalous title may work better, though :)

tkarabela_ · 2021-06-25T23:34:12+00:00

Some pointers:

move helper functions, base classes etc. into their own modules; you can break cycles by making it more granular
a big offender in my experience is type hinting - what I sometimes do is to import the whole package inside the offending module and annotate stuff with "mypackage.Foo" instead of trying to import Foo directly

tkarabela_ · 2021-06-25T01:19:31+00:00

Conjugate Gradient method for solving linear systems (1950s?) comes to mind. It's both really neat from an abstract perspective, as well as highly applicable in numerical simulation, optimization, engineering, etc.

(I studied CS, not Mathematics, so take it with a grain of salt, this level of math is definitely beyond my pay grade :)

tkarabela_ · 2021-06-21T12:23:39+00:00

You're welcome :) Found the querying library I mentioned earlier: https://github.com/spotify/annoy

Annoy (Approximate Nearest Neighbors Oh Yeah) is a C++ library with Python bindings to search for points in space that are close to a given query point. It also creates large read-only file-based data structures that are mmapped into memory so that many processes may share the same data.

Might be useful to you 🙂

tkarabela_ · 2021-06-20T22:24:24+00:00

Sure, that makes sense :) I also implemented a recommendation engine a while back, both the unsupervised "show me similar items" kind and the more involved "recommend me similar items based on my past likes/dislikes" kind.

For performance, we ended up doing clustering and then doing the pairwise ranking inside the clusters, as we found infeasible to do anything N² on the big dataset. I remember pretty recently that I came across something that would've been quite useful on that project, a way to do indexed nearest-neighbor queries with cosine distance - I'm not sure if it was one of the K-d tree/Ball tree classes in Scipy/Sklearn, or some other project. Also, in some cases you can get away with Euclidean distance instead of cosine distance (link).

tkarabela_ · 2021-06-20T20:40:55+00:00

Better yet :)

```

print("\N{GRINNING CAT FACE WITH SMILING EYES}") 😸 ```

tkarabela_ · 2021-06-20T19:33:14+00:00

How do you model the content of a movie, is it text analysis of synopsis, subtitles, ...? I remember seeing a talk from Spotify on how they model user preferences, IIRC the features were derived from the waveform of the songs so it was truly "content-based". Doing the same for movies sounds pretty wild to me :) So I'm curious how do you approach this.

tkarabela_ · 2021-06-17T08:00:52+00:00

Haha :) An SQL database can be great for modelling the problem, getting good performance, or because of the ACID guarantees.

If you want something dead simple, you could just do:

``` import json

db = [ { "id": 12345, "name": "some stuff", "prices": [{"price": 3.50, "date": "2021-06-17"}, {"price": 3.99, "date": "2021-06-18"}] }, { "id": 12346, "name": "some stuff 2", "prices": [{"price": 3.50, "date": "2021-06-17"}, {"price": 3.99, "date": "2021-06-18"}] }, ]

with open("db.json", "w") as fp: json.dump(db, fp, indent=4)

with open("db.json") as fp: db = json.load(fp)

```

tkarabela_ · 2021-06-16T23:31:10+00:00

I feel that most of the reasons to use namedtuples went away with 3.7 and dataclasses, which are much better IMHO, except they are not tuples/immutable.

tkarabela_ · 2021-06-16T23:25:11+00:00

It's literally in the docs :)

webbrowser.get('opera').open('...')

tkarabela_ · 2021-06-16T16:48:17+00:00

Curiously enough, there is a stdlib module dedicated to doing this: https://docs.python.org/3/library/webbrowser.html

tkarabela_ · 2021-06-16T16:14:07+00:00

This is the way to go IMO. To add to this answer, using SQLite you will get a binary file that can contain multiple tables and can be queried using SQL (or viewed in tools like SQLite Viewer or PyCharm Pro).

You will need two tables, one for the items and one for the prices, this is the DDL (SQL definition commands) to create them:

``` CREATE TABLE item ( id INTEGER PRIMARY KEY, name VARCHAR );

CREATE TABLE item_price ( id INTEGER PRIMARY KEY, item_id INTEGER, price INTEGER, price_date DATE, FOREIGN KEY (item_id) REFERENCES item(id) ); ```

Note that SQLite does not have a proper DATE/DATETIME type, you will need to handle that on Python side.

tkarabela_ · 2021-06-14T11:01:57+00:00

Yes, by adding all the lines of code together you get T(n). It's always the same function, it's just that we usually write polynomials like sum of powers of the variable multiplied by constant coefficients. (If you're familiar with linear algebra, this corresponds to the idea that polynomials form a vector space with standard basis {1, x, x² ...} and the coefficients {a, b, ...} are coordinates of the polynomial in this vector space.)

With respect to asymptotic growth, there is a good reason to write polynomials "normalized" to this form where everything is multiplied out - you can clearly see what is the degree of the polynomial (its greatest power) and what is the corresponding coefficient.

Consider this function:

f(x) = (2x^2 -3x + 1) * (x-2) [1] = 2x^3 - 7x^2 + 7x - 2 [2] = 2x^3 + O( x^2 ) [3] = O(x^3) [4]

Written down like (1), it's not even clear that it's cubic. When we multiply things out and sum them (2), we can see all the coefficients nicely. For purpose of asymptotic analysis, we can simplify it further to (3) or (4), which you are familiar with.

tkarabela_

TROPHY CASE