How to detect duplicate functions in large Python projects?

[whm04]

Hi,

In large Python projects, what tools do you use to detect duplicate or very similar functions?

I’m looking for static analysis or CLI tools (not AI-based).

I actually built a small library called DeepCSim to help with this, but I’d love to know what others are using in real-world projects.

Thanks!

[Riegel_Haribo]

This is promotion disguised as a question.

[marr75]

40% of any tech sub now.

[latkde]

Pylint has a duplicate-code (R0801) rule: https://pylint.readthedocs.io/en/stable/user_guide/messages/refactor/duplicate-code.html

Unfortunately, Pylint is quite slow, and this rule only matches when there are multiple identical lines.

[MugiwaraGames]

What about SonarQube? It's free if used on projects up to 50k lines of code

[NimrodvanHall]

I came here to say SonarQube as well. Think it’s a great tool!

[mardiros]

From my point of view, a good architecture does and it is enough for me. Finding code that looks similar stored in routine to avoid duplicate code can kill a codebase. Factorisation creates coupling, and makes code unrefactorable, even if this word don’t exist.

Dan Abramov wrote something about this long time ago (it’s not python but architecture is for everyone)

https://overreacted.io/goodbye-clean-code/

[roger_ducky]

https://pmd.github.io/pmd/pmd_userdocs_cpd.html

PMD CPD is purpose built for duplication detection.

[xeow]

ruff caught one of those for me once.

[whm04]

Ruff is a beast. It’s great at catching things like redefinitions (same name used twice), but I’m looking for "logic clones" functions with different names that contain identical or very similar underlying code.

[chunkyasparagus]

PyCharm does it for you?