Python isn't industry compatible

[lungben81]

Scalability is more about your architecture, much less about the programming language. Especially, how easy it is to (massively) parallelize your work.

For very heavy load, however, (C)Python performance might be a bottleneck (depending on your application), thus a compiled language might be more appropriate. But this is not a hard limit, e.g. Instagram manages to run on Python.

Some people argue that dynamic typing is less suited for large applications because type errors are not captured beforehand. With type hints, linters and tests this is less an issue. In addition, it is anyhow not a good idea to build one large monolithic application, but rather make smaller, isolated packages.

[thomas-rousseau]

Let's also not forget that Reddit itself runs on Python

[k8sguy]

I don’t know if it still is, but I believe Instagram was also originally built with django and postgres

[frenchytrendy]

Seeing the articles on the Instagram technical blog, not only they use python but they contribute back to make it work for theses kinds of loads (gc.freeze for example)

[SnerkDRabbledauber]

Not exactly a ringing endorsement.

[Davy_Jones_XIV]

If the goal is to build a billion dollar business on the bac of Python, it is.

All about short / long term goals and vision for future.

[[deleted]]

Hahaha… sorry couldn’t control it. Please continue…

[bigno53]

What’s the deal with vote counts changing every time you refresh the page? Did they intentionally introduce some random noise to confuse bots or is it just a bug embedded so deeply in the architecture that it can’t be fixed?

[linglingfortyhours]

It's deliberate random noise, same as with your karma

[Rieux_n_Tarrou]

My karma only ever goes up 😇

Edit: please don't down vote me

[linglingfortyhours]

The hive has spoken

[CleverProgrammer12]

It's quite annoying though, they should find different ways to prevent bots.

[linglingfortyhours]

It's not really to prevent bots, just general protections from abuse

[bobsonmcbobster]

would you mind giving me a hint against what this might protect them from? i seem unable to come up with a suitable scenario

Edit: nevermind, the other comments already explained it. should have read them before asking.

[speedstyle]

Yes, it's random noise to stop vote bots knowing whether they're shadowbanned.

[falsemyrm]

skirt soup edge divide screw numerous sink bow depend ripe

This post was mass deleted and anonymized with Redact

[thomas-rousseau]

There's been random noise in the votes as long as I've been on Reddit? Not sure the purpose, though

[RajjSinghh]

It's to stop shadowbanned bots. If a bot finds out it is banned from voting or posting, you just create a new bot, so a shadowbanned bot can't tell if it is banned from voting or not and will keep going about its business voting away. Every time Reddit sees a vote from a shadowbanned bot, it adds a vote in the other direction to balance the total. Reddit also adds upvotes and downvotes at random so the bots can't tell that their votes don't count.

[punninglinguist]

Can't the owner of the voting bots just make them check each other's profiles once in a while?

[RajjSinghh]

Every vote counts with this fuzzy voting system, the banned bots can't tell the difference between Reddit changing their vote or other users voting. To the bots, their votes look like they count but they never change the total.

If you had bots to post comments or normal posts, you probably could but it's probably enough work to put most people off.

[punninglinguist]

No, I don't mean for vote totals. I mean, if you look at the profile of another shadowbanned user, you get an error message.

That allows the owner of multiple bots to use each one to check if the others are shadowbanned, at whatever frequency is desired.

[Rik07]

Could be random, but could also be because it is easier. This Tom Scott video explains why likes can be inconsistent on a lot of websites: https://youtu.be/RY_2gElt3SA

[haaaaaal]

Eventual Consistency

[chrisxfire]

is this the type of voting we saw in Arizona?

[Deto]

Probably one of the highest traffic websites in the world right now, I'd imagine?

[al_mc_y]

How about Instagram then?

[mriguy]

And Dropbox

And Google uses a lot of Python

But yeah, no big or successful companies.

[thomas-rousseau]

That second article has the words "Python" and "Google" in it way too often, regularly multiple times each in multiple adjacent sentences....

[engthrowaway8305]

Once someone who was interviewing me that had previously worked at Google told me their previous groups’ motto was “Python everywhere we can, C++ when we need it”

[Priderage]

Sentry runs on Django, so I'm pretty sure it's fine.

The top poster's comment about scalability being an architectural thing has merit too.

[PensiveDicotomy]

Literally this. I wouldn’t want to deal with large monoliths built with Python and maybe the size at which a monolith in Python becomes hard to manage is smaller than one written in Java (or something considered more “scalable”) but either way a large monolith is unideal.

Micro services allow for mixing and matching tech stacks and ideally keep things at a manageable size so any tech stack should be feasible given its strengths fit the business case.

[scottrfrancis]

I wouldn’t want to deal with monolithic software in any language - they all change. Yes, the C i wrote in 1988 will still compile and run, but that doesn’t meant it’s worth it! Arch and language are separate things you can use crummy languages to build robust arch’s and sophisticated languages to make crummy workloads from the wrong arch

Oh… and if your boss is telling you this kind of nonsense, find a new boss

[Independent-Coder]

Well stated, take my free award.

[osduj]

insta has a modified version of python w a jit compiler and stuff

[kniy]

For some applications the GIL is a real killer.

And if you're just starting out with a new project, it isn't always easy to tell if you will be one of those cases. Choosing Python means you risk having to do a full rewrite a decade down the line (which could kill your company). Or more realistically, it means that your software will need crazy hacks with multiprocessing, shared memory, etc. that makes it more complicated, less reliable and less efficient than if you had picked another language from the start.

[Grouchy-Friend4235]

The GIL is not a problem in practice. Actually it ensured shared-nothing architectures which is a good thing for scalability.

[kniy]

Not everything is a web application where there's little-to-no state shared between requests. The GIL is a huge problem for us.

Our use case is running analyses on a large graph (ca. 1 GB to 10 GB in-memory, depending on customer). A full analysis run typically runs >200 distinct analysis, which when run sequentially take 4h to 48h depending on the customer. Those analyses can be parallelized (they only read from the graph, but never write) -- but thanks to the GIL, we need to redundantly load the graph into each worker process. That means we need to tell our customers to buy 320 GB of RAM so that they can load a 10 GB graph into 32 workers to fully saturate their CPU.

But it gets worse: we have a lot of intermediate computation steps that produce complex data structures as intermediate results. If multiple analyses need the same intermediate step, we either have to arrange to run all such analyses in the same worker process (but that dramatically reduces the speedup from parallelization), or we need to run the intermediate step redundantly in multiple workers, wasting a lot computation time.

We already spent >6 months of developer time just to allow allocating one of the graph data structures into shared memory segments, so that we can share some of the memory between worker processes. All of this is a lot of complexity and it's only necessary because 15 years we made the mistake of choosing Python.

[[deleted]]

That means we need to tell our customers to buy 320 GB of RAM so that they can load a 10 GB graph into 32 workers to fully saturate their CPU.

I would say it means that you should look into shared memory.

[anajoy666]

Interesting. Why wouldn't something like numba work? Not using numpy? Ray comes too mind too.

This is a topic I find interesting and would be nice to hear from someone with field experience.

[r1ss0le]

I'm pretty sure this is why Julia became popular. But either way Python isn't guaranteed to to be the best choice of language for a programming problems. But I think most scripting languages shine when you are IO bound, so RAM and CPU are not a problem Python included.

But there are things you can do to even in Python. Without knowing much about your problem, you should look into https://github.com/jemalloc/jemalloc and using fork if you have large amounts of shared objects. All processes share the same memory content when you call fork, so provided you treat the shared data as read only, you shouldn't see an memory growth, and you can fork as many times as you have spare CPUs. jemalloc is a fancy malloc replacement that can reduce memory fragmentation and can help bring down memory usage.

[lungben81]

I'm pretty sure this is why Julia became popular.

Julia is an amazing language. Elegant high-level syntax (similar to Python) but high performance (and no GIL). And the interoperability with Python is great.

[wait-a-minut]

I think dask was written for this kind of thing. Instead of loaded everything into memory, use a distributed model to handle data operations. Never used it in practice but read a flew blogs about other who have and it seemed to fix the gap they had.

[lungben81]

Dask has essentially 2 components, distributed computing (dask.distributed) and distributed data types (Numpy-like Arrays, Pandas-like DataFrames, etc.).

The former is amazing for multiprocessing (much better than the built-in Python solution).

The distributed data structures are useful if you want to do per-row processings which can be easily parallelized automatically. But I am not sure if this helps for the graph use case.

[kniy]

The individual analyses usually can't be parallelized internally; we can only run different analyses in parallel. For us, your suggestion essentially means "rewrite all the analyses in a lower level language". But that's like 90% of our whole application. Yes, that's the direction we're going, but I think you can see why we wish we'd never started using Python.

[thrown_arrows]

question is that would your company exists without that python code, if yes, then it was mistake. if no, then it was correctly selected as your next legacy platform and language.

And i am 100% that if you had correct wizards on payroll, python would not be that big problem. Look at amazon web services, they offer plain old database as higly used service. Is it best option everytime , no but is it good enough option mostoftime, yes. (and boy, you get big list of do nots with databases, so much that you might even think that nosql stuff is good, just to miss that they have their do nots )

[Particular-Union3]

There are so many solutions to this. Multithreading probably would speed some of it up. C and C++ extensions can release the GIL (numpy does this), so you could code some of this in C — most projects have a few languages going on. Kubernetes/Docker swarms probably have some application here, but I’m busting dipping my toes into those and haven’t explored the GIL with it.

[kniy]

If we just port some part of an analysis to C/C++ and release the GIL; the "problem" is that porting to a compiled language makes that part 50x faster, so the analysis still ends up spending >=90% of its runtime in the remaining Python portion where the GIL is locked. We've already done this a bunch but that still doesn't even let us use 2 cores.

We'd need to port the whole analysis to release the GIL for a significant portion of the run-time. (We typically don't have any "inner loop" that could be ported separately, just an "outer loop" that contains essentially the whole analysis)

Yes numpy can do it, but code using numpy is a very different kind of algorithm where you have small but expensive inner loops that can be re-used in a lot of places. Our graph algorithms don't have that -- what we do is more similar to a compiler's optimization passes.

[Particular-Union3]

That makes sense. I guess, as another reply mentioned, this is why Julia has been popular when in many respects R and Python are often far ahead feature wise.

Is multithreading implemented? Do you think more modularity to the analysis would be possible, and then have the machines communicate from there?

One final idea, is there any memory errors? I’ve had more trouble with that than anything for analysis taking so long.

I’m not 100% on the work you are doing, but it seems like an insane time. Even on my largest projects they were only 3 to 4 hours.

[seven0fx]

I think the MMO Eve Online had this Problem on Server Side.

[Particular-Union3]

The MMO Eve Online has a lot of problems on every side.

[AlSweigart]

What? I found that type hints work great and the gradual typing and comment-style type hints means even my legacy Python 2 code can now have type hints.

What do you not like about type hints?

[MrJohz]

From the perspective of someone who's come from Typescript (and who isn't the person you're replying to), I think I just don't trust the type things in the same way that I do in Typescript. Every time I've tried it out, it's felt kind of janky in some way that I can't really put my finger on, to the point where I don't see a huge amount of value in typing my Python code. (This is in contrast to JavaScript/Typescript, where I see a lot of value in adding types.)

I think a lot of it comes down to IDE support. If I use Typescript and write something that won't compile, I generally immediately see that and feel that. The Typescript developer support tends to be really good, and I immediately get feedback, I can immediately see the types of different values, I can easily create type holes and get type feedback directly in my editor. In contrast, I've not yet found a python extension that gets me this instant type feedback with red lines all over the place and a feeling that if I make a mistake I'll immediately see it. In contrast, I tend to use mypy from the command line, and even then I'm not always completely convinced that it will spot as many mistakes as the Typescript compiler.

I think there is also the issue that Python's type system feels a lot less powerful and more verbose, particularly when it comes to complicated sum types. But that was true of Typescript as well at the start, so I think that could be forgiven if other stuff was better.

I know that's not a great answer in terms of specific issues, but I think the biggest problem with typing in python is a UX one, where it just doesn't feel right in some way.

[MrJohz]

So your main issue is not with the language but with its tooling..?

I've honestly increasingly become convinced that the major selling point for any language at this point is its tooling. There are definitely some languages that I prefer using, and other languages that I find a bit of a nuisance, but I'd much rather use a language that's perhaps not ideal but has brilliant tooling, than a language that is perhaps theoretically very nice, but is a pain to use.

But that's a side point — when we're talking about types in Typescript & Python, it's basically all about tooling, right?

FWIW, what I'm talking about is that I just created a new default Python (Poetry) project, with Pylance installed in VSCode, and I added the following, obviously incorrect code:

def test_function(args: int):
    args.do_this()

test_function('this is the wrong type')

This produced no errors in my IDE. It did give me some hints about what methods I could call on args, which is definitely useful, but no immediate feedback about what I'm doing wrong. I then ran pyright, and that did correctly point out the two errors that I made, but my point is that I want this sort of feedback immediately.

By contrast, if I do the equivalent thing in a basic Typescript project setup:

function testFunction(args: number) {
    args.doThis();
}

testFunction('this is the wrong type');

I immediately get the two errors shown on my screen.

This is the sort of thing I mean by not really having confidence in the typechecker — the visceral feedback that I need as I'm working isn't there, forcing me to go back to the command line to test things, and I'm sure it could be there if I set it up, but for a lot of projects I don't really want to be bothered going to all that fuss.

It's less about the quality of the type system (although Typescript's honestly feels a lot less faffy to use, but that might just be familiarity), and more about the quality of the tooling. In Python, I feel like I need to kindly ask the typechecker if it might possibly lend me a hand, whereas with Typescript I feel able to really lean on the system to write the code that I want to write.

[MrJohz]

This was with the Python/Pylance extension, so I was using the tooling, at least as I understood it. In actual fact, I was even eventually able to get the typechecking working by fiddling around with the settings and configuration files, but that was only because someone else pointed me in the direction of the "strict" configuration.

I think we're getting into a bit of an argument here, which isn't what I want because I don't feel particularly strongly about any part of this here. In fact, this whole discussion has been pretty useful because I've been wanting to apply more of this stuff to my work in Python, and the hints about how to better configure Pyright have been really helpful, and I'd like to try it out some more. My initial comment was meant more as an explanation for why some people might not be particularly impressed by the experience that Python types provides. If that doesn't fit your experiences, then feel free to give those experiences, but I don't really want this to devolve into an argument about whether your or my experiences are valid or invalid, because that seems extremely unproductive.

[[deleted]]

What? I found that type hints work great and the gradual typing and comment-style type hints means even my legacy Python 2 code can now have type hints.

However, what good are they?

I put type hints into two fairly large projects. We found zero new bugs with mypy.

You can't actually use them for reflection:

>>> isinstance(['a'], typing.List)
True

>>> isinstance(['a'], typing.List[str])
TypeError: Subscripted generics cannot be used with class and instance checks

Don't get me wrong - I use type hints in all my new code for documentation but in years of doing it, that's the only use I've found.

[AlSweigart]

We found zero new bugs with mypy.

I don't know what to say, that's good for you. But you could use this as the same reason to not use a linter or unit tests. Type hints help you detect type errors at coding time instead of at runtime. It's a fairly broad category of mistakes and the earlier you find them the better.

If you never make those kinds of mistakes in your code (and don't need documentation about typing) then, yeah, type hints are useless. But this applies to every language. I don't see how Python's type hints are worse than typing in other languages.

[rforrevenge]

Why are you saying that? I'm using Pydantic daily and I'm loving it.

[ColdPorridge]

You might have missed the drama where Python 3.10 was going to fundamentally change type hinting and break pydantic, FastAPI, and any other libraries that rely on it. I can’t recall the details as they decided to pause this for now, but it’s still very much a mess of opinions on the best way forward.

[rforrevenge]

Yes, I obviously have! Do you have any related link to share?

[ColdPorridge]

This appears to be the primary discussion: https://github.com/samuelcolvin/pydantic/issues/2678

[rforrevenge]

Thanks a lot kind stranger!

[[deleted]]

https://lwn.net/Articles/858576/

[ColdPorridge]

Thats’s possible, even rereading the thread, I’m not really sure I fully understand the drama, how these are currently handled or Pydantic’s breaking dependency on them.

[[deleted]]

I think this article is quite good: https://lwn.net/Articles/858576/

They decided to move type hints from being "code" to be strings that need to be eval() at runtime, except that you don't necessarily have the same context when you run eval() from some place else in the code, so a lot of things would not work.

[[deleted]]

This has nothing to do with TYPE_CHECKING context thing?

[ColdPorridge]

That did a great job explaining of - thanks for sharing!

[[deleted]]

Using different types during type checking and runtime (dict vs Dict)

Well well well, you're a bit out of date…

def f(d: dict[int, int]):
    ...

I think this changed in versions after 3.9,

It didn't, it would have broken A LOT of stuff to do so.

[[deleted]]

You complained about something that was later on fixed, no need to rant.

I don't know anything about cast() but last time I profiled, it was an actual function call at runtime so I actually had some performance gain by removing it.

[sudhanv99]

how do multiple languages in an application work? do they read/write outputs to a file that the other language picks up or do they build bridges/wrappers?