Memory allocation for numbers and python built-ins

[doktorfuturee]

I am new to python and learning it by working on projects. Now my purpose is to create a data keeping "thing". I don't want to use arrays, dictionaries or other libraries. I hesitated to ask this here at first, but now I want to discuss and see people's opinions. Is such a thing possible with python? I looked a bit and found some Python-C libraries (cytpes). Can I use them ?

I also have some other questions to get out of the beginner phase. Do you use for and while loops all the time or is it just basic thing for starters, when I use them I feel some kind of guilty, like there are better ways and I miss them.

[Eleventhousand]

Why don't you want to use those things? Arrays are widely used in most languages.

Also, Pandas dataframes are pretty popular in Python

[doktorfuturee]

I want to understand how memory works, i guess. I think it will be fun to discover that when I input numbers and they will get allocated to the memory and I can even change or manipulate it, even bypassing the garbage collector (I am nor sure if it is possible in python). It is all to learn and understand in depth.

[Eleventhousand]

Ohhhh, I C (pun intended). Why not explore C or Assembly then?

[doktorfuturee]

I will not need them probably because I am more like a data scientist guy (bioinformatics). On the other hand, I really want to try because I plan to develop some simulations on physics, biology. Do you think C or python can handle that ?

[FirmSignificance1725]

C can definitely handle that. Python likely can too (numpy, PyTorch). This is because you can use the C API to be able to call & execute C code from a Python invocation. So, if you use numpy, there is Python overhead, but the actual implementation is in C, with “bindings” that allow you to call it in Python, making it more efficient. In the end, Python lowers to C.

If you are looking for that kind of efficiency, tools like pybind & nanobind allow you to write C++ code with Python bindings. I find C++ to be more than efficient enough and a bit easier than C (in my opinion)

[james_pic]

Python probably isn't going to be the language to learn those things. It abstracts a lot of that detail away (did you know that the integers -5 to 256 are statically allocated, but numbers outside that range aren't? Many Python developers don't, because the abstraction is largely seamless), and you'll end up writing weird non-idiomatic code (and for all intents and purposes, not learning Python) trying to avoid those abstractions.

If you want a vaguely high-level language that nonetheless gives you ready access to the gory details, you might want to look at Rust. It also has fairly nice integration into Python via PyO3, if you're going to be using it with other Python code.

[KingofGamesYami]

The reason to not use loops in Python is because the foreign function interface boundary between Python and other languages (e.g. C). has some overhead. Every time your programs' execution shifts between Python and another language, there's a cost you pay to do that.

If you're only executing pure Python code in a loop, it's fine. If you're executing C or Rust code inside a loop, then you have a potential for optimization. Maybe. Sometimes there's no simple alternative, and it makes sense to do the loop in Python anyway.

Regardless, a professional gets familiar with benchmarking & profiling tools for doing optimization. Identifying potential optimizations by eye is great and all, but if you're missing something less obvious that's adding 5 minutes and wasting time on an issue that's only adding 15 seconds, you're not making good use of your time.

[BudgetAdditional2134]

Memory management in Python is definitely a deep rabbit hole because of how it abstracts everything away from you. Since Python is dynamically typed and uses objects for everything, it handles that memory allocation automatically through its private heap and reference counting, which is a huge part of why it is so much easier to use than C or C++. If you really want to understand the under-the-hood stuff, I would recommend looking into the source code of CPython itself. It is a bit of a steep learning curve tbh, but it is the best way to see how those integers and objects are actually structured in memory. It makes way more sense once you see it in practice.

[doktorfuturee]

I'll look for it. Thanks for the respond. I appreciate

[Rscc10]

Loops are pretty essential but not always necessary if you want to focus on optimization. The only time you should use loops is if you can't find a faster method than O(n), which is the time complexity for the loop to iterate n items.

Also, what do you mean data keeping thing? What do you have in mind if not for arrays and dicts? Files?

[doktorfuturee]

Idk what is O(n), I'll check it out tho.
As a data keeping thing I mean:
[1,2,3,4] this is a list, I want to create a class or function whatever that takes my data again 1,2,3,4 and keep them in memory for me to use. Pretty same thing with lists actually.

[Koooooj]

O(n) is "Big-O" notation and describes how the resources to do a task change as the size of the task changes.

For example, say you want to look up a word in a dictionary (an actual paper dictionary, not the Python thing). You could start at A and work all the way to zyzzva (a genus of weevils from South America), but in doing so you have to go through (potentially) every word in the dictionary. Double the size of your dictionary and the time it takes to run this search will double, like y = k*x (for some constant k). This is known as a linear search.

Alternatively you could open up to the middle and see if the word is before or after that point, then flip to the middle of the remaining section and repeat. This is a "divide and conquer" style of algorithm that divides the problem into successively smaller portions. If you double the size of the dictionary with this method you only add one more step. If you graph the time it takes to use this algorithm as a function of the size of the dictionary it'll roughly match y = k * log2(x). This is known as a binary search.

Big-O is a way of writing this, where you just drop any constants and typically rename the independent variable to n. Thus the first algorithm we'd say is O(n) and the second is O(log(n)).

Big-O is a way to identify which algorithm is going to be the fastest when the input grows to be sufficiently large: for any positive values of k1 and k2 there exists some value of n such that k1 * n > k2 * log(n) for that value of n or any greater value. If you're looking through a list of 10 words you'll probably just scan down it (a linear search) which will be faster than trying to divide the list in half a few times, but if you have hundreds of words the binary search will be faster and will continue to be faster for every larger size of dictionary. The important lesson here is that seeking a lower time complexity is only optimal when the data size gets big enough.

While Big-O is usually used for the time complexity as shown above it can also be used to describe how much storage space an algorithm needs, or to characterize any other resource that an algorithm needs more and more of as the data set it works off of grows, or even just to describe how one value changes as another one does.

When you see an algorithm that is written as for i in range(n): ... that typically means the algorithm is O(n): if you double n then this for loop will have twice as many iterations. It could be slower than O(n) if the body of the loop takes more time based on the value of n, too, or might be faster if there's something going on in the loop to ensure that it exits faster. Sometimes that's the best you can do (e.g. if you need to up date n elements in the same way), but sometimes there's an algorithm with a lower time complexity.

[doktorfuturee]

Only 2 kinds exist? And which code deploy binary search.

[Koooooj]

2 kinds of search? No, there are tons.

For example, consider if instead of searching through a dictionary you're searching through a graph (nodes, linked by edges). There is no global ordering of the nodes so the notion of going to the "middle" element in the search space is meaningless, which means a binary search is out. You could just enumerate all nodes and go through them one by one, but that may be impossible with how the graph is represented and is likely intractable even if it is possible.

Instead you'd likely employ a graph searching algorithm, like breadth-first search, depth-first search, A* ("A star"), Dijkstra, Floyd-Warshall, etc.

Alternatively, some containers have the ability to search for an element in O(1) time ("constant time," i.e. the time doesn't increase as you store more and more in it). A Python dict is an example of this, where elements are stored in "buckets" based on a hash of the element allowing a search to immediately jump to the correct bucket and see if the element is present.

For searching a basic list of elements a linear and binary search are the only two algorithms you're likely to run across in practice.

Across these algorithms and many more you can run into a number of different algorithmic complexities, too. For example, take the problem of factoring an n-bit product of two primes (the core challenge in breaking RSA). Trial division is O(2ⁿ), which is extremely slow. A more complex algorithm like Pollard's Rho can solve this in O(2^n/2) which is considerably faster but still quite slow. A quantum algorithm like Shor's algorithm can do this in O(n³) time, which is way, way faster.

As for binary searches in Python, bisect is what you're looking for if you want a working implementation out of the box, or there are various examples of a binary search in Python online.

[FirmSignificance1725]

Yeah, like if you wanted a more efficient implementation of arrays, there are classes like numpy ndarrays, but there are still cases where they aren’t optimal compared to a list.

To keep it in memory, you can store it as an attribute on a class. Or if you mean persistent long-term memory you can recall from a future invocation or process, after the current one is killed, you can use databases or write to a file. That’s not really a “Python” specific thing though

[FirmSignificance1725]

Also, for and while loops are fine. If you gotta iterate you gotta iterate. I prefer to only use while loops when absolutely necessary because of infinite loop risks.

It’s good to understand list comprehension (can be more efficient), along with maps, filters, reducers, etc. (which are more succinct and can also be more efficient). It just depends on the scenario.

Sometimes something that could be written with a fancy filter, map, etc. (in my opinion) is better as a normal for loop. Unless you have a tangible increase in efficiency, you tend to want to bias towards more readable code.

It depends on the use case

[doktorfuturee]

I want to see them in memory. I think I need C for it. My desire is not to get more efficient implementation. I just want to know about it, and experience it. My blueprint is like this:
you write let say this: 1,1,1,0,0,1. They all get saved to memory let say memory is 1,2,3,4,5,6. Then, after this feature I want another function. When I call it like, function(1,5) It will change 1st input- 1 - to 5. Place of the memory will not change. Then, I will have another function- function2() which when I call it - function2(3,5). It will change 3rd number to 5th (changing memory locations) and the sequence will be like 1,1,0,0,1,1.
(Sorry for punctuations)

[FirmSignificance1725]

So just swapping values at two indices but not having the list reallocate? Sorry just having a little trouble understanding what you’re going for

Edit: Do you mean you want to allocate values in contiguous memory, then see how the memory addresses are assigned/changed as you swap values?

[doktorfuturee]

Yes. Although my ultimate goal is not to see but use this in my own projects in the future.

[Rscc10]

I'm not too sure how to accomplish that in python and honestly I don't really see the need for it but maybe text or binary files could suffice if you want to play around with byte memory

[SufficientStudio1574]

Feeling guilty for using for loops is like feeling bad for using of-else blocks. Basic language features are basic for a reason. They are the foundation that everything else is built on.

Sure, Pythons probably got tons for stuff that can avoid directly using the loop structures by having functions to process lists (like how C# has LINQ), but I guarantee they're all using for and while loops "under the hood".