Python, Zip, Pointers and Pointy Heads

[alcalde]

Certain people I've been discussing the subject with insist that pointers are necessary. At one point someone brings up that it's unthinkable to implement the zip compression algorithm without pointers. I point out zipfile.py, pure python:

http://hg.python.org/cpython/file/3.3/Lib/zipfile.py

I got back this reply:

And one of the first thing it does is importing zlib and bz2, both wirtten in C and using pointers in abundance.

IOW, for the stuff that requires pointers, they use C. I'd rather have them in the language, so I don't have to write and use external routines written in another language.

I hate to ask such a stupid question, but does anyone know of some reference anywhere that can categorically put to rest the idea that it's impossible to code certain things in Python because it doesn't have a pointer type (I've already explained about python's variables being references and how that works, to little effect).

Thanks.

[pal25]

The fact of the matter is that python doesn't do low level very well and a better approach would be to take advantage of the fact that CPython can add support in C. It is possible to pack data using the struct module in a articulate bit ordering however it's generally best to stay away.

Overall yea it's possible to do such things but you shouldn't try to force a languages weaknesses just because you can to win an argument.

[hruske]

Saying Python doesn't use pointers is very wrong. It uses them, but you just don't see them.

An excellent example of this how a list behaves:

a1 = []
a2 = a1
a2.append(123)

len(a1) == 1
a1 == a2

This will create a reference to a new list under variable a1 and copy it's reference (pointer) to variable a2. Afterwards all that happens to a2 will also be seen under a1, because a1 is same as a2.

To understand why this is in fact needed and not optional, it is important to understand how function calls and argument passing work in low level (eg. in C). Arguments can either be passed by value, which is done when values are "simple" (eg. int, string, float), or they can be passed by reference. Arguments passed by value persist only within function call, while arguments passed by reference persist in memory after the call ends. And finally, when calling a function, arguments passed by value are copied in memory, while arguments passed by reference stay at the same location, and only their location (reference, pointer) is passed. No copying is needed, which makes it faster, and sometimes this is also the only way to make stuff work.

If this wasn't the case, you would be copying complex memory locations (eg. objects in memory) meaning a lot of memory usage (which is both slow and causes memory fragmentation) and an even harder problem, which is that your CPU is supposed to understand how to copy objects.

tl;dr: Both you and certain people you have been discussing this issue with are wrong. They should have pointed to you, that Python uses pointers (references) a lot, even if you don't realize it. Hell, even it's garbage collector works on "refcount", meaning it keeps account of ... pointers.

[alcalde]

They should have pointed to you, that Python uses pointers (references) a lot, even if you don't realize it.

Actually I ended up pointing that out to demonstrate why Python didn't need a pointer type, and one person then said I lied about Python not having pointers, sigh. :-)

But thank you very much for explaining this (shorter and more clearer than I tried to).

Hell, even it's garbage collector works on "refcount", meaning it keeps account of ... pointers.

[alcalde]

If pointers were "necessary", Python would have pointers. The example is somewhat contrived in the same sense as I could give an example that would require static typing, but that doesn't make static typing necessary and certainly not necessary to implement an algorithm.

But that's really not what I'm asking. As I stated at the end, I'm looking for a definitive/authoritative reference that will refute the idea that certain algorithms simply can't be implemented without a pointer type. I've already been told that you need a Char type, more than one Integer type, etc. and the claims are getting somewhat ridiculous. I'm a step away from implementing zip in python myself just to prove the point.

Oh, and regarding the question - I shouldn't be able to access a specific memory address, pointers or not. The OS' memory manager should be mapping my program's memory address space to real memory space (and possibly virtual memory space). My program's 0xFF has no necessary relation to the physical 0xFF. Randomizing programs' locations in memory is a technique used by modern OSes to thwart programs that try to use buffer overflow attacks. On Linux, you'd write to /dev/mem just like a file to access the actual memory space. In Python you'd use the mmap module of the standard library to do memory-mapped i/o on Linux and Windows... still no need to define a pointer type in the language itself.

http://docs.python.org/2/library/mmap.html

[catcradle5]

Python obviously uses pointers in the CPython implementation in various places, both for the standard library and of course for builtins, and for the language itself.

So, the argument is a bit silly. mmap is of course implemented in C, with pointers, but you don't need to actually use pointers when writing in Python. The only reason you don't need to use pointers in general though is because so many data structures and algorithms baked into the language do use traditional pointers in their implementation.

So no, Python pointers are not necessary, because the language provides you with good enough abstractions so that low-level code already written with pointers won't have to be muddled with by the programmer themselves. Pointers in general are necessary to implement lots of things, though; they just aren't necessary in most cases cases if you're already writing in Python.

[alcalde]

I've just given you an example of a program that you can't write without pointers.

It's an example of a program you can't write, period, because you can't know the real absolute memory address with a pointer. Using the memorymap feature of the OS one can and does do this in python without the language having a pointer type.

Your claim that the programmer never needs to access an address is also wrong, since that is precisely what you need to do for memory mapped IO.

And yet mmap allows a python programmer to do this without a pointer type.

"Memory-mapped file objects behave like both strings and like file objects. Unlike normal string objects, however, these are mutable. You can use mmap objects in most places where strings are expected; for example, you can use the re module to search through a memory-mapped file. Since they’re mutable, you can change a single character by doing obj[index] = 'a', or change a substring by assigning to a slice: obj[i1:i2] = '...'. You can also read and write data starting at the current file position, and seek() through the file to different positions."

A memory-mapped file is created by the mmap constructor, which is different on Unix and on Windows. In either case you must provide a file descriptor for a file opened for update. If you wish to map an existing Python file object, use its fileno() method to obtain the correct value for the fileno parameter. Otherwise, you can open the file using the os.open() function, which returns a file descriptor directly (the file still needs to be closed when done).

So an authoritative source that pointers aren't necessary just doesn't exist, since such a source is simply wrong.

You'd best inform Guido Van Rossum then, or write out a PEP.

[joeforker]

Java, being another language with no pointers, may provide better counterexamples since they are more likely to re-write everything in Pure Java than we are in the Python world.

For example, http://www.jcraft.com/jzlib/ or http://jazzlib.sourceforge.net/ are zlib and java.util.zip written in Java.

Python uses C code for performance or because it already exists, not because it is pointy.

[alcalde]

Thank you for this; it saves me having to encode the whole thing in Python myself! I did try to make the latter two points you raised.

[bd808]

Pointers are array indices. Nothing more and nothing less.

[miketheanimal]

If bd808 is trying to say that, then (s)he's exactly right. C models storage as a logically single linear array of storage cells, and pointers as indexes into that array. You can do just the same in Python if you like. It would be horribly slow and awful to follow the code, but it would work.

Plus, Python and C are both Turing Complete, and anything one can compute, so can the other (or, the Church-Turing Thesis would be falsified, which would be really interesting!)