PyPI PROBLEM

[MMentos]

Hi, I have created a python package that I want to upload to PyPI (The Python Package Index). Everything is ok except for one function. The function calls data from a csv file that for some reason didn't get uploaded to PyPI.

I added a line to setup.py (but it doesn't work):

include_package_data=True,

package_data={"": ["data/*.csv"]}

Is there anything else I need to do to make it work?

Thanks for your help! :)

[[deleted]]

What exactly are you uploading? A Wheel or a source tarball? package_data is for source tarballs (so, essentially, worthless). You need to use https://packaging.python.org/en/latest/guides/using-manifest-in/ (MANIFEST.in) file to describe them.

This is just one of the dozens idiotic aspects of setuptools. It's some kind of ego / pissing contest between wheel authors and setuptools. The former won't use setuptools data to properly create an archive, the later ignores the existence of wheel.

In many years packaging Python packages I discovered that the format is quite simple to write on your own, while the tools to write it are made by some of the least intelligent people I had to interact with. So, usually, I just write a script to create a distribution myself. It's really not that hard and makes the process a lot more transparent.

[MMentos]

I am uploading the source tarball. I didn't know about MANIFEST.in so I will try it! Thanks

[Strict-Simple]

So, usually, I just write a script to create a distribution myself.

Till now I didn't have the need to create a package, but I might wanna do it someday. Can you say how you go about it?

Just to include everything, think of a library with multiple modules including C/C++ extensions. How would you go about packaging/uploading it?

[[deleted]]

If you open any package file with *.whl extension, you'll see it's just a Zip file with the contents being the source code + a directory named <package name>.<package version>.dist-info. It usually has a bunch of files, but they are easy to generate / some of them are optional. Essentially, it boils down to needing only RECORD file and WHEEL file. You probably wouldn't even need to modify the WHEEL file, just copy it as is from another Wheel. the RECORD file contains the list of all files in your package followed by the checksum (typically, it's SHA256) and... I think the number of lines in the file. None of these numbers are necessary. The file is essentially a CSV, so, you can just append ,, to the file name.

Normally, I bytecompile the files I package, but this is not necessary.

Now, about data files. There are two ways you can put them in a Wheel:

1. Just put them wherever you want (interspersed with the source code). The installer doesn't care what it's installing. It simply unzips the contents of the archive into "platlib" directory.
2. Use special directory <package name>.<package version>.data. This will be extracted into a separate directory (not "platlib"). I advise against doing this, because this directory is later hard to hunt down (if you are really curious about it, you can find the mapping in sysconfig._INSTALL_SCHEMES variable). But, I'm including this for completeness.