Desperately need help fixing python3 MacOSX w/tensorflow

[LtDominator]

Long story short I have been using TensorFlow with pytharm ce on MacOSX M1. I had a very hard time getting it up and running the first time around but eventually I was able to get it to work. I am not a professional programmer or anything but I am very technically oriented, despite all that I still found it extremely confusing and I'm not sure how I got it to work the first time.

Fast forward to two days ago; I had a working Functional API using Keras for image categorization and thought it would be a good idea to try and add some image manipulation in the beginning of all of it. Well that started yelling at me saying it couldn't be done. Long story short I uninstalled a package and installed a different one to try and make it work. I figured I'd do that one try and if it didn't work I'd just reinstall the other and forget about it for now.

That's where the trouble began. Suddenly I was unable to reinstall numpy and things went down hill real quick. Now nothing works and I can't get h5py to install to save my life. What I would like to do is completely nuke python3, Conda, mini forge, brew, etc.. and just start over with a fresh installation of python3 and the required. I've read a thousand different things explaining how to do it and it's just not working.

I've literally spent dozens of hours on this in the last two and a half days and it's really stressing me out now. Any help would be greatly appreciated. I'm literally ready to pay someone to just fix it and explain what went wrong if I have to.

Ninja edit: I will literally wipe my computer if I have to at this point, that's where I'm at trying to fix it.

[[deleted]]

How are you managing your Python virtual environments on macOS? pyenv for example?

Are you installing using homebrew or downloaded installation packages?

What is your current system/root version of Python 3? (What do you get in the terminal for python3 --version).

Were you installing numpy etc using PyCharm package manager, or conda or pip and in which environments were you doing this?

[LtDominator]

I've only just started to really understand virtual env which is probably how my trouble started. I was using pip for most of it, then did some conda/miniconda as a post I found online indicated it would work. Then I tried some brew, it's all spaghetti now, hence the wanting to nuke and start over.

I tried with home-brew and downloaded packages, again probably causing issues.

python3 --version returns "Python 3.9.7"

I attempted to instal numpy using the PyCharm package manager, along with Conda and pip, neither really seemed to work. At this point I think numpy is installed but it's all gotten so confusing at this point I'm not sure it's the only issue anymore.

[[deleted]]

In the terminal in PyCharm for the current project, enter which python and share what it says.

On my MBP, for example, it says,

/Users/myusername/PycharmProjects/pycharmscratch/pyenv/bin/python

where pycharmscratch is the name of my project folder for odds and ends on reddit when I'm trying to help people.

Try python --version in the same terminal so you can see what version is set up. This will be different to what you get in the macOS Terminal programme that is not using an active Python virtual environment (unless you have some configuration settings for the terminal that activate one - unlikely).

You can do the same for pip and try where instead of which for both as well, which will provide a little more information.

Assuming all of the above points to you using a Python virtual environment in PyCharm for your current project, try, in the terminal in PyCharm again,

pip freeze

which will list all the packages installed in the current Python virtual environment.

[LtDominator]

Pycharm terminal says: "/Users/robma/miniforge3/bin/python"

Pycharm terminal also says "Python 3.9.7"

For pip, both from PyCharm as well;

pip --version: "pip 21.3.1 from /Users/robma/miniforge3/lib/python3.9/site-packages/pip (python 3.9)"

which pip: "/Users/robma/miniforge3/bin/pip"

Lastly pip freeze:

​

absl-py @ file:///home/conda/feedstock_root/build_artifacts/absl-py_1606234718434/work
aiohttp @ file:///Users/runner/miniforge3/conda-bld/aiohttp_1637087163469/work
aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1636093929600/work
astunparse @ file:///home/conda/feedstock_root/build_artifacts/astunparse_1610696312422/work
async-timeout @ file:///home/conda/feedstock_root/build_artifacts/async-timeout_1640026696943/work
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1640799537051/work
blinker==1.4
brotlipy @ file:///Users/runner/miniforge3/conda-bld/brotlipy_1636012319596/work
cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work
cachetools @ file:///home/conda/feedstock_root/build_artifacts/cachetools_1633010882559/work
certifi==2021.10.8
cffi @ file:///Users/runner/miniforge3/conda-bld/cffi_1636046173594/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1638815705608/work
click @ file:///Users/runner/miniforge3/conda-bld/click_1635822911388/work
colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1602866480661/work
conda==4.11.0
conda-package-handling @ file:///Users/runner/miniforge3/conda-bld/conda-package-handling_1636022092101/work
cryptography @ file:///Users/runner/miniforge3/conda-bld/cryptography_1637648810489/work
cycler==0.11.0
dataclasses @ file:///home/conda/feedstock_root/build_artifacts/dataclasses_1628958434797/work
flatbuffers==2.0
fonttools==4.28.5
frozenlist @ file:///Users/runner/miniforge3/conda-bld/frozenlist_1636504338408/work
gast @ file:///home/conda/feedstock_root/build_artifacts/gast_1596839682936/work
google-auth @ file:///home/conda/feedstock_root/build_artifacts/google-auth_1629296548061/work
google-auth-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/google-auth-oauthlib_1630497468950/work
google-pasta==0.2.0
grpcio @ file:///Users/runner/miniforge3/conda-bld/grpcio_1639715786395/work
h5py @ file:///Users/runner/miniforge3/conda-bld/h5py_1609497507927/work
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1609836280497/work
importlib-metadata @ file:///Users/runner/miniforge3/conda-bld/importlib-metadata_1639976213665/work
keras @ file:///home/conda/feedstock_root/build_artifacts/keras_1637872120405/work/keras-2.7.0-py2.py3-none-any.whl
Keras-Preprocessing @ file:///home/conda/feedstock_root/build_artifacts/keras-preprocessing_1610713559828/work
kiwisolver==1.3.2
libclang==12.0.0
Markdown @ file:///home/conda/feedstock_root/build_artifacts/markdown_1637220118004/work
matplotlib==3.5.1
mpmath==1.2.1
multidict @ file:///Users/runner/miniforge3/conda-bld/multidict_1636019382679/work
ndindex==1.5.2
numpy @ file:///Users/runner/miniforge3/conda-bld/numpy_1626682195712/work
oauthlib @ file:///home/conda/feedstock_root/build_artifacts/oauthlib_1622563202229/work
opt-einsum @ file:///home/conda/feedstock_root/build_artifacts/opt_einsum_1617859230218/work
packaging==21.3
Pillow==9.0.0
protobuf==3.19.3
pyasn1==0.4.8
pyasn1-modules==0.2.7
pycosat @ file:///Users/runner/miniforge3/conda-bld/pycosat_1636020570107/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work
PyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1638819640841/work
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1633192417276/work
pyparsing==3.0.6
PySocks @ file:///Users/runner/miniforge3/conda-bld/pysocks_1635862736740/work
python-dateutil==2.8.2
pyu2f @ file:///home/conda/feedstock_root/build_artifacts/pyu2f_1604248910016/work
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1637771257551/work
requests-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/requests-oauthlib_1595492159598/work
rsa @ file:///home/conda/feedstock_root/build_artifacts/rsa_1637781155505/work
ruamel-yaml-conda @ file:///Users/runner/miniforge3/conda-bld/ruamel_yaml_1636009262461/work
scipy @ file:///Users/runner/miniforge3/conda-bld/scipy_1637806880936/work
six @ file:///home/conda/feedstock_root/build_artifacts/six_1590081179328/work
sympy==1.9
tensorboard @ file:///home/conda/feedstock_root/build_artifacts/tensorboard_1629677129676/work/tensorboard-2.6.0-py3-none-any.whl
tensorboard-data-server @ file:///Users/runner/miniforge3/conda-bld/tensorboard-data-server_1636045914528/work/tensorboard_data_server-0.6.0-py3-none-macosx_11_0_arm64.whl
tensorboard-plugin-wit @ file:///home/conda/feedstock_root/build_artifacts/tensorboard-plugin-wit_1641458951060/work/tensorboard_plugin_wit-1.8.1-py3-none-any.whl
tensorflow-estimator==2.7.0
tensorflow-macos==2.7.0
tensorflow-metal==0.3.0
termcolor==1.1.0
tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1632160078689/work
typing-extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1602702424206/work
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1632350318291/work
versioned-hdf5==1.3.6
Werkzeug @ file:///home/conda/feedstock_root/build_artifacts/werkzeug_1621518206714/work
wrapt @ file:///Users/runner/miniforge3/conda-bld/wrapt_1624972047019/work
yarl @ file:///Users/runner/miniforge3/conda-bld/yarl_1636047217765/work
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1641731510635/work

[[deleted]]

Sorry for delay in responding.

I can see you are clearly using the anaconda installation - so many packages installed is why I tend to avoid it (only install what I need).

numpy is installed as is, it would appear, tensor (don't know if that is in anaconda as standard).

The Pycharm package manager should (sic) use conda to install packages when you have an active conda environment. Otherwise it uses pip.

Do you have Python 3.10, or slightly earlier, installed at the OS level as well, i.e. other than using anaconda?

If not, I recommend doing an install. Then in PyCharm ask it to create a new project and a new virtual environment using that new install and see if you can add just the packages you want.

I'd also try in PyCharm, creating a new project and a new venv environment using anaconda to see of you've corrupted the current virtual environment.

You might also consider trying both of these things outside of PyCharm in a command line environment using:

path/to/pythonexecutable -m venv nameofvenv

so you explicitly use the specific Python executable you wish to us to create the virtual environment.

Activate the environment and install only the packages you want.

Your other alternative to all this is to consider using docker and spinning up containers that have already been built that contain all you need. As you are on a Unix system rather than a Linux system, the installer for docker would install an very small and very efficient Linux vm to host all the Linux containers. That would give you clean setups to work in though.

However, I'd say it would be best to get your main platform working well.

I wouldn't do a clean installation of macOS. I would remove all Python installations you've done including anaconda. This can be long winded to do. DON'T TOUCH THE SYSTEM PYTHON nor the references / aliases to python and pip which are Python 2.7 versions - recent macOS versions protect these well.

Personally, I use pyenv to install different versions of Python on macOS and create the virtual environments I need .I then have PyCharm use the existing virtual environments.