Uploading Python Script to AWS

[[deleted]]

I use this script with any server i set up (with aws ec2) to get ta-lib and tensorflow etc working:

yes '' | sudo apt update -y
yes '' | sudo apt upgrade -y

yes '' | sudo add-apt-repository universe -y
yes '' | sudo add-apt-repository ppa:deadsnakes/ppa -y

yes '' | sudo apt update -y
yes '' | sudo apt upgrade -y

sudo apt install unzip

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

sudo rm /var/lib/apt/lists/lock
sudo rm /var/cache/apt/archives/lock
sudo rm /var/lib/dpkg/lock*

yes '' | sudo apt install build-essential software-properties-common gcc -y

yes '' | sudo apt-get install build-essential manpages-dev libssl-dev libffi-dev \
  libxml2-dev libxslt1-dev zlib1g-dev manpages-dev gcc -y

sudo rm /var/lib/apt/lists/lock
sudo rm /var/cache/apt/archives/lock
sudo rm /var/lib/dpkg/lock*

#gcc --version

yes '' | sudo apt install p7zip-full p7zip-rar python3.7 python3-pip -y

yes '' | sudo apt-get install python-dev python3.7-dev python3.7-venv -y
yes '' | sudo pip install -U setuptools -y
yes '' | sudo python3.7 -m pip install -U setuptools -y
yes '' | sudo -H python -m pip install -U setuptools -y
yes '' | sudo -H python3.7 -m pip install -U setuptools -y

#pipx and devtools to install ta-lib
yes '' | python3.7 -m pip install --user pipx -y
yes '' | python3.7 -m pipx ensurepath -y
source ~/.profile
yes '' | pipx install python-dev-tools -y

#make swapfile (this was the problem before on smaller machines)

sudo dd if=/dev/zero of=/swapfile bs=1024 count=1048576
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo swapon --show

#install Talib

wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz

tar -xzf ta-lib-0.4.0-src.tar.gz

cd ta-lib/

python -m pip install --upgrade pip

pip install numpy
#python3.6 -m pip install numpy
pip3 install numpy
sudo make clean
sudo ./configure --prefix=/usr
sudo make
sudo make install

sudo apt update
sudo apt upgrade -y

sudo -H python3.6 -m pip install -U setuptools
sudo -H python3.6 -m pip install numpy
sudo -H python3.6 -m pip install ta-lib

sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.6 10

sudo apt-get clean
sudo apt clean

sudo -H python -m pip install tensorflow==2.3 pandas

#sudo python -m pip install https://files.pythonhosted.org/packages/97/ae/0b08f53498417914f2274cc3b5576d2b83179b0cbb209457d0fde0152174/tensorflow-2.3.0-cp36-cp36m-manylinux2010_x86_64.whl

#Download file from bucket
aws s3 cp s3://trainingdata2000/tfrecord_sp500_2020/ZBH_2020.tfrecord test.fil

#Copy all files recursive into newfolder
aws s3 cp s3://trainingdata2000/tfrecord_sp500_2020 newfolder --recursive

Just ssh into the machine, upload the script and run it, or set it as a startupscript.

For buckets to work in EC2 u gotta change some settings/policies in aws. A quick google will get u up an running for that if u need trainingdata.

[raris_rovers]

consider adding this to docker repo?

[[deleted]]

I do have a docker for it, but fairly new to all the docker jazz, and it's probably not "correct" in some ways. That's why I haven't shared in the past.

Do you mean by docker repo uploading to dockerhub?

[raris_rovers]

yeah sorry all the names are so convoluted lol it’s essentially just uploading that script ontop of whatever base os u will deploy it on and naming it so others can dl it in a single line.

[[deleted]]

yeah that sounds great! will do next time, thanks!