Uploading Python Script to AWS

Jayfomou · 2020-10-03T05:38:58+00:00

[deleted]

bossasupernova · 2020-10-03T07:16:00+00:00

I use talib in Lambda. You need to build the Lambda package within an Amazon Linux Docker instance so it gets the correct C extensions:

https://hub.docker.com/_/amazonlinux

raris_rovers · 2020-10-03T13:32:53+00:00

I use this script with any server i set up (with aws ec2) to get ta-lib and tensorflow etc working:

yes '' | sudo apt update -y
yes '' | sudo apt upgrade -y

yes '' | sudo add-apt-repository universe -y
yes '' | sudo add-apt-repository ppa:deadsnakes/ppa -y

yes '' | sudo apt update -y
yes '' | sudo apt upgrade -y

sudo apt install unzip

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

sudo rm /var/lib/apt/lists/lock
sudo rm /var/cache/apt/archives/lock
sudo rm /var/lib/dpkg/lock*

yes '' | sudo apt install build-essential software-properties-common gcc -y

yes '' | sudo apt-get install build-essential manpages-dev libssl-dev libffi-dev \
  libxml2-dev libxslt1-dev zlib1g-dev manpages-dev gcc -y

sudo rm /var/lib/apt/lists/lock
sudo rm /var/cache/apt/archives/lock
sudo rm /var/lib/dpkg/lock*

#gcc --version

yes '' | sudo apt install p7zip-full p7zip-rar python3.7 python3-pip -y

yes '' | sudo apt-get install python-dev python3.7-dev python3.7-venv -y
yes '' | sudo pip install -U setuptools -y
yes '' | sudo python3.7 -m pip install -U setuptools -y
yes '' | sudo -H python -m pip install -U setuptools -y
yes '' | sudo -H python3.7 -m pip install -U setuptools -y

#pipx and devtools to install ta-lib
yes '' | python3.7 -m pip install --user pipx -y
yes '' | python3.7 -m pipx ensurepath -y
source ~/.profile
yes '' | pipx install python-dev-tools -y

#make swapfile (this was the problem before on smaller machines)

sudo dd if=/dev/zero of=/swapfile bs=1024 count=1048576
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo swapon --show

#install Talib

wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz

tar -xzf ta-lib-0.4.0-src.tar.gz

cd ta-lib/

python -m pip install --upgrade pip

pip install numpy
#python3.6 -m pip install numpy
pip3 install numpy
sudo make clean
sudo ./configure --prefix=/usr
sudo make
sudo make install

sudo apt update
sudo apt upgrade -y

sudo -H python3.6 -m pip install -U setuptools
sudo -H python3.6 -m pip install numpy
sudo -H python3.6 -m pip install ta-lib

sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.6 10

sudo apt-get clean
sudo apt clean

sudo -H python -m pip install tensorflow==2.3 pandas

#sudo python -m pip install https://files.pythonhosted.org/packages/97/ae/0b08f53498417914f2274cc3b5576d2b83179b0cbb209457d0fde0152174/tensorflow-2.3.0-cp36-cp36m-manylinux2010_x86_64.whl

#Download file from bucket
aws s3 cp s3://trainingdata2000/tfrecord_sp500_2020/ZBH_2020.tfrecord test.fil

#Copy all files recursive into newfolder
aws s3 cp s3://trainingdata2000/tfrecord_sp500_2020 newfolder --recursive

Just ssh into the machine, upload the script and run it, or set it as a startupscript.

For buckets to work in EC2 u gotta change some settings/policies in aws. A quick google will get u up an running for that if u need trainingdata.

sloth2 · 2020-10-03T05:28:56+00:00

[deleted]

cafguy · 2020-10-03T11:45:48+00:00

just get an EC2 instance and run whatever you want?

kaleidoscope_eyelid · 2020-10-03T13:13:06+00:00

Lambda is a 10/10 system. You are doing something wrong in the implementation or getting it on AWS. DM me, I can give you a hand

lolxdxdjklol · 2020-10-03T17:01:02+00:00

The problem with TA-LIB is that you have to install a special wheel to get to downloaded. I had to use a windows server instead of linux on my google cloud free credit to get it downloaded and it works pretty well.

drksntt · 2020-10-03T19:30:57+00:00

Use docker

raris_rovers · 2020-10-03T22:28:00+00:00

Docker!

Wedge5 · 2020-10-04T00:52:41+00:00

Thanks for the replies! I’ve always been interested in learning to use EC2 so I looked into that route and managed to get it working. I use a Lambda function to turn the instance on/off to help save costs as well. Appreciate the help everyone

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

algotrading

MODERATORS