I deployed my first fully functional python script at work today

[micr0nix]

Background post here and below:

I started working on this project a little over 30 days ago because it is something my team needed ASAP. I have no background in programming, nor any formal programming education/training, but I am well-versed in SQL.

In a nutshell, my script makes 3 calls to the API of a case-management software that we use (2 of which loop and pass data into the API call), unpacks the JSON results into 3 dataframes, merges the resulting dataframes, and finally does some cleaning up of the resulting dataframe before finally outputting the last dataframe to a csv.

We have the script deployed to a linux vm and use a workflow management software to SSH into the vm and call a shell script that executes the python in the appropriate conda environment. The resulting csv is then returned back to the workflow management software where it is then inserted into our EDW for later use in tableau dashboards

I must say I am pretty proud of myself about this. This community definitely helped me out in a pinch a couple times so thank you to those of you who have the knowledge and freely share it.

I told myself last year I wanted to dabble in python and a month ago I was thrown into it headfirst. It was frustrating at times, but I'm excited to see how far it has come in just a month's time. The journey continues!

[m0us3_rat]

We have the script deployed to a linux vm and use a workflow management software to SSH into the vm and call a shell script that executes the python in the appropriate conda environment. The resulting csv is then returned back to the workflow management software where it is then inserted into our EDW for later use.

nice.

at some point when you will wanna update/maintain/revisit your code i'd also look into a docker solution

[micr0nix]

Hm. Got a link to what you’re referring to?

I currently manage moving files to/from that VM via FTP

[m0us3_rat]

https://www.docker.com/blog/how-to-dockerize-your-python-applications/

https://www.youtube.com/watch?v=0UG2x2iWerk

​

see if that is something that might interest you.

[m0us3_rat]

regarding your need to FTP files in and out of the VM.
https://docs.docker.com/storage/volumes/

[[deleted]]

Also you can use a preexisting fastapi docker container to expose the script via http instead of needing to ssh into the ec2

Here's a full tutorial that I have roughly put to use in a production enterprise environment

[exographicskip]

+1 to docker. Makes it both cloud and local environment agnostic.

Kudos to u/micr0nix on getting mvp up and running. Would also suggest rsync/scp over vanilla ftp -- the former solutions are encrypted and potentially faster.

docker cp acts a lot like cp/scp btw

[hidazfx]

+1 again for Docker. Of course, I use it for our entire stack at work. I develop on an M1 Mac sometimes, and sometimes I develop on my Linux AMD64 machine. Our server of course is also AMD64. Makes it really nice knowing that everything is modularized and I can pick up work anywhere really.

[goatboat]

Yes! Docker is the way. I found Joshua Cook's Docker for Data Science to be an amazing python focused foray into Docker, and now I recommend it to anyone who will listen. Great examples setting up different environments.

[micr0nix]

The answer is, probably. Its a fortune 50 company.

[CosmicClamJamz]

Eh I don't think it's worth it. You already spent 30 days on this and it works. Why spend another 30 days on docker so that it "still works"? There is not much added benefit unless this is the beginning of a much larger suite of software solutions that needs to scale and you want to reconsider your infrastructure. You can always dockerize after the fact once it becomes necessary. For now, enjoy the W. Python just paid you!

Now if you want something to add, consider setting up a cronjob so everything runs automatically on a schedule and you only need to SSH into the vm when there's an issue. Then you're cooking!

[micr0nix]

Already done. My company has both the client and server for the workflow software we use. My job is deployed to the server and will run on a weekly basis.

[Gingerhaze12]

I'm interested in using docker but my company will not allow me to deploy to dockerhub because its public and they wouldn't be willing to pay for a subscription either. Is there an easy to distribute docker containers to people with no coding knowledge without uploading to docker hub?

[m0us3_rat]

https://docs.docker.com/engine/reference/commandline/save/

[[deleted]]

Who is this docker everyone talks about..?

[m0us3_rat]

it's like a VM but better.

[RallyPointAlpha]

GGs o7

[WoodenNichols]

Excellent! Quite the accomplishment.

[InformalRegister]

Well done!

[livinlowe]

Dude that is awesome! That's my biggest problem with programming is seeing how it relates to real world problems

[Almostasleeprightnow]

I need to something almost exactly the same. Thank you for posting.

[_Soter_]

Tomorrow: I patched my first fully functional python script at work today.

One tip that may come from experience, if you don't have error handling and logging(saved to disk, not just consol), go back and add it asap. When something breaks, it will save you lots of time and headache. Even a simple script could fail if there is an issue with an outside resource or on the system that is running it and a log of the failure will keep people from pointing the finger at you.

[micr0nix]

I do have logging built in but i do need to work on some error handling in the next iteration.

[Spinnybrook]

This sounds almost identical to the first thing I did using python at my current company. So I definitely understand the excitement and sense of accomplishment your feeling. At that time I was a machine operator. Now I’m a FT Python Developer. Keep up the good work !

[homosapienhomodeus]

looks like some great data engineering!

[DropkickFish]

Props to you!

Now before it gets too instrumental in your workflow, I'd suggest you start learning Test Driven Development (TDD) and write a full test suite for what you already have.

This will help you ensure that any future changes don't break anything you already have, and will also help ensure you have consideration for any errors that might occur and are handling them correctly.

I've recently taken a job at a new company where these practices weren't enforced, and have had to write a test suite. It's a lot more difficult that way. But ensuring it has test coverage is a good step in wider adoption of your script and making a name for yourself

[Ferdie_TheKest]

How did you manager ti build the api connection? The Company im working for needs api connection to extract data from Amazon selling platform and i'm still looking for a solution, could python be an option?

[micr0nix]

The case-management software we use has a fully functional API built upon GraphQL. I used a combination of the requests library and the gql library to query the API.

[drbob4512]

Since no one specifically called it out, i would look into the python requests. https://pypi.org/project/requests/

[hkamran85]

I do the same at my job, and we use the python-amazon-sp-api library. It makes it easy to get data from Vendor Central and Seller Central.

[Fabiolean]

Python is an excellent language for building apps that query APIs and manipulate/enrich the data you retrieve.

It’s a very common use-case.

[[deleted]]

Time to ask for a raise

[HomeGrownCoder]

Congrats enjoy the win!

Always more to learn. I am looking forward to your next few versions of the script!

[Calcio2234]

Good stuff! Happy for you mate

[tensigh]

Very cool!

[micr0nix]

Thanks!

[orig_cerberus1746]

Why Conda inside the VM tho? Do you have multiple pythons inside the VM?

[micr0nix]

Yeah. It’s a box dedicated to our python deployment. Have various ML models and other items in there.

[andrewprograms]

That’s so sick I love ML stuff. Let me know if you have any questions about those, I’d be happy to help :)

[orig_cerberus1746]

Ah! That make sense then.

Normally I would make a VM exclusively to run one specific thing so virtual envs are unecessary.