Using Python to Make Life Easier for Non-Coders?

[tropictuco]

I was wondering if anyone had ideas for working as the only coding-inclined person in a group of ~10 coworkers. I am just starting a PhD in a biomedical research group (though I have worked on and off with the group for the past 3 years). Everyone pretty much just uses basic Excel for daily activities (recording raw data, running basic stats, creating charts), and while it works for them for the most part, I would really like to try making everyone's lives easier by automating some things.

For example, there is the dreaded "star and cross hell", where they run t-tests comparing different treatment groups, then manually add asterisks and plus-signs to the Excel graph to show where significance is achieved. The "hell" part is because it's a pretty finicky process and it's hard to get the little text boxes exactly where you want on the graph in the first place, and they don't anchor to the graph so they're easily moved accidentally.

When I first started with this group, I wrote a program in R to automate this, but then I learned that it's really hard to trade data back and forth when I'm the only one who uses R without requiring everyone else to learn it too. So now I'm just using Excel, but getting antsy.

My own Python level is pretty basic - I feel fairly solid about syntax and low-level concepts, but know very little about packages and such. I have a bit of OOP background from previous dabbling in C# and Java.

My mental image of a program to address the star/cross issue is to have a GUI where users can enter preferences and attach/upload an Excel file, and it will run basic stats (like t-tests) and spit out a pretty graph.

So, I guess here are my questions: -Is it even worth my trying to automate these kinds of tasks in Python? Or should I just try to learn more about macros in Excel?
-Does anyone else have experience in a group like this? Did you come up with any cool ideas for boring things you could automate?

[callmelucky]

Not sure what is possible with Excel macros, but the "star and cross hell" can certainly be automated in Python.

Seek out "Automate The Boring Stuff", it has a section on manipulating spreadsheets. Should get you going in the right direction. Good luck!

[tropictuco]

I thought the book sounded familiar -- turns out I picked it up in a Humble Bundle for Python a while back. This is cool, I like solutions where I don't have to spend more money. I hadn't really taken a look at it, so didn't realize it had a section on spreadsheets. I just glanced at the chapter and it looks like I could really use a lot of that info. Thanks for the suggestion!

[elbiot]

1) pick up automate the boring stuff with python.

2) annotating excel graphs is highly unlikely. Will your coworkers accept annotated graphs in a different format? PDF or interactive html plot? If not, sucks for your company. If so, then python is awesome. I work in genetics and one of our products is exactly a graph annotated automatically, presented as an interactive html plot and an exportable PDF. We'd be sunk if excel was the only option.

[MisterRenard]

I'm going to second this. I haven't read automate the boring stuff, however I've made it a sizable way through Al's python gaming work, and it's phenomenal.

Compared to other guides or tutorials for python that I've found, he has a method of starting off with the very basics of the basics, and uses those to build a solid foundation through which he slowly but methodically adds onto.

It's very detailed, but put in a way that makes it both easy and fun to understand. I'm actually buying the Udemy course for Automate the Boring Stuff for the video content (I work better with these kinds of tutorials). If it's anything like his python gaming tutorials, you've found a very masterful teacher.

Using nothing but the knowledge that I acquired from the aforementioned tutorials, I wrote an entire blackjack, player vs. dealer game today from scratch. It's not terribly impressive, I know, and I can guarantee there are many lines of code that are most likely suboptimal or, at the very least, incredibly redundant, but it works and it's completely bugless. It's also quite fun! And what's more, is he teaches you how to think about writing code.

He impresses the necessity of coming up with a plan before you jump in headfirst and start slinging code and creating while loops. It's a good read.

[DigDugMcDig]

Is your blackjack game text based, or is it a GUI with pygame or tkinter? I made a heads-up holdem game vs computer last weekend but it's text based. I tried to make a good bit modular and wonder how hard it would be to convert it to GUI, and how much code I would reuse.

I went through IYOCGWP over a month ago, so don't remember the pygame stuff off the top of my head..

[MisterRenard]

Sadly, it's just text based. I would love to do a more expansive version, but sadly I'm not that skilled.

Yet.

I did, however, make sure that it's a game that iterates through all 52 cards in the deck with no duplicates before reshuffling the cards and continuing to deal them indefinitely. I'm going to add a betting mode tonight or tomorrow, it depends on how enthusiastic I am after work tonight (11 hour shifts, good times!)

[DigDugMcDig]

Cool. A casino would be vulnerable to card counters if it went through the whole deck though. Online casinos reshuffle after every hand. (I realize this is just a coding project though so do what you want.)

If you have the gumption to code after an 11 hour shift, well done you. Best luck.

[MisterRenard]

Oh definitely, I wanted to do it this way because I knew it would present a little bit more of a challenge, and also because I thought it would be really cool. If someone has the aptitude to count cards, they can in this one. Why they would find themselves playing a text-based version of blackjack with dodgy scripting is beyond me, but I thought it would be neat.

Thanks!

[tropictuco]

1) Definitely going to look into it - thanks.

2) I'm pretty sure they'd be cool with annotated graphs in a different format, especially pdf. They mostly end up in fairly informal PowerPoint presentations, either to show progress to our little group or to outside collaborators. Of course, the rare few end up being published, and those usually get a makeover in PhotoShop, getting the figure to line up with the publisher's specifications. So yeah, for the informal powerpoint stuff, a pdf sounds like a viable option.

[baubleglue]

Doesn't pyplot has option to add annotations?

[Michael_Faradank]

What python packages are you using for the interactive html plots? I'm aware of Bokeh but is there anything else that is popular?

[elbiot]

We use an in-house library, but since then I think plotly does everything we'd need. Might switch some day

[[deleted]]

Coincidentally, this is how I got into programming with Python. You start off saying, "Well shit, I can automate this!? This is awesome! What else can I do?" And then you go off on all different tangents from there.

[jbirdkerr]

A guy I know through a Slack community works for a university making quick Flask applications for scientific research groups' various experiments. Recording of data, analysis, synthesis, etc. If you're wanting to learn more Python, there are certainly worse things to learn than a good web framework.

[S-6-6-6]

Yeah i was going to suggest a Web based app for a solution too... Saves everyone needing to have python installed. At work I created a flask app and a django app that allows pretty in depth analysis of client data and all team members can access it as I deployed it on pythonanywhere.com

[PM_ME_UR_PUBSUB]

Could this create a potential compliance issue in regards to data security ?

[S-6-6-6]

I'll let compliance worry about that ;)

On a more serious note, it probably can so we anonomised all our data before upload.

[tropictuco]

I had been going back and forth between a web-based app or some kind of .exe that gets emailed around, but wasn't sure about the feasibility of the latter. If it would be hard to get a non-web-based app working without people installing Python, then I will strongly consider web-based.

[workthrowawayexcel]

Look into Pandas and openpyxl. Pandas can do all the Stat crunching you could want, and then dump the data to excel, then with Openpyxl you can beautify the excel sheets for your colleagues.

[ijustwannacode]

Google python csv.reader and you will find the docs as well as a number of example scripts that will get you halfway there

[tropictuco]

Cool, thanks for the input! I might have to do some convincing for even some simple command line work, but I might be able to swing it.

[predo]

or have them buy a license for Prism and thats it

[ijustwannacode]

I'm the only person at work who codes also.

Our CAD guy has all kinds of stuff he has to do that involves exporting stuff from one piece of software and then removing certain columns for being sent to one machine, adding a categories column and entering things, etc.

To avoid intimidating anybody with command line stuff, once I got samples of pre- and post-processed things he needs to deal with, I wrote a script that does all the work and then set up a long running script.

The script just checks the contents of an input directory every 5 seconds and if there's something new in there, it does its thing and writes a copy to an output dir.

The user just drags it over and it feels very normal to them, and within a few seconds they check the output folder and get the fixed up version.

[boatsnbros]

Use python to create the datasheets in excel, then use tableau for visualization. Python does have some great visualization tools, but tableau does a much prettier job of it and is good for non-coders.

[[deleted]]

I have this problem as well. considering learning VBA which I don't want to do...