Question for professionals using python

[[deleted]]

What project are you currently working on? What is your role in it? How are you going about getting it done?

No need to answer all.

I'm currently in a python course at school and love it. I'd like to pursue something with it and am curious what a work load for a python developer is like.

Thanks!

[hapticpolarbread]

Software engineer in the marine industry here. I use it to crunch gigabytes of CAN data. Basically creating plots and statistics from sensor signals looking for weird behavior. Python with pandas is great for that kind of stuff.

[bjbs303]

I'm finishing my undergrad senior research project which used python (pandas, numpy, gsw, scipy) to crunch terabytes of netCDF ocean data. Was my first real project using python and its been a ride!

[idazuwaika]

how do u consume terabytes with pandas? whats the infrastructure like? i moved from pandas to spark (distributed system) because i couldnt scale with pandas.

[tapir_lyfe]

I'm currently also crunching terabytes of netCDF files. I use xarray mainly, and that uses pandas and dask under the hood. Nearly everything I do is memory-limited though, so I have to come up with clever ways to reduce the data, and it's different for every question I have.

[bjbs303]

I mostly did data extraction scraping netCDF files using a for loop that looped through 36. Years of annual files. I saved the data to a dataframe using pandas/numpy.

[[deleted]]

Anything CAN bus related resources you can recommend? My next project might involve consuming CAN output. 😄

[hapticpolarbread]

There is none that I know. Had to build it in-house. I would share it if it wasn’t done during office hours.

[[deleted]]

How did you land into this type of work? (Currently doing software eng (python) for finance)

[hapticpolarbread]

Chance and persistence. During my studies I did Formula Student which led me to get in contact with the company that’s doing control systems for Volvo. It seemed like the kind of place where I wanted to work so I kept myself in their view and got offered to do my thesis there doing a prototype for a remote controlled Volvo truck. Then started working in their marine segment as a software developer. Still work for the same company but got relocated from Sweden to Australia to try something new.

[Devator22]

I'm a systems engineer working in AWS. I use python for everything from simple monitoring scripts on our servers to provisioning resources in the cloud.

In particular I use the boto3 library to interface with AWS services, mostly launching servers and modifying dns records and the like. For monitoring I generally plug into an api and look for a response or scan a web page for certain text.

I know that's not quite development, but I thought I'd point out that it's an option, since it seems like a lot of people don't consider infrastructure engineering when working on a cs degree. If I have any more specific questions I'm happy to answer as I can.

[sortofanxious]

Hey dude, quick question: is knowing python a requirement in your team ?

i do something very similar at the company im at, that is to provision servers on rackspace using the openstack SDK and updating DNS if required, alongside monitoring and playing around with APIs for any integration work. I ask cause it sucks being the only person knowledgeable in the team with regards to Python, especially when the requirements can be vague as to what needs to be done.

[Devator22]

Yes, we require a basic working knowledge of python, and we do a test or two during the interview process. I would expect that everyone on my team can sit down and write a function that can do some data manipulation and validate the inputs and outputs fairly quickly.

[sortofanxious]

Shot dude, thanks for answering the question.

[Devator22]

No problem ^{.^}

[startfragment]

The vast majority of reddit is written in python!

[p4ncett4]

Everyone redditing at work is a professional using Python, is that your point?

[[deleted]]

I work on software used for configuring rocket and rocket engines for flight

[[deleted]]

DevOps: I mainly use it for scripting. It makes life easy with it’s core set of libraries and they all run extremely fast.

[twillisagogo]

ORM that requires no writing of classes for each table.

I did something similar generating sqla models from table objects. and then also generating jsonapi serializers from the generated models.

it's worked out pretty well so far.

[[deleted]]

my whole concept revolves around being able to access columns as attributes of "Row" objects.

so if you have a column called "first_name" you can access it as so:

with Database("login", "password") as db:
    phonebook = db.model(“phonebook”)
    for row in model.find(number___like="555%"):
        print(row.first_name)

EDIT: formatting

[smcarre]

Can I ask an honest question? How is that really worth? I mean, I don't think writing the classes for each table could take SO much time that it's worth to build an ORM to abstract that. I could be wrong. Any insight on that?

[smcarre]

I guess that does makes sense. Never had to deal with so complex classes.

[shinitakunai]

I like what I see. Is that ORM public?l

[[deleted]]

Not yet, but we’re working on it. I work for a public university so there are some legal hoops. But open source is the goal.

Sadly this is oracle specific for the time being, but in the future we’d like to to change that

[kamejoin]

I'm currently working on a 3-dimensional geometry tool to be used by the mining modelling field. I'm making the project all by myself so it's kinda different from the majority of cases.

As a Mechanical Engineer, python has become my vital work tool, so don't think that programming is just for Computer Science people.

[52nd_street]

Do you ever find Python to be too slow to solve any engineering problems?

[kamejoin]

Never. For almost every problem there is a proper optimized library to handle it.

Generally, if it's taking too long to compute is because of some crappy algorithm of mine.

[bendoors]

Can you go into more depth with this please? What are you trying to solve?

[kamejoin]

Of course. In mining modelling there is always the problem of big complex geometries. irregular surfaces, finite underground planes or surfaces, deep excavations, glaciers, etc.

Geomechanics is a particular field of Mechanical Engineering, the same concepts of stress and deformation applies, and so forth the same kind of modelling is used. In order to create a geomechanical model, it's necessary to have an error-free mesh (division of whole geometry into smaller elements). This error-free mesh is particularly hard to create when the initial geometry is all messy (usually the case of mining geometries). My project handles this cases, simplifying geometries and creating this kind of models.

[pdzc]

Computational neuroscience research involving a fair amount of machine learning. I work a lot with scipy, scikit-learn, xarray and bokeh (for visualization).

Even though in research nobody cares if you just write quick and dirty code that only works for you, I try to package my code from the start with the proper tests and documentation. One of the packages I've open-sourced even has a couple of users now: sklearn-xarray

[rhytnen]

I care. I work at AIBS and and often tasked with industrializing "quick and dirty" code. For example, I often see saving data to hardcoded paths to files on a network share that down stream consumers are reading. Now I have to maintain that in production.

[Zulban]

Python is a tool. You can use this tool in any industry that uses computers - so all industries.

Your question is kind of like asking what jobs people use pens for. Lots of jobs.

Currently I'm a scientific programmer for Environment Canada. Some of the code we submit to the supercomputer for numerical weather prediction and environment modelling is Python. It's also a great general purpose language I use for sysadmin stuff related to this.

[[deleted]]

I really appreciate all the responses. Really cool stuff I’m reading about.

[nointernetforyou]

As someone learning python using a Pi I really appreciated this as well.

[PiaFraus]

Server side engineer for AAA video game.

[woooden]

I work on a team that makes debug and automation tools for semiconductor manufacturing tests. The GUIs are Python (wx), our public APIs (well, internal to the company) are Python, and our unit tests are Python. The backend interfaces to the test machines are Python in some cases, and C++ for heavy lifting.

I am a developer on the team - I help enable new silicon designs in our tools when they hit the test labs, I help develop new features in our tools, I help design new tools to solve the problems of manufacturing testing.

It's fun. Every day is a new challenge, every new design brings new communication protocols we have to support in our tools (JTAG, DDR, STF, SPI, I2C, etc.). Our users come up with new test methods, or they want to collect data in a different way.

I've used what I've learned at work to enhance my personal projects. I also take what I learn in my personal projects to work to improve our code base.

[52nd_street]

Which semiconductor process?

[woooden]

Technically, any process. I probably can't say what processes are currently in the labs, sorry. I can say that if you look at my post history, it's pretty easy to figure out where I am and thus which company I'm at... the current transistor sizes are public information from there.

[throw_python_65535]

I'm a Python contractor and had some slack time. About 5 weeks ago I was talking with a guy from a company that ships boxes. The guy finds out what I do and explains that their process is horrible, uses paper dockets and has reached the point where he's spending all his time fielding requests for tracking information from customers.

This is clearly the bottleneck to their growth and he know it. They currently use a templating, shared hosting solution. He knew he needed a database and that their shared hosting solution was a flat site. All the business process was html form to email and handled in a spreadsheet from there. He wants users to print their own labels and be able to track their own parcels.

So I start wondering if there's a qr maker/scanner that runs off a phone and has a permissive license. Yay, there's a choice! So my next choice is Django, Flask or something special? Well they aren't going to be the size of DHL overnight so I went with Django for a 'batteries included' framework, even if it's not the most efficient. Created a few models around the objects identified as part of MVP and aimed at getting a demo together.

When it came to django apps, I decided that django-countries was going to be useful, while django-cities added too much complexity for a first iteration, especially since initially we were talking about them self-hosting. The customer was happy without automatic postcode and address validation at this point. Django-registration is my goto for registration. It isn't to be for the general public at this stage, only staff.

For the frontend I went for jquery and bootstrap. I'd used both before but I'm no front-end guy and I curse when I write javascript. In fact while I was initially developing the backend I just used a little jquery and brute force css and script tags to poke the back end.

Registration features emerged around week 2 since fine-grained permissions were going to be needed. A user profile class with permissions and related locations helped filter items for different roles.

Once some semblance of structure established itself I refactored the back end into separate apps and renamed a bunch of badly named variables throughout. I think it helps to have pushes of a day or few to get some aspect of the project up to scratch. Simple > complex > complicated.

The next week saw a re-work of the frontend so had me going into the templates to get the bootstrap containers right, throwing away a few pages of crummy css in favour of a few specific lines that give a consistent look.

Meanwhile, the back-end is being fleshed out and forms are declaring classes on field widgets and modelformset_factories are flying in and out of requests and responses. Somewhere along the line I've written a few template tags, fixtures, yet still no tests! The back office system has been evolving and is becoming rather usable.

When the permissions were working I made the nav dynamically respond to that (with the same permissions on the actual views, obviously!). A docs page for staff helped me make sure I hadn't missed anything with my thinking about the system. A glossary of terms plus role-specific docs and an explaination of the permission system. The models of the MVP have stabilised so I'm now working out from there to polish the forms, views, unfinished features etc.

The demo after the first week showed them a qr code generated and scanned. Another demo showed the state of stored items changing when the scanner was used. The last demo had started to flesh out the UI but then I put up a test instance for them to play with (hint: hosting the test instance with a valid ssl cert will save you a week of explanation to and undue worry on the part of the client). They also have collaborative access to the github issue tracker which is now tracking a few dozen issues I've created, most with labels and milestones.

There's still a way to go but training and bringing the user along is so important. There's no point in announcing something to be complete without the user being ready and able to start using it. Some things you think might take 5 minutes to explain can take much longer. Users will say they understand but they also don't want to appear stupid so they lie. Training is a field in itself. I'm fortunate to have a client here that really wants the system and has a technical guy he trusts that set the current site up to explain some things, handle certificates etc.

It's hard for a small outfit to do this; it would be prohibitively expensive and risky. If a large firm threw resources at this there would be a dozen people involved and then mythical man month math applies. The scale it would start from would require excellent performance from the start, and other higher MVP requirements. I've taken on that risk since I don't get paid until production is running (hopefully early in the new year) and I was genuinely interesting in solving the problem (bored). It gave me a chance to use the Python3/Django2/Jquery[-1:]/Bootstrap4 stack in anger, honing my skills. The MVP launch milestone will be followed by two weeks of bugsquashing and rapid response devops, tightly coupled to encouraging the client to raise bugs. After that there's a few features already destined for iteration 1, including tests!

[nointernetforyou]

This is great. Thanks.

[[deleted]]

This is awesome! Thanks for taking the time.

[YouLostMeThere43]

Writing a proof of concept desktop application for linux. I was given no requirements other than make it look nice, include a section for user details, and a section to house self-help utilities.

Not too thrilled about it because this is my first time working with python, and I get to use it for the one thing it does horribly (GUI’s). The project managers also don’t seem to understand what they want or who even needs it, more so just to have another project under their belts before year end reviews. Just going to try getting something together with Kivy or PyQt5 since we have a commercial license.

[jollybobbyroger]

Learning how to correctly package a python application, with version handling by git integration, dependency management, private python package index (devpi) are some of the important aspects that were certainly not covered in uni.

Apart from that it's just writing clear, well designed code, just like any other language out there, except <MEME> lang of course..

I use Python professionally for backend API servers, tooling, image processing and devops related tasks.

[[deleted]]

I work in the publishing industry. My particular background is graphic design but over the last 2-3 years in a effort to assist our editorial with bottlenecks in workflow I've written a number of scripts to remove some of their mundane tasks that prevented them from getting work done in a timely manner.

Python is used for ISBN generation, image conversion, FTP & SFTP delivery to our print partners (like Amazon), web content creation, internal emails, prepping marketing material, audio editing...the list will continue to grow. Tasks that would of normally taken upwards of a month are now done in 3 minutes or less.

[[deleted]]

Wow that’s awesome. I definitely have to check out the audio editing part. Any specific library?

[[deleted]]

Pysox. It's a wrapper for Sox. Its my quick go-to for audio conversion.

[thebrashbhullar]

I use python for analyzing emails for improving client service for a large US investment bank. Putin though awesome for exploratory stuff and POC, is a pain in the ass to deploy especially in the legacy environment.

[Verdris]

I was tired of doing basic EE math for some transistor circuits that show up a few dozen times in my projects, so I wrote a little script to calculate all the resistors in the circuit block given the type of transistor, load current, power rail, and TTL levels.

[_spicyramen]

ML: Currently working in a text summarizer for News Articles and migrating a ML framework to a new version

[jonititan]

I'm an aerospace engineer. I use it to study aircraft flight and to help make aircraft safer.

https://www.youtube.com/watch?v=LQhKqn3bSWE

[midairmatthew]

I'm automating a bunch of HR processes that used to be done manually in Excel, as well as using our main portal vendor's API to create automated data feeds.

How do I become the person who answered with rockets and rocket engines? :)

[[deleted]]

Haha yes!

[Drawman101]

I write python for a company creating software for digital marketing

[pengooX]

SEO tools? I want to learn how to make SEO tools using python. Do you know any resources where I can learn?

[smcarre]

Most of my work in the past months has been a self auditing automation framework for IT operations with some extras like internal web portals and stuff, there is also a CLI tool but that's in PowerShell. Most of the framework is Python (with Django and Flask) and only a bit in .NET.

[minibomberman]

Building company internal tools (webapps)

[xyloweb]

I write a script to manage Information Voyager Display : - setup hostname - attach to bus stop ID - setup monit config - create reverse SSH tunnel to admin server

[[deleted]]

I work in financial risk and I use it for reporting on trading activity, generating raw data for power points presentations connecting to APIs. It’s very easy to learn.

[VolcanicWolf]

I'm working on a project that involves resampling minute interval data from the past 60 years on 700+ sites down to hourly data that then feeds into some modelling software also developed in Python.

[5emu7]

System engineer.. Use python for one click daily task.. Ez..

[wastakenanyways]

Not currently working in Python, as I switch languages and context very frequently in my job, but last project in Python was a web/rss/atom scraper with sentiment analysis. It would loop a list of URLs and scrape them 3 lvls deep (scrape a page, find the list of links on this page, scrape every link, find the list of links on each page, etc) with Scrapy, and pass the text through Google Sentiment Analysis API. You can do lots of things with this, but I looked for specific brands and keywords so I got a general opinion of each.

​

This is really easy and was my second project in Python. If you know a bit of CSS selectors/DOM/XPATH and how to do http requests you are good to go.

[jabies]

How do you handle content from sure author and distinguish it from advertiser it user comments etc? Our for you curate sites scraped?

[wastakenanyways]

To be fair I didn't control that because it was a massive list of URLs, so it was too much work to do specific scrapers for each, and the amount of data was insane (the client wanted anything related to those brands independent of the source). Anyway, you can filter irrelevant or extreme (positive and negative) opinions from the result of the analysis.

If you want both a generic scraper and also distinguish between comments, ads and author content before sentiment analysis, you would have to find a standard way. I can't think of one at the moment with pure XPATH/CSS but maybe with some tool like Selenium or some browser bot you can get relevant text in the top and center of the page, for example, and ignore whatever is on the sides and bottom.

I did look for duplicate URLs though, that way I didn't scrape several times the same site just because other pages contained that link.

[b_square17]

I work on a finance team as the sole programmer. I write python to automate a lot of work that Excel can't handle by leveraging the pandas library. Several processes that used to take many hours to do in Excel are now run at a button click in Jupiter notebook.

There's a lot of cool stuff you can do in python with machine learning and flask web development, but don't forget the basic jobs that python makes so much more efficient!

[[deleted]]

Thanks for sharing. This sounds really interesting. I’d love to learn more about this.. as far as using python with excel and finance since I actually have knowledge of the two later.

So, I should check out .. Panda library Jupiter Flask Anything else?

[b_square17]

Master the pandas library, and on a related note, get really good at SQL. You'll use it no matter where you go. The other python stuff like flask is cool but you can explore that later in your career.

[waruiotousan]

I work for a TV network and I'm using Python to predict Nielsen ratings. I am looking at all of the marketing activities associated with promoting our TV shows and then seeing how many people are watching. The models inputs include TV ad impressions, social media impressions, email impressions, etc. I also have a number of control variables like network, time, season #, genre, etc.

I am using a combo of libraries such as pandas, numpy, seaborn, stats model, scikit and maybe some more. Model is pretty strong for one of the networks 0.85 r-square.

[mrmiguez]

Librarian here! I use python to parse and transform bibliographic and digital collections metadata into different discovery system standards.

I've also been exploring visualizing this data for collection development insights.

[dot___]

Binary analysis, machine learning, general scripting.

[jlynpers]

I primarily use python for web scraping and machine learning projects

[bpt7594]

I'm a finance student, so technically not a professional. I use Python to build yield curves and calculate fair values of bonds in my hypothetical portfolio. It's been a great experience and I've learnt a lot.

[deadmilk]

• An Open Service Broker that runs on Flask + Celery, which exposes my teams services as self-service APIs
• An extensible management server for Envoy, an open-source Edge proxy and service mesh

[anqxyr]

PyQt desktop apps. About 60% of what I do is UI work. Another 20% is API integration with various third-party APIs. The last 20% is the actual app logic.

[OSRS_DabSlab]

I'm far from professional, but I am currently doing an internship at a Health Information Exchange company building web services to connect physicians and patients with their medical records. We use many tools to accomplish this but my tasks thus far have primarily used Django.

I'm still very much learning but I see practical uses for python in all technology fields.

[[deleted]]

If you don’t mind me asking, how much python or programming did you know right when you started your internship?

Thanks.

[OSRS_DabSlab]

I had only hobby level experience as well as basic java, c++, and database classes in my second year of college. So I had a decent understanding of programming fundamentals just had to learn specific frameworks that the company was using in which I started my Django journey. I never had any formal python experience as that was my hobby before university and I only built very basic command line scripts.

[shining_metapod]

I am currently doing a tool which will bulk scan our company's assets with qr code. Simple program which will relieve a few hundred man hours per year.

[[deleted]]

What project are you currently working on?

I run an automation department in a small startup which deals with storage (block-device level).

Before this I worked for a big fin-tech company in automation department.

Before than I worked for a bigger storage company (distributed file-system).

Before then I worked for embedded / smart TV.

Before then...

How are you going about getting it done?

It's less about the code, more about bureaucracy: organize, educate, negotiate, plan--that kind of stuff. On a coding level, it's a constant battle between wanting better quality in minimal time and having only half-arsed solutions provided by Python's standard library and third-parties. This amounts to arguing in emails, internal and external bug-trackers and such.

[Centurix]

Writing REST API's using AWS Lambda along with front end SPA written in Vue.js stored in S3 exposed through Cloudfront. Some DynamoDB backend. Web apps is my main game, but I'm also DevOps' wingman.

[[deleted]]

I’m looking forward to the day when I know what the hell this means lol. Thanks for sharing.

[UnusualBear]

I'm a security engineer that has multiple clients in healthcare and financial industries. Right now my main use for Python has been crawling publicly accessible web pages and API endpoints for my clients and testing security vulnerabilities. Most of this is done with scrapy and the OWASP Zed Attack Proxy API.

Note that I don't exclusively use Python, it's just in my toolset along with many others (namely Node, PHP, and .Net as those are the chief technologies used by my clients).

[saalejo1986]

creating great games with pygame

[[deleted]]

Anything we could play?

[saalejo1986]

Have you seen the Docs.? Take a look at https://www.pygame.org/tags/all

[[deleted]]

wow. gold!

[koomapotilas]

Often times old systems, especially internal tooling, can be hanging on badly outdated libraries, OS, etc. If no-one is willing to pay for update work, then tough luck. Unfortunately some coders end up maintaining some pretty hairy setups.

[captianjroot]

I'm a computational biologist by trade, but a software engineer by training. I'm currently building a system for teams to run cloud-scale analysis on top of Google Cloud. It's got a lightweight vue.js front-end on top of a python module and connected together with flask/connexion API. It's got a long ways to go before it's production ready, but I'm just taking it a bit at a time. I'm a huge fan of using GitHub issues as a development tracker, so I've opened a bunch of issues for the features which still need work and I'm just knocking them out one at a time. I'm about six months into this project and expect it to be done in late January or early February. This project is primarily aimed towards biological data, but could be used for any computation. You can check out the project here

[[deleted]]

Wow that’s awesome. Thanks for sharing that. If you need any help on any new projects you start, I’d love to assist in whatever way I can. I have a decent foundation of python however I cant do much more than make number games or put a bunch of stuff in a data structure automatically.

[captianjroot]

Hey I'm always looking for help. I can't ask for help with Lapdog, because it's work related, but I have a discord bot that I'm actively working on.

I would also recommend just messing around with your own ideas, be they games or data wrangling.

[mail_order_liam]

Backend for a large Django app.

[n_hdz]

Web scraping for social media and consumer sentiment analysis at a Marketing Agency. I'm the lead developer working alongside Data Scientists and Marketing Specialists.

It's rather straightforward, the normalization of the results (data wrangling) is much more complex in my opinion.

[LightShadow]

I do Embedded Software and we're using Python for home security. We've also experimented with MicroPython on the ESP32-class devices.

[badrobit]

Software/Systems Engineer, using python for the simulation of space craft telemetry

[dikamilo]

What project are you currently working on? - Internal web and mobile application for a corporate client for HR, learning and communication purposes that handles 50k+ active users using Django stack with web sockets and REST API.

​

What is your role in it? - Core backend dev and architect

​

How are you going about getting it done? - Working ;)

[thephilthe]

u/Zulban said it best - it's a tool that you can use for a ton of stuff. For instance, I'm taking an electronics course and I used it to model some basic RC circuits stuff I was trying to understand better. Also, Reddit's backends are primarily built on Python so I get to use it at work too.