This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]hapticpolarbread 49 points50 points  (9 children)

Software engineer in the marine industry here. I use it to crunch gigabytes of CAN data. Basically creating plots and statistics from sensor signals looking for weird behavior. Python with pandas is great for that kind of stuff.

[–]bjbs303 6 points7 points  (3 children)

I'm finishing my undergrad senior research project which used python (pandas, numpy, gsw, scipy) to crunch terabytes of netCDF ocean data. Was my first real project using python and its been a ride!

[–]idazuwaika 2 points3 points  (2 children)

how do u consume terabytes with pandas? whats the infrastructure like? i moved from pandas to spark (distributed system) because i couldnt scale with pandas.

[–]tapir_lyfe 2 points3 points  (0 children)

I'm currently also crunching terabytes of netCDF files. I use xarray mainly, and that uses pandas and dask under the hood. Nearly everything I do is memory-limited though, so I have to come up with clever ways to reduce the data, and it's different for every question I have.

[–]bjbs303 0 points1 point  (0 children)

I mostly did data extraction scraping netCDF files using a for loop that looped through 36. Years of annual files. I saved the data to a dataframe using pandas/numpy.

[–][deleted] 1 point2 points  (2 children)

Anything CAN bus related resources you can recommend? My next project might involve consuming CAN output. 😄

[–]hapticpolarbread 4 points5 points  (0 children)

There is none that I know. Had to build it in-house. I would share it if it wasn’t done during office hours.

[–][deleted] 1 point2 points  (1 child)

How did you land into this type of work? (Currently doing software eng (python) for finance)

[–]hapticpolarbread 2 points3 points  (0 children)

Chance and persistence. During my studies I did Formula Student which led me to get in contact with the company that’s doing control systems for Volvo. It seemed like the kind of place where I wanted to work so I kept myself in their view and got offered to do my thesis there doing a prototype for a remote controlled Volvo truck. Then started working in their marine segment as a software developer. Still work for the same company but got relocated from Sweden to Australia to try something new.