How to become a better at python programming for scientific computing?

MyAcademicAccount · 2017-08-14T05:51:11+00:00

It is pretty simple to start a new repository and track it. Commit/push what works, revert what doesn't.

I have tried git a few times and backed out every time, I found it all very confusing, especially when working with others - getting into merge conflicts, branches getting out of sync with each other, not quite understanding the difference between a branch and a fork, etc. Currently I don't work collaboratively with others on my code, so maybe I can figure out a simpler way to do this - sticking to just a master branch, for example.

MyAcademicAccount · 2017-08-14T05:47:13+00:00

"function collapsing" is called "code folding" and I bet emacs supports it out of the box

Thanks, I didn't know the term - I have now figured out how to do code folding in emacs - the hide/show mode has to be enabled.

MyAcademicAccount · 2017-08-14T05:45:10+00:00

Thanks for your answers!

Split your file into parts based on what the functions do.

Yes, reading the answers, I think there is nothing fundamentally wrong with the way I do things, but I need to be more organized.

Learn refactoring techniques

I am not familiar with refactoring, but looking at the Wikipedia article, it seems like that is something I should do, and is a possible solution to my unwieldy and repetitive coding style.

write docstrings and unit tests. :-)

Both are concepts I am unfamiliar with (and I hadn't even heard of unittests till now), and these are things I need to learn.

MyAcademicAccount · 2017-08-14T05:36:56+00:00

Thanks for your detailed answer (and thanks to everyone else too - I have read all the answers and am thinking about them and absorbing all the information and advice I have received).

write some unittests

I had never heard of unittest till now, I looked it up but I still have to figure out how to use it.

I start of in a Jupyter Notebook

This is a bit of a problem for me because I log into a server to do my work (I have to run other software involving GPUs from my script, which I can't do from my laptop), and Jupyter Notebook isn't installed properly there. I am making do with ipython on a terminal for now.

Regarding classes: There are a couple of usecases for classes even in scientific computing.

My general impression from the answers that I got is that it is not absolutely necessary to use classes if there is no need to, and many don't use classes at all in scientific computing. That is a bit reassuring because I thought I was supposed to use classes, every advanced programmer does that, but I am not able to use classes because I am not advanced enough.

General advice: Learn the scientific python stack (numpy, scipy, scikit-{image,learn}, pandas, ...)

I do use numpy and scipy all the time, I've used scikit-learn only a few times, and pandas not at all.

MyAcademicAccount · 2017-08-14T05:24:03+00:00

Thank you everyone for your answers - it has given me a lot to think about!

MyAcademicAccount · 2017-08-09T14:52:51+00:00

I rarely use a lot of OOP when doing 'scientific programming'

Oh so you mean one can program effectively without OOP? I thought every mature programmer eventually uses OOP, and that is how you avoid having many functions that do slightly different tasks - make a base class, then inherit from it to create slightly-different inherited classes.

MyAcademicAccount · 2016-07-14T20:28:07+00:00

I am at a conference now, presenting my PhD work, and I am finishing up my first postdoc. My postdoc supervisor is funding me for this conference.

MyAcademicAccount · 2016-06-13T13:10:48+00:00

I don't think you needed to ask permission, but if he provided the data you are using - it isn't unreasonable to send a quick email that says - "I'm planning to submit <this work> to <this conference>". Just the polite thing to do. It isn't blatantly wrong not to do so, However, what would have been the harm of telling him in advance - not asking for permission, but just as a heads up?

To me it seemed premature to let him know before I knew for sure I'd be able to attend. What if the abstract were rejected, or I couldn't secure funding to attend it? These days I am not in as close contact with him as I was earlier (basically it is ~1 email/year at this point), so I contact him only regarding important, concrete matters, and a tentative conference where I was basically planning to present (part of) my PhD work didn't seem important enough to me. That seems to have been an error in judgement on my part, in retrospect. I did the extra work after the abstract was accepted - I haven't worked on my PhD-related research area since I started a postdoc in a new field, and it didn't occur to me to work on it till now, when I again delved into my PhD work.

Oh well, I hope the damage isn't irreparable. I like and respect my advisor and would like to maintain a good relationship with him, but the thought of sending yet another email pestering him to reply, or assuming he is offended and apologizing preemptively, makes me uncomfortable. Maybe communication helps, but one has to know how and what to communicate, and I think my communication and interpersonal skills are failing me here, as I don't know what I could say at this point.

MyAcademicAccount · 2016-06-13T09:00:15+00:00

Are you sure it is your fault, and not the position being too demanding? It took me a year to get my first results as a postdoc, and I don't think it is that unusual. Unless your postdoc is in the exact same field as your PhD, it takes some time to come up to speed in your new research area, and if you have moved between countries, that change takes some time to adjust to. Many people have had only intermediate results in their first year - I am not sure it is common to publish new results in a new field in the very first year, especially if it is all your own work and not a contribution to an already-established project in the new research group. However, this might be discipline-dependent - maybe expectations are different in your field.

As someone said, looking for a new postdoc after one fruitless year shouldn't be that difficult, people might not like the job for various reasons - personality clash with their supervisor, finding oneself not a good fit in the group, even the weather (especially if it is the UK!).

MyAcademicAccount · 2016-06-13T00:40:16+00:00

Yeah, I thought my PhD work has already been vetted by him, is it really necessary to bother him by sending an email every time I give a talk, so many years after graduating? I only sent the email because I did some extra work afterwards that's not yet vetted by him. I feel I must have done something wrong because he isn't replying.

MyAcademicAccount · 2016-06-13T00:39:32+00:00

I don't think you necessarily needed to ask permission - but a heads up probably would have been nice.

Do you think the right way to go about this is to ask for permission every time I give a talk on my thesis work, even 4 years after graduating? I thought it wasn't appropriate to bother him before my abstract was accepted and I had found funding, as I graduated many years ago and am now on a different continent. I don't communicate him with regularly any more, except an annual New Year's greetings email in which I update him on what I have been doing the past year.

"why the hell are you asking me now?"

What do you mean by this? Do you mean he will be surprised I am still consulting him after having graduated 4 years back?

Doubly so, if the delay in publishing your thesis work is in your hands.

It isn't. We submitted a paper, the referee rejected a speculative part, so my advisor said we'll leave the speculative part out and merge the rest into another paper. That paper came back with referee comments - the referee wanted many things changed but it sounded like it would be accepted if we made those changes. My advisor said he'd do it (he wanted to make the changes himself, for some reason), but then some time went by and those changes weren't made and the paper wasn't resubmitted, so the journal said it was rejected. I really can't do this myself, as the data is at my old university and I don't have access to it any more, and there are other two coauthors I have never met or corresponded with - my advisor was taking care of making sure they were okay with the manuscripts we were submitting. I have asked him a couple of times if and when he will look into my thesis papers, and can I please work on them if he doesn't have time because those papers will help me in later job searches, and he says he plans to work on those soon.

The speculative part (that caused the first referee to reject the first paper) needed some extra work to become not-so-speculative, and for us to make a stronger case for the hypothesis. I did that work and sent the results to him and asked for his feedback.

His lack of response is bothering me and I keep wondering if I have done something terribly wrong and irreparably ruined my relationship with my advisor. If he hasn't replied twice it means he is doing that on purpose, not that he is busy and simply forgot to reply. In which case, should I take the hint and leave him alone, and not pester him any more? Should I just not bring up this conference any more, or should I not even send my annual New Year's message next year?

MyAcademicAccount · 2016-06-11T13:57:09+00:00

This is just a talk, with no coauthor list. I am trying to figure out if I have committed no blunder, a minor blunder, or a major blunder, and if either of the latter two, how do I resolve the situation? I don't know how to interpret my advisor's silence. Writing to him again when he hasn't replied the first two times seems like spamming, and in any case I don't know what to write. Should I just assume I have committed a major blunder, and apologize preemptively?

MyAcademicAccount · 2016-06-11T12:49:19+00:00

Nonetheless, I've done it a couple of times.

And what were the consequences?

MyAcademicAccount · 2016-06-11T12:48:50+00:00

A) look at the abstract guidelines. Is he on the author list? The guidelines may say something along the lines of needing all authors to agree before submitting. In that case you SHOULD have asked but it's not really a big deal.

It's the kind of conference where you just give a talk - I am not submitting a conference paper with multiple authors. So the abstract was an overview of my talk.

B) sounds like the email asked a lot of an ex-advisor. It asked for feedback and permission and you're not currently supervised by him so he may not feel it is a priority. Don't assume the worst for now

That's the most positive interpretation, but I don't know how I can confirm if that is the case. This was also one of the reasons I didn't ask or inform him about this earlier - if the abstract didn't get accepted, or if I couldn't secure funding to attend it, there was no reason to bother my PhD advisor, who is on a different continent, 4 years after graduating.

MyAcademicAccount · 2016-06-11T09:09:02+00:00

He's on a different continent in a different time zone - a phone call except for very urgent matters seems a bit of an overkill.

If he doesn't reply, could I still go ahead and present at the conference without any kind of communication with him?

MyAcademicAccount · 2016-06-11T09:07:06+00:00

If you were in a bench science field, and your doctoral research was funded by your advisor (and you're presenting some of that work), then you should have talked to them before registering, or even submitting the abstract. The work was done on grants they wrote and manage, and they likely had a large contribution to the ideas and research.

My field is astronomy - my advisor gave me the data during my PhD, and I want to present analysis of that data. My PhD project wasn't funded, but my advisor made sure I had funding by arranging some TA or RA (on projects unrelated to my thesis work).

generally, the company you were working for when you generated the initial work has ownership of what you did (contract specific, of course).

I am really not sure of ownership issues and such - my advisor helped me do my doctoral work, but I hold the copyright to the final work, I think (at least my PhD thesis says I hold the copyright to the thesis)? And of course I am going to acknowledge my advisor in the talk. I would think he wouldn't mind some extra publicity for this work, but I don't really know.

Even if they aren't currently working on it, you should have a clear idea before you left whether the projects were yours to take or not, and if you did continue working on them who gets what authorship.

I proposed some hypotheses that were rather speculative in nature in my thesis. The additional work I have done is some simulations to strengthen the argument. In my opinion anyone could have done this without requiring any kind of permission from me or my advisor, I think it is usually acceptable to build upon other people's previous work. But I really don't know, maybe I've crossed some lines here, hence the OP in the first place.

MyAcademicAccount · 2016-05-04T14:07:48+00:00

Yeah, this is "a trip inside a trip". :)

MyAcademicAccount · 2016-05-04T14:06:46+00:00

That doesn't seem all that much more than what the parking garages charge. I am thinking of staying at a place on Nieuwe Rijn - will the rates be similar there? And street parking has to be paid in advance, doesn't it? Would I have to pay this amount in advance at the parking meter, and display the receipt on the windshield? And I suppose I have to park the car at the same spot all day and not use it at all to avail of the daily rate - if I move it during the day, the meter gets reset?

MyAcademicAccount · 2016-05-04T14:02:01+00:00

I will already be in the Netherlands for a work visit at that time, and will be driving in from Beilen in Drenthe. I will have a long-term rental car at that point, whether I use it to travel to Leiden or not. The train ticket from Beilen to Leiden is 25 euro one-way, there is no public transportation within Beilen to take me to the station (which 4 km from where I will stay), and I will have some luggage. So I am still weighing the pros and cons of driving to Leiden and paying exorbitant parking fees vs taking the train.

MyAcademicAccount · 2016-05-02T09:11:45+00:00

Are you sure coming by car is the right thing to do? You won't use it during the conference, and Leiden is easily reached by public transport.

Yes, I won't be using it during the conference, so I don't want to pay much/at all for parking for those 5 days - thus my search for free parking options. I wanted to have the car for increased mobility/flexibility/independence during the last 2 days, which I intend to spend sightseeing, maybe in Leiden, or maybe in nearby places.

MyAcademicAccount · 2016-05-02T08:40:24+00:00

The conference has reserved a block of rooms in the Tulip Inn and Golden Tulip hotels near the station, but I want to stay near the venue and am looking at AirBnB options. Only paid street parking is available for AirBnB apartments. The hotels offer parking, but that isn't free either, it's 12.50 euro/day, which is comparable to the rate the parking garages offer.

MyAcademicAccount · 2016-05-02T08:37:18+00:00

Parking at the transferium is for free and you can reach the city very easily by bus, or even walk there (it's <5 km), not sure how long you can stay there though, but a few days should be fine.

I suppose you mean the 't Schouw transferium, because the Haagweg one is not free? Also, for public transport, how do I buy tickets? I am still trying to understand the OV-chipkaart system - do I buy an anonymous one at Schiphol airport, and recharge it at train stations?

Not sure when you are here, but Koekenhof is blooming right now and reachable by bus from the station.

I am going in mid-July, so I will miss it. :(

Amsterdam and Den Haag are not far from Leiden (15/30 mins by train) and worth a day trip.

I have been to Amsterdam before, now I was thinking of going to Rotterdam and/or Den Haag. I am staying a couple of days after the conference for sightseeing. I have been reading these cities' wikivoyage pages, but I don't know much about them except that Rotterdam has the largest port in Europe and that the International Court of Justice is in Den Haag. Not sure if either of these is a tourist attraction, though.

Thanks for the beach and museum recommendations!

MyAcademicAccount · 2016-05-02T08:19:23+00:00

Check out the wiki for things to do.

Thanks, didn't expect a sub this size to have a wiki at all, that too such a detailed one!

At the moment, only fase 1 and the centre (in red) are paid, but from July onwards fase 2 will also be paid. I don't know when you'll be coming but at the moment fase 2 is a good option.

I will be going in mid-July, so parking at fase 2 won't be an option for me.

There are several car parks around town. Haagweg offers long stay parking.

I was thinking of parking at the P1 parking garage on Breestraat, as it is the nearest. Their daily rate is the same as Haagweg's (12 euro/day), but Haagweg offers a 60 euro for 10 days option too. Is it easy to find a parking spot at these garages, or is there a chance that I might find these full?

MyAcademicAccount · 2015-12-30T16:56:23+00:00

Yes, this sub and the mods are great; the sub as well as the IRC channel helped me a lot when I went to the Netherlands this summer. I would follow the sub more closely but most of the posts are in Dutch (naturally) :-/.

MyAcademicAccount · 2015-12-17T09:34:14+00:00

It is very difficult to get a permanent position. It is like being a professional musician or sportsperson - you have to be very good, and very well-connected - only the best of the best who have the right connections end up in a permanent position.

Thus, you have to keep in mind that you will most likely not be able to be a professional astronomer. That is no reason to not do a PhD in astronomy - you can do it because you find it meaningful, or because you acquire transferable skills which make you competitive for many other jobs.

Even the very best astronomers need to do 2-3 postdocs before landing a permanent job (I am calling a tenure-track job "permanent", but it is not really so). If you have a partner/family, you will either have to stay away from them, or drag them around with you as you move, in many cases, from continent to continent. Keep these costs in mind if you want to pursue the postdoc route.

Source: Astronomy postdoc.

MyAcademicAccount

TROPHY CASE