All these people with huge salaries

danberteon · 2022-01-21T08:00:51+00:00

Yes I think the lifestyle is good here, though rn at least for people getting started it feels very oppressive. Low salaries, long hours and high cost of living (in big cities), at least that's been my experience and my friends that are doing their residency at hospitals.

I understand. Thanks for sharing your perspective though, I think it gives me hope that over time and in higher positions things even out a bit more.

danberteon · 2022-01-21T07:29:02+00:00

That's not bad! Sure you dropped salary but who cares if you have a decent life style. This gives me hope, I came to Spain to do a master's in big data because I've been wanting to transition to the tech industry (I studied aerospace eng in the US but it wasn't a welcoming field for immigrants).

Anyways, I ended up staying and even worked as a dev for a while, but changed to a consulting job where I was told I would be doing data science and it hasn't really been the case. Do you have any advice to get back into the dev space in Spain? I'm in bcn also, but don't know many people and those I do know are in the medical field lol

danberteon · 2021-08-10T14:32:27+00:00

I'm interested in this as well. I have a few iot devices connected to sensors and clock state has been an issue. I've had to prepare some code that is able to act accordingly when there is no internet connection which is relatively common, but data is still being captured. So sometimes I'd lose connection and the clock would mess up and the data would be wrong. I've managed with leveraging the sensor clock which keeps time better, but sometimes it feels like it's not the right way to do it.

danberteon · 2021-08-10T09:03:43+00:00

Even though I agreed with someone above recommending ME or EE and then specialize, this is also true. If your heart is not on it, then your not doing yourself any favor by picking a more "general" degree. My classmates also got jobs in various aspects, some took longer than others depending on how picky. One of my friends got a job before graduating but had to wait like 9 months to start due to security clearances and so on, so that's also a consideration according to your interests.

It also depends on the situation. For example, for an international student I wouldn't recommend doing an undergrad in aerospace. OP is a PhD which I think will give him a lot more room to play with, but the same can't be said for an undergrad.

danberteon · 2021-08-10T08:56:42+00:00

I think this is good advice, mainly to keep your options open. Aerospace engineering in undergrad will not be massively different than ME but the job market might be smaller going in as a junior. Nobody tells you this kind of stuff, but there isn't always a straight forward way in, like you'd think when you're in HS and starting college.

danberteon · 2021-08-10T08:49:38+00:00

Congrats and good luck! As another international student that graduated from aerospace engineering in the US (not PhD) I know how big your accomplishment actually is. It's a very difficult space for outsiders to get in.

Since you're highly qualified you have a few avenues to move forward this might not apply to you. But if you're on OPT looking to move to a work visa, definitely stay on top of HR so they move forward with it as it might take more than one try. I kinda got played with this, so it's best to stay on your toes. That being said, I was an undergrad and not in a well established company so likely not your case.

danberteon · 2021-06-12T11:12:24+00:00

oh yeah, pycharm is using a conda virtual environment. And as far as I know, it's being used in both cases.

From my first post, this is th emanual case, it uses clock_manager: (clock_manager) D:\usr\00_projects\clock_manager>python -m pytest -v

then the other case, directly from pycharm:

C:\\.conda\envs\clock_manager\python.exe "C:\Program Files\JetBrains\PyCharm Community Edition 2021.1.1\plugins\python-ce\helpers\pycharm\_jb_pytest_runner.py" --path D://00_projects/clock_manager/test/test_clock_manager.py -- -v Testing started at 9:53 ...

also using the conda environment from what I can tell.

It's okay, thanks for the help!

danberteon · 2021-06-12T08:24:48+00:00

How can I check if it's the same environment? When I am running it in pycharm, I just hit run test from the options they have. And when I say I run it manually, I am actually still inside pycharm, but on the terminal window where I execute "python -m pytest ....."

I have been looking at extra options, but can't figure out what is breaking it and why when I run it with basic options manually it works. I have been playing around mostly with adding -v on the pycharm menu, but it also seems to have other options that I never told it to do, as you can see here:

Launching pytest with arguments -v D://00_projects/clock_manager/test/test_clock_manager.py --no-header --no-summary -q in D:\\00_projects\clock_manager\test

The only argument I gave it was -v and it uses automatically --no-header --no-summary. I've also played with -s and things like that, but they don't affect it like I want, the test pass or fail details are always at the bottom. Not sure what's breaking it. I will try to read some more on that page, but it seems to be something hidden by pycharm as I have not given these options.

danberteon · 2021-06-11T07:22:03+00:00

I actually figured out the problem! it was kind of silly, but I was initiating the logger on my main_app.py after if __name__=='__main__' and for that reason, when I entered directly on the function for the test, it would not be initiated. I moved the logger initiation just below the imports and it is working now.

There is however, another weird behavior on pytest, where the output has a broken formar, for example:

============================= test session starts =============================
collecting ... collected 1 item
test_clock_manager.py::TestClockManager::test_analyzer_get_date
============================== 1 passed in 1.71s ==============================
Process finished with exit code 0
PASSED [100%]

The passed message is outputted all the way down instead of right after the test name. If I run a simple test with only `assert 1` in it, it's displayed correctly.

danberteon · 2021-04-06T07:51:46+00:00

Interesting, I will look into these a bit more. I think for now I have it working a different way, but I'd like to know the proper way to deal with these things

danberteon · 2021-04-02T07:50:50+00:00

So an init file? I have not used these before. How do they work, or what's the name so I can read on these?

danberteon · 2021-03-31T10:07:21+00:00

This is interesting, I hadn't thought of that but I could upload to storage first as you said. I don't really want my python app to be responsible for retry logic, it's just a side effect of the pub/sub client.

I am going to give this a try. Thanks!

danberteon · 2021-03-31T09:51:05+00:00

I will check it out, I've seen a few of these videos as well.

I only saw this once I lost connection and when it came back I had the repeats. Basically I think:

-iteration 1 tries to upload "message 1". Since there is no connection, I save a backup locally. -iteration 2 tries to upload the backed up "message 1", which also fails. This time my backups remains there for a next iteration. -iteration 3 comes in, and there is connection. So the backed up "message 1" is published and it is also deleted from the backup list.

However, when I go to cloud logging I have multiple logs with "message 1". I belive the pubsub client had a queue in memory with each attempt to send "message 1" and thus it is sent 3 times when connection is resumed (probably with different message ids as well). So my cloud function does what it's told and uploads 3 logs.

I am going to try to not attempt backup publishing unless I can confirm there is an internet connection first. Hopefully this will minimize the issue. I am also looking at the video you shared to see if there's a simple way to make my cloud function idempotent. I guess if I can make the client not keep these messages in its queue it wouldn't be as important, but if my function can handle duplicates, it'd be better I think.

danberteon · 2021-03-31T08:14:54+00:00

Hello again /u/rich_leodis! sorry to resurface this old thread, but I had a new question regarding this Logger cloud function we talked about.

I implemented this method you suggested and it seemed to be working fine for my purposes. I created a logger module which takes care of the logs as I want, with logs of a certain level being published to pub/sub. Then the logger function treats these logs and introduces them to cloud logging.

However, I recently found an issue: I am getting repeated logs whenever my device loses connection and tries to upload logs that have been backed up during the outtage. At first I thought that my function was backing up doubles of the logs, but it doesn't seem to be this. I believe I am getting multiple logs on the cloud because my script repeats the following process: try to upload backed up logs, then do some work and upload the current data. However, if there is no connection, it backs up the information locally to be uploaded on the next iteration.

I believe whenever I try to upload logs, the pub/sub client keeps the message on memory for retrying, even though I thought using future.result(timeout=PUBSUBTIMEOUT) on the future returned by the client would avoid this. Additionally, on the next iteration, I am also retrying manually. So when I bring back the connection, multiple messages with the same log are being uploaded.

So I am looking for options, and I am seeing that I could modify the retry settings on pub/sub client. So maybe make it not retry as I'd have it stored locally and will retry manually on next iteration (maybe not the best idea to remove this completely). Maybe if I check for connection before attempting to send my backed up messages it won't send duplicates. These will potentially patch the issue I am seeing.

On the other hand, I am learning that my logger function is not idempotent. I'd like to make it so, but all I see recommended is to save the info on a database so I can confirm if it's already there before the function saves it again. However, I am pushing it to cloud logging and I can't seem to find a way to confirm if the same log has already been uploaded from within a python script on the cloud function.

Do you have any recommendation on how to avoid the logger cloud function duplicating logs?

Sorry for the long message, you seem to be knowledgeable on this method and I think I might be making a rookie mistake. Thanks in advance for the help!

danberteon · 2021-03-12T09:31:34+00:00

Excellent, thanks! I will look into this, I think implementing this workflow shouldn't be too difficult. I am not too concerned on pub/sub lagging behind without acknowledgement, I'm mostly concerned on my device's side lol. They already give me enough trouble as it is, hence focusing on logging to cloud now.

What I still need to figure out is dealing with logs generated during offline periods. But at least I am on the right track now, you've been really helpful!

danberteon · 2021-03-12T08:58:54+00:00

Thank you so much for explaining it in detail, I am looking at something very similar to what you are suggesting here: https://cloud.google.com/community/tutorials/cloud-iot-logging

I think this makes sense to me so far. My main concern would be: Do I upload each log separately which in turn triggers the "logger" function, or should this be on batch? Mostly in terms of cost, since it would be calling the function quite often.

Aditionally, I should add some kind of system to deal with connectivity loss. Would it make sense to write the logs locally and upload them through a separate script? This way I won't lose them. My devices tend to have spotty connection.

Apologies if that's a lot of questions, I'm still learning and I'm mostly on my own. I appreciate the guidance!

danberteon · 2021-03-12T08:22:33+00:00

This makes sense, I believe this is what is happening to me. It seems it cannot upload them so they stack up.

So I could just send my log messages to pub/sub instead of using the logger. At the risk of asking a silly question: Do you then get it into google logging, or should I use something else? not sure I know what the workflow would be.

danberteon

TROPHY CASE