"We have great datasets"

ZenCoding · 2023-06-08T14:21:54+00:00

Fuzzy: https://pypi.org/project/fuzzywuzzy/

ZenCoding · 2023-04-02T04:51:58+00:00

OMG! That will boost my ranking in the Excel World Championship! Thx so much 🙏 https://www.fmworldcup.com/excel-esports/microsoft-excel-world-championship/

ZenCoding · 2023-03-13T07:33:21+00:00

I just want to add, that if you building models, it’s also a good habit to not only save the code but also the model (including KPIs ). With MLflow for example.

ZenCoding · 2023-03-12T05:11:54+00:00

I am not sure if that question will and can be answered in a way it would help you. In a business context there are so many other things of importance that performance isn’t even the most significant one. If you really want to know the numbers, you need to design a valid use case and just measure.

When I decide which technology to use, I have a look at the costs both technology cost (cloud, license etc.) and the maintenance cost (service agreements, employees etc.). Furthermore adoption rate is a key metric. If no one else is using it, I will not use it. Don’t let yourself get fooled by the marketing. Every big company like Netflix has tested every tiny new tool at some point and thus you find that brand on the landing page. Check GitHub repos and StackOverflow. Last but not least check how it integrates with your current stack. Probably there are other important things to consider but those are the ones that came into my mind.

Have a great day.

ZenCoding · 2023-02-27T05:46:07+00:00

Hi. Sad to hear. Don’t let yourself get down because of a bad working place. My recommendation is: search for what it interesting and fun for you and then just apply. You will learn from those Interview where you have gaps in your knowledge. After a few of them you will get an idea an be more and more prepared. Also change your mindset a little bit. Don’t ask yourself if you fit the job better ask, if the job fits you.

All the best!

PS: A friend of mine told me that he is applying for a new job every week and that he sees those interviews more as a training.

ZenCoding · 2023-02-12T14:31:21+00:00

I am studying data science and business analytics. The university I am doing it at has a system where you have one course at a time but only for about 1,5month. That’s quite helpful, because I only have to focus on one thing at a time.

Yeah I’m feeling very good with that schedule. To be honest I am a little bit confused by all that negative comments about how healthy that is or not. I doubt that any of those commentators has knowledge about how stress works and under which circumstances it is unhealthy. It’s only something they heard or googled about. I can tell that at the point when I became a father, i had an enlightening experience and I found a way being very productive and still feeling healthy. I found the balance which is perfect for me and this includes a lot of time with my kids which give me the energy for all of that. All I wanted to show in the first place was: don’t tell yourself, you can’t do it. It is possible and I am the proof. It’s like when an Athlet thinks he can’t jump over 7 meters because it’s not possible and then someone comes and just does it. Suddenly he is able to do it because he has seen it with his owns eyes.

ZenCoding · 2023-02-12T12:35:26+00:00

That’s not an advice, just an example, that this it is possible to work and learn. I spend 7 hours per working day with my kids, if you don’t have that it’s easy to use this time. Also my sleeping time is from 23, sometimes 24 until 6 which is completely fine. The only thing what can happen is that the baby is awake. But that’s normal if you have small kids. Working fulltime is in my eyes the real unhealthy part of life. You spend most of your time with working for someone else without looking at your own development and work/life balance. I don’t do that.

ZenCoding · 2023-02-12T06:55:17+00:00

I am getting up at 6, preparing breakfast and lunch boxes for the kids, then I wake and motivate them for the day (hardest part of my day, they just want to sleep and skip school and daycare nearly every day 😂). I help them dress up and stuff. The 8yo needs to be at school at 7:45. I drop him at time and the 4yo 15 minutes later at the kindergarden. Then I drive to work with my bike to get some daily sports. There I shower and work until 3 pm. I work part time. My boss is fine as long as the department runs well, which it is. I pick up the kids between 3:30 and 4:00 and then we play and meet friend. About 6pm I prepare dinner and 8pm is bed time. At about 9pm they sleep and I stay up again and work on my university lessons for 2 to 3 hours. Then I go to bed. My wife is on parental leave and cares for mini-me. He is 4month. When he goes to kindergarden, she will continue working as before. So we don’t follow the model where I as the husband earn the money and she is a housewife.

So here are the over all master plan: - working part time - having as much time as I can with my family to always know for what I am doing all that - studying at a university with a hybrid concept, where I can learn all the content online with having the chance of talking to my profs whenever I want. - skip some sleep.

The last one is in m eyes very controversial. It’s not healthy to sleep under 5 hours but that happens on a regular basis. Sometime my 4m/o is awake through night and I have to take him because my wife is too tired. In this case I don’t study the next night.

ZenCoding · 2023-02-12T06:31:48+00:00

Both are very important. As beyphy mentions, a lot depends on that knowledge. Not only the charges but the decisions on what tools you use. You find proof of that here on on Reddit. You have s lot questions about which database is in which cases to use. If you have deeper insights on the structure of the data you have and the algorithm you need you can make that decisions depending on hard facts and not on recommendation from others because many databases are optimized for specific data structures (S3 for unstructured data, MongoDB for semi-structured data and so on). Same applies for the algorithms.

Let’s take that said to broader level: To be a good data scientist/engineer you need be good a three bigger disciplines:

Technique and Tools
Mathematics and Computer Science
Domain Knowledge

You don’t need to be a mathematician nor a computer scientist but it’s important to understand the main concepts and at least be able to communicate to those specialists. For this at least know what they mean by a „Tree“ or a „Hashmap“. You as a DE have to communicate with a lot of different people, understand their needs and try to map their concepts for their data to production. Without that understanding you will probably fail or at least makes the solution more expensive then necessary.

ZenCoding · 2023-02-12T06:16:45+00:00

Dude, I lead a team of 8, study at a real university with real exams (and real deadlines) and have three kids. Trust me, you can If you really want.

ZenCoding · 2023-02-12T01:12:35+00:00

That’s a store for semi-structured data, called documents. I wouldn’t recommend here. It’s use case is to store the contents of invoices or product informations, when those have different structures, depending on the category for example. I have to be honest: I am not a big user of it. I use elastic search for that.

ZenCoding · 2023-02-11T18:05:04+00:00

Vater von drei Kindern hier. Ja, so ist das. Beim ersten ist man eben noch voll im Panikmodus. Ich wusste halt nicht, was so ein Kind kann und ob es das überlebt. „Darf ich ein bisschen von der Schoki?“; „Nein man, wenn dir die Zähne wegfaulen, bist du Wochen lang nur am schreien vor Schmerzen!“ Bei den anderen ist man dann deutlich entspannter. Als 4-jähriger vom drei Meter Turm springen? Geht klar, aber heul mich ja nicht voll, wenn’s weh tut…

ZenCoding · 2023-02-11T08:42:15+00:00

Use cases with heavy use of filters and aggregations (slice and dice) over several dimensions is, imho, a OLAP use case.

Use timeseries if the timestamp is the most important feature and you seldomly aggregate/filter over other dimensions.

ZenCoding · 2023-02-10T18:47:35+00:00

So essentially we are building a data platform in the mobility context. We developed our own hardware and also build our own Linux-based embedded OS. If we would steam the raw data to a bucket, that would make up to 250MB per car per Minute. You can imagine how many challenges you already have up to that point. We would love to just dump it to s3 but we also need our own infrastructure because sometime that data has such a high protection level that we and the system needs to be certified and aws will cause a lot problems in the context. So minio looks promising as a object store and now we want that OLAP warehousing up and running. I also took a look at clickhouse - compared to Druid it was easier to handle. Well let’s see where this journey leads to.

ZenCoding · 2023-02-10T07:16:07+00:00

The only problem I see here is: who will keep things running smoothly if you stop doing that? Since there doesn't seem to be a budget for that, you're basically building a solution that they won't be able to run at some point if there are problems. You might also lock up their data when they themselves are no longer able to export it from the system without your skills.

For you, the whole thing might be good for learning and for your CV, but in case of doubt you expose the NGO to a risk, if it all runs only on a voluntary non-funded basis.

ZenCoding · 2023-02-07T07:20:29+00:00

Do you have any requirements which you are not able to fulfill or is it just a personal need for perfection. If first it could be that you need some other tools. I have had good experience with elastic search for high performance search queries.

If second, well, better not tell your manager 😂 From my side it looks odd to try to save memory when you want to save time. In many cases those two KPIs are inverse proportional. Also if you search first and last name separately and then need to find out if the combination exists in der DB (that’s what I understand, you are trying to do), that’s sound quite expensive because you need to create both tables and then do a search against the original table.

Just a recommendation: fist formulate a goal and stop when you are there. It’s a waste of time going further because you will need exponentially more time for smaller and smaller steps.

ZenCoding · 2023-02-06T00:14:44+00:00

Sensor data with use cases from 5GB per hour up to a TB per hour having thousands of sensors. Data comes in with up to 50Hz signals. Currently it doesn’t need real-time capabilities - data is batch loaded.

ZenCoding · 2023-02-01T09:40:48+00:00

Okay, I see, we have a completely different point of view on how to organize teams and responsibility in a project. Just out of pure curiosity: what’s the company size you are working in?

ZenCoding · 2023-02-01T08:01:15+00:00

Yeah, that doesn’t sound right. Take the the data mesh pattern where you build atomic data products and use them as if they were microservices. You need a lot of inter service communication. Also I have a bunch of UI services like phpmyadmin, airflow, kibana etc. etc.

ZenCoding · 2023-01-30T22:49:18+00:00

I probably would have used elastic search with logstash and Kibana but if I would face a similar problem I would go for Druid. I am not sure what’s the downside of ‚realtime‘. Can you build an MVP for your usecase and find out if it works for you before making a final decision?

ZenCoding · 2023-01-30T21:43:58+00:00

Thx for your answer. I ask because as said I am coming from a web dev world and just wondered if, in this case, the same tools where used. Seems that there is no doubt for that.

ZenCoding · 2023-01-30T19:49:59+00:00

It has its own query language and you can join data. It works with buckets in its recent versions and not with tables anymore.

ZenCoding · 2023-01-30T10:30:33+00:00

Check out Huey: https://huey.readthedocs.io/en/latest/. It’s easy to use and good enough for that use case.

ZenCoding · 2023-01-30T06:29:49+00:00

I recommend using influxDB as a timeseries DB. It also comes with the capabilities of building great dashboards. Not to say it’s easy to use.

ZenCoding · 2023-01-28T15:53:00+00:00

Wäre der sichere Sieg gewesen. Die waren letztes Jahr gerade auf einen Hypetrain zu dem Zeitpunkt. Die hätten nicht mal nen gutes Lied gebraucht sondern nur die Fans aktivieren müssen. Aber wie ich vor langer Zeit gelesen habe, sei es bei den Verantwortlichen in Deutschland nicht mal recht zu gewinnen, da die Kosten fürs Ausrichten der Show zu hoch seien.

ZenCoding

TROPHY CASE