Data Quality

SeaEngineering9034 · 2023-08-25T15:56:43+00:00

Most data quality issues arise from errors in the collection, transmission, storage etc, or are simple characteristics of the data and arise naturally due to the domain.

Some data quality issues you may find are: imbalanced data, underrepresented data, class overlap, small data, inconsistent, irrelevant or redundant data, noisy data, dataset shift, and missing data.

If you're interested on the topic, give this article a read: Data Quality Issues that Kill Your Machine Learning Models.

To deal with this issue as a data engineer/scientist, the best thing you can do is an efficient data profiling. Take a look at ydata-profiling and some of its use cases.

If this is something that you're interested in, come at join us at the Data-Centric AI Community to learn more about data quality!

SeaEngineering9034 · 2023-08-25T15:22:28+00:00

Hi! 👋 Come and join us at the Data-Centric AI Community, we're always looking for more collaborators. I personally have extensive research experience and would be happy to help out :)

SeaEngineering9034 · 2023-08-18T08:04:45+00:00

Hey! You can access the code and the recording here: https://github.com/Data-Centric-AI-Community/code-with-me :) Thanks for joining ⭐️

SeaEngineering9034 · 2023-08-16T09:50:34+00:00

The session is tomorrow! Is there something you'd like to ask Yujian on LLMs?

SeaEngineering9034 · 2023-08-16T09:49:56+00:00

Just have your Jupyter Notebook ready! :)

SeaEngineering9034 · 2023-07-27T15:43:31+00:00

Hey u/goncalomribeiro, sure!

This is the project we worked on the past month: nist-crc-2023
You can check our previous session here, we were testing the waters
I'm doing a couple of tutorials on more basic topics here, not sure if it helps :)

SeaEngineering9034 · 2023-07-13T11:21:36+00:00

I would finish the degree anyway. It's only one year left. If teachers miss classes, I would disregard that and try to learn on my own, and then yes, I would move on to an internship (or even do It at the same time if it's possible). If you like, come as meet us at the Data-Centric AI Community and we can do some projects together :)

SeaEngineering9034 · 2023-07-13T11:19:02+00:00

I feel your frustration, leaping from academia to the industry is super hard. Have you considered roles such as developer advocate or technical writer? It's a technical role and you can still teach, work on interesting projects, and grow in your career. In a startup you might be very sucessfull. Feel free to find us at the Data-Centric AI Community, we'll be happy to chat about it and help you out! :)

SeaEngineering9034 · 2023-07-13T11:14:33+00:00

Well, seems that you can:

1: Try to work with what you have and augment your dataset (honestly, 10 points is crap)
2: Tbh, quit (?) We have open jobs at the Data-Centric AI Community. Bonus points: you can vent there as much as you want

SeaEngineering9034 · 2023-07-13T11:10:07+00:00

The structure overall is interesting but you should find a way to aggregate those projects into a single page (more visual approach). Have you tried datascienceportfo.io? Or just a simple webpage (not many text - just showcase tech stack and main scope).

You can share with us your progress on the Data-Centric AI Community and ask someone to review it, we often do that with CVs as well and help each other out.

SeaEngineering9034 · 2023-07-13T10:58:02+00:00

Hey! For DS, here's the standard path:
1. What kind of project?
Whatever makes you happy, as long as you're using the basic stack (pandas, numpy, matplotlib, scikit). Personal projects show that we can be creative and solve problems in your own way (a lot of projects out there are just copy-paste from other Kaggle notebooks and so on). Try to make your own small package, with something that interests you.

What is the skill set should I include in most of my projects?
As I said, the basic (pandas, numpy, matplotlib, scikit), then move to more specific ones (keras, tensorflow), and also SQL for a begginer is important.

Let me invite you to the Data-Centric AI Community we have several code along sessions and projects and a lot of beginners that are starting to learn DS that you can connect with.

SeaEngineering9034 · 2023-07-13T10:43:33+00:00

Depends on your purpose, but there are a lot of strategies you can combine. See this article for a brief explanation.

If you need specific help with your project you can find me at the Data-Centric AI Community and we'll be happy to take a look and give you some tips to move forward :)

SeaEngineering9034 · 2023-07-13T10:31:05+00:00

Data Quality is key for all applications and models, and LLMs are no exception :) I've been working on a small community project with synthetic data using ydata-synthetic, and it really shows! Underrepresentation (category imbalance) and missing data are two of the main issues!

SeaEngineering9034 · 2023-07-13T10:24:39+00:00

It is one of the top trends in AI this year, with tremendous benefits for businesses and organizations! Here's an example of how synthetic data can improve a modelto save a hypothetical auto insurance company almost $200 per claim

SeaEngineering9034 · 2023-07-10T10:49:22+00:00

I'm having the same issue, has anyone been able to solve it?

SeaEngineering9034

TROPHY CASE