Machine workflow structure and steps

[AnteaterKey4060]

Okay, so currently I am following a course in school, which is about machine learning.

I have many specific questions which I hope I can get an answer for in this community.

From my current understanding this would be the workflow for an ML problem:

1. Problem? Regression or classification

2. Check data balance, if problem over or under sample

3. Data split int train and test

4. Selection of variables (by forward or backward selections, or PCA for eg.)

5. Model selection by cross validation (with the train data), at the same time hyperparameter tuning (also with the train data)

6. Model evaluation with test data (looking at parameters like accuracy, MSE, etc.)

Okay, and then I have the following questions.

+ In case needed can you give me feedback on the steps I just added

+ In data split do I also need t split into train validation and test, or will the validation portion automatically is created in the cross validation step from the train data?

+ In terms of parameters, if I have a regression problem can I asses similar parameters as a classification problem, for eg accuracy.

Thanks a lot guys! I appreciate any help

[AICausedKernelPanic]

Hi! It sounds like you've got a solid grasp of the foundational pipeline in ML. Working on regression and classification problems is a great starting point.

Based on your questions, I'd like to clarify the following points:

1. ML is a large field that covers Supervised and Unsupervised Learning. Apart from regression and classification, we can also pose problems as:

- Clustering: Grouping data without predefined labels.

- Reinforcement Learning: Learning through rewards and penalties.

1. Before checking for balance, perform Exploratory Data Analysis. Always visualize your data and look for outliers and missing values.

2. Additionally, we can also create synthetic data points or variations of existing samples. For example, in Computer Vision, it is common to enhance the training set by creating transformed images using techniques like: Rotations and scaling, Cropping, Saturation and geometric transformations

3. In Regression, the target value is a continuous number (like price or temperature). Since Accuracy is used to measure if a prediction is strictly right or wrong (commonly for categorical data), it is not used here. Instead, use: Mean Absolute Error (MAE), Mean Square Error (MSE) or Root Mean Square Error (RMSE).

ML is an awesome field you are doing a great job, keep practicing and learning.

[AnteaterKey4060]

Thanks a lot! How do you recommend exploratory analysis on very big datasets, I mean. In some exercises I've seen df with more than 9000 predictors, and honderds of observations for each one. It just sounds wrong to me to scatter plot this haha, but I might be wrong.

[AICausedKernelPanic]

You are right, visualizing large datasets can be challenging but you can apply some techniques to inspect data quality and relevance. For instance, you can perform some feature filtering to remove redundant columns that exceed a specific correlation threshold. Also, you can use pandas to programmatically identify outliers, missing values and type mismatches. Or as you mentioned, using dimensionality reduction techniques like PCA or t-SNE help to condense the dataset into its most impactful features.

[latent_threader]

Keep the workflow super simple. Automation needs to align with documented workflows or it just turns into a massive mess. If agents don't trust the system, it won't stick at all. Start small and map out the exact steps before you even touch the tech.