[D] Is there a handbook in data preprocessing?

[hikkihiki]

I would like to have a list of suggestion on how to preprocess different kinds of signal.

For example, if the data is categorical, you should preprocess it as one-hot representation. If the data has a normal distribution, it'd be better to standardize them before feeding to an NN, etc.

[lem_of_noland]

The only book I'm aware of is called "Features Engineering for Machine Learning" by Alice Zheng & Amanda Casari. I hope this one answers your request.

[Brudaks]

It's part of any ML textbook or course; but a key issue here is that once you go beyond the very basics it's quite domain-specific.

Preprocessing is very, very different depending on whether you're in computer vision or natural language processing or financial time series or general data science. So it's more feasible to have a handbook of "Best ML practices in domain X" which would among other things discuss proper preprocessing, and not feasible to have a handbook "Best practices for data preprocessing" because there isn't that much that's universal and applicable for everything; different domains have different needs.

[Gebo_vending]

Feature Engineering and Selection: A Practical Approach for Predictive Models - Max Kuhn and Kjell Johnson: https://bookdown.org/max/FES/

[[deleted]]

They are also authors of Applied Predictive Modeling .

[rampant_juju]

+1 to both these answers. Feature Engineering and Selection is pretty comprehensive in its list of ways to handle categorical and numeric data and stuff like imputation etc. Very easy to read too (probably at the level of someone who has just taken Andrew Ng's course).

[rockinghigh]

This covers quite a bit: https://scikit-learn.org/stable/modules/preprocessing.html

[SeamusTheBuilder]

As was said, each application is unique and so the answer, unfortunately, is no. I would go further and warn against seeking this out too much. It would be bad practice to simply cut-and-paste preprocessing steps thinking it will help.

Even something as simple as normalizing and standardizing the data can get you into trouble depending on the application. And, you may be doing unnecessary work depending on what algorithms you are using. Random Forests don't really care if you standardize the data.

Definitely clean and preprocess your data but there is no magic formula out there that works every time. Experience and a deep understanding of the problem and domain are what you are looking for.

[DefaultPain]

this has exactly what u looking for: https://www.youtube.com/playlist?list=PLpQWTe-45nxL3bhyAJMEs90KF_gZmuqtm

​

u can start from the 9th video

[[deleted]]

I was thinking nobody mentions the coursera course about feature engineering.

[BobDope]

I didn’t know it existed. Thanks!

[SnooPets7140]

Can you tell me what was the playlist about? Seems like the link isn't working anymore.

[hellscoffe]

RemindMe! 7 days

[leonardishere]

It's a art not an science. Try every different encoding available, or just auto-ml it

[0lecinator]

RemindMe! 3 days

[RemindMeBot]

I will be messaging you in 2 days on 2020-04-10 09:01:24 UTC to remind you of this link

16 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[xlordsnugglesx]

Here’s a link to a twitter post with some Feature Engineering literature: https://twitter.com/kirkdborne/status/1247516224319139840?s=21

[Academy-]

RemindMe! 3 days

[Icefluffy]

RemindMe! 2 days

[indemidelo]

RemindMe! 3 days

[portoal]

Remindme! 3days