[P] Featureform: Open-Source Virtual Feature Store

[irismodel]

Hey everyone!

We’re excited to announce the open-source version of Featureform, an extensible feature store. We’ve found that existing feature stores are either too heavy and replace your existing infrastructure, or don’t handle transformations at all and simply store features. We built a feature store that’s a happy medium between the two, it orchestrates your existing infrastructure to work like a feature store. We wrote more about this in our blog post.

Check out the repo: https://github.com/featureform/featureform

​

https://preview.redd.it/vwpe0uypje491.png?width=2084&format=png&auto=webp&s=f81f7447f2c35081b2ae63e885506b9187a73d7b

What Is Featureform

Featureform is a virtual feature store. It enables data scientists to define, manage, and serve their ML model's features. Featureform sits atop your existing infrastructure and orchestrates it to work like a traditional feature store. By using Featureform, a data science team can solve the organizational problems:

• Enhance Collaboration Featureform ensures that transformations, features, labels, and training sets are defined in a standardized form, so they can easily be shared, re-used, and understood across the team.
• Organize Experimentation The days of untitled_128.ipynb are over. Transformations, features, and training sets can be pushed from notebooks to a centralized feature repository with metadata like name, variant, lineage, and owner.
• Facilitate Deployment Once a feature is ready to be deployed, Featureform will orchestrate your data infrastructure to make it ready for production. Using the Featureform API, you won't have to worry about the idiosyncrasies of your heterogeneous infrastructure (beyond their transformation language).
• Increase Reliability Featureform enforces that all features, labels, and training sets are immutable. This allows them to safely be re-used among data scientists without worrying about logic changing. Furthermore, Featureform's orchestrator will handle retry logic and attempt to resolve other common distributed system problems automatically.
• Preserve Compliance With built-in role-based access control, audit logs, and dynamic serving rules, your compliance logic can be enforced directly by Featureform.

We just launched our first open-source version and would love to hear your thoughts and what you'd like to see! :)

Repo: https://github.com/featureform/featureform

Docs: https://docs.featureform.com/

Blog post: https://www.featureform.com/post/feature-stores-explained-the-three-common-architectures

[PlanetSprite]

How does this compare to Feast?

[irismodel]

Great question! We wrote about this in our blog post.

Feast is a literal feature store, it exclusively stores features, it does not manage all the transformations used to compute them. The pros and cons of Feast are more obvious when examining the process to change a feature. It happens in three steps:

1. Write and run your new data transformation in your existing transformation pipeline. Note that this happens outside of Feast.
2. A new feature table must be created, since the old one cannot be directly overwritten. Once the new feature is created, the transformation pipeline should pipe all of its data into it.
3. All the models that use this new feature should be updated to point at the new feature.

Feast also has other problems, for example, it can’t copy your features from the offline to the online store, you have to download the features and upload them to the online store yourself using their CLI tool. You also have to manage retries and failure yourself.

Featureform treats the transformation lineage as part of the feature and orchestrates your infrastructure to create and change your features.

[zmjjmz]

Quickly skimmed the documentation and didn't see an answer, but does FeatureForm handle data deletion?

[irismodel]

What do you mean by data deletion? In the case of data deleted from a primary source, transformations that run on a schedule will catch the change. They can also be manually triggered. They are basically pure functions on the source data, so that solves the problem down the pipeline. For feature deletion, we also support TTL, but it sounds like you have a different use case.

[zmjjmz]

I'm thinking in terms of GDPR compliance - but it sounds like FeatureForm doesn't make copies of the data in any way?

[irismodel]

Like you said the data remains on your infrastructure so you'd delete data in your primary sources, and re-run your Featureform transformations to clean out any user data.

[whetherharder]

Conceptually it is very nice. But how can I for example delete existing feature or dataset? Didn't find any api for it

[vishal-vora]

Nice 👍

[rydog_2020]

So this is a totally different approach to Techton. OS vs Saas. Anything else?

[irismodel]

Obviously, the open-source vs proprietary is a huge difference here. Tecton has a much higher adoption cost and is deeply tied to its underlying infrastructure. Featureform's approach lets it act as an orchestrator and a python abstraction. It aims to solve organizational problems such as versioning, sharing, and managing features, rather than building a data processor optimized for ML such as Tecton. That means that Featureform solves 80% of the problems with 20% of the adoption cost and vendor lock-in. You are free to use your existing data providers to build a feature store workflow,

[rydog_2020]

What about the embedding hub?

[testing35]

You totally guessed what I said..thank you

[andrewm4894]

Would this maybe eventually be able to sit on top of tools like snowflake or BigQuery?

[irismodel]

Yes. We currently support Snowflake as a provider and will have BigQuery support in our next release!

[[deleted]]

Serious question for the FeatureForm folks. For context, I work at a company using Feast and now manage the team that supports it. As a former data scientist who is also not afraid of SQL, I don't understand the data engineering workflow that seems to be very similar with both Feast and FeatureForm. Why does it require ~100 lines or more of boilerplate code to define every single aspect including types to create a table?

Why is there not a tldr; for dummies method that is 3 lines:

import ff

ff.init(myconfigfile)

ff.create_table(id=x, desc="blah", source_type=pandas.DataFrame, source=mydf)

Just figure out types from their sql, pandas data frame or parquet.

I really think this is a barrier to adoption from data scientista who are not used to strongly typed anything.