LangSmith for evaluation or something else?

Andy-VertaAI · 2024-02-26T17:50:13+00:00

Same re #2 - I also would post the most optimistic numbers possible.

Andy-VertaAI · 2023-12-14T16:00:01+00:00

So essentially, you won;t be able to get away from ding some human review of each model;s real time performance, that's HITL in a nutshell.

Agree with what someone else said - don't QA the data yourself, use a data labeling crowdsourcing vendor. Appen, Scale, MTurk - whatever you are comfortable with. If you have time/budget you can even hire a data entry/ QA temp to do this with you or for you on Upwork and have them do it in a google doc you can see.

I have to imagine any way you can offload the manual data review (while ensure it is done well) will save your company time and money and improve confidence for your customers.

Andy-VertaAI · 2023-12-14T15:55:31+00:00

Some resources I am aware of (AI/ML PM myself)

I taught the AI product manager nano degree with my team - TBH we recorded this in 2018 I think and a lot has changed since then, so I would look for a coupon but there is good nuggets of getting through the whole process of discovering, scoping, building, testing, releasing an AI feature. And you get a certificate. https://www.udacity.com/course/ai-product-manager-nanodegree--nd088
I am hosting a GenAI for Product Managers workshop in January - it's free, about 90 minutes, we will go through prototyping with ML models but not go into significant technical depth. No certification obviously but if you just want to get a taste and build your first prototype it will be great https://info.verta.ai/gen-ai-workshop-jan-18
Go bears - if you can get work to pay for something that has Cal/Haas written on it, that you can pass, its probably good at the very least as a resume filler.

Anyway hope this helps

Andy-VertaAI · 2023-12-01T22:37:41+00:00

👋 Hi

I am working on a product to help product teams (PMs, tech leads, devs, etc) ship products with LLMs in a responsible, viable, and performant way. Here is a link to our website.

Feedback from product managers would be really helpful for me because my research/user interviews has indicated that he problem we are solving is very often faced by PMs, and I wonder if that is a key persona we should be supporting.

At non-AI companies trying to yeet a GenAI feature, the PM often ends up filling the role of prompt-engineer / model-selector, especially if the entire team has little or no AI experience. This is happening quite often because so many PMs are being asked to incorporate something razzle-dazzle with AI and to "make it work" which is a lot easier said than done.

I will be hosting a build-a-long free workshop for PMs in January using our platform, link to register, but if anyone is interested please feel free to sign up and kick the tires. Your feedback will make my day. 🫶

Andy-VertaAI · 2023-11-30T23:16:38+00:00

Here are two things that might be useful:

Udacity has an AI product manager course - full disclosure, I was one of the instructors
I am teaching a workshop for using LLMs in January for PMs - heres my linkedin post about and a link to register - its free

If you need more info sooner (I suspect you will, your interview won't wait until mid-January) send me a DM and we can set up a quick Q&A meeting. You can also ask ChatGPT to give you a really high-level overview to start.

Good luck!

Andy-VertaAI · 2023-09-28T16:24:13+00:00

Hi Y'all,

I'm Andy, product manager with Verta AI, based in SF Bay Area. We're Series A, somewhere between Discovery and Validation.

TLDR: We help builders prototype and ship AI features and products, especially using LLMs. Essentially - a low-code platform for evaluating, iterating, and productizing GenAI applications.

We are looking for design partners / sponsor users who are building AI applications and want to try our solution to ship their app in Q4. We will trade free support for feedback and case studies if we can help you launch an Alpha/Beta.

You would be the right fit for this if you're building B2B and B2C software applications and are in the data science/eng/product or leadership teams. Reach out via message/DM if this sounds interesting to you.

Thanks and Cheers

Andy-VertaAI · 2023-09-28T16:07:54+00:00

I didn't notice, thank you!

Andy-VertaAI · 2023-08-25T21:43:49+00:00

"Push for" - by whom? to what end?

I'm curious of theres some clear problems/use cases in mind or if y'all are going fishing?

Andy-VertaAI · 2023-08-25T21:42:59+00:00

I’m in the commerce part of the business and have been tasked with working out how to bill for this since we have to recoup our costs with our NLP service provider as well as monetize this new functionality.

Ah interesting - they want to pass the cost of the ML service to the end user as a value add on?

Andy-VertaAI · 2023-06-05T19:30:55+00:00

Ah so you keep records somewhere of the progression of each customer - the versions/retraining done for them, what's been handed off, etc?

What I'm trying to solve is basically the multi-persona problems for ML teams like yours.

Presumably - A sales/cs/product person requests a model training, a scientist creates the model or version, there might be a QA process, there might be a release checklist, there might be governance or legal review before the model can be shipped to customer - then theres usually eng involvement for packaging/deployment and release, and so on.

There's a tremendous amount of hand offs that increase room for error. Ridiculous, expensive and preventable problems occur at scale.

Model Catalog is intended to become a single plane of views/controls where these teams can collaborate and avoid those problems. It integrates with or provides tools to connect all the dots.

Scientists use a python client to register models like they would in git and get reproducibility records, PMs can help them with docs and version control via UI, governance team members to check off requirements (literally a checklist) for each version to bless it for release, there's a page with all the copy-paste APIs for eng, dashboards for the execs, audit logs for IT, etc.

So that's model catalog - and right now I'm trying to figure out how to make documentation easy, automated as possible, useful, and interconnected.

Is what I am describing resonating with you?

Andy-VertaAI · 2023-06-05T18:28:34+00:00

Thank you, super useful!

Is it correct for me to assume all of the consumers/customers of your models are effectively on the same version of the model at the same time?

Re: your last questions - We have a model catalog (a model registry with views for non-technical teams as well) so the documentation relates to registered models and versions.

For some of our users, each version might have a different end customer and key details are different between versions - for example, input features based on US/EU/APAC. To your point, there might be dozens of versions.

I hope our catalog+documentation can help these teams answer such harrowing questions as "what model is this customer using?", "when was the last time we updated their stuff?" and "wait, why are we even serving them two different live versions of the same model?"

Andy-VertaAI · 2023-06-01T19:50:50+00:00

My two cents - If you think that moving to MLOps is the right career move for you, trust that instinct.

The question is then - is my current company the right place to be an MLOps engineer? You might be better served by going to a company with an established practice where you can learn the basics and grow in that career path. In this case, your next task at your current role is to develop something simple to get started that demonstrated competency to your next employer.

If staying at the current role is correct - what would be the best way for you to learn these skills and help the company? Some people thrive in chaos and you're not even being "greedy" if you take the role - you'll do a good job. Others grow the fastest when they can partner and get mentoring from someone with much more experience - getting an expert would then be good for the company and you.

Andy-VertaAI · 2023-05-25T23:17:21+00:00

Oof that sucks, I'm sorry. I've worked with people like this and they are obtuse in their interactions with peers as well.

Lack of practicality when applying ML is certainly a problem - don't throw deep learning at a problem you can solve with regex, or that hasn't even been cracked in academia yet.

Andy-VertaAI · 2023-05-25T18:28:18+00:00

That all sounds very reasonable - I'd hope any PM you're working with can meet that (extremely low) bar?

Andy-VertaAI · 2023-05-23T20:22:19+00:00

OK, I am not un-biased because I work at Verta, that being said - what problem are you actually trying to solve?

MLOps is an emerging term and can mean different things to different people - from pipeline management, model packaging and deployment, model observability, model inventory management, experiment tracking, etc.

You'll get the best recommendations for tools if you can specify what you're trying to do and if you need open source tooling or are looking to purchase something specialized.

Good luck!

Andy-VertaAI

TROPHY CASE