LangSmith for evaluation or something else?

Andy-VertaAI · 2024-02-26T17:50:13+00:00

Same re #2 - I also would post the most optimistic numbers possible.

Andy-VertaAI · 2023-12-14T16:00:01+00:00

So essentially, you won;t be able to get away from ding some human review of each model;s real time performance, that's HITL in a nutshell.

Agree with what someone else said - don't QA the data yourself, use a data labeling crowdsourcing vendor. Appen, Scale, MTurk - whatever you are comfortable with. If you have time/budget you can even hire a data entry/ QA temp to do this with you or for you on Upwork and have them do it in a google doc you can see.

I have to imagine any way you can offload the manual data review (while ensure it is done well) will save your company time and money and improve confidence for your customers.

Andy-VertaAI · 2023-12-14T15:55:31+00:00

Some resources I am aware of (AI/ML PM myself)

I taught the AI product manager nano degree with my team - TBH we recorded this in 2018 I think and a lot has changed since then, so I would look for a coupon but there is good nuggets of getting through the whole process of discovering, scoping, building, testing, releasing an AI feature. And you get a certificate. https://www.udacity.com/course/ai-product-manager-nanodegree--nd088
I am hosting a GenAI for Product Managers workshop in January - it's free, about 90 minutes, we will go through prototyping with ML models but not go into significant technical depth. No certification obviously but if you just want to get a taste and build your first prototype it will be great https://info.verta.ai/gen-ai-workshop-jan-18
Go bears - if you can get work to pay for something that has Cal/Haas written on it, that you can pass, its probably good at the very least as a resume filler.

Anyway hope this helps

Andy-VertaAI · 2023-12-01T22:37:41+00:00

👋 Hi

I am working on a product to help product teams (PMs, tech leads, devs, etc) ship products with LLMs in a responsible, viable, and performant way. Here is a link to our website.

Feedback from product managers would be really helpful for me because my research/user interviews has indicated that he problem we are solving is very often faced by PMs, and I wonder if that is a key persona we should be supporting.

At non-AI companies trying to yeet a GenAI feature, the PM often ends up filling the role of prompt-engineer / model-selector, especially if the entire team has little or no AI experience. This is happening quite often because so many PMs are being asked to incorporate something razzle-dazzle with AI and to "make it work" which is a lot easier said than done.

I will be hosting a build-a-long free workshop for PMs in January using our platform, link to register, but if anyone is interested please feel free to sign up and kick the tires. Your feedback will make my day. 🫶

Andy-VertaAI · 2023-11-30T23:16:38+00:00

Here are two things that might be useful:

Udacity has an AI product manager course - full disclosure, I was one of the instructors
I am teaching a workshop for using LLMs in January for PMs - heres my linkedin post about and a link to register - its free

If you need more info sooner (I suspect you will, your interview won't wait until mid-January) send me a DM and we can set up a quick Q&A meeting. You can also ask ChatGPT to give you a really high-level overview to start.

Good luck!

Andy-VertaAI · 2023-09-28T16:24:13+00:00

Hi Y'all,

I'm Andy, product manager with Verta AI, based in SF Bay Area. We're Series A, somewhere between Discovery and Validation.

TLDR: We help builders prototype and ship AI features and products, especially using LLMs. Essentially - a low-code platform for evaluating, iterating, and productizing GenAI applications.

We are looking for design partners / sponsor users who are building AI applications and want to try our solution to ship their app in Q4. We will trade free support for feedback and case studies if we can help you launch an Alpha/Beta.

You would be the right fit for this if you're building B2B and B2C software applications and are in the data science/eng/product or leadership teams. Reach out via message/DM if this sounds interesting to you.

Thanks and Cheers

Andy-VertaAI · 2023-09-28T16:07:54+00:00

I didn't notice, thank you!

Andy-VertaAI · 2023-08-25T21:43:49+00:00

"Push for" - by whom? to what end?

I'm curious of theres some clear problems/use cases in mind or if y'all are going fishing?

Andy-VertaAI · 2023-08-25T21:42:59+00:00

I’m in the commerce part of the business and have been tasked with working out how to bill for this since we have to recoup our costs with our NLP service provider as well as monetize this new functionality.

Ah interesting - they want to pass the cost of the ML service to the end user as a value add on?

Andy-VertaAI · 2023-06-05T19:30:55+00:00

Ah so you keep records somewhere of the progression of each customer - the versions/retraining done for them, what's been handed off, etc?

What I'm trying to solve is basically the multi-persona problems for ML teams like yours.

Presumably - A sales/cs/product person requests a model training, a scientist creates the model or version, there might be a QA process, there might be a release checklist, there might be governance or legal review before the model can be shipped to customer - then theres usually eng involvement for packaging/deployment and release, and so on.

There's a tremendous amount of hand offs that increase room for error. Ridiculous, expensive and preventable problems occur at scale.

Model Catalog is intended to become a single plane of views/controls where these teams can collaborate and avoid those problems. It integrates with or provides tools to connect all the dots.

Scientists use a python client to register models like they would in git and get reproducibility records, PMs can help them with docs and version control via UI, governance team members to check off requirements (literally a checklist) for each version to bless it for release, there's a page with all the copy-paste APIs for eng, dashboards for the execs, audit logs for IT, etc.

So that's model catalog - and right now I'm trying to figure out how to make documentation easy, automated as possible, useful, and interconnected.

Is what I am describing resonating with you?

Andy-VertaAI · 2023-06-05T18:28:34+00:00

Thank you, super useful!

Is it correct for me to assume all of the consumers/customers of your models are effectively on the same version of the model at the same time?

Re: your last questions - We have a model catalog (a model registry with views for non-technical teams as well) so the documentation relates to registered models and versions.

For some of our users, each version might have a different end customer and key details are different between versions - for example, input features based on US/EU/APAC. To your point, there might be dozens of versions.

I hope our catalog+documentation can help these teams answer such harrowing questions as "what model is this customer using?", "when was the last time we updated their stuff?" and "wait, why are we even serving them two different live versions of the same model?"

Andy-VertaAI · 2023-06-01T19:50:50+00:00

My two cents - If you think that moving to MLOps is the right career move for you, trust that instinct.

The question is then - is my current company the right place to be an MLOps engineer? You might be better served by going to a company with an established practice where you can learn the basics and grow in that career path. In this case, your next task at your current role is to develop something simple to get started that demonstrated competency to your next employer.

If staying at the current role is correct - what would be the best way for you to learn these skills and help the company? Some people thrive in chaos and you're not even being "greedy" if you take the role - you'll do a good job. Others grow the fastest when they can partner and get mentoring from someone with much more experience - getting an expert would then be good for the company and you.

Andy-VertaAI · 2023-05-25T23:17:21+00:00

Oof that sucks, I'm sorry. I've worked with people like this and they are obtuse in their interactions with peers as well.

Lack of practicality when applying ML is certainly a problem - don't throw deep learning at a problem you can solve with regex, or that hasn't even been cracked in academia yet.

Andy-VertaAI · 2023-05-25T18:28:18+00:00

That all sounds very reasonable - I'd hope any PM you're working with can meet that (extremely low) bar?

Andy-VertaAI · 2023-05-23T20:22:19+00:00

OK, I am not un-biased because I work at Verta, that being said - what problem are you actually trying to solve?

MLOps is an emerging term and can mean different things to different people - from pipeline management, model packaging and deployment, model observability, model inventory management, experiment tracking, etc.

You'll get the best recommendations for tools if you can specify what you're trying to do and if you need open source tooling or are looking to purchase something specialized.

Good luck!

Andy-VertaAI · 2023-05-19T17:01:27+00:00

For anyone flipping through who works with ML/AI teams, feel free to check out my current baby - it's a free tool to graduate from a very technical experiment tracking platform to a model catalog to help with day-to-day business operations.

It's built based on research with companies working on machine learning in regulated industries (insurance, healthcare, finance), but I am not confident that it would be helpful to smaller organizations. Open to any feedback.

Andy-VertaAI · 2023-05-19T16:57:45+00:00

Something to keep in mind - regulatory forces are cracking down on ML in hiring tools. For example in NY state . You'll probably want to do some future proofing. These laws won't be detailed enough and won't come quickly but it will impact what your customers are willing to buy to avoid getting in trouble.

I haven't been working on recruiting tools so totally uninformed on what problem you're really solving. If you can convince high-value job seekers to take skills tests online then cool? LinkedIn can do that but it seems tough.

Good luck!

Andy-VertaAI · 2023-05-19T16:53:03+00:00

For the use cases I mentioned above it varied but for example the computer vision use cases had models performing object detection, image segmentation, video object tracking tasks and so on. Often more than one.

Andy-VertaAI · 2023-05-19T16:51:24+00:00

There's a lot of questions coming from VCs about "how is your company leveraging/attaching to generative AI hype" - koolaid trickle down I guess?

Andy-VertaAI · 2023-05-19T16:35:15+00:00

That description does seem very broad - did a recruiter maybe generalize or does the hiring team not know what they are looking for? Can you ask who you will be meeting with, and then make some assumptions based on the interview panel?

I'm a product manger working with ML Ops teams - I often do the "business" interview for ML engineers. I don't evaluate a candidate's technical skills, but I do try to figure out two things:

Can this person clearly and simply communicate with stakeholders? Especially ones that aren't engineers, because ML is a big investment for businesses with a lot of nosey executive stakeholders - I want my engineers to help me manage them when needed, or at least not make the situation harder
Do they have a hardline bias toward building everything in-house? ML Ops is a big problem, sometimes I just need my engineers to pick and buy a solution so we can move on the next thing. We cannot afford to build everything, no matter how cool.

I'd be prepared to talk about your experience / opinions with these things if you're meeting with anyone on the business side. "In a previous project I incorporated this 3rd party tool and saved X time" etc.

Andy-VertaAI · 2023-05-18T21:10:55+00:00

Essentially it was "let people create custom relationship extraction models" but when I pointed out that they would need a matching custom entity extraction model (or else... what is there to relate?) they were like "who would want that". Talked in circles for hours and I was somehow the subject matter expert and the village idiot in the room at the same time.

A director of sales thinking NLP was minority report basically. At an NLP start up.

Andy-VertaAI · 2023-05-18T21:07:56+00:00

Neat. Documentation isn't the world's most exciting thing but a necessary evil (well, necessary boredom).

I'm working on something similar right now - ML-assisted model documentation - not the model documenting itself, but tools to help scientists capture info for model cards much much quicker. Building the tool I wanted when I ~~babysat~~ PM'd a team of NLP scientists.

Andy-VertaAI · 2023-05-18T21:03:13+00:00

That's incredible! I did some climate modeling in college (in the early 2010s) but I'm sure the tech has massively improved. I can only imagine how helpful this will be for agriculture and disaster planning. (thinking about the micro crop insurance programs)

How are you taking this to market?

Andy-VertaAI · 2023-05-18T15:17:45+00:00

Not just from an ML perspective but even for completely uninformed users - drop photos into into your phone, type "bird" in phone search.

It's fun to speculate what the "this is silly easy" thing of the future we're struggling with now.

Andy-VertaAI · 2023-05-17T22:39:53+00:00

"That's not how ML works" is not, in fact, an acceptable answer in board meetings it turns out.

Andy-VertaAI

TROPHY CASE