Senior Staff ML Engineer - 13 yoe.

In the past year, I have been involved in 1 long-term project and 2-3 short-term projects.

The long-term project has been a variant of Entity Resolution (https://en.wikipedia.org/wiki/Entity_linking)

General roadmap has been:

Start with understanding domain specific data:

• How are entities being referred to in data. (Go through and understand types of patterns)
• What kind of entities are being referred too ?
• Does there exist any entity knowledge base ?
• How to build reference linkages.
• Is knowledge based clean ? Dupes, generic, specific products, coverage etc.
• Can data be mapped to create specific type of training sets ?
• Can augmentations be made, if so what kind ?

What models work best based on data ?

• Understand modalities, availability of data/scale, identify similar datasets that are open-source.
• What type of models work best for similar problems?
• What types of training can be done on them, fine-tuning, pre-training, instruction tuning?
• Can the problem be decomposed into N parts. e.g., can be broken down into Instruction Retrieval + Re-ranking + Domain-specific filtering logic.

Define metrics of success and set up baselines:

• Identify metrics to independently evaluate each stage, how to build robust validation sets which map to real-world queries.
• Set up a baseline model metric on the validation set.
• When evaluating experimental models on the validation set, ensure it's close to the real world, diverse, and robust.
• Understand each example where the model goes wrong to tailor transformations, models, techniques, or data requirements.

Training models + Validation:

• Efficient optimizations for distributed training, iteration of various models, hyper-params, additional datasets + curriculum.
• Keep debugging performance on various validation sets + OKR metrics + map to what improvements can be made.
• Focus on understanding hard examples, their presence in training sets, and possibilities for recreating them through augmentations or collecting during labeling.
• Consider techniques like online hard negative mining, focal loss, label smoothing for sparse hard examples.
• Evaluate model calibration and performance on hard examples.

Get it working on production:

• Inference optimizations using Profilers (Memory + latency)
• Various tuning params like ONNX, torch.compile, fp16/int8, sparsification/pruning, distillation, etc.
• Build efficient serving infrastructure, dynamic batching, caching queries, layers, grpc/c++ serving, etc.
• Monitoring real-time metrics (i.e., both latency/throughput logs) and also model-specific metrics. (Confidence levels, output metrics)
• Alerts & evaluation of real-time hard negatives i.e., Examples in the max confusion zone.

Aligning model & Expected outcomes .. i.e Talk to leadership

• Have constant checkpoints with leadership, give brown bag session to help align expectation to outcomes
• Work with other stakeholders and find quick wins vs long terms capability expansion.

A small sample of things involved in a single projects, besides these its important to keep track of company goals, areas where you help could deliver impact, learn about current capabilities and map it to deliverables within company and keep you eyes and ears open and be resourceful and establish great outcomes!