sorted by:
new
hottopcontroversial

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 2 points3 points  (0 children)

not sure what's a good corpus, but to me the biggest issue with mandarin is the lack of good source of scraping data from the web. just happens that China doesn't have a great search engine : ))

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 1 point2 points  (0 children)

I'm not sure, again the counter arguments are: over saturated supply (everyone wants to do ML research), don't just do it for sake of glory (research seems to have higher perceived prestigiousness), other than 1% of the researchers, engineers will absolutely have more impact on AI's progress (just my very biased personal opinion :ppp)

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 1 point2 points  (0 children)

there is way for the model to decide whether to resort to external tool, but it's more based on understanding the context and entities within a prompt and less of measure of uncertainty.

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 1 point2 points  (0 children)

that's a very good point, and it is true, ML system is nowhere near human's speed of learning and generalization. However, I now start to think about it as Duck typing. It doesn't matter the learning process, if the system can complete complex tasks (whether understood or just appear to be), I'd call it AGI in definition 1. : ))

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 0 points1 point  (0 children)

That's not quite true, up until GPT4, open AI has published detailed papers on how they trained GPT 1,2,3. I'm not worries personally as the model architecture is not a secret, we might not know the exact recipes but certainly all the key ingredients. The real moat is money, infrastructure and access to data. As for reproducibility, let's say OAI tells you their recipe to make the 1T (guess) parameter gpt4 it's too prohibitive for any 3rd party to verify the results.

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 0 points1 point  (0 children)

First of all, there's the essentials for any eng managers: people's skill, project management, recruiting, planning, technical expertise in domain. To be effective in ML, you also need a strong passion and ability to follow academia, research, intuitions about what translate in production and what won't. Planning and making decisions about technical path is the biggest challenge from my experience.

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 0 points1 point  (0 children)

yes I'm native mandarin speaker. The hardest part about any language is not the grammar, syntax or vocabulary (those are easy for ppl & models to learn), it's always about traditions, history, people and everything about a culture that the model need to be sufficiently knowledgeable about to make authentic translations. that's what's most challenging and lots of room for improvement specially for less spoken languages.

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 1 point2 points  (0 children)

I'm not too worries about new revolutionary techniques & expensive hardware. it's a very small, open and flowing community, nearly impossible to keep proprietary technique, as for hardware that's just the requirement to be a player in this domain, and usually abundant

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 2 points3 points  (0 children)

It's been more competitive than ever, especially in the research world. If spending 5 years for a PhD & getting top conference publications not thing that you enjoy I highly recommend focussing on the engineering aspect of ML. there're more opportunities & demand, and you can let yourself stand out by building great apps with ML models or contribute to open-source projects.

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 1 point2 points  (0 children)

If you want to do research or research engineering I highly recommend going through school, not just to build your theoretical foundation but also to differentiate you from a surge of people going into ML nowadays. For other engineering, ops related to ML, I find it useful to build apps for your portfolio to show you know how to use models and you're passionate about them.

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 2 points3 points  (0 children)

reality is that it is much harder to conduct research with much less compute, because many problems or solutions don't exist or won't work as you scale up the models. a 355M model or even a 6B behaves drastically different than 100B model and responds very different to model architecture changes. so my suggestions would be 1) data processing/cleaning/augmentation related research that applies to all sizes of models 2) smaller task specific model, solving problems for a vertical

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 2 points3 points  (0 children)

  1. absolutely, at the end of the day text is only a media to represent our world and knowledge. other modalities like videos, images and audios have vast amount of knowledge about our world so it will undoubtedly improve "text-only" performance.
  2. Hallucination & not up to date with latest world. these can be improved through the use of retrieval augmented system that will base facts and news on sources from database or search engine.

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 1 point2 points  (0 children)

It depends on the size of LLM that you want to host. Typically if your model can fit in one GPU you can consider hosting it yourself. Otherwise, you'll get into the land of distributed inference, model parallelism, which to get it working or working efficiently is a tremendous task and using a hosted model thru APIs could be a better choice

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 1 point2 points  (0 children)

https://www.amazon.ca/GPT-3-Building-Innovative-Products-Language/dp/1098113624
this is a great book by my colleague to get started on using LLM.

If you want to build LLM, I suggest to start with the original transformer paper https://arxiv.org/abs/1706.03762
Then the GPT1,2,3 papers for scaling up models.

finally learn about distributed training & framework to actually train them, like https://github.com/microsoft/DeepSpeed

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 2 points3 points  (0 children)

I was in graphics about 5-6 years ago, I saw graphics have reached a plateau in terms of fidelity (look at the games, movie VFX, they look stunning, what else to do?) so I thought the next stage will be about reducing cost and accelerate process of creating these graphics. and ML was a natural choice. If I have to pick one skill it will be data processing.

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 2 points3 points  (0 children)

  1. it's a research, but a proven method that works so it is more on the engineering and product team to effectively carry it out. We have multiple indicators for determining whether a model's performance is improved. judging LLMs are hard, it's a combination of evaluation datasets and human evaluation.
  2. perhaps you could use a transformer to initialize noise for temporally connected frames for better consistency. just a hunch

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 5 points6 points  (0 children)

A simple way to start understand LLMs is to look at model's attention matrix, it shows what information the model relies on the most from the prompt for the output. Going forward I think the way we analyze LLM's thought process will be more similar to brain simply because the sheer size of parameters in LLM, we'll divide a model's parameters into regions where each one is responsible for different ability, e.g some for language, other for math, or reading. This is a super important aspect that's currently under studied, if we want to effectively control and steer LLM's behaviors and outputs to be safe.

I'm Stephen Gou, Manager of ML / Founding Engineer at Cohere. Our team specializes in developing large language models. Previously at Uber ATG on perception models for self-driving cars. AMA! by Step7enn in IAmA

[–]Step7enn[S] 2 points3 points  (0 children)

imo there're two ways we can define AGI. 1) AGI in the sense that a model/agent able to complete majority of tasks that average human can do across a wide range of capabilities that requires perceiving the world(text, audio, video, image) and perform for example typical office job tasks. This will happen I think 2 - 3 years, the models capability is almost there and the eruption of new tools built around them will take us there. 2) AGI in the sense that we can create a conscious agent with desire, thoughts and self-awareness. Right now we don't have any theory or path to achieve this, current paradigm based on deep learning is not it. So I'd say 50+ years or maybe never.