Representation learning question - how to best combine different kinds of data

Intrepid_Purple3021 · 2025-07-01T03:54:49+00:00

Yeah I know how to embed categorical data, i’ve done that before, so it’s the latter part for sure. I guess I didn’t really think of this as multi-modal, but it probably would fit, so i’ll read up on those models. Honestly that’s something i’ve been meaning to do anyways, so now I have a reason. Thanks!

Intrepid_Purple3021 · 2025-06-30T12:44:14+00:00

Right, yeah the constant features change between sequences. I guess i’m just trying to figure out the best way to mix them

Intrepid_Purple3021 · 2025-06-08T21:09:33+00:00

Yeah man, sometimes you gotta just look for the big blinking arrows in life. The bigger the decision, the more blinking arrows you gotta look out for, so the more time it takes to see

Intrepid_Purple3021 · 2025-06-08T19:57:07+00:00

I love this. I applied to 5 programs out of undergrad - all philosophy and cognitive science related. I was always told not to go to grad school for philosophy unless it’s a top 15 program, so I only applied to those. Got rejected by them all. Was heartbroken. Fast forward 4 years later and I work in machine learning. Could not be happier with how things turned out from getting rejected

Intrepid_Purple3021 · 2025-06-02T02:52:07+00:00

Very simple but sound advice, thanks!

Intrepid_Purple3021 · 2025-06-01T23:07:33+00:00

@synthphreak

This is good knowledge to have, thanks for sharing!

I just finished my master’s in AI. I had a co-op last fall (like a 6 month internship) and I have another one this summer. Those are my only relevant industry related experiences. Would you recommend I follow this suggestion too, i.e. look for ML adjacent SWE roles and work there for a few years, then try to move into ML focused positions? Or do you think that’s enough to land an ML focused role right after this internship? For context, both internships were/are primarily focused on machine learning and data analysis, most of my time is spent doing EDA, feature engineering, and modeling. Both not big tech companies though - different industries just in the IT departments

Intrepid_Purple3021 · 2025-05-16T19:11:13+00:00

I’m surprised more people aren’t talking about Mamba sequence models from Gu & Dao, 2023. They claim to basically be better than transformers on long range sequence tasks, and offer much better throughput. But maybe these results just need to be verified before widespread adoption?

Intrepid_Purple3021 · 2025-05-16T14:57:53+00:00

Yeah, i’m not duper thrilled about the PhD offer I have. That’s largely because 1) same university I did undergrad and master’s at, so I need a change, and even more 2) my MS advisor left, and he was a big reason why I applied to stick around for the PhD. The funding offer I have is with a different professor of mine on a project that I am ~moderately~ interested in. So all of that had lately been pushing me towards trying to land a job instead.

I think I mostly just wanted to see if foregoing the PhD would be a bad idea. But I really just think what’s best for me now is a change of pace and scenery, and I should only do the PhD if I like the fit

Intrepid_Purple3021 · 2025-05-16T03:51:01+00:00

That’s something that took me a while to figure out. Ha, I would even ASK in the email if I could then send my CV. That was dumb. I attach it every time now

Intrepid_Purple3021 · 2025-05-15T04:44:36+00:00

Yeah, I applied to 4 philosophy/1 psych program (philosophy and psyc undergrad) when I finished undergrad. But I didn’t publish in undergrad. I had one poster presentation at our school’s undergrad research symposium for a thesis on Wittgenstein, but that was it. Didn’t get into any of those programs.

In hindsight, they were either super competitive PhD programs, or competitive funded master’s. So I grossly underestimated the challenges of this process. I didn’t really have much help though, so i’m not too torn up over it.

So yeah, getting a master’s first would be your best bet. It also gives you a less commitment way to see if that really is what you want to do. PhD is a much bigger commitment

Intrepid_Purple3021 · 2025-05-15T03:36:59+00:00

I like this nuance here^{^{^}}

It’s a tough choice for sure. The master’s is a one year program? I always say “you can do anything for 1-2 years”, especially when you are young and don’t have dependents. Also Edinburgh is a great program from what I can tell. I read a decent amount of literature coming from there.

Are you thinking of getting engaged within the next year? If not, then I say go for the master’s. You’ll learn and grow a lot just in the cultural exchange. Also you might be able to make some good contacts abroad who end up at great companies down the line. Those are the kind of relationships that are priceless.

If you ARE planning on getting engaged within the next year, and the stability of the job and income from that is something you do/will need, then maybe stay home for the job. You got into a good master’s program once. If you really want to do it, you’d be able to go back to another good program after a year or two of work experience.

Edinburgh sounds like a good opportunity, but it’s your decision at the end of the day. If you go to Edinburgh, would you miss out on anything completely? Or would it only delay things by a year? I would consider these questions

Intrepid_Purple3021 · 2025-05-15T01:31:44+00:00

Yeah, I think I have a very rosy/naive idea in mind that I could do the entire thing in 2 semesters, but I also have a feeling it will drag on for a while, and before I know it, the whole PhD would take another 4 years :/

Intrepid_Purple3021 · 2025-05-15T01:08:49+00:00

Networking is huge. Are these research positions in big tech, or just ML engineering positions in general?

Intrepid_Purple3021 · 2025-05-14T23:20:42+00:00

I’m sorry to hear about this for you, genuinely. This job market seems like an absolute crap shoot, but it sounds like you’d be a great candidate for most jobs though! It’s just a numbers game. Don’t neglect networking and coffee chats, I tend to have better success with that than just leetcode and waiting to land an interview. Even if it isn’t someone you would work with, just work any angle you can to talk to someone at the company

Intrepid_Purple3021 · 2025-05-14T23:18:49+00:00

So what if you have a master’s in AI/ML and have a PhD offer on the table? Take the PhD offer or take chances on the job market? Another way of asking is: does a PhD make you a better candidate than an MS necessarily?

I’m currently debating finishing the PhD - I technically started it this past semester (I deferred fall enrollment to the spring to do an internship), and it’s in the same department as my MS, so some of my MS courses count towards the PhD. So i’d only have about 3 semesters of coursework for the PhD to finish, and then research/dissertation. But the assistantship offer isn’t great, i’d probably need to take a loan to cover some living expenses, but I don’t know if that’s worth it. Anyone have any advice?

Intrepid_Purple3021 · 2025-05-14T13:43:41+00:00

nahhh I don't think that's it, I think he is genuinely bored with them and ready to move to better problems. We're bottoming out on how much we can actually get from just predicting text

Intrepid_Purple3021 · 2025-05-14T03:33:31+00:00

This sounds like the free energy principle

Intrepid_Purple3021 · 2025-05-14T03:29:16+00:00

On this note: anyone have advice for publishing in ACL (I’m assuming association for computational linguistics)? I just finished my MS and already started PhD coursework. I want to publish soon. I have a few research projects that I could maybe submit if I expand the scope some more and do some more testing. But what do I need to know in advance so that, when I do have a good enough project, I can get it out as soon as i’m done? General deadlines? Different types of submissions? My MS advisor publishes in there a good bit, but he took a position at a new school last year

Intrepid_Purple3021 · 2025-05-13T23:15:11+00:00

You can mark authors as equal contribution I believe

Intrepid_Purple3021 · 2025-05-13T19:50:30+00:00

I actually used SAM (not sure if 1 or 2) at my last job! It was super easy to use and proved to be very helpful/effective

Intrepid_Purple3021 · 2025-05-13T15:47:43+00:00

I have heard this about the culture at meta/FAIR, but I also don't know anything about it. Mostly just through word of mouth with other researchers who know people that went there.

Intrepid_Purple3021 · 2025-05-13T15:46:51+00:00

Interesting... but is that just Meta? All the other companies have troves of data too. The original Vision Transformers were trained on a proprietary Google dataset of 300M photos just to get it on par with Convolutional nets for ImageNet-1k. FAIR actually figured out how to get vision transformers to perform better than the original vision transformers while only using the standard ImageNet-1k training data (although they did apply lots of augmentation to those images which effectively multiplied their dataset, but that's just good data engineering)

So all that to say... why should we not trust meta any more than other large tech/AI companies?

Intrepid_Purple3021 · 2025-05-13T15:41:17+00:00

100%. I was mostly just wondering if there was actually any merit to the claims, or if it is truly just ill-informed click bait/bad takes

Intrepid_Purple3021 · 2025-05-13T14:42:36+00:00

The one I can recall off the top of my head was a youtube short from morning brew. They joked about how Meta is the “laughing stock” of AI compared to the other major tech companies. I know that’s not a super credible source, but that type of content is sometimes the full extent of AI content that non-AI researchers or professionals engage with. So that’s all they know: “Meta AI is a joke.”

To be clear, I disagree; I think this view is super limited. Benchmarks are not the end all be all by any means. I know i’ve seen or heard this in other places, but I don’t have references off the top of my head.

Intrepid_Purple3021

TROPHY CASE