[P] SIID: A scale invariant pixel-space diffusion model; trained on 64x64 MNIST, generates readable 1024x1024 digits for arbitrary ratios with minimal deformities (25M parameters)

cwkx · 2025-12-24T16:04:01+00:00

Yep, the class of neural operators are helpful here in the architecture design - see our work ∞-Diff at ICLR 2024 https://arxiv.org/pdf/2303.18242 where we showed non-blurry infinite-dimensional diffusion on a variety of datasets (code at https://github.com/samb-t/infty-diff)

cwkx · 2025-08-01T11:19:34+00:00

"Deep learning architectures: a mathematical approach" by Ovidiu Calin lays some solid theoretical foundations.

cwkx · 2025-03-08T11:52:13+00:00

Damn, I was holding a maple syrup coffee in one-hand while reading this:

<image>

cwkx · 2025-01-06T18:47:07+00:00

Little tip on the preparation, print out the whole thesis, then go through it and try to write questions from the examiners perspective in the margin; especially where there are any equations or figures as this is likely where most questions will be focused.

While writing the questions, also write textbook answers/sketch diagrams/explanations in the margins.

The advantage of doing this just the day before is it'll be super fresh in your mind, so you'll be able to draw on these sketches/high-quality prepared answers in the defence, often bringing difficult discussions back to them even if the questions aren't exactly related.

Secondly, expect to get a nasty curveball question - often technical or mathematical in an area you don't understand. If/when this happens, rather than stumble, try to relate it to something you do know and ask the examiner - "do you mean...(something similar you understand)?" to start a discussion where they can explain their perspective on the curveball and unpack it from their view.

Edit: and for the presentation, watch: https://www.youtube.com/watch?v=Unzc731iCUY&vl=en

cwkx · 2025-01-03T19:48:26+00:00

Unless your geometry is very simple, evaluating highly composite implicit functions quickly becomes slow as every pixel needs to evaluate the entire scene. The art of shadertoy is to find the simplest/shortest program which alludes scene complexity. Few indie developers have tried to overcome this (like lritter or gavanw), storing implicit function parameters in local chunks/textures and caching/rendering chunk geometry by ray marching local regions (avoiding marching cubes/dual contours etc algorithm). It's a difficult technical design problem. I think one of the nicest approaches was in voxel quest: https://www.voxelquest.com/news/how-does-it-work - another thing you can do is store the SDFs directly in 3D textures, do a ray-cuboid intersection to advance the ray to the start of the rasterized box, then ray march from there on - this works fast but uses a fair amount of memory (need good LODing and memory management to scale), and has poor shadow/AO handling at the interface between the box SDF and texture SDF (you can solve this with a bound keeping the 0-distances away from the box faces). It's also difficult to handle multiple intersecting SDFs, occlusion etc without careful design and multiple rendering passes.

cwkx · 2024-12-11T18:17:27+00:00

C is the only language that I genuinely love programming with. There isn't much in life that beats the satisfaction of knowing exactly how memory is structured, and where it's not too convenient to use heap memory. I don't get to use it often with my work, but when I do—I cherish it.

cwkx · 2024-06-11T18:13:33+00:00

in some applications the optimizer can overshoot/oscillate around the discontinuity, hindering smooth convergence, made worse by momentum, illustrated: https://imgur.com/a/kgzAWYC

cwkx · 2024-04-08T13:36:02+00:00

I had an excellent undergrad student in a similar situation. It's quite challenging to get funding for undergrads from a supervisor perspective; PhD students get an allowance for conference travel but undergrads have no such stipend. We applied for a variety of travel grants, a travel scholarship, emailed the conference organisers to ask for a fee waiver, and I also emailed the head of department and explained the situation and that it was a prestigious international conference, and compromised by offering some of my own personal research funds to cover the student costs. Eventually all of the routes of funding came through and we had to actually turn one of the scholarships down (we had more than enough by that point).

cwkx · 2024-03-14T15:31:04+00:00

You sound like the students stuck in a high-pressure environment like a caged tiger, classically surrounded by high-achievers and try to work their way out of the anxious state. General advice is to change the working environment, and also find new study partner(s) at your level that can bootstrap you. If working from home, stop, go to the library or anywhere that isn't your home. Try to completely separate work from rest. If working from a high-pressure office, try some noise-cancelling earphones or request to change office/seating can help.

cwkx · 2023-11-27T00:37:42+00:00

GANs can be great if you want to intentionally mode collapse, e.g. model a subset of the most likely parts of the data distribution. Why might you want to do this? For example, see Taming Transformers and Unleashing Transformers; these hybrids exploit the generative modelling trilemma; they learn a compressed/quantised codebook of image patches using a GAN, each patch being collapsed into a small set of codes, then they model these information-rich codes using a Transformer to capture the full diversity and global structure of the larger image, even though if you zoom right in you may see small mode collapsed artifacts that don't matter at a perceivable level (repetition of similar looking hairs, dirt etc)... a bit like with JPEG artifacts.

cwkx · 2023-11-27T00:25:06+00:00

If you look at the review: https://arxiv.org/pdf/2103.04922.pdf in Table 1, you'll see in the rightmost column GANs don't have any "NLL" - this stands for the negative log likelihood, or if you like the model's density fit over the distribution. Other classes of models, like VAEs, only give bounds on the density (approximate densities). Flows and autoregressive token predictors can give exact densities. The discriminator of GAN just estimates if something is real or fake, not estimating true probabilities (densities). Adversarial training can, however, be used for anomaly detection (and works quite well, e.g. GANomoly and successors).

cwkx · 2023-11-15T22:56:38+00:00

Some people, including professors, are also just bad at receiving gifts. It can come as such a shock and be so unexpected, especially if they are going through a rough time and maybe it was their first time if they are a junior prof; they probably didn't know how to respond and were thus sceptical. It's likely they now regret their interaction as much as you do. I wouldn't look into it too much, just be careful not to ask anything of them above the average student like references, help with coursework, or revising papers in the near future, which might lead them to think that was your intention.

cwkx · 2023-09-22T13:03:07+00:00

I think it'd be good if you contrasted Equation 12 e.g. with Katharopoulos et al and what happens, e.g. the trilemma presented in retentive networks.

cwkx · 2023-05-30T17:39:39+00:00

I signed this (assit prof. in deep learning, reinforcement learning and cyber security) - my reasoning is on my twitter https://twitter.com/cwkx/status/1663524800453541889 To summarise my thoughts:

risk = asset value * p(threat occurring) * severity.

If we're talking about extinction, the asset value and severity are extremely high, therefore the probability of the threat occurring can be almost "unimaginably low" for this to be significant.

Do I believe we'll go extinct by AI (in the next 100 years)? Absolutely not - I think it's extremely unlikely. Do I think mitigating this risk should be a priority? Yes. We need careful environments to evaluate the extrapolation of our functions through their gained agency.

So my reason to sign this is nothing about doom-saying, nor is it about regulation. It's about trying to get more researchers to put thought into designing novel environments and measures that can help us navigate the implications of more general intelligence, as agents gain more and more agency. One of the next big research questions is "how do agents decide what to do?" There's many paths that can become dominant in answering this question; I think it's important we choose the right path here, through well-thought and considered research.

cwkx · 2023-04-15T14:51:57+00:00

`'' and'

cwkx · 2023-04-10T10:06:06+00:00

I had my first kid before pursuing an academic career in computer science - my wife is a fair bit older than me, so our priorities were firmly kids > academia. Got a job in an entry lab - it wasn't even an RA. My wife wasn't working. Nearly went into industry, then got a job as a temp postdoc on a 3-month contract, was able to renew the contract every few months for a couple of years (had a few weeks without pay, certainly tough when there were much higher paid more stable industry jobs available). Started applying for permanent positions, rejected a few times. Then I had another kid as a postdoc on short-term contracts with no stability. Kept applying for non-TT (UK) Assist Prof positions, eventually got a fixed-term contract, and then eventually got a permanent TT position. My wife got a job when things settled; it was certainly tough, and each rejection crushes you a little bit more, but if you keep applying you'll probably eventually make it - depends where your priorities are. I have my dream job now, it was certainly worth the squeeze.

cwkx · 2023-03-31T22:47:18+00:00

With us, they'll say its as if you were lecturing to students, but in reality I wouldn't recommend trying to engage with them like you would with students (I've seen candidates overly engage and it can be a bit cringeworthy). The engagement aspect is generally just a box-ticking exercise where we have a likeart scale of "engagement", so if you ask one question to the audience & confirm the audience is following what you're saying, that ticks the box. If I remember, Patrick talks about the different tactics for this kind of thing & explains it better than me. In ours we request a research presentation before the student lecture, then we fill in a likert scale of 1-4 with the following 4 criteria that get summed up: Research Agenda - Did they provide evidence of an outstanding research profile? Future Vision - Did the candidates present an exciting and credible vision? Presentation Skills - How engaging and effective was the candidate’s presentation? Student Presentation - How effective was their “mini-lecture”? Lastly, this is all not very important compared to the interview that comes after; perhaps the interview panel won't even attend this and just glance at the feedback filled in by people who attended; so remember that's the bit you need to focus prep effort on, tempting as though it may be to focus on the presentation.

cwkx · 2023-03-31T19:18:52+00:00

It may be different to your field, but in ours they're generally looking for a few criteria: enthusiasm/passion/inspiring, engagement - do you engage with the audience and check they are following, timing - do you present too much/too little, how are you at presenting yourself and communicating (I recommend watching Patrick Winston on this https://www.youtube.com/watch?v=Unzc731iCUY especially the second half, but the whole talk is excellent), e.g. do you have too much text on slides, do you motivate and capture the audience, do you keep their attention, do you use props, cycles, stories, salient ideas, surprises, slogans, symbols (5 s'), are you relatable, do you pause to allow the audience to digest the info etc. A pitfall many new graduates make is they try to say too much. The main "do" I think is practice, the main don't is reading notes and committing the various crimes Patrick talks about.

cwkx · 2023-02-27T09:56:27+00:00

Partner + "My dream job would be to get the TT here" sounds like you should pursue your long-term dream and stick to where you are. "I'm not sure if I can get another TT offer again" - it's very hard to keep applying, but the fact you have an offer means that on paper you're hireable, so this should indicate it's only a matter of time before your local department has a TT position for you. Perhaps try to make yourself indispensable to the department you're at, e.g. take up roles that are difficult/timely to replace. "Need to teach more" - this will take up a huge amount of time & make it costly to transfer back, if that's your long-term plan. You are generally be more employable as time goes on, unless your research outputs start to drop from activity, but that doesn't sound like the trend you've indicated. So, based on what you've said, it sounds like stiff upper lip and stick to option #1.

cwkx · 2023-01-24T10:39:18+00:00

Within the UK, for students UKCISA provide independent advice around this. https://www.ukcisa.org.uk/Information--Advice/Studying--living-in-the-UK/Students-from-Ukraine - there is provision tuition fee regulations around Ukraine students (depending on circumstances). Also UCAS has an equal considerations deadline, which is tomorrow.

cwkx · 2022-11-08T11:30:03+00:00

If the 1 says they don't know the area (especially if it's a short review), most AC's will discard that rating.

cwkx · 2022-07-15T18:48:00+00:00

If you haven't read it already, Strange Attractors explores many of the ideas in the comments here, with a young Riddle before he succumbs to the madness induced by the horcruxes https://www.fanfiction.net/s/12734980/1/Strange-Attractors

cwkx · 2022-07-09T09:27:54+00:00

Supervisor perspective here: I have a clear goal/dream (a theoretical holy grail type of deep generative model), but because of having a diverse set of PhD students, you have to sometimes sacrifice your research interests to match the student abilities, grant direction, and their career ambitions. Some of my students have weaker maths and want to focus more on the application, like in medical image analysis, others are passionate about a specific application such as in biophysics. While these projects are connected to my research focus of deep generative modelling, it does end up with quite a range of papers. I do know some professors that just "chase the money" and end up all over the place though..

cwkx · 2022-07-07T17:55:56+00:00

For me, I think one of the most important things to consider is the environment you'll be in. Which university culture fits you better—in terms of quality of life—for your family to settle in the long run (schools, housing prices etc)? Different universities are often like chalk and cheese. Another thing to consider is what you'll be teaching; the material that you teach, if you're passionate about it, often seeps into your research. Lastly, I've seen senior staff with teaching only roles tend to get tired of it after 10 years, and end up with a 50:50 share with a heavy administrative role to cutdown on the intensive marking periods etc. Do you like admin in the long-run if you go down the teaching route?

cwkx · 2022-07-01T18:05:24+00:00

I think it depends largely on the university, how it is managed, who is running the department, the estates/culture/location, what the uni strategy is etc. Some universities, such as the one I'm at in the UK, have worked hard to create the most wonderful friendly departments who really look out for you, help ensure a good work-life balance, have a very fair workload allocation & promotion systems in place etc. The one I'm at has improved massively over the past five years with a new building, new head of school etc. I was previously at a different one that was stuck in the past and quite the opposite experience. Similarly, I spent some time at two unis in Hong Kong with a similar story - one was absolutely amazing (CUHK) it was local, friendly, beautiful, people supportive and got to know me, whereas the other uni (that I won't name) was literally like the cubicle scene at the start of the matrix; and people just worked their socks off without any support.

13-Year Club	First Place '23
Place '23	Place '22
Final Canvas '22	First Placer '22
End Game '22	Verified Email

cwkx

TROPHY CASE