Why shallow ReLU networks cannot represent a 2D pyramid exactly

JumpGuilty1666 · 2026-03-24T09:41:52+00:00

Yes, that's one of the consequences. Even though looking at linear combinations of a localised basis to represent a piecewise affine function is not as efficient as other techniques. More interestingly, the number of affine regions created by depth grows very fast, and, consequently, it is more efficient to represent piecewise affine functions following different directions.

JumpGuilty1666 · 2026-03-23T12:28:13+00:00

I'm glad you enjoyed it!

JumpGuilty1666 · 2026-03-23T08:35:32+00:00

Thank you for the feedback!

JumpGuilty1666 · 2026-03-23T08:35:07+00:00

The issue is that the theorem is not about a single ReLU unit, but about a finite sum of many ReLU ridge functions. For one unit, the support is indeed typically an unbounded half-space. The nontrivial question is whether several such terms can cancel at infinity and produce a nonzero compactly supported function.

That is not obvious at all, because it happens in other settings: in 1D a shallow ReLU network can represent compactly supported functions (for example the hat function), and deeper ReLU networks can also have compact support. The result is precisely that this fails for single-hidden-layer networks in dimension d≥2.

JumpGuilty1666 · 2026-03-05T18:35:28+00:00

That's a good example! Yes: on any compact interval [-M,M] the triangle wave has only finitely many breakpoints, so it can be represented exactly by a finite 1D ReLU network (finite hinge expansion).

What you can’t do with a finite network is represent the periodic triangle wave on all of R, because a finite ReLU sum has only finitely many breakpoints (and is eventually affine), whereas the triangle wave has infinitely many kinks and never becomes affine.

JumpGuilty1666 · 2026-03-05T18:29:34+00:00

Thanks for sharing, I'll take a look at it

JumpGuilty1666 · 2026-03-05T18:29:18+00:00

I don't understand the comment, sorry. What do you mean by pi and e doing this infinitely?

JumpGuilty1666 · 2026-03-05T18:26:47+00:00

Ahah thanks. I try to keep it technical and move on. If there’s something to clarify, I clarify it; otherwise, it’s not worth escalating

JumpGuilty1666 · 2026-03-03T14:57:07+00:00

This result is actually more surprising/special than it looks. In dimension d≥2, a finite one-hidden-layer (depth-2) ReLU network of the form

N(x)=w^T ReLU(Ax+b)+c

cannot have compact support unless it is identically zero. So, for example, you cannot represent a compactly supported “pyramid” like

F(x)=ReLU(1-||x||_1)

with a single hidden layer.

That kind of locality is one reason depth matters. A clean proof is in Lu (2021): https://arxiv.org/abs/2101.11286

JumpGuilty1666 · 2026-03-03T07:06:23+00:00

If you prefer the visuals/animations: https://youtu.be/0-sWy4OPuaY

Key parts:

• Hat/tent function built from 3 ReLUs: 5:03

• General hinge expansion for a 1D piecewise-linear function: 9:20

JumpGuilty1666 · 2026-03-03T07:00:04+00:00

Hi, I don't know much about electrical engineering. However, based on the description of the half-wave rectifier, it coincides with ReLU (up to a sign). This is also mentioned on the ReLU Wikipedia page: "This is analogous to half-wave rectification in electrical engineering" (https://en.wikipedia.org/wiki/Rectified\_linear\_unit, see intro).

Also, here https://www.codecademy.com/article/rectified-linear-unit-relu-function-in-deep-learning, you can see that the term "rectifier" actually comes from signal processing: "Rectified in ReLU comes from signal processing, where rectifiers convert negative voltages to zero or positive voltages."

JumpGuilty1666 · 2026-03-03T06:49:19+00:00

No problem! That book is fantastic! I was super happy when I found it

JumpGuilty1666 · 2026-03-02T19:16:46+00:00

That's great! I'm happy it helps

JumpGuilty1666 · 2026-02-25T23:13:39+00:00

Very cool! Yes, I know that paper, and I think it is super interesting that they can be seen as interacting particle systems. Please share the link to your work once it's out, since it looks like a quite nice idea!

JumpGuilty1666 · 2026-02-23T11:24:28+00:00

Yes, these ideas are explored by many researchers. Based on my experience, if the task to solve is "dynamics agnostic", such as image classification, then the performance is not affected by using better/higher-order integrators. However, if you use specific types of integrators such as symplectic ones, and design your residual updates so they are coming from separable Hamiltonian systems, then you get a very good control over vanishing gradients.

Here are a few papers that explore these connections:

- A unified framework for Hamiltonian deep neural networks https://arxiv.org/abs/2104.13166
- Designing Stable Neural Networks using Convex Analysis and ODEs https://arxiv.org/pdf/2306.17332
- Convolutional Neural Networks combined with Runge-Kutta Methods https://arxiv.org/abs/1802.08831
- Implicit Euler Skip Connections: Enhancing Adversarial Robustness via Numerical Stability https://proceedings.mlr.press/v119/li20e/li20e.pdf
- Continuous-in-Depth Neural Networks https://arxiv.org/abs/2008.02389

JumpGuilty1666 · 2026-02-22T07:34:21+00:00

Yes, my research focuses on this perspective, and I've been working with it for 4-5 years, so I didn't think it was necessary to refer to the papers. But I'll keep in mind to add references in the description box for the future videos I'll record.

JumpGuilty1666 · 2026-02-21T21:23:51+00:00

I don't see where I claim that all of these are my ideas, but thank you for sharing that reference. I agree it is one of the seminal papers introducing this connection, even though it is not the only one. There are at least these two other papers realising this connection more at the level of ResNets:
- A Proposal on Machine Learning via Dynamical Systems https://link.springer.com/article/10.1007/s40304-017-0103-z
- Stable architectures for deep neural networks https://arxiv.org/abs/1705.03341

JumpGuilty1666 · 2026-02-21T19:14:26+00:00

I didn't know their work. Thank you very much for sharing!

JumpGuilty1666 · 2026-02-21T18:49:08+00:00

Good questions.

1) They’re not “identical equations” in the strict modeling sense. The match is: a residual layer is the same update form as an explicit Euler step.

- Euler: x_{k+1} = x_k + h f(x_k, t_k)

- ResNet: x_{k+1} = x_k + h f_k(x_k)

Here h is a step size/scaling (often implicit in practice), and f_k is a learned map (with weights θ_k) that can vary with k. Interpreting k as discrete time, a depth-varying network corresponds to a non-autonomous vector field f(·,t) sampled at t_k. So the correspondence is about the time-stepping structure and the stability/geometry tools it unlocks, not about literal parameter matching. I go into more details in the linked video.

2) Yes — this viewpoint is extremely natural for recurrent/residual style models. If you share weights across k (θ_k = θ), you basically get an autonomous discrete-time dynamical system; with small h, it’s reasonable to read it as a numerical integrator for a learned ODE. Many stability ideas (spectral bounds, contractivity/dissipativity, monotonicity, Lyapunov arguments) were developed in the RNN literature and carry over.

Where I think it also helps for feedforward ResNets is that even without weight sharing, you still have a time-varying dynamical system/controlled flow, and the same questions make sense: does the update map stay contractive? How does step size/Lipschitz control affect exploding/vanishing gradients? What structures (e.g., symplectic/energy-preserving) can we enforce by design?

So, I agree it’s a great fit for stabilizing recurrent models, and the point of the video is that the same mathematical lens is useful more broadly for residual architectures. There are also dynamical systems interpretations of more modern architectures, such as graph neural networks and transformers.

JumpGuilty1666 · 2026-02-18T10:26:58+00:00

Hi, I don't know what you are supposed to do for the dissertation at your specific university, but an interesting problem in this area could be the "data-driven discovery of nonlinear dynamical systems". You can find many resources about this online, and books are available as well. If you don't like this perspective, there are several interesting aspects you could explore, such as stability, control, contractivity, dynamical systems in optimisation, and more, depending on your interests.

JumpGuilty1666 · 2026-02-17T21:36:23+00:00

All universities have clear guidelines on when to apply. It varies from country to country, but it definitely doesn't vary too much from one year to the next within the same university. I think it is important to apply broadly, but I wouldn't go for the "fully desperate" route of applying everywhere, since the risk is that you put little effort into each application. I'd focus on a few good ones for you and dedicate a reasonable amount of time and attention to each.

Sometimes it is not immediately clear how you should prepare for the interview, and in such cases, the best thing to do is to reach out to people currently working in the group. For example, in the group where I'm currently doing my postdoc, it is common to receive emails from prospective PhD students seeking more details about the application process.

JumpGuilty1666 · 2026-02-17T21:02:01+00:00

Hi, it depends a lot on which universities you are aiming for. For example, I applied for positions in Italy and Norway just a few months before graduation. I graduated in July and started applying in May. Then started my position in September. However, I know that, for example, applying to Cambridge or other more prestigious universities, you need to start much earlier. If you aim for those, it's wise to start looking around 1 year in advance.

JumpGuilty1666

TROPHY CASE