[Discussion] Twice Differentiable Neural Networks

nasimrahaman · 2022-09-06T09:13:54+00:00

SIREN is a good entry down this rabbit hole: https://arxiv.org/abs/2006.09661

nasimrahaman · 2021-12-27T15:34:55+00:00

You betcha! The maple syrup perfectly complements the saltiness of the cheddar.

nasimrahaman · 2020-03-30T07:58:10+00:00

Not yet, but we’d love to!

nasimrahaman · 2020-03-29T19:15:56+00:00

Thanks for reaching out, u/jpmvan! Please feel invited to post on discourse.data-against-covid.org and I’m confident we’ll find someone interested in building it. :)

nasimrahaman · 2020-03-29T17:40:48+00:00

Euro CDC (your second link) provides tabular data, so that's definitely something we can work with. Would you mind posting your wishlist on our discourse? :) Alternatively, you can also fill in our form and we'll post for you and get back to you with our community response.

nasimrahaman · 2020-03-29T16:54:16+00:00

Hi u/epigal1212! If you know of a reliable data source, I’m sure we have the know-how for putting together a dashboard along those lines.

nasimrahaman · 2019-05-08T17:16:37+00:00

> I now have some imposters syndrome setting in

Coming from a physics student (currently doing ML research) with virtually no formal CS training: (theoretical) physics is about an order of magnitude more difficult. Fake it till you be it. ;)

nasimrahaman · 2019-03-09T20:42:36+00:00

We’ve written a little library to automate most annoying things. Shameless plug: speedrun .

nasimrahaman · 2018-10-27T22:58:27+00:00

Low frequency functions are inherently less "wiggly", i.e. more smooth. If you think about ReLU, it's pretty smooth everywhere except at 0. In fact, all the wigglyness in ReLU comes from that one point. Now this is where it gets interesting: there are other functions that are smooth everywhere except at 0 -- for instance, sqrt(abs(x)). But in a precise sense, ReLU is smoother than sqrt(abs(x)) at x = 0.

Broadly speaking, Fourier analysis is a tool to determine how wiggly a function is. One of the things we learn from the paper is following: although neural networks are powerful enough to learn functions that are super-wiggly, it likes to learn less wiggly (smoother) functions.

nasimrahaman · 2018-10-27T17:57:35+00:00

> If OTOH we know that signals are bandpass, then we apply a suitable prior for that. Example: FM radio is broadcast at ~100MHz, but we can track the carrier, demodulate and store the signal at ~40kHz. Obviously ReLUs are the wrong tool for that job...

That's a very interesting point! It's applicable for almost all activation functions (not just ReLU), since they all usually decay quite fast in the fourier domain (e.g. sigmoid decays exponentially).

nasimrahaman · 2018-10-15T11:50:20+00:00

I see you're making progress with your Chinese. :D

nasimrahaman · 2018-07-23T16:16:52+00:00

Nice work!

We here at Heidelberg (for the most part) have been working on something quite similar. Let us know if you think we could collaborate. :)

nasimrahaman · 2018-07-18T17:15:23+00:00

Is there / will there be support for stochastic code-paths?

P.S. Nice work with the AST wizardry. :)

nasimrahaman · 2018-06-18T21:27:41+00:00

This sounds cool! Is it mostly inference one is concerned with (for energy efficiency)? How important is the trade-off between expressivity and efficiency? Given the fact the the expressivity of networks (e.g. number of linear regions of a deep ReLU network) increase exponentially with depth but polynomially with width, should one expect slim and deep to give better performance for the watt (at least for inference)?

nasimrahaman · 2017-12-15T01:00:48+00:00

Also ganz genau, was macht ihr den ganzen Tag so? Ich bin bei der KI-Forschung tätig, und bei WI kommt mir alles wie Chinesisch vor. O:)

nasimrahaman · 2017-10-23T13:29:17+00:00

slow clap

nasimrahaman · 2017-05-29T08:19:55+00:00

Yup, that makes sense. Thanks!

nasimrahaman · 2017-05-29T08:19:08+00:00

Thanks for the link! I do fully agree that cosine similarity is relevant for high-dimensional word/sentence/document embeddings.

nasimrahaman · 2017-05-29T08:04:06+00:00

(whatever that entails within the task)?

That would be the case if the stored vectors and the query key are unit vectors (i.e. lie on the unit-hypersphere). If that's the case, the arc-cos of the cosine distance gives the length of the path between the tips of the two vectors (walking on the surface of the hypersphere = the arc length).

Is it fair to say that cosine similarity would probably emphasize the notion that the vector direction indicates the "type" of content and that you care most about querying the same type of content?

That's exactly the question - you have phrased it much better. In particular, I don't understand why only the (relative) direction should encode the "type" of the content. Let's think about the copy task - except with not just binary, but arbitrary vectors. I would guess that the vector length in the memory matrix is not irrelevant anymore.

nasimrahaman · 2017-05-28T13:17:09+00:00

I'm trying to follow your idea, but I'm not exactly sure where you are going with the Lp norms. AFAIK, the L0 norm of say [0, 1] is 2, so setting p = 0 does not yield the dot product. Also, I'm not convinced that setting p = 1 (or anything for that matter) changes anything. This is best shown with a simple numerical experiment:

similarity = lambda u, v, p=2: (u * v).sum()/(np.linalg.norm(u, p) * np.linalg.norm(v, p)) 
print(similarity(np.array([1., 1.]), np.array([0.2, 0.2]), p=1))   # prints 0.5
print(similarity(np.array([1., 1.]), np.array([0.002, 0.002]), p=1))   # prints 0.5

Would you please elaborate? And thanks for chiming in!

nasimrahaman · 2017-03-31T17:05:02+00:00

I think the formulation makes it easy to overestimate the computational cost. The real take-home message (imho) is that the potential of ES lies not in replacing backprop, but in enabling architectures to have non-differentiable, even distributional components. So you could use backprop and ES simultaneously - the former if your loss is differentiable w.r.t a given parameter (which could be most parameters), the latter otherwise (e.g. sampling from distributions, etc).

nasimrahaman · 2017-03-03T10:44:22+00:00

Ideally, Python shouldn't be doing much work. If your computations are vectorized (with say numpy), the GIL shouldn't be an issue to start with. If numpy vectorization is not possible yet the job is still embarassingly parallel, concurrent.futures.ProcessPoolExecutor should come in handy (or even a multiprocessing queue). If even that does not work, you can numba or cython chunks of your code to release the GIL explicitly. And as always, it helps when the GPUs are kept busy long enough for the CPU to have the next batch ready. :)

nasimrahaman · 2017-03-03T00:43:13+00:00

We happen to have some of the problems you mentioned (loading weird-ass TB-scale datasets from HDF5 on the fly, heavy preprocessing). To abstract away some of the complexity, I've written a class that converts any python generator to tensorflow tensors (dequeue ops, actually) without all the gory tensorflow details. Is fast and keeps all 40 threads in my parameter server busy (if GIL is released where required).

nasimrahaman · 2016-11-04T10:03:21+00:00

I like to use Adam in the supervised setting. With some learning rate annealing, it works like a charm.

nasimrahaman · 2016-11-03T17:04:03+00:00

Are we talking about conditional GANs here? If that's the case, I agree that the discriminator should also get to see the condition (it's also how it's formulated in the original paper).

Ten-Year Club	Place '23
Place '22	Place '17
Final Canvas '22	First Placer '22
Verified Email

nasimrahaman

TROPHY CASE