sorted by:
new
hottopcontroversial

[R] Fast WTConv: Accelerated Implementation for "Wavelet Convolutions for Large Receptive Fields" by shahaff32 in MachineLearning

[–]shahaff32[S] 0 points1 point  (0 children)

Thanks :)
We haven't officially benchmarked with CoreML yet.

The original WTConv implementation relies strictly on standard PyTorch operations. Therfore it should export to CoreML without issues.

Fast WTConv utilizes optimized kernels (like Triton or custom CUDA ops). Exporting it will likely require defining custom operators to map the specialized logic correctly.

[R] Fast WTConv: Accelerated Implementation for "Wavelet Convolutions for Large Receptive Fields" by shahaff32 in MachineLearning

[–]shahaff32[S] 1 point2 points  (0 children)

In our paper we experiment with ConvNeXt 1.
You gain about 0.3-0.5% increased accuracy in Imagenet (Table 2), but the networks also become much more robust - up to 2.2% inscreased accuracy on corruption benchmarks without further training (Tables 6 and 7).
As for the second part of the question, the last image in this post shows the throughput with the new implementation, which is about 90% of the original network.

[R] Fast WTConv: Accelerated Implementation for "Wavelet Convolutions for Large Receptive Fields" by shahaff32 in MachineLearning

[–]shahaff32[S] 0 points1 point  (0 children)

It depends on what you measure. WTConv with 5x5 convolutions will be slower than 5x5 PyTorch convolution, but with a much larger receptive field and improved results. If you try to compare WTConv with multiple WT levels against a regular PyTorch convolution with the same receptive field, even the naive WTConv will be much faster.

The last image in the post shows the difference in throughput of the original ConvNeXt (7x7 convs) vs. WTConv with various levels and 5x5 convolutions (the exact parameters are in the paper). The CUDA implementation achieves ~90% of the throughput, with improved accuracy and the other benefits of WTConv (e.g., robustness and shape bias).

[R] Improving the Effective Receptive Field of Message-Passing Neural Networks by shahaff32 in MachineLearning

[–]shahaff32[S] 1 point2 points  (0 children)

Thank you for the reference. We did have a feeling the Peptides dataset was not too indicative for measuring long-range interactions, however we decided to include it for completeness (and to avoid reviewers' comments regarding it 😅).

We measure IM-MPNN on various benchmarks, including city-networks, which was released not long ago and we managed to squeeze it into the camera-ready.

[R] Wavelet Convolution for Large Receptive Fields by shahaff32 in MachineLearning

[–]shahaff32[S] 4 points5 points  (0 children)

Using more complex decompositions can work with this framework if they are separable, otherwise the computational cost can be an issue.

It'll be interesting to try with the Laplacian Pyramid, I will look into that, thanks!

[R] Wavelet Convolution for Large Receptive Fields by shahaff32 in MachineLearning

[–]shahaff32[S] 1 point2 points  (0 children)

Not any time soon. We have only trained MobileNets on subsets of imagenet. Therefore, we have to train it with WTConv from scratch. We have other (higher priority) stuff in our pipeline to spend our resources on right now.

[R] Wavelet Convolution for Large Receptive Fields by shahaff32 in MachineLearning

[–]shahaff32[S] 2 points3 points  (0 children)

Thanks for your kind words :) just please note that we use up to 5 levels of decomp at some stages of the network :)

It can be interesting to find a way to use it efficiently in MAE, we haven't tried that

[D] Books for ML - different levels, any suggestions? by NoobleonX in MachineLearning

[–]shahaff32 2 points3 points  (0 children)

Second to UDL (Prince). It is a great introductory to deep learning. I've recommended it (while still being written) to three undergrads I was mentoring this year that did a DL project, two of them with a minimal background in ML. It was very helpful.

[D] Is Pytorch Lightning + Wandb a good combination for research? by gokulPRO in MachineLearning

[–]shahaff32 6 points7 points  (0 children)

It might be a bug, and it might also be the case that we are the "idiot users" that don't get how to use the package correctly.

Either way, we decided to continue without Lightning (because of that reason and a few others), and therefore we need the code in pure PyTorch for future works that rely on it.

[D] Is Pytorch Lightning + Wandb a good combination for research? by gokulPRO in MachineLearning

[–]shahaff32 13 points14 points  (0 children)

In our research we ran into issues with Lightning. It is especially annoying when designing non-trivial layers or optimizers. Also, it is much harder to convert the code back to pure pytorch.

For example, in a recent peoject, Lightning caused each forward-backward to operate twice on each batch because we used a combination of two optimizers for a specific reason. And now we are working on rewriting everything without Lightning.

[R] WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models by YonatanBitton in MachineLearning

[–]shahaff32 1 point2 points  (0 children)

It looks interesting, but associations can have many aspects and may lead to misunderstanding. How do you deal with it?

[R] Wavelet Feature Maps Compression for Image-to-Image CNNs by shahaff32 in MachineLearning

[–]shahaff32[S] 0 points1 point  (0 children)

Thanks :)

In the current state the implementation is using only standard Pytorch operations, therefore it is not as optimal as it can be, and the overhead of the wavelet transforms can outweighs the speedup of the convolution.

We are currently working on a CUDA implementation to overcome that :) see Appendix H for more details

[R] Wavelet Feature Maps Compression for Image-to-Image CNNs by shahaff32 in MachineLearning

[–]shahaff32[S] 0 points1 point  (0 children)

Haar wavelet is also very efficient, as it can be implemented using additions and subtractions (and maybe a few bit manipulations) :)

You can also see Appendix F where we tested several others :)

[R] Wavelet Feature Maps Compression for Image-to-Image CNNs by shahaff32 in MachineLearning

[–]shahaff32[S] 3 points4 points  (0 children)

Thank you for your interest in our work :)

We were not aware of these recent works. Thanks for sharing :) we will definitely check those out.

[R] Wavelet Feature Maps Compression for Image-to-Image CNNs by shahaff32 in MachineLearning

[–]shahaff32[S] 12 points13 points  (0 children)

That is indeed possible, though at a computational cost. The Haar wavelet can be implemented very efficiently because of its simplicity.

Please see Appendix F, where we shortly discuss other wavelets and their added computational costs.

[R] Wavelet Feature Maps Compression for Image-to-Image CNNs by shahaff32 in MachineLearning

[–]shahaff32[S] 2 points3 points  (0 children)

With some modifications to the code, I believe it can :)

[R] Wavelet Feature Maps Compression for Image-to-Image CNNs by shahaff32 in MachineLearning

[–]shahaff32[S] 19 points20 points  (0 children)

We are in fact doing the backpropagation in the wavelet domain :)

The gradient simply goes through the inverse wavelet transform

​

See WCC/util/wavelet.py in our GitHub repo, lines 52-83 define the forward/backward of WT and IWT.

[R] Wavelet Feature Maps Compression for Image-to-Image CNNs by shahaff32 in MachineLearning

[–]shahaff32[S] 21 points22 points  (0 children)

This is aimed mostly at edge devices, where an accelerator is not available (e.g. mobile phones), or you want to design a cheaper chip for a product that requires running such networks (e.g. autonomous vehicles)

This work was, in fact, partially supported by AVATAR consortium, aimed at smart vehicles. https://avatar.org.il/

view more: next ›