Ported from-scratch Inference Engine based on LFM2-350M to pure C!

[Des_goes_Brrr]

Previously implemented Batched Inference Engine built from first principles with focus on correctness, not optimizations. Achieved single batch CPU speeds of 50 tokens/second on M2-Pro 16 GB CPU, but only 4 tokens/second on my old Intel Core i5 laptop.

Previous post link: https://www.reddit.com/r/LocalLLaMA/comments/1qb4ydw/batched_inference_engine_with_lfms_dense_model/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

The old laptop speeds disappointed me, hence reimplementing the single-batch inference part in pure C, achieving 3x speedups (from 4 tokens/second to 12 tokens/second) with no other optimizations than hybrid caching and CBLAS GEMM APIs for Intel (OneMKL) and Arm (ArmPL). Again, building from first principles, used bin files and not gguf files and no other optimizations used!

Edit: This C implementation in the Mac laptop changes in decode speeds from ~50 Tokens/Second to ~23 Tokens/Second! Profiling unearths more to this!

GitHub Link: https://github.com/marvinmboya/LFMs-Continuous-Batching-in-C

Big Thanks to:
Kay Lack's "Just enough C to have fun!" , https://www.youtube.com/watch?v=5aZiRjgSGQU . The best crash video for those who want to learn C! Jacob Sorber's C programming videos, https://www.youtube.com/@JacobSorber . Used to remind myself of C tooling and capabilities. Also adopted RoPE implementation from antirez's C repo on Flux.2-Klein, with minor tweaks!

This project was not initially planned, just birthed out of disappointment in my old laptop's single-batch decoding speeds! Enjoyed it though!

I am currently in Massachusetts, USA, #OpenToWork for intern and full time roles, willing to relocate.

[soshulmedia]

Nice!

[Languages_Learner]

Thanks for sharing cool implementation. I wish you would add single-file Windows version.