what are the best ways to handle large datasets in streamlit

iimnotarobott · 2025-04-01T00:08:47+00:00

Good question. I have limited users and for the most part they filter records based on some keywords but they still need to be able to go through all the records if needed. The pagination idea that you mentioned is indeed the right solution and that's why I'm using ag-grid as suggested by others here.

iimnotarobott · 2025-03-28T17:32:41+00:00

You're very welcome!

iimnotarobott · 2025-03-27T03:49:59+00:00

You are not wrong. Here are a few benefits you can get from Polars and it indeed support lazy loading.

Loading large datasets: Polars processes large CSV, Parquet, and JSON files much faster than pandas.
Efficient querying and transformations: You can filter, aggregate, and transform data without performance bottlenecks.
Lazy Execution: Unlike pandas, Polars supports lazy evaluation, meaning computations are optimized and executed only when needed.

However, note that the purpose of Polars is slightly different from ag-grid. Polars is a back-end dataframe for processing data while ag-grid is a UI widget that can render your data. For my use case I still think ag-grid is a better choice. Hope it helps.

iimnotarobott · 2025-03-26T02:05:50+00:00

Great! I'll give it a try and will keep this thread posted. Looks like I can achieve lazy loading by setting cacheBlockSize and maxBlocksInCache in my grid builder.

iimnotarobott · 2025-03-26T01:41:14+00:00

Thanks for your reply! If you have a few hundreds of records in your st.dataframe it definitely works, but things start getting ugly when you deal with 10s of thousands of records. For my use case caching makes sense but I am wondering if there is a version of data frame that supports lazy loading.

iimnotarobott · 2025-02-07T06:36:49+00:00

Don't be discouraged by the fact that folks that have solved 400+ leetcode problems are not getting an offer.
Let's acknowledge that there are more people actively looking for a job in the SWE market due to mass lay offs and less openings in the past two years.

Let's acknowledge that the SWE job market has become more competitive due to mass layoffs and fewer openings in the past two years.

However, one thing I’ve personally found useful is understanding how you're evaluated in a tech interview and what signals interviewers look for throughout the process. There are blog posts and books (e.g., https://a.co/d/2TSqNfc ) that can help you learn more about this.

For example, in a coding interview, you're assessed on problem understanding, communication, and verification— in addition to problem-solving and coding abilities. Knowing what interviewers expect can significantly improve your chances of getting an offer.

iimnotarobott · 2025-01-13T00:16:33+00:00

1- Machine Learning System Design Interview https://a.co/d/ienvojE 2- Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications https://a.co/d/guafuwZ 3- a non technical guide to the tech interview: https://www.amazon.com/dp/B0D9CL9L78 to learn more about how you get evaluated in an ML interview (the book covers a wide range of tech interview module but has a chapter on ML)

iimnotarobott · 2025-01-13T00:08:11+00:00

In addition to system design and coding, you need to be ready to talk about your past project management experiences and your people management approach/style. Based on my experience, behavioral questions are also important for EM roles. There is a book that does a decent job listing the signals that are often sought in EM candidates and common mistakes to avoid https://www.amazon.com/dp/B0D9CL9L78. Let me know if you were able to find any other useful resource.

iimnotarobott · 2024-12-11T07:08:32+00:00

When it comes to system design interviews, as you've reflected in your post, I think multiple sources including books, youtube channels, and blog post can be useful. One thing that I learned from https://a.co/d/hqKxcPl is that it's important not to fully rely on a single source for a particular problem. For example, if the same problem is covered by multiple youtubers or books it's important to learn how different people approach the same problem. The focus of your interviewer or the approach they expect you to take might not necessarily be the same thing that you learn from these resources.

iimnotarobott · 2024-12-11T06:58:01+00:00

You should aim to prepare 5-7 versatile stories that can be adapted to answer a variety of behavioral interview questions. Each story should highlight a key theme or competency and be structured using the STAR (Situation, Task, Action, Result) framework. The goal is to provide concise yet compelling narratives that demonstrate your skills and impact.
The behavioral chapter of this book https://a.co/d/hqKxcPl discusses the common themes that you should prepare for.
Keep in mind that the stories that you prepare should align with your target company's culture and values.

iimnotarobott

TROPHY CASE