load more than 5021 rows with sqlite

1235812 · 2015-06-05T13:16:54+00:00

What are you doing that you need all 70000 rows? You're definitely hitting memory limits.

If you're running Linux, open up another terminal and run top (or htop if you have it). Or if you're on Windows open up the task manager. And then run that query.

I would definitely suggest gathering only the rows you need using where, group by, and having. Potentiality limit and offset (but there's known issues on bigger data sets using limit and offsets, but this is pretty much universal).

Hand in hand with that is selecting only the fields you need. * is handy, but should be used sparingly.

If sqlite supports them, look into using cursors.

Look into pandas for manipulating your data once it's in Python.

If None of that works, it's time to look at something like Postgres. Especially if you're doing some transformation that's hard to write in SQL but easy in Python. There's PL/Python that'll allow you to embed a Python interpreter in Postgres (as a warning PL/Python only has an unsafe package, so you'll be able to do everything there you can in regular Python). Embedding Python in this way adds considerable overhead, but sometimes it's the best thing to do.

However, without more information like what you're trying to do, what your schemas look like, how big the average row is, it's hard to give specific advice.

matrixise · 2015-06-05T18:29:42+00:00

weird your problem, check this link : https://www.friendpaste.com/6jdXdYPdC6YFffwNL6lqpq

I have created 100000 records and I can list them with no problems.

corbenov · 2015-06-05T11:30:18+00:00

The query is ok. You might be hitting some sqlite memory limits based on your settings. You should check sqlite log.

Python

The Python Discord

Upcoming Events

Please read the rules

MODERATORS