7

Hi I am using psycopg2 for postgres access.

I am trying to understand where "cursor" stores the returned rows. Does it store it in the database as a temporary table or is it on the clients end?

Does cursor (when you specify to fetch many rows) hit the database a query at a time or does it hit the database once ,get the first set of results then when you iterate over the returned values, it gets the next set (buffering).

I have read multiple articles on cursor but nothing really gives the inner working...

Thank you.

0

1 Answer 1

4

The dataset for a cursor is prepared by the server at the time of execution of the first FETCH. The client application receives only the results of subsequent FETCH statements.

If the server cannot use indexes to maintain a cursor, the temporary dataset is created. You can perform this simple test:

create table test(i int, v text);
insert into test
select i, i::text
from generate_series(1, 5000000) i;

Execute the statements in this script one by one:

begin;

declare cur cursor 
for select * from test
order by random();             -- 17 ms

fetch next cur;                -- 37294 ms (*)

fetch next cur;                -- 0 ms
fetch prior cur;               -- 0 ms
fetch absolute 1000000 cur;    -- 181 ms
fetch relative 1000000 cur;    -- 163 ms
fetch first cur;               -- 0 ms
fetch last cur;                -- 0 ms

rollback;

First FETCH (*) performs roughly around the same time as the creation of a similar temporary table:

create temp table temp_test as
select * from test
order by random();             -- 51684 ms

Some drivers may have their own implementation of cursor on client side. This should be explicitly described in the driver's documentation.

Sign up to request clarification or add additional context in comments.

2 Comments

PostgreSQL won't always materialize the data like that though. Sometimes it can fetch it only as-needed. It depends on the exact query plan. Frankly I'm surprised it seems to materialize it in this case. Also, note that this is with a server-side cursor. psycopg2's cursors map to server-side cursors, so behaviour will correspond pretty well, but this isn't necessarily true of other drivers. PgJDBC for example receives the whole result set into local memory by default.
@Craig, this is only an example. I chose the worst case to better illustrate the mechanism of use cursors. Thanks for the tip.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.