9

I have a .Net Core (C#) application that takes user requests via websocket and then creates a connection to a PostgreSQL database to manipulate/handle data.

When a user makes a new request to the backend, the endpoint function called creates a new SQL connection and run the query:

// Endpoint available via Websocket
public async Task someRequest(someClass someArg)
{
    /* Create a new SQL connection for this user's request */
    using var conn = new NpgsqlConnection(connstr.getConnStr());
    conn.Open();

    /* Call functions and pass this SQL connection for any queries to process this user request */
    somefunction(conn, someArg);
    anotherFunction(conn, someArg);

    /* Request processing is done */
    /* conn is closed automatically by the "using" statement above */
}

When this request is finished, the connection is closed by the using statement. However, this connection by default is returned to the Postgres "connection pool" and is shown as idle.

Since every new user request here creates a new SQL connection, those old "idle" SQL connections in the connection pool are never used again.

Currently:

  1. Due to these idle connections pooling up and reaching the max pool size, I have temporarily set the idle connection timeout very low. Otherwise these idle connections stack up and hit the artificial ceiling for open connections.
  2. I've also tried adding Pooling=false to the connection string. My understanding is this would stop the connection from idling once closed by the .Net app, but it seems to still be idling.

Question: What is the best practice for handling Postgres's connection pooling in .Net/C#?

  1. If I can more properly utilize the Postgres connection pool, it would be more efficient to re-use already opened connections than creating a new one for every user request.
  2. My idea for this was to have a function that creates new Postgres connections, keeps track of them, and hands them out to callers when a user makes a new request. Is this a terrible idea?
  3. Or, do I just keep pooling disabled/a very low idle timeout and create a new SQL connection per-request like I am now?

I've been unable to find many examples of properly utilizing Postgre's connection pool short of what I'm doing. This application has an average of 3,000-4,000 concurrent users at anytime, so I can't have a single static connection handling everything. What is the best practice for handling this in .Net?

EDIT So it looks like the pooling is native to NPGSQL and not Postgres. If a new connection is made with the same database, user, password it will use one of the idle "pooled" connections instead of opening another one.

The issue is that it seemed to not be doing so before I had disabled pooling. There was an error spammed that took the application down for an hour or more over the night.

The connection pool has been exhausted, either raise MaxPoolSize (currently 100) or Timeout (currently 15 seconds)

Now it's possible that it was really needing 100+ active connections at once... but my guess is that most of those were idle as I had seen before.

Edit #2 So I've tried now with allowing pooling (the default) and it seems to shoot up idle connections instantly as they're created by requests, but does not re-use these connections. Once it reaches the max pool cap, the application locks up due to no new connections/requests being able to be made.

DBeaver Img - Server Sessions: The red here is active connections, blue is idle.

enter image description here

Every single SQL connection in the application is created from a single/shared connection string environment variable. Host=IP;Port=somePort;Username=someUser;Password=somePass;Database=someDb;Maximum Pool Size=100 The only way I'm able to keep the application running is by setting idle_in_transaction_session_timeout to '10s' to clear out idle connections frequently as the pooling does not seem to work.

When I have postgres clearing out idle connections with idle_in_transaction_session_timeout and Pooling=false this is what my DB activity looks like:

enter image description here

I also ran a search in my code, every instance of making a new SQL connection has the using statement as well. As shown in the code example above. This should prevent any sort of connection leak.

enter image description here

Is there some sort of postgres config item that would cause this issue? The connection string is the same every time, and every connection has the C# using statement. I'm not sure why NPGSQL isn't re-using these idle connections when pooling is enabled.

I've tested spamming new connections in a loop on my dev server, and pooling seems to work just fine. So I can tell that the using statement format I have causes no issues. But if I enable pooling now on my production server, the idle connections instantly spam upwards as shown and hit the cap allowing no new connections. Metrics for the production server show ~1,000 transactions per second and ~4-5 active SQL sessions/connections a second. Is it possible that I just really need to increase the max pool limit?

5
  • Make sure that the connection string is the same every time you're opening a connection; if so, Npgsql's pool should work fine and get an existing idle connection from the pool. If you're still running into trouble, you can try increasing MaxPoolSize to see if that helps (in case your actual needs are higher). If that still doesn't work, check to see you're actually closing the connection everywhere (e.g. with using) - if you don't that would lead to a leak which would explain this. Commented Jan 12, 2021 at 23:47
  • @ShayRojansky I appreciate the response. For the life of me, the pooling does not seem to work. All SQL connections are created using a single environment variable that does not change. If I enable pooling in this connection string, idle connections will shoot up to 200 (the max) and no new connections will be allowed. Commented Jan 13, 2021 at 4:03
  • I created an API command to change the connection string on the fly to test swapping with and without pooling. Host=IP;Port=5432;Username=someUser;Password=somePass;Database=someDb;Maximum Pool Size=200 I'll edit the main post again to show some more details. Commented Jan 13, 2021 at 4:05
  • I've tested spamming new connections in a loop on my dev server, and pooling seems to work just fine. So I can tell that the using statement format I have causes no issues. But if I enable pooling now on my production server, the idle connections instantly spam upwards as shown and hit the cap allowing no new connections. Metrics for the production server show ~1,000 transactions per second and ~4-5 active SQL sessions/connections a second. Is it possible that I just really need to increase the max pool limit? I've tried setting that to 200, but that cap was hit within a few seconds. Commented Jan 13, 2021 at 5:00
  • If you need more than 200 connections at a given moment, then yes, you'll need to increase Max Pool Size - but that's quite a load (or it could mean your queries are running very slowly). However, I'd double-check that the connection string is always constant, etc... You may also get some more insight into what's going on by examining Npgsql's logs. Commented Jan 13, 2021 at 14:22

1 Answer 1

9

The issue ultimately seemed to be with the Postgres database configuration itself. We were receiving a lot of pq: could not resize shared memory segment. No space left on device errors.

The database runs in a Docker container. I increased the shm_size of the container to 6gb to utilize more of the resources it was running on.

From there, in the postgresql.conf file for the DB itself, there seemed to be a lot of confusion as to what to set the shared_buffers and max_connections to.

For shared_buffers this was set to 1/4th of the shm_size per the postgres docs: 1500M. This used to be set at 128M, which was way too low for the container memory size.

For max_connections this was lowered back down to the default 100. Even with ~3000-5000 concurrent users, the backend only utilized 5-10 active connections at any given moment.

Hopefully these config items can help someone else out in the future.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.