TimeoutError on multiple requests

[petritavd]

• asyncpg version: 0.27.0
• PostgreSQL version: 14.7
• Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
  the issue with a local PostgreSQL install?: We can reproduce it, not locally
• Python version: 3.10.2
• Platform: Backend: Linux, t3large AWS instance, Database: AWS RDS
• Do you use pgbouncer?: No
• Did you install asyncpg with pip?: Yes

We're using FastAPI ,and our current code for setting up the db connection is:

connection_arguments: Dict[str, Any] = {"server_settings": {"jit": "off"}}

ENUM_TYPES = (
    ("public", "sometype"), ...
)

async def register_enum_types(conn) -> None: 
    for enum_class, typename in ENUM_TYPES:
        await conn.set_builtin_type_codec(
            typename, schema=enum_class, codec_name="text"
        )


engine = create_async_engine(
    settings.uri,
    echo=False,
    connect_args=connection_arguments,
    pool_size=200,
    max_overflow=10,
)

async_sessionmaker = sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)


@event.listens_for(engine.sync_engine, "connect")
def on_connect(dbapi_connection, connection_record):
    dbapi_connection.run_async(register_enum_types)

We added the register_enum_types as we were thinking that the query:
SELECT t.oid, t.typelem AS elemtype, t.typtype AS kind FROM pg_catalog.pg_type AS t WHERE t.oid = $1

which we saw a lot in AWS RDS monitoring is causing the issue, but now we're not sure what is it anymore.

In logs when it happens we see:

File "/path/to/file/python3.10/site-packages/asyncpg/connect_utils.py", line 780, in _connect_addr raise asyncio.TimeoutError asyncio.exceptions.TimeoutError

and after some time, it gets back to normal.

[ilirosmanaj]

@elprans, any ideas on the above? It's happening pretty consistently and we've tried basically everything to get this sorted. Some input would be very welcome.

[elprans]

This seems like a connection timeout, not a query timeout, so I'd investigate the network configuration. How are you hosting your app?

Also, why do you need to register the codec alias for the enum types? Asyncpg already decodes them as text by default.

[ilirosmanaj]

The app is a FastAPI app running in EC2 - we're using uvicorn and start about 10 workers. The postgres instance is in AWS RDS (instance type db.m5.2xlarge), see instance details below:

What we notice is that the "Average active sessions" sometimes blows up even with not that high usage and causes these timeouts.

The reason for registering was because we sometimes notice spikes of the following query (even though we have jitt set to off) and as part off some investigations this was a proposed solution (but didn't help). The query:

SELECT t.oid, t.typelem AS elemtype, t.typtype AS kind FROM pg_catalog.pg_type AS t WHERE t.oid = $1.

I am not sure if the connection pool is being drained here but it's highly unlikely in my opinion.

Let me know your thoughts @elprans ✋

[ilirosmanaj]

Any ideas @elprans? Sorry for bugging you but it's quite an interesting thing.

[elprans]

Could it be that you've restarted your app at that time and all workers had their connection pools populated?

[elprans]

The reason for registering was because we sometimes notice spikes of the following query (even though we > have jitt set to off) and as part off some investigations this was a proposed solution (but didn't
help). The query:

SELECT t.oid, t.typelem AS elemtype, t.typtype AS kind FROM pg_catalog.pg_type AS t WHERE t.oid = $1.

This happens because you are registering codecs, the query is executed by set_type_codec/set_builtin_type_codec.

[zagortenej024]

Hey @ilirosmanaj, did you resolve this by any chance? We're having the exact same issues

[visal-123]

Hi @ilirosmanaj may I know if you already solved the issue?

[ilirosmanaj]

@visal-123 @zagortenej024 for us the issue was the workers. We were not using gunicorn, but rather uvicorn directly.

What helped is using the gunicorn with uvicorn workers - so it was likely a worker issue keeping some connections running. Let me know if this helps.

[visal-123]

@ilirosmanaj I also use gunicorn with uvicorn workers but still have this intermittent timeout issue.

And I have three instances of fastapi apps (3, 10, 24 workers), by the way.

But It's good you fixed the issue.

And @elprans, I would be glad to hear your thoughts on this.