Should use Host header to check server_hostname when using SSLContext

[arthru]

Long story short

When I use both an IP in the URL, a Host header, and a SSLContext that is configured to check the hostname, it checks the hostname using the IP

Expected behaviour

Checks the certificate using the hostname in the Host header, if it exists

Actual behaviour

Checks the certificate using the hostname in the URL that is actually the IP and fails with the following exception :

SSL handshake failed on verifying the certificate
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fb9db5c9ba8>
transport: <_SelectorSocketTransport fd=6 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/usr/lib64/python3.7/asyncio/sslproto.py", line 625, in _on_handshake_complete
    raise handshake_exc
  File "/usr/lib64/python3.7/asyncio/sslproto.py", line 189, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/lib64/python3.7/ssl.py", line 763, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '151.101.120.223'. (_ssl.c:1051)
SSL error in data received
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fb9db5c9ba8>
transport: <_SelectorSocketTransport closing fd=6 read=idle write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/usr/lib64/python3.7/asyncio/sslproto.py", line 526, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/lib64/python3.7/asyncio/sslproto.py", line 189, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/lib64/python3.7/ssl.py", line 763, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '151.101.120.223'. (_ssl.c:1051)
Traceback (most recent call last):
  File "/home/arthur/.virtualenvs/watchghost/lib/python3.7/site-packages/aiohttp/connector.py", line 822, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)
  File "/usr/lib64/python3.7/asyncio/base_events.py", line 975, in create_connection
    ssl_handshake_timeout=ssl_handshake_timeout)
  File "/usr/lib64/python3.7/asyncio/base_events.py", line 1003, in _create_connection_transport
    await waiter
  File "/usr/lib64/python3.7/asyncio/sslproto.py", line 526, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/lib64/python3.7/asyncio/sslproto.py", line 189, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/lib64/python3.7/ssl.py", line 763, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '151.101.120.223'. (_ssl.c:1051)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "aiohttp_ip_host.py", line 25, in <module>
    asyncio.run(failing_https_request())
  File "/usr/lib64/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.7/asyncio/base_events.py", line 573, in run_until_complete
    return future.result()
  File "aiohttp_ip_host.py", line 20, in failing_https_request
    ssl=context,
  File "/home/arthur/.virtualenvs/watchghost/lib/python3.7/site-packages/aiohttp/client.py", line 370, in _request
    timeout=timeout
  File "/home/arthur/.virtualenvs/watchghost/lib/python3.7/site-packages/aiohttp/connector.py", line 445, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/arthur/.virtualenvs/watchghost/lib/python3.7/site-packages/aiohttp/connector.py", line 757, in _create_connection
    req, traces, timeout)
  File "/home/arthur/.virtualenvs/watchghost/lib/python3.7/site-packages/aiohttp/connector.py", line 879, in _create_direct_connection
    raise last_exc
  File "/home/arthur/.virtualenvs/watchghost/lib/python3.7/site-packages/aiohttp/connector.py", line 862, in _create_direct_connection
    req=req, client_error=client_error)
  File "/home/arthur/.virtualenvs/watchghost/lib/python3.7/site-packages/aiohttp/connector.py", line 825, in _wrap_create_connection
    req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host 151.101.120.223:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '151.101.120.223'. (_ssl.c:1051)")]

Steps to reproduce

import asyncio
import aiohttp
import ssl


async def failing_https_request():
    url = 'https://151.101.120.223/'
    hostname = 'www.python.org'

    context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
    context.verify_mode = ssl.CERT_REQUIRED
    context.check_hostname = True
    context.load_default_certs()

    headers = {'Host': hostname}
    async with aiohttp.ClientSession() as session:
        response = await session.get(
            url,
            headers=headers,
            ssl=context,
        )
        print(response.status)


asyncio.run(failing_https_request())

Your environment

• Archlinux
• Python 3.7.1
• aiohttp 3.4.4 and master (da24574)

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are #3304 (Bracket IPv6 addresses in the HOST header), #3265 (Host header position causes certain sites to not respond), #2254 (when use aiodns), #1652 (Trailer headers), and #116 (Host the docs).

[asvetlov]

I'm wondering what is the use case?
Why you cannot use just DNS name?

[kxepal]

For such situations, the case is just to encrypt connection. But in this case you also want to turnoff verification since it's based on hostname.

[asvetlov]

Certificate verification is tightly coupled with hostnames, isn't it?
For me the feature request looks like as an ask for opening a way for cheating.

[kxepal]

Yes, it is. And that's leaves only one use case: just to encrypt data between peers. That's was always the case for self-signed certs.

For me the feature request looks like as an ask for opening a way for cheating.

Well, user have to explicitly turn off certificate verification with all the consequences. I think it should be already possible, right?

[arthru]

In fact, I want to check that a server is well configured (including https certificate) even if the dns is not set to that particular IP. The use case is to check a server is well configured before configuring the dns to target that server.

[asvetlov]

HTTP doesn't work this way.
For example, curl -H "Host: www.python.org" https://151.101.120.223/ fails.
Why should aiohttp support this subtle and error-prone behavior?

BTW, by standard HTTP cookies also behaves differently if a site is accessed by IP. DNS hostname is too important.

[asvetlov]

As a workaround, you can provide a custom resolver (implement aiohttp.AbstractResolver and pass an instance into TCPConnector).
This is the best that I can suggest.
Picking server_hostname from Host HTTP header is a security hole.

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.