error: [Errno 0] Error

[Safihre]

On Debian 9, with OpenSSL 1.1.0f 25 May 2017 CherryPy will thrown this (non-fatal) error on startup when using the 'server.ssl_module': 'builtin'. It does not happen when using 'server.ssl_module': 'pyopenssl'.

This is not just on our older CherryPy, but also on the latest CherryPy version 11 and both on Python 2.7 and Python 3.5.

We have been chasing this error for a while and first assumed it was a bug in Python. I was ready to report it to Python bug-tracker, but I cannot reproduce it using pure-python. So it really is something specific to what CherryPy does.

Code:

import cherrypy
print(cherrypy.__version__)

class RootServer:
    @cherrypy.expose
    def index(self, **keywords):
        return "it works!"

if __name__ == '__main__':
    server_config={
        'server.socket_host': '0.0.0.0',
        'server.socket_port': 9090,
        'server.ssl_module': 'builtin',
        #'server.ssl_module':'pyopenssl',
        'server.ssl_certificate':'/root/.sabnzbd/admin/server.cert',
        'server.ssl_private_key':'/root/.sabnzbd/admin/server.key'
    }

    cherrypy.config.update(server_config)
    cherrypy.quickstart(RootServer())

Error thrown by Python 2.7 and 3.5:

11.0.0
[11/Aug/2017:13:23:57] ENGINE Listening for SIGHUP.
[11/Aug/2017:13:23:57] ENGINE Listening for SIGTERM.
[11/Aug/2017:13:23:57] ENGINE Listening for SIGUSR1.
[11/Aug/2017:13:23:57] ENGINE Bus STARTING
CherryPy Checker:
The Application mounted at '' has an empty config.

[11/Aug/2017:13:23:57] ENGINE Started monitor thread '_TimeoutMonitor'.
[11/Aug/2017:13:23:57] ENGINE Started monitor thread 'Autoreloader'.
[11/Aug/2017:13:23:57] ENGINE Serving on https://0.0.0.0:9090
[11/Aug/2017:13:23:57] ENGINE Bus STARTED
[11/Aug/2017:13:23:57] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cheroot/server.py", line 1515, in start
    self.tick()
  File "/usr/local/lib/python2.7/dist-packages/cheroot/server.py", line 1590, in tick
    s, ssl_env = self.ssl_adapter.wrap(s)
  File "/usr/local/lib/python2.7/dist-packages/cheroot/ssl/builtin.py", line 73, in wrap
    server_side=True)
  File "/usr/lib/python2.7/ssl.py", line 363, in wrap_socket
    _context=self)
  File "/usr/lib/python2.7/ssl.py", line 611, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 840, in do_handshake
    self._sslobj.do_handshake()
error: [Errno 0] Error

Pure-python code version trying to reproduce all steps cherrypy performs, that does not thrown the error:

import socket, ssl
import fcntl

print "Start"
context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context.load_cert_chain(certfile="/root/.sabnzbd/admin/server.cert", keyfile="/root/.sabnzbd/admin/server.key")

bindsocket = socket.socket()
bindsocket.bind(('0.0.0.0', 9090))
bindsocket.listen(5)

while True:
    newsocket, fromaddr = bindsocket.accept()

    fd = newsocket.fileno()
    old_flags = fcntl.fcntl(fd, fcntl.F_GETFD)
    fcntl.fcntl(fd, fcntl.F_SETFD, old_flags | fcntl.FD_CLOEXEC)

    sslsoc = context.wrap_socket(newsocket, do_handshake_on_connect=True, server_side=True)
    request = sslsoc.read()
    print(request)
    print(sslsoc.cipher())
print "Done"

@sanderjo also created a guide to setting-up a docker to with Debian to test this:
sabnzbd/sabnzbd#1000
For non-docker-people like me, this command can be used to copy files into the docker:

sudo docker cp test.py DOCKER-ID:/test.py

[webknjaz]

// offtopic comment

sudo

I'd recommend adding your user to docker group instead, using root is considered harmful.

[webknjaz]

@Safihre thanks for your report and reproducible example. I'm not an expert in SSL and haven't had time to sort out related issues.

I know that some issues with SSL exist and here's another one: cherrypy/cheroot#6. Also, since SSL has moved to cheroot.ssl it sounds reasonable to try reproducing it with just pure cheroot.

Any ideas?

[Safihre]

@webknjaz Sure I can try with just cheroot, but how?
There is no documentation of just using cheroot that I can find and the example doesn't say how to include ssl:

    from cheroot import wsgi
    def my_crazy_app(environ, start_response):
        status = '200 OK'
        response_headers = [('Content-type','text/plain')]
        start_response(status, response_headers)
        return ['Hello world!']
    addr = '0.0.0.0', 8070
    server = wsgi.Server(addr, my_crazy_app)
    server.start()

[webknjaz]

probably:

import cheroot.server
ssl_module = 'builtin'
ssl_adapter_cls = cheroot.server.get_ssl_adapter_class(ssl_module)
server.ssl_adapter = ssl_adapter_cls(...corresponding args here)

[webknjaz]

I suspect it might be related to https://stackoverflow.com/a/38224367, which looks reasonable since you run frozen python app IIRC.

[webknjaz]

Ref: http://python.6.x6.nabble.com/Cherrypy-s-SSL-builtin-td5215127.html (https://groups.google.com/forum/#!topic/cherrypy-users/-TMVrdJ_VHI)

[webknjaz]

Ref: https://github.com/python/cpython/blob/07ca9af/Modules/_ssl.c#L919

[webknjaz]

https://wiki.openssl.org/index.php/Manual:SSL_do_handshake(3)#RETURN_VALUES

[webknjaz]

It looks like https://github.com/openssl/openssl/blob/1f5878b/ssl/ssl_lib.c#L3450-L3481 returns 0 and wrapper in cpython raises that as an error not adding additional insight.

[Safihre]

In the reported cases it's not run frozen, but as a regular python 2 application.

[webknjaz]

that's weird. I'd suspect it's maybe because of privileges. Have you tried that under sudo?

[webknjaz]

PyOpenSSL uses SSL_get_error() to reliably detect the underlying error, but stdlib implementation is very simplistic and it omits this. The comment in C-code in stdlib says that "0" should be treated as failure, but OpenSSL docs also state that's not a crash:

The TLS/SSL handshake was not successful but was shut down controlled and by the specifications of the TLS/SSL protocol. Call SSL_get_error() with the return value ret to find out the reason.

[Safihre]

Have no tried sudo since we use a port that should normally not require it.
And as described in the starting post, using a pure Python implementation does not throw this error. Only when using Cherrypy with seemingly the same setup.

[webknjaz]

Some dudes think that errno 0 shouldn't be raised as an exception https://bugs.python.org/msg299759

[webknjaz]

Ref. https://bugs.python.org/issue10808

[webknjaz]

Also, this might be related to OpenSSL version https://bugs.python.org/issue28689
Can you try w/ smth like 0.9? Just for checking purposes

[Safihre]

Yes it seems specific to this 1.1.0 version.
But again, only a problem with Cherrypy and not with standard Python.

[webknjaz]

Some of the complaints on the Internet seem to mention client connections, which close socket without fully completing the handshake.

[webknjaz]

apache/thrift#1321 suggest additionally catching OSError

[webknjaz]

So, during comparison with your example snippet I noticed that we also have fcntl modifications on bindsocket + SO_REUSEADDR sockopt
https://github.com/cherrypy/cheroot/blob/9219ea44ede356e1e9ef4cc38c18733d8fce2d5e/cheroot/server.py#L1574

Also you don't load a CA file

(FTR)

[webknjaz]

So I've done some debugging and it seems that during startup some of components tries to do a connection to server.
I guess it might happen to be a non-SSL connection (like plain HTTP), which would make sense to be failing.
I thought it might be Checker (_cpchecker), but it looks it doesn't do any health checks over HTTP.

[webknjaz]

(connection comes from the same process)

[webknjaz]

I've used this snippet and confirmed that the issue is not happening in pure Cheroot:

#! /usr/bin/env python

from cheroot import server, wsgi


addr = '0.0.0.0', 8070
ssl_module = 'builtin'
ssl_adapter_cls = server.get_ssl_adapter_class(ssl_module)
ssl_adapter_kwargs = {'certificate': '/root/.sabnzbd/admin/server.cert', 'private_key': '/root/.sabnzbd/admin/server.key'}


def my_crazy_app(environ, start_response):
    status = '200 OK'
    response_headers = [('Content-type','text/plain')]
    start_response(status, response_headers)
    return ['Hello world!']


server = wsgi.Server(addr, my_crazy_app)
server.ssl_adapter = ssl_adapter_cls(**ssl_adapter_kwargs)
server.start()

[webknjaz]

@Safihre I've checked it down to cherrypy==3 and confirmed that the behavior with SSL error being raised persists. So it's not a regression in cherrypy, but a regression in openssl. And old version of ssl wrapper from python stdlib does not seem to not take some changes in openssl into account, however pyopenssl (wrapping cryptography) is processing this correctly.

[Safihre]

Thanks :)

[len-samuelson]

This issue arose in my latest xubuntu 18.04 installation containing python3.6.5, cherrypy-15.0.0 and cheroot-6.3.1. Following the leads in this issue's comments led to a bit more detail and (for my installation) a workaround: Remove the test for python2.
issue1618_python3.txt

...Not ready to write a pull request, there are several testing dependencies as yet unresolved. :-)

In case the diff is useful, feel free to use it, I impose no license restrictions on its use.

[webknjaz]

@len-samuelson I'm inlining your patch since it's a better experience not to have to download files to just see text:

diff --git a/cheroot/ssl/builtin.py b/cheroot/ssl/builtin.py
index cc3866b1..2c472bc1 100644
--- a/cheroot/ssl/builtin.py
+++ b/cheroot/ssl/builtin.py
@@ -130,15 +130,22 @@ class BuiltinSSLAdapter(Adapter):
         except generic_socket_error as exc:
             """It is unclear why exactly this happens.

-            It's reproducible only under Python 2 with openssl>1.0 and stdlib
-            ``ssl`` wrapper, and only with CherryPy.
-            So it looks like some healthcheck tries to connect to this socket
-            during startup (from the same process).
+            It's reproducible under both Python 2 and Python3.6 with openssl>1.0
+            and stdlib ``ssl`` wrapper. CherryPy uses portend.py to verify that
+            the server port is ready for incoming connections (code ref
+            cherrypy/process/servers.py:260).  Portend tries to connect to a
+            port then close it without performing any action (such as starting
+            an SSL handshake). That causes what OPENSSL probably thinks is a
+            protocol violation, raising OSError with error 0. This exception
+            handler is extended to work with Python3 by removing the test for
+            python2, and assumes the OSError args (likely raised by python's
+            openssl code) are (0, 'Error').


             Ref: https://github.com/cherrypy/cherrypy/issues/1618
             """
-            if six.PY2 and exc.args == (0, 'Error'):
+            EMPTY_RESULT = None, {}
+            if exc.args == (0, 'Error'):
                 return EMPTY_RESULT
             raise
         return s, self.get_environ(s)

[webknjaz]

@Safihre that issue is about setting up some minimum TLS testing at all. It's not specific to the current one.

[webknjaz]

@msinn probably because everyone forgot about it :) Feel free to send a PR!

[webknjaz]

@Safihre
P.S. it should be reproducible by connecting to HTTPS-port via HTTP :)

[webknjaz]

Hey @len-samuelson @Safihre @msinn,

So I've pushed a bit more granular version of that patch. It looks like Python 3.7's ssl module handles this because it only supports 1.1, but older versions supported 1.0 and probably didn't fully cover errors of 1.1.

[mattdodge]

Am I not understanding the cause of this issue correctly or should this be an or instead of an and on this line?
https://github.com/cherrypy/cheroot/blob/48dba8024f93a8627876f0398d7d5b9d9e39d535/cheroot/ssl/builtin.py#L163

It is my understanding that if you're running Python <3.7 but OpenSSL 1.1 then you will get the weird 0 error code. So if you have 3.7 (which requires OpenSSL 1.1+ anyway) then you should be good. But if you had Python 3.6 for example we'd want to suppress the error by returning EMPTY_RESULT.

So I would think the conditional would either be this:

if is_error0 and IS_BELOW_PY37:
    return EMPTY_RESULT

or perhaps this to be more explicit

if is_error0 and (IS_BELOW_PY37 or IS_BELOW_OPENSSL11):
    return EMPTY_RESULT

but I don't the current syntax (with and) is going to do anything to help the original post (Python 3.5, OpenSSL 1.1). Am I talking crazy here?

[webknjaz]

Oh, sorry. It looks like I messed up :)

P.S. Kind reminder: it is always better to send a PR rather than post things in the closed issue where comments may be left unnoticed for various reasons.

[webknjaz]

Should be fine now. I've changed below 1.1 with above 1.0.

[mattdodge]

Perfect, thanks a lot!

[Lawouach]

Hey folks,

Im still seeing this issue with Python 3.7.1, openssl 1.1.0g, CherryPy 18.1.0 and cheroot 6.5.4

[14/Jan/2019:21:30:03] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
  File "/home/sylvain/.venv/lib/python3.7/site-packages/cheroot/server.py", line 1753, in serve
    self.tick()
  File "/home/sylvain/.venv/lib/python3.7/site-packages/cheroot/server.py", line 1958, in tick
    s, ssl_env = self.ssl_adapter.wrap(s)
  File "/home/sylvain/.venv/lib/python3.7/site-packages/cheroot/ssl/builtin.py", line 117, in wrap
    sock, do_handshake_on_connect=True, server_side=True,
  File "/usr/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/usr/lib/python3.7/ssl.py", line 853, in _create
    self.do_handshake()
  File "/usr/lib/python3.7/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

Is that expected?

• Sylvain

[webknjaz]

@Lawouach it's probably a bug. You should file an issue under Cheroot I guess and let's try to come up with a reproducer because it doesn't fail in CI. How did you check openssl version btw? What about earlier versions? I've done a massive TLS refactoring (and still doing it), can it be a regression?

[webknjaz]

And what about 3.7.2?

[Lawouach]

I haven't had the chance to update on my local machine.

For the openssl version:

$ python
Python 3.7.1 (default, Oct 22 2018, 11:21:55) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION_INFO < (1, 1)
False
>>> ssl.OPENSSL_VERSION_INFO
(1, 1, 0, 7, 15)

$ openssl version
OpenSSL 1.1.0g  2 Nov 2017

Today, I was actually setting a simple local CA using mkcert as per http://woile.github.io/posts/local-https-development-in-python-with-mkcert/

[webknjaz]

@Lawouach is that for testing? I've migrated Cheroot to trustme and it's pretty amazing!
https://github.com/cherrypy/cheroot/blob/d179b71c9a8c43d2595b6a892f41f44ccfd72bb4/cheroot/test/test_ssl.py

[webknjaz]

Here's how you identify which version stdlib ssl module is linked against:

$ python -c 'import ssl; print("\nOPENSSL_VERSION: " + ssl.OPENSSL_VERSION + "\nOPENSSL_VERSION_INFO: " + repr(ssl.OPENSSL_VERSION_INFO) + "\nOPENSSL_VERSION_NUMBER: " + repr(ssl.OPENSSL_VERSION_NUMBER))'

[Lawouach]

Here is the output:

$ python -c 'import ssl; print("\nOPENSSL_VERSION: " + ssl.OPENSSL_VERSION + "\nOPENSSL_VERSION_INFO: " + repr(ssl.OPENSSL_VERSION_INFO) + "\nOPENSSL_VERSION_NUMBER: " + repr(ssl.OPENSSL_VERSION_NUMBER))'

OPENSSL_VERSION: OpenSSL 1.1.0g  2 Nov 2017
OPENSSL_VERSION_INFO: (1, 1, 0, 7, 15)
OPENSSL_VERSION_NUMBER: 269484159

Thank you a lot for trustme. I wasn't aware of it and it is indeed for testing.

[webknjaz]

Could you please verify that Cheroot tests pass on your machine? It looks like depending on env combo some bits of ssl fail sometimes but I'm trying to be granular about the things I suppress.

[Lawouach]

All tests in cheroot master do pass fine indeed.

[webknjaz]

So maybe it's a misconfiguration in mkcert?

[webknjaz]

P.S. Here's a collection of TLS related testing tools I've found earlier while researching TLS: cherrypy/cheroot#95

[webknjaz]

@Lawouach oh and does it happen when you start CherryPy app but before you try sending any requests? It might be the checker thing which effectively punches port over HTTP to see that it's alive.

[Safihre]

@webknjaz This error is still there, also on Python 3.7 using default Python Windows installer.

https://bugs.python.org/issue31122

I did a pip install cherrypy -U, just to be sure.

3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] 
OpenSSL 1.1.0j 20 Nov 2018
Traceback (most recent call last):
  File "C:\Users\saf\Documents\GitHub\tests\py3\lib\site-packages\cheroot\server.py", line 1753, in serve
    self.tick()
  File "C:\Users\saf\Documents\GitHub\tests\py3\lib\site-packages\cheroot\server.py", line 1958, in tick
    s, ssl_env = self.ssl_adapter.wrap(s)
  File "C:\Users\saf\Documents\GitHub\tests\py3\lib\site-packages\cheroot\ssl\builtin.py", line 117, in wrap
    sock, do_handshake_on_connect=True, server_side=True,
  File "C:\Users\saf\AppData\Local\Programs\Python\Python37\Lib\ssl.py", line 412, in wrap_socket
    session=session
  File "C:\Users\saf\AppData\Local\Programs\Python\Python37\Lib\ssl.py", line 853, in _create
    self.do_handshake()
  File "C:\Users\saf\AppData\Local\Programs\Python\Python37\Lib\ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

[Safihre]

Anyone reading this, the fix for now is:

import cheroot.ssl.builtin
cheroot.ssl.builtin.IS_BELOW_PY37 = True

Before

cherrypy.engine.start()

[Safihre]

@webknjaz the patch for this in Python has been merged, finally: https://bugs.python.org/issue31122
However, it will only be fixed for recent versions of Python so I guess the workaround in the code should remain.