Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correctly handle unicode data in the HTTP runner #4601

Merged
merged 5 commits into from
Apr 23, 2019

Conversation

Kami
Copy link
Member

@Kami Kami commented Mar 15, 2019

HTTP runner didn't correctly handle data argument when it was an unicode string. When providing a textual (non-object / dictionary) data, requests expects bytes and not unicode type and that's what we should pass to the function to make it work correctly.

Before:

st2 run core.http url="http://httpbin.org/post" headers='{"Content-Type": "application/json; charset=utf-8"}' method=POST body='{ "msg": "ažž" }'
id: 5c8bafe5076129241e7452a2
status: failed
parameters: 
  body: "{ "msg": "ažž" }"
  headers:
    Content-Type: application/json; charset=utf-8
  method: POST
  url: http://httpbin.org/post
result: 
  error: '''ascii'' codec can''t encode characters in position 11-12: ordinal not in range(128)'
  traceback: "  File "/data/stanley/st2actions/st2actions/container/base.py", line 127, in _do_run
    (status, result, context) = runner.run(action_params)
  File "/data/stanley/contrib/runners/http_runner/http_runner/http_runner.py", line 102, in run
    result = client.run()
  File "/data/stanley/contrib/runners/http_runner/http_runner/http_runner.py", line 226, in run
    verify=self.verify
  File "/data/stanley/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/data/stanley/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/data/stanley/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/data/stanley/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/data/stanley/virtualenv/local/lib/python2.7/site-packages/urllib3-1.24.1-py2.7.egg/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/data/stanley/virtualenv/local/lib/python2.7/site-packages/urllib3-1.24.1-py2.7.egg/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 1057, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1097, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 1053, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 901, in _send_output
    self.send(message_body)
  File "/usr/lib/python2.7/httplib.py", line 873, in send
    self.sock.sendall(data)
  File "/data/stanley/virtualenv/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 401, in sendall
    tail = self.send(data, flags)
  File "/data/stanley/virtualenv/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 395, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/data/stanley/virtualenv/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 382, in _send_loop
    return send_method(data, *args)
"

After:

st2 run core.http url="http://httpbin.org/post" headers='{"Content-Type": "application/json; charset=utf-8"}' method=POST body='{ "msg": "ažž" }'

id: 5c8bb37007612924fa8c92cd
status: succeeded
parameters: 
  body: "{ "msg": "ažž" }"
  headers:
    Content-Type: application/json; charset=utf-8
  method: POST
  url: http://httpbin.org/post
result: 
  body:
    args: {}
    data: "{ "msg": "ažž" }"
    files: {}
    form: {}
    headers:
      Accept: '*/*'
      Accept-Encoding: gzip, deflate
      Content-Length: '18'
      Content-Type: application/json; charset=utf-8
      Host: httpbin.org
      User-Agent: st2/v3.0dev
      X-Stanley-Action: http
    json:
      msg: "ažž"
    url: https://httpbin.org/post
  headers:
    Access-Control-Allow-Credentials: 'true'
    Access-Control-Allow-Origin: '*'
    Connection: keep-alive
    Content-Encoding: gzip
    Content-Length: '288'
    Content-Type: application/json
    Date: Fri, 15 Mar 2019 14:17:40 GMT
    Server: nginx
  parsed: true
  status_code: 200

I confirmed the fix under Python 2.7 and Python 3.6.

Resolves #4599.

Thanks to @kknyxkk and @rsmartins78 for reporting this.

TODO

  • Tests
  • Changelog entry

@Kami Kami added the bug label Mar 15, 2019
Kami added 3 commits March 18, 2019 10:15
NOTE: It's very hard to write integration test for it.
@Kami Kami changed the title [WIP] Correctly handle unicode data in the HTTP runner Correctly handle unicode data in the HTTP runner Mar 18, 2019
@Kami Kami added this to the 3.0.0 milestone Mar 18, 2019
@Kami Kami merged commit 9996690 into master Apr 23, 2019
@Kami Kami deleted the fix_unicode_handling_in_core_http_runner branch April 23, 2019 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug Report - Unicode exception when executing core.http with special character on the body
2 participants