Tox based tests

[webknjaz]

This attempts to make tox a single source of truth for all the test envs (mainly CI, but it wires it into GNU make as well.

I didn't test deployment flow yet, but current results look promising in terms of clearly separating different envs/tasks + there's a nice ~10% speedup in Travis.

[codecov]

Codecov Report

Merging #263 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #263   +/-   ##
=======================================
  Coverage   99.71%   99.71%           
=======================================
  Files           4        4           
  Lines         355      355           
=======================================
  Hits          354      354           
  Misses          1        1

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0541083...e8b6ab6. Read the comment docs.

[asvetlov]

@webknjaz thank you for the PR
Sorry, I pretty busy now.
Please don't merge until my review/approval.
I'm ok to CI change but want to make sure that new process fits my everyday dev practices for running the test suite locally.

[webknjaz]

Sure, it's only a demo for now. I understand importance of non-breaking the current flow + I can also help you to figure out some benefits of working via tox for your tasks.

[webknjaz]

@asvetlov any updates so far?

[asvetlov]

Works fast.
I have 2 questions:

1. How to collect test coverage? make cov-dev is broken
2. How to test on different Python versions? Now I work on Python 3.6 virtual env by default, make test runs against it. When I need to test on different Python I switch to another venv (workon multidict-py37 for example) and run make test again. How to achieve it with tox?

[webknjaz]

1. How to collect test coverage? make cov-dev is broken

I suppose that it's reporter to codecov task is being broken, not coverage collection. Right?
https://ci.appveyor.com/project/aio-libs/multidict/build/1.0.0a0.dev34/job/qa0rtph3ajarjw6d#L954

I'll fix this by making tox not build a wheel in that env, I guess.

2. How to test on different Python versions? Now I work on Python 3.6 virtual env by default, make test runs against it. When I need to test on different Python I switch to another venv (workon multidict-py37 for example) and run make test again. How to achieve it with tox?

So, when you run tox now it pick up whatever points to python executable ATM and then creates env under .tox/python. You can explicitly run tox -e py36 and it will create another separate env .tox/py36 OR you can switch current python with any external tool you use (pyenv?) and then it will recreate .tox/python. Depending on what you want we can wire that into Makefile, but personally I prefer working with tox directly. Since we use pytest {posargs} in our config we can easily bypass args for pytest, so tox -- -k some_filter will result in pytest -k some_filter being executed in the proper env.
Also tox reuses envs if dependencies/env didn't change (and you didn't switch python).
P.S. You can make it run tests in several envs (matrix?) sequencially by spawning smth like tox -e py34,py35,py36,mypy and it will create separate envs for python 3.4, python 3.5, python 3.6 and env called mypy, so that test runs won't influence each other.

Bonus

I've wired up pytest-watch with pytest-testmon as tox -e watch, which effectifely watches your project modules and reruns only those test, which your changes affected.

[webknjaz]

mypy failure is unrelated to this PR, it's related to the version change. 0.610 completed without errors, 0.620 errors out.

[webknjaz]

coverage issue solved. and I've sped-up a number of auxiliary envs.

[webknjaz]

(Circle CI is failing because new image is still in process of building in Docker Hub)

[webknjaz]

I've merged this assuming that all questions are addressed now. However, I'm always open to cosmetic changes.