Testing

Summary: We provide some guidelines on how to run stress tests on your custom application - a crucial step before deployment.

As with all software systems, we test most of our functions. Specifically, we do end-to-end tests that simulate a client answering questions. We use the testing framework pytest to run the tests.

Below we will walk through how include your tests in pytest, how to run them with pytest and some features of pytest.

Generating your own tests

Each test file is designed to be independent of NEXT -- they are designed to have a machine running NEXT (as specified by NEXT_BACKEND_GLOBAL_HOST) and be run from your local machine.

These tests init an experiment on a machine running NEXT (specified by NEXT_BACKEND_GLOBAL_HOST) then simulate a client interacting with NEXT.

We found it easiest to modify existing code. For each file in fooApp/tests/test_api/py, we changed

1. How the experiment was initialized with initExp
2. What the user sent back while answering a getQuery call.

Structure

Conventions for Python test discovery, brief overview: All files named test_*.py or *_test.py are imported, and all functions named test_*() are run without arguments (unless configured).

Of course, it is possible to write a test that does not follow this convention then run this test with python some_test_file.py.

Features

• The default command recurses to find all tests (i.e., py.test runs all tests as defined above)
• The standard output stdout is (by default) ignored. You can view it by running py.test -s.
• pytest allows for package type imports -- i.e., we can say from next.utils import timeit
• pytest can run as many or as few tests as possible
  • one test can be run with py.test test_api.py. Even when cd'd deep in NEXT this still allow package level imports
  • only finds tests recursively for the current directory. Never in parent directories!

Testing the algorithms

The first iteration of algorithm testing should be done on your local machine.

The second stage of testing (writing a NEXT compatible algorithm) needs an input called "butler" (and all the other algorithms described in Algs.yaml).

The Butler is just a database; it can set and get certain keys. We can simulate that by writing to a file. The below implementation shows that when Db is a "database" class that writes to a file (complete implementation shown below).

import pickle
import os

class Butler:
    def __init__(self):
        self.algorithms = Db(name='butler-algs')
        self.experiment = Db(name='butler-exps')

    def job(self, task, task_args_json, ignore_result=True, time_limit=1):
        result = task(**task_args_json)
        if not ignore_result:
            return result

It's simple because there are various behaviors for when key/value are/are not defined. Additionally, key and value can be different types. For more detail, look in next/apps/Butler.py.

This means that your test would look something like:

class myAlg:
    def initExp(self, butler, ...): # ...
    def getQuery(self, butler, ...): # ...
    def processAnswer(self, butler, ...): #...
    def getModel(self, butler, ...): # ...

alg = myAlg()
butler = Butler()

alg.initExp(butler, ...)
for _ in range(100):
    q = alg.getQuery(butler, ...)
    ans = simulateUserResponse(q, ...)
    alg.processAnswer(self, butler, ...)

And the Db class implementation:

import pickle
import os


class Db:
    def __init__(self, name='butler'):
        self.data = {}
        os.system('rm -f .{}.pkl'.format(name))
        self.filename = '.{}.pkl'.format(name)

    def set(self, key=None, value=None):
        self.data.update({key: value})
        self.store()

    def append(self, key=None, value=None):
        self.get_db()
        if key not in self.data.keys():
            self.set(key=key, value=[value])
        else:
            self.data[key] += [value]
        self.store()
        return True

    def get(self, key=None):
        self.get_db()
        if type(key) == str:
            return self.data[key]
        else:
            return {k: self.data[k] for k in key}

    def increment(self, key=None):
        self.get_db()
        self.data[key] += 1
        self.store()
        return self.data[key]

    def get_db(self):
        with open(self.filename, 'rb') as f:
            self.data = pickle.load(f)
        return self.data

    def store(self):
        with open(self.filename, 'wb') as f:
            pickle.dump(self.data, f)