Python SDK version 1.1 (#263)

* style: fix linter errors in tests

* docs: correct inline documentation for MetricsApiConfig class

* refactor: move METRICS_API to MetricsApiConfig so that it can be overridden during SDK development

* feat: add a logger for debugging

* refactor: move submission code to its own file

* feat: add timeout parameter

* fix: remove unnecessary debugging statements

* feat: when the process is exiting, send any queued requests and wait for all outstanding network calls to complete

* refactor: move ALLOWED_HTTP_HOSTS enforcement into the core Metrics processing object, so it can be shared across multiple implementations (coming soon)

* docs: clean up README.md and make sure all parameters are documented

* feat: make it easier for users to override the new `timeout` parameter

* release: bump version to 1.1.0

* Ignore .envrc files (sometimes used in Python development)

* style: proper formatting with black

Author: RyanGWU82

.gitignore

@@ -1,3 +1,4 @@
+.envrc
 .vscode/
 node_modules/
 packages/*/node_modules/

packages/python/README.md

@@ -15,7 +15,7 @@ pip install readme-metrics
 
 ## Usage
 
-Just include the `MetricsMiddleware` into your API!
+Just include the `MetricsMiddleware` in any WSGI app!
 
 ```python
 from readme_metrics import MetricsApiConfig, MetricsMiddleware
@@ -25,8 +25,8 @@ app = Flask(__name__)
 app.wsgi_app = MetricsMiddleware(
     app.wsgi_app,
     MetricsApiConfig(
-        '<<apiKey>>',
-        lambda req: {
+        api_key='<<your-readme-api-key>>',
+        grouping_function=lambda req: {
             'api_key': 'unique api_key of the user',
             'label': 'label for us to show for this user (ie email, project name, user name, etc)',
             'email': 'email address for user'
@@ -39,25 +39,25 @@ app.wsgi_app = MetricsMiddleware(
 
 There are a few options you can pass in to change how the logs are sent to ReadMe. These can be passed in `MetricsApiConfig`.
 
-Ex)
-
 ```python
 MetricsApiConfig(
-    '<<apiKey>>',
-    lambda req: {
+    api_key='<<your-readme-api-key>>',
+    grouping_function=lambda req: {
         'api_key': 'unique api_key of the user',
         'label': 'label for us to show for this user (ie email, project name, user name, etc)',
         'email': 'email address for user'
     },
     buffer_length=1,
-    denylist=['credit_card'] # Prevents credit_card in the request from being sent to readme
+    denylist=['password'] # Prevents a request or response's "password" field from being sent to ReadMe
 )
 ```
 
-| Option             | Use                                                                                                                                                                                                                           |
-| :----------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| development_mode   | **default: false** If true, the log will be separate from normal production logs. This is great for separating staging or test data from data coming from customers                                                           |
-| denylist           | **optional** An array of keys from your API requests and responses headers and bodies that you wish to denylist from sending to ReadMe.<br /><br />If you configure a denylist, it will override any allowlist configuration. |
-| allowlist          | **optional** An array of keys from your API requests and responses headers and bodies that you only wish to send to ReadMe.                                                                                                   |
-| buffer_length      | **default: 10** Sets the number of API calls that should be recieved before the requests are sent to ReadMe                                                                                                                   |
-| allowed_http_hosts | A list of allowed http hosts for sending data to the ReadMe API.                                                                                                                                                              |
+| Option                 | Use                                                |
+| :--------------------- | :------------------------------------------------- |
+| development_mode       | **default: false** If true, the log will be separate from normal production logs. This is great for separating staging or test data from data coming from customers. |
+| background_worker_mode | **default: true** If true, requests to the ReadMe API will be made in a background thread. If false, the ReadMe API request will be made synchronously in the main thread, potentially slowing down your HTTP service. |
+| denylist               | **optional** An array of keys from your API requests and responses headers and bodies that are blocked from being sent to ReadMe. Both the request and response will be checked for these keys, in their HTTP headers, form fields, URL parameters, and JSON request/response bodies. JSON is only checked at the top level, so a nested field will still be sent even if its key matches one of the keys in `denylist`.<br /><br />If you configure a denylist, it will override any allowlist configuration. |
+| allowlist              | **optional** An array of keys from your API requests and responses headers and bodies that you only wish to send to ReadMe. All other semantics match `denylist`. Like `denylist`, only the top level of JSON request/response bodies are filtered. If this option is configured, **only** the whitelisted properties will be sent. |
+| buffer_length          | **default: 10** Sets the number of API calls that should be recieved before the requests are sent to ReadMe. |
+| allowed_http_hosts     | A list of HTTP hosts which should be logged to ReadMe. If this is present, a request will only be sent to ReadMe if its Host header matches one of the allowed hosts. |
+| timeout                | Timeout (in seconds) for calls back to the ReadMe Metrics API. Default 3 seconds. |

packages/python/readme_metrics/Metrics.py

@@ -1,3 +1,5 @@
+import atexit
+import math
 import queue
 import threading
 import requests
@@ -6,18 +8,19 @@
 
 from werkzeug import Request
 
-from readme_metrics import MetricsApiConfig, ResponseInfoWrapper
+from readme_metrics import MetricsApiConfig
+from readme_metrics.publisher import publish_batch
 from readme_metrics.PayloadBuilder import PayloadBuilder
+from readme_metrics.ResponseInfoWrapper import ResponseInfoWrapper
 
 
 class Metrics:
     """
-    This is the internal central controller classinvoked by the WSGI middleware. It
-    handles the creation, queueing, and submission of the requests.
+    This is the internal central controller class invoked by the ReadMe middleware. It
+    queues requests for submission. The submission is processed by readme_metrics.publisher.publish_batch().
     """
 
     PACKAGE_NAME: str = "readme/metrics"
-    METRICS_API: str = "https://metrics.readme.io"
 
     def __init__(self, config: MetricsApiConfig):
         """
@@ -37,38 +40,46 @@ def __init__(self, config: MetricsApiConfig):
         )
         self.queue = queue.Queue()
 
+        atexit.register(self.exit_handler)
+
     def process(self, request: Request, response: ResponseInfoWrapper) -> None:
         """Enqueues a request/response combination to be submitted the API.
 
         Args:
             request (Request): Request object
             response (ResponseInfoWrapper): Response object
         """
-        self.queue.put(self.payload_builder(request, response))
+        if not self.host_allowed(request.environ["HTTP_HOST"]):
+            self.config.LOGGER.debug(
+                f"Not enqueueing request, host {request.environ['HTTP_HOST']} not in ALLOWED_HTTP_HOSTS"
+            )
+            return
 
+        self.queue.put(self.payload_builder(request, response))
         if self.queue.qsize() >= self.config.BUFFER_LENGTH:
+            args = (self.config, self.queue)
             if self.config.IS_BACKGROUND_MODE:
-                threading.Thread(target=self._processAll, daemon=True).start()
+                thread = threading.Thread(target=publish_batch, daemon=True, args=args)
+                thread.start()
             else:
-                self._processAll()
+                publish_batch(*args)
 
-    def _processAll(self) -> None:
-        result_list = []
-        while not self.queue.empty():
-            obj = self.queue.get_nowait()
-            if obj:
-                result_list.append(obj)
+    def exit_handler(self) -> None:
+        if not self.queue.empty():
+            args = (self.config, self.queue)
+            for _ in range(math.ceil(self.queue.qsize() / self.config.BUFFER_LENGTH)):
+                if self.config.IS_BACKGROUND_MODE:
+                    thread = threading.Thread(
+                        target=publish_batch, daemon=True, args=args
+                    )
+                    thread.start()
+                else:
+                    publish_batch(*args)
+        self.queue.join()
 
-        payload = json.dumps(result_list)
-
-        version = importlib.import_module(__package__).__version__
-
-        readme_result = requests.post(
-            self.METRICS_API + "/request",
-            auth=(self.config.README_API_KEY, ""),
-            data=payload,
-            headers={
-                "Content-Type": "application/json",
-                "User-Agent": "readme-metrics-python@" + version,
-            },
-        )
+    def host_allowed(self, host):
+        if self.config.ALLOWED_HTTP_HOSTS:
+            return host in self.config.ALLOWED_HTTP_HOSTS
+        else:
+            # If the allowed_http_hosts has not been set (None by default), send off the data to be queued
+            return True

packages/python/readme_metrics/MetricsApiConfig.py

@@ -1,36 +1,43 @@
 from typing import List, Any, Callable
 
+from readme_metrics.util import util_build_logger
+
 
 class MetricsApiConfig:
     """ReadMe Metrics API configuration object
 
     Attributes:
-        README_API_KEY (str): (required) Your ReadMe API key
-        GROUPING_FUNCTION (lambda): (required) Grouping function to construct an
+        README_API_KEY (str) Your ReadMe API key
+        GROUPING_FUNCTION (lambda): Grouping function to construct an
             identity object. It receives the current request as a parameter, and must
             return a dictionary containing at least an "id" field, and optionally
             "label" and "email" fields.
 
             The main purpose of the identity object is to identify the API's caller.
-        BUFFER_LENGTH (int, optional): Number of requests to buffer before sending data
+        BUFFER_LENGTH (int): Number of requests to buffer before sending data
             to ReadMe. Defaults to 10.
-        IS_DEVELOPMENT_MODE (bool, optional): Determines whether you are running in
+        IS_DEVELOPMENT_MODE (bool): Determines whether you are running in
             development mode. Defaults to False.
-        IS_BACKGROUND_MODE (bool, optional):  Determines whether to issue the call to
+        IS_BACKGROUND_MODE (bool):  Determines whether to issue the call to
             the ReadMe API in a background thread. Defaults to True.
-        DENYLIST (List[str], optional): An array of headers and JSON body properties to
+        DENYLIST (List[str]): An array of headers and JSON body properties to
             skip sending to ReadMe.
 
             If you configure a denylist, it will override any allowlist configuration.
-        ALLOWLIST (List[str], optional): An array of headers and JSON body properties to
+        ALLOWLIST (List[str]): An array of headers and JSON body properties to
             send to ReadMe.
 
             If this option is configured, ONLY the allowlisted properties will be sent.
-        BLACKLIST (List[str], optional): Deprecated, prefer using an denylist.
-        WHITELIST (List[str], optional): Deprecated, prefer using an allowlist.
-        ALLOWED_HTTP_HOSTS (List[str] (optional)): A list of allowed http hosts for sending
+        ALLOWED_HTTP_HOSTS (List[str]): A list of allowed http hosts for sending
             data to the ReadMe API.
+        METRICS_API (str): Base URL of the ReadMe metrics API.
+        METRICS_API_TIMEOUT (int): Timeout (in seconds) for metrics API calls.
+        LOGGER (logging.Logger): Logger used by all classes and methods in the
+            readme_metrics packge. Defaults to a basic console logger with log level
+            CRITICAL.
 
+            You can adjust logging settings by manipulating LOGGER, or you can replace
+            LOGGER entirely with your application's Logger.
     """
 
     README_API_KEY: str = None
@@ -41,6 +48,8 @@ class MetricsApiConfig:
     DENYLIST: List[str] = []
     ALLOWLIST: List[str] = []
     ALLOWED_HTTP_HOSTS: List[str] = []
+    METRICS_API: str = "https://metrics.readme.io"
+    METRICS_API_TIMEOUT: int = 3
 
     def __init__(
         self,
@@ -54,6 +63,7 @@ def __init__(
         blacklist: List[str] = None,
         whitelist: List[str] = None,
         allowed_http_hosts: List[str] = None,
+        timeout: int = 3,
     ):
         """Initializes an instance of the MetricsApiConfig object
 
@@ -71,20 +81,29 @@ def __init__(
                 development mode. Defaults to False.
             background_worker_mode (bool, optional): Determines whether to issue the
                 call to the ReadMe API in a background thread. Defaults to True.
-            denylist (List[str], optional): An array of headers and JSON body
-                properties to skip sending to ReadMe. Defaults to None.
+            denylist (List[str], optional): An array of keys from your API requests and
+                responses headers and bodies that are blocked from being sent to ReadMe.
+                Both the request and response will be checked for these keys, in their
+                HTTP headers, form fields, URL parameters, and JSON request/response
+                bodies. JSON is only checked at the top level, so a nested field will
+                still be sent even if its key matches one of the keys in `denylist`.
+                Defaults to None.
 
                 If you configure a denylist, it will override any allowlist
                 configuration.
             allowlist (List[str], optional): An array of headers and JSON body
-                properties to send to ReadMe. Defaults to None.
+                properties to send to ReadMe. Similar semantics to `denylist`; defaults
+                to None.
 
                 If this option is configured, ONLY the whitelisted properties will be
                 sent.
             blacklist (List[str], optional): Deprecated, prefer denylist.
             whitelist (List[str], optional): Deprecated, prefer allowlist.
-            allowed_http_hosts (List[str], optional): A list of allowed http hosts for sending data
-                to the ReadMe API.
+            allowed_http_hosts (List[str], optional): A list of HTTP hosts which should be
+                logged to ReadMe. If this is present, requests will only be sent to ReadMe
+                whose Host header matches one of the allowed hosts.
+            timeout (int): Timeout (in seconds) for calls back to the ReadMe Metrics API.
+                Default 3 seconds.
         """
         self.README_API_KEY = api_key
         self.GROUPING_FUNCTION = grouping_function
@@ -94,3 +113,5 @@ def __init__(
         self.DENYLIST = denylist or blacklist or []
         self.ALLOWLIST = allowlist or whitelist or []
         self.ALLOWED_HTTP_HOSTS = allowed_http_hosts
+        self.METRICS_API_TIMEOUT = timeout
+        self.LOGGER = util_build_logger()

packages/python/readme_metrics/MetricsMiddleware.py

@@ -108,12 +108,7 @@ def _start_response(_status, _response_headers, *args):
                 )
 
                 # Send off data to be queued (and processed) by ReadMe if allowed
-                if self.config.ALLOWED_HTTP_HOSTS:
-                    if environ["HTTP_HOST"] in self.config.ALLOWED_HTTP_HOSTS:
-                        self.metrics_core.process(req, res)
-                else:
-                    # If the allowed_http_hosts has not been set (None by default), send off the data to be queued
-                    self.metrics_core.process(req, res)
+                self.metrics_core.process(req, res)
 
                 yield data
 

packages/python/readme_metrics/__init__.py

@@ -1,4 +1,4 @@
 from readme_metrics.MetricsApiConfig import MetricsApiConfig
 from readme_metrics.MetricsMiddleware import MetricsMiddleware
 
-__version__ = "1.0.6"
+__version__ = "1.1.0"

packages/python/readme_metrics/publisher.py

@@ -0,0 +1,48 @@
+import importlib
+import json
+import math
+from queue import Empty, Queue
+import time
+
+import requests
+
+
+def publish_batch(config, queue):
+    result_list = []
+    try:
+        try:
+            while not queue.empty() and len(result_list) < config.BUFFER_LENGTH:
+                payload = queue.get_nowait()
+                result_list.append(payload)
+        except Empty:
+            pass
+
+        if len(result_list) == 0:
+            return
+
+        version = importlib.import_module(__package__).__version__
+        url = config.METRICS_API + "/request"
+
+        readme_result = requests.post(
+            url,
+            auth=(config.README_API_KEY, ""),
+            data=json.dumps(result_list),
+            headers={
+                "Content-Type": "application/json",
+                "User-Agent": f"readme-metrics-python@{version}",
+            },
+            timeout=config.METRICS_API_TIMEOUT,
+        )
+        config.LOGGER.info(
+            f"POST to {url} with {len(result_list)} items returned {readme_result.status_code}"
+        )
+        if not readme_result.ok:
+            config.LOGGER.exception(readme_result.text)
+            raise Exception(f"POST to {url} returned {readme_result.status_code}")
+    except Exception as e:
+        # Errors in the Metrics SDK should never cause the application to
+        # throw an error. Log it but don't re-raise.
+        config.LOGGER.exception(e)
+    finally:
+        for _ in result_list:
+            queue.task_done()

packages/python/readme_metrics/tests/MetricsMiddleware_test.py

@@ -1,4 +1,4 @@
-import pytest
+import pytest  # pylint: disable=import-error
 import requests
 import json
 
@@ -49,10 +49,6 @@ def mockMiddlewareConfig():
     )
 
 
-# Verify that metrics has fields for metrics API and package name
-assert Metrics.METRICS_API != None
-assert Metrics.PACKAGE_NAME != None
-
 # Mock callback for handling middleware response
 class MetricsCoreMock:
     def process(self, req, res):