Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fake-data generation script + fixes for django-silk and django-debug-toolbar #1128

Merged
merged 7 commits into from
Jan 20, 2023

Conversation

joeyorlando
Copy link
Contributor

@joeyorlando joeyorlando commented Jan 12, 2023

What this PR does

Main stuff

  • add Python script to populate local Grafana/OnCall setup w/ large amounts of fake data. Right now the data types that can be generated are:
    • teams and Admin users via the Grafana API (must be synced manually by going into the UI before going onto the next step)
    • Calendar Schedules which have three 8h oncall-shifts, via the OnCall public API
  • fixes django-debug-toolbar when being run in docker-compose locally

Other stuff

  • documents how to easily modify the Grafana docker-compose container provisioning configuration
  • document solutions for two backend setup related issues encountered when running the engine/celery workers locally, outside of docker-compose, on an Apple silicon Mac
  • fixes small bug in grafana_plugin.helpers.client.APIClient.call_api where it would call response.json() for all requests, regardless of whether or not the response actually contained data or not
  • in engine/settings/dev.py, properly setup django-silk and document the steps to use it locally
  • make it possible to log out debug SQL queries by specifying DEV_DEBUG_VIEW_SQL_QUERIES env var, rather than having to uncomment out a section of settings/dev.py

Which issue(s) this PR fixes

  • Some local setup issues when trying to use django-silk and django-debug-toolbar
  • Makes it much easier to populate your local setup with a lot of fake data
  • Makes it possible to easily modify your local grafana's provisioning configuration

Checklist

  • Tests updated (N/A)
  • Documentation added (N/A)
  • CHANGELOG.md updated (N/A)

@joeyorlando joeyorlando changed the title fix django-debug-toolbar in docker-compose Fix slow RBAC SQL queries Jan 12, 2023
@@ -23,7 +23,7 @@ recurring-ical-events==0.1.16b0
slack-export-viewer==1.0.0
beautifulsoup4==4.8.1
social-auth-app-django==3.1.0
cryptography==39.0.0
cryptography==38.0.4 # version 39.0.0 introduced an issue - https://stackoverflow.com/a/75053968/3902555
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

version 39.0.0 does not work on Apple silicon Macs, version 38.0.4 does

# the below two lines make it possible to use django-debug-toolbar inside of docker locally
# https://knasmueller.net/fix-djangos-debug-toolbar-not-showing-inside-docker
hostname, _, ips = socket.gethostbyname_ex(socket.gethostname())
INTERNAL_IPS += [".".join(ip.split(".")[:-1] + ["1"]) for ip in ips]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this allows the django-debug-toolbar to work when running the backend inside of docker-compose

Comment on lines +19 to +20
- [distutils.errors.CompileError: command '/usr/bin/clang' failed with exit code 1](#distutilserrorscompileerror-command-usrbinclang-failed-with-exit-code-1)
- [symbol not found in flat namespace '\_EVP_DigestSignUpdate'](#symbol-not-found-in-flat-namespace-_evp_digestsignupdate)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these were the two issues I was experiencing when running make run-backend-celery locally on an Apple Silicon Mac

@joeyorlando joeyorlando force-pushed the jorlando/fix-slow-rbac-permissions-query branch from a4b120d to 96d0580 Compare January 16, 2023 09:28
@joeyorlando joeyorlando changed the title Fix slow RBAC SQL queries Optimize GET /schedules API endpoints Jan 19, 2023
@joeyorlando joeyorlando force-pushed the jorlando/fix-slow-rbac-permissions-query branch from 9be5db2 to d327f7f Compare January 20, 2023 07:07
don't call response.json() if there is no response body
(ex. in the case of a response for a HEAD call)
local grafana docker-compose container
when trying to run the server/celery worker
outside of docker on an M1 Mac
bump django-silk version to v5.0.3 (latest version)
- creating Grafana teams/users
- creating calendar schedules with three 8h oncall shifts
@joeyorlando joeyorlando force-pushed the jorlando/fix-slow-rbac-permissions-query branch 2 times, most recently from 6982986 to eaac9a7 Compare January 20, 2023 07:41
@joeyorlando joeyorlando changed the title Optimize GET /schedules API endpoints fake-data generation script + fixes for django-silk and django-debug-toolbar Jan 20, 2023
@joeyorlando joeyorlando marked this pull request as ready for review January 20, 2023 07:44
@joeyorlando joeyorlando requested review from a team and iskhakov January 20, 2023 07:44
@joeyorlando joeyorlando merged commit 98241b9 into dev Jan 20, 2023
@joeyorlando joeyorlando deleted the jorlando/fix-slow-rbac-permissions-query branch January 20, 2023 08:19
joeyorlando added a commit that referenced this pull request Jan 25, 2023
# What this PR does

Fixes slow internal`GET /schedules` endpoints. Using the fake-data
generation script in #1128, I generated 65 calendar schedules in my
local setup. This resulted in the following endpoint performance:
![Screenshot 2023-01-24 at 12 03
16](https://user-images.githubusercontent.com/9406895/214276618-1a9848ba-eb84-49ec-a099-fdd96beac93f.png)

The responses which show ~76 queries were run on the latest `dev`
branch. Responses w/ ~26 queries were run on this branch.

Additionally:
- add typing to a few methods in `apps/schedules/ical_utils.py`
- document `apps/api/permissions/__init__.py:user_is_authorized`
function

## Which issue(s) this PR fixes

grafana/oncall-private#1552

## Checklist

- [ ] Tests updated
- [ ] Documentation added
- [ ] `CHANGELOG.md` updated

Co-authored-by: Vadim Stepanov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants