pRAG

Conceptual model

How to run

Install python 3.11

brew install [email protected]
# you will also need cmake
brew install cmake
# you will also need libmagic for MIME type detection
brew install libmagic

Create a virtual environment and activate it.

python3.11 -m venv "venv"
source venv/bin/activate

for Windows, You need to activate by below command

venv\Scripts\activate

Install the dependencies

pip install --upgrade pip
pip install setuptools --upgrade
pip install -r requirements.txt

Local environment variables. Create .env file and put your secrets there. (It will not be committed to the repo)

OPENAI_API_KEY=sk-...

Next, you need to set CRYPTO_KEY as environment variable. to get the value run:

python application/shared/crypto/generate_crypto_key.py

and store the generated key in the .env file

API key authentication

The environment variable API_AUTHENTICATION_ENABLED controls whether to enable API key authentication for API endpoints. It is recommended to keep this turned off for local development. Otherwise, you should generate an API key through Django admin. The generated API key should be passed as the X-API-Key header in requests targeting API endpoints. You can add exceptions by modifying the PUBLIC_ENDPOINTS constant in settings.py.

Pycharm

To run have the project properly setup in pycharm. One needs to provide the path to the python interpreter.

Linting

We use flake8 for linting. To run it, run

cd application 
python -m flake8 --count

Docker

To run it with docker/compose, run

python manage.py collectstatic
docker compose --profile run_app up -d --build
# optional 
python manage.py createsuperuser
# instead of creating a superuser, you can use the existing super user that is in the database
# username: admin
# password: pragadmin24

To run the app/server locally outside docker

#stop the app in docker compose
docker-compose stop app
# or 
docker-compose down
docker compose up -d
# the app service will not be started if we do not activate profile run_app
python -m uvicorn application.asgi:app --host 0.0.0.0 --port 8001 --reload
## You will need a dependency services coming from docker-compose

You need to perform migration to initialize the database

python manage.py makemigrations
python manage.py migrate

Weaviate Vector database:

We use Weaviate as our default local vector database. It provides a graphQL interface to query the vectors. Visit https://console.semi.technology/ and connect to http://localhost:8080 an example graphQL query:

query {
  Get {
    Insurance {
      chapter_index
      document_id
      chunk
    }
  }
}
When we ingest documents a schema is autogenerated by Weaviate. In the case above Insurance represents the index = workspace
example query with object id from vector db
query {
  Get {
    Insurance(where: {
      operator: Equal,
      valueNumber: 1,
      path: ["document_id"]
    }) {
      _additional{id}
      chapter_index
      document_id
      chunk
    }
  }
}

Important urls:

• http://localhost:8001/api/docs
• http://localhost:8001/django/admin

login credentials for admin

• username: admin
• password: pragadmin24

Cheat sheets

Django cheatsheet: https://dev.to/ericchapman/my-beloved-django-cheat-sheet-2056

FastApi cheatsheet: https://lyz-code.github.io/blue-book/fastapi/

Local LLM

We are using ollama to provide us with local llms. At the moment the mac docker app does not support access to underlaying gpu. Therefore the host instance or another GPU instance has to run the llm model. More about ollama: https://python.langchain.com/docs/integrations/llms/ollama

1. install ollama using brew or other method 
2. ollama pull llama2 #ollama pull <model family>
3. ollama serve
# ollama will serve the model at localhost:11434
you can also try a bigger llama2 model. for example: llama2:13b-chat-q5_0
1. ollama pull llama2:13b-chat-q5_0
2. # stop ollama server
3. ollama serve

You can test you ollama server with:

curl http://localhost:11434/api/generate -d '{
  "model": "llama2:13b-chat-q5_0",
  "prompt": "Why is the sky blue?"
}'

We use host.docker.internal to access ollama from docker container.

Neo4j

We support Q&A for Neo4j. We have an example db with web shop data at:

url: bolt+s://a9b44a00.databases.neo4j.io:7687
user: neo4j
db: neo4j
pass: 'Yr68lFOnOhCi70FXTX9RFhDsyXfpwvNF83i90vK5Fi8'

Remarks:

• pyTorch 2.1.2 as of Jan 4th 2024 is not yet compatible with Python 3.12. Therefor the Preview (Nightly build is used)
• The code is still a mess and needs work on different areas. A lot of hardcoded values and so on
• The referenced external resources tend to be very unstable in terms of integration stability and breaking changes. Here is working set versions:
  • llama-index 0.9.25.post1
  • openai 1.6.1
  • pypdf 3.17.4
  • transformers 4.36.2
  • torch 2.3.0.dev20240104
  • torchaudio 2.2.0.dev20240104
  • torchvision 0.18.0.dev20240104