[Backend] KMS upload limits

[linear]

KMS Limits:

Uploads (Local)

• Single file size ? (50Mb / file )

• Single Folder total size ? No limits

• Concurrent file upload ? ( rate limit upload route, for now can't upload whole folders)

• # Files per folder ? -> No limits

• Total storage per tier of users ?

  • 500Mb free
  • 100GB premium

Knowledge linked to Brains ( sync & local )

• Total size for all brains:

  • free: 500 Mb
  • premium: 100Gb for all brains ( get size users in production)
  • No limits for # brains

• Nb or files ?

  • correct metric # of chunks in brain -> have the infos back to the user (too much chunks / files in brain .

• File size across syncs + local ? No diff

• # sync files per user to add to brain ?

  • Limit to # chunks in a brain

KM Processing

• # max concurrent processing of files ?
  • max queue size of 10_000 (check ) 4 worker
• # max concurrent processing files per user
  • fairness scheduler / per user
• Max time processing per file
  • p99 celery worker
• Max batch size sync
  • Randomize across users
  • Priorize refresh for connected users

[linear]

ENT-68 [Backend] KMS upload limits

[github-actions]

Thanks for your contributions, we'll be closing this issue as it has gone stale. Feel free to reopen if you'd like to continue the discussion.