Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve gravity parseLists on low-RAM systems #2320

Closed
wants to merge 1 commit into from

Conversation

DL6ER
Copy link
Member

@DL6ER DL6ER commented Mar 2, 2025

What does this implement/fix?

Use temp_store = FILE to avoid memory exhaustion for a very large number of lists in gravity parseLists. This is nothing else than extending the already merged pi-hole/pi-hole#5977 also to the parseList feature. I did not expect this to be affected, too, however, if single lists already have multi-million domains on them, then this is a problem is your Pi-hole is having only 512 MB total RAM (or even less...).


Related issue or feature (if applicable): Fixes #2194

Pull request in docs with documentation (if applicable): N/A


By submitting this pull request, I confirm the following:

  1. I have read and understood the contributors guide, as well as this entire template. I understand which branch to base my commits and Pull Requests against.
  2. I have commented my proposed changes within the code.
  3. I am willing to help maintain this change if there are issues with it later.
  4. It is compatible with the EUPL 1.2 license
  5. I have squashed any insignificant commits. (git rebase)

Checklist:

  • The code change is tested and works locally.
  • I based my code and PRs against the repositories developmental branch.
  • I signed off all commits. Pi-hole enforces the DCO for all contributions
  • I signed all my commits. Pi-hole requires signatures to verify authorship
  • I have read the above and my PR is ready for review.

…ber of lists in gravity parseLists

Signed-off-by: DL6ER <[email protected]>
@rdwebdesign
Copy link
Member

Looks like this is applied unconditionally in all cases.
Will this change make gravity slower in systems with enough memory? Or it doesn't matter?

@XhmikosR
Copy link

XhmikosR commented Mar 3, 2025

I don't think this is needed. The issue happens when creating the sqlite index. Not sure if this change hurts, but if you were to add this, shouldn't the same change be applied in all places?

Again, this doesn't help since the issue is when creating the index even on the development branch. I can reproduce with 256 RAM and ~2.400.000 domains. If I go down to around 1.400.00 domains RAM + ZRAM seems to be enough.

@DL6ER
Copy link
Member Author

DL6ER commented Mar 3, 2025

Superseded by #2321

@DL6ER DL6ER closed this Mar 3, 2025
@DL6ER DL6ER deleted the tweak/gravity_parseList_RAM branch March 3, 2025 07:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants