Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): Add gRPC Compression #4135

Open
walldiss opened this issue Feb 25, 2025 · 1 comment
Open

feat(core): Add gRPC Compression #4135

walldiss opened this issue Feb 25, 2025 · 1 comment
Labels
enhancement New feature or request

Comments

@walldiss
Copy link
Member

Implementation ideas

Problem
Right now, all gRPC traffic between Celestia Node and the Core server is uncompressed. This leads to bigger data transfers and slower performance. By enabling gRPC’s built-in compression, we can shrink the data payload, use less bandwidth, and speed up node operations.

Solution
Use gRPC options (e.g., GZIP,snappy,zstd)

https://grpc.io/docs/guides/compression/

@walldiss walldiss added the enhancement New feature or request label Feb 25, 2025
@aWN4Y25pa2EK
Copy link

aWN4Y25pa2EK commented Feb 28, 2025

Algorithm Compression Ratio Compression Speed Decompression Speed CPU Usage Memory Usage Streaming Random Access Library Support Notes
Zstandard (zstd) High Fast Fast Moderate Moderate Yes Yes Good Balanced performance, supports streaming and random access.
LZ4 Low to Medium Very Fast Very Fast Low Low Yes Limited Good Extremely fast, but lower compression ratio.
Snappy Low to Medium Very Fast Very Fast Low Low Yes Limited Good Good for short, repetitive data patterns.
Gzip Medium to High Slow Moderate Moderate Moderate No No Excellent Widely supported, but not designed for streaming or random access.
Brotli Very High Slow Moderate High High Yes No Good High compression ratio, slower compression speed.

Explanation:

  • Compression Ratio: How much the data can be compressed. Higher is better for reducing data size.
  • Compression Speed: How fast the algorithm compresses data. Faster is better for maintaining block time.
  • Decompression Speed: How fast the algorithm decompresses data. Faster is better for node performance.
  • CPU Usage: The computational resources required. Lower is better for scalability.
  • Memory Usage: Memory footprint during compression/decompression. Lower is better for handling large blocks.
  • Streaming: Ability to compress/decompress data in chunks. Important for large data sets.
  • Random Access: Ability to access specific parts of the compressed data without decompressing the entire block. Useful for partial block retrieval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants