fix: unreadable symbols in DS stream on client side #1750
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
IvanBelyakoff
force-pushed
the
merge/ds_unreadable_symbols
branch
from
bcec38c to
89c5ac0
Compare
revitteth
approved these changes
Feb 17, 2025
revitteth
added a commit
that referenced
this pull request
Feb 21, 2025
* Merge pull request #1565 from 0xPolygonHermez/fix/ws-http-same-port Fix/ws http same port * tweak(counters): poseidon 30 to 31 (#1587) (#1589) * add flag to reject transactions below default config (#1602) (#1604) * 2.61 fork upgrade changes (#1615) * allow the sequencer to always check for new fork information (#1606) * include all used info tree indexes in the witness * configurable witness unwind limit (#1617) * Fix/2.61 upgrade fork changes (#1630) * sequencer to go through stage loop whilst halted * updates to L1 processing when network is in a halted state * Fix/stop repeat overflow transactions (#1640) (#1644) * fix logging of hash in pool for counter overflows * better handling of single transactions overflowing batches * better handling of hash logging fix in pool for overflows Conflicts: zk/hermez_db/db.go zk/stages/stage_sequence_execute.go * resolve test issues around bad tx hash table (#1646) * 0tx block handling and moving to timers from tickers (#1673) * 0tx block handling and moving to timers from tickers * tweak(ci): stable branches triggers (#1675) * run ci on stable branches * tweak(ci): lint docs on stable --------- Co-authored-by: Max Revitt <[email protected]> Co-authored-by: Max Revitt <[email protected]> * sender lock to help prevent incorrect nonce checks (#1643) (#1671) (#1681) * sender lock to help prevent incorrect nonce checks (#1643) * sender lock to help prevent incorrect nonce checks helps to ensure the pool has finished working with a sent transaction before allowing any nonce checks to be performed. Only applies to pending nonce, latest won't use this lock. * tidy up on waiting for sender lock * DS client - add TLS support (#1553) (#1682) Add zkevm.l2-datastreamer-use-tls flag If set to true, a TLS connection will be used. This also enables SNI (Server Name Indication) support and allows connecting to ds server behind virtual hosting/load balancer Co-authored-by: Ivan Zubok <[email protected]> * fix: test SpawnSequencingStage timeouts issue (#1684) * perf: remove unneded mem alloc in combine collectors (#1546) (#1547) (#1686) Co-authored-by: Valentin Staykov <[email protected]> * fix: add check for tx is Create if init code size exceeds the (#1659) limit Co-authored-by: Max Revitt <[email protected]> * fix: switching to thread 0 (#1707) * fix: 0 tx blocks after limbo (#1708) (#1711) * fix: 0 tx blocks after limbo * Add executor-enabled flag. It disables using remote executor even when executor-urls option is filled in, which is helpful when testing limbo mode, when txs are still sent to the remote executor. * fix: recover sender only for those txs that are included (#1636) (#1710) * fix: recover sender only for those txs that are included * Moved recovering sender out of addTransaction. Default context used for sender recovery. * commit block update to state before removing mined transactions (#1717) without this change there is a very small window of time where you could request a pending nonce and the pool has had the transactions removed before the latest executed block is updated in the DB so you in fact get the "latest" nonce from state for an old block instead. we also slightly reduce the locks on the pool by combining the two slices of transactions that we want to remove from the pool. * fix: crash on nil block if no txs mined (#1716) * 2.61 deadlocks (#1713) * pool to manage flush locking internally * acquire semaphore before tracking tx begin * set no gossip to default to true This really isn't needed in zkevm networks as we don't support p2p gossip and it creates load on the DB that we simply don't need. Some of the deadlocks we've seen, once released, log errors from the gossip code on repeat so better we don't attempt it at all in this case Setting the flag to false will override this default so we can re-enable when the time suits * do not lock on waiting for the open transactions to close if we lock here then we can't close down already open transactions because they will deadlock as they enter the function * update to mdbx deadlock code test allows the test to run to completion without thread exhaustion * useful comment about the stacked locks in the pool best function * enable pool p2p gossip for unit tests that rely on the old default behaviour * remove unnecessary 2nd RO tx in send transaction * do not call track end on error acquiring the semaphore * increase timeout in stream client tests * reduce logging noise for common complaints (#1722) * temporarily halt processing tx for a sender when they encounter overflow (#1709) * temporarily halt processing tx for a sender when they encounter overflow previously during an overflow situation we would continue to attempt to process transactions for the same sender that were yielded. In networks that have a few very busy accounts this created nonce too high problems which we then incorrectly attempted to remove from the pool. Eventually this led to transactions sitting in the queued pool that actually had a valid nonce and should be processed. Now we pause processing for a sender that encountered an overflow until the next batch so that we don't create these problems. * adjustments to sendersToSkip after changes to sender recovery logic * adjustment of error handling (#1725) no more special treatment for nonce too high. They stick around in the pool for too long and we keep attempting to process them so they should be discarded like the others. this will also update other transactions in the pool for the same sender and move those over into the queued pool which also helps * fix(gas-pool): create a gas pool per block in normalcy mode on sequencer [stable 2.61] (#1720) * fix(gas-pool): create a gas pool per block in normalcy mode on sequencer fix(gas-pool): changed name normalcy gas pool * fix(gas-pool): fix for naming on gas pool and pull to block level in sequencer --------- Co-authored-by: Elliot Hallam <[email protected]> * tweak(deps): upgrade datastreamer (#1719) * fix(eth-logs): create eth logs max limit flag (#1706) (#1735) * create eth logs max limit flag fix(eth-logs): create eth logs max limit flag to prevent ddos risk fix(eth-logs): fix tests * fix(eth-logs): change message to eth logs max range message to block range too large * fix(eth-logs): add unit test for eth get logs using log max range * tweak(api): add block range max to limit response --------- Co-authored-by: Max Revitt <[email protected]> tweak(eth-logs): tests * tweak(tx-pool-grpc): add sub pool type to panic (#1733) tweak(tx-pool-grpc): log error but still if not valid type tweak(tx-pool-grpc): add sender * cleaner shut down (#1734) (#1736) shared mutex in the sync.Cond caused a panic on unlocking an unlocked mutex as it was shared with other logic * do not commit new block details to the DB during limbo processing (#1737) * do not commit new block details to the DB during limbo processing * do not clean up on unwind test completion * nonce issue detection and remedy in execution (#1741) * nonce issue detection and remedy in execution here we pause handling further transactions for the sender in this batch and also trigger the pool to perform a sender state change to remove nonce too low transactions and move nonce too high to queued * mop up discarded transactions during yielding correctly * Ignore DbInfo in unwind test * fix: respect transaction execution counters (#1757) * lock the pool flush mutex before calling yield (#1758) There was a race condition here where opening the db view for yielding didn't contain info from the DB that wasn't flushed completely yet. Now the flush can't start running until the DB view has been used and vice versa for yielding, we now wait for the flush to finish. Tested using the polycli load tool which is what highlighted this issue in the first place * fix(counters): smt depth fix and test (#1695) (#1751) * fix(counters): smt depth fix and test * fix(smt): set depth and get depth correctly * fix(smt): depth update Co-authored-by: Max Revitt <[email protected]> * fix: nil crash on uninitialized block var (#1763) * fix: unreadable symbols in DS stream on client side (#1750) * feat(witness-cache) write witness cache by batch and purge via flag feat(witness-cache) write witness cache per verified batch and offset wip feat(witness-cache) truncate witness cache properly feat(witness-cache) ad purge witness cache to prestart feat(web) removed logs for skipping witness cache stage feat(witness-cache) only cache trimmed and only get cached trimmed on rpc call feat(witness-cache) revert buildsmtfromwitness to take single witness * tweak(witness-cache): tweak witness cache flags for offset ahead and below * tweak(witness-cache): cache 1 down from highest seen * fix(rpc): stop getblockhash from crashing when sending hash (#1761) fix(rpc): allow block hash for eth get receipts * improve(smt/pkg/utils): improve the performance of HashContractBytecodeBigInt() (#1772) (#1780) * improve the performance of HashContractBytecodeBigInt() * update go.sum * update poseidongold to v0.0.3 Co-authored-by: Dumi Loghin <[email protected]> * fix(merge): fix merge artifact * fix(merge): dupe flag * fix(execute): separate flush context on shutdown --------- Co-authored-by: Moretti Georgiev <[email protected]> Co-authored-by: Scott Fairclough <[email protected]> Co-authored-by: Ivan Zubok <[email protected]> Co-authored-by: Ivan Belyakov <[email protected]> Co-authored-by: Valentin Staykov <[email protected]> Co-authored-by: Elliot Hallam <[email protected]> Co-authored-by: elliothllm <[email protected]> Co-authored-by: Dumi Loghin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merged #1698