[Import Genesis] Speeding up the import genesis time

[p0mvn]

Background

We need to understand what is happening during the import genesis and come up with a solution to improve time.

Acceptance Criteria

• Import genesis time is well understood
• A plan to mitigate the problem is created

[p0mvn]

Hi @moniya12 . Are you on this? If yes, what is the status? Thanks!

[catShaark]

@p0mvn, I did some work on this and so far I've reduced the importing time by maybe a half. I did the profiling by running this test file which contains the code needed for importing genesis used in the osmosisd start cmd.

[czarcas7ic]

@catShaark out of curiosity, how much time does it now take compared to what it did before? What changes did you make to achieve this?

[catShaark]

Changes I did :

• Improve the DelegatorSharesInvariant invariant check in the sdk. It reduces runtime from 241 to 161 sec in one of my computer.
• Makes the unmarshaling of modules' genesis state parallel. Currently, we're unmarshaling modules' GenesisState sequentially. Each module have this InitGenesis func which does the work of both unmarshaling the json.RawMessage into its self-defined GenesisState struct and running module initialization with the resulted GenesisState, for example. Right now we're iterating through all the modules in ModuleManager.InitGenesisOrder and call InitGenesis for each module we run through. I make some changes so that we unmarshal GenesisState for all modules parallelly, then run the module initialization logic for each modules sequentially in ModuleManager.InitGenesisOrder .This reduces runtime for about 30-40 sec. Tho this means adding new function for AppModule interface which is a big deal

[czarcas7ic]

@catShaark So just to give you some more context, this issue was in reference to state exported testnets. Basically, when we take a state export, modify the values, then start a new testnet off this new genesis the following happens:

1. Init genesis starts, this process takes roughly 15 minutes
2. We hit the block after the snapshot was taken
3. We then pause on this block for about an hour and a half, using a large amount of RAM (almost 64GB)
4. After we hit the next block, we hit blocks like normal

I think your changes will help with number 1, but number 3 is where the most time is spent. @moniya12 was working on some database changes to potentially help with this.

[catShaark]

take a state export = running osmosisd export right ?

[czarcas7ic]

@catShaark correct! That takes ~5-10 mins max

[catShaark]

I import a 1.8G genesis file ( exported from an osmosis mainet node ) and it takes only 200-300 sec. Do I do something wrong ? Can you give me a genesis.json file that you guys used

[czarcas7ic]

In our scenario, we are modifying the genesis the mint new coins to a wallet, change a validator's consensus key to the one that we have locally, and give it 99 percent of the voting power. You can read a little more about the process here https://github.com/osmosis-labs/osmosis/tree/main/tests/localosmosis#localosmosis-with-mainnet-state and see the python script being used to modify the genesis here https://github.com/osmosis-labs/osmosis/blob/main/cmd/osmosisd/cmd/testnetify/testnetify.py

Tomorrow I can provide you with a modified one with the required consensus key if desired

[catShaark]

thanks, I'd love to work on this to improve our test runtime overall
also, can you tell me what you think happened on step 3, why is it taking so long

[catShaark]

• Import genesis time is well understood

here is a pprof graph

, this is importing a 1.8G genesis file with improvement on staking invariant ( I run on a different computer so it takes 333 sec ) , I run in it with this test file.

[p0mvn]

@catShaark thanks for the pprof. From what I remember, we had a specific case where our pprof was taking more than an hour, halting for over 40 minutes due to processing data.

Seems that @czarcas7ic can guide us tomorrow in terms of how to get this genesis file.

[catShaark]

ohh thank you guys, this is something I really want to put my hand on.

[czarcas7ic]

Yeah honestly the more eyes on this the better. While it started off being just an annoying side problem, its gotten in the way a lot more than one would expect

[catShaark]

@czarcas7ic, this is still relevant right ? now that the incident is over, should we continue working on this ?

[czarcas7ic]

Yes very relevant, will get back to it tomorrow, thanks for the ping

[ValarDragon]

Closing this for now. We've temporarily mitigated the problem by pruning IBC state.

Further improvements should come from IAVL improvements and better atomicity structure in the SDK's usage of the database