Generalized bloat

[faddat]

I'm of the opinion that economic growth, along with with DEX implementation & IBC is going to lead to a scenario I've named "generalized bloat". It was originally named something else, but we will no longer refer to the problem this way.

https://forum.cosmos.network/t/proposal-draft-gaia-v8-rho-upgrade/6570/4

The Generalized bloat situation is going to at least affect validators and exchanges, but will also likely touch on other matters, as well.

Generalized bloat

• 16GB of RAM needed to process the first block #670
• importing genesis state is very slow #669
• Validators are more expensive to operate
• Sync times are lengthened (though state sync takes care of this)
• Various resource requirements go up

The above issues are from only a moderately bloated Gaia: IIRC, the codebase was clocking in ~300GB at the time.

After the DEX is added, I'm thinking that Gaia is going to get MUCH more bloated: Terabytes

I was thinking out loud about this in the Tendermint engineering slack, and realized that I should publish my thoughts, because if we're not testing what 1TB+ Gaia looks like, we should be.

I think it'd be a shame to strap on a best in class AMM system, and then get stuck because the load it puts on the network isn't manageable, smilar to ETH and Uniswap today.

[tac0turtle]

This is a concern of mine as well. I am hoping with the upcoming work around reworking storage this problem will becomes less present.

There is also ongoing work on the Tendermint side to make storage more efficient.

[faddat]

Any thoughts on how to set up a (preferably automated) test of a bloated network?

One could likely run simapp, and eventually you'd get a bloated gaia, but ideally you'd want a real world network scenario, with all the strange things that tend to happen only on a live network.

[faddat]

@shahankhatch this is making me think about system requirements for Gaia.

So, she runs on a pi:

but I suspect that pi has a bad microsd card. Unfortunately, I have a tough time getting legit microSD cards here (Hanoi). The ones that I have that are known-good, don't suffer from this write problem and I've got another Gaia coming up on another SD card and I think I am going to smash this one to prevent it getting in my workflow again.

anyway

I am wondering how fast and furious we should expect state growth to be when the liquidity module is added to gaia.

[faddat]

#817

[gituser]

It's definitely a big issue.

The blockchain data seems to be growing much faster comparing to the old gaia-v2.x, as of now it's already around 100GB and keeps growing.

And the worst thing is you can't use HDD as gaia seems to be loading old blocks and requires a lot of files being loaded quickly, if you try to move certain files to HDD gaia no longer syncs fast and starts to stall sync for hours.

Also noticed there is an increased CPU usage as well as I/O on these new gaia versions v4.2.x.

[zmanian]

We should expect running Cosmos validators and to a lesser extent full nodes to become substantially more cost and resource intensive over the coming years as the network faces competitive pressure to provide more services.

[faddat]

@zmanian I fully agree with you, and I don't think that is at all unreasonable.

Operating a validator will get more expensive.

I think that we can make some fairly significant improvements to efficiency and my plan is to use this issue to track those over time.

@gituser personally, I really don't think that we need to be concerned about supporting HDDs that spin. Those are truly on their way out.

[faddat]

Trying to:

https://github.com/arl/statsviz/blob/master/_example/default/main.go

But currently blocked by the M1 cpu issue #803 (not exactly blocked, I have other machines but, should work here on my M1, too)

[ChandraStation]

Hello, my name is Chalabi from Chandra Station. I am currently syncing 3 Gaia nodes for our val infra. Currently connected to 10 peers and syncing ~4 blocks per second so im sycning faster then block time but comparatively at an incredibly slow pace. Running on Gigabit Fiber 500 down to these specific machines 32 gb ram 16 core cpu. Mega slow, Ultra Fat. (Tips for speed pls)

[faddat]

@ChandraStation

Howdy!

Welcome to my lil thread about Gaia's bloat.

Some things I've changed about my setup-- none of which will actually help you:

• These days I only do archive nodes. You need them for relaying.
• My best connected Gaia node is: [email protected]:2010
• If you have our fat GAIA on a LAN near the Gaia you're syncing, this improves stuff.
• Snapshots
• State Sync

..... I say that none of these will help, because my latest gaia sync was painfully slow. You may wish to set gaia like:

• 300 inbound
• 60 outbound

I did.

It did not help.

You may wish to put a tenderseed in akash, we could perhaps try that after epoch today.

[tac0turtle]

for archive nodes there isn't much we can do outside of the core modules, as for pruned nodes there is a team researching what can be done. Id recommend opening an issue in the sdk as gaia is using the modules from there, not much gaia can do on its own.

[faddat]

tbh now that the name is more PC I think we should reopen it.