[Shipped] Reduce Blockchain size by 50%

Wow, this looks fantastic. Thanks for your hard work, and genius brains to make it work.

Hi @hungngo! Any updates for the past week ?)

Hello @andrey i make a typing mistake. Previous week update is already above . Please check it out, tks

Once this is complete, I vote that @hungngo deserves a unique incognito badge. “db Crusher” comes to mind.

4 Likes

I love that badge josh lol

2 Likes

Update 18 March 2020:
Last two day we encounter some minor issue, these issue only effect node in run time under some rare circumstance. Issues are:

  • When we insert new block, we have to make sure this procedure is atomic. So if node fail to insert new block, it must activate revert current state to a stable snapshot state (which is its previous state).
  • Our system has multiple shard chain and beacon chain, which is very complicated. So we check and make sure one-shard chain will get the right beacon data at the right time.

We managed this problem and solved it already. Now we continue testing phase and sync mainnet fullnode. At the end of this week, we may acheive:

  • Pass all QC testcase at devnet and testnet
  • Sync mainnet fullnode with new database version
  • Update new database version to shard5 testnet

Notice
At current time, choice for update new database version or not DOESN’T effect our network

4 Likes

database

@hungngo @Josh_Hamon @ning Database Crusher badge?

5 Likes

so cool @Mike_Wagner, i love it

Awesome! I like the idea of the db icon either looking short in the icon or even crushed like an aluminum can.

By the power vested in me by the internet - I hereby dub @hungngo the DATABASE CRUSHER.

Screen Shot 2020-03-19 at 10.48.13 PM

10 Likes

thanks guy, we will bring more good news soon

1 Like

Hi @hungngo,

I tried to get your code and run to sync data from the mainnet. I saw that at beacon block 90496, I couldn’t sync it because of this error log:

“2020-03-17 22:46:09.652 beaconprocess.go:302 [CRT] BlockChain log: FORK BEACON DETECTED, New Beacon Block Height 90496, Hash 44d925ddeed6b8a39f186c9d93356d4900eba3a988229f5d9a0beb1000c91715, Expected Previous Hash 378f3a45f83d14ec6e929065c6cf08e45dd583106223fce57b13c1e3febfa230, BUT Current Best State Height 90495 and Hash 378f3a45f83d14ec6e929065c6cf08e45dd583106223fce57b13c1e3febfa230
2020-03-17 22:46:09.652 synker.go:976 [ERR] BlockChain log: -1048: Fetch Beacon Block Error
-1001: block 378f3a45f83d14ec6e929065c6cf08e45dd583106223fce57b13c1e3febfa230 not exist
Get Beacon Block By Hash Error
github.com/incognitochain/incognito-chain/dataaccessobject/rawdbv2.NewRawdbError
/Users/autononous/go/src/github.com/incognitochain/incognito-chain/dataaccessobject/rawdbv2/error.go:157
github.com/incognitochain/incognito-chain/dataaccessobject/rawdbv2.GetBeaconBlockByHash
/Users/autononous/go/src/github.com/incognitochain/incognito-chain/dataaccessobject/rawdbv2/accessor_beacon.go:66
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).verifyPreProcessingBeaconBlock
/Users/autononous/go/src/github.com/incognitochain/incognito-chain/blockchain/beaconprocess.go:300
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).InsertBeaconBlock
/Users/autononous/go/src/github.com/incognitochain/incognito-chain/blockchain/beaconprocess.go:102”

I needed to restart node, and it synced continuously again. I think this is because beacon block 90495 could not be inserted into DB, but we update the best state of beacon chain. So we need to review the reversion of this process. When I review this logging text, I think it doesn’t make sense in this scenario.

Please have a look.

1 Like

@thaibao i solved this problem. This problem happened because node didn’t revert current state to previous state when it failed to insert new block

REPORT 14 Mar - 20 Mar 2020
Current Step

Finish:

  • Fixed bug and passed all QC testcase in dev/net, local network
  • Finish code review by @thaibao (let’s give him a big clap)

In progress:

  • We are moving to testnet, we deployed source code with database version 2 to shard 5 in testnet
  • Add time log and monitoring node operation with database version 2

To Do Next:

  • If things are smooth, we will move on mass testnet committes deployment and mainnet fullnode deployment
  • Merge code and support other team with our new database infrastructure

I have all the confident that our blockchain size will be 10 time smaller soon

2 Likes

Wonderful news over the weekend!

  • We have deployed a mainnet fullnode with database v2. The blockchain size in this fullnode is only 12GB compared to the current 150GB.
    Log rotation in two days consumed storage more than 5-months blockchain size. lol

Screen Shot 2020-03-22 at 21.54.43

6 Likes

Yeah, so great. We’re trying to move some old mainnet node to this DB version and run a while before moving all nodes to this great version.
This week
All testnet node will be moved for this version
Few mainnet nodes will be moved later

4 Likes

@aaron @ning let’s create a long topic explaining this. this is a big improvement that is super helpful to our validators! from 150gb to 12gb, that is indeed a wonderful news @hungngo :slight_smile:

cc @thaibao @0xkumi @binh

2 Likes

REPORT 21 Mar - 29 Mar 2020
Current Step

Finish:

  • Pass all QC testcase on both dev/net and test/net, no bug was found this week.
  • Testnet: All node in testnet now using the new database version
  • Mainnet: One node in each shard and beacon run with database version 2

In progress:

  • Testnet: monitoring node behavior
  • Mainnet: monitoring node behavior and continuously update node by node to database version 2

This weeks we received NO BAD NEWS. As scheduled, that we will deliver this feature to all node in mainnet within next 2-3 weeks.

2 Likes

Unfortunately, we having a trouble with instruction hash of beacon validator that use db v2. Maybe issue is in reproducing hash from instructions of block reward that mean beacon producer create a block with instruction hash which is different from beacon validator’s hash who using db v2 . @hungngo and @hyng are going to try hard to fix this issue earlly and we can retry on beacon validator again.

Add log error for this case

3 Likes

All great things take more trial and error than most people will ever know.
I am sure you will fix it.

1 Like