[Shipped] Reduce Blockchain size by 50%

[hungngo]

REPORT 22 Feb 2020 - 28 Feb 2020
Current Step:

@hungngo

5. Integration with incognito consensus version 1 (4 weeks)

Finish:

• Merge code from batching tx feature
• Local testing

In progress:

• Testing on dev/net, estimate end date 2 March 2020

To do next:

• Waiting to merge Incognito Mode for Smart Contract feature
• Add more unit test

[annie]

Great result, Hung!

As this project was funded last year and is now a work in progress, i’ve moved it to the right category.

Funds have been sent from November to February, and will continue to be disbursed every subsequent week dependent on progress.

[hungngo]

REPORT 2 Mar - 7 Mar 2020
Current Step:

@hungngo

5 Integration with consensus v1 30|3 Feb 2020|in progress

Finish:

• Merge code from Incognito Mode for Smart Contract feature
• Merge code from highway-v2
• Deploy into testnet and measure blockchain size

In Progress:

• This week, we deploy code into testnet and measure blockchain. Unfortunately, something wrong happened, blockchain size seem to grow faster. In details, when new block is added, block with greater height consumes more space than block with lower height. So we are analyzing and review our implementation to find out the cause
• Analysis and review estimate end date 13 March 2020

To do next:

• After analysis result, we will decide what should we do next

Note: In my opinion, the new technique is fine, I think we misuse something and make some mistake in implementation phase.

[Jamie]

Good luck solving this. Hopefully it is an obvious error that can be easily found and fixed.

[hungngo]

Good news is we detected the problem, and moving into debug phase

[hungngo]

REPORT 8 Mar - 13 Mar 2020
Current Step

@hungngo

|6.1|Review And Testing with consensus v1|60|10 Feb 2020|in progress

Finish:

• Fix data consumption problem last week
• Upgrade key - value schema
• Sync Testnet and Mainnet

** In Progress:**

• Sync Mainnet and continue to review and evaluation
• Prepare Mainnet deployment strategy

Result:
This week team got great result in reducing database:
a. Testnet:

• Fullnode: From 70GB to 15 GB
• Shard node (one shard): From 63GB to 4.7GB
  b. Mainnet:
• Beacon node: From 140GB to 2GB
• Shard node: in progress
• Fullnode: in progress

In general, we have reduce over 10 times blockchain size.

[Jamie]

Wow, this looks fantastic. Thanks for your hard work, and genius brains to make it work.

[andrey]

Hi @hungngo! Any updates for the past week ?)

[hungngo]

Hello @andrey i make a typing mistake. Previous week update is already above . Please check it out, tks

[Josh_Hamon]

Once this is complete, I vote that @hungngo deserves a unique incognito badge. “db Crusher” comes to mind.

[hungngo]

I love that badge josh lol

[hungngo]

Update 18 March 2020:
Last two day we encounter some minor issue, these issue only effect node in run time under some rare circumstance. Issues are:

• When we insert new block, we have to make sure this procedure is atomic. So if node fail to insert new block, it must activate revert current state to a stable snapshot state (which is its previous state).
• Our system has multiple shard chain and beacon chain, which is very complicated. So we check and make sure one-shard chain will get the right beacon data at the right time.

We managed this problem and solved it already. Now we continue testing phase and sync mainnet fullnode. At the end of this week, we may acheive:

• Pass all QC testcase at devnet and testnet
• Sync mainnet fullnode with new database version
• Update new database version to shard5 testnet

Notice
At current time, choice for update new database version or not DOESN’T effect our network

[Mike_Wagner]

@hungngo @Josh_Hamon @ning Database Crusher badge?

[hungngo]

so cool @Mike_Wagner, i love it

[Josh_Hamon]

Awesome! I like the idea of the db icon either looking short in the icon or even crushed like an aluminum can.

[ning]

By the power vested in me by the internet - I hereby dub @hungngo the DATABASE CRUSHER.

[hungngo]

thanks guy, we will bring more good news soon

[thaibao]

Hi @hungngo,

I tried to get your code and run to sync data from the mainnet. I saw that at beacon block 90496, I couldn’t sync it because of this error log:

“2020-03-17 22:46:09.652 beaconprocess.go:302 [CRT] BlockChain log: FORK BEACON DETECTED, New Beacon Block Height 90496, Hash 44d925ddeed6b8a39f186c9d93356d4900eba3a988229f5d9a0beb1000c91715, Expected Previous Hash 378f3a45f83d14ec6e929065c6cf08e45dd583106223fce57b13c1e3febfa230, BUT Current Best State Height 90495 and Hash 378f3a45f83d14ec6e929065c6cf08e45dd583106223fce57b13c1e3febfa230
2020-03-17 22:46:09.652 synker.go:976 [ERR] BlockChain log: -1048: Fetch Beacon Block Error
-1001: block 378f3a45f83d14ec6e929065c6cf08e45dd583106223fce57b13c1e3febfa230 not exist
Get Beacon Block By Hash Error
github.com/incognitochain/incognito-chain/dataaccessobject/rawdbv2.NewRawdbError
/Users/autononous/go/src/github.com/incognitochain/incognito-chain/dataaccessobject/rawdbv2/error.go:157
github.com/incognitochain/incognito-chain/dataaccessobject/rawdbv2.GetBeaconBlockByHash
/Users/autononous/go/src/github.com/incognitochain/incognito-chain/dataaccessobject/rawdbv2/accessor_beacon.go:66
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).verifyPreProcessingBeaconBlock
/Users/autononous/go/src/github.com/incognitochain/incognito-chain/blockchain/beaconprocess.go:300
github.com/incognitochain/incognito-chain/blockchain.(*BlockChain).InsertBeaconBlock
/Users/autononous/go/src/github.com/incognitochain/incognito-chain/blockchain/beaconprocess.go:102”

I needed to restart node, and it synced continuously again. I think this is because beacon block 90495 could not be inserted into DB, but we update the best state of beacon chain. So we need to review the reversion of this process. When I review this logging text, I think it doesn’t make sense in this scenario.

Please have a look.

[hungngo]

@thaibao i solved this problem. This problem happened because node didn’t revert current state to previous state when it failed to insert new block

[hungngo]

REPORT 14 Mar - 20 Mar 2020
Current Step

@hungngo

|6.1|Review And Testing with consensus v1|60|10 Feb 2020|in progress|
|6.2|Deploy new database with consensus v1|3||

Finish:

• Fixed bug and passed all QC testcase in dev/net, local network
• Finish code review by @thaibao (let’s give him a big clap)

In progress:

• We are moving to testnet, we deployed source code with database version 2 to shard 5 in testnet
• Add time log and monitoring node operation with database version 2

To Do Next:

• If things are smooth, we will move on mass testnet committes deployment and mainnet fullnode deployment
• Merge code and support other team with our new database infrastructure

I have all the confident that our blockchain size will be 10 time smaller soon