Hi everyone, here’s a follow up regarding today’s network issues.
As some of you will have noticed, the mainnet is now back up and running. What happened on the backend is fairly straightforward – a few servers where many of the core team host nodes went down last night. The outage is linked to a single data center that owns not just these servers, but also many others.
As you probably know, Incognito implements a variant of pBFT at the consensus layer. For every shard block to be created, the consensus layer requires more than ⅔ votes (signatures) from the current shard committee. As a result of the outage, there weren’t enough signatures necessary to produce a new shard block.
We immediately re-deployed our nodes across additional data centers, so that if any provider goes down in the future, there will be a much smaller risk of block creation being halted. This will do for the short term, but we have already begun work on longer-term solutions designed to keep the network stable and adaptable through multiple eventualities.
One such solution is Dynamic Committee Size. Once this is implemented (see Roadmap, Initiative 5, Objective 5.1), similar incidents will not halt block production. New blocks will continue to be produced even if some committee members go offline.
Additionally, we’re considering bringing slashing back. Slashing was part of the testnet consensus last year, but was later removed from the mainnet deployment as the initial design was deemed too stringent. We are currently revising our slashing policy so that it is friendly to validators, while remaining sufficiently rigorous to deter nodes from going offline.
Our aim, as it has always been, is to be as transparent as possible regarding every aspect of the project. So if you have any questions at all, please feel free to reach out.