Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
This list contains all stable releases in the cheqd-node v0.1.x family, and skips pre-release versions.
Our goal was to have a persistent network accessible to developers to build apps against. This was our target to get up and running with external partners, and we’re glad to have our launch partners onboard.
The genesis ceremony for the cheqd testnet was carried out on Wednesday, 21 July 2021. Prior to the genesis ceremony, we had asked all our launch partners to setup a node with a Cosmos cheqd node with the beta software code.
During the ceremony, we minted 1 billion CHEQ test tokens as well as the genesis configuration required to connect up all the nodes into the testnet. We’d planned for the ceremony to take 1.5 hours, but as these things go, we ended up extending this to 2 hours as to fix teething issues during peer configuration.
The balance minted with cheq test tokens was distributed as follows:
20% cheq test tokens distributed to the founding validators: We wanted to model the fact that in real life, different validator nodes would have different amounts staked.
80% cheq test tokens was set aside for future validators, treasury, and cold storage: We distributed the majority of the tokens minted as set aside for future use.
The cheqd testnet currently involves four Cosmos validator nodes, one from each founding launch partner. The functionalities available at the moment allow anyone with access to the nodes to:
Create Cosmos accounts that can hold cheq test tokens
Create DIDs that can be written to the cheqd testnet, and pay for it with cheq test tokens
Automation to speed up basic testnet cheqd node deployment
We wanted to get this code early with external partners, so that we can start capturing feedback from potential real node operators in the future.
To make the process of distributing the 20% cheq test tokens among the founding validators randomised, an appointed representative from each of the founding validators rolled three dice using DuckDuckGo. After multiple rounds of die rolls (and an impromptu guitar solo!) we ended up with this distribution:
Outlier Ventures at 9%
cheqd at 5%
Evernym at 4%
DIDx at 2%
As we open up the network to more validators, we’ll distribute more cheq test tokens on a randomised basis. We will be working in the background over the next few weeks to onboard additional validator nodes, initially as a manual process and then expanding it to make it public and open for anyone to join the network. We will continue working on tooling and services to make this process easier.
Our testnet launch in July 2021 gave us valuable feedback from our early adopters in being able to refine where our engineering teams should focus on. While we initially planned to release a Cosmos-based Decentralized Identifiers (DIDs) specification for August, the feedback we received was to focus on making the installation and upgrade processes easier.
We’ve therefore worked towards making installation packages and releases (along with documentation) available in the following formats:
Debian (.deb) package installer, targeting Ubuntu 20.04 LTS (the Linux distribution most Hyperledger Indy node operators should be familiar with)
Docker containers, as a more platform-independent way of deploying the cheqd blockchain node software
Linux binary packages, as a baseline for other Linux distributions
Instructions on how to build any of the above from source.
We’re also exploring support for other platforms in the future, for example, a Snap package for other Linux distributions.
One challenge with having a private beta was the logistical challenge of allowing prospective node operators to freely browse and decide to install the code at their own pace, rather than individually trying to coordinate access to the repositories.
While it’s early days, the cheqd Cosmos Command Line Interface (CLI) user guide we’ve started building defines what can be done on our testnet. We started this by asking the following:
Basic Cosmos node setup and the “cheq” token
Creating, managing, and configuring accounts and keys on a cheqd Cosmos node. These are set up steps that new node operators need to do to get started.
Our public-permissionless model is based on proof-of-stake, so we walk through how to stake and participate in network consensus.
Basic token functionality for holding and transferring our tokens, called “cheq”, to other accounts on the network.
Basic decentralised identity primitives
Writing Decentralized Identifiers (DIDs) entries on a ledger paying for DID writes on the cheqd testnet using testnet tokens. As our CEO, Fraser Edwards, and Javed Khattak (our CFO) described in this blog post why self-sovereign identity needs a token; we want to enable new business models for digital identity.
NYM is the term used by Hyperledger Indy for Decentralized Identifiers (DIDs) that are created on ledger. For the sake of explaining with similar concepts as current Hyperledger Indy implementations, on our testnet transactions to add a DID to the ledger are called NYM transactions. Future releases of our software are likely to replace the NYM terminology with DID for better understanding.
Full changelog: cheqd-node v0.1.19 release notes on GitHub
This list contains all stable releases in the cheqd-node v0.3.x family, and skips pre-release versions.
In November 2021, we released the cheqd mainnet with the support of our wide range of partners.
cheqd was always meant to be designed with decentralised identity at its core. As such, this release is where we implemented our foundational identity functionality.
Added functionality for decentralised identity, specifically Decentralised Identifiers (DIDs), utilising the cheqd DID method
Amended the genesis parameters
Full changelog: cheqd-node v0.3.1 release notes on GitHub
This list contains all stable releases in the cheqd-node v0.2.x family, and skips pre-release versions.
As we come towards our mainnet release, the purpose of this release is to test out the range of functionality we plan on releasing onto the cheqd mainnet.
Bump Cosmos SDK to v0.44.3
Swap the context field to top of the DIDDoc, before ID
Add identity specific tests
Add DIDDoc to Identity ADR
Get rid of symlinks and add ability to change HOME directory by user
Upgrade documentation by @anikitinDSR in #181
Get rid of symlinks and add ability to change HOME directory by @anikitinDSR in #185
Errors documentation by @asmarty in #171
Add identity specific tests by @VladimirWork in #170
Add DIDDoc to Identity ADR by @Toktar in #170
Bump cosmos sdk to v0.44.3 by @askolesov in #186
Add swagger docs by @Toktar in #174
Swap the context field to top of the DIDDoc, before ID by @Toktar in #199
Default configs + configuration subcommand by @askolesov in #187
Fix testnet docker image by @askolesov in #200
Full changelog: cheqd-node v0.2.7 release notes on GitHub.
Canonical list of cheqd Network upgrades
Throughout 2022 we released a breaking upgrade roughly once every quarter. Throughout 2023 we continued with this frequency, however releasing non-breaking point releases consistently. In 2024, we intend to release one major network upgrade per quarter.
To ensure we upgrade all nodes on the network in the most efficient way whilst maintaining due care to the upgrade process, we run the process in the following steps:
Testnet proposal
Testnet upgrade
Mainnet Upgrade Proposal
Mainnet On Chain Upgrade
Take a look at our network upgrades across the years, including Root Cause Analysis of any Network outage.
See below for all of our releases in 2022:
family
family
family
See below for all of our releases in 2021:
family
family
family
2021
View our network ugrades from 2021.
2022
View our network ugrades from 2022.
2023
View our network ugrades from 2023.
2024
View our network ugrades from 2024.
RCAs
View our Root Cause Analysis of Network outages.
This list contains all stable releases in the cheqd-node v0.4.x family, and skips pre-release versions.
Our packaged releases are currently compiled and tested for Ubuntu 20.04 LTS
, which is the recommended operating system for installation using Debian package or binaries.
For other operating systems, we needed to provide an alternative approach. Therefore, this point release was initiated which offers a pre-built Docker image releases for cheqd-node
.
Docker Compose Easy Setup (#250)
Add euox pipefail check for bash scripts (#266)
Test for positive case of upgrade process (#268)
Refactor Debian package installer to handle upgrade scenarios better (#279) (Andrew Nikitin)
Unified network configuration generation (#267)
Further docker, tests improvements (#280)
Updated recommended docker image version
Revert fastsync version to v0
(#256)
Fix small errors around postpurge script
Full changelog: cheqd-node v0.4.1 release notes on GitHub
This software upgrade proposal upgraded the version of cheqd-node software on our mainnet from v0.3.1, to v0.4.x.
Following the successful release of our mainnet in November 2021, we wanted to fast-follow with a some immediate improvements we felt necessary for the cheqd network.
This new node version is intended to enhance functionality currently available on v0.3.1. The upgrade to v0.4.x will be a breaking change that introduces new routes, plus fixes a few technical debt issues identified in the v0.3.x branch.
Bumped Cosmos SDK version from v0.44.3 to v0.44.5
Update DID operations in version v0.3.1 did not carry out a check on the keys used to authenticate the transaction.
In essence, this meant that any update DID operation could incorrectly update a DID or its associated DIDDoc. This has now been fixed.
Switched build system from using Starport to using Makefiles.
This gives greater control for future improvements and optimisation in adding support for other operating systems, database backends etc.
The node software binary has now been compiled with support for Ledger hardware wallet devices for key storage.
REST/gRPC endpoints for querying DIDs have been added. Documentation will be added to explain how these endpoints work.
Full changelog: cheqd-node v0.4.0 release notes on GitHub.
The following steps have been prepared by the cheqd development team for cheqd validators to follow in order to manually upgrade their node.
This patch is non-state-breaking. This means that validators can switch to the new binary, and restart their node without necessarily needing an on-chain software upgrade (which is intended for consensus-breaking changes).
Here’s what’s new in the latest cheqd node version - v0.6.0. This release is a state-breaking update that brings fundamental changes to the cheqd-node testnet and mainnet. The release notes below pertain to the cheqd-node changes only.
This upgrade includes 3 primary enhancements and new features, each of which helping accelerate cheqd to support a greater number of credential formats & SDKs, more use cases for decentralised identity applications and streamline the end-to-end developer experience.
Implementation of a Resources module, used to support DID-Linked Resources
Introduction of an Interactive Installer to streamline the developer onboarding experience
Introduction of Cosmovisor for automated upgrades.
What's changed?
The concept of resources in self-sovereign identity (SSI) ecosystems is not new, however, as we will discuss throughout this blog post, existing approaches to resources in SSI oblige adopters to make compromises between security, availability and interoperability. We first noticed this when we were looking at how we could securely reference credential schemas, something we will expand on throughout this post.
An initial use case for DID-Linked Resources is supporting AnonCreds on cheqd. This is an industry first as AnonCreds have to date been tightly bedded to the Indy ledger. We are using the Resources module to create DID-Linked Resources for schemas and CredDef, bringing AnonCreds support to the cheqd ledger in a W3C compliant format.
What's changed?
Although these were fit for purpose, allowing the vast majority of validators on the cheqd network to get set up these methods do require a higher level of technical skillset, and are more time-consuming.
The Interactive Installer offers a more intuitive and simple developer experience, which guides developers through a step-by-step journey, asking questions with the CLI along the way.
What's changed?
With an increasing number of validators on the cheqd network, we wanted to provide developers managing nodes the opportunity to more easily upgrade their node.
Attempts to create a backup of the user's configuration folder and private validator state files, when carrying out upgrades or downloading snapshots.
Frees up spaces by deleting the data folder when requesting a snapshot restore (only if the backup has been made).
Bumps Cosmovisor if necessary.
Separates out warning, error, info etc level messages based on success status of the action.
To enable debug level logs in case the installer fails, set an environment variable called PYTHONDEVMODE
to the value 1
on your system. This will provide additional diagnostic logging that can help pinpoint errors if the installer fails.
This version also bumps Cosmos SDK to v0.46.10.
The previous version of our node software had to be run in pruning="nothing"
setting due to an upstream issue with the Cosmos SDK iavl
package. This release fixes and restores support for setting pruning settings.
This release is a state-breaking update that brings fundamental changes to the cheqd-node testnet and mainnet. The release notes below pertain to the cheqd-node changes only. Changes to identity applications and tooling is summarised in a separate change log.
The release can be broken down into 6 primary areas of development:
Identity transaction pricing and burn
Enhancements to DIDs and DID-Linked Resource functionality
Changes to protobuf definitions required for enhancements to DID and DID-Linked Resource functionality
CosmosSDK version bumps
Miscellaneous fixes
Introduction of Behavioural Driven Development
In this release we’ve introduced new mechanisms to the cheqd’s tokenomics. Currently (prior to this release), writing a DID to cheqd mainnet only cost the gas fee, which was approximately 0.004 $CHEQ. This is extremely cheap when compared to networks such as Sovrin, where writing a DID costs $10, especially taking into account the value we are bringing as a network, protocol and team. Alongside this, $CHEQ is currently inflationary (where staking rewards come from), with only slashing, tombstoning or no-with-veto slashing providing any deflationary pressure.
Going forward, any changes required to periodically maintain these, especially before we implement pricing oracles to programmatically maintain stable dollar pricing for our partners, will require further votes.
Architectural decisions, diagrams and in-depth explanations of the scope and relationships will be shared shortly in a follow-up blog-post and document.
As mentioned above, the network is currently inflationary only aside from slashing, tombstoning and no-with-veto having deflationary effects. Implementing a burn mechanism will bring the system into equilibrium. The volume of tokens burnt will increase in the same way as distributions will, with the aim of bringing total supply back to one billion.
A number of core changes have been implemented to enable continued interoperability with products and applications across the decentralised identity ecosystem. These include:
Fetching historical versions of DIDDocs easily and without friction is an incredibly important feature for DIDDoc indexing in decentralised identity applications.
We have used the same versioning techniques for DIDDocs as was implemented for DID-Linked Resource versioning. This required an on-ledger change as previously we were using the ‘transaction hash’ of the on-ledger block as the DIDDoc ‘version’, which presents a problem as in the event a user has multiple DID operations in the same transaction, it becomes impossible to distinguish between the various versions of the DIDDocs. This has now been replaced with a UUID.
A number of other networks lack this feature and don’t provide the functionality to fetch historical DIDDocs, which can be a limiting factor for those building decentralised identity applications.
Previously when users created a UUID for a DID-Linked Resource they could use both lower and upper case in the UUID. The cheqd network, however, is only able to handle lower case characters. As such, we introduced a converter that would convert all UUID DIDs to lower case UUIDs.
As a result of this change, existing DIDs and Resources that were created prior to this upgrade, will be altered to lowercase if they previously contained uppercase characters.
The style for the unique identifier of a DID was initially based on the legacy Hyperledger Indy algorithm. We’ve brought it in line with the updated DID Indy specification requirements instead.
By default DID-Linked resources have a DID URL auto generated. For example, a DID URL for a DID-Linked Resource looks like this:
did:cheqd:testnet:z6jKUJA5YcZsNxZgsrQPKPipL2FRTf4s/resources/8140ec3a-d8bb-4f59-9784-a1cbf91a4a35
However, as part of our mission to drive the wider adoption of DID-Linked Resources beyond cheqd, and create the “web of trust”, we have enabled the ability to add an alternative URI for where that same resource can be found.
A checksum is a value that represents the number of bits in a transmission message, used to detect high-level errors within data transmissions. They provide a party on the receiving end information about the date to make sure that the full range of data is fully delivered. The checksum value itself is typically a long string of letters and numbers that act as a sort of fingerprint for a file or set of files to indicate the number of bits included in the transmission.
For example, if a user created a DID-Linked Resource, and 1 month later wanted to retrieve this, it would be possible to calculate the sha256 locally and then compare it with the “checksum” field from the resource in order to check whether this resource was changed or not.
Prior to this release, it was calculated in a less usable way, using “sum256 + data_bytes” where it only needed sum256 (which is what is used from v1.0.0 going forward).
To accommodate for the ledger enhancements, fundamental changes are required to the underlying code base which the ledger uses known as protobuf definitions.
Protobuf definitions can be seen as a template for variables in the ledger code-base; the existing release up to this point uses protobuf definitions v1, whilst this upgrade replaces these with v2 protobuf definitions. For example identity transaction pricing and the ability to provide an alternative URI for a DID did not exist in the previous version, meaning this feature would not be possible to implement.
To inspect the source of protobuf definitions, follow the links below. These are simply contained within the overall v1.0.0 upgrade so don’t require any additional steps for developers.
Previous protobuf definitions:
cheqdid.cheqdnode.cheqd.v1
cheqdid.cheqdnode.resource.v1
Updated:
cheqd.did.v2
cheqd.resource.v2
Another aspect of this change is based on best practices. When we launched the cheqd network we identified certain best-practices and designed an implementation approach which would allow us to roll out these conventions in a phased approach. This enabled us to move quickly, whilst not compromising anything on ledger in the initial stages. With this release the cheqd ledger is in line with overall style guides making it more flexible and robust for both our team and other developers to build on.
As is customary with the use of the underlying CosmosSDK and associated libraries, regular version bumps are required to keep the cheqd ledger in line with the wider ecosystem's changes, for example updates to IBC. In this release we’ve bumped the Cosmos SDK version to 46.6, crucially bringing key security patches as well as a bump to IBC to ibc-go v5.1.
cheqd-noded configure
command removedAs part of the overall ledger changes, the cheqd-noded configure
command has become an outdated and deprecated command. The command has been removed and documentation has been updated accordingly. This changes the manual installation of cheqd-node, and the Interactive Installer which previously relied on this.
The Interactive installer was using the legacy configure cmd ‘cheqd-noded’ to do the configuration for app.toml and config.toml files. As this is no longer used, the interactive installer no longer requires the configure command.
The following changes are now made by default when installing a node:
RPC port by default listens to world i.e from tcp://127.0.0.1:
to tcp://0.0.0.0:
The default minimum gas price is set to 50ncheq
, increased from 25ncheq
An option introduced for a user to enter persistent peers, log level and log format when doing interactive installation
A bug was identified where the Ledger nano hardware wallet was not working with cheqd transactions. This has been resolved in this release.
To date our testing approach has been focused around unit testing and more manual approaches to application testing. In this release we’ve introduced Behaviour Driven Development (BDD) at scale across the entirety of the cheqd node. BDD, is a framework which uses human-readable descriptions of product requirements as the basis for testing. Developers implement a well known vocabulary that can be understood by a regular user, to help identify the issues that arise.
We see this as an integral to building strong foundations that will enable developers to more easily build with cheqd, and streamline the overall development process for our core team as we progress further with ledger development centred on the implementation of cheqd payment rails.
See below for all of our releases in 2023:
family
This list contains all stable releases in the v0.6.x family, and skips pre-release versions.
Read Full changelog on GitHub:
This is in response to related to IBC protocol / ICS23.
This is . v0.6.9 fixes this vulnerability by patching to upstream.
The patch has already been applied to and one of the other large validators (at the time of writing this).
Read Full changelog on GitHub:
Read Full changelog on GitHub:
Read Full changelog on GitHub:
Our objective in building resources on cheqd is to improve the way resources are stored, referenced and retrieved for our partners and the broader SSI community, in line with the existing .
Please take a which provides a thorough explanation for your reasoning on this approach, and links to our documentation.
Prior to this upgrade, we offered developers looking to setup using a Ubuntu .deb
package file and installing a
Find out more about how to get started using the .
is a small process manager for Cosmos SDK application binaries that monitors the governance module for incoming chain upgrade proposals. If it sees a proposal that gets approved, cosmovisor can automatically download the new binary, stop the current binary, switch from the old binary to the new one, and finally restart the node with the new binary.
Read on GitHub:
DEV-1319 Add helper to convert ed25519 public keys to JWK by in
feat!: Add resource module (DEV-1281) by in
feat!: Interactive installer for cheqd-node with Cosmovisor support DEV-1321 by in
feat(ibc): Bump IBC to v3 and update associated upgrade tests DEV-807 by in
DEV-1044 Fix gas estimation for cheqd transactions by in
Proposal CLI Registration by in
chore: Bump Cosmos SDK to v0.45.4 by in
fix: Consensus issue with Resource.created by in
fix: Small fixes for installer after testing by in
fix: Fix helper message for CLI all-resource-versions by in
Dispatch workflow by in
DEV-520 - Chore: Added Github CodeQL Action by in
DEV-1096 Improve release processes by in
build(tooling): Fixing workflows to main branch version by in
Fix workflow calls to refer local versions instead of master by in
Add gofumpt check to the lint workflow by in
Add a workflow to lint Semantic PRs by in
fix: Get linter back by in
fix: Get rid of Debian package for release by in
test: Add Cosmovisor support for tests DEV-1278 by in
test: Add resources to upgrade integration tests by in
DEV-1171 Add additional instruction for updating from 0.4.x version by in
Bump cheqd version to 0.5.0 in docs by in
docs: Remove all documentation from cheqd-node to node-docs repo DEV-1320 by in
This list contains all stable releases in the v1.x.x family, and skips pre-release versions.
This version primarily fixes the , an off-ledger helper software for carrying out cheqd-node installations and upgrades. The installer now:
Set as well as .
Read Full changelog on GitHub:
This version creates a standalone Golang package called github.com/cheqd/cheqd-node/api
which is generated using the Pulsar generation file. This allows non-ledger applications such as the to consume a standalone package with just the API interaction elements, without needing to import the entire cheqd-node
Golang package.
Read Full changelog on GitHub:
Read Full changelog on GitHub:
Identity transactions such as and (e.g. visual representations of Verifiable Credentials; trust & revocation registries; schemas; documents; logos) writes will have a cost greater than gas which will increase rewards from the network as identity utility increases. At current price levels ($0.03-$0.04), writing a DID will equate to 66.6–50 $CHEQ, which leads us to where these tokens will be distributed upon writes. See detailed .
As can be seen in the showing the genesis parameters which define the costs in $CHEQ, there is also a “burn_factor” variable which sets the percentage of the tokens charged for identity transactions which will be directly burnt. This can be adjusted through on-ledger governance votes according to the cheqd community preferences.
Previously it was not possible to mark DIDs as deactivated on the cheqd network. This is now possible. This enables cheqd to support each of the required transactions for integrating with the .
UUIDs are . This means that UUIDs with both upper and lower case characters are semantically identical
As part of our interoperability efforts and support for AnonCreds with AFJ ( we needed to expand the supported identifiers on-ledger. The main barrier preventing full support for AnonCreds on cheqd using Aries SDKs is that AnonCreds is still dependent on the Indy SDK. When you call any function in the Indy SDK, you need to use the legacy Indy-style identifier format to carry out a transaction. Supporting Indy-style identifiers is therefore a stepping-stone in the wider AFJ support for cheqd.
When resolved ) you’ll find that this shows the IIW logo, used in our .
This is in the DID Core specification.
feat(installer): Get installer to handle multi-platform binaries DEV-1667 by in
feat: DeactivateDid transaction to mark DIDs as deactivated DEV-1774 by in
feat(proto): Proto API v2 DEV-1786 by in
feat(proto): Refactor app to support proto v2 by in
feat: DidDoc versioning DEV-1892 by in
fix(installer): Fixed binary permissions DEV-1826 by in
fix: Add resourceType for versioning functionality DEV-1703 by in
fix: Checksum calculation DEV-1649 by in
test: Fix installer by in
fix: Ability to write spaces in the moniker by in
fix(release): Bump semantic release package by in
fix(installer): Fixed binary permissions DEV-1826 by in
fix: Error handling by in
fix: Use proto marshaller instead of amino to derive sign bytes DEV-1694 by in
fix: Convert UUID DIDs to lower case UUID DEV-1705 by in
fix: Support Indy-style identifiers DEV-1750 by in
fix: Add support for Ledger nano by in
build(buf): Change buf.build package name by in
build: Buf.build config for generating proto DEV-1545 by in
build(docker): Move to Alpine images for Dockerfile by in
build: Backport workflow and other fixes to release/0.6.x branch DEV-1822 by in
build: Release with v-tags by in
security(tendermint): Bump github.com/tendermint/tendermint from 0.34.19 to 0.34.20 by in
test: Move unit testing to Gingko DEV-1652 by in
test: Integration tests merging state by in
test: Add interactive installer to workflow and get rid of deb by in .
See below for all of our cheqd-node
ledger software releases in 2024:
todo
This page offers a post mortem, root cause analysis, and lessons learnt from the v1.x cheqd network upgrade.
Date: 30th January 2023 Status: Resolved
In the lead up to cheqd v1.x mainnet upgrade, and during the eventual release and upgrade itself, we faced a number of challenges which resulted in a brief network halt of mainnet and the need for a fast-follow patched release.
As of the 8th February 2023, this patched release has now been applied across both testnet and mainnet validators, and the network is working as expected, with the issue identified during the upgrade now resolved.
This RCA provides a brief overview of the root cause of the issues identified, the fix itself released in v1.2.5, and the lessons learnt by the cheqd Product & Engineering team.
By the end of the day, we were able to identify three major three major types of issues we encountered:
“pruning = nothing”
The pruning issue was largely related to the DragonBerry patch, which was a fix of a high-risk security vulnerability (dubbed “Dragonberry”) related to IBC protocol / ICS23.
Within the patch a strict check was introduced on horizontal states in trees, and as a result, uneven heights, pruning + state-sync no longer were behaving as expected. After introducing the Dragonberry patch it’s been common that the store height does not equal the historic height. The store height not equalling the historic height is what caused our pruning issue. This is largely due to an known issue in the iavl state tree package + Cosmos SDK which occurs on store writes during upgrades.
On some validators we found there was a v0.6.x binary in the /usr/bin
folder, whereas under Cosmovisor that gets turned to a symlink from /usr/bin/cheqd-noded
to the actual binary in /home/cheqd/.cheqdnode/cosmovisor
.
Some nodes had an issue executing the migration, where they seemed to run into an issue with the new group module that Cosmos SDK introduced in v0.46.x, although this seems to be an edge case.
During the testnet upgrade we experienced a consensus error due to an issue with module versions among other issues (fortunately this is what testnet is for). As a result we started a fresh test network from block 0 + state export (testnet-6). With this fresh network, pruning was set to default ( pruning="default").
This is the default setting for every node. However, this default option also means that approximately 3.5 weeks of the state is kept which is not enough to catch the pruning issue early. Switching to more aggressive setting would’ve caught issues earlier, for example: pruning="custom"
, with:
Immediate resolution to get the network running was offered to us by our validator community, as it was an issue ~10 Cosmos chains had faced when getting to the latest version.
In order to get the network back up and running, the validators were advised to simply switch their pruning parameters, setting them to pruning = “nothing”. Once consensus was reached with validators who had switched to this, the network was restored.
Setting pruning to nothing was not an ideal solution, given it meant the network was not being pruned, resulting in an ever increasing chain size and increased storage requirements / higher costs for validators.
Going forward we should endeavour to create as near an identical environment for testing upgrades on testnet as possible. While it may not always be possible, due to modules like IBC not being available on testnet, we should make sure that the delta between mainnet and testnet is detailed comprehensively in advance of upgrades to make sure any potential issue can be quickly diagnosed.
Where there is any risk vector of a consensus fault going into an upgrade, full snapshots should be taken of the network before any upgrades are attempted.
The issue we encountered during this upgrade could have been discovered in advance by aligning more proactively with other Cosmos chains. With a Validator community that spans far more than the cheqd Network, we should try and identify any potential issues experienced on other chains before bumping Cosmos SDK or IBC versions. Likewise, we should be circulating any issues we uncover. We’ll be more actively using this space within the product site to report on these going forward.
While we’re thrilled that the Cosmovisor automatic installation process largely worked as intended for this upgrade, however we still need to be mindful of how technical minutiae like symlinks are handled differently with manual upgrades. Having a clear overview of the delta between manual and Cosmovisor upgrade patterns will help isolate any potential issues in future upgrades.
Reaching consensus initially at upgrade height took significantly longer than expected, due to the node managed by the cheqd team not being upgraded (cheqd node). This was because the cheqd node was taking a snapshot, at the halt height, to ensure there was a backup immediately before the upgrade to roll back to in case of a major upgrade failure. Going forward, rather than using this node for snapshots, a secondary node will be used to take snapshots to avoid delays.
Generally we have intended to upgrade on Tuesdays or Wednesdays, however this time due to urgency of this upgrade and team vacation we opted for a Monday. This however doesn’t allow much time to get everyone prepared, which we’ll avoid going forward.
When submitting the mainnet upgrade proposal, like all other Cosmos-based network, we log the intended upgrade block height. This is calculated by taking the current height, and adding on the number of blocks required to reach the agreed upgrade time and date. The number of blocks to add depends on the average time per block. Unfortunately we were out with our estimated time, with the upgrade height being reached two hours earlier then planned. As a result, certain members of the team were unavailable at the upgrade time. In the future, we'll improve the accuracy of our estimations, ensure team availability is confirmed for a wider time window, and add sufficient buffer time to allow for block time being faster or slower for whatever reason.
With network upgrades on this scale, a number of changes are required in our internal and external services that depend on the core ledger. Following the upgrade we begun identifying which services were knocked out due to dependency changes, but this wasn’t done in a systematic, coordinated manner. Going forward, we’ll create a check-list of all internal and external services that need to be checked and updated following upgrade, to reduce downtime.
Although generally running two forums (Discord & Slack) has worked, during this upgrade there was a clear misalignment which could have been resolved if everyone was conversing in one place. Slack has remained as the dominant location for communications with SSI vendor validators, and Discord for the Cosmos-based validators. Going forward we will review the use of two communities, and decide on how to coordinate better, likely reducing to one group.
. . .
Thank you to our validators for your patience and support throughout. Fortunately this didn’t cause any significant downtime, however it could have been avoided through more stringent checks on testnet, and more communication with other Cosmos SDK chains.
On Monday 30th January, the cheqd team initiated a major network upgrade, v1.x, which introduced identity transaction pricing, among other features and fixes to the network ().
Following the passing of the (proposal #12), At roughly 09:30 GMT, the network halted at block height 6,427,279, with the upgrade set to begin at height 6,427,280.
Over the subsequent 40 minutes, to 10:10 GMT, validators successfully upgraded to the new version, with consensus reached at 10:19 GMT. Shortly thereafter, following just ~10 blocks signed (), a number of validators reported errors and timeouts with their nodes, examples:
Separately, we also experienced dependency breaks on a number of the internal and external servies which use the network APIs following changes to API paths within this upgrade.
After investigating a route to solve this, we found a patch authored by , released in late November 2023, which had already been used as a solution for other Cosmos SDK based networks facing the same issue.
We also failed to effectively utility to provide updates. This generally needs to be incorporated more into the Product & Engineering teams ways of working, and in particular at critical points like upgrades.
You can also read more about about plans for the year ahead in our
This is a list of root cause analysis (RCA) of major network incidents/outages. These are intended to summarise any investigations in more detail.
ℹ️ To check current network status and incidents, please check status.cheqd.net instead. Our validator status monitoring tool also offers network-wide capability of monitoring issues with missing blocks.
January 2023: v1.x Upgrade
This list contains all stable releases in the cheqd-node v0.5.x family, and skips pre-release versions.
This new node version is intended to enhance functionality currently available on v0.4.x. The upgrade to v0.5.x will be a breaking change that introduces new routes, fixes a few technical debt issues identified and overall offers significant enhancements to the identity functionality and security of the network.
Full changelog: cheqd-node v0.5.0 release notes
What has changed?
Our did:cheqd Decentralized Identifier (DID) method provides a unique identification number for a DID Subject. This can be thought of in a very similar way to traditional bank cards. (Note: While the example of a bank card is used here, DIDs on ledger are typically written for companies, not individuals.)
Within a bank card, different sections of the card identify different actors, such as the Network, Issuer ID and a Unique Account Number - and when put together, you have a complete card number. DIDs are similar, in the sense that different components of the DID mean different things:
In this update, cheqd has updated the way that DIDs are created and resolved to ensure that the Unique Identifier is a Base58 encoded string (either 16 or 32 characters long). This was done to keep it consistent with the Hyperledger Indy DID method identifier specification.
Why it is important?
Prior to the upgrade, the ledger code itself was relatively “dumb”: the responsibility of ensuring that a valid unique identifier was created, matching our DID specification, would be done client-side. While the cheqd ledger did implement a uniqueness check to ensure that no other DIDs with the same identifier existed within a namespace, it didn’t implement checks to ensure that the identifier specification was met. (This is broadly how a lot of Hyperledger Indy SDKs currently do this.)
We believe that that there will be a blend of various levels of technical complexity in client-side SDKs that interact with DIDs, so relying on only client SDKs to implement this is an unsafe method. By adding checks on the ledger itself, it allows us to ensure consistency across a wide range of client SDKs.
We are additionally exploring future iterations which would allow unique identifiers to be defined as UUIDs instead of the Indy DID method technique as well.
What has changed?
In Hyperledger Indy, the assumption was that DID would have only one Verification Method and one Controller per DID. The DID Core specification has evolved since then to allow for multiple controllers and more complex Verification Relationships.
Learn more about DID’s (Decentralised Identifiers) and why they’re important on learn.cheqd.io.
The cheqd DID method allows multiple controllers, verification methods, and complex relationships to be defined which brings it in-line with the DID Core specification. But this also introduces new challenges. To illustrate through an example, let’s say there is:
A DID Document with three Verification Methods listed within the Verification Methods section of the DID Document (“DIDDoc”)
Next, there is a request to add a new Verification Method to the DID Document.
The question is: how should this DID Document be updated; who needs to sign to make this change?
Why it is important?
Prior to the upgrade, when a DID update request was made, the ledger would only ask for the existing Verification Method(s) listed in the DID Document to sign to approve the change. We have changed this to ensure that both the existing and new Verification Method keys need to sign the update request.
This is important because it will prove that the controller(s) of any new Verification Method(s) have control of the relevant key(s) that the DID update is trying to achieve. This is much more secure because it achieves a level of trust that all of the Verification Method(s) listed in the DID Document can be appropriately signed for by their Controller(s). It also, importantly, follows the DID Core Specification.
What has changed?
When a DID is resolved, it fetches resolution metadata in addition to the DID Document. This metadata contains entries such as when a DID was created as well as when it was most recently updated.
Prior to the upgrade, the Update metadata field would contain the same data as the Created field if the DID had never been updated. We’ve changed this to a null value instead of the same date/time, to make it easier for software to understand this scenario.
Why it is important?
If you see the Update field reads null
, it means the DIDDoc has never been updated. This clears up any potential confusion and aligns the DID resolution metadata.
What has changed?
We have changed the way we validate Verification Method keys, splitting this into:
Static validation;
Dynamic validation.
The former, static validation checks, are performed before the inclusion of a transaction in the mempool; and the latter, dynamic validation checks, are performed during the transaction execution once it is included in the blockchain.
Why it is important?
Splitting these is important because now errors in Verification Method keys can be caught earlier and flagged, before any transaction enters the mempool.
Note: The mempool serves the purpose of keeping track of transactions seen by all full-nodes. Full-nodes keep a mempool cache of the last ‘mempool.cache_size’ transactions they have seen, as a first line of defence to prevent replay attacks.
Therefore, the validation improvements will allow nodes to reject invalid transactions in the early stages without including them in the chain and charging the user.
This in turn speeds up the cheqd network as incorrect transactions are not being committed and incorrect Verification Method keys are not errored out when committing a transaction to the mempool.
We’re also using other work to build similar efficiency improvements:
This validator package for null checks, generic DID validity checks in static validation
This validator package for DID namespace checks in dynamic validation.
What has changed?
Within the DID Core specification there are two types of key in a Verification Method:
JSON Web Key (JWK); and
Public Key Multibase.
Previously, the only format of key supported was publicKeyMultibase
Having only one of these two options limits the scalability and interoperability of the cheqd network, as the supported Verification Method keys are limited.
Therefore, we implemented JSON Web Key (JWK) support.
Why it is important?
With JWK, cheqd now supports a broader selection of Public Key encoding schemes, which is an important improvement, given our focus on interoperability and eventual cross-ledger support.
What has changed?
Previously, we were using a Cosmos format of datetime, rather than what is defined in the DID core spec. Now this has been changed to align with DID Core.
Why it is important?
This aligns our DID Document metadata with DID Core, making it more semantically interoperable and spec compliant.
tx_hash
in the correct formatWhat has changed?
As cheqd is a chain built on the Cosmos SDK, it relies on Tendermint as its consensus mechanism.
This means that when something is written to the network, there is a Tendermint transaction hash that is created. This is specific to Cosmos chains.
Within DID Document metadata, there is a field called VersionID which, as the name suggests, denotes the specific version of the DID Document.
Previously, in this field, we were generating a hash from the transaction itself to calculate the DIDDoc version ID. We have now changed this to populate a Tendermint transaction hash.
Why it is important?
This is important because the DIDDoc versionId
can now be retrieved right after the creation of the DID, with the DID now more easily searchable on a Block Explorer. Previously, we needed to create the DID, then ask it to create version ID, then update.
This update streamlines the process and makes it more efficient.