This list contains all stable releases in the cheqd-node v0.1.x family, and skips pre-release versions.

v0.1.19: Testnet launch [August 2021]

Context

Our goal was to have a persistent network accessible to developers to build apps against. This was our target to get up and running with external partners, and we’re glad to have our launch partners onboard.

The genesis ceremony for the cheqd testnet was carried out on Wednesday, 21 July 2021. Prior to the genesis ceremony, we had asked all our launch partners to setup a node with a Cosmos cheqd node with the beta software code.

During the ceremony, we minted 1 billion CHEQ test tokens as well as the genesis configuration required to connect up all the nodes into the testnet. We’d planned for the ceremony to take 1.5 hours, but as these things go, we ended up extending this to 2 hours as to fix teething issues during peer configuration.

The balance minted with cheq test tokens was distributed as follows:

1. 20% cheq test tokens distributed to the founding validators: We wanted to model the fact that in real life, different validator nodes would have different amounts staked.

2. 80% cheq test tokens was set aside for future validators, treasury, and cold storage: We distributed the majority of the tokens minted as set aside for future use.

The cheqd testnet currently involves four Cosmos validator nodes, one from each founding launch partner. The functionalities available at the moment allow anyone with access to the nodes to:

1. Create Cosmos accounts that can hold cheq test tokens

2. Create DIDs that can be written to the cheqd testnet, and pay for it with cheq test tokens

3. Automation to speed up basic testnet cheqd node deployment

We wanted to get this code early with external partners, so that we can start capturing feedback from potential real node operators in the future.

To make the process of distributing the 20% cheq test tokens among the founding validators randomised, an appointed representative from each of the founding validators rolled three dice using DuckDuckGo. After multiple rounds of die rolls (and an impromptu guitar solo!) we ended up with this distribution:

1. Outlier Ventures at 9%

2. cheqd at 5%

3. Evernym at 4%

4. DIDx at 2%

As we open up the network to more validators, we’ll distribute more cheq test tokens on a randomised basis. We will be working in the background over the next few weeks to onboard additional validator nodes, initially as a manual process and then expanding it to make it public and open for anyone to join the network. We will continue working on tooling and services to make this process easier.

Major features

Our testnet launch in July 2021 gave us valuable feedback from our early adopters in being able to refine where our engineering teams should focus on. While we initially planned to release a Cosmos-based Decentralized Identifiers (DIDs) specification for August, the feedback we received was to focus on making the installation and upgrade processes easier.

We’ve therefore worked towards making installation packages and releases (along with documentation) available in the following formats:

1. Debian (.deb) package installer, targeting Ubuntu 20.04 LTS (the Linux distribution most Hyperledger Indy node operators should be familiar with)

2. Docker containers, as a more platform-independent way of deploying the cheqd blockchain node software

3. Linux binary packages, as a baseline for other Linux distributions

4. Instructions on how to build any of the above from source.

We’re also exploring support for other platforms in the future, for example, a Snap package for other Linux distributions.

One challenge with having a private beta was the logistical challenge of allowing prospective node operators to freely browse and decide to install the code at their own pace, rather than individually trying to coordinate access to the repositories.

Changelog

While it’s early days, the cheqd Cosmos Command Line Interface (CLI) user guide we’ve started building defines what can be done on our testnet. We started this by asking the following:

• Basic Cosmos node setup and the “cheq” token

  • Creating, managing, and configuring accounts and keys on a cheqd Cosmos node. These are set up steps that new node operators need to do to get started.

  • Our public-permissionless model is based on proof-of-stake, so we walk through how to stake and participate in network consensus.

  • Basic token functionality for holding and transferring our tokens, called “cheq”, to other accounts on the network.

• Basic decentralised identity primitives

  • Writing Decentralized Identifiers (DIDs) entries on a ledger paying for DID writes on the cheqd testnet using testnet tokens. As our CEO, Fraser Edwards, and Javed Khattak (our CFO) described in this blog post why self-sovereign identity needs a token; we want to enable new business models for digital identity.

  • NYM is the term used by Hyperledger Indy for Decentralized Identifiers (DIDs) that are created on ledger. For the sake of explaining with similar concepts as current Hyperledger Indy implementations, on our testnet transactions to add a DID to the ledger are called NYM transactions. Future releases of our software are likely to replace the NYM terminology with DID for better understanding.

Full changelog: cheqd-node v0.1.19 release notes on GitHub

This list contains all stable releases in the cheqd-node v0.2.x family, and skips pre-release versions.

v0.2.7 [November 2021]

As we come towards our mainnet release, the purpose of this release is to test out the range of functionality we plan on releasing onto the cheqd mainnet.

Major features

• Bump Cosmos SDK to v0.44.3

• Swap the context field to top of the DIDDoc, before ID

• Add identity specific tests

• Add DIDDoc to Identity ADR

• Get rid of symlinks and add ability to change HOME directory by user

Changelog

• Upgrade documentation by @anikitinDSR in #181

• Get rid of symlinks and add ability to change HOME directory by @anikitinDSR in #185

• Errors documentation by @asmarty in #171

• Add identity specific tests by @VladimirWork in #170

• Add DIDDoc to Identity ADR by @Toktar in #170

• Bump cosmos sdk to v0.44.3 by @askolesov in #186

• Add swagger docs by @Toktar in #174

• Swap the context field to top of the DIDDoc, before ID by @Toktar in #199

• Default configs + configuration subcommand by @askolesov in #187

• Fix testnet docker image by @askolesov in #200

Full changelog: cheqd-node v0.2.7 release notes on GitHub.

This list contains all stable releases in the cheqd-node v0.3.x family, and skips pre-release versions.

v0.3.1: Mainnet go-live [November 2021]

In November 2021, we released the cheqd mainnet with the support of our wide range of partners.

cheqd was always meant to be designed with decentralised identity at its core. As such, this release is where we implemented our foundational identity functionality.

Changelog

• Added functionality for decentralised identity, specifically Decentralised Identifiers (DIDs), utilising the cheqd DID method

• Amended the genesis parameters

• Integration with Keplr, OmniFlix

Full changelog: cheqd-node v0.3.1 release notes on GitHub

Context

Throughout 2022 we released a breaking upgrade roughly once every quarter. Throughout 2023 we continued with this frequency, however releasing non-breaking point releases consistently. In 2024, we intend to release one major network upgrade per quarter.

Release Timelines

To ensure we upgrade all nodes on the network in the most efficient way whilst maintaining due care to the upgrade process, we run the process in the following steps:

• Testnet proposal

• Testnet upgrade

• Mainnet Upgrade Proposal

• Mainnet On Chain Upgrade

Get started

Take a look at our network upgrades across the years, including Root Cause Analysis of any Network outage.

See below for all of our cheqd-node ledger software releases in 2022:

• v0.4.x family

• v0.5.x family

• v0.6.x family

This list contains all stable releases in the cheqd-node v1.x.x family, and skips pre-release versions.

v1.4.0: Fix interactive installer [March 2023]

This version primarily fixes the interactive installer, an off-ledger helper software for carrying out cheqd-node installations and upgrades. The installer now:

1. Attempts to create a backup of the user's configuration folder and private validator state files, when carrying out upgrades or downloading snapshots.

2. Frees up spaces by deleting the data folder when requesting a snapshot restore (only if the backup has been made).

3. Bumps Cosmovisor if necessary.

4. Set environment variables for cheqd-node as well as for Cosmovisor.

5. Separates out warning, error, info etc level messages based on success status of the action.

To enable debug level logs in case the installer fails, set an environment variable called PYTHONDEVMODE to the value 1 on your system. This will provide additional diagnostic logging that can help pinpoint errors if the installer fails.

Read Full changelog on GitHub: cheqd-node v1.4.0 release notes

v1.3.0: Fix pruning settings [February 2023]

This version creates a standalone Golang package called github.com/cheqd/cheqd-node/api which is generated using the Pulsar generation file. This allows non-ledger applications such as the did:cheqd Resolver to consume a standalone package with just the API interaction elements, without needing to import the entire cheqd-node Golang package.

This version also bumps Cosmos SDK to v0.46.10.

Read Full changelog on GitHub: cheqd-node v1.3.0 release notes

v1.2.5: Fix pruning settings [February 2023]

The previous version of our node software had to be run in pruning="nothing" setting due to an upstream issue with the Cosmos SDK iavl package. This release fixes and restores support for setting pruning settings.

Read Full changelog on GitHub: cheqd-node v1.2.5 release notes

v1.2.2: Custom identity transaction pricing, improvements to DIDs and DID-Linked Resources, Cosmos SDK v0.46.x [January 2022]

This release is a state-breaking update that brings fundamental changes to the cheqd-node testnet and mainnet. The release notes below pertain to the cheqd-node changes only. Changes to identity applications and tooling is summarised in a separate change log.

Summary

The release can be broken down into 6 primary areas of development:

1. Identity transaction pricing and burn

2. Enhancements to DIDs and DID-Linked Resource functionality

3. Changes to protobuf definitions required for enhancements to DID and DID-Linked Resource functionality

4. CosmosSDK version bumps

5. Miscellaneous fixes

6. Introduction of Behavioural Driven Development

1. Identity transaction pricing

In this release we’ve introduced new mechanisms to the cheqd’s tokenomics. Currently (prior to this release), writing a DID to cheqd mainnet only cost the gas fee, which was approximately 0.004 $CHEQ. This is extremely cheap when compared to networks such as Sovrin, where writing a DID costs $10, especially taking into account the value we are bringing as a network, protocol and team. Alongside this, $CHEQ is currently inflationary (where staking rewards come from), with only slashing, tombstoning or no-with-veto slashing providing any deflationary pressure.

Going forward, any changes required to periodically maintain these, especially before we implement pricing oracles to programmatically maintain stable dollar pricing for our partners, will require further votes.

Architectural decisions, diagrams and in-depth explanations of the scope and relationships will be shared shortly in a follow-up blog-post and document.

Mechanism 1: Identity transaction pricing

Identity transactions such as DIDs and DID-Linked Resources (e.g. visual representations of Verifiable Credentials; trust & revocation registries; schemas; documents; logos) writes will have a cost greater than gas which will increase rewards from the network as identity utility increases. At current price levels ($0.03-$0.04), writing a DID will equate to 66.6–50 $CHEQ, which leads us to where these tokens will be distributed upon writes. See detailed pricing and comparison to Sovrin here.

Mechanism 2: Burns

As mentioned above, the network is currently inflationary only aside from slashing, tombstoning and no-with-veto having deflationary effects. Implementing a burn mechanism will bring the system into equilibrium. The volume of tokens burnt will increase in the same way as distributions will, with the aim of bringing total supply back to one billion.

As can be seen in the image available here showing the genesis parameters which define the costs in $CHEQ, there is also a “burn_factor” variable which sets the percentage of the tokens charged for identity transactions which will be directly burnt. This can be adjusted through on-ledger governance votes according to the cheqd community preferences.

2. Enhancements to DIDs and DID-Linked Resource functionality

A number of core changes have been implemented to enable continued interoperability with products and applications across the decentralised identity ecosystem. These include:

Introduction of a DID deactivation transaction

Previously it was not possible to mark DIDs as deactivated on the cheqd network. This is now possible. This enables cheqd to support each of the required transactions for integrating with the Universal Registrar.

Historical version of DIDDocs can be stored and retrieved

Fetching historical versions of DIDDocs easily and without friction is an incredibly important feature for DIDDoc indexing in decentralised identity applications.

We have used the same versioning techniques for DIDDocs as was implemented for DID-Linked Resource versioning. This required an on-ledger change as previously we were using the ‘transaction hash’ of the on-ledger block as the DIDDoc ‘version’, which presents a problem as in the event a user has multiple DID operations in the same transaction, it becomes impossible to distinguish between the various versions of the DIDDocs. This has now been replaced with a UUID.

A number of other networks lack this feature and don’t provide the functionality to fetch historical DIDDocs, which can be a limiting factor for those building decentralised identity applications.

Enforce UUIDs to be in lower case by default for DIDs & DID-Linked Resources

UUIDs are not case sensitive. This means that UUIDs with both upper and lower case characters are semantically identical

Previously when users created a UUID for a DID-Linked Resource they could use both lower and upper case in the UUID. The cheqd network, however, is only able to handle lower case characters. As such, we introduced a converter that would convert all UUID DIDs to lower case UUIDs.

As a result of this change, existing DIDs and Resources that were created prior to this upgrade, will be altered to lowercase if they previously contained uppercase characters.

Bringing Indy-style DID identifiers in line with specification

The style for the unique identifier of a DID was initially based on the legacy Hyperledger Indy algorithm. We’ve brought it in line with the updated DID Indy specification requirements instead.

As part of our interoperability efforts and support for AnonCreds with AFJ (documented here we needed to expand the supported identifiers on-ledger. The main barrier preventing full support for AnonCreds on cheqd using Aries SDKs is that AnonCreds is still dependent on the Indy SDK. When you call any function in the Indy SDK, you need to use the legacy Indy-style identifier format to carry out a transaction. Supporting Indy-style identifiers is therefore a stepping-stone in the wider AFJ support for cheqd.

Ability to specify alternative URLs for DID-Linked Resources

By default DID-Linked resources have a DID URL auto generated. For example, a DID URL for a DID-Linked Resource looks like this:

did:cheqd:testnet:z6jKUJA5YcZsNxZgsrQPKPipL2FRTf4s/resources/8140ec3a-d8bb-4f59-9784-a1cbf91a4a35

When resolved (see here) you’ll find that this shows the IIW logo, used in our wallet demo.

However, as part of our mission to drive the wider adoption of DID-Linked Resources beyond cheqd, and create the “web of trust”, we have enabled the ability to add an alternative URI for where that same resource can be found.

This is similar to the ‘alsoKnownAs’ property in the DID Core specification.

Checksums for DID-Linked Resources now stored correctly

A checksum is a value that represents the number of bits in a transmission message, used to detect high-level errors within data transmissions. They provide a party on the receiving end information about the date to make sure that the full range of data is fully delivered. The checksum value itself is typically a long string of letters and numbers that act as a sort of fingerprint for a file or set of files to indicate the number of bits included in the transmission.

For example, if a user created a DID-Linked Resource, and 1 month later wanted to retrieve this, it would be possible to calculate the sha256 locally and then compare it with the “checksum” field from the resource in order to check whether this resource was changed or not.

Prior to this release, it was calculated in a less usable way, using “sum256 + data_bytes” where it only needed sum256 (which is what is used from v1.0.0 going forward).

3. Changes to protobuf definitions required for enhancements to DID and DID-Linked Resource functionality

To accommodate for the ledger enhancements, fundamental changes are required to the underlying code base which the ledger uses known as protobuf definitions.

Protobuf definitions can be seen as a template for variables in the ledger code-base; the existing release up to this point uses protobuf definitions v1, whilst this upgrade replaces these with v2 protobuf definitions. For example identity transaction pricing and the ability to provide an alternative URI for a DID did not exist in the previous version, meaning this feature would not be possible to implement.

To inspect the source of protobuf definitions, follow the links below. These are simply contained within the overall v1.0.0 upgrade so don’t require any additional steps for developers.

Previous protobuf definitions:

• cheqdid.cheqdnode.cheqd.v1

• cheqdid.cheqdnode.resource.v1

Updated:

• cheqd.did.v2

• cheqd.resource.v2

Another aspect of this change is based on best practices. When we launched the cheqd network we identified certain best-practices and designed an implementation approach which would allow us to roll out these conventions in a phased approach. This enabled us to move quickly, whilst not compromising anything on ledger in the initial stages. With this release the cheqd ledger is in line with overall style guides making it more flexible and robust for both our team and other developers to build on.

4. Cosmos SDK version bumps

As is customary with the use of the underlying CosmosSDK and associated libraries, regular version bumps are required to keep the cheqd ledger in line with the wider ecosystem's changes, for example updates to IBC. In this release we’ve bumped the Cosmos SDK version to 46.6, crucially bringing key security patches as well as a bump to IBC to ibc-go v5.1.

5. Miscellaneous fixes

cheqd-noded configure command removed

As part of the overall ledger changes, the cheqd-noded configure command has become an outdated and deprecated command. The command has been removed and documentation has been updated accordingly. This changes the manual installation of cheqd-node, and the Interactive Installer which previously relied on this.

Interactive Installer fixes

The Interactive installer was using the legacy configure cmd ‘cheqd-noded’ to do the configuration for app.toml and config.toml files. As this is no longer used, the interactive installer no longer requires the configure command.

The following changes are now made by default when installing a node:

• RPC port by default listens to world i.e from tcp://127.0.0.1: to tcp://0.0.0.0:

• The default minimum gas price is set to 50ncheq, increased from 25ncheq

• An option introduced for a user to enter persistent peers, log level and log format when doing interactive installation

Ledger Nano support fix

A bug was identified where the Ledger nano hardware wallet was not working with cheqd transactions. This has been resolved in this release.

6. Behaviour Driven Development

To date our testing approach has been focused around unit testing and more manual approaches to application testing. In this release we’ve introduced Behaviour Driven Development (BDD) at scale across the entirety of the cheqd node. BDD, is a framework which uses human-readable descriptions of product requirements as the basis for testing. Developers implement a well known vocabulary that can be understood by a regular user, to help identify the issues that arise.

We see this as an integral to building strong foundations that will enable developers to more easily build with cheqd, and streamline the overall development process for our core team as we progress further with ledger development centred on the implementation of cheqd payment rails.

Full changelog

Features

• feat(installer): Get installer to handle multi-platform binaries DEV-1667 by @ankurdotb in #376

• feat: DeactivateDid transaction to mark DIDs as deactivated DEV-1774 by @khornonzon in #332

• feat(proto): Proto API v2 DEV-1786 by @askolesov in #398

• feat(proto): Refactor app to support proto v2 by @askolesov in #436

• feat: DidDoc versioning DEV-1892 by @askolesov in #443

Bug fixes, patches & version bumps

• fix(installer): Fixed binary permissions DEV-1826 by @Eengineer1 in #414

• fix: Add resourceType for versioning functionality DEV-1703 by @anikitinDSR in #397

• fix: Checksum calculation DEV-1649 by @anikitinDSR in #396

• test: Fix installer by @anikitinDSR in #348

• fix: Ability to write spaces in the moniker by @Big0ak in #354

• fix(release): Bump semantic release package by @ankurdotb in #377

• fix(installer): Fixed binary permissions DEV-1826 by @Eengineer1 in #414

• fix: Error handling by @askolesov in #425

• fix: Use proto marshaller instead of amino to derive sign bytes DEV-1694 by @askolesov in #430

• fix: Convert UUID DIDs to lower case UUID DEV-1705 by @Toktar in #394

• fix: Support Indy-style identifiers DEV-1750 by @Toktar in #399

• fix: Add support for Ledger nano by @ankurdotb in #442

Build

• build(buf): Change buf.build package name by @ankurdotb in #453

• build: Buf.build config for generating proto DEV-1545 by @jay-dee7 in #339

• build(docker): Move to Alpine images for Dockerfile by @anikitinDSR in #356

• build: Backport workflow and other fixes to release/0.6.x branch DEV-1822 by @ankurdotb in #410

• build: Release with v-tags by @ankurdotb in #424

Security

• security(tendermint): Bump github.com/tendermint/tendermint from 0.34.19 to 0.34.20 by @dependabot in #367

Testing

• test: Move unit testing to Gingko DEV-1652 by @anikitinDSR in #373

• test: Integration tests merging state by @Eengineer1 in #419

• test: Add interactive installer to workflow and get rid of deb by @anikitinDSR in #347.

This is a list of root cause analysis (RCA) of major network incidents/outages. These are intended to summarise any investigations in more detail.

ℹ️ To check current network status and incidents, please check instead. Our also offers network-wide capability of monitoring issues with missing blocks.

Root Cause Analysis (RCA) List

2023

• January 2023:

See below for all of our releases in 2024:

• todo

See below for all of our releases in 2021:

• family

• family

• family

This list contains all stable releases in the v0.5.x family, and skips pre-release versions.

: Improved DID functionality [March 2022]

This new node version is intended to enhance functionality currently available on v0.4.x. The upgrade to v0.5.x will be a breaking change that introduces new routes, fixes a few technical debt issues identified and overall offers significant enhancements to the identity functionality and security of the network.

Full changelog:

Identity Improvements

Improved validation of did:cheqd identifiers

What has changed?

Our provides a unique identification number for a DID Subject. This can be thought of in a very similar way to traditional bank cards. (Note: While the example of a bank card is used here, DIDs on ledger are typically written for companies, not individuals.)

Within a bank card, different sections of the card identify different actors, such as the Network, Issuer ID and a Unique Account Number - and when put together, you have a complete card number. DIDs are similar, in the sense that different components of the DID mean different things:

In this update, cheqd has updated the way that DIDs are created and resolved to ensure that the Unique Identifier is a Base58 encoded string (either 16 or 32 characters long). This was done to keep it consistent with the

Why it is important?

Prior to the upgrade, the ledger code itself was relatively “dumb”: the responsibility of ensuring that a valid unique identifier was created, matching our DID specification, would be done client-side. While the cheqd ledger did implement a uniqueness check to ensure that no other DIDs with the same identifier existed within a namespace, it didn’t implement checks to ensure that the identifier specification was met. (This is broadly how a lot of Hyperledger Indy SDKs currently do this.)

We believe that that there will be a blend of various levels of technical complexity in client-side SDKs that interact with DIDs, so relying on only client SDKs to implement this is an unsafe method. By adding checks on the ledger itself, it allows us to ensure consistency across a wide range of client SDKs.

We are additionally exploring future iterations which would allow unique identifiers to be defined as instead of the Indy DID method technique as well.

More comprehensive cryptographic checks for DID updates

What has changed?

The cheqd DID method allows multiple controllers, verification methods, and complex relationships to be defined which brings it in-line with the DID Core specification. But this also introduces new challenges. To illustrate through an example, let’s say there is:

2. Next, there is a request to add a new Verification Method to the DID Document.

3. The question is: how should this DID Document be updated; who needs to sign to make this change?

Why it is important?

Prior to the upgrade, when a DID update request was made, the ledger would only ask for the existing Verification Method(s) listed in the DID Document to sign to approve the change. We have changed this to ensure that both the existing and new Verification Method keys need to sign the update request.

Improved DID Resolution metadata for updated DIDs

What has changed?

When a DID is resolved, it fetches resolution metadata in addition to the DID Document. This metadata contains entries such as when a DID was created as well as when it was most recently updated.

Prior to the upgrade, the Update metadata field would contain the same data as the Created field if the DID had never been updated. We’ve changed this to a null value instead of the same date/time, to make it easier for software to understand this scenario.

Why it is important?

If you see the Update field reads null, it means the DIDDoc has never been updated. This clears up any potential confusion and aligns the DID resolution metadata.

Implemented static validation for Verification Method keys

What has changed?

We have changed the way we validate Verification Method keys, splitting this into:

• Static validation;

• Dynamic validation.

The former, static validation checks, are performed before the inclusion of a transaction in the mempool; and the latter, dynamic validation checks, are performed during the transaction execution once it is included in the blockchain.

Why it is important?

Splitting these is important because now errors in Verification Method keys can be caught earlier and flagged, before any transaction enters the mempool.

Note: The mempool serves the purpose of keeping track of transactions seen by all full-nodes. Full-nodes keep a mempool cache of the last ‘mempool.cache_size’ transactions they have seen, as a first line of defence to prevent replay attacks.

Therefore, the validation improvements will allow nodes to reject invalid transactions in the early stages without including them in the chain and charging the user.

This in turn speeds up the cheqd network as incorrect transactions are not being committed and incorrect Verification Method keys are not errored out when committing a transaction to the mempool.

We’re also using other work to build similar efficiency improvements:

• This validator package for null checks, generic DID validity checks in static validation

• This validator package for DID namespace checks in dynamic validation.

Implemented JSON Web Key (JWK) support

What has changed?

Within the DID Core specification there are two types of key in a Verification Method:

• JSON Web Key (JWK); and

• Public Key Multibase.

Previously, the only format of key supported was publicKeyMultibase

Having only one of these two options limits the scalability and interoperability of the cheqd network, as the supported Verification Method keys are limited.

Therefore, we implemented JSON Web Key (JWK) support.

Why it is important?

With JWK, cheqd now supports a broader selection of Public Key encoding schemes, which is an important improvement, given our focus on interoperability and eventual cross-ledger support.

Tech Debt & Bug Fixes

Fixed date/time representation in DID Resolution

What has changed?

Previously, we were using a Cosmos format of datetime, rather than what is defined in the DID core spec. Now this has been changed to align with DID Core.

Why it is important?

This aligns our DID Document metadata with DID Core, making it more semantically interoperable and spec compliant.

DID metadata VersionId now populates a Tendermint’s tx_hash in the correct format

What has changed?

As cheqd is a chain built on the Cosmos SDK, it relies on Tendermint as its consensus mechanism.

This means that when something is written to the network, there is a Tendermint transaction hash that is created. This is specific to Cosmos chains.

Within DID Document metadata, there is a field called VersionID which, as the name suggests, denotes the specific version of the DID Document.

Previously, in this field, we were generating a hash from the transaction itself to calculate the DIDDoc version ID. We have now changed this to populate a Tendermint transaction hash.

Why it is important?

This is important because the DIDDoc versionId can now be retrieved right after the creation of the DID, with the DID now more easily searchable on a Block Explorer. Previously, we needed to create the DID, then ask it to create version ID, then update.

This update streamlines the process and makes it more efficient.

Date: 30th January 2023 Status: Resolved

Introduction

In the lead up to cheqd v1.x mainnet upgrade, and during the eventual release and upgrade itself, we faced a number of challenges which resulted in a brief network halt of mainnet and the need for a fast-follow patched release.

As of the 8th February 2023, this patched release has now been applied across both testnet and mainnet validators, and the network is working as expected, with the issue identified during the upgrade now resolved.

This RCA provides a brief overview of the root cause of the issues identified, the fix itself released in v1.2.5, and the lessons learnt by the cheqd Product & Engineering team.

Summary of events

On Monday 30th January, the cheqd team initiated a major network upgrade, v1.x, which introduced identity transaction pricing, among other features and fixes to the network (v1.x changelog).

Following the passing of the upgrade proposal (proposal #12), At roughly 09:30 GMT, the network halted at block height 6,427,279, with the upgrade set to begin at height 6,427,280.

Over the subsequent 40 minutes, to 10:10 GMT, validators successfully upgraded to the new version, with consensus reached at 10:19 GMT. Shortly thereafter, following just ~10 blocks signed (6,427,290), a number of validators reported errors and timeouts with their nodes, examples:

ERR Stopping peer for error err="pong timeout" module=p2p peer={"Data":{},"Logger":{\}}
Jan 30 13:01:32 ip-172-31-16-181 cosmovisor\[695507]: 1:01PM INF Timed out dur=3000 height=6427331 module=consensus round=0 step=3

Diagnosis

By the end of the day, we were able to identify three major three major types of issues we encountered:

1. Cosmos SDK v0.46.x upstream bug which requires “pruning = nothing”

The pruning issue was largely related to the DragonBerry patch, which was a fix of a high-risk security vulnerability (dubbed “Dragonberry”) related to IBC protocol / ICS23.

Within the patch a strict check was introduced on horizontal states in trees, and as a result, uneven heights, pruning + state-sync no longer were behaving as expected. After introducing the Dragonberry patch it’s been common that the store height does not equal the historic height. The store height not equalling the historic height is what caused our pruning issue. This is largely due to an known issue in the iavl state tree package + Cosmos SDK which occurs on store writes during upgrades.

2. Leftover legacy cheqd-noded versions

On some validators we found there was a v0.6.x binary in the /usr/bin folder, whereas under Cosmovisor that gets turned to a symlink from /usr/bin/cheqd-noded to the actual binary in /home/cheqd/.cheqdnode/cosmovisor.

3. Corrupted database

Some nodes had an issue executing the migration, where they seemed to run into an issue with the new group module that Cosmos SDK introduced in v0.46.x, although this seems to be an edge case.

Separately, we also experienced dependency breaks (dependency hell) on a number of the internal and external servies which use the network APIs following changes to API paths within this upgrade.

How come we didn’t pick this up on testnet?

During the testnet upgrade we experienced a consensus error due to an issue with module versions among other issues (fortunately this is what testnet is for). As a result we started a fresh test network from block 0 + state export (testnet-6). With this fresh network, pruning was set to default ( pruning="default").

This is the default setting for every node. However, this default option also means that approximately 3.5 weeks of the state is kept which is not enough to catch the pruning issue early. Switching to more aggressive setting would’ve caught issues earlier, for example: pruning="custom", with:

pruning="custom"
pruning-keep-recent=50
pruning-interval=10 

Resolution

Immediate resolution

Immediate resolution to get the network running was offered to us by our validator community, as it was an issue ~10 Cosmos chains had faced when getting to the latest version.

In order to get the network back up and running, the validators were advised to simply switch their pruning parameters, setting them to pruning = “nothing”. Once consensus was reached with validators who had switched to this, the network was restored.

Long-term solution

Setting pruning to nothing was not an ideal solution, given it meant the network was not being pruned, resulting in an ever increasing chain size and increased storage requirements / higher costs for validators.

After investigating a route to solve this, we found a patch authored by Chill Validator, released in late November 2023, which had already been used as a solution for other Cosmos SDK based networks facing the same issue.

Lessons learnt

1. Testnet extensive checks & snapshots

Going forward we should endeavour to create as near an identical environment for testing upgrades on testnet as possible. While it may not always be possible, due to modules like IBC not being available on testnet, we should make sure that the delta between mainnet and testnet is detailed comprehensively in advance of upgrades to make sure any potential issue can be quickly diagnosed.

Where there is any risk vector of a consensus fault going into an upgrade, full snapshots should be taken of the network before any upgrades are attempted.

2. Check other networks upgrade issues and downtimes in Cosmos ecosystem

The issue we encountered during this upgrade could have been discovered in advance by aligning more proactively with other Cosmos chains. With a Validator community that spans far more than the cheqd Network, we should try and identify any potential issues experienced on other chains before bumping Cosmos SDK or IBC versions. Likewise, we should be circulating any issues we uncover. We’ll be more actively using this space within the product site to report on these going forward.

3. Consider how Cosmosvisor preparations need to change if manual upgrade is required

While we’re thrilled that the Cosmovisor automatic installation process largely worked as intended for this upgrade, however we still need to be mindful of how technical minutiae like symlinks are handled differently with manual upgrades. Having a clear overview of the delta between manual and Cosmovisor upgrade patterns will help isolate any potential issues in future upgrades.

4. Have a non-critical (high voting power) node take backups and snapshots

Reaching consensus initially at upgrade height took significantly longer than expected, due to the node managed by the cheqd team not being upgraded (cheqd node). This was because the cheqd node was taking a snapshot, at the halt height, to ensure there was a backup immediately before the upgrade to roll back to in case of a major upgrade failure. Going forward, rather than using this node for snapshots, a secondary node will be used to take snapshots to avoid delays.

5. Upgrades should NOT take place on Mondays

Generally we have intended to upgrade on Tuesdays or Wednesdays, however this time due to urgency of this upgrade and team vacation we opted for a Monday. This however doesn’t allow much time to get everyone prepared, which we’ll avoid going forward.

6. Improve upgrade height / time forecasting calculations

When submitting the mainnet upgrade proposal, like all other Cosmos-based network, we log the intended upgrade block height. This is calculated by taking the current height, and adding on the number of blocks required to reach the agreed upgrade time and date. The number of blocks to add depends on the average time per block. Unfortunately we were out with our estimated time, with the upgrade height being reached two hours earlier then planned. As a result, certain members of the team were unavailable at the upgrade time. In the future, we'll improve the accuracy of our estimations, ensure team availability is confirmed for a wider time window, and add sufficient buffer time to allow for block time being faster or slower for whatever reason.

7. Create a dependent Internal & External services check-list

With network upgrades on this scale, a number of changes are required in our internal and external services that depend on the core ledger. Following the upgrade we begun identifying which services were knocked out due to dependency changes, but this wasn’t done in a systematic, coordinated manner. Going forward, we’ll create a check-list of all internal and external services that need to be checked and updated following upgrade, to reduce downtime.

8. Streamlining status updates, messaging & communications

Although generally running two forums (Discord & Slack) has worked, during this upgrade there was a clear misalignment which could have been resolved if everyone was conversing in one place. Slack has remained as the dominant location for communications with SSI vendor validators, and Discord for the Cosmos-based validators. Going forward we will review the use of two communities, and decide on how to coordinate better, likely reducing to one group.

We also failed to effectively utility status.cheqd.net to provide updates. This generally needs to be incorporated more into the Product & Engineering teams ways of working, and in particular at critical points like upgrades.

. . .

Thank you to our validators for your patience and support throughout. Fortunately this didn’t cause any significant downtime, however it could have been avoided through more stringent checks on testnet, and more communication with other Cosmos SDK chains.

You can also read more about about plans for the year ahead in our Our Product Vision for 2023 at cheqd 🔮.

See below for all of our cheqd-node ledger software releases in 2023:

• v1.x family

This list contains all stable releases in the cheqd-node v0.4.x family, and skips pre-release versions.

v0.4.1: Simplified Docker setup and stability improvements [March 2022]

Our packaged releases are currently compiled and tested for Ubuntu 20.04 LTS, which is the recommended operating system for installation using Debian package or binaries.

For other operating systems, we needed to provide an alternative approach. Therefore, this point release was initiated which offers a pre-built Docker image releases for cheqd-node.

Changelog

• Docker Compose Easy Setup (#250)

• Add euox pipefail check for bash scripts (#266)

• Test for positive case of upgrade process (#268)

• Refactor Debian package installer to handle upgrade scenarios better (#279) (Andrew Nikitin)

• Unified network configuration generation (#267)

• Further docker, tests improvements (#280)

• Updated recommended docker image version

• Revert fastsync version to v0 (#256)

• Fix small errors around postpurge script

Full changelog: cheqd-node v0.4.1 release notes on GitHub

v0.4.0: Stability improvements after mainnet launch [January 2022]

This software upgrade proposal upgraded the version of cheqd-node software on our mainnet from v0.3.1, to v0.4.x.

Following the successful release of our mainnet in November 2021, we wanted to fast-follow with a some immediate improvements we felt necessary for the cheqd network.

This new node version is intended to enhance functionality currently available on v0.3.1. The upgrade to v0.4.x will be a breaking change that introduces new routes, plus fixes a few technical debt issues identified in the v0.3.x branch.

Changelog

• Bumped Cosmos SDK version from v0.44.3 to v0.44.5

• Update DID operations in version v0.3.1 did not carry out a check on the keys used to authenticate the transaction.

  • In essence, this meant that any update DID operation could incorrectly update a DID or its associated DIDDoc. This has now been fixed.

• Switched build system from using Starport to using Makefiles.

  • This gives greater control for future improvements and optimisation in adding support for other operating systems, database backends etc.

• The node software binary has now been compiled with support for Ledger hardware wallet devices for key storage.

• REST/gRPC endpoints for querying DIDs have been added. Documentation will be added to explain how these endpoints work.

Full changelog: cheqd-node v0.4.0 release notes on GitHub.

This list contains all stable releases in the cheqd-node v0.6.x family, and skips pre-release versions.

v0.6.10: Genesis export fixes [December 2022]

Read Full changelog on GitHub: cheqd-node v0.6.10 release notes

v0.6.9: "Dragonberry" security vulnerability patch [October 2022]

The following steps have been prepared by the cheqd development team for cheqd validators to follow in order to manually upgrade their node.

This is in response to a high-risk security vulnerability (dubbed "Dragonberry") related to IBC protocol / ICS23.

This is closely related to the "Dragonfruit" security vulnerability published on Cosmos forums. v0.6.9 fixes this vulnerability by patching to Cosmos SDK v0.45.9 upstream.

Considerations

This patch is non-state-breaking. This means that validators can switch to the new binary, and restart their node without necessarily needing an on-chain software upgrade (which is intended for consensus-breaking changes).

The patch has already been applied to cheqd's own validator node and one of the other large validators (at the time of writing this).

Read Full changelog on GitHub: cheqd-node 0.6.9 release notes

v0.6.7: Multi-platform builds [August 2022]

Read Full changelog on GitHub: cheqd-node v0.6.7 release notes

v0.6.1: Installer and Docker fixes [August 2022]

Read Full changelog on GitHub: cheqd-node v0.6.1 release notes

v0.6.0: DID-Linked Resources & Tooling for an Improved Developer Experience [July 2022]

Here’s what’s new in the latest cheqd node version - v0.6.0. This release is a state-breaking update that brings fundamental changes to the cheqd-node testnet and mainnet. The release notes below pertain to the cheqd-node changes only.

Summary

This upgrade includes 3 primary enhancements and new features, each of which helping accelerate cheqd to support a greater number of credential formats & SDKs, more use cases for decentralised identity applications and streamline the end-to-end developer experience.

1. Implementation of a Resources module, used to support DID-Linked Resources

2. Introduction of an Interactive Installer to streamline the developer onboarding experience

3. Introduction of Cosmovisor for automated upgrades.

Feature Deep-dive

1. Implementation of a Resources module, used to support DID-Linked Resources

What's changed?

The concept of resources in self-sovereign identity (SSI) ecosystems is not new, however, as we will discuss throughout this blog post, existing approaches to resources in SSI oblige adopters to make compromises between security, availability and interoperability. We first noticed this when we were looking at how we could securely reference credential schemas, something we will expand on throughout this post.

Our objective in building resources on cheqd is to improve the way resources are stored, referenced and retrieved for our partners and the broader SSI community, in line with the existing W3C DID Core standard.

An initial use case for DID-Linked Resources is supporting AnonCreds on cheqd. This is an industry first as AnonCreds have to date been tightly bedded to the Indy ledger. We are using the Resources module to create DID-Linked Resources for schemas and CredDef, bringing AnonCreds support to the cheqd ledger in a W3C compliant format.

Please take a look at our blog which provides a thorough explanation for your reasoning on this approach, and links to our documentation.

2. Introduction of an Interactive Installer to streamline the developer onboarding experience

What's changed?

Prior to this upgrade, we offered developers looking to setup using a Ubuntu .deb package file and installing a with Docker

Although these were fit for purpose, allowing the vast majority of validators on the cheqd network to get set up these methods do require a higher level of technical skillset, and are more time-consuming.

The Interactive Installer offers a more intuitive and simple developer experience, which guides developers through a step-by-step journey, asking questions with the CLI along the way.

Find out more about how to get started using the Interactive Installer here.

3. Introduction of Cosmovisor for automated upgrades

What's changed?

With an increasing number of validators on the cheqd network, we wanted to provide developers managing nodes the opportunity to more easily upgrade their node.

Cosmovisor is a small process manager for Cosmos SDK application binaries that monitors the governance module for incoming chain upgrade proposals. If it sees a proposal that gets approved, cosmovisor can automatically download the new binary, stop the current binary, switch from the old binary to the new one, and finally restart the node with the new binary.

Full changelog

Read on GitHub: cheqd-node v0.6.0 release notes

Features

• DEV-1319 Add helper to convert ed25519 public keys to JWK by @askolesov in #321

• feat!: Add resource module (DEV-1281) by @Toktar in #330

• feat!: Interactive installer for cheqd-node with Cosmovisor support DEV-1321 by @anikitinDSR in #329

• feat(ibc): Bump IBC to v3 and update associated upgrade tests DEV-807 by @askolesov in #334

Bug fixes & patches

• DEV-1044 Fix gas estimation for cheqd transactions by @askolesov in #302

• Proposal CLI Registration by @keupsonite in #323

• chore: Bump Cosmos SDK to v0.45.4 by @askolesov in #328

• fix: Consensus issue with Resource.created by @Toktar in #344

• fix: Small fixes for installer after testing by @anikitinDSR in #343

• fix: Fix helper message for CLI all-resource-versions by @Toktar in #345

CI/CD

• Dispatch workflow by @ankurdotb in #308

• DEV-520 - Chore: Added Github CodeQL Action by @jay-dee7 in #303

• DEV-1096 Improve release processes by @anikitinDSR in #304

• build(tooling): Fixing workflows to main branch version by @ankurdotb in #313

• Fix workflow calls to refer local versions instead of master by @askolesov in #322

• Add gofumpt check to the lint workflow by @askolesov in #324

• Add a workflow to lint Semantic PRs by @askolesov in #326

• fix: Get linter back by @anikitinDSR in #338

• fix: Get rid of Debian package for release by @anikitinDSR in #341

Testing

• test: Add Cosmovisor support for tests DEV-1278 by @anikitinDSR in #320

• test: Add resources to upgrade integration tests by @Toktar in #340

Documentation

• DEV-1171 Add additional instruction for updating from 0.4.x version by @anikitinDSR in #301

• Bump cheqd version to 0.5.0 in docs by @Toktar in #307

• docs: Remove all documentation from cheqd-node to node-docs repo DEV-1320 by @ankurdotb in #333