On Documenting polkadot-sdk

I have been trying to improve polkadot-sdk's documentation for more than a year now. It has been a difficult journey, and admittedly beyond what I and a handful of others can reasonably solve.

As a lot of teams are applying to solve the same problems through governance and/or Decentralized Futures, I am sharing my retrospective of a year+ of working on this. I will also share how I believe teams getting funding will be most effective in working on this:.

To a large extent, I am wrapping up my contributions to this effort. My intention in writing this is to ensure all of my thoughts around it are publicly available, should they be useful to anyone in posterity. All of this is my personal opinion, and not Parity's.

flowchart
	polkadot[Polkadot as a Platform] --> polkadot_sdk
	polkadot_sdk --> onchain
	polkadot_sdk --> offchain 
	onchain --> substrate
	onchain --> frame
	onchain --> xcm
	offchain --> subxt
	offchain --> PAPI
	offchain --> PJS-API

Retrospective

The Bad

I will start by acknowledging that explaining polkadot[/-sdk] is hard. This is what I learned both from doing async documentation, and participating as a teacher in the Polkadot Blockchain Academy.

1. First, there is a lot of Web3 background knowledge that needs to be explained, or assumed. More troublesome, Polkadot itself is a unique Web3 system, and has lots of novel concepts: Origins, Extrinsic, Runtime, Runtime API, Forkless Runtime upgrades and its implications, Host Functions, Metadata, Dispatch, Core (Polkadot Core), Parachain. I am not even opening the jar of all XCM related concepts here.
2. Rust is a difficult language, and the intricate way we decided to use it does not help.
3. Moreover, the space moves fast. Look at the material around polkadot-sdk 2 years ago. A lot of it is less and less relevant today. Heck, even the name polkadot-sdk was not around, we are still in the process of re-establishing polkadot-sdk as a replacement of substrate^[1].

The Good

• The third issue is in general being solved by making Polkadot more stable as a whole. This is a strong promise that we hear from Parity and the fellowship: For our developers, we will make Polkadot stable, and stable for a long time. This also means less changes in polkadot-sdk, and therefore increasing the longevity of all educational material that is henceforth created.

• The second is not solve-able by us per se, as we are not the creators of Rust and cannot reverse the decision to use Rust. But, some of the great steps we have taken to improve this include:
  • pba-qualifier-exam being open-source from the get-go.
  • Rust state-machine tutorial from @shawntabrizi, exactly created to improve this situation.
  • Trait based programming guide for FRAME developers.
  • (more ideas here, to be published later^[2])

• The first issue is mainly what I have tried to solve in the last year. I believe a lot of educational content in the Polkadot space feels incoherent, because it lacks solid foundation. As in, there is a lot of common foundational knowledge, that many educators have to either:
  • Re-explain in their own words, risking fragmentation, or being poorly explained.
  • Skip, and assume the reader already knows it.

With this hypothesis, I believe there are two tools future educators can use to help with this:

1. polkadot-sdk-docs: A strong foundation of API docs, references on foundational topics, and examples that are guaranteed to be always correct (because they live in polkadot-sdk repository and are always compiled as a part of its CI). Some of the FAQ that might be asked about this crate is covered in:
   1. Why Rust Docs?
   2. Scope
   3. Principles
2. Recordings from the Polkadot Blockchain Academy. I hope to see the latest Singapore edition be out soon (as the XCM module has undergone significant changes), but for now the Hong Kong recordings are an excellent resource.

The Future

• there are still a number of core topics around polkadot-sdk for which I am personally not content with the degree of written documentation, and I intend to provide those as soon as I can:
  • How accounts are stored and behave in frame-system
  • Benchmarking and Weights
  • Fees: Why they exist, under which conditions one can be exempted, and what tools FRAME provides for this. This should include the new feeless_if macro, and incorporate new personhood ideas presented by Gavin in Polkadot Decoded 2024^[3].

• Explore converting this body of information to a markdown version for better readability and search and aesthetics. Possibly branded as a "Mastering Polkadot SDK book".

Wishlist

This is how I think teams applying for funding regarding documentation will be most impactful.

Forward-Looking

• I wish to see more content created around the latest topics in Polkadot, rather than rehashing existing ones. Of course, a large part of the old materials that we currently have are still relevant, but they should be carefully handpicked and vetted. The topics of the year that are exciting to me are:
  • It is going to be all about Agile-coretime going forward, especially on-demand. A number of great tutorials should start with writing a pallet, adding it to a parachain-ready template, and end with acquiring one on-demand block on Paseo, and producing a block. This guide by Bader from Web3 Foundation is along these lines.
  • Use of new developer tooling that simplify Developer Experience: Tanssi, PoP, Chopsticks, etc.^[4]
  • OZ Templates
  • OmniNode
  • Umbrella crates
  • The above 3 imply we can massively simplify our templates to
    1. Use the umbrella crates instead of dozens of them
    2. Use omni-nodes, and be mere Runtime Templates, removing the entire /node directory.
  • Transaction/Signed Extension, and other features that are unique to what you can do a blockchain, and you cannot do in a smart contract environment.
    • Where contracts fail, and runtimes/chains are needed - Ecosystem - Polkadot Forum
    • polkadot_sdk_docs::reference_docs::runtime_vs_smart_contract - Rust
  • Personhood.

All in all, I suggest educators to focus on creating tutorials on forward-looking topics, even if it means producing less.

You might argue that a lot of this is pretty advanced, and hard to grasp for those who are outside the circle of core contributors of polkadot-sdk, and you are right about that. And this brings me to the next point.

Expect Better

Expect better from core devs of polkadot-sdk. Foundational knowledge that explains the basics of an API and how it is intended to be used is not a nice-to-have, but a must. Core developers should provide the foundation for educators and technical writers in the space to create new content about the latest features. I hope to see most of this low level documentation to live in polkadot-sdk-docs.

Finally, to make this process fruitful, you, as educators, build on top of the core documentation and backlink to it. This creates a strong mutual incentive for both parties to benefit from one another and maintain provide their share of the work. More specifically, I would imagine that polkadot-sdk maintainers will be more and more encouraged to provide foundational API documentation, if they know these are often used.

Build on the shoulder of giants, as the saying goes.

Awesome Lists

There are so many cool developer tools out there, yet I am surprised to not often find them. I think the best way to aggregate these are indeed a standard awesome-list. We have an old one that is no longer maintained by Parity, and a new community driven awesome-dot. I hope to see a consolidation of the two into awesome-polkadot, with a dedicated maintainer.