TLDR: We’ve released a new cross-implementation Python test framework to find more interoperability bugs before they hit users. This is a great way to get involved in Lightning core development!

Once Upon A Time…

Two years ago, we started a small project to create a Lightning protocol test suite to go alongside the BOLT specifications. This involved a special language you’d write tests in, some Python code to interpret and run the tests, and a facility to smoothly connect to the node being tested.

However, writing these tests was hard, involving much hand calculation of what the results were supposed to be. Maintaining them as both the Lightning spec and Lightning’s various implementations were being separately updated was almost impossible.

Worse, the tests required nodes to be configured with exact, specific secret keys, which required modifications to the Lightning implementations themselves. Only c-lightning had the necessary invasive options to enable this (in developer mode).

And the tests were mainly designed to test well-trodden features already known to work — only in the case of some early dual-funding prototypes did Lisa use it to test some in-development features.

Despite all these shortcomings, the protocol tests proved their usefulness by finding numerous c-lightning bugs, including CVE-2019–12998/12999/13000 which could have led to the loss of funds in all implementations.

Ultimately, the test suite’s awkwardness meant that the repository has sat untouched, unloved, and bit-rotting for the last year.

But This Needs To Exist!

Yet, with the rate of spec changes increasing, the matrix of features supported increasing, and new Lightning implementations coming online, the need for a good protocol testing suite we can all use has never been clearer. There is no better example than the last c-lightning release, which required an emergency “dot” release because it didn’t interoperate with Eclair properly.

So for the last month, I’ve ignored almost everything else on my to-do list to completely rewrite the protocol testing suite into something the Lightning developer community can all use and improve. This was one cause of the next c-lightning release being delayed for a month; but we all agreed this was extremely important to the health of the whole ecosystem.

Introducing lnprototest

The result is lnprototest: a new, pure-Python3 test system.

Tests are simply a list of Events, like Msg to send a message and ExpectMsg to receive a response. But the tool goes much further: TryAll indicates multiple different variants should all be attempted, such as running the test with different features or trying reconnection in the middle of the test. The result is what Computer Scientists call a DAG and we automatically make sure all paths are tested.

It also has Events to create commitment transactions, add and remove HTLCs, and other things you’d expect. For example,you can say you expect a field to be a valid signature on the current commitment transaction. Testers can also write their own custom Events for more specialized testing and Events can be conditional on what features were negotiated or other runtime behavior.

By using the battle-tested pytest Python runner, we gain features — like debuggability and parallel operations — in a way that most Python people will immediately understand and the rest of us can trivially take advantage of.

We also wrote new Python packages for Lightning, which anyone can use:

• The pyln namespace packages, particularly the new pyln.proto.message module.
• pyln.spec.bolt1, bolt2, bolt4 and bolt7, which contain the infrastructure required for Lightning messages and are autogenerated from the specs themselves.

So Much Work Still To Do

The first release of lnprototest only has two “runners” which control the tested node: DummyRunner and clightning.Runner. We would love to see other teams plug their implementations in too: my experience with the tool leads me to believe that it would find at least one issue in any implementation (probably trivial), even with the limited tests we have now.

Regarding the limited tests—in particular, we are missing HTLC acceptance and forwarding tests, mainly because we haven’t written a Sphinx implementation in Python. Implementing that would be a major functionality advance.

There are also some Python corners which we could definitely neaten, too. We will work on these over the coming weeks.

Join Us!

If you’re a Python developer who’s curious about Lightning, or a Lightning developer who’s curious about Python (or just love the idea of testing!), we welcome you to jump in on the GitHub page. I can also be reached on #lightning-dev on IRC, but my Australian time zone might be an issue.

Let’s keep the sats flowing!

Note: This blog was originally posted at https://medium.com/blockstream/announcing-the-lnprototest-alpha-release-f43f46f2c05