Overview

• Purpose: Analysis of the Lightning Network’s performance
• Language: Python
• Code
• Documentation

The Probe Plugin

Since its launch on mainnet in early 2018, the Lightning Network has been growing rapidly, both in number of nodes and channels, as well as in overall capacity. While those numbers are very impressive they do not convey the full picture. In order to gauge the performance of the network, we need to look deeper at how successful payments in the network are, and how quickly they are being settled.

I was curious about the performance and stability properties of the network. In order to scratch that itch, I wrote a small plugin, a sort of tool belt, to inspect and probe the network.

In our quest to quantify how successful and fast payments in the Lightning Network are, we have two distinct approaches:

1. Based on our local view of the network, we can compute a theoretic route and determine whether we’d be able to reach the destination based on the current network topology;
2. Computing an actual route and attempting to perform a payment.

For the Probe plugin, I decided to take the second approach. Since we don’t want to spend huge amounts just to find a viable route, we will use a little trick that allows us to probe for free.

Usually when performing a payment we get an invoice from the destination containing various parameters we are to use, including a `payment_hash` that allows the recipient to claim the payment by releasing the matching payment_preimage once the payment attempt reaches her. What happens if we just generate a random payment_hash, for which the recipient doesn’t have a matching payment_preimage? Well, the recipient will report back an error, that is distinct from other errors.

Probe therefore selects a random node in the network, computes a route, generates a random payment_hash and then fires off a payment attempt. The payment attempt will return an error, since the recipient can’t claim it, but by looking at the information in the error we can not only determine whether the payment would have gone through, but also various details about failed attempts, such as the channel that failed, whether we used too little fees, or whether the failing node is out of sync with the bitcoin blockchain.

Probing allows us to learn quite a bit about the Lightning Network and it helps us to better understand the performance and stability properties of the network. But, there is a risk that it could have a negative impact on the network health, by temporarily locking up funds along the route, and even causing some channels to close in case of a stuck payment.

I do however believe that the positives outweigh the negatives:

• Funds are locked up for a very short time until the error gets propagated back to the prober, and the amounts we probe for are tiny;
• By continuously kicking the tire of the nodes we may rely on later, we make sure that potential failures are detected early, and not once we actually need them for a payment that needs to succeed. If we weren’t causing faulty nodes to drop out early, a faulty node could hold up a real payment;
• We learn a lot about the network topology, making our future payment attempts more reliable and faster;
• By providing probe traffic on the network we make the one attack that onion routing can’t protect against much more difficult: traffic analysis.

Technical Details

The goal of this (not so) short post is to show how even simple plugins that relies on very few moving parts can add a lot of value, allowing users to build their own utilities and instrument their own node. Plugins for c-lightning can be written in any language that can read JSON messages from stdin and write to stdout, so pretty much any language will do—I’m sure someone will figure out how to write a plugin in whitespace eventually.

For this post we will be using a small library called pylightning which provides a number of python wrappers that make writing a plugin very easy. It is heavily inspired by flask, and if you are familiar with the decorator-based approach to building web apps, you’ll feel right at home.

To get started we create the scaffolding to get the plugin up and running, and register a passthrough JSON-RPC call:

probe.py

GitHub Gist: instantly share code, notes, and snippets.

Gist262588213843476

There are a few things happening here:

1. We use the first line to tell the operating system to use python3 how to interpret and execute the file. This is necessary since c-lightning doesn’t really care about what language the plugin is written in, it’ll just take the executable and ask the OS to run it.
2. We import the plugin helper and create an instance.
3. We then use the @plugin.method() to register a method that we’ll expose over the JSON-RPC.
4. Finally we start the plugin event loop, which will accept incoming calls, forward logs to c-lightning and handle all of the other interactions.

After making the plugin executable (chmod +x probe.py) we can start c-lightning and have it run the plugin alongside:

cli.sh

GitHub Gist: instantly share code, notes, and snippets.

Gist262588213843476

By calling lightning-cli help we can verify that the plugin has registered the JSON-RPC method successfully:

cli.sh

GitHub Gist: instantly share code, notes, and snippets.

Gist262588213843476

Finally we can call the JSON-RPC method as well:

cli.sh

GitHub Gist: instantly share code, notes, and snippets.

Gist262588213843476

And we can also see that the print-statement above was logged in the lightningd logs printed onto the console as well:

logging.log

GitHub Gist: instantly share code, notes, and snippets.

Gist262588213843476

This is a solid foundation on which we can expand.

Probing the Network

Now that we have the basic scaffolding let’s implement the actual probing algorithm. What we’re going to do is very simple:

1. When the probe method gets called in the JSON-RPC, c-lightning will forward that call to the plugin;
2. We will then search for a route to the node in the network with the node_id we were told to probe;
3. We also generate a random payment_hash to send, along with the prove payment, to fail even though it may reach the destination;
4. We then initiate the payment attempt with these parameters and wait for it to complete;
5. Upon receiving a failure, which we expect since the payment must fail with a random payment_hash or we’d be in deep trouble cryptography-wise, we report that error back, along with some more context;
6. Translating this into code results in the following:

probe.py

GitHub Gist: instantly share code, notes, and snippets.

Gist262588213843476

There are really 3 outcomes when we probe:

1. We fail to find a route to the node with node_id. When randomly selecting a node this can happen quite often since many nodes are not as stable as they could be, but nodes that are backing a shop or service provider tend to be much more stable.
2. We find a route, however somewhere along the path a channel is currently unavailable. In this case the logic implemented in the pay plugin would update the network view, and try again until we succeed. The probe however just reports the result (there is interesting information in the failure after all).
3. We actually manage to reach the destination, indicated by a failure code 16399 which the specification defines as UNKNOWN_PAYMENT_DETAILS. This is exactly what we’d like to see :-)

And this is what a successful probe looks like in all its details:

probe.json

GitHub Gist: instantly share code, notes, and snippets.

Gist262588213843476

Investigating Stuck Payments

You might have heard that Lightning payments are fast, and we can see how fast by probing the network. For example the probe result above had a roundtrip time of just 2.56 seconds through 4 nodes and 3 channels. A lesser known fact is that they are very fast, unless they’re not. A payment may end up stuck if one of the nodes stops responding at the wrong time.

If a node accepts an HTLC but then stops responding, then we cannot determine whether the payment was forwarded to the next hop, or even reached the destination, or whether the payment was stopped short. In the former case the payment may end up eventually succeeding, while in the latter case the payment will not complete. It is this uncertainty that prevents us from just retrying the payment, and let the stuck payment eventually time out.

Furthermore, we don’t even get an error message back, meaning we can’t identify the node that failed. You might have guessed it, this is where the probe plugin can help: given a route that resulted in a stuck payment we can shorten that route and try if the prefix succeeds, fails, or also gets stuck. If you are familiar with networking tools you might recognize this as the traceroute utility, which allows similar inspection for IP based communication.

And just a few more lines of code will implement this:

traceroute.py

GitHub Gist: instantly share code, notes, and snippets.

Gist262588213843476

We should probably rearrange some of the code in order to reduce verbosity and better reuse some of the parts, but this listing is just to illustrate how easy it is to implement some really advanced logic with very little code.

Get Started

Here’s the quick-start install for c-lightning 0.7.2 and above, which downloads from our plugins GitHub repo, links it into the plugins subdirectory, then tells c-lightning to look for it:

getting-started.sh

GitHub Gist: instantly share code, notes, and snippets.

Gist262588213843476

The Probe plugin and many more live on Github in the lightningd repository, so why not give them a go, and if inspiration struck, feel free to let us know :-)

And for a comprehensive question and answers session on c-lighting development, check out my latest webinar on Youtube.

This was the first showcase in a series on c-lightning plugins. For a regularly updated list of available articles, head to our introduction to c-lightning plugins.

Note: This blog was originally posted at https://medium.com/p/bb92b7fd8cc3