Slides: https://lightningresidency.com/assets/presentations/Bosworth_Building_Applications.pdf

https://twitter.com/kanzure/status/1116327230135836673

Today I’m going to talk about building on lnd from the perspective of we want to have developers building on lnd. Developers, developers, developers. I just have so many slides today that let’s leave questions to the end if I have any time because I over made slides.

So what is the big idea behind lnd? What are we thinking about when we’re developing lnd and what do we want lnd to be? We want it to be something that works end-to-end. It’s a wallet. With Bitcoin Core you can run in no-wallet mode and a lot of people do. You can use a ssh tunnel and you can just have Bitcoin Core tell you what the consensus is. But with lnd we don’t want it to just do the protocol, we want it to really be actively helping you do everything involved with Lightning end-to-end. You as the app developer can just pay and receive and you don’t have to think about other things if you don’t really want to. We also want to develop lnd to be cutting edge in terms of we want the community to be using it. We want to be releasing it fast so people can hit problems quickly and help us figure them out what are the important problems and what are problems that we didn’t anticipate. We really want it to be community driven. We also want to soup to nuts APIs here. Not only do we want to do lnd APIs like Lightning APIs, we also want to do chain APIs. If you want to know information about the chain, we want lnd to be the point that you use. So a very comprehensive solution.

So let’s say you want to start building on lnd. What would be the best practices of how you’d get started building on lnd? I’d say right at the moment the best way to build on lnd is using the gRPC API. gRPC is an upgraded version of RPC. It works super fast, it’s small data, streaming. The downside of gRPC is that it has less support on maximal platforms because it is a little bit complicated. It is not quite as easy as sending a REST request though we do have a REST API. What we’re working on though in addition to this gRPC API is the ability to embed lnd right into your app so you’ll be able to create a mobile app. Right in the background there you can be running lnd. That’s something we’re working on ourselves with our desktop app that we would just run lnd in the background and that would be able to power the app functionality. For mobile that is going to require some more fixes to how the database works, to make sure it is conserving battery and not abusing the storage but we’re working on that. In the future we would want to make lnd usable as a library, as a very high level protocol way to really customize your own apps on Lightning. If you have your own specific use case, just like with Bitcoin if you’re building a huge application for Bitcoin you should actually not use Bitcoin Core. You should start to implement your own features, your own UTXO tracking, your own transaction building. In that case you want more of a library system. With lnd we also in the future want to move towards that.

One super cool thing about lnd is that it has independent backends. You can use Bitcoin Core as a backend, you can use btcd as a backend. In development, we’re working on a light backend, Neutrino. I would recommend to everybody using Bitcoin Core in production. I’ve tested out all the different types of backends and Bitcoin Core has the most community behind it, it is the most mature and it’s a great solution. I run all my stuff on Bitcoin Core. But I also like btcd because btcd is great if you’re running tests. With Bitcoin Core, you don’t really want to go in and recompile it all the time. With btcd if there’s something weird is going on you can go into the source code and change a little thing. You can see what’s going on. So btcd is great, I run all my tests on btcd and I run ally production on Bitcoin Core, They have similar APIs. In the works, very actively we’re working on Neutrino. Neutrino is our solution for how it is going to run on people’s mobile phones. We’re taking a look at all the problems of SPV, even incentive problems. Why would a Bitcoin Core node want to give you an answer to a query about your addresses? You’re not paying it, why should it give it to you? We’re working on from soup to nuts, privacy, security, incentives, all of that with Neutrino. The big challenge right now is to cover all the corner cases that can occur from a security point of view and also to optimize the crap out of Neutrino so that when you load up your Neutrino wallet you can be up and running in minutes. We have that in testing. On mainnet you can be ready to go in minutes with Neutrino. No more super long syncs. In the future, Neutrino could even be run on the server side. If you have your Bitcoin Core node somewhere and we could actually give you an authenticated connection so you could double check it. Even if somebody compromised your Bitcoin Core node, God forbid, they would still have an extra hurdle to overcome which is they would also have to fool your Neutrino checks which are pretty lightweight. Pruned mode is also coming. Pruned mode will work on lnd out of the box right now. It will start to degrade in certain scenarios. If you wanted to do a restore from seed it would have problems because it would say “Where are the blocks?”. If you get channel announcements that are from the past that are from before your pruning horizon, it will say “I don’t know what that channel is” because it is not aggressively thinking about transactions as it gets them.

I’m going to go into the basic features of lnd. When you set up a wallet you’ll see this new seed format that we’ve created. It is kind of like a regular seed format, BIP 39 that everybody is used to and everybody uses pretty much apart from Electrum. We’ve customized it based on the feedback that people had including the reasons Electrum didn’t adopt BIP 39. We’ve added this birthday field to it so that we know when you’ve created the seed. We’ve added a versioning system so that we can upgrade the seed and make it include more information so we know what we’re dealing with when we try to restore from seed. And we’ve added KDFs so you can encrypt your seed in a more secure way. Actually, lnd is two wallets. If you set out to make a comprehensive wallet for Lightning, you have the job of creating two wallets. You need to create a Lightning wallet and you need to create a good chain wallet. I would say there are no good chain wallets out there. You have a huge challenge ahead of you. You have to make a good chain wallet and you have to make a good Lightning wallet. lnd is trying to do that. We have both modes. lnd will try to expose to you all the different stages that HTLCs are going through on the chain. In certain situations Lightning transactions don’t work. We have a dispute between our peers and in that case we have to go to the chain. In some cases those are multi-stage processes with timelocks. And so lnd will try to explain to you what’s going on but that’s a work in progress.

Now you’ve got your wallet set up, you’re ready for the chain. The next step is to make your channels. The big challenge that you’ll see here when you run lnd is “I want to sell my service but how do people pay me? I have no channels.” You’re going to have to figure out how to solve that problem. That’s something we’re working on - ideas for how to improve that. That will be your first challenge as a merchant or as an app developer. One thing I would suggest when you’re making channels is to limit the min channel size. lnd has this feature where it says “what’s the minimum somebody can commit to when they’re making a channel with me?”. We don’t want the network to be everybody makes a tiny little channel. I don’t want every single user of Y’alls to create one channel with me and then slowly pay me 10 cents or 25 cents over the course of a month. I want to have just a few big channels and that’s the mode we’re thinking with lnd. Everybody makes a limited number of bigger channels that they use to pay other people through the network. That’s the greater idea of Lightning anyway. When you’re sending, that’s a lot easier than getting receiving capacity. With sending, you still need to think about it. With lnd there are a bunch of APIs you can use to look at the graph to say “who should I be opening channels with?”. We have this autopilot but I would not recommend using it in production. It is like a stub feature. Instead at the moment I would use the graph features to inspect what are the good nodes, what are the good channels. You can see a lot of information on the graph. Different peers will send out heartbeats. They’ll say “every week I will announce I’m still around”. Different peers will disable channels to other peers. If you do some easy analysis on the graph you’ll be able to see what are some stable nodes and some good nodes to make channels to. The future is we have this autopilot stub, that’s what we want to have working. We want the user experience to be that you can load up this autopilot, it will figure out from the graph what are good channels for you, it will make them for you and you won’t have to think about that.

When you’re receiving funds we have lots of ways to receive funds, chain funds, invoices. One thing I want to point out is what’s the nomenclature. This is something that confused me. When is something an invoice and when is something a payment request? The way I think about it is the payment request is what I talked about yesterday, the BOLT 11 format. That is a specific request where it includes routing hints or the description of the payment. An invoice is my local version of the payment request. The payment request is generated from the invoice. The invoice includes things you wouldn’t show in the payment request like the preimage. So lnd supports all the normal features of BOLT 11. You can set an expiration date, you can set a fallback to chain address though that is in progress, our support for that. You can set a description, you can expose hop hints to the world. Then lnd has some special cool features on invoices that are extra. We have an internal description field that you’re not going to reveal to everybody. If you want to keep some internal state about this payment you can put that in a special receipt field. We allow you to push in your own preimages. If you don’t want lnd to make the preimage for you you can create your own preimage. We’re building more features on top of custom preimages and more preimage control that will allow you to do cooler apps.

So when you’re processing receives what should you do with lnd and what can lnd let you do? The best practices use the invoice subscription APIs. Those are going to be extremely… APIs that are pushing announcements to you, “hey you’re being paid.” I would listen to those in two different ways. If you’re getting paid you get this subscription ping. The number one thing that I would suggest is that every time you get an invoice ping you go into the process payments stages. You’re like “OK I know about this invoice now, I can go and process that payment. I can figure out how to deal with that payment.” The second way I would deal with streaming payments is I would tell the users about that. A lot of times you’ll see these applications that are using Lightning where you pay and then you have to wait a little while and then you see it worked. But actually if you create a very narrow gap between the invoices subscription and the user on their interface, a lot of the time you’ll be able to show the user that they got their payment accepted before they even see that it was accepted because you have a direct connection a lot of times to the user. So you can say “payment received” even while the payment is still routing itself back to the user. So that makes it super fast, the experience very crisp. As far as processing payments I would not recommend using…. lnd is not made to be a total payment processing system. If you’re doing anything that has multiple stages like a delivery I would use an external database and move the payment through different stages. The person paid me so I would get that invoice ping. Then I would deal with that payment. I would set the payment in an external database to be processing. Then I would do the thing that they paid me to do. Once that was finished I would go back to the record and say “They successfully paid me and I successfully did the thing.” You would have the full invoice lifecycle from you got it to you sent it. Just like I said before about that user experience where you get the instant payment. One way to think about that also is up until you receive this HTLC locked to you, you never want to show anybody that pre-image. You want to make sure you’re making a good random pre-image or let lnd make the pre-image for you. Once you get the HTLC locked to you, you want to spray that everywhere. You don’t care about that pre-image and you want to get that pre-image to the user as fast as possible because that lets them know you paid. That’s how you should think about pre-images. Guard them until you’ve been paid and then be super liberal with them.

So sending payments, how you should approach sending payments with lnd? I would say the ideal practice would be you have this channel graph inside of lnd. It is keeping track of the overall network. Before you even make a payment you can ask your channel graph with a query “can I send here?”. You don’t even need to try the payment first. You can just say “will the payment even work?”. That’s not going out to the network, that’s just looking at your own database. Is there a route possible? Lnd supports sending to routes so you can choose your own path through the network. Maybe you have your own idea of how routing should be implemented. Maybe you want to go on a sneaky path for improved privacy. Or you want to go on a path where you want to reward your friends with extra fees and you’re going to pass along a little extra routing fees. The downside to watch out for if you use send to routes is that lnd will be learning from the payment process as it is paying. As it is trying to pay on different routes it will be feeding information and its learning about failures back into its attempt to pay. If you implement your own send to routes for payments you would also be responsible for feeding information back to yourself. Currently lnd doesn’t persist information about what are good routes any longer than 30 seconds. In the future you would be missing out on that feedback mechanism where we would be adding to a database like “This guy is fast and good at sending payments. This other path is not the best at sending it so let’s always try this guy.” You might actually think that you could do that better than we could so you could do that yourself. Lnd is a chain wallet as well so you can send out onchain. We support SegWit. We’re working on things like splicing out so you don’t have to interrupt your channels. If you ever want to send onchain you can keep all your funds in your channels. The only downside is if your peer goes offline and then you won’t be able to send instantly. Otherwise you will be able to send instantly and not interrupt your channels. You’d be able to be sending on channels, still forwarding, doing whatever you want and also be splicing out to somebody and there be no interruption. That’s something we’re working on. Another thing is you can do submarine swaps. Let’s say you want to send to a different chain. I want to send on Lightning offchain and I want to have the send be resolved on a different chain. It is a project I worked on before called submarine swaps. That allows you to tie those payments together with a HTLC so you can pay on Bitcoin offchain and then you can use a swap provider, who you don’t have to trust and they will resolve it on a different chain for you.

The balances in lnd are pretty complicated. In Bitcoin you kind of have two balances. You have your firm balance and your pending balance. In most wallets they’ll show you the normal balance and then they’ll have in parenthesis that you’re getting this probably if it does confirm. In lnd you have all sorts of balances. You have sweeping HTLCs and it can be a multistage process where you technically control it but you still have to wait until you can actually send it around. If you look at lnd APIs you’ll see that there’s all these different types of balances. We have a timelock balance, we have a balance in limbo, we have balances that we’re not sure how they’re going to happen. We’ve been talking about if you send a HTLC in the absolute worst case, it could be stuck for a week or so. If it’s stuck it doesn’t mean that it’s going to be refund. It could also be that it sends. That’s a weird type of balance, it’s pending now but maybe it’s going to come back. Balances are more tricky to think about in lnd. Channel balances are reflected in lnd and maybe we can do better there. We will adjust the balance based on the commitment fee. Your balance will actually be fluctuating all the time in lnd. It’s not like a Bitcoin wallet where the UTXO costs are hidden from you. In a channel wallet you’ll see the UTXO cost because the amount that you’ll be able to send will go up and down with fees. As fees go up, the amount that you’ll have to assign to the channel that will be unspendable, that will go up too so your effective balance will go down. It is the same in a chain wallet but you don’t see it. We’ll expose that to you. The only thing we’re not exposing to you right now is the reserve capacity. People talk about you need to reserve 1% for game theory of breaching.

We have a bunch of graph APIs. The thing with the graph API is that you see everybody else. The lnd philosophy about the graph is that we want the graph to be people who are signed up for routing. The rest of the people we want to be private channels so they would be off the graph. The graph data model is you have the nodes, so we’ll show you all the nodes. Then you have the channels and we’ll show you all the channels. Channels have two parts to them. The two parts are the policies. I have my policy and my policy means I’m willing to send to you but here is my baseline fee, you’ve got to send me 1 satoshi to send through this channel. The other guy will have an opposite policy. He’ll want 1 satoshi to send to this guy. There’s other things like the minimum HTLC that is allowed, the percentage fee that is allowed. We’re thinking about adding a max HTLC size, the maximum HTLC that can go through. The network has also added this idea of disabling. I’m not going to allow any sends to go that way. In the future we want the people who are public channels to only be people who are going to route. Otherwise you don’t need to tell me about the public channel. You can keep those private. If you’re an app developer, eventually we want to say “you just keep your channels private unless you’re dealing with routing stuff and that’s your deal with whoever you’ve selected as your routing nodes.” Technically a private channel can still route. If somebody knows about that channel and they make a request to you, private doesn’t mean you can’t route, it just means that you’re not advertising “Please route through me.”

As far as wallet security goes, you’ve set up your app, you’ve connected to your channels, it’s probably a good idea to do a security audit for all the things that can go wrong. In lnd we’ve been thinking about how to design this to be a secure hot wallet as best we can. One thing we have that people complain about but we believe in is that when the wallet restarts it locks. When you start up the wallet you enter a decryption key and then it goes and reads the wallet file off the disk and decrypts it using the key that you entered. Then it goes about its business and does its wallet stuff. People get upset because their server restarted or something and their wallet is now locked because it didn’t keep that decrypted file on the disk. That’s by design and we have a API that you can implement by yourself unlocking if you really want to. Our recommendation is don’t leave that decrypted wallet data on the disk for somebody to find if they get access to your machine. We also have another security model for the hot wallet which is this idea called macaroons. Macaroons kind of scope your access. Once you have your hot wallet that’s running on the machine, how does it know what to do? How does it know who to send to? We have this macaroon that people use called the admin macaroon that can do everything. When you message the hot wallet you can say “Send all your money to hacker” or whatever. We also have other macaroons. We have another macaroon which is an invoice macaroon. When the hot wallet receives a message using the invoice macaroon, the only thing it can do is make a new invoice. You can only receive money. That can be pretty cool for apps if you want to put that macaroon file on a machine that you don’t want to really worry about something going wrong because the worst that can happen is they can make invoices on your behalf. That is something we’re going to drill into. We’ll make macaroons that say “You can only send a certain amount per day. You can only send to these keys.” Macaroons are a very simple format actually, it’s just a signed attestation. The idea would be that you start up your node, it will read all the wallet data into memory. Then it will be checking the signature, did I say you would be allowed to use this? OK I did. You could then remove all the files that would allow you to arbitrarily move funds. You would be able to remove them, it would fail shut and you wouldn’t be able to remotely control it. The other thing to do in terms of security for your wallet is that you should not go offline for long periods of time. If you’re making smaller channels you should probably not go offline for longer than one day. If you’re making very large channels, lnd will automatically expand the amount of time that you have to contest if somebody is trying to cheat you. It will expand it to like two weeks. You should still plan to keep your nodes online, don’t let them go offline for weeks. The other thing I would suggest is scope how much you need. Don’t keep too much on the machine. If you don’t need to be sending a bunch through those channels you have mechanisms that you can push your channel capacity out to other people or back into your cold wallet, I would try to do that. I would think about how can I reduce the amount that I have exposed in this hot wallet.

Another thing that comes up a lot with running lnd is what if my computer crashes or my database gets corrupted or my hard drive dies? The story there is not the best that it could be right now. We don’t have a formal backup system at the moment. The best that you can do in production would be to use RAID to distribute at the file level your database across different disks. An easier step than that would be to copy the channel database to a different volume or different place. We’re very actively working on a thing called static backups which is the last snapshot of your channel and that’s something in the protocol already. That’s something that Eclair has implemented. That will give you the confidence that if the worst goes wrong, your channel database is totally corrupted or you just lost it or something, as long as you have the small files that tell you your last state with everybody, you’ll be able to close all your channels and get your money back in a reasonable way. That’s our first step that we’re working on at the moment. The final future that we’re thinking for how will you run this at a huge scale is that we would support a distributed database backend. We currently have more of a SQLite type of database. Something comes in and we record something in the database. It is fine if you’re running on a mobile phone or a small app but if you’re running a huge exchange with billions of dollars we’re working towards more of a distributed system. We approach that in the same way that we approach the Bitcoin Core, Neutrino, btcd backends. We have a clearly defined interface of how we need to persist all this data. Currently we run boltdb and we would create a new interface that will go to a database that would do more consistency. The tradeoff would probably be that it would be slower because you would have to be sending it to multiple places.

Lnd supports Litecoin so we don’t just Bitcoin, we also support Litecoin. The support is not all it could be because we need more contributors from Litecoin. We’re busy, we need to focus on just getting Lightning running. It’s also pretty straightforward to do swaps with altcoins. That’s a project that I worked on before I joined Lightning Labs, the idea of submarine swaps. It doesn’t require a very complicated script, a complicated protocol to actually arrange for a low trust swap. A lot of work goes over to lnd. Everything that I did when I was working on swaps I built with lnd because lnd is keeping track of all the edge cases and problems that can happen. Lnd is a work in progress where it is trying to solve the problems and a lot of problems that you’d face when you’re doing swaps. One example is people maybe didn’t think about the condition where I’m doing a swap with somebody and the pre-image is actually larger than Bitcoin can allow for pre-images. Lightning is a vetted protocol so every contract that is made on Lightning checks the pre-image to make sure that it is 32 bytes. In the swap case you can just forget about that because you know that the bounds of that pre-image are definitely going to be 32 bytes and that’s guaranteed by the network. In the future of alts I really believe that we will just not need to care about alts as developers. That will be another thing that the routing network takes care of. We will just say we want to receive money, this is money that we care about and we’ll be able to have the routing network take care of swapping between different things in a way that we don’t have to trust those people to do it. I think that will make the app experience much better. I talk to people that are building apps, building services and what’s a big feature that people are requesting? They’re requesting can you support my altcoin, can you support my coin? It is compelling because they’re going to spend some money. In the future you can outsource that. You can say “Yeah I support it” but actually you don’t support it, somebody else is doing the swap providing. Eventually even the network will just do it and you’ll be able to take whatever you want.

My final point is when you’re making these apps, how should you be doing it? The number one thing I think to emphasize is please don’t make a closed source Bitcoin wallet. People have this idea that they can make a closed source Bitcoin wallet and people should then trust it. You should trust your money to a wallet that you don’t know what it’s doing. I think that’s ridiculous. I really would like people who are building on lnd…. It is one thing if you make a backend service, a backend app or something. If you give people an app that they’re going to use on their phone or on their computer you should definitely make it open source. Not a weird open source like you can download this compressed file or whatever. A real licensed open source wallet. We also have our own wallet project that is separate from lnd which is an app in React Native that is designed to run on different platforms. You can take a look at it. If you want to build on lnd you can take a look at our Lightning desktop app to see how it interacts with lnd as an embedded component. That is GPL, that’s kind of like our attempt to say “That’s something that we’re building as an end user system.” We’re thinking about iterating on that. That’s not the platform, the platform that we want you to build on is lnd. The other thing that I’d like to say beyond closed source wallets is that when you’re developing your apps, especially if you’re developing a wallet app, you should be a good citizen on the network. Lightning protocol development has gone to a lot of trouble to make sure that this is private and this is delivering lower fees and it’s delivering micro transactions and all these cool features. You as a developer don’t have to do all these things. Instead of being private you could just program your wallet to send all the information to your server. Maybe you want to see all the things that people are doing, you might like that. Or maybe you want to not have it be trustless. You want the wallet to do a decrypted backup back to your server so that if the user forgets their password they can just email you and you’ll just email them back a decrypted version or something. I would say try to avoid that, keep that in mind. We’re trying to work towards this thing and we’re working on this stuff with the intent that people are going to use it properly. We don’t want to be building software for people who are going to be breaking the rules, breaking the stuff that we’re so carefully trying to build: privacy, security, that kind of stuff. That’s the thing I’d keep in mind. I understand there’s some flexibility but that’s what I would go for. Trying to be a good citizen.

Q - You mentioned before that with splicing out you basically have to rely on the fact that the other node is online. Why not make the commitment transaction in a way that they are always splicing out….?

A - You could reserve a certain portion? Once you start to close off funds and you have a different way out you start to lose the protections of the commitment transaction. One thing I’ve thought about is making different versions of the commitment transaction but the other pushback would be that it’s added complexity. We’re having to keep track of different states. That’s maybe something we can think about in future as an option but I wouldn’t look for it right away.

Q - You said there’s two wallets, the chain wallet and the channel…? Do you attempt to be compatible with other types of clients like c-lightning….?

A - There’s lots of things. The revocation basepoint… there’s a bunch of secrets. There’s some derivation thing but there’s still state that you have to keep around or things that you got from the peer. On the other question, are we doing stuff to be compatible with other wallets? We’re trying to push the boundaries and in that way we can’t be compatible. We took BIP39 and we’re trying to make an improved version of it that we hope could become a standard and we would push it out as a standard. There’s going to be debate, should we have a longer string or a shorter string, is a birthday really needed, is versioning really needed? So far I think we haven’t really been keeping with a standard where you can take your wallet and plug it in somewhere else. I think the practice that people use right now of taking a seed from one wallet to another wallet is actually a bad idea because wallets have different subtle ways of working. If the spec isn’t followed in exactly the same way between the two different clients you could lose funds. I wouldn’t say that’s a near term goal but it sounds cool.

Q - …

A - The advertising layer is an optional layer. For example, you don’t even really need the channel IDs. Right now you can see all the channel IDs but all you really need is the pubkey graph. You could say “Mr pubkey A can you send to Mr pubkey B?” He could just figure that out himself. All you’re really doing is trading a secret. I think they’ll be different modes for how routing works other than just the existing channel graph. This is the baseline that we have where we have a set of people who have said that they’re willing to route and they’re willing to give us some information about how they’re willing to route with base fees and percentages. As a global system we can add in different ways to do that. The existing system isn’t made to be totally… of doing swaps because you advertise which chain you have. You can connect to different people and negotiate like “I support this chain” and “I support this chain.” I think the tricky part to doing those swaps is you need to coordinate timelocks across different times. One chain is going faster so each timelock needs to be from a different point than the other time. The rates have to be different so what rate are we using? In Bitcoin to Bitcoin it is pretty easy, you just do a percentage. But in altcoin to Bitcoin you’re going to be like “I’m going to use this rate.” It is not crazy different, we could accomplish it if people wanted to do it.

Q - You recommend people use gRPC… There are some clients that don’t support gRPC. Do you plan to keep supporting the HTTP one….?

A - I wouldn’t say we’re going to deprecate the REST API. I’m just saying how does a REST API come about? It is derived from the gRPC. We focus more on using gRPC internally, lncli actually uses gRPC. That’s going to have the best support always because we’re going to run into it and we’re going to building for that. Before I joined Lightning Labs I had this problem of gRPC is not well supported because I made a Mac desktop wallet. Then I was like how does gRPC integrate with Swift? gRPC is in development so it doesn’t work too well with Swift. I made a little shin server that talked to gRPC and then fed out my own REST results in an easy to parse format. Overall our vision for how you should talk to the daemon is gRPC has great features, it is super fast, it does streaming, it’s authenticated. That’s where we want to go. We’re hoping the ecosystem falls along with us.