Subscribe to RCast on iTunes | Google Play | Stitcher
Chris Boscolo, Kelly Foster, and Greg Meredith discuss the underlying architecture of RChain and what’s in store as main net approaches.
Transcript
Greg: Today’s RCast is “All Things Node.” I’m going to let Chris and Kelly drive because as project managers and product owners they really are the go-to people for “All Things Node.” Kelly, I was hoping you guys could talk a little bit about where the node is and give a walkthrough for anyone who is interested in launching a node and what they might expect when they launch a node.
Kelly: Sure. From a technical and delivery of the node component standpoint, the node being the primary part of the access to the RChain network and platform: at this point in development—we are in the pre-release phase working towards our Mercury release—the component of node is largely done. There are some features: adding improvements for users that are on the current backlog and planned prior to being code-complete for Mercury, but largely the node team is having the opportunity to work on optimizations, which is a really nice place to be in a project as large and complicated as this.
We released RNode 0.9.5 this week. With that came some of those user feature improvements to node as well as some of the optimization work. That’s what will continue to be in the releases of nodes leading up to being complete for Mercury.
We’re really glad and like the place where node is. We’re also taking a lot of cues from users of RNode in terms of making those prioritization decisions related to optimization. We have an excellent group in the community that participates weekly in our node testing sessions. Those happen Thursdays at 1400 UTC.
We appreciate the opportunity not only to watch them use node, but to hear them talk about things that would make it better, easier, nicer, or the ways that they’re trying to use node to either build dApps as developers or validation on the platform. The voice of the user is a really important part of our, I guess, lesson learning for the Dev team as we think about how to continue to iterate on node and make it the best it can be for the Mercury release.
Greg: Chris, did you have anything to add?
Chris: A lot to add, Greg. What I was hoping to share was to give people an overview, first of all, how RChain works, and then talk about nodes and their role in how the RChain blockchain works. I was wondering if I could turn the tables on you, Greg, and ask you a question: to give an overview of Rholang, the tuple space, and produce consumes.
Then I want to fill the rest of the story with what a validator node does as part of the RChain network (to basically produce the blockchain) and what the observer nodes do to be a part of the blockchain. Greg, can you give a brief overview of Rholang, the tuple space, how code gets deployed, and how the system grows?
Greg: Sure, absolutely. Thanks for the opportunity. Rholang and the tuple space fit together hand in glove. Rholang is the user-facing manifestation of the execution mechanisms. It’s the language in which programmers, developers, smart contract writers, and other living creatures express their intentions or their desires with respect to computation. As we’ve expressed many, many times, it’s built up from the Rho calculus, which is a bare-bones, minimal model of concurrent execution that’s a variant or an elaboration of the Pi calculus. The essential features are waiting for input at a channel and producing input to a channel.
If Rholang is the language, then you need some mechanism that’s going to interpret that language and effect the instructions that are being given through that language. The tuple space is the mechanism. You can think of the tuple space in a lot of different ways. The origins of the tuple space go all the way back to David Gelernter’s idea of a tuple space as a coordination mechanism for concurrent execution. Essentially, you have addresses and you can put data at those addresses or you can get data from those addresses. That was the original idea of the tuple space as contemplated by David Gelernter. Gelernter’s ideas caught on quite a bit in the late eighties and early nineties. There were many, many, many implementations of tuple space-like gadgets in Java and JVM realizations.
I came along and suggested that there’s a variant of the tuple space idea that is a little cleaner and gives rise to a much broader range of designs and also gives rise to some interesting optimizations. The basic variation is that at a particular address you can either put data or you can put a continuation. If you imagine that you have a Rholang program that wants to consume some data from a particular address. That would be expressed in the Rholang form: for Y from X do P—that’s the basic form. Y is where you’re expecting the data to be deposited if you get it out of the address X. In the tuple space, if you go and look up whatever X points to as an address, if there’s some data already sitting there, and the most bare-bones implementation that will be taken out of that address no longer available to other consumers, and it will be bound as a value to the variable Y for reference in the program or process P. That’s the plain vanilla execution.
But if it turns out that there isn’t any data at that address, instead what you’ll do is take the continuation that’s waiting for a Y and then we’ll run P and put that in the address X. That’s the other half that you can take out of this.
There are two other paths that we can look at. We’ve just looked at two. One is looking for data and then what do you do if there’s data there and what if you do if there isn’t data there. The counterpart to that is producing data. If four is a notion of consumption than the sending data on a channel construct in Rholang—so X Bang some data value like Q—also has two possible execution paths. If there isn’t any data at X when you’re trying to put Q in the address X, then you’ll just deposit Q into that address.
If, on the other hand, there is a continuation waiting there, then you will pull that continuation out of the tuple space and supply the data Q to it. If that continuation were there because we had done a for comprehension, then Q would be bound as the value to the variable Y when you’re running P. That’s the basic idea.
For a blockchain application, we elaborate this to make sure that the steps of production and consumption are gated by a resource. In this particular case, the resource is called Phlogiston (or flow). You can only take the next step if there’s some flow available to allow you to take that step to either deposit the data or consume the data. That’s the basic operations and how Rholang and tuple space fit together. Does that answer your question, Chris?
Chris: Yes. That tuple space is RChain. The tuple space is the blockchain. If you want to send or consume— that is, deploy Rholang that sends and consumes—you would construct your Rholang and you would want it to get sent out to all of the nodes so that the tuple space for all of the nodes would get updated with that new deploy that just happened.
It’s important for everybody to understand that each node has a copy on that tuple space that is up to date with everybody else’s node, depending on which blocks they’ve seen thus far. There are two types of nodes. There are what we call validator nodes and there are what we call observer nodes.
Observer nodes are just nodes that receive the blocks and update their tuple space, so they always have the most up-to-date picture of the blockchain. Validators, on the other hand, are the nodes that can actually propose updates to the tuple space. That happens when a user does a deploy. If a user wants to deploy some Rholang, they will construct a deploy and they’ll sign it. That flow that you just talked about, Greg, is bound to an address stored in a REV vault. The signature of that deploy is done mapping to that same REV vault. That’s where the payment comes from when somebody deploys new Rholang to RChain.
I thought it’d be really good to start there so people understand that if you want to run a node, even an observer node, you have to have hardware that’s sufficiently large enough to be able to receive all the blocks and update the tuple space. Depending on what’s going on on the network, at enough speed and compute capacity to keep up, start there, and then maybe have some questions we want to drill in on.
Greg: That was really brilliant. I would want to point out that there are some nuanced distinctions that, as a software architect and computer scientist, I would want to make. In particular, the tuple space is a general purpose store; you don’t actually have to use it for a blockchain. You can take the Rholang tuple space thing and use it in the same way that many enterprises use SAP HANA. In fact, this is an architecture that I have rolled out at the enterprise level in many cases in commercial settings.
What’s nice about Rholang as an update to that architecture is that it provides a very detailed query language. Because of its pattern matching capabilities, it becomes a query language against the form of the data that’s in the tuple space store. On top of the tuple space store, then we build a tri of blocks that are associated with the specific format that is supporting the blockchain operations. That’s an added functionality that refines the architecture to fit specifically for the blockchain setup.
Chris: That’s a good distinction. And from an outsider, if I could just see the network without having to run a node or anything like that, I would see all of these blocks which actually contain Rholang on them being sent amongst all the nodes. That’s from the blockchain perspective, what you see on the wire is deployed Rholang. That’s a good point to bring out. The tuple space that’s being used on the nodes themselves is an implementation. There could be potentially other implementations so long as they adhere to the protocol.
Greg: Right. And you can also use the tuple space to implement other things. One of the contemplated uses of node, and it has been a contemplated use of node before I ever even heard of blockchain, is as middleware. It’s a particularly effective workflow engine and middleware coordination architecture. So it’s not just about blockchain, but it turns out that blockchain is a great application for this kind of technology.
This lines up with how we’re thinking about the market. We really believe that the node architecture will be great for the management of digital assets as the first set of applications that you can go after. We also think it’ll be quite good in the enterprise setting for a lot of the kind of workflow things that enterprises typically have to do and other kinds of applications have to do. Whatever kind of digital asset or other kinds of asset you’re tracking, these kinds of workflows across a collection of digital assets becomes really important.
If you’re running some reporting and accounting or if you’re triggering other jobs that are related to a condition with respect to the digital assets. For example, in the network telephony, it may be that in the case that when you replace a particular component of the network, the record that keeps track of the component is deployed in the field is updated, that then triggers a service call or things like that.
Those are the kinds of things that the architecture is actually contemplated for. It allows us to fit blockchain kinds of applications directly into that larger framework of applications. That’s why I was mentioning these distinctions.
Chris: Awesome. I don’t know if people care about the nitty-gritty detail, but just to cover some of those things, node software itself is written in Scala, for those people that aren’t involved with this project, which runs in the Java virtual machine. This is really standard enterprise deployed software. Anybody that’s managed other enterprise deployments of Java-based technology running node will be very comfortable. Kelly, we have some other things we were going to talk about in terms of high-level stuff. Rather than hearing my voice the whole time, I’ll send it back to you.
Kelly: Building on what Greg was saying, node serves a variety of purposes. You mentioned, Chris, the work of validation on the blockchain platform and the role of an observer node, and more specifically the role of an observer node as a way to understand what might be happening on the blockchain. But the other part of node is it supports the evaluation and compilation of Rholang. It’s a place where developers can, number one, get to know Rholang, and then number two, use and develop with it.
Greg: Are you guys at a point where you could launch a node and talk about some of the flags that a standard user might put on the command line? Do we have wrappers for those kinds of command line calls?
Kelly: Sure. I’m not sure I’m clear what you mean by “wrapper.”
Greg: A lot of times, if you’ve been in the role of Dev Ops or Sys Admin, you might have this complicated command line with all the flags tweaked for a particular purpose. Then you just put it into a script.
Kelly: Yes, we do have a very complicated run command line. We don’t have an official, “this is the RChain way of wrapping that up and simplifying it.” But I do observe our SRE team doing that, as well as other node operators, depending on what they’re used to doing and how they like to do things, which has been fun to observe. Different people use it differently.
From a command line point, in terms of just flags, if you installed the software and you bring it up and you’ll get help. It’s very clear terms of all the flags that are available to get node running, including information about how you could customize the configuration file. We’re using the Hokan file format for that, which has been fairly popular with people who are using node. There’s also, within that help system, all of the information you would need to deploy Rholang to your node if you’d like to launch a validating node on public test net.
There are some things that are currently changing. We’re changing our key format for node. Currently, it’s using Ed25519, and we’re changing it over to Secp256k1. We’re pleased with this most recent release of node that we have the ability to generate those keys at the command line. That was certainly nice to have an improvement for node users.
As we are in this pre-release phase, from time to time we are advising users to use flags to help support node performance as we continue to work towards better—and in the end, final—solutions for how node runs. From time to time we will publish those in our development Wiki, as well as on the #development discord, to let node users know that for the next few weeks we’re going to be using this flag and to keep node running and to do the things that you’d like to do with it.
Chris: Just to add in on that, Kelly, one of the more popular flags (or command line ways) that a lot of developers end up using a node for is when writing Rholang and they want to test their Rholang locally. They can run RNode eval and then pass in the file which is containing their Rholang code. That’s a very common cycle, especially for new developers that are new to Rholang who are coming along and trying to write their first Hello World program, to see it actually executing on their node software.
RNode itself is built on that, is one piece of software, but it plays a couple of different roles. As we’ve already talked about, it can be a validator or observer, so you can launch it and have it play the role of a validator or you can launch it and have it play the role of an observer, and you can also use that same node software to talk to the validators are running so that you can get your Rholang deployed onto the blockchain through that deploy command.
So the same RNode software is used for all three, both running as a server and operating as a client. In addition to that, the node software has the ability for you to connect to it over the network using either gRPC or, depending on what you want to do, you can use a gRPC connection to send the code you want to deploy to a node that’s already writing or you can collect metrics from the node over HTTP. That fills out some of the different ways that people would end up calling node and using node.
Greg: That’s all really useful and important information. What about logging? Can you guys talk a little bit about logging and how as a user, especially as a dApp developer, I might collect log information?
Kelly: There are three ways that we see people doing it and ways that we as the development team are doing it. The console log is useful. It’s a key way to observe what’s happening. For example, you deploy Rholang to your node, and you can see the logging of that and be able to tell that not only was your deployment successful, but then you can validate that it in fact happens.
There’s some very good immediate feedback via the console. Of course, when you join your node to a network, the console gets fairly loud fairly quickly. We see people using tools like log aggregation to help parse what’s coming through to be able to set filters to look for certain types of events. The node is logging things related to the comm layer; it’s also logging things related to the consensus protocol. You can use a log aggregator to hone in on what you’re interested in.
Then node allows you to pull out that file. We do this a lot when we’re trying to investigate and track down bugs. When users are finding bugs, we immediately ask them to pull their RNode log. It is automatically generated and is very helpful when people can pull that out and share that with us as we work to investigate bugs as they arise. So there’s a variety of ways of getting information out of node.
Greg: That’s pretty awesome. It sounds like it’s going through its paces. When you got well-used software than these basic patterns, being able to evaluate some Rholang and being able to look at some logs, those are things that people who are actually using software need to do. When we see it’s easy to get that information, it’s a good sign that the software is being shaken out well. What about the deployment of a number of nodes? Where are we in terms of being able to launch a number of nodes and get them to connect with each other?
Kelly: Tamash and Adam, who are on our SRE team, have really created nice tools to help support that effort. I can’t talk in great detail in terms of exactly what they’re doing, but at a very high level, they’re using Rundeck to help orchestrate and launch the network. Then we are using also Rundeck to automate. It supports us, for example, running automated testing as well as to jump in from time to time an automated cadence of events—for example, a microburst of deployments and proposals so that we can look in that moment what’s happening with the network. We launched a variety of networks this way. We have a public test and that we orchestrate with Rundeck. It also supports our development networks as well.
The team has also created tooling to support monitoring of networks. This is really important, especially if you’re interested in being a validator, you have some responsibilities to uptime and having your node to be performant always. They’re using Nagios to help with monitoring and alerting and making sure that they can be responsive when nodes are struggling.
That happens from time to time. Nodes aren’t always going to be doing their best. There are certain thresholds that you can set: this node is struggling, it’s likely to fail, and they can quickly jump in and bring it down. Because we have the block store, they don’t have to lose the state of the blockchain. They just can bring it down, then they can restart it, and it will pick up where it left off. The only catch-up involved is whatever time the node was down, they would have to catch up from that point forward, which is really great.
In the early days of running test net, that happened on the regular. As we continue to learn and debug and improve node to be performant over long periods of time, we’re very pleased to say at this point that that’s not the norm anymore. Those are those odd moments where node is struggling, we need to kill it and bring it back. The monitoring software really helps support that.
Greg: Very good. I know from talking to a lot of folks in the exchanges that they do that kind of starting a node software and then shooting it in the head and having it restart all the time inside the major exchanges. This is par for the course. We’re very much on track to be able to slot into an exchange that is already running Ethereum nodes and Omni nodes and Bitcoin nodes. We’re rapidly coming to the point where we’re ceteris paribus in terms of those basic features. Where are we in terms of the testing. We’re in test net two, is that correct?
Kelly: We’re very close to test net 2. We’ve got a bug to fix and a feature to complete and then we’ll be ready to launch test net 2. We have continued to update test net one. We distinguish test nets 1 and 2 In terms of features that are available on those tests nets. While we have been in the test net 1 phase, we have updated the software and the network three times to add other features that have come in but not features that are milestone markers for the launch of test net 2.
That’s going well. Regular users of the RChain public test net have been extraordinarily invaluable, not only in terms of debugging but also to continue to put load on that network and add to watch it work in that blockchain row.
In our development network, we use that to test a variety of scenarios that require a lot of time. Our CI test suite is hefty and it does a lot related to integration tests are things that we know that node needs to be able to do and things that we know, for example, also consensus needs to be able to do. But you really benefit from having the opportunity to quickly bring up the network and let it run for a few days to observe those things over time, and to stress it periodically with little microbursts. We’re using our def network very heavily, and we’re in the process of building out our hardening networks that will also support those really long-running and tests that we intend to be both broad and deep to really understand and stress the platform.
Chris: Just to tack on a piece of fun information: this morning we had the Thursday test net community meeting. Ed Eykholt has been working on his own wallet. On this morning’s call, against test net, he demoed transferring REVs using his wallet software that he’s built, and that was really exciting to see, from end to end transferring REV from one address to another on test net.
Greg: That’s great news. We should do a demo of that on the hangout next week.
Kelly: We did record it today. It’s on the Youtube channel. It’s worth having him demo it again at some point. Kudos to Ed for all the work that he’s put into building this application. I also appreciate the collaboration we’ve had with him because he helps make this platform better too. It was really exciting to see that happen today.
Greg: That’s great news. Are there any points that you guys feel like we should cover in an “All Things Node” podcast? What are some points that might interest our listeners?
Kelly: We know from observing onboarding new users to node that some people opt to run node on their local hardware. In that scenario, one of the biggest hurdles to onboarding is network configuration and net-busting to be able to connect to the network. We do have some documentation about that on the developer Wiki. It’s surmountable. We observe people being able to do that. And that’s what I’ve observed some people having the biggest challenge with when they’re first starting to work with node.
We also know that a lot of users, including our development team, are running on cloud services and that seems to go very well. We do have some people in Asia using node and they have some interesting challenges, sometimes related to being able to hold on to their IP address. The node team has that on their backlog as we continue to investigate that they can stay connected at the moment that their IP address might change.
There is an RNode user guide on the developer Wiki. Everything that you need to know to run node is in fact in the node software. And then by calling for help, you can learn almost everything you would need to know. The user guide for RNode augments that in plain English and provides additional context for node users. I recommend taking a look at that as a way to get started.
There are different ways to run and operate node. We’re about a 50/50 split in terms of people who are running it on, for example, Linux, or who are running in Docker. There is success in both of those scenarios.
One of the things that it’s been interesting to observe is that there’s a lot of different ways that you can install, run, and operate your node and meet you where you are in terms of the technical person that you are. It’s really neat. It makes it accessible to a lot of different types of people.
Greg: That’s really good. Are we at a point now where we can have a bunch of nodes launched across a bunch of different Docker containers and they ended up connecting into a network?
Kelly: I don’t know that we’ve done that and I don’t know that we will be prevented from doing that.
Greg: The only fiddly bits are jumping from Docker container to Docker container, which can be kind of a little bit fiddly.
Chris: I can’t think of any reason why that would not work. With local development, that’s the way I’ve done all mine, so it should be totally possible.
The one other thing is things that we’re going to be adding very soon. We’re going to be adding the ability for users to, as Kelly already mentioned, key jam is one of the features that’s there, but we’re also going to make it so that the private keys for users get stored in an encrypted file and you will use a password to unlock them when you want to launch node commands. Currently, when you launch a validator, you pass that private key on the command line, which obviously is very insecure because if anybody were to see the systems that are running and looked to command lines they’d find all of these private keys. That change is coming after the next sprint.
Any other major features, Kelly? Am I forgetting any of that?
Kelly: That’s the biggest one. We’ve known about it for quite some time and getting node to the place where it is today and having the time and resources to do it. We’re really excited to deliver that.
Greg: Great. I really appreciate you guys taking time out of your busy schedules to chat about All Things Node. I’m pretty excited about the software. I think we’re close to delivering the vision—at least the first iteration. I like the shape that it’s coming to. It should be fairly easy for a wide range of users to take it and run with it and do fun things with it.
One of the things I’m most interested in does, there’s this old adage from William Gibson, who wrote Neuromancer: the street finds its own uses for things. That’s what I’m hoping. Once the software gets out into the wild a bit more, that people start using it in ways that we never contemplated.
Chris: Just to pile on, Greg. There was another demo this morning in which Tomislaw did exactly What you’re talking about, Greg, where he modified the RNode software to be able to allow him to write essentially a Javascript client that speaks WebSocket and sends Rholang to his node software and execute commands by doing Rholang from his client web browser.
Greg: Oh, that’s awesome.
Chris: It’s all also in that recording. You’re exactly right. Once people get their hands on these tools, they can start applying their creative ideas they have and take it in directions that were never contemplated at the beginning.