Private Ethereum Blockchain on a Custom Kubernetes Cluster
Companies and even governments and other organizations nowadays start to evaluate blockchain technologies more and more, and the former picture of a typical slow-in-adoption-dinosaur-enterprise-company is already changing as it has clear benefits to invest in a more modern and scalable infrastructure like Kubernetes to run their mission-critical workloads.
A system like kubernetes can also be used as a playground to try out things, and this was our initial motivation to start working on kuberneteth during a Hackweek at SUSE. It's really not much. In fact it's not more than a small script to generate manifests via templates. Nothing fancy but it's enough to do the trick. That's mostly Kubernetes' fault as you don't need anything else than state files and an operator to deploy and maintain the most complex application stacks. So if you have access to a running Kubernetes instance, and you would like to create your own private Ethereum cluster, you're almost there, read on!
Introduction and Some History
After pushing some updates recently we have tagged the initial version to v1.0.0 as it has worked fine for that use case. It was a demonstration of a non scalable, fixed node cluster setup with the nice eth-netstats monitoring dashboard.
However with the recent updates it is now possible to create a configurable cluster with as many nodes as you wish, and it's scalable. It means no more than the manifests are written in a way that Kubernetes is able to scale the deployment up and down. Think of a new geth release with a critical fix, and you can now smoothly roll-out an update of the containers without any downtime.
Configuring the Cluster
To specify the most common configuration options on the highest level, there is a file called kuberneteth.yaml To generate the config file for geth there is a template called node.toml.erbwhich contains variables that are filled with the values that are defined in the nodes[*].geth.[*] section of kuberneteth.yaml
There is also a keystore for accessing the Ether that has been mined in the cluster. You need to first upload your private key through the Kubernetes Secrets API, or you just use the default existing one in the keystore folder linked from the title of this section. It's more than good enough if you only want to play around with it (password is 'linux').
Drawdown is that the keyfile itself will be a ConfigMap Object, which means it is stored in plain text. Nevertheless keys that are generated by geth are encrypted with a password, so someone with the keyfile doesn't necessarily control your funds right away.You can easily generate a key via geth account new.
Extending Customization
Though not all configuration options are accessible from the high level in kuberneteth.yaml, it is possible to either:
add key value pairs to it and then adapt the templates to write those variables
or just hard code the templates if it should be quick
Kubernetes Objects
Let's go through the deployment and see how the various Kubernetes Objects fit into the picture of this use case.
DaemonSet
The bootnodes' configuration setup runs as a DaemonSet because we need to have a valid bootnode URI present on all hosts (k8s-nodes). This address is written via a simple script as part of a customised docker image that has the bootnode binary baked into it (the official geth docker image doesn't provide it)
Service
A geth node usually exposes 2 ports for interaction:
8545 for the json rpc interface
30303 for the ipc interface
So with a Service Object we make sure to have one IP address (ClusterIP) for each geth-node pod routing to the rpc/ipc endpoints. Internally we have one direct consumer present, which is the monitoring Deployment, but you might also want to connect externally through a wallet, or an ethereum browser like e.g. mist or parity that allows you to deploy smart contracts and perform transactions.
ConfigMap
All configuration files like:
Geth's toml config
the genesis block configuration
the monitor's config file
are mounted as volumes through a ConfigMap Object. More dynamic values that need to be in a config file (e.g. bootnode addresses which are not known from the start) are added through an initContainer via sed.
Deployment
A Deployment Object is definitely what you want in most cases, as it allows you to manage, update and scale your applications more easily. So for all applications in this use case:
Ethereum Nodes
Monitoring
we are using a Deployment Object.
Secret
A private key to perform transactions and to access funds that have been generated through mining needs to be present on a geth-node.We could just use a ConfigMap Object, as the key is most likely a text file (e.g. json) which has an encryption within, so it doesn't expose the private key in plain text. But to be more secure we can make use of the Secrets functionality that Kubernetes offers.The downside is only that it requires one additional step, to upload your keyfile once to Kubernetes, before you deploy the cluster as the secrets will be mounted as volumes.
kubectl create secret generic geth-key --from-file /path/to/keyfile
So you can choose between those two ways by just setting keystore.secret to true or false in kuberneteth.yaml
Benefits of Kubernetes
There are some reasons why Kubernetes is a good choice to deploy and manage larger scale application clusters (like a private Ethereum cluster).
First of all the deployment itself can be easily coded, as in the philosophy of "infrastructure as code", or even better and more accurate:
"infrastructure as a state"
You can define your state once, and Kubernetes makes sure that this state is always met:
If a part of the cluster misbehaves, Kubernetes will fix it (aka self-healing)
…It’s quite similar to the consensus theory of Ethereum:
If a part of the cluster misbehaves, Ethereum will fix it (aka kick out the node from the network … aka tell it to fork off)
The second reason is maintenance. A task that has to be done by an administrator. A typical maintenance task of an Ethereum node, no matter which network, is an update of the client itself. Imagine a new geth version that fixes a critical bug, needs to be rolled out to all the cluster nodes. Kubernetes has a built in mechanism for doing this in a non disruptive manner, called the RollingUpdate. This is defined as a strategy in our Deployment and thus we can easily update the client without disrupting the whole cluster.
So once the cluster is running and there are some blocks written, try to:
update kuberneteth.yaml where it says geth.versione.g. from stable(which at the time of writing is 1.7.2) to v1.7.3
run ./kuberneteth to write the new deployment.yaml or update deployment.yaml directly in case you don't want to update all nodes at once
run kubectl apply -f deployment.yaml
(optional) watch the status of the rollout: kubectl rollout status deployment geth-n83192ad8965300b2-deployment
recreate the monitor to write the new monitor config: kubectl scale deployment monitor-deployment --replicas 0; kubectl scale deployment monitor-deployment --replicas 1;
and you will now see an updated version of geth picking up from the block where it left off.
The Ethereum Network Cluster is Running
…after configuring with kuberneteth.yaml and running this commands:
./kuberneteth # creates the manifest yaml
kubectl apply -f deployment.yaml # deploys the manifest to kubernetes
Looking at the eth-netstats monitoring dashboard
Let's have a look at our cluster through the monitoring Deployment
kubectl port-forward $(kubectl get po | grep monitor | awk '{print $1}') 3001:3001
and then visit http://localhost:3001 in the browser.
We can see a dashboard with all kinds of interesting statistics about our cluster.
Deploying a distributed application
Now it's time to deploy distributed applications, called Dapps. Essentially a Dapp is a back-end program that runs on the blockchain and is also referred to as a smart contract.
The language of choice is Solidity which is fairly easy to learn, as it feels a bit like JavaScript. There are also other implementations like e.g. serpent which do the same thing: Compile to code that will be interpreted by the Ethereum Virtual Machine (EVM) and ensures a given program is executed in a non-disruptive and censorship-proof way. The Ethereum platform provides this reliable environment.
To deploy a smart contract one can use an Ethereum browser or wallet like mist or parity, or connect via geth to an interactive console (geth attach), or write JavaScript code via the web3.js bindings, or issue raw curl commands to the geth json-rpc. It's now just a matter of choosing the right client that fits you best.
In this example we chose the mist wallet as it is really easy to use and runs on any platform.
Let's pick one of the nodes, it doesn't matter which one as all nodes export a json-rpc port
kubectl port-forward geth-n12b143ba4b727847-deployment-3182110105–34bz9 8545:8545
and now run mist with the parameters to connect to localhost:8545
ethereumwallet --rpc http://127.0.0.1:8545 --node geth --network test
In the tab CONTRACTS/Deploy a new Contract we can perform this transaction, provided we have some ether to interact/transact with the blockchain.
First we copy + paste our contract source code into the interactive contract editor, and then it will be automatically compiled.
We can now select the Block Caster contract, which already inherits the mortal contract, press the Deploy button, and now it's just a matter of signing the transaction (with our decryption password of the private key).
Now we can interact with the contract. Let's create a new account, which will be called consumer. From this account we will sign an instance of our deployed contract, which just means we will consume something (the Dapp).
To sign any contract we need to send the new user consumer some ether.
Now we can go to the address of the contract, it's a normal address like any other account on the blockchain, and interact with it.
It looks like a real contract (in real life on paper) as we can see some form fields we need to fill, and in the end we need to sign it. But we don't sign by hand, in Ethereum we sign by a private key and a password.
As the function we pick broadcast. The user address is our identification, meaning this address/account/user wants to sign that contract.The name is just an identifier for the video itself. The length means how long (in seconds) we want to play this video.The url is an address on the www where the video can be fetched (this should probably be IPFS or some other decentralized storage, but here it can be any server) On the next block, the transaction will be processed, and the web application that listens to this contract will play the video for 12 seconds…