mudRPC, gRPC for unstable networks

mudRPC is RPC for networks that go away: a call is a message, a reply may come hours later or expire, nothing blocks. The Go binding is one package over the C ABI in bindings/c/mudrpc.h through cgo: mud.go, the peer, with no dependency beyond the standard library, and mud_grpc.go beside it, the drop-in for gRPC, with the google.golang.org/grpc and google.golang.org/protobuf modules the product already has.

Integrate

1. Build the library. A Rust toolchain is the one build dependency, until prebuilt binaries ship at 3.0:

cargo build --release -p mudrpc-c

That gives target/release/libmudrpc.dylib (.so on Linux). Copy the folder bindings/go into your module as a package named mud, with mudrpc.h beside its two files (a product with no gRPC leaves mud_grpc.go out), or from 3.0 go get github.com/jacob-mellor/mudRPC/bindings/go/v3, the same two files as a module, and tell cgo where the library is: CGO_LDFLAGS="-L<dir>" in the environment of go build and go run, <dir> the library’s folder, and -ldflags="-extldflags=-Wl,-rpath,<dir>" on the command, once, for the final link alone (in CGO_LDFLAGS the rpath reaches every cgo package and the linker warns of the duplicate).

2. Make one call. The smallest complete program: two peers in one process, a call and a reply moved by hand between them.

package main

import (
	"crypto/rand"
	"fmt"
	"os"

	"example/mud" // the package you copied in, under your module's path
)

func main() {
	key := make([]byte, 32) // the fleet key: 32 bytes from the OS, provisioned to every peer; never a constant and never a passphrase
	rand.Read(key)
	epoch := uint32(1)                        // persisted beside the key; raise it on every boot and restore, write it first
	gcs, err := mud.New("gcs", key, 1, epoch) // roster numbers; 0 cannot send
	if err != nil {
		panic(err)
	}
	drone, _ := mud.New("drone", key, 2, epoch)
	drone.Listen("nav.Flight") // the drone's transport will subscribe to nav.Flight

	gcs.Call(0, "nav.Flight", "Arm", []byte{8, 1}, 30) // now 0; a protobuf request, which the generated code encodes for you

	peers := []*mud.Peer{gcs, drone}
	for now := uint32(0); now < 3; now++ { // your loop: pump each peer, every tick
		for i, p := range peers {
			for o, _ := p.Outgoing(now); o != nil; o, _ = p.Outgoing(now) {
				if !o.Listen { // hand o.Topic, o.Wire, o.TTL to your transport; a listen: subscribe to o.Topic
					in, _ := peers[1-i].Deliver(now, o.Topic, o.Wire)
					switch in.Kind {
					case mud.Call:
						fmt.Println("drone:", in.Method)
						peers[1-i].Reply(now, in, []byte("armed"))
					case mud.Reply:
						fmt.Println("gcs: status", in.Status, string(in.Bytes))
						os.Exit(0)
					case mud.Dropped:
						fmt.Println("dropped:", in.Dropped) // log in development, count in the field
					}
				}
				p.Sent() // only after the transport took it; if it refused, break instead
			}
			for e, _ := p.Tick(now); e != nil; e, _ = p.Tick(now) {
				fmt.Println("expired")
			}
		}
	}
	os.Exit(2)
}

go run . prints drone: Arm then gcs: status 0 armed and exits 0.

3. The loop. Every tick, drain Outgoing(now) into your transport, calling Sent() once it took each item (if it refused, break: the item is there next tick); feed what it received to Deliver(now, topic, wire); call Tick(now). Outgoing also yields listen items, one for the peer’s own name (its reply inbox) and one per Listen, and what a peer hears is what its transport subscribes to. now is a uint32, your uptime in whole seconds from a clock that only moves forward; the 12-byte id Call returns is the Id on the reply that answers it, Seq and End carry streams, Status is gRPC’s status number and mud.StatusName(status) its name. Incoming.Kind is mud.Call, mud.Reply, mud.Event (a publish), mud.Expired (from Tick, with the call’s Id), mud.Dropped, mud.Flood or mud.Rollback, and all but Expired and Dropped name their Sender. Save(), mud.Restore(saved, epoch), Renew(epoch) and Close() keep and end a peer under the host contract at the top of bindings/c/mudrpc.h, which every binding shares; every error is a *mud.Error whose Code is that header’s. A peer’s methods serialize on a mutex, so any goroutine may call them, one at a time; the drop-in’s channel owns its peer and lends it out through Use.

4. Streams. Answer one call with ReplyItem(now, call, seq, status, detail, payload, end) for seq 0, 1, 2… below 64 (above it -3), end true on the last item only; Reply(now, call, payload) is seq 0, status 0 and end, a unary reply. The items arrive in any order, each a Reply with its Seq and End, and the call completes once every Seq up to the lowest End it has seen has arrived, or is Expired at its ttl with one missing.

5. A signed fleet and the cipher. target/release/mud provision fleet 3 writes the fleet key, one device file per sender and the roster, as the C skill describes; a device builds mud.New(name, key, sender, epoch, mud.Config{Seed: seed, Budget: 1000}) from them and adds Roster(sender, publicKey, nil) for every sender it believes, with mud.PublicKey(seed) and Revoke(sender). Config{Hub: true} believes 256 senders in place of 16. The cipher is Config’s zero value, AES-256-GCM, unless Cipher: mud.CipherChaCha20Poly1305 is set; a fleet runs one.

6. UDP. mud.NewUdp("10.0.0.2:7000", "10.0.0.3:7000") sends each wire to every listed peer and hears only them, on the topics its peer listens to; a public address, 0.0.0.0 included, is -1 unless the library was built with the cargo feature internet and MUDRPC_INTERNET=listen-on-public-addresses, which no release is (mud.Internet() says which), and a bind the system refuses is -2. Send(item) takes an outgoing item and fails -2 when no peer took a wire; Poll() gives the next topic and wire, the wire nil when none waits; after Close(), every call is mud.ErrClosed.

Drop-in for gRPC

What every language’s drop-in shares, the one-line change, gRPC without HTTP/2, the dependencies and what is not carried, is From gRPC in the README.

The product keeps its .proto, protoc with protoc-gen-go and protoc-gen-go-grpc, and the client and server they generate. mud_grpc.go is a grpc.ClientConnInterface, what pb.NewDroneClient takes, and a grpc.ServiceRegistrar, what pb.RegisterDroneServer takes, over one peer and its radio; no grpc.NewClient, no grpc.NewServer and no listener:

// was: conn, err := grpc.NewClient("10.0.0.3:5001", grpc.WithTransportCredentials(insecure.NewCredentials()))
radio, err := mud.NewUdp("10.0.0.2:7000", "10.0.0.3:7000")
peer, err := mud.New("gcs", fleetKey, 1, epoch)
conn := mud.NewChannel(peer, radio)
drone := pb.NewDroneClient(conn)
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
reply, err := drone.Arm(ctx, &pb.ArmRequest{Id: 7})

A server is pb.RegisterDroneServer(conn, &droneService{}), the line that took grpc.NewServer(), in any process; one channel is a client and a server at once, and there is no Serve to call. While it lives the peer is reached through conn.Use(func(p *mud.Peer) { saved, err = p.Save() }), the same for a key’s Stage, Rotate and Retire, and the function must not call the channel but to Close it. Another radio is a mud.RadioFunc, taking each outgoing item on the pump’s goroutine with no lock held, so it may wait on a goroutine that calls into the channel, and it hands every wire it receives to conn.Deliver(topic, wire); the first error it returns on an item goes to the function of the mud.Fault option, run on the pump after its round with no lock held, and the item is tried for a second, then let go. The default fault writes to stderr, as grpc logs. mud.TTL(secs) sets the life of a call made with no deadline. conn.Close() ends every call Unavailable and stops the pump, from any goroutine, a handler, Use, the radio’s Send or the fault function included; it returns once the pump has stopped, or at once while any goroutine is inside Use or Send, the pump ending on its next round. A channel closed under its owner, by a peer or socket gone, answers every call so; close the channel before the peer.

cargo test -p mudrpc-c --test abi -- --include-ignored the_go_drop_in builds protoc-gen-go and protoc-gen-go-grpc from the modules bindings/c/tests/grpc-go/go.sum pins, generates drone.proto with protoc, and runs a grpc-go client and server through every case above, over two peers, over UDP and against a native peer, plain and under the race detector, and step 2’s program; it needs Go and protoc.

Verify

The host mistakes the contract names, each with what it answers:

Mistake Answer Kind
Restore or Renew with an epoch at or below the saved one *Error, Code == -6 error
sender 0 sending Code == -7, at the first send error
a key or seed of 32 equal bytes, or of another length Code == -1 error
Save bytes of another length to Restore Code == -4 error
Hub without a seed Code == -1 error
ttl 0, or past a week sent as 1, or a week clamp
a topic or name over 64 bytes, or a payload over the ceiling Code == -3 error
an empty name or topic, or one that is not UTF-8 Code == -1 error
a reply to a call this peer does not have Deliver returns Dropped == 5 silent
16 calls pending, or 16 items queued undrained Code == -2: decide what not to queue; Pending and Queued count them error
the 33rd send in one second Code == -9: the next second sends error

Test at each rung

Known edges