At Jane Street, we end up writing lots of messaging protocols, and many of these protocols end up being simple RPC-style protocols, i.e., protocols with a client and a server, where communication is done in a simple query/response style.

I've always found the writing of these protocols rather unsatisfying, because I could never find a clean way of writing down the types. In the following, I'd like to describe some nice tricks I've learned recently for specifying these protocols more cleanly.

A Simple Example

I'll start with a concrete example: a set of RPCs for accessing a remote filesystem. Here are the signatures for a set of functions that we want to make available via RPC.

  1. type path = Path of string list with sexp
  2. type 'a result = Ok of 'a | Error of string with sexp
  4. val listdir : path -> string list result
  5. val read_file : path -> string result
  6. val move : path * path -> unit result
  7. val put_file : path * string -> unit result
  8. val file_size : path -> int result
  9. val file_exists : path -> bool

The with sexp appended to the end of the type definitions comes from the Jane Street's publicly available sexplib macros. These macros generate functions for converting values to and from s-expressions. This is fantastically helpful for writing messaging protocols, since it gives you a simple hassle-free mechanism for serializing values over the wire. (Unfortunately, s-expression generation is not the fastest thing in the world, which is why we've written a set of binary serialization macros for high-performance messaging applications, which we intend to release.)

The usual next step would be to build up two types, one for the queries and one for the responses. Here's what you might write down to support the functions shown above.

  1. module Request = struct
  2. type t = | Listdir of path
  3. | Read_file of path
  4. | Move of path * path
  5. | Put_file of path * string
  6. | File_size of path
  7. | File_exists of path
  8. with sexp
  9. end
  11. module Response = struct
  12. type t = | Ok
  13. | Error of string
  14. | File_size of int
  15. | Contents of string list
  16. | File_exists of bool
  17. with sexp
  18. end

In some ways, this is great. The types are simple to write down and understand, and you get the wire protocol virtually for free from the s-expression converters. And both the server and the client code are pretty easy to write. Let's look at how that code might look.

First, let's assume we have functions for sending and receiving s-expressions over some connection object, with the following signature:

  1. val send : conn -> Sexp.t -> unit
  2. val recv : conn -> Sexp.t

Then the server code should look something like this:

  1. let handle_query conn =
  2. let module Q = Query in
  3. let module R = Response in
  4. let msg = Q.t_of_sexp (recv conn) in
  5. let resp =
  6. match query with
  7. | Q.Listdir path ->
  8. begin match listdir path with
  9. | Ok x -> R.Contents x
  10. | Error s -> R.Error s
  11. end
  12. | Q.Read_file path ->
  13. .
  14. .
  15. .
  16. in
  17. send (R.sexp_of_t resp)

And the client code could look like something this:

  1. let rpc_listdir conn path =
  2. let module Q = Query in
  3. let module R = Response in
  4. send conn (Q.sexp_of_t (Q.Listdir path));
  5. match R.t_of_sexp (recv conn) with
  6. | R.Contents x -> Ok x
  7. | R.Error s -> Error s
  8. | _ -> assert false

Unfortunately, to make this all work, you've been forced to turn your type definitions sideways: rather than specifying for each RPC a pair of a request type and a response type, as you do in the specification of ordinary function type, you have to specify all the requests and all the responses at once. And there's nothing in the types tying the two sides together. This means that there is no consistency check between the server code and the client code. In particular, the server code could receive a File_size query and return Contents, or Ok, when really it should only be returning either a File_size or Error, and you would only catch it at runtime.

Specifying RPCs with Embeddings

But all is not lost! With just a little bit of infrastructure, we can specify our protocol in a way that ties together the client and server pieces. The first thing we need is something that we're going to call an embedding, but which you might see referred to elsewhere as an embedding-projection pair. An embedding is basically a pair of functions, one for converting values of a given type into some universal type, and the other for converting back from the universal type. (For another take on universal types, take a look at this post from Steven). The universal type we'll use is S-expressions:

  1. type 'a embedding = { inj: 'a -> Sexp.t;
  2. prj: Sexp.t -> 'a; }

It's worth noting that the projection function is always going to be partial, meaning it will fail on some inputs. In this case, we'll encode that partiality with exceptions, since our s-expression macro library generates conversion functions that throw exceptions when a value doesn't parse. But it's often better to explicitly encode the partiality in the return type of the projection function.

We can now write up a type that specifies the type of the RPC from which we can derive both the client and server code.

  1. module RPC = struct
  2. type ('a,'b) t = {
  3. tag: string;
  4. query: 'a embedding;
  5. resp: 'b embedding;
  6. }
  7. end

Here's a how you could write the RPC.t corresponding to the listdir function:

  1. module RPC_specs = struct
  2. type listdir_resp = string list result with sexp
  3. let listdir = { RPC.
  4. tag = "listdir";
  5. query = { inj = sexp_of_path;
  6. prj = path_of_sexp; };
  7. resp = { inj = sexp_of_listdir_resp;
  8. prj = listdir_resp_of_sexp; };
  9. }
  11. .
  12. .
  13. .
  15. end

One slightly annoying aspect of the above code is that we had to define the type listdir_resp purely for the purpose of getting the corresponding s-expression converters. At some point, we should do a post on type-indexed values to explain how one could get around the need for such a declaration.

Note that the above specifies the interface, but not actually the function used to implement the RPC on the server side. The embeddings basically specify the types of the requests and responses, and the tag is used to distinguish different RPCs on the wire.

As you may have noticed, an ('a,'b) RPC.t corresponds to a function of type 'a -> 'b. We can put this correspondence to work by writing a function that takes an ('a,'b) RPC.t and an ordinary function of type

  1. 'a -> 'b

and produces an RPC handler. We'll write down a simple implementation below.

  1. type full_query = string * Sexp.t with sexp
  2. (* The first part is the tag, the second half is the s-expression for the arguments to the query. We only declare this type to get the s-expression converters *)
  4. module Handler : sig
  5. type t
  6. val implement : ('a,'b) RPC.t -> ('a -> 'b) -> t
  7. val handle : t list -> Sexp.t -> Sexp.t
  8. end
  9. =
  10. struct
  11. type t = { tag: string;
  12. handle: Sexp.t -> Sexp.t; }
  14. let implement rpc f =
  15. { tag = rpc.RPC.tag;
  16. handle = (fun sexp ->
  17. let query = rpc.RPC.query.prj sexp in
  18. rpc.RPC.resp.inj (f query)); }
  20. let handle handlers sexp =
  21. let (tag,query_sexp) = full_query_of_sexp sexp in
  22. let handler = List.find ~f:(fun x -> x.tag = tag) handlers in
  23. handler.handle query_sexp
  24. end

Using the RPC.t's we started writing as part of the RPC_specs module, we can now write the server as follows:

  1. let handle_query conn =
  2. let query = recv conn in
  3. let resp =
  4. Handler.handle [ Handler.implement RPC_specs.listdir listdir;
  5. Handler.implement RPC_specs.read_file read_file;
  6. Handler.implement RPC_specs.move move;
  7. Handler.implement RPC_specs.put_file put_file;
  8. Handler.implement RPC_specs.file_size file_size;]
  9. query
  10. in
  11. send conn resp

And we can implement the client side just as easily.

  1. let query rpc conn x =
  2. let query_sexp = rpc.RPC.query.inj x in
  3. send (sexp_of_full_query (rpc.RPC.tag,query_sexp));
  4. rpc.RPC.resp.prj (recv conn)
  6. module Client : sig
  7. val listdir : path -> string list result
  8. val read_file : path -> string result
  9. val move : path * path -> unit result
  10. val put_file : path * string -> unit result
  11. val file_size : path -> int result
  12. val file_exists : path -> bool
  13. end
  14. =
  15. struct
  16. let listdir = query RPC_specs.listdir
  17. let read_file = query RPC_specs.read_file
  18. let move = query RPC_specs.move
  19. let put_file = query RPC_specs.put_file
  20. let file_size = query RPC_specs.file_size
  21. let file_exists = query RPC_specs.file_exists
  22. end

Pleasantly, the signature of the Client module is exactly the same as the signature of the underlying functions we're exposing via RPC.

To be clear, this is far from a complete implementation --- particularly notable is the weak error handling, and we haven't said anything about how to deal with versioning of the protocol. But even though the implementation we've sketched out is a toy, we think this approach scales well to a full implementation.

There are still some problems. Although we've added static checks for some errors, we've eliminated some others. For instance, it's now possible for the user to specify multiple RPC.t's with the same tag, and there's no guarantee that the server has exhaustively implemented all of expected RPC.t's. I'm not aware of a clean way of getting all of these static checks working cleanly together in the same implementation.