Self Adjusting DOM and Diffable Data

In my last post, I gave some simple examples showing how you could use self adjusting computations, or SAC, as embodied by our Incremental library, to incrementalize the computation of virtual dom nodes. In this post, I’d like to discuss how we can extend this approach to more realistic scales, and some of the extensions to Incremental itself that are required to get there.

Along the way, we’ll discuss the way that diffable data structures relate to self adjusting computations, and how we can use ordinary pure data types and functional updates to those data types as the basis for building efficient incremental computations.

» Continue Reading.

Self Adjusting DOM

I’ve been thinking recently about how to structure dynamic web applications, and in particular about the role that incremental computation should play.

In this post, I’d like to outline an approach we’ve been experimenting with internally which uses Incremental, a general purpose library for building so-called self adjusting computations. Self adjusting computations are essentially graph-structured computations that can be updated efficiently when their inputs change.

I’ll describe this all in terms of OCaml, which is the language we’re doing these experiments in (courtesy of the excellent js_of_ocaml), but the ideas should be applicable to other languages.

» Continue Reading.

Incremental computation and the web

I’ve recently been thinking about the world of JavaScript and web applications. That’s odd for me, since I know almost nothing about the web. Indeed, Jane Street’s use of web technologies is quite minimal — nearly all of our user interfaces are text based, and all told we’ve been pretty happy with that.

But there are real limitations to console apps, and if you need something richer that’s cross-platform, the web is pretty appealing. For us it’s made yet more appealing by the fact that OCaml, our language of choice, compiles into JavaScript via js_of_ocaml.

So recently, when a few people internally got interested in trying out JavaScript-based UIs, I dug in a little to try to understand the landscape, and help us figure out what approach to take.

» Continue Reading.

Why OCaml?

Here’s a post from a talk I gave this last summer during our internship program about why we use OCaml. It spends a lot of time on how OCaml fits into the space of programming language designs, and why we think OCaml is in a real sweet spot in that design space, especially for the kind of work we do at Jane Street.

Warning: it’s a very informal talk, with lots of questions and answers from the audience, not all of which are clearly audible, for which I apologize. Still, I hope people will get something out of it.

Testing with expectations

Testing is important, and it’s hard to get people to do as much of it as they should. Testing tools matter because the smoother the process is, the more tests people will write.

Especially in the functional programming world, most of the talk about testing tools is focused on tools for property-based testing, like the various and sundry quickcheck-style systems. These are great, but sometimes, you don’t want to write down properties — what you want is to write your tests in terms of simple, concrete scenarios.

We’ve recently added support for what we call expect tests, a kind of test optimized for this kind testing. Expect tests allow you to write test scenarios without having to manually write out the output generated by the code you’re testing. Instead, that output is captured and recorded automatically for you, in a way that makes it easy to integrate into the source of the test.

» Continue Reading.

Quickcheck for Core

Automated testing is a powerful tool for finding bugs and specifying correctness properties of code. Haskell’s Quickcheck library is the most well-known automated testing library, based on over 15 years of research into how to write property-base tests, generate useful sources of inputs, and report manageable counterexamples. Jane Street’s Core library has not had anything comparable up until now; version 113.00 of Core finally has a version of Quickcheck, integrating automated testing with our other facilities like s-expression reporting for counterexample values, and support for asynchronous tests using Async.


Motivation

There are at least two other Quickcheck implementations on OPAM. Why write implementation N+1?

» Continue Reading.

rsync rounds timestamps to the nearest second

I’m not sure how I’ve managed to use rsync for so many years without ever noticing this, but hey, you learn something new every day!

[cperl@localhost ~]$ rpm -q --qf '%{name}-%{version}-%{release}\n' rsync
rsync-3.0.6-12.el6

[cperl@localhost ~]$ touch foo
[cperl@localhost ~]$ stat --format "%y" foo
2015-09-24 14:07:05.349357260 -0400
 
[cperl@localhost ~]$ rsync -a foo bar
[cperl@localhost ~]$ stat --format "%y" bar
2015-09-24 14:07:05.000000000 -0400
 
[cperl@localhost ~]$ cp -a foo baz
[cperl@localhost ~]$ stat --format "%y" baz
2015-09-24 14:07:05.349357260 -0400

No (functional) experience required

Jane Street is a serious functional programming shop. We use OCaml, a statically typed functional language for almost everything and have what is probably the largest OCaml codebase anywhere.

This leads lots of people to think that they shouldn’t even bother applying, under the assumption that we are only interested in hiring deep functional programming gurus. I think people get to this conclusion in part because they think of functional languages, especially those with fancy type systems, as arcane tools that can only be used effectively after deep study.

To the contrary, one of the reasons we started building production systems with OCaml was that it was relatively easy to understand, even for people with no formal CS background. Since then, we’ve had good experiences taking students with no functional experience at all and getting them to the point of being able to complete a project in just a few weeks. We also have a very successful “OCaml Bootcamp” program, where over four weeks, we train all of the incoming traders and many other non-developer hires on OCaml and our development tools and libraries. By the end, most of them are able to create useful applications.

» Continue Reading.

Introducing Incremental

I’m pleased to announce the release of Incremental (well commented mli here), a powerful library for building self-adjusting computations, i.e., computations that can be updated efficiently when their inputs change.

At its simplest, you can think of a self-adjusting computation as a fancy spreadsheet. In a spreadsheet, each cell contains either simple data, or an equation that describes how the value in this cell should be derived from values in other cells. Collectively, this amounts to a graph-structured computation, and one of the critical optimizations in Excel is that when some of the cells change, Excel only recomputes the parts of the graph that depend on those changed cells.

» Continue Reading.

Converting a code base from camlp4 to ppx

As with many projects in the OCaml world, at Jane Street we have been working on migrating from camlp4 to ppx. After having developed equivalent ppx rewriters for our camlp4 syntax extensions, the last step is to actually translate the code source of all our libraries and applications from the camlp4 syntax to the standard OCaml syntax with extension points and attributes.

For instance to translate code using pa_ounit and pa_test, we have to rewrite:

TEST = <:test_result< int >> ~expect:42 (f x)

to:

let%test _ = [%test_result: int] ~expect:42 (f x)

For small to medium projects it is enough to just take a couple hours to translate the source code by hand. But at Jane Street where we have a huge OCaml code base making extensive use of camlp4, it is simply not realistic. So we needed a tool to do that for us.

» Continue Reading.

4