When Bash Scripts Bite

There are abundant resources online trying to scare programmers away from using shell scripts. Most of them, if anything, succeed in convincing the reader to blindly put something that resembles

set -euo pipefail

at the top of their scripts. Let's focus on the "-e" flag. What does this do? Well, here are descriptions of this flag from the first two results on Google for "writing safe bash scripts":

  • "If a command fails, set -e will make the whole script exit, instead of just resuming on the next line" (https://sipb.mit.edu/doc/safe-shell/)
  • "This tells bash that it should exit the script if any statement returns a non-true return value." (http://www.davidpashley.com/articles/writing-robust-shell-scripts/)

Unfortunately, this is bash we are talking about and the story is never that simple.

A couple months ago, a particular production bash script (if that doesn't sound horrifying, hopefully it will by the end of this post) failed in the worst kind of way: silently. The script generates a list of valid users at Jane Street and pushes this out to our inbound mail servers. It looks something like:

set -euo pipefail
...
echo "($(ldap-query-for-valid-users))" > "/tmp/all-users.sexp"
...
push-all-users-if-different

On this one particular day, a file was deployed with the contents "()". But why didn't set -e cause the script to exit when ldap-query-for-valid-users failed? A quick look at the bash man page answers this question. It turns out that there are a couple surprising subtleties to this flag. Here are a couple:

set -e works on "simple commands"

A script will exit early if the exit status of a simple command is nonzero. So how is a simple command executed? In short, bash does all expansions and checks to see if there is still a command to run. If there is a command to run, the exit status of the simple command is the exit status of the command. If there is not a command to run, the exit status of the simple command is the exit status of the last command substitution performed. Here are some example commands that all have exit status 0, so would not cause a set -e script to exit:

# echo, local and export are commands that always have exit status 0
echo "$(/bin/false)"
local foo="$(/bin/false)"
export foo="$(/bin/false)"

# the last command substitution has exit status 0
foo="$(/bin/false)$(/bin/true)"

set -e does not get passed to subshells in command substitution (without --posix)

Here is an example consequence of this:

set -e

foo() {
    /bin/false
    echo "foo"
}
echo "$(foo)"

Running this script with bash will print "foo" while running this with bash --posix (or sh) will not. Both scripts will exit with status 0.

Tangible takeaway

This is not to say that something like set -euo pipefail should not be used at the top of all bash scripts, but it should not give you a false sense of security. Like all production code, you must reason about all failure conditions and ensure they are handled appropriately. Even if you are some kind of bash expert who knows all these subtleties, chances are your peers do not. The execution of shell scripts is subtle and confusing, and for production code, there is likely a better tool for the job.

This is not the performance you were looking for: the tricks systems play on us

It's often surprising just how much software performance depends on how the software is deployed. All the time and effort you've invested in optimization can be erased by a few bad decisions in scheduler policy, affinity, or background workload on a server.

So here are a few things I check for when an app's performance is unexpectedly bad. These are things that should apply to any OS running on a modern server, but the specific tools I'll mention are for Linux.

Are you running the binary you think you are?

It's funny how often seemingly bizarre problems have simple explanations. So I start by checking that I'm really running the right binary. I use md5sum to get a hash:

md5sum /path/to/your/executable

Then I verify that the hash matches the md5sum of the app I was trying to deploy. On Linux, you do the same trick to check a running binary via the proc filesystem if you know the process's PID:

md5sum /proc/$PID/exe

Are the dynamic libraries in use the same?

Sometimes the same app will perform unexpectedly because the dynamic libraries are not what you'd expect. Ldd can tell you which libraries will be linked at startup time:

ldd /path/to/your/executable 
# or 
ldd /proc/$PID/exe

Did you affinitize the process?

With Linux, you can restrict the cores that a process runs on. That can be a benefit because it helps keep the process's data warm in the processor's cache. For a single-threaded app, affinitizing to a single core might be the right choice, but a busy multi-threaded app may require multiple cores.

And you can see which cores a process is able to run on via taskset -p $PID. Taskset can also be used to control which cores a process runs on.

Don't forget about NUMA effects

Modern servers use NUMA, which means that latency and throughput to RAM, disk or the network depends on which core an application is running on. Though the penalty is small for each operation (in the range of hundreds of nanoseconds), when aggregated across an application the affect can be noticable.

Keep each application close to the things it uses. If an application uses the network, then affinitize the application to a core that's on the same NUMA node as the network adapter that it's using.

On Linux, you can the topology of your hardware using numactl -H. Here's sample output:

available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14
node 0 size: 65442 MB
node 0 free: 63882 MB
node 1 cpus: 1 3 5 7 9 11 13 15
node 1 size: 65536 MB
node 1 free: 63515 MB
node distances:
node   0   1 
  0:  10  21 
  1:  21  10 

The output tells you that there are 2 nodes, each with 64 GB of RAM and 8 cores.

What about other processes?

Just because you affinitized your app to a specific core doesn't mean that other apps won't also use that core. So once you start affinitizing one app, you'll want to affinitize the other apps on the server as well.

For a while, the Linux kernel has a command line option to reserve cores from boot time: isolcpus. For instance, booting Linux with the kernel parameter isolcpus=1,3-5 tells the kernel that by default, no process should be scheduled cores 1, 3, 4 and 5. However, we as well as others have found that isolcpus can lead to unintended behavior where load is concentrated rather than spread across cores, so we don't use it.

Affinity and other hardware

If an app uses a lot of peripherals (e.g. network or storage), make sure the app is affinitized to the same NUMA node as the peripheral.

To check the NUMA node of an ethernet device, you can use sysfs:

cat /sys/class/net/$ETH/device/numa_node

The Linux tool hwloc-ls will also tell you how system components map to NUMA nodes.

Machine setup

Sometimes the problem isn't with how the software is deployed but the performance difference comes from the machine itself: either its hardware or software setup is not quite what you'd expect.

Performance on a virtual machine is often quite a bit worse than on a physical machine. You can check if a machine is virtual by looking for the hypervisor flag in /proc/cpuinfo:

grep -q '^flags.* hypervisor.*' /proc/cpuinfo && echo this is a VM

Is this the software you expect?

For starters, you can figure out the version of the Linux kernel on a machine with uname -a. Different kernels can behave very differently on the same workload.

You can also use your OS package manager to list all the packages and versions. Often I'll run the same command on two servers and diff the output:

function hdiff () { diff -u <(ssh $1 $3) <(ssh $2 $3) }

You can use this to diff the software installed on two hosts:

hdiff $host1 $host2 "dkpg -l"

Is the hardware what you expect?

As a first step, check the processor model and speed via cat /proc/cpuinfo.

DMI can tell you many things about the hardware you're dealing with:

hdiff $host1 $host2 "sudo /usr/sbin/dmidecode"

The output of dmidecode is huge and very detailed. One thing to pay particular attention to is the version of the BIOS:

BIOS Information Vendor: Computers Inc. Version: 1.5.1 Release Date: 06/23/2012

Finally, when dealing with the unexpected, it never hurts to check whether the server you're running on has been rebooted recently enough:

$ uptime 13:16:41 up 300 days, 9:21, 1 user, load average: 0.00, 0.00, 0.00

Three hundred days is way too long.

Summary

Advances in server architecture have led to spectacular performance gains, but often the gains are only realized when apps are tuned properly. This post only scratches the surface of the issues in performance tuning. Still, I've found these tools useful and I hope you will too.

By the way, if you enjoy solving these sorts of problems, Jane Street is hiring!

rsync rounds timestamps to the nearest second

I'm not sure how I've managed to use rsync for so many years without ever noticing this, but hey, you learn something new every day!

[cperl@localhost ~]$ rpm -q --qf '%{name}-%{version}-%{release}\n' rsync
rsync-3.0.6-12.el6

[cperl@localhost ~]$ touch foo
[cperl@localhost ~]$ stat --format "%y" foo
2015-09-24 14:07:05.349357260 -0400
 
[cperl@localhost ~]$ rsync -a foo bar
[cperl@localhost ~]$ stat --format "%y" bar
2015-09-24 14:07:05.000000000 -0400
 
[cperl@localhost ~]$ cp -a foo baz
[cperl@localhost ~]$ stat --format "%y" baz
2015-09-24 14:07:05.349357260 -0400

Reverse web proxy in ~50 lines of BASH

In the spirit of reinventing the wheel for fun, I hacked this together as a quick challenge to myself last week. It's a little rough around the edges, but I thought it was too cute not to share. If you have any bug fixes, please post them in the comments.

#!/bin/bash

set -e -u

function usage() {
  echo "USAGE: ${0} parent {port} {lower_bound_backend_port} {upper_bound_backend_port}"
}

[[ $# < 1 ]] && usage && exit 1

mode=${1};shift
case ${mode} in
  parent)
    PORT=${1};shift
    LOWER=${1};shift
    UPPER=${1};shift
    socat TCP-LISTEN:${PORT},fork,reuseaddr "EXEC:${0} child ${LOWER} ${UPPER}"
    ;;
  child)
    LOWER=${1};shift
    UPPER=${1};shift
    COUNT=0
    PORT=$(shuf -i ${LOWER}-${UPPER} -n 1)
    let "RANGE = UPPER - LOWER"
    SUCCESS=false
    while ((COUNT <= RANGE)) || ${SUCCESS}; do 
      set +e
      if socat STDIN TCP:127.0.0.1:${PORT},connect-timeout=2; then
        SUCCESS=true
        break
      else
        echo "unable to connect to port ${PORT}" >&2
      fi
      set -e
      let COUNT+=1
      let PORT+=1
      if ((PORT > UPPER )); then
        let 'PORT = LOWER'
      fi
    done
    if ! ${SUCCESS}; then
      echo "HTTP/1.1 500 Internal Server Error"
      echo
      echo "All REDACTED servers are down.  Please report to REDACTED@janestreet.com."
      exit 1
    fi
    ;;
  *)
    usage
    exit 1
    ;;
esac

Inspecting the Environment of a Running Process

Sometimes its useful to be able see the values of environment variables in running processes. We can use the following test program to see how well we can accomplish this:

#define _GNU_SOURCE
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char **argv)
{
        int n;
        char *envstr;
        while((n = scanf("%as", &envstr)) != EOF) {
                putenv(envstr);
        }
        return 0;
}

This program just reads strings from stdin and then basically passes them on to `putenv(3)1 so we have any easy way to modify our environment.

Now, lets run it with env -i to reset the environment to something pretty sparse:

[cperl@localhost ~]$ gcc -Wall t.c -o t
[cperl@localhost ~]$ env -i FOO=bar ./t

First, lets see what we can get out of /proc/{pid}/environ, as googling for this problem will undoubtedly point you in this direction (including ps eww which reads /proc/{pid}/environ):

[cperl@localhost ~]$ cat /proc/$(pgrep -x t)/environ | xargs -r0 -n1 echo
FOO=bar

Great, so that looks like its our answer!

Unfortunately, /proc/{pid}/environ only reflects the environment of the process when it started and does not reflect any calls that process might have made to putenv(3) or setenv(3) (you can experiment with the above program substituting in setenv(3) for putenv(3) and playing with overwrite to see what you get).

We can see that if we feed some data into our program, causing calls to putenv(3):

[cperl@localhost ~]$ env -i FOO=bar ./t
BAR=baz

And then check /proc/{pid}/environ again:

[cperl@localhost ~]$ cat /proc/$(pgrep -x t)/environ | xargs -r0 -n1 echo
FOO=bar

However, we can verify the data is really there if we attach with gdb and iterate over the environ(7) array directly:

[cperl@localhost ~]$ gdb ./t $(pgrep -x t)
...
(gdb) set $i = 0
(gdb) while (environ[$i] != 0)
 >print environ[$i++]
 >end
$1 = 0x7fffc8e42fec "FOO=bar"
$2 = 0x12d4080 "BAR=baz"

Unfortunately, I'm not aware of any other way to get this "dynamic environment info" (except for other ptrace based solutions). Obviously attaching to production processes with gdb (or ptrace in general) isn't a great idea. Most of the time you'll probably be fine inspecting /proc/{pid}/environ and verifying (via source code inspection) that that process you care about doesn't make any calls to putenv(3) or setenv(3) for the variables whose values you are interested in.

If you have any better ideas about how to get this information, please share in the comments!

Inspecting Internal TCP State on Linux

Sometimes it can be useful to inspect the state of a TCP endpoint. Things such as the current congestion window, the retransmission timeout (RTO), duplicate ack threshold, etc. are not reflected in the segments that flow over the wire. Therefore, just looking at packet captures can leave you scratching your head as to why a TCP connection is behaving a certain way.

Using the Linux ss utility coupled with crash, its not too difficult to inspect some of the internal TCP state for a socket on Linux. Figuring out the meaning of all variables and how they relate to the variables referenced in the many TCP RFCs and papers is another matter, but at least we can get some idea of what is going on.

First, you can ask ss to give you information about, say, NFS sockets in use on a given client system:

[cperl@localhost ~]$ ss -eipn '( dport = :nfs )'
State       Recv-Q Send-Q    Local Address:Port    Peer Address:Port 
ESTAB       0      0         192.168.1.10:975      192.168.1.200:2049   ino:12453 sk:ffff8802305a0800
     ts sack cubic wscale:6,7 rto:201 rtt:1.875/0.75 ato:40 cwnd:10 ssthresh:40 send 61.8Mbps rcv_rtt:1.875 rcv_space:1814280
ESTAB       0      0         192.168.1.10:971      192.168.1.201:2049   ino:16576 sk:ffff88022f14d6c0
     ts sack cubic wscale:6,7 rto:202 rtt:2.125/1.75 ato:40 cwnd:10 ssthresh:405 send 54.5Mbps rcv_rtt:5 rcv_space:3011258

Internally, ss uses the tcp_diag kernel module to extract extra information (this is done via an AF_NETLINK socket).

A lot of interesting TCP state is provided in this output. For example, you can see the current retransmission timeout ("rto"), the current buffer space available for receiving data ("rcv_space"), the congestion control algorithm ("cubic") and you can see what the window scale option for the connection is (the number before the comma is the scaling applied to the window offered by the remote endpoint and the number after the comma is the scaling the remote endpoint will be applying to the window offered by us (i.e. its the Window Scale option we sent in our initial SYN). Some of the other variables are interesting too, but going into details on all of them is beyond the scope of this blog post.

If you're really, really interested in the kernel's internal state, you can also take the address of the struct sock that ss gave you (e.g. sk:ffff8802305a0800) and inspect it with crash:

[cperl@localhost ~]$ sudo crash -e emacs
...
      KERNEL: /usr/lib/debug/lib/modules/2.6.32-431.1.2.0.1.el6.x86_64/vmlinux
    DUMPFILE: /dev/crash
        CPUS: 4
        DATE: Tue Jul  1 15:26:19 2014
      UPTIME: 1 days, 07:32:48
LOAD AVERAGE: 0.08, 0.05, 0.01
       TASKS: 871
    NODENAME: localhost
     RELEASE: 2.6.32-431.1.2.0.1.el6.x86_64
     VERSION: #1 SMP Fri Dec 13 13:06:13 UTC 2013
     MACHINE: x86_64  (2992 Mhz)
      MEMORY: 7.9 GB
         PID: 29732
     COMMAND: "crash"
        TASK: ffff88013a928080  [THREAD_INFO: ffff88011b548000]
         CPU: 1
       STATE: TASK_RUNNING (ACTIVE)

crash> struct tcp_sock.rcv_nxt,snd_una,reordering ffff8802305a0800
  rcv_nxt = 3794247234
  snd_una = 2557966926
  reordering = 3

Because of the way Linux stores the structures in memory, you can just cast the struct sock to a struct tcp_sock. If you leave off the specific members in the "struct" invocation above you can get a recursive dump of all the fields and the structures embedded within (its just too large to be useful in this blog post).

It's possible you might not be able to get what you want just using crash and may want to turn to a tool like SystemTap to further figure out what is going on, but this is a decent place to start.

Why change is hard

At Jane Street we have a number of systems that are vital to the operation of the firm. As the company has grown, so have these systems, and in the process acquired a few oddities. The below is a post I wrote a while ago for our internal blog to give people outside of tech an idea of why it often takes a long time to add features to these systems or straighten out some of their weird behaviors.

Imagine you're driving your car. It's an okay car. It drives. It may be a little rusty, and the engine makes some rattling sounds, but you're not too concerned. Actually, upon closer inspection you notice one of the tires is flat. Funny, you must've been driving like this for years. Come to think of it, you did always have to pull pretty hard on the steering wheel to avoid veering off the road... You should probably do something about this.

Easy, just stop and change the tire, should only take a few minutes. But here's the catch: you can never stop. Well. That's annoying. You can probably still change the tire if you spend a lot of time preparing, make a careful plan, and then risk your life.

Assuming you survive the tire change, there's still that rattling noise in the engine. That's obviously going to be an issue sooner or later, and no amount of acrobatics is going to help you change the engine whilst driving. Hm. So I lied before. You can stop the car, but only for a few minutes at a time, and you'd better be damn sure the car will start again when you need to go on. So you can't just stop at a garage and change the engine - that'll take too long and who knows whether they'll connect all the hoses and gears and stuff1 the right way on the first try. You adopt the obvious solution:

  1. At a quick stop, have a new engine strapped to the roof of your car
  2. Connect things up so that you can switch between the old and the new engine2
  3. At another quick stop remove the old engine
  4. Yet another quick stop, move the new engine from the roof into the engine compartment
  5. Spend the next 6 months shortening excess hose, smoothing out dents in your roof, and scrubbing away oil stains in the upholstery.

The bottom line is that when you want to make any kind of change to a system that can never be shutdown or can only be down for very short amounts of time, you probably can't (or at least shouldn't) just make that change in one go. You have to break it down into very small, well understood steps, not all of which may directly contribute to what you actually want to achieve.

[1] It should be evident at this point that I don't know anything about cars.

[2] See [1].

Disabling Chrome’s x-webkit-speech vulnerability

It's been a busy couple of weeks for Internet security! Almost unnoticed amongst the 'Heartbleed' fallout was a post on Guy Aharonovsky's blog detailing how Google Chrome's speech-to-text features can be used to snoop on anything you say near your computer — via a single tag attribute and some CSS.

The exploit, in a nutshell:

A text box with the x-webkit-speech attribute lets the user click a microphone icon and speak text into the box. With some simple stylesheet tricks, the blogger shows how to hide the text box (and subsequent pop-up) so that speech can be captured without the user's knowledge.

Okay, so that's Not Good. How do we fix it?

The Chrome devs responded quickly (especially once the proof-of-concept was made public), removing x-webkit-speech support from the upcoming Chrome v36. But that's not due for stable release until mid-May — we needed something to prevent this method of snooping in the meantime.

Luckily, Chrome has a pretty awesome Extension system, so it was near-trivial to build a proof-of-concept extension that simply removes the 'x-webkit-speech' attribute from any <input> tag on the page — the first draft was just a boilerplate 'manifest' file and 4 lines of code, but it worked!

After some testing the plugin was extended to listen for DOM changes (so it could detect if a speech input was added to the page via Javascript). Additionally, 'page icon' was added to give UI feedback that speech had been disabled, which the user can click to re-enable speech input if desired.

The extension is available in the Chrome Web Store.

How Does Automount Work Anyway?

Introduction

Autofs/automount is a combination of a user space program and some pieces in the kernel that work together to allow filesystems (many kinds of filesystems, but often NFS) to be mounted "just in time" and then be unmounted when they are no longer in use. As soon as a process wants to access an automounted filesystem, the kernel intercepts the access and passes control to a user space program (typically automount, but systemd now supports some automount functionality as well).

The user space program does whatever is necessary to mount the file system (which usually involves some invocation of mount(8)) and then reports success or failure back to the kernel so the kernel can either allow the process to continue to, or signal a failure.

We use automount for a number of things here at Jane Street. Recently, users started reporting that directories that shouldn't exist (i.e. some path on an automounted filesystem for which the automount daemon has no configuration) were spontaneously appearing and not going away. Most commonly, users were seeing dubious output from commands like hg root and hg status, both cases in which Mercurial calls stat(2) on any path that seems like it might be a ".hg" directory. The problem was that ".hg" directories kept popping up in places where they didn't actually exist, causing these stat(2) calls to succeed and Mercurial to believe it had found a valid repository directory. Because attempts to access this ghost ".hg" directory obviously fail, Mercurial provides odd output for hg root and hg status. We were stumped so I dug into automount to try to find out where things were going wrong.

Debugging

I'm a big fan of dynamic tracing tools such as DTrace, SystemTap and Ktap for troubleshooting problems like this. In this instance I used a SystemTap script (developed as I went and presented at the end of this blog post) coupled with simply browsing the available source code to better understand the automount daemon's behavior and ultimately present enough information to the right people to get our problem fixed.

For further info about using Dynamic Tracing to better understand your systems, I highly recommend reading Brendan Gregg's book, DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X and FreeBSD.

Maps

The configuration of the automount daemon is based on maps. The automount daemon takes as input the location where it can find its master map. That map can be in a file, LDAP or anywhere else but the automount daemon needs a master map (see auto.master(5)).

Assuming the master map is a file, each line of the master map consists of several fields. There is the mountpoint for the entry, the type (i.e. one of "file", "program", "yp" or several others), the format and then any options that are to be applied to this master map entry. If the master map entry specifies a mount point of "/-" then the map that it references is considered "direct". Otherwise, it is considered "indirect."

In addition, both direct and indirect mounts can have "offsets" (see autofs4-mount-control.txt, auto.master(5) and autofs(5) for further details).

In the end, from the view point of autofs filesystems, it's important to remember that with indirect mounts the autofs filesystem is mounted on the mountpoint specified in the master map (or in submaps if you have multiple levels of nesting) and then filesystems are automounted as subdirectories of the autofs filesystem. For mounts that are direct, direct with offsets and indirect with offsets the autofs filesystem is mounted at the point where the automounted filesystem will ultimately be mounted. As such, once that filesystem is mounted, the underlying autofs filesystem has been shadowed.

We use both indirect and indirect with offset mounts here at Jane Street.

User to Kernel Communication

Interfaces to the Kernel

All communication from user space to the kernel is done via ioctl(2).

The latest version of this interface, v5, uses a distinct set of ioctls from prior versions. The newer interface has advantages over the older interfaces, so it always makes sense to use the newer interface when you can.

For example, automount(8) from the autofs package will always try to use the new interface and only if it runs into some kind of a problem will it fall back to the older interface (e.g. if access to "/dev/autofs" is denied due to SELinux).

This document has a good description of the problem and motivation for the newer interface. If you really want to understand automount/autofs, you should read that document.

For the rest of this post, I'll focus specifically on the newer interface, which all new implementations should be using.

The New Interface

Some of the main points of the newer interface are:

  • All ioctls are issued to the "/dev/autofs" device node. Previously the user space daemon needed to open the directory where the autofs filesystem was mounted and use that file descriptor to issue ioctls.

  • There is a command that allows the daemon to request that the kernel open the autofs filesystem at a given path. The return value of this command is the file descriptor that the daemon should use for all subsequent requests pertaining to that filesystem. This allows you to access an autofs mount point even if that mount point is currently shadowed by an automounted mount. This can happen with direct maps and any maps that use offsets if the automount daemon was restarted while automounted filesystems are still in use.

  • There are 14 different types of requests (or commands) that can be sent from the user space daemon down to the kernel:

enum {
  /* Get various version info */
  AUTOFS_DEV_IOCTL_VERSION_CMD = 0x71,
  AUTOFS_DEV_IOCTL_PROTOVER_CMD,
  AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD,

  /* Open mount ioctl fd */
  AUTOFS_DEV_IOCTL_OPENMOUNT_CMD,

  /* Close mount ioctl fd */
  AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD,

  /* Mount/expire status returns */
  AUTOFS_DEV_IOCTL_READY_CMD,
  AUTOFS_DEV_IOCTL_FAIL_CMD,

  /* Activate/deactivate autofs mount */
  AUTOFS_DEV_IOCTL_SETPIPEFD_CMD,
  AUTOFS_DEV_IOCTL_CATATONIC_CMD,

  /* Expiry timeout */
  AUTOFS_DEV_IOCTL_TIMEOUT_CMD,

  /* Get mount last requesting uid and gid */
  AUTOFS_DEV_IOCTL_REQUESTER_CMD,

  /* Check for eligible expire candidates */
  AUTOFS_DEV_IOCTL_EXPIRE_CMD,

  /* Request busy status */
  AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD,

  /* Check if path is a mountpoint */
  AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD,
};
  • The parameter that is passed with each of the possible ioctl requests is always the following structure (filled in and accessed with different member fields depending on the ioctl request):
struct autofs_dev_ioctl {
  __u32 ver_major;
  __u32 ver_minor;
  __u32 size;       /* total size of data passed in, including this struct */
  __s32 ioctlfd;    /* automount command fd */

  /* Command parameters */

  union {
    struct args_protover     protover;
    struct args_protosubver  protosubver;
    struct args_openmount    openmount;
    struct args_ready        ready;
    struct args_fail         fail;
    struct args_setpipefd    setpipefd;
    struct args_timeout      timeout;
    struct args_requester    requester;
    struct args_expire       expire;
    struct args_askumount    askumount;
    struct args_ismountpoint ismountpoint;
  };

  char path[0];
};
   

The size field encodes the total size of the parameter (i.e. the total size of the current struct autofs_dev_ioctl being passed). The ioctlfd field encodes what autofs filesystem this ioctl is meant to operate on and the rest are ioctl specific arguments.

Each of the structs referenced in the union type provide field names (and data types) that allow access to the particular arguments that were passed in. I won't go through each of them, but as an example, here is the definition of struct args_requester:

struct args_requester {
  __u32   uid;
  __u32   gid;
};

This allows the kernel to write code like param->requester.uid to access the uid field in the parameter to the AUTOFS_DEV_IOCTL_REQUESTER_CMD ioctl call.

The Kernel Ioctl Dispatcher

When the autofs4 kernel module is loaded (yes, it's the autofs4 kernel module that supports the version 5 protocol), it creates the /dev/autofs device node and registers the proper components such that ioctls issued from user space to a file descriptor associated with /dev/autofs will wind up invoking the kernel function autofs_dev_ioctl.

That function uses a dispatch table to map the ioctl request it receives to a dedicated function in the kernel:

static ioctl_fn lookup_dev_ioctl(unsigned int cmd)
{
  static struct {
    int cmd;
    ioctl_fn fn;
  } _ioctls[] = {
    {cmd_idx(AUTOFS_DEV_IOCTL_VERSION_CMD),      NULL},
    {cmd_idx(AUTOFS_DEV_IOCTL_PROTOVER_CMD),     autofs_dev_ioctl_protover},
    {cmd_idx(AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD),  autofs_dev_ioctl_protosubver},
    {cmd_idx(AUTOFS_DEV_IOCTL_OPENMOUNT_CMD),    autofs_dev_ioctl_openmount},
    {cmd_idx(AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD),   autofs_dev_ioctl_closemount},
    {cmd_idx(AUTOFS_DEV_IOCTL_READY_CMD),        autofs_dev_ioctl_ready},
    {cmd_idx(AUTOFS_DEV_IOCTL_FAIL_CMD),         autofs_dev_ioctl_fail},
    {cmd_idx(AUTOFS_DEV_IOCTL_SETPIPEFD_CMD),    autofs_dev_ioctl_setpipefd},
    {cmd_idx(AUTOFS_DEV_IOCTL_CATATONIC_CMD),    autofs_dev_ioctl_catatonic},
    {cmd_idx(AUTOFS_DEV_IOCTL_TIMEOUT_CMD),      autofs_dev_ioctl_timeout},
    {cmd_idx(AUTOFS_DEV_IOCTL_REQUESTER_CMD),    autofs_dev_ioctl_requester},
    {cmd_idx(AUTOFS_DEV_IOCTL_EXPIRE_CMD),       autofs_dev_ioctl_expire},
    {cmd_idx(AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD),    autofs_dev_ioctl_askumount},
    {cmd_idx(AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD), autofs_dev_ioctl_ismountpoint}
  };
  unsigned int idx = cmd_idx(cmd);

  return (idx >= ARRAY_SIZE(_ioctls)) ? NULL : _ioctls[idx].fn;
}

One point worth noting here is that AUTOFS_DEV_IOCTL_VERSION_CMD doesn't actually do very much (i.e. its function pointer is NULL).

There is logic in the kernel to validate that the major and minor fields passed in the autofs_dev_ioctl with the ioctl request are valid and then update the major and minor fields with the kernel's values. However, this validation happens for all ioctls, so a AUTOFS_DEV_IOCTL_VERSION_CMD really is a nop.

In practice, its invocation looks like:

1396615913363043073:  automount(18060)(0xffff8803d19f8aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
1396615913363055840:  automount(18060)(0xffff8803d19f8aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1

Note: The large numbers at the beginning of the line are nanosecond timestamps. All of the examples showing tracing of various parts of autofs/automount look similar to the above.

Kernel to User Communication

Establishing A Communication Channel

Kernel to user space communication can happen in a few different ways.

First, the kernel can communicate with user space via the return value from ioctl(2). This is the least interesting communication channel.

Second, the kernel can (and does) modify the data in the autofs_dev_ioctl passed to it. Once the ioctl call completes, the daemon can access whatever output parameters should be available in the autofs_dev_ioctl (which depends on the specific ioctl that was requested).

Third, notification of missing mounts or the need to umount existing mounts happens via a regular unix pipe(2). The daemon creates a pipe just before it mounts a given autofs filesystem and passes the write end of that pipe as the `fd' parameter to the mount command:

1396451770814396287:  automount(18388)(0xffff8803d5ae0aa0) -> mount: dev: /etc/auto.foo, dir: /foo, type: autofs, data: fd=7,pgrp=18388,minproto=5,maxproto=5,indirect

In addtion, it passes the process group id of its process to the kernel so when the kernel receives VFS requests for the autofs filesystems, it can determine whether it should pass the requests to the automount daemon or should operate in "Oz mode" and let the process "see the man behind the curtain."

I didn't make that up, it's in the kernel sources and it's really called oz_mode.

In the case where you're mounting a new autofs filesystem, giving the kernel the pipe to use for communication is fairly straightforward. But, what about the case where the daemon has just been restarted and the autofs filesystems that you wish to operate on are actually shadowed by other (say NFS) filesystems?

The AUTOFS_DEV_IOCTL_OPENMOUNT_CMD ioctl was created to address this problem. This ioctl's parameter includes a path (as part of struct autofs_dev_ioctl described earlier). The return from this call will have ioctlfd field set to -1 for the call, since we're asking for the mount point to be opened.

Once you have an ioctlfd with which to operate, you can create a pipe and issue a AUTOFS_DEV_IOCTL_SETPIPEFD_CMD ioctl which has the same effect as passing the fd argument to mount(8). However, there is the additional restriction that you cannot issue a AUTOFS_DEV_IOCTL_SETPIPEFD_CMD against an autofs filesystem without first having set that autofs filesystem to be "catatonic" with AUTOFS_DEV_IOCTL_CATATONIC_CMD.

See autofs4-mount-control.txt for additional details.

Message Format

Messages from the kernel to the user space daemon via the pipe have the following format:

struct autofs_v5_packet {
  struct autofs_packet_hdr hdr;
  autofs_wqt_t wait_queue_token;
  __u32 dev;
  __u64 ino;
  __u32 uid;
  __u32 gid;
  __u32 pid;
  __u32 tgid;
  __u32 len;
  char name[NAME_MAX+1];
};

Note: This structure has some interesting history associated with it. The structure is 300 bytes, but due to alignment issues, on x86_64, the compiler pads it to 304.

This isn't a problem if both the kernel and the user space daemon are both 32 bit or both 64 bit, but becomes a problem when the daemon is 32 bit, but the kernel is 64 bit. In that case, the daemon is expecting a 300 byte packet from the kernel but will receive a 304 byte packet.

You can read more about this problem in this lwn.net article.

Message Types

There are only 4 types of messages that are sent from the kernel to the user space daemon in the v5 protocol. From the automount source:

static int handle_packet(struct autofs_point *ap)
{
  union autofs_v5_packet_union pkt;

  if (get_pkt(ap, &pkt))
    return -1;

  debug(ap->logopt, "type = %d", pkt.hdr.type);

  switch (pkt.hdr.type) {
  case autofs_ptype_missing_indirect:
    return handle_packet_missing_indirect(ap, &pkt.v5_packet);

  case autofs_ptype_missing_direct:
    return handle_packet_missing_direct(ap, &pkt.v5_packet);

  case autofs_ptype_expire_indirect:
    return handle_packet_expire_indirect(ap, &pkt.v5_packet);

  case autofs_ptype_expire_direct:
    return handle_packet_expire_direct(ap, &pkt.v5_packet);
  }
  error(ap->logopt, "unknown packet type %d", pkt.hdr.type);
  return -1;
}

When the user space daemon receives one of these messages, it knows what autofs filesystem it is for because the message arrives on the pipe that the daemon explicitly created for this communication. As a result, the packet itself doesn't need to identify what filesystem it is associated with.

However, one very important piece of information that comes through the pipe is the wait_queue_token. This is used by the user space daemon in its ioctl back to the kernel to notify the kernel whether or not it was able to process the request (via AUTOFS_DEV_IOCTL_READY_CMD or AUTOFS_DEV_IOCTL_FAIL_CMD) and uniquely identifies the associated request.

Messages of type autofs_ptype_missing_direct and autofs_ptype_missing_indirect are sent when a process is trying to access a directory entry and the kernel needs the userspace daemon to resolve it.

Messages of type autofs_ptype_expire_direct and autofs_ptype_expire_indirect are sent when the kernel has realized that a mount is no longer active and that it can be umounted, possibly in response to a AUTOFS_DEV_IOCTL_EXPIRE_CMD command from the daemon. That last case would mean the daemon makes a synchronous ioctl call into the kernel, the kernel roots around for a while, figures out something can be unmounted, then sends a message back to the daemon, which must be handled concurrently.

For example, here is automount reqeuesting an immediate expiration of /a/b/c, which causes (1) the kernel to call back into the daemon, (2) the daemon to perform the umount and signal success (or failure) via another ioctl, and finally (3) the orginal ioctl to complete.

1396454302262508340:   automount(18388)(0xffff8803d3cda040) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_EXPIRE_CMD: major: 1, minor 0, size: 24, ioctlfd: 307: how: AUTOFS_EXP_IMMEDIATE (/a/b/c)
1396454302262523253:   automount(18388)(0xffff8803d3cda040) -> autofs4_notify_daemon: autofs_ptype_expire_direct: pipefd: 48, proto: 5: {.hdr={.proto_version=5, .type=6}, .wait_queue_token=13052, .dev=1048629, .ino=9285341, .uid=0, .gid=0, .pid=486, .tgid=18388, .len=16, .name="ffff8800488dcbc0"}
1396454302262714273:   automount(18388)(0xffff8803d6cd0080) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 57, ioctlfd: -1: type: 0, path: /a/b/c
1396454302262728637:   automount(18388)(0xffff8803d6cd0080) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 57, ioctlfd: -1: devid: 38, magic: 0x6969
1396454302289469919:   umount.nfs( 489)(0xffff8803d64b4aa0) -> umount: name: /a/b/c
1396454302289942712:   automount(18388)(0xffff8803d6cd0080) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 57, ioctlfd: -1: type: 0, path: /a/b/c
1396454302289956871:   automount(18388)(0xffff8803d6cd0080) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 57, ioctlfd: -1: devid: 1048629, magic: 0x187
1396454302289984247:   automount(18388)(0xffff8803d6cd0080) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_READY_CMD: major: 1, minor 0, size: 24, ioctlfd: 307: token: 13052 (/a/b/c)
1396454302290005575:   automount(18388)(0xffff8803d6cd0080) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_READY_CMD: major: 1, minor 0, size: 24, ioctlfd: 307: token: 13052 (/a/b/c)
1396454302290072586:   automount(18388)(0xffff8803d3cda040) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_EXPIRE_CMD: major: 1, minor 0, size: 24, ioctlfd: 307: how: AUTOFS_EXP_IMMEDIATE ()

Note: In the above output the path in parenthesis is not actually a part of the ioctl. As part of the SystemTap script used to trace this I am taking the given ioctlfd and looking it up in the open file descriptors for the automount process and translating it into a path. This made it easier to match up various calls that automount daemon was making to better understand its behavior.

The User Space Daemon

At startup the user space daemon will do some initial sanity checking. To do this it mounts a few temporary autofs filesystems and issues ioctls to them to ensure things are as it expects them to be (version numbers match up etc.).

The startup steps look like the following:

automount(29803)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
automount(29803)(0xffff8803d6cd1540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
automount(29803)(0xffff8803d6cd1540) -> mount: dev: automount, dir: /tmp/autoQzUuVP, type: autofs, data: fd=5,pgrp=29803,minproto=3,maxproto=5
automount(29803)(0xffff8803d6cd1540) -> autofs4_fill_super: s=0xffff8803d40b8000 data=0xffff88034ce14000 silent=0x0
automount(29803)(0xffff8803d6cd1540) <- autofs4_fill_super: {.magic=?, .pipefd=?, .pipe=?, .oz_pgrp=?, .catatonic=?, .version=?, .sub_version=?, .min_proto=?, .max_proto=?, .exp_timeout=?, .type=?, .reghost_enabled=?, .needs_reghost=?, .sb=?, .wq_mutex={...}, .pipe_mutex={...}, .fs_lock={...}, .queues=?, .lookup_lock={...}, .active_list={...}, .expiring_list={...}}
automount(29803)(0xffff8803d6cd1540) <- mount
automount(29803)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 40, ioctlfd: -1: devid: 23, path: /tmp/autoQzUuVP
automount(29803)(0xffff8803d6cd1540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 40, ioctlfd: 5: devid: 23, path: /tmp/autoQzUuVP
automount(29803)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 5
automount(29803)(0xffff8803d6cd1540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 5
automount(29803)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 0
automount(29803)(0xffff8803d6cd1540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 5
automount(29803)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 0
automount(29803)(0xffff8803d6cd1540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 2
automount(29803)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 (/tmp/autoQzUuVP)
automount(29803)(0xffff8803d6cd1540)  umount: name: /tmp/autoQzUuVP
    mount(29807)(0xffff8803d24d2080) -> mount: dev: /tmp/autoEXHYpt, dir: /tmp/auto4X1sU6, type: none, data: 
    mount(29807)(0xffff8803d24d2080) <- mount
   umount(29808)(0xffff8803d269e040) -> umount: name: /tmp/auto4X1sU6
    mount(29810)(0xffff8803d81aeae0) -> mount: dev: /tmp/autoQgrTpK, dir: /tmp/autoWZ7jVn, type: none, data: 
    mount(29810)(0xffff8803d81aeae0) <- mount
   umount(29811)(0xffff8803d56d4080) -> umount: name: /tmp/autoWZ7jVn

Next, the daemon consults its maps, sets up the necessary autofs filesystems and waits on the pipes it has setup for communication from the kernel.

What do the various Ioctls Do?

A lot of this detail is described in autofs4-mount-control.txt, but I've gone through each one to show examples of their actual invocation as observed with SystemTap. If something I've written here conflicts with the document linked, I'm probably wrong.

AUTOFS_DEV_IOCTL_VERSION_CMD

As mentioned earlier, this ioctl is basically a no-op. It does some validation on the major and minor version numbers that you pass in the parameter (All ioctls on /dev/autofs do this same validation).

Specifically it validates that the major number is equal to AUTOFS_DEV_IOCTL_VERSION_MAJOR and that the minor number is less than or equal to AUTOFS_DEV_IOCTL_VERSION_MINOR:

#define AUTOFS_DEV_IOCTL_VERSION_MAJOR    1
#define AUTOFS_DEV_IOCTL_VERSION_MINOR    0

These same major and minor numbers are included in all ioctls from user space to kernel space.

It's also worth noting that the ioctlfd in this request is simply set to -1.

1396617872001439657:  automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
1396617872001453752:  automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1

AUTOFS_DEV_IOCTL_PROTOVER_CMD

This ioctl is issued on a per autofs filesystem basis and therefore requires a valid ioctlfd argument to identify the autofs file system in question.

The parameter to the ioctl contains a struct args_protover which is initially 0. On return from the ioctl, the kernel will have filled out this struct with the version field of the struct autofs_sb_info associated with this autofs filesystem in the kernel.

In practice, this call is only made once on startup for a temporary test autofs filesystem and is never used again after that.

1396617872001615443:  automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 40, ioctlfd: -1: devid: 21, path: /tmp/autoV2cNiF
1396617872001623807:  automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 40, ioctlfd: 5: devid: 21, path: /tmp/autoV2cNiF
1396617872001629141:  automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 5
1396617872001633477:  automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 5

(These prior commands are run to open a file descriptor for the mount point after mounting)

1396617872001636885:  automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 0
1396617872001648323:  automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 5

AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD

This ioctl is issued on a per autofs filesystem basis and therefore requires a valid ioctlfd argument to identify the autofs file system in question.

The parameter to the ioctl contains a struct args_protosubver which is initially 0. On return from the ioctl, the kernel will have filled out this struct with the sub_version field of struct autofs_sb_info associated with this autofs filesystem in the kernel.

In practice, this call is only made once on startup for a temporary test autofs filesystem and is never used again after that.

1396617872001651494:  automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 0
1396617872001654267:  automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 2

AUTOFS_DEV_IOCTL_OPENMOUNT_CMD

This ioctl is issued after an autofs filesystem is mounted in order to obtain the ioctlfd that can then be used for subsequent ioctls referencing that autofs file system. It's purpose is to obtain the ioctlfd and therefore you do not need to have a valid ioctlfd in order to issue this ioctl.

This actually opens a new file descriptor in the process for the directory which is the mount point of the autofs file system.

The parameter to the ioctl contains a struct args_openmount. In addition, the parameter contains the actual path of the autofs filesystem that you are trying to open as a string.

In practice this ioctl is issued once for each autofs filesystem that the user space daemon is managing.

1396879456563815509:  automount(10681)(0xffff8803d82ce040) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 31, ioctlfd: -1: devid: 24, path: /a/b/c
1396879456563823041:  automount(10681)(0xffff8803d82ce040) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 31, ioctlfd: 28: devid: 24, path: /a/b/c
[cperl@tot-qws-u12114d ~]$ sudo lsof -p $(pgrep -f /etc/auto.master) | grep /a/b/c
automount 10681 root   28r   DIR   0,24        0 16335017 /a/b/c

As you can see, the ioctlfd field is -1 on entry to the ioctl and is filled out by the kernel. Furthermore, we can confirm with lsof that the file descriptor is now open for the directory.

AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD

This ioctl is issued to signal that the user space daemon no longer needs to issue ioctls for a given autofs filesystem.

For this ioctl, there are no additional details passed in a structure. The only piece of information needed is the ioctlfd that the daemon is requesting to be closed.

This actually closes the file descriptor in the process.

In practice, this ioctl is used to close down the temporary autofs filesystem that is used during early initialization to check the various version information. Otherwise it only appears to be used when shutting down automount.

1396617872001657498:  automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 (/tmp/autoV2cNiF)
1396617872001663579:  automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 ()

AUTOFS_DEV_IOCTL_READY_CMD

This ioctl is one of two ioctls issued in response to a request from the kernel. This particular ioctl signals that the request processing was successful and the kernel can continue.

The parameter to the ioctl contains a struct args_ready. This contains the same token that was received from the kernel on the pipe where the request originated. This is how the kernel can match a response to its initial request (along with the fact that the ioctlfd means its for a particular autofs filesystem).

In practice, this ioctl is used all the time. Every time the user space daemon successfully handles automounting a filesystem, it communicates it to the kernel via this ioctl.

1396617916437019856:         hg(30925)(0xffff8803d18c4aa0) -> autofs4_notify_daemon: autofs_ptype_missing_indirect: pipefd: 7, proto: 5: {.hdr={.proto_version=5, .type=3}, .wait_queue_token=52225, .dev=21, .ino=11251159, .uid=12114, .gid=32771, .pid=30925, .tgid=30925, .len=6, .name=".hg"}
...
1396617916440211973:  automount(30790)(0xffff8803d11db500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_READY_CMD: major: 1, minor 0, size: 24, ioctlfd: 11: token: 52225 (/foo)
1396617916440230486:  automount(30790)(0xffff8803d11db500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_READY_CMD: major: 1, minor 0, size: 24, ioctlfd: 11: token: 52225 (/foo)

In the above output I've included the request from hg that caused the request to be sent to the automount daemon (the kernel function which does that notification is autofs4_notify_daemon). Here you can see the matching of wait_queue_token in the request and token in the response.

AUTOFS_DEV_IOCTL_FAIL_CMD

This ioctl is the second of the two ioctls issued in response to a request from the kernel. This command is similar to the ready command described above, but indicates failure.

The parameter to the ioctl contains a struct args_fail which contains both the token that the daemon received from the kernel as well as a status code.

This status code is what the kernel should return to the process that requested the access. Many times this will be ENOENT, but it can also be other things like ENOMEM or ENAMETOOLONG.

Its worth pointing out that the kernel uses negative numbers to communicate errors and therefore the status returned is -2 or -ENOENT. For instance, when the automounter calls get_ioctl_ops()->send_fail, the argument that it passes is always something like -ENOENT, or -ENOMEM or -ENAMETOOLONG.

1396617921296295641:       stat(30998)(0xffff8803d191c040) -> autofs4_notify_daemon: autofs_ptype_missing_indirect: pipefd: 7, proto: 5: {.hdr={.proto_version=5, .type=3}, .wait_queue_token=52228, .dev=21, .ino=11251159, .uid=12114, .gid=32771, .pid=30998, .tgid=30998, .len=3, .name=".hg"}
...
1396617921296848457:  automount(30790)(0xffff8803d19f8040) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_FAIL_CMD: major: 1, minor 0, size: 24, ioctlfd: 11: token: 52228, status: -2 (/foo)
1396617921296870360:  automount(30790)(0xffff8803d19f8040) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_FAIL_CMD: major: 1, minor 0, size: 24, ioctlfd: 11: token: 52228, status: -2 (/foo)

As you can see here, the status code for the failure is -ENOENT.

AUTOFS_DEV_IOCTL_SETPIPEFD_CMD

This ioctl can be used to set the pipe file descriptor that the kernel will use to send notifications to the user space daemon. The other option would be to pass the file descriptor during the actual call to mount(8). This ioctl only makes sense when you already have a valid ioctlfd to issue it against.

If the automount daemon was restarted and some of its autofs filesystems that its supposed to be managing are shadowed by the real mounts (and therefore still mounted), there is no way to establish this pipe file descriptor. This is the purpose of this ioctl.

1396617872058414208:  automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_SETPIPEFD_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: pipefd: 13 (/home)
1396617872058417490:  automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_SETPIPEFD_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: pipefd: 13 (/home)

Additional data from autofs4-mount-control.txt:

The call requires an initialized struct autofs_dev_ioctl with the ioctlfd field set to the descriptor obtained from the open call and the arg1 field set to descriptor of the pipe. On success the call also sets the process group id used to identify the controlling process (eg. the owning automount(8) daemon) to the process group of the caller.

AUTOFS_DEV_IOCTL_CATATONIC_CMD

This ioctl is issued against a specific autofs filesystem and therefore you must have a valid ioctlfd to use before it can be issued.

This ioctl asks the kernel to mark the struct autofs_sb_info as no longer being responsive to mount requests. In addition, it closes the kernel's side of the pipe.

If an autofs filesystem is marked as cataonic, then there is no longer any "magic" going on. Requests are not dispatched to the user space daemon and all processes are allowed to see the raw filesystem (not just the process id that was given as the pgrp mount option when the filesystem was mounted).

More details from autofs4-mount-control.txt state that this command is a prerequisite for AUTOFS_DEV_IOCTL_SETPIPEFD_CMD:

In order to protect mounts against incorrectly setting the pipe descriptor we also require that the autofs mount be catatonic (see next call).

And a few examples of its use:

1396617872001439657:  automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
1396617872001453752:  automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_VERSION_CMD: major: 1, minor 0, size: 24, ioctlfd: -1
1396617872001538014:  automount(30790)(0xffff8803d5ae0aa0) -> mount: dev: automount, dir: /tmp/autoV2cNiF, type: autofs, data: fd=5,pgrp=30790,minproto=3,maxproto=5
1396617872001570079:  automount(30790)(0xffff8803d5ae0aa0) -> autofs4_fill_super: s=0xffff8803d8ba5c00 data=0xffff880210282000 silent=0x0
1396617872001582788:  automount(30790)(0xffff8803d5ae0aa0) <- autofs4_fill_super: {.magic=?, .pipefd=?, .pipe=?, .oz_pgrp=?, .catatonic=?, .version=?, .sub_version=?, .min_proto=?, .max_proto=?, .exp_timeout=?, .type=?, .reghost_enabled=?, .needs_reghost=?, .sb=?, .wq_mutex={...}, .pipe_mutex={...}, .fs_lock={...}, .queues=?, .lookup_lock={...}, .active_list={...}, .expiring_list={...}}
1396617872001602201:  automount(30790)(0xffff8803d5ae0aa0) <- mount
1396617872001615443:  automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 40, ioctlfd: -1: devid: 21, path: /tmp/autoV2cNiF
1396617872001623807:  automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 40, ioctlfd: 5: devid: 21, path: /tmp/autoV2cNiF
1396617872001629141:  automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 (/tmp/autoV2cNiF)
1396617872001633477:  automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 (/tmp/autoV2cNiF)
1396617872001636885:  automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 0 (/tmp/autoV2cNiF)
1396617872001648323:  automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 5 (/tmp/autoV2cNiF)
1396617872001651494:  automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 0 (/tmp/autoV2cNiF)
1396617872001654267:  automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD: major: 1, minor 0, size: 24, ioctlfd: 5: 2 (/tmp/autoV2cNiF)
1396617872001657498:  automount(30790)(0xffff8803d5ae0aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 (/tmp/autoV2cNiF)
1396617872001663579:  automount(30790)(0xffff8803d5ae0aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 5 ()
1396617872001668692:  automount(30790)(0xffff8803d5ae0aa0) -> umount: name: /tmp/autoV2cNiF
1396617872005398387:      mount(30794)(0xffff8803d82cd500) -> mount: dev: /tmp/auto3HiGR8, dir: /tmp/auto8FUzqC, type: none, data: 
1396617872005425776:      mount(30794)(0xffff8803d82cd500) <- mount
1396617872006666370:     umount(30795)(0xffff8803d67cd500) -> umount: name: /tmp/auto8FUzqC
1396617872009924917:      mount(30797)(0xffff8803d18c4040) -> mount: dev: /tmp/autoFHRl05, dir: /tmp/autowC47zz, type: none, data: 
1396617872009966875:      mount(30797)(0xffff8803d18c4040) <- mount
1396617872011350661:     umount(30798)(0xffff8803d269e040) -> umount: name: /tmp/autowC47zz

OR

1396617872058378954:  automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 30, ioctlfd: -1: type: 1, path: /home
1396617872058386150:  automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 30, ioctlfd: -1: devid: 22, magic: 0x187
1396617872058391034:  automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 30, ioctlfd: -1: devid: 22, path: /home
1396617872058396234:  automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_OPENMOUNT_CMD: major: 1, minor 0, size: 30, ioctlfd: 17: devid: 22, path: /home
1396617872058407576:  automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 17 (/home)
1396617872058410942:  automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_CATATONIC_CMD: major: 1, minor 0, size: 24, ioctlfd: 17 (/home)
1396617872058414208:  automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_SETPIPEFD_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: pipefd: 13 (/home)
1396617872058417490:  automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_SETPIPEFD_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: pipefd: 13 (/home)
1396617872058420869:  automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_TIMEOUT_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: timeout: 604800 (/home)
1396617872058426251:  automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_TIMEOUT_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: timeout: 604800 (/home)
1396617872058436785:  automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 30, ioctlfd: 17: type: 0, path: /home
1396617872058441090:  automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 30, ioctlfd: 17: devid: 22, magic: 0x0
1396617872058471734:  automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 36, ioctlfd: -1: type: 0, path: /home/cperl
1396617872058477028:  automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 36, ioctlfd: -1: devid: 27, magic: 0x6969
1396617872058480975:  automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_REQUESTER_CMD: major: 1, minor 0, size: 36, ioctlfd: 17: uid: 0, gid: 0 (/home)
1396617872058484928:  automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_REQUESTER_CMD: major: 1, minor 0, size: 36, ioctlfd: 17: uid: 0, gid: 0 (/home)

AUTOFS_DEV_IOCTL_TIMEOUT_CMD

This ioctl is issued against a specific autofs filesystem and therefore you must have a valid ioctlfd to use before it can be issued.

The parameter to the ioctl contains a struct args_timeout. This contains the timeout for the autofs filesystem in seconds.

Internally the kernel converts this to jiffies and stores it in the exp_timeout field of the associated struct autofs_sb_info.

1396617872058420869:  automount(30790)(0xffff8803d67cd500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_TIMEOUT_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: timeout: 604800 (/home)
1396617872058426251:  automount(30790)(0xffff8803d67cd500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_TIMEOUT_CMD: major: 1, minor 0, size: 24, ioctlfd: 17: timeout: 604800 (/home)

AUTOFS_DEV_IOCTL_REQUESTER_CMD

This ioctl is issued against a specific autofs filesystem and therefore you must have a valid ioctlfd to use before it can be issued.

The parameter to the ioctl contains a struct args_requester. This struct is initially all 0 on the call and output parameters are filled in before returning from the ioctl.

1396530871075227821:  automount(29803)(0xffff8803d8aeb540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_REQUESTER_CMD: major: 1, minor 0, size: 46, ioctlfd: 24: uid: 0, gid: 0 (/home/cperl)
1396530871075232024:  automount(29803)(0xffff8803d8aeb540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_REQUESTER_CMD: major: 1, minor 0, size: 46, ioctlfd: 24: uid: 12114, gid: 32771 (/home/cperl)

The rationale for this ioctl as quoted from autofs4-mount-control.txt:

In addition, to be able to reconstruct a mount tree that has busy mounts, the uid and gid of the last user that triggered the mount needs to be available because these can be used as macro substitution variables in autofs maps. They are recorded at mount request time and an operation has been added to retrieve them.

AUTOFS_DEV_IOCTL_EXPIRE_CMD

This ioctl is issued against a specific autofs filesystem and therefore you must have a valid ioctlfd to use before it can be issued.

The parameter to the ioctl contains a struct args_expire which is a bit flag in which two flags can be set:

/* Mask for expire behaviour */
#define AUTOFS_EXP_IMMEDIATE    1
#define AUTOFS_EXP_LEAVES       2

In practice, I've seen this ioctl called with no flags set, or just AUTOFS_EXP_IMMEDIATE set. Its possible that it will use AUTOFS_EXP_LEAVES under some circumstances, but none that I've hit yet.

1396470011128585542:  automount(24018)(0xffff8803d49f8aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_EXPIRE_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: how: AUTOFS_EXP_IMMEDIATE (/a/b/c)
1396470011128619772:  automount(24018)(0xffff8803d49f8aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_EXPIRE_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: how: AUTOFS_EXP_IMMEDIATE (/a/b/c)

OR

1396453375060672511:  automount(18388)(0xffff8803d6cd1540) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_EXPIRE_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: how:  (/a/b/c)
1396453375060687029:  automount(18388)(0xffff8803d6cd1540) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_EXPIRE_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: how:  (/a/b/c)

Further details from autofs4-mount-control.txt:

Issue an expire request to the kernel for an autofs mount. Typically this ioctl is called until no further expire candidates are found. The call requires an initialized struct autofs_dev_ioctl with the ioctlfd field set to the descriptor obtained from the open call. In addition an immediate expire, independent of the mount timeout, can be requested by setting the arg1 field to 1. If no expire candidates can be found the ioctl returns -1 with errno set to EAGAIN. This call causes the kernel module to check the mount corresponding to the given ioctlfd for mounts that can be expired, issues an expire request back to the daemon and waits for completion.

AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD

This ioctl is issued against a specific autofs filesystem and therefore you must have a valid ioctlfd to use before it can be issued.

The parameter to the ioctl contains a struct args_askumount. That number is set to 0 by the user space daemon and is either set 1 to indicate the mount point may be unmounted, or left at zero to indicate that the mount point is still busy.

1396453065009385966:  automount(18388)(0xffff8803d64b4aa0) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: may_umount: 0 (/foo)
1396453065009390765:  automount(18388)(0xffff8803d64b4aa0) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: may_umount: 0 (/foo)

OR

1396454296928731407:  automount(18388)(0xffff8803d5ae0040) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: may_umount: 0 (/a/b/c)
1396454296928738658:  automount(18388)(0xffff8803d5ae0040) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD: major: 1, minor 0, size: 24, ioctlfd: 34: may_umount: 1 (/a/b/c)

AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD

This ioctl can be issued with or without an ioctlfd. If anioctlfd is passed, that is used to find the mount point to operate on, otherwise you must set ioctlfd to -1 and pass a string in the argument.

The parameter to the ioctl contains a struct args_ismountpoint. This struct is a union that defines different arguments for the way in and for the way out.

On the way in, it contains a string path to inquire about. On return it fills out the devid and the magic number. Using the magic number, the user space daemon can determine what type of filesystem is mounted at that path.

For example the following are defined in include/linux/magic.h:

#define AUTOFS_SUPER_MAGIC    0x0187
#define NFS_SUPER_MAGIC       0x6969

And you can see examples of the returned values from the calls. The user space daemon uses this information to determine what action it might need to take.

Below are three examples of what it might return (autofs, nfs, or nothing). The nothing case would correspond to a directory that had been created beneath an indirect autofs mount point for ghosting:

1396454169249848027:  automount(18388)(0xffff8803d2643500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 44, ioctlfd: -1: type: 0, path: /home/cperl
1396454169249853221:  automount(18388)(0xffff8803d2643500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 44, ioctlfd: -1: devid: 200, magic: 0x187

1396454169249858444:  automount(18388)(0xffff8803d2643500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 44, ioctlfd: -1: type: 0, path: /foo
1396454169249863856:  automount(18388)(0xffff8803d2643500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 44, ioctlfd: -1: devid: 38, magic: 0x6969

1396454169257121837:  automount(18388)(0xffff8803d11db500) -> autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 47, ioctlfd: -1: type: 0, path: /home/dlobraico
1396454169257128955:  automount(18388)(0xffff8803d11db500) <- autofs_dev_ioctl: AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD: major: 1, minor 0, size: 47, ioctlfd: -1: devid: 25, magic: 0x0

Relevant quotes from autofs4-mount-control.txt:

Since we're re-implementing the control interface, a couple of other problems with the existing interface have been addressed. First, when a mount or expire operation completes a status is returned to the kernel by either a "send ready" or a "send fail" operation. The "send fail" operation of the ioctl interface could only ever send ENOENT so the re-implementation allows user space to send an actual status. Another expensive operation in user space, for those using very large maps, is discovering if a mount is present. Usually this involves scanning /proc/mounts and since it needs to be done quite often it can introduce significant overhead when there are many entries in the mount table. An operation to lookup the mount status of a mount point dentry (covered or not) has also been added.

The call requires an initialized struct autofs_dev_ioctl. There are two possible variations. Both use the path field set to the path of the mount point to check and the size field adjusted appropriately. One uses the ioctlfd field to identify a specific mount point to check while the other variation uses the path and optionally arg1 set to an autofs mount type. The call returns 1 if this is a mount point and sets arg1 to the device number of the mount and field arg2 to the relevant super block magic number (described below) or 0 if it isn't a mountpoint. In both cases the the device number (as returned by new_encode_dev()) is returned in field arg1.

SystemTap Script

Below is the SystemTap script that came out of debugging/tracing automount(8). The script started off very small and targeted to see what was happening inside the kernel, but continually grew over time.

Each time I answered one question about how automount worked I would uncover several more. This script should not be considered a general purpose tool as it is placing specific user space probes at specific lines in the automount source and therefore would certainly need to be adjusted if used in the future (i.e. line numbers will certainly change).

One particular limitation of SystemTap (or perhaps user space spacing probing, I haven't looked that hard) that I came across while investigating this is that the SystemTap script that places user space probes must be setup and running before the process that those probes are meant trace.

If you start the process first (or leave the process running), you won't see any of the user space stuff, which was ultimately quite important to understanding automount(8)'s behavior and finding the bug in its handling of negative cache entries.

/*
 * !!! DO NOT USE THIS ON A PRODUCTION MACHINE !!!
 *
 * Run this with:
 * stap -vv --vp 10101 \
 *   -d /usr/sbin/automount \
 *   -d /usr/lib64/autofs/lookup_file.so \
 *   -d /usr/lib64/autofs/parse_sun.so \
 *   -d /usr/lib64/autofs/mount_nfs.so \
 *   --ldd 
 *   -D MAXSTRINGLEN=1024 \
 *   -g \
 *   ./autofs.stp
 */

/* for selective tracing */
global trace;

/* for constant to string translation */
global _autofs_kernel_notify_types[4]
global _autofs_nsswitch_return_status[5]

probe begin {
    /** Autofs kernel to user space notify types **/
    _autofs_kernel_notify_types[0x3] = "autofs_ptype_missing_indirect";
    _autofs_kernel_notify_types[0x4] = "autofs_ptype_expire_indirect";
    _autofs_kernel_notify_types[0x5] = "autofs_ptype_missing_direct";
    _autofs_kernel_notify_types[0x6] = "autofs_ptype_expire_direct";

    /** Autofs nss return codes from include/nsswitch.h **/
    _autofs_nsswitch_return_status[0x0] = "NSS_STATUS_SUCCESS";
    _autofs_nsswitch_return_status[0x1] = "NSS_STATUS_NOTFOUND";
    _autofs_nsswitch_return_status[0x2] = "NSS_STATUS_UNAVAIL";
    _autofs_nsswitch_return_status[0x4] = "NSS_STATUS_TRYAGAIN";
    _autofs_nsswitch_return_status[0x5] = "NSS_STATUS_MAX";

}

/* Stolen from /usr/share/doc/systemtap-client-1.8/examples/process/pfiles.stp */
function task_file_handle_d_path:string (task:long, fd:long) %{ /* pure */
    struct task_struct *p = (struct task_struct *)((long)STAP_ARG_task);
    struct files_struct *files;
    char *page = NULL;
    struct file *filp;
    struct dentry *dentry;
    struct vfsmount *vfsmnt;
    char *path = NULL;

    rcu_read_lock();
    if ((files = kread(&p->files)) &&
        // We need GFP_ATOMIC since we're inside a lock so we
        // can't sleep.
        (page = (char *)__get_free_page(GFP_ATOMIC)) &&
        (filp = fcheck_files(files, STAP_ARG_fd))) {

#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,26)
        /* git commit 9d1bc601 */
        path = d_path(&filp->f_path, page, PAGE_SIZE);
#else
        dentry = kread(&filp->f_dentry);
        vfsmnt = kread(&filp->f_vfsmnt);

        if (dentry && vfsmnt) {
            path = d_path(dentry, vfsmnt, page, PAGE_SIZE);
        }
#endif
        if (path && !IS_ERR(path)) {
            snprintf(STAP_RETVALUE, MAXSTRINGLEN, "%s", path);
        }
    }
    CATCH_DEREF_FAULT();

    if (page) free_page((unsigned long)page);

    rcu_read_unlock();
%}

function log_common:string () {
    return sprintf("%d: %10s(%5d)(0x%x)", gettimeofday_ns(), execname(), pid(), task_current());
}

function autofs_dev_ioctl2str:string (cmd:long, pkt:long, outgoing:long) {
    major   = user_uint32(pkt);
    minor   = user_uint32(pkt+4);
    size    = user_uint32(pkt+8);
    ioctlfd = user_int32(pkt+12);

    task = task_current();
    common = sprintf("major: %d, minor %d, size: %d, ioctlfd: %d", major, minor, size, ioctlfd);

    cmd = cmd & 255;
    if (cmd == 0x71) {
        s = "AUTOFS_DEV_IOCTL_VERSION_CMD";
        return sprintf("%s: %s", s, common);
    }

    else if (cmd == 0x72) {
        s = "AUTOFS_DEV_IOCTL_PROTOVER_CMD";
        v = user_uint32(pkt+16);
        return sprintf("%s: %s: %d", s, common, v);
    }

    else if (cmd == 0x73) {
        s = "AUTOFS_DEV_IOCTL_PROTOSUBVER_CMD";
        v = user_uint32(pkt+16);
        return sprintf("%s: %s: %d", s, common, v);
    }

    else if (cmd == 0x74) {
        s = "AUTOFS_DEV_IOCTL_OPENMOUNT_CMD";
        sz = size - 24;
        devid = user_uint32(pkt+16);
        path  = user_string_n(pkt+24, sz);
        return sprintf("%s: %s: devid: %d, path: %s", s, common, devid, path);
    }

    else if (cmd == 0x75) {
        s = "AUTOFS_DEV_IOCTL_CLOSEMOUNT_CMD"; 
        path = task_file_handle_d_path(task, ioctlfd);
        return sprintf("%s: %s (%s)", s, common, path);
    }

    else if (cmd == 0x76) {
        s = "AUTOFS_DEV_IOCTL_READY_CMD";
        token = user_uint32(pkt+16);
        path = task_file_handle_d_path(task, ioctlfd);
        return sprintf("%s: %s: token: %d (%s)", s, common, token, path);
    }

    else if (cmd == 0x77) {
        s = "AUTOFS_DEV_IOCTL_FAIL_CMD";
        token = user_uint32(pkt+16);
        status = user_int32(pkt+20);
        path = task_file_handle_d_path(task, ioctlfd);
        return sprintf("%s: %s: token: %d, status: %d (%s)", s, common, token, status, path);
    }

    else if (cmd == 0x78) {
        s = "AUTOFS_DEV_IOCTL_SETPIPEFD_CMD";
        pipefd = user_int32(pkt+16);
        path = task_file_handle_d_path(task, ioctlfd);
        return sprintf("%s: %s: pipefd: %d (%s)", s, common, pipefd, path);
    }

    else if (cmd == 0x79) {
        s = "AUTOFS_DEV_IOCTL_CATATONIC_CMD";
        path = task_file_handle_d_path(task, ioctlfd);
        return sprintf("%s: %s (%s)", s, common, path);
    }

    else if (cmd == 0x7A) {
        s = "AUTOFS_DEV_IOCTL_TIMEOUT_CMD";
        timeout = user_uint64(pkt+16);
        path = task_file_handle_d_path(task, ioctlfd);
        return sprintf("%s: %s: timeout: %d (%s)", s, common, timeout, path);
    }
    
    else if (cmd == 0x7B) {
        s = "AUTOFS_DEV_IOCTL_REQUESTER_CMD";
        uid = user_uint32(pkt+16);
        gid = user_uint32(pkt+20);
        path = task_file_handle_d_path(task, ioctlfd);
        return sprintf("%s: %s: uid: %d, gid: %d (%s)", s, common, uid, gid, path);
    }

    else if (cmd == 0x7C) {
        s = "AUTOFS_DEV_IOCTL_EXPIRE_CMD";
        how = user_uint32(pkt+16);
        path = task_file_handle_d_path(task, ioctlfd);
        h = "";

        if (how & 1) {
            h = "AUTOFS_EXP_IMMEDIATE";
        }

        if (how & 2) {
            t = "AUTOFS_EXP_LEAVES";
            h = h == "" ? t : sprintf("%s|%s", h, t);
        }

        return sprintf("%s: %s: how: %s (%s)", s, common, h, path);
    }

    else if (cmd == 0x7D) {
        s = "AUTOFS_DEV_IOCTL_ASKUMOUNT_CMD";
        may_umount = user_uint32(pkt+16);
        path = task_file_handle_d_path(task, ioctlfd);
        return sprintf("%s: %s: may_umount: %d (%s)", s, common, may_umount, path);
    }

    /* ISMOUNTPIONT is interpretted different on the way in than out */
    else if (cmd == 0x7E) {
        s = "AUTOFS_DEV_IOCTL_ISMOUNTPOINT_CMD";
        if (!outgoing) {
            sz = size - 24;
            type = user_uint32(pkt+16);
            path = user_string_n(pkt+24, sz);
            return sprintf("%s: %s: type: %d, path: %s", s, common, type, path);
        }
        else {
            devid = user_uint32(pkt+16);
            magic = user_uint32(pkt+20);
            return sprintf("%s: %s: devid: %d, magic: 0x%X", s, common, devid, magic);
        }
    }

    else {
        return "UNKNOWN"
    }
}

function autofs_kernel_notify2str:string (typ:long) {
    return typ in _autofs_kernel_notify_types ? _autofs_kernel_notify_types[typ] : "UNKNOWN"
}

function nsswitch_status2str:string (typ:long) {
    return typ in _autofs_nsswitch_return_status ?  _autofs_nsswitch_return_status[typ] : "NSS_STATUS_UNKNOWN"
}

probe module("autofs4").function("autofs_dev_ioctl").call
{
    if (execname() == "automount") {
        printf("%s -> %s: %s\n",
            log_common(),
            "autofs_dev_ioctl",
            autofs_dev_ioctl2str($command, $u, 0));
    }
}

probe module("autofs4").function("autofs_dev_ioctl").return
{
    if (execname() == "automount") {
        printf("%s  %s: dev: %s, dir: %s, type: %s, data: %s\n",
            log_common(), "mount", dev, dir, typ, dat);
}

probe kernel.function("sys_mount").return
{
        printf("%s  %s: name: %s\n", log_common(), "umount", name);
}

probe module("autofs4").function("autofs4_fill_super").call
{
    if (execname() == "automount") {
        printf("%s -> %s: %s\n", log_common(), probefunc(), $$parms);
    }
}

probe module("autofs4").function("autofs4_fill_super").return
{
    if (execname() == "automount") {
        printf("%s  %s: %s: pipefd: %d, proto: %d: %s\n",
        log_common(),
        probefunc(),
        autofs_kernel_notify2str($pkt->v5_pkt->hdr->type),
        $sbi->pipefd,
        $pkt->v5_pkt->hdr->proto_version,
        $pkt->v5_pkt->v5_packet$$);
}

/* user space probes, requires the debuginfo package for the version of autofs
 * be installed, and for the backtraces requires debuginfo for glibc */
probe process("/usr/sbin/automount").function("handle_packet_missing_indirect").call {
    printf("%s -> %s: AP: %s: PACKET: %s\n", log_common(), probefunc(), $ap$$, $pkt$$);
}

probe process("/usr/sbin/automount").function("handle_packet_missing_indirect").return {
    printf("%s  %s: %s\n", log_common(), probefunc(), path);
    print_ubacktrace();
}

probe process("/usr/sbin/automount").function("mkdir_path").return {
    printf("%s  %s: %s\n", log_common(), "st_add_task", "ST_READMAP");
        print_ubacktrace();
    }
}

probe process("/usr/sbin/automount").function("st_add_task").return {
    t = task_current();
    if (trace[t]) {
        delete trace[t];
        printf("%s  %s: %s\n", log_common(), probefunc(), $$parms$$);
}

probe process("/usr/lib64/autofs/lookup_file.so").function("lookup_mount").return {
    printf("%s  %s: %s\n", log_common(), "lookup_ghost", $me$$);
}

probe  process("/usr/sbin/automount").function("cache_update").call,
       process("/usr/sbin/automount").function("cache_add").call {
    key = user_string($key);
    t = task_current();
    trace[t] = 1;
    printf("%s -> %s: (KEY: %s): %s\n", log_common(), probefunc(), key, $$parms$$);
    print_ubacktrace();
}

probe  process("/usr/sbin/automount").function("cache_update").return,
    process("/usr/sbin/automount").function("cache_add").return {
    t = task_current();
    if (trace[t]) {
        delete trace[t];
        printf("%s  %s: %s\n", log_common(), "lookup_prune_cache", $$parms$$);
}

probe process("/usr/sbin/automount").function("lookup_prune_cache").return {
    printf("%s  %s: %s\n", log_common(), "lookup_prune_one_cache", $me$$);
}

probe process("/usr/sbin/automount").statement("st_readmap@daemon/state.c:587") {
    printf("%s  -> %s: now: %d\n", log_common(), "st_readmap (starting do_readmap thread)", $ra->now);
}

What’s 2013 + 50? 1969, of course!

What happens when the latest CentOS 6.4/RHEL/FreeBSD GnuTLS certtool gets used to generate a TLS certificate with a 18250-day validity period? Time travel back in time, is what.

Note: This applies to the CentOS-released GnuTLS v 2.8.5. Latest source distribution is 3.2.4 Curiously enough, even in FreeBSD (by way of a counterpoint), gnutls "stable" is 2.12.23, and devel is 2.99.4_1. Professionals call this sort of a thing a "hint". FreeBSD's 2.12.23 also has the described behavior. FreeBSD's 2.99.4_1 cannot be downloaded via the usual "portinstall" mechanism - it has a known security vulnerability which hasn't been patched and portaudit does its best impersonation of The Grumpy Cat meme and says "No".

So, you'd like to use CentOS' certtool to create a self-signed certificate? Sure, no problem.

First step, create a skeleton certificate authority.

  1. $ rpm -qf which certtool gnutls-utils-2.8.5-10.el6_4.2.x86_64
  2. $ uname -a; cat /etc/redhat-release Linux buildhost 3.9.5pm1 #3 SMP PREEMPT Thu Jun 13 11:20:29 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux CentOS release 6.4 (Final)

First, CA private key:

  1. $ certtool -p --outfile ca-temp-key.pem
  2. Generating a 2048 bit RSA private key...

All good.

Next, CA signing certificate:

  1. $ certtool -s --load-privkey ca-temp-key.pem --outfile ca-test-signing.pem
  2. Generating a self signed certificate...
  3. Please enter the details of the certificate's distinguished name. Just press enter to ignore a field.
  4. Country name (2 chars): US
  5. Organization name: Jane Street
  6. Organizational unit name: Systems
  7. Locality name: US
  8. State or province name: NY
  9. Common name: test-ca.janestreet.com
  10. UID:
  11. This field should not be used in new certificates.
  12. E-mail:
  13. Enter the certificate's serial number in decimal (default: 1378834082):
  14.  
  15. Activation/Expiration time. The certificate will expire in (days): 18250
  16.  
  17. Extensions.
  18. Does the certificate belong to an authority? (y/N): y
  19. Path length constraint (decimal, -1 for no constraint):
  20. Is this a TLS web client certificate? (y/N):
  21. Is this also a TLS web server certificate? (y/N):
  22. Enter the e-mail of the subject of the certificate:
  23. Will the certificate be used to sign other certificates? (y/N): y
  24. Will the certificate be used to sign CRLs? (y/N):
  25. Will the certificate be used to sign code? (y/N):
  26. Will the certificate be used to sign OCSP requests? (y/N):
  27. Will the certificate be used for time stamping? (y/N):
  28. Enter the URI of the CRL distribution point:
  29. X.509 Certificate Information:
  30. Version: 3
  31. Serial Number (hex): 522f56a2
  32. Validity:
  33. Not Before: Tue Sep 10 17:28:04 UTC 2013
  34. Not After: Wed Dec 31 23:59:59 UTC 1969
  35. Subject: C=US,O=Jane Street,OU=Systems,L=US,ST=NY,CN=test-ca.janestreet.com
  36. Subject Public Key Algorithm: RSA
  37. Modulus (bits 2048):
  38. ce:cb:49:2c:3d:a2:e2:97:6f:71:df:43:e1:fa:b1:14
  39. 1e:b1:e5:51:13:1c:cc:7c:18:38:29:bf:08:70:f1:35
  40. d9:5d:ad:51:dc:0e:9d:f9:e6:ec:53:20:b0:04:fe:cb
  41. 0e:a6:45:27:c0:f2:cc:34:45:fd:97:2c:11:b7:86:e9
  42. 8f:9f:58:fa:90:ac:e7:9f:4e:a0:7f:8e:eb:5b:6f:15
  43. 17:8d:82:a1:30:cf:3f:37:a8:44:6a:1d:2e:3b:69:36
  44. 3e:34:c5:2a:f3:d2:2b:1f:81:ec:25:81:76:0e:1d:b9
  45. 7f:12:23:a2:af:b7:e5:9b:f7:f6:be:c4:23:65:f1:4a
  46. 63:fc:ec:92:5b:fc:f0:2c:6b:80:ee:fb:54:bf:7f:16
  47. 33:b8:26:e5:d4:f4:ec:86:18:26:3e:31:5f:66:cf:0c
  48. 81:cd:ef:c2:ec:ad:fc:26:07:2d:67:94:de:98:c2:32
  49. d4:6e:59:31:6a:35:1d:db:19:b4:a5:27:6b:94:be:8a
  50. 77:2f:8c:7c:6b:cb:af:71:62:fa:7a:41:e5:da:63:5b
  51. 95:d1:05:62:56:33:07:67:8c:bf:3f:64:11:dc:84:69
  52. e6:f2:b7:f2:6c:a0:e1:36:fc:e3:00:c0:11:26:dd:44
  53. f0:ca:02:97:67:70:15:85:34:e9:ca:d6:60:a4:37:8b
  54. Exponent (bits 24):
  55. 01:00:01
  56. Extensions:
  57. Basic Constraints (critical):
  58. Certificate Authority (CA): TRUE
  59. Key Usage (critical):
  60. Certificate signing.
  61. Subject Key Identifier (not critical):
  62. d7dfcb520769255a65638e6dc3b899648dd4e447
  63. Other Information:
  64. Public Key Id:
  65. d7dfcb520769255a65638e6dc3b899648dd4e447

And here's the crux of the issue:

  1. Validity:
  2. Not Before: Tue Sep 10 17:28:04 UTC 2013
  3. Not After: Wed Dec 31 23:59:59 UTC 1969

Not after 1969? Yeah... Let me get my flux capacitor and a DeLorean and get back to you.

FreeBSD's 2.12.3:

  1. $ certtool -p --outfile ca-temp-key.pem
  2. Generating a 2432 bit RSA private key...
  3. $certtool -s --load-privkey=ca-temp-key.pem --outfile ca-test-signing.pem
  4. Generating a self signed certificate...
  5. Please enter the details of the certificate's distinguished name. Just press enter to ignore a field.
  6. Country name (2 chars): US
  7. Organization name: Jane Street
  8. Organizational unit name: Systems
  9. Locality name: New York
  10. State or province name: NY
  11. Common name: test-ca.janestreet.com
  12. UID:
  13. This field should not be used in new certificates.
  14. E-mail:
  15. Enter the certificate's serial number in decimal (default: 1378834456):
  16.  
  17. Activation/Expiration time.
  18. The certificate will expire in (days): 18250
  19.  
  20. Extensions.
  21. Does the certificate belong to an authority? (y/N): y
  22. Path length constraint (decimal, -1 for no constraint):
  23. Is this a TLS web client certificate? (y/N):<br />
  24. Will the certificate be used for IPsec IKE operations? (y/N):
  25. Is this also a TLS web server certificate? (y/N):
  26. Enter the e-mail of the subject of the certificate:
  27. Will the certificate be used to sign other certificates? (y/N): y
  28. Will the certificate be used to sign CRLs? (y/N):
  29. Will the certificate be used to sign code? (y/N):
  30. Will the certificate be used to sign OCSP requests? (y/N):
  31. Will the certificate be used for time stamping? (y/N):
  32. Enter the URI of the CRL distribution point:
  33. X.509 Certificate Information:
  34. Version: 3
  35. Serial Number (hex): 522f5818
  36. Validity:
  37. Not Before: Tue Sep 10 17:34:17 UTC 2013
  38. Not After: Thu Jan 01 00:00:00 UTC 1970
  39. Subject: C=US,O=Jane Street,OU=Systems,L=New York,ST=NY,CN=test-ca.janestreet.com
  40. Subject Public Key Algorithm: RSA
  41. Certificate Security Level: Normal
  42. Modulus (bits 2432):
  43. 00:ea:3e:bf:c2:bb:55:90:4f:e1:d3:da:2b:3e:b2:81
  44. 64:97:8f:db:70:27:ad:94:ae:1d:dd:ab:28:73:6e:60
  45. 2a:39:8a:c0:1b:2c:ae:1e:f7:ce:c5:dc:01:8a:9e:31
  46. 15:e3:e5:9c:67:63:05:ec:24:6b:0c:74:7d:6b:ae:bc
  47. ba:8b:4c:fd:b8:2b:37:74:f1:10:39:a1:c7:f3:fb:dc
  48. b8:09:80:2f:a5:8b:79:13:66:e0:8b:93:56:3b:3b:dd
  49. fb:6d:78:49:cf:c6:5c:57:f0:5d:1f:2d:73:98:b2:eb
  50. 1e:10:be:0e:e7:de:2b:9b:d2:88:e0:49:34:a9:30:28
  51. ad:4c:60:8c:11:50:bb:25:c2:e5:88:0a:4d:6a:84:a9
  52. 48:2e:07:ed:dc:e0:04:9c:bd:90:2b:fb:10:92:ca:8d
  53. cc:51:4f:f8:fa:d2:51:a4:12:50:75:e6:e5:87:f2:67
  54. 5f:17:4e:12:63:4c:aa:70:2e:20:b9:07:63:1d:41:89
  55. f4:f7:7f:c7:91:55:05:49:94:ff:7f:1b:dc:23:59:08
  56. 15:c0:9f:13:c7:90:bf:c0:c1:8f:02:9b:6f:28:71:e4
  57. 1e:90:0b:1f:7b:f6:4b:1a:2d:1f:24:d4:d4:6d:11:3a
  58. 3d:e2:7e:41:d1:0d:1c:88:da:db:29:5a:1d:4d:62:c3
  59. ac:c6:dc:2c:e9:d9:7d:3d:fc:af:3a:10:fe:3a:b7:bc
  60. 8a:f1:ed:9b:85:89:b6:e2:e8:0c:36:df:55:c6:60:7a
  61. 1c:1c:3d:54:7f:d7:d5:ea:1c:0d:d1:0c:c6:ef:99:cf
  62. 5d
  63. Exponent (bits 24):
  64. 01:00:01
  65. Extensions:
  66. Basic Constraints (critical):
  67. Certificate Authority (CA): TRUE
  68. Key Usage (critical):
  69. Certificate signing.
  70. Subject Key Identifier (not critical):
  71. 6ec09c8592ba3904a301051b60223a5e50cad333
  72. Other Information:
  73. Public Key Id:
  74. 6ec09c8592ba3904a301051b60223a5e50cad333
  75. Is the above information ok? (y/N): n

GnuTLS 3.2.4 (compiled from source):

  1. gnutls-3.2.4/src$ ./certtool -p --outfile ca-temp-key.pem
  2. Generating a 2432 bit RSA private key...
  3.  
  4. gnutls-3.2.4/src$ ./certtool -s --load-privkey=ca-temp-key.pem --outfile ca-test-signing.pem
  5. Generating a self signed certificate...
  6. Please enter the details of the certificate's distinguished name. Just press enter to ignore a field.
  7. Common name: test-ca.janestreet.com
  8. UID:
  9. Organizational unit name: Systems
  10. Organization name: Jane Street
  11. Locality name: New York
  12. State or province name: NY
  13. Country name (2 chars): US
  14. Enter the subject's domain component (DC): janestreet.com
  15. Enter the subject's domain component (DC):
  16. This field should not be used in new certificates.
  17. E-mail:
  18. Enter the certificate's serial number in decimal (default: 1378836930):</p>
  19.  
  20. <p>Activation/Expiration time.
  21. The certificate will expire in (days): 18250</p>
  22.  
  23. <p>Extensions.
  24. Does the certificate belong to an authority? (y/N): y
  25. Path length constraint (decimal, -1 for no constraint):
  26. Is this a TLS web client certificate? (y/N):
  27. Will the certificate be used for IPsec IKE operations? (y/N):
  28. Is this a TLS web server certificate? (y/N):
  29. Enter a dnsName of the subject of the certificate:<br />
  30. Enter a URI of the subject of the certificate:
  31. Enter the IP address of the subject of the certificate:
  32. Enter the e-mail of the subject of the certificate:
  33. Will the certificate be used to sign other certificates? (y/N): y
  34. Will the certificate be used to sign CRLs? (y/N):
  35. Will the certificate be used to sign code? (y/N):
  36. Will the certificate be used to sign OCSP requests? (y/N):
  37. Will the certificate be used for time stamping? (y/N):
  38. Enter the URI of the CRL distribution point:
  39. X.509 Certificate Information:
  40. Version: 3
  41. Serial Number (hex): 522f61c2
  42. Validity:
  43. Not Before: Tue Sep 10 18:15:31 UTC 2013
  44. Not After: Wed Aug 29 18:15:31 UTC 2063
  45. Subject: CN=test-ca.janestreet.com,OU=Systems,O=Jane Street,L=New York,ST=NY,C=US,DC=janestreet.com
  46. Subject Public Key Algorithm: RSA
  47. Algorithm Security Level: Normal (2432 bits)
  48. Modulus (bits 2432):
  49. 00:b9:f0:d3:81:b1:d6:09:71:45:47:e6:66:ac:41:0b
  50. 93:93:b3:68:28:60:08:5e:e4:ba:9e:43:5f:b5:05:55
  51. 24:f0:34:ab:11:8a:fe:74:9e:d2:f8:e4:ab:c6:5c:f3
  52. 2c:f9:0b:b4:4c:26:b9:3d:58:3b:16:73:85:28:95:13
  53. ec:7d:7c:8b:38:c8:fa:08:64:de:5e:f5:9a:f5:70:1c
  54. cb:d4:d0:4a:e7:ad:5b:20:89:cc:29:91:c0:58:3b:dd
  55. 38:f8:6f:56:f5:9b:25:05:44:ae:f9:9d:67:0b:59:96
  56. b7:da:4c:24:37:84:a5:f6:8f:32:5b:ae:e3:e8:ac:d2
  57. 1b:7d:b4:67:42:f7:60:95:30:e4:8e:fa:4d:db:5b:65
  58. 4f:f3:04:ca:94:74:d0:b2:42:20:8f:be:22:1b:77:34
  59. 34:00:7d:0f:1a:7f:33:5a:56:b7:c6:88:9b:68:5b:7d
  60. 84:d6:c4:c2:3e:8a:b5:40:6e:35:64:10:46:b1:28:ac
  61. 8c:1f:2c:55:98:14:96:9c:e9:17:93:d3:28:30:04:8e
  62. 7d:9e:ae:55:77:13:c5:7b:1b:cd:e1:d9:85:62:66:ad
  63. 64:14:11:f3:2a:a4:f2:9a:88:36:d7:b9:7d:3f:c7:8f
  64. 45:7c:b9:7d:11:73:da:c3:36:5e:12:e3:8a:8f:94:c1
  65. 4e:33:be:e6:2c:49:d4:cf:39:d8:38:7c:fd:c5:7d:06
  66. 1d:2d:87:8e:ea:7e:80:f7:aa:25:bf:e8:a7:0f:17:c7
  67. 12:e7:21:05:aa:3a:0c:9a:a8:1c:86:98:fc:ea:30:40
  68. 29
  69. Exponent (bits 24):
  70. 01:00:01
  71. Extensions:
  72. Basic Constraints (critical):
  73. Certificate Authority (CA): TRUE
  74. Key Usage (critical):
  75. Certificate signing.
  76. Subject Key Identifier (not critical):
  77. 683cf71bc67af324655d661ecd7043a0707e3ee7
  78. Other Information:
  79. Public Key Id:
  80. 683cf71bc67af324655d661ecd7043a0707e3ee7
  81. Public key's random art:
  82. +--[ RSA 2432]----+
  83. | . . .+o.|
  84. | + . +o|
  85. | o . .*|
  86. | . . o. =.|
  87. | = S oo...|
  88. | . o o o + |
  89. | * . E |
  90. | oo= |
  91. | ...o. |
  92. +-----------------+
  93. Is the above information ok? (y/N):

Lovely, that.

On CentOS 6.4, rpm --whatdepends and rpm --requires show very few (at least in our general install) direct dependencies.

Parallel to that, it is not certain why GnuTLS in CentOS/RHEL and FreeBSD (there is also evidence that Debian and Ubuntu are in a similar version paths) use an old(er) version of GnuTLS. There is a distinct possibility of an ABI change since there is a major version number jump.

4