Chapter 12. Reflection

Go provides a mechanism to do the following on variables without knowing their types at compile time:

This mechanism is called reflection. Reflection also lets us treat types themselves as first-class values.

This chapter covers Go's reflection features on how they increase the expressiveness of the language, and in particular how they are crucial to the implementation of two important APIs:

Reflection is also essential to the template mechanism provided by the text/template and html/template packages as seen in Section 4.6. However, reflection is complex to reason about and not for casual use, so although these packages are implemented using reflection, they do not expose reflection in their own APIs.

Why Reflection?

Sometimes we need to write a function capable of dealing uniformly with values of types, which have one of the following traits:

A familiar example is the formatting logic within fmt.Fprintf, which can usefully print an arbitrary value of any type, even a user-defined one. Let's try to implement a function like it using what we know already. For simplicity, our function will accept one argument and will return the result as a string like fmt.Sprint does, so we'll call it Sprint.

We start with a type switch that tests whether the argument defines a String method and call it if so. We then add switch cases that test the values' dynamic type against each of the basic types: string, int, bool, etc., and perform the appropriate formatting operation in each case.

func Sprint(x interface{}) string {
    type stringer interface {
        String() string
    }
    switch x := x.(type) {
    case stringer:
        return x.String()
    case string:
        return x
    case int:
        return strconv.Itoa(x)
    // ...similar cases for int16, uint32, and so on...
    case bool:
    if x {
        return "true"
        }
        return "false"
    default:
        // array, chan, func, map, pointer, slice, struct
        return "???"
    }
}

But how do we deal with other types?

Without a way to inspect the representation of values of unknown types, we quickly get stuck. What we need is reflection.

reflect.Type and reflect.Value

Reflection is provided by the reflect package. It defines two important types, Type and Value. A Type represents a Go type. It is an interface with many methods for discriminating among types and inspecting their components, like the fields of a struct or the parameters of a function. The sole implementation of reflect.Type is the type descriptor (Section 7.5), the same entity that identifies the dynamic type of an interface value.

The reflect.TypeOf function accepts any interface{} and returns its dynamic type as a reflect.Type:

t := reflect.TypeOf(3)  // a reflect.Type
fmt.Println(t.String()) // "int"
fmt.Println(t)          // "int"

The TypeOf(3) call above assigns the value 3 to the interface{} parameter. Recall from Section 7.5 that an assignment from a concrete value to an interface type performs an implicit interface conversion, which creates an interface value consisting of two components:

Because reflect.TypeOf returns an interface value's dynamic type, it always returns a concrete type. For example, the code below prints "*os.File", not "io.Writer". Later, we will see that reflect.Type is also capable of representing interface types.

var w io.Writer = os.Stdout
fmt.Println(reflect.TypeOf(w)) // "*os.File"

Notice that reflect.Type satisfies fmt.Stringer. Because printing the dynamic type of an interface value is useful for debugging and logging, fmt.Printf provides a shorthand, %T, that uses reflect.TypeOf internally:

fmt.Printf("%T\n", 3) // "int"

The other important type in the reflect package is Value. A reflect.Value can hold a value of any type. The reflect.ValueOf function accepts any interface{} and returns a reflect.Value containing the interface's dynamic value. As with reflect.TypeOf, the results of reflect.ValueOf are always concrete, but a reflect.Value can also hold interface values.

v := reflect.ValueOf(3) // a reflect.Value
fmt.Println(v)          // "3"
fmt.Printf("%v\n", v)   // "3"
fmt.Println(v.String()) // NOTE: "<int Value>"

Like reflect.Type, reflect.Value also satisfies fmt.Stringer, but unless the Value holds a string, the result of the String method reveals only the type. Instead, use the fmt package's %v verb, which treats reflect.Values specially.

Calling the Type method on a Value returns its type as a reflect.Type:

t := v.Type()            // a reflect.Type
fmt.Println(t.String())  // "int"

The inverse operation to reflect.ValueOf is the reflect.Value.Interface method. It returns an interface{} holding the same concrete value as the reflect.Value:

v := reflect.ValueOf(3) // a reflect.Value
x := v.Interface()      // an interface{}
i := x.(int)            // an int
fmt.Printf("%d\n", i)   // "3"

A reflect.Value and an interface{} (empty interface) can both hold arbitrary values. What is the difference between them?

Let's use the methods of a Value for our second attempt at a general formatting function called format.Any.

Instead of a type switch, we use reflect.Value's Kind method to discriminate the cases. Although there are infinitely many types, there are only a finite number of kinds of type:

gopl.io/ch12/format/format.go

package format

import (
    "reflect"
    "strconv"
)

// Any formats any value as a string.
func Any(value interface{}) string {
    return formatAtom(reflect.ValueOf(value))
}

// formatAtom formats a value without inspecting its internal structure.
func formatAtom(v reflect.Value) string {
    switch v.Kind() {
    case reflect.Invalid:
        return "invalid"
    case reflect.Int, reflect.Int8, reflect.Int16,
        reflect.Int32, reflect.Int64:
        return strconv.FormatInt(v.Int(), 10)
    case reflect.Uint, reflect.Uint8, reflect.Uint16,
        reflect.Uint32, reflect.Uint64, reflect.Uintptr:
        return strconv.FormatUint(v.Uint(), 10)
    // ...floating-point and complex cases omitted for brevity...
    case reflect.Bool:
        return strconv.FormatBool(v.Bool())
    case reflect.String:
        return strconv.Quote(v.String())
    case reflect.Chan, reflect.Func, reflect.Ptr, reflect.Slice, reflect.Map:
        return v.Type().String() + " 0x" +
            strconv.FormatUint(uint64(v.Pointer()), 16)
    default: // reflect.Array, reflect.Struct, reflect.Interface
        return v.Type().String() + " value"
    }
}

This function treats each value as an indivisible thing with no internal structure (hence formatAtom). For aggregate types (structs and arrays) and interfaces it prints only the type of the value, and for reference types (channels, functions, pointers, slices, and maps), it prints the type and the reference address in hexadecimal. This is less than ideal but still a major improvement. Since Kind is concerned only with the underlying representation, format.Any also works for named types. For example:

var x int64 = 1
var d time.Duration = 1 * time.Nanosecond
fmt.Println(format.Any(x))                  // "1"
fmt.Println(format.Any(d))                  // "1"
fmt.Println(format.Any([]int64{x}))         // "[]int64 0x8202b87b0"
fmt.Println(format.Any([]time.Duration{d})) // "[]time.Duration 0x8202b87e0"

Display, a Recursive Value Printer

This section is about how to improve the display of composite types. Rather than try to copy fmt.Sprint exactly, we'll build a debugging utility function called Display that, given an arbitrarily complex value x, prints the complete structure of that value, labeling each element with the path by which it was found.

e, _ := eval.Parse("sqrt(A / pi)")
Display("e", e)

In the call above, the argument to Display is a syntax tree from the expression evaluator in Section 7.9. eval.Parse is from gopl.io/ch7/eval/parse.go.

The output of Display is shown below:

Display e (eval.call):
e.fn = "sqrt"
e.args[0].type = eval.binary
e.args[0].value.op = 47
e.args[0].value.x.type = eval.Var
e.args[0].value.x.value = "A"
e.args[0].value.y.type = eval.Var
e.args[0].value.y.value = "pi"

Where possible, you should avoid exposing reflection in the API of a package. We'll define an unexported function display to do the real work of the recursion, and export Display, a simple wrapper around it that accepts an interface{} parameter:

gopl.io/ch12/display/display.go

func Display(name string, x interface{}) {
    fmt.Printf("Display %s (%T):\n", name, x)
    display(name, reflect.ValueOf(x))
}

display uses the formatAtom function defined earlier to print elementary values, e.g. basic types, functions, and channels, but uses the methods of reflect.Value to recursively display each component of a more complex type. As the recursion descends, the path string, which initially describes the starting value (for instance, "e"), will be augmented to indicate how we reached the current value (for instance, "e.args[0].value").

func display(path string, v reflect.Value) {
    switch v.Kind() {
    case reflect.Invalid:
        fmt.Printf("%s = invalid\n", path)
    case reflect.Slice, reflect.Array:
        for i := 0; i < v.Len(); i++ {
            display(fmt.Sprintf("%s[%d]", path, i), v.Index(i))
        }
    case reflect.Struct:
        for i := 0; i < v.NumField(); i++ {
            fieldPath := fmt.Sprintf("%s.%s", path, v.Type().Field(i).Name)
            display(fieldPath, v.Field(i))
        }
    case reflect.Map:
        for _, key := range v.MapKeys() {
            display(fmt.Sprintf("%s[%s]", path,
                formatAtom(key)), v.MapIndex(key))
        }
    case reflect.Ptr:
        if v.IsNil() {
            fmt.Printf("%s = nil\n", path)
        } else {
            display(fmt.Sprintf("(*%s)", path), v.Elem())
        }
    case reflect.Interface:
        if v.IsNil() {
            fmt.Printf("%s = nil\n", path)
        } else {
            fmt.Printf("%s.type = %s\n", path, v.Elem().Type())
            display(path+".value", v.Elem())
        }
    default: // basic types, channels, funcs
        fmt.Printf("%s = %s\n", path, formatAtom(v))
    }
}

In the above code:

Let's use Display on some example types. The Movie type below is a slight variation on the one in Section 4.5:

type Movie struct {
    Title, Subtitle string
    Year            int
    Color           bool
    Actor           map[string]string
    Oscars          []string
    Sequel          *string
}

Let's declare a value of this type and see what Display does with it:

strangelove := Movie{
    Title:    "Dr. Strangelove",
    Subtitle: "How I Learned to Stop Worrying and Love the Bomb",
    Year:     1964,
    Color:    false,
    Actor: map[string]string{
        "Dr. Strangelove":            "Peter Sellers",
        "Grp. Capt. Lionel Mandrake": "Peter Sellers",
        "Pres. Merkin Muffley":       "Peter Sellers",
        "Gen. Buck Turgidson":        "George C. Scott",
        "Brig. Gen. Jack D. Ripper":  "Sterling Hayden",
        `Maj. T.J. "King" Kong`:      "Slim Pickens",
    },
    Oscars: []string{
        "Best Actor (Nomin.)",
        "Best Adapted Screenplay (Nomin.)",
        "Best Director (Nomin.)",
        "Best Picture (Nomin.)",
    },
}

The call Display("strangelove", strangelove) prints:

Display strangelove (display.Movie):
strangelove.Title = "Dr. Strangelove"
strangelove.Subtitle = "How I Learned to Stop Worrying and Love the Bomb"
strangelove.Year = 1964
strangelove.Color = false
strangelove.Actor["Gen. Buck Turgidson"] = "George C. Scott"
strangelove.Actor["Brig. Gen. Jack D. Ripper"] = "Sterling Hayden"
strangelove.Actor["Maj. T.J. \"King\" Kong"] = "Slim Pickens"
strangelove.Actor["Dr. Strangelove"] = "Peter Sellers"
strangelove.Actor["Grp. Capt. Lionel Mandrake"] = "Peter Sellers"
strangelove.Actor["Pres. Merkin Muffley"] = "Peter Sellers"
strangelove.Oscars[0] = "Best Actor (Nomin.)"
strangelove.Oscars[1] = "Best Adapted Screenplay (Nomin.)"
strangelove.Oscars[2] = "Best Director (Nomin.)"
strangelove.Oscars[3] = "Best Picture (Nomin.)"
strangelove.Sequel = nil

We can use Display to display the internals of library types, such as *os.File:

Display("os.Stderr", os.Stderr)
// Output:
// Display os.Stderr (*os.File):
// (*(*os.Stderr).file).fd = 2
// (*(*os.Stderr).file).name = "/dev/stderr"
// (*(*os.Stderr).file).nepipe = 0

Notice that even unexported fields are visible to reflection. Beware that the particular output of this example may vary across platforms and may change over time as libraries evolve. (Those fields are private for a reason.)

We can even apply Display to a reflect.Value and watch it traverse the internal representation of the type descriptor for *os.File. The output of the call Display("rV", reflect.ValueOf(os.Stderr)) is shown below (though your output may vary:

Display rV (reflect.Value):
(*rV.typ).size = 8
(*rV.typ).hash = 871609668
(*rV.typ).align = 8
(*rV.typ).fieldAlign = 8
(*rV.typ).kind = 22
(*(*rV.typ).string) = "*os.File"
(*(*(*rV.typ).uncommonType).methods[0].name) = "Chdir"
(*(*(*(*rV.typ).uncommonType).methods[0].mtyp).string) = "func() error"
(*(*(*(*rV.typ).uncommonType).methods[0].typ).string) = "func(*os.File) error"
...

Observe the difference between the following two examples:

var i interface{} = 3
Display("i", i)
// Output:
// Display i (int):
// i = 3

Display("&i", &i)
// Output:
// Display &i (*interface {}):
// (*&i).type = int
// (*&i).value = 3

As currently implemented, Display will never terminate if it encounters a cycle in the object graph, such as this linked list that eats its own tail:

// a struct that points to itself
type Cycle struct{ Value int; Tail *Cycle }
var c Cycle
c = Cycle{42, &c}
Display("c", c)

Display prints this ever-growing expansion:

Display c (display.Cycle):
c.Value = 42
(*c.Tail).Value = 42
(*(*c.Tail).Tail).Value = 42
(*(*(*c.Tail).Tail).Tail).Value = 42
...ad infinitum...

Many Go programs contain at least some cyclic data. Making Display robust against such cycles is tricky, requiring additional bookkeeping to record the set of references that have been followed so far (which is also costly). A general solution requires unsafe language features, as we will see in Section 13.3.

Cycles pose less of a problem for fmt.Sprint because it rarely tries to print the complete structure. For example, when it encounters a pointer, it breaks the recursion by printing the pointer's numeric value. It can get stuck trying to print a slice or map that contains itself as an element, but such rare cases do not warrant the considerable extra trouble of handling cycles.

Example: Encoding S-Expressions

(skipped) [p338-341]

Setting Variables with reflect.Value

In previous sections, reflection has only interpreted values in our program. The point of this section is to change them.

Recall that some Go expressions like x, x.f[1], and *p denote variables, but others like x + 1 and f(2) do not. A variable is an addressable storage location that contains a value, and its value may be updated through that address.

A similar distinction applies to reflect.Values. Some are addressable; others are not. Consider the following declarations:

x := 2                     // value   type   variable?
a := reflect.ValueOf(2)    // 2       int    no
b := reflect.ValueOf(x)    // 2       int    no
c := reflect.ValueOf(&x)   // &x      *int   no
d := c.Elem()              // 2       int    yes (x)

We can ask a reflect.Value whether it is addressable through its CanAddr method:

fmt.Println(a.CanAddr()) // "false"
fmt.Println(b.CanAddr()) // "false"
fmt.Println(c.CanAddr()) // "false"
fmt.Println(d.CanAddr()) // "true"

We obtain an addressable reflect.Value whenever we indirect through a pointer, even if we started from a non-addressable Value. All the usual rules for addressability have analogs for reflection. For example, since the slice indexing expression e[i] implicitly follows a pointer, it is addressable even if the expression e is not. By analogy, reflect.ValueOf(e).Index(i) refers to a variable, and is thus addressable even if reflect.ValueOf(e) is not.

To recover the variable from an addressable reflect.Value requires three steps:

  1. We call Addr(), which returns a Value holding a pointer to the variable.
  2. We call Interface() on this Value, which returns an interface{} value containing the pointer.
  3. If we know the type of the variable, we can use a type assertion to retrieve the contents of the interface as an ordinary pointer. We can then update the variable through the pointer:
x := 2
d := reflect.ValueOf(&x).Elem()   // d refers to the variable x
px := d.Addr().Interface().(*int) // px := &x
*px = 3                           // x = 3
fmt.Println(x)                    // "3"

Alternatively, we can update the variable referred to by an addressable reflect.Value directly without using a pointer, by calling the reflect.Value.Set method:

d.Set(reflect.ValueOf(4))
fmt.Println(x) // "4"

The same assignability checks, which are ordinarily performed by the compiler, are done at run time by the Set methods. In the above code, the variable and the value both have type int, but if the variable had been an int64, the program would panic, so it's crucial to make sure the value is assignable to the type of the variable:

d.Set(reflect.ValueOf(int64(5))) // panic: int64 is not assignable to int

Calling Set on a non-addressable reflect.Value panics too:

x := 2
b := reflect.ValueOf(x)
b.Set(reflect.ValueOf(3)) // panic: Set using unaddressable value

There are variants of Set specialized for certain groups of basic types: SetInt, SetUint, SetString, SetFloat, and so on:

d := reflect.ValueOf(&x).Elem()
d.SetInt(3)
fmt.Println(x) // "3"

These methods are more forgiving. For example, SetInt will succeed so long as the variable's type is some kind of signed integer, or even a named type whose underlying type is a signed integer, and if the value is too large it will be quietly truncated to fit. But tread carefully: calling SetInt on a reflect.Value that refers to an interface{} variable will panic, even though Set would succeed.

x := 1
rx := reflect.ValueOf(&x).Elem()
rx.SetInt(2)                     // OK, x = 2
rx.Set(reflect.ValueOf(3))       // OK, x = 3
rx.SetString("hello")            // panic: string is not assignable to int
rx.Set(reflect.ValueOf("hello")) // panic: string is not assignable to int
var y interface{}
ry := reflect.ValueOf(&y).Elem()
ry.SetInt(2)                     // panic: SetInt called on interface Value
ry.Set(reflect.ValueOf(3))       // OK, y = int(3)
ry.SetString("hello")            // panic: SetString called on interface Value
ry.Set(reflect.ValueOf("hello")) // OK, y = "hello"

When we applied Display to os.Stdout, we found that reflection can read the values of unexported struct fields that are inaccessible according to the usual rules of the language, like the fd int field of an os.File struct on a Unix-like platform. However, reflection cannot update such values:

stdout := reflect.ValueOf(os.Stdout).Elem() // *os.Stdout, an os.File var
fmt.Println(stdout.Type())                  // "os.File"
fd := stdout.FieldByName("fd")
fmt.Println(fd.Int()) // "1"
fd.SetInt(2)          // panic: unexported field

An addressable reflect.Value records whether it was obtained by traversing an unexported struct field and, if so, disallows modification. Consequently, CanAddr is not usually the right check to use before setting a variable. The related method CanSet reports whether a reflect.Value is addressable and settable:

fmt.Println(fd.CanAddr(), fd.CanSet()) // "true false"