Overview — Anaconda Enterprise 4.4 documentation

Anaconda Enterprise: Documentation

# Overview

## Motivation and Example

Dask.delayed lets you parallelize custom code. It is useful whenever your problem doesn’t quite fit a high-level parallel object like dask.array or dask.dataframe but could still benefit from parallelism. Dask.delayed works by delaying your function evaluations and putting them into a dask graph. Dask.delayed is useful when wrapping existing code or when handling non-standard problems.

Consider the following example:

    def inc(x):
        return x + 1

    def double(x):
        return x + 2

    def add(x, y):
        return x + y

    data = [1, 2, 3, 4, 5]

    output = []
    for x in data:
        a = inc(x)
        b = double(x)
        c = add(a, b)
        output.append(c)

    total = sum(output)

As written this code runs sequentially in a single thread. However we see that a lot of this could be executed in parallel. We use the `delayed` function to parallelize this code by turning it into a dask graph. We slightly modify our code by wrapping functions in `delayed`. This delays the execution of the function and generates a dask graph instead.

    from dask import delayed

    output = []
    for x in data:
        a = delayed(inc)(x)
        b = delayed(double)(x)
        c = delayed(add)(a, b)
        output.append(c)

    total = delayed(sum)(output)

We used the `delayed` function to wrap the function calls that we want to turn into tasks. None of the `inc`, `double`, `add` or `sum` calls have happened yet, instead the object `total` is a `Delayed` result that contains a task graph of the entire computation. Looking at the graph we see clear opportunities for parallel execution. The dask schedulers will exploit this parallelism, generally improving performance. (although not in this example, because these functions are already very small and fast.)

    total.visualize()  # see image to the right

We can now compute this lazy result to execute the graph in parallel:

    >>> total.compute()
    45

## Delayed Function

The `dask.delayed` interface consists of one function, `delayed`:

- `delayed` wraps functions

  > Wraps functions. Can be used as a decorator, or around function calls directly (i.e. `delayed(foo)(a, b, c)`). Outputs from functions wrapped in `delayed` are proxy objects of type `Delayed` that contain a graph of all operations done to get to this result.

- `delayed` wraps objects

  > Wraps objects. Used to create `Delayed` proxies directly.

`Delayed` objects can be thought of as representing a key in the dask. A `Delayed` supports most python operations, each of which creates another `Delayed` representing the result:

- Most operators (`*`, `-`, and so on)
- Item access and slicing (`a[0]`)
- Attribute access (`a.size`)
- Method calls (`a.index(0)`)

Operations that aren’t supported include:

- Mutating operators (`a += 1`)
- Mutating magics such as `__setitem__`/`__setattr__` (`a[0] = 1`, `a.foo = 1`)
- Iteration. (`for i in a: ...`)
- Use as a predicate (`if a: ...`)

The last two points in particular mean that `Delayed` objects cannot be used for control flow, meaning that no `Delayed` can appear in a loop or if statement. In other words you can’t iterate over a `Delayed` object, or use it as part of a condition in an if statement, but `Delayed` object can be used in a body of a loop or if statement (i.e. the example above is fine, but if `data` was a `Delayed` object it wouldn’t be). Even with this limitation, many workflows can easily be parallelized.