Working with files

Warning

This tutorial was written for Bonobo 0.5, while the current stable version is Bonobo 0.6.

Please be aware that some things changed.

A summary of changes is available in the migration guide from 0.5 to 0.6.

Bonobo would be pointless if the aim was just to uppercase small lists of strings.

In fact, Bonobo should not be used if you don’t expect any gain from parallelization/distribution of tasks.

Some background…

Let’s take the following graph:

digraph { rankdir = LR; BEGIN [shape="point"]; BEGIN -> "A" -> "B" -> "C"; "B" -> "D"; }

When run, the execution strategy wraps every component in a thread (assuming you’re using the default bonobo.strategies.ThreadPoolExecutorStrategy).

Bonobo will send each line of data in the input node’s thread (here, A). Now, each time A yields or returns something, it will be pushed on B input queue.Queue, and will be consumed by B’s thread. Meanwhile, A will continue to run, if it’s not done.

When there is more than one node linked as the output of a node (for example, with B, C, and D), the same thing happens except that each result coming out of B will be sent to both on C and D input queue.Queue.

One thing to keep in mind here is that as the objects are passed from thread to thread, you need to write “pure” transformations (see Best Practices).

You generally don’t have to think about it. Just be aware that your nodes will run in parallel, and don’t worry too much about nodes running blocking operations, as they will run in parallel. As soon as a line of output is ready, the next nodes will start consuming it.

That being said, let’s manipulate some files.

Reading a file

There are a few component builders available in Bonobo that let you read from (or write to) files.

All readers work the same way. They need a filesystem to work with, and open a “path” they will read from.

We’ll use a text file that was generated using Bonobo from the “liste-des-cafes-a-un-euro” dataset made available by Mairie de Paris under the Open Database License (ODbL). You can explore the original dataset.

You’ll need the “coffeeshops.txt” example dataset, available in Bonobo’s repository:

$ curl https://raw.githubusercontent.com/python-bonobo/bonobo/master/bonobo/examples/datasets/coffeeshops.txt > `python3 -c 'import bonobo; print(bonobo.get_examples_path("datasets/coffeeshops.txt"))'`

Note

The “example dataset download” step will be easier in the future.

https://github.com/python-bonobo/bonobo/issues/134

You can also run this example as a module (but you’ll still need the dataset…):

$ bonobo run -m bonobo.examples.tutorials.tut02e01_read

Note

Don’t focus too much on the get_services() function for now. It is required, with this exact name, but we’ll get into that in a few minutes.

Writing to files

Let’s split this file’s each lines on the first comma and store a json file mapping coffee names to their addresses.

Here are, like the readers, the classes available to write files

Let’s write a first implementation:

(run it with bonobo run -m bonobo.examples.tutorials.tut02e02_write or bonobo run myfile.py)

If you read the output file, you’ll see it misses the “map” part of the problem.

Let’s extend bonobo.io.JsonWriter to finish the job:

(run it with bonobo run -m bonobo.examples.tutorials.tut02e03_writeasmap or bonobo run myfile.py)

It should produce a nice map.

We favored a bit hackish solution here instead of constructing a map in python then passing the whole to json.dumps() because we want to work with streams, if you have to construct the whole data structure in python, you’ll loose a lot of bonobo’s benefits.