Nodes

Module

bonobo.nodes

class FixedWindow(*args, **kwargs)[source]

Bases: bonobo.config.configurables.Configurable

Transformation factory to create fixed windows of inputs, as lists.

For example, if the input is successively 1, 2, 3, 4, etc. and you pass it through a FixedWindow(2), you’ll get lists of elements 2 by 2: [1, 2], [3, 4], …

Parameters

length (int) –

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

buffer
length
class Limit(*args, **kwargs)[source]

Bases: bonobo.config.configurables.Configurable

Creates a Limit() node, that will only let go through the first n rows (defined by the limit option), unmodified.

limit

Number of rows to let go through.

TODO: simplify into a closure building factory?

Parameters

limit

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

counter
limit
class PrettyPrinter(*args, **kwargs)[source]

Bases: bonobo.config.configurables.Configurable

Parameters
  • filter

    A filter that determine what to print.

    Default is to ignore any key starting with an underscore and none values.

  • max_width (int) –

    If set, truncates the output values longer than this to this width.

    Default: 80

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

format_console(index, key, value, *, fields=None)[source]
format_quiet(index, key, value, *, fields=None)[source]
print_console(context, *args, **kwargs)[source]
print_jupyter(context, *args)[source]
print_quiet(context, *args, **kwargs)[source]
context
filter

A filter that determine what to print.

Default is to ignore any key starting with an underscore and none values.

max_width

If set, truncates the output values longer than this to this width.

Default: 80

class CsvReader(*args, **kwargs)[source]

Bases: bonobo.nodes.io.file.FileReader, bonobo.nodes.io.csv.CsvHandler

Reads a CSV and yield the values as dicts.

Parameters
  • path (str) – Path to use within the provided filesystem.

  • eol (str) –

    Character to use as line separator.

    Default: ‘n’

  • encoding (str) –

    Encoding.

    Default: ‘utf-8’

  • fs (str) –

    The filesystem instance to use.

    Default: ‘fs’

  • mode (str) –

    What mode to use for open() call.

    Default: ‘r’

  • output_fields (ensure_tuple) – Specify the field names of output lines. Mutually exclusive with “output_type”.

  • output_type – Specify the type of output lines. Mutually exclusive with “output_fields”.

  • delimiter (str) –

  • quotechar (str) –

  • escapechar (str) –

  • doublequote (str) –

  • skipinitialspace (str) –

  • lineterminator (str) –

  • quoting (int) –

  • headers

  • fields (ensure_tuple) –

  • skip (int) – If set and greater than zero, the reader will skip this amount of lines.

  • reader_factory

    Builds the CSV reader, a.k.a an object we can iterate, each iteration giving one line of fields, as an iterable.

    Defaults to builtin csv.reader(…), but can be overriden to fit your special needs.

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

read(file, context, *, fs)[source]

Write a row on the next line of given file. Prefix is used for newlines.

reader_factory

Builds the CSV reader, a.k.a an object we can iterate, each iteration giving one line of fields, as an iterable.

Defaults to builtin csv.reader(…), but can be overriden to fit your special needs.

skip

If set and greater than zero, the reader will skip this amount of lines.

class CsvWriter(*args, **kwargs)[source]

Bases: bonobo.nodes.io.file.FileWriter, bonobo.nodes.io.csv.CsvHandler

Parameters
  • path (str) – Path to use within the provided filesystem.

  • writer_factory

    Builds the CSV writer, a.k.a an object we can pass a field collection to be written as one line in the target file.

    Defaults to builtin csv.writer(…).writerow, but can be overriden to fit your special needs.

  • eol (str) –

    Character to use as line separator.

    Default: ‘n’

  • encoding (str) –

    Encoding.

    Default: ‘utf-8’

  • fs (str) –

    The filesystem instance to use.

    Default: ‘fs’

  • mode (str) –

    What mode to use for open() call.

    Default: ‘w+’

  • delimiter (str) –

  • quotechar (str) –

  • escapechar (str) –

  • doublequote (str) –

  • skipinitialspace (str) –

  • lineterminator (str) –

  • quoting (int) –

  • headers

  • fields (ensure_tuple) –

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

write(file, context, *values, fs)[source]

Write a row on the next line of opened file in context.

writer_factory

Builds the CSV writer, a.k.a an object we can pass a field collection to be written as one line in the target file.

Defaults to builtin csv.writer(…).writerow, but can be overriden to fit your special needs.

class FileReader(*args, **kwargs)[source]

Bases: bonobo.nodes.io.base.Reader, bonobo.nodes.io.base.FileHandler

Component factory for file-like readers.

On its own, it can be used to read a file and yield one row per line, trimming the “eol” character at the end if present. Extending it is usually the right way to create more specific file readers (like json, csv, etc.)

Parameters
  • path (str) – Path to use within the provided filesystem.

  • eol (str) –

    Character to use as line separator.

    Default: ‘n’

  • encoding (str) –

    Encoding.

    Default: ‘utf-8’

  • fs (str) –

    The filesystem instance to use.

    Default: ‘fs’

  • mode (str) –

    What mode to use for open() call.

    Default: ‘r’

  • output_fields (ensure_tuple) – Specify the field names of output lines. Mutually exclusive with “output_type”.

  • output_type – Specify the type of output lines. Mutually exclusive with “output_fields”.

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

read(file, *, fs)[source]

Write a row on the next line of given file. Prefix is used for newlines.

mode

What mode to use for open() call.

Default: ‘r’

output
output_fields

Specify the field names of output lines. Mutually exclusive with “output_type”.

output_type

Specify the type of output lines. Mutually exclusive with “output_fields”.

class FileWriter(*args, **kwargs)[source]

Bases: bonobo.nodes.io.base.Writer, bonobo.nodes.io.base.FileHandler

Component factory for file or file-like writers.

On its own, it can be used to write in a file one line per row that comes into this component. Extending it is usually the right way to create more specific file writers (like json, csv, etc.)

Parameters
  • path (str) – Path to use within the provided filesystem.

  • eol (str) –

    Character to use as line separator.

    Default: ‘n’

  • encoding (str) –

    Encoding.

    Default: ‘utf-8’

  • fs (str) –

    The filesystem instance to use.

    Default: ‘fs’

  • mode (str) –

    What mode to use for open() call.

    Default: ‘w+’

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

write(file, context, line, *, fs)[source]

Write a row on the next line of opened file in context.

mode

What mode to use for open() call.

Default: ‘w+’

class JsonReader(*args, **kwargs)[source]

Bases: bonobo.nodes.io.json.JsonHandler, bonobo.nodes.io.file.FileReader

Parameters
  • path (str) – Path to use within the provided filesystem.

  • eol (str) –

    Character to use as line separator.

    Default: ‘n’

  • encoding (str) –

    Encoding.

    Default: ‘utf-8’

  • fs (str) –

    The filesystem instance to use.

    Default: ‘fs’

  • mode (str) –

    What mode to use for open() call.

    Default: ‘r’

  • output_fields (ensure_tuple) – Specify the field names of output lines. Mutually exclusive with “output_type”.

  • output_type – Specify the type of output lines. Mutually exclusive with “output_fields”.

  • loader

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

read(file, *, fs)[source]

Write a row on the next line of given file. Prefix is used for newlines.

loader
class JsonWriter(*args, **kwargs)[source]

Bases: bonobo.nodes.io.json.JsonHandler, bonobo.nodes.io.file.FileWriter

Parameters
  • path (str) – Path to use within the provided filesystem.

  • eol (str) –

    Character to use as line separator.

    Default: ‘n’

  • encoding (str) –

    Encoding.

    Default: ‘utf-8’

  • fs (str) –

    The filesystem instance to use.

    Default: ‘fs’

  • mode (str) –

    What mode to use for open() call.

    Default: ‘w+’

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

write(file, context, *args, fs)[source]

Write a json row on the next line of file pointed by ctx.file.

Parameters
  • ctx

  • row

envelope
class LdjsonReader(*args, **kwargs)[source]

Bases: bonobo.nodes.io.json.LdjsonHandler, bonobo.nodes.io.json.JsonReader

Read a stream of line-delimited JSON objects (one object per line).

Not to be mistaken with JSON-LD (where LD stands for linked data).

Parameters
  • path (str) – Path to use within the provided filesystem.

  • eol (str) –

    Character to use as line separator.

    Default: ‘n’

  • encoding (str) –

    Encoding.

    Default: ‘utf-8’

  • fs (str) –

    The filesystem instance to use.

    Default: ‘fs’

  • mode (str) –

    What mode to use for open() call.

    Default: ‘r’

  • output_fields (ensure_tuple) – Specify the field names of output lines. Mutually exclusive with “output_type”.

  • output_type – Specify the type of output lines. Mutually exclusive with “output_fields”.

  • loader

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

read(file, *, fs)[source]

Write a row on the next line of given file. Prefix is used for newlines.

class LdjsonWriter(*args, **kwargs)[source]

Bases: bonobo.nodes.io.json.LdjsonHandler, bonobo.nodes.io.json.JsonWriter

Write a stream of Line-delimited JSON objects (one object per line).

Not to be mistaken with JSON-LD (where LD stands for linked data).

Parameters
  • path (str) – Path to use within the provided filesystem.

  • eol (str) –

    Character to use as line separator.

    Default: ‘n’

  • encoding (str) –

    Encoding.

    Default: ‘utf-8’

  • fs (str) –

    The filesystem instance to use.

    Default: ‘fs’

  • mode (str) –

    What mode to use for open() call.

    Default: ‘w+’

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

class PickleReader(*args, **kwargs)[source]

Bases: bonobo.nodes.io.file.FileReader, bonobo.nodes.io.pickle.PickleHandler

Reads a Python pickle object and yields the items in dicts.

Parameters
  • path (str) – Path to use within the provided filesystem.

  • eol (str) –

    Character to use as line separator.

    Default: ‘n’

  • encoding (str) –

    Encoding.

    Default: ‘utf-8’

  • fs (str) –

    The filesystem instance to use.

    Default: ‘fs’

  • output_fields (ensure_tuple) – Specify the field names of output lines. Mutually exclusive with “output_type”.

  • output_type – Specify the type of output lines. Mutually exclusive with “output_fields”.

  • fields (tuple) –

  • mode (str) –

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

read(file, context, *, fs)[source]

Write a row on the next line of given file. Prefix is used for newlines.

mode
class PickleWriter(*args, **kwargs)[source]

Bases: bonobo.nodes.io.file.FileWriter, bonobo.nodes.io.pickle.PickleHandler

Parameters
  • path (str) – Path to use within the provided filesystem.

  • eol (str) –

    Character to use as line separator.

    Default: ‘n’

  • encoding (str) –

    Encoding.

    Default: ‘utf-8’

  • fs (str) –

    The filesystem instance to use.

    Default: ‘fs’

  • fields (tuple) –

  • mode (str) –

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

write(file, context, item, *, fs)[source]

Write a pickled item to the opened file.

mode
class Filter(*args, **kwargs)[source]

Bases: bonobo.config.configurables.Configurable

Filter out hashes from the stream depending on the filter callable return value, when called with the

current hash as parameter.

Can be used as a decorator on a filter callable.

filter

A callable used to filter lines.

If the callable returns a true-ish value, the input will be passed unmodified to the next items.

Otherwise, it’ll be burnt.

Parameters

filter

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

filter
class RateLimited(*args, **kwargs)[source]

Bases: bonobo.config.configurables.Configurable

Parameters
  • handler

  • initial (int) –

  • period (int) –

  • amount (int) –

Custom instance builder. If not all options are fulfilled, will return a PartiallyConfigured instance which is just a functools.partial object that behaves like a Configurable instance.

The special _final argument can be used to force final instance to be created, or an error raised if options are missing.

Parameters
  • args

  • _final – bool

  • kwargs

Returns

Configurable or PartiallyConfigured

amount
bucket
handler
initial
period
Format(**formats)[source]
OrderFields(fields)[source]

Transformation factory to reorder fields in a data stream.

Parameters

fields

Returns

callable

Rename(**translations)[source]
SetFields(fields)[source]

Transformation factory that sets the field names on first iteration, without touching the values.

Parameters

fields

Returns

callable

Tee(f)[source]
UnpackItems(*items, fields=None, defaults=None)[source]
>>> UnpackItems(0)
Parameters
  • items

  • fields

  • defaults

Returns

callable

count(counter)[source]
identity(x)[source]
noop(*args, **kwargs)[source]