Starting the IPython controller and engines

To use IPython for parallel computing, you need to start one instance of the controller and one or more instances of the engine. The controller and each engine can run on different machines or on the same machine. Because of this, there are many different possibilities.

Broadly speaking, there are two ways of going about starting a controller and engines:

• In an automated manner using the ipcluster command.
• In a more manual way using the ipcontroller and ipengine commands.

This document describes both of these methods. We recommend that new users start with the ipcluster command as it simplifies many common usage cases.

General considerations

Before delving into the details about how you can start a controller and engines using the various methods, we outline some of the general issues that come up when starting the controller and engines. These things come up no matter which method you use to start your IPython cluster.

Let’s say that you want to start the controller on host0 and engines on hosts host1-hostn. The following steps are then required:

1. Start the controller on host0 by running ipcontroller on host0.
2. Move the FURL file (ipcontroller-engine.furl) created by the controller from host0 to hosts host1-hostn.
3. Start the engines on hosts host1-hostn by running ipengine. This command has to be told where the FURL file (ipcontroller-engine.furl) is located.

At this point, the controller and engines will be connected. By default, the FURL files created by the controller are put into the ~/.ipython/security directory. If the engines share a filesystem with the controller, step 2 can be skipped as the engines will automatically look at that location.

The final step required required to actually use the running controller from a client is to move the FURL files ipcontroller-mec.furl and ipcontroller-tc.furl from host0 to the host where the clients will be run. If these file are put into the ~/.ipython/security directory of the client’s host, they will be found automatically. Otherwise, the full path to them has to be is tls literal">something.furl. The default location of these files is the ~./ipython/security directory.

To connect and authenticate to the controller an engine or client simply needs to present an appropriate FURL (that was originally created by the controller) to the controller. Thus, the FURL files need to be copied to a location where the clients and engines can find them. Typically, this is the ~./ipython/security directory on the host where the client/engine is running (which could be a different host than the controller). Once the FURL files are copied over, everything should work fine.

Currently, there are three FURL files that the controller creates:

ipcontroller-engine.furl

This FURL file is the key that gives an engine the ability to connect to a controller.

ipcontroller-tc.furl

This FURL file is the key that a TaskClient must use to connect to the task interface of a controller.

ipcontroller-mec.furl

This FURL file is the key that a MultiEngineClient must use to connect to the multiengine interface of a controller.

More details of how these FURL files are used are given below.

A detailed description of the security model and its implementation in IPython can be found here.

Getting Started

To use IPython for parallel computing, you need to start one instance of the controller and one or more instances of the engine. Initially, it is best to simply start a controller and engines on a single host using the ipcluster command. To start a controller and 4 engines on you localhost, just do:

$ ipcluster local -n 4

More details about starting the IPython controller and engines can be found here

Once you have started the IPython controller and one or more engines, you are ready to use the engines to do something useful. To make sure everything is working correctly, try the following commands:

In [1]: from IPython.kernel import client

In [2]: mec = client.MultiEngineClient()

In [4]: mec.get_ids()
Out[4]: [0, 1, 2, 3]

In [5]: mec.execute('print "Hello World"')
Out[5]:
<Results List>
[0] In [1]: print "Hello World"
[0] Out[1]: Hello World

[1] In [1]: print "Hello World"
[1] Out[1]: Hello World

[2] In [1]: print "Hello World"
[2] Out[1]: Hello World

[3] In [1]: print "Hello World"
[3] Out[1]: Hello World

Remember, a client also needs to present a FURL file to the controller. How does this happen? When a multiengine client is created with no arguments, the client tries to find the corresponding FURL file in the local ~./ipython/security directory. If it finds it, you are set. If you have put the FURL file in a different location or it has a different name, create the client like this:

mec = client.MultiEngineClient('/path/to/my/ipcontroller-mec.furl')

Same thing hold true of creating a task client:

tc = client.TaskClient('/path/to/my/ipcontroller-tc.furl')

You are now ready to learn more about the MultiEngine and Task interfaces to the controller.

Note

Don’t forget that the engine, multiengine client and task client all have different furl files. You must move each of these around to an appropriate location so that the engines and clients can use them to connect to the controller.

[Capability]

Capability-based security, http://en.wikipedia.org/wiki/Capability-based_security