Example notebook - nb2kg

This notebook is an example of connecting a jupyter notebook to IBM Analytics Engine (IAE) using nb2kg. It is expected that you will run this notebook on a local jupyter environment (i.e. not from DSX).

The notebook connecting to IAE is spun up in a docker container. This is because:

• When you enable the nb2kg extension, all kernels run on the configured Kernel Gateway, instead of on the Notebook server host. The extension does not support local kernels. If nb2kg is installed in your local notebook environment, it would prevent you from running local kernels. Thus using docker prevents me from corrupting your local notebook environment.
• Setting up a notebook environment with all the dependencies for nb2kg can be tricky. Docker allows me to provide you with an environment that is pre-configured.

Python is used to interact with docker because this was easier to script in a notebook. However, if you run docker with sudo, this notebook may not work for you.

WARNING: This project is just a demo of connecting to IAE using nb2kg. Your mileage may vary.

! pip install --quiet docker

Load the IBM Analytics Engine (IAE) credentials. See this example notebook for creating an IAE instance and saving the vcap.json credentials file.

import json
vcap = json.load(open('./vcap.json'))

import docker
client = docker.from_env()
api_client = docker.APIClient()

The docker images can be quite large and take a long time to load. Here we load the parent image and regularly print some output so you can see what is going on.

import json

lines_processed = 0

api_client = docker.APIClient()
for line in api_client.pull('jupyter/minimal-notebook:fa77fe99579b', tag=None, stream=True):
    if lines_processed % 25 == 0:
        try:
            status = json.loads(str(line)[2:-5]) # strip quotes and newline
            if 'progressDetail' in status:
                print(status['progressDetail'])
        except:
            pass
    lines_processed += 1

I have created a custom docker environment that uses nb2kg. This environment has been kept as simple as possible to make it easy for you to adapt to to your own requirements.

image = client.images.build(
    path='https://github.com/snowch/docker_jupyter_notebook_kg.git',
    tag='docker_jupyter_notebook_kg')

print(image.tags)

Delete any containers hanging around from a previous run of this notebook.

WARNING: Ensure you backup any work you want to keep before running this command.

from datetime import datetime as dt
import dateutil.parser

for cont in client.containers.list(filters={ 'name': 'iae_nb2kg_example' }):
    created = dt.utcnow() - dateutil.parser.parse(cont.attrs['Created']).replace(tzinfo=None)
    print("Name: {} | Status: {} | Age (H:M:S): {}".format(
                         cont.attrs['Name'],
                         cont.attrs['State']['Status'],
                         created
                         ))
    #print(json.dumps(cont.attrs, indent=4, sort_keys=True))
    cont.kill()

Run the notebook. You may need to change the LOCAL_PORT if this port is not free on your local machine.

Change the LOCAL_NOTEBOOKS_FOLDER to a folder on your local machine where you want your notebooks in the ‘work’ folder to be saved.

LOCAL_PORT = 8899
LOCAL_NOTEBOOKS_FOLDER = '/Users/snowch/Desktop/notebooks'

container = client.containers.run(
    image = image,
    volumes = { LOCAL_NOTEBOOKS_FOLDER : '/home/jovyan/work' },
    ports = {str(LOCAL_PORT)+'/tcp': LOCAL_PORT},
    environment = {
        'NB_PORT': LOCAL_PORT,
        'KG_HTTP_USER': vcap['cluster']['user'],
        'KG_HTTP_PASS': vcap['cluster']['password'],
        'KG_URL': vcap['cluster']['service_endpoints']['notebook_gateway'],
        'KG_WS_URL': vcap['cluster']['service_endpoints']['notebook_gateway_websocket'],
        'KG_CONNECT_TIMEOUT': '50.0',
        'KG_REQUEST_TIMEOUT': '40.0'
    },
    detach=True,
    stdout=True,
    stderr=True,
    remove=True,
    name='iae_nb2kg_example'
    )

The next cell prints the log output. Ensure there are no errors reported.

You should see a url, e.g.

Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://0.0.0.0:8899/?token=12345

Click on the url in the output to open it in your browser.

From here, you should be able to create a new notebook:

from IPython.lib import backgroundjobs as bg
jobs = bg.BackgroundJobManager()

def printlogs():
    for line in container.logs(stream=True):
        print(str(line)[2:-3]) # strip quotes and newline
    sys.stdout.flush()

jobs.new('printlogs()')
jobs.status()

finally, remove the notebook docker container

container.kill()