How to add visual annotations to an image using python in a jupyter notebook?

Question

The overall objective is to view medical images from some server, add visual annotations to the image (i.e., highlighting/circling a tumor in red), then upload those images back to the server with the annotation as some sort of metadata.

The viewing/uploading/storing metadata is essentially complete, but I have yet to find any packages that allow me to draw on the images within jupyter notebooks. This could either manipulate the pixels or essentially create a new image on top of the displayed image.

I'm currently displaying the image data using matplotlib.

I've seen some use of javascript to track mouse movement and keystrokes, but can't figure out how to use that to my advantage. I can use any package, so long as it works within jupyter.

I don't think Jupyter is the right tool for the job, it seems you want to develop a web application. — Eduardo
– Eduardo, Commented Jun 6, 2016 at 23:21
@Edu While I do agree with you, unfortunately this is my current limitation with work :/ — sutartmelson
– sutartmelson, Commented Jun 6, 2016 at 23:39
To clarify your question, are you saying that you'd like to be able to first display the raw image and then by tracking mouse movement and clicks draw circles and arrows on that image- and do it all within the context of a Jupyter notebook? — michael_j_ward
– michael_j_ward, Commented Jun 6, 2016 at 23:48
@michael_j_ward, Correct. Obviously, displaying the image is simple enough. Tracking mouse movement and clicks is possible within the context. Just not sure how to apply it to drawing, if its possible. — sutartmelson
– sutartmelson, Commented Jun 6, 2016 at 23:52
@sutartmelson, got it. Perhaps this could be useful: matplotlib.org/examples/pylab_examples/cursor_demo.html And this stackoverflow discussion: stackoverflow.com/questions/9136938/… I have never done anything like this myself though. If you find a solution please share it. — michael_j_ward
– michael_j_ward, Commented Jun 7, 2016 at 0:30

sutartmelson · Accepted Answer · 2016-06-07 18:17:48Z

3

Thanks to the suggestions from @michael_j_ward, I found a solution. I looked through those discussions and tutorials as well as read the documentation for the matplotlib axes. This is what I came up with/altered

import matplotlib.pyplot as plt
from matplotlib.lines import Line2D
import numpy as np
import dicom


class Annotator(object):
    def __init__(self, axes):
        self.axes = axes

        self.xdata = []
        self.ydata = []

    def mouse_move(self, event):
        if not event.inaxes:
            return

        x, y = event.xdata, event.ydata

        self.xdata.append(x)
        self.ydata.append(y)
        line = Line2D(self.xdata,self.ydata)
        line.set_color('r')
        self.axes.add_line(line)

        plt.draw()

    def mouse_release(self, event):
        # Erase x and y data for new line
        self.xdata = []
        self.ydata = []

path = '../sample.dcm'

data = dicom.read_file(path)

img = data.pixel_array

fig, axes = plt.subplots()
axes.imshow(img[0])
plt.axis("off")
plt.gray()
annotator = Annotator(axes)
plt.connect('motion_notify_event', cursor.mouse_move)
plt.connect('button_release_event', cursor.mouse_release)

axes.plot()

plt.show()

screenshot

It allows me to open images and draw on them to highlight or annotate significant portions of the image.

answered Jun 7, 2016 at 18:17

sutartmelson

1422 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

MrMartin Over a year ago

Where is cursor from? I'm getting NameError: name 'cursor' is not defined

Kaushalya Over a year ago

@MrMartin I think it should be annotator.

Kaushalya Over a year ago

Did this work in a Jupyter notebook? I couldn't get it to draw anything on the image.

Kevin K. · Accepted Answer · 2016-06-06 23:16:45Z

-2

Inside Jupyter notebook you can use:

%matplotlib inline

answered Jun 6, 2016 at 23:16

Kevin K.

1,4272 gold badges15 silver badges19 bronze badges

Collectives™ on Stack Overflow

How to add visual annotations to an image using python in a jupyter notebook?

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related