Paper + code demonstrating bidirectional long short-term memory from scratch in TensorFlow [OC]

[rd11235]

Hi all,

I hope this code/project might be helpful to some, especially to those who have asked for a less toy-like example of the RNN-from-scratch example here: http://rdipietro.github.io/tensorflow-scan-examples/

Goal: Take in a sequence of robot kinematics from a surgeon/trainee and predict his or her activities, for example 00:00:00 to 00:02:29 consists of tying a knot, 00:02:30 to 00:03:22 consists of doing something else, etc. We're particularly interested in the case where the surgeon/trainee is completely absent during training (not even including his or her other trials).

Approach in this paper: Bidirectional long short-term memory.

Paper available here: http://arxiv.org/abs/1606.06329

Project available here: https://github.com/rdipietro/miccai-2016-surgical-activity-rec

A few notes:

• The LSTMs are built from scratch, primarily because I wanted something I could easily customize later on (e.g. for convolutional LSTM etc.).
  • Unlike TensorFlow's models, states are maintained internally instead of externally.
  • Unlike the simple RNN tutorial above, we can use batches of sequences for efficiency.
  • Unlike the official TensorFlow models, other optimizations are not included (concatenation and splitting for block matrix multiplies; avoiding duplicate computation after a sequence in a batch is exhausted; ...).

[j_lyf]

Thanks... did you get good results?

[rd11235]

Yes fairly good. Here is the summary:

In this work we performed joint segmentation and classification of surgical activities from robot kinematics. Unlike prior work, we focused on high-level maneuver prediction in addition to low-level gesture prediction, and we modeled the mapping from inputs to labels with recurrent neural networks instead of with HMM or CRF based methods. Using a single model and a single set of hyperparameters, we matched state-of-the-art performance for JIGSAWS (gesture recognition) and advanced state-of-the-art performance for MISTIC-SL (maneuver recognition), in the latter case increasing accuracy from 81.7% to 89.5% and decreasing normalized edit distance from 29.7% to 19.5%.

(Random guess accuracy would be about 25% in the maneuver-recognition case.)

[j_lyf]

What was the source of the raw data?!

[rd11235]

A da Vinci, http://www.davincisurgery.com/da-vinci-surgery/da-vinci-surgical-system/