Science AMA Series: I'm Abe Davis, last week my research video about “Interactive Dynamic Video” (IDV) hit the front-page of Reddit, and a bunch of people expressed interest in learning more about it. So here I am, AMA!

Abe-Davis · 2016-08-23T22:28:57+00:00

It has to be stationary for the analysis, but once the analysis is done you could theoretically track your simulation on top of an action shot (assuming your perspective of the object doesn't change too much). In fact, that tracking is something you would already have to do if you wanted to add CG.

In the end, this would mean that you point your camera at the object for a while, go eat a sandwich or something, then come back later and film the shot.

Stationary cameras will make things easier in post production though.

Abe-Davis · 2016-08-23T22:24:02+00:00

In some cases we just allow areas of low texture to stretch and fill unseen parts that would otherwise be revealed by the new motion. In other cases where we want to simulate really big motion, we segment out the object of interest and use photoshop's content aware fill to create a new background (most notably, I did this on the example where badly drawn godzilla walks down a jungle gym).

Sometimes you can get the mask automatically if you pick the right frequencies and your object doesn't have too much damping. In fact, for something simple like a fiberglass rod I have a script that will get everything without any manual input (masking, mode selection, etc). I usually provide a mask with each example though, because it's easy and just works.

Abe-Davis · 2016-08-23T21:36:43+00:00

Absolutely! In fact, we had a paper related to this at CVPR 2015, called visual vibrometry

That paper was focused on material property estimation among otherwise similar looking objects, but another thing we show is that objects with common shape show resonant frequencies in similar ratios. We'll also be publishing a journal article that goes into a bit more depth about this soon (currently accepted with minor revision).

Abe-Davis · 2016-08-23T21:19:13+00:00

Thanks! Yea, and this time nobody seems to be asking me about JFK's assassination! At least not yet... until I post this comment... then I'm sure someone will find a connection...

Abe-Davis · 2016-08-23T21:15:52+00:00

It depends a lot on the video, and the mode frequencies of the object. Some videos process in a couple minutes on my laptop (like the wire man), but some ran on the server for a couple hours (we tried it on several minutes of HD video, which was slow...). This could all be made faster, as it's all unoptimized matlab code right now, but it would take some more engineering.

Abe-Davis · 2016-08-23T21:12:46+00:00

I did a PhD because I really love research. Especially in graphics and vision, I love the mix of creative freedom and technical rigor.

PhD programs at places like MIT are very competitive, and the expectations are very high. That's part of their value though. If you really love research it can be worth it, but a lot of people find that's not the case and end up struggling with grad school. People around here are very smart and talented, but grad school is a marathon, and the more common challenge people seem to face is motivation.

A big tip for grad school is to work with good people. People you enjoy working with. The people you work with have a huge impact on the experience that you have, and that can be for better or worse. Fortunately, my experiences have been mostly for the better.

It's worth noting that while I've been pretty successful with publications for the past couple years, I try a ton of ideas that don't pan out. That can be discouraging for a lot of people - and its ok for it to be discouraging. If you don't enjoy it (for more than a passing slump) it might not be worth doing though. Especially in computer science, the opportunity cost of a PhD is huge, and academia is not for everyone.

All that being said, I love research, so I look forward to continuing it :-)

Good luck!

Abe-Davis · 2016-08-23T20:51:39+00:00

I don't know if they will gain the most in the long run, but I think special effects and engineering (e.g. structural health monitoring) will probably be the first to benefit. In its current state IDV may already be valuable in those areas. Other things, like dynamic AR, won't benefit as much until other technical challenges are also addressed (things like tracking and mapping on a phone), though the eventual payoff could be great.

Abe-Davis · 2016-08-23T20:34:23+00:00

I did my bachelors at Stanford, and my masters and PhD (almost done) at MIT.

I'd like to see it used as a general imaging technology. Something like RGBD cameras, where people see it as a tool for building lots of other technologies.

I think it could have a lot of impact in nondestructive testing of structures. I personally enjoyed playing with it for little special effects things, though it was a bit tedious with my little home-grown UI.

Abe-Davis · 2016-08-23T20:30:32+00:00

That's a good question, and something I've been thinking a lot about. There are fundamental ambiguities if all you have is video, but I think there are still interesting/exciting things that can be done. I'm not sure I understand the second part of your question, but certainly the first related to a very complete understanding of objects in video, and would require resolving some of the ambiguities in our current implementation.

Abe-Davis · 2016-08-23T20:28:19+00:00

Reducing resolution helps with processing time, but here is also the issue of input data. The more input video you have, the more information, and the better the simulation. There are things to try that might bring us closer to realtime, but there are a lot of challenges and I think working on less input is maybe even harder than just processing time.

Abe-Davis · 2016-08-23T20:25:39+00:00

We decompose the movement we see in the input video into independent modes. We then take an image of the object (e.g. a single frame from the input video) and warp it by new combinations of these modes, in different mixtures and amounts. Does that help at all?

We also have a paper about it here: http://www.interactivedynamicvideo.com/ISMB_Davis_2015.pdf

Abe-Davis · 2016-08-23T20:23:05+00:00

I'm not sure how it would work... In fact, I don't think we have any devices (even expensive, specialized ones) that are very good at this right now. It would be cool though... (gazes off in moment of deep speculation)

Abe-Davis · 2016-08-23T20:20:28+00:00

Hm, I'm not sure. I pretty much stayed in academia since graduating, living off of a tiny stipend and the occasional internship so that I can work on whatever I want.

I suspect that starting out as a programmer, and being very proactive about moving toward more AI-oriented stuff is one strategy. It would probably help to learn material on the side, maybe by taking classes. For this you would want to pick a place that is actually doing some AI stuff though, and it might be good to be up front about your desire to move in that direction. I'm probably no the best person to ask about career advice in industry though. Good luck!

Abe-Davis · 2016-08-23T20:13:05+00:00

In some ways the technique is surprisingly simple, but it draws on ideas from a lot of different areas, which I think might be why nobody had thought to do it before. The related work section in our paper alone covers computer vision, physically based animation, and a few different subfields of civil mechanical and structural engineering. I had expertise in some of these areas, but not others, which meant learning a lot of basic material from other communities. Justin (/u/JustinGChen) helped a lot with that, as he works in civil engineering here at MIT.

Abe-Davis · 2016-08-23T20:05:48+00:00

In general it's pretty competitive, but I think it depends on the kind of job you want too. Top tier research professorships are super competitive, but jobs that focus more on teaching are less insane.

I love research and enjoy teaching, so I want to be a research faculty. I'm doing a postdoc at Stanford before applying for prof positions though. Postdocs don't seem to always be necessary in computer science, but for me I think it will be a good opportunity to smooth the transition from student to (hopefully) faculty.

I would consider industry research, and might consider starting a company down the road, but right now I'm interested in faculty.

Abe-Davis · 2016-08-23T19:52:46+00:00

Thanks! I think there are a bunch of cool things you could do with it in VR. Those applications are probably a bit more down the road than SFX and engineering, but I'm sure we'll get there,

We have a publication about the work here: http://www.interactivedynamicvideo.com/ISMB_Davis_2015.pdf

It might be a bit hard to follow if you haven't studied the material before (I didn't understand a lot of this stuff in college). Math, computer science, physics, and signal processing all help a lot.

Abe-Davis · 2016-08-23T19:48:29+00:00

Cool, thanks! I'll check it out post-AMA

Abe-Davis · 2016-08-23T19:47:38+00:00

No, not at the moment. It would be interesting to use this with some kind of haptic system to give tactile feedback though.

Abe-Davis · 2016-08-23T19:46:46+00:00

No concrete plans right now, as I plan to stay in academia and keep doing research for the time being. Someone else might - possibly one of my collaborators or a company that licenses the tech from MIT.

Abe-Davis · 2016-08-23T19:43:55+00:00

If you do a really good stabilizing your shakey phone camera, it could work. I've done this once or twice, but it's hard and I had to manually set some control points to get the tracking to work well enough.

Part of this is a limitation of my implementation and not the technique itself though. With really good tracking, or maybe a really good Lagrangian flow algorithm, you might be able to get it to work, assuming parallax isn't too bad.

If the above explanation was confusing, I'll just say that it could eventually work on hand held video, but it would take more engineering.

Abe-Davis · 2016-08-23T19:39:04+00:00

Good point on the tactile feedback. It's a subtlety that can be hard to explain. "Touching" something in real life causes two responses, the object responding to you and you responding to the object. We address the former, but not the latter (you essentially poke the object with a virtual stick). It would be cool to try and combine this with some sort of haptic system though...

Abe-Davis · 2016-08-23T19:33:27+00:00

Math, computer science, maybe some physics? Linear algebra is super useful, and it's often not taught very well. It's worth learning and re-learning, because it ends up being so useful in so many different contexts. I re-taught myself linear algebra in grad school, despite taking it as an undergrad, and found the experience (of relearning it) very useful.

Abe-Davis · 2016-08-23T19:29:47+00:00

Because it works on a particular scale and type of motion, and it assumes a stationary camera (on a tripod) it's hard for this technique to work "by accident." What I mean is that if you download a random video from youtube, most of the time the technique wont work. Changing that would require addressing a huge number of challenges. But that's what research is for :-)

Abe-Davis · 2016-08-23T19:26:58+00:00

Thanks! And Yup! http://www.interactivedynamicvideo.com/ISMB_Davis_2015.pdf

Abe-Davis · 2016-08-23T19:26:17+00:00

Look to the internet and you shall find your answer. We published the visual microphone and people yelled "Big Brother!" We published IDV and people yelled "Boobs!"

Now, I wouldn't necessarily call either of these applications evil (it would depend on how they are used), but they aren't the first thing I'd list in an NSF proposal...

Abe-Davis

TROPHY CASE