From CS 294-10 Visualization Sp10

Jump to: navigation, search

Part 1

I am currently involved in a project that involves estimating the depth of images from pictures of images (specifically, top-down pictures of terrain). We have ground-truth depth maps of these images, and I am to produce depth maps that are *similar* to those ground-truth maps. Similarity is difficult to define --- we are never quite perfect, and often the ways in which our estimated depth maps differ from the true depth maps are extremely difficult to judge.

Currently, I visualize depth-maps in a standard GIS (geographic information system) style, in which height is encoded by hue. This may seem unnatural (as we know that hue is poorly suited to encoding numerical values), but often works very well, as often it is the level sets of a depth map that one is interested in, and hue is very good for matching similar values. The most difficult thing is visualizing fine-scale detail, like differences within a small range of heights.

My goal is to make an interactive visualization system in which one can explore two depth maps, side-by-side, and interactively specify the range of heights that the user views. This would be the initial interface:


Where the image on the left is my (slightly incorrect) reconstruction, and the image on the right is ground-truth. It's clear that my reconstruction isn't quite perfect, as it looks slightly "noisy" or "bumpy" in places.

Our first method of interaction is inspired by the Timeboxes interface of Hochheiser and Schneiderman. If the user is interested in getting a detailed visualization of some small part of the image, he can click and drag, and select all heights on a line between the start- and end-points of the line that he has drawn. This will then set the range of heights being visualized to the range of the heights that were just selected:


We believe that this is an effective method for allowing the user to adjust the color scale such that a specific area of interest is made more clear. Selecting a line seems sensible, as other methods (selecting a rectangle, or a circle) seem rigid and poorly suited to this domain.

The resulting rendering of the map seems very effective for visualizing errors between the two images. Differences between the two images, such as the lower-left region, immediately "pop out" as being different, while it is easy to rapidly discern what other parts of the depth-maps are very similar.

The second method of interaction is inspired by the HomeFinder visualization. The user can directly interact with the color range. By clicking and dragging the center of the bar, the user can shift the range of heights being viewed, with the rendered image tightly coupled to the bar while the user drags it:


The user can also drag the extents of the bar, to change one end of the range of the heights being viewed. Again, the rendered image will be updated dynamically as the bar is dragged.


We believe that the tight-coupling in this method of visualization will be extremely helpful, especially when dragging a very narrow bar. By "sliding" vertically through the depth maps, the user should get a very fine-detail view of the discrepancies between the two images, while still feeling like he/she is "seeing" the entire depth-map at once.

Part 2

Code for my project, in the form of Matlab scripts, can be found here: File:JonBarronA4.zip

The final visualization is very similar to how it was initially described. A walk through:


One immediate difference I made was to add the intensity-based shading of the surface, which amounts to rendering the surface (with Lambertian reflectance) as though it was lit from the direction of the camera. This immediately gives the viewer a sense of depth, and better illustrates the shape of the surface. Most importantly, this shading stays the same as the color scale is varied, which allows the user to see the surface that is outside of the color range.

The user can click and drag to select a linear region of the surface:


And after the user finishes dragging, the colormap is updated. The line is drawn in real-time as the user drags, but we didn't feel that tight coupling was necessary for the color map, so the image is not updated until the user finishes dragging.


The user can also modify the color range by manipulating the color bar on the right. The design of the color bar has been modified a bit since the initial proposal. First, the limitation of matlab required that I couldn't use a rainbow-colored bar as an interactive element (such things cannot be clicked and dragged), so I made the bar half-rainbow (to inform the user about the orientation and scale of the color scheme) and half-gray (which the user can manipulate). On the top and bottom of the bar, are two wide, solid-colored tabs, which denote the ends of the range. These tabs, when clicked, modify the extents of the color bar, as opposed to clicking the gray bar, which shifts the entire color bar. The unfortunate necessity of having to use the gray bar actually complements the tabs nicely, as they stand out by spanning the entire width of the bar. This makes it easy for the user to distinguish between shifting the bar and dragging the tabs, which was a very serious ambiguity in our initial design.

Here, the user drags one end of the bar:


And then shifts the bar up through the image.

File:5.png File:6.png

The image is re-rendered in real-time as the bar is manipulated. This tight-coupling gives the user an increased sense of interaction, and is much more convenient than repeatedly dragging and then releasing the bar by small increments.

I added a drop-down tab in the lower left to allow the user to try different color schemes. We toyed around with encoding height with the following schemes: hue (shown), saturation (red vs white, blue vs white), blue-white-yellow, red-blue (a subset of hue, shown below), and red-white-blue. I found hue to be most effective, but I'm biased because I've been looking at hue-based plots for a while, so having the option of other color schemes seems sensible. We also added a reset button to allow the user to rescale the colors to the initial scale.


Happily, I discovered that the line selection interface has a nice degenerate case, in which the user simply clicks, and the color range is rescaled such that: blue == above the point that was clicked, and red == below the point that was clicked. This makes repeatedly clicking on the image an effective way of visualizing different level-sets:

File:8.png File:9.png

To recap, changes between the prototype and the final stage: Shading of the surface encoded with image intensity, redesigned color bar, addition of color-range drop-down and reset button, and the discovery of the degenerate clicking mode.

The development process was fairly straightforward. I already had code for rendering a heightmap according to some arbitrary color map, and the nature of this shape-from-shading project meant that I had access to code for rendering a surface with some arbitrary lighting. Most of my time was spent grappling with Matlabs admittedly mediocre GUI tools, and in experimenting with different color maps and shading methods, eventually settling on the overhead lighting angle and a hue-based color scheme that was similar to what I had started with. Overall, development took about a day and a half of solid work.

I'm fairly pleased with the overall visualization, though I feel that it doesn't work very well towards my initial stated purpose, which is analyzing the errors in my surface reconstruction. It may be that a technique that explicitly visualizes the *difference* between the two height maps was the way to go. This tool's primary success is as a tool for navigating a single height map, which still has merit.

[add comment]
Personal tools