CS294-69 Image Manipulation and Computational Photography

From CS294-69 Image Manipulation and Computational Photography Fa11

Revision as of 22:27, 4 December 2011 by Samii (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search


Digital cameras have made it very easy and cheap to capture the world around us. Yet, the images our cameras record often do not match our experience of a particular scene, or our communicative intent. The best visual depictions usually require extensive editing of the raw imagery. Today, photographers primarily use low-level image-processing tools (e.g. Photoshop, GIMP, Picasa, etc.) and often end up directly touching individual pixels. In this course we will study techniques and algorithms that allow users to manipulate visual imagery at a much higher level. We will focus on recent research that brings together ideas from computer graphics, computer vision, image processing and human-computer interaction.

The course is open to graduates (sign up for CS294-69) and advanced undergraduates (CS194-69) who are familiar with basic concepts in computer graphic, computer vision and signal processing. We will also assume good understanding of linear algebra. Students who have taken an undergraduate graphics course will have the necessary background. Finally we will assume a willingness to learn Matlab or other suitable languages/APIs as necessary. Send me (Maneesh) email if you are worried about whether you have the background for the course.



Note: Readings may be changed and updated as the semester progresses.

Week 1: Introduction/Color

Aug 29: Introduction (Slides)

Assignment 0: due by 5pm Mon Sep 12: Hybrid Images

Aug 31: Forming Color Images (Slides)

  • Chapters 2.2.1, 2.2.2 and 2.3.2 (pp.60-68 and pp.80-90). Computer Vision: Algorithms and Applications. Rick Szeliski. 2010. (pdf)
Accessing Matlab for Assignment 0
  • Using the username and password that we provide, you can log into the Windows PCs in 199 Cory or the Linux PCs in 330 Soda and run /share/b/bin/matlab from the command line. This will open Matlab for you. You can find more information on how to run Matlab on this website.
Signing up for Paper Presentations
  • You can sign up here for the paper presentations. Remember that if an undergraduate student signed up as a paper presenter, only graduate students can sign up for being the discussant of that paper and vice versa.

Week 2: Sampling, Filtering, Reconstruction

Sep 5: Holiday - no class

Sep 6: MATLAB Tutorial - The tutorial will take place 3 to 4pm at 330 Soda Hall. Here is the document that I used for the Matlab introduction today. Change the file extension to *.m and you can directly open this file in Matlab.

Sep 7: Sampling, Filtering, Reconstruction (Slides)

  • Pyramid-based Computer Graphics. J.M. Ogden, E.H. Adelson, J.R. Bergen and P.J. Burt. RCA Engineer 1984. (pdf)
  • Hybrid Images. A. Oliva, A. Torralba and P. Schyns. SIGGRAPH 2005. (pdf)
Optional Readings
  • Laplacian Pyramid as a Compact Image Code. E.H. Adelson and P.J. Burt. IEEE Trans. on Communication 1983. (pdf)
Student Presentations
  • We posted instructions for helping you prepare the student presentations.

Week 3: Texture Synthesis

Sep 12: Texture Synthesis (Slides)

Due Assignment 0: due by 5pm : Hybrid Images
Assignment 1: due by 5pm Mon Sep 26: Texture Synthesis
  • Pyramid-Based Texture Analysis/Synthesis. David J. Heeger and James R. Bergeny. SIGGRAPH 95. (pdf)
  • Texture Synthesis by Non-parametric Sampling. Alexei A. Efros and Thomas K. Leung. ICCV 1999 (pdf)
Optional Readings
  • Video Textures. Arno Schoedl, Richard Szeliski, David H. Salesin, Irfan Essa. SIGGRAPH 2000 (pdf)
  • Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images. Jeremy De Bonet. SIGGRAPH 97. (pdf)
  • Fast Texture Synthesis using Tree-structured Vector Quantization. Li-Yi Wei and Marc Levoy. SIGGRAPH 2000 (pdf)

Sep 14: Texture Synthesis (Student Presentations) (Image Quilting Slides, Region Filling Slides, PatchMatch Slides)

  • Image Analogies. Aaron Hertzmann, Charles E. Jacobs ,Nuria Oliver, Brian Curless and David H. Salesin. SIGGRAPH 2001 (pdf)
  • Region-Filling and Object Removal by Exemplar-Based Inpainting. Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. CVPR 2004. (pdf)
  • PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. Connelly Barnes, Eli Shechtman, Adam Finkelstein and Dan B Goldman. SIGGRAPH 2009 (pdf)
Optional Readings
  • Image Quilting for Texture Synthesis and Transfer. Alexei Efros and William Freeman. SIGGRAPH 2001. (pdf)
  • Image Completion with Structure Propagation. Jian Sun, Lu Yuan, Jiaya Jia and Heung-Yeung Shum. SIGGRAPH 2005.(pdf)
  • Synthesizing Structured Image Hybrids. Eric Risser, Charles Han, Rozenn Dahyot and Eitan Grinspun. SIGGRAPH 2010 (pdf)

Week 4: Camera Optics/Lenses/Depth of Field

Sep 19: Camera Optics/Lenses/Depth of Field (Slides)

  • Chapter 2.1 (focus on 2.1.5 (projection) and 2.1.6 (lens distortion)). Computer Vision: Algorithms and Applications. Rick Szeliski. 2010. (pdf)

Sep 21: Camera Optics/Lenses/Depth of Field (Student Presentations) (Blur to Affect Perceived Distance and Size Slides, Defocus Magnification Slides)

  • Using Blur to Affect Perceived Distance and Size. Robert Held, Emily Cooper, James O’Brien, and Martin Banks. TOG 2010. (pdf)
  • Defocus Magnification. Soonmin Bae and Frédo Durand. EUROGRAPHICS 2007. (pdf)
  • Capturing and Viewing Gigapixel Images. Johannes Kopf, Matt Uyttendaele, Oliver Deussen, Michael F. Cohen. SIGGRAPH 2007. (pdf)

Week 5: Light Fields and Camera Arrays

Sep 26: Light Fields and Camera Arrays (Slides)

Due Assignment 1: due by 5pm : Texture Synthesis
Assignment 2: due by 5pm Mon Oct 10: Light Fields/HDR/Tone-Mapping
  • Light Field Rendering. Marc Levoy and Pat Hanrahan. SIGGRAPH 96 (pdf)
  • High Performance Imaging Using Large Camera Arrays. Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez. SIGGRAPH 2005 (pdf)
Optional Readings
  • The Lumigraph. Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski and Michael F. Cohen. SIGGRAPH 96. (pdf)

Sep 28: Light Fields and Camera Arrays (Student Presentations) (Dynamically Reparameterized Light Fields Slides, Unstructured Lumigraph Slides, Fourier Slice Photography Slides)

  • Unstructured Lumigraph Rendering. Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler and Michael Cohen. SIGGRAPH 2001. (pdf)
  • Fourier Slice Photography. Ren Ng. SIGGRAPH 2005 (pdf)
Optional Readings
  • Dynamically Reparameterized Light Fields. Aaron Isaksen, Leonard McMillan and Steven Gortler. SIGGRAPH 2000. (pdf)
  • Light Field Photography with a Hand-held Plenoptic Camera. Ren Ng, Marc Levoy, Mathieu Bredif, Gene Duval, Mark Horowitz and Pat Hanrahan. Tech Report 2005. (pdf)

Week 6: HDR/Tone-Mapping

Oct 3: HDR/Tone-Mapping (Slides)

  • Recovering High Dynamic Range Radiance Maps from Photographs. Paul E. Debevec and Jitendra Malik. SIGGRAPH 97. (pdf)
  • LCIS: A Boundary Hierarchy for Detail-Preserving Contrast Reduction. Jack Tumblin and Greg Turk. SIGGRAPH 1999. (pdf)
  • Fast Bilateral Filtering for the Display of High-Dynamic-Range Images. Fredo Durand and Julie Dorsey. SIGGRAPH 2002. (pdf)
Optional Readings
  • Tone Reproduction for Realistic Images. Jack Tumblin. Computer Graphics and Applications 1993. (pdf)
  • Bilateral Filtering: Theory and Applications. Sylvain Paris, Pierre Kornprobst, Jack Tumblin, and Frédo Durand. Foundations and Trends in Computer Graphics and Vision 2009. (pdf)
  • Digital Photography with Flash and No-Flash Image Pairs. Georg Petschnigg, Maneesh Agrawala, Hugues Hoppe, Richard Szeliski, Michael Cohen and Kentaro Toyama. SIGGRAPH 2004. (pdf)
  • Flash Photography Enhancement via Intrinsic Relighting. Elmar Eisemann and Frédo Durand. SIGGRAPH 2004. (pdf)

Oct 5: HDR/Tone-Mapping (Student Presentation) (Domain Transform for Edge-Aware Image and Video Processing Slides,Tone Mapping for Low-Light Conditions,Photographic Tone Reproduction for Digital Images)

  • Photographic Tone Reproduction for Digital Images. Erik Reinhard, Michael Stark, Peter Shirley and James Ferwerda. SIGGRAPH 2002. (pdf)
  • Perceptually Based Tone Mapping for Low-Light Conditions. Adam G. Kirk and James F. O’Brien. SIGGRAPH 2011. (pdf)
  • Real-time Edge-Aware Image Processing with the Bilateral Grid. Jiawen Chen, Sylvain Paris, and Frédo Durand. SIGGRAPH 2007. (pdf)
Optional Readings
  • Fast High-Dimensional Filtering Using the Permutohedral Lattice. Andrew Adams, Jongmin Baek and Abe Davis. Eurographics 2010. (pdf)
  • Domain Transform for Edge-Aware Image and Video Processing. Eduardo Gastal and Manuel Oliveira. SIGGRAPH 2011. (pdf)

Week 7: Gradient Domain Techniques

Oct 10: Gradient Domain Techniques (Slides)

Due Assignment 2: due by 5pm : Light Fields/HDR/Tone-Mapping
Assignment 3: due by 5pm Mon Oct 24: Gradient Domain/Resizing and Warping
  • Gradient Domain High Dynamic Range Compression. Raanan Fattal, Dani Lischinski and Michael Werman. SIGGRAPH 2002. (pdf)
  • Poisson Image Editing. Patrick Perez, Michel Gangnet and Andrew Blake. SIGGRAPH 2003. (pdf)
Optional Readings
  • Image Editing in the Contour Domain. James Elder and Rick Goldberg. CVPR 1998. (pdf)

Oct 12: Gradient Domain Techniques (Student Presentations) (Gradient Shop Slides,Diffusion Curves Slides,Efficient Gradient-Domain Compositing Using Quadtrees Slides)

  • Diffusion Curves. Alexandrina Orzan, Adrien Bousseau, Holger Winnemoeller, Pascal Barla, Joelle Thollot and David Salesin. SIGGRAPH 2008. (pdf)
  • Real-time Gadient Domain Painting. James McCann and Nancy Pollard. SIGGRAPH 2008. (pdf)
Optional Readings
  • Efficient Gradient-Domain Compositing Using Quadtrees. Aseem Agarwala. SIGGRAPH 2007. (pdf)
  • GradientShop: A Gradient-Domain Optimization Framework for Image and Video Filtering. Pravin Bhat, C. Lawrence Zitnick, Michael Cohen and Brian Curless. (pdf)

Week 8: Image Resizing and Warping

Oct 17: Image Resizing and Warping (Slides)

  • Seam Carving for Content-Aware Image Resizing. Shai Avidan and Ariel Shamir. SIGGRAPH 2007. (pdf)
  • Optimized Scale-and-Stretch for Image Resizing. Yu-Shuen Wang, Chiew-Lan Tai, Olga Sorkine and Tong-Yee Lee. SIGGRAPH Asia 2008. (pdf)
Optional Readings
  • Optimizing Content-Preserving Projections for Wide-Angle Images. Robert Carroll, Maneesh Agrawala and Aseem Agarwala. SIGGRAPH 2009. (pdf)
  • Image Warps for Artistic Perspective Manipulation. Robert Carroll, Aseem Agarwala, Maneesh Agrawala. SIGGRAPH 2010. (pdf)
  • Feature-Aware Texturing. Ran Gal, Olga Sorkine and Daniel Cohen-Or. EGSR 2006. (pdf)

Oct 19: Image Resizing and Warping (Student Presentations) (Improved Seam Carving - Multi-Operator Media Retargeting Slides,Comparative Study of Image Retargeting Slides,Nonlinear Disparity Mapping for Stereoscopic 3D Slides)

  • Multi-operator Media Retargeting. Michael Rubinstein, Ariel Shamir, Shai Avidan. SIGGRAPH 2009. (pdf)
  • A Comparative Study of Image Retargeting. Michael Rubinstein, Diego Gutierrez, Olga Sorkine, Ariel Shamir. SIGGRAPH ASIA 2010. (pdf)
Optional Readings
  • Improved Seam Carving for Video Retargeting. Michael Rubinstein, Ariel Shamir, Shai Avidan. SIGGRAPH 2008. (pdf)
  • Nonlinear Disparity Mapping for Stereoscopic 3D . M. Lang, A. Hornung, O. Wang, S. Poulakos, A. Smolic and M. Gross. SIGGRAPH 2010. (pdf)

Week 9: Matting

Oct 24: Matting (Slides)

Due Assignment 3: due by 5pm: Gradient Domain/Resizing and Warping
Final Project: Proposals due 10/31, Proposal Presentations 10/31 and 11/2, Final Presentations 11/28 and 11/30, Final Paper 12/7
  • A Bayesian Approach to Digital Matting. Yung-Yu Chuang, Brian Curless, David H. Salesin and Richard Szeliski. SIGGRAPH 2001. (pdf)
  • Image and Video Matting: A Survey. Jue Wang and Michael Cohen. Foundations and Trends in Computer Graphics and Vision. (pdf)
Optional Readings
  • Alpha Estimation in Natural Images. Mark Ruzon and Carlo Tomassi. CVPR 2000. (pdf)

Oct 26: Matting (Student Presentations) (Poisson Matting Slides,A Closed-Form Solution to Natural Image Matting Slides)

  • Poisson Matting. Jian Sun, Jiaya Jia, Chi-Keung Tang and Heung-Yeung Shum. SIGGRAPH 04. (pdf)
  • A Closed-Form Solution to Natural Image Matting. Anat Levin, Dani Lischinski, and Yair Weiss. PAMI 2008. (pdf)

Week 10: Student Project Presentations

Oct 31: Student Presentations: Final Project Proposal

  • Rohan Nagesh,
  • Michael Tao,
  • Eileen Bai & Philip Ly
  • Sally Ahn & Soham Mehta,
  • Vasily Volkov & Stacy Hsueh
  • Sean Arietta,
  • Jiamin Bai,
  • Nikhil Naikal

Nov 2: Student Presentations: Final Project Proposal

  • Yean Jin Lee & Yin-Chia Yeh,
  • Dai Bui,
  • Armin Samaii, Tim Althoff,
  • Aaron Eidelson & Dustin Shean,
  • Robin Gaestel & Moeka Takagi,
  • Vera Dadok,
  • Hong Wu & Viraj Kulkarni,
  • Jeff Donahue,
  • Nancy Wang

Week 11: Graphcut Techniques

Nov 7: Graphcut Techniques (Slides)

  • Graphcut Textures: Image and Video Synthesis Using Graph Cuts. Vivek Kwatra , Arno Schödl , Irfan Essa , Greg Turk and Aaron Bobick. SIGGRAPH 2003. (pdf)
  • Interactive Digital Photomontage, Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn. SIGGRAPH 2004 (pdf)

Nov 9: Graphcut Techniques (Student Presentations) (Grabcut and Lazy-Snapping Slides,Video Cutout Slides,Photographing Long Scenes Slides)

  • Grabcut: Interactive Foreground Extraction Using Iterated Graph Cuts. Carsten Rother, Vladimir Kolmogorov and Andrew Blake. SIGGRAPH 2004. (pdf)
  • Interactive Video Cutout. Jue Wang, Pravin Bhat, R. Alex Colburn, Maneesh Agrawala, Michael F. Cohen. SIGGRAPH 2005. (pdf)
Optional Readings
  • Lazy Snapping. Yin Li, Jian Sun, Chi-Keung Tang and Heung-Yeung Shum. SIGGRAPH 2004. (pdf)
  • Photographing Long Scenes with Multi-Viewpoint Panoramas. Aseem Agarwala, Maneesh Agrawala, Michael Cohen, David Salesin and Richard Szeliski. SIGGRAPH 2006. (pdf)
  • Panoramic Video Textures. Aseem Agarwala, Ke Colin Zheng, Chris Pal, Maneesh Agrawala, Michael Cohen, Brian Curless, David Salesin and Richard Szeliski. SIGGRAPH 2005. (pdf)

Week 12: Image Relighting

Nov 14: Image Relighting (Slides)

  • Acquiring the Reflectance Field of a Human Face. Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. SIGGRAPH 2000. (pdf)
  • Conveying Shape and Features with Image-Based Relighting. David Akers, Frank Losasso, Jeff Klinger, Maneesh Agrawala, John Rick and Pat Hanrahan. Visualization 2003. (pdf)
  • Environment Matting and Compositing. Zongker, Werner, Curless, and Salesin, and Szeliski. SIGGRAPH 1999. (pdf)
Optional Readings
  • Multiscale Shape and Detail Enhancement from Multi-light Image Collections. Rannan Fattal, Maneesh Agrawala and Szymon Rusinkiewicz. SIGGRAPH 2007. (pdf)

Nov 16: Image Relighting (Student Presentations) (Intrinsic Images Slides,Illumination Decomposition Slides,Inverse Light Transport Slides)

  • A Theory of Inverse Light Transport. Steven M. Seitz, Yasuyuki Matsushita and Kiriakos Kutulakos. CVPR 2005. (pdf)
  • User-assisted Intrinsic Images. Adrien Bousseau, Sylvain Paris, and Frédo Durand. SIGGRAPH Asia 2009. (pdf)
Optional Readings
  • Illumination Decomposition for Material Recoloring with Consistent Interreflections. Robert Carroll, Ravi Ramamoorthi and Maneesh Agrawala. SIGGRAPH 2011. (pdf)
  • Light Mixture Estimation for Spatially Varying White Balance. Eugene Hsu, Tom Mertens, Sylvain Paris, Shai Avidan, and Frédo Durand. SIGGRAPH 2008. (pdf)
  • Fast Separation of Direct and Global Components of a Scene using High Frequency Illumination. S.K. Nayar, G. Krishnan, M. D. Grossberg, R. Raskar. SIGGRAPH 2006. (pdf)
  • Optical Computing for Fast Light Transport Analysis. Matthew O'Tool and Kiriakos Kutulakos. SIGGRAPH Asia 2010. (pdf)

Week 13: Internet Image Collections

Nov 21: Internet Image Collections (Slides)

  • Photo Clip Art. Jean-François Lalonde, Derek Hoiem, Alexei A. Efros, Carsten Rother, John Winn, Antonio Criminisi. SIGGRAPH 2007. (pdf)
  • Scene Completion Using Millions of Photographs. James Hays, Alexei A. Efros. SIGGRAPH 2007. (pdf)
Optional Readings
  • 80 million tiny images: a large dataset for non-parametric object and scene recognition. Antonio Torralba, Rob Fergus and William Freeman. IEEE PAMI 2008. (pdf)

Nov 23: No Class

Week 14

Nov 28:Internet Image Collections (Student Presentations) (Geolocation Slides,Scene Summarization Slides,Where is Waldo Slides)

  • Image Sequence Geolocation with Human Travel Priors. Evangelos Kalogerakis, Olga Vesselova, James Hays, Alexei A. Efros, Aaron Hertzmann. ICCV 2009. ( pdf)
  • Scene Summarization for Online Image Collections. Ian Simon, Noah Snavely and Steven M. Seitz. ICCV 2007. (pdf)
Optional Readings
  • Where's Waldo: Matching People in Images of Crowds. Rahul Garg, Deva Ramanan, Steve Seitz and Noah Snavely. CVPR 2011. (pdf)

Nov 30: (Student Presentations)

  • Yean Jin Lee & Yin-Chia Yeh,
  • Dai Bui,
  • Armin Samaii, Tim Althoff,
  • Aaron Eidelson & Dustin Shean,
  • Jiamin Bai,
  • Jeff Donahue,
  • Nancy Wang
  • Wesley Willett

Week 15

Dec 5: Student Presentations

  • Vera Dadok,
  • Michael Tao,
  • Eileen Bai & Philip Ly
  • Sally Ahn & Soham Mehta,
  • Vasily Volkov & Stacy Hsueh
  • Sean Arietta,
  • Nikhil Naikal
  • Hong Wu & Viraj Kulkarni, Rohan Nagesh
  • Robin Gaestel & Moeka Takagi,

Dec 7: No Lecture - RRR Week

Week 16

Dec 12: Final Exam Week


Course Numbers: CS294-69 for grads, CS194-69 for undergrads
Instructor: Maneesh Agrawala (maneesh at cs.berkeley.edu)
GSI: Floraine Berthouzoz (floraine at cs.berkeley.edu)
Meeting: 320 Soda Hall, Mon-Wed 5-6:30pm

Discussion Board: CS294-69 discussion on Piazza (Signup for the service and look for UCB CS294-69)

Office Hours:

  • Maneesh: 535 Soda Hall, M: 4-5pm and by appointment
  • Floraine: by appointment

Textbooks: There is no textbook for this course. Readings will be linked from this wiki.

Got Questions?

If you have questions related to this class, post a message onto the discussion board. Discussing ideas and responding to questions onto the discussion board will count towards your class participation for this course. We will also check the discussion board regularly to help with your questions.

If a question cannot be resolved through the discussion board, send mail to the GSI of this class, Floraine Berthouzoz (floraine at cs.berkeley.edu) and use CS294-69 in the subject line. This class being a advanced course, we assume that you are comfortable programming. We will thus not respond to questions related to your code, but we will of course make sure that you understand at a high level the algorithms that you need to implement.

If you have questions about research opportunities, please email Maneesh Agrawala (maneesh at cs.berkeley.edu) or stop by during his office hours.

Format and Requirements

The format of the course will be discussion based. Each week the instructor will present an overview of a topic on Mondays and students will lead presentations or structured critiques of selected papers on Wednesdays. All students will be expected to actively discuss each paper each week.

In addition students will be asked to implement the algorithms described in one or two research papers every couple of weeks. The goal of these implementations is to build up your programming skills and create a toolbox of algorithms that may be suitable for further research projects. We will not be grading these assignments in detail. We would like you to simply demonstrate that the basic ideas work.

The final weeks of the course will be devoted to implementing a final project. We expect the final project to make a new research contribution in some aspect of image manipulation and computational photography.

Class Participation and Presentation (20%)

Short Programming Assignments (30%)

Final Project (50%)

Plagiarism Policy: Assignments should consist primarily of your original work, building off of others' work--including 3rd party libraries, public source code examples, and design ideas--is acceptable and in most cases encouraged. However, failure to cite such sources will result in score deductions proportional to the severity of the oversight.

How to use and edit this wiki

New to wikis? Read the Wiki editing guide.

To contribute to this wiki, you'll need to first create an account. Please use your full name as your user name as in this example. Afterwards, please add some descriptive information about yourself on your personal page -- click your login name (next to the person icon) at the top of the page to access your personal page.


The lectures slides for this course are based on those of many other instructors. I am especially grateful to Fredo Durand, Alexei Efros, Bill Freeman, James Hays, Derek Hoiem and Marc Levoy, for the work they have put into preparing excellent teaching materials.

[add comment]
Personal tools