CS294-69 Image Manipulation and Computational Photography
From CS294-69 Image Manipulation and Computational Photography Fa11
Digital cameras have made it very easy and cheap to capture the world around us. Yet, the images our cameras record often do not match our experience of a particular scene, or our communicative intent. The best visual depictions usually require extensive editing of the raw imagery. Today, photographers primarily use low-level image-processing tools (e.g. Photoshop, GIMP, Picasa, etc.) and often end up directly touching individual pixels. In this course we will study techniques and algorithms that allow users to manipulate visual imagery at a much higher level. We will focus on recent research that brings together ideas from computer graphics, computer vision, image processing and human-computer interaction.
The course is open to graduates (sign up for CS294-69) and advanced undergraduates (CS194-69) who are familiar with basic concepts in computer graphic, computer vision and signal processing. We will also assume good understanding of linear algebra. Students who have taken an undergraduate graphics course will have the necessary background. Finally we will assume a willingness to learn Matlab or other suitable languages/APIs as necessary. Send me (Maneesh) email if you are worried about whether you have the background for the course.
Schedule
Note: Readings may be changed and updated as the semester progresses.
Week 1: Introduction/Color
Aug 29: Introduction (Slides)
- Assignment 0: due by 5pm Mon Sep 12: Hybrid Images
Aug 31: Forming Color Images (Slides)
- Readings
- Chapters 2.2.1, 2.2.2 and 2.3.2 (pp.60-68 and pp.80-90). Computer Vision: Algorithms and Applications. Rick Szeliski. 2010. (pdf)
- Accessing Matlab for Assignment 0
- Using the username and password that we provide, you can log into the Windows PCs in 199 Cory or the Linux PCs in 330 Soda and run /share/b/bin/matlab from the command line. This will open Matlab for you. You can find more information on how to run Matlab on this website.
- Signing up for Paper Presentations
- You can sign up here for the paper presentations. Remember that if an undergraduate student signed up as a paper presenter, only graduate students can sign up for being the discussant of that paper and vice versa.
Week 2: Sampling, Filtering, Reconstruction
Sep 5: Holiday - no class
Sep 6: MATLAB Tutorial - The tutorial will take place 3 to 4pm at 330 Soda Hall. Here is the document that I used for the Matlab introduction today. Change the file extension to *.m and you can directly open this file in Matlab.
Sep 7: Sampling, Filtering, Reconstruction (Slides)
- Readings
- Optional Readings
- Laplacian Pyramid as a Compact Image Code. E.H. Adelson and P.J. Burt. IEEE Trans. on Communication 1983. (pdf)
- Student Presentations
- We posted instructions for helping you prepare the student presentations.
Week 3: Texture Synthesis
Sep 12: Texture Synthesis (Slides)
- Due Assignment 0: due by 5pm : Hybrid Images
- Assignment 1: due by 5pm Mon Sep 26: Texture Synthesis
- Readings
- Optional Readings
- Video Textures. Arno Schoedl, Richard Szeliski, David H. Salesin, Irfan Essa. SIGGRAPH 2000 (pdf)
- Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images. Jeremy De Bonet. SIGGRAPH 97. (pdf)
- Fast Texture Synthesis using Tree-structured Vector Quantization. Li-Yi Wei and Marc Levoy. SIGGRAPH 2000 (pdf)
Sep 14: Texture Synthesis (Student Presentations) (Image Quilting Slides, Region Filling Slides, PatchMatch Slides)
- Readings
- Image Analogies. Aaron Hertzmann, Charles E. Jacobs ,Nuria Oliver, Brian Curless and David H. Salesin. SIGGRAPH 2001 (pdf)
- Region-Filling and Object Removal by Exemplar-Based Inpainting. Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. CVPR 2004. (pdf)
- PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. Connelly Barnes, Eli Shechtman, Adam Finkelstein and Dan B Goldman. SIGGRAPH 2009 (pdf)
- Optional Readings
- Image Quilting for Texture Synthesis and Transfer. Alexei Efros and William Freeman. SIGGRAPH 2001. (pdf)
- Image Completion with Structure Propagation. Jian Sun, Lu Yuan, Jiaya Jia and Heung-Yeung Shum. SIGGRAPH 2005.(pdf)
- Synthesizing Structured Image Hybrids. Eric Risser, Charles Han, Rozenn Dahyot and Eitan Grinspun. SIGGRAPH 2010 (pdf)
Week 4: Camera Optics/Lenses/Depth of Field
Sep 19: Camera Optics/Lenses/Depth of Field (Slides)
- Readings
- Chapter 2.1 (focus on 2.1.5 (projection) and 2.1.6 (lens distortion)). Computer Vision: Algorithms and Applications. Rick Szeliski. 2010. (pdf)
Sep 21: Camera Optics/Lenses/Depth of Field (Student Presentations) (Blur to Affect Perceived Distance and Size Slides, Defocus Magnification Slides)
- Readings
- Using Blur to Affect Perceived Distance and Size. Robert Held, Emily Cooper, James O’Brien, and Martin Banks. TOG 2010. (pdf)
- Defocus Magnification. Soonmin Bae and Frédo Durand. EUROGRAPHICS 2007. (pdf)
- Capturing and Viewing Gigapixel Images. Johannes Kopf, Matt Uyttendaele, Oliver Deussen, Michael F. Cohen. SIGGRAPH 2007. (pdf)
Week 5: Light Fields and Camera Arrays
Sep 26: Light Fields and Camera Arrays (Slides)
- Due Assignment 1: due by 5pm : Texture Synthesis
- Assignment 2: due by 5pm Mon Oct 10: Light Fields/HDR/Tone-Mapping
- Readings
- Optional Readings
- The Lumigraph. Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski and Michael F. Cohen. SIGGRAPH 96. (pdf)
Sep 28: Light Fields and Camera Arrays (Student Presentations) (Dynamically Reparameterized Light Fields Slides, Unstructured Lumigraph Slides, Fourier Slice Photography Slides)
- Readings
- Optional Readings
Week 6: HDR/Tone-Mapping
Oct 3: HDR/Tone-Mapping (Slides)
- Readings
- Recovering High Dynamic Range Radiance Maps from Photographs. Paul E. Debevec and Jitendra Malik. SIGGRAPH 97. (pdf)
- LCIS: A Boundary Hierarchy for Detail-Preserving Contrast Reduction. Jack Tumblin and Greg Turk. SIGGRAPH 1999. (pdf)
- Fast Bilateral Filtering for the Display of High-Dynamic-Range Images. Fredo Durand and Julie Dorsey. SIGGRAPH 2002. (pdf)
- Optional Readings
- Tone Reproduction for Realistic Images. Jack Tumblin. Computer Graphics and Applications 1993. (pdf)
- Bilateral Filtering: Theory and Applications. Sylvain Paris, Pierre Kornprobst, Jack Tumblin, and Frédo Durand. Foundations and Trends in Computer Graphics and Vision 2009. (pdf)
- Digital Photography with Flash and No-Flash Image Pairs. Georg Petschnigg, Maneesh Agrawala, Hugues Hoppe, Richard Szeliski, Michael Cohen and Kentaro Toyama. SIGGRAPH 2004. (pdf)
- Flash Photography Enhancement via Intrinsic Relighting. Elmar Eisemann and Frédo Durand. SIGGRAPH 2004. (pdf)
Oct 5: HDR/Tone-Mapping (Student Presentation) (Domain Transform for Edge-Aware Image and Video Processing Slides,Tone Mapping for Low-Light Conditions,Photographic Tone Reproduction for Digital Images)
- Readings
- Photographic Tone Reproduction for Digital Images. Erik Reinhard, Michael Stark, Peter Shirley and James Ferwerda. SIGGRAPH 2002. (pdf)
- Perceptually Based Tone Mapping for Low-Light Conditions. Adam G. Kirk and James F. O’Brien. SIGGRAPH 2011. (pdf)
- Real-time Edge-Aware Image Processing with the Bilateral Grid. Jiawen Chen, Sylvain Paris, and Frédo Durand. SIGGRAPH 2007. (pdf)
- Optional Readings
Week 7: Gradient Domain Techniques
Oct 10: Gradient Domain Techniques (Slides)
- Due Assignment 2: due by 5pm : Light Fields/HDR/Tone-Mapping
- Assignment 3: due by 5pm Mon Oct 24: Gradient Domain/Resizing and Warping
- Readings
- Optional Readings
- Image Editing in the Contour Domain. James Elder and Rick Goldberg. CVPR 1998. (pdf)
Oct 12: Gradient Domain Techniques (Student Presentations) (Gradient Shop Slides,Diffusion Curves Slides,Efficient Gradient-Domain Compositing Using Quadtrees Slides)
- Readings
- Optional Readings
Week 8: Image Resizing and Warping
Oct 17: Image Resizing and Warping (Slides)
- Readings
- Optional Readings
- Optimizing Content-Preserving Projections for Wide-Angle Images. Robert Carroll, Maneesh Agrawala and Aseem Agarwala. SIGGRAPH 2009. (pdf)
- Image Warps for Artistic Perspective Manipulation. Robert Carroll, Aseem Agarwala, Maneesh Agrawala. SIGGRAPH 2010. (pdf)
- Feature-Aware Texturing. Ran Gal, Olga Sorkine and Daniel Cohen-Or. EGSR 2006. (pdf)
Oct 19: Image Resizing and Warping (Student Presentations) (Improved Seam Carving - Multi-Operator Media Retargeting Slides,Comparative Study of Image Retargeting Slides,Nonlinear Disparity Mapping for Stereoscopic 3D Slides)
- Readings
- Optional Readings
Week 9: Matting
Oct 24: Matting (Slides)
- Due Assignment 3: due by 5pm: Gradient Domain/Resizing and Warping
- Final Project: Proposals due 10/31, Proposal Presentations 10/31 and 11/2, Final Presentations 11/28 and 11/30, Final Paper 12/7
- Readings
- Optional Readings
- Alpha Estimation in Natural Images. Mark Ruzon and Carlo Tomassi. CVPR 2000. (pdf)
Oct 26: Matting (Student Presentations) (Poisson Matting Slides,A Closed-Form Solution to Natural Image Matting Slides)
- Readings
Week 10: Student Project Presentations
Oct 31: Student Presentations: Final Project Proposal
- Rohan Nagesh,
- Michael Tao,
- Eileen Bai & Philip Ly
- Sally Ahn & Soham Mehta,
- Vasily Volkov & Stacy Hsueh
- Sean Arietta,
- Jiamin Bai,
- Nikhil Naikal
Nov 2: Student Presentations: Final Project Proposal
- Yean Jin Lee & Yin-Chia Yeh,
- Dai Bui,
- Armin Samaii, Tim Althoff,
- Aaron Eidelson & Dustin Shean,
- Robin Gaestel & Moeka Takagi,
- Vera Dadok,
- Hong Wu & Viraj Kulkarni,
- Jeff Donahue,
- Nancy Wang
Week 11: Graphcut Techniques
Nov 7: Graphcut Techniques (Slides)
- Readings
Nov 9: Graphcut Techniques (Student Presentations) (Grabcut and Lazy-Snapping Slides,Video Cutout Slides,Photographing Long Scenes Slides)
- Readings
- Optional Readings
- Lazy Snapping. Yin Li, Jian Sun, Chi-Keung Tang and Heung-Yeung Shum. SIGGRAPH 2004. (pdf)
- Photographing Long Scenes with Multi-Viewpoint Panoramas. Aseem Agarwala, Maneesh Agrawala, Michael Cohen, David Salesin and Richard Szeliski. SIGGRAPH 2006. (pdf)
- Panoramic Video Textures. Aseem Agarwala, Ke Colin Zheng, Chris Pal, Maneesh Agrawala, Michael Cohen, Brian Curless, David Salesin and Richard Szeliski. SIGGRAPH 2005. (pdf)
Week 12: Image Relighting
Nov 14: Image Relighting (Slides)
- Readings
- Acquiring the Reflectance Field of a Human Face. Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. SIGGRAPH 2000. (pdf)
- Conveying Shape and Features with Image-Based Relighting. David Akers, Frank Losasso, Jeff Klinger, Maneesh Agrawala, John Rick and Pat Hanrahan. Visualization 2003. (pdf)
- Environment Matting and Compositing. Zongker, Werner, Curless, and Salesin, and Szeliski. SIGGRAPH 1999. (pdf)
- Optional Readings
- Multiscale Shape and Detail Enhancement from Multi-light Image Collections. Rannan Fattal, Maneesh Agrawala and Szymon Rusinkiewicz. SIGGRAPH 2007. (pdf)
Nov 16: Image Relighting (Student Presentations) (Intrinsic Images Slides,Illumination Decomposition Slides,Inverse Light Transport Slides)
- Readings
- Optional Readings
- Illumination Decomposition for Material Recoloring with Consistent Interreflections. Robert Carroll, Ravi Ramamoorthi and Maneesh Agrawala. SIGGRAPH 2011. (pdf)
- Light Mixture Estimation for Spatially Varying White Balance. Eugene Hsu, Tom Mertens, Sylvain Paris, Shai Avidan, and Frédo Durand. SIGGRAPH 2008. (pdf)
- Fast Separation of Direct and Global Components of a Scene using High Frequency Illumination. S.K. Nayar, G. Krishnan, M. D. Grossberg, R. Raskar. SIGGRAPH 2006. (pdf)
- Optical Computing for Fast Light Transport Analysis. Matthew O'Tool and Kiriakos Kutulakos. SIGGRAPH Asia 2010. (pdf)
Week 13: Internet Image Collections
Nov 21: Internet Image Collections (Slides)
- Readings
- Optional Readings
- 80 million tiny images: a large dataset for non-parametric object and scene recognition. Antonio Torralba, Rob Fergus and William Freeman. IEEE PAMI 2008. (pdf)
Nov 23: No Class
Week 14
Nov 28:Internet Image Collections (Student Presentations) (Geolocation Slides,Scene Summarization Slides,Where is Waldo Slides)
- Readings
- Optional Readings
- Where's Waldo: Matching People in Images of Crowds. Rahul Garg, Deva Ramanan, Steve Seitz and Noah Snavely. CVPR 2011. (pdf)
Nov 30: (Student Presentations)
- Yean Jin Lee & Yin-Chia Yeh,
- Dai Bui,
- Armin Samaii, Tim Althoff,
- Aaron Eidelson & Dustin Shean,
- Jiamin Bai,
- Jeff Donahue,
- Nancy Wang
- Wesley Willett
Week 15
Dec 5: Student Presentations
- Vera Dadok,
- Michael Tao,
- Eileen Bai & Philip Ly
- Sally Ahn & Soham Mehta,
- Vasily Volkov & Stacy Hsueh
- Sean Arietta,
- Nikhil Naikal
- Hong Wu & Viraj Kulkarni, Rohan Nagesh
- Robin Gaestel & Moeka Takagi,
Dec 7: No Lecture - RRR Week
Week 16
Dec 12: Final Exam Week
Information
Course Numbers: CS294-69 for grads, CS194-69 for undergrads
Instructor: Maneesh Agrawala (maneesh at cs.berkeley.edu)
GSI: Floraine Berthouzoz (floraine at cs.berkeley.edu)
Meeting: 320 Soda Hall, Mon-Wed 5-6:30pm
Discussion Board: CS294-69 discussion on Piazza (Signup for the service and look for UCB CS294-69)
Office Hours:
- Maneesh: 535 Soda Hall, M: 4-5pm and by appointment
- Floraine: by appointment
Textbooks: There is no textbook for this course. Readings will be linked from this wiki.
Got Questions?
If you have questions related to this class, post a message onto the discussion board. Discussing ideas and responding to questions onto the discussion board will count towards your class participation for this course. We will also check the discussion board regularly to help with your questions.
If a question cannot be resolved through the discussion board, send mail to the GSI of this class, Floraine Berthouzoz (floraine at cs.berkeley.edu) and use CS294-69 in the subject line. This class being a advanced course, we assume that you are comfortable programming. We will thus not respond to questions related to your code, but we will of course make sure that you understand at a high level the algorithms that you need to implement.
If you have questions about research opportunities, please email Maneesh Agrawala (maneesh at cs.berkeley.edu) or stop by during his office hours.
Format and Requirements
The format of the course will be discussion based. Each week the instructor will present an overview of a topic on Mondays and students will lead presentations or structured critiques of selected papers on Wednesdays. All students will be expected to actively discuss each paper each week.
In addition students will be asked to implement the algorithms described in one or two research papers every couple of weeks. The goal of these implementations is to build up your programming skills and create a toolbox of algorithms that may be suitable for further research projects. We will not be grading these assignments in detail. We would like you to simply demonstrate that the basic ideas work.
The final weeks of the course will be devoted to implementing a final project. We expect the final project to make a new research contribution in some aspect of image manipulation and computational photography.
Class Participation and Presentation (20%)
Short Programming Assignments (30%)
Final Project (50%)
Plagiarism Policy: Assignments should consist primarily of your original work, building off of others' work--including 3rd party libraries, public source code examples, and design ideas--is acceptable and in most cases encouraged. However, failure to cite such sources will result in score deductions proportional to the severity of the oversight.
How to use and edit this wiki
New to wikis? Read the Wiki editing guide.
To contribute to this wiki, you'll need to first create an account. Please use your full name as your user name as in this example. Afterwards, please add some descriptive information about yourself on your personal page -- click your login name (next to the person icon) at the top of the page to access your personal page.
Credits
The lectures slides for this course are based on those of many other instructors. I am especially grateful to Fredo Durand, Alexei Efros, Bill Freeman, James Hays, Derek Hoiem and Marc Levoy, for the work they have put into preparing excellent teaching materials.
