Thread by @docmilanfar

(1/n) This is my brief and incomplete history of digital and computational photography.

Part I: From film to digital standalone cameras.

(Notes: These are based on a talk I've given many times. Opinions are mine, not my employer's).

(2/n) Smartphones have changed the world -- of this there is no doubt. Nearly everyone on earth who can afford to have one does. They've spawned trillion-dollar economy & decimated the digital camera market.

(3/n) Most importantly, they changed the way we live, relate to others & the world. The first camera phone was sold in '00 when taking pictures with your phone was an oddity, and sharing pictures online was unheard-of. In the decades that followed, we saw the world in a new way.

(4/n) Photography is historically fascinating, and many social aspects of photography have deep roots in the past.
But here, I'll mostly focus on the recent developments of the last two decades. Here's an over-simplified timeline.

(4/n) The 1st digital camera was built in '75 by Steve Sasson at Kodak: “[Kodak execs] were convinced that no one would ever want to look at their pictures on a screen.” The image was captured on a cassette tape, which could then be shown on a TV screen via a playback computer.

(5/n) Though Sasson's invention was not widely appreciated at the time, 35 years later he received a long overdue recognition: The National Medal of Technology, the highest honor that can be given to an engineer by the President of the United States.

(6/n) Digital cameras would go on to fulfill a strong human craving for instant gratification. But make no mistake, this craving has its roots in the long past. Interestingly, the original Polaroid built for this very purpose became widely used around the same time.

(7/n) Digital cameras became commercially available in '90. One of the very first was the Fotoman: with 0.1Mp and a steep price tag, it wasn't a big hit. Digital SLRs were heavy, awkward, and absurdly pricey. Some attached external storage had to be shoulder-carried separately.

(8/n) In the 90's all cameras were CCD based. But CMOS technology would eventually win over by being cheaper and more power efficient. While CMOS was introduced in '93, it would be at least another decade before CMOS-based SOCs would enable mass affordable (and mobile) cameras.

(9/n) The early 2000's brought digital compacts and SLRs (still using CCDs.) The "point-and shoot" compacts introduced easy photography to the masses. Most people in their 20s (e.g. my kids) will remember their childhood thru photos on these compacts. SLRs were 10x more pricey.

(10/n) Interestingly, if we fast-forward to today for a comparison, the point-shoots are still about the same price, and DSLRs are still about 10x pricier. The difference? Optics are far better on both, w/ more pixels, and all use CMOS. i.e. we get far better value for the price.

(11/n) Despite the great value, today ~90% of all photos are captured on smartphones, not on digital standalone cameras. Why? Because computational photography changed the rules, bringing to photography new modes of capture, processing, and sharing.

That story next in Part II

(12/n) As promised, here is Part II:

From digital standalone cameras to mobile devices that are more camera than phone.

Complex hardware in cameras has given way to small form factors. Yet, high expectations remain for software / clever algorithms to produce great images.

(13/n) The 1st patent on mobile cameras was 25 yrs ago by Kodak. The illustrations are strikingly prescient of what early camera phones would look like. Yet Kodak didn't push hard to commercialize either digital or mobile cameras for fear they would eat into their film business.

(14/n) The first commercial camera phone was put on the market in Japan only five years later in 2000, around the same time as the compact cameras' height of popularity.

https://twitter.com/docmilanfar/status/1246986617534861319?s=20

(15/n) The J-phone by Sharp was the first commercially sold mobile device to have a camera. But it wasn't much of a camera. And the device, at $500, was comparable in price to compact cameras that took far better pictures. It was largely a commercial flop but a part of history.

(16/n) The "flip" phones were early success stories of the camera phone market. They appeared in '02 in the US, marketed by the large mobile carriers. In '03 the first forward-facing camera appeared. Back then it was named the "video call" camera. The "selfie" area had begun.

(17/n) The market was upended in 2007 with the introduction of the iPhone. But the iPhone was all about the display, the configurable UI and the app ecosystem. The camera was just an afterthought. Steve Jobs' historic 90min keynote mentioned it just once

(18/n) In the midst of the so-called "megapixel wars", the original iPhone camera was not a major selling point, nor produced competitive images by any subjective quality standards.

(19/n) The inflection point was 2010 when two key factors converged to make smartphones (iOS and Android) the camera of choice for consumers: 1) improved display resolution; 2) 4G networks. Essentially, these innovations made photos easy to take, view, and share right *on-device*

(20/n) Starting in 2010, the iPhone 4 and other smartphones deployed screen resolutions better than 300 dpi, rivaling the Retina display on laptops. The history of improving mobile displays correlates beautifully with the collapse of the standalone camera market.

(21/n) Meanwhile, the effect of faster network speeds brought by 4G contributed to the explosion of social media/sharing aspects of mobile photography. A typical image that might have been impractical to share before could now be shared in half a second.

(22/n) The next decade would be the era of mobile imaging. But mobile photography-as-popular-artform didn't come altogether organically. It was popularized by a few notable influencers like @chasejarvis. His "Best Camera" app in '09 was pioneering.

(23/n) This was the first photo app that allowed you to filter, crop, and share photos to social networks. It beat Instagram to market by a year. It also beat Instagram to 1M downloads. But in the end lost out to the free Instagram app.

Compelling story: chasejarvis.com/blog/my-bigges…

(24/n) The engineering challenge in the last decade has been to see if we can produce images with a mobile camera that rival those made with a point-and-shoot or even DSLR. There are of course huge physical limitations. A typical mobile camera gathers 100s of times less light.

(25/n) The mobile devices' form factor, battery, memory bandwidth etc. also dictate how large the image sensor can be. This adversely affects light-gathering efficiency.

(26/n) Cornered by physical limitations of a single camera, device makers began to compete not just with mega-pixels, but with added hardware. If you can't have a bigger, better camera, then have more of them! But each camera served a different use (zoom, uwide, portrait, etc)

(27/n) What we really wanted all along though was (a) more light, (b) better dynamic range, and (c) higher (true optical) resolution. We can't use a flash all the time. Long exposures give motion blur. Two bracketed exposures are hard to fuse without ghosting. Compromise: Bursts

(28/n) Burst processing is the workhorse of modern computational photography. When we press the shutter button, instead of a single shot we capture a series of them. We then very carefully align, merge, and enhance them to produce the final picture. Adaptation to the scene is key

(29/n) Compare this to the classic pipeline using a single raw capture and interpolation to produce the final image. Many fewer degrees of freedom to balance motion blur/noise/resolution. And 2/3 of colors in your picture are made-up!
Great tutorial here
eecs.yorku.ca/~mbrown/ICCV19…

(30/n) One of the first commercial uses of burst photography was our pipeline in Google Glass. This evolved to what is now the HDR+ and Super Res Zoom pipelines powering the Pixel phones' cameras. Many (if not most) other device makers employ burst imaging in one form or another.

(31/n) The punchline is that we've gotten closer to what standalone compact and DSLR cameras can do. While we'll likely never reach the quality of a full-frame DSLR, we have proven we can get comparable or better images than micro 4/3 cameras.

(32/n) The industry has come a long way in just 10 years. Most devices can do very high quality zoom by smartly combining optics, hardware, and software. And we can shoot stunning photos in extreme low light.

(33/n) Consumers will continue to expect high quality from their smartphones cameras.
(Curious note: making a phone call does not even appear in the top 5 list of important features!)

(34/n) But nearly everyone who can afford to have a smartphone already has one. And consumers are more value conscious; waiting longer to replace phones. Device makers will be wise to bring more value to users. And these constraints will spur new innovations in the decade to come

(35/35) I would be remiss to end this thread without a nice cat picture (yes, on a mobile device). This is our cat Fuzzy. She keeps me company in the back yard as I work on my tweetorials.

Thank you for reading. 😀🙏