What are MDMs?

I created images in the gallery above using a unique and strange camera lens: the Canon RF5.2mm f/2.8L Dual Fisheye .

Here's a sample image that comes straight out of the RF5.2mm f/2.8L Dual Fisheye lens:

Each of the two lenses on this lens captures a 190° field of view vertically and horizontally, meaning the lens captures photons from behind itself. Both of the circular image areas above contain a view of the other lens.

Most humans can see in three dimensions because, among other reasons, we have two eyes separated by a short distance known as your interpupillary distance, or IPD. When I captured a photo with this lens, I was capturing the same subject from two unique points of view separated by 60mm. If we manipulate and view the images in a specific way, we see what the camera captured in 3D.

Using Canon's EOS VR Utility , we process the raw images into 8192x4096px equirectangular images, which maps the surface of a spherical image onto a flat image. The processing software also masks out areas of the image that can't comfortably be viewed in 3D, such as the nearby view of the other lens.

Using popular 3D graphics library Three.js , we project each half of those equirectangular images to the inside of their own half-spheres. We then position those spheres in front of the virtual camera. The left eye will see the left image mapped to one sphere, and the right eye will see the right image mapped to a different sphere in the same position.

Multidimensional Memory Audio Capture

Using a Zoom H2n Portable Audio Recorder , I captured Ambisonic B-format audio along with many of these photos. This audio is processed and played back using the AmbisonicsJS library . If you wear stereo headphones while listening to this audio, it will sound like the audio's source is coming from a location in three-dimensional space.

There's something very special and unique about pairing a still photograph with spatial audio. Here's what's important to consider while capturing spatial audio for MDM:

• To maintain the illusion of presence in the real world, it's critical to capture audio from the same location, orientation, and time as the original photograph.
• Capture audio for at least 30 seconds.
• Create audio that loops infinitely. For maximum effect, ensure the audio's loop point is imperceptible.
• As with capturing audio more generally, it's important to maintain similar perceived loudness between individual captures. Large jumps in volume between captures can be uncomfortable for viewers.

Viewing MDMs

There are two ways to view the Multidimensional Memories in this gallery:

1. In three dimensions, using a VR headset like Meta Quest 3 or Valve Index, rotating your head to look around
2. In two dimensions, on your computer or phone screen, using your mouse or finger to pan around

If you have a VR headset, and you're running the necessary software, you'll see an "Enter VR" button at the bottom of each image.

If you click that button and put on your headset, you'll see the photos in 3D. You can use the buttons to your left and right to move between images. The red button below you will exit VR mode.

If you're one of the people out there with a VR setup capable of viewing these images, I'd love to hear about your experience. Contact me, and let's chat.