# Reality Design Inspire — AR catalog A catalog of the most creative augmented-reality creators, from the first pioneers to today, and their AR works, compiled by Reality Design Lab as idea material for teaching. Each work lists its video, core idea, key technique and a classroom exercise. https://augmented.reality.design · 2026-09-28 · 711 creators · 2991 works ## How an AI assistant should use this file - Ground AR ideas in specific works below and name the work and creator you are drawing on. - Combine interaction patterns from different works to propose new ideas. - Reuse and adapt the “Try it” exercises for lessons and workshops. - Do not invent details that are not stated here; the video link is the reference. ## Key creators and guided tours ### Creative-coding roots The most-cited people in the network. Many later creators were found by following their trail. #### Zach Lieberman Sound, text and the body placed in space: the origin of poetic AR. Zach Lieberman co-founded openFrameworks and the School for Poetic Computation. For twenty years he has done one thing again and again: give invisible things — sound, handwriting, motion — a shape in space. His AR pieces are often technically simple; their strength is an idea that is clear, playful and instantly understood. **What to learn:** How to turn a technical capability (tracking, recording, drawing a line) into a poetic idea you can explain in one sentence. 1. **Drawn** (2005) — The 2005 starting point: hand-drawn ink comes alive once the camera sees it. Watch this first to understand his idea of working between paper and screen. https://vimeo.com/4732884 2. **The EyeWriter** (2009) — Eye tracking lets a paralyzed graffiti artist draw again; also a model of open-source collaboration. Technology in service of one specific person. https://www.youtube.com/watch?v=84H-xLrLvvk 3. **Audio in AR space** (2017) — The seed of this project: every sound stays where it was made, and walking back through it plays it in reverse. The most poetic response to ARKit's arrival. https://vimeo.com/290238447 4. **Weird Type** (2018) — With Molmol Kuo, turned into an app anyone can use: write in the air and let text follow motion. The step from personal experiment to public tool. https://www.youtube.com/watch?v=UzNCGqE9MhE 5. **Poem World (Snap Spectacles)** (2021) — On Snap Spectacles, with Shantell Martin: a poem you read by walking through it. Your footsteps set the reading order. https://x.com/zachlieberman/status/1408879174429949953 #### Golan Levin A source of interactive-art education: making voice, shadow and hands visible as form. Golan Levin directs the STUDIO for Creative Inquiry at Carnegie Mellon and has taught many creative coders. His works often amplify a person's voice or body into visuals, and keep asking: how does a machine see a person? **What to learn:** Another meaning of “augment”: not adding virtual objects, but making what people already have — voice, gesture, shadow — visible. 1. **Messa di Voce (installation)** (2005) — Speech and song become shapes that pour out of the mouth. A classic of projection AR, made with Zach Lieberman and others. https://vimeo.com/221802940 2. **Footfalls** (2006) — Stomping makes virtual objects rain down onto your shadow. Sound, body and projection in one piece. https://vimeo.com/227566535 3. **Augmented Hand Series** (2014) — Your hand, transformed live: an extra finger, fingers that bend on their own. The smallest augmentation, the strongest bodily effect. https://vimeo.com/111951283 4. **ARKit re-code of Jeffrey Shaw's Golden Calf** (2017) — Jeffrey Shaw's 1994 Golden Calf, re-made with ARKit. Good for discussing AR's lineage. https://vimeo.com/269478756 #### Kyle McDonald The most-connected person in the network: face tracking and machine vision as artistic material. Kyle McDonald wrote ofxFaceTracker, the most popular early open-source face tracker, and many face-AR works are built on his code. His pieces keep testing what identity and privacy mean in front of a camera. **What to learn:** How to open-source a low-level capability (face tracking) and then use it to make work that is thought-provoking, even unsettling. 1. **Face Substitution** (2011) — Real-time face swap in 2011, years before Snapchat. The root of every face filter that followed. https://vimeo.com/29348533 2. **Sharing Faces** (2013) — People in two cities “swap” similar faces through a mirror: an AR mirror that connects strangers. https://vimeo.com/96549043 3. **Light Leaks** (2013) — Dozens of mirror balls and computer-vision-calibrated projectors fill a room with moving points of light. With Jonas Jongejan. https://vimeo.com/66167082 4. **Highsight** (2015) — A “virtual bungee jump through a real space”: visitors in a harness see the live feed of a camera dropping along a wire. Body sensation and viewpoint are split apart. https://vimeo.com/144061990 #### Design I/O — Theo Watson & Emily Gobeille The benchmark for children's interactive projection: full-body interaction, ideal for classroom re-creation. Design I/O — Theo Watson (co-founder of openFrameworks) and Emily Gobeille — has built large interactive spaces for museums for more than a decade. Children use their bodies, sand and paper to interact with a room-sized virtual ecosystem. **What to learn:** How to design for people who will never read instructions: no interface, no buttons, instant feedback to the body. 1. **L.A.S.E.R. Tag** (2007) — Graffiti on a building with a laser pointer (with Graffiti Research Lab). Architecture-scale projection AR, open-sourced. https://www.youtube.com/watch?v=LtZq2q43Jkc 2. **Funky Forest** (2007) — Children grow trees with their bodies and steer water to the forest. Where the Design I/O style began. https://vimeo.com/3872687 3. **Puppet Parade** (2011) — Raise your arms to puppeteer giant projected creatures. One of the most direct full-body interactions. https://vimeo.com/34824490 4. **Connected Worlds** (2015) — A large ecosystem at the New York Hall of Science: six habitats linked by water, where children decide together how the ecosystem evolves. A model of multiplayer collaboration. https://vimeo.com/131585517 5. **Living Library** (2016) — Turn a real page and the animals on it come alive. Paper plus projection; easy to adapt into a class assignment. https://vimeo.com/203193098 ### Research origins Many AR interaction ideas first appeared in these labs' papers and demo videos. #### Microsoft Research — Hrvoje Benko & Andy Wilson Turning a whole room into a display: the research source of projected spatial AR. Hrvoje Benko and Andy Wilson spent over a decade at Microsoft Research on “AR without glasses”: projectors plus depth cameras that make walls, tables and furniture interactive. Many later mixed-reality game ideas first appeared in their demos. **What to learn:** The spatial-AR mindset: understand the room's geometry first, then decide where content appears and how it aligns with reality. 1. **LightSpace** (2010) — 2010: several projectors and depth cameras turn any surface into a touchscreen, and you can carry content in your hand. https://www.youtube.com/watch?v=xx5kBqxyaHE 2. **IllumiRoom** (2013) — The game spills out of the TV onto the whole wall. Minimal hardware, maximum field of view. https://www.youtube.com/watch?v=L2w-XqW7bF4 3. **RoomAlive** (2014) — The living room becomes a game, with monsters climbing out of the walls. A forerunner of today's Quest mixed-reality games. https://www.youtube.com/watch?v=GYkRRbP7m8s 4. **Mano-a-Mano (Dyadic Projected SAR)** (2014) — Two people face off and each sees correct perspective. Multiplayer AR without headsets. https://www.youtube.com/watch?v=Df7fZAYVAIE 5. **Room2Room** (2016) — A remote person is captured in 3D and projected life-size into your room, sitting on your real couch. An early form of telepresence. https://www.youtube.com/watch?v=2o6krhxpUGk #### MIT Tangible Media Group — Hiroshi Ishii Giving digital information a physical body: the origin of shape-changing interfaces. Hiroshi Ishii's MIT Tangible Media Group proposed “Tangible Bits” and “Radical Atoms”: digital information should not be trapped behind glass but be touchable and movable. Their pin-display series remains among the most striking demos. **What to learn:** “Reverse AR”: instead of overlaying the virtual on the real, reality's physical shape changes with digital information. 1. **Sublimate** (2013) — Virtual graphics and a physical pin array merge: the same shape is half real, half projected. https://vimeo.com/63284274 2. **inFORM** (2013) — The most famous one: 900 moving pins take the shape of objects in real time, and even shake hands remotely. It changes what you think an interface is. https://vimeo.com/79179138 3. **Physical Telepresence** (2014) — A remote person pushes real objects on your table through the pin display. Telepresence moves from seeing to touching. https://vimeo.com/108402837 4. **TRANSFORM** (2014) — Pin displays become furniture and art at Milan Design Week. A research prototype turned into design language. https://vimeo.com/98880732 #### MIT Fluid Interfaces — Pattie Maes & Pranav Mistry Fingers and walls as the interface: wearable AR's first mass audience. In 2009 Pranav Mistry, in Pattie Maes's MIT Fluid Interfaces group, showed SixthSense: a projector and camera worn around the neck let you operate information on your palm, a newspaper or a wall. The TED talk made the world imagine “AR you carry” for the first time. **What to learn:** How to demonstrate a big idea with the cheapest hardware (projector, camera, colored finger caps) and a story people understand. 1. **SixthSense (WUW – Wear Ur World)** (2009) — Dial on your palm, play video on a newspaper, frame a photo with your fingers. The iconic demo of wearable projection AR. https://vimeo.com/17567999 2. **LuminAR** (2010) — A lamp that moves, projects and recognizes objects on the desk. AR hidden inside everyday furniture. https://vimeo.com/52899256 3. **AfterMath: Seeing the Future through AR** (2015) — Point a tablet at a physical scene and AR predicts what happens next — where things will fall or roll. It augments time, not only space. https://vimeo.com/130279501 4. **HoloARt: Painting with Holograms** (2016) — Painting in the air with HoloLens, blended with the real room. An early creation tool on headsets. https://vimeo.com/202652308 #### Ryo Suzuki — Programmable Reality Lab Draw a line and reality starts moving: the newest generation of AR interaction research. Ryo Suzuki leads the Programmable Reality Lab at the University of Calgary and publishes several AR interaction papers at CHI and UIST nearly every year. His work links hand-drawn sketches to real-world motion, ideal for education and science visualization. **What to learn:** Dynamic sketching: a line the user draws is no longer a static annotation but a live parameter that tracks, measures and simulates. 1. **RealitySketch** (2020) — Draw a line in AR and it binds to a real object, measuring angles and plotting graphs live. His signature work. https://www.youtube.com/watch?v=L0p-BNU9rXU 2. **RealityTalk** (2022) — While you speak, matching AR graphics and keywords appear; the presenter needs no slides. https://www.youtube.com/watch?v=vfIMeICV-7c 3. **Augmented Math** (2023) — Point at a static math problem in a textbook and it becomes an AR figure you can drag and animate. https://www.youtube.com/watch?v=Zv6JQ5T-qn0 4. **Augmented Physics** (2024) — Physics diagrams from a textbook become interactive simulations. A direct example of AR for education. https://www.youtube.com/watch?v=HUgYeA3BKfk ### Open prototypers Whenever a new device or API appears, they publish a small experiment. Ideal for a prototype-a-week style of teaching. #### Ian Curtis A spatial experiment every week: the 2025–26 frontier of WebAR and generated worlds. Ian Curtis (@XRarchitect) is an architecture-trained designer who worked at 8th Wall and Niantic and is now at World Labs. He posts his experiments on X and lists exactly which tools he used. His recent direction: placing AI-generated 3D worlds (Gaussian splats) into real rooms. **What to learn:** Prototyping in public: build a demo the night a new tool ships, film 30 seconds, and write down the stack. 1. **8th Wall moon portal** (2023) — 8th Wall's moon portal: open a door on the street and walk onto the moon. A model WebAR portal. https://64uauol9km.ufs.sh/f/SNo22QO8bFjm1Y3nWaemQEOeBiztFDvnIw7rqPp4adJoyxKf 2. **Step into a persistent redesign, 1:1** (2025) — AI designs a new living room, generates it in 3D, and VPS aligns it 1:1 to the real room so you can walk into the redesign. https://x.com/XRarchitect/status/1968356682888823060 3. **Off-axis projection with face tracking** (2026) — A plain webcam tracks your face and turns a laptop screen into a window with depth. 3D without a headset. https://x.com/XRarchitect/status/2008221154100580600 4. **Shrunk to action-figure size on the coffee table** (2026) — Shrink yourself to action-figure size and explore a tiny world on the coffee table. Changing scale is one of the most reliable sources of AR surprise. https://x.com/XRarchitect/status/2053874498630468078 5. **A 20-million-splat San Francisco on the table** (2026) — A 20-million-splat San Francisco on the table; move the phone closer and dive into the streets. City-scale tabletop AR. https://x.com/XRarchitect/status/2071471052086194197 #### Takashi Yoshinaga An AR experimenter across more than a decade and almost every device generation. Takashi Yoshinaga began with medical AR research in 2009 and has since prototyped on Tango, ARCore, HoloLens, iPad LiDAR and Quest 3; he is also an important educator in Japan's AR community. His experiments often solve real problems with clever tricks, such as streaming holograms through YouTube Live. **What to learn:** Device-independent ideas: re-make the same idea (copying reality, sending a hologram of a person) on each new generation of hardware and see how it improves. 1. **Streaming Holograms to ARCore via YouTube** (2019) — Holograms streamed to phone AR through YouTube Live. Solving a hard problem with existing infrastructure. https://www.youtube.com/watch?v=IeaHt6bBw1o 2. **Spatial Copy and Paste Shared with ARCore** (2020) — Select a piece of the real room on HoloLens 2, “copy” it as a mesh, and “paste” it into another person's ARCore phone view. https://www.youtube.com/watch?v=hEeXlnFK_24 3. **Holo Cat: Perspective Expression by Head Tracking** (2021) — Head tracking gives a cat on a flat screen correct perspective. Compare it with Ian Curtis's off-axis projection. https://www.youtube.com/watch?v=9kvIlsb8HQE 4. **Mixed Reality Door for Meta Quest 3** (2023) — A real door in the room becomes a portal: open it in Quest 3 passthrough and a virtual world is behind it. https://www.youtube.com/watch?v=VW7ELpkVmIQ 5. **DejaViewer: Replaying Past Motion in AR** (2025) — Replay a past moment of human motion on the spot, like an instant replay in space. https://www.youtube.com/watch?v=kau5hRnEGdI #### Laan Labs (Chris Laan) The most imaginative early ARKit prototypes, several of which became products. In the months after ARKit launched in 2017, Chris Laan's Laan Labs released a string of viral demos: an AR tape measure, spray paint in the air, cockroaches crawling everywhere. They defined the early picture of what phone AR could do, and some became hit apps. **What to learn:** One API capability = one idea: plane detection, feature points and occlusion can each become a small, playful app. 1. **AirMeasure – AR Tape Measure** (2017) — An AR tape measure: one of the first proofs that phone AR was useful. Apple later built in its own Measure app. https://www.youtube.com/watch?v=z7DYC_zbZCM 2. **ARSpray – AR Spray Paint** (2017) — Use the phone as a spray can to paint on walls and in the air. Spatial drawing for everyone. https://www.youtube.com/watch?v=xlvaMxTfwBg 3. **AR Roaches Crawling on Everything** (2018) — A swarm of cockroaches crawls over real tables and objects. Spatial understanding used for instant horror. https://www.youtube.com/watch?v=nnrK5ziihx8 4. **Augmented Reality 3D Gaussian Splatting in Real Time** (2023) — 2023: Gaussian splats rendered in real-time AR on a phone. From 2017 point clouds to today's new representation. https://www.youtube.com/watch?v=jfiG2tohCAo 5. **AirDraw – Finger Paint AR for Vision Pro** (2024) — Finger-painting in the air on Vision Pro. The same spatial-drawing idea, re-made on a new device. https://www.youtube.com/watch?v=x_Vo7I8pc2s ### Art & stage Answers to “why make AR at all”: technology turned into a point of view, a performance, or an experience in public space. #### Daito Manabe / Rhizomatiks AR as live performance: dancers, drones and mixed-reality stages. Daito Manabe and Rhizomatiks work with choreographer MIKIKO and the dance company ELEVENPLAY, combining tracking, projection, drones and AR broadcast into live shows — from Perfume's stages to the Tokyo segment of the Rio 2016 closing ceremony. Their work covers more interaction types than any other creator here. **What to learn:** AR as a live medium: everything must run in real time with zero failures, for the audience in the room and the one watching on screens. 1. **24 drones** (2015) — Three dancers and 24 glowing drones share the stage; the drones' formations of light shift with the choreography and frame the dancers. https://www.youtube.com/watch?v=cYWvKudIIJ8 2. **border (Rhizomatiks Research x ELEVENPLAY)** (2015) — Audience members ride automated wheelchairs wearing headsets; real and virtual dancers merge seamlessly. The signature mixed-reality stage. https://www.youtube.com/watch?v=DdYw3ZZ0XLQ 3. **Fencing Visualized** (2016) — The tips of fencing blades are tracked and drawn on the TV broadcast in real time. AR helps viewers follow a sport too fast to see. https://www.youtube.com/watch?v=h2DXCAWI8gU 4. **discrete figures (with Kyle McDonald)** (2018) — With Kyle McDonald: machine-learning-generated virtual dancers share the stage with real ones. https://www.youtube.com/watch?v=hauXQQhwbgM 5. **border 2021** (2021) — The 2021 version of border. Compare it with 2015 to see how technology changes the same work. https://www.youtube.com/watch?v=rqmO7FA2vdw #### Keiichi Matsuda A critical imagination of AR's future: the author of HYPER-REALITY. Keiichi Matsuda is an architecture-trained designer and director. His 2016 short HYPER-REALITY shows a city drowned in ads and interfaces and is still the most-cited work in debates about AR ethics. He led design at Leap Motion; his studio Liquid City now builds spatial AI agents. **What to learn:** Design fiction: use a short film to imagine and critique a future that does not exist yet, rather than only building a working demo. 1. **Augmented (hyper)Reality: Domestic Robocop** (2010) — A 2010 graduate project: every object in a kitchen is covered in interfaces. The seed of HYPER-REALITY. https://www.youtube.com/watch?v=fSfKlCmYcLc 2. **HYPER-REALITY** (2016) — Essential viewing: ads, gamification and identity systems flood a whole city. A strong opener for a class on AR ethics. https://www.youtube.com/watch?v=YJg02ivYzSs 3. **Merger** (2018) — A worker who spends all day in AR, the line between her and the machine blurring. The dark side of “AR at work”. https://www.youtube.com/watch?v=SqW2dEkiD-Y 4. **Agents (Liquid City)** (2023) — A Liquid City short: AI agents appear as characters in AR glasses and negotiate the protagonist's day. From critique to the question: what relationship do we want with AI? https://www.youtube.com/watch?v=bkKv2AHpn8E #### Rafael Lozano-Hemmer Bodies, shadows and projection in public space: the “Relational Architecture” series. Since the 1990s Rafael Lozano-Hemmer has staged huge interactive projections in city squares, which he calls Relational Architecture: passers-by's shadows, heartbeats and voices rewrite the surface of buildings. He showed that thousands of people can augment a space together without any personal device. **What to learn:** AR in public space: no app, no headset — anyone walking by becomes part of the work. 1. **Body Movies (Relational Architecture 6)** (2001) — Passers-by's shadows fall on a building and reveal portraits of other people inside them. A milestone of public projection AR. https://www.youtube.com/watch?v=g-CNxFiXZDY 2. **Under Scan (Relational Architecture 11)** (2005) — A square flooded with white light; video portraits of local people appear in pedestrians' shadows, look up at them, and vanish when they leave. https://www.youtube.com/watch?v=Bfn14sLJmyU 3. **Subtitled Public** (2005) — Each person entering the room gets a random verb projected on their body, following them around. Labeling people as a metaphor for tracking technology. https://vimeo.com/1089123902 4. **Zoom Pavilion (with Krzysztof Wodiczko)** (2015) — Face-detecting cameras amplify the relationships between visitors onto the walls. Surveillance technology made into art. https://www.youtube.com/watch?v=ENWBRsvn7qA #### Universal Everything (Matt Pyke) Mirror-like interaction and moving form: the digital art studio with the strongest visual language. Universal Everything, founded by Matt Pyke in Sheffield, is known for moving human figures: motion capture driving endlessly varied characters. Most of their interactive works are magic mirrors — you move, and another version of you transforms in the glass. **What to learn:** Visual style is interaction: with the same body tracking, the form design decides whether people stop and dance in front of the mirror. 1. **Future You** (2019) — A motion-capture mirror at the Barbican: every visitor becomes a synthetic robotic figure, and the harder you move the stranger it gets (47,000 possible responses). https://vimeo.com/718890921 2. **Super You (AR app)** (2020) — Their moving characters as a phone AR app, so anyone can summon a giant “super self”. https://www.youtube.com/watch?v=-zPQczdMCVc 3. **HyperSpace AR** (2021) — LiDAR understands the real room so abstract digital life can grow inside it. https://www.youtube.com/watch?v=1KuwkTdab0o 4. **Look Up - Augmented Reality Architecture** (2024) — AR architecture: look up at the skyline and new moving structures appear on top of buildings. https://www.youtube.com/watch?v=e5ELC-FcJXs 5. **Chameleon** (2024) — A digital being that imitates passers-by's movement and appearance on street screens in East London, later Venice and Beijing. The magic mirror in public space. https://www.youtube.com/watch?v=EkGLZ_Ck_-w #### Adrien M & Claire B (Adrien Mondot & Claire Bardainne) Images with weight that breathe like matter: from the juggler's stage to AR on paper and city walls. Adrien Mondot was a computer science researcher and juggler who, from the mid-2000s, wrote custom software, eMotion, so projected objects would obey physics. From 2011 Mondot co-led the company with visual artist Claire Bardainne (since 2023 Mondot has continued solo as cie Adrien M). Their images are never a backdrop: they get thrown, blown away and pushed aside like another performer. Over twenty years the same idea moved from stage projection to interactive exhibitions, and then to hand-held AR on pop-up books, posters and city walls. **What to learn:** Design digital images as a material: set their weight, elasticity and rhythm first, then decide how people live with them. And notice how one animated world can be moved across four carriers: stage, exhibition, book and street. 1. **Cinématique** (2010) — The starting point: a juggler and a dancer perform inside projected space generated live by eMotion and governed by physics. Watch this first to see where their idea of image as matter comes from. https://www.youtube.com/watch?v=CknHVdMZ5xg 2. **Hakanaï** (2013) — A dancer inside a cube of tulle while the surrounding grids and letters bend and scatter with the dancer's movement. The signature piece of projected, stage-scale AR. https://www.youtube.com/watch?v=xvJNia3z11I 3. **XYZT, Abstract Landscapes** (2011) — The stage technology handed to visitors: ten installations where touch, breath and footsteps push letters and particles around. The step from watching a performance to becoming the performer. https://www.youtube.com/watch?v=N7q9k9Z3HIk 4. **Acqua Alta – Crossing the Mirror (AR pop-up book)** (2020) — The same story on paper: point a tablet at a black-and-white pop-up book and tiny figures live through an ink flood inside a paper house. Stage projection shrunk to the palm of your hand for the first time. https://vimeo.com/462776850 5. **Faune** (2021) — The last stop takes it to the street: with Brest Brest Brest, posters on city walls hide animals that only appear through a phone. AR as a way to train passers-by to pay attention. https://vimeo.com/566519444 ## Salient works Simple works where the concept jumps out: one idea, minimal means, understood in seconds. ### Borrowed senses Perceive the world through a sense you don't have, or someone else's. - **FeltSight** — Botao 'Amber' Hu (2025): Borrow a star-nosed mole's sense: you feel the space around you through your fingertips instead of seeing it. https://www.youtube.com/watch?v=7Pq6s3VnD0A - **Live caption glasses for the Deaf and hard of hearing** — Cayden Pierce (Mentra / AugmentOS) (2025): Subtitles for real life, floating next to the person who is speaking. https://www.youtube.com/watch?v=CRyhfPrfywU - **EchoVision** — Botao 'Amber' Hu (2024): Put on the mask and call out: your voice lights up the room in echoes, so you see like a bat. https://www.youtube.com/watch?v=0JyHmEApctg - **Empa** — Katie Dale-Everett Dance (KDE Dance) (2019): Two people swap eyes and dance hand in hand, each seeing through the other's view. https://www.youtube.com/watch?v=TKyzy7ttthU - **Visual Noise Reduction HMD** — Kiyoshi Kiyokawa (2019): Noise-cancelling headphones, but for the eyes: moving distractions around you fade to grey. https://www.youtube.com/watch?v=vIKYG9NIeIU - **ChromaGlasses** — Tobias Langlotz, Stefanie Zollmann (2018): Glasses that correct color instead of focus, so people with color-vision deficiency can tell confused colors apart. https://www.youtube.com/watch?v=kBe-pfc7PrY - **Somen-to-Ramen Taste Modulation** — Kiyoshi Kiyokawa (2018): Plain noodles shown as ramen start to taste like ramen, so sight changes flavor. https://www.youtube.com/watch?v=z53xW_BIaZE - **AdaptiVisor** — Yuta Itoh (2017): Sunglasses that darken only the glaring pixels of the world and leave the rest untouched. https://www.youtube.com/watch?v=fi4PjGIbuu8 - **Learning to See** — Memo Akten (2017): A neural network sees cloth and cables on a table as waves or fire, showing that a machine sees only what it was trained on. https://vimeo.com/260612034 - **Parallel Eyes** — Shunichi Kasahara (2016): Four people see through each other's eyes at once and try to play hide-and-seek. https://vimeo.com/175105689 - **CHILDHOOD: Egocentric Smaller-person Experience** — Jun Nishida (2015): A low camera and small hands let an adult see and grasp the world as a child does. https://vimeo.com/120369920 - **In the Eyes of the Animal** — Marshmallow Laser Feast (2015): Stand in a forest and see it through the senses of a mosquito, frog or owl. https://vimeo.com/140057053 - **Upside-Down Goggles** — Carsten Höller (1994): Goggles flip the world upside down, and you have to learn to walk and pour again. https://www.youtube.com/watch?v=Ct3c9PzS6yE ### Invisible made visible Something that is always there but never seen (air, motion, data) becomes visible. - **Micro Art** — Stijn Spanhove, Pavlo Tkachenko (2026): Point at a rock or leaf and see the imagined microscopic world living inside it. https://www.youtube.com/watch?v=esOODTn0RVU - **CookAR** — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2024): Kitchen tools glow safe where you can grab them and warn where they can hurt you. https://www.youtube.com/watch?v=gtJhme8em7U - **I-XRAY** — AnhPhu Nguyen & Caine Ardayfio (2024): Look at a stranger and their name and address appear, showing how exposed everyone already is. https://x.com/AnhPhuNguyen1/status/1840786336992682409 - **Solid Light (Tate Modern)** — Anthony McCall (2024): A projected beam becomes a solid-looking volume that you walk into and cut with your body. https://www.youtube.com/watch?v=1OVzjGO5iS4 - **Soul Paint** — Sarah Ticho (2024): Paint your feelings onto a body where you feel them, then walk among everyone else's. https://www.youtube.com/watch?v=BmYAtoJA9Wk - **AR CO2 Visualization System** — Kiyoshi Kiyokawa (2023): The air you share becomes visible: invisible CO2 turns into a colored cloud filling the stale corners of the room. https://www.youtube.com/watch?v=zsOv6N_jAXU - **Controlling Air Conditioning via Augmented Reality** — Jason Orlosky (2021): Invisible airflow becomes visible, and you steer the air conditioner by grabbing the stream itself. https://www.youtube.com/watch?v=i3OctJhaXKI - **Extreme Measures** — FIELD (FIELD.IO / FIELD.SYSTEMS), SPACE10 (IKEA's research and design lab) (2021): Measure a gap by inflating a virtual elephant until it fills the space. https://vimeo.com/554342553 - **Fog Sculpture (Neue Nationalgalerie)** — Fujiko Nakaya (2021): A sculpture made of fog that you walk into and that the wind keeps reshaping. https://www.youtube.com/watch?v=k7VeOZbUZyk - **Ghost Forest** — Maya Lin (2021): Trees killed by rising seas are planted in a Manhattan park, bringing a distant climate loss into the city. https://www.youtube.com/watch?v=0DHpGPwHm2I - **Pulse Topology** — Rafael Lozano-Hemmer (2021): Thousands of bulbs each flicker with a recorded heartbeat, so you walk through a landscape of pulses. https://www.youtube.com/watch?v=5Zy39kMf_3o - **Connected** — Roelof Knol (2020): Lines of light on the floor join the strangers standing in the room, so each person's personal space becomes shared. https://www.youtube.com/watch?v=ArFcUEoxKfo - **Kitchen Timers in Augmented Reality** — Morten Just (2018): Countdown timers float right above the pots they are timing. https://www.youtube.com/watch?v=3c114LvhJIA - **LUMOS** — Matt DesLauriers (2018): A cold blue beacon turns warm the moment it senses your body heat. https://vimeo.com/261406016 - **SWIM: Sequential Wave Imprinting Machine** — Steve Mann (2017): A sweeping row of lights freezes a sound or radio wave in the air, exactly where the wave really is. https://www.youtube.com/watch?v=wKfwufI2hrQ - **Fencing Visualized** — Daito Manabe / Rhizomatiks (2016): Sword tips too fast to see are drawn as trails over live footage, so the bout becomes readable. https://www.youtube.com/watch?v=h2DXCAWI8gU - **Laplacian Vision** — Yuta Itoh (2016): A line in the air shows where a thrown ball is about to go, so you see a moment into the future. https://www.youtube.com/watch?v=2GD7KQOHiMs - **Parsing our Silent Language** — Kat Sullivan (2016): Body language gets live captions: 'Interested' or 'Disinterested' appears over each person. https://vimeo.com/165629185 - **Treehugger: Wawona** — Marshmallow Laser Feast (2016): Hugging a tree is the interface that reveals the water flowing inside it. https://www.youtube.com/watch?v=if0wfysmoMU - **Caress of the Gaze** — Behnaz Farahi (2015): A garment ripples wherever someone is looking at you, so a gaze becomes something you can see. https://vimeo.com/152363295 - **Ice Watch** — Olafur Eliasson (2014): Blocks of Greenland ice melt in a city square, so people can touch climate change with their hands. https://www.youtube.com/watch?v=qd-JRGBKSXA - **Thermal Touch** — Metaio (Thomas Alt & Peter Meier) (2014): The warm spot your fingertip leaves on any surface becomes the click. https://www.youtube.com/watch?v=K2XL0qnu4Z4 - **Ballet Rotoscope** — Masahiko Sato + EUPHRATES (2011): Glowing curves and figures traced over a ballerina reveal the geometry hidden in every pose. https://www.youtube.com/watch?v=yzJk6ww3LD0 - **Immaterials: Light painting WiFi** — Timo Arnall (2011): A rod of lights walked down the street draws the invisible WiFi signal as a glowing wall. https://www.youtube.com/watch?v=cxdjfOkPu-E - **Pulse Room** — Rafael Lozano-Hemmer (2006): Your heartbeat becomes one flashing bulb among hundreds left by earlier visitors. https://www.youtube.com/watch?v=R3benqCGVLI - **Wind Wand** — Len Lye (2000): One tall bending rod shows a whole city where the wind is. https://www.youtube.com/watch?v=HH2gjPPi0XQ - **Boundary Functions** — Scott Snibbe (1998): Lines drawn between people on the floor show each person's personal space, which shrinks as more people arrive. https://www.youtube.com/watch?v=5wA3lKcDrlM - **Smoke Machine (Mouvements de l'air)** — Étienne-Jules Marey (1901): Lines of smoke make the invisible flow of air visible around an object. https://www.youtube.com/watch?v=95XOEN4f8EE ### Sound in space Sound gets a place, a shape or a body in space. - **Ultrasound VR** — Universal Everything (Matt Pyke) (2026): Drawing becomes composing: every stroke you paint in the air turns into a sound that plays from where you drew it. https://www.youtube.com/watch?v=95_WMsapSuA - **Audio Personas** — Sean Follmer (2025): Everyone carries a personal sound around their body, so you hear who is nearby before you look. https://www.youtube.com/watch?v=l8lis-JPIzA - **Sonify Anything** — Laura Schütz (2025): A virtual ball sounds like glass, wood or metal depending on the real thing it hits. https://www.youtube.com/watch?v=tN6JE3kNUbk - **CON/TOUCH #2** — Samuele Albani (2023): An instrument that only makes sound when two people touch each other's shoulders. https://vimeo.com/862691895 - **World Ensemble** — Koki Ibukuro (asus4) (2023): Tap the buildings on a street and each one becomes an instrument in a song. https://www.youtube.com/watch?v=adra9QppRO0 - **Live Music with My Clothes in AR** — Lucas Rizzotto (2022): Your outfit becomes an instrument: tap different parts of your clothes to play music. https://www.youtube.com/watch?v=Vpr1i3-Ekl4 - **I Turned My Girlfriend into a Musical Instrument** — Lucas Rizzotto (2021): Every movement of a dancer's body plays a note, so the person becomes the instrument. https://www.youtube.com/watch?v=R2KZ34OZlwk - **Piano&Dancer** — Instituto Stocos (Muriel Romero, Pablo Palacio & Daniel Bisig) (2021): A piano with no pianist plays itself in answer to a dancer who never touches it. https://www.youtube.com/watch?v=_3gEitmSdis - **Audio in AR space** — Zach Lieberman (2017): Every sound stays where it was made, so walking back through the trail plays it again. https://vimeo.com/290238447 - **Hidden Stories** — Red Paper Heart (2015): Press a cup to the wallpaper and overhear the secret stories of everyday objects. https://vimeo.com/132444721 - **cloud piano** — David Bowen (2014): Passing clouds press the keys of a real piano, so the sky plays music. https://vimeo.com/101857804 - **Ishin-Den-Shin** — Ivan Poupyrev (2013): Touching someone's ear passes a whisper through your finger, so a message travels by touch alone. https://www.youtube.com/watch?v=Iw1FhmY1sIU - **Voice Tunnel** — Rafael Lozano-Hemmer (2013): Your voice becomes a wave of light that runs down a real tunnel. https://www.youtube.com/watch?v=jmRnLUVt4kE - **Moc** — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2010): Whistle and a tree grows from your breath; stop and it stops. https://vimeo.com/11989814 - **Messa di Voce (installation)** — Golan Levin, Zach Lieberman (2005): Your voice grows visible shapes out of your mouth that you can then push around with your body. https://vimeo.com/221802940 - **The Voices of Oakland** — Blair MacIntyre — Georgia Tech Augmented Environments Lab (2005): Walk through a cemetery and hear the people buried there tell their stories at their own graves. https://www.youtube.com/watch?v=mHoe2NfTV7U - **Touching (e-touching)** — Palindrome (Robert Wechsler), Frieder Weiss (2004): Skin-to-skin contact between two dancers becomes audible, so every touch is heard. https://www.youtube.com/watch?v=FfjU-lbHdvo - **Very Nervous System** — David Rokeby (1986): In an empty room, your body's movement is instantly turned into music. https://www.youtube.com/watch?v=qdvyuvfKVU0 ### Body remix Your body is changed, extended, swapped or turned into something else. - **Generative Muscle Stimulation** — Pedro Lopes (2026): Ask the AI for help and it answers by moving your own hand through your muscles. https://www.youtube.com/watch?v=pJM2Z8mmwAw - **Morphing Identity** — Shunichi Kasahara (2023): Your face slowly melts into the face of the person sitting across from you, live. https://www.youtube.com/watch?v=rfmIuaV9Ny0 - **Electrical Head Actuation** — Jun Nishida, Pedro Lopes (2022): Instead of drawing an arrow to show you where to look, the interface turns your head for you. https://www.youtube.com/watch?v=vqpH9gNGpts - **Parallel Ping-Pong** — Shunichi Kasahara (2022): One mind, two bodies: a single person plays two ping-pong matches at the same time. https://www.youtube.com/watch?v=q1XAmaMdEiE - **Distant Hand** — Sander Veenhof (2021): Your hand stretches far out in AR so you can touch things across the room. https://www.youtube.com/watch?v=ZdudbnDh7tk - **HandMorph** — Jun Nishida, Pedro Lopes (2020): Wearing a child-sized mechanical hand makes the real world feel oversized, with no screen at all. https://www.youtube.com/watch?v=5o2wPy5hl0w - **Kat & Cassie Make A Ballet** — Kat Sullivan (2020): One ballerina, captured live, is multiplied into an entire corps de ballet. https://www.youtube.com/watch?v=cSspDP7H5tg - **Social Distancing Shirt** — Clémence Debaig (Unwired Dance Theatre) (2020): A shirt that says 'You are too close' until the person near you steps back. https://www.youtube.com/watch?v=zKxfqXSyp3M - **Super You (AR app)** — Universal Everything (Matt Pyke) (2020): Point the phone at yourself and your body becomes a giant flowing digital being; the whole work is that one transformation. https://www.youtube.com/watch?v=-zPQczdMCVc - **YoPuppet** — Hart Woolery (2020CV) (2019): Open and close your hand and a cartoon puppet talks, the oldest puppet trick redone in AR. https://www.youtube.com/watch?v=zd-LKYu5QDQ - **Affordance++** — Pedro Lopes (2015): Objects move your hand with electrical stimulation to show you how to use them. https://www.youtube.com/watch?v=Gz4dphzBb6I - **Augmented Hand Series** — Golan Levin, Chris Sugrue, Kyle McDonald (2014): Your own hand grows an extra finger on screen, live, and it feels uncanny. https://vimeo.com/111951283 - **Dhalsim: Real-Time Body Transformation** — Keita Higuchi (2014): Your live body, but your arm stretches across the room to punch or grab. https://www.youtube.com/watch?v=g2bg_vBVW1w - **Rain Room** — Random International (Hannes Koch & Florian Ortkrass) (2012): Rain falls everywhere except on you, so your body becomes the one dry spot in a storm. https://www.youtube.com/watch?v=FslABAyj2OA - **Body Swap** — Chris O'Shea (2011): You control the on-screen body of the person beside you while they control yours. https://vimeo.com/20745353 - **Eyeshine** — Golan Levin, Kyle McDonald (2011): The glint in your own eyes is caught and shown back so they glow like a night animal's. https://vimeo.com/29356492 - **Sandbox (Relational Architecture 17)** — Rafael Lozano-Hemmer (2010): The hand you put in a small sandbox appears as a giant hand over the whole beach. https://www.youtube.com/watch?v=GotOBu_14fc - **The Salt Satyagraha Online: Gandhi's March to Dandi** — Joseph DeLappe (2008): Walking 240 miles on a treadmill carries an avatar along Gandhi's Salt March in Second Life. https://www.youtube.com/watch?v=34SxiWwOvHw - **Finger Gloves and body extensions** — Rebecca Horn (1972): Gloves with metre-long fingers let you touch the room from far away. https://www.youtube.com/watch?v=6uEkq3IBIf0 ### Shadows & mirrors Your shadow or reflection starts to behave differently. - **AR Sundial** — Max van Leeuwen (2025): A virtual sundial lit by the real sun casts a shadow that tells the true time. https://x.com/maksvanleeuwen/status/1953056369818439790 - **Chameleon** — Universal Everything (Matt Pyke) (2024): A digital creature on the street copies how passers-by move and look, like a chameleon. https://www.youtube.com/watch?v=EkGLZ_Ck_-w - **Prototype: Someone - a random mirror** — Universal Everything (Matt Pyke) (2024): A mirror that never shows you, only someone else moving the way you move. https://www.youtube.com/watch?v=OS3rX7QU468 - **The Building (Liberty Science Center)** — Leandro Erlich (2023): Lie on a facade laid flat on the floor and a mirror overhead shows you hanging from a building. https://www.youtube.com/watch?v=heWqEr-MqEI - **Just your shadow** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2022): A dancer negotiates with her own shadow, which refuses to simply follow. https://www.youtube.com/watch?v=iWfpUCYw3Kg - **CMY Shadows Mirror** — Daniel Rozin (2021): Three colored shadows overlap on the wall and mix into a colored reflection of you. https://vimeo.com/1198095563 - **Antivanity Mirror** — Neil Mendoza (2020): A mirror that turns away whenever you try to look at yourself. https://vimeo.com/398041909 - **Painted Mirror** — Theo Watson, Emily Gobeille (2020): A mirror that paints you: the longer you stand still, the clearer your portrait becomes. https://vimeo.com/416474904 - **Fabric Mirror** — Daniel Rozin (2019): Strips of cloth turn their light or dark side to weave a soft picture of whoever stands in front. https://vimeo.com/317577759 - **Fragment Shadow** — Shunichi Kasahara (2019): One body casts several colored shadows at once, turning an everyday shadow into something strange. https://vimeo.com/322671485 - **Hello, Shadow!** — Joon Moon (Joon Y. Moon / 문준용) (2019): Turn a light around a sculpture and its shadow on the wall becomes a shape the object could never cast. https://www.youtube.com/watch?v=RbwEf1QGA8U - **Floating by shadow offset** — Shengzhi Wu (2018): Move only the shadow and the object seems to lift off the table, showing the cheapest depth cue there is. https://x.com/Wu_Shengzhi/status/1059284500159578113 - **Move Mirror** — Irene Alvarado, Google Creative Lab (2018): Your pose becomes a search query that finds strangers striking the same pose, frame by frame. https://www.youtube.com/watch?v=JvzkFJW6LIU - **Echo (Flip-Discs)** — BREAKFAST (Andrew Zolty) (2016): A wall of flip-discs clatters into your silhouette and keeps a fading echo of how you just moved. https://www.youtube.com/watch?v=3kYKPIh3TCk - **Elephant Walk** — Michael Flückiger (2015): Ride a bike at night and its shadow is an elephant walking at your speed. https://vimeo.com/142179343 - **Penguins Mirror** — Daniel Rozin (2015): Hundreds of toy penguins turn their black backs or white bellies to form your silhouette. https://www.youtube.com/watch?v=QlrnjjfLkTI - **PomPom Mirror** — Daniel Rozin (2015): A mirror made of soft fur pom-poms that shows your silhouette as it moves. https://vimeo.com/128375543 - **shadow (drone with a spotlight)** — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2015): Drones carry the spotlights, so as they fly the dancer's shadows multiply and turn. https://www.youtube.com/watch?v=hX2TneyE41Q - **Parade** — Dpt. (2014): Push the lamp and the shadows of two sad vases start to dance. https://vimeo.com/96615251 - **ZEROTIME** — Children of the Light (Christopher Gabriel & Arnout Hulskamp) (2014): The room stays dark until your phone torch hits the reflective tubes, and you get back only the light you send. https://vimeo.com/129778294 - **Dalston House (Bâtiment)** — Leandro Erlich (2013): A house facade lies on the ground under a tilted mirror, so visitors lying on it appear to hang and climb up the building. https://www.youtube.com/watch?v=AFF3BAnSNn4 - **Vestige** — Rob Mulholland (2012): Human figures made of mirrors vanish into the forest they reflect. https://www.youtube.com/watch?v=gf4BOH9vod4 - **Augmented Shadow** — Joon Moon (Joon Y. Moon / 문준용) (2010): Plain white blocks cast shadows in which little houses, trees and people grow and live. https://www.youtube.com/watch?v=0arZMuPK58w - **Shadow works** — Kumi Yamashita (2010): A few abstract pieces on a wall cast the shadow of a detailed human face. https://www.youtube.com/watch?v=65BRB3kuvv8 - **Your uncertain shadow (colour)** — Olafur Eliasson (2010): Five colored lamps split one shadow into a row of colored shadows that move with you. https://www.youtube.com/watch?v=PeBH6fTQNSc - **Audience** — Random International (Hannes Koch & Florian Ortkrass), Chris O'Shea (2008): A crowd of mirrors turns in unison to stare at you, so the visitor becomes the one being watched. https://www.youtube.com/watch?v=JuKi35j3Dwk - **Under Scan (Relational Architecture 11)** — Rafael Lozano-Hemmer (2005): A stranger's portrait appears inside your shadow and looks up at you, turning a passing shadow into a meeting. https://www.youtube.com/watch?v=Bfn14sLJmyU - **Shadows** — Frieder Weiss (2003): A dancer's shadow is drawn by a computer, so it lags, mirrors and finally walks away from her. https://www.youtube.com/watch?v=gm_doxsdqG4 - **Shadow (Screen Series)** — Scott Snibbe (2002): After you walk away your shadow stays behind and repeats what you just did. https://www.youtube.com/watch?v=pdxYv-_70-s - **Body Movies (Relational Architecture 6)** — Rafael Lozano-Hemmer (2001): Hidden portraits on a facade only show inside passers-by's giant shadows, so your shadow is the window. https://www.youtube.com/watch?v=g-CNxFiXZDY - **Text Rain (with Romy Achituv)** — Camille Utterback (1999): Letters of a poem fall like rain and pile up on your silhouette, so reading becomes a physical act. https://www.youtube.com/watch?v=GYvyuL-Mkjg - **Lunch with a Helmet On** — Shigeo Fukuda (福田繁雄) (1987): A tangle of forks and knives casts the shadow of a motorcycle. https://www.youtube.com/watch?v=aFUGx-DaKz4 - **TV Buddha** — Nam June Paik (白南准) (1974): A Buddha sits watching a live image of itself on a television. https://www.youtube.com/watch?v=MVwR1n0IZAs ### Windows & portals A frame, door or hole opens onto another place. - **A picture frame that is a window** — Ian Curtis (2026): A picture frame on the wall is really a window: look into it and another 3D world sits behind the wall. https://x.com/XRarchitect/status/2052418499516170377 - **Unbounded: Object–Boundary Interaction in Mixed Reality** — Zhuoyue Lyu (2026): A ball rolls off a real table and keeps falling through the real floor, as if every surface were only a boundary you can switch off. https://www.youtube.com/watch?v=FL3hmRgtYUM - **Mixed Reality Door for Meta Quest 3** — Takashi Yoshinaga (2023): Open the real door in your home and a virtual world is waiting on the other side. https://www.youtube.com/watch?v=VW7ELpkVmIQ - **AR Portal Gun** — Max van Leeuwen (2022): Shoot two portals onto real walls and look through one to see your room from the other. https://x.com/maksvanleeuwen/status/1556715819316412418 - **Depth lines on paper** — Roelof Knol (2022): A plain paper card becomes a hole into the table, and the hole moves when you slide the card. https://x.com/CurieuxExplorer/status/1544719441790992385 - **La Ferita (The Wound)** — JR (2021): A paper tear in a closed museum's stone facade lets you see the halls inside. https://www.youtube.com/watch?v=P-FWZ5-tyws - **A Futuristic Portal to My Best Friend** — Lucas Rizzotto (2020): A portal in the wall opens into a friend's home during lockdown, so two people live as virtual roommates. https://www.youtube.com/watch?v=713T4x89kMk - **AR Cut & Paste** — Cyril Diagne (2020): Point a phone at a real object to cut it out, then point at your screen to paste it in. https://x.com/cyrildiagne/status/1259441154606669824 - **'Hole in the head' occlusion effect** — Luke Hurd (2019): A hole opens in your head and there is a tiny world inside. https://www.youtube.com/watch?v=j7klMWrJFHs - **The Secret of the Great Pyramid** — JR (2019): Paper pasted on a flat courtyard makes the Louvre pyramid seem to rise out of a deep quarry. https://www.youtube.com/watch?v=rsnpm1_IXbw - **The Ring Brought to Life in AR** — Abhishek Singh (2018): The girl from The Ring crawls out of your own TV and into your living room. https://www.youtube.com/watch?v=g_WBVi-bu9Q - **Augmented Reality's A-ha Moment (Take On Me AR portal)** — Trixi Studios (Chip Sineni) (2017): Step through a door into a pencil-sketch world where even your own hand turns into line art. https://www.youtube.com/watch?v=ZBdRAdSosv4 - **Descension** — Anish Kapoor (2017): A black whirlpool spins in the ground as if the floor were draining into the earth. https://www.youtube.com/watch?v=p8knuUS4w-Q - **The Bottomless Pit** — Mark Skwarek, Will Pappenheimer (2012): A virtual hole in a San Jose plaza goes straight through the Earth, and people drop things into it to reach the other side. https://www.youtube.com/watch?v=LuWh7DNtc9U - **Delicate Boundaries** — Chris Sugrue (2007): Light bugs crawl out of the screen and onto the hand that touches it, so the image escapes its frame. https://vimeo.com/1007230 - **Head Tracking for Desktop VR Displays** — Johnny Chung Lee (2007): Track the viewer's head and an ordinary TV turns into a window with real depth behind it. https://www.youtube.com/watch?v=Jd3-eiid-Uw - **Out of Bounds** — Chris O'Shea (2007): Shine an X-ray flashlight at a wall and see the hidden room behind it. https://vimeo.com/1333176 - **levelHead** — Julian Oliver (2007): Rooms hide inside a cube in your hand, and tilting the real cube walks a tiny figure through them. https://www.youtube.com/watch?v=UJYKSFANuaQ - **Sky Mirror** — Anish Kapoor (2001): A huge dish mirror brings the sky down to the ground. https://www.youtube.com/watch?v=Ee1nAvEZTwY - **Swimming Pool** — Leandro Erlich (1999): Look down into the pool and people stand at the bottom, dry and breathing. https://www.youtube.com/watch?v=NT7gjhHq9d0 ### Time & traces The past stays in place, replays, or leaves a visible trace. - **Replaying a broadcast basketball play on a real court in XR** — Stijn Spanhove (2025): Last night's televised play happens again, in 3D, on the court you are standing on. https://x.com/stspanho/status/1992980633652478223 - **The Gates in augmented reality (Bloomberg Connects)** — Christo and Jeanne-Claude (2025): A vanished artwork returns to the exact paths in Central Park where it once stood. https://www.youtube.com/watch?v=ncGYO75uATA - **Entomographies** — Xavi Bou (2024): The flight of an insect is left behind as a glowing line in the air. https://vimeo.com/1050040741 - **Hand-frame photos left in space** — Aidan Wolf (2024): Frame a shot with your fingers and the photo stays hanging at that spot for the next person to find. https://x.com/Aidan_Wolf/status/1838941247148278146 - **Paris 1924** — Paris AR Studio (Snap) (2024): Hold up your phone and the street you are standing on goes back exactly one hundred years. https://www.youtube.com/watch?v=XqI0etsPbKo - **Project Revival** — Anrick Bregman (Studio ANRK), Nexus Studios (2024): Stand before a broken artefact and see it whole again, exactly where it sits. https://www.youtube.com/watch?v=nr9tFXYf0Iw - **Schiphol People's Clock** — Maarten Baas (2024): The hands of a giant clock are people, one after another, showing the time with their bodies. https://www.youtube.com/watch?v=WqxXsbquzOE - **Spatial Vacuuming** — Daniel Beauchamp (Pushmatrix) (2024): While you vacuum, the floor you have cleaned gets painted over, so the missed spots stand out. https://x.com/pushmatrix/status/1749797146961006716 - **Clo(o)k: Human-Time Interactions Through a Clock That "Looks"** — Zhuoyue Lyu (2023): A clock that watches you: time speeds up when you look away and stops while people talk. https://www.youtube.com/watch?v=A3jYe0NNDAk - **Echoes** — Torin Blankensmith (2023): Every pose you strike is answered by the most similar pose of someone who stood there before you. https://x.com/blankensmithing/status/1737307286572298738 - **This Fragile Earth: Day to Night** — Stephen Wilkes (2023): One photograph holds a whole day, from daylight on one side to night on the other. https://www.youtube.com/watch?v=IlTY4KNGzAY - **Real-Time Body Clones** — Max van Leeuwen (2022): As you move, frozen 3D copies of your body stay behind in the room. https://x.com/maksvanleeuwen/status/1527804730629574656 - **SUPER SLIT SCAN** — Kitasenju Design (Takayuki Watanabe) (2022): Time is smeared across the frame, so moving people turn into stretched sculptures of their own motion. https://www.youtube.com/watch?v=y3HyKMhGeTI - **Murmurations** — Xavi Bou (2020): Tracing every starling turns a flock into one moving sculpture in the sky. https://www.youtube.com/watch?v=hScBionqFBA - **SlitScanCam: realtime slit-scan camera** — Keijiro Takahashi (2020): Each slice of the live image comes from a different moment, so moving people stretch through time. https://vimeo.com/494895371 - **The Life** — Marina Abramović (2019): The artist's holographic double performs in the room with you, so a performance piece can exist without the performer. https://www.youtube.com/watch?v=VeajXYdTEiE - **Time Travel in AR** — Nathan Gitter (2018): A wall painted over every day shows its earlier layers in place, so the phone becomes a window into the past. https://x.com/nathangitter/status/1020733723183124480 - **AR Wormhole** — Jonas Jongejan, Google Creative Lab (2017): Step through a portal and look around the same place as it was ten seconds ago. https://www.youtube.com/watch?v=Xo0_B3pNNnA - **Another Time (Margate)** — Antony Gormley (2017): A lone iron figure in the sea is swallowed and revealed by the tide every day. https://www.youtube.com/watch?v=3fUZA7ylRJk - **Light Capsules x Neon Museum** — Craig Winslow (2017): Projection fakes the glow of broken neon signs so they seem lit again. https://vimeo.com/207339810 - **A pound of flesh for 50p** — Alex Chinneck (2016): A house built from wax bricks melts like a candle over a month. https://www.youtube.com/watch?v=5X144p9qA1w - **Ornithographies** — Xavi Bou (2016): Stacking the frames of a bird's flight into one image draws the invisible shape its wings leave in the air. https://vimeo.com/561365138 - **Schiphol Clock** — Maarten Baas (2016): A man inside a giant airport clock repaints the hands by hand every minute. https://www.youtube.com/watch?v=e_3KY2gWDwg - **Lichtgrenze** — Christopher Bauder / WHITEvoid (2014): A line of glowing balloons redraws the vanished Berlin Wall through the city, then lets it float away. https://www.youtube.com/watch?v=Sq0Ecpakna4 - **Exit Glacier Augmented Reality Terminus Project** — Nathan Shafer (2012): In an empty Alaskan valley, the ice of Exit Glacier rises again where its edge stood decades ago. https://www.youtube.com/watch?v=PQac4zosOVw - **Carnation Rain (Largo do Carmo)** — Tamiko Thiel (2011): Virtual carnations rain down on the square where the Carnation Revolution happened, marking history in place. https://www.youtube.com/watch?v=VxBEr_bq_0k - **Please Empty Your Pockets** — Rafael Lozano-Hemmer (2010): Your object leaves the conveyor belt, but its image stays behind among everyone else's. https://www.youtube.com/watch?v=arK7V_jDlNI - **Bird Watching (small brains series)** — Dennis Hlynsky (2009): Every bird leaves a trail, so ordinary flight turns into drawings in the sky. https://www.youtube.com/watch?v=BA5iOn_toVQ - **Day to Night** — Stephen Wilkes (2009): One photograph of a city runs from day on the left to night on the right. https://www.youtube.com/watch?v=afev0ZjAhUA - **Time remap – Anamorphose temporelle** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2009): Each row of the image comes from a different moment, so a moving body twists into a sculpture of its own past. https://vimeo.com/7878518 - **Footprints of History** — Cai Guo-Qiang (蔡国强) (2008): Giant footprints of fireworks walk one by one across the city toward the stadium. https://www.youtube.com/watch?v=kiRyECW2UXk - **Daisies** — Theo Watson (2005): Flowers wilt under your feet and grow back after you leave, so your presence leaves a visible mark. https://vimeo.com/463536634 - **Liquid Time Series** — Camille Utterback (2002): The closer you step to the image, the further that patch slips into the past, so distance becomes time. https://www.youtube.com/watch?v=qSHmx45AF_k - **Siluetas** — Ana Mendieta (1973): The outline of her body, pressed into earth and water, stays after she is gone. https://www.youtube.com/watch?v=xyS5gsw-A5Q - **A Line Made by Walking** — Richard Long (1967): Walking back and forth across a field leaves a line in the grass. https://www.youtube.com/watch?v=5eVaZRaQWRQ ### Rewriting reality What is already there is erased, replaced or re-labelled, often as a statement. - **La Caverne du Pont Neuf** — JR (2026): Paris's oldest bridge disappears inside a rock cave. https://www.youtube.com/watch?v=RngYQyEu54k - **Ad Block** — Stijn Spanhove (2025): An ad blocker for the street: billboards in front of you simply get covered up. https://www.youtube.com/watch?v=KLiEm74cw9Q - **EARLL** — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2024): Pick up an everyday object and its word in the language you are learning appears. https://www.youtube.com/watch?v=AjdQsxkPVkU - **Head-tracked Subtitles** — Jason Orlosky (2024): Real life gets subtitles, shown beneath whoever is speaking, the way films caption dialogue. https://www.youtube.com/watch?v=Oc91ouDpgio - **Integrating Real-World Distractions into Virtual Reality** — Yujie Tao, Pedro Lopes (2022): A real fan's gust becomes wind in the virtual world instead of breaking the illusion. https://www.youtube.com/watch?v=PO8ZlQGYMY8 - **Reality Channels (Liquid City)** — Keiichi Matsuda (2022): Flip through the city like TV channels: the same street, re-skinned with each switch. https://www.youtube.com/watch?v=0tyswdePMmA - **Reverse AR** — Russ Maschmeyer, Shopify Spatial Commerce Team (2022): AR turned inside out: the product stays real and the room around it becomes virtual. https://x.com/russ_maschmeyer/status/1562450155080597505 - **Summer Snow** — Ninsky (2022): A sunny summer street is suddenly buried in snow. https://x.com/Ninsky_AR/status/1556994932363214852 - **Eiffel Tower anamorphosis (Trocadéro)** — JR (2021): From one spot, a flat paper collage turns the ground into a cliff with the Eiffel Tower on its edge. https://www.youtube.com/watch?v=ptUCuYhYL1c - **Floor Pit (AR 100 Days, Day 45)** — Satoshi Hattori (2021): The real floor cracks into tiles and falls away, opening a pit under your feet. https://x.com/shmdevelop/status/1368503261515247617 - **L'Arc de Triomphe, Wrapped** — Christo and Jeanne-Claude (2021): Wrapping a monument in fabric makes a familiar landmark look new. https://www.youtube.com/watch?v=NUXIn9Fj7GI - **Squarepusher 'Terminal Slam' music video** — Daito Manabe / Rhizomatiks (2020): Through the glasses, Shibuya's billboards are erased and replaced by music visuals, AR that subtracts instead of adds. https://www.youtube.com/watch?v=GlhV-OKHecI - **Landmarkers** — Snap Inc. (Snapchat Lenses) (2019): The real Eiffel Tower melts or sprays rainbows on camera, turning a monument into a toy. https://www.youtube.com/watch?v=jNs9kdSAOCA - **Notable Women** — Nexus Studios (2019): AR puts portraits of historic women onto US banknotes, asking who deserves to be on money. https://www.youtube.com/watch?v=DuxXeeGg-T0 - **Cercles concentriques excentriques (Carcassonne)** — Felice Varini (2018): Yellow shards painted across a whole medieval city snap into perfect circles from one single spot. https://www.youtube.com/watch?v=Iybcb-pKLQY - **ARKit re-code of Jeffrey Shaw's Golden Calf** — Golan Levin (2017): An empty plinth holds a golden calf that exists only through the phone. https://vimeo.com/269478756 - **Giants: Kikito** — JR (2017): A giant toddler peers over the US–Mexico border fence, turning a wall into a curious look. https://www.youtube.com/watch?v=M_LMgzMnG7Q - **Mirage** — Doug Aitken (2017): A house covered in mirrors disappears into the desert it reflects. https://www.youtube.com/watch?v=Gn3BUue9nEc - **SUN** — Random Studio (2017): Bounce a giant ball and the sun rises and sets with it. https://vimeo.com/212766197 - **Support** — Lorenzo Quinn (2017): Two giant hands rise from the canal to hold up a sinking Venice building. https://www.youtube.com/watch?v=OJjvUjczmgE - **JR au Louvre** — JR (2016): The Louvre pyramid is covered with a photo of the palace behind it, so from one spot it simply vanishes. https://www.youtube.com/watch?v=M9VkFxEwINY - **Safe Passage (Konzerthaus Berlin)** — Ai Weiwei (艾未未) (2016): Refugees' life jackets wrap the columns of a Berlin concert hall, bringing the sea crossing onto a landmark. https://www.youtube.com/watch?v=8faWQdQ_JgY - **Halo Content** — Jason Orlosky, Kiyoshi Kiyokawa (2015): Notifications move out of the way when a person walks into view and form a halo around their face. https://www.youtube.com/watch?v=9Vbs1oLpk38 - **Level of Confidence** — Rafael Lozano-Hemmer (2015): A face-recognition camera searches every visitor's face for 43 missing students, turning surveillance into a memorial. https://vimeo.com/953969845 - **Queen Victoria** — Krzysztof Wodiczko (2014): A colonial statue takes on the faces and voices of local residents and tells their stories. https://www.youtube.com/watch?v=oRmT13QlY78 - **Les Voyageurs** — Bruno Catalano (2013): Travellers with their bodies torn away, so the harbour behind them shows through. https://www.youtube.com/watch?v=bZJ2sE-xyik - **Discovering Columbus** — Tatzu Nishi (2012): A living room built around a statue six storeys up lets you sit next to it on the sofa. https://www.youtube.com/watch?v=rF76AmzRGxs - **Mao Wants This Dollar!** — Geoffrey Alan Rhodes (2011): Any dollar bill seen through a phone turns into an IOU in which Mao says the United States owes China one dollar. https://www.youtube.com/watch?v=ex-EHCLcm7I - **Your rainbow panorama** — Olafur Eliasson (2011): Walk around a ring of coloured glass and the city changes colour with every step. https://www.youtube.com/watch?v=ZsMCfOW0SRA - **Nimbus** — Berndnaut Smilde (2010): A real cloud floats for a moment in the middle of an empty room. https://www.youtube.com/watch?v=I_Uu8epvcw0 - **The Great Firewall and Sunken City** — Lily & Honglei (Lily Xiying Yang & Honglei Li) (2010): The Great Firewall becomes a real wall that blocks a Brooklyn street for anyone looking through a phone. https://www.youtube.com/watch?v=BD3BvwFIPuU - **The Leak in Your Hometown** — Mark Skwarek (2010): Point at any BP logo and crude oil gushes out of it, turning a brand into the scene of its own spill. https://www.youtube.com/watch?v=V6-BbqANr04 - **We AR in MoMA** — Sander Veenhof, Mark Skwarek (2010): An uninvited AR show hangs inside MoMA, showing that no one controls who exhibits in virtual space. https://www.youtube.com/watch?v=b9T2LVM7ynM - **Word Lens** — Otavio Good (Quest Visual) (2010): Point the camera at a foreign sign and the words are replaced in place by your language. https://www.youtube.com/watch?v=h2OfQdYrHRs - **erasAR: Statue of Liberty erased** — Mark Skwarek (2010): Point a phone at Liberty Island and the Statue of Liberty is gone, leaving only her empty pedestal. https://www.youtube.com/watch?v=afaggs_RJ7U - **de-surveillance** — Sander Veenhof (2009): Hold up something blue and you vanish from a police surveillance camera's screen. https://www.youtube.com/watch?v=SBxdBHO9Ozo - **The Artvertiser** — Julian Oliver (2008): Look through binoculars and street ads are replaced by artworks, so you can switch off advertising. https://www.youtube.com/watch?v=z4a8n8hotI4 - **Rubber Duck** — Florentijn Hofman (2007): A bath toy blown up to building size turns the whole harbour into a bathtub. https://www.youtube.com/watch?v=G4-2UUz4Px4 - **Morphovision** — Toshio Iwai (岩井俊雄) (2005): Scanning light makes a solid spinning house bend and melt like rubber. https://www.youtube.com/watch?v=GbXeybKgIyY - **RedBall Project** — Kurt Perschke (2001): A giant red ball gets squeezed into a different gap in the city each day. https://www.youtube.com/watch?v=kNSpgFoUR6E - **Tijuana Projection** — Krzysztof Wodiczko (2001): A factory worker's live face and voice fill a giant public dome, so an unheard person takes over a monument. https://www.youtube.com/watch?v=dI_85KyAFWk - **Zerseher / De-viewer** — ART+COM Studios (Joachim Sauter) (1992): Wherever you look at the painting, your gaze smears it away. https://vimeo.com/386256001 ### Things come alive Paper, drawings and everyday objects become interactive or animate. - **Bringing Everyday Objects to Life with AI-Powered Talking Characters** — Fabrice Matulic (Preferred Networks) (2025): Every household object gets a face and a personality that come from what it is. https://www.youtube.com/watch?v=RkRc7TRDPDQ - **Bringing an action figure to life in AR** — Stijn Spanhove (2025): Photograph a toy and a living copy of it walks around next to the real one. https://x.com/stspanho/status/1910708661279773101 - **Myaku Myaku AR** — Koki Ibukuro (asus4) (2025): A vase of flowers turns into a wobbling blob covered in eyes while the rest of the room stays real. https://github.com/user-attachments/assets/03961874-421f-41bc-a1d8-f9e0e07b4fe4 - **Sketched Reality** — Ryo Suzuki — Programmable Reality Lab (2022): A spring you draw on paper can bounce a real robot, and the robot can push your drawings back. https://www.youtube.com/watch?v=xy-IeVgoEpY - **plant drone** — David Bowen (2022): A living plant flies the drone, and its flight path is drawn in the night sky. https://vimeo.com/709247945 - **+Panic** — Schnelle Bunte Bilder (with kling klang klong) (2021): Projected fish come close only if you move calmly; rush and panic ripples through the swarm. https://vimeo.com/654523297 - **Faune** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2021): Posters on city walls hide animals that only come out when you stop and look through a phone. https://vimeo.com/566519444 - **Lune** — Julie Curtiss, Acute Art (2021): However you walk around her, the painted figure keeps turning her back to you. https://vimeo.com/812460459 - **A Working Marauder's Map – Harry Potter AR** — Lucas Rizzotto (2020): A parchment Marauder's Map that really shows where everyone in the castle is right now. https://www.youtube.com/watch?v=A3v6dLlylU8 - **Acqua Alta – Crossing the Mirror (AR pop-up book)** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2020): A paper pop-up book where tiny figures live and drown in an ink flood you can only see through a tablet. https://vimeo.com/462776850 - **Bosch AR** — Anrick Bregman (Studio ANRK) (2020): Monsters from a 500-year-old Bosch painting walk out of the canvas into your street. https://www.youtube.com/watch?v=iznOf-dOIbw - **Earth Speakr** — Olafur Eliasson (2020): Children lend their faces and voices to trees and rivers so the planet can speak. https://www.youtube.com/watch?v=oneExExNwZw - **ElaMorph Projection** — Ishikawa Watanabe Laboratory (University of Tokyo) (2020): Projected light alone makes a rigid object look as stretchy as rubber. https://www.youtube.com/watch?v=uWq-a52X-7g - **Rubens Cupid** — Skullmapping (Filip Sterckx & Antoon Verbeeck) (2020): A cupid escapes a Rubens painting, flies over travellers, and points to where the original hangs. https://www.youtube.com/watch?v=JREE37Q5cNM - **AR eyedropper for smart lights** — Shengzhi Wu (2019): An eyedropper sucks a color from any real object and squirts it into a lamp, so the world becomes a palette. https://x.com/Wu_Shengzhi/status/1113284624355135489 - **Bringing Tattoos to Life Using AR** — Abhishek Singh (2019): Point at a tattoo and it animates and tells the personal story behind it. https://www.youtube.com/watch?v=ojnAHSCNFLk - **DoodleLens** — Aidan Wolf (2019): A doodle on paper peels off the page and comes alive in the room. https://x.com/Aidan_Wolf/status/1124997833159929856 - **Pull a sticky note out of an iPad** — Shengzhi Wu (2019): You pinch a sketch and pull it out of the iPad screen like a page from a book, which a child grasps at once. https://x.com/Wu_Shengzhi/status/1127773999876976640 - **My Room Is Another Fish Bowl** — Philippe Parreno (2018): Balloon fish drift at head height and bump into you, so the room becomes a fish tank. https://www.youtube.com/watch?v=B21g3MFnpUI - **Please Feed the Lions** — Es Devlin (2018): Feed a word to a lion in Trafalgar Square and it roars back a line of poetry. https://www.youtube.com/watch?v=SpZ-k9woZ14 - **Drifter** — Studio DRIFT (2017): A heavy concrete block floats slowly through the room like a cloud. https://www.youtube.com/watch?v=2QJ-Zn-yXs4 - **HotStepper** — Nexus Studios (2017): A small character walks ahead of you on the real street to show the way. https://www.youtube.com/watch?v=kR7RBUhnQmU - **InstaSaber** — Hart Woolery (2020CV) (2017): A rolled-up sheet of paper turns into a glowing lightsaber in your hand. https://www.youtube.com/watch?v=MWd7shj59PA - **Magic Sudoku** — Brad Dwyer (2017): Point at a paper Sudoku and the answers appear written in its empty squares. https://x.com/braddwyer/status/910030265006923776 - **Enthusiast Overlay** — Cyril Diagne (2016): Hanging tennis balls get scribbled cartoon faces and bodies and bounce around with endless enthusiasm. https://vimeo.com/224709486 - **Gallery Invasion** — Skullmapping (Filip Sterckx & Antoon Verbeeck) (2016): A painted character escapes its frame and runs around the gallery walls. https://www.youtube.com/watch?v=APpw6ZKIQ3I - **Le Petit Chef** — Skullmapping (Filip Sterckx & Antoon Verbeeck) (2015): A tiny projected chef cooks your meal right on your dinner plate. https://www.youtube.com/watch?v=yBJEP4lsRFY - **Study for Fifteen Points** — Random International (Hannes Koch & Florian Ortkrass) (2015): Fifteen moving dots of light are enough to see a person walking. https://www.youtube.com/watch?v=qnkxo7CWACs - **Waterlily Invasion** — Tamiko Thiel (2014): Water lilies that feed on your gaze grow over you the longer you look at them through the phone. https://www.youtube.com/watch?v=B1wVL_YSodE - **Reality Editor: Programming Smarter Objects** — Valentin Heun, MIT Fluid Interfaces — Pattie Maes & Pranav Mistry (2013): Drag a line between two real objects on your phone and they are wired together. https://vimeo.com/74073989 - **Sketch Aquarium** — teamLab (2013): A fish a child colors on paper is scanned in and swims away in the shared aquarium. https://www.youtube.com/watch?v=AnAqB7LZUb8 - **Strandbeest evolution (archive)** — Theo Jansen (2013): Creatures made of plastic tubes walk along the beach powered only by the wind. https://www.youtube.com/watch?v=MYGJ9jrbpvg - **Augmented Reality Sandbox** — Oliver Kreylos (2012): Pile up real sand and contour lines and water follow it instantly, so shaping a landscape is just moving sand. https://www.youtube.com/watch?v=j9JXtTj0mzE - **Super Mario Brush** — Cyril Diagne (2012): A Super Mario level drawn by a child on paper becomes the level Mario actually runs on. https://vimeo.com/46281850 - **LightSpace** — Microsoft Research — Hrvoje Benko & Andy Wilson (2010): Scoop a projected photo off the table, carry it in your hand and pour it onto the wall, as if data were liquid. https://www.youtube.com/watch?v=xx5kBqxyaHE - **Grandfather Clock (Real Time)** — Maarten Baas (2009): A man living inside a grandfather clock redraws its hands by hand every minute. https://www.youtube.com/watch?v=aYD-CDMhnmI - **Little People** — Slinkachu (2006): Tiny figures turn a drain or a crisp packet on the street into a whole scene. https://www.youtube.com/watch?v=VqTSqUOHTtg - **Wildlife** — Karolina Sobecka (2006): A projected tiger runs along the buildings beside a moving car, keeping pace and stopping when the car stops. https://vimeo.com/6400445 - **Pedestrian** — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2002): A tiny city of walking people appears inside the pavement under your feet. https://vimeo.com/186079084 - **Pick-and-Drop** — Jun Rekimoto (1997): Pick up a file with a pen on one screen and drop it on another, treating data as a physical object. https://www.youtube.com/watch?v=rFw9aMubL-Y - **Surface Tension** — Rafael Lozano-Hemmer (1992): A giant eye on the wall follows you wherever you move. https://www.youtube.com/watch?v=JXLoLPkzdto ### Drawing in space Drawing or writing directly in the air, where the mark itself is the idea. - **I/O Brush re-creation (Genuary)** — Torin Blankensmith (2025): Dip your hand into the real world to pick up its color and texture, then paint with it. https://x.com/blankensmithing/status/1875375318946795571 - **Poem World (Snap Spectacles)** — Zach Lieberman, Shantell Martin (2021): A poem is laid out in space, so reading it means walking through it. https://x.com/zachlieberman/status/1408879174429949953 - **Weird Type skate trails** — Zach Lieberman, Molmol Kuo (2020): A skateboard leaves a trail of words behind it, so a ride becomes a sentence in the air. https://x.com/zachlieberman/status/1271947807709069317 - **Drawalong AR** — Google Creative Lab (2019): A drawing tutorial video floats over your sketchbook as tracing paper. https://www.youtube.com/watch?v=S9g1PqY19jE - **Weird Cuts** — Zach Lieberman, Molmol Kuo (2019): Cut shapes out of the camera image and hang them in mid-air, making a collage in space. https://www.youtube.com/watch?v=fU0iFi0PXxI - **Just a Line** — Google Creative Lab (2018): You can only draw one white line in the air, and that constraint makes spatial drawing obvious to anyone. https://www.youtube.com/watch?v=IOKwGCQJVCw - **Stenciling in Augmented Reality** — Morten Just (2018): A picture cast onto paper lets anyone trace a good drawing by hand. https://www.youtube.com/watch?v=KjWL_zHOysM - **A Walk through the Line** — Chiharu Shiota (塩田千春) (2017): Threads knotted through a building let you walk inside a drawing in the air. https://www.youtube.com/watch?v=5UmbVYwyiqQ - **Invisible Highway** — Judith Amores, Anna Fusté (2017): Draw a road in AR and a real toy car drives along it. https://www.youtube.com/watch?v=9ZhqsuWF96I - **Water Light Graffiti** — Antonin Fourneau (2012): Paint the wall with plain water and it glows, then fades as it dries. https://vimeo.com/47080920 - **Making Future Magic: iPad light painting** — BERG (Jack Schulze, Matt Webb, Timo Arnall) (2010): A moving iPad becomes a brush that paints solid 3D words in the air. https://www.youtube.com/watch?v=C-dVnG8drxs - **The EyeWriter** — Zach Lieberman, Theo Watson, Chris Sugrue (2009): A paralyzed graffiti artist draws with his eyes and the tags appear live on city buildings. https://www.youtube.com/watch?v=84H-xLrLvvk - **bit.fall** — Julius Popp (2006): Falling water drops spell words that exist only while they fall. https://www.youtube.com/watch?v=gg9LWsfqqrk - **Pencil Mask (Bleistiftmaske)** — Rebecca Horn (1972): A mask of pencils turns every movement of the head into a line on the wall. https://www.youtube.com/watch?v=Eh9JH7daSbg - **Picasso Light Drawings** — Gjon Mili (1949): Picasso draws in the air with a light, and the drawing hangs there for one exposure. https://www.youtube.com/watch?v=X-i9eqlRzks ### Play & together A game or shared moment built on one simple rule. - **TouchPort / Allow me into your dream** — Botao 'Amber' Hu (2026): A handshake and a pull is all it takes to merge two people's separate mixed realities into one. https://www.youtube.com/watch?v=uMSXnUxnxd4 - **1D ARCADE** — 1024 Architecture (François Wunschel & Pier Schneider) (2025): A whole arcade game squeezed into a single line of light, and you still understand it at a glance. https://vimeo.com/1098205588 - **Projected fish school for cats** — Roelof Knol (2024): Projected fish scatter across the floor when a cat chases them: AR for an audience that cannot read screens. https://x.com/Rainmaker1973/status/1772239372521263467 - **The Floor Is Lava!** — Figmin XR (Overlay) (2023): The childhood game made literal: the floor really turns to lava and only the furniture is safe. https://www.youtube.com/watch?v=aOXpbDBwNyM - **Touch Grass lens** — Aidan Wolf (2023): The internet joke taken literally: the lens checks that you really touched grass and gives you proof. https://x.com/Aidan_Wolf/status/1701565816385179877 - **Hot Potato multiplayer lens** — Aidan Wolf (2021): Friends pass a ticking virtual potato phone to phone until it blows up on someone, a party game anyone already knows. https://x.com/Aidan_Wolf/status/1395445443333476352 - **Real World Splatoon (AR 100 Days, Day 39)** — Satoshi Hattori (2021): Paint balls splat onto your real walls and furniture, as in Splatoon. https://x.com/shmdevelop/status/1361732615988932612 - **Fort Builder** — FIELD (FIELD.IO / FIELD.SYSTEMS), SPACE10 (IKEA's research and design lab) (2020): Stack digital copies of your own cushions and chairs into a fort, then let it crash down without breaking anything. https://vimeo.com/554339068 - **Beyond Walls** — Saype (Guillaume Legros) (2019): Giant painted hands clasp each other on grass, city after city, forming a chain around the world. https://www.youtube.com/watch?v=GbX3m2mG644 - **Musical Chairs Against the Avengers in AR** — Abhishek Singh (2019): Musical chairs where the rivals are life-size virtual Avengers competing for real chairs. https://www.youtube.com/watch?v=mqd_6gdXgpk - **Bot Party** — Phoenix Perry (2017): Players can only pass messages between their little bots by touching skin to skin, so the body becomes the network cable. https://vimeo.com/252776298 - **CyberSnake – Holographic Snake** — Lucas Rizzotto (2017): You are the snake, walking through your home and dodging your own growing tail. https://www.youtube.com/watch?v=1SprJQz_pGU - **Laser Cat AR** — Abhishek Singh (2017): The whole game is one gesture: tease a virtual kitten on your floor with a laser dot. https://www.youtube.com/watch?v=LYszXfkwYeQ - **Rainbrow – Eyebrow-Controlled Game** — Nathan Gitter (2017): Raise your eyebrows to fly up: your face is the whole controller. https://www.youtube.com/watch?v=9k_9BKA3w_Q - **Portals for Mortals** — Jamie Gledhill (2016): Each doorway of the sculpture plays one part of a fanfare, so the full tune only sounds when a group walks through together. https://vimeo.com/180005726 - **Imaginary Reality Gaming: Quantum Basketball** — Christian Holz (2013): A basketball game with no ball: players only know where it is from how everyone else moves. https://www.youtube.com/watch?v=NNirAkibYGc - **Appel d'Air** — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2012): A child's breath on a tiny windmill comes back as a gust from a giant fan. https://vimeo.com/74689338 - **PixelPyros** — Seb Lee-Delisle (2012): Wave your hand and a firework launches from that exact spot. https://vimeo.com/61174060 - **Face Raiders** — Nintendo (2011): Your own face turns into the enemy that bursts through the walls of your room. https://www.youtube.com/watch?v=bh-YrWvdUCs - **Hand from Above** — Chris O'Shea (2009): A giant hand reaches into the big-screen feed of a square and picks up passers-by like toys. https://vimeo.com/7042266 - **Subtitled Public** — Rafael Lozano-Hemmer (2005): Each visitor is labeled with a projected verb, and the only way to pass it on is to touch someone else. https://vimeo.com/1089123902 - **The obliteration room** — Yayoi Kusama (草間彌生) (2002): Every visitor adds colored dots until a white room disappears under them. https://www.youtube.com/watch?v=-xNzr-fJHQw ## Visual effects (with source code) Real-time visual effects from the Unity VFX community, most with open source code; building blocks that translate easily to AR. ### Particles & swarms Thousands of points that flow, flock, burst or trail. - **Flying Embers** — Rob Cupisz (2025): Model how a real camera sees a moving hot spark instead of stretching a sprite. https://x.com/robcupisz/status/1883842759825715217 · Source code: https://github.com/robcupisz/flying-embers - **Constellation Plexus Ribbons** — Mirza Beig (2024): Connect nearby points with light and a cloud of dots becomes a constellation. https://x.com/TheMirzaBeig/status/1840974568216813609 · Source code: https://github.com/MirzaBeig/Constellation-Plexus - **Rendering Millions of Particles** — Acerola (Garrett Gunnell) (2024): Apply a few random affine maps over and over and a fractal appears out of points. https://www.youtube.com/watch?v=1L-x_DH3Uvg · Source code: https://github.com/GarrettGunnell/Iterated-Function-Systems - **VFX Graph sketch: glowing lines on a moving figure** — Keijiro Takahashi (2024): Draw a body only with the lines that follow it. https://x.com/_kzr/status/1790748365518725597 · Source code: https://github.com/keijiro/VfxGraphTestbed3 - **VFX Graph custom HLSL: plexus network** — Keijiro Takahashi (2023): Connect nearby points and a swarm becomes a web. https://x.com/_kzr/status/1812469561704468638 · Source code: https://github.com/keijiro/VFXCustomCode - **VFX Sketchbook (Godot 4)** — Thibaud Goiffon (HeyTibo) (2023): A sketchbook habit: one quick effect at a time, all shared openly. https://www.youtube.com/watch?v=M8iTroIF1ag · Source code: https://github.com/gtibo/VFX-sketchbook-Godot-4.x - **StickShow: a sea of glow sticks** — Keijiro Takahashi (2022): A crowd is a repeated object with small differences. https://x.com/_kzr/status/1564979744642179073 · Source code: https://github.com/keijiro/StickShow - **godot-particles** — Leon Denise (2022): A minimal GPU-particle template for an open-source engine. https://x.com/leondenise/status/1509126396203126784 · Source code: https://github.com/leon196/godot-particles - **Aura VFX Samples** — rngtm (Kamosoba) (2021): A convincing aura is a few simple meshes and particles with textures that flow over them. https://x.com/rn49rn49/status/1346612720234205185 · Source code: https://github.com/rngtm/Unity-VFXToolBox - **Coding Adventure: Ant and Slime Simulations** — Sebastian Lague (2021): Simple agents that sense and deposit trails grow living networks. https://www.youtube.com/watch?v=X-iSQQgOd1A · Source code: https://github.com/SebLague/Slime-Simulation - **Compute shader data into VFX Graph** — Keijiro Takahashi (2021): Simulate anywhere, render with VFX Graph. https://x.com/_kzr/status/1418439725631754244 · Source code: https://github.com/keijiro/VfxGraphGraphicsBufferTest - **GPU Fog Particles** — Mirza Beig (2021): Fog made purely from noise can hug the world without hard edges. https://x.com/TheMirzaBeig/status/1471820398056677376 · Source code: https://github.com/MirzaBeig/GPU-Fog-Particles - **GPU Instanced Flocking on URP** — Kodai Takao (2021): A school of thousands only needs local rules and the GPU. https://www.youtube.com/watch?v=FXcFn8QM8dk · Source code: https://github.com/kodai100/Unity_BoidsSimulationOnURP - **Ghost of Tsushima lanterns (VFX Graph)** — Codrin Mihail (Kodrin) (2021): Recreate a memorable game atmosphere entirely on the GPU. https://x.com/CodrinMihail/status/1358992850009088000 · Source code: https://github.com/Kodrin/GoTLanterns-VFXGraph - **HDRP VFX Graph Workshop (VFX-Essentials)** — Codrin Mihail (Kodrin) (2021): Teach VFX Graph by starting from waves: position particles with simple math before anything else. https://x.com/CodrinMihail/status/1366480535862837256 · Source code: https://github.com/Kodrin/VFX-Essentials - **Particle and LiDAR Scan (AR 100 Days, Day 25)** — Satoshi Hattori (2021): Once particles can feel real geometry, every object becomes a landscape for them. https://x.com/shmdevelop/status/1354909729928044545 · Source code: https://github.com/satoshi0212/AR_100Days - **SushiVfx: vaporizing a sushi** — Keijiro Takahashi (2021): Real food, scanned, becomes a playful effect. https://x.com/_kzr/status/1442857433483726859 · Source code: https://github.com/keijiro/SushiVfx - **VFX Graph Dragon** — Tasuku Takahashi (2021): Build a creature out of particles, so one dragon becomes a thousand for free. https://x.com/supertask_jp/status/1463189020964364289 · Source code: https://github.com/TranscendVFX/VFXGraphIntermediate - **VFX Graph with 300,000 particles** — Keijiro Takahashi (2021): Scale changes the feeling: a crowd of particles becomes a material. https://x.com/_kzr/status/1400753544999817216 · Source code: https://github.com/keijiro/VfxGraphTestbed2 - **Abcvfx: Alembic animation to VFX Graph** — Keijiro Takahashi (2020): Any baked animation can feed a particle system. https://x.com/_kzr/status/1269876564486807552 · Source code: https://github.com/keijiro/Abcvfx - **GPU-FlowField** — Codrin Mihail (Kodrin) (2020): Let a noise-based vector field steer a huge particle swarm. https://x.com/CodrinMihail/status/1279055017094205440 · Source code: https://github.com/Kodrin/GPU-FlowField - **Krbv: colorful particle strip tunnel** — Keijiro Takahashi (2020): Ribbons moving past the camera create a sense of speed. https://x.com/_kzr/status/1219572530236641285 · Source code: https://github.com/keijiro/Krbv - **Particle depth of field with VFX Graph** — Keijiro Takahashi (2020): Fake camera blur per particle to add depth cheaply. https://x.com/_kzr/status/1290633852990287872 · Source code: https://github.com/keijiro/DofVfxSamples - **Sword fighting effects with VFX Graph** — Keijiro Takahashi (2020): A trail turns a fast motion into a readable shape. https://x.com/_kzr/status/1276106752728031232 · Source code: https://github.com/keijiro/VfxGraphTestbed - **3D Physarum Transport Networks** — Benoit Arbelot (2019): Slime-mould logic in 3D: simple agents following their own trails build networks. https://vimeo.com/379589358 · Source code: https://github.com/Barbelot/Physarum3D - **AR Foundation + VFX Graph** — Dan Miller (2019): Any VFX Graph effect becomes an AR effect once the render pipeline and the camera background agree. https://x.com/DanMillerDev/status/1186744214392049664 · Source code: https://github.com/DanMillerDev/ARFoundation_VFX - **Coding Adventure: Boids** — Sebastian Lague (2019): Separation, alignment and cohesion are enough to make a flock. https://www.youtube.com/watch?v=bqtqltqcQhw · Source code: https://github.com/SebLague/Boids - **GeoVfx: world population as particles** — Keijiro Takahashi (2019): Data becomes landscape when every value is a particle. https://x.com/_kzr/status/1429739880259612685 · Source code: https://github.com/keijiro/GeoVfx - **Mixed Reality Theater** — CJT (CJT-Jackton) (2019): Stage effects no longer need to be physical: every seat sees them through AR glasses. https://www.youtube.com/watch?v=EEx642qwkxE · Source code: https://github.com/CJT-Jackton/Mixed-Reality-Theater - **Physarum with Vector Fields** — Deniz Bicer (2019): Add one outside force to an emergent system and it starts drawing with intent. https://x.com/ojelibalon/status/1168248320529174535 · Source code: https://github.com/DenizBicer/Physarum - **Project North Star: Strange Attractors in VFX Graph** — Tasuku Takahashi (2019): Let a chaotic equation, not an animator, choreograph a swarm of light in the room. https://www.youtube.com/watch?v=kLG291XVf_k · Source code: https://github.com/supertask/VFXNorthStar - **Smrvfx: particles from a skinned mesh** — Keijiro Takahashi (2019): Any animated body can become an emitter. https://x.com/_kzr/status/1114513038302830592 · Source code: https://github.com/keijiro/Smrvfx - **VFX Graph Sketch1012** — Keijiro Takahashi (2019): Daily small sketches are a way to learn a tool. https://x.com/_kzr/status/1187004957821505536 · Source code: https://github.com/keijiro/VfxGraphTestbed - **VFX Graph Sun with 2 Million Particles** — Dilmer Valecillos (2019): Millions of tiny points are enough to build a star. https://www.youtube.com/watch?v=f1BHXqeokSE · Source code: https://github.com/dilmerv/UnityVFXMillionsOfParticles - **VfxPyro: interactive fireworks** — Keijiro Takahashi (2019): A classic firework is the simplest lesson in spawn, burst and trail. https://x.com/_kzr/status/1179427868587130880 · Source code: https://github.com/keijiro/VfxPyro - **Demographics of Israel: Data Sculpture** — Yuma Yanagisawa (2018): Let a dataset set the size of a living particle sculpture instead of drawing a chart. https://www.youtube.com/watch?v=qop7sk6H4GA · Source code: https://github.com/yumayanagisawa/Unity-Visual-Effects-Graph-Practice - **GPUVoxelParticleSystem (unity-voxel)** — Masatatsu Nakamura (mattatz) (2018): Turn any mesh into thousands of voxel particles that can leave and return to their home position. https://x.com/mattatz/status/951031022342242304 · Source code: https://github.com/mattatz/unity-voxel - **ShaderToy to Unity: Particle Collision** — Przemyslaw Zaworski (2018): Particle physics can live entirely in a GPU buffer. https://www.youtube.com/watch?v=gUhTbheuoyo · Source code: https://github.com/przemyslawzaworski/Unity3D-CG-programming - **Spaceship Demo** — Unity Visual Effect Graph team (Thomas Iché, Julien Fryer, Vlad Neykov and the Unity Demo Team) (2018): Show everything a GPU particle graph can do inside one playable space. https://www.youtube.com/watch?v=rqMcPZoEc3U · Source code: https://github.com/Unity-Technologies/SpaceshipDemo - **GpuTrail: 100,000 Trails** — fuqunaga (2017): Trails, usually expensive, can be fully GPU-driven at massive scale. https://www.youtube.com/watch?v=sS2MYj6LceY · Source code: https://github.com/fuqunaga/GpuTrail - **Swarm** — Keijiro Takahashi (2017): Constrain a noise-driven swarm so it hugs the surface of an object. https://vimeo.com/219277691 · Source code: https://github.com/keijiro/Swarm - **Holographic ribbons for VRDG+H** — Keijiro Takahashi (2016): An optical virtual image makes a Unity statue and ribbons float in the air and dance to music. https://x.com/_kzr/status/709712057235406848 - **Screen Space Collision GPU Particles** — hecomi (2016): If you can see a surface, particles can bounce off it: use the depth buffer as the collider. https://www.youtube.com/watch?v=G-4k8Zur7zY · Source code: https://github.com/hecomi/UnityPseudoInstancedGPUParticles - **Screen Space Particle Collision** — Hironori Sugino (sugi-cho) (2016): Use what the camera sees as the physics world for particles. https://www.youtube.com/watch?v=E81EVRG0SlU · Source code: https://github.com/sugi-cho/Unity-GPU-Particle - **Kvant Spray** — Keijiro Takahashi (2015): Treat any mesh as a particle and spray huge numbers of them. https://vimeo.com/117040444 · Source code: https://github.com/keijiro/KvantSpray - **Kvant Swarm** — Keijiro Takahashi (2015): Draw each particle's path as a line, so a swarm turns into flowing strands. https://vimeo.com/134624419 · Source code: https://github.com/keijiro/KvantSwarm - **Boids** — Keijiro Takahashi (2014): Three local rules are enough to make a lifelike flock. https://vimeo.com/87151096 · Source code: https://github.com/keijiro/Boids - **Fragments** — Keijiro Takahashi (2014): Better particle effects come from interesting shapes, not just from more particles. https://vimeo.com/102452671 · Source code: https://github.com/keijiro/Fragments - **Turbulent Flow (particle animations)** — Keijiro Takahashi (2013): Shape a particle system with external forces instead of emitter settings. https://vimeo.com/80264404 · Source code: https://github.com/keijiro/unity-particle-animations ### Point clouds & depth Depth cameras, LiDAR and volumetric video turned into living 3D material. - **QuestRoomScan** — Arghya Sur (Genesis Interactive) (2026): The headset can scan the room while you use it, so the real space becomes editable material. https://www.youtube.com/watch?v=lEn3GkH7Yao · Source code: https://github.com/genesisinteractive/QuestRoomScan - **Triangle Splatting importer** — Keijiro Takahashi (2025): Scenes captured as triangles run on any normal renderer. https://x.com/_kzr/status/1939302534285582621 · Source code: https://github.com/keijiro/TriangleSplattingTest - **Depth Scanner for Meta Quest 3** — Appletea (2024): The headset already sees depth; just show the user what it sees. https://x.com/Appletea_VRC/status/1861910477389086953 · Source code: https://github.com/Appletea0673/Depth-Scanner-Project - **Metavido VFX (WebGPU)** — Keijiro Takahashi (2024): Put a phone video with depth onto a web page, so anyone can walk around its effects in a browser. https://x.com/_kzr/status/1828366682689061280 · Source code: https://github.com/keijiro/MetavidoVFX - **Rcam3 for Masaya Matsuura's concert** — Keijiro Takahashi (2024): Use a phone's LiDAR as a volumetric camera on stage to generate live show visuals. https://x.com/_kzr/status/1871169155254435978 · Source code: https://github.com/keijiro/Rcam3 - **SplatVFX: 3D Gaussian Splatting in VFX Graph** — Keijiro Takahashi (2023): A photoreal capture that is also a particle system. https://x.com/_kzr/status/1714214841265856932 · Source code: https://github.com/keijiro/SplatVFX - **Unity Gaussian Splatting** — Aras Pranckevičius (2023): A photo capture can be rendered directly as millions of fuzzy points, with no meshes or textures. https://www.youtube.com/watch?v=iccfV0YlWVI · Source code: https://github.com/aras-p/UnityGaussianSplatting - **AR VJing + Volumetric Video** — Tasuku Takahashi (2022): VJ a captured human body in AR the way you would VJ a video clip. https://x.com/supertask_jp/status/1504792152315871238 · Source code: https://github.com/supertask/4DVFXProject - **BibcamStage: live show at Channel 22** — Keijiro Takahashi (2022): Everyday walks recorded with a phone become stage material. https://www.youtube.com/watch?v=v3mRlMx_6Is · Source code: https://github.com/keijiro/BibcamStage - **VFX with Unity AR Foundation** — Keijiro Takahashi (2022): Grow light effects along the real environment live on a phone. https://x.com/_kzr/status/1601509152395706369 · Source code: https://github.com/keijiro/Rcam2 - **Walking a Taiwanese street with iPhone LiDAR** — Keijiro Takahashi (2022): Turn LiDAR's short sight into a dreamlike style of street scene. https://x.com/_kzr/status/1607672355710709760 - **AR MeshWave** — Takahiro "Poly" Horikawa (2021): A light wave that only exists on real surfaces makes the scanned room itself the effect. https://x.com/thorikawa/status/1387068106757992449 · Source code: https://github.com/thorikawa/ar-meshwave - **ARVolumeVFX — Eyez Matrix** — Eyez (2021): Rebuild the real room out of particles, live, from the device's depth sensor. https://x.com/EyezCG/status/1460742859661869059 · Source code: https://github.com/EyezLee/ARVolumeVFX - **Adding VFX to a Bibcam clip** — Keijiro Takahashi (2021): Add space-aware post effects to a depth video that has already been recorded. https://x.com/_kzr/status/1460618910990929926 · Source code: https://github.com/keijiro/BibcamVfx - **Bibcam test in Shibuya** — Keijiro Takahashi (2021): Burn depth and camera pose into the video as you record, so you can later add effects that fit the street space. https://vimeo.com/651111230 · Source code: https://github.com/keijiro/Bibcam - **KinFuSDFVFX** — Kaito Tsutsumi (にー兄さん / drumath2237) (2021): Scan a real space once and give particles its shape to hold on to. https://www.youtube.com/watch?v=ApPVp6Z3cgE · Source code: https://github.com/drumath2237/KinFuSDFVFX - **LiDAR Radar (AR 100 Days, Day 21)** — Satoshi Hattori (2021): Show people what the device sees, then let them see it from above. https://x.com/shmdevelop/status/1353238818824970241 · Source code: https://github.com/satoshi0212/AR_100Days - **OAK-D-Lite stereo depth particles** — Keijiro Takahashi (2021): A cheap stereo camera is enough for body particles. https://x.com/_kzr/status/1474726909917667330 · Source code: https://github.com/keijiro/DepthAITestbed - **PointCloudExplorer** — Masatatsu Nakamura (mattatz) (2021): Treat a real place scanned as points as a living material, not a finished model. https://x.com/mattatz/status/1351050543746211842 · Source code: https://github.com/mattatz/PointCloudExplorer - **Space Voxels (AR 100 Days, Day 20)** — Satoshi Hattori (2021): Paint the room into voxels that borrow their color from reality. https://x.com/shmdevelop/status/1352835578509709312 · Source code: https://github.com/satoshi0212/AR_100Days - **4DViews volumetric video with VFX Graph** — Keijiro Takahashi (2020): Studio volumetric capture can be remixed like any other footage. https://x.com/_kzr/status/1270712472853340167 · Source code: https://github.com/keijiro/4DViewsTest2 - **Akvj: Azure Kinect VJ set** — Keijiro Takahashi (2020): A depth camera becomes a live visual instrument. https://vimeo.com/424260614 · Source code: https://github.com/keijiro/Akvj - **Layered Light Field Video on Quest** — Julien Kipp (julienkay) (2020): Volumetric video can be a stack of video-textured shells rather than a point cloud. https://x.com/julien_kaye/status/1279487242608742401 · Source code: https://github.com/julienkay/LightfieldVideoUnity - **PcxEffects3: point cloud effects** — Keijiro Takahashi (2020): A static scan becomes a living, breathing place. https://x.com/_kzr/status/1293928930647261184 · Source code: https://github.com/keijiro/PcxEffects3 - **Point Cloud Garden** — Yuma Yanagisawa (2020): A scanned place becomes a living material when its points can move. https://www.youtube.com/watch?v=J6QHswn6Zdg · Source code: https://github.com/yumayanagisawa/Unity-Point-Cloud-VFX-Graph - **Rcam2: Volumetric AR camera rig** — Keijiro Takahashi (2020): Use an iPad's LiDAR as a mobile volumetric camera so effects grow right on the real room. https://vimeo.com/461782810 · Source code: https://github.com/keijiro/Rcam2 - **Akvfx: Azure Kinect with VFX Graph** — Keijiro Takahashi (2019): Treat a depth sensor as a live particle source. https://x.com/_kzr/status/1163456709894922240 · Source code: https://github.com/keijiro/Akvfx - **Dkvfx: Depthkit volumetric video with VFX Graph** — Keijiro Takahashi (2019): Recorded volumetric video becomes raw material for particle effects. https://x.com/_kzr/status/1105456612162994177 · Source code: https://github.com/keijiro/Dkvfx - **Project North Star: VFX Graph with RealSense** — Tasuku Takahashi (2019): Feed a live depth camera back into an AR headset so the room itself becomes particle material. https://www.youtube.com/watch?v=oYYl_ALUDQI · Source code: https://github.com/supertask/VFXNorthStar - **Rcam at Channel #20 (live with umio)** — Keijiro Takahashi (2019): Scan a performer into a live point cloud, then let particles tear the body apart and rebuild it as concert visuals. https://vimeo.com/346711967 · Source code: https://github.com/keijiro/Rcam - **Record3D → Unity VFX Graph Point Cloud Streaming** — Marek Šimoník (Record3D) (2019): The depth camera already in your pocket can replace a Kinect for live volumetric effects. https://x.com/record3d/status/1175234894529798145 · Source code: https://github.com/marek-simonik/record3d_unity_demo - **Rsvfx: RealSense depth to VFX Graph** — Keijiro Takahashi (2019): A cheap depth camera turns a live person into particle material. https://x.com/_kzr/status/1099299041463066624 · Source code: https://github.com/keijiro/Rsvfx - **VFX sketches with Depthkit and Unity** — Keijiro Takahashi (2019): One recording, many visual personalities. https://vimeo.com/383216987 · Source code: https://github.com/keijiro/DkvfxSketches - **VFXGraphSandbox: Runtime Point Cache** — fuqunaga (2019): Generate the point cache live instead of baking it, so any animated mesh can become particles. https://x.com/fuqunaga/status/1209023053016924161 · Source code: https://github.com/fuqunaga/VFXGraphSandbox - **Depth Camera Voxel Particles** — Hironori Sugino (sugi-cho) (2018): Rebuild a live body from cubes that can break away. https://www.youtube.com/watch?v=r6qXq082iH4 · Source code: https://github.com/sugi-cho/RealSenseVisualize - **Pcx point cloud renderer** — Keijiro Takahashi (2017): Make raw scan data a first-class, animatable material inside a game engine. https://vimeo.com/239850990 · Source code: https://github.com/keijiro/Pcx - **Kinect V2 Depth into the G-Buffer** — hecomi (2016): Treat the depth camera image as part of the renderer, not as a texture. https://www.youtube.com/watch?v=Nl3lxlz0qME · Source code: https://github.com/hecomi/UnityKinectV2DeferredRendering ### Body & face effects Effects driven by segmentation, pose, hands or face tracking. - **BodyPix body part tracking** — Keijiro Takahashi (2023): Know which part of the body is where, and give each part its own effect. https://x.com/_kzr/status/1626200056033599491 · Source code: https://github.com/keijiro/BodyPixSample - **Interactive VJ with Azure Kinect** — Tasuku Takahashi (2023): Turn your own body into the VJ source instead of a video clip. https://x.com/supertask_jp/status/1612166613931032577 · Source code: https://github.com/supertask/InteractiveVJ - **Virtual Wearable** — Tasuku Takahashi (2023): Replace rectangular screens with interfaces that are worn on the body like jewellery. https://x.com/supertask_jp/status/1635235099989143552 · Source code: https://github.com/supertask/VirtualWearable - **Dithered Hand as Physarum Stimulus** — Deniz Bicer (2022): An image becomes food: the slime draws a picture by eating it. https://x.com/ojelibalon/status/1477660120200974338 · Source code: https://github.com/DenizBicer/Physarum - **VFX Dancer** — Tasuku Takahashi (2022): Make motion visible by letting the body leave its own history behind as light. https://x.com/supertask_jp/status/1524424117948088321 · Source code: https://github.com/TranscendVFX/VFXGraphIntermediate - **3D Face Landmarks in Shaders** — SCRN (2021): Put the whole face-tracking pipeline on the GPU so an avatar can see its owner without plugins. https://x.com/SCRNinVR/status/1404187468153327617 · Source code: https://github.com/SCRN-VRC/3D-Face-Landmark-in-UnityCG-HLSL - **ARVolumeVFX — Human particles** — Eyez (2021): Use the AR human stencil to keep particles only inside a person's body. https://user-images.githubusercontent.com/18374192/142727769-b8595802-7458-47a5-8802-a7c10fb85871.mp4 · Source code: https://github.com/EyezLee/ARVolumeVFX - **BlazeFace face filters on Barracuda** — Keijiro Takahashi (2021): Six keypoints are enough to decorate a face. https://x.com/_kzr/status/1378352493134929926 · Source code: https://github.com/keijiro/BlazeFaceBarracuda - **BodyPix visual effects (NNCam)** — Keijiro Takahashi (2021): Cut the person out of the background in real time so effects only touch the body. https://vimeo.com/580670067 · Source code: https://github.com/keijiro/NNCam - **FaceMeshBarracuda face & eye tracker** — Keijiro Takahashi (2021): Get a face mesh and eye tracking from one camera in Unity for mask and gaze effects. https://vimeo.com/545493860 · Source code: https://github.com/keijiro/FaceMeshBarracuda - **HandPoseBarracuda hand tracker** — Keijiro Takahashi (2021): Get finger joints in Unity from an ordinary camera as a building block for gesture effects. https://vimeo.com/545493967 · Source code: https://github.com/keijiro/HandPoseBarracuda - **IrisBarracuda: eye and iris tracking** — Keijiro Takahashi (2021): Knowing where the iris is opens up gaze-driven effects. https://x.com/_kzr/status/1382324941861769220 · Source code: https://github.com/keijiro/IrisBarracuda - **Neon Human (AR 100 Days, Day 33)** — Satoshi Hattori (2021): Draw light along the edge of the human silhouette to turn anyone into a music-video character. https://x.com/shmdevelop/status/1358940972499959808 · Source code: https://github.com/satoshi0212/AR_100Days - **UltraFace: realtime emoji face overlay** — Keijiro Takahashi (2021): Face detection plus a sticker is the simplest face filter. https://x.com/_kzr/status/1361657191401365505 · Source code: https://github.com/keijiro/UltraFaceBarracuda - **Hand-Tracked Forces on VFX Particles (Quest)** — Dilmer Valecillos (2020): Your hands become force fields for particles. https://www.youtube.com/watch?v=EyMF2Wo1awo · Source code: https://github.com/dilmerv/OculusQuestHandTrackingPhysicsURP - **AR Body Tracking with Head and Hand Particles** — Dilmer Valecillos (2019): Give real people superpowers by attaching effects to their tracked joints. https://www.youtube.com/watch?v=jxvBrfuyusU · Source code: https://github.com/dilmerv/UnityARFoundationEssentials - **Skeletal Geometric Effects** — Keijiro Takahashi (2018): Build rich geometry from nothing more than a skeleton's bones. https://va.media.tumblr.com/tumblr_pcsc78PRjA1qio469.mp4 · Source code: https://github.com/keijiro/SkeletalGeometricEffects - **GlitchDancer** — Keijiro Takahashi (2017): Show that a phone can run club-grade real-time 3D visuals. https://vimeo.com/198537336 · Source code: https://github.com/keijiro/GlitchDancer - **Skinner** — Keijiro Takahashi (2016): Use every vertex of a moving body as an emitter, so the motion itself paints the effect. https://vimeo.com/197396746 · Source code: https://github.com/keijiro/Skinner ### Audio-reactive Visuals that listen: sound and music shape the effect. - **DrumPadVFX: finger drum visualizer** — Keijiro Takahashi (2024): Each pad gets its own visual voice. https://x.com/_kzr/status/1849430841119973885 · Source code: https://github.com/keijiro/DrumPadVFX - **Pollen VFX Composition** — Jonathan Thorpe (Sonosthesia) (2024): Grow an organic form whose opening and shedding are timed by sound. https://www.youtube.com/watch?v=MOeGDgMatwg · Source code: https://github.com/jbat100/sonosthesia-unity-demo-deform - **Sonosthesia: GPU Sound Visualisation** — Jonathan Thorpe (Sonosthesia) (2024): Break sound into a few descriptor streams and wire each one to a visual parameter. https://x.com/johnnyfrenchy/status/1752349779223330997 · Source code: https://github.com/jbat100/sonosthesia-unity-demo-deform - **Spark VFX Composition** — Jonathan Thorpe (Sonosthesia) (2024): Let each part of the music own one part of the effect: the core, the sparks, the glow. https://www.youtube.com/watch?v=_2siOZ5pIWQ · Source code: https://github.com/jbat100/sonosthesia-unity-demo-deform - **World Ensemble** — Koki Ibukuro (asus4) (2023): Use the geometry of the city as a sequencer: every building you tag becomes a voice in the song. https://www.youtube.com/watch?v=adra9QppRO0 · Source code: https://github.com/asus4/WorldEnsemble - **Generative VFX Music Video (Bitwig x Unity)** — Kaito Tsutsumi (にー兄さん / drumath2237) (2022): Let probability, not a timeline, direct both the music and the visuals. https://www.youtube.com/watch?v=gARJwIS5VGc · Source code: https://github.com/drumath2237/Generative-VFX-Bitwig-Sandbox - **VzoVfx: triggering VFX from Bitwig** — Keijiro Takahashi (2022): Let the music software drive the visuals directly. https://x.com/_kzr/status/1492752963206406148 · Source code: https://github.com/keijiro/VzoVfx - **Audio Reactive FaceMap (AR 100 Days, Day 52)** — Satoshi Hattori (2021): Let the face become a VU meter. https://x.com/shmdevelop/status/1412363580406517765 · Source code: https://github.com/satoshi0212/AR_100Days - **Voice Recognition with Shaders** — SCRN (2021): Recover speech from lip-sync signals when the raw audio is out of reach. https://x.com/SCRNinVR/status/1367343416644886531 · Source code: https://github.com/SCRN-VRC/Voice-Recognition-Shader - **Khoreo: procedural dance with the MC-101** — Keijiro Takahashi (2020): The music plays the dancer. https://x.com/_kzr/status/1291723680490254336 · Source code: https://github.com/keijiro/Khoreo - **Grubo: MC-101 live performance at Channel #21** — Keijiro Takahashi (2019): One small groovebox drives both the music and the visuals. https://vimeo.com/379562830 · Source code: https://github.com/keijiro/Grubo - **LaspVfx: audio-reactive VFX with LASP** — Keijiro Takahashi (2019): Let the sound itself draw the shape. https://x.com/_kzr/status/1116337708782067713 · Source code: https://github.com/keijiro/LaspVfx - **VFX controlled by MIDI notes (Minis)** — Keijiro Takahashi (2019): Play visuals like an instrument. https://x.com/_kzr/status/1194287720203374592 · Source code: https://github.com/keijiro/VfxMinisExamples - **Museum (Channel 18)** — Keijiro Takahashi (2018): Mix improvised code with prepared 3D props, like a museum curated live. https://www.youtube.com/watch?v=r-nbIpdn-Lk · Source code: https://github.com/keijiro/Museum - **OP-Z videolab visuals** — Keijiro Takahashi (2018): Let a synthesizer's sequencer play 3D visuals directly. https://vimeo.com/307285370 · Source code: https://github.com/keijiro/VideolabTest - **Beta (live coding at Channel 16)** — Keijiro Takahashi (2017): Live-code the texture of a 3D world while the music plays. https://vimeo.com/213872212 · Source code: https://github.com/keijiro/Beta - **LASP audio-reactive particles** — Keijiro Takahashi (2017): Make visuals react to sound with no perceptible delay. https://va.media.tumblr.com/tumblr_otamd9rnf21qio469.mp4 · Source code: https://github.com/keijiro/Lasp - **Republic (FEMM live)** — Keijiro Takahashi (2017): Match real-time 3D visuals to a pop act's choreography and music. https://x.com/tokyomax/status/878736675228090368 · Source code: https://github.com/keijiro/Republic - **Seido (静動)** — Keijiro Takahashi (2017): Turn a row of screens into one rhythmic, public-domain visual instrument. https://va.media.tumblr.com/tumblr_p19olvfZwP1qio469.mp4 · Source code: https://github.com/keijiro/Seido - **ShaderSketches** — Keijiro Takahashi (2017): Keep a daily habit of tiny shader sketches. https://va.media.tumblr.com/tumblr_ooco6l0oEQ1qio469.mp4 · Source code: https://github.com/keijiro/ShaderSketches - **TouchDesignerPoliceAudioReactive** — Hiroaki Oishi (irishoak) (2017): Make an audio-reactive piece whose subject is a typo everyone makes. https://vimeo.com/245430053 · Source code: https://github.com/hiroakioishi/TDPoliceAudioReactive - **Trinity (Channel 17)** — Keijiro Takahashi (2017): Let each drum hit trigger a different GPU effect across three screens. https://va.media.tumblr.com/tumblr_ou5rt7l30F1qio469_720.mp4 · Source code: https://github.com/keijiro/Trinity - **VJ04 (Channel #10)** — Keijiro Takahashi (2015): Use physically based rendering to make club visuals look like product shots in motion. https://www.youtube.com/watch?v=jLmbjudThlA · Source code: https://github.com/keijiro/VJ04 - **VJ05 (Channel #12)** — Keijiro Takahashi (2015): Build a reusable live-visual rig where every animated parameter is exposed to sound and MIDI. https://x.com/shutamegai/status/619489135749824513 · Source code: https://github.com/keijiro/VJ05 - **VJ01: Guitar Songs** — Keijiro Takahashi (2014): A generative visual system that listens to a song. https://vimeo.com/90006189 · Source code: https://github.com/keijiro/VJ01 - **VJ02** — Keijiro Takahashi (2014): Play a 3D game-engine scene like an instrument, with faders. https://vimeo.com/104780871 · Source code: https://github.com/keijiro/VJ02 ### Procedural forms & growth Geometry that grows, extrudes, tiles or deforms by rules. - **Dryad** — Owen (owenyuwono) (2026): A seed number is enough to grow a unique tree, down to each leaf. https://x.com/owenyuwono/status/2096193851648258351 · Source code: https://github.com/owenyuwono/dryad - **MeshSlicer: cutting objects in real time** — Keijiro Takahashi (2026): Cut any mesh and fill the wound. https://x.com/_kzr/status/2074120483688374287 · Source code: https://github.com/keijiro/MeshSlicer - **Infinite Grass Field** — Youssef Afella (2024): Grow grass only where the camera can see it, so a field can be infinite at a fixed cost. https://x.com/Youssef_Afella/status/1831049793629724836 · Source code: https://github.com/Youssef-Afella/UnityURP-InfiniteGrass - **Unity Runtime Shader** — Wita (witalosk) (2024): Edit an effect while it is running, so tuning happens in front of the audience or inside the scene. https://x.com/witalosk/status/1805984379543638156 · Source code: https://github.com/witalosk/UnityRuntimeShader - **VFX Graph as Mesh Deformer** — Ming Wai Chan (cinight) (2023): Treat every vertex as a particle and all of VFX Graph's forces become mesh deformers. https://www.youtube.com/watch?v=dmoOpK7DHRY · Source code: https://github.com/cinight/MinimalCompute - **Metawire: wireframe primitives for VFX** — Keijiro Takahashi (2022): Wireframes give a technical, holographic look with very little geometry. https://x.com/_kzr/status/1537797028817735680 · Source code: https://github.com/keijiro/Metawire - **Procedural walk with Animation Rigging** — Keijiro Takahashi (2022): Motion can be computed instead of keyframed. https://x.com/_kzr/status/1574788205010112513 · Source code: https://github.com/keijiro/CharacterRigTest - **Drawing with Epicycles (DFT)** — Ming Wai Chan (cinight) (2021): Any drawing can be rebuilt from rotating circles. https://www.youtube.com/watch?v=3YWKfmuZ1aw · Source code: https://github.com/cinight/MinimalCompute - **How Games Render So Much Grass** — Acerola (Garrett Gunnell) (2021): A meadow is one blade drawn a million times with small differences. https://www.youtube.com/watch?v=Y0Ko0kvwfgA · Source code: https://github.com/GarrettGunnell/Grass - **NoiseBall6: compute shader mesh on mobile** — Keijiro Takahashi (2021): Deform geometry directly on the GPU for heavy effects on phones. https://x.com/_kzr/status/1402611991118712841 · Source code: https://github.com/keijiro/NoiseBall6 - **Procedural Ring Mesh VFX** — rngtm (Kamosoba) (2021): Generate the effect's geometry in code, so its shape can be tuned like a parameter. https://x.com/rn49rn49/status/1347823897211666432 · Source code: https://github.com/rngtm/Unity-VFXToolBox - **3D Moebius Transformations** — Shahriar Shahrabi (2020): Bend space instead of objects: send the scene to a 4D sphere, rotate it, and bring it back. https://www.youtube.com/watch?v=81XDjBiuuEI · Source code: https://github.com/IRCSS/3D-Moebius-Transformations-Vertex-shader-in-Unity-3D - **IMMATERIA: Forms On Forms** — Isaac Cohen (Cabbibo) (2020): Chain simple compute-shader 'lifeforms' so one simulation lives on the surface of another. https://www.youtube.com/watch?v=GC5f3d3uanM · Source code: https://github.com/cabbibo/IMMATERIA - **Mesh Deformation with Compute Shaders on Quest** — Shahriar Shahrabi (2020): Make virtual surfaces soft to the touch on standalone mobile hardware. https://www.youtube.com/watch?v=IVy6T5_9r2c · Source code: https://github.com/IRCSS/Mesh-Deformation-With-Compute-Shader-Oculus-Quest-Unity - **Procedural Painting with Genetic Evolution** — Shahriar Shahrabi (2020): Let evolution, not a filter, decide where each brush stroke goes. https://www.youtube.com/watch?v=--YygVe0Eu4 · Source code: https://github.com/IRCSS/Procedural-painting - **VertexAnimationJob: multithreaded vertex animation** — Keijiro Takahashi (2020): Push every vertex every frame, and a static model comes alive. https://x.com/_kzr/status/1217326699714494464 · Source code: https://github.com/keijiro/VertexAnimationJob - **VfxCrystal: growing crystals** — Keijiro Takahashi (2020): Mesh particles can build solid-looking forms. https://x.com/_kzr/status/1333403226813595649 · Source code: https://github.com/keijiro/VfxCrystal - **WfcMaze: Wave Function Collapse maze** — Keijiro Takahashi (2020): Local rules can build a coherent structure. https://x.com/_kzr/status/1249358391975137282 · Source code: https://github.com/keijiro/WfcMaze - **Coding Adventure: Hydraulic Erosion** — Sebastian Lague (2019): Let simulated rain, not an artist, carve the landscape. https://www.youtube.com/watch?v=eaXk97ujbPQ · Source code: https://github.com/SebLague/Hydraulic-Erosion - **Coding Adventure: Marching Cubes** — Sebastian Lague (2019): Turn a field of numbers into a surface you can carve. https://www.youtube.com/watch?v=M3iI2l0ltbE · Source code: https://github.com/SebLague/Marching-Cubes - **NoiseBall5: mesh deformed by the Job System** — Keijiro Takahashi (2019): A living blob is just noise applied to vertices. https://x.com/_kzr/status/1211628843343081472 · Source code: https://github.com/keijiro/NoiseBall5 - **ProcCharVfx: procedural letters and Matrix rain** — Keijiro Takahashi (2019): Generate an alphabet instead of loading one. https://x.com/_kzr/status/1209489319544549376 · Source code: https://github.com/keijiro/ProcCharVfx - **CellularGrowth** — Masatatsu Nakamura (mattatz) (2018): Let simple push-and-divide rules for thousands of cells grow an organic form on the GPU. https://x.com/mattatz/status/1002190822064861184 · Source code: https://github.com/mattatz/CellularGrowth - **CurveModifier** — Leon Denise (2018): Bend any mesh along a path, as Blender's curve modifier does, but live in the engine. https://x.com/leondenise/status/965584020989005824 · Source code: https://github.com/leon196/CurveModifier - **GPUVoxelSkinnedMesh (unity-voxel)** — Masatatsu Nakamura (mattatz) (2018): Voxelize a moving, skinned body in real time so motion itself becomes blocky sculpture. https://x.com/mattatz/status/955682645131718657 · Source code: https://github.com/mattatz/unity-voxel - **Procedural dance (PuppetTest)** — Keijiro Takahashi (2018): Generate dance from a few sine waves and noise instead of motion capture. https://vimeo.com/255257338 · Source code: https://github.com/keijiro/PuppetTest - **Reaction-Diffusion: Coral** — Andy Duboc (2018): Two chemicals and two rates are enough to grow coral. https://x.com/andyduboc/status/1069928765277593601 · Source code: https://github.com/andydbc/unity-reaction-diffusion - **3D Reaction Diffusion (Marching Cubes)** — kaiware007 (2017): Run reaction-diffusion in 3D and turn the chemical field into a living surface. https://x.com/kaiware007/status/867068705741586432 · Source code: https://github.com/kaiware007/UnityReactionDiffusion3D - **Cloner** — Keijiro Takahashi (2017): Build a complex organic shape out of many clones of one simple part. https://vimeo.com/218961301 · Source code: https://github.com/keijiro/Cloner - **Conway's Game of Life on the GPU** — fuqunaga (2017): Four rules on a grid produce endless emergent life. https://www.youtube.com/watch?v=Oxo22vWgzKc · Source code: https://github.com/fuqunaga/GpuLifeGame - **Conway's Game of Life with Compute Shaders** — World of Zero (Sam Wronski) (2017): Four simple rules on a grid generate endless living patterns. https://www.youtube.com/watch?v=ItPTBSeGjdM · Source code: https://github.com/WorldOfZero/UnityVisualizations - **RDSystem (reaction-diffusion)** — Keijiro Takahashi (2017): Let a chemistry-style simulation paint living textures. https://vimeo.com/217373413 · Source code: https://github.com/keijiro/RDSystem - **unity-teddy** — Masatatsu Nakamura (mattatz) (2016): A 2D doodle becomes a 3D object the moment the stroke closes. https://www.youtube.com/watch?v=ncvcNBKO6kk · Source code: https://github.com/mattatz/unity-teddy - **ManyCubes** — Keijiro Takahashi (2015): Get rich motion from one shader and a hash function. https://va.media.tumblr.com/tumblr_nwkrsraePx1qio469.mp4 · Source code: https://github.com/keijiro/ManyCubes - **Subatomic (geometric mirroring)** — Keijiro Takahashi (2015): Make a kaleidoscope that exists in 3D space. https://va.media.tumblr.com/tumblr_nv2ru4onKo1qio469.mp4 · Source code: https://github.com/keijiro/SpektrSubatomic - **TextAnimation** — Keijiro Takahashi (2015): Animate typography entirely in a shader. https://va.media.tumblr.com/tumblr_nxhz6iOsZv1qio469.mp4 · Source code: https://github.com/keijiro/TextAnimation - **Kvant Deformer** — Keijiro Takahashi (2014): Let noise sculpt a surface continuously on the GPU. https://vimeo.com/86002232 · Source code: https://github.com/keijiro/Kvant - **Sunburst effects** — Keijiro Takahashi (2013): Generate classic radial burst graphics as live meshes. https://vimeo.com/76949095 · Source code: https://github.com/keijiro/unity-sunburst-effects - **Undulation** — Keijiro Takahashi (2013): Contrast slow motion in the scene with a fast-moving camera. https://vimeo.com/80716975 · Source code: https://github.com/keijiro/unity-undulation - **Virtual Kinetic Sculpture** — Keijiro Takahashi (2013): Rebuild a physical kinetic sculpture as a virtual one. https://vimeo.com/80520888 · Source code: https://github.com/keijiro/unity-kinetic-sculpture ### Shaders & surfaces Dissolves, holograms, glitches and materials that change what a surface looks like. - **Fake Real Glass** — Youssef Afella (2026): Glass only has to look right from the camera, so fake the bending of light in screen space. https://x.com/Youssef_Afella/status/2020632755399213068 · Source code: https://github.com/Youssef-Afella/UnityURP-FakeRealGlass - **Galaxy Water** — Mirza Beig (2026): Water that reflects a universe instead of the sky. https://x.com/TheMirzaBeig/status/2008528798397198550 · Source code: https://github.com/MirzaBeig/Galaxy-Water - **LightGridShader: LED display look** — Keijiro Takahashi (2025): Imitate a physical display's structure to make screens feel real. https://x.com/_kzr/status/1962445754007765480 · Source code: https://github.com/keijiro/LightGridShader - **Radiance Cascades 2D GI** — Youssef Afella (2025): Compute light the way it really travels, but cheaply, by sharing rays between nearby points. https://x.com/Youssef_Afella/status/1896237865484636513 · Source code: https://github.com/Youssef-Afella/UnityURP-RadianceCascades2DGI - **Surface-Stable Fractal Dithering** — Rune Skovbo Johansen (2025): Dither that belongs to the surface, not the screen. https://www.youtube.com/watch?v=EzjWBmhO_1E · Source code: https://github.com/runevision/Dither3D - **Chromatic Distortion Sphere** — Mirza Beig (2024): A sphere that refracts the world with a rainbow edge instantly reads as magic. https://www.youtube.com/watch?v=IkBZLo4ROU0 · Source code: https://github.com/MirzaBeig/Chromatic-Distortion-Sphere - **FloatingHUD: floating interface effect** — Keijiro Takahashi (2024): Interface graphics as a visual effect in space. https://x.com/_kzr/status/1897620890919280893 · Source code: https://github.com/keijiro/FloatingHUD - **Shader Graph Material Examples for visionOS** — Yasuhito Nagatomo (2024): A small library of readable node graphs is the fastest way into shaders on a new platform. https://x.com/AtarayoSD/status/1791701412147253643 · Source code: https://github.com/ynagatomo/SGMExamples - **Uber Stylized Water** — MatrixRex (2024): Depth-based colour and foam make any shoreline read as water. https://www.youtube.com/watch?v=NtMFK1EBNDU · Source code: https://github.com/MatrixRex/Uber-Stylized-Water - **AIShader: ChatGPT shader generator** — Keijiro Takahashi (2023): Describe a surface in words and get a shader. https://x.com/_kzr/status/1632634562399600640 · Source code: https://github.com/keijiro/AIShader - **Shell Texturing Fur** — Acerola (Garrett Gunnell) (2023): Stack enough thin slices and a flat surface becomes fuzzy volume. https://www.youtube.com/watch?v=9dr-tRQzij4 · Source code: https://github.com/GarrettGunnell/Shell-Texturing - **Cel Shading (Devlog 3)** — Robin Seibold (2022): Quantise light into a few bands and a 3D model reads like hand-painted animation. https://www.youtube.com/watch?v=gw31oF9qITw · Source code: https://github.com/Robinseibold/Unity-URP-CelShadeLit - **Impossible Geometry with Stencil Shaders** — Daniel Ilett (2022): Space can be bigger on the inside if each window only shows its own world. https://www.youtube.com/watch?v=EzM8LGzMjmc · Source code: https://github.com/daniel-ilett/shaders-impossible-geom - **LEDScreenShader** — Tatsuro Ogata (llcheesell) (2022): Imitate the physical structure of a display, not just its image, to make a virtual screen believable. https://x.com/llcheesell/status/1492332637976936448 · Source code: https://github.com/llcheesell/LEDScreenShader - **Floor Pit (AR 100 Days, Day 45)** — Satoshi Hattori (2021): Cut the camera image into pieces and let reality itself fall apart. https://x.com/shmdevelop/status/1368503261515247617 · Source code: https://github.com/satoshi0212/AR_100Days - **Fully Functional Portals in URP** — Daniel Ilett (2021): Recreate the magic of Portal with render textures in a modern pipeline. https://www.youtube.com/watch?v=PkGjYig8avo · Source code: https://github.com/daniel-ilett/portals-urp - **Real World Splatoon (AR 100 Days, Day 39)** — Satoshi Hattori (2021): Turn the scanned room into a paintable game surface. https://x.com/shmdevelop/status/1361732615988932612 · Source code: https://github.com/satoshi0212/AR_100Days - **Stencil Portal Halloween Scene** — Shahriar Shahrabi (2021): A doorway can hold a whole world that exists only when you look through it. https://www.youtube.com/watch?v=gGeP34_6d2A · Source code: https://github.com/IRCSS/Unity-Stencil-Portal - **UnityFurURP (Shell / Fin Fur Shader)** — hecomi (2021): Stack many transparent copies of a surface and it turns into fur. https://www.youtube.com/watch?v=Hab3dcumtXU · Source code: https://github.com/hecomi/UnityFurURP - **AR Foundation Demos: AR Shaders** — Dan Miller (2020): Shaders that ground and reveal virtual content are what make AR look real, and they can be shared as a library. https://x.com/DanMillerDev/status/1294377908555800576 · Source code: https://github.com/Unity-Technologies/arfoundation-demos - **Coding Adventure: Portals** — Sebastian Lague (2020): A portal is a second camera whose image is pasted exactly where the doorway is. https://www.youtube.com/watch?v=cWpFZbjtSQg · Source code: https://github.com/SebLague/Portals - **Eyeball: procedural iris that follows you** — Keijiro Takahashi (2020): An object that looks back at you feels alive. https://x.com/_kzr/status/1321000166585856000 · Source code: https://github.com/keijiro/Eyeball - **Grass Geometry Shader (URP)** — Cyan (Cyanilux) (2020): Grow a whole meadow from a plane by letting the GPU emit the blades. https://x.com/Cyanilux/status/1255870774558232582 · Source code: https://github.com/Cyanilux/URP_GrassGeometryShader - **Matrix VFX** — Shahriar Shahrabi (2020): Any object can be rewritten as digital rain that follows its shape. https://www.youtube.com/watch?v=8l7cujPLw84 · Source code: https://github.com/IRCSS/MatrixVFX - **Unity URP Toon Lit Shader Example** — Colin Leung (NiloCat) (2020): Stylised lighting is a few deliberate rules on top of real lights, not a filter. https://www.youtube.com/watch?v=gcUCTLF5hwE · Source code: https://github.com/ColinLeung-NiloCat/UnityURPToonLitShaderExample - **Crystal Caustics** — CJT (CJT-Jackton) (2019): Bake the expensive physics once, then replay it cheaply as light. https://www.youtube.com/watch?v=-mrXdcObRNk · Source code: https://github.com/CJT-Jackton/Crystal-Caustics - **Texture Painting on Meshes** — Shahriar Shahrabi (2019): Painting on an object should feel like painting on the real thing, not on a screen. https://www.youtube.com/watch?v=GmCZZrV004A · Source code: https://github.com/IRCSS/TexturePaint - **Flipper (ADIRECTOR Channel)** — Keijiro Takahashi (2018): Make a digital show feel like flipping through a printed book. https://va.media.tumblr.com/tumblr_pd1oevl48u1qio469.mp4 · Source code: https://github.com/keijiro/Flipper - **PortalGate** — fuqunaga (2018): Two linked holes in walls rewrite how space connects. https://www.youtube.com/watch?v=jzud9m-NgnA · Source code: https://github.com/fuqunaga/PortalGate - **See-Through Objects Shader** — Przemyslaw Zaworski (2018): A mask decides where a solid object stops being solid. https://www.youtube.com/watch?v=1lw2hrVuPYk · Source code: https://github.com/przemyslawzaworski/Unity3D-CG-programming - **Shader Graph moving-lines globe** — Keijiro Takahashi (2018): Show that a node graph can make a striking animated material. https://va.media.tumblr.com/tumblr_pfv4dgueZM1qio469.mp4 · Source code: https://github.com/keijiro/ShaderGraphExamples - **Contour Map Shader** — World of Zero (Sam Wronski) (2017): Any surface can be read as a map by slicing it at regular heights. https://www.youtube.com/watch?v=AK8oV4BzrW4 · Source code: https://github.com/WorldOfZero/UnityVisualizations - **Energy Shield Shader** — World of Zero (Sam Wronski) (2017): Every impact leaves a ripple on an invisible barrier. https://www.youtube.com/watch?v=NeZcAYJdkv4 · Source code: https://github.com/WorldOfZero/UnityVisualizations - **GDisintegrator** — Keijiro Takahashi (2017): Treat disintegration as a controllable, reversible material state. https://vimeo.com/241520939 · Source code: https://github.com/keijiro/GDisintegrator - **GVoxelizer** — Keijiro Takahashi (2017): Turn any mesh into a voxel transition without preprocessing it. https://vimeo.com/241191777 · Source code: https://github.com/keijiro/GVoxelizer - **InkPainter with Google Tango** — EsProgram (2017): Graffiti on the real world by painting onto its scanned mesh. https://www.youtube.com/watch?v=dA-rf3lkRg0 · Source code: https://github.com/EsProgram/InkPainter - **Parallax Occlusion Mapping Hole** — Erik Nordeus (Habrador) (2017): A flat surface can fake depth by shifting its texture with the view angle. https://www.youtube.com/watch?v=nvok7temQuI · Source code: https://github.com/Habrador/Unity-Advanced-Shaders-Tutorial - **Prisma (VRDG+H #4)** — Keijiro Takahashi (2017): Place virtual objects between performers and audience by projecting onto a transparent screen. https://www.youtube.com/watch?v=oY6uCfNb-Ng · Source code: https://github.com/keijiro/Prisma - **Projection Spray (VR Spray Drawing)** — Hironori Sugino (sugi-cho) (2017): A spray can is just a projector that writes into the texture of whatever it hits. https://www.youtube.com/watch?v=TTv6YPWNLxY · Source code: https://github.com/sugi-cho/ProjectionSpray - **Sea of Clouds** — kaiware007 (2017): Fake a volumetric cloud sea cheaply with a displaced, soft-shaded surface. https://www.youtube.com/watch?v=i8HpYQ0hmio · Source code: https://github.com/kaiware007/UnitySeaOfClouds - **The Magic Revealing Flashlight Shader** — World of Zero (Sam Wronski) (2017): Light itself becomes the key that reveals hidden content. https://www.youtube.com/watch?v=b4utgRuIekk · Source code: https://github.com/WorldOfZero/UnityVisualizations - **Unity-ShaderSketches** — setchi (2017): A daily habit of tiny fragment-shader sketches builds a library of moving patterns. https://x.com/setchi/status/947253545111134209 · Source code: https://github.com/setchi/Unity-ShaderSketches - **Flipbook** — Keijiro Takahashi (2016): Turn time into pages of a physical book. https://va.media.tumblr.com/tumblr_pcorcg1yV01qio469.mp4 · Source code: https://github.com/keijiro/Flipbook - **InkPainter** — EsProgram (2016): Paint directly onto any mesh at runtime, like spraying ink on real objects. https://www.youtube.com/watch?v=TWGK5UQ6KsU · Source code: https://github.com/EsProgram/InkPainter - **Spektr Scatter (polygon scatter)** — Keijiro Takahashi (2015): Build a poly-dissolve that works on any model with a single slider. https://va.media.tumblr.com/tumblr_nvmqi6Tmuc1qio469.mp4 · Source code: https://github.com/keijiro/SpektrScatter - **Cut-out Fx** — Keijiro Takahashi (2014): A shader-only dissolve that can reveal or hide anything. https://vimeo.com/84364022 · Source code: https://github.com/keijiro/CutoutFxTest - **SlicerFx** — Keijiro Takahashi (2014): Make a 3D scene look like it is being scanned in slices. https://vimeo.com/102444263 · Source code: https://github.com/keijiro/SlicerFx - **SonarFx** — Keijiro Takahashi (2014): Reveal a space with an expanding pulse of light. https://vimeo.com/102398137 · Source code: https://github.com/keijiro/SonarFx - **Water Drops (pseudo refraction)** — Keijiro Takahashi (2014): Fake refraction cheaply enough to use it everywhere. https://vimeo.com/85640039 · Source code: https://github.com/keijiro/UnityRefractionShader - **Portal Room Cubes via Stencil Buffer** — NoiseCrime (2013): Several rooms can share the same space if each is only visible through its own face. https://www.youtube.com/watch?v=5DKIP9N-OB4 · Source code: https://github.com/noisecrime/Unity-StencilPortalRoomCube ### Raymarching & SDF Shapes defined by distance fields: smooth blends, infinite detail, impossible spaces. - **SdfManipulator** — Tooster (T3sT3ro) (2024): Model with distance fields the way you would with clay, without writing shader code. https://github.com/user-attachments/assets/f52581a2-1cf7-49b1-a8d6-941ec3a6ccc7 · Source code: https://github.com/T3sT3ro/SdfManipulator - **Spark2D** — Singtaa (2024): Treat every shape as a distance function, and outlines, glows and blends come for free. https://x.com/Singtaa/status/1837274411667444140 · Source code: https://github.com/Singtaa/Spark2D - **Death Star Ray Marching Shader** — Erik Nordeus (Habrador) (2022): Two spheres and a subtraction make an iconic shape without a single polygon. https://www.youtube.com/watch?v=_MgfKdxa6lE · Source code: https://github.com/Habrador/Unity-Advanced-Shaders-Tutorial - **Mesh-to-SDF** — Unity Visual Effect Graph team (Thomas Iché, Julien Fryer, Vlad Neykov and the Unity Demo Team) (2022): Give a moving body a distance field every frame and effects can feel its shape. https://user-images.githubusercontent.com/6276154/201238758-cc87ec4a-a65a-4cb2-b3cc-123576ab9ea2.mov · Source code: https://github.com/Unity-Technologies/com.unity.demoteam.mesh-to-sdf - **Raymarching with ShadowCaster** — SCRN (2022): Make distance-field shapes behave like real objects by marching them in the shadow pass too. https://x.com/SCRNinVR/status/1536860644758245379 · Source code: https://github.com/SCRN-VRC/Raymarching-with-ShadowCaster - **mesh-to-sdf VFX Examples** — Rob Cupisz (2022): Rebuild the character's shape as a field every frame, and any number of effects can hug, avoid or fill it. https://x.com/robcupisz/status/1591063441225437187 · Source code: https://github.com/robcupisz/mesh-to-sdf-examples - **ComputeMarchingCubes: GPU isosurfaces** — Keijiro Takahashi (2021): Turn any 3D field into a solid surface in real time. https://x.com/_kzr/status/1403359710577786881 · Source code: https://github.com/keijiro/ComputeMarchingCubes - **Shaderworlds** — Jure Triglav (2021): A fragment shader is already a world; WebXR just lets you stand inside it. https://www.youtube.com/watch?v=XoRwpPPUAbc · Source code: https://github.com/jure/shaderworlds - **4D Explorer (4D Raymarching)** — Jelle Vermandere (2020): See a 4D world by walking through its 3D slices. https://www.youtube.com/watch?v=nUExziADzjc · Source code: https://github.com/Jellevermandere/4D-Raymarching - **Coding Adventure: Clouds** — Sebastian Lague (2019): Clouds are noise that light travels through. https://www.youtube.com/watch?v=4QOcCGI6xOU · Source code: https://github.com/SebLague/Clouds - **Coding Adventure: Ray Marching** — Sebastian Lague (2019): Describe shapes as distance formulas and you get smooth blends and infinite fractals for free. https://www.youtube.com/watch?v=Cp5WWtMoeKg · Source code: https://github.com/SebLague/Ray-Marching - **MeshToSDF** — Aman Tiwari (2019): If a moving body becomes a distance field, particles can feel its surface. https://x.com/aman_gif/status/1100187796302635008 · Source code: https://github.com/aman-tiwari/MeshToSDF - **Metaballs** — Yuma Yanagisawa (2019): Blobs that merge smoothly feel alive. https://www.youtube.com/watch?v=kuLUqNNlN4g · Source code: https://github.com/yumayanagisawa/Unity-Metaballs - **Raymarching in Unity** — Shahriar Shahrabi (2019): Mix raymarched shapes with ordinary geometry so both live in one world. https://www.youtube.com/watch?v=87YvrkrymG0 · Source code: https://github.com/IRCSS/UnityRaymarching - **SdfVfxSamples: particles shaped by distance fields** — Keijiro Takahashi (2019): An invisible shape can be revealed by what flows around it. https://x.com/_kzr/status/1821182526184100126 · Source code: https://github.com/keijiro/SdfVfxSamples - **Realtime Volumetric Clouds in Unity** — Jaagup Kuhi (2018): Clouds are noise sampled along a ray, lit by how much sun reaches each step. https://www.youtube.com/watch?v=zOyHy969Y6I · Source code: https://github.com/jaagupku/volumetric-clouds - **unity-volume-rendering** — Masatatsu Nakamura (mattatz) (2017): Raymarch through a 3D texture so volumetric data becomes a glowing object you can slice. https://x.com/mattatz/status/935704854046969856 · Source code: https://github.com/mattatz/unity-volume-rendering - **CloudSkybox** — Keijiro Takahashi (2016): Put real volumetric clouds into the sky without any geometry. https://www.youtube.com/watch?v=_QC6dXTMMwE · Source code: https://github.com/keijiro/CloudSkybox - **Volumetric Lights for Unity** — Michal Skalsky (2016): Make the air visible by marching rays through the light. https://www.youtube.com/watch?v=JPxLCYXB-8A · Source code: https://github.com/SlightlyMad/VolumetricLights - **uRaymarching** — hecomi (2016): Write only a distance function and get a full, lit shader that behaves like any other object. https://www.youtube.com/watch?v=AppyVflAagc · Source code: https://github.com/hecomi/uRaymarching - **Water Surface (isosurface)** — Keijiro Takahashi (2013): Build liquid shapes from invisible fields instead of modelled meshes. https://vimeo.com/82601169 · Source code: https://github.com/keijiro/unity-isosurface-test ### Fluids & physics Simulated fluid, smoke, cloth and soft bodies. - **Poseidon** — Owen (owenyuwono) (2026): A film-grade ocean technique, now a web page. https://x.com/owenyuwono/status/2064260229207142583 · Source code: https://github.com/owenyuwono/poseidon - **Tiamat** — Owen (owenyuwono) (2026): Tilt the phone and the water inside it moves like water in a glass. https://x.com/owenyuwono/status/2063441468690411593 · Source code: https://github.com/owenyuwono/tiamat - **Fluo: fluid and spectral color visualizer** — Keijiro Takahashi (2025): The camera image becomes the dye in a fluid. https://x.com/_kzr/status/1962518821111280037 · Source code: https://github.com/keijiro/Fluo - **Pigment-based color mixing: fluid art** — Keijiro Takahashi (2025): Mix colors the way paint does, not the way light does. https://x.com/_kzr/status/1952365070601855069 · Source code: https://github.com/keijiro/PigmentTest - **TrackpadFluid: ten-finger fluid** — Keijiro Takahashi (2025): A trackpad becomes a multi-finger paint pool. https://x.com/_kzr/status/1956332698592805220 · Source code: https://github.com/keijiro/TrackpadFluid - **Coding Adventure: Simulating Fluids** — Sebastian Lague (2023): Water is just many particles that try to keep a constant density. https://www.youtube.com/watch?v=rSKMYc1CQHE · Source code: https://github.com/SebLague/Fluid-Sim - **Counter-Strike 2 Smoke Grenades** — Acerola (Garrett Gunnell) (2023): Smoke that knows the room: it floods the free space and can be pushed away. https://www.youtube.com/watch?v=ryB8hT5TMSg · Source code: https://github.com/GarrettGunnell/CS2-Smoke-Grenades - **Eulerian Fluid Simulator** — Erik Nordeus (Habrador) (2023): A fluid is just a grid of velocities kept divergence-free. https://www.youtube.com/watch?v=6Jw1CsTOkDg · Source code: https://github.com/Habrador/Ten-Minute-Physics-Unity - **GPU Fluid 3D** — Ming Wai Chan (cinight) (2023): A GPU fluid needs only a few kernels: advect, project, repeat. https://www.youtube.com/watch?v=1Yi4Wek994M · Source code: https://github.com/cinight/MinimalCompute - **Simulating the Entire Ocean** — Acerola (Garrett Gunnell) (2023): Real oceans are sums of thousands of waves, and the GPU can add them up every frame. https://www.youtube.com/watch?v=yPfagLeUa7k · Source code: https://github.com/GarrettGunnell/Water - **VFX Graph Smoke Portal Sample** — Unity Visual Effect Graph team (Thomas Iché, Julien Fryer, Vlad Neykov and the Unity Demo Team) (2023): Smoke that is lit like real smoke makes a portal feel physical. https://www.youtube.com/watch?v=57cKxN3XdEY · Source code: https://github.com/Unity-Technologies/VisualEffectGraph-Samples - **Volumetric fog from VFX Graph** — Keijiro Takahashi (2023): Particles can write into the fog instead of being sprites. https://x.com/_kzr/status/1615973816286744578 · Source code: https://github.com/keijiro/VolumetricVfxTest - **VAT Utils for VFX Graph** — Alexandre Rivaux (Bonjour Lab) (2022): Simulate offline, replay anywhere: baked physics becomes a real-time particle output. https://www.youtube.com/watch?v=Ms41cq63dQw · Source code: https://github.com/Bonjour-Interactive-Lab/Unity3D-VATUtils - **3D Volumetric Fire** — Tasuku Takahashi (2021): Simulate fire as a real 3D gas instead of faking it with billboards. https://x.com/supertask_jp/status/1358792459543662592 · Source code: https://github.com/supertask/UnityVolumetricFire3D - **Coriolis** — Benoit Arbelot (2021): The same growth rule reads as cells, rivers or galaxies depending on the scale. https://vimeo.com/543236276 · Source code: https://github.com/Barbelot/Physarum3D - **Interactive Volumetric Fog with Fluid Dynamics (The Vast Land)** — Shahriar Shahrabi (2021): Fog becomes something you can push: a fluid simulation drives a raymarched fog volume. https://www.youtube.com/watch?v=hMhNhgnOGN8 · Source code: https://github.com/IRCSS/Compute-Shaders-Fluid-Dynamic- - **MLS-MPM Fluid in Unity** — Tasuku Takahashi (2021): Use the same particle-grid method as film VFX to get believable liquid at interactive speed. https://x.com/supertask_jp/status/1416315883832561673 · Source code: https://github.com/supertask/Unity-MLS-MPM-Fluid-Test - **WaterRW** — Noboru Seto (ruccho) (2021): Simulate only the surface of the water, and it can be both cheap and interactive. https://x.com/ruccho_vector/status/1461970803428405248 · Source code: https://github.com/ruccho/WaterRW - **BurstWig: flowing hair-like strands** — Keijiro Takahashi (2020): Strands with simple spring physics feel organic. https://x.com/_kzr/status/1258039132372135941 · Source code: https://github.com/keijiro/BurstWig - **Chimera: Particle Advection** — Benoit Arbelot (2020): Let a fluid, not a noise function, decide where particles go. https://x.com/BenoitArbelot/status/1297605129114390534 · Source code: https://github.com/Barbelot/Chimera - **Dynamic hair and petals (2D-Unity-Experiments)** — Angela He (zephyo) (2020): Give flat 2D art physical secondary motion with a chain of joints. https://x.com/zephybite/status/1305539495912144897 · Source code: https://github.com/zephyo/2D-Unity-Experiments - **Fluid Simulation in Compute Shaders** — Shahriar Shahrabi (2020): A full fluid solver fits in a handful of compute-shader passes and can paint any surface with moving ink. https://www.youtube.com/watch?v=GkrQy5JUyZk · Source code: https://github.com/IRCSS/Compute-Shaders-Fluid-Dynamic- - **HdrpVatExample: baked fluid and cloth with VAT** — Keijiro Takahashi (2020): Simulate once offline, play it back anywhere. https://x.com/_kzr/status/1220338439117127680 · Source code: https://github.com/keijiro/HdrpVatExample - **IMMATERIA: My First Gooey Mesh** — Isaac Cohen (Cabbibo) (2020): Make any mesh gooey by simulating its vertices as springy particles on the GPU. https://www.youtube.com/watch?v=H_08R0Mzq54 · Source code: https://github.com/cabbibo/IMMATERIA - **Lights of Nibel** — Benoit Arbelot (2020): Light as a liquid that creatures stir as they pass. https://vimeo.com/458749435 · Source code: https://github.com/Barbelot/Chimera - **StableFluids** — Keijiro Takahashi (2018): Turn a touch surface into a hands-on liquid painting instrument. https://vimeo.com/277872734 · Source code: https://github.com/keijiro/StableFluids - **WaveShaderDemo (water surface shader)** — EsProgram (2018): A minimal, readable wave simulation that makes any plane react like water. https://www.youtube.com/watch?v=HBlMCAyFIkA · Source code: https://github.com/EsProgram/WaveShaderDemo - **WaveformProvider** — EsProgram (2018): Store the water surface as a texture and let a wave equation run on it. https://www.youtube.com/watch?v=3RgRCOm0NAQ · Source code: https://github.com/EsProgram/WaveformProvider - **GPU Accelerated 2D Wave Simulation** — Kodai Takao (2017): A few lines of physics on the GPU turn a surface into a living membrane. https://www.youtube.com/watch?v=XqwTrwWG-_4 · Source code: https://github.com/kodai100/Unity_Waves - **Position Based Fluid** — Kodai Takao (2017): Fluids stay stable if you correct positions instead of forces. https://www.youtube.com/watch?v=N-XVl2Jip3M · Source code: https://github.com/kodai100/Unity_PositionBasedFluid - **ShaderToy to Unity: Fluid Dynamics** — Przemyslaw Zaworski (2017): A self-feeding texture is enough to make convincing, never-repeating fluid. https://www.youtube.com/watch?v=8mBqEpj_3tk · Source code: https://github.com/przemyslawzaworski/Unity3D-CG-programming - **Unity FLIP Fluid** — Kodai Takao (2017): Combine particles and grids to get detailed yet stable water. https://www.youtube.com/watch?v=JRTQ6Kgi_Wk · Source code: https://github.com/kodai100/Unity_FLIPFluid - **Water Surface Simulation with CustomRenderTexture** — hecomi (2017): A ripple simulation fits in a texture that updates itself every frame. https://www.youtube.com/watch?v=jclxfdS3a3w · Source code: https://github.com/hecomi/UnityWaterSurface - **Wave Propagation Shader (Water Ripple)** — Yuma Yanagisawa (2017): Turn any image into a pond. https://www.youtube.com/watch?v=rK5AAb-1pgE · Source code: https://github.com/yumayanagisawa/Unity-Wave-Propagation-Water-Ripple - **Tornado Simulator** — Erik Nordeus (Habrador) (2016): A tornado is a few forces: pull in, lift up, spin around. https://www.youtube.com/watch?v=IlhdLQ5NU5E · Source code: https://github.com/Habrador/Unity-Tornado-Simulator - **Mesh-Based Watercolor Paint** — Nakata Nobuyuki (nobnak) (2014): Simulate watercolour as growing, deforming polygons rather than pixels. https://vimeo.com/114418780 · Source code: https://github.com/nobnak/PaintingWithPolygons - **Real-time Water Caustics Simulation** — Nakata Nobuyuki (nobnak) (2014): Compute where light lands after passing through a moving water surface. https://vimeo.com/110753770 · Source code: https://github.com/nobnak/CausticsUnity ### Camera & screen effects Post-processing on the camera image: datamosh, slit-scan, feedback, stylisation. - **FlashGlitch: trigger-based glitch** — Keijiro Takahashi (2026): A glitch that hits like a drum. https://x.com/_kzr/status/2030279235081060715 · Source code: https://github.com/keijiro/FlashGlitch - **Karbon: live camera visuals with Launchpad** — Keijiro Takahashi (2026): Play camera effects like a drum machine. https://x.com/_kzr/status/2034570627886239876 · Source code: https://github.com/keijiro/Karbon - **KinoGlitch URP: analog and digital glitch** — Keijiro Takahashi (2026): Broken signals as an expressive style. https://x.com/_kzr/status/2025558651768070209 · Source code: https://github.com/keijiro/KinoGlitchURP - **Light leak effect** — Keijiro Takahashi (2026): Borrow a film-camera accident as a mood. https://x.com/_kzr/status/2028469137132114330 · Source code: https://github.com/keijiro/LightLeakEffectExample - **Pencil Effect** — Przemyslaw Zaworski (2026): Turn any live image into a moving sketchbook page. https://www.youtube.com/watch?v=Km6WS_T9ER0 · Source code: https://github.com/przemyslawzaworski/Unity3D-CG-programming - **StrobePages: page-turning post effect** — Keijiro Takahashi (2026): Turn smooth motion into a stack of pages. https://x.com/_kzr/status/2021954733439848713 · Source code: https://github.com/keijiro/StrobePages - **MiniBokeh: lightweight depth of field** — Keijiro Takahashi (2025): A cheap bokeh makes a scene feel photographed. https://x.com/_kzr/status/1959598355149180928 · Source code: https://github.com/keijiro/MiniBokeh - **Passthrough Camera Shaders (QuestCameraKit)** — Thomas Ratliff, Roberto Coviello (2025): Once the headset camera is readable, the real world can be filtered like a photo, in place. https://x.com/devtom7/status/1902033672041091453 · Source code: https://github.com/xrdevrob/QuestCameraKit - **Realtime optical flow glitch machine** — Keijiro Takahashi (2025): Use motion itself to push pixels around. https://x.com/_kzr/status/1886351525569843694 · Source code: https://github.com/keijiro/OpticalFlowTest - **Supermosh** — Nino Filiu (2025): Datamosh without codecs or command lines, right in the browser. https://www.youtube.com/watch?v=M1OCjF-aJyo · Source code: https://github.com/supermosh/supermosh.github.io - **VolFx** — Andrei Iurin (NullTale) (2024): Post-processing does not have to apply to the whole screen; it can be a brush for chosen layers. https://www.youtube.com/watch?v=0Byz2CEw-y8 · Source code: https://github.com/NullTale/VolFx - **Duotone image effect** — Keijiro Takahashi (2023): Reduce the palette to make an image graphic. https://x.com/_kzr/status/1789384955136733476 · Source code: https://github.com/keijiro/Duotone - **Gamma: live coding at GitHub Universe Recap** — Keijiro Takahashi (2023): Writing code in front of an audience is the performance. https://www.youtube.com/watch?v=gA9beOCv8s0 · Source code: https://github.com/keijiro/Gamma - **GiLight2D: Bad Apple!! in 2D Ray Tracing** — Andrei Iurin (NullTale) (2023): Treat every bright pixel as a light and let rays find the shadows. https://www.youtube.com/watch?v=fNq0HUg6L8o · Source code: https://github.com/NullTale/GiLight2D - **Pixel Sorting** — Acerola (Garrett Gunnell) (2023): A classic still-image glitch, made fast enough to run on live video. https://www.youtube.com/watch?v=HMmmBDRy-jE · Source code: https://github.com/GarrettGunnell/Pixel-Sorting - **Anime Speed Lines** — Mirza Beig (2022): Borrow the manga shorthand for speed and surprise as a live camera effect. https://user-images.githubusercontent.com/37354140/151656459-d99d2d36-06ed-4889-8a10-17cabd1626d8.mp4 · Source code: https://github.com/MirzaBeig/Anime-Speed-Lines - **Color Quantization and Dithering** — Acerola (Garrett Gunnell) (2022): Fewer colours plus a clever pattern read as a strong style instead of a loss. https://www.youtube.com/watch?v=8wOUe32Pt-E · Source code: https://github.com/GarrettGunnell/Post-Processing - **Kuwahara Filter** — Acerola (Garrett Gunnell) (2022): Pick the calmest neighbourhood around every pixel and the image paints itself. https://www.youtube.com/watch?v=LDhN-JK3U9g · Source code: https://github.com/GarrettGunnell/Post-Processing - **Post-Processing Scan** — Mirza Beig (2022): A single expanding sphere reveals the hidden shape of the world. https://x.com/TheMirzaBeig/status/1809173668456792524 · Source code: https://github.com/MirzaBeig/Post-Processing-Scan - **Speed lines post effect** — Keijiro Takahashi (2022): A comic convention turned into a screen effect. https://x.com/_kzr/status/1506969479158243331 · Source code: https://github.com/keijiro/SimplePostEffects - **Fake Stop Motion** — Eric Freeman (2021): Change the time signature of motion, not the look of pixels, to get a hand-made feel. https://x.com/_ericfreeman/status/1369299130678382593 · Source code: https://github.com/EricFreeman/FakeStopMotion - **Flipbook2: flip book with watercolor** — Keijiro Takahashi (2021): Show video as a physical object: a stack of pages. https://x.com/_kzr/status/1398540851941314568 · Source code: https://github.com/keijiro/Flipbook2 - **Fogged Line World (AR 100 Days, Day 35)** — Satoshi Hattori (2021): Use depth as a style switch: reality near, drawing far. https://x.com/shmdevelop/status/1359887965984464897 · Source code: https://github.com/satoshi0212/AR_100Days - **KinoAqua: watercolor effect with VFX Graph** — Keijiro Takahashi (2021): A painterly filter makes computer graphics feel handmade. https://x.com/_kzr/status/1392482946393677829 · Source code: https://github.com/keijiro/KinoAqua - **Pink Smog (AR 100 Days, Day 37)** — Satoshi Hattori (2021): Real depth makes fake weather believable: the fog thickens exactly where the world gets farther away. https://x.com/shmdevelop/status/1361336054523961349 · Source code: https://github.com/satoshi0212/AR_100Days - **Screen-Space Outlines** — Robin Seibold (2021): Edges found in depth and normals turn any render into an illustration. https://www.youtube.com/watch?v=LMqio9NsqmM · Source code: https://github.com/Robinseibold/Unity-URP-Outlines - **Space Distortion (AR 100 Days, Day 2)** — Satoshi Hattori (2021): Bend the camera image locally so reality looks like it has a lens in it. https://x.com/shmdevelop/status/1345363102141845504 · Source code: https://github.com/satoshi0212/AR_100Days - **URP Screen Space Cavity** — MalyaWka (Pavel) (2021): Emphasise geometry by drawing its curvature, the way an illustrator inks edges. https://x.com/_malyawka_/status/1471816770604441603 · Source code: https://github.com/malyawka/URP-ScreenSpaceCavity - **Anime-inspired scene (2D-Unity-Experiments)** — Angela He (zephyo) (2020): Composite flat illustration layers with shaders until they feel like a living anime background. https://x.com/zephybite/status/1310626494935257088 · Source code: https://github.com/zephyo/2D-Unity-Experiments - **Cubism Shader** — Shahriar Shahrabi (2020): Show several viewpoints at once, as Picasso did, but live. https://www.youtube.com/watch?v=_DwnvbPxZTM · Source code: https://github.com/IRCSS/Cubism-Shader - **KinoFeedback2: frame feedback with emoji particles** — Keijiro Takahashi (2020): Feed the last frame back into the next one. https://x.com/_kzr/status/1306955390513364992 · Source code: https://github.com/keijiro/KinoFeedback2 - **MonoFxSketches: monochrome screen effects** — Keijiro Takahashi (2020): Strong graphic effects work best with only two colors. https://x.com/_kzr/status/1333749477732098048 · Source code: https://github.com/keijiro/MonoFxSketches - **Rain shader (2D-Unity-Experiments)** — Angela He (zephyo) (2020): Make the screen itself feel like wet glass between viewer and world. https://x.com/zephybite/status/1308833244712927236 · Source code: https://github.com/zephyo/2D-Unity-Experiments - **Retro CRT Shader** — Cyan (Cyanilux) (2020): Turn a clean render into a warm, flickering memory of old screens. https://x.com/Cyanilux/status/1301866507669057536 · Source code: https://github.com/Cyanilux/URP_RetroCRTShader - **SlitScanCam: realtime slit-scan camera** — Keijiro Takahashi (2020): Show time along one axis of the image. https://vimeo.com/494895371 · Source code: https://github.com/keijiro/SlitScanCam - **Sparkle Reflection Effect** — Ming Wai Chan (cinight) (2020): Let the image itself decide where the sparkles go. https://www.youtube.com/watch?v=Yh8PZ77jdSc · Source code: https://github.com/cinight/MinimalCompute - **URP-PSX** — Codrin Mihail (Kodrin) (2020): Deliberately degrade rendering to evoke a specific era of graphics. https://x.com/CodrinMihail/status/1254786582021697537 · Source code: https://github.com/Kodrin/URP-PSX - **Webcam Effect with Shader Graph** — Andy Duboc (2020): The camera feed is just another texture you can shade. https://x.com/andyduboc/status/1279416830478684160 · Source code: https://github.com/andydbc/WebcamEffect - **AR Foundation Post Processing** — Dan Miller (2019): Make virtual objects sit in the camera image by giving them the same lens as the real camera. https://x.com/DanMillerDev/status/1180163735668707328 · Source code: https://github.com/DanMillerDev/ARFoundation_PostProcessing - **KinoEight: 8-bit style post effect** — Keijiro Takahashi (2019): Old hardware limits become a visual style. https://x.com/_kzr/status/1208045085452955653 · Source code: https://github.com/keijiro/KinoEight - **ZIGSIM-Parallax (PC as a portal)** — AMAGI (Takayosi Amagi) (2019): Track the viewer's head with a phone and render off-axis perspective so a flat screen feels like a hole in the wall. https://x.com/amagitakayosi/status/1197225844676026368 · Source code: https://github.com/fand/ZIGSIM-Parallax - **Night Vision Camera Effect** — World of Zero (Sam Wronski) (2017): Depth becomes light: what is close to the camera is what you can see. https://www.youtube.com/watch?v=mElPESXcakM · Source code: https://github.com/WorldOfZero/UnityVisualizations - **Pepper's Cone** — Xuan Luo (2017): A cheap cone plus a pre-warped image makes a walk-around hologram. https://www.youtube.com/watch?v=dppCXj11I_Q · Source code: https://github.com/roxanneluo/Pepper-s-Cone-Unity - **Raindrops Shader** — Yuma Yanagisawa (2017): A rainy window can be drawn entirely in one shader. https://www.youtube.com/watch?v=dQSLjsDAzw0 · Source code: https://github.com/yumayanagisawa/Unity-Raindrops - **VideoPlayerEffects (keying)** — Keijiro Takahashi (2017): Place filmed people into a real-time 3D world with a shader. https://va.media.tumblr.com/tumblr_om9qfzPwgn1qio469_480.mp4 · Source code: https://github.com/keijiro/VideoPlayerEffects - **Fluid2D Blur Image Effect** — Hiroaki Oishi (irishoak) (2016): Use a fluid simulation as a brush that distorts the screen image. https://vimeo.com/150463085 · Source code: https://github.com/hiroakioishi/UnityFluid2DBlurImageEffect - **Phantom (Phantom Sketch Mod.)** — Keijiro Takahashi (2016): Push a game engine's post-processing stack to cinematic quality for a live show. https://x.com/tokyomax/status/787249881899347976 · Source code: https://github.com/keijiro/Phantom - **Spectrum (Liquidroom 2016)** — Keijiro Takahashi (2016): Combine a whole library of screen effects into one playable party instrument. https://va.media.tumblr.com/tumblr_oj1b8wYz3O1qio469_720.mp4 · Source code: https://github.com/keijiro/Spectrum - **Star glow shader (Kawase Light Streak)** — Nakata Nobuyuki (nobnak) (2016): Recreate a photographic star filter in real time. https://vimeo.com/173164170 · Source code: https://github.com/nobnak/KawaseLightStreakUnity - **WaterColor Filter** — Nakata Nobuyuki (nobnak) (2016): Turn any real-time scene into a moving watercolour illustration. https://vimeo.com/152248671 · Source code: https://github.com/nobnak/WaterColorFilter - **KinoIsoline** — Keijiro Takahashi (2015): See a 3D scene as moving contour lines. https://va.media.tumblr.com/tumblr_nva7mmMPqD1qio469.mp4 · Source code: https://github.com/keijiro/KinoIsoline - **PhotoMosaic** — Keijiro Takahashi (2015): Render a scene out of a photo album. https://va.media.tumblr.com/tumblr_nwf56pAxZd1qio469.mp4 · Source code: https://github.com/keijiro/PhotoMosaic - **Smoke Blur Effect** — Nakata Nobuyuki (nobnak) (2015): Blur an image along a flow field so it looks like it is turning into smoke. https://vimeo.com/148481861 · Source code: https://github.com/nobnak/SmokeBlur - **Depthcue** — Keijiro Takahashi (2014): Use depth as a stylistic color ramp. https://vimeo.com/101211958 · Source code: https://github.com/keijiro/Depthcue - **GlitchFx** — Keijiro Takahashi (2014): Use controlled digital corruption as an aesthetic. https://vimeo.com/102398104 · Source code: https://github.com/keijiro/GlitchFx - **HexBokeh** — Keijiro Takahashi (2014): Bring the character of a physical lens aperture into real-time graphics. https://vimeo.com/103702704 · Source code: https://github.com/keijiro/HexBokeh - **SketchyFx** — Keijiro Takahashi (2014): Stack standard image effects to get a hand-drawn look. https://vimeo.com/97597887 · Source code: https://github.com/keijiro/SketchyFx ### AI-driven effects Neural networks running in the engine: segmentation, style transfer, generative models. - **Robust Video Matting on Mac** — Keijiro Takahashi (2026): Clean mattes make any person a compositing layer. https://x.com/_kzr/status/2101943586606678108 · Source code: https://github.com/keijiro/unity-rvm-coreml - **Myaku Myaku AR** — Koki Ibukuro (asus4) (2025): Segment real objects and re-skin them as a character, so the world itself becomes the costume. https://github.com/user-attachments/assets/03961874-421f-41bc-a1d8-f9e0e07b4fe4 · Source code: https://github.com/asus4/MyakuMyakuAR - **Dcam2: Stable Diffusion VJ set with DUB-Russell** — Keijiro Takahashi (2024): Mix generative AI with body tracking for live visuals. https://www.youtube.com/watch?v=qa4jv5JhKhM · Source code: https://github.com/keijiro/Dcam2 - **Dcam: realtime Stable Diffusion in live performance** — Keijiro Takahashi (2023): Run image generation fast enough to become a live visual. https://www.youtube.com/watch?v=iVi-7oz67OU · Source code: https://github.com/keijiro/Dcam - **Genesis: 3D Worlds from AI Skyboxes** — Julien Kipp (julienkay) (2023): Give a flat AI panorama depth and it becomes a stage set you can step into. https://x.com/julien_kaye/status/1626694253437939712 · Source code: https://github.com/julienkay/com.doji.genesis - **SNeRG in Unity** — Julien Kipp (julienkay) (2023): Raymarch a neural capture like a volume texture and it becomes a place you can stand in. https://x.com/julien_kaye/status/1632880010942992386 · Source code: https://github.com/julienkay/SNeRG-Unity-Viewer - **MoCap VFX** — Adrián Ciborro Montes (2022): No mocap suit: a neural network turns a webcam into a particle body. https://www.youtube.com/watch?v=z2Kst0t0PBA · Source code: https://github.com/adcimon/mocap-vfx - **MobileNeRF in Unity** — Julien Kipp (julienkay) (2022): A neural capture becomes a normal game asset when the network is baked into a shader. https://x.com/julien_kaye/status/1556257212829417474 · Source code: https://github.com/julienkay/MobileNeRF-Unity-Viewer - **M-LSD line detection as VFX** — Keijiro Takahashi (2021): The architecture of a room becomes the drawing. https://x.com/_kzr/status/1413426397054332930 · Source code: https://github.com/keijiro/MlsdBarracuda - **SelfieBarracuda: virtual background on phones** — Keijiro Takahashi (2021): Segmentation separates the person from the world, and each can get its own effect. https://x.com/_kzr/status/1405518336230793223 · Source code: https://github.com/keijiro/SelfieBarracuda - **TinyYOLOv2 object detection in Unity** — Keijiro Takahashi (2021): Let the engine know what it is looking at. https://x.com/_kzr/status/1353349183252533249 · Source code: https://github.com/keijiro/TinyYOLOv2Barracuda - **YOLOv4-tiny in Fragment Shaders** — SCRN (2021): Run a neural network where scripts are not allowed: inside the shader itself. https://x.com/SCRNinVR/status/1380238589238206465 · Source code: https://github.com/SCRN-VRC/YOLOv4-Tiny-in-UnityCG-HLSL - **MediaPipe FaceMesh on Unity** — Koki Ibukuro (asus4) (2020): A face filter engine you own: run the face model yourself instead of relying on a platform's lens tools. https://www.youtube.com/watch?v=ctFZjSzx6Vc · Source code: https://github.com/asus4/tf-lite-unity-sample - **Pix2Pix in a Fragment Shader** — SCRN (2020): A GAN can be just another material on a surface. https://x.com/SCRNinVR/status/1317299768301735936 · Source code: https://github.com/SCRN-VRC/Pix2Pix-in-a-Fragment-Shader - **Ngx** — Keijiro Takahashi (2018): A neural network that hallucinates an infinite music video from a short clip. https://vimeo.com/294399440 · Source code: https://github.com/keijiro/Ngx - **Pix2Pix for Unity** — Keijiro Takahashi (2018): Use real-time image-to-image translation as a drawing tool. https://vimeo.com/287778343 · Source code: https://github.com/keijiro/Pix2Pix ## AI × AR AR where AI is central to the idea: models that understand the scene, agents in space, generated worlds, reality restyled live. ### AI that sees Vision and language models that recognise, read and explain what the camera sees. - **AI Building Blocks: object detection in mixed reality** — Dilmer Valecillos (2026): Drop-in blocks make the headset name what it sees. https://www.youtube.com/watch?v=QIa6frPMcSQ - **Carveout: object understanding inside Gaussian splats** — Hugues Bruyère (2026): A scan of a real place becomes a box of named, movable objects. https://x.com/smallfly/status/2076061670322094401 - **IronSight: 4D replays from Ray-Ban Meta clips** — Bilawal Sidhu (2026): First-person clips become replayable spaces you can re-enter from any angle. https://x.com/bilawalsidhu/status/2074536788853665831 - **Live Scene Analyzer: describing actions and checking tasks** — Takashi Yoshinaga (2026): Your glasses watch what your hands do and keep your checklist for you. https://www.youtube.com/watch?v=O7LS4NdFOfQ - **Meta Ray-Ban AI City Tour Guide** — Stijn Spanhove (2026): An AI city guide in Ray-Ban Meta glasses describes whatever you are looking at as you walk. https://www.youtube.com/watch?v=IA0LGSA4Lgw - **Navig-AI-tion** — Mar Gonzalez-Franco (2026): Navigation you can hear, grounded in what is actually around you. https://www.youtube.com/watch?v=CZt26nSjfao - **On-device vision AI: scene understanding and prompt-based OCR** — Takashi Yoshinaga (2026): Say what to read instead of framing it inside a scan box. https://www.youtube.com/watch?v=gVoTzhzCqSQ - **One-click custom object detector for Spectacles** — Stijn Spanhove (2026): Teaching glasses to recognise your own objects should take a click, not a weekend. https://x.com/stspanho/status/2066207120199208985 - **Ray-Ban Meta product scanner** — Juampi (2026): Look at any product and hear what it is and where it comes from. https://x.com/juampitech/status/2068387708276527265 - **Spatial Search for Meta Quest** — Takashi Yoshinaga (2026): Search your room the way you search the web. https://www.youtube.com/watch?v=CpNNvYJ_eqo - **Standing inside your RAG embeddings** — Stijn Spanhove (2026): Walk around the vector space that powers a chatbot, instead of reading its answer. https://x.com/stspanho/status/2024877056035622974 - **AR pothole detection, grading and geotagging** — Virendra Suryawanshi (2025): Inspect a street by looking at it; the headset keeps the report. https://x.com/Virendra0698/status/2001634595494855158 - **Ad Block** — Stijn Spanhove (2025): A real-world ad blocker that finds the billboards in front of you and covers them up. https://www.youtube.com/watch?v=KLiEm74cw9Q - **Android XR glasses live demo (TED)** — Microsoft Research Cambridge / I3D — Shahram Izadi (2025): Walk on stage wearing AI glasses that remember where you just put your keys. https://www.youtube.com/watch?v=gElClXpg4J0 - **Fruit Defense** — Stijn Spanhove, Pavlo Tkachenko (2025): The glasses recognize real objects around you and make them part of a tower-defense game. https://www.youtube.com/watch?v=KlC__Y0J-Wc - **Guided Reality** — Ryo Suzuki — Programmable Reality Lab (2025): An LLM automatically generates AR instructions pinned to the real objects you need to handle. https://www.youtube.com/watch?v=n6jMzQ6Z2Ic - **Hands-free Gemini assistant with Vision Pro camera access** — Stijn Spanhove (2025): An assistant that shares your point of view instead of waiting for a photo. https://x.com/stspanho/status/1875989390344032340 - **Live caption glasses for the Deaf and hard of hearing** — Cayden Pierce (Mentra / AugmentOS) (2025): Subtitles for real life, placed where you are already looking. https://www.youtube.com/watch?v=CRyhfPrfywU - **QuestCameraKit: passthrough camera with object detection and more** — Roberto Coviello (2025): Once the headset camera is open, every AI model becomes a mixed-reality feature. https://www.youtube.com/watch?v=1z3pcMJbnRA - **Ray-Ban Meta safety assistant with a cloud VLM** — Stijn Spanhove (2025): Glasses that quietly watch for danger and tap you when they see it. https://x.com/stspanho/status/1996894608597700645 - **Reality Proxy** — Mar Gonzalez-Franco (2025): Interact with a stand-in instead of the hard-to-reach real thing. https://www.youtube.com/watch?v=F2ul_68PrD0 - **RealitySummary** — Ryo Suzuki — Programmable Reality Lab (2025): Printed paper gets an AI sidekick that writes in the margins of space. https://www.youtube.com/watch?v=JxtctjERJBU - **Remote object detection with Hugging Face on Spectacles** — Alessio Grancini (2025): Borrow any open vision model from the web and see its answers on the world. https://www.youtube.com/watch?v=6FuHyGiO8dc - **Replaying a broadcast basketball play on a real court in XR** — Stijn Spanhove (2025): Watch last night's play happen again on the court you are standing on. https://x.com/stspanho/status/1992980633652478223 - **Sonify Anything** — Laura Schütz (2025): Let the materials of the real world decide how virtual collisions sound. https://www.youtube.com/watch?v=tN6JE3kNUbk - **Spectacles AI-powered translation challenge** — Snap Inc. (Snapchat Lenses) (2025): Subtitles for real life let strangers build something together. https://www.youtube.com/watch?v=hR4W_Qtzij4 - **ARSports** — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2024): Let a vision model highlight the only things that matter in a fast game: the ball and the people. https://www.youtube.com/watch?v=2IvhNwAAl4E - **Augmented Physics** — Ryo Suzuki — Programmable Reality Lab, Rubaiat Habib Kazi (2024): Photograph a physics diagram in a textbook and it comes alive as a simulation you can tune. https://www.youtube.com/watch?v=HUgYeA3BKfk - **CookAR** — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2024): Color the world by what is safe to grab and what can hurt you. https://www.youtube.com/watch?v=gtJhme8em7U - **EARLL** — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2024): Your hands choose the lesson: touch an object and learn its name. https://www.youtube.com/watch?v=AjdQsxkPVkU - **Frame: open-source AI glasses** — Brilliant Labs (2024): Open hardware so anyone can build their own seeing, talking glasses. https://www.youtube.com/watch?v=OS6GMsYyXdo - **GazeGPT** — Zinn Labs (Robert Konrad, Nitish Padmanaban, Kevin Boyle) (2024): Use your gaze as the pointer for an AI that looks through your glasses. https://www.youtube.com/watch?v=AuDFHHTK_m8 - **GazePointAR** — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2024): Replace the pronouns in a spoken question with whatever your eyes and finger are pointing at. https://www.youtube.com/watch?v=3DebNQ1kpe0 - **Human I/O** — Ruofei Du (2024): Before deciding how to talk to you, the interface first asks which of your senses and limbs are free. https://www.youtube.com/watch?v=vHBCLixDEYs - **I-XRAY** — AnhPhu Nguyen & Caine Ardayfio (2024): A privacy warning built as a working demo: glasses that dox at a glance. https://x.com/AnhPhuNguyen1/status/1840786336992682409 - **Interfacing AI through physical paper** — Lukas Moro (2024): Paper stays the interface; the headset quietly turns it into a conversation with AI. https://x.com/lukas_moro/status/1838207092471050645 - **Math Notes on real paper** — Lukas Moro (2024): Could mixed reality bring a renaissance of paper as a productivity tool? https://x.com/lukas_moro/status/1829487148078412019 - **Mistral-OUI: contextual AR UI** — Alessio Grancini (2024): Whatever you look at, AI generates a matching interface for it on the spot. https://www.youtube.com/watch?v=2Bcnh3w1CCg - **OmniActions** — Meta Reality Labs Research — AI assistants for AR glasses (2024): Turn anything you perceive into a ready-made next action. https://www.youtube.com/watch?v=DZ8CXS2iS0o - **Pocket Buddy** — Max van Leeuwen (2024): Scan your friend into a pet that asks you to find real things. https://www.youtube.com/watch?v=lCiCceLHw_s - **Project Astra: a universal AI assistant that sees** — Google Android XR & DeepMind (Project Astra) (2024): A single assistant that lives in your camera and keeps the context of your day. https://www.youtube.com/watch?v=hIIlJt8JERI - **SituationAdapt** — Christian Holz (2024): Let a vision-language model act as the social and safety editor of your floating UI. https://www.youtube.com/watch?v=3C_87tDsTGw - **WorldScribe** — Anhong Guo — Human-AI Lab (University of Michigan) (2024): A narrator for the world that decides what to say, how much, and when to stay quiet. https://www.youtube.com/watch?v=9dRExJzyqxw - **XR-Objects: Augmented Object Intelligence** — Mar Gonzalez-Franco, Ruofei Du (2024): Treat every physical object as if it were a digital one you can click on. https://www.youtube.com/watch?v=4DjPuf-oyq8 - **AI + AR Shopping Concept** — Russ Maschmeyer, Shopify Spatial Commerce Team (2023): Let AI answer shopping questions by lighting up the real world instead of a chat window. https://x.com/russ_maschmeyer/status/1640741787105984512 - **AR Lens for Blind People** — Xulipa (Allan Yde) (2023): Use AR's understanding of space for someone who cannot see the screen at all. https://www.youtube.com/watch?v=pRbeE_P1tdM - **Bubbleu** — Youngki Lee (Seoul National University) (2023): Treat the AI's mistakes as game material instead of hiding them. https://www.youtube.com/watch?v=nrq8WfGFB8s - **Convoscope: live definitions and insights during conversation** — Cayden Pierce (Mentra / AugmentOS) (2023): An assistant that listens along and whispers context into your eye. https://www.youtube.com/watch?v=3n6DzuYQ_v8 - **Look, Pinch, Ask (Vision Pro concept)** — Russ Maschmeyer, Shopify Spatial Commerce Team (2023): Ask about the thing you are looking at, the moment you are curious. https://x.com/russ_maschmeyer/status/1696520326266634557 - **Peridot** — Niantic (John Hanke) (2023): An AR pet that recognizes grass, water and sand and plays with the real environment. https://www.youtube.com/watch?v=HFi2pJrAEdc - **Real-Time AR Effect on Cat** — Takashi Yoshinaga (2023): Track a house cat in real time and wrap it in AR effects that follow it around. https://www.youtube.com/watch?v=ExRmlztPP7M - **Touch Grass lens** — Aidan Wolf (2023): It checks that you really touched grass and hands you proof that you went outside. https://x.com/Aidan_Wolf/status/1701565816385179877 - **XAIR** — Meta Reality Labs Research — AI assistants for AR glasses (2023): Treat 'why is the AI showing me this?' as a first-class AR design question. https://www.youtube.com/watch?v=yugkzrBbTCs - **arGPT fact-checking on the Monocle** — Brilliant Labs (2023): A fact-checker that sits on your glasses during a conversation. https://www.youtube.com/watch?v=VyHFm3OrDPo - **PRE-FIGURES** — Uncharted Limbo Collective (George Adamopoulos, Eleana Polychronaki, Chris Waters), Alexander Whitley Dance Company (2022): Archive dancers step out of old film and share the stage with living ones. https://www.youtube.com/watch?v=i_Zr_9x_1zk - **Ukemochi** — Kiyoshi Kiyokawa (2022): Let only the food cross the boundary between the real and the virtual world. https://www.youtube.com/watch?v=dFAB1Uxz3zU - **Chain of Traceability** — FIELD (FIELD.IO / FIELD.SYSTEMS), SPACE10 (IKEA's research and design lab) (2021): Let objects tell their own life story, from source to second life. https://www.youtube.com/watch?v=BkPHI3FM7lg - **M-LSD line detection as VFX** — Keijiro Takahashi (2021): The architecture of a room becomes the drawing. https://x.com/_kzr/status/1413426397054332930 - **TinyYOLOv2 object detection in Unity** — Keijiro Takahashi (2021): Let the engine know what it is looking at. https://x.com/_kzr/status/1353349183252533249 - **Voice Recognition with Shaders** — SCRN (2021): Recover speech from lip-sync signals when the raw audio is out of reach. https://x.com/SCRNinVR/status/1367343416644886531 - **YOLOv4-tiny in Fragment Shaders** — SCRN (2021): Run a neural network where scripts are not allowed: inside the shader itself. https://x.com/SCRNinVR/status/1380238589238206465 - **AR Cut & Paste** — Cyril Diagne (2020): Copy and paste between the physical world and your computer with a phone camera. https://x.com/cyrildiagne/status/1259441154606669824 - **Copy printed text to desktop with AR+ML** — Cyril Diagne (2020): Treat printed text as something you can pick up and drop into a digital document. https://x.com/cyrildiagne/status/1262047009411907585 - **Augmented Reality Language Learning** — Jason Orlosky (2019): Turn your own home into a vocabulary flashcard deck. https://www.youtube.com/watch?v=4SkkWc396J4 - **Paper Classifier (openFrameworks + Wekinator)** — Jorge Guevara (2019): A plain piece of paper becomes a five-button controller once the camera learns its shapes. https://www.youtube.com/watch?v=y2tAz2yMLz4 - **Avocado Toast with ARKit** — Laan Labs (Chris Laan) (2017): Recognize a real avocado and turn it into avocado toast. https://www.youtube.com/watch?v=oikVikS2Ge0 - **Fencing tracking and visualization system** — Daito Manabe / Rhizomatiks (2017): Fencing blade tips too fast for the eye are drawn live as trails over the broadcast. https://www.youtube.com/watch?v=3W1_5io-NYQ - **Magic Sudoku** — Brad Dwyer (2017): Point at a Sudoku on paper and the answers are written right into the empty squares. https://x.com/braddwyer/status/910030265006923776 - **SemanticPaint** — Microsoft Research Cambridge / I3D — Shahram Izadi (2015): Point and speak while scanning a room, and the system learns to recognize every piece of furniture. https://www.youtube.com/watch?v=z_TcWC7yjj0 - **Word Lens for Glass** — Otavio Good (Quest Visual) (2013): Look at a sign through the glasses and the translation appears right in your view. https://www.youtube.com/watch?v=pZKWW3rzT2Q - **Word Lens** — Otavio Good (Quest Visual) (2010): Point at a foreign sign and the words turn into your own language on the spot, like magic. https://www.youtube.com/watch?v=h2OfQdYrHRs ### Agents & companions Characters, guides and assistants that live in your space, talk and act. - **AI Building Blocks: multimodal LLMs in mixed reality** — Dilmer Valecillos (2026): Ask your room a question and let the model see it before answering. https://www.youtube.com/watch?v=Q8BFLkRYOy0 - **AgentHands** — Ruofei Du (2026): Give a talking AI hands, so it can point instead of describe. https://www.youtube.com/watch?v=O4Gma0XqRgM - **ChatMuse** — Chen Chen & Shaoze Zhou (2026): An AI prompter whispering from the wings of a real conversation. https://www.youtube.com/watch?v=IMoYRhvsHsI - **Gemini Vision Navigator** — FireDragonGameStudio (2026): Indoor navigation without beacons: an agent that reads the room and tells you where to go. https://www.youtube.com/watch?v=m61hl5wy9cE - **Physio3D: mixed-reality physiotherapy with a real-time AI voice coach** — Physio 3D (2026): A therapist who talks back is placed where you actually exercise. https://www.youtube.com/watch?v=TlQLjDA5XSY - **Sai-Fi: talk to a computer-use agent through Ray-Ban glasses** — Simular (2026): Your glasses delegate; an agent does the clicking at home. https://x.com/SimularAI/status/2090926852974297577 - **Smart glasses AI tour guide** — Cayden Pierce (Mentra / AugmentOS) (2026): A personal guide for any street, built in a weekend on open glasses. https://www.youtube.com/watch?v=pEm8m59k3KQ - **Spectacles controlling a Reachy Mini robot** — Alessio Grancini (2026): AR glasses become the remote control and face for an AI robot on your desk. https://www.youtube.com/watch?v=7-7kT0e0knQ - **AI smart glasses that order a Big Mac** — Cayden Pierce (Mentra / AugmentOS) (2025): Glasses as the remote control for an agent that acts in the world for you. https://www.youtube.com/watch?v=ILG9XfiRlPM - **AI wayfinding companion (Spatial 1)** — Ian Curtis (2025): An AI wayfinding companion worn on the chest that watches the road and talks to you. https://x.com/XRarchitect/status/1948817329493512337 - **AiGet** — Shengdong Zhao — Synteraction Lab (2025): Turn idle moments into curiosity: the AI notices what you glance at and tells you something you did not know. https://www.youtube.com/watch?v=Z1WACpnZodg - **Android XR glasses live demo with Gemini (Google I/O 2025)** — Google Android XR & DeepMind (Project Astra) (2025): Glasses with an assistant that remembers what you saw a minute ago. https://www.youtube.com/watch?v=7nv1snJRCEI - **Bringing Everyday Objects to Life with AI-Powered Talking Characters** — Fabrice Matulic (Preferred Networks) (2025): Give every object a voice and a personality that come from what the object is. https://www.youtube.com/watch?v=RkRc7TRDPDQ - **EmBARDiment** — Mar Gonzalez-Franco (2025): Your gaze history is the prompt. https://www.youtube.com/watch?v=Af2lxWTFiz4 - **Halo and Noa: glasses that remember what you forgot** — Brilliant Labs (2025): Glasses as a memory that notices what slipped through the cracks. https://www.youtube.com/watch?v=Xp06a3n4LZU - **Huk, the Jaguaress** — Violeta Ayala (2025): A film that looks at you, judges you and answers. https://www.youtube.com/watch?v=4cDaYOr3QBM - **If My Apple Can Talk** — Yu Wang & Yulu Lu (Beihang University) (2025): An object's physical traits become the design brief for its AI personality. https://www.youtube.com/watch?v=XyrT846glfE - **Meta Ray-Ban Display live demo (Connect 2025)** — Meta Reality Labs (Quest mixed reality) (2025): An AI assistant you see, not just hear, answering in the corner of your eye. https://www.youtube.com/watch?v=gZ9IsB72nVk - **ProMemAssist** — Meta Reality Labs Research — AI assistants for AR glasses (2025): An assistant that knows not only what to say but when your mind has room for it. https://www.youtube.com/watch?v=4fd5p7sR6Mo - **Project Jade - Spatial Agents (Liquid City)** — Keiichi Matsuda (2025): An AI companion that sees what you see and answers you out in space. https://www.youtube.com/watch?v=ZwyGUZIoa8Q - **Revisiting Put-That-There** — Mar Gonzalez-Franco (2025): Speech plus pointing is the oldest multimodal idea, and LLMs finally make its ambiguity solvable. https://www.youtube.com/watch?v=iyBkON2r9QI - **Rubikon** — Anhong Guo — Human-AI Lab (University of Michigan) (2025): Keep the real object in your hands but let AI rewrite its state, so every practice round targets your weak spot. https://www.youtube.com/watch?v=4Jes_0nd3kY - **Sensible Agent** — Ruofei Du (2025): A good assistant reads the social situation before it speaks. https://www.youtube.com/watch?v=iYayksCTYGI - **Wisp World (Liquid City)** — Keiichi Matsuda (2025): An AI and AR game in which you go on adventures with a talking little spirit in your real room. https://www.youtube.com/watch?v=CtP9FERiL0w - **Even AI on Even G1 glasses** — Even Realities (2024): An assistant whose answers are private text in your line of sight. https://www.youtube.com/watch?v=Quam8jLa8pY - **Hybrid Thinking smart glasses** — Cayden Pierce (Mentra / AugmentOS) (2024): Thinking with an AI in the middle of a real conversation, not after it. https://www.youtube.com/watch?v=DWyLFKIVrfM - **Las Awichas** — Violeta Ayala (2024): Grandmothers' wisdom is carried by AR animals that listen back. https://www.youtube.com/watch?v=maAXs0HzNA8 - **Memoro** — MIT Fluid Interfaces — Pattie Maes & Pranav Mistry (2024): A memory prosthesis that speaks up only with the one fact you are missing. https://www.youtube.com/watch?v=fOmvnHcGcXA - **Moon Story (Scientific and Fantastical)** — James Landay — Stanford HCI (2024): Blend a fantasy story with real science and let the characters answer the child's own questions. https://www.youtube.com/watch?v=ZpKhhMcS8Jw - **PANDALens** — Shengdong Zhao — Synteraction Lab (2024): Glasses that interview you during the trip and write the diary with you. https://www.youtube.com/watch?v=WceDSG4I4cw - **Agents (Liquid City)** — Keiichi Matsuda (2023): Give AI assistants a body as characters living in AR glasses, and think about how people relate to agents. https://www.youtube.com/watch?v=bkKv2AHpn8E - **Matrix Stockroom** — Russ Maschmeyer, Shopify Spatial Commerce Team (2023): Show search results as an infinite room you walk through, then talk to narrow it down. https://x.com/russ_maschmeyer/status/1613218237969494017 - **Voice In My Head** — Lauren Lee McCarthy, Kyle McDonald (2023): An AI inner voice in your earpiece whispers live advice while you talk to people. https://www.youtube.com/watch?v=B2-dV8IrhWo - **Wol: AI owl guide to the redwoods** — Ian Curtis (2023): Talk with an AR owl that shows you around the redwood forest. https://www.youtube.com/watch?v=GUgAw-uU46o - **City of Sparkles** — Botao 'Amber' Hu (2022): How might an AI perceive our city? Let the audience see it through the AI's eyes. https://vimeo.com/777136892 - **Digital Buddy** — FIELD (FIELD.IO / FIELD.SYSTEMS), SPACE10 (IKEA's research and design lab) (2021): A companion on your side that reads the fine print for you. https://vimeo.com/558597213 - **us+** — Lauren Lee McCarthy, Kyle McDonald (2013): An AI "social coach" in video calls that reads your face and words and nudges you on screen in real time. https://vimeo.com/81903116 - **Remembrance Agent** — Thad Starner (1996): A just-in-time memory: the wearable listens to what you are doing and surfaces your own past notes without being asked. https://www.youtube.com/watch?v=k-zThJX920w - **Neuro-Baby** — Naoko Tosa (1993): A virtual being that responds to how you feel, not what you say. https://www.youtube.com/watch?v=etEIbMZ6uUY ### Generate the world Say or sketch it and it appears: text, voice or drawings turned into 3D content in place. - **A picture frame that is a window** — Ian Curtis (2026): A 'picture frame' on the wall is really a window into another 3D world. https://x.com/XRarchitect/status/2052418499516170377 - **A pocket holodeck in WebAR** — Ian Curtis (2026): Each AI-generated scene is only 5 MB, so a few of them placed around you become a holodeck you can carry in your pocket. https://x.com/XRarchitect/status/2014383901268181253 - **An LLM turns a home scan into an editable scene with a portal** — Bilawal Sidhu (2026): Multimodal LLMs may have crossed the chasm on inverse graphics: scans become worlds you can direct. https://x.com/bilawalsidhu/status/2096810261730505075 - **Living room redesign as WebAR splat** — Ian Curtis (2026): Overlay an AI-designed living room on the old one at 1:1 scale, and walk into the 'future room' with a phone. https://x.com/XRarchitect/status/2076932513822617966 - **Off-axis window into a generated world** — Ian Curtis (2026): Block out a scene with simple shapes, let AI fill in the detail, then use face tracking to turn the screen into a window onto that world. https://x.com/XRarchitect/status/2034675907898876133 - **Re-coaching a World Cup goal in AR** — Stijn Spanhove (2026): Pause a real World Cup goal, move a defender, and let AI re-render a different ending. https://www.youtube.com/watch?v=uPpCLfgWbdA - **Shrunk to action-figure size on the coffee table** — Ian Curtis (2026): Shrink yourself to action-figure size and explore a miniature world on the coffee table. https://x.com/XRarchitect/status/2053874498630468078 - **Snap2VoxelAR** — Takashi Yoshinaga (2026): Snap a photo and get back a toy-block version of the world. https://www.youtube.com/watch?v=57gkK-xGxKQ - **Tap to launch orbs into a generated world** — Ian Curtis (2026): Overlay an AI-generated 3D world on a real room and tap to fire glowing orbs into it. https://x.com/XRarchitect/status/2077641433855795594 - **Turn your memories into 3D Gaussian splats** — Takashi Yoshinaga (2026): An old photo becomes a place you can lean into. https://www.youtube.com/watch?v=5nlFjTWZSE8 - **Bringing an action figure to life in AR** — Stijn Spanhove (2025): Photograph a toy and it steps out of its box as a living AR character. https://x.com/stspanho/status/1910708661279773101 - **Bubbles** — Pavlo Tkachenko (2025): Type some words to generate 3D objects, then fill the space around you with them, each sealed in a bubble. https://www.youtube.com/watch?v=TAPU-f4w7xs - **Chain-of-thought models designing XR puzzles** — Stijn Spanhove (2025): Let a reasoning model design the level, then step inside it. https://x.com/stspanho/status/1885352446806970708 - **Fireside Tales** — Yegor Ryabtsov, Stijn Spanhove, Pavlo Tkachenko (2025): A campfire where the spoken story instantly becomes pictures floating in the shared space. https://www.youtube.com/watch?v=XDcy9xj_BII - **From 2D video to 4D Gaussian splats on Vision Pro** — Stijn Spanhove (2025): Any flat video can become a scene you walk around, one generated frame at a time. https://x.com/stspanho/status/2004310347486761051 - **Generative-AI earthquake rehearsal overlaid on a real room** — Tomoki Itamiya (2025): Rehearse a disaster in the exact room where it could happen. https://x.com/t_itamiya/status/1914896389152133364 - **Life Stories × World Labs: Night Lake** — Keiichi Matsuda (2025): A spoken memory becomes a place other people can walk through. https://www.youtube.com/watch?v=w5fp-otQhQU - **Midjourney → splat → AR** — Ian Curtis (2025): Go from one AI image to an AR scene you can stand inside. https://x.com/XRarchitect/status/1954372725667254569 - **Morning stroll into a portal** — Ian Curtis (2025): Walk through a door into a world generated by AI. https://x.com/XRarchitect/status/1970878813338051024 - **Photo to WebAR world in real time** — Ian Curtis (2025): A single photo becomes an AR scene you can walk into. https://x.com/XRarchitect/status/1980654061323255906 - **Redesigned living room, 1.5 MB per splat** — Ian Curtis (2025): Switch between several AI-generated living-room designs right where they will go. https://x.com/XRarchitect/status/1995541338335678801 - **Shape n' Swarm** — Ken Nakagaki (2025): Sketch a shape with your hands and say one sentence, and a robot swarm forms it and brings it to life. https://www.youtube.com/watch?v=5u0M9yL7tyY - **Sketch2Terrain** — Tianyi Xiao (ETH Zurich) (2025): Sketch a few ridgelines in the air and let AI fill in the whole landscape. https://www.youtube.com/watch?v=DPvd_EmxzVg - **Spectacles AI release: Snap 3D and LLM integrations** — Snap Inc. (Snapchat Lenses) (2025): AR glasses where you can ask for a 3D object and it appears. https://www.youtube.com/watch?v=qpi1JTnPwgs - **Step inside an AI-generated movie scene** — Stijn Spanhove (2025): A film scene you stand inside, rebuilt entirely by generative models. https://x.com/stspanho/status/2005626245166313514 - **Step into a persistent redesign, 1:1** — Ian Curtis (2025): AI design, AI generation and precise localization let you step into a redesigned living room at 1:1 scale in your own home. https://x.com/XRarchitect/status/1968356682888823060 - **Thing2Reality** — Ruofei Du (2025): Anything you can show on camera can instantly become a shared 3D object. https://www.youtube.com/watch?v=ZClLOeu6y5A - **Turning a child's drawing into a playable XR world** — Stijn Spanhove (2025): A paper drawing is the level design; an AI world model builds the game. https://x.com/stspanho/status/1961138591934947462 - **XR Worlds** — Stijn Spanhove (2025): Turn a 2D doodle on paper into a playable XR world. https://www.youtube.com/watch?v=h7ll6AB4rcc - **AI-generated 3D models with Luma Genie in mixed reality** — Figmin XR (Overlay) (2024): Say an object and place it on your real table a minute later. https://www.youtube.com/watch?v=PFY1mk4gzjs - **BlendScape** — Balasaravanan Thoravi Kumaravel (2024): Let meeting participants generate the shared place they meet in. https://www.youtube.com/watch?v=maqZbVyuGBA - **LLMR** — Fernanda De La Torre (LLMR, with Microsoft) (2024): Language becomes the runtime: objects, behaviors and scene edits are generated as code while you are inside the world. https://www.youtube.com/watch?v=9YdYkgNNpE8 - **Prompt-your-own AR drone show** — Stijn Spanhove (2024): Anyone can direct a drone show by describing it. https://x.com/stspanho/status/1843725659585294651 - **SonifyAR** — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2024): Let AI decide what an AR scene should sound like from real materials and virtual actions. https://www.youtube.com/watch?v=2iXS2tvjwKg - **SpaceBlender** — Balasaravanan Thoravi Kumaravel (2024): Merge everyone's real rooms into one imagined shared room. https://www.youtube.com/watch?v=wQqJHcrOado - **The Keywords Karaoke** — Fabio Lattanzi Antinori (2024): Sing your neighbourhood's search history. https://vimeo.com/952736898 - **ARephotography** — Stefanie Zollmann (2023): Turn an old photograph into a 3D time window placed on the same spot today. https://www.youtube.com/watch?v=nGZBKTKC1us - **Running AR videos through NeRF and Gaussian splatting** — Bilawal Sidhu (2023): Bake AR into reality: once reconstructed, the virtual and the real are the same material. https://x.com/bilawalsidhu/status/1694086197029503466 - **Visual Captions** — Ruofei Du, Alex Olwal (2023): Live visuals that follow the conversation, like captions but in pictures. https://www.youtube.com/watch?v=Dv4lsS-f8Bc - **AI Product Genie** — Russ Maschmeyer, Shopify Spatial Commerce Team (2022): Speak a wish, see the custom product in your room, buy it instantly. https://x.com/russ_maschmeyer/status/1569700294673702912 - **The Welcome Chorus** — Yuri Suzuki (2019): A public choir of horns that learns new lyrics from passers-by. https://www.youtube.com/watch?v=pB1TBwACzsE - **Ngx** — Keijiro Takahashi (2018): A neural network that hallucinates an infinite music video from a short clip. https://vimeo.com/294399440 - **Pix2Pix for Unity** — Keijiro Takahashi (2018): Use real-time image-to-image translation as a drawing tool. https://vimeo.com/287778343 - **WDCH Dreams** — Refik Anadol (2018): Let a concert hall use AI to 'dream' its hundred years of memories and project them onto its facade. https://www.youtube.com/watch?v=PuMVVsoiLPM ### Restyle reality Live diffusion and style transfer that repaint the world you are looking at. - **AI-transformed video call on Spectacles** — Alessio Grancini (2026): Telepresence where the person you see is re-imagined as they talk. https://x.com/alessiograncini/status/2079985282510631174 - **Art Morph (Hands-on Hallucinations) at the Cleveland Museum of Art** — Hugues Bruyère (2026): Your own hands become the brush that turns you into a museum painting. https://x.com/smallfly/status/2089105114183381042 - **Prop-Chromeleon** — Ludwig Sidenmark & Haoyu Wang (2026): Reskin the object in your hand with a prompt, using its real shape as the constraint. https://www.youtube.com/watch?v=IKRM-_YxGpM - **Real-time AI style transfer on AR glasses** — Alessio Grancini (2026): A reality filter that lives on your glasses rather than your phone screen. https://x.com/alessiograncini/status/2082150582467387856 - **Room restyled as Studio Ghibli, live on Vision Pro** — Wenbo Tao (2026): Rebuild your room as editable objects first, then restyle it any way you like. https://x.com/wenbq_me/status/2104174834276061569 - **AI Teleport** — Stijn Spanhove (2025): Generative AI 'teleports' your surroundings into another world in real time. https://www.youtube.com/watch?v=GvBRlCcgJ_g - **Decart XR: speak a prompt and transform the world** — Roberto Coviello (2025): Say a world and step into it without leaving your room. https://x.com/xrdevrob/status/1974535441811677627 - **Frame-a-scene spatial pixel art (first Spectacles lens)** — Stijn Spanhove (2025): Your fingers are the camera; AI turns what you frame into pixel art in place. https://x.com/stspanho/status/1894776068600447319 - **Mixed Reality + Diffusion with the Quest Passthrough Camera API** — Hugues Bruyère (2025): Paint over the real room with a prompt, from inside the headset. https://x.com/smallfly/status/1901403937321750862 - **Real-time AI video restyling an XR room** — Stijn Spanhove (2025): Type a style and your room is redecorated around you, live. https://x.com/stspanho/status/1980751025930661934 - **Stable Diffusion AR with Quest passthrough camera access** — xr masiso (2025): The first day the headset can see, it can also dream about what it sees. https://www.youtube.com/watch?v=FXFgkAmvpgo - **AR with a magnifying glass** — Dpt. (2024): A magnifying glass that reveals imaginary worlds inside ordinary things. https://vimeo.com/973282394 - **Dcam2: Stable Diffusion VJ set with DUB-Russell** — Keijiro Takahashi (2024): Mix generative AI with body tracking for live visuals. https://www.youtube.com/watch?v=qa4jv5JhKhM - **DreamSpace** — Bangbang Yang (2024): Repaint your real room with a sentence while keeping its real geometry. https://www.youtube.com/watch?v=Ni5kXdj7cfw - **StreamDiffusionTD: real-time diffusion in TouchDesigner** — dotsimulate (Lyell Hintz) (2024): Any live video signal can be restyled by a prompt as it happens. https://www.youtube.com/watch?v=X4rlC6y1ahw - **WebAR capture enhanced with GenAI video-to-video** — Stijn Spanhove (2024): Use AR as a rough sketch and let AI finish the rendering. https://x.com/stspanho/status/1845952462517424401 - **AI style transfer on live AR camera** — Sander Veenhof (2023): AI turns the world in front of you into a painting in real time. https://www.youtube.com/watch?v=XU8D91ljJ9A - **Clueless Closet** — Russ Maschmeyer, Shopify Spatial Commerce Team (2023): Movie-magic try-on: see yourself in any outfit combination instantly. https://x.com/russ_maschmeyer/status/1650892732611858434 - **Dcam: realtime Stable Diffusion in live performance** — Keijiro Takahashi (2023): Run image generation fast enough to become a live visual. https://www.youtube.com/watch?v=iVi-7oz67OU - **Generative AI Virtual Try-on research** — ZERO10 (2023): Swap real-time 3D for generative images: the model imagines you in the outfit. https://x.com/zero10_ar/status/1729535657444286935 - **Fashion Twin** — GoSpooky (2022): Borrow someone else's outfit with your camera. https://x.com/GospookyHQ/status/1593200088758640640 - **Lil Cartoon Bratz Doll** — Paige Piskin, Snap Inc. (Snapchat Lenses) (2021): Turn your face into a 90s fashion-doll cartoon in real time. https://www.youtube.com/watch?v=UIexjgFSXjc - **Neural Cameras** — Tobias Langlotz, Stefanie Zollmann (2021): Make virtual objects look filmed by the same imperfect camera as everything else. https://www.youtube.com/watch?v=cGc8vBVbIvI - **TransforMR** — Christian Holz (2021): Keep reality's motion but swap every object for something from another world. https://www.youtube.com/watch?v=RsxdGwRvvEU - **Pix2Pix in a Fragment Shader** — SCRN (2020): A GAN can be just another material on a surface. https://x.com/SCRNinVR/status/1317299768301735936 - **Learning to See** — Memo Akten (2017): As hands move cloth and cables on a table, AI sees them live as waves, fire or nebulae. https://vimeo.com/260612034 - **Realistic AR Brush Texture with Deep Learning** — Laan Labs (Chris Laan) (2017): A neural network turns photos into oil-paint brushwork and hangs them on real walls. https://www.youtube.com/watch?v=9_2o5E1rmBI - **Cubist Mirror** — Gene Kogan (2016): A mirror that reflects you as a Cubist painting. https://vimeo.com/167910860 ### AI and the body Pose, face and motion models that read the body and answer it. - **A Moving Sanctuary** — Random Studio (2026): A room that breathes with you. https://vimeo.com/1196639577 - **Generative Muscle Stimulation** — Pedro Lopes (2026): Instead of answering in words, the AI answers by moving your body. https://www.youtube.com/watch?v=pJM2Z8mmwAw - **Meta glasses fitness coach** — Oscar Falmer (2026): A coach that watches your form from your own point of view. https://x.com/OscarFalmer/status/2018276017089990824 - **Mirror** — Alexander Whitley Dance Company (2026): An AI mirror that starts faithful and slowly lies. https://www.youtube.com/watch?v=zjHSgImXydw - **ViSTAR** — Harvard Visual Computing Group — Hanspeter Pfister (2026): Let an LLM read your 3D motion data and talk like a coach. https://www.youtube.com/watch?v=aaCfLeenGdE - **Body Oracle Translator** — Danlin Huang, Botao 'Amber' Hu (2025): Translate body postures into AI-generated 'oracle-bone script' in real time. https://www.youtube.com/watch?v=GXeLSfZnXAQ - **Video2MR** — Ryo Suzuki — Programmable Reality Lab (2025): Lift the person out of a YouTube video and make them your 3D trainer. https://www.youtube.com/watch?v=f1L-2US25PU - **AI-driven virtual try-on** — Stijn Spanhove (2024): The mirror becomes a fitting room that can dress you in anything. https://x.com/stspanho/status/1869053825082134912 - **LILITH.AEON** — AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) (2024): A virtual being dances back at the crowd, so every visit becomes a new duet between people and an AI. https://vimeo.com/925210197 - **Motion as input for real-time image generation** — Lukas Moro (2024): Dance the prompt instead of typing it. https://x.com/lukas_moro/status/1842953164498100617 - **Echoes** — Torin Blankensmith (2023): Every move you make summons the most similar move made by an earlier visitor. https://x.com/blankensmithing/status/1737307286572298738 - **Self Absorbed** — Tim Murray-Browne (2023): Navigate an AI's memory of your life with your body, not a mouse. https://www.youtube.com/watch?v=JKg-6fHRT9U - **Nonverbal Interactions with Soli Radar** — Google ATAP (Advanced Technology and Projects) (2022): Devices that respond to social cues like approach and turning away, the way people do. https://www.youtube.com/watch?v=r-eh2K4HCzI - **PAN+TILT** — Gibson/Martelli (Ruth Gibson & Bruno Martelli) (2022): Use a somatic dance technique as the controller for a virtual world. https://vimeo.com/758291757 - **Unlearning Language** — Lauren Lee McCarthy, Kyle McDonald (2022): In a room watched constantly by AI, people have to invent new ways of communicating that machines cannot read. https://vimeo.com/780628671 - **Engendered Otherness (Ai Transmutations)** — Kianí del Valle (2021): A live dancer leads a chorus of AI-generated creatures that only exist while she moves. https://www.youtube.com/watch?v=hXiDYN1IFfY - **BODY** — Random Studio (2018): Use your whole body as a search term. https://vimeo.com/314018273 - **Move Mirror** — Irene Alvarado, Google Creative Lab (2018): Your body becomes a search query that finds strangers striking the same pose. https://www.youtube.com/watch?v=JvzkFJW6LIU - **TendAR** — Tender Claws (Samantha Gorman & Danny Cannizzaro) (2018): A small AR fish that feeds on your expressions and emotions. https://www.youtube.com/watch?v=v4YMQRwsNr8 - **discrete figures (with Kyle McDonald)** — Daito Manabe / Rhizomatiks (2018): Machine-learning-generated virtual dancers share the stage with real ones. https://www.youtube.com/watch?v=hauXQQhwbgM - **Parsing our Silent Language** — Kat Sullivan (2016): Make the unspoken signals of body language visible as captions. https://vimeo.com/165629185 - **Level of Confidence** — Rafael Lozano-Hemmer (2015): Face recognition searches every visitor's face for the 43 missing students, turning surveillance into a memorial. https://vimeo.com/953969845 - **000000swan** — Phoenix Perry (2011): Teach the computer your own gestures instead of using its built-in ones. https://www.youtube.com/watch?v=dpW0wRkijQY - **ASL recognition from a wearable camera** — Thad Starner (1998): Point the camera at your own hands so the wearable understands the language they speak. https://www.youtube.com/watch?v=XWQN68ySnGk ### Co-creation tools Authoring, prototyping and design tools where AI and a person make AR together. - **AI Agent Vision: rebuilding a photo as an XR sketch** — Figmin XR (Overlay) (2026): An AI agent that sketches in 3D next to you, starting from a photo. https://www.youtube.com/watch?v=Sbk1b6-lcK8 - **Agent Center for Spectacles** — Snap Inc. (Snapchat Lenses) (2026): Your coding agents follow you around as floating panels. https://www.youtube.com/watch?v=U0cA-bVxLgM - **Arcade Hoops: a mixed-reality game built with AI in 30 days** — Dilmer Valecillos (2026): One developer plus coding agents can ship a mixed-reality game in a month. https://www.youtube.com/watch?v=eS0f7W6Y6cE - **Splat worlds as AR camera stand-ins** — Ian Curtis (2026): Use an AR phone as a virtual camera to block shots inside a generated 3D scene, then hand the footage to a video model for the final look. https://x.com/XRarchitect/status/2072207377106153797 - **Touching grass: two-prompt WebAR** — Ian Curtis (2026): Two prompts produce an outdoor WebAR prototype: the AI writes the code, and you play with virtual objects right on the grass. https://x.com/XRarchitect/status/2072573377714471177 - **AI-powered scripting: a school of fish** — Figmin XR (Overlay) (2025): Behaviours, not just objects, can be generated from words. https://www.youtube.com/watch?v=o_bxxVkN1Ag - **AI-scripted Playable Character** — Figmin XR (Overlay) (2025): AI writes the script that makes a little robot run and jump around your room. https://www.youtube.com/watch?v=z1DbE2WMZRg - **Capybara** — Andrés Monroy-Hernández & Lei Zhang — Princeton HCI Lab (2025): Make children the creators of AI-powered AR, not just its audience. https://www.youtube.com/watch?v=paJP8B8btVk - **ChatGPT 5 writing mixed-reality scripts in Figmin XR** — Figmin XR (Overlay) (2025): Describe a behaviour, and an object in your room starts doing it. https://www.youtube.com/watch?v=eY-538AArdI - **From prompt to lens with an MCP server for Lens Studio** — Alessio Grancini (2025): The AR editor becomes something an AI agent can operate for you. https://www.youtube.com/watch?v=bhWIryYyVvM - **ImaginateAR** — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2025): Speak a story into the place where you stand and let AI build it there. https://www.youtube.com/watch?v=G9mH1HKSPkg - **Mixed Reality RC Airplane** — Figmin XR (Overlay) (2025): An AI-made RC plane flies around the living room and crashes into real walls. https://www.youtube.com/watch?v=-jNwln2vMAg - **The prompt engineer and the director** — Hugues Bruyère (2025): Generative AR works best as a conversation between two people. https://x.com/smallfly/status/1911543395694027201 - **Vibe-coding an AR game inside Quest 3 with Gemini 3** — Jon Barron (2025): The holodeck is real: describe a game and play it in your room a minute later. https://x.com/jon_barron/status/1990909319701758157 - **VideoCraft** — Boyu Li (2025): Use MR to direct a generative video model with precise spatial layers. https://www.youtube.com/watch?v=uvozXgniRok - **XR Blocks and Vibe Coding XR** — Ruofei Du (2025): Make XR prototyping as quick as describing the idea in a sentence. https://www.youtube.com/watch?v=nknCzIxHHzw - **agentAR** — Karthik Ramani — Convergence Design Lab (Purdue) (2025): Let an AI agent assemble AR apps from research-proven building blocks while you only describe what you want. https://www.youtube.com/watch?v=vVUakvZij0Q - **Real-time diffusion as a muse while painting on paper** — Lukas Moro (2024): Keep the pencil; let the AI dream alongside your drawing. https://x.com/lukas_moro/status/1847299759603699906 - **AIShader: ChatGPT shader generator** — Keijiro Takahashi (2023): Describe a surface in words and get a shader. https://x.com/_kzr/status/1632634562399600640 - **AR + AI effect: a generated scene that reacts to your hands** — Karen X Cheng (2023): Use generative fill to paint the world, then let your hands play inside it. https://x.com/karenxcheng/status/1661474527673749504 - **Creating Lenses with GenAI Tools** — Anne Horel (2023): Use generative AI as a sketchbook that feeds straight into a live AR effect. https://www.youtube.com/watch?v=YUGlBTneq0M - **Teachable Reality** — Ryo Suzuki — Programmable Reality Lab (2023): Demonstrate with any object at hand a few times and it becomes a physical controller for AR effects. https://www.youtube.com/watch?v=JssiyfrhIJw - **Tetris in WebAR, written by ChatGPT** — Ian Curtis (2023): Let AI write the code and stand a game of Tetris up in the living room. https://x.com/XRarchitect/status/1737260350792425957 - **Drawn Together** — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2012): Draw a duet with a machine that replies in 3D above your paper. https://vimeo.com/37623623 ## Related art (not AR, but inspires AR) Land art, light, projection, fireworks, illusions and installations that already do in the real world what AR tries to do. ### Land & environment Works made in and with the landscape: the world itself becomes the canvas. - **Strandbeest Evolution 2025** — Theo Jansen (2025): Thirty-five years of one idea show that depth comes from iterating on a single system, a lesson for long-running AR projects. https://www.youtube.com/watch?v=ANhA94ZqnEQ - **The Living Pyramid** — Agnes Denes (2025): A monumental shape made of living plants changes with the seasons, showing how an AR form can have a growing skin. https://www.youtube.com/watch?v=gTMUGHzXK-o - **Sculpt the World** — Jon Foreman (2024): Imposing a simple order (size, colour) on natural material reads immediately as intention, useful for AR layout and generative placement. https://www.youtube.com/watch?v=yGKes_A5nIM - **A Year in a Catalan Forest** — Spencer Byles (2023): Works hidden off the path reward exploration, like geo-anchored AR pieces placed away from main routes. https://www.youtube.com/watch?v=GcJbxM9iIEo - **City** — Michael Heizer (2022): A walkable abstract architecture in the desert shows how pure geometry at huge scale can feel like a lost civilisation, a mood AR world-builders chase. https://www.youtube.com/watch?v=iWCThfpaxN8 - **KAWS:HOLIDAY Changbai Mountain** — KAWS (Brian Donnelly) (2022): Making the character from the landscape's own material (snow) ties it to the season, an idea for AR objects whose material changes with weather. https://www.youtube.com/watch?v=zqGiPif83ds - **Fog Sculpture (Neue Nationalgalerie)** — Fujiko Nakaya (2021): Fog is a real volume that people walk into and that the weather shapes, the benchmark for any volumetric AR effect. https://www.youtube.com/watch?v=k7VeOZbUZyk - **GROW** — Studio Roosegaarde (Daan Roosegaarde) (2021): Lighting a working landscape shows the hidden value of a place people normally drive past, a strong brief for AR in agriculture. https://www.youtube.com/watch?v=BId_104fAZI - **Ghost Forest** — Maya Lin (2021): Bringing the evidence of climate change into a busy park makes a distant event physical, a direct brief for climate-data AR. https://www.youtube.com/watch?v=0DHpGPwHm2I - **L'Arc de Triomphe, Wrapped** — Christo and Jeanne-Claude (2021): A familiar monument turned into a soft, shimmering object is exactly what a city-scale AR material swap feels like. https://www.youtube.com/watch?v=NUXIn9Fj7GI - **Strandbeest Evolution 2021** — Theo Jansen (2021): Treating each version as a species with a life cycle turns iteration into narrative, a way to present AR prototypes as a lineage. https://www.youtube.com/watch?v=C97kMKwZ2-g - **World in Progress** — Saype (Guillaume Legros) (2020): A picture of children drawing the world on the ground is a meta image of what location AR does: people drawing on the planet. https://www.youtube.com/watch?v=n5IkFCcqzsM - **Beyond Walls** — Saype (Guillaume Legros) (2019): One motif repeated city by city builds a world-spanning chain, a model for a location AR campaign where each site adds a link. https://www.youtube.com/watch?v=GbX3m2mG644 - **Beyond Walls, Champ de Mars** — Saype (Guillaume Legros) (2019): Placing the image under a famous viewpoint (the Eiffel Tower) guarantees an audience from above, a tactic for AR content anchored near lookouts. https://www.youtube.com/watch?v=xYIs1lqAAJs - **Tehachapi** — JR (2019): An image made for the view from above, in a place people cannot leave, lets people be seen by the world; AR can give voice to spaces that are otherwise closed. https://www.youtube.com/watch?v=WEkarbrmsIw - **Arcosanti** — Jim Denevan (2018): An ephemeral drawing beside a utopian architecture shows how a temporary layer can talk to a permanent site, as AR often does. https://www.youtube.com/watch?v=671Ur90JlhQ - **Message from Future** — Saype (Guillaume Legros) (2018): A message painted into a landscape can speak for a cause at the scale of a city, a strong use of large-scale location AR. https://www.youtube.com/watch?v=CPZgThf1h4M - **The London Mastaba** — Christo and Jeanne-Claude (2018): A single abstract volume on water, doubled by its reflection, shows how simple geometry plus reflection can feel monumental in AR. https://www.youtube.com/watch?v=P2vno5rLldE - **Transience** — Richard Shilling (2018): Sorting found things by colour instantly makes them look designed, a fast exercise that translates directly to AR collecting and arranging. https://www.youtube.com/watch?v=0O3d4n3_FuU - **Leaning into the Wind** — Andy Goldsworthy (2017): A body-shaped dry patch on wet ground is a trace of presence that disappears in minutes, a direct idea for AR silhouettes left by users. https://www.youtube.com/watch?v=BQYGbfVfpm0 - **Guided group beach mandala** — Andres Amador (2016): Splitting a big image into simple local tasks lets strangers co-create, a model for multi-user AR drawing sessions. https://www.youtube.com/watch?v=tVxKGuW9U4E - **Leysin land art (world record)** — Saype (Guillaume Legros) (2016): At this size the image becomes landscape, and it disappears as the grass grows: scale and decay are both part of the work. https://www.youtube.com/watch?v=W_CiGuMlhHw - **Seven Magic Mountains** — Ugo Rondinone (2016): Bright artificial colour against a natural landscape reads instantly as 'placed', the look many AR objects have, used here on purpose. https://www.youtube.com/watch?v=GDqbobusO80 - **The Floating Piers** — Christo and Jeanne-Claude (2016): Walking on water is felt through the body (the piers move under your feet), a reminder that AR can borrow real physical sensations. https://www.youtube.com/watch?v=h9KMY970tXk - **Horizon Line Chamber** — Chris Drury (2015): A dark room that shows the outside world live is the ancestor of AR passthrough: it reframes a familiar view by changing how it is delivered. https://www.youtube.com/watch?v=bo748C3iz2U - **Land Art au col des Aravis** — Saype (Guillaume Legros) (2015): A picture painted on a slope is designed for the view from across the valley, an early lesson in anamorphic placement for AR. https://www.youtube.com/watch?v=YdlpCru_x1Q - **Earth Wall** — Andy Goldsworthy (2014): An object half-inside a wall reads as something passing through a boundary, a classic AR occlusion effect in physical form. https://www.youtube.com/watch?v=I051qmxvDlE - **Earthscapes** — Andres Amador (2014): Working inside a fixed window of time (between tides) is a design constraint that AR events can borrow. https://www.youtube.com/watch?v=T_tIG5mo1DM - **East-West/West-East** — Richard Serra (2014): Level tops over uneven ground turn a desert into a measuring device, a strong idea for AR objects that reveal terrain by staying level. https://www.youtube.com/watch?v=0VbVxY98zG0 - **Ice Watch** — Olafur Eliasson (2014): Touching melting ice makes climate data physical and felt, a lesson for data AR: let people feel the number in their hands. https://www.youtube.com/watch?v=qd-JRGBKSXA - **Riverbed** — Olafur Eliasson (2014): Replacing a museum's floor with a landscape you have to climb over changes how the body moves, a strong case for full-room AR replacement. https://www.youtube.com/watch?v=s_xA_8ps_Co - **Sand drawings** — Jim Denevan (2014): Drawing at the scale of walking and seeing at the scale of flying is exactly the gap AR can close with a live overhead view. https://www.youtube.com/watch?v=Trdv3F7lcWI - **Van Gogh Path** — Studio Roosegaarde (Daan Roosegaarde) (2014): A path that becomes a painting when you ride it at night ties art to movement and route, a model for AR guidance lines with character. https://www.youtube.com/watch?v=f68cdc27HWg - **St. Catharines** — Jim Denevan (2013): Nested circles made one by one show how a big pattern can be built step by step by a single person, like incremental AR drawing. https://www.youtube.com/watch?v=aU7mrOdypgY - **Strandbeest evolution (archive)** — Theo Jansen (2013): A machine that walks by the wind makes an invisible force legible as a creature, a good model for AR creatures driven by real sensor data. https://www.youtube.com/watch?v=MYGJ9jrbpvg - **Tree Fall** — Andy Goldsworthy (2013): Bringing a tree indoors and coating the room around it creates a portal-like space, the same effect AR world replacement aims for. https://www.youtube.com/watch?v=qfKnT-8uq0I - **Floating and scattering installations** — Cornelia Konrads (2012): Natural objects frozen mid-explosion look like a paused physics simulation, an easy bridge to AR physics effects. https://www.youtube.com/watch?v=V8trnhNs4t0 - **Mandala I** — Andres Amador (2012): Radial symmetry lets a human-scale gesture add up to a readable image from above, a principle for AR ground patterns. https://www.youtube.com/watch?v=AYDsFNkG1vE - **Richard Long at The Hepworth Wakefield** — Richard Long (2012): Bringing a walk's materials indoors as a precise floor pattern shows how an outdoor trace can be translated into a room-scale layout, as AR can do. https://www.youtube.com/watch?v=tarLr9BL45w - **Untilled** — Pierre Huyghe (2012): The work is a system of living parts left to run by itself, a model for AR ecosystems that evolve whether or not anyone is watching. https://www.youtube.com/watch?v=CyXy7Aok_WM - **Carbon Sink: What Goes Around, Comes Around** — Chris Drury (2011): A simple spiral of two materials tells a cause-and-effect story; AR data art can be this direct and this local. https://www.youtube.com/watch?v=tHgpIlfjyWM - **Wood Line** — Andy Goldsworthy (2011): A single line on the ground turns a forest into a path and a drawing at once, the simplest form of guidance AR can give. https://www.youtube.com/watch?v=97ap3DwHK1o - **Horizon Field** — Antony Gormley (2010): All figures sit on one invisible contour line, so the landscape itself draws a horizontal line through the mountains: an AR layer can reveal a hidden datum in the same way. https://www.youtube.com/watch?v=RT7_-XyDHEA - **Lake Baikal** — Jim Denevan (2010): The drawing's scale is so large that only aircraft can see it, raising the question of who an AR work is for when its full view is private. https://www.youtube.com/watch?v=3CkmjXuZpsw - **Dismemberment, Site 1 (Gibbs Farm)** — Anish Kapoor (2009): A single stretched surface across a valley shows how one bold shape can frame an entire landscape. https://www.youtube.com/watch?v=2CdmszOKzo0 - **Installations at Montalvo Arts Center** — Chris Drury (2009): Echoing a pattern you find in nature (veins, rivers, whirlpools) at a new scale makes the hidden structure visible, a strategy for AR generative forms. https://www.youtube.com/watch?v=sj1dSbK_Npg - **Storm King Wavefield** — Maya Lin (2009): A frozen water pattern made of earth lets people 'swim' with their bodies, a model for AR terrain that users walk through rather than look at. https://www.youtube.com/watch?v=JW0Cbrlyhcg - **Spire** — Andy Goldsworthy (2008): The work is designed to be absorbed by the forest over decades, a model for AR installations with a planned lifespan. https://www.youtube.com/watch?v=5xtjTu7TWuY - **The New York City Waterfalls** — Olafur Eliasson (2008): Showing the scaffold behind the illusion lets people see both the effect and how it is made, a stance many AR works could adopt. https://www.youtube.com/watch?v=6wUwV0eDDQI - **Tree of 40 Fruit** — Sam Van Aken (2008): An artwork that reveals its design only in one season and keeps an archive of lost varieties alive shows how living data can be an artwork. https://www.youtube.com/watch?v=uU2L5nTSHtc - **Stellar Axis: Antarctica** — Lita Albuquerque (2006): Mapping the sky onto the ground turns astronomy into a walkable place, a clear template for AR star maps anchored to terrain. https://www.youtube.com/watch?v=gvxU7GSXiqI - **Time Horizon** — Antony Gormley (2006): Keeping every figure on one absolute height makes the uneven ground visible, a way to use AR objects as measuring devices for terrain. https://www.youtube.com/watch?v=3ncox_YTRyU - **Moerenuma Park** — Isamu Noguchi (2005): A whole park conceived as one sculpture shows how land shape itself can be the interface for play, a guide for AR in parks. https://www.youtube.com/watch?v=ajgj-f-sDVM - **The Gates, Central Park** — Christo and Jeanne-Claude (2005): Repeating one element along every path turns a park's circulation into a visible drawing, a strong pattern for AR wayfinding. https://www.youtube.com/watch?v=CcSkyeXdHS4 - **Inside Australia** — Antony Gormley (2003): Scanning a whole community and placing their slimmed bodies in the desert makes a town's population visible as a landscape, something AR could do with volunteers' scans. https://www.youtube.com/watch?v=VeIDvX23HYs - **Refuges d'Art** — Andy Goldsworthy (2002): Art that you reach only by walking days through a landscape makes the journey part of the work, a model for long-distance AR trails. https://www.youtube.com/watch?v=Bgcwm27Ttfs - **Rivers and Tides** — Andy Goldsworthy (2001): The artwork includes its own disappearance: designing for how a piece decays or is taken by nature is as important as how it appears. https://www.youtube.com/watch?v=AT3lveJmjY8 - **Sculpture park and houses to watch the sunset** — Not Vital (1999): A building that exists only to frame one moment of the day is a pure example of designing an experience around time and place, as AR can. https://www.youtube.com/watch?v=-KAxEmX3a2Y - **Storm King Wall** — Andy Goldsworthy (1998): A line that dodges trees and dives under water shows how an anchored path can react to the site rather than ignore it. https://www.youtube.com/watch?v=tsezrORBFj0 - **Up and Under** — Nancy Holt (1998): Tunnels through a mound are viewfinders that tell you exactly where to stand and look, like AR markers that reward one viewpoint. https://www.youtube.com/watch?v=iaIY1ZCMIkc - **sanctuarium** — herman de vries (1997): Protecting a patch of nature and letting time do the work makes an invisible process the artwork, a stance for slow AR. https://www.youtube.com/watch?v=3vaA37kDS44 - **De Groene Kathedraal (The Green Cathedral)** — Marinus Boezem (1996): An invisible building drawn with trees is exactly an architectural overlay on land, and its 'negative' twin shows the same plan by subtraction. https://www.youtube.com/watch?v=yye2qZ-o0lk - **Tree Mountain – A Living Time Capsule** — Agnes Denes (1996): One person, one tree, one position in a pattern: a shared artwork that lasts centuries, a model for persistent multi-user AR. https://www.youtube.com/watch?v=nmVFGwNeWcc - **Solar Rotary** — Nancy Holt (1995): Using the sun as a pointer that visits dated plaques turns a plaza into a calendar, a direct model for time-based AR annotations. https://www.youtube.com/watch?v=5kO2k3SPtuc - **Wrapped Reichstag** — Christo and Jeanne-Claude (1995): Hiding a loaded symbol makes people look at it again; an AR 'wrap' filter on a monument can do the same. https://www.youtube.com/watch?v=4zYKa6xmbjQ - **The Umbrellas, Japan–USA** — Christo and Jeanne-Claude (1991): Two simultaneous sites in two countries, one colour each, show how a shared event can link distant places, a model for synchronous multi-site AR. https://www.youtube.com/watch?v=S_2kQyHgBPg - **Stones and Flies: Richard Long in the Sahara** — Richard Long (1988): Simple geometric marks (line, circle) instantly read as human in a vast landscape, a lesson in how little an AR mark needs. https://www.youtube.com/watch?v=hB_EAlSc7uE - **Dark Star Park** — Nancy Holt (1984): A shadow that fits its outline once a year is a clock, an idea AR can replicate by matching virtual shadows to the real sun. https://www.youtube.com/watch?v=K-t5FDC-J1M - **Surrounded Islands** — Christo and Jeanne-Claude (1983): Outlining existing things with a bright halo makes them visible as a set, a basic AR highlighting technique at landscape scale. https://www.youtube.com/watch?v=nfDGK_WSFro - **Vietnam Veterans Memorial** — Maya Lin (1982): Reflection puts the living among the names, and the wall grows taller as you walk in, a model for data AR that uses the viewer's body and path. https://www.youtube.com/watch?v=wuxjTxxQUTs - **Wheatfield – A Confrontation** — Agnes Denes (1982): A living field in the wrong place is a sharp confrontation of values, a template for AR that plants nature where it 'should not' be. https://www.youtube.com/watch?v=zxYjGy3csTU - **The Lightning Field** — Walter De Maria (1977): A strict grid laid over wild land makes both light and terrain readable; AR grids can do the same for invisible fields. https://www.youtube.com/watch?v=iozjJgjOsUg - **The New York Earth Room** — Walter De Maria (1977): Filling a whole room with one natural material replaces the interior with a landscape, the physical version of AR room replacement. https://www.youtube.com/watch?v=krF9DEH327w - **The Vertical Earth Kilometer** — Walter De Maria (1977): An artwork almost entirely invisible and known only by its description shows that AR can point to things we can never see, such as depth. https://www.youtube.com/watch?v=oCwu3H0m35o - **Running Fence** — Christo and Jeanne-Claude (1976): A line that follows the land for 40 kilometres makes the topography readable, a strong idea for AR lines drawn on terrain data. https://www.youtube.com/watch?v=nBVpgN4JAsE - **Sun Tunnels** — Nancy Holt (1976): The tunnels frame the sky at precise moments and places, the same logic as an AR experience that only resolves at one time and position. https://www.youtube.com/watch?v=f8OLToLVYCw - **Amarillo Ramp** — Robert Smithson (1973): A ramp that curls up and stops invites the body to walk into the sky, a strong lead-in shape for AR content that continues where the ground ends. https://www.youtube.com/watch?v=jvC8TYCqlug - **Valley Curtain** — Christo and Jeanne-Claude (1972): A thin coloured plane across a valley shows how one flat element can re-scale a whole landscape, and how weather is part of the design. https://www.youtube.com/watch?v=nDpkNFcmEWQ - **Broken Circle/Spiral Hill** — Robert Smithson (1971): Pairing a flat figure (the circle) with a climbable one (the hill) gives two viewpoints on the same work, a classic AR composition pattern. https://www.youtube.com/watch?v=7uW4BLGhhcs - **Star Axis** — Charles Ross (1971): Each step up the stair corresponds to a moment in the precession cycle, so walking becomes time travel through astronomy, a strong model for data-bound AR paths. https://www.youtube.com/watch?v=RJ1UVKOv03o - **Partially Buried Woodshed** — Robert Smithson (1970): An artwork that is meant to decay turns entropy into content, a useful stance for AR works that degrade on purpose. https://www.youtube.com/watch?v=9KXbUVekRg4 - **Spiral Jetty** — Robert Smithson (1970): A shape that you can walk along but only fully see from the air shows how AR content can reward both the ground view and the overview. https://www.youtube.com/watch?v=xWd_YGHjWKM - **Double Negative** — Michael Heizer (1969): A sculpture made of removed material shows that absence can be the object, a direct idea for diminished-reality AR. https://www.youtube.com/watch?v=1U0Q6MZmw3c - **Wrapped Coast** — Christo and Jeanne-Claude (1969): Wrapping hides detail and reveals form: a single material laid over a landscape is the physical version of an AR mesh shader. https://www.youtube.com/watch?v=lU5oBJ0zoV8 ### Bodies & figures in place Figures and objects placed where you don't expect them, changing how a place feels. - **People I Saw But Never Met** — Zadok Ben-David (2025): A crowd of strangers at a small scale lets you walk among everyone at once, like an AR visualisation of a population. https://www.youtube.com/watch?v=SS3J5j-XAfc - **Rose Wonders (Burning Man)** — Thomas Dambo (2025): A giant character in an empty desert becomes a gathering point for a temporary city, a model for AR landmarks at festivals. https://www.youtube.com/watch?v=_YrbYB2PtWs - **Long Leif, the biggest troll** — Thomas Dambo (2024): A network of characters with a story across a region shows how to design a multi-site AR world with one mythology. https://www.youtube.com/watch?v=U5Kx20pRwkY - **Sound Mirrors** — Collusion (2023): A town listens to its young people through a giant ear and a projected image. https://www.youtube.com/watch?v=U5ol3hcQY4w - **Sonic Bloom** — Yuri Suzuki (2021): Horns that let strangers talk to each other across a street. https://vimeo.com/610583719 - **The Cost of Your Words** — Fabio Lattanzi Antinori (2020): Show passers-by what their words are worth to an algorithm. https://vimeo.com/501403867 - **Water's Soul** — Jaume Plensa (2020): A 'hush' gesture aimed at a whole skyline speaks to a city at once, a model for AR figures that address a distant view. https://www.youtube.com/watch?v=jIRANB8uLkM - **Behind the Walls** — Jaume Plensa (2019): A giant figure that refuses to look makes viewers aware of their own looking, a strong idea for gaze-aware AR. https://www.youtube.com/watch?v=mXyEnS7SHyo - **Building Bridges** — Lorenzo Quinn (2019): Hands reaching across water to meet make a bridge out of a gesture, a model for shared AR interactions between two sides of a space. https://www.youtube.com/watch?v=-s2QOSx4yZM - **Coral Greenhouse** — Jason deCaires Taylor (2019): A human-made structure designed to be colonised by life flips the relation between architecture and nature, a thought for AR spaces that host real processes. https://www.youtube.com/watch?v=oSDIfIg5Fko - **Forest Giants (Bernheim Forest)** — Thomas Dambo (2019): Characters hidden along trails turn a walk into a search, the physical version of an AR scavenger hunt. https://www.youtube.com/watch?v=Db0x_ptA8FU - **KAWS:HOLIDAY Hong Kong** — KAWS (Brian Donnelly) (2019): A floating giant that moves between harbour locations shows how an AR character can appear in different places on different days. https://www.youtube.com/watch?v=eCoEU74N7r4 - **KAWS:HOLIDAY Japan** — KAWS (Brian Donnelly) (2019): Pairing a giant character with an iconic natural landmark gives every photo two anchors, a composition trick for AR. https://www.youtube.com/watch?v=JXiE03O3sW4 - **KAWS:HOLIDAY Taipei** — KAWS (Brian Donnelly) (2019): A soft, casual giant in front of a formal monument changes how the square feels, a contrast AR creators can use deliberately. https://www.youtube.com/watch?v=vq8KoNSZ7uc - **SVIATOVID** — BARTKRESA studio (Bart Kresa) (2019): A sculpture with a face on every side rewards walking around it; AR objects should give each viewing direction its own reason to look. https://www.youtube.com/watch?v=G8eV_Qy7nGQ - **Caterpillar with Green Stripe** — Reuben Margolin (2018): A creature's liveliness can come from a single traveling wave; AR characters can feel alive with nothing more than a phase-shifted loop. https://www.youtube.com/watch?v=RF0o9Em1HnY - **EVERY THING EVERY TIME** — Naho Matsuda, AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) (2018): Turn a city's live sensor data into poetry, shown on the street where it happens. https://www.youtube.com/watch?v=6bGqpGS-H88 - **Julia** — Jaume Plensa (2018): Seen from the side, the thin head almost disappears; from the front it fills the square, a lesson in designing AR objects that change with angle. https://www.youtube.com/watch?v=4W9Q45QjeME - **KAWS:HOLIDAY Seoul** — KAWS (Brian Donnelly) (2018): A giant character doing something ordinary (sunbathing) in a real place is instantly shareable, the same reason giant AR characters spread online. https://www.youtube.com/watch?v=cgtF21GOTHc - **Liverpool Mountain** — Ugo Rondinone (2018): The same object placed in a new setting (desert to harbour) changes its meaning, which AR creators can test instantly by relocating content. https://www.youtube.com/watch?v=Y81OJiom72Y - **658 prepared dc-motors, cotton balls, cardboard boxes 70x70x70cm** — Zimoun (2017): A grid of identical cells becomes a landscape of sound when each cell is slightly out of sync; in AR, repetition plus tiny randomness reads as life. https://www.youtube.com/watch?v=YF1SmQcJrM8 - **Another Time (Margate)** — Antony Gormley (2017): One figure in the sea becomes a clock for the tide, showing how an AR object can visualise a slow natural rhythm by how much of it is visible. https://www.youtube.com/watch?v=3fUZA7ylRJk - **Good Fences Make Good Neighbors** — Ai Weiwei (艾未未) (2017): Spreading one theme over hundreds of city sites makes a whole city the exhibition, a model for citywide AR shows. https://www.youtube.com/watch?v=kOzRdfxRa90 - **Law of the Journey** — Ai Weiwei (艾未未) (2017): Scaling up a news image until you must walk along it turns a picture into a body experience, what AR can do with a photograph. https://www.youtube.com/watch?v=1D54wIG2a34 - **Support** — Lorenzo Quinn (2017): Giant hands that seem to hold up a real building make a warning feel physical, a strong idea for AR interaction with architecture. https://www.youtube.com/watch?v=OJjvUjczmgE - **192 prepared dc-motors, wooden sticks 2.4m** — Zimoun (2016): A forest of long lines makes a room's height and depth legible; AR designers can use vertical elements the same way to reveal space. https://www.youtube.com/watch?v=oLpwK6mhhXs - **Cement Eclipses (Utsira)** — Isaac Cordal (2016): Tiny figures in a huge landscape reverse the usual scale, making viewers crouch and look closely, as small AR figures do. https://www.youtube.com/watch?v=G3N3_-Hzbc4 - **Giants, Rio 2016** — JR (2016): A body frozen in mid-action at the scale of a building makes the city part of the movement, a trick for AR motion stills. https://www.youtube.com/watch?v=HsILNmNMu-w - **Museo Atlántico** — Jason deCaires Taylor (2016): A procession of figures walking toward a threshold turns the sea floor into a narrative scene, a pattern for AR storytelling with frozen crowds. https://www.youtube.com/watch?v=Xhxckg-_kdI - **Rio 2016 Olympic Cauldron** — Anthony Howe (2016): A small real flame amplified by a moving reflective frame shows how AR can enlarge a modest real event rather than replace it. https://www.youtube.com/watch?v=xpZmwQD_-Wk - **Safe Passage (Konzerthaus Berlin)** — Ai Weiwei (艾未未) (2016): Covering a cultural landmark with the objects of a crisis brings a distant fact onto a local building, a strategy for AR data overlays on monuments. https://www.youtube.com/watch?v=8faWQdQ_JgY - **Asinas** — Jennifer Townley (2015): Apparent motion can emerge from two rotations; AR designers can create flowing effects without moving objects, just by phasing their parts. https://www.youtube.com/watch?v=P6EvXt-C9LI - **Heartbeat (Invasions)** — Charles Pétillon (2015): A mass of one simple element plus a slow pulse makes a building feel alive, a cheap and effective AR composition. https://www.youtube.com/watch?v=2OCAqJkjPHs - **LAND** — Antony Gormley (2015): Spreading five figures across a country connects distant sites into one work, a model for multi-location AR campaigns. https://www.youtube.com/watch?v=AEJpwbKRVBY - **Sixty Eight** — Nils Völker (2015): Slow, organic motion on a hard surface makes architecture feel alive — AR can animate walls with subtle breathing instead of loud effects. https://www.youtube.com/watch?v=l5R2BBSuCXI - **Study for Fifteen Points** — Random International (Hannes Koch & Florian Ortkrass) (2015): The brain needs only 15 moving points to see a person (biological motion) — AR characters can be radically minimal. https://www.youtube.com/watch?v=qnkxo7CWACs - **The Dappled Light of the Sun** — Conrad Shawcross (2015): A structure overhead can shape the light you walk through — outdoor AR can use the sky and sun as part of the composition. https://www.youtube.com/watch?v=n5u1RIIi0p0 - **A Million Times at Changi** — Humans since 1982 (Per Emanuelsson & Bastian Bischoff) (2014): Scale turns a clock into architecture — the same grid logic works on a phone screen or a building-sized AR overlay. https://www.youtube.com/watch?v=d2dUlZjuGj0 - **Dark Matter** — Troika (Eva Rucki, Conny Freyer & Sebastien Noel) (2014): A 3D object has as many identities as viewpoints — AR designers can hide a message that only resolves from one angle. https://www.youtube.com/watch?v=17f47By0DjI - **Di-Octo** — Anthony Howe (2014): Counter-rotating layers create depth and moiré; AR designers can use layered opposite motions to make a virtual object feel volumetric and alive. https://www.youtube.com/watch?v=UvgyHLAxgDc - **HippopoThames** — Florentijn Hofman (2014): An animal half-submerged reads as alive and moving through the real river, a good use of the water line as an AR occlusion boundary. https://www.youtube.com/watch?v=byJIXyGlWkY - **Intrude** — Amanda Parer (2014): Cute, glowing giants in a city at night are the simplest crowd-pleasing AR scene, and here they carry an ecological message. https://www.youtube.com/watch?v=0QpZ-02W-2g - **Liminal Air Space-Time** — Shinji Ohmaki (大巻伸嗣) (2014): Air is invisible until something shows it — AR can visualize flow, wind and breath through a single responsive surface. https://www.youtube.com/watch?v=UqXWbbkRiJ8 - **Moon Rabbit** — Florentijn Hofman (2014): Linking a giant figure to a local legend and festival gives it meaning beyond scale, a lesson for culturally specific AR. https://www.youtube.com/watch?v=TzN0A_UB8ww - **Ocean Atlas** — Jason deCaires Taylor (2014): A figure that carries the boundary between two worlds (water and air) is a strong image for AR works that sit on a surface between realities. https://www.youtube.com/watch?v=k81odhXg2Lw - **Shogyo Mujo (with Josh Harker)** — BARTKRESA studio (Bart Kresa) (2014): A neutral white form is the perfect AR target: one physical shape can carry infinite materials and stories. https://www.youtube.com/watch?v=yD99kirMs48 - **Street installations** — Mark Jenkins (2014): Realistic bodies in impossible poses test people's reactions, which is the social experiment behind every public AR character. https://www.youtube.com/watch?v=XeT11IfxIyM - **A Million Times** — Humans since 1982 (Per Emanuelsson & Bastian Bischoff) (2013): Many identical rotating parts can move like a flock — emergent patterns from simple rules are an efficient way to animate AR grids. https://www.youtube.com/watch?v=XdaKTnqotbE - **Cubes** — Jennifer Townley (2013): Slow phase drift between identical parts creates a narrative arc of order, chaos and order again, a simple structure for time-based AR pieces. https://www.youtube.com/watch?v=PQnHcoyhBs0 - **Forever Bicycles** — Ai Weiwei (艾未未) (2013): Thousands of identical parts create moiré-like patterns that change with the viewer's position, a direct recipe for instanced AR structures. https://www.youtube.com/watch?v=5Rj3x7vKU4Y - **Human Nature** — Ugo Rondinone (2013): Crude, ancient-looking figures in a polished business plaza create strong contrast, a design cue for AR figures that do not blend in. https://www.youtube.com/watch?v=Jlk2SVAtdAk - **Skytower** — Rob Mulholland (2013): Steel drawn like wind-blown willow shows how a form can record a force, a starting point for AR sculptures shaped by real wind data. https://www.youtube.com/watch?v=QVUayqFIOIw - **Stickwork** — Patrick Dougherty (2013): Line drawn in space with natural material shows how volumetric sketching can make architecture; AR 3D drawing apps can reach the same feeling. https://www.youtube.com/watch?v=2WpVAq0qb-c - **Symphonie Cinétique – The Poetry of Motion** — ART+COM Studios (Joachim Sauter) (2013): Movement itself can be the melody — AR scenes feel alive when objects move in time with sound rather than simply appearing. https://vimeo.com/70938823 - **The Shape of a Circle in the Mind of a Square** — Troika (Eva Rucki, Conny Freyer & Sebastien Noel) (2013): Rotation reveals that circle and square can be the same object — a clear demo of why AR must be judged from many viewpoints. https://www.youtube.com/watch?v=eEUkYyhnB4M - **ArcelorMittal Orbit** — Anish Kapoor (2012): A tower that twists around itself turns climbing into looping motion, a reference for AR landmarks you move through rather than look at. https://www.youtube.com/watch?v=3xwVfgnJyeI - **Breathing Flower** — Choi Jeong-hwa (최정화) (2012): One slow looping motion gives an object life, a minimal animation principle for AR sculptures. https://www.youtube.com/watch?v=K0FS7Z_EMt4 - **Cement Eclipses (Málaga)** — Isaac Cordal (2012): Small interventions that reward attention are a subtle alternative to loud AR content. https://www.youtube.com/watch?v=j_CDLuqlFgw - **Discovering Columbus** — Tatzu Nishi (2012): Changing the room around a monument changes what it means; AR portals can wrap a statue in an unexpected interior the same way. https://www.youtube.com/watch?v=rF76AmzRGxs - **Hyper-Matrix (Hyundai Motor Group Pavilion, Expo 2012 Yeosu)** — Jonpasang (전파상) (2012): A wall that physically moves is shocking because walls never move — AR can make solid architecture appear to breathe. https://www.youtube.com/watch?v=nGz1vyCNe4I - **In Cloud Light III** — Anthony Howe (2012): Reflection plus rotation make a sculpture borrow colour from the sky; AR materials that reflect the real environment gain the same site-specific shimmer. https://www.youtube.com/watch?v=JTWqo5H-aig - **Kinetic Rain** — ART+COM Studios (Joachim Sauter) (2012): A slow, synchronized swarm above people's heads turns a transit space into a place to pause — AR can use the ceiling as a stage, not just the floor. https://vimeo.com/45188800 - **Levitated Mass** — Michael Heizer (2012): Walking under something heavy that seems to float creates real bodily tension, a cue AR can borrow with carefully scaled overhead objects. https://www.youtube.com/watch?v=MD_6az-OI_A - **Thirty Six** — Nils Völker (2012): A tiny grid is enough to read waves — sequencing matters more than resolution when designing AR motion. https://www.youtube.com/watch?v=GW39zLyr1Jc - **275 prepared dc-motors, filler wire 1.0mm** — Zimoun (2011): Covering an architectural surface with one repeated gesture reframes the whole room; AR can do the same by instancing one tiny behaviour across a scanned mesh. https://www.youtube.com/watch?v=sGkSrYx6mZE - **Cloud Cities** — Tomás Saraceno (2011): Clusters of connected cells suggest how an AR city could grow modularly in the air above us. https://www.youtube.com/watch?v=-VWAeLxq8gE - **Echo** — Jaume Plensa (2011): A dreaming face among skyscrapers slows the pace of a busy street, showing how an AR figure can set a mood rather than deliver information. https://www.youtube.com/watch?v=J7nrzfkwGNQ - **Inside Out Project** — JR (2011): A simple, repeatable format that anyone can deploy turns an artwork into a global platform, the dream of every social AR template. https://www.youtube.com/watch?v=NgagcqxMB5U - **Lookout Rabbit** — Florentijn Hofman (2011): A giant figure you can enter and look out of turns a sculpture into a viewpoint, an idea for AR creatures that become portals. https://www.youtube.com/watch?v=umn8NRidbes - **Steelman** — Florentijn Hofman (2011): A sleepy giant toy in a housing estate gives a neighbourhood a friendly mascot, a gentle model for community AR characters. https://www.youtube.com/watch?v=I2GhmlB4HOI - **The Merlion Hotel** — Tatzu Nishi (2011): Turning a public icon into a private room flips the scale of intimacy, a prompt for AR rooms built around landmarks. https://www.youtube.com/watch?v=yDYlr47crIk - **Thixotropes** — Troika (Eva Rucki, Conny Freyer & Sebastien Noel) (2011): Motion can create volume that isn't there — a single moving line in AR can suggest a full solid. https://www.youtube.com/watch?v=nYfP4qGLy94 - **6 Times** — Antony Gormley (2010): Placing a figure at intervals along a river turns a walk into a narrative sequence, a structure location-based AR tours can copy. https://www.youtube.com/watch?v=9sNoyjInDJs - **Border Crossers** — Chico MacMurtrie / Amorphic Robot Works (2010): A figure that physically reaches across a real line gives the line meaning; AR can place bodies that bridge real boundaries on site. https://www.youtube.com/watch?v=8yeyn_8PSPU - **Circle of Animals/Zodiac Heads** — Ai Weiwei (艾未未) (2010): Re-staging lost heritage objects at monumental size in a public square asks who owns them, a question AR restitution projects also face. https://www.youtube.com/watch?v=kWNFuG4BTfU - **ClockClock** — Humans since 1982 (Per Emanuelsson & Bastian Bischoff) (2010): A familiar object used in an unfamiliar grid becomes a new display — AR can recombine everyday objects as pixels. https://www.youtube.com/watch?v=lMdloSwiNbQ - **Exposure** — Antony Gormley (2010): The figure reads as a body from far away and as pure structure up close, a lesson in designing AR objects for two viewing distances. https://www.youtube.com/watch?v=I3MG5UIi0hs - **Nelson's Ship in a Bottle** — Yinka Shonibare (2010): An answer placed on a plinth beside an old monument rewrites the square's story, the way AR counter-monuments can. https://www.youtube.com/watch?v=voEgrnPqKxo - **Sunflower Seeds** — Ai Weiwei (艾未未) (2010): A landscape made of countless individual, handmade units shows the one and the many at once, a strong idea for AR crowds and data points. https://www.youtube.com/watch?v=PueYywpkJW8 - **The Silent Evolution (MUSA)** — Jason deCaires Taylor (2010): A crowd that nature slowly transforms makes time and ecology visible, a model for AR figures that age with their environment. https://www.youtube.com/watch?v=oip5M3IJ4bI - **Dream** — Jaume Plensa (2009): A calm face on a scarred landscape changes what the place means for a whole town, showing that an AR landmark can carry collective memory. https://www.youtube.com/watch?v=qHuEnPjo6Rk - **AD INFINITUM** — Ralfonso (2008): A form that reads differently from every angle rewards walking around it, the key behaviour to design for in world-anchored AR. https://www.youtube.com/watch?v=ioOzYWPKNyk - **Cloud** — Troika (Eva Rucki, Conny Freyer & Sebastien Noel) (2008): A display wrapped around a sculptural form reads as a living skin — AR content feels more physical when it follows the curvature of a real object. https://www.youtube.com/watch?v=42hgPLL8IrA - **Evolution and Theory** — Zadok Ben-David (2008): A crowd of tiny strange figures on sand becomes a world to lean over and explore, like a tabletop AR diorama. https://www.youtube.com/watch?v=os4bvw8n7zU - **Kinetic Sculpture – The Shape of Things to Come** — ART+COM Studios (Joachim Sauter) (2008): Hundreds of points, each moving only up and down, can draw any 3D shape — the physical ancestor of AR point clouds and voxel morphs. https://vimeo.com/8554267 - **Tree Huts** — Tadashi Kawamata (川俣正) (2008): Small structures attached to real surfaces at unexpected heights are a physical preview of facade-anchored AR content. https://www.youtube.com/watch?v=MacwNa-vP88 - **Untitled Sound Objects (with Pe Lang)** — Zimoun (2008): Small, framed sound objects show that an AR piece can be intimate and wall-sized rather than room-sized. https://www.youtube.com/watch?v=DMPfed58xb0 - **Women Are Heroes** — JR (2008): Pasting local faces on the architecture they live in gives a community authorship of the view, a model for participatory AR portraits. https://www.youtube.com/watch?v=ji2rv5yw5Hw - **Event Horizon** — Antony Gormley (2007): A figure on a skyline makes people look up and scan the city, showing that one well-placed AR character can redirect a crowd's attention. https://www.youtube.com/watch?v=d6Y9-tgSUsQ - **Face 2 Face** — JR (2007): Putting two faces side by side that people expect to see apart creates empathy, a pairing strategy AR face filters rarely use. https://www.youtube.com/watch?v=4u_G0G6Jog4 - **Magic Wave** — Reuben Margolin (2007): Seeing the mechanism beside the effect teaches AR designers to expose the 'engine' of an effect as part of the experience. https://www.youtube.com/watch?v=4ZjAQkkqtZI - **Rubber Duck** — Florentijn Hofman (2007): Scaling up a bath toy makes the city look like a bathtub, a perfect example of how size change alone creates a new story in AR. https://www.youtube.com/watch?v=G4-2UUz4Px4 - **Little People** — Slinkachu (2006): Street details become full stories at 1:87 scale; AR can re-scale everyday surfaces into worlds in the same way. https://www.youtube.com/watch?v=VqTSqUOHTtg - **Sixteen Birds** — Chico MacMurtrie / Amorphic Robot Works (2006): Moving creatures along fixed paths overhead fills a big volume with life using very simple motion — a cheap trick for AR flocks. https://www.youtube.com/watch?v=3xxTv3-DWUI - **Square Wave** — Reuben Margolin (2005): Two simple sine waves added together already produce a surface that feels alive; AR motion can be built from the same small set of summed waves. https://www.youtube.com/watch?v=PvcZ2DffaYA - **Crown Fountain** — Jaume Plensa (2004): A giant face that suddenly does something physical (spits water) creates delight and play, a good cue for AR characters that trigger real-world effects. https://www.youtube.com/watch?v=zGYTvFFgBAQ - **RedBall Project** — Kurt Perschke (2001): One simple object that deforms to each site's geometry is the ideal spatial-mapping demo, and people come back daily to see where it goes next. https://www.youtube.com/watch?v=kNSpgFoUR6E - **Sphère (Esfera)** — Julio Le Parc (2001): A volume built from suspended particles is both an object and a cloud, a good way to present AR objects that should feel light and permeable. https://www.youtube.com/watch?v=O2WIqtDg11U - **Annular Eclipse** — George Rickey (2000): Rare alignments are rewarding moments; AR works can design for the chance moment when parts line up from the viewer's point of view. https://www.youtube.com/watch?v=dUIJmM2psKY - **Wind Wand** — Len Lye (2000): A single flexible line on a coastline makes the invisible wind visible to the whole city; AR can visualise invisible forces with the simplest form. https://www.youtube.com/watch?v=HH2gjPPi0XQ - **Maman** — Louise Bourgeois (1999): A giant creature you can walk under creates awe and a feeling of being protected, a scale relationship AR creatures can reproduce. https://www.youtube.com/watch?v=UTsFuZ80OyY - **Angel of the North** — Antony Gormley (1998): A single giant figure on a hill becomes a landmark people navigate by, a reminder that the scale and silhouette of an AR landmark matter more than detail. https://www.youtube.com/watch?v=i79ZI_KMzjw - **The Tarot Garden** — Niki de Saint Phalle (1998): A walkable set of characters from one symbolic system turns a landscape into a story you move through, a model for AR theme trails. https://www.youtube.com/watch?v=fLcmyExFqdM - **Another Place** — Antony Gormley (1997): Many identical figures facing one horizon turn a whole beach into a stage: the same trick an AR scene can use by spreading copies of one anchor across a site. https://www.youtube.com/watch?v=MN1M3p14c8M - **Cory's Yellow Chair** — Arthur Ganson (1997): The exploded view is a powerful AR move: taking a real object apart in space and reassembling it explains it and delights at once. https://www.youtube.com/watch?v=fFG-Lk9c2CI - **Field for the British Isles** — Antony Gormley (1993): Tens of thousands of eyes looking back at you reverse who is watching whom, a strong idea for AR crowds that react to gaze. https://www.youtube.com/watch?v=ZP-2wUaXnjU - **Machine with Concrete** — Arthur Ganson (1992): Motion can be made to disappear into deep time; an AR piece can show a process whose end lies far beyond a human life. https://www.youtube.com/watch?v=5q-BH-tvxEg - **Puppy** — Jeff Koons (1992): A giant cute figure with a living, changing skin shows how an AR object's surface can follow seasons. https://www.youtube.com/watch?v=DdUyxFRwz_g - **Margot's Cat** — Arthur Ganson (1991): Timing and anticipation make inanimate objects feel intentional; AR animations should borrow the rhythm of a real creature rather than smooth tweens (year approximate). https://www.youtube.com/watch?v=a6aicIcQJvc - **Concert for Anarchy** — Rebecca Horn (1990): Hanging a familiar object upside down and letting it erupt on a timer turns a museum room into suspense; AR objects gain drama from long stillness and sudden, rare events. https://www.youtube.com/watch?v=l1y1d7AcNNY - **Two Lines Oblique Gyratory II** — George Rickey (1989): Two lines moving in wind are enough to animate a whole landscape; AR objects outdoors can be kept minimal and let the environment drive them. https://www.youtube.com/watch?v=Hih0BDuKGik - **Machine with Wishbone** — Arthur Ganson (1988): Giving a found object a gait gives it a character; AR can animate everyday objects with minimal, well-chosen motion. https://www.youtube.com/watch?v=4pZXoayEL78 - **Spoonbridge and Cherry** — Claes Oldenburg and Coosje van Bruggen (1988): An everyday object at giant scale makes a whole garden feel like a tabletop, the most common and effective trick in consumer AR. https://www.youtube.com/watch?v=nb37ZIP4NV4 - **Wild Carrot II** — George Rickey (1987): Abstract steel that moves like a plant borrows the behaviour of nature without imitating its look; AR plants can do the same with motion alone. https://www.youtube.com/watch?v=GqzjY_l8_2c - **Fontaines (Sphérades), Palais-Royal** — Pol Bury (1985): Reflective, slowly moving objects blend into their surroundings by mirroring them; AR objects with real-time environment reflections sit more naturally in a place. https://www.youtube.com/watch?v=wVyVdvV1XsA - **Kleines Federrad (Small Feather Wheel)** — Rebecca Horn (1982): A single slow opening gesture, borrowed from an animal, is enough to make a machine feel alive; AR objects can borrow the same body language. https://www.youtube.com/watch?v=ZOeO6vB1uxU - **Fontaine (Fondation Maeght)** — Pol Bury (1978): Movement driven by slowly accumulating weight creates anticipation; AR can build suspense with gradual build-ups that release in a single gesture. https://www.youtube.com/watch?v=8vl6onFHh7o - **Méta-Harmonie** — Jean Tinguely (1978): Letting visitors see every moving part that makes a sound turns noise into music; AR sound objects are more engaging when their mechanism is visible. https://www.youtube.com/watch?v=-Xp2jjSaRVg - **Flip and Two Twisters (Trilogy)** — Len Lye (1977): Motion and sound from the same material feel inseparable; AR objects gain presence when their sound comes directly from how they move. https://www.youtube.com/watch?v=A8TXkUSxB5Y - **Sculptures à cordes** — Pol Bury (1974): Motion at the edge of perception creates unease and attention; AR can use nearly imperceptible change to reward patient looking. https://www.youtube.com/watch?v=DYzYSDz8vo0 - **TV Buddha** — Nam June Paik (白南准) (1974): A closed loop between object, camera and screen makes watching itself the subject; AR mirror and selfie effects can be designed around who is watching whom. https://www.youtube.com/watch?v=MVwR1n0IZAs - **Finger Gloves and body extensions** — Rebecca Horn (1972): Extending a finger by a metre changes how a whole room feels; AR tools that lengthen reach or add body parts do the same, and the body should feel them first. https://www.youtube.com/watch?v=6uEkq3IBIf0 - **Musical sculptures (Musicales)** — Takis (1966): An instrument played by magnetism instead of hands makes invisible forces audible; AR can sonify invisible data or fields in the same spirit. https://www.youtube.com/watch?v=jBK_m68TVf0 - **Narcissus Garden** — Yayoi Kusama (草間彌生) (1966): Many small mirrors scattered over a landscape multiply it into countless tiny worlds, a direct recipe for instanced reflective AR objects. https://www.youtube.com/watch?v=HxMwouPTU7M - **Heureka** — Jean Tinguely (1964): A machine that performs on a public schedule gathers people at a place and time; AR works in public space can use timed appearances to create shared moments. https://www.youtube.com/watch?v=DpbegMX2m6o - **Homage to New York** — Jean Tinguely (1960): A work designed to exist only as a one-time event shows that ephemerality and failure can be the experience; AR can also stage content that happens once and is gone. https://www.youtube.com/watch?v=6dgGu2w3Qvo - **Magnetic sculptures (Télésculptures)** — Takis (1960): Showing an invisible force through its effect on objects is exactly how AR can make fields, signals or data feel physical. https://www.youtube.com/watch?v=Q5ktkR-xSoM - **Blade** — Len Lye (1959): Pushing a simple material to its limit makes an event; AR effects can build tension by slowly increasing energy until something rings or snaps. https://www.youtube.com/watch?v=MAjXg3RkUxs - **Fountain** — Len Lye (1959): Rigid material moving like liquid surprises the eye; AR effects can play with material expectations by giving solid things fluid motion. https://www.youtube.com/watch?v=5L9gvFmZJVY - **Strutturazione pulsante (Pulsating Structure)** — Gianni Colombo (1959): A surface that breathes by moving small pieces is a physical pixel display; AR can make real walls appear to pulse with the same displacement idea. https://www.youtube.com/watch?v=vb_p3SxIBak - **Black Mobile with Hole** — Alexander Calder (1954): A few balanced parts moving at different speeds create endless, never-repeating compositions; ambient AR objects can rely on simple physics instead of animation. https://www.youtube.com/watch?v=Nj-V1rUM75k - **Triple Gong** — Alexander Calder (1948): Sound triggered by chance collisions makes a moving object feel alive and unpredictable; AR physics objects can generate sound the same way. https://www.youtube.com/watch?v=l0kybG2OabA - **Two Spheres** — Alexander Calder (1931): Two dots moving in space are enough to suggest orbits and relationships; minimal AR motion can carry a whole story. https://www.youtube.com/watch?v=pqWi5HDB72Q ### Light & space Light as a material: rooms, skies and volumes you walk into. - **Lucida** — Lachlan Turczan (2025): Make light feel like a material you can touch and shape. https://www.youtube.com/watch?v=Rne2JgYM0s8 - **NARCISSE** — NONOTAK (Noemi Schipfer & Takami Nakamoto) (2025): Light that watches its own reflection move. https://www.youtube.com/watch?v=rXw_K5YlPOU - **Solid Light (Tate Modern)** — Anthony McCall (2024): When light has volume, the audience becomes part of the image: other people's bodies interrupting the beam are content, which is how shared AR scenes should feel. https://www.youtube.com/watch?v=1OVzjGO5iS4 - **Thanet Warn(m)ing** — Aphra Shemza (2024): Stand inside sixty years of local warming, felt as colour and volume. https://www.youtube.com/watch?v=ZT3MwIFmRx4 - **Your Voices** — Es Devlin (2023): A slowly rotating object lets text be read from any side of a plaza; in AR, rotating an anchored object can replace UI for showing many messages in one place. https://www.youtube.com/watch?v=_ZAm4Zvg0XE - **Come Home Again** — Es Devlin (2022): Drawing hundreds of named species on one lit object makes an abstract list feel like a congregation; AR can turn data about a place into figures that surround the visitor. https://www.youtube.com/watch?v=p5rGKOhkwIY - **Silent Fall** — Studio Swine / A.A. Murakami (Azusa Murakami & Alexander Groves) (2022): Constant gentle falling, like snow, is calming and endlessly watchable — AR ambient effects work best at slow, steady rates. https://www.youtube.com/watch?v=c-nBDKNiOdo - **World Lines (Colour Light Waves)** — Paul Friedlander (2022): Persistence of vision can draw a 3D shape in thin air; many 'holographic' AR effects are just this — a fast-moving line that the eye fuses into a volume. https://www.youtube.com/watch?v=4BxK173q2XI - **Between Light** — Children of the Light (Christopher Gabriel & Arnout Hulskamp) (2019): A single line of light becomes a horizon that rises and sets inside a church. https://vimeo.com/493404347 - **Fragment Shadow** — Shunichi Kasahara (2019): Several projectors split one person's shadow into multiple colored shadows. https://vimeo.com/322671485 - **Light Wave Tree (Skopje)** — Paul Friedlander (2019): An organic form built from pure motion reads as alive; AR plants and creatures can also be made from motion rather than from detailed models. https://www.youtube.com/watch?v=IMBguNvpoHc - **TEMPEST** — Matt DesLauriers (2019): One button lets a passer-by throw lightning across a whole building. https://vimeo.com/317006348 - **LUX FORMAE (Solid Light Festival, Rome)** — Bordos.ArtWorks (László Zsolt Bordos) (2018): Pure geometric light that follows the building's own lines can be more powerful than illustration; AR overlays can simply trace and emphasise real structure. https://www.youtube.com/watch?v=wQynpuovlIc - **SKALAR (with Kangding Ray)** — Christopher Bauder / WHITEvoid (2018): Mirrors multiply a few light sources into a whole architecture — reflective virtual surfaces can multiply the reach of AR content. https://www.youtube.com/watch?v=8-hxsm8lDps - **Seconds Pass** — Aphra Shemza, Tim Murray-Browne (2018): A statistic becomes a heartbeat of light you cannot look away from. https://www.youtube.com/watch?v=NhKyJ1yPlVw - **TRANSITO** — Children of the Light (Christopher Gabriel & Arnout Hulskamp) (2018): Waves of light make a corridor feel infinite and pull you along it. https://vimeo.com/278128603 - **GRID (with Robert Henke)** — Christopher Bauder / WHITEvoid (2017): Tilting a line of light turns it into a plane — a few controllable lines can sketch whole architectures around the viewer. https://www.youtube.com/watch?v=gT2Mnbs8vYs - **Light Harp (Shambala Festival)** — Paul Friedlander (2017): Turning an instrument into light makes sound visible; AR music tools can show vibration as form instead of as a flat visualiser. https://www.youtube.com/watch?v=3cMw1ZAor54 - **Origin of String Theory (Phaeno)** — Paul Friedlander (2017): A physics idea becomes understandable when you can watch it happen in 3D; AR science lessons can make abstract waves visible around a real object. https://www.youtube.com/watch?v=1j0SblgkNbQ - **Sea of Time – TOHOKU** — Tatsuo Miyajima (宮島達男) (2017): Letting each participant set one parameter of their own light makes a shared memorial personal; multi-user AR can collect such small choices into one collective piece. https://www.youtube.com/watch?v=I7YyVE1touE - **Spinning Cosmos** — Paul Friedlander (2017): A simple physical mechanism plus darkness can create a sense of scale far larger than the object; AR scenes gain the same power from a dark, reduced surrounding. https://www.youtube.com/watch?v=hoGE_-4kvA0 - **Tree of Ténéré** — Studio DRIFT (2017): One large, lit landmark in an empty landscape becomes a meeting place; AR can put a single glowing anchor in open space for people to gather around. https://www.youtube.com/watch?v=bFz6i3GqeBY - **Warping Halos** — Children of the Light (Christopher Gabriel & Arnout Hulskamp) (2017): A single spinning ring of light turns into a halo that bends the room. https://vimeo.com/233282594 - **Crossing** — Anthony McCall (2016): Projecting down instead of across turns light into standing figures: vertical volumes feel like companions, a cue for sizing AR presences to the human body. https://www.youtube.com/watch?v=yNbitJwJzDs - **DEEP WEB (with Robert Henke)** — Christopher Bauder / WHITEvoid (2016): Lines only appear when light hits an object in the air — AR can use real or virtual 'anchors' so that graphics connect points in space. https://www.youtube.com/watch?v=n2bf8L7G6WE - **Gravity and Grace** — Shinji Ohmaki (大巻伸嗣) (2016): A single light inside a patterned shell can fill every surface of a room, including people — AR light projection should land on bodies too. https://www.youtube.com/watch?v=g46PZLXk01A - **LUCID** — 3hund (Friedrich van Schoor & Tarek Mawad) / Glowing Atoms (2016): One luminous line placed in a vast landscape reframes the whole view; a single, well-placed AR primitive can do more than a crowded scene. https://vimeo.com/186269874 - **Light Barrier Third Edition** — Kimchi and Chips (2016): A larger Light Barrier where light condenses in fog into volumetric images and objects. https://vimeo.com/218354021 - **Moving Stained Glass** — Red Paper Heart (2016): Stained glass that moves, lit only by the sun. https://vimeo.com/151795207 - **Our Colour Reflection** — Liz West (2016): Looking down to see up: a floor of reflective tiles lets people explore a ceiling; AR floor layers can reveal what is above or behind the viewer. https://www.youtube.com/watch?v=k74_GTdTiYg - **Our Spectral Vision** — Liz West (2016): Colour can explain science: pairing a spectrum of light with how different animals see it is a model for AR that shows the world through another species' eyes. https://www.youtube.com/watch?v=DbpB9oOzj3w - **Our Time** — United Visual Artists (UVA) (2016): Changing the tempo of a whole space changes how long a minute feels — AR can shape subjective time, not just space. https://www.youtube.com/watch?v=lNDMno_Gids - **untitled (dawn to dusk)** — Robert Irwin (2016): The work is the changing light, not the building: an AR piece can be designed around time of day, looking different every hour at the same place. https://www.youtube.com/watch?v=hgXeWyOd6LU - **Atmosphere** — Chris Fraser (2015): Putting the light source at the centre and people around it makes every visitor's shadow part of the image; AR effects that use the viewer's silhouette invite participation. https://www.youtube.com/watch?v=StJYBpOYqhc - **Nebula** — Reuben Margolin (2015): A volume of points moving together reads as one creature; AR particle clouds gain presence when they move as coherent waves rather than noise. https://www.youtube.com/watch?v=RDQsp9pPnjM - **RGB|CMY Kinetic** — ART+COM Studios (Joachim Sauter) (2015): Colour appears where light paths cross — an AR designer can design with virtual light and shadow mixing, not just with coloured objects. https://vimeo.com/386253674 - **Swarm Study / VII** — Random International (Hannes Koch & Florian Ortkrass) (2015): Collective motion reads as life at any scale — a few hundred AR particles following flocking rules can feel like creatures. https://www.youtube.com/watch?v=ajV8A5Y2_dE - **The Diffusion of Light** — Jayson Haebich (2015): One beam of white light turned into a room full of colour by broken glass. https://vimeo.com/127403161 - **The Infinite Crystal Universe** — teamLab (2015): A phone can be a remote control for a physical light volume shared by a crowd; AR can hand the same shared space to many users at once. https://www.youtube.com/watch?v=3LFegfRrwtg - **yellowbluepink** — Ann Veronica Janssens (2015): Colour can be something you are inside rather than something you look at; volumetric AR effects around the viewer shift the experience from watching to being immersed. https://www.youtube.com/watch?v=qxVIZAHFs6M - **EPILOG** — Schnelle Bunte Bilder (with kling klang klong) (2014): Move and the light around you turns into walls and tunnels. https://vimeo.com/99909498 - **Elastic Light** — Jayson Haebich (2014): A gesture interface made of pure light in mid-air. https://vimeo.com/103230160 - **Foresta Lumina** — Moment Factory (2014): A story told in light along a forest trail at night. https://www.youtube.com/watch?v=AIMcZtSUiFo - **Lichtgrenze** — Christopher Bauder / WHITEvoid (2014): Marking a vanished boundary on the real map, then letting people release it, is location-based AR in physical form. https://www.youtube.com/watch?v=Sq0Ecpakna4 - **Light Barrier** — Kimchi and Chips (2014): A mirror array bends projected light into haze to form floating shapes of light. https://www.youtube.com/watch?v=Dp7c_0v2TRw - **Momentum** — United Visual Artists (UVA) (2014): Movement of light makes time tangible — you feel the rhythm of the room through shadows passing over your own body. https://www.youtube.com/watch?v=B5FjivaKSyA - **Shylight** — Studio DRIFT (2014): An object that opens and closes feels like it breathes and has moods; AR objects gain life from simple cycles of growth and retreat. https://www.youtube.com/watch?v=LE9NPhGEUF4 - **UNION 3** — Ralfonso (2014): Designing for both day and night states is essential in outdoor AR, where lighting conditions change what content reads well. https://www.youtube.com/watch?v=Mhv469IURPk - **Arcades** — Troika (Eva Rucki, Conny Freyer & Sebastien Noel) (2013): Light should travel in straight lines, so when it looks curved we stop and look — AR can break physical expectations in small, elegant ways. https://www.youtube.com/watch?v=WMzHXYQrfgQ - **Aten Reign** — James Turrell (2013): Layered, softly lit rings erase edges and make a building feel bottomless; AR can use nested translucent shells to change how deep a real ceiling looks. https://www.youtube.com/watch?v=Wih1K71hYlk - **In Passing** — Chris Fraser (2013): A slit turns a lamp into a drawing tool in space; in AR, thin light planes that respond to walking can guide people through a building. https://www.youtube.com/watch?v=A9lJ3vhTztk - **Light Leaks** — Kyle McDonald, Jonas Jongejan (2013): Dozens of disco balls reflect projector light and turn a whole room into a space of drifting light points. https://vimeo.com/66167082 - **Line Segments Space** — Kimchi and Chips (2013): A web of nylon threads in the dark, lit by precise projection into flowing 3D shapes. https://vimeo.com/111610020 - **Marquee** — Philippe Parreno (2013): A familiar signal with its content removed makes people notice the signal itself; AR UI can be exhibited as an empty frame to make users feel anticipation. https://www.youtube.com/watch?v=h8pMX6jO2Z4 - **Netykavka** — INITI (Dan Gregor) (2013): Make a beam of light feel like a solid object you can touch. https://vimeo.com/72222918 - **Submergence** — Squidsoup (2013): A sparse grid of points is enough to make 3D forms you can stand inside; AR point clouds can create presence without heavy meshes. https://www.youtube.com/watch?v=aJHRHa1D6OE - **The Bay Lights** — Leo Villareal (2013): An existing structure's own geometry can be the display; AR at city scale works best when it follows the lines of real infrastructure. https://www.youtube.com/watch?v=dcTkhDMyIk4 - **Vanishing Point** — United Visual Artists (UVA) (2013): Perspective is a construction you can stand inside — AR designers can make the vanishing point itself a visible object. https://www.youtube.com/watch?v=RAlhQZJRlTA - **test pattern [100m version]** — Ryoji Ikeda (池田亮司) (2013): Pure data rendered at human scale becomes a landscape; a floor-anchored AR layer can turn any stream (network traffic, sound) into terrain you walk on. https://www.youtube.com/watch?v=RZ-dLYmoGW4 - **49 Nord 6 EST 68 VEN 12 FL** — Doug Wheeler (2012): Remove corners and shadows and a room becomes infinite; in AR, hiding the edges of virtual content is often more convincing than adding detail. https://www.youtube.com/watch?v=aBJv_SdIgUQ - **Buckyball** — Leo Villareal (2012): A molecule enlarged to the size of a pavilion becomes a gathering place; AR can blow up tiny scientific structures into public sculptures that people walk around. https://www.youtube.com/watch?v=gyDJArirU7I - **Laser interaction prototype** — Jayson Haebich (2012): Laser beams you can touch. https://vimeo.com/44904580 - **Twilight Epiphany** — James Turrell (2012): Colour next to a window changes the colour we see through it; an AR layer placed around a real view can re-tint that view without touching it. https://www.youtube.com/watch?v=GrD5ylEn1wQ - **Exploded Views** — Jim Campbell (2011): The eye completes very low-resolution images; AR does not need high fidelity to be convincing, and distance-based detail can be a design choice. https://www.youtube.com/watch?v=W4T5kECTYaE - **The Walking Cube** — 1024 Architecture (François Wunschel & Pier Schneider) (2011): Combine small physical motion with light to give a simple shape a personality. https://vimeo.com/131077465 - **Your rainbow panorama** — Olafur Eliasson (2011): Where you stand decides what filter you see: in AR, mapping a colour or effect to compass direction turns walking into the interface. https://www.youtube.com/watch?v=ZsMCfOW0SRA - **Falling Light** — Troika (Eva Rucki, Conny Freyer & Sebastien Noel) (2010): Caustics make light look like a liquid — animated light spots on the real floor are a cheap, convincing AR effect. https://www.youtube.com/watch?v=jfTp1ltayik - **Mobility – Reflective Kinematronic II** — ART+COM Studios (Joachim Sauter) (2010): One light source plus many angled reflectors can write anywhere in a room — AR designers can think of reflection as a way to place content on distant surfaces. https://vimeo.com/67559505 - **Ocean of Light** — Squidsoup (2010): Think of light as a spatial medium with resolution: designing for a low-resolution 3D grid trains students to make forms readable with very few points. https://www.youtube.com/watch?v=QzXaM_KXze4 - **Scattered Light** — Jim Campbell (2010): Placing a ghostly crowd above a real park mirrors the people in it; AR can show a place's past or typical visitors as low-resolution presences. https://www.youtube.com/watch?v=9xnI8p5xGPg - **Swarm Light** — Random International (Hannes Koch & Florian Ortkrass) (2010): A swarm drawn in real 3D pixels feels alive in a way a screen never does — AR flocks placed in real space inherit this power. https://www.youtube.com/watch?v=Bhda3ASVMuE - **Within without** — James Turrell (2010): A long approach prepares the eyes: before showing AR content, a staged transition (darkness, water, a tunnel) makes the reveal much stronger. https://www.youtube.com/watch?v=biScm3kGMdY - **Your uncertain shadow (colour)** — Olafur Eliasson (2010): Five colored lamps split your shadow into a string of rainbows. https://www.youtube.com/watch?v=PeBH6fTQNSc - **Chorus** — United Visual Artists (UVA) (2009): Phase is a composition tool: identical pendulums slightly out of sync produce endless patterns — useful for any repeating AR animation. https://www.youtube.com/watch?v=dEqIbzMKpN0 - **Slow Arc Inside a Cube IV** — Conrad Shawcross (2009): Moving one light inside a simple object turns the whole room into a moving drawing — AR shadows from virtual lights can transform real spaces. https://www.youtube.com/watch?v=nxPzEa2YSFw - **The Wolfsburg Project** — James Turrell (2009): Remove every edge and texture and the brain loses depth; AR designers can use uniform fields of colour to dissolve a room, or add a single object to re-anchor it. https://www.youtube.com/watch?v=QWekIcZaKns - **Wind to Light** — Jason Bruges Studio (2009): Wiring each light to its own local sensor makes an invisible field (wind) readable at a glance; AR can overlay the same kind of field map on a real place. https://www.youtube.com/watch?v=lwLozMxaMv8 - **For the Guggenheim** — Jenny Holzer (2008): Text that follows the architecture's path turns reading into walking; AR captions and messages can be laid along a route instead of floating in front of the face. https://www.youtube.com/watch?v=neLvdTd1tWs - **Multiverse** — Leo Villareal (2008): Light moving faster or slower than the viewer changes the sense of speed; AR motion cues in a corridor can make people feel they are accelerating or floating. https://www.youtube.com/watch?v=_TMobatWhok - **bit.flow** — Julius Popp (2008): Pixels do not need a screen: any flowing material can carry an image through space, which suggests AR content that travels along real pipes, rails or paths. https://www.youtube.com/watch?v=OD19qyL9a3Q - **ATOM (with Robert Henke)** — Christopher Bauder / WHITEvoid (2007): A low-resolution grid of 64 lights is enough to make a 3D display — AR effects don't need many elements if their motion is rhythmic. https://www.youtube.com/watch?v=3SpEX2Scwso - **Breath** — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2007): A building's facade breathes in time with music. https://www.youtube.com/watch?v=A6SAIdHukGg - **You and I, Horizontal II** — Anthony McCall (2006): A slowly changing line on a flat plane becomes a moving architecture once it has depth: animate 2D shapes over time and they read as spatial rooms in AR. https://www.youtube.com/watch?v=HgzcbQlwT6w - **bit.fall** — Julius Popp (2006): Information that exists only while it falls teaches AR designers that text in space can be temporary, physical-feeling and tied to gravity. https://www.youtube.com/watch?v=gg9LWsfqqrk - **Fragile Future** — Studio DRIFT (2005): Pairing the most fragile natural object with technology makes both feel precious; AR designers can let real, delicate things carry the digital layer. https://www.youtube.com/watch?v=Jg8YH32s2Mo - **Round rainbow** — Olafur Eliasson (2005): One small rotating object can paint a whole room with light; in AR, a single animated emitter can be enough to occupy and redefine a space. https://www.youtube.com/watch?v=BuYH6nLMR-c - **Counter Void** — Tatsuo Miyajima (宮島達男) (2003): Numbers counting at different speeds can speak about life and loss; a public AR piece can carry memory by being switched on only on certain days. https://www.youtube.com/watch?v=HSO-5P1i71k - **The weather project** — Olafur Eliasson (2003): Half an object plus a mirror makes a whole one, and a mirrored ceiling turns the audience into the show: an AR scene can add the viewers themselves to the spectacle. https://www.youtube.com/watch?v=_1Vgeose43g - **Blue, Red and Yellow** — Ann Veronica Janssens (2001): Three primary lights and fog are enough to erase architecture; AR designers can create a strong sense of place with atmosphere instead of objects. https://www.youtube.com/watch?v=B5tvvN5-CCA - **Room for one colour** — Olafur Eliasson (1997): Changing the light, not the objects, changes the whole world at once; a global colour grade in AR can transform a room more strongly than any added 3D model. https://www.youtube.com/watch?v=hd077pa-5CI - **Santa Maria Annunciata in Chiesa Rossa** — Dan Flavin (1997): Light can follow the story of a building (entrance, nave, altar); AR experiences in architecture can be sequenced along the route people already walk. https://www.youtube.com/watch?v=45Ax3XGczBo - **Beauty** — Olafur Eliasson (1993): The rainbow exists only between light, droplets and your eye: effects that depend on viewer position are personal, and AR can make every visitor's version different. https://www.youtube.com/watch?v=vQ0Eq097RKA - **Signaux lumineux (Bassin Takis, La Défense)** — Takis (1990): Blinking lights at different heights in a field read as a language of signals; AR wayfinding and data layers can borrow this calm, distributed signalling. https://www.youtube.com/watch?v=Z6WjEcBXGPs - **Lightmobile** — Eric Staller (1985): Light content does not have to stay on buildings: attaching animated light to a moving object makes the street itself the stage, a cue for AR content that rides on vehicles. https://www.youtube.com/watch?v=6TxVpY_3iPc - **untitled (Dia:Bridgehampton)** — Dan Flavin (1983): Off-the-shelf lights arranged with care become art; AR designers can make strong work from default primitives (a line, a quad) instead of custom assets. https://www.youtube.com/watch?v=8bPqlQEPUO0 - **Roden Crater** — James Turrell (1979): Architecture can act as a time-based lens on the sky; an AR experience can likewise be tied to a place and a moment, opening only when sun or moon line up. https://www.youtube.com/watch?v=g0g6JFYRKxQ - **Scrim veil—Black rectangle—Natural light** — Robert Irwin (1977): A barely visible surface changes how a whole room is read; semi-transparent AR planes can re-divide real spaces without blocking them. https://www.youtube.com/watch?v=J_oHzl8qG-0 - **Line Describing a Cone** — Anthony McCall (1973): The projector beam, not the screen, is the artwork: an AR designer can treat the volume between device and surface as a place to put content. https://www.youtube.com/watch?v=1-HWsxPnNNY - **untitled (to Donna) II** — Dan Flavin (1971): A corner is the most ignored part of a room; placing light there changes the whole geometry, a reminder to use corners and edges as AR anchors. https://www.youtube.com/watch?v=c6uz8fy1sM4 - **Chronos 10** — Nicolas Schöffer (1969): A sculpture and its projected shadow play form one work; AR objects can be designed together with the light and shadow they cast on real surfaces. https://www.youtube.com/watch?v=jhWNVgya--E - **Electronic Light Ballet** — Otto Piene (1969): The same light choreography can live in a room or on a screen; AR designers can prototype spatial light pieces as simple films first. https://www.youtube.com/watch?v=M13FEIZgn0g - **Lumino** — Nicolas Schöffer (1968): A kinetic light artwork made small enough for a living room; ambient AR pieces can likewise live quietly on a shelf instead of demanding attention. https://www.youtube.com/watch?v=F2qpfmguwNs - **Chromosaturation** — Carlos Cruz-Diez (1965): Colour can be an environment you walk through instead of a surface you look at; AR colour grading of the whole passthrough view can do the same. https://www.youtube.com/watch?v=RDN_yN3rwMA - **Transchromie** — Carlos Cruz-Diez (1965): A physical filter that recolours the real world depending on where you stand is the direct ancestor of AR colour filters. https://www.youtube.com/watch?v=FvZ60oj30WQ - **Continuel mobile, lumière (A)** — Julio Le Parc (1963): Many tiny reflectors driven by air produce endless variation with no programming, a model for particle effects that feel physical rather than scripted. https://www.youtube.com/watch?v=47Q_QDZadxY - **Continuel-lumière plafond** — Julio Le Parc (1963): A few moving mirrors can turn any ceiling into a live, ever-changing display; AR can reuse a room's surfaces the same way instead of floating screens. https://www.youtube.com/watch?v=tH5wCntSfVc - **Lumière en mouvement** — Julio Le Parc (1962): Light that moves over walls and people makes the whole room, including the visitor, part of the image, which is the promise of room-scale AR. https://www.youtube.com/watch?v=NcwQ8Q087v4 - **Light Ballet (Lichtballett)** — Otto Piene (1961): One rotating lamp with holes turns an entire room into a moving display; AR can get huge spatial effects from one small source projected onto real surfaces. https://www.youtube.com/watch?v=MitGag5UUVE - **Light Prop for an Electric Stage (Light-Space Modulator)** — László Moholy-Nagy (1930): The object matters less than the light it throws into the room; in AR, the effect an object has on the space around it can be more important than the object itself. https://www.youtube.com/watch?v=QHdK19meZTk ### Projection & mapping Images projected onto buildings, objects and bodies: the ancestor of AR overlays. - **Future whiteboard: marker strokes that come alive** — Roelof Knol (2026): Handwriting becomes the trigger for light. https://x.com/tokufxug/status/2072491697557361021 - **Growing** — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2025): Let passers-by feed or disturb a spreading organism. https://vimeo.com/1159696549 - **Pose Tracking R&D on the Aston Webb Building** — Holosphere (2025): Wave your arms and a whole building moves with you. https://www.youtube.com/watch?v=HBz0KWN4GJ4 - **Shadow Tuner** — Rafael Lozano-Hemmer (2025): Passers-by cast their shadows onto a giant Earth balloon and change how the planet looks. https://vimeo.com/1067505877 - **University of Birmingham 125th Anniversary Projection Show** — Holosphere (2025): A building's facade opens to show what is inside it. https://www.youtube.com/watch?v=wt2GkcPV9DY - **Bioluminescence** — Cybersaur Arts (Josh Garner) (2024): Everyone's hand-drawn deep-sea creature ends up glowing on a heritage building. https://www.youtube.com/watch?v=V9e2UwIVXME - **Dynamicland intro** — Dynamicland — Bret Victor & collaborators (2024): Every program is a physical object in the room that you can see and touch. https://www.youtube.com/watch?v=5Q9r-AEzRMA - **Projected fish school for cats** — Roelof Knol (2024): AR for an audience that cannot read screens: a cat. https://x.com/Rainmaker1973/status/1772239372521263467 - **Voyage au Centre de la Terre (Padirac cave mapping)** — Gamgie (Clément Rignault) (2024): Use a real cave as the screen and the stage for a journey to the centre of the Earth. https://www.youtube.com/watch?v=Lsv5H5XheLk - **light lines** — Roelof Knol (2024): Real objects bend the light around them. https://x.com/orbithm/status/1812187426590200184 - **Circle intersections: interactive pendulum projection** — Roelof Knol (2023): Physics as a drawing tool: gravity animates the geometry. https://x.com/CurieuxExplorer/status/1621037555729960960 - **Improvising cellular playgrounds in Realtalk** — Dynamicland — Bret Victor & collaborators (2023): Improvise an evolving cellular-automaton playground from paper and markers. https://www.youtube.com/watch?v=nT4E5HkpLjo - **Morphose (Cie Ultreia)** — Gamgie (Clément Rignault) (2023): Let old paintings spill beyond their frames onto the room. https://www.youtube.com/watch?v=nBenQHOFNK8 - **Re-Vision (The Treasury, Petra)** — Maxin10sity (2023): On a sacred or heritage site, restraint is part of the design: light that only traces what is already there respects the place more than a new story would. https://www.youtube.com/watch?v=0VqvHN74oNA - **Augmented Shadow: Chasing Stars in Shadow** — Joon Moon (Joon Y. Moon / 문준용) (2022): Scale a tabletop shadow trick into a room-sized story that visitors light up themselves. https://vimeo.com/782602226 - **Biomolecular design in Realtalk** — Dynamicland — Bret Victor & collaborators (2022): Scientists gather around a table and design protein molecules together with paper and physical objects. https://www.youtube.com/watch?v=GmY_BrwWnCA - **Circle intersections: interactive magnets on whiteboard** — Roelof Knol (2022): Play with geometry by hand: magnets are the control points of a live diagram. https://x.com/CurieuxExplorer/status/1526756672999288832 - **Dots on paper** — Roelof Knol (2022): A single sheet of paper is enough as a screen, a controller and a stage. https://x.com/WevolverApp/status/1597712101006577664 - **Louise Bourgeois x Jenny Holzer: Projections, Basel** — Jenny Holzer (2022): Another artist's private words become public architecture; AR can carry a museum's content out into the city around it. https://www.youtube.com/watch?v=6v_nPUaa_OY - **Monuments (Virginia Tech Drillfield)** — Craig Walsh (2022): Replacing the statue with a living face on a tree asks who deserves a monument; AR can place alternative monuments in public space the same way. https://www.youtube.com/watch?v=82wypNISZpw - **NAKED Yorumode 2022 Heian Jingu** — NAKED, INC. (Ryotaro Muramatsu) (2022): Giving each visitor a handheld light makes them a participant in the show, a model for AR where the phone itself is the lantern. https://www.youtube.com/watch?v=TEHFyH44eBs - **Seaside** — 3hund (Friedrich van Schoor & Tarek Mawad) / Glowing Atoms (2022): Found objects can carry the story: a shell lit from outside becomes a small stage, and AR can pick out real objects rather than add new ones. https://vimeo.com/785576368 - **The Reading Light** — Red Paper Heart (2022): Words that exist only where the light shines. https://vimeo.com/759623238 - **Voronoi cells drawn by a pendulum** — Roelof Knol (2022): Let a pendulum, not a person, conduct a generative drawing. https://x.com/CurieuxExplorer/status/1553404644566507521 - **Innerworld Prism** — Marlena Myles (2021): A giant projection tells a dream about letting go of the self and returning to nature. https://www.youtube.com/watch?v=Er7oZ2n1u8s - **Interactive projection mapping on paper** — Roelof Knol (2021): A printed card becomes the source that the projected drawing grows out of. https://x.com/CurieuxExplorer/status/1439277743212937224 - **The Intergalactic Hanseatic League** — Collusion (2021): Historic buildings receive messages from the future about the town's climate. https://www.youtube.com/watch?v=iz8NmgJvhd0 - **VarioLight 2: Rhythmic Gymnastics** — Ishikawa Watanabe Laboratory (University of Tokyo) (2021): Projection chases a gymnast's ball and ribbon and colors them during the routine. https://www.youtube.com/watch?v=9X66YZTb_hA - **Bruegel's Creatures** — Skullmapping (Filip Sterckx & Antoon Verbeeck) (2020): Colouring and animating a real print in place shows how AR can add a layer to an existing artefact without replacing it. https://www.youtube.com/watch?v=8hellCoFdu4 - **Dynamicland (progress report)** — Dynamicland — Bret Victor & collaborators (2020): A computer the size of a building with no screens: code on paper, projections on tables, and people programming together around them. https://www.youtube.com/watch?v=x8-7E0IT5K0 - **ElaMorph Projection** — Ishikawa Watanabe Laboratory (University of Tokyo) (2020): Projection alone makes a hard object look as stretchy as rubber. https://www.youtube.com/watch?v=uWq-a52X-7g - **Embryogenesis (with Gabriel Schama)** — Limelight (Viktor Vicsek & team) (2020): A detailed physical object gives projected light something to hold on to; AR effects also read best when they follow the real depth and edges of an object. https://www.youtube.com/watch?v=9UdrqrwuR4c - **Micromonumental Mapping** — Limelight (Viktor Vicsek & team) (2020): A miniature model is a cheap, controllable stand-in for a building; a tabletop AR prototype is the fastest way to test a large site-specific idea. https://www.youtube.com/watch?v=5y02p_JcJ2s - **Rubens Cupid** — Skullmapping (Filip Sterckx & Antoon Verbeeck) (2020): An artwork that leaves its frame to guide you to the real thing is a model for AR wayfinding and museum invitations. https://www.youtube.com/watch?v=JREE37Q5cNM - **Augmented Painting** — visiophone (Rodrigo Carvalho) (2019): A still painting becomes a physics world for moving light. https://vimeo.com/317035732 - **Digital Supernova** — Miguel Chevalier (2019): Turn a cathedral vault into a sky of constantly exploding digital supernovae. https://www.youtube.com/watch?v=ZbSWZL_YZ3w - **NightBloom at the Conservatory of Flowers** — Lightform (Brett Jones & Kevin Karsch) (2019): Project onto real plants so the greenhouse 'blooms' at night. https://www.youtube.com/watch?v=45tjfOd4vzg - **Stuck inside** — Clémence Debaig (Unwired Dance Theatre) (2019): Make a flat wall of folded paper feel like it has an inside. https://www.youtube.com/watch?v=B14B_qgilMU - **Dockyard 3D Projection Mapping / Yokohama Odyssey** — Nobumichi Asai (2018): Turn a century-old stone dock into a projection stage for a seafaring epic. https://www.youtube.com/watch?v=vguPITbNorQ - **DynaFlash v2 and Post Reality** — Ishikawa Watanabe Laboratory (University of Tokyo) (2018): High-speed projection makes objects look as if their material has been rewritten, so reality itself becomes the screen. https://www.youtube.com/watch?v=QDppJ9NWtaE - **It's Lit! Projected AR Murals** — Lightform (Brett Jones & Kevin Karsch) (2018): A hand-painted mural plus projected animation makes still letters light up. https://www.youtube.com/watch?v=XTR-NhJ9em8 - **La Tabla** — Dynamicland — Bret Victor & collaborators (2018): A 'programmable' game played on a projection table with real cards and pieces. https://www.youtube.com/watch?v=VTHvNasQyu8 - **MIDAS Projection** — Ishikawa Watanabe Laboratory (University of Tokyo) (2018): Without markers or models, make a moving object look like it is made of a different material. https://www.youtube.com/watch?v=c40cxE-dfPg - **Melting Memories** — Refik Anadol (2018): Biosignals become compelling when turned into matter-like forms; AR can visualise a user's heartbeat or breath as an object in the room instead of a number. https://www.youtube.com/watch?v=__l2sl5Z1GQ - **Museo del Prado 200 years** — Onionlab (2018): A museum can show what is inside on its outside; AR can turn any building's walls into a preview of its contents. https://www.youtube.com/watch?v=H9cjG69TfNM - **Renegade Projected AR in Downtown Vegas** — Lightform (Brett Jones & Kevin Karsch) (2018): Ride a projector bike that scans and projects as it goes, turning city walls into AR murals. https://www.youtube.com/watch?v=_RFa7f8If2c - **Universe of Water Particles on a Rock where People Gather** — teamLab (2018): The body as a physical obstacle is the simplest interaction there is; AR water, smoke or crowds that flow around the user need no instruction. https://www.youtube.com/watch?v=WWuDTBpPZbA - **VarioLight** — Ishikawa Watanabe Laboratory (University of Tokyo) (2018): High-speed steering mirrors chase a dancer across the stage, landing images precisely on a moving body. https://www.youtube.com/watch?v=XEseo-orRDI - **WDCH Dreams** — Refik Anadol (2018): Let a concert hall use AI to 'dream' its hundred years of memories and project them onto its facade. https://www.youtube.com/watch?v=PuMVVsoiLPM - **AURA** — Moment Factory (2017): A church interior turned into an immersive space of light and image. https://www.youtube.com/watch?v=FV3XdOda3zM - **Game of Thrones Premiere, Walt Disney Concert Hall** — BARTKRESA studio (Bart Kresa) (2017): Free-form architecture forces content to follow curves instead of rectangles; AR on curved surfaces must be designed for the surface, not just placed on it. https://www.youtube.com/watch?v=-TEylnercdA - **INORI (Prayer)** — Nobumichi Asai (2017): Tracking and projection at a thousand frames per second keep face imagery locked onto intense dance. https://www.youtube.com/watch?v=9n0ZAwt23VU - **Jenny Holzer at Blenheim Palace** — Jenny Holzer (2017): Placing personal testimony on a monument of power creates a strong contrast; AR can layer many small human voices over an official place. https://www.youtube.com/watch?v=pKIQNbuIqpE - **Kacho-fugetsu** — Nobumichi Asai (2017): Seasonal motifs of flowers, birds, wind and moon are projected onto a moving face. https://www.youtube.com/watch?v=5cS7KLUPfZQ - **Light Capsules x Neon Museum** — Craig Winslow (2017): Fake the light of a dead object so that it looks alive again. https://vimeo.com/207339810 - **Light Sketch — El Cosmico** — Craig Winslow (2017): When the plan fails, map whatever you have with you. https://vimeo.com/242350117 - **Makeup Lamps** — Disney Research — Ivan Poupyrev, Robert Sumner & colleagues (2017): High-speed projection paints moving makeup onto a performer's face in real time. https://www.youtube.com/watch?v=Ilgu3aFCphs - **Mudança de Dança** — Harshini J. Karunaratne (2017): Scale a dancer up to the size of a building. https://vimeo.com/248122645 - **TOKYO ART CITY by NAKED** — NAKED, INC. (Ryotaro Muramatsu) (2017): Projecting onto a miniature city gives you a god's-eye view of a place, the same tabletop viewpoint that makes AR maps and city models so readable. https://www.youtube.com/watch?v=PECqN-vX_J4 - **Time Waterfall** — Tatsuo Miyajima (宮島達男) (2017): Turning a skyscraper into a clock of falling numbers shows how city-scale surfaces can hold one simple idea; geo-AR can do the same on any tall building. https://www.youtube.com/watch?v=A-JMlkH7_BA - **Cité Mémoire** — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (2016): Turning the city into a walkable anthology of site-specific scenes, with the phone carrying the sound, is a blueprint for location-based AR storytelling. https://www.youtube.com/watch?v=B_OmCLGKAWY - **Combining Shape-Changing Interfaces and Spatial AR** — David Lindlbauer (2016): Projection plus a shape-changing object makes the object seem to reach past its own edges. https://www.youtube.com/watch?v=fWREdKL2Kus - **Crystal Cloud (Fête des Lumières)** — URBANSCREEN (2016): A dialogue between crystal and cloud performed on a deconstructivist building. https://www.youtube.com/watch?v=82jYUtxbsCQ - **Dare to Dream** — Ouchhh (Ferdi Alıcı & Eylül Duranağaç) (2016): Let dreams appear on a face as a living, tracked mask of light. https://vimeo.com/160396410 - **Down the Rabbit Hole** — The Macula (2016): A façade as a rabbit hole: the building opens onto another world. https://vimeo.com/181344271 - **Drawing on the Water Surface Created by the Dance of Koi and People - Infinity** — teamLab (2016): Real water and projected fish together feel more alive than either alone; AR gains presence when it plays on a real material such as water or sand. https://www.youtube.com/watch?v=qAebQICA-fE - **Dynamic Projection Mapping onto Deforming Non-Rigid Surfaces** — Ishikawa Watanabe Laboratory (University of Tokyo) (2016): However the cloth is crumpled or folded, the projected pattern bends with it as if printed on. https://www.youtube.com/watch?v=-bh1MHuA5jU - **Gallery Invasion** — Skullmapping (Filip Sterckx & Antoon Verbeeck) (2016): When content can leave its frame and travel through the room, the room becomes the interface; AR can let characters jump from a poster into space the same way. https://www.youtube.com/watch?v=APpw6ZKIQ3I - **Hypercard in the World** — Dynamicland — Bret Victor & collaborators (2016): Turn paper cards into programs so the whole room becomes a computer people program together. https://www.youtube.com/watch?v=uI7J3II59lc - **Interconnection** — Limelight (Viktor Vicsek & team) (2016): Zooming between scales on one fixed surface is an easy-to-read storytelling device; AR can zoom through scales around a real object in the same way. https://www.youtube.com/watch?v=a1v4W95wJnM - **LEGACY (Schlosslichtspiele Karlsruhe)** — Maxin10sity (2016): A building can reveal what is inside it; AR can likewise 'open' a museum, a machine or a wall to show hidden content in place. https://www.youtube.com/watch?v=VeGZ6p37now - **Light Capsule 004 — Astoria, Oregon** — Craig Winslow (2016): Use light to temporarily restore a city's forgotten layer of history. https://vimeo.com/173006252 - **Light Capsule 013 — Detroit x Miss Van** — Craig Winslow (2016): Add a time layer to a mural: light animates what paint can only freeze. https://vimeo.com/186274660 - **Light Capsules — London Design Festival** — Craig Winslow (2016): A tour of the city's ghost signs, each lit up for one night. https://vimeo.com/183522909 - **Living Library** — Theo Watson, Emily Gobeille (2016): A giant real book whose illustrations move when you turn and touch the pages. https://vimeo.com/203193098 - **Luminoscope** — Harshini J. Karunaratne (2016): Found junk plus projection plus a hand sensor becomes a responsive sculpture. https://vimeo.com/178385157 - **Multi-player Pong on Globe4D** — Rick Companje (2016): A classic flat game wrapped onto a sphere so players stand around it. https://vimeo.com/170910332 - **Parsing our Silent Language** — Kat Sullivan (2016): Make the unspoken signals of body language visible as captions. https://vimeo.com/165629185 - **Remote Memories (with Laurent Delforge / Before Tigers)** — Yannick Jacquet (Legoman) (2016): Fragments of memory pieced into a panoramic projection you have to take in slowly. https://www.youtube.com/watch?v=v23ZJmWdmo4 - **Room2Room** — Microsoft Research — Hrvoje Benko & Andy Wilson (2016): Project a faraway person at life size onto your sofa, with no headset required. https://www.youtube.com/watch?v=2o6krhxpUGk - **TOKYO TOWER CITY LIGHT FANTASIA – Spring Concerto** — NAKED, INC. (Ryotaro Muramatsu) (2016): A window is already a see-through display; drawing on it over a real skyline is exactly the optical idea of AR glasses. https://www.youtube.com/watch?v=TwAxq5i648Q - **Unifield | Projection on Lasercut Sculpture** — Can Büyükberber (2016): Projection mapping on a laser-cut sculpture makes it look like a 4D object is passing through it. https://www.youtube.com/watch?v=H8jOvMXsZgc - **connected colors** — Nobumichi Asai (2016): Flower and bird patterns bloom live on a tracked face. https://www.youtube.com/watch?v=nMvFwC3bo_E - **t-shirt mapping** — Nicola Buttari (PROFORMA Videodesign) (2016): Clothing becomes a screen when projection follows the body. https://vimeo.com/179772378 - **Blueprint** — Joanie Lemercier (AntiVJ) (2015): Light builds and unbuilds structures on a blueprint's line drawing. https://www.youtube.com/watch?v=OozmLRPL1zQ - **Central Children's Store permanent 3D mapping** — Sila Sveta (2015): A flat wall can pretend to have depth if the animation invents it; AR on flat surfaces also works by drawing convincing fake depth. https://www.youtube.com/watch?v=gzCXYFK6IfU - **DELTΔ (Echolyse)** — Olivier Ratsi (2015): A simple geometric frame can act as a portal once light suggests depth inside it; AR portals work best when they sit in a clear physical frame. https://vimeo.com/155655808 - **DynaFlash** — Ishikawa Watanabe Laboratory (University of Tokyo) (2015): A thousand-frame projector keeps the image stuck to the paper waving in your hand. https://www.youtube.com/watch?v=L8kjdObjZpY - **ESCAPE (St. Ludmila Church, Signal Festival Prague)** — Bordos.ArtWorks (László Zsolt Bordos) (2015): Treating architecture as something that can break and release what is inside it is a strong dramatic arc for any AR piece anchored on a building. https://www.youtube.com/watch?v=ih8vLl75v7U - **Fiat Lux: Illuminating Our Common Home** — Obscura Digital (2015): Placing living nature on the most symbolic human architecture creates a strong contrast; AR overlays gain meaning from what the place already stands for. https://www.youtube.com/watch?v=jUlxYv3egLo - **Future Ruins** — Romain Tardy (2015): Use light and structure to imagine how future people will see today's buildings as ruins. https://vimeo.com/151385179 - **Globe4D** — Rick Companje (2015): Touch and spin a glowing globe to travel through space and time. https://www.youtube.com/watch?v=WUL-u_Cx6uM - **Le Petit Chef** — Skullmapping (Filip Sterckx & Antoon Verbeeck) (2015): The best AR scale is sometimes tiny: a character sized to your plate turns an ordinary object into a stage and makes the wait for food a story. https://www.youtube.com/watch?v=yBJEP4lsRFY - **Parallel Universes (Bolshoi Theatre)** — Maxin10sity (2015): Showing several 'versions' of the same real building in turn is a strong way to tell time or alternate realities; AR can do this on any street corner. https://www.youtube.com/watch?v=2dh9Zl2vQZk - **Projecting Change: Empire State Building** — Obscura Digital (2015): Scaling an animal to the size of a skyscraper makes an abstract statistic unforgettable; AR can use scale shifts to make data emotional. https://www.youtube.com/watch?v=T6FQvFJG9dc - **Projecting West** — Craig Winslow (2015): A road trip where each stop gets its own site-specific story in light. https://vimeo.com/138903302 - **Reflection Study (interactive edition)** — Zach Lieberman (2015): Move acrylic pieces on a light table and the software bounces simulated light off them to draw shapes and letters. https://vimeo.com/159142972 - **SIM/NEBULA** — The Macula, Can Büyükberber (2015): Map the inside of a concert hall so the architecture performs alongside the orchestra. https://vimeo.com/138894725 - **Spider Projection v.02 (Nuit Blanche 2015)** — 3hund (Friedrich van Schoor & Tarek Mawad) / Glowing Atoms (2015): Windows are ready-made screens with depth behind them; AR can treat every window as a portal into a hidden interior. https://vimeo.com/152629846 - **Tactum** — Madeline Gannon, Golan Levin (2015): Pinch a 3D-printable piece of jewelry into shape right on your own arm. https://www.youtube.com/watch?v=tOVommpNzPA - **Unfold 01 | Projection on Print** — Can Büyükberber (2015): A projection laid exactly over a print makes the flat image fold and breathe like a living thing. https://www.youtube.com/watch?v=pfeEgUcj0Ds - **WO1 Hologram** — Skullmapping (Filip Sterckx & Antoon Verbeeck) (2015): Placing ghostly figures in the real site of an event gives memory a location; site-specific AR memorials work the same way. https://www.youtube.com/watch?v=yKqmxWbItck - **Bioluminescent Forest (Projections in the Forest)** — 3hund (Friedrich van Schoor & Tarek Mawad) / Glowing Atoms (2014): Light only the details nature already has, like a leaf's veins or a mushroom cap, and the real world looks enchanted rather than covered; AR overlays work best the same way. https://vimeo.com/115082758 - **FUJI** — Joanie Lemercier (AntiVJ) (2014): Projection brings light and weather to a pencil drawing of Mount Fuji. https://www.youtube.com/watch?v=gVuWcyXHMoI - **Mano-a-Mano (Dyadic Projected SAR)** — Microsoft Research — Hrvoje Benko & Andy Wilson (2014): Two people face each other, both see correct 3D perspective in the projected space, and throw fireballs at each other. https://www.youtube.com/watch?v=Df7fZAYVAIE - **OMOTE** — Nobumichi Asai (2014): Turn a face into a tracked projection canvas where digital makeup moves with every expression. https://www.youtube.com/watch?v=18y9RsYNLYw - **Parade** — Dpt. (2014): Push the lamp and the shadows start to dance. https://vimeo.com/96615251 - **Queen Victoria** — Krzysztof Wodiczko (2014): A colonial monument becomes a microphone for the community living around it. https://www.youtube.com/watch?v=oRmT13QlY78 - **RoomAlive** — Microsoft Research — Hrvoje Benko & Andy Wilson (2014): Every wall and piece of furniture in the living room becomes a game board, with monsters crawling out of the couch. https://www.youtube.com/watch?v=GYkRRbP7m8s - **RoomAlive: The Other Resident** — Microsoft Research — Hrvoje Benko & Andy Wilson (2014): Use room-scale projection to stage a haunting in your own living room. https://www.youtube.com/watch?v=NXxVXQYlSXc - **SOLANUM (Augmented Sculpture)** — URBANSCREEN (2014): A layered physical sculpture and a close-fitting virtual skin merge into one. https://www.youtube.com/watch?v=ZSQ3N8FKyb8 - **iMapp Bucharest 555** — The Macula, Maxin10sity (2014): At building scale, a single illusion can make thousands of people feel the ground move. https://www.youtube.com/watch?v=tcq2gG1qxwc - **template/variant/friend/stranger** — Tony Oursler (2014): Showing the tracking mesh itself turns an invisible technology into the subject; AR face filters can reveal their landmarks instead of hiding them. https://www.youtube.com/watch?v=--d7NG_DWgk - **26th Floor (Augmented Sculpture)** — URBANSCREEN (2013): Sculpture and projection are designed together, giving a white form a flowing skin of light. https://www.youtube.com/watch?v=4BUhNy1iUhc - **Evolució** — Onionlab (2013): Abstract geometry that follows the real edges of a building teaches the eye where the structure is; AR can use simple lines to reveal architecture. https://www.youtube.com/watch?v=mhQ50QFsnMc - **IllumiRoom** — Microsoft Research — Hrvoje Benko & Andy Wilson (2013): The game bursts out of the TV: snow drifts down the living-room walls and the whole room shakes when things explode. https://www.youtube.com/watch?v=L2w-XqW7bF4 - **Interactive Video Mapping on Canvas** — Benjamin Kuperberg (2013): Pictures that only appear when someone is looking. https://www.youtube.com/watch?v=EsDzHx_JTcc - **Kinetic projection mapping** — Dpt. (2013): Projection mapping that stays glued to a moving object. https://vimeo.com/83618926 - **Memory of Form and Matter** — Chris Sugrue (2013): Project simulations onto a 3D-printed sculpture so the still object seems to breathe like a living thing. https://vimeo.com/163678148 - **Multitouch Wood Bar with NecTouch** — Benjamin Kuperberg (2013): Any piece of furniture can become a touchscreen. https://www.youtube.com/watch?v=T_Vm18xWYKM - **Onion Skin (Echolyse)** — Olivier Ratsi (2013): A projected image can cancel or exaggerate the real geometry it sits on; AR can flatten, fold or unfold a room corner in the same way. https://vimeo.com/155308126 - **Space Shuttle Endeavour Projection Mapping** — BARTKRESA studio (Bart Kresa) (2013): A real historic object is more moving than a replica; AR in museums should augment the artefact itself rather than a screen next to it. https://www.youtube.com/watch?v=_pTiq2nrvDc - **The Ark (with Squeaky Lobster)** — Romain Tardy (2013): Project onto a real grove of cacti and turn the plants into a glowing archive. https://vimeo.com/70131252 - **3D Mocap Mapping (with Motek Entertainment)** — NuFormer (2012): A live puppeteer behind a virtual giant makes a projection feel alive; AR characters gain the same presence when a hidden human drives them. https://www.youtube.com/watch?v=hwROCMIOkZo - **3D Video Mapping – Matenadaran, Yerevan** — NuFormer (2012): The best overlay tells the story of the very building it covers, which is the strongest reason to anchor AR content to a specific place. https://www.youtube.com/watch?v=vjL8zCveLdc - **Abraham Lincoln: War Veteran Projection** — Krzysztof Wodiczko (2012): Let living people borrow a monument's body to tell the stories it never told. https://www.youtube.com/watch?v=81nxZhbqv-Y - **Archifon** — INITI (Dan Gregor) (2012): Play architecture like an instrument by pointing at it. https://vimeo.com/37920250 - **Assembly** — Kimchi and Chips (2012): 5,500 hanging blocks are painted with projected pixels, letting digital forms live in real space. https://vimeo.com/42707293 - **Augmented Reality Sandbox** — Oliver Kreylos (2012): Pile up sand by hand while projection draws contour lines live; hold out a hand to make it rain down the dunes. https://www.youtube.com/watch?v=j9JXtTj0mzE - **Beamatron** — Microsoft Research — Hrvoje Benko & Andy Wilson (2012): A head-turning projector robot lets a virtual RC car drive over the real furniture in a room. https://www.youtube.com/watch?v=L9yccRm3Zu8 - **LightGuide** — Rajinder Sodhi, Microsoft Research — Hrvoje Benko & Andy Wilson (2012): Project movement cues straight onto your hand and let the hand follow the light. https://www.youtube.com/watch?v=vNaw9GpuVLQ - **Lighting the Sails (Sydney Opera House)** — URBANSCREEN (2012): The sails of the Sydney Opera House are taken apart, rebuilt and made to breathe in projection. https://www.youtube.com/watch?v=snT8psrPxmA - **MirageTable** — Microsoft Research — Hrvoje Benko & Andy Wilson (2012): A curved desk with stereo projection lets you grab virtual objects by hand and sit "across the table" from someone far away. https://www.youtube.com/watch?v=ll2K4tPD47E - **O (Omicron) (with Thomas Vaquié)** — Romain Tardy (2012): Use projection to reveal the structural beauty of a century-old concrete dome. https://vimeo.com/41486619 - **Sagrada Familia - Ode à la Vie** — Moment Factory (2012): A projection show that makes the stone walls of the Sagrada Família grow, flood and bloom. https://www.youtube.com/watch?v=RS-OTtIsBKY - **Stars (Skyway Festival, Toruń)** — Limelight (Viktor Vicsek & team) (2012): Linking a building to the local story (here, astronomy in Copernicus's city) gives abstract visuals a reason to be on that exact wall; AR content also works best when it is site-specific. https://www.youtube.com/watch?v=RjBro_MbGjo - **Stereoscopic 3D mapping, Museum of Art and History (Mapping Festival Geneva)** — Bordos.ArtWorks (László Zsolt Bordos) (2012): Adding stereo depth to a building projection makes content leave the surface and occupy the air in front of it, which is what a head-worn AR display does by default. https://www.youtube.com/watch?v=n36cpSmKt1c - **Super Mario Brush** — Cyril Diagne (2012): A paper drawing becomes a playable video game level. https://vimeo.com/46281850 - **re-flex (Z33)** — Pablo Valbuena (2012): An installation where light reveals and warps the geometry of a room. https://www.youtube.com/watch?v=ixrj2kFhGik - **Augmented Dance Floor (ADF')** — Beam'Art (Benjamin Petit & Antoine Vanel) (2011): The dancing crowd sees itself as light on the building in front of it. https://vimeo.com/26668124 - **Chase No Face / BELL** — Zach Lieberman (2011): Every graphic on the faces in this music video was projected live, with no post-production. https://vimeo.com/26649425 - **Dancing House** — Klaus Obermaier (2011): When you move, the whole old building twists and dances with you. https://www.youtube.com/watch?v=su3Zrpck4P8 - **Eyjafjallajökull** — Joanie Lemercier (AntiVJ) (2011): Projection makes a hand-drawn line mountain look like an erupting volcano. https://www.youtube.com/watch?v=hmV1rTUlJ2s - **Lit Tree** — Kimchi and Chips (2011): Use the leaves of a real tree as 3D pixels, so a visitor's hand gesture lights up inside the tree. https://vimeo.com/24049819 - **Luminous Flux** — The Macula, INITI (Dan Gregor) (2011): A city celebrates a building by letting light rewrite its stone. https://vimeo.com/26827092 - **Mécaniques Discursives (with Fred Penelle)** — Yannick Jacquet (Legoman) (2011): Projection makes the printed machine on the wall actually turn. https://www.youtube.com/watch?v=3URvRzYkSGo - **Para-Site (Mattress Factory)** — Pablo Valbuena (2011): Light projections that live like parasites on the corners of a building. https://www.youtube.com/watch?v=h4XQAOH7liE - **Santa Eulalia 3D Projection Mapping** — Onionlab (2011): Treating a façade as a material that can be cut and sewn ties the illusion to the brand story; AR effects on buildings feel stronger when the building 'behaves' like the subject. https://www.youtube.com/watch?v=QbEkt4UBmWA - **Sony PlayStation Realtime Projection Mapping** — Marshmallow Laser Feast, Memo Akten (2011): Projection mapping that follows moving objects turns an ordinary room into a film set. https://vimeo.com/34021153 - **Things are not what they seem (MIGZ Festival)** — Sila Sveta (2011): The title says it all: mapping is about making solid things look unstable, and AR can make the real world feel unsure of itself in the same way. https://www.youtube.com/watch?v=JkvKEtHl6XM - **YouTube Symphony Orchestra, Sydney Opera House** — Obscura Digital (2011): Visuals that react to live music tie a large audience to one shared moment; sound-reactive AR can make co-located viewers feel the same beat. https://www.youtube.com/watch?v=EOFuzE42z58 - **Crossings** — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2010): Let figures walk across a building's broken geometry as if it were terrain. https://vimeo.com/15550692 - **Face projection test** — Zach Lieberman, Daito Manabe / Rhizomatiks (2010): Project graphics straight onto a moving face so the face becomes a live screen. https://www.youtube.com/watch?v=o-Ke6mu-F-c - **Guggenheim YouTube Play** — Obscura Digital (2010): Mapping inside and outside at once makes one building a continuous experience; AR can link an exterior view with what happens inside. https://www.youtube.com/watch?v=hSyoEDz2MAg - **HOME (Alice Springs)** — Craig Walsh (2010): The people of a place are the best content for that place; location-based AR can make residents visible on their own streets. https://vimeo.com/35550168 - **Incursion, 25.033°N / 121.633°E (Taipei)** — Craig Walsh (2010): Overlooked infrastructure becomes a stage when something unexpected appears in it at the right scale, a strong idea for city-scale AR. https://vimeo.com/34827533 - **Mosaïka (Parliament Hill, Ottawa)** — Moment Factory (2010): A national building can hold many small personal stories; civic AR works when it layers real voices onto symbolic places. https://vimeo.com/15168693 - **Night Lights** — Zach Lieberman (2010): Turn a whole ferry building into a magnifier for the body, blowing up silhouettes and gestures five storeys high. https://vimeo.com/8525186 - **Perspective Lyrique** — 1024 Architecture (François Wunschel & Pier Schneider) (2010): Give a building a face and let the crowd's voice control its expression. https://vimeo.com/18888136 - **TETRA.TENNIS** — 1024 Architecture (François Wunschel & Pier Schneider) (2010): A sports field is a ready-made canvas whose lines can come alive. https://vimeo.com/19163299 - **The 600 Years** — The Macula (2010): Make a monument narrate its own biography by pretending to take it apart. https://vimeo.com/15749093 - **3D Video Mapping – Volvo, Frankfurt** — NuFormer (2009): A facade becomes believable 3D content only when the virtual geometry is modelled from the real one; AR occlusion and anchoring work the same way. https://www.youtube.com/watch?v=ar-sVuv63dM - **555 KUBIK** — URBANSCREEN (2009): What if a house were dreaming? Make a boxy building look like it is bending and folding. https://www.youtube.com/watch?v=H8qcml3smAA - **AntiVJ - SONGDO** — Joanie Lemercier (AntiVJ) (2009): An audiovisual projection on the buildings of New Songdo City in South Korea. https://www.youtube.com/watch?v=8tQpubnD0LQ - **Chase** — Karolina Sobecka (2009): Projected characters chase a moving car along the street. https://vimeo.com/6400072 - **Déshérence (AntiVJ)** — Romain Tardy (2009): Make a building facade peel, crumble and reassemble in projection. https://vimeo.com/12622615 - **Filmmuseum augmented sand sculpture** — Theo Watson, Emily Gobeille (2009): Use projection to unveil the future film museum on a sand sculpture step by step. https://vimeo.com/6521600 - **Humannature (Adelaide)** — Craig Walsh (2009): A face on an organic, uneven surface feels alive because the surface breaks and animates it; AR faces on real objects gain presence from the material they sit on. https://vimeo.com/8420194 - **AntiVJ - Grote Kerk** — Joanie Lemercier (AntiVJ) (2008): A live projection performance inside a cathedral. https://www.youtube.com/watch?v=OksyXOMTnG4 - **Augmented Space (The Hague City Hall)** — Pablo Valbuena (2008): Project along a building's own structural lines to bring a city hall facade to life. https://www.youtube.com/watch?v=eGzBayXZOJw - **Generative Graffiti** — Theo Watson (2008): Particle graffiti grows out of a hotel's lit windows and is projected back onto the facade. https://vimeo.com/463526242 - **Le Moulin à Images (The Image Mill)** — Robert Lepage / Ex Machina (2008): An industrial ruin becomes a history book when you project the city's past onto it; site-specific AR can turn any large surface into a storytelling wall. https://www.youtube.com/watch?v=6JsNKXwavqY - **Projection for Chicago** — Jenny Holzer (2008): A poem projected slowly on a building makes a whole street read together; location-based AR text can create the same shared, public reading. https://www.youtube.com/watch?v=QeOOFKzQxS0 - **Augmented Sculpture v1.2** — Pablo Valbuena (2007): Precise projection makes still white blocks warp through space and time. https://www.youtube.com/watch?v=5nzhV0x3_qM - **L.A.S.E.R. Tag** — Theo Watson (2007): Tag a whole building with a laser pointer: a camera tracks the dot and a projector paints giant strokes in real time. https://www.youtube.com/watch?v=LtZq2q43Jkc - **AntiVJ - projection on building** — Joanie Lemercier (AntiVJ) (2006): An early experiment projecting live visuals onto a building and playing with its structure. https://www.youtube.com/watch?v=L64-nqZsgjo - **Hungarian Parliament 1956–2006** — Limelight (Viktor Vicsek & team) (2006): A building connected to a historical event is the best screen for remembering it: AR memorials can place archive images on the very place where things happened. https://www.youtube.com/watch?v=DoMY-vbb5NI - **Wildlife** — Karolina Sobecka (2006): Project a tiger from a moving car onto roadside buildings so it runs alongside the car. https://vimeo.com/6400445 - **Pedestrian** — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2002): A tiny city of walking people appears in the pavement you are standing on. https://vimeo.com/186079084 - **The Influence Machine** — Tony Oursler (2000): Projecting onto smoke and foliage makes images feel like ghosts in a place; semi-transparent, wavering AR content can read as presence rather than as a screen. https://www.youtube.com/watch?v=OzcuLP2vkWg - **Ever Is Over All** — Pipilotti Rist (1997): Two images meeting in a corner make the viewer stitch cause and effect together; AR can split a story between two surfaces so the viewer's head turns do the editing. https://www.youtube.com/watch?v=IDydC2EYVDo - **Colors** — Tony Oursler (1995): A long, looping monologue gives a projected character a mind; in AR, voice and persistence can matter more than animation quality. https://www.youtube.com/watch?v=lq30GmG4_RA - **Electronic effigies (talking dummies)** — Tony Oursler (1994): A face projected onto the plainest object is enough to create a character with a point of view; AR characters do not need detailed bodies to feel present. https://www.youtube.com/watch?v=RNrwxYeHw-I - **Grass photographs** — Ackroyd & Harvey (Heather Ackroyd and Dan Harvey) (1990): Light as a slow printer on a living surface shows projection as a biological process, a poetic ancestor of projection AR. https://www.youtube.com/watch?v=RgUp8mZ_W7Q - **EAT** — Michael Naimark (1989): A familiar ritual (ordering, lifting the cloche) frames a virtual object so well that people accept it instantly; AR works best when it borrows an everyday script. https://www.youtube.com/watch?v=KJD381TlEIs - **Hirshhorn Museum, Washington, D.C.** — Krzysztof Wodiczko (1988): Project a body onto a building so the architecture speaks about the power around it. https://www.youtube.com/watch?v=XYih-aS6JK8 - **Displacements** — Michael Naimark (1980): Projecting a scene back onto its own shape makes it look real; anything that does not match the surface becomes a ghost, a lesson in how registration decides what feels solid in AR. https://www.youtube.com/watch?v=bMDr_CFFgWE - **Talking Head Projection** — Michael Naimark (1980): Presence comes from matching the shape and the motion of the surface to the image; an AR avatar needs a body that moves with it, not just a flat video. https://www.youtube.com/watch?v=uo8CPZdw0d4 - **The Haunted Mansion: Madame Leota and Little Leota (projected faces)** — Walt Disney Imagineering (Haunted Mansion team: Yale Gracey, Rolly Crump, Marc Davis, X Atencio) (1969): Projecting a face onto a head-shaped form gives far more presence than a flat screen; it is the root of face projection and of AR avatars anchored on physical props. https://www.youtube.com/watch?v=kaMSvaN_EM4 - **Chromointerference (Chromointerferent environment)** — Carlos Cruz-Diez (1964): Projected patterns that slide over walls and people make the body part of the display, showing how AR overlays can dress people as well as surfaces. https://www.youtube.com/watch?v=Fma9IV8ojug ### Sky, fire & drones Drawing in the sky with fireworks, smoke, balloons and drone swarms. - **America's founding story in the sky (Boston Pops)** — Nova Sky Stories (2026): A drone swarm can tell a story in scenes, like a storyboard in the sky; AR narratives outdoors also need few, bold frames that read from far away. https://www.youtube.com/watch?v=xiQuQFh13c0 - **Drone text performance, Museum of Islamic Art, Doha** — Jenny Holzer (2026): Words written in the sky are read by an entire crowd at once; AR text in the sky can do the same, but for every phone pointing up. https://www.youtube.com/watch?v=MwJdMC7g7Ug - **Electric Sky** — Studio DRIFT (2025): A swarm can imitate weather instead of pictures; AR sky content does not have to show images, it can behave like natural phenomena. https://www.youtube.com/watch?v=uApCaBR8Gl0 - **Grace for the World: the Sistine Chapel in drones** — Nova Sky Stories (2025): Moving a ceiling painting into the open sky turns a private indoor view into a public one; AR can release images from inside buildings and show them outside. https://www.youtube.com/watch?v=PozwuT-67pc - **Drones and fireworks at the Eiffel Tower** — Groupe F (2024): Drones draw precise images while fireworks give raw energy; AR design can pair a readable, precise layer with a looser particle layer for impact. https://www.youtube.com/watch?v=6VaMZJaD_Xo - **When the Sky Blooms with Sakura** — Cai Guo-Qiang (蔡国强) (2023): The same effect means something different in a place with memory; site-specific AR gains weight when its imagery answers the history of the exact location. https://www.youtube.com/watch?v=I2uIi0GT8Qg - **Earthtime 1.26 Munich** — Janet Echelman (2021): A title that names an invisible change (a shorter day) gives a floating form a reason to exist; AR works benefit from linking a visible shape to an unseen fact. https://www.youtube.com/watch?v=CYbeC1txB1g - **Forever de Young** — Judy Chicago (2021): Colour can dress an entire building for a few minutes; AR can temporarily re-skin a landmark with a transient layer rather than a permanent overlay. https://www.youtube.com/watch?v=CeScvBuGGP0 - **Virtual Sky Mapping – The Dream** — Maxin10sity (2021): This is AR by broadcast: virtual content locked to a real landmark in a camera image, which is exactly what a phone AR fireworks show would be. https://www.youtube.com/watch?v=RhbLeV2m0xE - **Fly with Aerocene Pacha** — Tomás Saraceno (2020): A human lifted by the sun over a threatened landscape links a body, a message and a place, the three things a strong location AR work needs. https://www.youtube.com/watch?v=AG_UXEXg_Mk - **KAWS:HOLIDAY Space** — KAWS (Brian Donnelly) (2020): A small figure filmed against the curve of the Earth plays with scale in the opposite direction from the giants, a reminder that AR can shrink as well as enlarge. https://www.youtube.com/watch?v=JwWDtlr2oQI - **A Purple Poem for Miami** — Judy Chicago (2019): A single dominant colour can title a whole event; limiting an AR palette to one hue makes a site-wide effect feel authored rather than noisy. https://www.youtube.com/watch?v=FV9PijQOEds - **New Horizon** — Doug Aitken (2019): A reflective object in the sky becomes a gathering point for a community, showing how a shared AR landmark can structure an event. https://www.youtube.com/watch?v=u3C7cAa6Q8A - **Pleins feux sur la Tour Eiffel** — Groupe F (2018): Firing from the landmark instead of behind it makes the building a performer; AR effects gain force when they emit from the architecture's own structure. https://www.youtube.com/watch?v=vr0qFE9DCVE - **Franchise Freedom** — Studio DRIFT (2017): Simple local rules produce a sky that feels alive; AR swarms should follow flocking rules rather than scripted paths to feel natural. https://www.youtube.com/watch?v=wCyNjt8TABk - **Memorial Rebirth** — Shinji Ohmaki (大巻伸嗣) (2017): Something that fills the air and vanishes brings strangers together — ephemeral, city-scale AR events can do the same. https://www.youtube.com/watch?v=SykNfKqt9kk - **Earthtime 1.8 London** — Janet Echelman (2016): Scientific data can become a form that floats over daily life; AR can lift abstract datasets into the air above the places people pass every day. https://www.youtube.com/watch?v=FrM1KiS6JPo - **Aerocene** — Tomás Saraceno (2015): A shared, open tool for flying without fuel turns the sky into a commons, a frame for collaborative AR in the air. https://www.youtube.com/watch?v=FPMrSkF_7BY - **As If It Were Already Here** — Janet Echelman (2015): A sky object can hold the memory of what used to be on the ground below; AR can place a trace of a vanished street right above where it was. https://www.youtube.com/watch?v=avciVzTLb9Y - **Drone 100** — Ars Electronica Futurelab (Spaxels) (2015): Going from ten points to a hundred changes what shapes can be drawn; AR designers should test at what density a cloud of points starts to read as an image. https://www.youtube.com/watch?v=7cegKFOW5fM - **Sky Ladder** — Cai Guo-Qiang (蔡国强) (2015): A single vertical line that climbs away from the viewer turns the whole sky into a destination; in AR, one tall anchored object can give an empty horizon a direction and a story. https://www.youtube.com/watch?v=4t2jLOcbZh0 - **Composition for a drone** — Mária Júdová (2014): Make a flying robot into a musical instrument whose score is space. https://vimeo.com/96177923 - **Elegy: Explosion Event for the Opening of The Ninth Wave** — Cai Guo-Qiang (蔡国强) (2014): An effect in the sky can carry grief instead of celebration; AR designers can use pace, colour and fading to set an emotional tone, not only to impress. https://www.youtube.com/watch?v=G5O7VbNmfWE - **Skies Painted with Unnumbered Sparks** — Janet Echelman (2014): Phones in the crowd can paint on a shared canvas in the sky; the same model works for AR, where many small inputs add up to one large shared image. https://www.youtube.com/watch?v=uNQ0f_Vg3T0 - **spectra** — Ryoji Ikeda (池田亮司) (2014): One vertical beam can mark a place for a whole city; AR landmarks visible from far away (a pillar, a column) help people find and gather at a location. https://www.youtube.com/watch?v=qtgvxElYA9k - **One Night Stand** — Cai Guo-Qiang (蔡国强) (2013): A city river is a ready-made stage with a long axis and a reflecting floor; AR events can use existing urban lines like rivers and bridges to organise the audience's view. https://www.youtube.com/watch?v=8vf0mJHR01k - **Spaxels** — Ars Electronica Futurelab (Spaxels) (2013): Each drone is a pixel in 3D space, so the sky becomes a display you can walk under; AR designers can think of floating points as a low-resolution volumetric screen. https://www.youtube.com/watch?v=COdkA5-ideA - **A Butterfly for Pomona** — Judy Chicago (2012): A drawing made on the ground for people watching from above uses the stands as the viewpoint; AR ground drawings should be designed for where the audience actually stands. https://www.youtube.com/watch?v=vqnNHP9SNxk - **Meet Your Creator** — Marshmallow Laser Feast, Memo Akten (2012): Flying robots as moving light pixels that draw in the air of a theater. https://www.youtube.com/watch?v=JLAKXJG1trU - **Mystery Circle** — Cai Guo-Qiang (蔡国强) (2012): A building facade can act as a launch surface, not just a screen; AR content that emerges from and returns to real architecture feels more rooted than content floating in front of it. https://www.youtube.com/watch?v=kAECemP1xWY - **the radar** — Ryoji Ikeda (池田亮司) (2012): Showing the sky's hidden data under the real sky gives people a second sky; AR star maps work best when aligned with where the stars really are. https://www.youtube.com/watch?v=WNJpQVoU5ZE - **Black Ceremony** — Cai Guo-Qiang (蔡国强) (2011): Fireworks do not need darkness: dark smoke shapes read best against a bright sky, a reminder that daylight AR content should use dark, high-contrast silhouettes instead of glow. https://www.youtube.com/watch?v=8DZgNNnhvUM - **Flight Assembled Architecture (with Gramazio Kohler)** — Raffaello D'Andrea / Verity (2011): Flying machines can build in places people cannot reach; AR can preview such aerial construction by showing each flight path and each placed piece in sequence. https://www.youtube.com/watch?v=uhfBB5mm2HM - **Her Secret is Patience** — Janet Echelman (2009): A soft, light form hung over a city makes the invisible wind visible; AR sky objects can reveal real forces by moving with them. https://www.youtube.com/watch?v=0rwy6IS0cHo - **Footprints of History** — Cai Guo-Qiang (蔡国强) (2008): A repeated mark that moves across a city makes an invisible giant walk; AR can use a sequence of simple anchored marks to suggest a presence far larger than the screen. https://www.youtube.com/watch?v=kiRyECW2UXk - **Poetic Cosmos of the Breath** — Tomás Saraceno (2007): Everyday waste turned into a flying object shows how humble materials can carry big visions, an attitude useful for low-tech AR prototypes. https://www.youtube.com/watch?v=qHOsO-IYpbw - **Transient Rainbow** — Cai Guo-Qiang (蔡国强) (2002): A familiar sky shape made by unexpected means holds attention precisely because it disappears; short, one-time AR events can be more memorable than persistent content. https://www.youtube.com/watch?v=goH0BJEYvM8 - **Atmospheres** — Judy Chicago (1970): Colour released into the air changes the mood of a whole landscape for a few minutes; AR can tint the atmosphere around viewers instead of adding objects to it. https://www.youtube.com/watch?v=Re8isNQ5S6M ### Perspective & illusion Anamorphic paintings, mirrors and tricks that only resolve from one point of view. - **La Caverne du Pont Neuf** — JR (2026): Turning a bridge into a cave is world replacement at city scale, and pairing it with a phone lens shows how physical and AR layers can tell one story. https://www.youtube.com/watch?v=RngYQyEu54k - **Optical theatre for 'From Kalila wa Dimna to La Fontaine' (Louvre Abu Dhabi)** — Pierrick Sorin (2024): A museum vitrine can host a living scene without a headset; the same framing (a box, a window, a lit set) helps AR content feel placed and intentional. https://www.youtube.com/watch?v=8IjXXyl3GwA - **Reality Rifts** — Christian Holz (2023): Remove the cause but keep the effect, and people imagine the rest. https://www.youtube.com/watch?v=68oIgasJ0hs - **The Building (Liberty Science Center)** — Leandro Erlich (2023): A proven illusion becomes a play space when you design the mirror as a stage: AR selfie effects work best when the camera image is the show and the physical set is the backstage. https://www.youtube.com/watch?v=heWqEr-MqEI - **Voronoi Depth** — Roelof Knol (2023): Fake depth in the floor, then let real objects disturb it. https://x.com/CurieuxExplorer/status/1716746012939931835 - **Depth lines on paper** — Roelof Knol (2022): A plain card becomes a portal into the table. https://x.com/CurieuxExplorer/status/1544719441790992385 - **Real-time perspective bounce detection** — Roelof Knol (2022): A bounce is enough input to make a flat table feel deep. https://x.com/Rainmaker1973/status/1531553569995952129 - **Eiffel Tower anamorphosis (Trocadéro)** — JR (2021): A single viewpoint turns a flat print into a cliff that swallows a monument; AR can use the same 'magic spot' to guide where people stand. https://www.youtube.com/watch?v=ptUCuYhYL1c - **Forest of Us** — Es Devlin (2021): One shape seen from the inside (a lung as a forest) can reframe a whole topic; AR scale shifts, putting the user inside a body or a tree, teach through the body. https://www.youtube.com/watch?v=9n-hBp3th4E - **Holo Cat: Perspective Expression by Head Tracking** — Takashi Yoshinaga (2021): Head tracking makes a cat on a flat screen look three-dimensional. https://www.youtube.com/watch?v=9kvIlsb8HQE - **La Ferita (The Wound)** — JR (2021): A fake tear in a real wall that shows the interior is exactly an AR portal, done in paper. https://www.youtube.com/watch?v=P-FWZ5-tyws - **Frame Perspective** — Olivier Ratsi (2019): An anamorphic frame anchored to real buildings is AR without a screen: it only 'works' from the viewpoint the designer chose. https://vimeo.com/314461942 - **Hello, Shadow!** — Joon Moon (Joon Y. Moon / 문준용) (2019): The light in your hand becomes a lens that reveals shadows the object does not actually cast. https://www.youtube.com/watch?v=RbwEf1QGA8U - **Mirage Gstaad** — Doug Aitken (2019): The same object in a new landscape becomes a new work, which shows why AR assets should be designed to take on the character of each site. https://www.youtube.com/watch?v=wdJP-sIGgS4 - **The Secret of the Great Pyramid** — JR (2019): An anamorphic ground image that reveals depth from one viewpoint is the analogue ancestor of AR depth illusions, and it dissolved as people walked on it. https://www.youtube.com/watch?v=rsnpm1_IXbw - **Cercles concentriques excentriques (Carcassonne)** — Felice Varini (2018): One vantage point turns a city into a single image: AR designers can mark a 'sweet spot' where scattered anchored fragments line up, and make the walk toward it part of the experience. https://www.youtube.com/watch?v=Iybcb-pKLQY - **Fragmented Plane** — Olivier Ratsi (2018): One flat image spread over many depths shows how fragile a 2D reading of 3D space is; AR designers can play with that moment of breaking. https://vimeo.com/260737007 - **Changing the Appearance of Real-World Objects by Modifying Their Surroundings** — David Lindlbauer (2017): Leave the object alone and change only the light around it, and it seems to change color. https://www.youtube.com/watch?v=2gez_joXaiE - **Descension** — Anish Kapoor (2017): A hole that seems to swallow the ground is the physical version of an AR 'portal into the floor' and shows how motion sells depth. https://www.youtube.com/watch?v=p8knuUS4w-Q - **Drifter** — Studio DRIFT (2017): Taking weight away from a heavy object breaks our expectations more than any effect; AR can make the heaviest real-looking thing float. https://www.youtube.com/watch?v=2QJ-Zn-yXs4 - **Giants: Kikito** — JR (2017): A child's gaze across a wall turns a political barrier into a human moment, showing how an image aligned with a real structure can change its meaning. https://www.youtube.com/watch?v=M_LMgzMnG7Q - **Mirage** — Doug Aitken (2017): A mirrored object is only its surroundings, reframed: a perfect physical reference for environment-reflective AR materials and 'invisible' objects. https://www.youtube.com/watch?v=Gn3BUue9nEc - **Mixed Reality room (RnD tests)** — THÉORIZ (David-Alexandre Chanel, Jonathan Richer & team) (2017): Head-tracked projection turns bare walls into windows onto virtual space without headsets. https://www.youtube.com/watch?v=NrO5WYG5QIw - **Portal** — Kuflex (Igor Tatarnikov & Denis Perevalov) (2017): Step inside a kaleidoscope and become its pattern. https://vimeo.com/246672516 - **Shape** — Olivier Ratsi (2017): An illusion that works only from one spot turns the viewer into a participant who must search; AR anchors can reward a precise viewpoint in the same way. https://vimeo.com/248975465 - **A pound of flesh for 50p** — Alex Chinneck (2016): A building that changes over weeks makes time the material, a model for AR structures that decay slowly across visits. https://www.youtube.com/watch?v=5X144p9qA1w - **Axioma** — Onionlab (2016): Pure geometry floating in front of a real façade shows how depth cues alone, without story, can hold an audience; AR primitives can be the content. https://www.youtube.com/watch?v=q2wT_WgOHJ4 - **Dynamic VR Display** — Daito Manabe / Rhizomatiks (2016): Tracking the viewer's head and redrawing the image in real time merges flat screens and real objects into one 3D space. https://www.youtube.com/watch?v=G7ZQ4KiX1JE - **JR au Louvre** — JR (2016): Printing the background onto the object makes it vanish from one viewpoint: the physical version of diminished reality in AR. https://www.youtube.com/watch?v=M9VkFxEwINY - **Mirror Maze** — Es Devlin (2016): A maze of mirrors hides where the real walls are, so projected content seems to float everywhere; AR can use the same confusion between real and reflected space. https://www.youtube.com/watch?v=IMbLlN_6AD8 - **The Cloud** — Leandro Erlich (2016): Volume can be faked with a stack of flat layers: slicing a 3D shape into printed planes is a cheap way to make holograms or 'volumetric' AR content that reads from many angles. https://www.youtube.com/watch?v=Fw3JzEJu9F4 - **Diplopia** — Onionlab (2015): Showing depth through two slightly different images reveals the core mechanism behind every stereo AR headset. https://www.youtube.com/watch?v=OJId9dEOZ2Q - **Musée de l'Homme in situ** — Georges Rousse (2015): The camera is the only true viewer: in AR the phone is also a single eye, so a shape painted for one lens can feel more magical than one that works from everywhere. https://www.youtube.com/watch?v=CB2ZIcahpaU - **Optic Cloak** — Conrad Shawcross (2015): Camouflage can be dynamic: patterns that shift with your movement make a solid building look immaterial — a lesson for diminished-reality AR. https://www.youtube.com/watch?v=QVjBKCZX-w4 - **Pulled by the Roots** — Leandro Erlich (2015): An impossible object placed in a real public space works best when it borrows a familiar machine: attach AR fantasy to real cranes, poles or bridges so the city seems to be doing it. https://www.youtube.com/watch?v=A58zmN67Z90 - **Contact** — Olafur Eliasson (2014): A single horizon line plus a mirror is enough to create a world beyond the wall; AR portals can work with equally minimal cues. https://www.youtube.com/watch?v=NXXuyotR8us - **Golden Mirror Carousel** — Carsten Höller (2014): Slowing a familiar ride down turns fun into contemplation: in AR, changing only the speed of a known motion can make viewers notice their own body and surroundings. https://www.youtube.com/watch?v=UQKF1dtvN3Y - **Gun Country** — Michael Murphy (Perceptual Art) (2014): One set of points can hold two readings from two viewpoints: AR artworks can hide a second, opposing message that appears only when the visitor walks to the other side. https://www.youtube.com/watch?v=802kLR9_cSk - **HoloFire (57fire)** — Ruofei Du (2014): Power a virtual fire with the real heat of your hand. https://www.youtube.com/watch?v=4cvCBN_ARlY - **Still** — Rob Mulholland (2014): A figure standing in water that only shows water is a clean example of camouflage through reflection, useful for subtle AR presence. https://www.youtube.com/watch?v=3VsdvqXvJYI - **Take my lightning but don't steal my thunder** — Alex Chinneck (2014): A levitating building is a classic AR demo made physical; the hidden support shows how illusions depend on concealed anchors. https://www.youtube.com/watch?v=9C-HTHAUfk0 - **Trois ellipses ouvertes en désordre** — Felice Varini (2014): A work can have two lives, fragments on the ground and a whole image from above: AR city games can hide pieces in the street and reveal the full shape only from a high viewpoint. https://www.youtube.com/watch?v=9QFJvZVmBgs - **Across the Buildings** — Felice Varini (2013): Graphics that fold over real architecture reveal its shape: an AR overlay that is designed flat from one view shows off the depth of the building from every other view. https://www.youtube.com/watch?v=HTDRz9A89lw - **Art Project in Miyagi** — Georges Rousse (2013): A single aligned shape can hold a community's memory of a place: AR memorial layers work when locals help decide where the one meaningful viewpoint is. https://www.youtube.com/watch?v=vSQtrKtdYEg - **Box** — Bot & Dolly (2013): Robot arms move the canvases and the camera, a live magic trick where projection and motion stay perfectly in sync. https://www.youtube.com/watch?v=lX6JcybgDFo - **DAYDREAM** — NONOTAK (Noemi Schipfer & Takami Nakamoto) (2013): Two layers of projected lines are enough to make a solid room feel liquid. https://www.youtube.com/watch?v=JZQmg-jP9CQ - **Dalston House (Bâtiment)** — Leandro Erlich (2013): Swap the gravity axis and let people perform for their own reflection: an AR mirror that re-orients the floor turns every visitor into an acrobat without any rendering tricks on their body. https://www.youtube.com/watch?v=AFF3BAnSNn4 - **Felice Varini at the Grand Palais** — Felice Varini (2013): Searching for the viewpoint is itself the interaction: an AR piece can reward people for moving their body to one pose instead of tapping a button. https://www.youtube.com/watch?v=-DjKShvvrjQ - **From the knees of my nose to the belly of my toes** — Alex Chinneck (2013): Deforming a real facade with real materials is exactly what AR 'melt' filters do digitally, and it shows how much credibility depends on material detail. https://www.youtube.com/watch?v=v4Uxx5F_RI8 - **Les Voyageurs** — Bruno Catalano (2013): The missing body is filled by the real landscape: the most direct physical example of diminished reality and see-through AR. https://www.youtube.com/watch?v=bZJ2sE-xyik - **TESSERACT (HyperCube)** — 1024 Architecture (François Wunschel & Pier Schneider) (2013): Use a real 3D frame to make a fourth dimension feel visible. https://vimeo.com/79702430 - **eMotion × Leap Motion – Pepper's ghost test** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2013): Your hands sculpt light that hovers in mid-air, without any headset. https://vimeo.com/71216887 - **Spider Projection / Araneola** — 3hund (Friedrich van Schoor & Tarek Mawad) / Glowing Atoms (2012): Filming something real at miniature scale and projecting it at building scale creates a believable giant; matching scale and viewpoint matters more than polish. https://vimeo.com/37176398 - **Vestige** — Rob Mulholland (2012): Figures made only of reflections are the physical form of an 'invisible' AR character that is noticed only when it moves or catches light. https://www.youtube.com/watch?v=gf4BOH9vod4 - **White Roads in the Red Matrix** — Olivier Ratsi (2012): A few lines of light can fake a deep space; AR designers can suggest depth with perspective cues alone before adding any 3D model. https://vimeo.com/52538265 - **Bending Space: the Durham Project** — Georges Rousse (2011): Buildings about to change are perfect hosts for temporary images: AR can give abandoned or transitional spaces a last visible layer before they disappear. https://www.youtube.com/watch?v=cnwTgXymWf0 - **Perceptual Shift** — Michael Murphy (Perceptual Art) (2011): Depth hides the image and one viewpoint reveals it: a cloud of AR particles can carry a message that people have to physically find, which is far more memorable than showing it. https://www.youtube.com/watch?v=fh9Hx-Uujno - **Snowflake Sculpture, Time Square Seoul** — Moritz Waldemeyer (2011): An anamorphic sculpture rewards the one person who finds the right spot. https://www.youtube.com/watch?v=dxBVrMfJtuM - **Blackfield** — Zadok Ben-David (2010): The same field tells two opposite stories depending on where you stand, a direct model for viewpoint-dependent AR. https://www.youtube.com/watch?v=F7h5blRk-V8 - **Nimbus** — Berndnaut Smilde (2010): An outdoor thing in an indoor place instantly feels uncanny; AR's strongest effects often move one familiar element into the wrong context. https://www.youtube.com/watch?v=I_Uu8epvcw0 - **Shadow works** — Kumi Yamashita (2010): The shadow, not the object, is the content: in AR a virtual light can make a real object cast an unexpected virtual shadow, a quiet way to add story to everyday things. https://www.youtube.com/watch?v=65BRB3kuvv8 - **Base 8** — Chris Sugrue (2008): Reach into a world floating in a glass reflection, and geometry grows out from between your fingers. https://vimeo.com/30834797 - **Cloud Gate** — Anish Kapoor (2006): A mirror object puts the public and the city inside the sculpture; people come to see themselves, which is also why AR mirror filters spread. https://www.youtube.com/watch?v=0r_EQ4RqaAk - **Morphovision** — Toshio Iwai (岩井俊雄) (2005): By controlling which slice of a real object is lit and when, you can deform reality itself; this is diminished and altered reality done with light. https://www.youtube.com/watch?v=GbXeybKgIyY - **La situazione antispettiva** — Olafur Eliasson (2003): Mirrors placed at exact angles turn one view into an infinite pattern; AR can use the live camera image as raw material for geometric multiplication. https://www.youtube.com/watch?v=d8yLRu2QwDk - **Sky Mirror** — Anish Kapoor (2001): Bringing the sky down to eye level with a mirror is exactly what an AR portal to the sky can do on a lawn. https://www.youtube.com/watch?v=Ee1nAvEZTwY - **Swimming Pool** — Leandro Erlich (1999): Two audiences looking at each other through one surface create the story: an AR scene can give 'inside' and 'outside' viewers different roles in the same illusion. https://www.youtube.com/watch?v=NT7gjhHq9d0 - **Titre variable n°1 (théâtre optique)** — Pierrick Sorin (1999): A tilted sheet of glass is the cheapest optical see-through display: it teaches how to place a virtual figure among real objects and how lighting decides whether it looks present. https://www.youtube.com/watch?v=O11LIyjcxiw - **Shadow sculptures (Dirty White Trash (With Gulls))** — Tim Noble & Sue Webster (1998): The gap between a messy object and its clean projection is the punchline: AR can reveal a hidden portrait inside clutter by choosing where the virtual light comes from. https://www.youtube.com/watch?v=u7DGkkxBS5w - **Upside-Down Goggles** — Carsten Höller (1994): Perception is trainable: a headset or phone passthrough filter that simply flips or delays the world is already a powerful experience, no content needed. https://www.youtube.com/watch?v=Ct3c9PzS6yE - **Lunch with a Helmet On** — Shigeo Fukuda (福田繁雄) (1987): A playful title plus one light turns junk into a joke: AR pieces that pair a mundane object with a surprising projected identity are easy to understand in five seconds. https://www.youtube.com/watch?v=aFUGx-DaKz4 - **The Haunted Mansion: Grand Hall ghosts (Pepper's ghost)** — Walt Disney Imagineering (Haunted Mansion team: Yale Gracey, Rolly Crump, Marc Davis, X Atencio) (1969): Room-scale Pepper's ghost is still the simplest 'hologram': it teaches that a virtual layer must be lit, scaled and staged against the real set to be believed. https://www.youtube.com/watch?v=Dx-_SYsjwJk - **Vibration (Blue and Black)** — Jesús Rafael Soto (1966): Motion parallax between a near layer and a fine patterned layer creates a shimmering effect that only exists while the viewer moves, a cheap trick for making AR overlays feel alive. https://www.youtube.com/watch?v=n8eaoKmo9TY - **Reverspective paintings** — Patrick Hughes (1964): Reversing depth cues makes a static object feel alive: AR designers can exploit the same conflict between shading and parallax to make flat labels or signs seem to watch the viewer. https://www.youtube.com/watch?v=I-OuPNZmX-0 - **Physichromie** — Carlos Cruz-Diez (1959): The colour you see depends on where you stand, a physical version of view-dependent materials that AR can exploit to reward walking around an object. https://www.youtube.com/watch?v=NIMnBoQcsbo ### Traces of time & motion Making invisible paths visible: flight, movement, light painting, time made into form. - **Fluctus** — Xavi Bou (2026): Changing the viewpoint (from below the sky to above the bird) reveals a new structure, a reminder that AR can offer impossible camera angles on real things. https://www.youtube.com/watch?v=83Pg95Bc12M - **Entomographies** — Xavi Bou (2024): The same trace method works at insect scale, so an AR trail effect can make the tiny motion around a flower visible. https://vimeo.com/1050040741 - **One for Sorrow** — Xavi Bou (2024): Mirroring natural traces makes people read meaning into them, a way for AR to turn random motion into a message. https://vimeo.com/941987903 - **Schiphol People's Clock** — Maarten Baas (2024): Letting many people 'be' the interface makes it communal — AR experiences can be built from crowd-contributed recordings. https://www.youtube.com/watch?v=WqxXsbquzOE - **Illuminate** — Chelsi Alise Cocking (2023): What would it be like if we could see our movement? https://vimeo.com/850306904 - **This Fragile Earth: Day to Night** — Stephen Wilkes (2023): Compressing a day of animal visits into one image reveals patterns of use no single moment shows; AR can do the same for how people use a space. https://www.youtube.com/watch?v=IlTY4KNGzAY - **Fiat Lux** — Darren Pearson (DARIUSTWIN) (2022): Story beats can be staged in real locations using only drawn light; AR stories can likewise lean on sparse, glowing marks rather than heavy 3D assets. https://www.youtube.com/watch?v=RF7GNqIQNn0 - **Field of View** — Freya Björg Olafson (2022): Your movement paints itself as a glitch trail. https://vimeo.com/721137520 - **Tous les oiseaux du monde** — Xavi Bou (2022): Showing the machine's bounding boxes on nature exposes what the camera 'sees', a useful critical move for AR object detection. https://vimeo.com/763921141 - **plant drone** — David Bowen (2022): Letting a non-human pilot draw in the sky and capturing the path with long exposure is a ready-made recipe for AR light trails. https://vimeo.com/709247945 - **Emergence** — Xavi Bou (2021): Slowing and inverting reality is enough to reveal a hidden system, a cheap and strong AR filter idea. https://www.youtube.com/watch?v=iAOUMbgTkgw - **Murmurations** — Xavi Bou (2020): Tracing a flock shows that form can emerge from many local decisions without a leader, an idea for AR swarms that react to a predator (the user). https://www.youtube.com/watch?v=hScBionqFBA - **Phaser** — Jennifer Townley (2019): The drawing is a record of motion over time; AR can leave persistent trails that turn movement into a lasting artefact. https://www.youtube.com/watch?v=vjtY-Sykf_Q - **She Lights The Night** — Darren Pearson (DARIUSTWIN) (2018): A light-painted character can light up its own surroundings; in AR, virtual characters that cast light on real surfaces feel far more grounded. https://www.youtube.com/watch?v=iOU2kjWp9jg - **Kill the Lights** — Darren Pearson (DARIUSTWIN) (2017): A character drawn only as outlines can live convincingly in a real landscape; AR characters do not need full shading to feel present. https://www.youtube.com/watch?v=HaWYh7YLazc - **SWIM: Sequential Wave Imprinting Machine** — Steve Mann (2017): Phenomenal augmented reality: make real physical phenomena, not graphics, visible exactly where they exist. https://www.youtube.com/watch?v=wKfwufI2hrQ - **Dancing with the Kinect** — Kat Sullivan (2016): Your silhouette leaves coloured echoes on the wall. https://vimeo.com/157173015 - **In 20 Steps** — Studio DRIFT (2016): Many simple moving parts can draw a single motion through time; AR can break a movement into sampled positions to make it visible and study it. https://www.youtube.com/watch?v=CFjtJBGlKM8 - **Kung Fu Motion Visualization** — Tobias Gremmler (2016): Making a movement's path visible teaches the movement; AR can leave the same traces around a real athlete or dancer for learning and spectacle. https://www.youtube.com/watch?v=RwJG62tRjGU - **Motion Exposure: whitewater kayak** — Stephen Orlando (2016): Colour-coding time along a motion path turns one stroke into a readable diagram, a visual grammar for AR motion trails. https://www.youtube.com/watch?v=gO9IQDsL2-s - **Ornithographies** — Xavi Bou (2016): Stacking time into one frame makes the invisible shape of flight visible: the basic recipe for any AR motion trail over real footage. https://vimeo.com/561365138 - **Schiphol Clock** — Maarten Baas (2016): An endless small human task in a public place gives travellers a moment of empathy — AR in transit spaces can use tiny human stories. https://www.youtube.com/watch?v=e_3KY2gWDwg - **Sisyphus** — Bruce Shapiro (2016): A single moving point leaves a readable trace in a real material; AR trails that interact with real surfaces can feel as calm and physical as this. https://www.youtube.com/watch?v=_Y3IA9fuXG0 - **toki-** — Akinori Goto (2016): Light chooses the frame: an AR designer learns that a sliced reveal can turn a dense, unreadable object into a clear animated moment. https://www.youtube.com/watch?v=4R9Enhw2Qd0 - **Blooms** — John Edmark (2015): When the flash is synchronised to the geometry, a static object becomes animation; AR depends on the same sync between rendering and real motion. https://www.youtube.com/watch?v=1cSR3FTQTyc - **Empreintes** — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2015): Make a gesture tangible by printing it into the air. https://vimeo.com/152416235 - **Quality Visualizing Tool (Kat Sullivan & Sergio Mora-Diaz, CCL 2015)** — Motion Bank / Choreographic Coding Lab (2015): Make invisible movement qualities visible, one visual rule per quality. https://vimeo.com/138653312 - **BLINK OF AN EYE** — Schnelle Bunte Bilder (with kling klang klong) (2014): See your recent past moving next to you. https://vimeo.com/101410848 - **Micromigrations** — Dennis Hlynsky (2014): The same trace method works at many scales, from insects to flocks, so a single AR effect can reveal many kinds of motion in nature. https://www.youtube.com/watch?v=ML6n2t6uy1Q - **Blue Shirt** — Gregory Barsamian (2013): Everyday objects in impossible motion are the essence of good AR surprise: familiar enough to believe, strange enough to remember. https://www.youtube.com/watch?v=mZoTY-CP8BA - **Timepiece** — Conrad Shawcross (2013): Turning a clock into a moving light makes time something you stand inside — AR can externalize invisible rhythms at architectural scale. https://www.youtube.com/watch?v=zkSl7KlnQm8 - **Forms** — Memo Akten (2012): Turning an athlete's motion into a sculpture of lines and surfaces shows how AR can make the invisible physics of a body visible around the body itself. https://www.youtube.com/watch?v=zCiTqZf170I - **Group Walk, Birmingham** — Hamish Fulton (2012): A simple shared rule for walking becomes the artwork; an AR score can guide groups through a site in the same way. https://www.youtube.com/watch?v=2XaX4ktvQ_0 - **Gunpowder drawings** — Cai Guo-Qiang (蔡国强) (2012): The drawing is the record of a few seconds of energy; AR can likewise leave a permanent trace of a brief action, turning a moment into a mark in space. https://www.youtube.com/watch?v=c-QIj7E6CR8 - **Lunar Trails** — Seb Lee-Delisle (2012): Your path through a screen game is drawn out physically on the wall. https://vimeo.com/54043239 - **Robot Readable World** — Timo Arnall (2012): The machine's view is itself a visual language; exposing tracking points and meshes can be a deliberate AR aesthetic instead of a debug mode. https://www.youtube.com/watch?v=7DWsMzyX9so - **Ballet Rotoscope** — Masahiko Sato + EUPHRATES (2011): Reveal the hidden geometry of a dance by drawing its trajectories on top of the body. https://www.youtube.com/watch?v=yzJk6ww3LD0 - **Drawing with the Body** — visiophone (Rodrigo Carvalho) (2011): Every joint of the body is a brush that paints in the air. https://vimeo.com/19142510 - **Immaterials: Light painting WiFi** — Timo Arnall (2011): Walking a sensor through the city and drawing its readings in place is situated data visualization — the core promise of AR data overlays. https://www.youtube.com/watch?v=cxdjfOkPu-E - **Loops** — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2011): Portray a person by their motion, not their appearance. https://vimeo.com/25509279 - **Study of Time** — Random International (Hannes Koch & Florian Ortkrass) (2011): Time can be felt as a moving light rather than read as numbers — AR clocks can be ambient instead of literal. https://www.youtube.com/watch?v=Qq9qhtbXv28 - **Traces** — James Alliban (2011): A Kinect turns visitors' movements into trails of light. https://www.youtube.com/watch?v=0KvCqF-dS2U - **Whatever happened, Happened** — Daniel Palacios (2011): Change we never see happen can be made visible by drawing its trace step by step — AR can do the same by recording and replaying slow processes where they happened. https://vimeo.com/27441259 - **dENiZEN — Light Painting Video** — Lichtfaktor (2011): Many people drawing together in one frame produces a density no single artist could reach; shared AR drawing sessions can build the same collective scenes. https://www.youtube.com/watch?v=RAe30VG5CI4 - **particles** — Daito Manabe / Rhizomatiks (2011): Timing lights on moving objects can draw volumes that do not exist — a physical version of persistence-of-vision effects that AR can extend with tracked objects. https://www.youtube.com/watch?v=OcDec4RRavA - **After Ghostcatching** — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2010): A dancer's captured motion becomes drawn ghosts that escape their frame. https://vimeo.com/26407428 - **Making Future Magic: iPad light painting** — BERG (Jack Schulze, Matt Webb, Timo Arnall) (2010): A moving screen can sweep out a volume: slicing a 3D model over time is a physical preview of how AR renders form in space. https://www.youtube.com/watch?v=C-dVnG8drxs - **Please Empty Your Pockets** — Rafael Lozano-Hemmer (2010): An object leaving behind its image turns a simple action into an archive of everyone who came before — a model for AR ghosts of past visitors. https://www.youtube.com/watch?v=arK7V_jDlNI - **Stainless — Shinjuku** — Adam Magyar (2010): Changing the time scale of a crowd turns strangers into portraits; AR can slow a busy place down so people notice the individuals in it. https://www.youtube.com/watch?v=hS5NBUG3BYs - **Variations on Pi (Light Drawing Machine)** — Nils Völker (2010): A number sequence becomes a visible trace when motion is recorded over time — AR trails can reveal data as drawings. https://www.youtube.com/watch?v=C5koC8cYe4Y - **Bird Watching (small brains series)** — Dennis Hlynsky (2009): A cheap camera plus frame accumulation reveals hidden patterns of motion: the simplest version of an AR trail effect. https://www.youtube.com/watch?v=BA5iOn_toVQ - **Day to Night** — Stephen Wilkes (2009): Time can be laid out across space so one glance shows a whole day: AR can map time onto a direction, letting viewers read history by looking left to right. https://www.youtube.com/watch?v=afev0ZjAhUA - **Grandfather Clock (Real Time)** — Maarten Baas (2009): A screen inside a familiar object makes it feel inhabited — AR can place 'someone inside' everyday objects. https://www.youtube.com/watch?v=aYD-CDMhnmI - **Immaterials: the ghost in the field** — Timo Arnall (2009): Invisible technical fields have shapes: AR can show the real reach of sensors, beacons and NFC so people understand where interactions work. https://vimeo.com/7022707 - **Sweepers' Clock** — Maarten Baas (2009): Time shown by a human performance feels warm and absurd — AR clocks, timers and progress bars could be performed rather than drawn. https://www.youtube.com/watch?v=0eBVFLlKO80 - **Time remap – Anamorphose temporelle** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2009): Turn time into an axis of space, so a moving body becomes a sculpture of its own recent past. https://vimeo.com/7878518 - **growth modeling device** — David Bowen (2009): Sampling a living thing at intervals and freezing each sample in space turns time into a line of objects — the core of any AR time-lapse trail. https://vimeo.com/20966301 - **Cybrid Landscape** — Chris O'Shea (2008): Footsteps in a real building erode a virtual landscape. https://vimeo.com/1330818 - **Light Graffiti in Cologne (Tracks feature)** — Lichtfaktor (2008): Graffiti made of light leaves no mark on the wall; AR street art can be just as temporary and still tied to a specific corner. https://www.youtube.com/watch?v=swPAKxTh-aM - **The Very Angry Caterpillar** — Lichtfaktor (2008): Stop-motion light drawing gives a character a real path through a real room — a low-tech prototype for AR characters that walk on real surfaces. https://www.youtube.com/watch?v=BpK5QuSUcLo - **Traces** — Chris O'Shea (2008): People's movements leave light trails in the projection, like long-exposure car lights. https://vimeo.com/1821234 - **Abundance** — Camille Utterback (2007): The paths of people crossing a plaza are drawn live as a giant projection on the building. https://www.youtube.com/watch?v=xgRFUsVVb84 - **Urban Flow** — Adam Magyar (2006): A camera that records only one line turns time into the horizontal axis; AR slit-scan filters can make the passage of people readable at a glance. https://www.youtube.com/watch?v=erkk1aPiI1k - **Temporary Printing Machine** — Random International (Hannes Koch & Florian Ortkrass) (2005): An image that fades invites people to make more — temporary AR marks are playful because they do not last. https://www.youtube.com/watch?v=YgNFpS0bxY8 - **Time Scan** — Daniel Rozin (2004): Slit-scan turns time into a spatial axis, one of the simplest ways to make motion visible in a live AR camera effect. https://vimeo.com/130223120 - **Untitled 5 (External Measures series)** — Camille Utterback (2004): People's pauses and movements leave growing brushstrokes on the wall. https://www.youtube.com/watch?v=zOydp3DXWrk - **Liquid Time Series** — Camille Utterback (2002): The closer you get to the screen, the further that patch of the picture slips back into the past. https://www.youtube.com/watch?v=qSHmx45AF_k - **Seen** — David Rokeby (2002): Letting crowds draw their own paths across a square shows the hidden choreography of a place — a classic AR trace effect waiting to be anchored on site. https://www.youtube.com/watch?v=0Ai49w7QPPY - **Die Falle** — Gregory Barsamian (1998): Showing an illusion in a science context invites people to explain what they see; AR can pair wonder with a reveal of how the trick works. https://www.youtube.com/watch?v=-rAExrwhzag - **Juggler** — Gregory Barsamian (1997): Darkness plus a timed flash hides the mechanism and leaves only the illusion; AR benefits from similarly controlling what the viewer can and cannot see. https://www.youtube.com/watch?v=koA9YxMZF9Q - **Open Shutter: Long-Exposure Construction Series** — Michael Wesely (1997): A site's history can be shown as translucent layers in one view; AR 'time layers' over buildings can borrow this ghosted look. https://www.youtube.com/watch?v=uGuSf_FuYXM - **Feral Fount** — Gregory Barsamian (1996): Physical morphing in front of your eyes is more uncanny than screen animation; AR transformations of real objects can aim for the same dream logic. https://www.youtube.com/watch?v=Ixg44H4EVeo - **Tornado** — Ned Kahn (1996): Fog is a way to see flow; AR can use particles in the same way to show otherwise invisible currents, and invite the hand to disturb them. https://www.youtube.com/watch?v=wMDM0x01gy4 - **Watch** — David Rokeby (1995): Splitting a live view into 'what moves' and 'what stays' is a simple filter that reveals time; AR reality filters can do this on the camera feed. https://www.youtube.com/watch?v=LCDRZWNU7Ck - **The Fourth Dimension** — Zbigniew Rybczyński (1988): Delaying each row of the image makes time bend bodies; slit-scan is one of the simplest and strongest time-warp effects for AR cameras. https://www.youtube.com/watch?v=LlVh0TCDDgU - **Steps** — Zbigniew Rybczyński (1987): Placing living people inside historic footage is the ancestor of AR history walks; the friction between the two times is the content. https://www.youtube.com/watch?v=vBaH03PrOhE - **Time Stratum II (時間層II)** — Toshio Iwai (岩井俊雄) (1985): Any light source with a rhythm can be a strobe; AR designers learn that timing, not hardware, is what creates animation. https://www.youtube.com/watch?v=iHHQ9SRBn1s - **Narcissus** — Norman McLaren (1983): Dancing with your own copy is a compelling interaction: a delayed or mirrored AR clone of the user makes a partner out of oneself. https://www.youtube.com/watch?v=YN9Iu0z2waI - **Seascapes** — Hiroshi Sugimoto (杉本博司) (1980): One fixed rule repeated across many places makes a whole series feel like one view: AR installations can use a constant framing to connect distant sites. https://www.youtube.com/watch?v=JWh4t67e5GM - **Tango** — Zbigniew Rybczyński (1980): Many looping recordings can share one real room if their paths are choreographed; AR spaces can hold layers of past visitors without clutter. https://www.youtube.com/watch?v=WcySR3RmenE - **Theaters** — Hiroshi Sugimoto (杉本博司) (1976): Adding up an entire film yields only light: an AR piece can show the sum of everything that happened in a place rather than any single moment. https://www.youtube.com/watch?v=ZspPvi-Dg_k - **Siluetas** — Ana Mendieta (1973): A body-shaped mark left in nature holds presence after the person has gone, a direct brief for AR silhouettes left at places. https://www.youtube.com/watch?v=xyS5gsw-A5Q - **Ballet Adagio** — Norman McLaren (1972): Slowing time exposes the mechanics of skill; AR replays of real performers at slow speed make an effective teaching layer. https://www.youtube.com/watch?v=aTlPYCz7UfY - **Pencil Mask (Bleistiftmaske)** — Rebecca Horn (1972): The face becomes a drawing tool and every small head movement leaves a mark, showing that head pose alone is expressive enough to draw with. https://www.youtube.com/watch?v=Eh9JH7daSbg - **Pas de deux** — Norman McLaren (1968): Delayed copies of a body turn movement into form; the same echo effect on a live AR body track makes dance visible as sculpture. https://www.youtube.com/watch?v=WopqmACy5XI - **A Line Made by Walking** — Richard Long (1967): The body's repeated path becomes the drawing: the purest form of a location-based trace that AR can record and replay. https://www.youtube.com/watch?v=5eVaZRaQWRQ - **Bullet through Apple** — Harold "Doc" Edgerton (1964): Violent, invisible events become calm and readable when frozen; AR replays can slow dangerous physics to a speed where people can study them safely. https://www.youtube.com/watch?v=22J7-ypzkYk - **Méta-Matic drawing machines** — Jean Tinguely (1959): A machine that draws turns an invisible process into a visible trace; AR drawing tools can make the maker, human or machine, part of the show. https://www.youtube.com/watch?v=VxoqVvQeil0 - **Milk Drop Coronet** — Harold "Doc" Edgerton (1957): Freezing one exact instant turns ordinary events into sculptures: AR can let people pause the world at a chosen moment and walk around it. https://www.youtube.com/watch?v=zOGmwd7kauE - **Neighbours** — Norman McLaren (1952): Sampling a real body at the wrong rate makes it move like a cartoon; AR frame-dropping and pose-snapping can give real people superpowers. https://www.youtube.com/watch?v=e_aSowDUUaY - **Picasso Light Drawings** — Gjon Mili (1949): A drawing in the air lasts only as long as someone records it: AR finally makes these gestures persistent and walkable. https://www.youtube.com/watch?v=X-i9eqlRzks - **Quicker 'n a Wink** — Harold "Doc" Edgerton (1940): A strobe light lets a still eye see a fast world; AR effects that flash or sample the camera feed can reveal hidden rhythms in ordinary motion. https://www.youtube.com/watch?v=gspK_Bi0aoQ - **Ein Lichtspiel Schwarz Weiss Grau** — László Moholy-Nagy (1930): Filming a light machine with overlays and multiple exposures reveals motion as layered traces, which is how AR can show time and movement in a single frame. https://www.youtube.com/watch?v=sRssg9hSYAI - **Cyclegraphs and Chronocyclegraphs** — Frank & Lillian Gilbreth (1914): A light trail is also a measurement: AR hand-tracking trails can double as feedback that teaches a smoother, shorter motion. https://www.youtube.com/watch?v=FgYC1qQ9lmA - **Smoke Machine (Mouvements de l'air)** — Étienne-Jules Marey (1901): Invisible fields become readable when you seed them with visible tracers: AR can show wind, WiFi or sound as streams of particles bending around real objects. https://www.youtube.com/watch?v=95XOEN4f8EE - **Chronophotographic Films** — Étienne-Jules Marey (1890): Slow motion reveals rules the eye cannot see, like a cat righting itself mid-air: AR replays at a fraction of real speed can teach physical skills. https://www.youtube.com/watch?v=BIKwns3y2_w - **Animal Locomotion** — Eadweard Muybridge (1887): Shooting one motion from several sides at once is volumetric capture in its earliest form — the idea behind today's AR volumetric people. https://www.youtube.com/watch?v=07x7KhuwwFE - **Flight of a Gull (zoetrope sculptures)** — Étienne-Jules Marey (1887): Motion can be stored as a ring of physical keyframes: an AR designer can animate by swapping solid sculptures rather than deforming one mesh. https://www.youtube.com/watch?v=0wXNrersYT0 - **Chronophotography on a Fixed Plate** — Étienne-Jules Marey (1882): Reduce the body to a few bright lines and a whole movement fits in one image: stick-figure trails are often clearer than full avatars in AR. https://www.youtube.com/watch?v=Q02SAf_eUmU - **The Attitudes of Animals in Motion: Athletes** — Eadweard Muybridge (1881): A measuring grid behind a moving body turns a gesture into data you can read — the same role a visible AR floor grid can play when teaching movement. https://www.youtube.com/watch?v=RReKrWokC7Q - **Zoopraxiscope** — Eadweard Muybridge (1879): Motion is an illusion built from still frames plus a shutter: an AR designer can make any looping animation feel physical by tying it to a real rotating object. https://www.youtube.com/watch?v=_eGcxp-TeFA - **The Horse in Motion (Sallie Gardner at a Gallop)** — Eadweard Muybridge (1878): Space the moments of a movement along a line and people can walk through time: lay out a motion as a row of frozen poses in AR and let the viewer's position choose the frame. https://www.youtube.com/watch?v=1X9UmOps-ag ### Responsive installations Physical installations that sense people and answer them with light, sound or motion. - **A Moving Sanctuary** — Random Studio (2026): A room that breathes with you. https://vimeo.com/1196639577 - **The Mora Constellation** — Moritz Waldemeyer (2026): A light field that notices your presence the way a candle does. https://www.youtube.com/watch?v=N9BuisYqWZM - **1D ARCADE** — 1024 Architecture (François Wunschel & Pier Schneider) (2025): A whole video game squeezed into one line of light. https://vimeo.com/1098205588 - **Anatomy of Embodied Ecologies** — Camille Dunlop (2025): Your movement becomes a force of nature inside a living digital mudflat. https://vimeo.com/1085797818 - **Buoyancé** — Ken Nakagaki (2025): Robots reel helium balloons up and down to form an interactive 3D display in mid-air. https://www.youtube.com/watch?v=v9mNKIILt_s - **Huk, the Jaguaress** — Violeta Ayala (2025): A film that looks at you, judges you and answers. https://www.youtube.com/watch?v=4cDaYOr3QBM - **Shape n' Swarm** — Ken Nakagaki (2025): Sketch a shape with your hands and say one sentence, and a robot swarm forms it and brings it to life. https://www.youtube.com/watch?v=5u0M9yL7tyY - **Chameleon** — Universal Everything (Matt Pyke) (2024): A digital chameleon creature on the street that imitates you. https://www.youtube.com/watch?v=EkGLZ_Ck_-w - **Floral Resonance** — Christian Brinkmann (2024): Make the silent relationship between a person and a plant audible and visible. https://x.com/publicartad/status/2005927172448698436 - **Hi-Tech Garden** — HsienYu Cheng (鄭先喻), AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) (2024): Let children 'hear' a plant respond when they touch it. https://vimeo.com/970162531 - **Hydrogen Wave** — Universal Everything (Matt Pyke) (2024): Push waves of 'hydrogen' particles with your body. https://www.youtube.com/watch?v=M2SOCYnWIiU - **Immersive Memory (AlUla)** — Torin Blankensmith (2024): Among the ruins of an old town, visitors' movements call up images and spatial sound from the memory of the incense route. https://x.com/blankensmithing/status/1858627004398268491 - **Prototype: Someone - a random mirror** — Universal Everything (Matt Pyke) (2024): The mirror always shows 'someone else'. https://www.youtube.com/watch?v=OS3rX7QU468 - **Talk to the Mushrooms (Mushroom Music)** — Dom Scott (2024): Augment a real forest floor with light that listens, without any screen at all. https://www.youtube.com/watch?v=VN7wOe7Wvxw - **The Keywords Karaoke** — Fabio Lattanzi Antinori (2024): Sing your neighbourhood's search history. https://vimeo.com/952736898 - **walking in circles** — Roelof Knol (2024): Your walking path rearranges the architecture of light around you. https://x.com/genmediaclub/status/1866604059240853525 - **Amazonia – The Interactive Forest** — Schnelle Bunte Bilder (with kling klang klong) (2023): Walk into a rainforest wall and let your movement uncover its stories. https://vimeo.com/813835066 - **Around 7 Meters is more fun** — HsienYu Cheng (鄭先喻) (2023): Split drawing and seeing across 7 metres so two strangers have to talk to make one image. https://vimeo.com/865813598 - **Clo(o)k: Human-Time Interactions Through a Clock That "Looks"** — Zhuoyue Lyu (2023): An everyday object that notices your attention can bend the time it shows: time rushes when ignored and stops for a conversation. https://www.youtube.com/watch?v=A3jYe0NNDAk - **Echoes** — Torin Blankensmith (2023): Every move you make summons the most similar move made by an earlier visitor. https://x.com/blankensmithing/status/1737307286572298738 - **Passaggi // Presenze** — Samuele Albani (2023): Doorways and footsteps become the score of a room. https://vimeo.com/862699079 - **Self Absorbed** — Tim Murray-Browne (2023): Navigate an AI's memory of your life with your body, not a mouse. https://www.youtube.com/watch?v=JKg-6fHRT9U - **Turning RC Cars into Pets** — Jelle Vermandere (2023): Give RC cars personalities so they wander the floor and respond to you like pets. https://www.youtube.com/watch?v=9FBi0qbLiio - **distanze** — Samuele Albani (2023): Play music with the space between two bodies. https://vimeo.com/812529965 - **the space in between** — Roelof Knol (2023): Make the empty space between people and walls the thing that reacts. https://www.youtube.com/watch?v=Ypd0NXoVUi0 - **FACING LIFE [In Doc Tri Nation]** — Klaus Obermaier (2022): Your face and movement drive a virtual child who is gradually disciplined by society. https://www.youtube.com/watch?v=DSdO6zoP1Zc - **FLUX** — Ksawery Komputery (Ksawery Kirklewski) (2022): Visualise the invisible signal that carries us across networks by turning bodies into flowing light data. https://www.youtube.com/watch?v=MKt0uAd9a1w - **INNER LIFE** — MAOTIK (Mathieu Le Sourd) (2022): Walking becomes painting and composing on a floor that behaves like a living organism. https://vimeo.com/1064295716 - **Motion-Reactive Physarum** — Uncharted Limbo Collective (George Adamopoulos, Eleana Polychronaki, Chris Waters) (2022): A living slime mould that grows toward the people moving in front of it. https://www.youtube.com/watch?v=Ir8NH9M7ZhI - **Unlearning Language** — Lauren Lee McCarthy, Kyle McDonald (2022): In a room watched constantly by AI, people have to invent new ways of communicating that machines cannot read. https://vimeo.com/780628671 - **+Panic** — Schnelle Bunte Bilder (with kling klang klong) (2021): A school of fish that only trusts you if you are calm. https://vimeo.com/654523297 - **Augmented Games** — Moment Factory (2021): Turn any space into a projected game field you can run and jump on. https://www.youtube.com/watch?v=jhLKSY6NB5I - **CMY Shadows Mirror** — Daniel Rozin (2021): Overlapping cyan, magenta and yellow shadows paint a colored mirror image of you. https://vimeo.com/1198095563 - **ENTER** — Ksawery Komputery (Ksawery Kirklewski) (2021): A doorway that shows you as signal, as if about to step through a network. https://vimeo.com/606803803 - **Infinity** — Universal Everything (Matt Pyke) (2021): The arrival of viewers changes an endless crowd of walkers. https://vimeo.com/523660693 - **Life Forces** — Tin & Ed (Tin Nguyen & Edward Cutting) (2021): Your body becomes pollen, fungus or rock in a living diorama. https://vimeo.com/906183149 - **Poem Pavilion (UK Pavilion, Expo 2020 Dubai)** — Es Devlin (2021): A building that writes with its visitors' words is a public, readable record of who came; shared AR text anchored on architecture can do the same at any scale. https://www.youtube.com/watch?v=91RiJ1vNMmg - **Pulse Topology** — Rafael Lozano-Hemmer (2021): Hanging data at different heights makes a walkable terrain of light; AR can let people walk through data landscapes at body scale. https://www.youtube.com/watch?v=5Zy39kMf_3o - **Realtime interactive football wall** — Roelof Knol (2021): Any wall becomes a playable surface when the projector knows where the ball lands. https://x.com/Rainmaker1973/status/1535625398670614529 - **The Multitude** — Collusion, Jamie Gledhill (2021): Your projected body is the game controller in a quest to save nature. https://www.youtube.com/watch?v=NywjgxG6JrM - **World Skies** — BREAKFAST (Andrew Zolty) (2021): Bringing a faraway sky into a room creates a window, not a screen — AR portals can use live data from real places. https://www.youtube.com/watch?v=JUkegoJ8nx8 - **shemza.digital #5** — Aphra Shemza (2021): A painted arch becomes a real doorway that lights up to welcome you. https://www.youtube.com/watch?v=BAdUb-LwuXs - **Antivanity Mirror** — Neil Mendoza (2020): A mirror that refuses to let you look at yourself. https://vimeo.com/398041909 - **Connected** — Roelof Knol (2020): Draw the invisible lines between strangers on the floor. https://www.youtube.com/watch?v=ArFcUEoxKfo - **Effluve (Faire corps)** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2020): Let visitors breathe life into a cloud of light with their bodies. https://www.youtube.com/watch?v=HC47brh6tio - **Instrument for Dissonance** — Lily Hassioti (2020): A self-playing machine whose rhythm is interrupted by your presence. https://www.youtube.com/watch?v=7j4NHuRxjGU - **Meander** — Philip Beesley / Living Architecture Systems Group (2020): A responsive space with its own slowly changing mood feels more alive than one that reacts identically every time — a lesson for persistent AR worlds. https://www.youtube.com/watch?v=nWijIywLUnw - **Remnant** — Theo Watson, Emily Gobeille (2020): Reach out to gather matter into a star until it explodes into a supernova and leaves a black hole. https://vimeo.com/393540065 - **Brixel Mirror** — BREAKFAST (Andrew Zolty) (2019): A mirror made of real mirrors, each one choosing where to reflect, blends physical and computed reflection — like AR on a real surface. https://www.youtube.com/watch?v=Z-5cVpWhp30 - **Evolution of the Garden** — Lily Hassioti (2019): A garden you play by touching its leaves. https://www.youtube.com/watch?v=V6MjBPvO5gU - **FIELD** — Theo Watson, Emily Gobeille (2019): Pollinate a virtual flower field with your body, attract butterflies, and watch the seasons turn. https://vimeo.com/322353545 - **Fabric Mirror** — Daniel Rozin (2019): Even cloth can become a display when each strip has two faces, which suggests AR textiles that flip to show hidden content. https://vimeo.com/317577759 - **Faire corps** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2019): An exhibition space that people enter together and change with their bodies. https://www.youtube.com/watch?v=VJmHz4OgnMo - **Future You** — Universal Everything (Matt Pyke) (2019): A motion-capture mirror shows 'future you', and the more you move, the stranger it gets. https://vimeo.com/718890921 - **GLADES** — Nick Hardeman (2019): Your own silhouette becomes the source of ecological harm, so the message is felt, not read. https://vimeo.com/323247011 - **Iridescence** — Behnaz Farahi (2019): Flipping two-coloured elements is a cheap, readable display — an AR designer can reuse the flip as a visual language for attention and mood. https://vimeo.com/325446043 - **Journey of Colour** — Moritz Waldemeyer (2019): Use a physical sample as a remote control for the color of a whole space. https://www.youtube.com/watch?v=kiTv0UH4rRU - **Propagason** — Gamgie (Clément Rignault) (2019): Send your voice into space and watch it travel. https://vimeo.com/362786908 - **Talking Heads** — Limelight (Viktor Vicsek & team) (2019): Faces made only of light still read as characters; an AR agent does not need a realistic face, only readable expressions and turn-taking. https://www.youtube.com/watch?v=fhL3gYWpXig - **The Welcome Chorus** — Yuri Suzuki (2019): A public choir of horns that learns new lyrics from passers-by. https://www.youtube.com/watch?v=pB1TBwACzsE - **BODY** — Random Studio (2018): Use your whole body as a search term. https://vimeo.com/314018273 - **Friction** — visiophone (Rodrigo Carvalho) (2018): Play light and sound in the air between your hands. https://vimeo.com/312071200 - **L'ombre de la vapeur** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2018): Make an invisible process, alcohol evaporating and the fungus living on it, into a landscape you can walk through. https://vimeo.com/278181935 - **LUMOS** — Matt DesLauriers (2018): A winter beacon that turns warm when it feels your body heat. https://vimeo.com/261406016 - **Laser Duck Hunt** — Seb Lee-Delisle (2018): A screen game escapes onto the wall as laser light, played with its original toy gun. https://vimeo.com/263303651 - **Lightning Catchers** — Seb Lee-Delisle (2018): Catch virtual lightning with a real glowing stick. https://vimeo.com/264245459 - **Nebula** — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2018): Your presence is the wind that reshapes a landscape. https://vimeo.com/318206534 - **Please Feed the Lions** — Es Devlin (2018): Giving a monument a voice that answers the public turns a statue into a conversation; AR layers on landmarks work best when they respond to what people bring. https://www.youtube.com/watch?v=SpZ-k9woZ14 - **Strange Stranger** — Alexander Whitley Dance Company (2018): Your 'data shadow' stays behind in the room after you leave. https://vimeo.com/354179753 - **After ALife Ahead** — Pierre Huyghe (2017): A living process drives both physical architecture and digital overlays, showing how AR can be steered by a real-world signal rather than the user. https://www.youtube.com/watch?v=eWre6dlUAbo - **Close Encounters** — visiophone (Rodrigo Carvalho) (2017): Walking up to a piece of street furniture opens a conversation with aliens. https://vimeo.com/319699135 - **Eyemote** — Kuflex (Igor Tatarnikov & Denis Perevalov) (2017): Turn things on with a glance. https://vimeo.com/245804445 - **FANTASTIC VOYAGE** — Schnelle Bunte Bilder (with kling klang klong) (2017): Shine a light inside an object and it comes alive. https://vimeo.com/207717295 - **Learning to See** — Memo Akten (2017): As hands move cloth and cables on a table, AI sees them live as waves, fire or nebulae. https://vimeo.com/260612034 - **Meadow** — Studio DRIFT (2017): Flipping a natural scene upside down and putting it overhead gives visitors a new relation to it; AR scenes can be placed on ceilings, not just floors. https://www.youtube.com/watch?v=jhUIFldl7h8 - **Mimic** — Theo Watson, Emily Gobeille (2017): A machine that forms a different relationship with each person shows how behaviour and memory, not visuals, make a character feel alive — key for AR companions. https://vimeo.com/207140893 - **New Spring (for COS)** — Studio Swine / A.A. Murakami (Azusa Murakami & Alexander Groves) (2017): A form that lives a few seconds and responds to your hand is irresistible — ephemeral, touchable AR objects hold attention better than permanent ones. https://www.youtube.com/watch?v=lOrkjk6FqIs - **Opale** — Behnaz Farahi (2017): Clothing that reacts to others' emotions shows a new kind of expressive interface; AR face filters can move from decorating yourself to answering others. https://vimeo.com/232258166 - **Post-Truth and Beauty** — Tim Murray-Browne, Aphra Shemza (2017): Truth as something you only ever see partially, depending on where your head is. https://www.youtube.com/watch?v=tVVIV_sNlaA - **Real Life Arcade Game** — Michael Flückiger (2017): A video game whose character is a real person following your button presses. https://vimeo.com/199587998 - **SUN** — Random Studio (2017): Bounce a ball to move the sun. https://vimeo.com/212766197 - **Studio Play** — Theo Watson, Emily Gobeille (2017): Use your body to uncover and zoom into museum artworks. https://vimeo.com/203162198 - **Symbiosis** — Kuflex (Igor Tatarnikov & Denis Perevalov) (2017): Your body becomes the core of a projected creature. https://vimeo.com/226068670 - **Control No Control** — Iregular (Daniel Iregui) (2016): Let the audience feel how much — and how little — control they have over a living surface. https://vimeo.com/1171876250 - **Demonz (INITI Playground)** — INITI (Dan Gregor) (2016): Turn any flat surface into a giant touch screen for physical ball games. https://vimeo.com/166247151 - **Echo (Flip-Discs)** — BREAKFAST (Andrew Zolty) (2016): A low-resolution mirror with sound is more magical than a perfect one — AR body effects can be coarse if they are tactile and audible. https://www.youtube.com/watch?v=3kYKPIh3TCk - **Forest of Resonating Lamps - One Stroke** — teamLab (2016): A single local action that ripples through a whole system shows one person's influence at room scale; AR can propagate a tap into a visible chain reaction. https://www.youtube.com/watch?v=cEMniCKnOD0 - **Graffiti Nature** — teamLab (2016): Drawn animals enter a projected ecosystem where they hunt and can be stepped on. https://www.youtube.com/watch?v=OomhbW3bffs - **Hag-Seed interactive** — Zach Lieberman (2016): Turn the words of a novel into a projected scene you play with your body. https://vimeo.com/222236331 - **I am Sound** — Tamiko Thiel (2016): Your face becomes a musical score played on the screen that shows it. https://www.youtube.com/watch?v=d66v3GFFEtU - **Laser Light Synths** — Seb Lee-Delisle (2016): Anyone can play music, and every note becomes light that fills the space. https://vimeo.com/314844029 - **Luma Paint — Interactive Light Graffiti** — Lichtfaktor (2016): Turning a slow photographic technique into instant feedback makes it a public game: live accumulation of traces is a strong AR interaction on its own. https://www.youtube.com/watch?v=_c7eTvKNdYU - **MotionComposer** — Palindrome (Robert Wechsler) (2016): Everyone is a dancer and a musician if the instrument listens to whatever movement they have. https://www.youtube.com/watch?v=4aDj7Ma_HkE - **Portals for Mortals** — Jamie Gledhill (2016): Walk through all four doorways together to play the whole fanfare. https://vimeo.com/180005726 - **Wilderness Wiggle** — Chris O'Shea (2016): A motion-sensing wilderness play wall in an Alaskan hospital where animals and landscape answer children's movements. https://vimeo.com/271539058 - **Δ∞ [Infinite Delta]** — visiophone (Rodrigo Carvalho) (2016): An architectural surface that reshapes itself around the people near it. https://vimeo.com/176050475 - **ANIMA iki** — onformative (Cedric Kiefer & Julia Laub) (2015): A glowing orb that behaves like a shy living being. https://vimeo.com/128767230 - **CHOREO** — Harshini J. Karunaratne (2015): Play a recorded dancer like a video game character. https://vimeo.com/191660123 - **Caress of the Gaze** — Behnaz Farahi (2015): Making another person's gaze visible on your body turns an invisible social signal into a physical response — the same move as gaze-driven AR effects. https://vimeo.com/152363295 - **Composition X** — Aphra Shemza (2015): Visitors mix the colour of the sculpture together just by where they stand. https://www.youtube.com/watch?v=hmh0cZxwpNw - **Connected Worlds** — Theo Watson, Emily Gobeille (2015): The whole hall is one ecosystem: children move real logs to steer water and plant seeds by hand, shaping six habitats. https://vimeo.com/131585517 - **EGO (with Stefano D'Alessio & Martina Menegon)** — Klaus Obermaier (2015): Your movements warp your mirror image into an abstract shape that still looks like you. https://www.youtube.com/watch?v=KzDifurF9wQ - **Elements** — Theo Watson, Emily Gobeille (2015): Each person becomes an element and shapes the projected world with their body. https://vimeo.com/197332386 - **Entangled** — Camille Utterback (2015): People on either side of a sheer screen tangle up each other's traces. https://www.youtube.com/watch?v=cmKSwen2GAw - **Floating Flower Garden** — teamLab (2015): Content that clears space around the user instead of crowding them feels welcoming; AR clutter should retreat as the user approaches. https://www.youtube.com/watch?v=7i-sPiuipco - **Flylight** — Studio DRIFT (2015): Light alone can behave like a flock that notices you; AR responses can be as simple as brightness waves moving away from a person. https://www.youtube.com/watch?v=oeJMBVEYweA - **Klanglichter** — Onat Hekimoglu & Tobias Kreter (2015): Touch photons: make light beams both the instrument and the joystick. https://vimeo.com/128600307 - **Level of Confidence** — Rafael Lozano-Hemmer (2015): Face recognition searches every visitor's face for the 43 missing students, turning surveillance into a memorial. https://vimeo.com/953969845 - **Penguins Mirror** — Daniel Rozin (2015): 450 toy penguins turn around, their black backs and white bellies forming your silhouette. https://www.youtube.com/watch?v=QlrnjjfLkTI - **Pixel Waves** — Miguel Chevalier (2015): Walk through a sea of pixel light whose waves move with you. https://vimeo.com/1208277695 - **Play Table** — Jamie Gledhill (2015): A table where strangers play with the same projected objects from all sides. https://vimeo.com/142236663 - **PomPom Mirror** — Daniel Rozin (2015): Rendering a live body with a soft, unexpected material makes the reflection funny and tender — AR body effects gain character from the choice of 'pixel' material. https://vimeo.com/128375543 - **Story of Light (Star Stomp)** — Gene Kogan (2015): Stomp on the floor to make both sound and light. https://vimeo.com/120653546 - **Synapse** — Behnaz Farahi (2015): Showing an inner state (attention) on the outside of the body is a strong idea for AR wearables and social filters. https://vimeo.com/139237974 - **Where things start from** — Tim Murray-Browne (2015): Reward stillness and slowness with sound instead of fast gestures. https://www.youtube.com/watch?v=mhtq0pi2iYI - **Zoom Pavilion (with Krzysztof Wodiczko)** — Rafael Lozano-Hemmer (2015): Surveillance cameras zoom in on how you relate to strangers and fill the whole room with it. https://www.youtube.com/watch?v=ENWBRsvn7qA - **Dhalsim: Real-Time Body Transformation** — Keita Higuchi (2014): Your own live body, but with arms that stretch across the room. https://www.youtube.com/watch?v=g2bg_vBVW1w - **Electricus** — Jamie Gledhill (2014): Your movement charges your own body with electricity on screen. https://vimeo.com/87755772 - **If The Walls Had Eyes** — Luxloop (2014): The digital eyes that watch us online become real eyes on the wall. https://vimeo.com/113986453 - **Luminescence** — Jamie Gledhill (2014): Energy threads connect everyone who steps into the frame. https://vimeo.com/93631167 - **MOMENTUM (Schnelle Bunte Bilder & kling klang klong)** — Motion Bank / Choreographic Coding Lab (2014): Your body becomes a liquid creature whose motion is also the soundtrack. https://vimeo.com/112193826 - **Magic Carpets** — Miguel Chevalier (2014): Turn the floor of an old building into a magic carpet that changes with every step. https://www.youtube.com/watch?v=LvNhZrWVByM - **Noisy Skeleton** — THÉORIZ (David-Alexandre Chanel, Jonathan Richer & team) (2014): A dialogue between a body and a machine that listens with sound and space. https://vimeo.com/103438556 - **Portée/** — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2014): Unfold a music score into space and let people play it by touching the lines. https://vimeo.com/118226187 - **STEPSEQUENCER** — Schnelle Bunte Bilder (with kling klang klong) (2014): Your dance steps compose the beat. https://vimeo.com/101091090 - **SWARM** — Mark Coniglio / Troika Ranch (with Dawn Stoppiello) (2014): The audience is the dance company, and the piece only progresses through collective action. https://vimeo.com/86373749 - **Silhouettes** — Gene Kogan (2014): Paint walls with your shadow. https://vimeo.com/113887700 - **Sketch Town** — teamLab (2014): Hand-drawn cars and houses move into a shared projected city. https://www.youtube.com/watch?v=kQ_17zapssI - **TRANSFORM** — MIT Tangible Media Group — Hiroshi Ishii (2014): A living table that heaves like ocean waves and carries objects around by itself. https://vimeo.com/98880732 - **cloud piano** — David Bowen (2014): Reading the sky as a score shows that any live view can become an instrument — a direct pattern for camera-driven AR sound. https://vimeo.com/101857804 - **1000 Hands** — Universal Everything (Matt Pyke) (2013): Drawings by a thousand visitors grow into 3D forms and join one shared projected landscape. https://www.youtube.com/watch?v=yECE20Cq0mE - **329 prepared dc-motors, cotton balls, toluene tank** — Zimoun (2013): The space is the instrument: an AR designer learns that simple agents placed along a real surface can make the architecture itself sound and feel alive. https://www.youtube.com/watch?v=8jOBgFJSXxg - **Angles Mirror** — Daniel Rozin (2013): Using orientation instead of colour as the pixel value shows that angle alone can carry an image — a useful idea for AR fields of arrows, grass or fins. https://vimeo.com/61823984 - **Cave of Sounds** — Tim Murray-Browne (2013): A circle of strange instruments turns strangers into a band without instructions. https://vimeo.com/76453883 - **Conduct The Orchestra** — Random Studio (2013): Anyone can be the conductor of a great orchestra. https://vimeo.com/67575564 - **D.I.G.I.T.** — Teehan+Lax Labs (2013): A mirror made of calculator digits. https://vimeo.com/79332227 - **Galets Magiques** — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2013): Shake a real pebble to reveal the animals hiding under it. https://vimeo.com/64186902 - **Garden of Russolo** — Yuri Suzuki (2013): Speak into a box and hear your voice turned into noise music. https://vimeo.com/75251985 - **Interactive Window Projection – Zierikzee** — NuFormer (2013): An ordinary window becomes a portal the moment it notices you, a small-scale idea that transfers directly to AR shop fronts. https://www.youtube.com/watch?v=vXLq8LAtaE0 - **Interactive particles @ La Bifurk** — Benjamin Kuperberg (2013): Turn a dancing body into drifting sand. https://www.youtube.com/watch?v=9N--dSXwP9c - **Kinect / Leap controlled realtime creature** — onformative (Cedric Kiefer & Julia Laub) (2013): Puppeteer a creature with your bare hands. https://vimeo.com/71086950 - **Laser Forest** — Marshmallow Laser Feast (2013): A walk-through instrument where every touch becomes sound and light in space. https://www.youtube.com/watch?v=gDZqJ4RPVYc - **PLAYGROUND** — Red Paper Heart (2013): Walk into the music and see the room answer every sound. https://vimeo.com/61893280 - **Réflexions** — Dpt. (2013): A mirror that listens and deliberately gets you wrong. https://vimeo.com/83617391 - **Sharing Faces** — Kyle McDonald (2013): A cross-border mirror: you see the face of someone in another country with your same expression and pose. https://vimeo.com/96549043 - **Sketch Aquarium** — teamLab (2013): Fish drawn by children are scanned in and swim in a projected aquarium. https://www.youtube.com/watch?v=AnAqB7LZUb8 - **Strike A Pose (Gallery One)** — Local Projects (2013): Understand a sculpture by imitating it with your own body. https://vimeo.com/60866008 - **The Living, Breathing Wall** — Behnaz Farahi (2013): A wall that recognises speech and answers with shape shows how architecture can listen; AR can let real walls 'listen' the same way. https://vimeo.com/81572631 - **Voice Tunnel** — Rafael Lozano-Hemmer (2013): A voice that travels as light along a real corridor turns sound into something you can see move — a clear pattern for spatial audio visualisation in AR. https://www.youtube.com/watch?v=jmRnLUVt4kE - **Woodland Wiggle** — Chris O'Shea (2013): A wall-sized interactive fairy-tale forest in a hospital, where children paint, make music and call up weather with their bodies. https://vimeo.com/59349284 - **fly revolver** — David Bowen (2013): Handing control to an unpredictable living agent creates tension no script can — AR characters can borrow behaviour from real creatures tracked by the camera. https://vimeo.com/71826503 - **Appel d'Air** — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2012): A small breath gets a big answer. https://vimeo.com/74689338 - **Augmented Silhouette** — Beam'Art (Benjamin Petit & Antoine Vanel) (2012): Your silhouette becomes a light body that throws particles. https://vimeo.com/39887510 - **Botanicus Interacticus** — Ivan Poupyrev (2012): Stroke a real plant and it answers with sound, light and shadow. https://www.youtube.com/watch?v=17QOyr2d5-I - **Drawn Together** — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2012): Draw a duet with a machine that replies in 3D above your paper. https://vimeo.com/37623623 - **Firewall** — Aaron Sherwood (2012): A soft surface that you can physically push makes digital content feel touchable — a reminder that AR benefits from real material feedback. https://vimeo.com/54882144 - **Future Self (with Wayne McGregor)** — Random International (Hannes Koch & Florian Ortkrass) (2012): An array of LED rods forms a "light double" that mirrors your movements. https://www.youtube.com/watch?v=Jqn1cMY8oGM - **I Spy** — Neil Mendoza (2012): Our gadgets finally stare back at us. https://vimeo.com/55122295 - **It's You** — Karolina Sobecka (2012): A projected crowd hides a secret, and only steps aside when you lean in. https://vimeo.com/35266165 - **Loup-garou** — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2012): Stand in front of the moon and become a werewolf. https://vimeo.com/119942169 - **NikeFuel Station** — onformative (Cedric Kiefer & Julia Laub) (2012): Your body rebuilt as a cloud of glowing pixels. https://vimeo.com/44338220 - **PixelPyros** — Seb Lee-Delisle (2012): Wave your hand and a firework launches from that spot; the crowd choreographs the show. https://vimeo.com/61174060 - **Rain Room** — Random International (Hannes Koch & Florian Ortkrass) (2012): Walk through rain that stops just for you. https://www.youtube.com/watch?v=FslABAyj2OA - **SENSE #1** — Ralfonso (2012): Combining ambient information (time) with response to presence makes a public piece useful and playful at once, a good pattern for civic AR. https://www.youtube.com/watch?v=bCaWkHyEwUE - **Sibyl** — Philip Beesley / Living Architecture Systems Group (2012): A suspended form that trembles when you look up at it shows how overhead space — often ignored in AR — can host responsive content. https://www.youtube.com/watch?v=jTc14p5ognM - **Sirènes Sylvestres** — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2012): Lights that wake up as you come near and lead you into the dark. https://vimeo.com/74054426 - **Starfield** — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2012): A playground swing becomes a spaceship. https://vimeo.com/36892768 - **Stop-iT** — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2012): A wall of post-it notes is a music sequencer. https://vimeo.com/37536157 - **The Nature Trail** — Jason Bruges Studio (2012): Content that appears only when someone passes and walks with them turns a stressful route into a companion — a direct template for location-based AR wayfinding. https://www.youtube.com/watch?v=8j2RegeSwYM - **The Treachery of Sanctuary** — Chris Milk (2012): Across three screens your shadow falls apart, is devoured, then grows wings and flies. https://www.youtube.com/watch?v=I5__9hq-yas - **Water Light Graffiti** — Antonin Fourneau (2012): Painting with an everyday material that leaves no mark teaches AR designers to link virtual ink to a real gesture and to let drawings fade on their own. https://vimeo.com/47080920 - **3D Video Mapping Interactivity Test** — NuFormer (2011): Letting the audience drive a building-sized image turns passive watching into play, which is what makes location AR memorable. https://www.youtube.com/watch?v=3HrGkyoCGmg - **Dancing With Swarming Particles** — visiophone (Rodrigo Carvalho) (2011): Your body is a magnet that gathers a swarm into your own shape. https://vimeo.com/21052774 - **Eyeshine** — Golan Levin, Kyle McDonald (2011): Catch the red-eye glint in visitors' eyes so they see their own eyes glowing like a night animal's. https://vimeo.com/29356492 - **Flow** — Frieder Weiss (2011): The floor behaves like water that remembers every step. https://vimeo.com/41397711 - **Inside - Out** — Hellicar & Lewis (Pete Hellicar & Joel Gethin Lewis) (2011): Project people from one space live into another so both sides end up watching each other. https://vimeo.com/30795891 - **Night Bright** — Theo Watson, Emily Gobeille (2011): Use your body as a flashlight to light up a night forest and follow sounds to find hidden animals. https://vimeo.com/29193895 - **Puppet Parade** — Theo Watson, Emily Gobeille (2011): Work giant projected puppets with your arms while other kids walk into the scene to feed them. https://vimeo.com/34824490 - **Wind Arbor** — Ned Kahn (2011): Shadows cast inside tell visitors what is happening outside; AR can likewise let outdoor data leak indoors as light and shadow. https://www.youtube.com/watch?v=XZxZSHukZpo - **XYZT, Abstract Landscapes** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2011): Touch, blow and walk to reshape landscapes made of letters and particles. https://www.youtube.com/watch?v=N7q9k9Z3HIk - **tele-present water** — David Bowen (2011): A remote sea can be rebuilt as a physical mesh in a room; AR can do the same with a virtual surface anchored to the floor. https://vimeo.com/27614383 - **Augmented Shadow** — Joon Moon (Joon Y. Moon / 문준용) (2010): Fake the shadows of real objects so a hidden story can live inside them. https://www.youtube.com/watch?v=0arZMuPK58w - **DUNE** — Studio Roosegaarde (Daan Roosegaarde) (2010): Augment a public path with nature-like light that notices the people using it. https://www.youtube.com/watch?v=nf-q5zs8HgE - **Feedback (with Todd Vanderlin)** — Hellicar & Lewis (Pete Hellicar & Joel Gethin Lewis) (2010): Feed people's movements back into themselves as layer upon layer of visual echoes. https://vimeo.com/19598568 - **Fragments of RGB** — onformative (Cedric Kiefer & Julia Laub) (2010): A screen that falls apart into pixels when you get close. https://vimeo.com/21235126 - **Hylozoic Ground** — Philip Beesley / Living Architecture Systems Group (2010): Thousands of tiny local reactions add up to a room that feels alive; in AR, many small agents reacting to nearby people beat one big scripted animation. https://www.youtube.com/watch?v=jgL2ppDmNtA - **Mimosa** — Jason Bruges Studio (2010): Borrowing a plant's reflex (the mimosa folds when touched) gives a responsive surface an instantly readable personality — useful for AR objects that should feel alive. https://www.youtube.com/watch?v=srYK8pEYnMc - **Moc** — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2010): Your breath makes a tree grow. https://vimeo.com/11989814 - **One Hundred and Eight** — Nils Völker (2010): Cheap, soft material with good timing feels alive — AR effects gain life from rhythm and reaction, not detail. https://www.youtube.com/watch?v=1BfHY5sFGJM - **Pulse Index** — Rafael Lozano-Hemmer (2010): Showing each new person largest and letting older entries shrink into a crowd is a clear visual grammar for shared AR walls of contributions. https://www.youtube.com/watch?v=sdRWsCVAklQ - **Rust Mirror** — Daniel Rozin (2010): Combining a live reflection with a slow environmental effect (rain) makes a mirror feel like a place with weather — an idea for layering ambient effects on AR selfies. https://vimeo.com/9256258 - **Sandbox (Relational Architecture 17)** — Rafael Lozano-Hemmer (2010): The hand you reach into a small sandbox is projected as a giant hand covering the whole beach. https://www.youtube.com/watch?v=GotOBu_14fc - **ShadowFighter** — Peter Uithoven (2010): Your shadow fights your opponent's shadow. https://vimeo.com/12486954 - **White Heat** — Hellicar & Lewis (Pete Hellicar & Joel Gethin Lewis) (2010): A swarm of rule-driven light creatures gathers, scatters and evolves around visitors and books. https://vimeo.com/15633768 - **Beacon** — Chris O'Shea (2009): A field of lights that notices you and turns to look. https://vimeo.com/1025054392 - **CueLight interactive pool table** — Obscura Digital (2009): Augmenting an existing game with visual feedback keeps its rules and adds delight; the best AR games often start from a familiar physical game. https://www.youtube.com/watch?v=e0_wloTBxUM - **Decrypted Reflections** — Chris Sugrue (2009): Move a set of mirrors to break a projected generative image apart and bounce it all over the room. https://vimeo.com/7652087 - **Interzone** — visiophone (Rodrigo Carvalho) (2009): Turn a corridor you pass through every day into a zone that notices you. https://vimeo.com/9827907 - **Sniff (with James George)** — Karolina Sobecka (2009): A virtual dog in a shop window follows passers-by and judges whether you are friendly or mean to it. https://vimeo.com/6400266 - **Terrarium** — Theo Watson, Emily Gobeille (2009): Talk to the installation and your voice becomes food that keeps a virtual ecosystem alive. https://vimeo.com/5269088 - **You Fade to Light** — Random International (Hannes Koch & Florian Ortkrass) (2009): Turn your mirror image into light that slowly fades away. https://www.youtube.com/watch?v=aIVKVoFYvZ8 - **tele-present wind** — David Bowen (2009): Invisible wind from somewhere else becomes visible indoors by copying its motion onto many objects — a model for AR scenes driven by live remote data. https://vimeo.com/20963294 - **Audience** — Random International (Hannes Koch & Florian Ortkrass), Chris O'Shea (2008): A crowd of mirrors turns its heads in unison to stare at you like an audience. https://www.youtube.com/watch?v=JuKi35j3Dwk - **Body Paint** — Memo Akten (2008): Your body is the brush: dance in front of a projected wall and paint splashes and flows with your moves. https://vimeo.com/3576457 - **Drawing from Life** — Camille Utterback (2008): Write your live mirror image as a portrait made from the four letters of DNA. https://www.youtube.com/watch?v=E4AJY959514 - **Fractal Flowers** — Miguel Chevalier (2008): A fractal garden that bends and blooms as you come close. https://vimeo.com/796298809 - **Mirrors Mirror** — Daniel Rozin (2008): Countless small mirrors reflect the room's light and dark to piece together your image. https://www.youtube.com/watch?v=oKum2u7oLwc - **Pulse Spiral** — Rafael Lozano-Hemmer (2008): A spiral turns a history of inputs into a readable shape, so the newest heartbeat sits at the start and the oldest fades at the top — a layout idea for AR timelines. https://www.youtube.com/watch?v=9hwEpbjlaek - **Central Mosaic** — Scott Snibbe (2007): Each passer-by's shadow takes the center until newcomers push it to the edge of a fractal mosaic. https://www.youtube.com/watch?v=ZdgX4PUyfvw - **Delicate Boundaries** — Chris Sugrue (2007): Little bugs made of light crawl out of the screen and onto the hands of whoever touches it. https://vimeo.com/1007230 - **Funky Forest** — Theo Watson, Emily Gobeille (2007): Kids plant trees with their bodies and use real 'logs' to redirect a projected river to water the forest. https://vimeo.com/3872687 - **Ghost Pole Propagator** — Golan Levin (2007): Distill a person's silhouette into skeleton-like lines projected on the wall. https://vimeo.com/222999706 - **Inflatable Architectural Body** — Chico MacMurtrie / Amorphic Robot Works (2007): Soft inflation makes huge forms feel gentle and alive; AR objects can gain the same presence with slow breathing growth instead of instant appearance. https://www.youtube.com/watch?v=vqXJpuglpa0 - **Interstitial Fragment Processor** — Golan Levin (2007): Turn the gaps between people's shadows into solid things that fall, bounce and make sounds. https://vimeo.com/86071976 - **Out of Bounds** — Chris O'Shea (2007): Shine an 'X-ray flashlight' at the wall to see the hidden space behind it. https://vimeo.com/1333176 - **Peg Mirror** — Daniel Rozin (2007): Wooden pegs turn their angled faces to build a portrait out of shadows. https://www.youtube.com/watch?v=dghosA-zI6k - **Weave Mirror** — Daniel Rozin (2007): Weave your image into being, strand by strand. https://www.youtube.com/watch?v=ushJnQfjbF0 - **Close-Up (ShadowBox 2)** — Rafael Lozano-Hemmer (2006): Your shadow is filled with surveillance videos of everyone who looked at the piece before you. https://www.youtube.com/watch?v=gGbCF2oPayM - **Eye Contact (ShadowBox 1)** — Rafael Lozano-Hemmer (2006): The moment you show up, hundreds of people on the screen wake up together and stare at you. https://www.youtube.com/watch?v=SzIwx-oX3U8 - **Footfalls** — Golan Levin, Zach Lieberman (2006): The harder you stomp, the more virtual objects fall from the sky, and you can catch them with your shadow and throw them back. https://vimeo.com/227566535 - **Pulse Room** — Rafael Lozano-Hemmer (2006): A room that keeps a queue of visitors' heartbeats shows how biometric input can become a shared, cumulative space — a model for multi-user AR memories. https://www.youtube.com/watch?v=R3benqCGVLI - **Third Person (ShadowBox 4)** — Rafael Lozano-Hemmer (2006): Build your shadow portrait out of every verb in the dictionary. https://www.youtube.com/watch?v=w-EWZ1r1Yos - **Volume** — United Visual Artists (UVA) (2006): People become the performers when a space answers their movement — AR in public spaces works best when crowds can 'play' it together. https://www.youtube.com/watch?v=vbglOn9ea-Y - **Waves** — Daniel Palacios (2006): One line that goes from calm to chaos as a crowd grows teaches an AR designer to map audience activity to a single, readable visual parameter. https://vimeo.com/12075151 - **Daisies** — Theo Watson (2005): Flowers wilt under your feet and grow back after you leave. https://vimeo.com/463536634 - **Make Like a Tree** — Scott Snibbe (2005): Your shadow becomes a ghost in the forest, wandering off between the trees. https://www.youtube.com/watch?v=CPFF3-di2PU - **Messa di Voce (installation)** — Golan Levin, Zach Lieberman (2005): Speech and song grow visible shapes out of your mouth that you can push with your body. https://vimeo.com/221802940 - **Scrapple** — Golan Levin (2005): Whatever you toss on the table becomes a score that plays live and gets marked up by projection. https://vimeo.com/227633208 - **Shadow Bag** — Scott Snibbe (2005): Your shadow stops obeying: sometimes it follows you, sometimes it walks off on its own. https://www.youtube.com/watch?v=ybt4DJ7ptpQ - **Subtitled Public** — Rafael Lozano-Hemmer (2005): Everyone gets a verb label projected on their body, and the only way to pass it on is to touch someone else. https://vimeo.com/1089123902 - **Under Scan (Relational Architecture 11)** — Rafael Lozano-Hemmer (2005): Walk across the square and a stranger appears in your shadow, looking up at you. https://www.youtube.com/watch?v=Bfn14sLJmyU - **Visceral Cinema: Chien** — Scott Snibbe (2005): Visitors' shadows walk into a surrealist film and take over the lead role. https://www.youtube.com/watch?v=xgxkUH6PrIE - **Articulated Cloud** — Ned Kahn (2004): A soft, moving skin makes architecture feel weather-sensitive; AR façades can borrow this idea of a responsive veil over a fixed structure. https://www.youtube.com/watch?v=hy9OrQyl1dE - **Cause and Effect** — Scott Snibbe (2004): Your movements are recorded and jostle with the shadows of earlier visitors, forming a chain of cause and effect across time. https://www.youtube.com/watch?v=Db5i2Nz1jHk - **Compliant** — Scott Snibbe (2003): Make the screen behave like a rubber sheet that people's shadows push and squash. https://www.youtube.com/watch?v=PSdvN7c25mI - **Frequency and Volume (Relational Architecture 9)** — Rafael Lozano-Hemmer (2003): Tune a radio with your shadow on the wall to hear the city's invisible airwaves. https://www.youtube.com/watch?v=hWd8uv8U4tc - **Schlamp** — Frieder Weiss (2003): Let passers-by paint with their bodies on a projected surface. https://www.youtube.com/watch?v=2BKTlqt4GFA - **Shiny Balls Mirror** — Daniel Rozin (2003): A mirror made of tiny mirrors creates two reflections at once, a playful layering an AR designer can reuse for picture-in-picture views. https://vimeo.com/57244184 - **Deep Walls** — Scott Snibbe (2002): Record passing shadows and keep them looping forever in 'shadow cupboards' on the wall. https://www.youtube.com/watch?v=X7h9ckxlxtc - **Shadow (Screen Series)** — Scott Snibbe (2002): You have left, but your shadow stays behind and repeats what you just did. https://www.youtube.com/watch?v=pdxYv-_70-s - **Technorama Facade** — Ned Kahn (2002): Every pixel here is a physical sensor and actuator in one, a reminder that the best AR displays may be the world's own surfaces reacting. https://www.youtube.com/watch?v=hVyo5ICl5-I - **Trash Mirror** — Daniel Rozin (2002): Make a mirror that reflects you out of scraps of street trash. https://www.youtube.com/watch?v=R0dLo3HB4P8 - **Body Movies (Relational Architecture 6)** — Rafael Lozano-Hemmer (2001): Your giant shadow is the only window through which strangers' portraits can be seen. https://www.youtube.com/watch?v=g-CNxFiXZDY - **HypoSurface** — dECOi / Mark Goulthorpe (HypoSurface) (2001): A physical surface that moves like a screen shows what AR could add to real walls: content that seems to deform the wall itself rather than float in front of it. https://www.youtube.com/watch?v=ANXQRJ2zksI - **Wind Veil** — Ned Kahn (2000): Making an invisible force visible across a surface is AR's core promise; Kahn does it at building scale with no electronics. https://www.youtube.com/watch?v=ZDPqrA4-jK0 - **Unconscious Flow** — Naoko Tosa (1999): Show the hidden emotional relationship between two people. https://www.youtube.com/watch?v=eF4DsTolxJM - **Vectorial Elevation** — Rafael Lozano-Hemmer (1999): Letting remote people design something that appears over a real city shows how participation can reach physical space — a template for web-authored, site-anchored AR. https://www.youtube.com/watch?v=C4xx8sirByI - **Wooden Mirror** — Daniel Rozin (1999): Piece together your live mirror image from turning wooden tiles. https://www.youtube.com/watch?v=1ZPJ0U_kpNg - **Boundary Functions** — Scott Snibbe (1998): Draw invisible 'personal space' on the floor as Voronoi lines, so the more people there are, the less space each gets. https://www.youtube.com/watch?v=5wA3lKcDrlM - **Neuro-Baby** — Naoko Tosa (1993): A virtual being that responds to how you feel, not what you say. https://www.youtube.com/watch?v=etEIbMZ6uUY - **Surface Tension** — Rafael Lozano-Hemmer (1992): Being watched back by an image is one of the strongest ways to make a virtual thing feel present — AR characters that track the user's position use the same trick. https://www.youtube.com/watch?v=JXLoLPkzdto - **Zerseher / De-viewer** — ART+COM Studios (Joachim Sauter) (1992): The act of looking can destroy what you look at — gaze is a powerful and underused input for headset AR. https://vimeo.com/386256001 - **The Giver of Names** — David Rokeby (1991): A machine that names objects badly but beautifully shows how recognition can be poetic; AR object labels do not have to be literal. https://www.youtube.com/watch?v=sO9RggYz24Q - **Very Nervous System** — David Rokeby (1986): The whole body as the interface, with no visible device, is the root of camera-based interaction; AR sound can use the same idea with body tracking. https://www.youtube.com/watch?v=qdvyuvfKVU0 - **CRITTER (VIDEOPLACE)** — Myron Krueger (1984): A virtual pet that understands the shape of your body. https://www.youtube.com/watch?v=VdrujesfIBQ - **VIDEOPLACE** — Myron Krueger (1975): Your camera silhouette is your body in the computer world, no gear required. https://www.youtube.com/watch?v=d4DUIeXSEpk - **The Senster** — Edward Ihnatowicz (1970): Shy behaviour, not spectacle, made people stay with it; AR characters that approach gentle attention and retreat from noise feel far more alive. https://www.youtube.com/watch?v=1jDt5unArNk - **SAM (Sound Activated Mobile)** — Edward Ihnatowicz (1968): Turning to face a voice is the simplest signal of attention; AR objects that orient to the speaker feel social with almost no other behaviour. https://www.youtube.com/watch?v=8b52qpyV__g - **Magnet TV** — Nam June Paik (白南准) (1965): A physical object distorts a screen image in real time; AR can let real objects act as lenses or magnets that bend virtual content around them. https://www.youtube.com/watch?v=44DJOtaU-b8 - **Participation TV** — Nam June Paik (白南准) (1963): One of the first artworks where the audience's voice draws on a screen; sound-reactive AR effects start here. https://www.youtube.com/watch?v=DimYjLdQMHc - **Tour Lumière Cybernétique (Liège)** — Nicolas Schöffer (1961): A landmark that reads its environment and answers with light makes a city site feel alive; geolocated AR landmarks can react to live weather or city data the same way. https://www.youtube.com/watch?v=y1nHehuu7Jw - **CYSP 1** — Nicolas Schöffer (1956): The first sculpture with its own senses and behaviour; AR objects become characters as soon as they react to light, sound or people. https://www.youtube.com/watch?v=u8nxktU1R6E ### Stage & performance Dance, theatre and concerts where digital imagery shares the stage with performers. - **Inner Room (dance)** — Zhou Zhou (2026): The dancer's body becomes the weather of the room. https://www.youtube.com/watch?v=oZopGxgotVo - **Mirror** — Alexander Whitley Dance Company (2026): An AI mirror that starts faithful and slowly lies. https://www.youtube.com/watch?v=zjHSgImXydw - **The holographic ballet Swan Lake** — Sila Sveta (2026): Wrapping layered holographic screens around dancers turns a flat stage into a volume; AR ballet can place swans and lakes in front of, beside and behind the dancer. https://www.youtube.com/watch?v=RfKX8A98s-I - **La Tournoyante x Gamgie** — Gamgie (Clément Rignault) (2025): Projected particles that dance with acrobats as a live partner. https://www.youtube.com/watch?v=XBd41JohXq8 - **The Last Swan** — Alexander Whitley Dance Company (2025): A classical ballet becomes a projection room the audience can enter and continue. https://vimeo.com/1219574400 - **The Placeholders** — Julie C. Stamm (2025): Make a stage where a camera, a dataset and dancers invent meaning together live. https://vimeo.com/1118720620 - **nino** — Gibson/Martelli (Ruth Gibson & Bruno Martelli) (2025): Suit-free mocap turns every visitor into a puppeteer of a dreamlike avatar. https://vimeo.com/1081511795 - **Between Information and Noise** — Jorge Guevara (2024): Let the audience sit inside the dance and see both the body and its virtual trace at once. https://www.youtube.com/watch?v=QfJdhuRkY_s - **Encyclies** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2024): The image plays next to the pianist as a second instrument. https://vimeo.com/1057582668 - **Hatsune Miku live (MIKU EXPO 2024, Vancouver)** — Crypton Future Media (Hatsune Miku live concerts) (2024): A virtual performer becomes 'real' through shared rhythm, lighting and crowd ritual; social AR can borrow the same cues (timing, call-and-response, shared lights). https://www.youtube.com/watch?v=b6VhAvtekeU - **Monolith** — Uncharted Limbo Collective (George Adamopoulos, Eleana Polychronaki, Chris Waters) (2024): A dancer's body stirs a virtual fluid as if dancing in water. https://www.youtube.com/watch?v=Hdep8X88SWA - **WOW+FLUTTER** — Gibson/Martelli (Ruth Gibson & Bruno Martelli) (2024): Dancers in two cities share one body of non-human avatars. https://vimeo.com/1018110955 - **Waterhall Mocap Performance** — Holosphere (2024): A dancer's limbs leave painted trails that hang in the air behind them. https://www.youtube.com/watch?v=_p_85sK5is4 - **Becoming Performance in VR** — Jorge Guevara (2023): Ask what a headset audience needs in order to feel a dancer who is only there as data. https://www.youtube.com/watch?v=aKd1FepHuiQ - **Homme Plissé Issey Miyake show, Palais de Tokyo** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2023): A runway where the space itself pleats like the clothes. https://vimeo.com/793397205 - **Piano piano** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2023): Music you can see: piano notes and juggled balls turn into brushstrokes in space. https://vimeo.com/893723919 - **REfract** — Zelia ZZ Tan (2023): A dancer and her avatars share one body and argue over who leads. https://www.youtube.com/watch?v=PsYzQYleWsk - **Thermal Improvisations** — Lily Hassioti (2023): Play the room with body heat that only a thermal camera can see. https://www.youtube.com/watch?v=A5ROdOCii0Y - **Virtual Stage x Holosphere** — Birmingham Royal Ballet (Virtual Stage) (2023): A ballet dancer paints a 35-metre wall with his body in real time. https://www.youtube.com/watch?v=I0y7QCaBFm4 - **Choreographic Coding Lab at A+E Lab (CCL-14)** — Motion Bank / Choreographic Coding Lab, AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) (2022): A one-week lab format where code meets choreographic thinking, and dancers test prototypes on stage the same day. https://vimeo.com/759923547 - **DISCORDANCE** — Clémence Debaig (Unwired Dance Theatre) (2022): Two cities, one virtual stage, three kinds of audience. https://www.youtube.com/watch?v=p_1sgQmuf-4 - **Embodied Machine** — Instituto Stocos (Muriel Romero, Pablo Palacio & Daniel Bisig) (2022): Turn stage lights into limbs that extend the dancer's gestures. https://www.youtube.com/watch?v=ntp-rhbwV9c - **Just your shadow** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2022): A dancer dances with a shadow that does not quite obey. https://www.youtube.com/watch?v=iWfpUCYw3Kg - **PRE-FIGURES** — Uncharted Limbo Collective (George Adamopoulos, Eleana Polychronaki, Chris Waters), Alexander Whitley Dance Company (2022): Archive dancers step out of old film and share the stage with living ones. https://www.youtube.com/watch?v=i_Zr_9x_1zk - **Pattern Regression** — Alexander Whitley Dance Company, Memo Akten (2022): A human and a lighting machine hold a kinetic conversation in a gallery. https://vimeo.com/778138186 - **Playscape: How to Build a Galaxy** — Katie Dale-Everett Dance (KDE Dance) (2022): Dancing bodies literally build a galaxy that children can watch grow. https://www.youtube.com/watch?v=bhecnzfh3D4 - **Sensing Adjacency** — Zelia ZZ Tan (2022): Choreograph the gap between a body and its imperfect digital double. https://www.youtube.com/watch?v=drXmMBy0a_o - **Anti-Body** — Alexander Whitley Dance Company (2021): Live motion capture turns dancers into the light and scenery of the show. https://www.youtube.com/watch?v=IZpirMOBwLg - **Björk: Arisen My Senses (concert visuals)** — Tobias Gremmler (2021): Reformatting the same visual system for a new shape (wide stage vs. circle) shows that good stage visuals are a system, not a single video, which is how AR effects should be designed too. https://www.youtube.com/watch?v=r-aQX5spvyM - **Engendered Otherness (Ai Transmutations)** — Kianí del Valle (2021): A live dancer leads a chorus of AI-generated creatures that only exist while she moves. https://www.youtube.com/watch?v=hXiDYN1IFfY - **Levitation 2** — Sila Sveta (2021): Illusions of weightlessness depend on hiding the support and controlling the viewpoint; AR can make real people seem to levitate with the same discipline. https://www.youtube.com/watch?v=IYvbl4KY4-A - **Piano&Dancer** — Instituto Stocos (Muriel Romero, Pablo Palacio & Daniel Bisig) (2021): A dancer plays a piano from across the room with her body alone. https://www.youtube.com/watch?v=_3gEitmSdis - **Anti-Gone** — Theo Triantafyllidis (2020): A swamp theater where a motion-captured performer drives a virtual creature in real time. https://www.youtube.com/watch?v=qMvq9pYd2PI - **Deep Dancing** — Irini Kalaitzidi (2020): Hear how someone moves, not only see it. https://vimeo.com/783283941 - **Kat & Cassie Make A Ballet** — Kat Sullivan (2020): One dancer, captured live, becomes an entire corps de ballet. https://www.youtube.com/watch?v=cSspDP7H5tg - **Research lab with Daniell Alnuma** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2020): A spinning body becomes the center of gravity for a field of lines. https://vimeo.com/479608301 - **S . P . A . C . E .** — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2020): Break down and reassemble dancers' body data to create geometric worlds the body itself cannot form. https://www.youtube.com/watch?v=bY0lMfl1rpI - **Acqua Alta – Noir d'encre** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2019): A drawn world of ink that floods, flows and grows hair in dialogue with the performers. https://www.youtube.com/watch?v=FOF1xKHe7e0 - **BLINK** — Marion Tränkle / NOW (2019): Blinking, which we never notice, becomes the score of a performance. https://www.youtube.com/watch?v=vEc037P5tiI - **Björk Cornucopia: Body Memory (stage visuals)** — Tobias Gremmler (2019): Visuals that look like they grow out of the performer's body blur the line between costume and environment, which is the natural home of performance AR. https://www.youtube.com/watch?v=GaQfixl2Ss4 - **Bounce (elasticity test)** — Marion Tränkle / NOW (2019): Measure how a body bounces and let the whole room bounce with it. https://www.youtube.com/watch?v=uiQmSlH7bEo - **Cubee** — enra (Nobuyuki Hanabusa) (2019): A flat screen can suggest depth when the body pretends to enter it, a cheap trick for making AR portals feel deep. https://www.youtube.com/watch?v=lW6mEJMA-mA - **Holotronica Live (British Science Festival)** — Holotronica (Stuart Warren-Hill) (2019): Combining stereo 3D with a floating screen pushes images out into the room; AR music sets can place visuals between performer and audience rather than on a back wall. https://www.youtube.com/watch?v=0SyxkCGWHpw - **MÆ – Motion Aftereffect** — Freya Björg Olafson (2019): Stage the gap between the body in the headset and the body in the room. https://vimeo.com/428830273 - **Roots** — 1927 (Suzanne Andrade & Paul Barritt) (2019): Folk tales told through simple drawn projection show that a story needs style more than fidelity; AR storytelling can choose a handmade look too. https://www.youtube.com/watch?v=l-rx8_Hnz-k - **Équinoxe** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2019): Turn a live concert into a journey through a space built from light. https://vimeo.com/343654541 - **Camouflage** — Mária Júdová (2018): Play hide-and-seek between a real dancer and a headset wearer, watched in two worlds at once. https://vimeo.com/285149672 - **Everywhen** — Mária Júdová (2018): Stage history repeating itself as a body caught in looping images. https://vimeo.com/259204752 - **Lightflow (performance)** — visiophone (Rodrigo Carvalho) (2018): A landscape's water flow becomes a light instrument a dancer can play. https://vimeo.com/312068065 - **Live VR Painting at the Louvre** — Anna Zhilyaeva (Anna Dream Brush) (2018): Step inside a famous painting and rebuild it as a space. https://www.youtube.com/watch?v=Zs3n07Clw7A - **Media Scenography for Chinese Theater (with Tim Yip)** — Tobias Gremmler (2018): Building a stage language from a few elemental materials keeps digital scenery coherent; AR worlds also feel stronger when every effect comes from one material palette. https://www.youtube.com/watch?v=ua5EtzWgFnw - **Median** — Hiroaki Umeda (S20) (2018): A shared projected field turns several bodies into one system; multi-user AR can show relationships between people the same way. https://www.youtube.com/watch?v=cP9q7SBmjGA - **See Me Now?** — Marion Tränkle / NOW (2018): Use light to flip who watches whom in a building made for surveillance. https://vimeo.com/308889681 - **SpinWall** — Benjamin Kuperberg (2018): Make the props in a performer's hands and the wall behind them one light show. https://www.youtube.com/watch?v=dUlzz_6AU3s - **Virtual Actors in Chinese Opera** — Tobias Gremmler (2018): A virtual performer can be built from a tradition's costume language instead of a realistic human; AR characters can borrow cultural form rather than photorealism. https://www.youtube.com/watch?v=UbbJVAts-D8 - **discrete figures (with Kyle McDonald)** — Daito Manabe / Rhizomatiks (2018): Machine-learning-generated virtual dancers share the stage with real ones. https://www.youtube.com/watch?v=hauXQQhwbgM - **Dürer's Dog (Ballett Nürnberg)** — Frieder Weiss (2017): A painter's etchings come alive as a stage that moves with the dancers. https://www.youtube.com/watch?v=wopUITjQCVo - **Metallica WorldWired Tour indoor drone swarm** — Raffaello D'Andrea / Verity (2017): Points of light above an audience create a shared focus without screens; AR concerts can place a few floating lights overhead instead of a flat virtual display. https://www.youtube.com/watch?v=A4TBvBfPNVg - **phosphere** — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2017): Markers on dancers' wrists steer 24 projectors, turning gestures into geometry of light in space. https://www.youtube.com/watch?v=W7X6UqXm9eY - **CPA [Consistent Partial Attention]** — Freya Björg Olafson (2016): A dance score delivered through screens, including one in front of the dancer's eye. https://vimeo.com/179255945 - **24 drones** — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2015): Twenty-four glowing drones form a dancing constellation of light. https://www.youtube.com/watch?v=cYWvKudIIJ8 - **887** — Robert Lepage / Ex Machina (2015): Scale jumps between a model and a life-size projection make memory spatial; AR can do this by letting a tabletop model open into a room-sized scene. https://www.youtube.com/watch?v=B3mPXPkiJtk - **Butterfly under Glass** — Frieder Weiss (2015): A dancer pinned like a specimen inside living, reactive images. https://www.youtube.com/watch?v=GiSOWS9XQmM - **Dancing Light / D.O.PE.** — Klaus Obermaier (2015): Make projected light itself a dance partner for the performer. https://www.youtube.com/watch?v=KXoN8zNc6sg - **Intensional Particle** — Hiroaki Umeda (S20) (2015): Treating the body as a force that bends a virtual field is an AR interaction model that needs no buttons at all. https://www.youtube.com/watch?v=jwYUo-yyXCg - **Le mouvement de l'air** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2015): Turn invisible air currents into visible imagery swirling around the dancers. https://www.youtube.com/watch?v=xsskbGYq7lc - **Pattern Recognition** — Alexander Whitley Dance Company, Memo Akten (2015): The lighting rig watches, learns and becomes a dance partner. https://vimeo.com/136252155 - **SHIRO** — NONOTAK (Noemi Schipfer & Takami Nakamoto) (2015): Performers disappear inside a room made only of projected light. https://www.youtube.com/watch?v=DcGHcsyuXuE - **TORN** — Kat Sullivan (2015): Film a dancer as a 3D volume you can tear apart and fly around. https://vimeo.com/137311875 - **This Floating World** — Tim Murray-Browne (2015): The dancer grows the landscape she dances in, like a vine shaped by a wall. https://www.youtube.com/watch?v=D1ZEzkMCNsI - **shadow (drone with a spotlight)** — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2015): Spotlights carried by drones make the dancer's shadows part of the dance. https://www.youtube.com/watch?v=hX2TneyE41Q - **Breakdown** — visiophone (Rodrigo Carvalho) (2014): Perform against a world whose physics keep changing. https://vimeo.com/95846156 - **Golem** — 1927 (Suzanne Andrade & Paul Barritt) (2014): A projected assistant that slowly takes control is a sharp warning for AR assistants; the form carries the critique. https://www.youtube.com/watch?v=1oizDDrC2i8 - **Heliopolis** — Pablo Ventura (2014): A city that records every citizen's path as a visible trace. https://vimeo.com/109680105 - **Icare (Icarus)** — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (2014): Flight is easy to fake when the surroundings move instead of the body; AR can make people feel they fly by moving the world around them. https://www.youtube.com/watch?v=loExb1r9ayw - **Jeu de modes** — Palindrome (Robert Wechsler) (2014): Map the scale of a gesture, small, conversational or explosive, to a mode of listening. https://vimeo.com/108061408 - **Pathfinder** — Christian Mio Loclair / Waltz Binaire, Motion Bank / Choreographic Coding Lab (2014): Reverse the usual pipeline: the graphics lead and the dancers follow. https://www.youtube.com/watch?v=8Uh0aK3ATS8 - **Pixel (with Mourad Merzouki / Compagnie Käfig)** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2014): Hip-hop dancers move together with pixels of light on the floor. https://www.youtube.com/watch?v=z_Hu57QTqqE - **Porsche Macan: 3D Mapping & Dance Performance** — Sila Sveta (2014): Pairing a real dancer with a projected double creates a dialogue between body and image; AR performance can use the same trick with a virtual clone. https://www.youtube.com/watch?v=Xadmwk2qAVE - **Prince Mio improvise to Interactive Sound** — Christian Mio Loclair / Waltz Binaire (2014): The dancer does not dance to the music; the music comes out of the dance. https://www.youtube.com/watch?v=RtUEML0ncJk - **SPARKED (with Cirque du Soleil)** — Raffaello D'Andrea / Verity (2014): A flying lampshade becomes a character through timing and motion alone; AR objects can gain personality from how they move toward and away from people. https://www.youtube.com/watch?v=6C8OJsHfmpI - **Shiver** — Compagnie Nicole Seiler (2014): Your own live double, projected onto you, starts to behave suspiciously. https://vimeo.com/92021691 - **The ADA Project** — Conrad Shawcross (2014): A robot becomes a dancer when its motion follows music — AR characters don't need faces, only well-timed movement. https://www.youtube.com/watch?v=ndSth0XSEUw - **The Measures Taken** — Alexander Whitley Dance Company, Marshmallow Laser Feast (2014): A stage that is drawn by the way a machine 'sees' the dancers. https://vimeo.com/85073837 - **Torque starter** — enra (Nobuyuki Hanabusa) (2014): Mixing a real prop with virtual consequences is more convincing than a fully virtual object; AR shines when a real thing triggers the effect. https://www.youtube.com/watch?v=3JdT4fDi4iI - **Visions of America: Amériques** — Refik Anadol (2014): Let a conductor's gestures conduct the projections across an entire concert hall in real time. https://www.youtube.com/watch?v=U-9VAPC92Bw - **Waltz Binaire — Dance and Interactive Media (Diesel Reboot)** — Christian Mio Loclair / Waltz Binaire (2014): A laptop, a Kinect and a projector are enough to make spontaneous dance visible. https://www.youtube.com/watch?v=TPDrSMfbaI0 - **fly (dance with drones)** — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2014): Make drones dance partners that fly in sync with the dancers. https://www.youtube.com/watch?v=HQLORg5COiU - **Choreophony** — Pablo Ventura (2013): Choreography that writes its own polyphonic soundtrack. https://www.youtube.com/watch?v=EtQ31OHI60o - **FLOW 1** — Christian Mio Loclair / Waltz Binaire (2013): Give the computer access to a street dancer's private flow. https://www.youtube.com/watch?v=ISKV1BeB3pM - **FUMA-KAI** — enra (Nobuyuki Hanabusa) (2013): Graphics that seem to come from a gesture make the performer look powerful; AR effects should appear to start from the user's hand. https://www.youtube.com/watch?v=HhJeNgjIKVw - **Gravitacional (visiophone, CCL Frankfurt 2013)** — Motion Bank / Choreographic Coding Lab (2013): Turn the dancer's hands into gravity wells that sculpt a particle cloud. https://vimeo.com/81705600 - **HYPER_** — Freya Björg Olafson (2013): Make a live body and a stereoscopic image trade places. https://vimeo.com/104413592 - **Hakanaï** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2013): A dancer in a cube of gauze converses with live projected lines, like a short poem. https://www.youtube.com/watch?v=xvJNia3z11I - **Hologauze** — Holotronica (Stuart Warren-Hill) (2013): A semi-transparent layer between audience and stage creates floating content without glasses; the same layering logic (content in front of, around and behind real people) is the grammar of AR. https://www.youtube.com/watch?v=LctpKrzGvQE - **Lumière** — Robert Henke (2013): When every visual event is also a sound event, the audience believes the image; AR feedback that pairs light and sound one-to-one feels physical. https://www.youtube.com/watch?v=fJzGd_KfLeY - **Needles and Opium (Les Aiguilles et l'opium)** — Robert Lepage / Ex Machina (2013): Rotate the room and gravity looks different; AR can reorient a virtual room around the user for the same disorientation. https://www.youtube.com/watch?v=6jWBCnIHe9A - **Nosaj Thing 'Eclipse/Blue' music video** — Daito Manabe / Rhizomatiks (2013): Motion-captured projections reshape themselves around the dancer's body in real time. https://www.youtube.com/watch?v=_woNBiIyOKI - **Perfume at Cannes Lions** — Daito Manabe / Rhizomatiks (2013): A pop group pushes translucent screens around the stage while live projection merges with their moves. https://www.youtube.com/watch?v=UbLLIhCvTQ8 - **Projection + Dance** — Gene Kogan (2013): Projection that knows where the dancer's body is in 3D. https://vimeo.com/81914893 - **cube (Sónar Tokyo)** — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2013): Dancers move inside cube frames as their motion turns into live light and graphics. https://www.youtube.com/watch?v=zBm3mJiJzh8 - **pleiades** — enra (Nobuyuki Hanabusa) (2013): Two bodies and one shared cosmos show how a virtual layer can unite performers; shared AR works when everyone sees the same object respond. https://www.youtube.com/watch?v=0813gcZ1Uw8 - **Dance Jockey** — visiophone (Rodrigo Carvalho) (2012): The dancer is the DJ and VJ at once. https://vimeo.com/43462444 - **ISAM Live 2.0** — Amon Tobin — ISAM Live (with V Squared Labs, Vita Motus, Leviathan, Blasthaus) (2012): Showing and hiding the performer through the mapped surface plays with what is real behind the image, a key tension in AR occlusion. https://www.youtube.com/watch?v=ba_4oOpn9Qw - **Madonna Super Bowl XLVI Halftime Show** — Moment Factory (2012): Projecting on the floor seen from above transforms a whole venue; AR designers can treat the ground plane as the biggest, most flexible canvas. https://www.youtube.com/watch?v=WqseMKmd3Bo - **Nikola Tesla in Sound and Light** — Marco Tempest (2012): A pop-up book becomes a projection-mapped stage for a live magic story. https://vimeo.com/42402467 - **The Magic Flute (Die Zauberflöte, with Barrie Kosky)** — 1927 (Suzanne Andrade & Paul Barritt) (2012): Pinning performers to fixed points on a projected wall trades freedom for perfect registration, a useful trade-off for AR scenes that must stay aligned. https://www.youtube.com/watch?v=wDTtRhxtm7E - **primitive** — enra (Nobuyuki Hanabusa) (2012): Virtual objects feel solid when the body reacts at the exact right frame; timing, not tracking, sells the contact. https://www.youtube.com/watch?v=IALr6M2NXsE - **superposition** — Ryoji Ikeda (池田亮司) (2012): Performers can be data operators rather than dancers; in AR theatre, visible live input (typing, tapping) makes the digital layer feel caused by people. https://www.youtube.com/watch?v=0ivkmVDg4D0 - **000000swan** — Phoenix Perry (2011): Teach the computer your own gestures instead of using its built-in ones. https://www.youtube.com/watch?v=dpW0wRkijQY - **Coïncidence** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2011): A duet between a body and a live-played image, where letters behave like juggled objects. https://vimeo.com/35528568 - **DUMMY lab** — Frieder Weiss (2011): Acrobatics against a wall of light that answers every fall and lift. https://www.youtube.com/watch?v=FnbvSRylnPs - **Divide By Zero** — Hellicar & Lewis (Pete Hellicar & Joel Gethin Lewis) (2011): A dance under hypnotic guidance turns unconscious movement into a visible interface. https://vimeo.com/19487686 - **Du, Liebe** — Frieder Weiss (2011): Projection mapping that follows people instead of buildings. https://www.youtube.com/watch?v=nZojuHerXR4 - **Holistic Strata** — Hiroaki Umeda (S20) (2011): A particle field that behaves like a medium around the dancer shows how AR can make empty space feel thick and physical. https://www.youtube.com/watch?v=2uQqxLpnueE - **ISAM Live** — Amon Tobin — ISAM Live (with V Squared Labs, Vita Motus, Leviathan, Blasthaus) (2011): When the whole set is one mapped object, the performer becomes part of the image; AR stages can wrap virtual worlds around a real body the same way. https://www.youtube.com/watch?v=WLrt7-kIgIM - **Interactive Body Projection Mapping (Hypermetrop)** — Beam'Art (Benjamin Petit & Antoine Vanel) (2011): Clothe moving dancers in live projected light. https://vimeo.com/34609484 - **La Belle et la Bête** — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (2011): Transformation is the most powerful AR moment: showing a body change in place, right next to real people, is more affecting than any new world. https://www.youtube.com/watch?v=qExGC7Lpc7g - **Little Magic Stories** — Chris O'Shea (2011): Children act on a small stage while the characters and sets they drew come alive through projection. https://vimeo.com/20196781 - **MOTIV** — Russ Maschmeyer (2011): Let the body, not an instrument, control the emotion of music. https://www.youtube.com/watch?v=nKl2Wjto4zI - **Make the Line Dance** — 1024 Architecture (François Wunschel & Pier Schneider) (2011): Projection mapping onto a moving body instead of a static building. https://vimeo.com/21308228 - **Stocos** — Instituto Stocos (Muriel Romero, Pablo Palacio & Daniel Bisig) (2011): Let dancers and a simulated swarm choreograph each other on the same floor. https://www.youtube.com/watch?v=MRTGNMYyGUY - **Bernstein's Mass, Royal Festival Hall** — Yeast Culture (2010): With a huge cast on stage, images must frame people rather than cover them; AR in crowded spaces should leave room for the humans. https://www.youtube.com/watch?v=XXnUAMvwSps - **Cinématique** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2010): Let a dancer push projected letters and lines as if they were objects with weight. https://www.youtube.com/watch?v=CknHVdMZ5xg - **Der Ring des Nibelungen (Metropolitan Opera)** — Robert Lepage / Ex Machina (2010): A moving physical surface plus projection gives virtual landscapes real slopes to walk on, the same pairing AR needs between content and real geometry. https://www.youtube.com/watch?v=1_ssNfEXu_0 - **Ghost Projection for Richard III** — Golan Levin (2010): Project static noise only onto the 'ghost' actors on stage, turning them into phantoms. https://vimeo.com/11013624 - **Intangible States (with Stray Dogs)** — Yannick Jacquet (Legoman) (2010): Project an urban drifter's dreams onto a stage built from a pile of boxes. https://www.youtube.com/watch?v=cgU7F42vgj4 - **Kylie Minogue — Get Outta My Way** — Frieder Weiss (2010): Pop choreography shot live inside a responsive light set instead of adding effects in post. https://www.youtube.com/watch?v=BHGaW8lBlSk - **Magic Projection (live at TEDxTokyo)** — Marco Tempest (2010): A handheld blank board becomes a living screen because the projection follows it. https://vimeo.com/11801074 - **Reactive Stage** — visiophone (Rodrigo Carvalho) (2010): Layer projections on sheer fabric to put a dancer inside the image. https://vimeo.com/12171899 - **The Animals and Children Took to the Streets** — 1927 (Suzanne Andrade & Paul Barritt) (2010): Precise alignment between a live head and a drawn body makes one character from two media; AR face and body effects rely on the same registration. https://www.youtube.com/watch?v=hTNeZsIRaKA - **the concept of ... (here and now)** — Klaus Obermaier (2010): Choreograph for both the audience and the camera so bodies merge into impossible shapes on screen. https://www.youtube.com/watch?v=afbQQM6RTe8 - **AVATAR** — Freya Björg Olafson (2009): Perform the body as an internet profile picture come to life. https://vimeo.com/6602039 - **Live Motion Capture with Matthew Barley (with Centroid)** — Yeast Culture (2009): A musician's gesture is already choreography; capturing it lets the instrument play the visuals as well as the sound, as body-tracked AR can do. https://www.youtube.com/watch?v=BWXJtxHLZGA - **Ludovico Einaudi at the Royal Albert Hall** — Yeast Culture (2009): Quiet music needs quiet images; AR for a performance should match tempo and restraint, not compete with the performer. https://www.youtube.com/watch?v=laOTwxhmsyA - **loopdiver** — Mark Coniglio / Troika Ranch (with Dawn Stoppiello) (2009): Humans perform an edit made by a computer, glitches and all. https://vimeo.com/15062917 - **Adapting for Distortion** — Hiroaki Umeda (S20) (2008): Projecting a pattern onto a body changes how we read its shape, a direct lesson for AR body effects and camouflage. https://www.youtube.com/watch?v=N1i3MA13v9I - **BOOMBOX** — 1024 Architecture (François Wunschel & Pier Schneider) (2008): A stage made of simple boxes becomes a musical instrument of light. https://vimeo.com/15734398 - **Eye Movement** — Palindrome (Robert Wechsler) (2008): Turn the smallest, fastest movement of the body, the eyes, into an instrument. https://www.youtube.com/watch?v=bFnqHgmNPwE - **Mortal Engine (Chunky Move)** — Frieder Weiss (2008): Darkness as the default, with the body only existing where the computer chooses to light it. https://www.youtube.com/watch?v=sbjOMualLVs - **Nine Inch Nails: Lights in the Sky** — Moment Factory (2008): When images respond to where the performer stands, the screen stops being a backdrop and becomes a partner; body-tracked AR should react to people, not just sit behind them. https://www.youtube.com/watch?v=iD7BNesOIGE - **Prince Charles hologram (World Future Energy Summit)** — Musion (Musion Eyeliner) (2008): A virtual presence can replace travel when it respects the stage conventions of a real speaker; this is the idea behind holographic telepresence in AR. https://www.youtube.com/watch?v=wphytMJ_F_A - **Telstra live hologram telepresence** — Musion (Musion Eyeliner) (2008): Live telepresence changes a projection effect into a conversation; the moment the remote person can answer, the audience treats them as present. https://www.youtube.com/watch?v=P3jhFXmNUt8 - **Between the Devil and the Deep Blue Sea** — 1927 (Suzanne Andrade & Paul Barritt) (2007): A drawn world becomes inhabitable when a real person stands in it; hand-drawn AR can feel warmer than realistic 3D. https://www.youtube.com/watch?v=0xIKO3--kq8 - **La Pietra del paragone (Rossini, Théâtre du Châtelet)** — Pierrick Sorin (2007): Showing the real stage and the composited image side by side turns the seam into the show, a model for AR performances where the audience watches both the performer and the augmented feed. https://www.youtube.com/watch?v=CzjS3MCjBzI - **NORMAN** — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (2007): Hand-drawn animation next to a real dancer shows that AR does not have to be photoreal; a line that reacts to a body is already magic. https://www.youtube.com/watch?v=62OwWsqt59g - **16 [R]evolutions** — Mark Coniglio / Troika Ranch (with Dawn Stoppiello) (2006): Each gesture leaves a living 3D trace that becomes part of the set. https://www.youtube.com/watch?v=LJ3_AOBX6TM - **A Human Conversation** — Palindrome (Robert Wechsler) (2006): Let a silent conversation of gestures be heard as music. https://www.youtube.com/watch?v=WFgEO5G2jsE - **Glow (Chunky Move)** — Frieder Weiss (2006): Light that behaves like a living skin, growing out of the body's silhouette. https://www.youtube.com/watch?v=C4He543_a80 - **Jenseits der Schatten** — Palindrome (Robert Wechsler), Frieder Weiss (2006): Stage Plato's cave literally: the audience watches bodies and the shadows the system makes of them. https://www.youtube.com/watch?v=LQAGrlC6xnY - **John McEnroe hologram (Wimbledon Lawn Tennis Museum)** — Musion (Musion Eyeliner) (2006): A virtual guide placed in the exact room where history happened is more convincing than any screen; AR museum guides should stand in the space, not in a panel. https://www.youtube.com/watch?v=SfjAxPYpQck - **Kubic's Cube** — Pablo Ventura (2006): A robot becomes the only dancer, and the audience walks around it like a sculpture. https://www.youtube.com/watch?v=xBki9nYeZ9Q - **Le Sacre du Printemps (The Rite of Spring)** — Klaus Obermaier (2006): Turn a dancer into a live stereoscopic 3D image so the audience watches a 'reshaped body' through 3D glasses. https://www.youtube.com/watch?v=_6NQf-a5UAc - **Pixel Babes** — Compagnie Nicole Seiler (2006): Media images of perfect bodies are literally pasted onto real ones. https://vimeo.com/190371062 - **Convergence 1.0** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2005): A juggler juggles with objects that only exist inside the computer. https://vimeo.com/954911 - **Dolls/Dolls Live** — Compagnie Nicole Seiler (2005): Put projected phantoms and living dancers in the same dark room and let visitors meet both. https://vimeo.com/190992843 - **Gorillaz hologram (MTV Europe Music Awards)** — Musion (Musion Eyeliner) (2005): A virtual character sharing a real stage is the core promise of AR performance, and this shows that a flat image works if the reflection, scale and lighting match the stage. https://www.youtube.com/watch?v=CRViE4N-u5Y - **La Tempête (The Tempest)** — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (2005): Magic in a story is the natural home for mixed reality: when a plot already contains spirits, virtual characters feel motivated rather than decorative. https://www.youtube.com/watch?v=rsiOM3-Oaf8 - **Richard Branson hologram (Virgin Digital launch)** — Musion (Musion Eyeliner) (2005): The designers chose to make the figure slightly transparent on purpose; AR avatars also need a visual 'this is virtual' cue so people trust what they see. https://www.youtube.com/watch?v=x4FO6HFe6rA - **Talking Bodies** — Palindrome (Robert Wechsler), Frieder Weiss (2005): Treat the moving body as a speaker that produces both sound and text. https://www.youtube.com/watch?v=4YKeRrsx9BM - **The Andersen Project (Le projet Andersen)** — Robert Lepage / Ex Machina (2005): Small physical cut-outs in a projected world give the performer real places to enter, the way AR portals are strongest when framed by a real doorway. https://www.youtube.com/watch?v=zMIhuuncY6I - **Apparition (with Ars Electronica Futurelab)** — Klaus Obermaier (2004): Dancers' moves generate projections live on the backdrop and on their bodies, so body and image pull on each other. https://www.youtube.com/watch?v=-wVq41Bi2yE - **Madame K** — Compagnie Nicole Seiler (2004): Projection as make-up: the image a body wants to show versus the body itself. https://vimeo.com/190536002 - **Surfacing** — Mark Coniglio / Troika Ranch (with Dawn Stoppiello) (2004): Your image is confined while your body is free, so which one are you? https://vimeo.com/111712677 - **Future of Memory** — Mark Coniglio / Troika Ranch (with Dawn Stoppiello) (2003): Live video as memory: what the camera keeps is distorted each time it is replayed. https://vimeo.com/112552170 - **Shadows** — Frieder Weiss (2003): Your shadow stops obeying you. https://www.youtube.com/watch?v=gm_doxsdqG4 - **Anima** — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (2002): The clash between a real body and its projected double makes the audience feel the gap between body and soul; AR doubles can carry meaning, not just spectacle. https://www.youtube.com/watch?v=MMFtNe2Er_k - **Brother/Sister (Blinde Liebe)** — Palindrome (Robert Wechsler), Frieder Weiss (2002): Let the dancers' bodies play part of an opera score in real time. https://www.youtube.com/watch?v=g8aWaOn5eNo - **Maibaum (Baila Mi Ritmo)** — Palindrome (Robert Wechsler), Frieder Weiss (2002): Make the costume itself part of the music, and let the dancers play it with their movement. https://www.youtube.com/watch?v=_pHoyecfO8k - **Vivisector (with Chris Haring)** — Klaus Obermaier (2002): Light the dancers only with projection so their bodies get sliced and dissolved by light. https://www.youtube.com/watch?v=Q0YEFX6Nk9k - **VorOrt** — Palindrome (Robert Wechsler) (2002): Stitch three kinds of presence, recorded, live on screen and live in the room, into one continuous walk. https://www.youtube.com/watch?v=9tFnMzH5UZg - **while going to a condition** — Hiroaki Umeda (S20) (2002): One body and one field of light is enough; AR designers can learn how little content is needed when timing and contrast are strong. https://www.youtube.com/watch?v=XQE444jISmk - **Heisenberg's Uncertainty Principle** — Palindrome (Robert Wechsler), Frieder Weiss (2001): Use projection on a see-through screen to make the dancers' position uncertain. https://www.youtube.com/watch?v=Xgnw39zrkCc - **Reine Rien** — Mark Coniglio / Troika Ranch (with Dawn Stoppiello) (2001): Dancing makes weather; stillness makes silence. https://vimeo.com/111764665 - **ZONE (Kuka robot choreography)** — Pablo Ventura (2001): Choreograph an industrial robot as a dancer next to humans. https://www.youtube.com/watch?v=PD1i6rItAfA - **...seine hohle Form...** — Frieder Weiss (2000): A piece of music that cannot be heard unless someone dances it. https://vimeo.com/8895552 - **The Far Side of the Moon (La Face cachée de la Lune)** — Robert Lepage / Ex Machina (2000): One reconfigurable surface can stand for many places; AR can reuse a single real wall the same way instead of building new sets. https://www.youtube.com/watch?v=K_HI70pLYhc - **memorandum** — Dumb Type (1999): A semi-transparent layer between the viewer and the performer is exactly an AR display; memorandum shows how text and image on that layer can rewrite the person behind it. https://www.youtube.com/watch?v=H9MlE1eXvGM - **D.A.V.E. – digital amplified video engine (with Chris Haring)** — Klaus Obermaier (1998): Use the dancer's own body as the screen and project impossible transformations onto it. https://www.youtube.com/watch?v=1bhNjYTQFQY - **OR** — Dumb Type (1997): Flashes of light can make a real person appear and disappear, a stage-scale version of the fades and occlusion an AR designer uses to add or remove things from reality. https://www.youtube.com/watch?v=2ZMYPXvR_TU - **Music Plays Images x Images Play Music (with Ryuichi Sakamoto)** — Toshio Iwai (岩井俊雄) (1996): Placing imagery exactly over the source of sound fuses the two senses; AR music visuals work best when they sit on the instrument itself. https://www.youtube.com/watch?v=QRHwP82WiK4 - **Grand Hôtel des Étrangers** — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (1994): A single room shared by one real body and several projected ones is the basic unit of mixed-reality drama. https://www.youtube.com/watch?v=JJF4dC69Kg8 - **pH** — Dumb Type (1990): A machine that sweeps the space turns people into data being read; AR interfaces can also 'scan' users, and designers can make that sensing visible and physical. https://www.youtube.com/watch?v=kjWm4zm94lE - **Kouzelný cirkus (Wonderful Circus)** — Laterna Magika (Alfréd Radok & Josef Svoboda) (1977): Comedy is a good test for mixed reality: if the gag between the real clown and the projected one lands, the audience has accepted both as the same world. https://www.youtube.com/watch?v=LrUMYz5_Syc - **TV Cello** — Nam June Paik (白南准) (1971): An instrument that is also a screen merges performer, sound and image into one object, a model for AR instruments that show what they play. https://www.youtube.com/watch?v=-9lnbIGHzUM - **Robot K-456** — Nam June Paik (白南准) (1964): Taking a performing machine into the street makes the city its stage; location-based AR characters can perform in public space the same way. https://www.youtube.com/watch?v=JciiqCsRhdY - **Laterna Magika (Expo 58 programme and repertoire)** — Laterna Magika (Alfréd Radok & Josef Svoboda) (1958): A live body and a filmed body can pass one action back and forth; the illusion lives in the handover, the same seam an AR designer must hide when a virtual character meets a real person. https://www.youtube.com/watch?v=pmXrna52etU - **Cirque Calder (Calder's Circus)** — Alexander Calder (1926): A tabletop world of tiny performers animated by hand is still one of the most charming formats for AR: small scale, close attention and visible manipulation. https://www.youtube.com/watch?v=t6jwnu8Izy0 ### Immersive rooms & VR Rooms, domes and virtual worlds that surround you completely. - **EchoVision (Ars Electronica Deep Space 8K edition)** — Jiabao Li, Botao 'Amber' Hu (2025): On a giant wall and floor, visitors' calls turn into bat-like echolocation waves. https://www.youtube.com/watch?v=W0TSj-LilLw - **Protist Reverie** — Uncharted Limbo Collective (George Adamopoulos, Eleana Polychronaki, Chris Waters) (2025): A digital organism lives across the whole dome and breathes with live music. https://www.youtube.com/watch?v=zgehFbLJuZ4 - **The Long Fall: A descent into the Ocean's Living Memory** — Jiabao Li (2025): Sink down with the plankton and see how the ocean remembers carbon. https://www.youtube.com/watch?v=fe-Xd3gxcsU - **En amour** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2024): A walk-through ritual that lets a group feel a love story as changing light and sound. https://vimeo.com/963236356 - **LILITH.AEON** — AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) (2024): A virtual being dances back at the crowd, so every visit becomes a new duet between people and an AI. https://vimeo.com/925210197 - **Phish at Sphere** — Moment Factory (2024): An improvising band needs improvising visuals; live AR performances need real-time operators, not only pre-baked scenes. https://www.youtube.com/watch?v=aP-y5CDJBUs - **Soul Paint** — Sarah Ticho (2024): Paint your feelings where they live in your body, then see everyone else's. https://www.youtube.com/watch?v=BmYAtoJA9Wk - **Graphical Representation of Dance in VR (Motion Bank)** — Jorge Guevara (2023): A dance score you can step inside instead of read. https://www.youtube.com/watch?v=DeNW3NMeh-Q - **Inner Room (installation)** — Zhou Zhou (2023): Your hand sculpts a room and its sound at the same time. https://www.youtube.com/watch?v=sa7peuzAn6w - **SunForceOceanLife** — Ernesto Neto (2023): When the floor itself gives way, visitors feel a space with their whole body; AR can borrow this by tying virtual space to balance, steps and posture, not only to sight. https://www.youtube.com/watch?v=vT4zn_y6Kiw - **Dernière minute** — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2022): A ritual of passage in which the floor you lie on turns into rising water. https://vimeo.com/725276687 - **PAN+TILT** — Gibson/Martelli (Ruth Gibson & Bruno Martelli) (2022): Use a somatic dance technique as the controller for a virtual world. https://vimeo.com/758291757 - **Social Sacrifice** — Studio DRIFT (2022): Being inside the swarm feels different from watching it; AR swarms can be placed around the viewer's body so they move through the group. https://www.youtube.com/watch?v=s9RAZ0PG6T8 - **AIive: Interactive Visualization and Sonification of Neural Networks in Virtual Reality** — Zhuoyue Lyu (2021): Make an invisible process tangible: a learning machine you can walk around, touch and hear getting better. https://www.youtube.com/watch?v=kE-NDBdZ9AU - **Constellation of the Flesh** — Mária Júdová (2021): Can the dance of trance be digitised and felt from inside? https://vimeo.com/577351369 - **Grove** — Philip Beesley / Living Architecture Systems Group (2021): Surrounding a visitor with many soft, reacting elements creates immersion without any screen — a reminder that AR can build space with layers, not just objects. https://www.youtube.com/watch?v=f-wQeHNae48 - **KID A MNESIA EXHIBITION** — Stanley Donwood (with Radiohead) (2021): An album becomes architecture: each song is a room you walk through. https://www.youtube.com/watch?v=AOinMjQ9jo8 - **Sleepwalking in the Forbidden City** — Cai Guo-Qiang (蔡国强) (2021): Virtual space lets an artist do the fireworks that are forbidden in reality; AR and VR can stage impossible events in sensitive places without touching them. https://www.youtube.com/watch?v=BIBFvH-Whqs - **The Changing Same** — Scatter / Depthkit (James George, Alexander Porter, Yasmin Elayat) (2021): Time travel through one place to show that history repeats in cycles. https://www.youtube.com/watch?v=tugi4v7S32o - **DAZZLE: A Re-assembly of Bodies** — Gibson/Martelli (Ruth Gibson & Bruno Martelli) (2020): A ball where everyone, audience included, dances as a live avatar. https://vimeo.com/868428016 - **KYKEON** — Mária Júdová (2020): Use VR to invent a shared ritual built from dancers' motion. https://vimeo.com/493291875 - **Quantum Memories** — Refik Anadol (2020): A flat screen can feel volumetric when the content is rendered for one sweet spot; AR designers can use forced perspective to fake depth on flat surfaces. https://www.youtube.com/watch?v=2Rlgphw9Jxw - **Empa** — Katie Dale-Everett Dance (KDE Dance) (2019): See yourself through your partner's eyes while you move together. https://www.youtube.com/watch?v=TKyzy7ttthU - **FLOWERS BY NAKED 2019** — NAKED, INC. (Ryotaro Muramatsu) (2019): Mixing real flowers with projected ones blurs which is which, the core pleasure of a good AR blend. https://www.youtube.com/watch?v=KAjNzRf9Y-k - **Machine Hallucination** — Refik Anadol (2019): Walking through a model's latent space makes an invisible dataset feel like a place; AR can present data as an environment you stand inside rather than a chart. https://www.youtube.com/watch?v=x1EVhNM-uf4 - **data-verse** — Ryoji Ikeda (池田亮司) (2019): Zooming through orders of magnitude in one gesture explains scale better than any label; an AR 'powers of ten' zoom anchored in the room makes science tangible. https://www.youtube.com/watch?v=BmEz0nTl4qw - **GaiaMotherTree** — Ernesto Neto (2018): A room-within-a-room placed in a busy public hall creates a pause zone; AR can carve a calm, soft space inside a crowded place without moving anything physical. https://www.youtube.com/watch?v=eKhyy3qEYms - **It Will End in Stars** — Nathalie Djurberg & Hans Berg, Acute Art (2018): The artwork looks back at you and reacts to your presence. https://www.youtube.com/watch?v=25laN3xx2GM - **My Room Is Another Fish Bowl** — Philippe Parreno (2018): Objects that drift at eye level and react to air turn a room into water without any screen; AR creatures feel present when they avoid or bump into the body. https://www.youtube.com/watch?v=B21g3MFnpUI - **The Other Way** — Shengzhi Wu (2018): Use a real bicycle as the controller so the body feels the journey. https://www.youtube.com/watch?v=xcUKUKx6DDo - **We Live in an Ocean of Air** — Marshmallow Laser Feast (2018): Visualize the breath exchange between people and trees in a shared space. https://vimeo.com/303589503 - **teamLab Borderless** — teamLab (2018): Content that wanders between spaces makes a building feel alive and worth exploring; location-based AR can let characters migrate between anchors. https://www.youtube.com/watch?v=Xy6Vz3rkd1w - **A Colossal Wave** — Marshmallow Laser Feast (2017): Mix a headset vision with real physical impacts so the whole room shakes with the virtual event. https://vimeo.com/244047652 - **A Walk through the Line** — Chiharu Shiota (塩田千春) (2017): A line drawing can be scaled up until the viewer is inside it; AR drawing tools become stronger when strokes are big enough to walk through rather than look at. https://www.youtube.com/watch?v=5UmbVYwyiqQ - **DUST** — Mária Júdová (2017): Experience dance from the point of view of a speck of dust inside it. https://vimeo.com/210525711 - **WHIST** — AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) (2017): The audience's own attention and wandering assemble a personal cut of a Freudian dance story. https://vimeo.com/799520099 - **Zero Days VR** — Scatter / Depthkit (James George, Alexander Porter, Yasmin Elayat) (2017): Give an invisible event a body and a space you can stand in. https://www.youtube.com/watch?v=E_NZEdeh2cA - **Anywhen** — Philippe Parreno (2016): An exhibition can be a timeline, not a set of objects; AR experiences can choreograph when things appear, not only where. https://www.youtube.com/watch?v=M1RWxQaM5mc - **Dead Mall Chunks** — Claire Hentschker (2016): Use other people's videos to archive places that are disappearing. https://www.youtube.com/watch?v=QL7JgrXcJHw - **Flock: A Holojam Experience** — Ken Perlin — NYU Future Reality Lab (2016): A group of people become a flock of birds together in a shared space. https://www.youtube.com/watch?v=BZXRX-kDj5M - **Pixel Forest** — Pipilotti Rist (2016): Exploding a screen into its pixels lets people walk inside an image; volumetric AR content can be shown as sparse points that surround the body. https://www.youtube.com/watch?v=yRnDHu0Fmtk - **Treehugger: Wawona** — Marshmallow Laser Feast (2016): A physical hug is the interface to the invisible life inside a tree. https://www.youtube.com/watch?v=if0wfysmoMU - **Uncertain Journey** — Chiharu Shiota (塩田千春) (2016): Thousands of thin lines in one color can turn an empty room into a solid-feeling volume; in AR, line density alone can define a space without any walls. https://www.youtube.com/watch?v=6S3RCLaPhMg - **Underwater Pavilions** — Doug Aitken (2016): A sculpture you can only visit by entering another element (water) shows how an AR piece can require a special trip or state to be seen. https://www.youtube.com/watch?v=gzGgzf361ZE - **Decision (Isomeric Slides)** — Carsten Höller (2015): Choice is a material: an AR tour can split into two paths at a physical doorway, so the visitor's decision becomes part of the work. https://www.youtube.com/watch?v=yAzyXVCelGE - **Flowers and People, Cannot be Controlled but Live Together** — teamLab (2015): Projected flowers bloom or fall as visitors touch them or stand still. https://www.youtube.com/watch?v=arafX3Es6JQ - **H {N)Y P N(Y} OSIS** — Philippe Parreno (2015): Moving the audience itself (rotating the seats) is a strong way to direct attention; AR can guide a viewer's body instead of adding arrows. https://www.youtube.com/watch?v=Opa4J9VXReo - **Highsight** — Kyle McDonald (2015): Put on a headset and see through a real camera on a wire as it plunges through a model and into the crowd. https://vimeo.com/144061990 - **Holojam** — Ken Perlin — NYU Future Reality Lab (2015): A group in wireless headsets becomes cartoon avatars in one room and draws in the air together. https://www.youtube.com/watch?v=kzx5igORwk4 - **Imponderable** — Tony Oursler (2015): The history of spirit photography is a history of mixed reality; tricks that once convinced people ghosts were real are still the core of believable AR. https://www.youtube.com/watch?v=zqyEt7YtNB8 - **In the Eyes of the Animal** — Marshmallow Laser Feast (2015): Swap human senses for animal senses inside the very place you are standing. https://vimeo.com/140057053 - **Infinity Room** — Refik Anadol (2015): Projection and mirrors make a small room look endless. https://www.youtube.com/watch?v=p9Cj1PdmtMA - **MIRROR (M2)** — Schnelle Bunte Bilder (with kling klang klong) (2015): See yourself dissolve into the sound you make. https://vimeo.com/140405048 - **Quantum Space** — Kuflex (Igor Tatarnikov & Denis Perevalov) (2015): Walk into a room and your body dissolves into quanta of light. https://vimeo.com/120944206 - **The Key in the Hand** — Chiharu Shiota (塩田千春) (2015): Thousands of personal objects hung in one web make memory visible as a volume; AR can let a crowd's contributions hang in a shared space the same way. https://www.youtube.com/watch?v=u_M40SwNw0w - **VERSUS** — NONOTAK (Noemi Schipfer & Takami Nakamoto) (2015): A whole dome becomes one moving optical illusion. https://www.youtube.com/watch?v=OrkI6WW2bIo - **CLOUDS** — Scatter / Depthkit (James George, Alexander Porter, Yasmin Elayat) (2014): Film people as data, so the documentary itself can be rearranged and explored like code. https://www.youtube.com/watch?v=KefV_ZAsOxo - **Houston Penetrable** — Jesús Rafael Soto (2014): A volume that is solid from afar and dissolves when you enter it: AR can hide shapes in point grids that only resolve from certain distances or angles. https://www.youtube.com/watch?v=976Ghk1Ue4E - **Leviathan Mixed Reality Props** — USC World Building Media Lab (Alex McDowell, Bradley Newman et al.) (2014): Touch the virtual world through real furniture. https://www.youtube.com/watch?v=BLH5VtcN9iE - **320° Licht** — URBANSCREEN (2013): Turn the inside of a giant gas holder into an ever-changing space of light. https://www.youtube.com/watch?v=X_31XyGBL7U - **Infinity Mirrored Room – The Souls of Millions of Light Years Away** — Yayoi Kusama (草間彌生) (2013): A strict time limit turns a simple light loop into a precious moment; AR experiences can use scarcity of time the same way instead of running forever. https://www.youtube.com/watch?v=7dd3ZtB4Cjs - **in orbit** — Tomás Saraceno (2013): A shared surface that transmits everyone's movement makes multiplayer presence physical, a model for shared AR spaces where others' actions are felt. https://www.youtube.com/watch?v=ROqL-8h_7DM - **SENSESCAPES** — Schnelle Bunte Bilder (with kling klang klong) (2012): Stir a design epoch with your body. https://vimeo.com/50854847 - **Infinity Mirrored Room – Filled with the Brilliance of Life** — Yayoi Kusama (草間彌生) (2011): When the body is inside the reflection, the viewer becomes part of the pattern; AR that includes the user's own reflection feels far more immersive than AR the user only looks at. https://www.youtube.com/watch?v=bT7i507OnOw - **the transfinite** — Ryoji Ikeda (池田亮司) (2011): Scale changes meaning: the same data that is abstract on a laptop becomes overwhelming at 12 metres; AR can play with the size of information to make it felt. https://www.youtube.com/watch?v=q7SEYpWPczk - **Din blinde passager (Your blind passenger)** — Olafur Eliasson (2010): When sight is reduced, the body navigates by sound and touch; AR does not have to add more to see, it can also take visibility away to make people feel space. https://www.youtube.com/watch?v=wYxxNA_WTs8 - **Aftermath of Obliteration of Eternity** — Yayoi Kusama (草間彌生) (2009): Light that appears and then vanishes on a timed breath reads as life and death; an AR designer can give virtual objects a short, visible lifespan to carry emotion. https://www.youtube.com/watch?v=aUViLK-bBaY - **Your atmospheric colour atlas** — Olafur Eliasson (2009): A room can be a colour map you walk through: AR can assign colour or sound to positions on a floor grid so that location becomes the controller. https://www.youtube.com/watch?v=lRSY61HVlBM - **Pour Your Body Out (7354 Cubic Meters)** — Pipilotti Rist (2008): Giving people a place to lie down changes how long they stay; AR experiences gain depth when they design the body's posture, not only the content. https://www.youtube.com/watch?v=89vgdELbVyQ - **The Salt Satyagraha Online: Gandhi's March to Dandi** — Joseph DeLappe (2008): A real body walking in a room carries its avatar through a shared virtual world. https://www.youtube.com/watch?v=34SxiWwOvHw - **Point A to B** — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2007): Split one virtual space across two screens so the gap itself becomes part of the course. https://vimeo.com/4377157 - **Léviathan Thot** — Ernesto Neto (2006): A soft form hung in a hard, historic space changes how the whole building feels; AR overlays can soften or re-body architecture instead of just labeling it. https://www.youtube.com/watch?v=zd-VE7eOLoQ - **Test Site** — Carsten Höller (2006): An artwork can be a body experience rather than an image: AR works gain force when they change how people move through a building, not only what they see. https://www.youtube.com/watch?v=3xC53y2DQGc - **dead-in-iraq** — Joseph DeLappe (2006): Use a shared virtual space as a temporary memorial that its own players cannot ignore. https://www.youtube.com/watch?v=ejcZ3TR5YTs - **Homo sapiens sapiens** — Pipilotti Rist (2005): Putting the image where paintings of heaven used to be reuses the building's own logic; AR works well when content respects where people already look in a place. https://www.youtube.com/watch?v=2lanq0jk1mQ - **ZENetic Computer** — Naoko Tosa (2004): A computer that asks you questions instead of answering them. https://www.youtube.com/watch?v=RYeT75F7ezQ - **Fireflies on the Water** — Yayoi Kusama (草間彌生) (2002): A few points of light plus mirrors on every side are enough to make a closet feel like the night sky, so an AR scene does not need many assets, only the right reflections. https://www.youtube.com/watch?v=ahZh_Qpb8ik - **The obliteration room** — Yayoi Kusama (草間彌生) (2002): Letting every visitor add one small mark turns a room into a shared, growing record; persistent multi-user AR annotations can make a space change over time the same way. https://www.youtube.com/watch?v=-xNzr-fJHQw - **Penetrable BBL Bleu** — Jesús Rafael Soto (1999): The artwork is completed by bodies moving through it; in AR, letting virtual strands part and swing around the user makes presence physical. https://www.youtube.com/watch?v=7zW2y9tqJBo - **Hand-drawn Spaces** — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (1998): Motion-captured dancers drawn by hand travel through the room between screens. https://vimeo.com/32776116 - **Be Now Here** — Michael Naimark (1995): Moving the viewer's body with the image (the rotating floor) creates a strong sense of place, a reminder that AR presence comes from the body, not only the eyes. https://www.youtube.com/watch?v=J2-VQFPYftM - **Le Cyclop** — Jean Tinguely (1994): A monumental object hidden in a forest that you enter becomes a destination; AR can hide large walk-in worlds at specific sites that reward the journey. https://www.youtube.com/watch?v=-lELc3auHgM - **Lovers (Teiji Furuhashi)** — Dumb Type (1994): Life-size figures that walk around you on the walls make a room feel inhabited; scale and eye contact matter more than realism when placing virtual people. https://www.youtube.com/watch?v=ZAaYEZN7EwI - **TV Garden** — Nam June Paik (白南准) (1974): Screens planted among living things become part of an ecosystem; AR content feels more at home when it is distributed through the real environment instead of floating in front of it. https://www.youtube.com/watch?v=KtWmmoYyYzI - **Topoestesia (itinerario programmato)** — Gianni Colombo (1970): Changing the ground under the feet changes perception more than any image; AR experiences can guide walking paths and pacing to shape how a space is felt. https://www.youtube.com/watch?v=nkeF48-8Vto - **Spazio elastico (Elastic Space)** — Gianni Colombo (1967): Distorting the grid that defines a space makes the space itself feel elastic; AR can warp the perceived room with glowing guide lines instead of objects. https://www.youtube.com/watch?v=gA24n_wGuss - **Room of the Present (Raum der Gegenwart)** — László Moholy-Nagy (1930): A room designed as a showcase for new media, where the space itself is part of the exhibit; AR exhibitions can treat the room as the interface. https://www.youtube.com/watch?v=OZXl5Jfg07U ## All creators and works ### Keijiro Takahashi *Creative coder; developer advocate at Unity Technologies Japan* Unity Japan engineer who open-sources hundreds of real-time VFX, depth-camera and ML-tracking experiments on GitHub and uses them in live concert visuals. #### Sunburst effects — Keijiro Takahashi (2013) - Video: https://vimeo.com/76949095 - Source code: https://github.com/keijiro/unity-sunburst-effects - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, C#, mesh animation - Idea: Generate classic radial burst graphics as live meshes. - What it is: Rays of light burst from a center and rotate in shifting stripes, like animated manga background effects. - Technique: A C# script builds and animates triangle-fan meshes each frame, varying ray width, length and rotation. - Try it: Attach a rotating sunburst mesh behind a real person's head using AR face tracking. Twist: let the burst speed up with the person's smile. #### Turbulent Flow (particle animations) — Keijiro Takahashi (2013) - Video: https://vimeo.com/80264404 - Source code: https://github.com/keijiro/unity-particle-animations - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Shuriken particle system - Idea: Shape a particle system with external forces instead of emitter settings. - What it is: Clouds of particles are pushed around by turbulence, attractors and bouncing forces in a series of short studies. - Technique: A script reads Shuriken particles every frame, applies custom forces such as noise turbulence and point attractors, and writes them back. - Try it: Place a particle cloud in AR and add an attractor at the phone position so the cloud follows the viewer. Twist: add a second attractor at a classmate's phone and let the cloud choose between them. #### Undulation — Keijiro Takahashi (2013) - Video: https://vimeo.com/80716975 - Source code: https://github.com/keijiro/unity-undulation - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Perlin noise - Idea: Contrast slow motion in the scene with a fast-moving camera. - What it is: A field of slow-moving objects rises and falls in waves while a fast camera flies through them. - Technique: Perlin noise drives the height and rotation of a grid of objects, while the camera follows a fast path through the field. - Try it: Fill a room in AR with a field of cubes that rise and fall in noise waves, and let students walk through it with the phone. Twist: make the waves calmer the slower the viewer walks. #### Virtual Kinetic Sculpture — Keijiro Takahashi (2013) - Video: https://vimeo.com/80520888 - Source code: https://github.com/keijiro/unity-kinetic-sculpture - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, C# - Idea: Rebuild a physical kinetic sculpture as a virtual one. - What it is: Hundreds of suspended elements move up and down in interlocking waves, like a kinetic sculpture hanging in a museum atrium. - Technique: Each element's height is a sum of sine waves whose phases depend on its grid position, so simple math produces rolling interference patterns. - Try it: Hang a virtual kinetic sculpture of spheres from a real ceiling in AR and compute each sphere's height from sine waves. Twist: let a raised hand in front of the camera add a new wave. #### Water Surface (isosurface) — Keijiro Takahashi (2013) - Video: https://vimeo.com/82601169 - Source code: https://github.com/keijiro/unity-isosurface-test - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, marching cubes - Idea: Build liquid shapes from invisible fields instead of modelled meshes. - What it is: A blobby liquid surface forms, wobbles and splits, generated as an isosurface from moving sources. - Technique: Metaball-like sources add up to a scalar field, and a marching cubes mesh generator extracts the isosurface every frame. - Try it: Generate a metaball liquid with marching cubes in AR and let the blobs sit on a real table. Twist: add a source at the tracked fingertip so the user can pull liquid out of the puddle. #### Boids — Keijiro Takahashi (2014) - Video: https://vimeo.com/87151096 - Source code: https://github.com/keijiro/Boids - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, C# - Idea: Three local rules are enough to make a lifelike flock. - What it is: A flock of simple shapes moves as one body, splitting and regrouping as it chases a moving target. - Technique: Classic Reynolds boids: each agent steers by separation, alignment and cohesion with its neighbours plus attraction to a target, computed in C# every frame. - Try it: Spawn a flock of paper birds in AR Foundation that follows the phone's position through the room. Twist: add AR planes as obstacles so the flock swerves around real tables and walls. #### Cut-out Fx — Keijiro Takahashi (2014) - Video: https://vimeo.com/84364022 - Source code: https://github.com/keijiro/CutoutFxTest - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, shader - Idea: A shader-only dissolve that can reveal or hide anything. - What it is: Objects appear and disappear by being eaten away along animated, noisy edges. - Technique: An alpha cut-out shader compares a noise texture against an animated threshold and discards fragments below it, giving shapes a ragged, moving boundary. - Try it: Make virtual furniture in an AR Foundation scene dissolve in when it is first placed and dissolve out when the user deletes it. Twist: use a hand-drawn texture instead of noise so each student's object dissolves in their own handwriting. #### Depthcue — Keijiro Takahashi (2014) - Video: https://vimeo.com/101211958 - Source code: https://github.com/keijiro/Depthcue - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Unity, image effect, depth buffer - Idea: Use depth as a stylistic color ramp. - What it is: Objects fade into flat color as they move away, turning depth into a clean graphic gradient. - Technique: An image effect reads the depth buffer and blends each pixel toward a gradient color according to its linear depth, a stylised form of depth cueing. - Try it: Use the AR depth texture (LiDAR or ARCore Depth) to tint the real camera image by distance on a phone. Twist: invert it so that near objects vanish and only the far wall stays visible. #### Fragments — Keijiro Takahashi (2014) - Video: https://vimeo.com/102452671 - Source code: https://github.com/keijiro/Fragments - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Shuriken particle system - Idea: Better particle effects come from interesting shapes, not just from more particles. - What it is: Shards, splinters and fragments tumble through space as particles, showing a mesh set designed for particle effects. - Technique: A public-domain library of low-poly fragment meshes is used as mesh particles in Unity's Shuriken particle system, with lighting and rotation giving each shard a solid look. - Try it: Use custom shard meshes with VFX Graph mesh output to make a real window or mirror appear to shatter in AR. Twist: gather the shards back into place when the viewer claps. #### GlitchFx — Keijiro Takahashi (2014) - Video: https://vimeo.com/102398104 - Source code: https://github.com/keijiro/GlitchFx - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, image effect - Idea: Use controlled digital corruption as an aesthetic. - What it is: The whole image breaks into blocks, color shifts and tearing, like a corrupted digital video signal. - Technique: A post-processing shader offsets blocks of the screen using a random noise texture that is refreshed at irregular intervals and splits the color channels; it is the precursor of KinoGlitch. - Try it: Apply a block-glitch post effect to the AR camera image only inside the silhouette of a virtual object. Twist: make the glitch stronger the faster the phone moves. #### HexBokeh — Keijiro Takahashi (2014) - Video: https://vimeo.com/103702704 - Source code: https://github.com/keijiro/HexBokeh - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, image effect - Idea: Bring the character of a physical lens aperture into real-time graphics. - What it is: Out-of-focus highlights bloom into hexagonal shapes, like the bokeh of a real camera lens. - Technique: A depth-of-field image effect uses three skewed separable blur passes (McIntosh's technique) to build hexagonal bokeh from the depth buffer at low cost. - Try it: Add a shallow depth-of-field effect with custom-shaped bokeh to an AR scene so virtual objects appear photographed with a real lens. Twist: let students design their own aperture shape, such as a star or heart. #### Kvant Deformer — Keijiro Takahashi (2014) - Video: https://vimeo.com/86002232 - Source code: https://github.com/keijiro/Kvant - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, vertex shader - Idea: Let noise sculpt a surface continuously on the GPU. - What it is: A smooth mesh bulges, twists and ripples under flowing noise, an early test of Keijiro's GPU effects library Kvant. - Technique: A vertex shader displaces each vertex along its normal by animated 3D noise, and normals are recalculated so lighting follows the deformation. - Try it: Deform a virtual blob placed on a table in AR with a noise vertex shader in Shader Graph. Twist: tie the noise strength to the microphone volume so the blob reacts when people talk. #### SketchyFx — Keijiro Takahashi (2014) - Video: https://vimeo.com/97597887 - Source code: https://github.com/keijiro/SketchyFx - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, image effects - Idea: Stack standard image effects to get a hand-drawn look. - What it is: A 3D scene with a walking robot is rendered as if it were drawn in pencil on crumpled paper. - Technique: Edge detection, grayscale conversion and pencil and paper texture overlays from Unity's standard image effects are chained into a sketch filter. - Try it: Apply a pencil-sketch filter to the AR camera image so the real room looks drawn, while virtual objects stay in full color. Twist: reverse it so only virtual objects look sketched. #### SlicerFx — Keijiro Takahashi (2014) - Video: https://vimeo.com/102444263 - Source code: https://github.com/keijiro/SlicerFx - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Unity, replacement shader - Idea: Make a 3D scene look like it is being scanned in slices. - What it is: Every surface in the scene is cut into thin glowing stripes that slide along, as if the world were being scanned slice by slice. - Technique: A replacement shader computes each fragment's world position and discards or lights it depending on where that position falls within a repeating slice interval, which runs even on Unity Free. - Try it: Apply the slicing stripe shader to the LiDAR scene mesh from AR Foundation so the real room appears to be scanned in bands. Twist: let the bands start from wherever the user taps. #### SonarFx — Keijiro Takahashi (2014) - Video: https://vimeo.com/102398137 - Source code: https://github.com/keijiro/SonarFx - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Unity, replacement shader - Idea: Reveal a space with an expanding pulse of light. - What it is: Rings of light pulse outward across every surface, like a sonar wave revealing a dark scene. - Technique: A replacement shader measures each fragment's distance from an origin point in world space and adds moving emissive bands with a sharp wavefront and fading tail. - Try it: Put the sonar shader on the AR mesh of a room so pulses spread out from the phone's position. Twist: trigger a new pulse whenever the microphone hears a clap. #### VJ01: Guitar Songs — Keijiro Takahashi (2014) - Video: https://vimeo.com/90006189 - Source code: https://github.com/keijiro/VJ01 - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, Unity, audio analysis - Idea: A generative visual system that listens to a song. - What it is: An experimental visual generator made in Unity animates abstract forms to guitar music by Masaya Matsuura. - Technique: Audio levels analysed in Unity drive procedural geometry, colors and camera moves, likely through an early version of Keijiro's Reaktion toolkit. - Try it: Feed the phone microphone's spectrum into an AR scene where one object per frequency band grows and glows on a table. Twist: pick a song and have each student design the object for one instrument. #### VJ02 — Keijiro Takahashi (2014) - Video: https://vimeo.com/104780871 - Source code: https://github.com/keijiro/VJ02 - Interaction: Voice & Sound, Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, Reaktion, MIDI - Idea: Play a 3D game-engine scene like an instrument, with faders. - What it is: Scrolling abstract geometry, spheres and shifting camera angles are performed live with a MIDI controller as club visuals for Channel #8. - Technique: The Reaktion toolkit maps audio levels and MIDI CC faders (a KORG nanoKONTROL) to scroll speed, object selection and camera yaw and pitch in a Unity scene. - Try it: Build a small AR stage on a table and map three on-screen or MIDI sliders to scroll speed, object type and camera angle. Twist: let loud sounds from the microphone override the sliders. #### Water Drops (pseudo refraction) — Keijiro Takahashi (2014) - Video: https://vimeo.com/85640039 - Source code: https://github.com/keijiro/UnityRefractionShader - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, shader - Idea: Fake refraction cheaply enough to use it everywhere. - What it is: Clear water drops roll through a scene, bending and reflecting the environment behind them. - Technique: A pseudo refraction shader samples an environment cube map along a bent view vector and mixes in reflections with a Fresnel term, simulating only the first refraction. - Try it: Drop virtual water blobs on a real table in AR and use the AR environment probe as the cube map so they refract the real room. Twist: let the blobs merge when they touch. #### KinoIsoline — Keijiro Takahashi (2015) - Video: https://va.media.tumblr.com/tumblr_nva7mmMPqD1qio469.mp4 - Source code: https://github.com/keijiro/KinoIsoline - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Unity, image effect, depth buffer - Idea: See a 3D scene as moving contour lines. - What it is: Glowing contour lines sweep over the scene, drawing it like a topographic map in motion. - Technique: An image effect reconstructs world-space position from the depth buffer and draws lines wherever that position crosses a repeating interval along a chosen axis, with scrolling offset. - Try it: Draw scrolling contour lines on the real room using the AR depth map or LiDAR mesh on a phone. Twist: let the lines rise with the viewer's height so the room seems to flood. #### Kvant Spray — Keijiro Takahashi (2015) - Video: https://vimeo.com/117040444 - Source code: https://github.com/keijiro/KvantSpray - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, GPGPU, instancing - Idea: Treat any mesh as a particle and spray huge numbers of them. - What it is: A continuous spray of thousands of small objects bursts out, tumbles and fades in a smooth GPU-driven stream. - Technique: Positions, velocities and rotations are simulated in float render textures by fragment shaders and read in the vertex shader of a pre-built bulk mesh, so many instances are drawn in few draw calls. - Try it: Spray thousands of small meshes in AR with VFX Graph mesh output, emitting from wherever the user taps on a detected plane. Twist: let the particles collide with the AR plane and pile up. #### Kvant Swarm — Keijiro Takahashi (2015) - Video: https://vimeo.com/134624419 - Source code: https://github.com/keijiro/KvantSwarm - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, GPGPU, render texture - Idea: Draw each particle's path as a line, so a swarm turns into flowing strands. - What it is: Thousands of flowing lines swirl through space like smoke trails or bundles of silk. - Technique: Particle positions and their history are updated in float render textures by fragment shaders (GPGPU before compute shaders were common in Unity) and rendered as line strips following a noise field. - Try it: Recreate the flowing strands with VFX Graph particle strips anchored above a table in AR Foundation. Twist: make the strands bend away from the phone as if the viewer were blowing on them. #### ManyCubes — Keijiro Takahashi (2015) - Video: https://va.media.tumblr.com/tumblr_nwkrsraePx1qio469.mp4 - Source code: https://github.com/keijiro/ManyCubes - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, shader, hash function - Idea: Get rich motion from one shader and a hash function. - What it is: A dense cluster of cubes shuffles, rotates and reorganises itself in a looping animation. - Technique: A cube-cluster shader positions and rotates each cube from an xxHash-based random value and time, and the scene is finished with ambient occlusion, bokeh and color grading. - Try it: Fill a real shelf in AR with a cluster of small cubes that shuffle every few seconds. Twist: let each cube carry a photo taken by a classmate. #### PhotoMosaic — Keijiro Takahashi (2015) - Video: https://va.media.tumblr.com/tumblr_nwf56pAxZd1qio469.mp4 - Source code: https://github.com/keijiro/PhotoMosaic - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, image effect - Idea: Render a scene out of a photo album. - What it is: The live 3D scene is rebuilt from hundreds of small photographs, each chosen to match the color of its tile. - Technique: An image effect samples the average color of each screen tile and picks the closest photo from an album texture atlas of 256x256 images, drawing it in the tile. - Try it: Turn the AR camera view into a photo mosaic built from the class's own photos. Twist: only virtual objects are rendered as mosaic while the real world stays normal. #### Spektr Scatter (polygon scatter) — Keijiro Takahashi (2015) - Video: https://va.media.tumblr.com/tumblr_nvmqi6Tmuc1qio469.mp4 - Source code: https://github.com/keijiro/SpektrScatter - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, shader - Idea: Build a poly-dissolve that works on any model with a single slider. - What it is: A solid model breaks apart into scattered polygons that fly off and dissolve, then can come back together. - Technique: A shader works on a mesh whose triangles are split apart, moving each triangle along a direction and rotation derived from its centroid and a noise hash as the scatter amount increases. - Try it: Scatter the polygons of a virtual object in AR when the viewer walks through it, and reassemble it when they step away. Twist: let the fragments settle on the real floor plane. #### Subatomic (geometric mirroring) — Keijiro Takahashi (2015) - Video: https://va.media.tumblr.com/tumblr_nv2ru4onKo1qio469.mp4 - Source code: https://github.com/keijiro/SpektrSubatomic - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, mesh renderer - Idea: Make a kaleidoscope that exists in 3D space. - What it is: A mesh is mirrored many times around a point, creating a kaleidoscope that is a real 3D object rather than a flat screen effect. - Technique: A custom mesh renderer draws the source mesh several times with reflection matrices in world space, so the mirroring behaves like a mesh modifier with correct lighting (inspired by the subatomic sequence in Ant-Man). - Try it: Mirror a small virtual object eight times around an anchor in AR so viewers can walk around a 3D kaleidoscope. Twist: include the AR face mesh so the viewer's face becomes part of it. #### TextAnimation — Keijiro Takahashi (2015) - Video: https://va.media.tumblr.com/tumblr_nxhz6iOsZv1qio469.mp4 - Source code: https://github.com/keijiro/TextAnimation - Interaction: Information & UI, Perception & Effects - Platform & tech: Desktop, Unity, shader - Idea: Animate typography entirely in a shader. - What it is: Letters and glyphs assemble, scatter and flicker in a prototype of shader-driven motion typography. - Technique: A text animation shader offsets and reveals each glyph quad using its index and time, so many letters animate on the GPU without per-letter scripts. - Try it: Write a short sentence in AR space with TextMeshPro and animate each letter in a shader so it assembles when the viewer looks at it. Twist: letters scatter again when the viewer looks away. #### VJ04 (Channel #10) — Keijiro Takahashi (2015) - Video: https://www.youtube.com/watch?v=jLmbjudThlA - Source code: https://github.com/keijiro/VJ04 - Interaction: Voice & Sound, Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, Reaktion, MIDI, image-based lighting - Idea: Use physically based rendering to make club visuals look like product shots in motion. - What it is: Glossy 3D forms and figures pulse and morph to Nyolfen's live set at Channel #10, projected as club visuals. - Technique: Unity scenes lit with HDR image-based lighting (sIBL archive) are driven by Reaktion controllers mapped to audio input and MIDI CCs. - Try it: Light a virtual chrome object in AR with the real environment probe and make it pulse to the music the phone hears. Twist: swap its material on every drum hit. #### VJ05 (Channel #12) — Keijiro Takahashi (2015) - Video: https://x.com/shutamegai/status/619489135749824513 - Source code: https://github.com/keijiro/VJ05 - Interaction: Voice & Sound, Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, Reaktion, MIDI - Idea: Build a reusable live-visual rig where every animated parameter is exposed to sound and MIDI. - What it is: Live visuals for Katsuhiro Chiba at Channel #12, where HDR-lit 3D models and animations move with the music; the clip is an audience recording from the club. - Technique: All models and animations are driven by Reaktion controllers (Reaktor components) that take audio analysis, optionally via Data Racket, and MIDI CC input. - Try it: Expose five parameters of an AR scene (scale, color, rotation speed, spawn rate, camera shake) to the phone microphone and perform a one-minute set with a partner. Twist: the partner controls the parameters from a second phone over the network. #### CloudSkybox — Keijiro Takahashi (2016) - Video: https://www.youtube.com/watch?v=_QC6dXTMMwE - Source code: https://github.com/keijiro/CloudSkybox - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Desktop, Unity, raymarching, skybox shader - Idea: Put real volumetric clouds into the sky without any geometry. - What it is: A sky full of soft volumetric clouds drifts and changes, drawn entirely inside a skybox shader. - Technique: An extension of Unity's procedural skybox raymarches through a layer of animated 3D noise density and accumulates light scattering per step. - Try it: Cut a portal in a real ceiling in AR and show the volumetric cloud sky through it. Twist: link cloud density to the local weather forecast. #### Flipbook — Keijiro Takahashi (2016) - Video: https://va.media.tumblr.com/tumblr_pcorcg1yV01qio469.mp4 - Source code: https://github.com/keijiro/Flipbook - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, vertex shader - Idea: Turn time into pages of a physical book. - What it is: A live 3D scene is sliced into pages that flip over like a book, each page freezing a moment of the animation. - Technique: Frames are captured into a ring of render textures, and a vertex shader bends page meshes around a spine with speed parameters controlling the flip. - Try it: Record the AR camera view into pages and flip them on a virtual book lying on a real desk. Twist: let the viewer turn pages by swiping in the air with hand tracking. #### Holographic ribbons for VRDG+H — Keijiro Takahashi (2016) - Video: https://x.com/_kzr/status/709712057235406848 - Interaction: Projection, Performance, Perception & Effects - Platform & tech: Projection, Unity, holographic display - Idea: An optical virtual image makes a Unity statue and ribbons float in the air and dance to music. - What it is: A floating statue and ribbons appear as virtual images in mid-air using a holographic optical setup, animated to DUB-Russell's music at the VRDG+H show. - Technique: Unity renders are shown through a holographic / Pepper's-ghost-style optical element so the image is perceived as a virtual image floating in space (likely a half-mirror or holographic screen). - Try it: Make a Pepper's ghost with a tilted sheet of clear acrylic and a phone screen so a small 3D statue appears to float. Twist: ribbons spin around it when you clap. #### Phantom (Phantom Sketch Mod.) — Keijiro Takahashi (2016) - Video: https://x.com/tokyomax/status/787249881899347976 - Source code: https://github.com/keijiro/Phantom - Interaction: Voice & Sound, Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, multi-display, post-processing - Idea: Push a game engine's post-processing stack to cinematic quality for a live show. - What it is: Heavy, cinematic real-time visuals with bloom, depth of field and motion blur were projected for Merzbow and other acts at the Phantom Sketch Mod. event in October 2016. - Technique: Unity's multi-display feature shows a VJ control UI on the monitor and sends clean renders to the projector, while GPU-heavy image effects (SSAO, DOF, motion blur and Kino filters) run at 60 fps on a GTX 1070. - Try it: Build a two-device AR show: one phone is the control panel for bloom, blur and color, the other shows the AR scene to the audience. Twist: hand the control phone to someone in the audience. #### Skinner — Keijiro Takahashi (2016) - Video: https://vimeo.com/197396746 - Source code: https://github.com/keijiro/Skinner - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, GPU particles, replacement shader - Idea: Use every vertex of a moving body as an emitter, so the motion itself paints the effect. - What it is: A dancing character sheds thousands of particles, glowing trails and glitchy triangle fragments straight from its animated skin while it moves. - Technique: A replacement shader bakes the skinned mesh's vertex positions for the current and previous frame into render textures, and GPU particle, trail and glitch renderers read those textures to spawn and move their elements. - Try it: Import a Mixamo dancer into Unity, sample its skinned mesh in a VFX Graph particle system and place the dancer on a detected floor with AR Foundation on a phone. Twist: switch to ARKit body tracking so a classmate's own body sheds the particles. #### Spectrum (Liquidroom 2016) — Keijiro Takahashi (2016) - Video: https://va.media.tumblr.com/tumblr_oj1b8wYz3O1qio469_720.mp4 - Source code: https://github.com/keijiro/Spectrum - Interaction: Voice & Sound, Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, Kino post effects - Idea: Combine a whole library of screen effects into one playable party instrument. - What it is: A real-time VJ rig for the 2016 end-of-year party at Liquidroom layers 3D scenes with datamosh, feedback, contour and binary-style screen effects. - Technique: The Unity project stacks Keijiro's Kino image effects (Datamosh, Feedback, Contour, Binary and others) on top of animated scenes and switches them live. - Try it: Stack three screen effects (feedback, datamosh, contour) on an AR camera view and switch them with on-screen buttons during a song. Twist: let the song's bass level choose the effect automatically. #### Beta (live coding at Channel 16) — Keijiro Takahashi (2017) - Video: https://vimeo.com/213872212 - Source code: https://github.com/keijiro/Beta - Interaction: Voice & Sound, Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, KodeLife, GLSL, Spout - Idea: Live-code the texture of a 3D world while the music plays. - What it is: Abstract shader patterns are live-coded in KodeLife and projected onto objects inside a Unity 3D scene, mixing improvisation with prepared assets. - Technique: KodeLife renders audio-reactive GLSL fragment shaders, the frames are shared with Unity through Spout, and Unity projects them onto meshes and overlays while a nanoKONTROL2 controls extra effects. - Try it: Live-code a fragment shader in a browser editor and use it as the texture of a virtual sculpture in a WebXR scene. Twist: each student gets 60 seconds to edit the code before passing it on. #### Cloner — Keijiro Takahashi (2017) - Video: https://vimeo.com/218961301 - Source code: https://github.com/keijiro/Cloner - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, compute shader, GPU instancing - Idea: Build a complex organic shape out of many clones of one simple part. - What it is: Thousands of small copies of a mesh sit on the vertices of a base model and ripple, twist and pulse together as one living form. - Technique: A compute shader animates a transform for each vertex of the base model from noise, and GPU procedural instancing draws the template mesh at every vertex with very little CPU cost. - Try it: Place a sphere in AR and clone a tiny cube onto each of its vertices with VFX Graph or GPU instancing, animating the cubes' scale with noise. Twist: use the AR face mesh as the base so the clones cover a real face. #### GDisintegrator — Keijiro Takahashi (2017) - Video: https://vimeo.com/241520939 - Source code: https://github.com/keijiro/GDisintegrator - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, geometry shader - Idea: Treat disintegration as a controllable, reversible material state. - What it is: A figure's surface peels off into drifting triangles that swirl away like ash, and then the body is restored. - Technique: A geometry shader detaches each triangle when a moving threshold reaches it, then moves, spins and shrinks the triangle with noise over time; it is the companion effect to GVoxelizer. - Try it: Add a triangle-disintegration shader to a virtual statue in AR Foundation and drive the threshold with the distance between phone and statue. Twist: make the triangles fly toward the viewer's camera instead of away. #### GVoxelizer — Keijiro Takahashi (2017) - Video: https://vimeo.com/241191777 - Source code: https://github.com/keijiro/GVoxelizer - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, geometry shader - Idea: Turn any mesh into a voxel transition without preprocessing it. - What it is: An animated character dissolves into flying voxel cubes and then reassembles, as if it were being teleported block by block. - Technique: A geometry shader turns each triangle into a cube once a moving effector plane passes it, then pushes and scales the cubes with noise, so the effect runs in real time on any mesh. - Try it: Apply a voxel-dissolve shader to a scanned object placed with AR Foundation so it breaks into cubes as the phone gets closer. Twist: reverse it so the object only builds itself up when the viewer steps back. #### GlitchDancer — Keijiro Takahashi (2017) - Video: https://vimeo.com/198537336 - Source code: https://github.com/keijiro/GlitchDancer - Interaction: Hands & Body, Performance, Perception & Effects - Platform & tech: Phone, Unity, iOS, Metal, motion capture - Idea: Show that a phone can run club-grade real-time 3D visuals. - What it is: A motion-captured dancer keeps breaking into shifting glitch fragments, rendered live at 60 fps on an iPhone as a pocket VJ tool. - Technique: Mocap clips drive a skinned character whose triangles are displaced and recolored by Skinner-style glitch shaders, tuned to hold 60 fps on an iPhone 6S; part of it was used for the Teenage Engineering OP-Z demo at NAMM 2017. - Try it: Build a phone AR scene where a mocap dancer stands on a table and its triangles glitch apart on every beat the microphone detects. Twist: tap the screen to freeze the glitch and walk around the frozen fragments. #### LASP audio-reactive particles — Keijiro Takahashi (2017) - Video: https://va.media.tumblr.com/tumblr_otamd9rnf21qio469.mp4 - Source code: https://github.com/keijiro/Lasp - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, Unity, LASP, native audio plugin - Idea: Make visuals react to sound with no perceptible delay. - What it is: A particle system bursts and swells in tight sync with live audio input, demonstrating a low-latency audio plugin for Unity. - Technique: LASP is a native plugin that captures audio input with minimal buffering and runs a three-band filter bank, and the band levels drive particle emission and size in real time. - Try it: Use microphone levels split into low, mid and high bands to drive three VFX Graph emitters placed around a speaker in AR. Twist: measure the delay with a clap and try to make it shorter. #### Pcx point cloud renderer — Keijiro Takahashi (2017) - Video: https://vimeo.com/239850990 - Source code: https://github.com/keijiro/Pcx - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Unity, PLY, compute buffer - Idea: Make raw scan data a first-class, animatable material inside a game engine. - What it is: Scanned point clouds imported from .ply files are drawn in Unity as dense colored dots and animated with custom point shaders. - Technique: A custom asset importer reads binary PLY files into point-topology meshes or ComputeBuffers, and point and disk shaders render and animate millions of points; later versions also bake them into VFX Graph point caches (the video shows its predecessor PlypcImporter). - Try it: Scan a room corner with a LiDAR phone app, export it as .ply, import it with Pcx and re-anchor the cloud onto the real corner in AR Foundation. Twist: make the points scatter wherever a hand appears in front of the camera. #### Prisma (VRDG+H #4) — Keijiro Takahashi (2017) - Video: https://www.youtube.com/watch?v=oY6uCfNb-Ng - Source code: https://github.com/keijiro/Prisma - Interaction: Projection, Performance, Perception & Effects - Platform & tech: Projection, Unity, Pepper's ghost, Shadow Slicer - Idea: Place virtual objects between performers and audience by projecting onto a transparent screen. - What it is: Huge floating figures and shapes appear as Pepper's ghost holograms in front of Merzbow and XXX Residents, with their shadows falling on a screen behind the performers. - Technique: A large transparent screen at the front of DMM VR Theater creates the Pepper's ghost image, and a Shadow Slicer technique renders shadows onto the background screen without regenerating shadow maps, which kept the show at full frame rate. - Try it: Render a Unity scene to a phone laid under a tilted sheet of acrylic to make a desktop Pepper's ghost, then view the same scene with HoloKit to compare the two kinds of holograms. Twist: make the virtual figure cast a shadow onto a real card behind the acrylic. #### RDSystem (reaction-diffusion) — Keijiro Takahashi (2017) - Video: https://vimeo.com/217373413 - Source code: https://github.com/keijiro/RDSystem - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Custom Render Texture - Idea: Let a chemistry-style simulation paint living textures. - What it is: Organic spots and maze-like stripes grow and flow across surfaces in real time, driven by a reaction-diffusion simulation. - Technique: A Gray-Scott reaction-diffusion system runs inside a Unity Custom Render Texture that updates itself every frame, and the result drives the color and displacement of surface shaders. - Try it: Run a reaction-diffusion Custom Render Texture on a quad anchored to a real wall in AR and seed new growth where the user taps. Twist: seed it from the edges of the camera image so real objects start the pattern. #### Republic (FEMM live) — Keijiro Takahashi (2017) - Video: https://x.com/tokyomax/status/878736675228090368 - Source code: https://github.com/keijiro/Republic - Interaction: Voice & Sound, Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, multi-display - Idea: Match real-time 3D visuals to a pop act's choreography and music. - What it is: Real-time visuals for FEMM's live performance at REPUBLIC, WOMB Shibuya, surround the performers with 3D graphics synced to the music. - Technique: A Unity project drives multiple displays on Windows with DX11, switching prepared 3D scenes and effects in time with the set. - Try it: Choreograph a 30-second AR sequence to a song: place three virtual objects around a dancer and trigger their animations on cue from a phone. Twist: let the dancer trigger the cues with gestures seen by a second phone's camera. #### Seido (静動) — Keijiro Takahashi (2017) - Video: https://va.media.tumblr.com/tumblr_p19olvfZwP1qio469.mp4 - Source code: https://github.com/keijiro/Seido - Interaction: Voice & Sound, Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, Klak, multi-display - Idea: Turn a row of screens into one rhythmic, public-domain visual instrument. - What it is: A seven-display rig shows live visuals for Masaya Matsuura's concerts in Osaka and Tokyo, with calm and dynamic scenes alternating across the screens. - Technique: Unity's multi-display output feeds seven screens, and Klak components (audio input, cosine gradients, interpolators, game-view layouter) animate colors and motion from the music. - Try it: Line up several phones on a shelf, each running the same AR scene from a different virtual camera, and sync them to one song. Twist: let the audience rearrange the phones during the song. #### ShaderSketches — Keijiro Takahashi (2017) - Video: https://va.media.tumblr.com/tumblr_ooco6l0oEQ1qio469.mp4 - Source code: https://github.com/keijiro/ShaderSketches - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, KodeLife, GLSL - Idea: Keep a daily habit of tiny shader sketches. - What it is: A collection of small fragment-shader sketches written while commuting; in this one, geometric patterns pulse with an audio spectrum input. - Technique: Each sketch is a single GLSL fragment shader in KodeLife; this one reads the spectrum texture that KodeLife provides from audio input and maps bands to shape size and color. - Try it: Write a one-screen audio-reactive shader and show it on a virtual poster in a WebXR scene that listens to the microphone. Twist: everyone writes a new sketch on the way to class for a week. #### Swarm — Keijiro Takahashi (2017) - Video: https://vimeo.com/219277691 - Source code: https://github.com/keijiro/Swarm - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Unity, compute shader, GPU instancing - Idea: Constrain a noise-driven swarm so it hugs the surface of an object. - What it is: Thousands of glowing lines crawl over the surface of a shape, like insects swarming across a sculpture. - Technique: A compute shader moves particles through a divergence-free noise field while a distance-field volume pulls them toward the object surface, and the trails are drawn with procedural instancing (DrawMeshInstancedIndirect). - Try it: Bake a signed distance field of a real object, such as a box matching a table, and let a VFX Graph swarm crawl over it in AR Foundation. Twist: use the phone's LiDAR mesh so the swarm crawls over the actual room. #### Trinity (Channel 17) — Keijiro Takahashi (2017) - Video: https://va.media.tumblr.com/tumblr_ou5rt7l30F1qio469_720.mp4 - Source code: https://github.com/keijiro/Trinity - Interaction: Voice & Sound, Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, OSC, compute shader - Idea: Let each drum hit trigger a different GPU effect across three screens. - What it is: Triple-projector live visuals for DUB-Russell's set at Channel 17 react to kicks and snares with cloned, crawling and instanced geometry. - Technique: Audio from the default recording device and OSC triggers (kick and snare) drive Cloner, distance-field volume and instancing effects across three XGA projector outputs. - Try it: Map kick and snare detection from the phone microphone to two different VFX Graph bursts in an AR scene. Twist: put three phones side by side as a triptych and let each show a different reaction. #### VideoPlayerEffects (keying) — Keijiro Takahashi (2017) - Video: https://va.media.tumblr.com/tumblr_om9qfzPwgn1qio469_480.mp4 - Source code: https://github.com/keijiro/VideoPlayerEffects - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, VideoPlayer, shader - Idea: Place filmed people into a real-time 3D world with a shader. - What it is: Green-screen footage played by Unity's VideoPlayer is keyed live, with improved spill suppression so the subject blends into a 3D scene. - Technique: A chroma-key shader computes a matte from the green channel difference and removes green spill from the edges; the keyer later moved to the ProcAmp repository. - Try it: Film a classmate against a green wall, key the clip with a shader and place it as a life-size video billboard in AR. Twist: make the billboard always face the viewer so the person follows you around. #### Flipper (ADIRECTOR Channel) — Keijiro Takahashi (2018) - Video: https://va.media.tumblr.com/tumblr_pd1oevl48u1qio469.mp4 - Source code: https://github.com/keijiro/Flipper - Interaction: Voice & Sound, Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, geometry shader - Idea: Make a digital show feel like flipping through a printed book. - What it is: Live visuals for ADIRECTOR Channel in Omotesando combine flip-book page turning with ribbons and shapes grown from animated skeletons. - Technique: The flipbook renderer captures frames into pages that turn with a bending mesh, combined with skeletal geometric effects built with geometry-shader instancing. - Try it: Capture the AR camera view every half second and let the pages flip past on a virtual book lying on a real table. Twist: flip forward on a beat and backward on silence. #### Museum (Channel 18) — Keijiro Takahashi (2018) - Video: https://www.youtube.com/watch?v=r-nbIpdn-Lk - Source code: https://github.com/keijiro/Museum - Interaction: Voice & Sound, Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, KodeLife, GLSL, Spout - Idea: Mix improvised code with prepared 3D props, like a museum curated live. - What it is: Classical statues float in a Unity scene whose backgrounds and surfaces are live-coded in KodeLife during DUB-Russell's set at Channel 18. - Technique: KodeLife's live-coded GLSL output is shared with Unity through Spout in real time, where it becomes the backdrop and texture for scanned statues with glitch post effects. - Try it: Place a scanned statue in AR and texture it with a shader that students live-edit in a browser. Twist: switch statues on every chorus of the song. #### Ngx — Keijiro Takahashi (2018) - Video: https://vimeo.com/294399440 - Source code: https://github.com/keijiro/Ngx - Interaction: Performance, Perception & Effects - Platform & tech: Desktop, Projection, Unity, pix2pix, compute shader - Idea: A neural network that hallucinates an infinite music video from a short clip. - What it is: Endless abstract VJ footage is generated frame by frame by a neural network that predicts the next frame, and two models can be crossfaded live. - Technique: pix2pix is trained as a next-frame predictor on pairs of consecutive frames from a clip (for example Beeple's VJ loops), and each output frame is fed back as the next input; two models are mixed by interpolating their outputs. - Try it: Show a feedback-loop generative model on a floating AR screen and let viewers crossfade two models by moving the phone left or right. Twist: seed the loop with the live camera image of the room. #### OP-Z videolab visuals — Keijiro Takahashi (2018) - Video: https://vimeo.com/307285370 - Source code: https://github.com/keijiro/VideolabTest - Interaction: Voice & Sound, Performance - Platform & tech: Phone, Unity, OP-Z videolab, iOS - Idea: Let a synthesizer's sequencer play 3D visuals directly. - What it is: Short 3D visual loops made in Unity are played by the steps and tracks of the Teenage Engineering OP-Z synthesizer and rendered live on an iPhone. - Technique: Scenes built with the OP-Z videolab package are exported as videopaks; the OP-Z iOS app maps sequencer tracks and notes to animation triggers and vertex-shader parameters. - Try it: Map MIDI notes (or beats detected by the microphone) to VFX Graph events in a phone AR scene, one effect per drum. Twist: anchor each drum's effect to a different real object in the room. #### Pix2Pix for Unity — Keijiro Takahashi (2018) - Video: https://vimeo.com/287778343 - Source code: https://github.com/keijiro/Pix2Pix - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, Unity, compute shader, pix2pix - Idea: Use real-time image-to-image translation as a drawing tool. - What it is: A sketch drawn on screen is turned instantly into a photo-like image, such as edges into a cat, by a neural network running inside Unity. - Technique: A hand-written compute-shader inference engine runs a pre-trained pix2pix generator (a U-Net) in Unity without any external machine-learning framework. - Try it: Run a small image-to-image model with Unity Sentis on the phone camera feed and show the result on a floating AR panel. Twist: let students draw strokes in AR space and translate the drawing live. #### Procedural dance (PuppetTest) — Keijiro Takahashi (2018) - Video: https://vimeo.com/255257338 - Source code: https://github.com/keijiro/PuppetTest - Interaction: Performance, Perception & Effects - Platform & tech: Desktop, Unity, humanoid IK - Idea: Generate dance from a few sine waves and noise instead of motion capture. - What it is: A crowd of humanoid figures dances without any recorded animation, each one swaying with its own rhythm. - Technique: Sine curves and Perlin noise set target positions for the hands, feet and hips, and Unity's humanoid IK rig solves the full pose every frame. - Try it: Place a few procedurally dancing figures on a table in AR, driving Animation Rigging IK targets with sine waves and noise. Twist: set their tempo from the music the phone microphone hears. #### Shader Graph moving-lines globe — Keijiro Takahashi (2018) - Video: https://va.media.tumblr.com/tumblr_pfv4dgueZM1qio469.mp4 - Source code: https://github.com/keijiro/ShaderGraphExamples - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Shader Graph - Idea: Show that a node graph can make a striking animated material. - What it is: A globe is covered with flowing lines that scroll across its surface, built entirely with a node-based shader. - Technique: A Shader Graph (CC0 example) uses spherical coordinates, a scrolling time offset and step functions to draw moving line patterns with emission. - Try it: Recreate the moving-lines material in Shader Graph and apply it to a virtual globe anchored on a real table in AR. Twist: make the lines flow toward the city where each student was born. #### Skeletal Geometric Effects — Keijiro Takahashi (2018) - Video: https://va.media.tumblr.com/tumblr_pcsc78PRjA1qio469.mp4 - Source code: https://github.com/keijiro/SkeletalGeometricEffects - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, geometry shader, instancing - Idea: Build rich geometry from nothing more than a skeleton's bones. - What it is: Simple line segments along a dancing skeleton grow into complex ribbons and geometric shapes that follow the movement. - Technique: Geometry-shader instancing expands each bone's line segment into many triangles on the GPU, for example 16 line segments into about 16,000 animated ribbon triangles. - Try it: Take the joint positions from ARKit body tracking and draw ribbons between them with a VFX Graph or line shader on a phone. Twist: make the ribbons leave a trail that stays in the room after the dancer leaves. #### StableFluids — Keijiro Takahashi (2018) - Video: https://vimeo.com/277872734 - Source code: https://github.com/keijiro/StableFluids - Interaction: Hands & Body, Drawing & Making, Perception & Effects - Platform & tech: Desktop, Unity, compute shader, Sensel Morph - Idea: Turn a touch surface into a hands-on liquid painting instrument. - What it is: Colored ink swirls through a real-time fluid simulation that reacts to fingers pressing on a Sensel Morph touch pad. - Technique: Jos Stam's Stable Fluids (advection, diffusion and pressure projection) runs in compute shaders, and multi-touch pressure from the Sensel Morph injects force and dye (demo from SenselExamples). - Try it: Run the StableFluids solver on a plane in AR Foundation and inject force where finger touches raycast onto the plane. Twist: use hand tracking so waving a hand above the table stirs the fluid. #### Akvfx: Azure Kinect with VFX Graph — Keijiro Takahashi (2019) - Video: https://x.com/_kzr/status/1163456709894922240 - Source code: https://github.com/keijiro/Akvfx - Interaction: Hands & Body, Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Azure Kinect - Idea: Treat a depth sensor as a live particle source. - What it is: A person in front of an Azure Kinect appears as a live 3D point cloud that breaks into sparks and lines as they move. - Technique: The plugin converts Azure Kinect color and depth frames into position and color attribute maps on the GPU; VFX Graph spawns particles from these maps every frame. - Try it: Recreate the effect on a phone: use AR Foundation's human depth or LiDAR depth to spawn points on people, and view it through HoloKit. Twist: let the points lag one second behind, so each person leaves a ghost. #### Dkvfx: Depthkit volumetric video with VFX Graph — Keijiro Takahashi (2019) - Video: https://x.com/_kzr/status/1105456612162994177 - Source code: https://github.com/keijiro/Dkvfx - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Depthkit - Idea: Recorded volumetric video becomes raw material for particle effects. - What it is: A dancer recorded with Depthkit plays back in 3D and is re-rendered as particles, lines and Kinect-style effects that can be viewed from any angle. - Technique: The Depthkit combined-per-pixel video (color plus encoded depth) is decoded in a shader into position and color maps, which VFX Graph uses as a point source. - Try it: Record a 10-second clip of a friend with a depth-capable phone app (Record3D or a Depthkit export), play it in Unity as a point source, then place the result on a table with AR Foundation. Twist: let the figure dissolve into particles as you walk closer. #### GeoVfx: world population as particles — Keijiro Takahashi (2019) - Video: https://x.com/_kzr/status/1429739880259612685 - Source code: https://github.com/keijiro/GeoVfx - Interaction: Information & UI, Location & City - Platform & tech: Desktop, Unity, VFX Graph, GraphicsBuffer - Idea: Data becomes landscape when every value is a particle. - What it is: A globe is built from particles whose heights show where people live, based on NASA SEDAC population data. - Technique: Gridded population data is loaded into a GraphicsBuffer and VFX Graph maps each cell's latitude and longitude onto a sphere, scaling the particle height by density. - Try it: Load a small open dataset (e.g., city air quality) and render it as a particle globe that sits on your desk in AR Foundation. Twist: let people pinch to fly into one country. #### Grubo: MC-101 live performance at Channel #21 — Keijiro Takahashi (2019) - Video: https://vimeo.com/379562830 - Source code: https://github.com/keijiro/Grubo - Interaction: Voice & Sound, Performance - Platform & tech: Projection, Desktop, Unity, LASP, Minis, Roland MC-101 - Idea: One small groovebox drives both the music and the visuals. - What it is: Keijiro plays a Roland MC-101 groovebox on stage while a Unity visualizer reacts to each track with shapes and particles projected behind him. - Technique: The MC-101's multichannel USB audio is analyzed per track with LASP and its MIDI clock and notes arrive through Minis, so each part triggers its own effect. - Try it: Split a song into drums and melody, map each stem to its own AR Foundation VFX object in the room, and perform it live. Twist: let the audience move the objects by walking around with the phone. #### KinoEight: 8-bit style post effect — Keijiro Takahashi (2019) - Video: https://x.com/_kzr/status/1208045085452955653 - Source code: https://github.com/keijiro/KinoEight - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HDRP, Post-processing - Idea: Old hardware limits become a visual style. - What it is: A modern 3D scene is redrawn as a low-resolution image with an 8-color palette, dithering and tile-based color clash like old home computers. - Technique: A full-screen pass downsamples the image, picks the two best palette colors per 8x8 tile and dithers between them with a Bayer matrix. - Try it: Apply an 8-bit palette shader to the AR Foundation camera feed so the real room looks like a 1980s game screen. Twist: keep one real object in full color. #### LaspVfx: audio-reactive VFX with LASP — Keijiro Takahashi (2019) - Video: https://x.com/_kzr/status/1116337708782067713 - Source code: https://github.com/keijiro/LaspVfx - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, LASP - Idea: Let the sound itself draw the shape. - What it is: Live audio is split into low, mid and high bands and drawn as three glowing waveforms and particle bursts that move with the music. - Technique: LASP captures low-latency audio input and filters it into bands; property binders push the waveform buffer and band levels into VFX Graph as textures and floats. - Try it: Place a VFX Graph in AR Foundation that reads the phone microphone level and makes particles burst around a real speaker. Twist: map bass to particle size and treble to color, then play two songs and compare. #### NoiseBall5: mesh deformed by the Job System — Keijiro Takahashi (2019) - Video: https://x.com/_kzr/status/1211628843343081472 - Source code: https://github.com/keijiro/NoiseBall5 - Interaction: Perception & Effects - Platform & tech: Desktop, Phone, Unity, C# Job System, Burst - Idea: A living blob is just noise applied to vertices. - What it is: A sphere wobbles and ripples with noise every frame, its vertices moved by multithreaded C# jobs; it also runs on iPhone. - Technique: Unity's new Mesh API writes vertex data from Burst-compiled C# jobs directly into the mesh buffer, avoiding extra copies. - Try it: Anchor a noise-deformed sphere to a real object with AR Foundation image tracking and let it pulse. Twist: drive the noise strength with the microphone. #### ProcCharVfx: procedural letters and Matrix rain — Keijiro Takahashi (2019) - Video: https://x.com/_kzr/status/1209489319544549376 - Source code: https://github.com/keijiro/ProcCharVfx - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Shader Graph - Idea: Generate an alphabet instead of loading one. - What it is: Columns of invented glyphs rain down the screen like the Matrix; every letter is generated by a shader rather than drawn from a font. - Technique: A Shader Graph builds each glyph from a random seed on a small grid of strokes; VFX Graph spawns thousands of quads and passes each a seed so every character is unique. - Try it: Make a procedural glyph shader and pour glyph rain down a real wall in AR Foundation, using plane detection to find the wall. Twist: let the rain part around people's silhouettes. #### Rcam at Channel #20 (live with umio) — Keijiro Takahashi (2019) - Video: https://vimeo.com/346711967 - Source code: https://github.com/keijiro/Rcam - Interaction: Performance, Hands & Body, Perception & Effects - Platform & tech: Projection, Desktop, Unity, VFX Graph, Intel RealSense - Idea: Scan a performer into a live point cloud, then let particles tear the body apart and rebuild it as concert visuals. - What it is: A depth camera on stage streams a volumetric point cloud of the performer into Unity, where VFX Graph particles dissolve and rebuild her body in real time for a live concert. - Technique: A RealSense depth camera sends color+depth frames to Unity, which converts them into a point cloud that drives VFX Graph particles projected behind the performer. - Try it: Use Kinect, phone depth or MediaPipe body segmentation to turn a classmate into a particle silhouette projected on the wall. Twist: the louder the music, the more the particles scatter. #### Rsvfx: RealSense depth to VFX Graph — Keijiro Takahashi (2019) - Video: https://x.com/_kzr/status/1099299041463066624 - Source code: https://github.com/keijiro/Rsvfx - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Intel RealSense - Idea: A cheap depth camera turns a live person into particle material. - What it is: A person filmed by an Intel RealSense D415 turns into a live cloud of glowing particles that drift away from their body in real time. - Technique: A native plugin converts each RealSense depth frame into position and color textures (attribute maps), which a VFX Graph samples to spawn particles on the captured surface. - Try it: Build an AR Foundation scene on a LiDAR iPhone that reads the environment depth texture into a VFX Graph and spawns particles on your classmate's body. Twist: make the particles fall like sand only when the person stands still. #### SdfVfxSamples: particles shaped by distance fields — Keijiro Takahashi (2019) - Video: https://x.com/_kzr/status/1821182526184100126 - Source code: https://github.com/keijiro/SdfVfxSamples - Interaction: Perception & Effects - Platform & tech: Desktop, Web, Unity, VFX Graph, SDF, WebGPU - Idea: An invisible shape can be revealed by what flows around it. - What it is: Particle trails are pulled onto the surface of a hidden shape, tracing its outline as they flow; the samples run in a web browser via WebGPU. - Technique: A signed distance field baked from a mesh gives each particle the distance and direction to the surface; VFX Graph uses it for attraction forces and collision. - Try it: Bake an SDF from a scanned object, then use it in AR Foundation so particles wrap around the real object. Twist: swap the SDF every 10 seconds so the swarm morphs between objects. #### Smrvfx: particles from a skinned mesh — Keijiro Takahashi (2019) - Video: https://x.com/_kzr/status/1114513038302830592 - Source code: https://github.com/keijiro/Smrvfx - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph - Idea: Any animated body can become an emitter. - What it is: An animated character sheds streams of particles and ribbons from its moving body, so the dance leaves glowing trails in the air. - Technique: A script bakes the skinned mesh's vertex positions and velocities into textures every frame (later replaced by VFX Graph's Sample Skinned Mesh operator), and particles spawn from those points. - Try it: Rig a Mixamo character, drive it with AR Foundation body tracking on an iPhone, and emit VFX Graph particles from its skinned mesh so it overlays a real dancer. Twist: emit particles only from the fastest-moving limb. #### VFX Graph Sketch1012 — Keijiro Takahashi (2019) - Video: https://x.com/_kzr/status/1187004957821505536 - Source code: https://github.com/keijiro/VfxGraphTestbed - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph - Idea: Daily small sketches are a way to learn a tool. - What it is: A short abstract sketch where thousands of particles and strips swirl, fold and change color in a dark space. - Technique: VFX Graph combines noise-driven forces, particle strips and color-over-life gradients; the graph is kept small so each sketch takes about an hour. - Try it: Make one VFX Graph sketch in under an hour, then place it in your room with AR Foundation so the swirl floats above a real object. Twist: give yourself one constraint, such as one color and one force. #### VFX controlled by MIDI notes (Minis) — Keijiro Takahashi (2019) - Video: https://x.com/_kzr/status/1194287720203374592 - Source code: https://github.com/keijiro/VfxMinisExamples - Interaction: Voice & Sound, Tangible Objects, Performance - Platform & tech: Desktop, Web, Unity, VFX Graph, Minis, MIDI - Idea: Play visuals like an instrument. - What it is: Pressing keys on a velocity-sensitive MIDI controller fires bursts of particles; harder hits make bigger bursts and several keys can play at once. - Technique: Minis exposes MIDI devices to Unity's Input System; input binders map note-on events and velocity to VFX Graph events and parameters without scripting. - Try it: Connect a MIDI keyboard (or a Web MIDI page) and make each key spawn a burst at a fixed spot in an AR Foundation scene, so the room becomes a keyboard. Twist: assign each octave to a different wall. #### VFX sketches with Depthkit and Unity — Keijiro Takahashi (2019) - Video: https://vimeo.com/383216987 - Source code: https://github.com/keijiro/DkvfxSketches - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Depthkit, Azure Kinect - Idea: One recording, many visual personalities. - What it is: A compilation of volumetric recordings of performers turned into particle strips, electric arcs and flowing point clouds. - Technique: Depthkit and Azure Kinect footage is decoded into point maps; separate VFX Graphs draw strips, lightning-like lines and noise-advected points from the same data. - Try it: Take one volumetric clip and build three different VFX Graph looks for it, then switch between them in AR Foundation by tapping. Twist: tie each look to a different room in your building. #### VfxPyro: interactive fireworks — Keijiro Takahashi (2019) - Video: https://x.com/_kzr/status/1179427868587130880 - Source code: https://github.com/keijiro/VfxPyro - Interaction: Play, Perception & Effects - Platform & tech: Desktop, Web, Unity, VFX Graph, URP - Idea: A classic firework is the simplest lesson in spawn, burst and trail. - What it is: Clicking launches fireworks that rise, burst into sparkling shells and fade with trails, all simulated as GPU particles. - Technique: VFX Graph uses GPU events: a rocket particle triggers a burst system on death, and each spark spawns trail particles, giving a three-stage chain. - Try it: Port the firework graph to AR Foundation so tapping the phone screen launches a firework from the real floor where you tapped. Twist: launch from the spot a friend points at instead. #### 4DViews volumetric video with VFX Graph — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1270712472853340167 - Source code: https://github.com/keijiro/4DViewsTest2 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, 4DViews, Alembic - Idea: Studio volumetric capture can be remixed like any other footage. - What it is: A volumetric recording of a performer is replayed in Unity and melts into drifting particles and outlines. - Technique: The 4DViews clip is converted to Alembic meshes plus a HAP texture; a converter samples the meshes into point clouds that VFX Graph uses as a source. - Try it: Place a free volumetric clip on the floor with AR Foundation and let it dissolve into particles when viewers circle around it. Twist: freeze the performer when someone stands in front of them. #### Abcvfx: Alembic animation to VFX Graph — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1269876564486807552 - Source code: https://github.com/keijiro/Abcvfx - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Alembic - Idea: Any baked animation can feed a particle system. - What it is: An Alembic-animated shape is turned into a stream of particles that follow and peel off its moving surface. - Technique: The Alembic mesh is sampled each frame (now via VFX Graph's dynamic mesh input) and particles spawn on its vertices with inherited velocity. - Try it: Export a short Blender animation as Alembic, emit particles from it and place it on a real shelf with AR Foundation. Twist: particles stick to the real surfaces they fall on. #### Akvj: Azure Kinect VJ set — Keijiro Takahashi (2020) - Video: https://vimeo.com/424260614 - Source code: https://github.com/keijiro/Akvj - Interaction: Hands & Body, Performance - Platform & tech: Projection, Desktop, Unity, HDRP, VFX Graph, Azure Kinect - Idea: A depth camera becomes a live visual instrument. - What it is: Performers captured by an Azure Kinect are turned live into point clouds, meshes and particle storms for a VJ show. - Technique: Akvfx turns Kinect depth into attribute maps and a surface mesh; HDRP renders several VFX Graph and shader effects that the VJ switches live. - Try it: Build three body effects on a LiDAR iPhone with AR Foundation and switch them with a MIDI pad or on-screen buttons during a short performance. Twist: let the dancer switch effects with a pose. #### BurstWig: flowing hair-like strands — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1258039132372135941 - Source code: https://github.com/keijiro/BurstWig - Interaction: Face, Perception & Effects - Platform & tech: Desktop, Web, Unity, Burst, C# Job System, VFX Graph, HDRP - Idea: Strands with simple spring physics feel organic. - What it is: Thousands of long strands grow out of a moving object and swing and flow like hair in the wind. - Technique: Each strand is a chain of points solved with Verlet-style spring constraints in Burst-compiled jobs, then drawn with VFX Graph particle strips. - Try it: Attach a wig of strands to a tracked face with AR Foundation face tracking so it swings as you turn your head. Twist: let the wind come from the microphone. #### Eyeball: procedural iris that follows you — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1321000166585856000 - Source code: https://github.com/keijiro/Eyeball - Interaction: Gaze & Attention, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Shader Graph - Idea: An object that looks back at you feels alive. - What it is: A realistic eyeball with a procedurally generated iris rolls to follow the mouse, its pupil tightening and widening. - Technique: A VFX Shader Graph builds the iris from polar-coordinate noise and layered rings, and a script rotates the eye toward the pointer. - Try it: Place a procedural eyeball in AR Foundation that always turns to face the phone camera. Twist: fill a real wall with dozens of eyes that blink when you come close. #### HdrpVatExample: baked fluid and cloth with VAT — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1220338439117127680 - Source code: https://github.com/keijiro/HdrpVatExample - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HDRP, Shader Graph, VFX Graph, Houdini - Idea: Simulate once offline, play it back anywhere. - What it is: A splashing liquid and soft cloth, simulated offline in Houdini, play back in real time as smooth animated meshes inside Unity. - Technique: Houdini exports vertex animation textures (VAT) that store per-frame positions and normals; a Shader Graph reads them in the vertex stage and interpolates between frames. - Try it: Bake a simple splash or cloth VAT (Houdini Apprentice or Blender), and play it on a real table surface in AR Foundation. Twist: trigger the splash when a real cup is placed on an image marker. #### Khoreo: procedural dance with the MC-101 — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1291723680490254336 - Source code: https://github.com/keijiro/Khoreo - Interaction: Voice & Sound, Hands & Body, Performance - Platform & tech: Projection, Desktop, Unity, VFX Graph, Roland MC-101, MIDI - Idea: The music plays the dancer. - What it is: A procedurally animated dancer made of light moves in sync with music played live on a Roland MC-101, surrounded by particles shed from its body. - Technique: Procedural motion scripts animate the character while MIDI clock from the MC-101 syncs the timing; Smrvfx emits particles from the skinned mesh. - Try it: Place a rigged character on the floor with AR Foundation and make its moves change with the beat of a song. Twist: let a second phone's microphone control a second dancer. #### KinoFeedback2: frame feedback with emoji particles — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1306955390513364992 - Source code: https://github.com/keijiro/KinoFeedback2 - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HDRP, Custom Pass - Idea: Feed the last frame back into the next one. - What it is: Emoji particles leave looping, zooming echoes of themselves, like pointing a video camera at its own monitor. - Technique: An HDRP custom pass copies the previous frame, scales, rotates and hue-shifts it slightly, and blends it under the new frame. - Try it: Add a feedback pass to the AR Foundation camera image so moving people leave spiraling trails. Twist: the trail only appears while the phone is still. #### Krbv: colorful particle strip tunnel — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1219572530236641285 - Source code: https://github.com/keijiro/Krbv - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph - Idea: Ribbons moving past the camera create a sense of speed. - What it is: The camera flies through an endless tunnel made of bright colored ribbons that twist and flow past. - Technique: VFX Graph spawns particle strips on a cylinder, moves them toward the camera and colors them with a gradient over time. - Try it: Build a ribbon tunnel and anchor its mouth to a real doorway in AR Foundation, so walking through the door feels like entering it. Twist: the tunnel speeds up with your walking speed. #### MonoFxSketches: monochrome screen effects — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1333749477732098048 - Source code: https://github.com/keijiro/MonoFxSketches - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Shader - Idea: Strong graphic effects work best with only two colors. - What it is: A set of black-and-white full-screen effects that split, slide and posterize the image in rhythmic patterns. - Technique: Full-screen shaders threshold the image and apply random slice offsets and mirroring driven by time and random seeds. - Try it: Apply a random-split monochrome shader to the AR Foundation camera feed and sync the splits to a metronome. Twist: people stay unsplit using human segmentation. #### Particle depth of field with VFX Graph — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1290633852990287872 - Source code: https://github.com/keijiro/DofVfxSamples - Interaction: Perception & Effects - Platform & tech: Desktop, Web, Unity, VFX Graph, WebGPU - Idea: Fake camera blur per particle to add depth cheaply. - What it is: Glowing particles in the foreground and background blur into soft bokeh discs while those at the focal distance stay sharp. - Technique: A custom VFX Graph block computes each particle's circle of confusion from its distance to the focal plane and scales and fades a soft disc sprite accordingly. - Try it: Use DoF particles in an AR Foundation scene so dust motes near the phone look blurred and those around a real object look sharp. Twist: move the focus to whatever the center of the screen touches. #### PcxEffects3: point cloud effects — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1293928930647261184 - Source code: https://github.com/keijiro/PcxEffects3 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Pcx - Idea: A static scan becomes a living, breathing place. - What it is: A photogrammetry point cloud of a scanned scene flickers, scatters and reforms as animated particles. - Technique: Pcx imports the .ply point cloud as a point cache, and VFX Graph reads positions and colors from it to spawn and animate particles. - Try it: Scan a corner of your room with Polycam, import the points, and align the particle version to the real corner in AR Foundation. Twist: let the particles drift away wherever the phone points. #### Rcam2: Volumetric AR camera rig — Keijiro Takahashi (2020) - Video: https://vimeo.com/461782810 - Source code: https://github.com/keijiro/Rcam2 - Interaction: Spatial Mapping, Perception & Effects, Performance - Platform & tech: Phone, Desktop, Unity, ARKit, LiDAR, VFX Graph, NDI - Idea: Use an iPad's LiDAR as a mobile volumetric camera so effects grow right on the real room. - What it is: An iPad Pro LiDAR rig streams color, depth and camera pose to a PC, where Unity wraps the real scene in glowing particle and line effects that stay locked to the room. - Technique: ARKit on an iPad Pro captures LiDAR depth, color and camera pose, streams them over NDI to a PC, and Unity reprojects the depth into world space to spawn VFX Graph effects. - Try it: Use mesh scanning in Reality Composer or Unity AR Foundation to make particles flow along real walls and furniture. Twist: only the surfaces of objects you have pointed at light up. #### SlitScanCam: realtime slit-scan camera — Keijiro Takahashi (2020) - Video: https://vimeo.com/494895371 - Source code: https://github.com/keijiro/SlitScanCam - Interaction: Perception & Effects, Hands & Body - Platform & tech: Desktop, Unity, Shader, Webcam - Idea: Show time along one axis of the image. - What it is: A live camera image is rebuilt from slices taken at different moments, so moving people stretch and bend into elastic shapes. - Technique: Recent camera frames are stored in a texture array ring buffer, and each row of the output image samples a different frame by its vertical position. - Try it: Build a slit-scan filter on the AR Foundation camera feed and walk slowly past a friend. Twist: tie the time delay to depth, so nearer things lag more. #### Sword fighting effects with VFX Graph — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1276106752728031232 - Source code: https://github.com/keijiro/VfxGraphTestbed - Interaction: Hands & Body, Play - Platform & tech: Desktop, Unity, VFX Graph - Idea: A trail turns a fast motion into a readable shape. - What it is: Swinging blades leave bright arcs of light and sparks, like a fighting game. - Technique: VFX Graph particle strips follow a transform on the blade tip and hilt, fading out along their length, with spark bursts triggered on direction changes. - Try it: Attach a light trail to your phone in AR Foundation so swinging it draws arcs in the room. Twist: two players duel and the trails collide. #### VertexAnimationJob: multithreaded vertex animation — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1217326699714494464 - Source code: https://github.com/keijiro/VertexAnimationJob - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, C# Job System, Burst - Idea: Push every vertex every frame, and a static model comes alive. - What it is: Dense meshes twist, wave and break apart in real time while all CPU cores work on the vertex animation. - Technique: IJobParallelFor jobs compiled with Burst compute vertex positions in parallel and write them straight into mesh buffers using the Mesh API from Unity 2019.3. - Try it: Write a job that makes a scanned object ripple and place it in AR Foundation next to the real object. Twist: the ripple spreads from the point you tap. #### VfxCrystal: growing crystals — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1333403226813595649 - Source code: https://github.com/keijiro/VfxCrystal - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph - Idea: Mesh particles can build solid-looking forms. - What it is: Sharp crystal shards grow, spin and scatter in a small looping animation made entirely with particles. - Technique: VFX Graph outputs lit mesh particles with random orientation and scale over life, arranged around a center to read as a crystal cluster. - Try it: Grow a crystal cluster on a real surface detected by AR Foundation, spawning shards where you tap. Twist: crystals grow faster where the room is louder. #### WfcMaze: Wave Function Collapse maze — Keijiro Takahashi (2020) - Video: https://x.com/_kzr/status/1249358391975137282 - Source code: https://github.com/keijiro/WfcMaze - Interaction: Play, Spatial Mapping - Platform & tech: Desktop, Unity, WFC, URP - Idea: Local rules can build a coherent structure. - What it is: A 3D maze of connected blocks assembles itself piece by piece, each tile chosen to fit its neighbors. - Technique: Wave Function Collapse keeps a list of allowed tiles per cell, collapses the cell with the fewest options and propagates the constraints to neighbors. - Try it: Generate a small WFC maze on a detected table plane in AR Foundation and let a tiny character walk through it. Twist: regenerate the maze when you tap the table. #### Adding VFX to a Bibcam clip — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1460618910990929926 - Source code: https://github.com/keijiro/BibcamVfx - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, Desktop, Unity, VFX Graph, LiDAR - Idea: Add space-aware post effects to a depth video that has already been recorded. - What it is: A recorded Bibcam walk is post-processed so that particles and light trails wrap around the real buildings and people, with the effect following the real depth of the street. - Technique: Depth and pose embedded in the video let VFX Graph sample the real geometry per frame, so particles can spawn on and collide with surfaces in the footage. - Try it: Import a depth video into TouchDesigner and use the depth channel to control particle density so nearby people turn into points of light. Twist: shift the particle colors over time from day to night. #### Bibcam test in Shibuya — Keijiro Takahashi (2021) - Video: https://vimeo.com/651111230 - Source code: https://github.com/keijiro/Bibcam - Interaction: Spatial Mapping, Perception & Effects, Location & City - Platform & tech: Phone, Unity, ARKit, LiDAR, VFX Graph - Idea: Burn depth and camera pose into the video as you record, so you can later add effects that fit the street space. - What it is: Street footage shot on an iPhone 13 Pro Max records depth and camera motion inside the video itself, so later the crossing in Shibuya can be re-lit and filled with 3D effects. - Technique: Bibcam encodes the LiDAR depth map and camera pose as metadata burned into each recorded video frame, so Unity can later reconstruct the 3D scene for post-processed AR VFX. - Try it: Record a short campus clip with a LiDAR phone (or export one with Record3D), then add snow or glowing lines to the real space in Unity or Blender. Twist: make the effects appear only beyond 2 meters from the camera. #### BlazeFace face filters on Barracuda — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1378352493134929926 - Source code: https://github.com/keijiro/BlazeFaceBarracuda - Interaction: Face - Platform & tech: Desktop, Unity, Barracuda, MediaPipe BlazeFace - Idea: Six keypoints are enough to decorate a face. - What it is: A webcam face gets simple AR decorations placed on the eyes, nose and mouth, following head motion in real time. - Technique: MediaPipe BlazeFace runs on Barracuda and outputs face boxes with six keypoints; weighted non-maximum suppression stabilizes them before the decorations are drawn. - Try it: Design a face filter from keypoints and run it with AR Foundation face tracking on a phone. Twist: the decoration reacts when two faces are close together. #### BodyPix visual effects (NNCam) — Keijiro Takahashi (2021) - Video: https://vimeo.com/580670067 - Source code: https://github.com/keijiro/NNCam - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, Barracuda, BodyPix - Idea: Cut the person out of the background in real time so effects only touch the body. - What it is: A webcam segments the person from the background and body parts in real time, so particles, outlines and glitches can wrap only the human body in the live image. - Technique: The BodyPix person-and-part segmentation network runs through Unity Barracuda, and its mask texture is used to emit and confine VFX Graph particles to the body. - Try it: Use MediaPipe Selfie Segmentation or TouchDesigner to turn a classmate's body into flowing ink while the background stays still. Twist: only a raised hand catches fire. #### Compute shader data into VFX Graph — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1418439725631754244 - Source code: https://github.com/keijiro/VfxGraphGraphicsBufferTest - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Compute Shader, GraphicsBuffer - Idea: Simulate anywhere, render with VFX Graph. - What it is: Particles move in patterns computed by a separate compute shader, showing GPU simulation results fed straight into VFX Graph. - Technique: A compute shader writes positions into a GraphicsBuffer, and VFX Graph's buffer operators read them each frame without baking to textures. - Try it: Write a small compute simulation (e.g., boids), render it with VFX Graph and place it in AR Foundation above a table. Twist: the flock avoids the phone. #### ComputeMarchingCubes: GPU isosurfaces — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1403359710577786881 - Source code: https://github.com/keijiro/ComputeMarchingCubes - Interaction: Perception & Effects, Information & UI - Platform & tech: Desktop, Phone, Unity, Compute Shader, Marching Cubes - Idea: Turn any 3D field into a solid surface in real time. - What it is: A blobby, smoothly shaded surface forms and morphs in real time from a volume of values, and the same code can show CT scan data. - Technique: A compute shader runs marching cubes over a scalar grid, writes triangles straight into a mesh's GraphicsBuffer and reconstructs smooth normals from the field gradient. - Try it: Build a metaball sculpture whose blobs follow the phone and anchor it in the room with AR Foundation. Twist: add a blob wherever someone claps. #### FaceMeshBarracuda face & eye tracker — Keijiro Takahashi (2021) - Video: https://vimeo.com/545493860 - Source code: https://github.com/keijiro/FaceMeshBarracuda - Interaction: Face, Gaze & Attention - Platform & tech: Desktop, Phone, Unity, Barracuda, MediaPipe - Idea: Get a face mesh and eye tracking from one camera in Unity for mask and gaze effects. - What it is: A webcam face and iris tracker running in Unity overlays a dense face mesh and eye landmarks on the live video, ready for mask and gaze effects. - Technique: MediaPipe face-mesh and iris models run via Barracuda to return 468 face vertices plus iris landmarks, which are used to texture and deform an overlay mesh. - Try it: Use Spark, Lens Studio or MediaPipe FaceMesh to make a third-eye mask that follows your eye movements. Twist: the mask swaps to a different face when you blink. #### Flipbook2: flip book with watercolor — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1398540851941314568 - Source code: https://github.com/keijiro/Flipbook2 - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, Unity, HDRP, Shader Graph, KinoAqua - Idea: Show video as a physical object: a stack of pages. - What it is: A video clip is turned into a paper flip book whose pages turn in 3D, each page painted in watercolor style. - Technique: Frames are captured into a texture array and applied to page meshes that curl with a Shader Graph vertex deformation, then KinoAqua adds the watercolor look. - Try it: Capture phone camera frames and flip them as pages of a virtual book lying on a real table in AR Foundation. Twist: flip speed follows how fast you swipe. #### HandPoseBarracuda hand tracker — Keijiro Takahashi (2021) - Video: https://vimeo.com/545493967 - Source code: https://github.com/keijiro/HandPoseBarracuda - Interaction: Hands & Body - Platform & tech: Desktop, Phone, Unity, Barracuda, MediaPipe - Idea: Get finger joints in Unity from an ordinary camera as a building block for gesture effects. - What it is: A single webcam tracks all finger joints in real time inside Unity, drawing a 3D hand skeleton over the live image as a building block for gesture-driven effects. - Technique: MediaPipe palm-detection and hand-landmark neural networks run on the GPU through Unity Barracuda to output 21 3D keypoints per frame. - Try it: Track fingers with MediaPipe Hands (the web version is enough) and make your fingertips drag colored light trails across the camera view. Twist: when you pinch, the light trail becomes a rope you can grab. #### IrisBarracuda: eye and iris tracking — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1382324941861769220 - Source code: https://github.com/keijiro/IrisBarracuda - Interaction: Face, Gaze & Attention - Platform & tech: Desktop, Unity, Barracuda, MediaPipe Iris - Idea: Knowing where the iris is opens up gaze-driven effects. - What it is: A close-up of an eye is tracked in real time with points outlining the eyelid and the iris circle. - Technique: The MediaPipe iris landmark model runs on Barracuda on a cropped eye region and returns eyelid contour points and the iris center and radius. - Try it: Use AR Foundation eye tracking (or this model) to make particles shoot from where the user looks. Twist: blinking resets the scene. #### KinoAqua: watercolor effect with VFX Graph — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1392482946393677829 - Source code: https://github.com/keijiro/KinoAqua - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, HDRP, VFX Graph - Idea: A painterly filter makes computer graphics feel handmade. - What it is: Particle animations are rendered as if painted in watercolor, with soft edges, paper texture and pigment pooling. - Technique: A post-processing pass detects edges, darkens them like pigment pooling, adds noise-based wobble and multiplies a paper texture. - Try it: Apply a watercolor post effect to AR Foundation so both the real room and virtual objects look painted. Twist: the painting gets wetter where you point the phone for longer. #### M-LSD line detection as VFX — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1413426397054332930 - Source code: https://github.com/keijiro/MlsdBarracuda - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Unity, Barracuda, M-LSD, VFX Graph - Idea: The architecture of a room becomes the drawing. - What it is: Straight lines detected in a live camera image, like room edges and furniture outlines, are turned into glowing wires and particle sparks. - Technique: The M-LSD line segment detection model runs on Barracuda; detected segments are passed to VFX Graph through a buffer to spawn particles along them. - Try it: Detect lines in the AR Foundation camera image and trace them with sparks so the room's edges light up. Twist: only horizontal lines glow. #### NoiseBall6: compute shader mesh on mobile — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1402611991118712841 - Source code: https://github.com/keijiro/NoiseBall6 - Interaction: Perception & Effects - Platform & tech: Phone, Desktop, Unity, Compute Shader, Mesh API - Idea: Deform geometry directly on the GPU for heavy effects on phones. - What it is: A noisy, spiky ball deforms every frame on the GPU and runs on mobile phones as well as desktop. - Technique: The Mesh API exposes vertex and index buffers as GraphicsBuffers, so a compute shader rewrites the mesh without any CPU work. - Try it: Write a compute shader that deforms a mesh anchored with AR Foundation, and test its frame rate on your phone. Twist: the spikes point toward the nearest person. #### OAK-D-Lite stereo depth particles — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1474726909917667330 - Source code: https://github.com/keijiro/DepthAITestbed - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, DepthAI, OAK-D-Lite - Idea: A cheap stereo camera is enough for body particles. - What it is: A person seen by a small OAK-D-Lite stereo camera becomes a flowing particle figure in real time. - Technique: A small native plugin reads stereo depth from the DepthAI device, converts it to a position map and feeds VFX Graph. - Try it: Compare the depth from a stereo camera with a LiDAR iPhone in AR Foundation using the same particle graph. Twist: mix both sources in one scene. #### SelfieBarracuda: virtual background on phones — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1405518336230793223 - Source code: https://github.com/keijiro/SelfieBarracuda - Interaction: Hands & Body, Portals & Worlds - Platform & tech: Phone, Desktop, Unity, Barracuda, MediaPipe Selfie Segmentation - Idea: Segmentation separates the person from the world, and each can get its own effect. - What it is: A person is cut out from their surroundings in real time and placed on a new background, running smoothly on an iPhone X and a Pixel 5. - Technique: The MediaPipe Selfie segmentation model runs on Barracuda on the GPU and outputs a person mask used to composite the camera image over another background. - Try it: Use a segmentation mask in AR Foundation to replace everything except people with a VFX Graph world. Twist: the background world reacts to how much space the person fills. #### SushiVfx: vaporizing a sushi — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1442857433483726859 - Source code: https://github.com/keijiro/SushiVfx - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Photogrammetry - Idea: Real food, scanned, becomes a playful effect. - What it is: A photoscanned piece of sushi dissolves into a cloud of colored particles and reassembles. - Technique: VFX Graph samples positions and colors from the scanned mesh surface, and particles are pushed off by noise and pulled back to their original points. - Try it: Scan a snack with Polycam, make it vaporize into particles, and place it on the real plate with AR Foundation. Twist: it vaporizes when you bring a real fork close. #### TinyYOLOv2 object detection in Unity — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1353349183252533249 - Source code: https://github.com/keijiro/TinyYOLOv2Barracuda - Interaction: Information & UI, Tangible Objects - Platform & tech: Desktop, Unity, Barracuda, YOLO - Idea: Let the engine know what it is looking at. - What it is: A live video shows people, cars and other objects outlined with labeled boxes, detected by a neural network inside Unity. - Technique: The Tiny YOLOv2 ONNX model runs on Barracuda on the GPU; a compute shader decodes the output grid into boxes and removes overlaps before drawing. - Try it: Run an object detector on the AR Foundation camera feed and spawn a VFX Graph effect on every detected cup or person. Twist: each class gets its own effect. #### UltraFace: realtime emoji face overlay — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1361657191401365505 - Source code: https://github.com/keijiro/UltraFaceBarracuda - Interaction: Face - Platform & tech: Desktop, Unity, Barracuda, UltraFace - Idea: Face detection plus a sticker is the simplest face filter. - What it is: Every face in a video, even in a crowd, is covered with an emoji that follows it in real time. - Technique: The lightweight UltraFace detector runs on Barracuda; its bounding boxes position emoji sprites over each detected face. - Try it: Detect faces in the AR Foundation camera feed and hide each one behind a different emoji. Twist: the emoji changes with the person's distance to the phone. #### VFX Graph with 300,000 particles — Keijiro Takahashi (2021) - Video: https://x.com/_kzr/status/1400753544999817216 - Source code: https://github.com/keijiro/VfxGraphTestbed2 - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph - Idea: Scale changes the feeling: a crowd of particles becomes a material. - What it is: Hundreds of thousands of glowing particles swirl in dense ribbons and clouds, all simulated on the GPU in real time. - Technique: VFX Graph simulates particles with curl-noise style turbulence and additive blending, relying on GPU compute to update 300,000 particles per frame. - Try it: Test how many particles your phone can handle in an AR Foundation scene and design a piece at that limit. Twist: tie the particle count to the room's brightness. #### BibcamStage: live show at Channel 22 — Keijiro Takahashi (2022) - Video: https://www.youtube.com/watch?v=v3mRlMx_6Is - Source code: https://github.com/keijiro/BibcamStage - Interaction: Performance, Spatial Mapping, Location & City - Platform & tech: Projection, Phone, Unity, VFX Graph, LiDAR, Bibcam, Bitwig - Idea: Everyday walks recorded with a phone become stage material. - What it is: In a live audiovisual set, iPhone LiDAR recordings of city streets are replayed as volumetric scenes with particles and glitches, while generative music plays from Bitwig. - Technique: Bibcam videos store color, depth, human stencil and camera pose in one frame; Unity decodes them to rebuild the scene in 3D and applies VFX Graph effects synced to the music. - Try it: Record a short walk with a LiDAR phone (Record3D or Metavido), replay it as a point cloud on your desk with AR Foundation, and set it to music. Twist: the scene rebuilds only on the beat. #### Metawire: wireframe primitives for VFX — Keijiro Takahashi (2022) - Video: https://x.com/_kzr/status/1537797028817735680 - Source code: https://github.com/keijiro/Metawire - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Mesh, Scripted Importer - Idea: Wireframes give a technical, holographic look with very little geometry. - What it is: Glowing wireframe spheres, boxes and rings rotate and pulse as clean line graphics. - Technique: A scripted asset importer generates line-topology meshes for primitive shapes, which are drawn with emissive materials or used as VFX sources. - Try it: Wrap a real object in a wireframe box that tracks it with AR Foundation image or object tracking. Twist: the wireframe unfolds when you touch the object. #### Procedural walk with Animation Rigging — Keijiro Takahashi (2022) - Video: https://x.com/_kzr/status/1574788205010112513 - Source code: https://github.com/keijiro/CharacterRigTest - Interaction: Play, Spatial Mapping - Platform & tech: Desktop, Unity, Animation Rigging - Idea: Motion can be computed instead of keyframed. - What it is: A character walks and turns under keyboard control with no animation clips; its feet find the ground and its body leans naturally. - Technique: Animation Rigging constraints (two-bone IK and multi-aim) are driven by scripts that plan foot steps procedurally from the body's velocity. - Try it: Make a small procedural walker that follows your phone around a real floor using AR Foundation plane detection. Twist: let it climb onto real steps it detects. #### Speed lines post effect — Keijiro Takahashi (2022) - Video: https://x.com/_kzr/status/1506969479158243331 - Source code: https://github.com/keijiro/SimplePostEffects - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, Shader Graph - Idea: A comic convention turned into a screen effect. - What it is: Manga-style speed lines radiate from the center of the screen over a 3D scene, like a comic panel in motion. - Technique: A full-screen Shader Graph draws radial lines with polar-coordinate noise, animated over time and injected with a URP renderer feature. - Try it: Add speed lines to the AR Foundation camera image whenever the phone moves fast. Twist: the lines point toward a hidden virtual object. #### StickShow: a sea of glow sticks — Keijiro Takahashi (2022) - Video: https://x.com/_kzr/status/1564979744642179073 - Source code: https://github.com/keijiro/StickShow - Interaction: Performance, Voice & Sound - Platform & tech: Desktop, Unity, GPU Instancing, Shader Graph - Idea: A crowd is a repeated object with small differences. - What it is: A concert crowd of thousands of colored glow sticks sways in waves in front of a stage, all drawn in a single instanced call. - Technique: Graphics.RenderMeshInstanced draws one stick mesh thousands of times; per-instance data (color, phase) goes to a Shader Graph that swings each stick with a sine wave. - Try it: Fill a real floor detected by AR Foundation with a field of instanced glow sticks that sway to music. Twist: sticks near the viewer turn to face them. #### VFX with Unity AR Foundation — Keijiro Takahashi (2022) - Video: https://x.com/_kzr/status/1601509152395706369 - Source code: https://github.com/keijiro/Rcam2 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, Unity, AR Foundation, ARKit, LiDAR, VFX Graph - Idea: Grow light effects along the real environment live on a phone. - What it is: Live on an iPhone, glowing particles and lines grow over the real room and people in front of the camera, anchored to the LiDAR-scanned environment. - Technique: AR Foundation exposes the LiDAR environment depth and human stencil textures, which VFX Graph samples on-device to emit particles from real surfaces. - Try it: Use depth textures in AR Foundation or Lens Studio to make fireflies rise from the room's surfaces. Twist: fireflies scatter wherever a person walks. #### VzoVfx: triggering VFX from Bitwig — Keijiro Takahashi (2022) - Video: https://x.com/_kzr/status/1492752963206406148 - Source code: https://github.com/keijiro/VzoVfx - Interaction: Voice & Sound, Performance - Platform & tech: Desktop, Unity, VFX Graph, OSC, Bitwig - Idea: Let the music software drive the visuals directly. - What it is: Drum hits in a Bitwig music project trigger particle bursts in Unity in exact sync with the beat. - Technique: The VZO plugin sends note and parameter events from the DAW over OSC, and VzoVfx binders turn them into VFX Graph events with timing compensation. - Try it: Send OSC from a music app to a phone running AR Foundation so each drum hit bursts particles on a real object. Twist: map each drum to a different object in the room. #### Walking a Taiwanese street with iPhone LiDAR — Keijiro Takahashi (2022) - Video: https://x.com/_kzr/status/1607672355710709760 - Interaction: Spatial Mapping, Location & City, Perception & Effects - Platform & tech: Phone, Unity, ARKit, LiDAR - Idea: Turn LiDAR's short sight into a dreamlike style of street scene. - What it is: A night walk through a street in Taiwan is visualized as a sparse point-cloud world built live from the iPhone LiDAR, showing both the magic and the 5-meter range limit of the sensor. - Technique: LiDAR depth samples are reprojected into world-space points each frame and accumulated, so the scene is reconstructed as a trail of colored points around the walker. - Try it: Scan a corridor with Polycam or Scaniverse, export the point cloud, and use three.js to show only the points within 5 meters of you. Twist: make the point cloud flicker with your footsteps. #### AIShader: ChatGPT shader generator — Keijiro Takahashi (2023) - Video: https://x.com/_kzr/status/1632634562399600640 - Source code: https://github.com/keijiro/AIShader - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, Unity, ChatGPT, ShaderLab - Idea: Describe a surface in words and get a shader. - What it is: Typing a short text prompt in the Unity editor produces a working shader, and the material on screen instantly changes to match the description. - Technique: An editor extension sends the prompt with a fixed template to the ChatGPT API and imports the returned code as a Unity shader asset. - Try it: Generate three shaders from text prompts with an LLM, apply them to scanned real objects and place them back in the room with AR Foundation. Twist: the audience writes the prompts. #### BodyPix body part tracking — Keijiro Takahashi (2023) - Video: https://x.com/_kzr/status/1626200056033599491 - Source code: https://github.com/keijiro/BodyPixSample - Interaction: Hands & Body - Platform & tech: Desktop, Unity, Barracuda, BodyPix - Idea: Know which part of the body is where, and give each part its own effect. - What it is: A webcam image of a person is split into colored body parts in real time, with keypoints tracking arms, legs and face. - Technique: The BodyPix model runs on Barracuda/Sentis and outputs a 24-part segmentation map plus keypoints; shaders color each part and a sample uses keypoints as game input. - Try it: Use body segmentation in AR Foundation to make only your arms emit particles while the rest of the body stays normal. Twist: pixelate the face automatically for privacy. #### Dcam: realtime Stable Diffusion in live performance — Keijiro Takahashi (2023) - Video: https://www.youtube.com/watch?v=iVi-7oz67OU - Source code: https://github.com/keijiro/Dcam - Interaction: Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, Stable Diffusion, Core ML, NDI - Idea: Run image generation fast enough to become a live visual. - What it is: During Metome's live show, the camera image of the stage is repainted by Stable Diffusion image-to-image a few times per second and projected as flowing painted frames. - Technique: Apple's Core ML port of Stable Diffusion runs image-to-image on a Mac inside Unity, while an iPhone streams camera video and controls over NDI. - Try it: Send phone camera frames to an image-to-image model (e.g., a fast SDXL-Turbo server) and show the result as a floating panel beside the real scene in AR Foundation. Twist: the prompt changes with the song section. #### Duotone image effect — Keijiro Takahashi (2023) - Video: https://x.com/_kzr/status/1789384955136733476 - Source code: https://github.com/keijiro/Duotone - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, Render Graph - Idea: Reduce the palette to make an image graphic. - What it is: A full-color scene is redrawn with just two colors plus dithering, giving a printed, poster-like look. - Technique: A URP full-screen pass maps luminance to a two-color gradient with extra dark and highlight colors and applies ordered dithering. - Try it: Apply a duotone filter to the AR Foundation camera feed so the whole room looks like a risograph print. Twist: each person gets their own color pair. #### Gamma: live coding at GitHub Universe Recap — Keijiro Takahashi (2023) - Video: https://www.youtube.com/watch?v=gA9beOCv8s0 - Source code: https://github.com/keijiro/Gamma - Interaction: Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, KodeLife, GLSL, Syphon - Idea: Writing code in front of an audience is the performance. - What it is: At a party in Tokyo, Keijiro live-codes shaders in KodeLife while a DJ plays; the code layers feedback swirls and color over a 3D scene rendered in Unity. - Technique: Unity renders a 3D scene and likely shares it with KodeLife through Syphon, where a live-edited GLSL shader adds feedback and color effects while the code stays visible on screen. - Try it: Live-code a fragment shader (KodeLife or shadertoy-style WebGL) that processes a phone camera feed, then show it on a floating AR panel in the room. Twist: the audience shouts parameter values. #### SplatVFX: 3D Gaussian Splatting in VFX Graph — Keijiro Takahashi (2023) - Video: https://x.com/_kzr/status/1714214841265856932 - Source code: https://github.com/keijiro/SplatVFX - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Web, Unity, VFX Graph, Gaussian Splatting - Idea: A photoreal capture that is also a particle system. - What it is: A photoreal 3D scene captured with Gaussian Splatting renders in Unity as millions of soft splats, and can be scattered like particles. - Technique: Gaussian splat data (position, scale, rotation, color) is loaded into buffers and VFX Graph draws each splat as an oriented, alpha-blended quad sorted by depth. - Try it: Capture a splat of an object with a phone app (Polycam or Luma), render it next to the real object in AR Foundation, and make it dissolve when touched. Twist: swap the real object and its splat and ask viewers which is real. #### VFX Graph custom HLSL: plexus network — Keijiro Takahashi (2023) - Video: https://x.com/_kzr/status/1812469561704468638 - Source code: https://github.com/keijiro/VFXCustomCode - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, Unity, VFX Graph, HLSL - Idea: Connect nearby points and a swarm becomes a web. - What it is: Glowing points drift in space and connect to their nearest neighbors with thin lines, forming a moving network. - Technique: A Custom HLSL node in VFX Graph searches a buffer for each particle's nearest neighbor and outputs the target position used to draw connecting lines. - Try it: Spawn points on real surfaces from AR Foundation meshing and connect neighbors into a glowing web that covers the room. Twist: lines snap when someone walks through them. #### Volumetric fog from VFX Graph — Keijiro Takahashi (2023) - Video: https://x.com/_kzr/status/1615973816286744578 - Source code: https://github.com/keijiro/VolumetricVfxTest - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HDRP, VFX Graph - Idea: Particles can write into the fog instead of being sprites. - What it is: Soft purple smoke rises from a teapot and is lit as real volumetric fog, blending with the scene lighting. - Technique: VFX Graph's volumetric fog output (Unity 2023.1 HDRP) injects particle density into the local volumetric fog buffer so smoke receives scene lighting and shadows. - Try it: Make smoke rise from a real cup in AR Foundation using soft particles, lit by the estimated room light. Twist: blowing into the microphone pushes the smoke away. #### Dcam2: Stable Diffusion VJ set with DUB-Russell — Keijiro Takahashi (2024) - Video: https://www.youtube.com/watch?v=qa4jv5JhKhM - Source code: https://github.com/keijiro/Dcam2 - Interaction: Performance, Hands & Body - Platform & tech: Projection, Desktop, Unity, Stable Diffusion, Core ML, BodyPix - Idea: Mix generative AI with body tracking for live visuals. - What it is: At Channel 24, DUB-Russell's live set is accompanied by visuals in which camera footage is repainted by Stable Diffusion into glitchy, saturated illustrations. - Technique: Stable Diffusion image-to-image runs in real time on a Mac, combined with BodyPix segmentation and keypoints to place extra effects on performers (link between this video and the Dcam2 repo is likely but not stated). - Try it: Build a phone AR filter that segments people and sends only the background to an image-to-image model, then composites the painted background behind them. Twist: the style changes each time the beat drops. #### DrumPadVFX: finger drum visualizer — Keijiro Takahashi (2024) - Video: https://x.com/_kzr/status/1849430841119973885 - Source code: https://github.com/keijiro/DrumPadVFX - Interaction: Voice & Sound, Tangible Objects, Performance - Platform & tech: Desktop, Unity, VFX Graph, Minis, MIDI - Idea: Each pad gets its own visual voice. - What it is: Each hit on a Yamaha finger drum pad launches a matching burst of shapes and particles on screen, in sync with the rhythm. - Technique: Minis reads MIDI from the pad through the Input System and input binders fire VFX Graph events per note, built without writing code. - Try it: Map each pad of a MIDI controller (or on-screen pads) to an effect anchored on a different real object in AR Foundation. Twist: a combo of pads triggers a bigger effect that links all objects. #### FloatingHUD: floating interface effect — Keijiro Takahashi (2024) - Video: https://x.com/_kzr/status/1897620890919280893 - Source code: https://github.com/keijiro/FloatingHUD - Interaction: Information & UI, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Shader Graph, URP - Idea: Interface graphics as a visual effect in space. - What it is: Transparent HUD panels, grids and scan lines float around an object in 3D space like a sci-fi interface. - Technique: VFX Graph spawns quads textured with Shader Graph HUD elements and a filtered webcam feed, arranged and animated around a center. - Try it: Surround a real object with floating HUD panels in AR Foundation that show made-up data about it. Twist: the panels show real data, such as the room's noise level. #### Metavido VFX (WebGPU) — Keijiro Takahashi (2024) - Video: https://x.com/_kzr/status/1828366682689061280 - Source code: https://github.com/keijiro/MetavidoVFX - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, Web, Unity, WebGPU, VFX Graph, LiDAR - Idea: Put a phone video with depth onto a web page, so anyone can walk around its effects in a browser. - What it is: Volumetric clips captured with an iPhone Pro play back in a web browser, where the recorded person and space are re-rendered as particles and effects in 3D. - Technique: Metavido stores LiDAR depth and camera metadata inside a standard video stream, which a Unity WebGPU build decodes to reconstruct points for VFX in the browser. - Try it: Import a Record3D or phone depth video into three.js and rebuild a dance clip as a point cloud on a web page. Twist: when a viewer clicks, switch to a 'sand' material that collapses. #### Rcam3 for Masaya Matsuura's concert — Keijiro Takahashi (2024) - Video: https://x.com/_kzr/status/1871169155254435978 - Source code: https://github.com/keijiro/Rcam3 - Interaction: Performance, Hands & Body, Perception & Effects - Platform & tech: Phone, Projection, Unity, VFX Graph, ARKit, LiDAR - Idea: Use a phone's LiDAR as a volumetric camera on stage to generate live show visuals. - What it is: A fixed iPhone on stage captures the performers' depth, and Unity turns their silhouettes into large-scale VFX shown behind the band during Masaya Matsuura's concert. - Technique: An iPhone controller app streams color and LiDAR depth to a host PC, where the depth is converted to point positions that drive VFX Graph for projection. - Try it: Film a musician with a phone and turn their body outline into particles in TouchDesigner, projected behind them. Twist: make the drum hits burst the particles apart. #### VFX Graph sketch: glowing lines on a moving figure — Keijiro Takahashi (2024) - Video: https://x.com/_kzr/status/1790748365518725597 - Source code: https://github.com/keijiro/VfxGraphTestbed3 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph - Idea: Draw a body only with the lines that follow it. - What it is: Red glowing lines trace and orbit around a moving figure in the dark, drawing its motion in light. - Technique: VFX Graph samples positions on an animated skinned mesh and draws particle strips that follow those points with a lag (likely using mesh sampling and strip outputs). - Try it: Use AR Foundation body tracking to attach light strips to a real dancer's joints and view it through HoloKit. Twist: the strips remember the last 5 seconds of movement. #### Fluo: fluid and spectral color visualizer — Keijiro Takahashi (2025) - Video: https://x.com/_kzr/status/1962518821111280037 - Source code: https://github.com/keijiro/Fluo - Interaction: Performance, Perception & Effects - Platform & tech: Projection, Phone, Unity, Stable Fluids, spectral.js, NDI - Idea: The camera image becomes the dye in a fluid. - What it is: Live camera footage dissolves into a bright, flowing fluid of mixed pigments, used as visuals at club events. - Technique: Stable Fluids advects camera colors as dye while spectral.js mixes them like pigments; an iPhone streams the camera and controls to the host over NDI. - Try it: Stream the phone camera into a fluid shader and show the result on an AR panel that floats in front of the performer. Twist: the performer's movement adds force to the fluid. #### LightGridShader: LED display look — Keijiro Takahashi (2025) - Video: https://x.com/_kzr/status/1962445754007765480 - Source code: https://github.com/keijiro/LightGridShader - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, Shader Graph - Idea: Imitate a physical display's structure to make screens feel real. - What it is: Images and animations are shown as if on a giant LED screen, with visible diodes glowing and bleeding into soft bokeh. - Technique: A Shader Graph quantizes UVs into a grid, samples the source per cell and draws each cell as an emissive dot, combined with MiniBokeh blur. - Try it: Place a virtual LED billboard on a real building facade in AR Foundation that shows the phone camera feed. Twist: the LEDs flicker off where people's shadows fall. #### MiniBokeh: lightweight depth of field — Keijiro Takahashi (2025) - Video: https://x.com/_kzr/status/1959598355149180928 - Source code: https://github.com/keijiro/MiniBokeh - Interaction: Perception & Effects - Platform & tech: Desktop, Web, Unity, URP - Idea: A cheap bokeh makes a scene feel photographed. - What it is: Scenes with small bright lights get soft, round bokeh blur in the background, cheap enough for mobile and web. - Technique: Instead of a depth texture, the effect assumes one reference plane and applies a separable circular blur whose size grows with distance from that plane. - Try it: Add a bokeh blur to virtual lights in an AR Foundation scene so they match the phone camera's real blur. Twist: focus follows the nearest face. #### Pigment-based color mixing: fluid art — Keijiro Takahashi (2025) - Video: https://x.com/_kzr/status/1952365070601855069 - Source code: https://github.com/keijiro/PigmentTest - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, Unity, spectral.js, Shader - Idea: Mix colors the way paint does, not the way light does. - What it is: A webcam image is swirled into flowing paint where colors mix like real pigments, yellow and blue making green rather than gray. - Technique: Likely spectral.js-style Kubelka-Munk mixing converts colors to spectral reflectance before blending, applied inside a fluid advection shader. - Try it: Turn the AR Foundation camera feed into pigment paint that swirls where you drag your finger. Twist: freeze the painting onto a real wall as a virtual canvas. #### Realtime optical flow glitch machine — Keijiro Takahashi (2025) - Video: https://x.com/_kzr/status/1886351525569843694 - Source code: https://github.com/keijiro/OpticalFlowTest - Interaction: Perception & Effects, Hands & Body - Platform & tech: Desktop, Unity, Compute Shader, Optical Flow - Idea: Use motion itself to push pixels around. - What it is: Live video smears and melts in the direction things move, like a datamosh glitch, but computed in real time. - Technique: A compute shader estimates optical flow with the Lucas-Kanade method, and a feedback pass displaces the previous frame along the flow vectors to fake datamoshing. - Try it: Apply an optical-flow smear to the AR Foundation camera feed so waving hands drag the room with them. Twist: only a virtual object's area is smeared. #### TrackpadFluid: ten-finger fluid — Keijiro Takahashi (2025) - Video: https://x.com/_kzr/status/1956332698592805220 - Source code: https://github.com/keijiro/TrackpadFluid - Interaction: Hands & Body, Drawing & Making - Platform & tech: Desktop, Unity, Stable Fluids, Native Plugin - Idea: A trackpad becomes a multi-finger paint pool. - What it is: All ten fingers on a Mac trackpad stir a colorful fluid simulation at once, each finger pushing its own swirl. - Technique: A native plugin reads raw multitouch points from the trackpad and injects forces and dye into a GPU Stable Fluids solver. - Try it: Run a stable fluids simulation on a virtual pool lying on a real table in AR Foundation and stir it with multitouch on the phone screen. Twist: two phones stir the same pool. #### Triangle Splatting importer — Keijiro Takahashi (2025) - Video: https://x.com/_kzr/status/1939302534285582621 - Source code: https://github.com/keijiro/TriangleSplattingTest - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Unity, Triangle Splatting, Gemini CLI - Idea: Scenes captured as triangles run on any normal renderer. - What it is: A photoreal scene reconstructed with Triangle Splatting is loaded into Unity as a cloud of small colored triangles that can be viewed from any angle. - Technique: A custom importer (written with an AI coding agent) reads the .off triangle soup exported by Triangle Splatting and builds a vertex-colored mesh. - Try it: Import a captured triangle soup and place it at real scale in AR Foundation, then let triangles fly off as particles when touched. Twist: reassemble the scene only when viewed from the capture position. #### FlashGlitch: trigger-based glitch — Keijiro Takahashi (2026) - Video: https://x.com/_kzr/status/2030279235081060715 - Source code: https://github.com/keijiro/FlashGlitch - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, Unity, URP - Idea: A glitch that hits like a drum. - What it is: Short, sharp bursts of glitch flash across the image on each trigger, made to hit in time with musical beats. - Technique: A URP renderer feature plays a short randomized glitch envelope (displacement, color inversion, slicing) whenever a trigger method is called. - Try it: Trigger a glitch flash on the AR Foundation camera feed from each kick drum detected by the microphone. Twist: the glitch is only visible through HoloKit on one eye. #### Karbon: live camera visuals with Launchpad — Keijiro Takahashi (2026) - Video: https://x.com/_kzr/status/2034570627886239876 - Source code: https://github.com/keijiro/Karbon - Interaction: Performance, Tangible Objects - Platform & tech: Projection, Desktop, Unity, URP, Minis, Launchpad - Idea: Play camera effects like a drum machine. - What it is: Live camera feeds of a singer are sliced, tinted and glitched in real time for a club event, controlled by pressing pads on a Novation Launchpad. - Technique: A UVC capture device brings the camera into Unity URP, where a stack of custom full-screen effects is mapped to Launchpad pads via Minis, including polyphonic aftertouch. - Try it: Map four camera effects on a phone AR app to four on-screen pads and perform them live over a friend's music. Twist: pressure or hold time controls the strength. #### KinoGlitch URP: analog and digital glitch — Keijiro Takahashi (2026) - Video: https://x.com/_kzr/status/2025558651768070209 - Source code: https://github.com/keijiro/KinoGlitchURP - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP - Idea: Broken signals as an expressive style. - What it is: A scene is hit with scan-line jitter, color drift, block noise and a new horizontal ripple, recreating broken video signals. - Technique: Lightweight full-screen URP passes offset UVs per line and per block using noise, split RGB channels and are driven by camera-attached controller components. - Try it: Glitch the AR Foundation camera feed whenever a virtual object appears or disappears, so the reality seems to break. Twist: glitch strength grows the closer you get to a hidden anchor. #### Light leak effect — Keijiro Takahashi (2026) - Video: https://x.com/_kzr/status/2028469137132114330 - Source code: https://github.com/keijiro/LightLeakEffectExample - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, VFX Graph, Shader Graph - Idea: Borrow a film-camera accident as a mood. - What it is: Warm streaks and blobs of light drift across the frame like film light leaks, blending over the scene. - Technique: Light leak sources made with Shader Graph and VFX Graph are rendered by an offscreen camera into a RenderTexture, then blurred and composited onto the main image. - Try it: Add drifting light leaks to the AR Foundation camera feed and tie their color to the real room's light estimate. Twist: leaks enter from the direction of the nearest window. #### MeshSlicer: cutting objects in real time — Keijiro Takahashi (2026) - Video: https://x.com/_kzr/status/2074120483688374287 - Source code: https://github.com/keijiro/MeshSlicer - Interaction: Play, Hands & Body - Platform & tech: Desktop, Unity, Unity MCP, Physics - Idea: Cut any mesh and fill the wound. - What it is: Wooden crates are cut along a plane and fall apart into pieces with clean new faces, in an interactive demo built with AI coding agents. - Technique: The slicer splits triangles by a plane, generates cap geometry for the cut surface and turns the halves into physics objects; the code was written through Unity MCP with AI agents. - Try it: Slice virtual objects placed on a real table in AR Foundation by swiping the phone through them. Twist: slice a scan of a real object and hide the real one. #### Robust Video Matting on Mac — Keijiro Takahashi (2026) - Video: https://x.com/_kzr/status/2101943586606678108 - Source code: https://github.com/keijiro/unity-rvm-coreml - Interaction: Hands & Body, Portals & Worlds - Platform & tech: Desktop, Unity, Core ML, Metal, RVM - Idea: Clean mattes make any person a compositing layer. - What it is: People in a video are cut out cleanly from difficult backgrounds, hair included, in real time inside Unity. - Technique: The Robust Video Matting network runs through Core ML and Metal, using its recurrent state across frames to keep the alpha matte stable. - Try it: Cut out a performer with a matting model and place them as a floating cutout in an AR Foundation scene. Twist: surround the cutout with particles that only touch the background. #### StrobePages: page-turning post effect — Keijiro Takahashi (2026) - Video: https://x.com/_kzr/status/2021954733439848713 - Source code: https://github.com/keijiro/StrobePages - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, Renderer Feature - Idea: Turn smooth motion into a stack of pages. - What it is: The live scene is captured every few frames and each new capture flips over the previous one like a page, giving a stuttering flipbook look. - Technique: A URP renderer feature captures frames at a fixed interval and renders a page mesh that curls from the new frame onto the held previous frame. - Try it: Apply a page-flip effect to the AR Foundation camera feed and trigger each flip with a tap. Twist: flip automatically on each beat of a song. ### Ian Curtis *Product designer & creative technologist (XRarchitect); designer at World Labs* Architect-turned-XR designer known online as @XRarchitect; worked on AR at 8th Wall, Niantic and Niantic Spatial and now designs world-model tools at World Labs. #### A robot that tidies up ornaments — Ian Curtis (2021) - Video: https://x.com/XRarchitect/status/1474792447712669705 - Interaction: Spatial Mapping, Play - Platform & tech: Phone, Web, 8th Wall, A-Frame - Idea: A virtual robot tidies up Christmas ornaments on your real floor. - What it is: A virtual robot rolls around the real floor collecting fallen Christmas ornaments in WebAR. - Technique: Virtual ornaments and a robot share the detected floor plane, and the robot steers toward the nearest ornament with a simple seek behavior and collects it on contact. - Try it: Scatter a few small objects at random on the AR floor and have a small robot pick them up in order of distance. Twist: tap the floor with your phone to set up roadblocks for the robot. #### Candy cane tower vs. robot — Ian Curtis (2021) - Video: https://x.com/XRarchitect/status/1476342490349277190 - Interaction: Play, Spatial Mapping - Platform & tech: Phone, Web, 8th Wall, A-Frame, physics - Idea: Build a topple-able virtual candy tower on the floor, then release a troublemaking robot. - What it is: He stacks a physics-driven candy-cane tower on the floor in WebAR, and a small virtual robot knocks it down. - Technique: A physics engine simulates stacked rigid bodies on an invisible ground collider at the detected floor, and a scripted robot applies forces on contact to topple the tower. - Try it: Use A-Frame or Unity physics to build an AR block tower on the floor and add a small robot that crashes into it. Twist: let the audience steer the robot by tilting their phones. #### Resize your present with a photo scan — Ian Curtis (2021) - Video: https://x.com/XRarchitect/status/1475916160830951426 - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Phone, Web, photogrammetry, 8th Wall, A-Frame - Idea: Scan a real gift into a 3D model and scale it freely in AR. - What it is: A real gift is photo-scanned into a 3D model and re-imported into AR, where it can be scaled up or down. - Technique: Photogrammetry turns photos of a real object into a textured mesh, which is re-imported into AR and placed with hit test, with a pinch gesture mapped to uniform scale. - Try it: Scan a small object with Polycam or Object Capture, bring it back into AR and pinch to scale it up to the size of the room. Twist: once enlarged, you can walk inside it. #### 8th Wall hand tracking UX — Ian Curtis (2023) - Video: https://www.youtube.com/watch?v=i9hpAFDFnQo - Interaction: Hands & Body - Platform & tech: Phone, Web, 8th Wall, hand tracking - Idea: Interaction guidelines for grabbing and pinching virtual objects with your hands in web AR. - What it is: Gesture patterns, guidelines and demos for hand-tracked interaction in browser-based AR. - Technique: Browser hand tracking estimates 3D hand keypoints, and gestures such as pinch or open palm are classified from fingertip distances and mapped to AR actions. - Try it: Use MediaPipe Hands to recognize two gestures, pinch and open hand, and use them to grab and release an AR object. Twist: invent a new gesture and write a one-line instruction for classmates to test. #### 8th Wall moon portal — Ian Curtis (2023) - Video: https://64uauol9km.ufs.sh/f/SNo22QO8bFjm1Y3nWaemQEOeBiztFDvnIw7rqPp4adJoyxKf - Interaction: Portals & Worlds - Platform & tech: Phone, Web, 8th Wall R17 - Idea: Open a door to the moon right on the street. - What it is: Showcase WebAR portal to the moon, built to demonstrate the R17 engine's portal rendering. - Technique: A portal frame uses stencil masking so a lunar skybox and terrain render only through the opening, and an inside/outside test switches the full environment on when the camera crosses the plane. - Try it: Build a portal to the moon with a stencil, and use low-gravity physics inside so objects drift down slowly. Twist: after you step through, even the sound picked up by the phone microphone becomes muffled. #### Augmented Reality 2023 (compilation) — Ian Curtis (2023) - Video: https://x.com/XRarchitect/status/1736803320021528631 - Interaction: Spatial Mapping, Portals & Worlds - Platform & tech: Phone, Web, 8th Wall - Idea: A year of WebAR experiments, handy as a quick tour of what AR can do. - What it is: A reel of his favourite real-time WebAR experiences built during 2023. - Technique: A reel of browser AR pieces built on the same core toolkit (SLAM world tracking, hit-test placement, image and face targets), showing how far a small set of techniques can stretch. - Try it: Look back at the AR sketches you made this semester, cut them into a 60-second reel, and label each clip with the core technique it uses. Twist: make a 'failure reel' using only the prototypes that didn't work. #### Tetris in WebAR, written by ChatGPT — Ian Curtis (2023) - Video: https://x.com/XRarchitect/status/1737260350792425957 - Interaction: Play, Spatial Mapping - Platform & tech: Phone, Web, 8th Wall, ChatGPT - Idea: Let AI write the code and stand a game of Tetris up in the living room. - What it is: A playable Tetris board floating in the room, with the WebAR code generated by ChatGPT. - Technique: A 2D Tetris grid is rendered as a world-anchored vertical plane of cube meshes placed via hit test, with standard game logic running on a tick timer and on-screen buttons for input. - Try it: Have AI generate a Tetris game floating in AR and log every bug you have to fix by hand. Twist: when blocks land, they become physics blocks on the real floor that can be knocked over. #### Wol: AI owl guide to the redwoods — Ian Curtis (2023) - Video: https://www.youtube.com/watch?v=GUgAw-uU46o - Interaction: Voice & Sound, Information & UI - Platform & tech: Phone, Headset, Web, 8th Wall, LLM, Metaversal Deployment - Idea: Talk with an AR owl that shows you around the redwood forest. - What it is: An AR owl you talk to about the redwood forest ecosystem, running on phones and MR headsets from one web build. - Technique: An anchored animated character is wired to speech recognition, an LLM prompted with forest-ecology context, and text-to-speech, and one web build adapts to phone or headset input. - Try it: Place an animal character in AR and use speech recognition, an LLM and TTS so it answers questions about an ecosystem. Twist: give the character a boundary of knowledge and make it admit when a question goes beyond it. #### Niantic Studio spatial anchoring — Ian Curtis (2024) - Video: https://64uauol9km.ufs.sh/f/SNo22QO8bFjmPINkCFSvHen4wCFE5MZxyIKhr8BN3u2A7Djs - Interaction: Location & City, Spatial Mapping - Platform & tech: Phone, Web, Niantic Studio, VPS - Idea: Pick a real place in the editor and pin AR content to it precisely. - What it is: Developer workflow and demos for placing AR at real-world locations with VPS, including physics-driven content anchored to places. - Technique: VPS-localized anchors give a stable real-world pose, and a physics engine simulates content relative to that anchor, so rigid bodies collide with approximated real geometry at a specific place. - Try it: Localize at a fixed spot with an image marker, place physics balls that roll down a set of steps, and approximate the steps with simple colliders. Twist: let passers-by 'push' the balls with their phones. #### Scaniverse Gaussian splat map — Ian Curtis (2024) - Video: https://www.youtube.com/watch?v=1LQh_lm0VoM - Interaction: Location & City, Information & UI - Platform & tech: Phone, Web, Scaniverse, Gaussian Splatting - Idea: Browse real places scanned by people around the world on a map, then drop one into AR with a single tap. - What it is: UX for browsing Gaussian-splat captures of real places on a map, then jumping from map to 3D to AR views. - Technique: Geotagged splat captures are indexed on a map, and the UI moves from 2D map to an orbitable 3D viewer to an AR placement mode, reusing the same asset at each step. - Try it: Design and prototype a three-step 'map → 3D preview → AR view' flow in Figma or on the web, featuring three objects you scanned yourself. Twist: attach a voice message from the person who made each scan. #### Simulon AR VFX — Ian Curtis (2024) - Video: https://64uauol9km.ufs.sh/f/SNo22QO8bFjmnu8NCeEI5SrPLVvJzlUujsQFZmiWyCA3oX6K - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, Simulon, Blender - Idea: Shoot visual effects on location with phone AR, with lighting that matches the real environment. - What it is: Blender physics simulations composited into real locations using Simulon's AR capture and real-world lighting. - Technique: The tracked camera path and an estimated environment light probe from the real location are imported into Blender, where simulations are rendered with matching lighting and composited over the footage. - Try it: Read ambient light intensity and direction with AR Foundation light estimation so that the shadows of virtual objects on a table line up with those of real objects. Twist: make the virtual objects react differently when the lights are switched off and on. #### AI wayfinding companion (Spatial 1) — Ian Curtis (2025) - Video: https://x.com/XRarchitect/status/1948817329493512337 - Interaction: Location & City, Voice & Sound - Platform & tech: Wearable, Niantic VPS, Gemini, ElevenLabs - Idea: An AI wayfinding companion worn on the chest that watches the road and talks to you. - What it is: A 3D-printed wearable that sees what you see, localizes with VPS, and speaks directions with Gemini and ElevenLabs. - Technique: A worn camera sends frames for VPS localization and to a multimodal LLM, which turns the pose and scene description into spoken directions through text-to-speech, giving audio-only AR guidance. - Try it: Take a photo with a phone camera and send it to a multimodal model, asking it to give a one-sentence spoken direction toward a preset destination. Twist: make the guide a character with a personality, for example one who only speaks in metaphors. #### Augmented Reality Portal — Ian Curtis (2025) - Video: https://x.com/XRarchitect/status/1922897331461996689 - Interaction: Portals & Worlds - Platform & tech: Phone, Web, 8th Wall - Idea: An AR portal you can walk into just by opening a link. - What it is: A walk-in web AR portal that anyone can try from a link. - Technique: A walk-in portal uses a stencil or depth-mask shader on the doorway so the inner world is visible only through the frame, and it is distributed as a link through browser-based SLAM. - Try it: Build a portal that opens from a link using 8th Wall or WebXR, with a 360 panorama inside the door. Twist: let people outside hear sounds from inside, getting clearer as they approach. #### Desk-side web AR experiment — Ian Curtis (2025) - Video: https://x.com/XRarchitect/status/1976297639630266571 - Interaction: Spatial Mapping - Platform & tech: Phone, Web, WebAR - Idea: A quick, improvised WebAR experiment on the desk. - What it is: A quick web AR sketch built at his desk. - Technique: A minimal WebAR sketch most likely uses surface detection plus a hit test to place a single object, showing how fast a browser AR idea can be tested. - Try it: Give yourself 30 minutes to build a minimal working AR sketch at your desk from a WebXR template, and screen-record it. Twist: you may use only one kind of geometry, but it has to express an emotion. #### Location-triggered AR scene — Ian Curtis (2025) - Video: https://x.com/XRarchitect/status/1955008892443955566 - Interaction: Location & City - Platform & tech: Phone, VPS, WebAR - Idea: The AR story only appears once you walk to a particular place. - What it is: An older location-based AR experience: arrive at the spot, open the experience, and the scene comes to life in place. - Technique: A location check (GPS geofence or VPS localization) gates the experience, and once the device is localized the scene is instantiated at a fixed real-world pose. - Try it: Pick a spot on campus, use GPS distance to check whether someone has arrived, and only then make a character appear in AR. Twist: the character appears only when two people are there at the same time. #### Midjourney → splat → AR — Ian Curtis (2025) - Video: https://x.com/XRarchitect/status/1954372725667254569 - Interaction: Portals & Worlds - Platform & tech: Phone, Web, Midjourney, Gaussian Splatting, WebAR - Idea: Go from one AI image to an AR scene you can stand inside. - What it is: A generated image turned into an 8 MB Gaussian splat and viewed in web AR. - Technique: An AI image is lifted into a Gaussian splat by an image-to-3D pipeline, then compressed and rendered in a WebAR splat viewer on a hit-tested surface. - Try it: Generate a scene image with a text-to-image tool, convert it to 3D and place it in phone AR, then compare how the composition changed. Twist: have the whole class use the same prompt and line the results up as an AR gallery. #### Mixed Reality Map of San Francisco — Ian Curtis (2025) - Video: https://x.com/XRarchitect/status/1954989245787169202 - Interaction: Location & City, Information & UI - Platform & tech: Phone, Web, Mapbox, WebAR - Idea: A city map where tapping a pin opens AR right where you stand. - What it is: A web app for discovering location-based AR across San Francisco: open it on a phone, tap a pin, and launch the AR on the spot with no download. - Technique: A Mapbox web map lists geotagged AR experiences as pins, and tapping a pin deep-links to a WebAR URL, avoiding app installs. - Try it: Make a campus map with Mapbox or Leaflet where each pin links to a WebAR page. Twist: a link unlocks only when you are within 50 meters of its pin. #### Morning stroll into a portal — Ian Curtis (2025) - Video: https://x.com/XRarchitect/status/1970878813338051024 - Interaction: Portals & Worlds - Platform & tech: Phone, WebAR, World Labs - Idea: Walk through a door into a world generated by AI. - What it is: Walking through a doorway-sized portal into a World Labs-generated environment. - Technique: A doorway mesh placed on the ground uses a stencil buffer to show a generated environment only through the opening, and a camera-crossing check flips the render so the user ends up inside. - Try it: Place a door with AR Foundation plane detection, render a forest inside it with a stencil, and switch to a full surrounding environment once the user walks through. Twist: make the other side of the door the place where you lived as a child. #### Niantic Spatial launch + WebAR portal — Ian Curtis (2025) - Video: https://www.youtube.com/watch?v=kOyz2pNSPhk - Interaction: Portals & Worlds, Location & City - Platform & tech: Phone, Web, 8th Wall, Niantic Spatial - Idea: Turn the company launch itself into a WebAR portal you can walk into. - What it is: Brand and product vision for Niantic Spatial's debut, including a working WebAR portal released with the launch film. - Technique: A launch-film WebAR portal likely reuses stencil portal rendering on a hit-tested surface, so viewers can open the same world shown in the film on their own phones. - Try it: Shoot a 30-second concept film for a fictional product and pair it with an AR portal viewers can open. Twist: make the scene inside the portal continue from the film's last frame. #### Photo to WebAR world in real time — Ian Curtis (2025) - Video: https://x.com/XRarchitect/status/1980654061323255906 - Interaction: Portals & Worlds - Platform & tech: Phone, Web, WebAR, World Labs - Idea: A single photo becomes an AR scene you can walk into. - What it is: A single photo becomes a 5 MB Gaussian splat scene that runs in real time as a WebAR experience. - Technique: A single photo is converted by an image-to-3D model into a compact splat, which is then loaded in a WebAR viewer and placed on a detected surface. - Try it: Take a photo, turn it into a model with an image-to-3D tool, import it into WebAR on a tabletop, and compare it with the original photo to find what the generation got wrong. Twist: pick a painting or an old photo on purpose and see what it looks like in 3D. #### Redesigned living room, 1.5 MB per splat — Ian Curtis (2025) - Video: https://x.com/XRarchitect/status/1995541338335678801 - Interaction: Spatial Mapping, Information & UI - Platform & tech: Phone, Web, WebAR, three.js, World Labs - Idea: Switch between several AI-generated living-room designs right where they will go. - What it is: Nano Banana image → World Labs world → WebAR: several living-room redesigns as tiny .spz splats he can switch between in place. - Technique: Multiple compressed .spz splats share one aligned anchor in the room, and a UI toggle swaps which splat is visible so each redesign appears in the same place. - Try it: Prepare layout models in three styles for the same corner, share one anchor among them, and switch between them with a button. Twist: let classmates vote for their favorite and show the vote count in AR. #### Step into a persistent redesign, 1:1 — Ian Curtis (2025) - Video: https://x.com/XRarchitect/status/1968356682888823060 - Interaction: Spatial Mapping, Portals & Worlds - Platform & tech: Phone, Niantic VPS, World Labs, Gemini - Idea: AI design, AI generation and precise localization let you step into a redesigned living room at 1:1 scale in your own home. - What it is: Gemini designs a new living room, World Labs generates it in 3D, and VPS localizes it to his real room at 1:1 scale so he can step into it. - Technique: VPS localizes the phone against a prior map of the room, so a 1:1 generated redesign loads at the same real-world pose every session instead of needing manual alignment. - Try it: Save a room layout with ARKit ARWorldMap or persistent anchors in AR Foundation, and restore it to the same place automatically the next time the app opens. Twist: shift the furniture slightly each time based on that day's weather. #### VPS-anchored WebAR demo — Ian Curtis (2025) - Video: https://x.com/XRarchitect/status/1922797867514683753 - Interaction: Location & City - Platform & tech: Phone, Web, Niantic VPS, WebAR - Idea: Content pinned to a real place, so everyone who comes sees the same thing in the same spot. - What it is: A real-time WebAR experience anchored to a specific real-world location with a visual positioning system. - Technique: A browser-based VPS matches camera frames against a pre-scanned location map to recover a 6DoF pose, so content is anchored to an exact real-world spot rather than to the first detected plane. - Try it: Simulate VPS with image tracking (put a poster at a real location as the marker) and anchor content at a fixed spot. Twist: the content appears only during a certain time window. #### three.js in real life — Ian Curtis (2025) - Video: https://x.com/XRarchitect/status/1940566978185633835 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, Web, three.js, WebAR - Idea: Move the effects of a web 3D engine out into the real world. - What it is: A three.js scene pulled off the screen and placed in the physical world with AR. - Technique: An existing three.js scene is wrapped in a WebXR immersive-ar session, placed at a hit-test point, and scaled to real-world units. - Try it: Pick one of the official three.js examples, add a WebXR AR session and a hit test, and put it on the real floor. Twist: change its scale to make it very large or very small, and notice how the experience changes. #### A 20-million-splat San Francisco on the table — Ian Curtis (2026) - Video: https://x.com/XRarchitect/status/2071471052086194197 - Interaction: Location & City, Spatial Mapping - Platform & tech: Phone, Web, WebAR, three.js, SparkJS - Idea: Shrink all of San Francisco into a tabletop model, then move the phone closer to dive into its streets. - What it is: A 20M-splat scan of San Francisco runs on a phone in WebAR; moving the phone closer zooms into individual streets. - Technique: A very large splat city is streamed with level-of-detail chunking, placed on a detected tabletop plane, and the renderer loads finer splats as the camera distance to each chunk shrinks. - Try it: Place a 3D model of a city or campus on a tabletop with three LOD levels, and switch to the high-detail version when the phone moves closer. Twist: when the phone approaches a particular building, play a one-line voice introduction about it. #### A picture frame that is a window — Ian Curtis (2026) - Video: https://x.com/XRarchitect/status/2052418499516170377 - Interaction: Portals & Worlds - Platform & tech: Headset, Web, WebXR, Marble, Cursor - Idea: A 'picture frame' on the wall is really a window into another 3D world. - What it is: A WebXR picture frame on the office wall that shows a persistent generated 3D world behind it, like a window into another place. - Technique: A frame is anchored to a wall via plane detection, and a stencil mask limits rendering of a persistent 3D world to the frame's rectangle, so parallax makes it read as a window. - Try it: Use AR Foundation to detect a wall plane and place a picture frame on it, then render a distant scene inside the frame with a stencil. Twist: make the time of day and weather inside the frame follow the real clock. #### A pocket holodeck in WebAR — Ian Curtis (2026) - Video: https://x.com/XRarchitect/status/2014383901268181253 - Interaction: Portals & Worlds, Spatial Mapping - Platform & tech: Phone, Web, WebAR, SparkJS - Idea: Each AI-generated scene is only 5 MB, so a few of them placed around you become a holodeck you can carry in your pocket. - What it is: Gaussian splats of about 5 MB each, generated from single images, placed around him in web AR like a miniature holodeck. - Technique: Several small single-image splats (about 5 MB each) are placed at fixed offsets around the user's starting pose in WebAR, forming a ring of scenes the user turns to face. - Try it: Make four small scenes with a single-image-to-3D tool (or simple 3D models as stand-ins) and place them around yourself facing north, south, east and west. Twist: let the scene in each direction stand for a different time of day. #### Coit Tower in the living room — Ian Curtis (2026) - Video: https://x.com/XRarchitect/status/2051683566292222009 - Interaction: Spatial Mapping, Location & City - Platform & tech: Headset, Web, WebXR, Spark 2.0 LoD, Quest 3 - Idea: Bring a huge scan of a real landmark into the living room and walk around it in a headset. - What it is: A 40M-splat scan of San Francisco's Coit Tower placed in his living room, running fully on-device on Quest 3 through WebXR. - Technique: A 40M-splat scan runs on a standalone headset through hierarchical LoD that culls and simplifies distant splats, keeping the per-frame splat budget within mobile GPU limits. - Try it: Pick a high-poly model, measure the frame rate of the original and a decimated version on HoloKit or a phone, and find the limit for smooth playback. Twist: use that budget to design a scene that brings a landmark into your living room. #### Game Boy graphics in real life — Ian Curtis (2026) - Video: https://x.com/XRarchitect/status/2082706695646777746 - Interaction: Perception & Effects, Play - Platform & tech: Phone, WebAR - Idea: Bring the Game Boy pixel-art look into a real living room. - What it is: A living-room test that brings chunky Game Boy-style graphics into the real room. - Technique: Low-poly content is rendered with nearest-neighbor texture filtering, a limited four-color palette and a pixelation post-process, then anchored to room planes so the retro style contrasts with the camera feed. - Try it: Add a pixelation and four-color palette post-processing shader to an AR scene and place a low-poly character on the floor. Twist: apply the post-process to the camera feed too, so the whole real room turns into Game Boy style. #### Gaussian splat capture in Quest mixed reality — Ian Curtis (2026) - Video: https://x.com/XRarchitect/status/2056211832613089323 - Interaction: Spatial Mapping - Platform & tech: Headset, Web, WebXR, Spark, Quest - Idea: Put a scan of a real place into MR so it sits side by side with reality. - What it is: A friend's Gaussian splat capture viewed in Quest passthrough MR via WebXR and Spark level-of-detail streaming. - Technique: A captured splat is rendered over passthrough in WebXR immersive-ar mode, with LoD streaming loading detail progressively so the capture appears quickly and sharpens as it streams. - Try it: Scan a classmate or an object with Scaniverse or Polycam, export it, and view it overlaid on real space in WebXR. Twist: gather several scans together into a small 'scan museum'. #### Living room redesign as WebAR splat — Ian Curtis (2026) - Video: https://x.com/XRarchitect/status/2076932513822617966 - Interaction: Spatial Mapping, Portals & Worlds - Platform & tech: Phone, Web, 8th Wall, SparkJS, World Labs - Idea: Overlay an AI-designed living room on the old one at 1:1 scale, and walk into the 'future room' with a phone. - What it is: A generated interior redesign of his living room rendered as a Gaussian splat and aligned in WebAR, so he can walk through the future room on his phone. - Technique: A generated splat of the redesigned room is placed at 1:1 scale and manually aligned (origin + yaw) to the real room, so SLAM tracking lets the user walk through the future interior on a phone. - Try it: Photograph your room and have an image model redesign it, then lay out simplified furniture in any 3D tool and place it in AR at 1:1 scale, aligned with the real room. Twist: add a slider that fades between 'now' and 'after the makeover'. #### Off-axis projection with face tracking — Ian Curtis (2026) - Video: https://x.com/XRarchitect/status/2008221154100580600 - Interaction: Face, Perception & Effects - Platform & tech: Desktop, Web, MediaPipe, three.js, off-axis projection - Idea: A webcam tracks your face and turns an ordinary screen into a window with depth that shifts as your head moves. - What it is: A head-coupled perspective demo: MediaPipe tracks the viewer's face so a 3D scanned shoe on a flat screen shifts perspective like a window. - Technique: MediaPipe face landmarks estimate the eye position relative to the screen, and an off-axis (asymmetric frustum) projection is recomputed each frame so the monitor behaves like a window with correct parallax. - Try it: Estimate head position with MediaPipe Face Mesh and update an asymmetric frustum projection in three.js from the head coordinates, so a shoe on screen shows parallax. Twist: make the object 'lean out' past the screen plane and test at what angle the illusion breaks. #### Off-axis window into a generated world — Ian Curtis (2026) - Video: https://x.com/XRarchitect/status/2034675907898876133 - Interaction: Face, Portals & Worlds - Platform & tech: Desktop, Web, MediaPipe, three.js, World Labs - Idea: Block out a scene with simple shapes, let AI fill in the detail, then use face tracking to turn the screen into a window onto that world. - What it is: Blender layout turned into a detailed World Labs splat, viewed through a face-tracked off-axis projection so the monitor becomes a window. - Technique: A generated splat world is rendered with a head-coupled off-axis projection driven by webcam face tracking, placing the scene behind the monitor plane so it reads as a view through glass. - Try it: Reuse the face-tracked off-axis projection and replace what lies behind the screen with a long corridor or canyon to exaggerate the sense of depth. Twist: as your head moves closer to the screen, let the fog in the scene gradually clear. #### Shrunk to action-figure size on the coffee table — Ian Curtis (2026) - Video: https://x.com/XRarchitect/status/2053874498630468078 - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Headset, Web, WebXR, three.js, SparkJS, World Labs - Idea: Shrink yourself to action-figure size and explore a miniature world on the coffee table. - What it is: In mixed reality he shrinks himself to action-figure scale and explores a tiny generated world sitting on his living-room coffee table. - Technique: A miniature world is placed on the coffee table in passthrough MR, and scaling the user rig (or inversely scaling the world) changes the effective eye height so the table feels human-sized. - Try it: Place a small scene on a tabletop in WebXR or Unity, and when a button is pressed, scale the whole scene up 20 times around the tabletop origin. Twist: after scaling, let real objects on the table (like a cup) become 'giant landmarks' in the scene. #### Splat worlds as AR camera stand-ins — Ian Curtis (2026) - Video: https://x.com/XRarchitect/status/2072207377106153797 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, 3D Gaussian Splatting, AR camera, video diffusion - Idea: Use an AR phone as a virtual camera to block shots inside a generated 3D scene, then hand the footage to a video model for the final look. - What it is: Generated 3DGS environments act as a spatial reference for directing real-time AR camera moves; the capture is then restyled by a video model. - Technique: A tracked phone acts as a virtual camera inside a splat environment, recording its 6DoF pose path as a camera move, and the rendered pass is then restyled frame-by-frame with a video diffusion model. - Try it: Record the phone's camera pose path with AR Foundation, then use it in Unity to drive a virtual camera through a prebuilt scene and export the video. Twist: pass the exported video to a video stylization model to give it a hand-drawn look. #### Tap to launch orbs into a generated world — Ian Curtis (2026) - Video: https://x.com/XRarchitect/status/2077641433855795594 - Interaction: Play, Spatial Mapping - Platform & tech: Phone, Web, 8th Wall, three.js, SparkJS, World Labs - Idea: Overlay an AI-generated 3D world on a real room and tap to fire glowing orbs into it. - What it is: A WebAR scene built from a World Labs Gaussian splat of his room; tapping the phone fires glowing orbs that fly through the space. - Technique: A pre-generated Gaussian splat of the room is aligned to the SLAM world origin, and each tap spawns an emissive sphere at the camera position with a forward velocity vector, so projectiles appear to fly through the captured space. - Try it: Use three.js and WebXR to spawn glowing balls at the camera position and launch them along the line of sight, with a simple room model as the 'world'. Twist: make the balls bounce off walls or leave a spot of light where they hit. #### Touching grass: two-prompt WebAR — Ian Curtis (2026) - Video: https://x.com/XRarchitect/status/2072573377714471177 - Interaction: Spatial Mapping, Hands & Body - Platform & tech: Phone, Web, 8th Wall, three.js, LLM coding - Idea: Two prompts produce an outdoor WebAR prototype: the AI writes the code, and you play with virtual objects right on the grass. - What it is: An outdoor WebAR prototype with SLAM tracking, built with two prompts to an AI coding model, where he drags virtual objects across real grass. - Technique: Outdoor SLAM with ground-plane detection lets a raycast from the touch point move an object along the detected plane, so dragged objects stay pinned to the grass. - Try it: Get an AI to write a WebAR drag-and-drop scene using only two or three prompts, and record each prompt and its result. Twist: make the dragged object leave a trail line on the grass. ### Adrien M & Claire B (Adrien Mondot & Claire Bardainne) *Digital art and performance company* Company combining juggler-programmer Adrien Mondot and visual artist Claire Bardainne; their live, physics-driven projections (eMotion software) and AR books/exhibitions make drawings breathe around dancers and visitors. #### Convergence 1.0 — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2005) - Video: https://vimeo.com/954911 - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, eMotion, physics simulation, real-time projection - Idea: A juggler juggles with objects that only exist inside the computer. - What it is: Adrien Mondot's founding piece: a juggler plays with real balls while virtual balls and objects, generated live by the software eMotion and projected behind the performer, join the routine so real and synthetic trajectories meet on stage. - Technique: Custom real-time software (the first version of eMotion) simulates objects with physical properties and projects them behind the performer; the match between real throws and virtual balls is likely a mix of rehearsed timing and partial tracking. - Try it: Project a simple 2D physics scene (p5.js + matter.js) on a wall and have one student toss a real ball in front of it while a partner launches matching virtual balls with key presses; rehearse until real and virtual throws read as one routine. Twist: track a brightly colored ball with the webcam so the virtual balls bounce off the real one. #### Time remap – Anamorphose temporelle — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2009) - Video: https://vimeo.com/7878518 - Interaction: Perception & Effects, Hands & Body - Platform & tech: Desktop, Projection, Quartz Composer, slit-scan, GPU - Idea: Turn time into an axis of space, so a moving body becomes a sculpture of its own recent past. - What it is: Two people move in a park while each horizontal band of the image comes from a slightly different moment, so their bodies stretch, twist and smear through time. - Technique: A custom Quartz Composer plug-in stores the live video stream as a 3D texture (x, y, time) and samples each row from a different delay on the GPU at 60 fps; this slit-scan effect also runs live for visitors in the XYZT exhibition. - Try it: Build a live slit-scan mirror in p5.js or TouchDesigner: keep a buffer of the last 120 webcam frames and draw each row from an older frame the lower it sits on screen, then let students hunt for poses that look impossible. Twist: make the delay depend on distance from a tracked hand instead of the row, so time bends around a point. #### Cinématique — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2010) - Video: https://www.youtube.com/watch?v=CknHVdMZ5xg - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, eMotion, physics simulation, real-time projection - Idea: Let a dancer push projected letters and lines as if they were objects with weight. - What it is: A dancer and a juggler-programmer perform inside projected letters, lines and particles that behave with physics and react to their gestures in real time. - Technique: Adrien Mondot's eMotion software runs physics-based particle and line systems, driven live by a digital performer and by motion sensing of the dancers. - Try it: Make text particles with gravity in p5.js or Processing, project them on a wall, and let classmates push them away with hand gestures (via camera). Twist: once scattered, the letters slowly reassemble into a new sentence. #### Coïncidence — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2011) - Video: https://vimeo.com/35528568 - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, eMotion, real-time projection, live digital performance - Idea: A duet between a body and a live-played image, where letters behave like juggled objects. - What it is: A 20-minute juggled and danced duet in a minimal black space: abstract shapes and words projected around Adrien Mondot come alive, fall and scatter with the juggler's gestures, while Claire Bardainne plays the images live. - Technique: Projected typography and particles are driven by eMotion's physics engine and steered live by a digital performer, so the timing comes from a human partner rather than full body tracking. - Try it: Pair students as performer and image-player: one moves in front of a projection, the other drives falling words in a p5.js + matter.js sketch with a MIDI controller or keyboard, and they rehearse until the words seem to answer the body. Twist: swap roles without rehearsal and compare how much the illusion depends on timing. #### XYZT, Abstract Landscapes — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2011) - Video: https://www.youtube.com/watch?v=N7q9k9Z3HIk - Interaction: Hands & Body, Projection, Voice & Sound - Platform & tech: Projection, eMotion, depth/camera sensing, projection - Idea: Touch, blow and walk to reshape landscapes made of letters and particles. - What it is: An exhibition of ten interactive installations where visitors touch, blow on and walk through projected typographic and particle landscapes. - Technique: Floor and wall projections use camera, depth and touch sensing to feed eMotion physics systems, so visuals respond like material. - Try it: Build a floor-projected particle pool and use an overhead camera so people walking through 'part' the particles. Twist: use a microphone to let the strength of a breath control which way the particles scatter. #### Hakanaï — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2013) - Video: https://www.youtube.com/watch?v=xvJNia3z11I - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, eMotion, tulle projection, live visuals - Idea: A dancer in a cube of gauze converses with live projected lines, like a short poem. - What it is: A dancer inside a cube of tulle interacts with projected lines, grids and letters that are performed live around her; afterwards the audience can enter the cube. - Technique: Four-sided projection on tulle scrims is 'played' live by a digital performer using eMotion, with motion sensing linking the images to the dancer. - Try it: Enclose a small cube with four pieces of gauze or white cloth, project line animations onto it, and have a classmate improvise with the lines inside. Twist: let the audience control the lines from their phones to interact with the person inside. #### eMotion × Leap Motion – Pepper's ghost test — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2013) - Video: https://vimeo.com/71216887 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, eMotion, Leap Motion, Pepper's ghost - Idea: Your hands sculpt light that hovers in mid-air, without any headset. - What it is: A first test with the company's own software: hands tracked by a Leap Motion controller shape glowing strands of light that seem to float inside a small glass box in front of them. - Technique: eMotion, the company's real-time motion and physics software, reads hand positions from a Leap Motion and renders particle strands that are reflected in an angled glass sheet (Pepper's ghost), so they appear to float in physical space. - Try it: Build a Pepper's ghost box from a clear plastic sheet angled at 45° over a tablet lying face up, run a browser particle sketch driven by MediaPipe Hands from a laptop webcam and mirror it to the tablet, and tune colors on black until the image reads as floating. Twist: let two people's hands pull on the same particles. #### Pixel (with Mourad Merzouki / Compagnie Käfig) — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2014) - Video: https://www.youtube.com/watch?v=z_Hu57QTqqE - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, eMotion, floor projection - Idea: Hip-hop dancers move together with pixels of light on the floor. - What it is: Eleven hip-hop and circus dancers perform with projected points of light on the floor and backdrop that move, ripple and cascade with their bodies. - Technique: Floor and back projections of particle fields are generated live with eMotion and synchronised with the choreography by a digital performer. - Try it: Project a field of light dots on the floor and have classmates breakdance while the dots ripple outward from their steps. Twist: make the dots form a "river" that can only be crossed when two people work together. #### Le mouvement de l'air — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2015) - Video: https://www.youtube.com/watch?v=xsskbGYq7lc - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, eMotion, fluid simulation - Idea: Turn invisible air currents into visible imagery swirling around the dancers. - What it is: Three dancers move through projected wind, water and particle landscapes that swirl around their bodies as if air had become visible. - Technique: Real-time fluid and particle simulations in eMotion are projected on floor and backdrop and steered live with the dancers' movement. - Try it: Project a web fluid simulation (WebGL fluid) and use camera optical flow so classmates' movements stir up the "air". Twist: sync real wind from a fan with the projection. #### La neige n'a pas de sens — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2016) - Video: https://www.youtube.com/watch?v=U7-BHx3B_jA - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Phone, image-tracked AR, mobile app, particle systems - Idea: A printed art book whose pages hold living, moving matter. - What it is: The company's first monograph is also an AR book: pointing a tablet at six of its pages makes rain, a cloud, a wave, a tornado, a cocoon and grass rise out of the paper and move. - Technique: A custom mobile app recognizes the printed page images and anchors real-time generative animations of natural phenomena, likely eMotion-style particle systems, onto each page. - Try it: Print three abstract black-and-white patterns, make them image targets in Lens Studio, AR Foundation or MindAR, and attach a particle effect that grows out of each (rain, smoke, grass); test how the print design changes the feel of the animation. Twist: make the matter respond to tilting the book, as if it could spill off the page. #### Mirages & miracles — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2017) - Video: https://www.youtube.com/watch?v=hJBTqYbhTtA - Interaction: Tangible Objects, Perception & Effects, Drawing & Making - Platform & tech: Phone, Projection, image-tracked AR, Pepper's ghost - Idea: Seen through a tablet, real stones and hand drawings are home to tiny breathing people. - What it is: An exhibition of drawn stones, landscapes and figures that come alive through tablets and holographic displays, where tiny dancers climb and breathe inside real rocks. - Technique: Image-tracked AR on tablets and Pepper's-ghost holographic boxes overlay captured dancers onto physical stones and drawings. - Try it: Make image-recognition AR for a stone or a hand drawing (e.g., with Unity or 8th Wall) so a recorded dance of a tiny figure appears on it. Twist: build a small Pepper's ghost box from a phone screen and a transparent sheet. #### L'ombre de la vapeur — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2018) - Video: https://vimeo.com/278181935 - Interaction: Projection, Hands & Body, Location & City - Platform & tech: Projection, eMotion, floor projection, presence sensing - Idea: Make an invisible process, alcohol evaporating and the fungus living on it, into a landscape you can walk through. - What it is: A 900 m² immersive, interactive installation at the Fondation d'entreprise Martell in Cognac: a vast cloud of crumpled reflective material hangs over a floor of projected particles that visitors walk through and disturb, a tribute to the Torula fungus that once fed on the building's evaporating alcohol. - Technique: Floor projection with presence sensing likely feeds eMotion particle systems that scatter around visitors, combined with a large physical sculpture that catches the light, so digital and material vapour share one space. - Try it: Choose an invisible process in your building (radiator heat, humidity, Wi-Fi traffic) and project a particle field on the floor that visitors disturb when a top-down webcam sees them; add one physical object that reflects or blocks the projection. Twist: make the particles slowly drift back toward where the invisible source really is. #### Acqua Alta – Noir d'encre — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2019) - Video: https://www.youtube.com/watch?v=FOF1xKHe7e0 - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, eMotion, real-time projection, hand-drawn animation - Idea: A drawn world of ink that floods, flows and grows hair in dialogue with the performers. - What it is: The stage version of the Acqua Alta triptych: two performers play a couple whose house is swallowed by a sea of ink, and the projected black-and-white drawings of water and living hair move with their bodies. - Technique: Projections on the floor and on a vertical screen are played live with eMotion, so hand-drawn animation behaves like fluid and hair that react to where the performers are. - Try it: Draw a short ink-style loop (a wave, a strand of hair), project it on the floor, and use a top-down webcam with frame differencing so the strokes flow away from anyone who steps in; stage a one-minute scene in which the image is a second character. Twist: continue the same story on paper as an AR image target, as the company did with its pop-up book. #### Faire corps — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2019) - Video: https://www.youtube.com/watch?v=VJmHz4OgnMo - Interaction: Hands & Body, Projection, Shared & Social - Platform & tech: Projection, camera sensing, real-time projection - Idea: An exhibition space that people enter together and change with their bodies. - What it is: An 'exhibition-experience' at Gaîté Lyrique where visitors enter projected, interactive environments that react to their bodies and invite them to move together. - Technique: Camera and depth sensing of visitors drives real-time projected particle and line environments on floors and walls. - Try it: Design a projected space that changes only when three or more people move together (for example, jumping together changes the colors). Twist: add a hidden image that appears only when everyone is completely still. #### Équinoxe — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2019) - Video: https://vimeo.com/343654541 - Interaction: Performance, Projection, Voice & Sound - Platform & tech: Projection, eMotion, tulle projection, live visuals - Idea: Turn a live concert into a journey through a space built from light. - What it is: A concert-show with the band Limousine and dancer Akiko Kajihara, conceived as a nod to Hakanaï: musicians and dancer perform inside moving landscapes of projected lines and grids that unfold like a road movie. - Technique: Images are projected on tulle and a back screen and performed live by Adrien Mondot, likely following the music's tempo and the dancer's movement through eMotion. - Try it: Pick a three-minute instrumental track and build a projected perspective grid in TouchDesigner or p5.js that travels forward with the beat; one student dances in front while another plays the image live. Twist: let bass energy from audio analysis bend the grid so the space breathes with the music. #### Acqua Alta – Crossing the Mirror (AR pop-up book) — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2020) - Video: https://vimeo.com/462776850 - Interaction: Tangible Objects, Portals & Worlds, Drawing & Making - Platform & tech: Phone, marker-based AR, motion capture, mobile app - Idea: Through a phone, ink-drawn figures dance between the pages of a pop-up book and get swept up by a flood. - What it is: A black-and-white ink pop-up book that, viewed through a phone or tablet, fills with animated rain, rising water and two dancing figures moving through its pages. - Technique: Marker-based AR recognises each pop-up spread and anchors motion-captured dance animation and water simulations to the paper geometry. - Try it: Make a three-page paper pop-up book and use image-recognition AR to play an animation on each page. Twist: the order in which pages are turned changes the ending. #### Effluve (Faire corps) — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2020) - Video: https://www.youtube.com/watch?v=HC47brh6tio - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, projection, motion tracking, particle physics - Idea: Let visitors breathe life into a cloud of light with their bodies. - What it is: In the Faire corps exhibition, visitors stand before a projected field of particles that drift like vapour and respond to their gestures and presence. - Technique: Body tracking likely drives a physics-based particle system (probably the studio's own eMotion tool) projected at human scale. - Try it: Project a particle cloud on a wall and push it with a webcam optical-flow field so gestures blow the particles like mist. Twist: particles slowly form the silhouette of the last visitor. #### Research lab with Daniell Alnuma — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2020) - Video: https://vimeo.com/479608301 - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, eMotion, floor projection, body tracking - Idea: A spinning body becomes the center of gravity for a field of lines. - What it is: An unedited improvisation from a lockdown research lab with dancer Daniell Alnuma: the dancer spins at the center of a floor projection while white lines spiral around the body, as if the rotation were drawing the vortex. - Technique: A floor projection runs a real-time line simulation, likely eMotion, whose center of attraction follows the dancer (likely via camera tracking); the video is a single improvised take with no post-production. - Try it: Project a p5.js vector field on the floor whose center follows a person tracked by a top-down webcam (blob centroid), and ask a student to improvise for two minutes using only spins and slow walks. Twist: add a second attractor for a second dancer and watch the lines negotiate between them. #### Faune — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2021) - Video: https://vimeo.com/566519444 - Interaction: Tangible Objects, Location & City, Play - Platform & tech: Phone, image-tracked AR, mobile app, posters - Idea: Street posters hide animals that only appear when you stop and look through your phone. - What it is: Ten large posters, co-signed with the graphic design collective Brest Brest Brest, are pasted on city walls; seen through a free AR app, each printed landscape expands and the animal hidden in it bounces, slips away and reappears. - Technique: A custom mobile app tracks each printed poster as an image target and plays hand-drawn animations that extend the graphic landscape beyond the edges of the paper. - Try it: Design one A3 poster whose landscape hides an animal, make it an image target in Lens Studio, AR Foundation or MindAR, and animate the animal escaping from its hiding place; paste it on campus and map a short trail. Twist: the animal only moves once the viewer has held still for three seconds. #### Dernière minute — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2022) - Video: https://vimeo.com/725276687 - Interaction: Projection, Shared & Social, Perception & Effects - Platform & tech: Projection, eMotion, floor projection, immersive sound - Idea: A ritual of passage in which the floor you lie on turns into rising water. - What it is: A 30-minute immersive installation-experience about thresholds, rooted in scattering a father's ashes by the sea: visitors sit and lie on a floor of projected water and particles that ripple, rise and dissolve around them to music by Olivier Mellano. - Technique: Floor and wall projections of filmed and simulated water are choreographed with a composed soundtrack into a fixed 30-minute sequence, likely with parts that respond to visitors' presence. - Try it: Design a five-minute projected ritual for one threshold moment (leaving home, a goodbye): choose one material (water, ash, light), project it on a floor where people lie down, and script how it changes with the music. Twist: let the material react to how still the group stays, measured by a top-down camera. #### Just your shadow — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2022) - Video: https://www.youtube.com/watch?v=iWfpUCYw3Kg - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, eMotion, real-time projection - Idea: A dancer dances with a shadow that does not quite obey. - What it is: A short choreographic piece co-created with choreographer Jann Gallois (Cie BurnOut) for ARTE's immersive night at the 2022 Festival d'Avignon, in the Célestins church: a body negotiates with its own shadow, which extends into luminous matter around the dancers. - Technique: Live projection around the dancer; the shadow-like light matter is likely generated in real time with eMotion and played by an operator in sync with the choreography. - Try it: Use a webcam silhouette (MediaPipe Selfie Segmentation) to project a student's shadow on a wall with a one-to-two-second delay and a small offset, then choreograph 60 seconds in which the dancer leads, follows and argues with it. Twist: let the shadow sometimes refuse to move, and design how the dancer reacts. #### Homme Plissé Issey Miyake show, Palais de Tokyo — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2023) - Video: https://vimeo.com/793397205 - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, eMotion, large-scale projection - Idea: A runway where the space itself pleats like the clothes. - What it is: Art direction for the Homme Plissé Issey Miyake show at Paris Fashion Week: a vast fluid membrane of particles appears on the walls and floor around the runway, pulsing, rippling and folding around dancing models. - Technique: Large-scale projection of a real-time particle membrane, likely built with eMotion, blurs the perception of space and gravity and is timed to the models' choreography. - Try it: Pick one fabric behavior (pleating, crumpling, fluttering) and build a projected particle cloth in TouchDesigner or p5.js that performs it on a wall; have a classmate walk past like a model and make the cloth fold as they pass. Twist: project onto the garment itself instead of the wall. #### Piano piano — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2023) - Video: https://vimeo.com/893723919 - Interaction: Voice & Sound, Performance, Projection - Platform & tech: Projection, real-time projection, audio-reactive visuals - Idea: Music you can see: piano notes and juggled balls turn into brushstrokes in space. - What it is: A musical and visual duet: babx plays the piano while Adrien Mondot juggles and plays live images, and the piano and both performers are immersed in a space of moving projected marks. - Technique: Projection surrounds the stage and is generated live by Mondot's own software, likely reacting to the piano's sound and to Mondot's juggling and hand gestures. - Try it: Connect a MIDI keyboard (or microphone pitch detection) to a p5.js sketch so each note leaves a brushstroke whose height follows its pitch, project it behind the player, and perform a two-minute piece. Twist: add a second performer who throws a ball and catches strokes with it through webcam tracking. #### En amour — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2024) - Video: https://vimeo.com/963236356 - Interaction: Projection, Shared & Social, Voice & Sound - Platform & tech: Projection, projection, presence sensing, spatial sound - Idea: A walk-through ritual that lets a group feel a love story as changing light and sound. - What it is: An immersive, interactive installation-experience commissioned by the Musée de la musique – Philharmonie de Paris, with music by Laurent Bardainne and selected for the Cannes 2024 immersive competition: visitors move through projected light, colors, words and sounds that trace a symbolic metamorphosis around love and separation. - Technique: Projection on floors, walls and tulle is combined with visitor sensing and a composed soundtrack; the balance between interactive and scripted moments likely changes from scene to scene. - Try it: Storyboard a three-station walk-through (meeting, closeness, parting) and give each station one projected element and one sound that respond to how many people a webcam counts there. Twist: make the final station react only to pairs standing close together. #### Encyclies — Adrien M & Claire B (Adrien Mondot & Claire Bardainne) (2024) - Video: https://vimeo.com/1057582668 - Interaction: Voice & Sound, Performance, Projection - Platform & tech: Projection, real-time projection, live visuals - Idea: The image plays next to the pianist as a second instrument. - What it is: A light musical and visual duet with pianist Nathalie Morazin, the second piece of Mondot's Synesthésies cycle, in which images animated live act as a full partner to the piano, between abstraction and impressionism. - Technique: A compact touring setup projects images that Mondot animates live with custom software, likely steering particle and brush systems by hand in time with the score. - Try it: Choose a short piano piece and write an image score: three visual behaviors mapped to three controller knobs, then play the images live next to a recording until the pairing feels like a duet. Twist: replace the recording with a live musician who cannot see the screen. ### Stijn Spanhove *XR developer at In The Pocket; Snap Spectacles creator* Belgian XR developer who builds rapid AR-glasses proofs of concept for Snap Spectacles and Meta AI glasses, often pairing them with generative AI. #### AI-driven virtual try-on — Stijn Spanhove (2024) - Video: https://x.com/stspanho/status/1869053825082134912 - Interaction: Hands & Body, Face - Platform & tech: Phone, Web, Mattercraft, Zappar, generative AI - Idea: The mirror becomes a fitting room that can dress you in anything. - What it is: A browser-based try-on concept dresses the user in a chosen garment using generative AI, with default outfits and a simple shop flow. - Technique: A photo of the user and a garment image are sent to a virtual try-on diffusion model; the result is shown inside a WebAR/Mattercraft interface. - Try it: Build a webcam page that captures a selfie and calls a hosted try-on model with three garment images, then shows the results side by side. Twist: try on clothes from a painting in a museum. #### Prompt-your-own AR drone show — Stijn Spanhove (2024) - Video: https://x.com/stspanho/status/1843725659585294651 - Interaction: Location & City, Performance - Platform & tech: Phone, Flux LoRA, Luma Dream Machine, AR - Idea: Anyone can direct a drone show by describing it. - What it is: Users type a prompt to design a drone formation; an image model draws it as a drone-light figure, a video model animates it, and it plays in the night sky in AR. - Technique: A Flux LoRA trained on drone-show imagery generates the formation, Luma's Dream Machine animates it into a clip, and the clip is placed as a large billboard in the AR sky. - Try it: Generate a 'drone light' image from a prompt, turn the bright pixels into 200 glowing points, and animate them rising into formation above a real building in phone AR. Twist: the formation is chosen by the audience's votes. #### WebAR capture enhanced with GenAI video-to-video — Stijn Spanhove (2024) - Video: https://x.com/stspanho/status/1845952462517424401 - Interaction: Perception & Effects - Platform & tech: Phone, Web, WebAR, Runway Gen-3 Alpha Turbo - Idea: Use AR as a rough sketch and let AI finish the rendering. - What it is: Rough AR content captured in the browser is passed through Runway's video-to-video model so the virtual objects look far more photoreal in the final clip. - Technique: A screen recording of WebAR is used as structure for a video-to-video diffusion model, which keeps the motion and layout but re-renders lighting and materials. - Try it: Record a 5-second AR clip with a grey placeholder object, then run it through any video-to-video AI tool with a prompt describing the object. Twist: keep the same clip and try three prompts that change the object's story. #### AI Teleport — Stijn Spanhove (2025) - Video: https://www.youtube.com/watch?v=GvBRlCcgJ_g - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Headset, Snap Spectacles, Google Gemini - Idea: Generative AI 'teleports' your surroundings into another world in real time. - What it is: A portal lens for Spectacles that uses Google Gemini to generatively restyle the user's surroundings in real time. - Technique: Camera frames are sent to a multimodal image model that restyles the scene, and the result is displayed back as a portal overlay aligned to the view. - Try it: Photograph the scene in front of you with a phone, send it to an image-editing model to change its style, then paste it back in place as an AR picture frame. Twist: let a classmate choose the target style with a single word. #### Ad Block — Stijn Spanhove (2025) - Video: https://www.youtube.com/watch?v=KLiEm74cw9Q - Interaction: Portals & Worlds, Perception & Effects, Information & UI - Platform & tech: Headset, Snap Spectacles, Google Gemini - Idea: A real-world ad blocker that finds the billboards in front of you and covers them up. - What it is: An experimental Spectacles app that works as a real-world ad blocker: Gemini detects billboards and ads in view and covers them. - Technique: A vision-language model detects ads and returns bounding boxes, which are projected into 3D and covered with opaque or replacement quads in the wearer's view. - Try it: Use object detection to find logos or posters in the frame and cover them in AR with a solid color block or your own artwork. Twist: instead of hiding the ad, rewrite its text into one honest sentence. #### Bringing an action figure to life in AR — Stijn Spanhove (2025) - Video: https://x.com/stspanho/status/1910708661279773101 - Interaction: Tangible Objects, Play - Platform & tech: Phone, Web, OpenAI image generation, Tripo AI, 8th Wall - Idea: Photograph a toy and it steps out of its box as a living AR character. - What it is: A photo of a toy action figure is turned into a rigged 3D character that walks around in WebAR next to the real toy. - Technique: An image model cleans up the figure into a character sheet, Tripo AI converts it into a textured, rigged 3D model, and 8th Wall places and animates it in the browser. - Try it: Photograph a small object, turn it into a 3D model with a free image-to-3D service, auto-rig it and place it on a table with WebAR or Reality Composer. Twist: make the AR copy mirror whatever pose you put the real toy in. #### Chain-of-thought models designing XR puzzles — Stijn Spanhove (2025) - Video: https://x.com/stspanho/status/1885352446806970708 - Interaction: Play - Platform & tech: Web, Headset, DeepSeek, three.js, WebXR - Idea: Let a reasoning model design the level, then step inside it. - What it is: A reasoning model is asked to invent small puzzle games, and its generated three.js code runs directly as an XR puzzle. - Technique: A chain-of-thought LLM (DeepSeek) is prompted for a puzzle rule set and outputs three.js scene code, which is loaded into a WebXR page. - Try it: Ask an LLM for a one-page WebXR puzzle (for example 'arrange four cubes so their shadows spell a letter'), paste the code into a template and test it on a phone. Twist: the class must solve it without knowing the rule, then guess the prompt. #### Frame-a-scene spatial pixel art (first Spectacles lens) — Stijn Spanhove (2025) - Video: https://x.com/stspanho/status/1894776068600447319 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Headset, Snap Spectacles, Lens Studio, generative AI - Idea: Your fingers are the camera; AI turns what you frame into pixel art in place. - What it is: Making a frame with your fingers captures part of the view, and the lens generates pixel art from it that sits back in place, blending into the surroundings. - Technique: A hand-tracking gesture defines a crop of the camera frame, which is sent to an image model with a pixel-art prompt, and the result is anchored as a quad at the framed location. - Try it: In Lens Studio or a phone AR app, let users drag a rectangle on the camera view, send the crop to an image API with a style prompt, and pin the result where it was taken. Twist: each classmate chooses a different art style, and the room becomes a collage. #### From 2D video to 4D Gaussian splats on Vision Pro — Stijn Spanhove (2025) - Video: https://x.com/stspanho/status/2004310347486761051 - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Headset, Apple Vision Pro, Apple SHARP, PlayCanvas, Gaussian splatting - Idea: Any flat video can become a scene you walk around, one generated frame at a time. - What it is: Frames of the short film Big Buck Bunny are each turned into a Gaussian splat with Apple's SHARP model, compressed, and played back as a rough volumetric animation in Vision Pro. - Technique: A single-image-to-3D model (SHARP) predicts a splat per frame; the splats are compressed with PlayCanvas tools and swapped every frame to fake a 4D sequence. - Try it: Take 24 frames from a short clip, turn each into a 3D splat with a free image-to-3D tool, and flip through them in a WebXR viewer on a table. Twist: freeze time when the viewer walks behind the scene. #### Fruit Defense — Stijn Spanhove, Pavlo Tkachenko (2025) - Video: https://www.youtube.com/watch?v=KlC__Y0J-Wc - Interaction: Tangible Objects, Shared & Social, Play - Platform & tech: Headset, Snap Spectacles, Lens Studio - Idea: The glasses recognize real objects around you and make them part of a tower-defense game. - What it is: A single- and multiplayer Spectacles game in which real-world objects detected around you become active gameplay elements to defend. - Technique: An on-device object detection model finds real objects in view, and their estimated 3D positions become gameplay entities that enemies target. - Try it: Use object recognition in Lens Studio or on a phone to find a cup on the table, make it the 'base' you must protect, and send small AR monsters swarming in from all sides. Twist: switch to a different object and the enemy type changes with it. #### Hands-free Gemini assistant with Vision Pro camera access — Stijn Spanhove (2025) - Video: https://x.com/stspanho/status/1875989390344032340 - Interaction: Voice & Sound, Information & UI - Platform & tech: Headset, Apple Vision Pro, Enterprise camera access, Gemini 2.0 Flash - Idea: An assistant that shares your point of view instead of waiting for a photo. - What it is: Using Vision Pro camera access, Gemini 2.0 Flash sees what the wearer sees and answers spoken questions in real time without any hands. - Technique: Main-camera frames and microphone audio are streamed to the Gemini Live (multimodal realtime) API, and responses are spoken and shown as floating text. - Try it: Use a phone in a headset viewer with a free multimodal live API to build 'ask what you see'; test it on a bookshelf or a fridge. Twist: give the assistant a persona that only speaks in questions. #### Ray-Ban Meta safety assistant with a cloud VLM — Stijn Spanhove (2025) - Video: https://x.com/stspanho/status/1996894608597700645 - Interaction: Information & UI, Voice & Sound - Platform & tech: Wearable, Meta Wearables Device Access Toolkit, VLM, Ray-Ban Meta - Idea: Glasses that quietly watch for danger and tap you when they see it. - What it is: Ray-Ban Meta glasses stream video to a vision-language model in the cloud that checks for safety risks, and the glasses speak back and vibrate when something is wrong. - Technique: Meta's Wearables SDK exposes the camera stream to a phone app, which sends frames to a VLM with a safety prompt and returns spoken and haptic alerts. - Try it: Make a phone web app that sends a camera frame every two seconds to a vision model asking 'is anything unsafe here?' and speaks the answer. Twist: tune it for a specific place, like a kitchen or a workshop. #### Real-time AI video restyling an XR room — Stijn Spanhove (2025) - Video: https://x.com/stspanho/status/1980751025930661934 - Interaction: Perception & Effects, Voice & Sound, Spatial Mapping - Platform & tech: Headset, real-time video diffusion, XR - Idea: Type a style and your room is redecorated around you, live. - What it is: A room seen in XR is repainted by a live video model: first as a 'modern, Ikea style room', then as an 'industrial loft', with updates synced slowly to avoid dizziness. - Technique: The headset view is streamed to a real-time video-to-video diffusion model driven by a text prompt, and the restyled frames are composited back in XR at a reduced update rate to limit motion sickness. - Try it: Capture a phone camera feed, send it to a hosted real-time diffusion API with two style prompts, and show the result full-screen in a cardboard or HoloKit viewer. Twist: let the room slowly drift from one style to another as you walk. #### Replaying a broadcast basketball play on a real court in XR — Stijn Spanhove (2025) - Video: https://x.com/stspanho/status/1992980633652478223 - Interaction: Location & City, Play, Information & UI - Platform & tech: Headset, pose estimation, player tracking, XR - Idea: Watch last night's play happen again on the court you are standing on. - What it is: From a single 2D TV broadcast, the court and players are detected and tracked, and the play is reconstructed in 3D and mapped onto a real basketball court for viewing in XR. - Technique: Court-line (pitch) detection gives a homography from the broadcast to the court plane, player detection and tracking give 2D paths, and these are lifted to 3D avatars placed on the real court. - Try it: Download a short sports clip, track the players with a free detection model, map their feet to the floor plane using four court corners, and replay dots on a tabletop court in AR. Twist: let the viewer pause and move one player to test a different play. #### Starship rocket in XR — Stijn Spanhove (2025) - Video: https://www.youtube.com/watch?v=LOF3rx0zSgg - Interaction: Tangible Objects, Play - Platform & tech: Headset, Snap Spectacles, Lens Studio, 3D printing - Idea: Launch a Starship, then guide the booster back to a 3D-printed launch tower on the table. - What it is: A Spectacles experience that lets you launch a SpaceX Starship and guide the booster back to a 3D-printed launch tower on your table. - Technique: The printed tower is registered as a tracked object or placed anchor, and a simple physics or scripted trajectory lets the user steer the booster back to the physical catch point. - Try it: Use a real object (a paper cup) as the landing pad and steer a rocket in AR back into its mouth, making it explode if it drifts too far. Twist: add random crosswinds. #### Step inside an AI-generated movie scene — Stijn Spanhove (2025) - Video: https://x.com/stspanho/status/2005626245166313514 - Interaction: Portals & Worlds, Performance - Platform & tech: Headset, Apple Vision Pro, Gaussian splatting, generative video - Idea: A film scene you stand inside, rebuilt entirely by generative models. - What it is: A proof of concept that lets the viewer walk into the Dojo from The Matrix, where the environment, characters and movement were all generated by AI and replayed around them in XR. - Technique: Likely a generative video model produces the fight, which is then lifted into 3D (Gaussian splats or 4D reconstruction) and streamed to the headset as a volumetric sequence. - Try it: Generate a 5-second clip of two characters in a room, convert key frames into splats with a single-image-to-3D tool, and play them in sequence in WebXR at room scale. Twist: let the viewer's position choose which camera angle the AI generates next. #### Turning a child's drawing into a playable XR world — Stijn Spanhove (2025) - Video: https://x.com/stspanho/status/1961138591934947462 - Interaction: Drawing & Making, Play, Portals & Worlds - Platform & tech: Headset, Snap Spectacles, Mirage 2, world model - Idea: A paper drawing is the level design; an AI world model builds the game. - What it is: His niece's drawing is fed into the Mirage 2 world model and becomes a playable world that opens up in front of the viewer on Spectacles. - Technique: A photo of the drawing prompts an interactive world model (Mirage 2) that generates frames in response to controls; the stream is shown as a floating window or portal in the Spectacles lens. - Try it: Have students draw a level on paper, photograph it and generate a 3D environment with a free image-to-world or image-to-3D tool; place it on the table in AR as a tiny diorama. Twist: the child who drew it gives voice commands to change the weather. #### XR Worlds — Stijn Spanhove (2025) - Video: https://www.youtube.com/watch?v=h7ll6AB4rcc - Interaction: Drawing & Making, Portals & Worlds - Platform & tech: Headset, Snap Spectacles, Mirage 2, generative AI - Idea: Turn a 2D doodle on paper into a playable XR world. - What it is: A Spectacles proof of concept that turns a 2D drawing on paper into a playable XR world, using a generative world model. - Technique: A camera capture of a paper drawing is sent to a generative world model that returns a playable environment, which is then displayed in the glasses. - Try it: Photograph a hand-drawn level sketch, detect its lines as platforms, and generate an AR level a ball can roll through. Twist: use pens of different colors to stand for different physical materials. #### LEGO AR glasses proof of concept — Stijn Spanhove, Pavlo Tkachenko (2026) - Video: https://www.youtube.com/watch?v=7E2HXzrLLZE - Interaction: Tangible Objects, Play - Platform & tech: Headset, Snap Spectacles, LEGO Smart Bricks - Idea: AR characters and effects are layered onto LEGO smart bricks, so the physical bricks drive the virtual content. - What it is: A Spectacles proof of concept that adds AR effects and characters to LEGO Smart Bricks builds, reacting to the physical bricks in play. - Technique: Signals from the smart bricks (state, motion, identity) are read over a wireless link and mapped to AR effects anchored to the build's tracked position. - Try it: Drive an AR effect with a micro:bit or phone sensor data, so that shaking a real brick makes sparks burst out in AR. Twist: when two bricks come close, a short conversation appears. #### Meta Ray-Ban AI City Tour Guide — Stijn Spanhove (2026) - Video: https://www.youtube.com/watch?v=IA0LGSA4Lgw - Interaction: Location & City, Voice & Sound, Information & UI - Platform & tech: Wearable, Meta Wearables SDK, Ray-Ban Meta - Idea: An AI city guide in Ray-Ban Meta glasses describes whatever you are looking at as you walk. - What it is: A proof of concept built at In The Pocket: a location-aware AI tour guide on Meta Ray-Ban glasses that walks you through Ghent and describes what you look at. - Technique: GPS location and a captured camera frame are sent to a multimodal LLM, which returns a spoken description through the glasses, giving audio-first AR without a display. - Try it: Build a campus audio tour: send a photo plus GPS to a multimodal model and have it introduce the building in front of you in a 30-second voice clip. Twist: make the guide speak in the voice of a historical figure. #### Micro Art — Stijn Spanhove, Pavlo Tkachenko (2026) - Video: https://www.youtube.com/watch?v=esOODTn0RVU - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Phone, Headset, Lens Studio, generative AI - Idea: Point at an object in nature to reveal an imagined microscopic world inside it. - What it is: An AR lens that reveals an imagined microscopic world inside plants, rocks and other things you point it at in nature. - Technique: The lens segments or classifies the pointed-at natural object and anchors a generated microscopic scene to it, likely using a pre-generated asset per category. - Try it: Point the camera at a leaf or a stone and show an imagined microscopic world magnified on its surface in AR. Twist: give each material its own family of microscopic creatures. #### One-click custom object detector for Spectacles — Stijn Spanhove (2026) - Video: https://x.com/stspanho/status/2066207120199208985 - Interaction: Tangible Objects, Information & UI - Platform & tech: Headset, Desktop, Snap Spectacles, SnapML, SAM 3, YOLOv7-tiny, ONNX - Idea: Teaching glasses to recognise your own objects should take a click, not a weekend. - What it is: A pipeline turns raw footage into a trained object detector for Spectacles in one click: SAM 3 labels the objects, a small YOLO model trains in the cloud, and a SnapML-ready ONNX file drops into Lens Studio. - Technique: A segmentation foundation model (SAM 3) auto-labels frames, which train a YOLOv7-tiny detector exported to ONNX for on-device SnapML inference. - Try it: Film 30 seconds of one object, auto-label it with a free segmentation tool, train a tiny detector (Teachable Machine or YOLO in Colab) and trigger an AR label when it is seen. Twist: detect objects that should not be there and make them glow. #### Re-coaching a World Cup goal in AR — Stijn Spanhove (2026) - Video: https://www.youtube.com/watch?v=uPpCLfgWbdA - Interaction: Information & UI, Play - Platform & tech: Headset, AR, generative video - Idea: Pause a real World Cup goal, move a defender, and let AI re-render a different ending. - What it is: A real World Cup goal is rebuilt as a tabletop AR scene; you pause it, move a defender, and AI re-renders the alternate ending photorealistically. - Technique: Player positions from match tracking data are replayed as tabletop 3D figures, and after the user edits a position the altered layout conditions a generative video model to render a new ending. - Try it: Turn a goal from a match into a tabletop AR replay (keyframed simple figures and a ball) that can be paused so defenders can be dragged. Twist: after a change, recompute the ball's trajectory with simple physics. #### Standing inside your RAG embeddings — Stijn Spanhove (2026) - Video: https://x.com/stspanho/status/2024877056035622974 - Interaction: Information & UI, Voice & Sound - Platform & tech: Headset, Apple Vision Pro, embeddings, RAG, LLM - Idea: Walk around the vector space that powers a chatbot, instead of reading its answer. - What it is: A Vision Pro prototype places the user inside a cloud of course descriptions; semantically similar courses float next to each other, and a spoken question lights up the matches. - Technique: Text is embedded into high-dimensional vectors, reduced to 3D positions (likely with UMAP or t-SNE), and a voice query is embedded the same way so its nearest neighbours can be highlighted in space. - Try it: Embed 50 short texts (song lyrics, class notes) with a free embedding API, project them to 3D and place them as floating labels in a WebXR room; ask a question aloud and highlight the nearest five. Twist: let two people's questions pull the cloud in different directions. ### Shengzhi Wu *Staff product designer for AR and AI (Meta; formerly Google AR and Android XR)* Interaction designer and creative coder trained at Carnegie Mellon's School of Design and ETC. At Google he designed ARCore, AR in Search, Google Lens and Android XR features; his CMU thesis prototypes on multi-device hand gestures in AR reached over a million views on Twitter and became a CHI 2020 paper. #### Dinosaur Passage: a HoloLens museum experience — Shengzhi Wu (2017) - Video: https://www.youtube.com/watch?v=Z-6CveTUFYY - Interaction: Location & City, Information & UI, Spatial Mapping - Platform & tech: Headset, HoloLens, Unity - Idea: Bring extinct animals back at life size into the museum hall where their bones are shown. - What it is: A mixed-reality museum concept in which visitors wearing HoloLens see a life-size T. rex stride through the museum hall among virtual plants. - Technique: HoloLens spatial mapping anchors an animated dinosaur and vegetation to the hall's floor so they stay put as visitors walk around. - Try it: Place a life-size animal model in a school hallway with phone AR and design a 3-stop guided walk around it. Twist: it reacts when a visitor gets too close. #### AR dial on a physical puck — Shengzhi Wu (2018) - Video: https://www.youtube.com/watch?v=N8HpIkYmpmA - Interaction: Tangible Objects, Hands & Body, Information & UI - Platform & tech: Phone, Vuforia, Unity - Idea: Give a dumb physical object a digital function by tracking how you turn it. - What it is: Rotating a blank physical puck on the desk turns it into a virtual dial: an AR ring and angle readout follow the real rotation. - Technique: Vuforia tracks a marker on the puck; its yaw angle relative to the base drives a ring UI and a numeric value in Unity. - Try it: Stick an image marker on a bottle cap and use its rotation in phone AR to control music volume or a light. Twist: map rotation to time and scrub through a day in the room. #### ARCore procedural bridges — Shengzhi Wu (2018) - Video: https://x.com/Wu_Shengzhi/status/1059281909166104576 - Interaction: Spatial Mapping, Play - Platform & tech: Phone, ARCore, Unity - Idea: Let the real room's furniture define the level layout of an AR game. - What it is: An internship prototype for Google Daydream: game levels are generated from the real room, with bridges built automatically between the detected surfaces of furniture and floor. - Technique: ARCore plane detection returns surfaces at different heights; a generator connects nearby plane edges with procedurally built bridge meshes. - Try it: Write a small script that places a path between every pair of detected planes in AR Foundation. Twist: bridges collapse after the player crosses them once. #### Finger ray for far interaction — Shengzhi Wu (2018) - Video: https://x.com/Wu_Shengzhi/status/1069317433511591936 - Interaction: Hands & Body, Gaze & Attention - Platform & tech: Headset, Leap Motion, Oculus Rift, Unity - Idea: Extend bare-hand input beyond arm's reach with a pointing ray plus a pinch. - What it is: Instead of only touching what is within arm's reach, a ray shoots from the index finger so that simple pinch gestures can select and move posters hanging far away. - Technique: A raycast from the camera through the tracked index fingertip picks targets, and Leap Motion pinch state confirms selection and drag. - Try it: Using a webcam hand tracker (e.g. MediaPipe), build a pointer that selects items on a far wall projection when you pinch. Twist: the ray bends toward the nearest target like a magnet. #### Floating by shadow offset — Shengzhi Wu (2018) - Video: https://x.com/Wu_Shengzhi/status/1059284500159578113 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, ARCore, Unity - Idea: Shadows are the cheapest depth cue: move the shadow and the object seems to lift off. - What it is: A small study showing that a virtual object reads as floating above a real surface just by separating it from its shadow, an idea inspired by designer Dixon Lo. - Technique: A blob shadow stays on the detected plane while the object's height is animated, so the gap between them conveys elevation. - Try it: Put one virtual ball on a table in phone AR and test three versions: no shadow, attached shadow, offset shadow; ask classmates which floats. Twist: make the shadow move in the opposite direction of the ball. #### Floating islands between real surfaces — Shengzhi Wu (2018) - Video: https://x.com/Wu_Shengzhi/status/1061653324536168449 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, ARCore, Unity - Idea: Fill the empty space between real surfaces with generated content so AR fits the room. - What it is: Another Google prototype: low-poly floating islands, towers and trees are generated to fill the gaps between detected surfaces, so the fantasy scene seems to grow out of the room. - Technique: Multiple ARCore planes are used as anchors; islands are spawned at heights between them with procedural decoration. - Try it: Generate a floating object at the midpoint height between the floor and a table in AR Foundation. Twist: its species depends on the distance between the two surfaces. #### Hand-summoned UI panel — Shengzhi Wu (2018) - Video: https://x.com/Wu_Shengzhi/status/1072330175554170880 - Interaction: Hands & Body, Information & UI - Platform & tech: Headset, Leap Motion, Oculus Rift, Unity - Idea: UI lives in the world but comes to your hand only when you need it. - What it is: A floating control panel glides into arm's reach only when the hand reaches for it and drifts back into the world when done; he imagined it for controlling smart lights, with a slider inspired by Leap Motion's Cat Explorer. - Technique: A hand-proximity trigger lerps the panel between a world anchor and a hand-relative anchor, with Leap Motion pinch detection driving the slider. - Try it: Create a WebXR or phone-AR panel that stays on a wall but slides toward the camera when you walk closer than one meter. Twist: it retreats faster if you look away. #### Jumping across real surfaces — Shengzhi Wu (2018) - Video: https://x.com/Wu_Shengzhi/status/1058150946541395969 - Interaction: Play, Spatial Mapping - Platform & tech: Phone, ARCore, Unity - Idea: Real furniture becomes platforms in a platformer game. - What it is: A character controlled by an on-screen joystick jumps between several real surfaces detected by ARCore, turning a table and the floor into platforms. - Technique: ARCore planes get colliders so a physics-driven character controller can land on them. - Try it: Build a one-button jumping game on AR Foundation planes. Twist: the character can only land on surfaces of a certain color. #### Pull a bookmark to trigger AR — Shengzhi Wu (2018) - Video: https://x.com/Wu_Shengzhi/status/1062782838314020865 - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, Vuforia, Unity - Idea: Use a natural physical action (pulling, opening, sliding) as the switch for AR content. - What it is: Pulling a printed bookmark out of a book reveals an image target that makes AR information appear, using the book's own affordance as the trigger. - Technique: Vuforia image tracking only detects the target once it is exposed, so the physical reveal itself acts as the trigger. - Try it: Hide an image marker inside an envelope, drawer or book and make AR content appear only when it is revealed. Twist: content changes depending on how far it is pulled out. #### The Other Way — Shengzhi Wu (2018) - Video: https://www.youtube.com/watch?v=xcUKUKx6DDo - Interaction: Tangible Objects, Hands & Body, Play - Platform & tech: Headset, HTC Vive, Vive Tracker, Unity - Idea: Use a real bicycle as the controller so the body feels the journey. - What it is: An interactive VR story built in two weeks at CMU's Entertainment Technology Center: players sit on a real bicycle fitted with a Vive Tracker, so riding it carries them through the story; it was later shown at IEEE VR 2020. - Technique: A Vive Tracker on the bike mimics steering and motion in VR, so the physical prop and the virtual path stay in sync. - Try it: Turn a chair, bike or scooter into a controller by taping a phone to it and reading its motion in a WebXR scene. Twist: the story branches depending on which way you lean. #### AR eyedropper for smart lights — Shengzhi Wu (2019) - Video: https://x.com/Wu_Shengzhi/status/1113284624355135489 - Interaction: Hands & Body, Tangible Objects, Perception & Effects - Platform & tech: Headset, ZED Mini, Leap Motion, Oculus Rift, Philips Hue, Unity - Idea: Treat the whole visible world as a color palette you can sample and pour into connected devices. - What it is: A virtual eyedropper sucks a color from any real object with a pinch and squirts it into a smart lamp, which changes to that color, accompanied by the Deluminator sound from Harry Potter. - Technique: A ray from the fingertip hits the passthrough video texture to read the pixel color, and virtual colliders registered to the real lamps send that color to the smart bulbs over their API. - Try it: Make a phone app that samples the color at the screen center and sends it to a smart bulb or a web page background. Twist: mix two sampled colors before pouring. #### Desk weather diorama — Shengzhi Wu (2019) - Video: https://x.com/Wu_Shengzhi/status/1107392653510037505 - Interaction: Hands & Body, Information & UI - Platform & tech: Headset, ZED Mini, Leap Motion, Oculus Rift, Unity - Idea: Turn a data widget into a tiny physical-feeling landscape on your desk. - What it is: A small 3D mountain diorama showing Pittsburgh's temperature and a three-day forecast sits on his real desk; he flips through days with his bare hand in passthrough AR. - Technique: ZED Mini stereo passthrough on an Oculus Rift gives depth-correct AR, with Leap Motion hands pressing the day buttons on a Unity diorama. - Try it: Design a tabletop AR diorama that visualizes one live data feed (weather, bus times, air quality). Twist: the terrain itself must change shape with the data. #### Dissolving point-cloud face — Shengzhi Wu (2019) - Video: https://x.com/Wu_Shengzhi/status/1098847193283391488 - Interaction: Face, Perception & Effects - Platform & tech: Desktop, Unity, Visual Effect Graph, depth camera - Idea: Turn a depth scan of a person into living particles. - What it is: A face captured as a depth image is rebuilt from a million real-time particles that dissolve and drift away. - Technique: The depth image is sampled into a point cache and fed to Unity's GPU Visual Effect Graph, where noise fields dissolve the particles. - Try it: Use a phone's depth or a TrueDepth face capture to spawn particles on a face and blow them away with a voice volume trigger. Twist: the face reforms only when you stay silent. #### Pull a 3D model out of the monitor like a drawer — Shengzhi Wu (2019) - Video: https://x.com/Wu_Shengzhi/status/1155725029310521344 - Interaction: Hands & Body, Information & UI - Platform & tech: Headset, Desktop, ZED Mini, Leap Motion, Oculus Rift, Unity - Idea: A monitor is a drawer: pull to take the 3D model out into the world, push to put it back. - What it is: While modeling on a desktop editor, he reaches toward the monitor and slides the 3D model out as if opening a drawer, so it lands at full size on the real desk for review. - Technique: The monitor's position is calibrated in passthrough AR, the editor's current mesh is mirrored into Unity, and a hand-tracked pull along the screen normal drives the drawer motion (likely a constrained slider joint). - Try it: Prototype a phone-AR 'drawer' on a laptop: detect the screen with an image marker and let a swipe toward the camera slide a 3D object out onto the table. Twist: the object keeps the scale it had on screen until you let go. #### Pull a sticky note out of an iPad — Shengzhi Wu (2019) - Video: https://x.com/Wu_Shengzhi/status/1127773999876976640 - Interaction: Hands & Body, Drawing & Making, Information & UI - Platform & tech: Headset, Desktop, ZED Mini, Leap Motion, Oculus Rift, iPad, Unity - Idea: The tablet screen becomes a book you can pull pages out of, bridging 2D creation and 3D placement. - What it is: He sketches a sticky note on an iPad, then pinches its edge and pulls it out of the screen into the room, where it springs free like a note slipped out of a book; a six-year-old understood it at once. - Technique: Passthrough AR (ZED Mini on a Rift) plus Leap Motion hand tracking; the iPad's drawing is streamed to Unity and a spring joint animates the note as it leaves the registered screen plane. - Try it: Build a WebAR page where a doodle drawn on the phone screen can be flicked up to become a note pinned in the room. Twist: notes stick only to vertical surfaces and fall off horizontal ones. #### Scent Memo — Shengzhi Wu (2019) - Video: https://www.youtube.com/watch?v=msQrZAsQUd0 - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, image recognition, scent diffuser - Idea: Objects become keys that unlock memories through images and smell. - What it is: Pointing a phone at a keepsake shows photos and memories linked to it, while a connected scent diffuser next to it releases a matching smell. - Technique: Likely object or image recognition in a phone app that looks up linked media and signals a networked diffuser to release a stored scent. - Try it: Tag three personal objects with image markers and make each one open a photo memory in phone AR. Twist: add a sound or smell cue that plays before the image appears. #### Slurp in AR — Shengzhi Wu (2019) - Video: https://x.com/Wu_Shengzhi/status/1096214064873852928 - Interaction: Hands & Body, Information & UI - Platform & tech: Headset, Leap Motion, Oculus Rift, Unity - Idea: Move digital content between places with a single squeeze-and-release gesture, even when it is far away. - What it is: Inspired by the MIT Media Lab's 'Slurp' eyedropper, a hand-held virtual dropper sucks up a distant piece of content and squeezes it out somewhere else; he later revisited the idea in the DIS 2022 paper '"Slurp" Revisited'. - Technique: Leap Motion hand tracking drives a virtual dropper; a pinch while pointing at an object 'slurps' it into the bulb, and releasing over a target plays the reverse animation. - Try it: Build a two-step phone-AR interaction: tap to suck a virtual object into a cursor, walk elsewhere, tap to squirt it out. Twist: the dropper can only hold one thing, so picking up a second swaps them. #### Aquatic: an AR journey into the deep sea — Shengzhi Wu (2020) - Video: https://www.youtube.com/watch?v=GntDf6c7xsY - Interaction: Portals & Worlds, Spatial Mapping, Perception & Effects - Platform & tech: Headset, AR headset, Unity - Idea: Flood an ordinary room with the mysterious life of the deep sea. - What it is: Wearing an AR headset, a visitor sees glowing deep-sea creatures and sparks of bioluminescence bloom over the coffee table and drift through a sunny office lounge. - Technique: Likely a head-worn AR display anchoring particle-based creatures to a table surface, with VFX for bioluminescent trails. - Try it: Anchor one bioluminescent creature to a real object in phone AR and let it swim around that object. Twist: it only glows when the room lights are off. #### Google Search EDU AR — Shengzhi Wu (2020) - Video: https://x.com/Wu_Shengzhi/status/1259007666459774977 - Interaction: Information & UI, Spatial Mapping - Platform & tech: Phone, Web, ARCore, Scene Viewer, Google Search - Idea: Every search result can become an object you can walk around in your own room. - What it is: Searching for a topic such as the muscular or skeletal system on a phone lets students place a life-size 3D anatomy model in their room; he helped launch the content and designed the auto-scale feature used by AR animals and dinosaurs in Search. - Technique: Search results open Scene Viewer, which places a glTF model on an ARCore plane and auto-scales it to fit the available space. - Try it: Build a model-viewer web page that shows a school topic in AR at true scale. Twist: add a button that shrinks it to fit on the desk and label what changes. #### Mergereality: multi-device gestural interaction in AR — Shengzhi Wu (2020) - Video: https://www.youtube.com/watch?v=s-NUr2AZ1nA - Interaction: Hands & Body, Information & UI, Tangible Objects - Platform & tech: Headset, Desktop, Oculus Rift, ZED Mini, Leap Motion, Unity - Idea: Screens and devices become physical containers that your hands can reach into and pull content out of. - What it is: His CMU thesis and CHI 2020 paper: a set of passthrough-AR prototypes where bare hands move content between tablets, monitors, smart lights and the room, each borrowing a physical affordance such as a drawer, a book page or a dropper. - Technique: A VR headset with a ZED Mini stereo camera provides video passthrough, Leap Motion tracks the hands, and each screen's pose is registered in Unity so that pinches near its edge hand content over between 2D and 3D. - Try it: Pick one everyday metaphor (drawer, pocket, envelope) and prototype a phone-AR gesture that moves a photo from a laptop screen into the room using an image marker on the screen. Twist: make the reverse gesture put it back. #### AR athletes in Google Search — Shengzhi Wu (2021) - Video: https://x.com/Google/status/1394712843333816324 - Interaction: Hands & Body, Information & UI - Platform & tech: Phone, Web, ARCore, Scene Viewer, volumetric capture - Idea: Bring star athletes' movements into your living room at life size. - What it is: Announced at Google I/O 2021: searching for athletes such as Simone Biles, Naomi Osaka or Megan Rapinoe lets you watch a captured 3D performance of their signature moves in your room; he worked on it from the early concept. - Technique: Likely volumetric or motion-captured animations delivered as 3D assets through Search and placed with ARCore Scene Viewer. - Try it: Record a classmate doing one move with a phone and turn it into an animated 3D figure placed in AR. Twist: let viewers slow it down to study the form. #### Robot arm follows a toio car — Shengzhi Wu (2021) - Video: https://x.com/Wu_Shengzhi/status/1396120848126210049 - Interaction: Tangible Objects, Play - Platform & tech: Phone, Desktop, myCobot, toio, ROS, Arduino - Idea: Let robots and tiny toy cars share one coordinate system so AR effects can join the physical play. - What it is: A small myCobot robot arm always turns to face a moving toio toy car without any computer vision, and simple AR collision effects burst when the cars meet. - Technique: toio cars report their absolute position from the printed mat, which is sent over Wi-Fi to the arm's board so it computes the facing angle, while an AR layer uses the same positions for collision effects. - Try it: Use a toio, a robot vacuum or a phone on wheels as a tracked object and make a phone-AR effect follow it. Twist: the effect should predict where it will be in one second. #### Opportunistic Interfaces (Ad hoc UI) — Shengzhi Wu, Ruofei Du, Alex Olwal (2022) - Video: https://www.youtube.com/watch?v=r3TSYB41RMI - Interaction: Tangible Objects, Hands & Body, Information & UI - Platform & tech: Phone, Headset, 6DoF object tracking, Unity - Idea: Whatever is at hand can become the controller. - What it is: A CHI 2022 Interactivity with Ruofei Du, Alex Olwal and colleagues at Google: any everyday object picked up from the desk, such as a cup or a marker, instantly becomes a tangible 6DoF controller for AR content. - Technique: Likely real-time 6DoF pose estimation of arbitrary objects from the camera, mapping object translation and rotation to UI parameters on the fly. - Try it: Use a marker taped to any object and map its tilt and distance to two sliders in phone AR. Twist: let the user choose the mapping by shaking the object. #### RetroSphere — Shengzhi Wu (2023) - Video: https://www.youtube.com/watch?v=y0vKO8r2PEc - Interaction: Tangible Objects, Hands & Body - Platform & tech: Headset, retroreflective markers, IR cameras - Idea: Tracked AR input does not need electronics: shiny passive markers are enough. - What it is: An IMWUT/UbiComp 2023 system with Google researchers: passive, battery-free controllers covered in retroreflective spheres are tracked in 3D by a self-contained AR headset. - Technique: Near-infrared illumination makes the retroreflective spheres appear as bright blobs to the headset's cameras, and stereo triangulation recovers their 3D positions. - Try it: Tape retroreflective stickers to a stick and track the bright spots with a phone camera and flashlight in a browser. Twist: turn the stick into a light saber or paintbrush. ### Microsoft Research — Hrvoje Benko & Andy Wilson *Principal researchers at Microsoft Research Redmond (Benko later at Meta Reality Labs Research)* Andy Wilson (PlayAnywhere, TouchLight, Surface) and Hrvoje Benko led a decade of projector-and-depth-camera spatial AR at Microsoft Research, with interns such as Brett Jones, Rajinder Sodhi and Tomislav Pejsa. Their rooms turn every wall, table and body into an interactive display. #### PlayAnywhere — Microsoft Research — Hrvoje Benko & Andy Wilson (2005) - Video: https://www.youtube.com/watch?v=o8DBRgxpZYA - Interaction: Projection, Hands & Body, Tangible Objects - Platform & tech: Projection, projector, IR camera, computer vision - Idea: A projector and camera on a small cart turn any table into a touch-enabled game table. - What it is: Andy Wilson's compact front-projected tabletop: a projector and camera on a small cart turn any table into a touch and object-sensing surface for games, maps and photo sharing. - Technique: A short-throw projector and an IR camera with IR illumination sense finger touches by analyzing shadows and contact, and use visual codes and optical flow to detect objects and paper on an ordinary table. - Try it: Mount a small projector and a phone camera on a stand looking down at a table, judge a 'touch' from the distance between a finger and its shadow, and build a projected jigsaw puzzle or draggable map. Twist: make real objects on the table (cups, keys) act as obstacles in the game. #### LightSpace — Microsoft Research — Hrvoje Benko & Andy Wilson (2010) - Video: https://www.youtube.com/watch?v=xx5kBqxyaHE - Interaction: Projection, Hands & Body, Spatial Mapping - Platform & tech: Projection, depth cameras, projectors - Idea: Scoop a projected photo off the table, carry it in your hand, and pour it onto the wall. - What it is: Depth cameras and projectors make a whole room interactive: pick up a projected photo from a table, carry it on your hand and drop it onto a wall. - Technique: Several calibrated depth cameras and projectors share one room coordinate system, and virtual 'depth camera' projections of the point cloud detect when a hand touches a surface or holds a projected item, so content can move between surfaces and the body. - Try it: With one projector and one depth camera (or MediaPipe palm detection), let people 'pick up' a photo projected on a table, have it follow the palm, and 'drop' it onto the wall. Twist: make the photo change while it is carried (flip over, age, or turn into sound). #### Beamatron — Microsoft Research — Hrvoje Benko & Andy Wilson (2012) - Video: https://www.youtube.com/watch?v=L9yccRm3Zu8 - Interaction: Projection, Spatial Mapping, Play - Platform & tech: Projection, steerable projector, Kinect - Idea: A head-turning projector robot lets a virtual RC car drive over the real furniture in a room. - What it is: A steerable projector and Kinect on a pan-tilt head follow people around a room, projecting a virtual car that drives over real furniture and floors. - Technique: A projector and Kinect sit on a motorized pan-tilt head; the room is pre-scanned into a 3D model, and imagery is pre-distorted per surface as the head steers so content stays correct while moving across furniture. - Try it: Put a small projector on a pan-tilt mount turned by hand or servo, switch the image in TouchDesigner based on the angle, and make a projected little car "drive" along the wall and floor. Twist: have the car chase a particular person in the room (detected by a camera). #### Holoflector — Microsoft Research — Hrvoje Benko & Andy Wilson (2012) - Video: https://www.youtube.com/watch?v=2Xv6FnM1SrE - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Kinect, half-silvered mirror - Idea: A magic mirror where your reflection can catch a virtual ball, with the image and the virtual objects lined up exactly. - What it is: A large half-silvered mirror with a display behind it and Kinect body tracking: virtual objects appear in your reflection and react to your real body. - Technique: A half-silvered mirror in front of an LCD combines the real reflection with rendered graphics, and Kinect skeleton tracking places virtual objects so they align with the user's reflected body. - Try it: Build a "magic mirror" from a tablet and an acrylic half-mirror (or a mirrored camera feed), and use MediaPipe Pose so a virtual ball lands on your reflected shoulder and can be brushed off. Twist: your reflection makes a different move from the one you make. #### MirageTable — Microsoft Research — Hrvoje Benko & Andy Wilson (2012) - Video: https://www.youtube.com/watch?v=ll2K4tPD47E - Interaction: Projection, Hands & Body, Shared & Social - Platform & tech: Projection, Kinect, stereo projector, shutter glasses - Idea: A curved desk with stereo projection lets you grab virtual objects by hand and sit "across the table" from someone far away. - What it is: A curved projected tabletop with stereo glasses and Kinect: users grab virtual 3D objects with bare hands and meet a remote partner rendered as if sitting across the table. - Technique: A Kinect captures the user and the remote partner as depth meshes, head-tracked stereo projection onto a curved screen renders view-dependent 3D, and a physics engine lets hand geometry push virtual objects. - Try it: Build a parallax tabletop in three.js driven by face tracking (MediaPipe Face): a camera tracks your head, and 3D objects on the screen or projection shift with your viewpoint to create an "off-screen" illusion. Twist: put the other person's video from another computer into this 3D space and make a small "sitting across the table" game. #### IllumiRoom — Microsoft Research — Hrvoje Benko & Andy Wilson (2013) - Video: https://www.youtube.com/watch?v=L2w-XqW7bF4 - Interaction: Projection, Play, Perception & Effects - Platform & tech: Projection, Kinect, projector, Xbox - Idea: The game bursts out of the TV: snow drifts down the living-room walls and the whole room shakes when things explode. - What it is: A projector around the TV extends Xbox games onto the living-room walls and furniture: snow falls in the room, the room shakes on explosions, the field of view grows beyond the screen. - Technique: A Kinect scans the geometry around the TV, the projector is calibrated to it, and game-driven effects (peripheral extension, edge highlighting, radial distortion) are rendered with depth-aware projection mapping around the screen. - Try it: Aim a projector at the wall around a monitor, grab the colors at the screen's edges in real time in TouchDesigner and extend them outward, and shake the projected image whenever an explosion happens on screen. Twist: make the room effects run against the story, for example the more intense the game, the quieter the room. #### Mano-a-Mano (Dyadic Projected SAR) — Microsoft Research — Hrvoje Benko & Andy Wilson (2014) - Video: https://www.youtube.com/watch?v=Df7fZAYVAIE - Interaction: Projection, Shared & Social, Play - Platform & tech: Projection, projectors, Kinect, view-dependent rendering - Idea: Two people face each other, both see correct 3D perspective in the projected space, and throw fireballs at each other. - What it is: Two people face each other in a projected room and each sees correct perspective 3D objects between them: throwing fireballs, passing a ball, playing catch with virtual objects. - Technique: Kinect tracks both users' heads, and each projected object is rendered with view-dependent perspective for one user so that, on the surfaces behind the opposite person, it appears to float in 3D between them. - Try it: Use MediaPipe Face to track one classmate's head position and render a ball "floating in mid-air" on the projection wall from their viewpoint, while another person "throws" it over with a gesture. Twist: the two take turns getting the correct viewpoint, then discuss what a third bystander saw. #### RoomAlive — Microsoft Research — Hrvoje Benko & Andy Wilson (2014) - Video: https://www.youtube.com/watch?v=GYkRRbP7m8s - Interaction: Projection, Spatial Mapping, Play - Platform & tech: Projection, Kinect, projectors, RoomAlive Toolkit - Idea: Every wall and piece of furniture in the living room becomes a game board, with monsters crawling out of the couch. - What it is: Multiple auto-calibrating projector-camera units cover an entire living room so games can happen on every surface: whack-a-mole creatures crawl out of the couch, and you dodge traps with your body. - Technique: Each projector-camera unit auto-calibrates by projecting Gray-code structured light that its Kinect observes, yielding a unified 3D room model onto which games are rendered with per-surface projection mapping and depth-based body tracking. - Try it: Calibrate a projector and camera with Gray-code structured light (or manual four-corner mapping) and turn a corner of the room into whack-a-mole: little monsters pop out from the edge of the couch, and MediaPipe detects where your hand slaps them. Twist: make the monsters dodge you instead of waiting to be hit. #### RoomAlive: The Other Resident — Microsoft Research — Hrvoje Benko & Andy Wilson (2014) - Video: https://www.youtube.com/watch?v=NXxVXQYlSXc - Interaction: Projection, Play, Portals & Worlds - Platform & tech: Projection, RoomAlive Toolkit - Idea: Use room-scale projection to stage a haunting in your own living room. - What it is: A spooky narrative game built with the RoomAlive projection system, where a ghostly presence haunts the real room around the player. - Technique: Built on the RoomAlive Toolkit's calibrated projector-camera units, the game uses the scanned room model to place projected effects (shadows, apparitions, stains) on specific furniture, triggered by the player's tracked position. - Try it: Use one projector to create a 3-minute horror story in a corner of a dorm or classroom: when a camera detects the viewer reaching a certain spot, a shadow appears on the wall and a stain seeps from the door frame. Twist: the "ghost" only moves when the viewer has their back to it. #### FoveAR — Microsoft Research — Hrvoje Benko & Andy Wilson (2015) - Video: https://www.youtube.com/watch?v=A5M6xk1Itgo - Interaction: Projection, Perception & Effects, Play - Platform & tech: Headset, Projection, optical see-through HMD, projectors - Idea: The glasses handle the center of your view and projection fills the edges, stitching together a very wide AR field of view. - What it is: Combines an optical see-through headset for the central field of view with room projection for the periphery, giving AR a very wide field of view. - Technique: An optical see-through HMD renders the central view while calibrated room projectors render the same scene in the periphery, with head tracking keeping both views registered. - Try it: Have one student wear HoloKit to look at a virtual object in the center while a projector renders the outer part of the same scene on the surrounding walls (rain, a starry sky), and compare how immersive it feels with and without the projection. Twist: deliberately make the peripheral and central imagery tell different stories. #### Room2Room — Microsoft Research — Hrvoje Benko & Andy Wilson (2016) - Video: https://www.youtube.com/watch?v=2o6krhxpUGk - Interaction: Shared & Social, Projection - Platform & tech: Projection, Kinect, projectors - Idea: Project a faraway person at life size onto your sofa, with no headset required. - What it is: Life-size telepresence: a remote person is captured in 3D and projected into your room, sitting on your real couch at the correct scale. - Technique: A Kinect captures the remote person as a colored depth mesh, and projector-camera calibration plus the local user's head tracking renders them life-size with perspective correction onto a real piece of furniture. - Try it: Cut out a remote classmate from their video with MediaPipe Selfie Segmentation, project them at life size sitting on a real local chair, and talk over a video call. Twist: the remote person appears only when you look at the chair. #### SnapToReality — Microsoft Research — Hrvoje Benko & Andy Wilson (2016) - Video: https://www.youtube.com/watch?v=phImnaIVQOQ - Interaction: Spatial Mapping, Drawing & Making - Platform & tech: Headset, depth sensing, HoloLens-style AR - Idea: Holograms snap onto real table edges and walls, like the snap-to-align guides in slide software. - What it is: Virtual objects in AR automatically snap to real-world edges and planes, making precise alignment of holograms with furniture effortless. - Technique: Planes and dominant edges are extracted from the depth data in real time, and virtual objects being manipulated are snapped to the nearest aligned constraint when within a distance and angle threshold. - Try it: In AR Foundation, make a virtual picture frame snap to the nearest detected plane and plane edge while being dragged, when it is within 5 cm. Twist: design an "anti-snap" so some objects always float away from real surfaces. ### Daito Manabe / Rhizomatiks *Artist, programmer and DJ; co-founder of Rhizomatiks* Daito Manabe and Rhizomatiks (with Motoi Ishibashi and choreographer MIKIKO / ELEVENPLAY) build live shows that mix drones, projection, motion capture and AR broadcast, including Perfume's stage shows and the Rio 2016 Tokyo handover. #### particles — Daito Manabe / Rhizomatiks (2011) - Video: https://www.youtube.com/watch?v=OcDec4RRavA - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, LEDs, wireless control, rail, position tracking - Idea: Timing lights on moving objects can draw volumes that do not exist — a physical version of persistence-of-vision effects that AR can extend with tracked objects. - What it is: Balls with built-in LEDs roll along a large spiral rail and blink at precisely timed moments, so their light points draw floating shapes and afterimages in the dark. - Technique: The position of each ball on the rail is tracked or predicted, and wireless control fires its LED when it passes a point that belongs to the target 3D shape. - Try it: Attach a virtual light to a real rolling ball or toy train tracked in AR and switch it on only when it crosses the surface of an invisible 3D shape, leaving a short trail. Twist: let the audience pick the shape with a voice command. #### Nosaj Thing 'Eclipse/Blue' music video — Daito Manabe / Rhizomatiks (2013) - Video: https://www.youtube.com/watch?v=_woNBiIyOKI - Interaction: Projection, Hands & Body, Performance - Platform & tech: Projection, motion capture, projection mapping, openFrameworks - Idea: Motion-captured projections reshape themselves around the dancer's body in real time. - What it is: A dancer performs inside real-time projected graphics driven by motion capture, so the visuals wrap her body and the space live. - Technique: Motion capture of the dancer drives real-time openFrameworks graphics that are projected onto the stage and her body, with the projector calibrated to the capture volume. - Try it: Track a person with Kinect or MediaPipe Pose and project lines and particles onto the wall behind them that deform with their movement. Twist: the graphics only appear when the person is still, and dissolve faster the faster they move. #### Perfume at Cannes Lions — Daito Manabe / Rhizomatiks (2013) - Video: https://www.youtube.com/watch?v=UbLLIhCvTQ8 - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, motion capture, projection mapping, 3D scanning - Idea: A pop group pushes translucent screens around the stage while live projection merges with their moves. - What it is: Perfume performs with semi-translucent screens they move on stage; motion capture and a 3D-scanned venue let projections map onto the screens and dancers in real time, blending stage and virtual space. - Technique: Optical motion capture tracks the moving screens and dancers, and a 3D scan of the venue defines a virtual stage in which projector renders are updated per frame to stay mapped onto moving surfaces. - Try it: Put an ArUco or AR marker on a handheld whiteboard, track its position with a camera so the projector always casts the image onto the moving board, and choreograph a one-minute dance. Twist: when two boards come close, the image "flows" from one to the other. #### Seamless MR (hanacam test) — Daito Manabe / Rhizomatiks (2014) - Video: https://www.youtube.com/watch?v=FPMkIWUTVXo - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, camera tracking, real-time compositing - Idea: An MR test where a tracked camera blends CG seamlessly into a live scene. - What it is: A test of seamless mixed-reality video where a tracked camera composites CG into a live scene in real time. - Technique: A tracked camera (encoder or optical tracking) supplies real-time pose to a render engine whose virtual camera matches it, so CG is keyed and composited into the live feed without drift. - Try it: Use the phone's ARKit pose as a virtual camera in Unity to composite a CG object into live-action video, and compare how real it looks with and without shadows and occlusion. Twist: have the CG object only "peek" in from the edge of the frame, creating a presence that is there one moment and gone the next. #### border (Rhizomatiks Research x ELEVENPLAY) — Daito Manabe / Rhizomatiks (2015) - Video: https://www.youtube.com/watch?v=DdYw3ZZ0XLQ - Interaction: Performance, Portals & Worlds, Perception & Effects - Platform & tech: Headset, HMD, robotic wheelchairs, motion capture - Idea: Ride an automated wheelchair in a headset while real and virtual dancers weave past you. - What it is: Audience members ride automated wheelchairs wearing headsets whose view mixes live dancers with virtual ones, so they cannot tell which bodies are real. - Technique: Motion capture tracks both the audience wheelchairs and real dancers in a shared coordinate system, so HMD-rendered virtual dancers and scenery are registered precisely to the physical stage. - Try it: Place a virtual dancer in the classroom with HoloKit or phone AR, seat the viewer on a wheeled chair and have classmates push them along a fixed route while real and virtual dancers take turns appearing. Twist: the viewer cannot tell when a real dancer has replaced a virtual one, and has to guess afterward. #### drone + AR test — Daito Manabe / Rhizomatiks (2015) - Video: https://www.youtube.com/watch?v=Wvr81vk2STU - Interaction: Perception & Effects, Performance - Platform & tech: Desktop, drones, motion capture, AR compositing - Idea: Overlay AR graphics on drones in flight, in real time. - What it is: Tests where flying drones are tracked and AR graphics are composited onto them in real time, previewing AR for live drone performances. - Technique: Optical motion capture (e.g. retroreflective markers on drones) gives each drone's 6DoF pose, which drives CG attached to it and composited into the tracked camera view. - Try it: Stick an image marker on a remote-control car or a handheld object and use AR image tracking to overlay wings, trails and other graphics on it. Twist: have the trail record its complete path, ending up as a "motion painting". #### Dynamic VR Display — Daito Manabe / Rhizomatiks (2016) - Video: https://www.youtube.com/watch?v=G7ZQ4KiX1JE - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Desktop, head tracking, motion capture - Idea: Tracking the viewer's head and redrawing the image in real time merges flat screens and real objects into one 3D space. - What it is: A head-tracked display setup in which 3D graphics are re-rendered from the viewer's eye position in real time, so flat screens combined with physical objects read as a continuous 3D space. - Technique: Head tracking feeds an off-axis (asymmetric frustum) projection so each display renders the scene from the viewer's eye position, making flat screens behave like windows into a consistent 3D space. - Try it: Estimate head position with MediaPipe Face Mesh and build an off-axis projection in three.js so a laptop screen becomes a "window box" with depth. Twist: put an object or text in the box that only comes together from one particular angle. #### Fencing Visualized — Daito Manabe / Rhizomatiks (2016) - Video: https://www.youtube.com/watch?v=h2DXCAWI8gU - Interaction: Information & UI, Performance - Platform & tech: Desktop, computer vision, AR broadcast - Idea: Blade-tip trails drawn live over the broadcast make hard-to-see fencing visible. - What it is: An AR sports-broadcast system that tracks the tips of fencing swords and draws their trajectories over live footage so audiences can follow the fast bouts. - Technique: Computer vision (likely multi-camera marker-less tracking with deep learning) localises the sword tips in each frame, and their 3D trajectories are rendered as trails over the broadcast camera image. - Try it: Record the trajectory of a waving colored pen with MediaPipe Hands or color tracking, and overlay a fading band of light on the video. Twist: pick a fast motion you know well (bouncing a ball, typing, flipping pages) and make a "visualized broadcast" of it. #### live AR system test — Daito Manabe / Rhizomatiks (2016) - Video: https://www.youtube.com/watch?v=B-RHS6Y4Onk - Interaction: Performance, Spatial Mapping - Platform & tech: Desktop, camera tracking, real-time rendering - Idea: A live AR system that locks virtual graphics to the real stage during a live broadcast. - What it is: A test of a live AR system for stage shows, with virtual graphics anchored to the physical stage as the broadcast camera moves. - Technique: A broadcast camera with tracking (encoders or optical) streams its pose and lens data to a real-time engine that renders stage-anchored graphics matched to the camera's view. - Try it: Use a phone as the "broadcast camera" and add fixed virtual lights and scenery to a small tabletop stage in Unity AR while another classmate performs on it. Twist: make the virtual scenery change with the performer's voice volume. #### Fencing tracking and visualization system — Daito Manabe / Rhizomatiks (2017) - Video: https://www.youtube.com/watch?v=3W1_5io-NYQ - Interaction: Information & UI, Performance - Platform & tech: Desktop, computer vision, deep learning, motion capture - Idea: Fencing blade tips too fast for the eye are drawn live as trails over the broadcast. - What it is: An AR broadcast system (with Dentsu Lab Tokyo) that detects fencing sword tips too fast for the eye and overlays their trajectories live; later versions are markerless using deep learning. - Technique: High-speed cameras detect the sword tips (first with markers, later with a trained deep-learning keypoint detector), triangulate their 3D positions, and render trajectory trails over the broadcast feed. - Try it: Film a badminton swing or someone wiping a whiteboard in slow motion on a phone, and use color tracking or MediaPipe hand landmarks to draw the fast trajectory over the video. Twist: treat the trajectories as calligraphy and compare whose strokes look best. #### discrete figures (with Kyle McDonald) — Daito Manabe / Rhizomatiks (2018) - Video: https://www.youtube.com/watch?v=hauXQQhwbgM - Interaction: Performance, Hands & Body, Projection - Platform & tech: Projection, machine learning, drones, motion capture - Idea: Machine-learning-generated virtual dancers share the stage with real ones. - What it is: A dance performance mixing dancers, drones, projection and machine-learning 'generated dancers' that appear alongside and on the performers. - Technique: Motion capture of dancers trains or drives ML-generated virtual dancers, which are rendered into projection mapped onto stage elements and drones whose positions are tracked in the same space. - Try it: Record a classmate's dance skeleton data and generate a virtual partner in the projection that 'copies' them and gradually drifts off, sharing the stage with the real dancer. Twist: each round, the virtual partner generates from the previous round's deviations, and see how far it strays from the human. #### Squarepusher 'Terminal Slam' music video — Daito Manabe / Rhizomatiks (2020) - Video: https://www.youtube.com/watch?v=GlhV-OKHecI - Interaction: Portals & Worlds, Location & City, Voice & Sound - Platform & tech: Headset, diminished reality, computer vision - Idea: Put on MR glasses and Shibuya's ads are erased and replaced with music visuals. - What it is: Through mixed-reality glasses, Shibuya's billboards and ads are erased and replaced by glitching visuals synced to the music (diminished reality). - Technique: Diminished reality: the camera feed is analysed to detect billboard regions, which are masked, inpainted and replaced with music-synced visuals, then re-rendered into the headset view. - Try it: Use the phone camera to detect and cover text or logos in the frame (with color segmentation or image tracking), and fill them with rhythm-driven glitch graphics. Twist: replace the covered ads with blank space or sky to experience a city without ads. ### Takashi Yoshinaga *AR/VR researcher & rapid prototyper (ISIT Fukuoka); Microsoft MVP for Mixed Reality* Researcher at ISIT Fukuoka with a PhD in AR-based remote healthcare, who has posted hundreds of AR prototypes since 2009 and runs the AR Fukuoka community; creator of HoloTuberKit. #### AR Tele-Echography (2009) — Takashi Yoshinaga (2009) - Video: https://www.youtube.com/watch?v=GY3UlelftKU - Interaction: Information & UI, Shared & Social, Tangible Objects - Platform & tech: Desktop, marker tracking, OpenCV - Idea: A remote doctor uses AR overlays to guide a local helper in handling an ultrasound probe. - What it is: A remote doctor guides a local operator's ultrasound probe in real time using AR overlays that show where and how to move it. - Technique: Fiducial markers on the probe and patient are tracked with OpenCV, and the remote doctor's target probe pose is streamed and rendered as an overlay guiding the local operator. - Try it: With two phones, have one person drag a virtual arrow on a remote screen while the other's AR view shows the arrow in sync, guiding them to move a real object on the table. Twist: use it to teach someone remotely how to fold a paper crane. #### HoloKit + Tango Demo — Takashi Yoshinaga, Botao 'Amber' Hu (2017) - Video: https://www.youtube.com/watch?v=sbLEsmwM1s4 - Interaction: Spatial Mapping - Platform & tech: Headset, Phone, HoloKit, Google Tango, Unity - Idea: Early spatially stable stereo AR from a cardboard headset and a Tango phone. - What it is: An early test of HoloKit's cardboard stereo optics driven by a Google Tango phone, placing stable holograms in the room. - Technique: Uses Tango's motion tracking for 6DoF pose and renders a side-by-side stereo image through HoloKit's mirror-and-Fresnel optics. - Try it: Change a phone AR scene to side-by-side stereo rendering and view a cube fixed on a table through a cardboard viewer. Twist: compare the sense of depth in the single-eye view and the stereo view. #### Holoportation for ARCore with Multiple Azure Kinect — Takashi Yoshinaga (2019) - Video: https://www.youtube.com/watch?v=4oJJN2eH6U0 - Interaction: Shared & Social, Hands & Body - Platform & tech: Phone, ARCore, Azure Kinect, Unity - Idea: Several Azure Kinects capture a person's point cloud live and beam it into phone AR as a hologram. - What it is: Several Azure Kinects capture a person as a live point cloud that is streamed into an ARCore phone as a life-size hologram. - Technique: Multiple calibrated Azure Kinect RGB-D streams are fused into a colored point cloud, streamed over the network, and rendered at life size at an ARCore anchor. - Try it: Record an RGB-D video of a person with the iPhone depth camera (Record3D or similar), convert it to a point cloud, and replay it life-size on the floor in AR. Twist: make the point cloud shake or scatter with the loudness of sound. #### Streaming Holograms to ARCore via YouTube — Takashi Yoshinaga (2019) - Video: https://www.youtube.com/watch?v=IeaHt6bBw1o - Interaction: Shared & Social, Perception & Effects - Platform & tech: Phone, ARCore, Azure Kinect, YouTube Live - Idea: Encode hologram point clouds into an ordinary YouTube live stream and decode them into AR holograms on a phone. - What it is: Encodes RGB-D point clouds into an ordinary YouTube live stream so anyone's phone can decode and view them as AR holograms (the basis of HoloTuberKit). - Technique: Depth and color are packed side by side into a regular video frame, streamed via YouTube Live, and a client shader decodes depth per pixel to rebuild a point cloud in AR. - Try it: Combine RGB and depth into a side-by-side video, use a shader in three.js to rebuild the depth as a point cloud, and place it in a WebXR scene. Twist: try different depth encodings and compare the noise after compression. #### HoloBox with HoloLens 2 — Takashi Yoshinaga (2020) - Video: https://www.youtube.com/watch?v=ZE_F2NsOSfQ - Interaction: Portals & Worlds, Tangible Objects - Platform & tech: Headset, Desktop, HoloLens 2, Unity - Idea: Virtual objects travel seamlessly between a 3D display box and the HoloLens space. - What it is: A physical box-shaped display and HoloLens 2 share synchronised objects that move seamlessly between the screen's 3D window and the air around it. - Technique: The box display renders an off-axis perspective matched to the HoloLens user's head pose, and a shared coordinate frame lets objects hand over seamlessly between screen rendering and headset rendering. - Try it: Use a monitor as an off-axis perspective window (tracking head position with a phone), and when an object leaves the edge of the screen, have it continue in an AR phone. Twist: objects that fall off the screen onto the desk bounce. #### LiDAR Point Cloud Streaming with Cartoon Effect — Takashi Yoshinaga (2020) - Video: https://www.youtube.com/watch?v=cKILsUOLkFU - Interaction: Perception & Effects, Shared & Social - Platform & tech: Phone, ARKit, LiDAR, Unity - Idea: Render a live point cloud as a cartoon-style remote space. - What it is: The iPad's live point cloud of a room and person is re-rendered on the receiving side as a hand-drawn cartoon. - Technique: The received point cloud is rendered with a non-photorealistic shader (edge detection from depth discontinuities plus flat color quantization) to look hand-drawn. - Try it: Apply color quantization and an outline shader to a point cloud or depth video to create a cartoon style. Twist: switch styles with the speaker's mood, judged by how loud they are. #### Real-Time LiDAR Point Cloud Streaming from iPad Pro — Takashi Yoshinaga (2020) - Video: https://www.youtube.com/watch?v=Y4erikakae0 - Interaction: Spatial Mapping, Shared & Social - Platform & tech: Phone, ARKit, LiDAR - Idea: Stream an iPad's live LiDAR point cloud to remote devices. - What it is: Streams the iPad Pro's live LiDAR point cloud to a remote PC and other XR devices, later with a cartoon-style rendering. - Technique: Depth-map pixels are unprojected to world-space points using camera intrinsics and pose each frame, then compressed and streamed over the network for remote rendering. - Try it: Take one point every N pixels from an iPhone depth map, unproject them into a point cloud, and send it over WebSocket to a three.js page on a computer for display. Twist: render the points on the receiving side in a different style, such as blocks or text. #### Room Scan with iPad LiDAR and AR Visualization — Takashi Yoshinaga (2020) - Video: https://www.youtube.com/watch?v=VBDFOMj5vD0 - Interaction: Spatial Mapping - Platform & tech: Phone, ARKit, LiDAR, Unity - Idea: Scan a room with iPad LiDAR and overlay the mesh back onto reality to view it. - What it is: A prototype room scanner that builds a coloured mesh of the room with the new iPad Pro LiDAR and overlays it back onto reality. - Technique: ARKit scene reconstruction produces a LiDAR mesh, vertex colors are sampled by projecting camera frames onto it, and the colored mesh is overlaid back on the room. - Try it: Generate a live room mesh with AR Foundation's ARMeshManager and overlay it on the real space with a wireframe material. Twist: color walls, floor and ceiling differently based on the mesh normal direction. #### Spatial Copy and Paste Shared with ARCore — Takashi Yoshinaga (2020) - Video: https://www.youtube.com/watch?v=hEeXlnFK_24 - Interaction: Spatial Mapping, Shared & Social, Portals & Worlds - Platform & tech: Headset, Phone, HoloLens 2, ARCore, Unity - Idea: Copy a piece of real space on HoloLens and paste it into someone else's phone AR. - What it is: Selects a region of the real room on HoloLens 2, 'copies' it as a mesh, and pastes the captured space into another user's ARCore view. - Technique: A bounding box selects part of the HoloLens 2 spatial mesh, the cropped mesh with colors is sent over the network, and the ARCore client places it at a chosen anchor. - Try it: Scan a corner of a desk with Polycam, export the model, and paste it onto another table in a second phone's AR. Twist: shrink it tenfold when pasting to make a miniature landscape. #### AR Light Saber with MediaPipe and p5.js — Takashi Yoshinaga (2021) - Video: https://www.youtube.com/watch?v=pL_q-nHelCU - Interaction: Hands & Body, Play - Platform & tech: Web, MediaPipe, p5.js - Idea: Gesture tracking in a web page grows a lightsaber out of your hand. - What it is: A browser toy that detects your hand with MediaPipe and draws a glowing lightsaber extending from it on the webcam feed. - Technique: MediaPipe Hands returns 2D landmarks each frame, and a glowing line is drawn along the direction from the wrist through the knuckles in p5.js over the webcam feed. - Try it: Use MediaPipe Hands and p5.js to draw a glowing lightsaber from the wrist along the fingers that leaves a trail as you swing it. Twist: when the sabers in both hands cross, they throw sparks and make a sound. #### AR Shooter Shared Between Nreal Light and iPad — Takashi Yoshinaga (2021) - Video: https://www.youtube.com/watch?v=X6MYN9y_K3w - Interaction: Shared & Social, Play - Platform & tech: Headset, Phone, Nreal Light, ARKit, Unity - Idea: A glasses player and an iPad player team up for AR shooting in the same space. - What it is: A co-located AR shooting game in which a glasses wearer and an iPad player see and fight the same virtual targets. - Technique: Both devices align to a shared reference (likely a common image marker or anchor) so their coordinate frames match, and target states and hits are synced over the network. - Try it: Use a printed image as a shared origin so a HoloKit headset and a phone see the same set of targets and sync hit states. Twist: give the two players unequal abilities, with one shooting and the other moving the targets. #### Holo Cat: Perspective Expression by Head Tracking — Takashi Yoshinaga (2021) - Video: https://www.youtube.com/watch?v=9kvIlsb8HQE - Interaction: Gaze & Attention, Perception & Effects - Platform & tech: Desktop, head tracking, Holo-SDK - Idea: Head tracking makes a cat on a flat screen look three-dimensional. - What it is: A flat screen shows his cat as a 3D hologram whose perspective shifts with the viewer's head, with no headset needed. - Technique: The viewer's head is tracked (likely by a webcam face tracker) and the 3D scene is rendered with an off-axis projection matched to the screen rectangle, creating motion parallax so the display acts as a window. - Try it: Track head position with MediaPipe Face Mesh and build an off-axis projection in three.js so a small box on screen seems to have real depth. Twist: the character inside the box turns its head to look at you. #### Copy and Paste Room Mesh with Meta Quest 3 — Takashi Yoshinaga (2023) - Video: https://www.youtube.com/watch?v=jqPZ_m5opuY - Interaction: Spatial Mapping, Portals & Worlds - Platform & tech: Headset, Meta Quest 3, Unity - Idea: In Quest 3, copy a chunk of the real room's mesh and paste it somewhere else. - What it is: Grabs a chunk of the scanned room mesh and pastes duplicates of real furniture elsewhere in passthrough. - Technique: The Quest scene mesh is cropped with a user-placed bounding box, duplicated, and re-rendered with passthrough-sampled colors at a new position. - Try it: On a LiDAR phone, cut out a 'chair' region from the mesh and place a copy on the other side of the room. Twist: make the copied furniture slowly melt or sprout plants. #### Mixed Reality Door for Meta Quest 3 — Takashi Yoshinaga (2023) - Video: https://www.youtube.com/watch?v=VW7ELpkVmIQ - Interaction: Portals & Worlds, Tangible Objects - Platform & tech: Headset, Meta Quest 3, Unity - Idea: Turn a real door at home into a portal to a virtual world. - What it is: A real door in the room becomes a portal: opening it in passthrough reveals a virtual world behind it. - Technique: A virtual portal plane is aligned to the real door frame in passthrough, with a stencil or depth mask so the virtual world is visible only where the real door opening is. - Try it: Manually align a stencil door to a real doorframe with AR Foundation and render a forest inside it. Twist: change the world behind the door based on the day's date or weather. #### Real-Time AR Effect on Cat — Takashi Yoshinaga (2023) - Video: https://www.youtube.com/watch?v=ExRmlztPP7M - Interaction: Perception & Effects, Hands & Body - Platform & tech: Phone, Unity, computer vision - Idea: Track a house cat in real time and wrap it in AR effects that follow it around. - What it is: Tracks his pet cat in real time and wraps it in animated AR effects that follow the animal as it moves. - Technique: An animal detection or segmentation model tracks the cat's bounding box or mask in real time, and particle effects are emitted from its position in screen or world space. - Try it: Use MediaPipe Object Detection (or YOLO) on a web page to recognize a cat or a stuffed toy animal, and generate a halo of particles that follows it. Twist: the faster the animal moves, the more exaggerated the effect. #### DejaViewer: Replaying Past Motion in AR — Takashi Yoshinaga (2025) - Video: https://www.youtube.com/watch?v=kau5hRnEGdI - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Headset, Meta Quest 3, Passthrough Camera API - Idea: Record the motion that just happened in front of you and replay the past on the spot as a point cloud. - What it is: Records motion in the field of view from Quest colour and depth cameras and replays it on the spot as a ghostly point cloud. - Technique: Passthrough color and depth frames are recorded, frame differencing isolates moving regions, and those regions are unprojected to point clouds that replay in place with fading transparency. - Try it: Use a phone depth camera to record someone walking past, keep only the moving parts as a point cloud, and loop them in place as semi-transparent 'afterimages'. Twist: the older an afterimage is, the more it breaks apart. #### StickAR: Spatial Notes Beyond Typing — Takashi Yoshinaga (2025) - Video: https://www.youtube.com/watch?v=o_zFX04Ircg - Interaction: Drawing & Making, Voice & Sound, Information & UI - Platform & tech: Headset, Meta Quest 3, OCR, Unity - Idea: Skip the keyboard and turn handwritten or spoken notes into sticky notes placed in space. - What it is: Hand-written or spoken notes are recognised and pinned as 3D sticky notes in space, avoiding keyboards in AR/VR. - Technique: OCR or speech recognition converts handwritten or spoken input to text, which is spawned as a 3D sticky-note panel anchored at the hand or gaze position. - Try it: Turn speech into text with the Web Speech API and generate a 3D sticky note on the wall hit by a ray from the center of the screen. Twist: color the notes automatically by content type (to-do or idea). #### Live Scene Analyzer: describing actions and checking tasks — Takashi Yoshinaga (2026) - Video: https://www.youtube.com/watch?v=O7LS4NdFOfQ - Interaction: Information & UI, Hands & Body - Platform & tech: Headset, VLM, first-person video, AR headset - Idea: Your glasses watch what your hands do and keep your checklist for you. - What it is: A first-person AR headset feed is read by an AI model almost in real time: in Description Mode it narrates actions like typing, using a phone or petting a cat; in Scenario Mode it ticks off steps of a morning routine. - Technique: Short clips or frames from the headset camera are sent with a system prompt to a vision-language model that returns an action description or a task-completion verdict. - Try it: Mount a phone on your head, send a frame every three seconds to a vision model with a five-step checklist (for example making tea), and tick steps on screen as they are seen. Twist: the checklist is a dance routine. #### On-device vision AI: scene understanding and prompt-based OCR — Takashi Yoshinaga (2026) - Video: https://www.youtube.com/watch?v=gVoTzhzCqSQ - Interaction: Information & UI, Tangible Objects - Platform & tech: Phone, on-device LLM, OCR, smartphone - Idea: Say what to read instead of framing it inside a scan box. - What it is: An LLM running entirely on a smartphone, without internet, either summarises the scene or reads only the text on an object you name, such as a label or a sign, and organises it into meaningful units. - Technique: A small on-device multimodal model receives the camera frame plus a natural-language target and returns structured text, replacing the fixed region of interest of classic OCR. - Try it: Build a phone page that sends a photo and a spoken request ('read the ingredients on the red box') to a vision model and pins the answer next to the object in AR. Twist: ask it to read things that have no text, like a face or a plant. #### Snap2VoxelAR — Takashi Yoshinaga (2026) - Video: https://www.youtube.com/watch?v=57gkK-xGxKQ - Interaction: Tangible Objects, Drawing & Making - Platform & tech: Headset, Meta Quest, XREAL One, GPT-5.5, Codex, voxel - Idea: Snap a photo and get back a toy-block version of the world. - What it is: A photo taken with the headset camera is converted by an AI workflow into voxel art and displayed back in AR next to the real object. - Technique: Captured images are sent to a coding agent (Codex with GPT-5.5) that writes and runs an image-to-voxel conversion, and the resulting voxel model is loaded into the AR scene. - Try it: Photograph an object, downsample it into a 16x16x16 voxel grid with a script (or an image-to-3D tool plus voxeliser), and place the blocks in AR beside the real thing. Twist: let the voxel copy slowly fall apart when you look away. #### Spatial Search for Meta Quest — Takashi Yoshinaga (2026) - Video: https://www.youtube.com/watch?v=CpNNvYJ_eqo - Interaction: Spatial Mapping, Voice & Sound, Information & UI - Platform & tech: Headset, Meta Quest 3, open-vocabulary search, point cloud - Idea: Search your room the way you search the web. - What it is: Type or say what you are looking for, and the matching points in Quest's spatial data of the room light up as the search result. - Technique: Image features from the headset camera are attached to the spatial point cloud (likely with a CLIP-style embedding), and a text query is embedded and matched against them to highlight regions. - Try it: Take 20 photos around a room with their positions (from a phone AR session), embed them with CLIP, and on a text query place a glowing marker where the best photo was taken. Twist: search for feelings, like 'somewhere cosy'. #### Turn your memories into 3D Gaussian splats — Takashi Yoshinaga (2026) - Video: https://www.youtube.com/watch?v=5nlFjTWZSE8 - Interaction: Portals & Worlds, Tangible Objects - Platform & tech: Headset, Apple SHARP, Gaussian splatting, Meta Quest 3 - Idea: An old photo becomes a place you can lean into. - What it is: Ordinary photos, even old film prints, are turned into 3D Gaussian splats and viewed in AR on Quest, so a memory stands on the table as a small 3D scene. - Technique: A single-image-to-3D model (Apple SHARP) predicts splats from each photo, which are then rendered in passthrough AR on Quest. - Try it: Ask each student to bring one family photo, convert it with a free image-to-3D tool, and display it in WebAR on their desk. Twist: place the 3D photo exactly where it was originally taken. ### Theo Watson *Artist; co-founder of openFrameworks and Design I/O* Artist and creative technologist, co-creator of openFrameworks, member of Graffiti Research Lab and F.A.T. Lab, and co-founder of the interactive studio Design I/O with Emily Gobeille. #### Daisies — Theo Watson (2005) - Video: https://vimeo.com/463536634 - Interaction: Hands & Body, Projection - Platform & tech: Projection, computer vision - Idea: Flowers wilt under your feet and grow back after you leave. - What it is: A floor projection of daisies that wilt and die under visitors' feet and quickly grow back after they walk away; version 2.0 grows algorithmically generated flowers. - Technique: An overhead camera with background subtraction detects where feet are on the floor projection, and a per-flower state (height, health) decays under occlusion and regrows over time. - Try it: Use an overhead camera and floor projection to make a patch of grass that gets trampled and slowly grows back after people leave (p5.js or TouchDesigner). Twist: the shape of the flowers that grow after someone walks over depends on how long they stayed. #### Funky Forest — Theo Watson, Emily Gobeille (2007) - Video: https://vimeo.com/3872687 - Interaction: Hands & Body, Tangible Objects, Projection - Platform & tech: Projection, openFrameworks, computer vision - Idea: Kids plant trees with their bodies and use real 'logs' to redirect a projected river to water the forest. - What it is: An interactive projected forest ecosystem: children grow trees by pressing their bodies into the forest and use physical logs and pillow 'rocks' to divert a digital stream to water them. - Technique: Overhead and frontal cameras segment children's bodies and tracked physical objects, which act as emitters and obstacles in a 2D fluid/particle stream simulation projected onto the floor and walls. - Try it: Project a flowing particle stream on the floor, use a camera to recognize cardboard boxes or books as movable barriers, and guide the water to trees at the edge of the screen. Twist: how a tree grows depends on which classmates the water it drank came from. #### L.A.S.E.R. Tag — Theo Watson (2007) - Video: https://www.youtube.com/watch?v=LtZq2q43Jkc - Interaction: Drawing & Making, Projection, Location & City - Platform & tech: Projection, openFrameworks, computer vision - Idea: Tag a whole building with a laser pointer: a camera tracks the dot and a projector paints giant strokes in real time. - What it is: Graffiti Research Lab's system for tagging buildings with a green laser pointer: a camera tracks the laser dot and a high-powered projector paints it as giant graffiti on the facade. - Technique: A camera watches the building, thresholds for the bright green laser dot, maps its position through a camera-to-projector homography, and draws strokes at that point with the projector. - Try it: Use a camera to track the bright dot of a laser pointer or phone flashlight and, after four-point calibration, paint thick strokes on a projected wall (p5.js or TouchDesigner). Twist: the stroke changes brush with drawing speed, dripping ink when slow and spraying paint when fast. #### Generative Graffiti — Theo Watson (2008) - Video: https://vimeo.com/463526242 - Interaction: Projection, Location & City, Perception & Effects - Platform & tech: Projection, openFrameworks, computer vision - Idea: Particle graffiti grows out of a hotel's lit windows and is projected back onto the facade. - What it is: With Graffiti Research Lab: particles spawn from the lit windows of the Maritime Hotel in New York, attracted to each other and repelled by dark windows, projected back onto the building. - Technique: A camera image of the facade is thresholded to find lit windows, which act as particle emitters and attractors while dark windows act as repellers, projected back via camera-projector registration. - Try it: Take a night photo of a teaching building, use the lit windows as particle sources and the dark ones as repellers, and project the animation back onto the same wall or a model. Twist: the particles stand for the 'late-night energy' of the people inside, and they migrate when a window goes dark. #### Filmmuseum augmented sand sculpture — Theo Watson, Emily Gobeille (2009) - Video: https://vimeo.com/6521600 - Interaction: Projection, Tangible Objects - Platform & tech: Projection, projection mapping, openFrameworks - Idea: Use projection to unveil the future film museum on a sand sculpture step by step. - What it is: An augmented projection that unveiled a five-metre sand sculpture of Amsterdam's future Film Museum building in stages using projection mapping. - Technique: Projection mapping aligns projector output to the physical sculpture using a 3D model or manually warped masks, then reveals textures and animations region by region. - Try it: Make a small building from clay or cardboard, use masks in TouchDesigner or MadMapper to align the projection with each face, and 'unveil' it in stages. Twist: have the projection tell the building's whole life, from construction to ruin. #### Knee Deep — Theo Watson, Emily Gobeille (2009) - Video: https://vimeo.com/8805152 - Interaction: Hands & Body, Portals & Worlds - Platform & tech: Projection, openFrameworks, computer vision - Idea: Step into worlds of different scales and see yourself standing in oceans, cities and even planets. - What it is: Children stomp into projected worlds of different scales with their feet, seeing themselves composited into oceans, cities and planets from impossible points of view. - Technique: Chroma keying or background subtraction isolates children's feet and legs from an overhead camera, which are then composited at a different scale into pre-rendered or live 3D worlds. - Try it: Use MediaPipe body segmentation to cut out a classmate, scale them and composite them into a city aerial photo or a microscope image, then project it on the floor so everyone can 'step into' the picture. Twist: switch scale with every step, from an ant's view to a planet's. #### Terrarium — Theo Watson, Emily Gobeille (2009) - Video: https://vimeo.com/5269088 - Interaction: Voice & Sound, Projection - Platform & tech: Projection, openFrameworks - Idea: Talk to the installation and your voice becomes food that keeps a virtual ecosystem alive. - What it is: A sonic ecosystem powered by visitors' voices: sounds enter through vents and are digested by creatures that grow and sustain life in the projected world. - Technique: Microphones feed amplitude and pitch features into an agent-based simulation, where incoming sounds become food particles and creatures grow according to what they consume. - Try it: Use p5.sound to turn sounds into 'food' particles that drop into a projected world, where simple creatures grow when they eat and starve when they do not. Twist: different pitches are different foods, and everyone uses their voices to 'feed' one shared species. #### Fat Tag, Meet Projector — Theo Watson (2010) - Video: https://vimeo.com/8617601 - Interaction: Drawing & Making, Projection, Location & City - Platform & tech: Phone, Projection, iOS, openFrameworks - Idea: Finger graffiti on a phone goes straight onto a street wall: the simplest projection bombing. - What it is: The Fat Tag iPhone graffiti app wired straight into a pocket projector, turning finger-drawn tags into instant projection bombing on city walls. - Technique: Touch strokes from a phone drawing app are streamed (likely over OSC or network) to a laptop that renders them to a pico projector aimed at a wall. - Try it: Make a phone web drawing pad, stream strokes over WebSocket to a computer connected to a projector, and project them onto a wall. Twist: allow only one kind of symbol, and have the whole class build a 'tag map' together on the same wall. #### Rise and Fall — Theo Watson, Emily Gobeille (2010) - Video: https://vimeo.com/10078874 - Interaction: Tangible Objects, Play - Platform & tech: Desktop, openFrameworks, marker tracking - Idea: Rotate a magazine cover in front of a webcam and the world on it unfolds a story. - What it is: An interactive front-and-back magazine cover for Boards Magazine: holding the printed cover to a webcam and rotating it steers a story world rendered on top of it. - Technique: Fiducial marker tracking on the printed cover estimates its 6-DoF pose from a webcam, and the cover's rotation angle drives gravity and scene state in a 3D world rendered on top. - Try it: Use 8th Wall or AR Foundation image tracking to recognize a poster you designed, let the poster's tilt decide which way a virtual figure slides, and give the front and back each their own world. Twist: flipping the poster makes the stories of the two worlds affect each other. #### Night Bright — Theo Watson, Emily Gobeille (2011) - Video: https://vimeo.com/29193895 - Interaction: Hands & Body, Projection, Play - Platform & tech: Projection, openFrameworks, computer vision - Idea: Use your body as a flashlight to light up a night forest and follow sounds to find hidden animals. - What it is: Children use their bodies as flashlights to light up a projected nighttime forest and play hide-and-seek with nocturnal creatures they locate by sound. - Technique: Depth or camera-based body tracking positions a light mask on each child, and a projected night scene is revealed only inside those regions, with spatial audio cues pointing to hidden creatures. - Try it: Build an all-black projected scene, track people's hands or bodies with MediaPipe so imagery appears only where the 'body flashlight' shines, and use left/right audio channels to hint where animals are. Twist: once all the animals are found, they turn around and 'shine' on you. #### Puppet Parade — Theo Watson, Emily Gobeille (2011) - Video: https://vimeo.com/34824490 - Interaction: Hands & Body, Projection, Shared & Social - Platform & tech: Projection, Kinect, openFrameworks - Idea: Work giant projected puppets with your arms while other kids walk into the scene to feed them. - What it is: Children puppeteer giant projected creatures with their arms via Kinect skeleton tracking, while others step into the scene to pet and feed them. - Technique: Kinect skeleton tracking maps the shoulder-elbow-wrist chain to the joints of a projected puppet, with separate body detection for others who walk in to interact. - Try it: Use MediaPipe Pose to map three arm joints onto a large projected marionette, and let a second person walk into the frame to feed it. Twist: two people each control half of the puppet's body and must cooperate to make it walk. #### Skataviz — Theo Watson, Emily Gobeille (2012) - Video: https://vimeo.com/53565956 - Interaction: Information & UI, Tangible Objects - Platform & tech: Phone, iOS, openFrameworks, gyroscope - Idea: Strap a phone to a skateboard and turn board flips into 3D data trails over the video. - What it is: An iPhone attached to a skateboard records its motion and visualizes the board's rotation and tricks as 3D data augmented over live video of the run. - Technique: The phone's gyroscope and accelerometer are fused (IMU sensor fusion) into an orientation stream that animates a 3D board model and data traces overlaid on synchronized video. - Try it: Strap a phone to a skateboard, bicycle or basketball, use the DeviceOrientation API in a web page to record its rotation, and overlay a 3D trail on the video. Twist: visualize the data as an animal that "calls out" when a trick happens. #### Weather Worlds — Theo Watson, Emily Gobeille (2013) - Video: https://vimeo.com/69084390 - Interaction: Hands & Body, Portals & Worlds - Platform & tech: Projection, Desktop, openFrameworks, computer vision - Idea: Children step into the picture and gain the power to control weather, waving up storms and lightning. - What it is: Real-time green-screening places children inside a dynamic weather world where body gestures conjure storms, tornadoes and lightning. - Technique: Green-screen chroma keying composites children into a 3D weather scene, and gesture detection from the silhouette (arm raise, spin) triggers particle weather systems. - Try it: Use MediaPipe segmentation to cut people into a weather scene projected on the wall, where raising a hand brings rain and spinning brings wind. Twist: trigger weather with emotions, for example laughing clears the sky and covering your face brings fog. #### Connected Worlds — Theo Watson, Emily Gobeille (2015) - Video: https://vimeo.com/131585517 - Interaction: Hands & Body, Tangible Objects, Shared & Social - Platform & tech: Projection, openFrameworks, Kinect - Idea: The whole hall is one ecosystem: children move real logs to steer water and plant seeds by hand, shaping six habitats. - What it is: A large-scale ecosystem at the New York Hall of Science: six projected habitats linked by a 3000 sq ft interactive floor and 45 ft waterfall; children move physical logs to route water and plant seeds with gestures. - Technique: Many Kinect depth cameras track the floor to detect physical logs and people, which redirect a projected 2D fluid simulation of water flowing between networked habitat simulations. - Try it: Project a stream of water onto the floor, use an overhead camera to detect a few colored boards that redirect the flow, and split the water between two small ecosystems on the wall (desert and rainforest). Twist: create a resource conflict between the two ecosystems so the class has to negotiate how the water is shared. #### Elements — Theo Watson, Emily Gobeille (2015) - Video: https://vimeo.com/197332386 - Interaction: Hands & Body, Projection - Platform & tech: Projection, openFrameworks, depth cameras - Idea: Each person becomes an element and shapes the projected world with their body. - What it is: Visitors embody Earth, Wind, Fire or Water and sculpt a projected environment with their movements. - Technique: Depth cameras track each visitor's body and gesture velocity, which are mapped to element-specific particle and fluid systems (fire, wind, water, earth) in a projected environment. - Try it: Use MediaPipe Pose to assign one element to each of four classmates, so their arm speed spawns fire, wind, water or earth particles that are projected on the wall. Twist: when two elements meet they create something new, such as water plus fire making mist. #### Living Library — Theo Watson, Emily Gobeille (2016) - Video: https://vimeo.com/203193098 - Interaction: Tangible Objects, Projection, Hands & Body - Platform & tech: Projection, openFrameworks, depth camera, laser projector - Idea: A giant real book whose illustrations move when you turn and touch the pages. - What it is: A giant projected book with real pages: a depth camera tracks hands so touching and turning pages triggers animated illustrations on the paper. - Technique: A depth camera over a blank physical book detects hand touches and page turns, and a projector calibrated to the book surface projects illustrations aligned to the pages. - Try it: Make a handmade book with blank pages, use a camera to detect which page is open (colored patches on the page corners help), and project the matching animation onto the page so whatever a finger touches moves. Twist: write a story whose ending changes only when you flip back to the previous page. #### Mimic — Theo Watson, Emily Gobeille (2017) - Video: https://vimeo.com/207140893 - Interaction: Hands & Body, Shared & Social, Play - Platform & tech: Desktop, industrial robot arm, depth sensors, openFrameworks - Idea: A machine that forms a different relationship with each person shows how behaviour and memory, not visuals, make a character feel alive — key for AR companions. - What it is: An industrial robot arm tracks the people around it and responds through gesture alone, building an impression of each person and becoming more trusting, curious or cautious with them. - Technique: Depth sensors track visitors, and custom software models per-person states such as trust, interest and curiosity that drive the robot arm's motion style. - Try it: Make an AR creature on a table that tracks each person's phone position and keeps a trust score per person — approaching those who move slowly and hiding from fast movers. Twist: let it greet returning visitors differently. #### Studio Play — Theo Watson, Emily Gobeille (2017) - Video: https://vimeo.com/203162198 - Interaction: Hands & Body, Information & UI - Platform & tech: Projection, Desktop, openFrameworks, depth cameras - Idea: Use your body to uncover and zoom into museum artworks. - What it is: A gallery of installations at the Cleveland Museum of Art where families use movement to explore the collection, including a 4K wall that reveals and zooms artworks by body position. - Technique: Depth cameras estimate each visitor's position and distance from the wall, mapping it to pan and zoom within a high-resolution image of artworks. - Try it: Use MediaPipe to estimate how far a person is from the screen and whether they stand left or right, and zoom into the matching region of a high-resolution famous painting. Twist: the figures in the painting turn their heads to look at where you stand. #### FIELD — Theo Watson, Emily Gobeille (2019) - Video: https://vimeo.com/322353545 - Interaction: Hands & Body, Projection - Platform & tech: Projection, openFrameworks - Idea: Pollinate a virtual flower field with your body, attract butterflies, and watch the seasons turn. - What it is: An ever-changing ecosystem where visitors pollinate the environment with their bodies: movement spreads color, makes flowers bloom and attracts butterflies across the seasons. - Technique: Camera or depth-based body tracking deposits color into a persistent field texture as visitors move, which drives flower spawning and butterfly agents that follow color gradients. - Try it: Use projection and body tracking so that people leave color where they walk, with flowers growing where color is dense and butterflies flying toward it. Twist: the scene slowly changes season in real time, and in winter flowers bloom only where two people stand close together. #### Painted Mirror — Theo Watson, Emily Gobeille (2020) - Video: https://vimeo.com/416474904 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, openFrameworks, depth camera - Idea: A mirror that paints: the longer you stand still, the clearer your portrait becomes in an illustrated world. - What it is: A mirror installation for Springville Public Library that separates you from your surroundings with a depth camera and paints your portrait into an illustrated world, sharpening as you stand still. - Technique: A depth camera segments the person from the background, and a stroke-based painterly renderer draws the portrait with brush size shrinking the longer the person stands still. - Try it: Use MediaPipe segmentation and p5.js brush particles to paint a person in an oil-painting style, where strokes get finer and clearer the longer they stay still and blur as soon as they move. Twist: replace the background with a scene from your school to make a graduation-photo mirror. #### Remnant — Theo Watson, Emily Gobeille (2020) - Video: https://vimeo.com/393540065 - Interaction: Hands & Body, Projection - Platform & tech: Projection, openFrameworks - Idea: Reach out to gather matter into a star until it explodes into a supernova and leaves a black hole. - What it is: Visitors hold out their hands to gather matter into a star, pushing it through its lifecycle until it explodes into supernovae that leave nebulae, pulsars and black holes. - Technique: Hand tracking (likely depth camera) detects open-hand gathering gestures, attracting particles to a point whose mass accumulates and triggers staged particle effects for each stellar lifecycle phase. - Try it: Use MediaPipe Hands to detect a gathering gesture that pulls projected particles into a star, which explodes into a nebula once it is full. Twist: the number of participants decides the star's color and lifespan, so one person alone can only make a small red dwarf. ### Max van Leeuwen *AR artist and creative technologist; Snap Lens Studio ambassador; co-founder of Max & Liisi* Dutch digital artist trained in visual effects at the Netherlands Film Academy who moved into real-time augmented reality, first at the Amsterdam AR studio GoSpooky and then with Liisi Mononen as Max & Liisi. He builds playful Lens Studio and Spectacles experiments, open-source tools and commercial lenses (Olivia Rodrigo, Snapchat), and led AR discovery workshops and an XR Sunday Social talk at A+E Lab in Chatham. #### Holiday Lights AR — Max van Leeuwen (2020) - Video: https://x.com/maksvanleeuwen/status/1474054362746003461 - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Phone, Lens Studio, world mesh, LiDAR - Idea: Decorate any room with light strings by drawing on its walls. - What it is: Tap and drag to hang strings of festive lights on the walls and furniture of any room; the lights stick to the real surfaces. Made at GoSpooky and updated to use Snap's world meshing on ARKit and ARCore. - Technique: Screen touches are raycast against a real-time world mesh (LiDAR or an ML fallback), and a string of emissive bulbs is spawned along the hit points so it follows the real geometry. - Try it: Build a WebXR hit-test app that places a glowing bulb wherever the finger drags across a detected surface, connected by a wire. Twist: make the bulbs blink in sequence to music from the phone microphone. #### Butterflies AR — Max van Leeuwen (2021) - Video: https://x.com/maksvanleeuwen/status/1421467861743247367 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, Lens Studio, LiDAR, world mesh - Idea: Let virtual butterflies land on the real objects around you. - What it is: A flock of butterflies flutters around the room, lands on real surfaces such as plants and railings, avoids bumping into them and follows a moving point in space; he released it as a free Lens Studio asset. - Technique: Each butterfly steers with simple flocking rules and raycasts against the LiDAR world mesh to find landing spots and avoid collisions; count, flight radius and flying 'character' are adjustable at runtime. - Try it: Make five AR butterflies in WebXR or AR Foundation that fly randomly and land on the nearest detected plane after a few seconds. Twist: they fly away when the phone moves too close. #### Neon Face Paint — Max van Leeuwen (2021) - Video: https://x.com/maksvanleeuwen/status/1469362123725807621 - Interaction: Face, Drawing & Making - Platform & tech: Phone, Lens Studio, face tracking, Snapchat - Idea: Paint glowing neon lines directly onto your own face. - What it is: A Snapchat lens made with Liisi Mononen for Snap's Call for Creation in which you paint glowing neon strokes onto your own face; it was nominated in the Art & Expression category at Lens Fest. - Technique: Touch strokes are likely projected onto the tracked face mesh and stored in its UV space, so the paint sticks to the skin as the face moves, with a bloom glow pass on top. - Try it: Use a face-mesh filter tool (Lens Studio, Effect House or MediaPipe Face Mesh) to let users draw on a face texture with a finger. Twist: the paint slowly drips down over time. #### AR Portal Gun — Max van Leeuwen (2022) - Video: https://x.com/maksvanleeuwen/status/1556715819316412418 - Interaction: Portals & Worlds, Spatial Mapping - Platform & tech: Phone, Lens Studio, Custom Landmarker, Metascan, LiDAR - Idea: Shoot portals into real walls and see your own room from somewhere else. - What it is: A working portal gun on a phone: you shoot two portals onto the walls of his living room or office and look through one to see the room from the other portal's position, running in real time. - Technique: The rooms were pre-scanned with Metascan and aligned with Snap's Custom Landmarker, so the lens can render the scanned 3D room from the second portal's viewpoint; LiDAR or ML depth handles the wall hits. - Try it: Scan a room with Polycam, load the model into a WebXR or Unity scene aligned to the real room, and place a portal whose view is rendered by a second camera at another spot. Twist: let the user tap to move the exit portal. #### AR Treasure Hunt — Max van Leeuwen (2022) - Video: https://www.youtube.com/watch?v=QL6RaRtzVGY - Interaction: Spatial Mapping, Play - Platform & tech: Phone, Lens Studio, LiDAR, world mesh - Idea: Turn a scan of your own room into a pirate treasure map. - What it is: A pirate scavenger hunt that adapts to your room: a cannon faces the nearest wall and blasts a hole in it, minigames lead you on, and a hand-drawn pirate map of your actual surroundings shows the red X, made with Liisi Mononen at GoSpooky. - Technique: Real-time world meshing (LiDAR, with a machine-learning fallback) builds a model of the room; the map is rendered from that mesh in a hand-drawn style, and debris uses baked camera textures so the 'broken wall' matches the real one. - Try it: Use a LiDAR phone or WebXR plane detection to find the largest wall in the room and place an AR target there; draw a top-down sketch map of the detected planes. Twist: hide the treasure behind the wall so players must walk to find it. #### Bold and Brash IRL — Max van Leeuwen (2022) - Video: https://www.youtube.com/watch?v=FI8s77qrstM - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Phone, Lens Studio, image marker, Houdini - Idea: Let a famous cartoon painting act out its own scene. - What it is: A real copy of Squidward's famous 'Bold and Brash' painting from SpongeBob is augmented with a silly AR scene that recreates the cartoon moment, made with Cyriel Verkuijlen. - Technique: The physical painting is tracked as an image marker in Lens Studio, and animated 3D elements modelled in Houdini are anchored to its frame. - Try it: Pick a poster or painting in the school, track it as an image target and animate one element leaving the frame. Twist: the element reacts differently when the viewer steps closer. #### Exploding Stuff in AR — Max van Leeuwen (2022) - Video: https://x.com/maksvanleeuwen/status/1598809529977737217 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, Lens Studio, LiDAR, physics - Idea: Make real surroundings look destructible. - What it is: Objects in the room appear to blow up: smoke and debris burst out and bounce off the real floor and furniture, and they pick up the colours and textures of the surroundings, created for a project with BLNK Digital. - Technique: A real-time LiDAR world mesh acts as a physics collider for debris, and the debris and smoke sample the camera image so they match the environment. - Try it: In AR Foundation with meshing or Reality Composer, spawn 30 physics cubes that fall and bounce on the real floor and table when you tap. Twist: colour each cube with the camera pixel where it was spawned. #### Real-Time Body Clones — Max van Leeuwen (2022) - Video: https://x.com/maksvanleeuwen/status/1527804730629574656 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, Lens Studio, 3D body mesh, MeshBuilder - Idea: Leave frozen 3D copies of yourself behind as you move. - What it is: A Snapchat lens that freezes 3D copies of your body as you move; tap to leave a single clone or hold to paint a trail of textured bodies through the room. - Technique: The MeshBuilder API copies the current tracked 3D body mesh into a static mesh, then the camera texture from that moment is projected back onto it so the clone keeps your real clothes and colours. - Try it: Use a body-segmentation lens or MediaPipe selfie segmentation to snapshot the cut-out person every second and leave the snapshots floating in the scene. Twist: make older clones slowly fade and drift upward. #### Any Object as a Push Button — Max van Leeuwen (2023) - Video: https://x.com/maksvanleeuwen/status/1706358333785989315 - Interaction: Tangible Objects, Hands & Body - Platform & tech: Headset, Spectacles, Lens Studio, hand tracking - Idea: Any real object can become a button you press with your finger. - What it is: A work-in-progress setup for Spectacles that turns any physical object into a push button: pressing the real object with your finger triggers a virtual response, giving AR interaction a physical feel. - Technique: Likely a virtual button collider is placed on the chosen object, and Spectacles fingertip tracking detects when the finger moves into it, triggering animation and sound. - Try it: Place an invisible button collider on a real mug using WebXR hand tracking or a Quest, and play a sound when the fingertip touches it. Twist: each object in the room plays a different note, making an instrument out of the desk. #### Hijacking Magritte's Treachery of Images — Max van Leeuwen (2023) - Video: https://x.com/maksvanleeuwen/status/1649856445230247936 - Interaction: Tangible Objects, Location & City, Perception & Effects - Platform & tech: Phone, Lens Studio, image marker - Idea: Continue a famous artwork's narrative in AR, right in the museum. - What it is: At LACMA he 'hijacked' René Magritte's famous painting of a pipe ('This is not a pipe'), using AR to continue the painting's story on the museum wall. - Technique: The painting is likely tracked as an image marker in Lens Studio, with animated 3D content anchored to the canvas so the added story plays out in front of the original. - Try it: Choose a reproduction of a well-known painting, track it as an image target, and add one AR animation that answers the painting's question or joke. Twist: make the AR continuation disagree with the original. #### Nuclei AR — Max van Leeuwen (2023) - Video: https://x.com/maksvanleeuwen/status/1630701599432536066 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, Lens Studio, 3D body tracking, VFX editor, inpainting - Idea: Replace a person with a particle cloud that keeps their colours. - What it is: Thousands of glowing particles cling to a person's body in 3D and copy their colours, while the real person underneath is painted out, so they seem to dissolve into a cloud of 'nuclei'. It was shown at the Max Ernst Museum and on Snap's AR mirrors. - Technique: 3D body tracking drives particle emitters on the body mesh, particles sample the camera colour at their position, and person inpainting removes the real body before the particles are composited on top. - Try it: Use MediaPipe pose or a Lens Studio body template to emit particles from each joint, coloured by the camera pixel beneath them, over a blurred background. Twist: particles scatter when the person moves fast and regroup when they stand still. #### PS1 Hagrid in AR — Max van Leeuwen (2023) - Video: https://www.youtube.com/watch?v=E65qwxsuUwQ - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, Lens Studio, Custom Landmarker, Polycam, shaders - Idea: Overlay a retro video-game level, glitches included, on a real place. - What it is: A nostalgic level built on top of his real office: low-poly Hagrid and other props from the PlayStation 1 Harry Potter games stand along a path, complete with wobbly vertices and warped textures. - Technique: The office was 3D-scanned with Polycam and turned into a Custom Landmarker for alignment; custom shaders fake PS1 artefacts such as affine texture sampling and vertex snapping. - Try it: Scan a school corridor with Polycam, align a WebXR or Lens Studio scene to it and place three low-poly characters along the path. Twist: add a vertex-snapping shader so everything wobbles like a 1990s console. #### Polygon Studio — Max van Leeuwen (2023) - Video: https://www.youtube.com/watch?v=S_k8jysvK4c - Interaction: Drawing & Making, Hands & Body, Shared & Social - Platform & tech: Headset, Spectacles, Lens Studio, hand tracking, QR codes - Idea: Sculpt 3D models with your bare hands in the room, together with friends. - What it is: An open-source low-poly 3D modelling tool for AR glasses: you pinch and drag vertices with your fingers to build meshes in the air, walk around them, model together with friends in a shared session, and export the result as a sequence of QR codes. - Technique: Spectacles hand tracking detects pinch gestures that grab and move mesh vertices, the mesh is rebuilt in real time with a MeshBuilder, the scene is synced via Snap's connected-lens multiplayer, and the model data is encoded into QR codes for export. - Try it: Build a WebXR or Quest hand-tracking scene where pinching creates a point in the air and every three points form a triangle. Twist: let two people add points to the same shared shape. #### City Carpet AR — Max van Leeuwen (2024) - Video: https://www.youtube.com/watch?v=D1RazB0V3QA - Interaction: Tangible Objects, Play - Platform & tech: Headset, Phone, Spectacles, Lens Studio, image marker - Idea: Bring a printed toy map to life with autonomous traffic. - What it is: A children's city play-rug comes alive: tiny cars drive along the printed roads, stop at red lights and move aside for police cars, seen through Spectacles or the Snapchat app. Left running, the carpet city eventually jams into full gridlock. - Technique: The rug is tracked as an image marker; cars follow hand-placed waypoints along the roads and use simple rules (collision checks, traffic lights, yielding to police) with only a few cars checking at a time for performance. - Try it: Print a simple road map on A3, track it as an image target in Lens Studio or AR Foundation and make three cars loop along waypoints. Twist: add a tappable traffic light that makes all cars stop. #### Pocket Buddy — Max van Leeuwen (2024) - Video: https://www.youtube.com/watch?v=lCiCceLHw_s - Interaction: Play, Tangible Objects, Face - Platform & tech: Phone, Lens Studio, Snapchat, object detection, persistent storage - Idea: Scan your friend into a pet that asks you to find real things. - What it is: A virtual-pet game where you scan a friend to turn them into a pocket-sized pet; the pet has wishes, and you keep it happy by finding the requested real objects (over 30 are recognised) with your camera. It won Lens of the Year at Lens Fest 2024. - Technique: The lens captures the person's texture and name into persistent storage, maps it onto a small animated character, and uses an on-device object-detection model to check whether the item the pet wants is in view. - Try it: Make a phone AR pet with Teachable Machine or a COCO object detector that asks for one everyday object at a time and reacts happily when the camera sees it. Twist: the pet's mood slowly drops if you ignore it for a minute. #### AR Flower Drawing — Max van Leeuwen (2025) - Video: https://x.com/maksvanleeuwen/status/1950507520977440932 - Interaction: Drawing & Making, Play, Location & City - Platform & tech: Headset, Spectacles, Lens Studio, leaderboard - Idea: Draw a line of flowers through the world and compete on its length. - What it is: A Spectacles lens in which a trail of flowers grows along the path you draw in the world; the length of your flower line is submitted to a global leaderboard (his record is about 20 metres). - Technique: Likely a hand-tracked drawing path is converted into a trail mesh populated with instanced flowers, with its total length measured in world units and sent to an online leaderboard. - Try it: Build a WebXR drawing app that spawns a flower every 20 cm along the finger's path and shows the total length in metres. Twist: the flowers wilt if the line crosses itself. #### AR Sundial — Max van Leeuwen (2025) - Video: https://x.com/maksvanleeuwen/status/1953056369818439790 - Interaction: Location & City, Information & UI, Perception & Effects - Platform & tech: Phone, Lens Studio, GPS, real-time shadows - Idea: A virtual object whose shadow is cast by the real sun and tells the time. - What it is: A virtual antique sundial placed outdoors that actually tells the time: it is rotated for your latitude and lit by the sun's real position, so its AR shadow reads the true solar time. - Technique: GPS location and time of day compute the sun's azimuth and elevation, which drive the directional light of his self-shadowing renderer (LUX) so virtual shadows align with real ones. - Try it: Use a sun-position library (e.g. SunCalc) in a WebXR app to aim a directional light and cast a virtual pole's shadow on the ground outdoors. Twist: compare it with a real stick's shadow at three times of day. #### Age of Mythology in AR — Max van Leeuwen (2025) - Video: https://www.youtube.com/watch?v=SHRz9ZCFdgo - Interaction: Play, Spatial Mapping - Platform & tech: Headset, Spectacles, Lens Studio - Idea: Play a classic top-down strategy game as a diorama on your own floor. - What it is: A fan-art homage to the classic real-time strategy game: tiny villagers, temples and water from Age of Mythology appear on his living-room floor and outdoors, and he commands units with his hands through Snap Spectacles. - Technique: Low-poly game scenes are placed on detected ground planes in Lens Studio and recorded live on Spectacles; hand tracking selects and moves units, and the water uses looping sprite-sheet animation in a shader. - Try it: Build a tabletop mini-game in AR Foundation or WebXR where the player taps to send five tiny units to collect objects around a real table. Twist: make the units avoid real obstacles such as a cup placed on the table. #### Specs Racer — Max van Leeuwen (2025) - Video: https://x.com/maksvanleeuwen/status/2001680542304239964 - Interaction: Play, Tangible Objects, Shared & Social - Platform & tech: Headset, Spectacles, Lens Studio, steering wheel, pedals - Idea: Race in AR glasses using real steering wheels and pedals. - What it is: An arcade racing game for Snap Spectacles played in a physical installation with real steering wheels, pedals and start lights; Snap built the rig and he developed the game with Ninsky. - Technique: Physical controllers are likely connected to the Spectacles lens over a network bridge, and a race track with physics cars is rendered in the glasses on top of the real room. - Try it: Connect a cheap USB gamepad or a phone's gyroscope to a WebXR scene and drive a toy car around a track drawn on the real floor. Twist: two players with two phones race on the same floor track. #### Dancing Self-Scan on Your Hand — Max van Leeuwen (2026) - Video: https://x.com/maksvanleeuwen/status/2103112713560002911 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Headset, Web, WebXR, Spectacles browser, 3D scan, hand tracking - Idea: A personal website that puts a tiny dancing you on the visitor's hand. - What it is: His portfolio website, opened in the Spectacles browser, loads a small dancing 3D scan of himself that stands on the visitor's hand through WebXR. - Technique: A rigged, animated 3D scan is loaded in a three.js WebXR session and parented to the tracked hand joints exposed by the WebXR Hand Input API. - Try it: Scan yourself with Polycam, rig it with Mixamo and show it dancing on a hand in a three.js WebXR page on Quest or Spectacles. Twist: the figure jumps to the other hand when you clap. #### Fourier Series in AR — Max van Leeuwen (2026) - Video: https://x.com/maksvanleeuwen/status/2076380515016348075 - Interaction: Drawing & Making, Hands & Body, Information & UI - Platform & tech: Headset, Spectacles, Lens Studio, hand tracking - Idea: Watch spinning circles redraw the shape you just drew in the air. - What it is: Inspired by 3Blue1Brown, you draw a shape in the air with your hand on Spectacles and a chain of spinning circles (epicycles) redraws it; opening your hand stabilises the drawing. - Technique: The hand-drawn path is sampled, a discrete Fourier transform turns it into rotating vectors, and these are rendered as nested circles in 3D whose tip traces the shape. - Try it: Record a finger or mouse drawing as 100 points, compute its DFT in JavaScript and animate the epicycles in a WebXR scene. Twist: let students limit the number of circles to see the drawing simplify. ### Jerome Etienne *Creator of AR.js; WebGL/three.js developer (learningthreejs)* French web developer behind learningthreejs and AR.js, the open-source library that brought 60fps marker-based AR to any phone browser in 2017. #### Augmented Gesture with Post-its — Jerome Etienne (2012) - Video: https://www.youtube.com/watch?v=k8R1y0oqiic - Interaction: Hands & Body, Tangible Objects - Platform & tech: Web, augmentedgesture.js, WebRTC - Idea: Stick Post-its on your fingers and they become a web gesture tracker. - What it is: Cheap coloured Post-it notes on the fingers become gesture trackers for controlling a web 3D scene. - Technique: Color thresholding on the webcam image finds the centroids of brightly colored Post-its on the fingers, which serve as low-cost gesture input points. - Try it: Track colored Post-its on your fingers with color thresholding in a web page and use their positions to rotate a three.js object. Twist: detect a "pinch" when the two colors get close to each other. #### Augmented Pong in 3D — Jerome Etienne (2012) - Video: https://www.youtube.com/watch?v=KGNjXXpKRpE - Interaction: Tangible Objects, Play - Platform & tech: Web, three.js, WebRTC - Idea: Hold physical markers as paddles and play 3D Pong in the browser. - What it is: A 3D Pong game in which the paddles are driven by physical markers held in front of the webcam. - Technique: Two fiducial markers are tracked from the webcam, and their screen positions are mapped to paddle positions in a three.js Pong game. - Try it: Track two marker cards with AR.js as paddles and use three.js to make a ball bounce back and forth between them. Twist: the ball speed changes with the distance between the two cards. #### Augmented Reality in Your Browser — Jerome Etienne (2012) - Video: https://www.youtube.com/watch?v=rzLuJxTraos - Interaction: Tangible Objects - Platform & tech: Web, JSARToolKit, WebRTC, three.js - Idea: Web AR already running in the browser back in 2012. - What it is: A 2012 screencast showing marker-based AR running in a desktop web browser via WebRTC webcam access and JSARToolKit. - Technique: WebRTC getUserMedia streams the webcam into a canvas, JSARToolKit detects square fiducial markers and estimates their pose, and three.js renders 3D content with that pose over the video. - Try it: Use AR.js (or MindAR) to recognize a printed Hiro marker in a web page and display a spinning 3D model. Twist: make the model change color according to how far the marker is tilted. #### Punch a Doom Character in AR — Jerome Etienne (2012) - Video: https://www.youtube.com/watch?v=Aa9945MGRL0 - Interaction: Hands & Body, Play - Platform & tech: Web, three.js, WebRTC - Idea: Throw punches at the camera to hit a Doom monster in the browser. - What it is: A browser mini-game in which you punch a Doom monster that appears in your webcam view. - Technique: Frame differencing (motion detection) in regions of the webcam image registers punches at the monster's screen position, rendered in three.js over the video. - Try it: Build camera frame differencing in p5.js so the monster gets knocked flying when a big motion is detected in a certain region of the frame. Twist: the monster dodges, so you have to punch from different directions. #### AR Swimming Pool — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=2u3xB9bXozs - Interaction: Portals & Worlds, Tangible Objects - Platform & tech: Web, Phone, AR.js - Idea: A mini swimming pool with a rubber duck and fish sunk into the tabletop. - What it is: A tiny pool with a rubber duck and fish lives inside a marker on the desk, seen through a cut-out in the table. - Technique: A pool geometry sunk below the marker plane is framed by a depth-only occluder around the opening, creating the illusion of a cavity cut into the table. - Try it: Build a small pool model below a marker, use occluder faces to make it look like a hole dug into the table, and fill it with swimming fish. Twist: replace the pool with an underground ant nest. #### AR-Code: QR Code to AR in Under 5 Seconds — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=GSlGQ_Kh54I - Interaction: Tangible Objects, Information & UI - Platform & tech: Web, Phone, AR.js - Idea: Scan a QR code and land in web AR in under five seconds. - What it is: Scanning a QR code opens a web page that immediately shows AR content on the same marker, measuring cold-start time to AR. - Technique: A QR code encodes the URL of a web AR page and doubles as the tracking marker (or sits beside one), so scanning it launches AR content aligned to the same printed code. - Try it: Generate a QR code that holds the address of an AR web page and print an AR.js marker next to it, so scanning it makes the content appear right on the marker. Twist: turn it into a "talking business card" for a course assignment. #### AR.js on Tango with WebAR Playground — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=pXF-eb6eB7c - Interaction: Spatial Mapping - Platform & tech: Web, Phone, AR.js, Tango, A-Frame - Idea: Markerless web AR running on a Tango phone. - What it is: Markerless web AR on a Google Tango phone, placing A-Frame objects directly on real surfaces. - Technique: Tango's depth camera and motion tracking provide 6DoF pose and surface hits in the browser, letting A-Frame entities be placed on real surfaces without markers. - Try it: Use the WebXR Hit Test API (Chrome on Android) to place an A-Frame object on the floor, and compare it with the AR.js marker approach. Twist: every tap on the floor plants a tree that grows. #### AR.js – Efficient Augmented Reality for the Web — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=0MtvjFg7tik - Interaction: Tangible Objects - Platform & tech: Web, Phone, AR.js, ARToolKit, three.js - Idea: Pure web AR with no install that runs at 60 fps even on an old phone. - What it is: Launch demo of AR.js: pure-JavaScript marker AR running at 60fps on an old phone, with no app install. - Technique: An Emscripten-compiled ARToolKit runs marker pose estimation in the browser and feeds the pose to a three.js camera, giving marker AR with no app install. - Try it: Write a marker AR web page in no more than 10 lines of HTML using AR.js and A-Frame, deploy it to GitHub Pages, and open it on a phone. Twist: design your own pattern marker to replace the default Hiro. #### Area Learning with Multi-Markers — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=ibzcbBLzDZU - Interaction: Spatial Mapping, Tangible Objects - Platform & tech: Web, Phone, AR.js - Idea: Several markers learned together let web AR cover a whole room. - What it is: Several markers scattered around a room are learned together so content stays anchored across a larger area, even under hand occlusion. - Technique: Relative poses between several markers are learned once, so any visible marker can recover the shared coordinate frame, keeping content anchored when others are occluded. - Try it: Stick four markers on a table, record their relative positions first, then show the same large model whenever any one of them is visible. Twist: cover markers with your hand and compare the stability of single-marker and multi-marker tracking. #### Augmenting Any Webpage with AR.js — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=q2IFgPYOpq8 - Interaction: Information & UI - Platform & tech: Web, AR.js - Idea: Add one snippet to any web page and its content jumps out of the page. - What it is: A snippet that adds AR to an ordinary web page, so visitors see 3D content popping out of the page itself. - Technique: A JavaScript snippet overlays a transparent WebGL canvas on the page and ties 3D content to DOM elements (or a marker), so models appear to pop out of ordinary HTML. - Try it: Add a transparent three.js canvas to your portfolio page so a 3D object pops out of a certain image when it scrolls into view. Twist: use WebXR so the same page lets phone users place the object on their desk. #### Hatsune Miku Dancing in My Living Room — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=oXxqEoMNH0Y - Interaction: Performance, Spatial Mapping - Platform & tech: Web, Phone, AR.js, A-Frame, Tango - Idea: Hatsune Miku dances on your living room floor. - What it is: A dancing Hatsune Miku model performs on the living room floor using AR.js on Tango and A-Frame. - Technique: An MMD character with dance animation is loaded in A-Frame and placed on the floor using Tango's markerless tracking, so it stays fixed while the viewer walks around. - Try it: Use WebXR plane detection to put a dancing Mixamo character on the floor and play music. Twist: make the character's dance rhythm follow the beat detected by the microphone. #### Hole in the Wall with AR.js — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=gEHAs9pLIJY - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Web, Phone, AR.js - Idea: A single marker opens a hole in the wall that shows the space inside it. - What it is: A marker on a surface opens a convincing hole that reveals a virtual space inside the wall. - Technique: An inverted box rendered behind the marker plane is surrounded by an invisible depth-only occluder mesh, so the virtual interior is only visible through the opening. - Try it: Use three.js or AR.js to build a box opening upward below a marker, surrounded by depth-only occluder faces, forming a hole in the tabletop. Twist: put a small creature in the hole that peeks out. #### Hologram Bird in AR.js — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=xrsZuHpJEsA - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Web, Phone, AR.js - Idea: A Star Wars-style hologram bird jumps out of a business card to deliver a message. - What it is: A Star Wars-style hologram bird delivers a message from a marker, prototyping an AR business card. - Technique: A marker on a business card triggers an animated bird rendered with an additive, scanline hologram shader and synced audio message. - Try it: Use AR.js to show a small character with a scanline hologram shader on your own business-card marker, playing a 10-second self-introduction. Twist: the character says a different greeting depending on the time of day. #### Invisibility Cloak for AR.js — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=9xnORdOyLq8 - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Web, Phone, AR.js - Idea: A Harry Potter-style invisibility cloak that makes the marker and the objects on it vanish. - What it is: A Harry Potter-style cloak effect that makes the marker and objects on it vanish. - Technique: An occluder mesh over the marker writes depth only, and a background patch (a captured or procedural texture) is drawn in its place so the marker and objects appear to vanish. - Try it: Place a plane above a marker that shows a pre-shot photo of the background, so small objects on the marker become "invisible." Twist: add a rippling distortion around the edge of the invisible area, like a magic cloak. #### Liquid Marker Matrix Style — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=N1LJQTHQXc0 - Interaction: Perception & Effects - Platform & tech: Web, Phone, AR.js - Idea: The camera image around a marker ripples like liquid. - What it is: The camera image around a marker ripples like liquid, 'liquifying reality' in the style of The Matrix. - Technique: The video texture around the marker is remapped in a fragment shader with time-varying sinusoidal UV offsets centered on the marker pose, producing a ripple distortion. - Try it: Use a shader in three.js to apply a sine-wave ripple distortion to the camera image, centered on the marker. Twist: tap the screen to drop a new "water drop" at that spot. #### Measure It with AR.js — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=dIEZwmjuaUA - Interaction: Information & UI, Tangible Objects - Platform & tech: Web, Phone, AR.js - Idea: Measure the distance between two AR markers live in the browser. - What it is: Measures the distance between two AR markers in the browser and draws it as a live ruler. - Technique: Two tracked markers each give a pose in camera space, and the distance between their origins is drawn as a line with a live numeric label. - Try it: Use two markers to measure the distance between two objects on a table and show the number live. Twist: play an alert sound when the distance reaches a set value, turning it into a "keep your social distance" game. #### Refraction in AR.js — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=2wySBEzetGg - Interaction: Perception & Effects - Platform & tech: Web, Phone, AR.js, three.js - Idea: Virtual glass objects refract the real scene and blend into reality. - What it is: Glass-like virtual objects refract the live camera feed so they blend into the real desk; a larger glass torus fills his apartment. - Technique: The live camera frame is used as an environment texture, and a refraction shader offsets its lookup by the surface normal so glass objects bend the real scene behind them. - Try it: In three.js with WebXR, use the camera image as an environment map and put a glass sphere with a refractive material on the table. Twist: shape the glass object as 3D letters of your own name. #### Two-Way AR Portal with A-Frame 360 — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=AI-g2wV9_VA - Interaction: Portals & Worlds - Platform & tech: Web, Phone, AR.js, A-Frame, Tango - Idea: A two-way portal: from outside you see a 360 world, and from inside you look back at the real room. - What it is: A walk-through portal door that shows a 360 world from outside and the real room from inside. - Technique: A stencil-masked door shows a 360 panorama from outside, and when the camera crosses the door plane the rendering inverts so the panorama surrounds the viewer and the real room is seen through the door. - Try it: Place a door with three.js and WebXR that shows a 360 panorama inside; after you walk through, the panorama surrounds you and the real room is visible through the door. Twist: use a panorama you shot yourself. #### Walking in VR Using AR.js Tracking — Jerome Etienne (2017) - Video: https://www.youtube.com/watch?v=DoR7usCBHNM - Interaction: Spatial Mapping, Hands & Body - Platform & tech: Web, Phone, AR.js, A-Frame - Idea: Borrow AR.js positioning to actually walk into the A-Frame logo in VR. - What it is: Uses AR.js positional tracking to physically walk into the A-Frame logo in virtual reality. - Technique: AR positional tracking drives the VR camera, so physical walking translates 1:1 into movement through a fully virtual A-Frame scene. - Try it: Track the phone's position with WebXR, replace the view with a fully virtual scene, and have students walk into a giant logo in the classroom. Twist: align the walls of the virtual scene with the real classroom walls so no one bumps into them. #### WebXR Studio on Augmented Website — Jerome Etienne (2018) - Video: https://www.youtube.com/watch?v=UryjzCDPLig - Interaction: Drawing & Making, Information & UI - Platform & tech: Web, Phone, WebXR, AR.js - Idea: Assemble AR scenes in the browser and publish them straight to an 'augmented website'. - What it is: A browser authoring tool that lets anyone compose AR scenes and publish them on an 'augmented website' viewable on any phone. - Technique: A browser editor serializes a scene graph of 3D assets and anchor settings to a URL, and a WebXR/AR.js viewer loads that description on any phone. - Try it: Build a simple editor page with A-Frame that generates a share link after you drag three objects into place, so anyone opening the link sees the same layout in AR. Twist: let visitors add an object too, so the scene keeps growing. ### Rafael Lozano-Hemmer *Media artist; founder of Antimodular Research* Mexican-Canadian artist whose 'relational architecture' uses surveillance cameras, robotic projectors and biometric sensors to let the public take over city squares and façades; represented Mexico at the 2007 Venice Biennale. #### Surface Tension — Rafael Lozano-Hemmer (1992) - Video: https://www.youtube.com/watch?v=JXLoLPkzdto - Interaction: Gaze & Attention, Hands & Body, Projection - Platform & tech: Projection, camera tracking, video projection - Idea: Being watched back by an image is one of the strongest ways to make a virtual thing feel present — AR characters that track the user's position use the same trick. - What it is: A giant projected human eye on a wall follows the viewer wherever they move in the room. - Technique: A camera tracking system estimates the viewer's position and selects or interpolates pre-recorded video of an eye looking in that direction. - Try it: Place a large AR eye on a real wall that rotates to look at the phone's position, and blinks when the user looks directly at it. Twist: add a second eye that only follows other people. #### Vectorial Elevation — Rafael Lozano-Hemmer (1999) - Video: https://www.youtube.com/watch?v=C4xx8sirByI - Interaction: Location & City, Shared & Social, Information & UI - Platform & tech: Web, Desktop, robotic searchlights, web interface, webcams - Idea: Letting remote people design something that appears over a real city shows how participation can reach physical space — a template for web-authored, site-anchored AR. - What it is: Robotic searchlights around Mexico City's Zócalo form huge light sculptures in the night sky, each designed by people over the internet and shown with their name. - Technique: A web interface with a 3D model of the square lets users aim 18 robotic searchlights; designs are queued and executed in sequence while webcams document each one back to its author. - Try it: Build a web page where remote users place light beams on a 3D model of your campus, then show the queued designs one by one as AR beams over the real site. Twist: credit each design with the author's city floating in the sky. #### Body Movies (Relational Architecture 6) — Rafael Lozano-Hemmer (2001) - Video: https://www.youtube.com/watch?v=g-CNxFiXZDY - Interaction: Hands & Body, Projection, Location & City - Platform & tech: Projection, robotic projectors, camera tracking, xenon lights - Idea: Your giant shadow is the only window through which strangers' portraits can be seen. - What it is: Thousands of street portraits are projected onto a building façade but only become visible inside the giant shadows of passers-by, whose silhouettes grow from 2 to 25 metres depending on their distance from floor lights. - Technique: Robotic projectors show portraits on a façade washed out by bright floor lamps; a camera tracks shadow regions, and when a shadow matches a portrait the scene advances. - Try it: Have one projector show images while a strong lamp washes the picture out, so the images are visible only inside classmates' shadows. Twist: use a camera to check whether a shadow fully covers a face, and trigger the next image when it does. #### Frequency and Volume (Relational Architecture 9) — Rafael Lozano-Hemmer (2003) - Video: https://www.youtube.com/watch?v=hWd8uv8U4tc - Interaction: Hands & Body, Voice & Sound, Projection - Platform & tech: Projection, camera tracking, radio scanners - Idea: Tune a radio with your shadow on the wall to hear the city's invisible airwaves. - What it is: Visitors' shadows on a wall tune a radio: the position of the shadow selects a frequency and its size sets the volume, making the invisible airwaves audible and contested. - Technique: A camera tracks shadow position and size on the wall and maps them to frequency and volume of an array of computer-controlled radio receivers. - Try it: Use a camera to detect a shadow on the wall, with its horizontal position choosing which audio clip plays and its size setting the volume. Twist: when several shadows overlap, their sounds mix or interfere with each other. #### Subtitled Public — Rafael Lozano-Hemmer (2005) - Video: https://vimeo.com/1089123902 - Interaction: Hands & Body, Projection, Shared & Social - Platform & tech: Projection, surveillance tracking, projection on bodies - Idea: Everyone gets a verb label projected on their body, and the only way to pass it on is to touch someone else. - What it is: In an empty room a surveillance system detects each visitor and projects a random third-person verb onto their chest; the only way to get rid of your word is to touch someone else, who then takes it. - Technique: Overhead camera tracking follows each person and steers projections of text onto their body, swapping labels when two tracked blobs touch. - Try it: Track classmates with an overhead camera and project a word onto each person, swapping words when two people touch. Twist: replace the words with adjectives the classmates wrote themselves, and watch how the labels flow. #### Under Scan (Relational Architecture 11) — Rafael Lozano-Hemmer (2005) - Video: https://www.youtube.com/watch?v=Bfn14sLJmyU - Interaction: Hands & Body, Projection, Location & City - Platform & tech: Projection, computer vision tracking, high-power projectors - Idea: Walk across the square and a stranger appears in your shadow, looking up at you. - What it is: Town squares are flooded with white light; as pedestrians walk, video portraits of local people appear inside their shadows, look up at them, and vanish when they leave. - Technique: Computer vision tracks pedestrians from above and predicts their path, so projectors place a video portrait precisely inside each person's shadow a moment ahead of them. - Try it: Project white light onto the ground, use a camera to find where shadows are, and project classmates' pre-recorded 'looking up' videos inside them. Twist: the longer a shadow stays, the longer the sentence the person in the video speaks. #### Close-Up (ShadowBox 2) — Rafael Lozano-Hemmer (2006) - Video: https://www.youtube.com/watch?v=gGbCF2oPayM - Interaction: Hands & Body, Perception & Effects, Shared & Social - Platform & tech: Desktop, camera tracking, video database - Idea: Your shadow is filled with surveillance videos of everyone who looked at the piece before you. - What it is: The viewer's shadow on a screen is filled with up to 800 tiny surveillance videos of people who recently looked at the piece — and the viewer is recorded for the next person. - Technique: A camera records each viewer while their silhouette mask reveals a mosaic of previously recorded viewer clips; the database keeps the most recent 10,000. - Try it: Use a camera to record each visitor while filling their silhouette with a grid of small videos recorded earlier. Twist: the shadow shows only people who visited on the same day as the current viewer. #### Eye Contact (ShadowBox 1) — Rafael Lozano-Hemmer (2006) - Video: https://www.youtube.com/watch?v=SzIwx-oX3U8 - Interaction: Gaze & Attention, Hands & Body, Perception & Effects - Platform & tech: Desktop, built-in camera tracking, video grid - Idea: The moment you show up, hundreds of people on the screen wake up together and stare at you. - What it is: A display shows hundreds of tiny videos of people lying down; when a viewer is detected, all of them wake up and turn to stare directly at the viewer. - Technique: A built-in camera detects the viewer's presence and silhouette, switching each video tile from its idle loop to a 'turn and look' clip. - Try it: Have each classmate record a 'fall asleep, then turn to look at the camera' clip, assemble them into a grid, and trigger it when the camera detects a person. Twist: only the people in cells covered by a shadow wake up. #### Pulse Room — Rafael Lozano-Hemmer (2006) - Video: https://www.youtube.com/watch?v=R3benqCGVLI - Interaction: Hands & Body, Information & UI, Shared & Social - Platform & tech: Desktop, heart-rate sensor, incandescent bulbs, dimmers - Idea: A room that keeps a queue of visitors' heartbeats shows how biometric input can become a shared, cumulative space — a model for multi-user AR memories. - What it is: Hundreds of clear incandescent bulbs hang in a room; a visitor holds a sensor, their heartbeat makes the nearest bulb flash, and each new heartbeat pushes older ones down the grid. - Technique: A handheld heart-rate sensor detects the pulse and a controller dims each bulb to that rhythm, shifting the recording one bulb along the grid each time a new person grips the sensor. - Try it: Hang a grid of virtual bulbs from the ceiling in AR and make each one pulse to a heartbeat captured from a phone-camera fingertip reading, with newer beats pushing older ones away. Twist: let the grid persist across visitors via a shared anchor. #### Third Person (ShadowBox 4) — Rafael Lozano-Hemmer (2006) - Video: https://www.youtube.com/watch?v=w-EWZ1r1Yos - Interaction: Hands & Body, Information & UI, Perception & Effects - Platform & tech: Desktop, camera silhouette, text rendering - Idea: Build your shadow portrait out of every verb in the dictionary. - What it is: The viewer's shadow is drawn entirely out of the verbs of a dictionary in English, Spanish or French, like a portrait written in actions. - Technique: Camera silhouette masking selects which cells of a dense text grid are rendered, with word size varying with distance from the screen. - Try it: In p5.js, fill the silhouette area from the camera with text (a set of words the class picks). Twist: the closer you get to the screen, the smaller and denser the words become. #### Pulse Spiral — Rafael Lozano-Hemmer (2008) - Video: https://www.youtube.com/watch?v=9hwEpbjlaek - Interaction: Hands & Body, Information & UI, Shared & Social - Platform & tech: Desktop, heart-rate sensors, light bulbs - Idea: A spiral turns a history of inputs into a readable shape, so the newest heartbeat sits at the start and the oldest fades at the top — a layout idea for AR timelines. - What it is: A spiral chandelier of hundreds of light bulbs hangs overhead; heartbeats read from visitors below travel up the spiral as flickering light. - Technique: Heart-rate sensors feed a controller that dims bulbs to each pulse and shifts the recordings along the spiral as new ones arrive. - Try it: Model an AR helix of glowing points above a table that records each participant's tap rhythm and shifts it upward as the next person taps. Twist: let people find their own earlier rhythm by looking up. #### Please Empty Your Pockets — Rafael Lozano-Hemmer (2010) - Video: https://www.youtube.com/watch?v=arK7V_jDlNI - Interaction: Tangible Objects, Projection, Shared & Social - Platform & tech: Projection, conveyor belt, scanner, projector, database - Idea: An object leaving behind its image turns a simple action into an archive of everyone who came before — a model for AR ghosts of past visitors. - What it is: Visitors place objects from their pockets on a conveyor belt; a camera scans them and when the real object comes out the other end, its image stays behind on the belt among thousands of earlier objects. - Technique: An overhead scanner photographs each object and a projector re-projects the stored images onto the moving belt in the same position, drawing from a database of about 600,000 items. - Try it: Photograph objects people put on a table, then leave their AR images in place after the real objects are removed, slowly drifting away like a conveyor. Twist: mix in images from yesterday's session so the table remembers. #### Pulse Index — Rafael Lozano-Hemmer (2010) - Video: https://www.youtube.com/watch?v=sdRWsCVAklQ - Interaction: Hands & Body, Information & UI, Shared & Social - Platform & tech: Projection, pulse sensor, microscope camera, projector - Idea: Showing each new person largest and letting older entries shrink into a crowd is a clear visual grammar for shared AR walls of contributions. - What it is: A visitor puts a finger into a sensor; a camera records their fingerprint and heartbeat, and the enlarged pulsing fingerprint joins a wall of thousands of earlier participants. - Technique: A microscope camera captures the fingerprint while a pulse sensor provides the rhythm, and software places the new recording at the top of a grid that shrinks older ones. - Try it: Build a shared AR wall where each visitor's selfie thumbnail pulses to their tapped rhythm, newest largest, older ones shrinking to the edges. Twist: let a visitor search the wall for themselves by walking along it. #### Sandbox (Relational Architecture 17) — Rafael Lozano-Hemmer (2010) - Video: https://www.youtube.com/watch?v=GotOBu_14fc - Interaction: Hands & Body, Projection, Shared & Social - Platform & tech: Projection, infrared cameras, high-power projectors - Idea: The hand you reach into a small sandbox is projected as a giant hand covering the whole beach. - What it is: On Santa Monica beach, tiny projections of beachgoers appear in two small sandboxes; when participants reach in, cameras relay their hands live to giant projectors that cover 8,000 m² of beach. - Technique: Infrared cameras capture people on the beach and hands over the sandboxes, cross-projecting each feed at opposite scales (tiny people in the box, giant hands on the beach). - Try it: Have one phone film hands on a table and another film people on the sports field, and project each feed into the other's space. Twist: let the tiny people and the giant hands 'touch' each other in the projection and trigger sounds. #### Voice Tunnel — Rafael Lozano-Hemmer (2013) - Video: https://www.youtube.com/watch?v=jmRnLUVt4kE - Interaction: Voice & Sound, Location & City, Shared & Social - Platform & tech: Desktop, spotlights, speakers, intercom, custom software - Idea: A voice that travels as light along a real corridor turns sound into something you can see move — a clear pattern for spatial audio visualisation in AR. - What it is: New York's Park Avenue Tunnel is closed to cars and lined with 300 theatre spotlights; people speak into an intercom and their voice becomes flickering light and sound that travels down the tunnel. - Technique: Each recording's amplitude modulates the brightness of one spotlight and 150 speakers along the tunnel; new recordings push older ones further along the queue. - Try it: Line a real hallway with AR lights whose brightness follows recorded voices, and make each new voice push the older ones one light farther away. Twist: let listeners walk to a light to hear that voice spatially. #### Level of Confidence — Rafael Lozano-Hemmer (2015) - Video: https://vimeo.com/953969845 - Interaction: Face, Information & UI - Platform & tech: Desktop, face recognition, OpenCV - Idea: Face recognition searches every visitor's face for the 43 missing students, turning surveillance into a memorial. - What it is: A face-recognition camera trained on the faces of the 43 disappeared Ayotzinapa students looks for them in each visitor's face and shows which student they most resemble and how confident the match is. - Technique: Eigenface/Fisherface and LBP face-recognition algorithms compare the live face against the students' portraits and display the best match with its confidence score. - Try it: Use a web face-recognition library (face-api.js) to compare visitors with a set of photos of historical figures and display the similarity scores. Twist: have the class discuss and choose a group worth "searching for" as the dataset. #### Zoom Pavilion (with Krzysztof Wodiczko) — Rafael Lozano-Hemmer (2015) - Video: https://www.youtube.com/watch?v=ENWBRsvn7qA - Interaction: Face, Projection, Shared & Social - Platform & tech: Projection, face detection, robotic zoom cameras - Idea: Surveillance cameras zoom in on how you relate to strangers and fill the whole room with it. - What it is: An immersive room of projections from 12 computerized surveillance cameras that detect visitors' faces and zoom in on them and on the spatial relationships between strangers. - Technique: Face-detection and tracking algorithms steer robotic zoom cameras and project their live feeds, recording distances between detected people. - Try it: Use a webcam to detect faces, automatically "digitally zoom" in on each one, and project the result onto a wall. Twist: compute the distance between two people and draw a line marking it in the image. #### Pulse Topology — Rafael Lozano-Hemmer (2021) - Video: https://www.youtube.com/watch?v=5Zy39kMf_3o - Interaction: Hands & Body, Spatial Mapping, Shared & Social - Platform & tech: Desktop, heart-rate sensors, LED bulbs - Idea: Hanging data at different heights makes a walkable terrain of light; AR can let people walk through data landscapes at body scale. - What it is: Thousands of small light bulbs hang at different heights to form a landscape of peaks and valleys that visitors walk through, each bulb flickering with a recorded heartbeat. - Technique: Heart-rate sensors at the edges record visitors' pulses, which are distributed across about 3,000 bulbs suspended to a sculpted height map. - Try it: Hang a field of virtual bulbs at heights from a terrain map across a room in AR and make each flicker to recorded taps; let viewers walk through at waist height. Twist: fill valleys with the newest data and raise older data into peaks. #### Shadow Tuner — Rafael Lozano-Hemmer (2025) - Video: https://vimeo.com/1067505877 - Interaction: Hands & Body, Projection, Location & City - Platform & tech: Projection, camera silhouette, spherical projection - Idea: Passers-by cast their shadows onto a giant Earth balloon and change how the planet looks. - What it is: A rotating, upside-down Earth is projected onto a giant spherical balloon; passers-by at a camera station cast their shadows onto the globe, reshaping how the planet appears. - Technique: A camera at a station captures the passer-by's silhouette and composites it into the projected (or LED) spherical Earth animation in real time. - Try it: Project a spinning Earth onto a white balloon or globe lamp, and use a camera silhouette to 'cast' a shadow onto it. Twist: the area covered by the shadow shows a real data point about that region. ### Russ Maschmeyer *Designer and product lead; ex-lead of Shopify's Spatial Commerce lab, Primer co-founder* Interaction designer who turned a hacked Kinect into a musical instrument for his SVA thesis, led search design at Facebook, co-founded the AR home-design app Primer, and ran Shopify's Spatial Commerce lab (2021-2024) before leading GenAI design at Meta and moving to Stripe in 2026. #### MOTIV — Russ Maschmeyer (2011) - Video: https://www.youtube.com/watch?v=nKl2Wjto4zI - Interaction: Hands & Body, Voice & Sound, Performance - Platform & tech: Desktop, Kinect, MIDI - Idea: Let the body, not an instrument, control the emotion of music. - What it is: Russ's SVA MFA thesis: a hacked Kinect follows a performer's body so that gestures shape the velocity and tempo of a playing song, while on-screen graphics mirror and guide the moves. - Technique: A Kinect depth camera tracks the performer's movement and maps it in real time to expression parameters such as note velocity and tempo of a multi-track MIDI sequence. - Try it: Map one tracked body feature (hand height or arm spread) to the volume and tempo of a looping track with a webcam pose tracker, then perform a 30-second piece. Twist: map two performers to different instruments so they must play together. #### AI Product Genie — Russ Maschmeyer, Shopify Spatial Commerce Team (2022) - Video: https://x.com/russ_maschmeyer/status/1569700294673702912 - Interaction: Voice & Sound, Drawing & Making - Platform & tech: Phone, Stable Diffusion, ARKit - Idea: Speak a wish, see the custom product in your room, buy it instantly. - What it is: Built right after Stable Diffusion came out: you speak a wish to a genie, it generates a custom product design such as a printed shirt or a plush toy, previews it in AR, and sends it to on-demand production. - Technique: Speech becomes a Stable Diffusion prompt, the generated image is applied as a texture to a 3D product template shown in AR, and print-on-demand fulfilment likely sits behind the buy button. - Try it: Build a pipeline where a spoken prompt generates an image (any text-to-image API) that is wrapped onto a 3D mug or T-shirt and placed in AR. Twist: let a second student remix the wish by voice before it is placed. #### Home Goods in AR (Shop app) — Russ Maschmeyer, Shopify Spatial Commerce Team (2022) - Video: https://x.com/russ_maschmeyer/status/1483577639356960769 - Interaction: Spatial Mapping, Information & UI - Platform & tech: Phone, ARKit, AR Quick Look, USDZ - Idea: Put AR one tap away inside everyday shopping instead of in a separate app. - What it is: After Shopify brought on the Primer team (whose app previewed paint, tile and wallpaper on real walls), they shipped a 'Visualize before you buy' section in the Shop app where shoppers drop furniture and lamps into their own room. - Technique: Likely built on ARKit and AR Quick Look with USDZ product models, so any listing with a 3D model opens a true-scale preview in the shopper's room. - Try it: Build a three-product mini store page where each product opens a 1:1 AR preview (model-viewer or Quick Look), then run a quick 'which would you buy' test with classmates. Twist: add a 'will it fit?' check that warns when the model is larger than a measured spot. #### Instructional AR — Russ Maschmeyer, Shopify Spatial Commerce Team (2022) - Video: https://x.com/russ_maschmeyer/status/1602706821403361282 - Interaction: Tangible Objects, Voice & Sound, Information & UI - Platform & tech: Headset, Meta Quest Pro, passthrough, Unity, voice commands - Idea: Replace a 15-page manual with instructions that live on the parts themselves. - What it is: Hands-free assembly with a Quest Pro: parts laid out on the floor are identified, wobbly guide lines show which pieces connect, tracked parts snap together with playful effects, and voice commands move between steps. - Technique: Quest Pro controllers attached to physical parts give inside-out 6DoF tracking of the pieces, so Unity can draw 3D connection guides in passthrough while voice commands advance the steps. - Try it: Pick a simple 3-5 step build (LEGO or a paper model) and make an AR guide that anchors each step's arrow or ghost part on the real pieces, with a voice command for 'next'. Twist: make the guide react when a step is done wrong. #### Retail Research Kit — Russ Maschmeyer, Shopify Spatial Commerce Team (2022) - Video: https://x.com/russ_maschmeyer/status/1592554373519142913 - Interaction: Gaze & Attention, Spatial Mapping, Information & UI - Platform & tech: Headset, Meta Quest Pro, eye tracking, Unity, Polycam - Idea: Turn a headset's eye tracking into affordable shopper research for small shops. - What it is: Volunteers wore a Quest Pro while shopping in Paxton Gate, a San Francisco curiosity shop; their gaze and touches were replayed on a 3D scan of the store as heatmaps so a small merchant can see what draws attention. - Technique: The shop was scanned with Polycam and aligned in Unity; Quest Pro eye-tracking rays and hand-tracking contacts are painted onto the mesh with a texture-painting shader to build 3D gaze and touch heatmaps. - Try it: Scan a classroom shelf, log where testers look or point (head direction in phone AR is fine), and paint the hits onto the scan as a heatmap. Twist: compare heatmaps for two different product arrangements. #### Reverse AR — Russ Maschmeyer, Shopify Spatial Commerce Team (2022) - Video: https://x.com/russ_maschmeyer/status/1562450155080597505 - Interaction: Portals & Worlds, Spatial Mapping, Shared & Social - Platform & tech: Phone, Unity, AR Foundation, LiDAR, RoomPlan, Polycam - Idea: Flip AR: keep the product real and make the room virtual. - What it is: Instead of placing a virtual sofa in a real room, a shopper sits on a real sofa in the store while the phone wraps a 3D scan of their own living room around it; two shoppers can share the view and spin the room with gestures. - Technique: The home is captured with RoomPlan or photogrammetry, and in the store the real furniture is cut out of the camera feed with LiDAR depth-based segmentation inside a bounding box, then composited into the virtual room in Unity with AR Foundation. - Try it: Scan a room at home with a free photogrammetry app, bring it to class, and composite a real chair (cut out with a depth or person mask) into it on a phone. Twist: place the real object in an impossible room, such as underwater or on the Moon. #### Space Eraser — Russ Maschmeyer, Shopify Spatial Commerce Team (2022) - Video: https://x.com/russ_maschmeyer/status/1544666221181734912 - Interaction: Spatial Mapping, Portals & Worlds - Platform & tech: Phone, ARKit, RoomPlan, LiDAR, ARWorldMap - Idea: Give your room a reset button before you shop for furniture. - What it is: An iPhone scans a room with Apple's RoomPlan, textures the scanned boxes with live camera pixels and lays this digital twin 1:1 over the room, so you can erase the sofa, the dog or everything and then swipe through new furniture sets. - Technique: RoomPlan's semantically labelled USDZ boxes are realigned to the room with ARWorldMap, textured from live camera frames plus LiDAR depth, and the hidden areas are likely filled with ML inpainting. - Try it: Scan one corner of the classroom (RoomPlan, Polycam or a hand-built box model), overlay it in AR, and let users tap an object to hide it behind a painted wall texture. Twist: instead of erasing, swap the object for a funny replacement of the same size. #### Spatial Admin — Russ Maschmeyer, Shopify Spatial Commerce Team (2022) - Video: https://x.com/russ_maschmeyer/status/1578022121883676673 - Interaction: Hands & Body, Drawing & Making, Information & UI - Platform & tech: Headset, Meta Quest Pro, Unity, Meta Interaction SDK, passthrough - Idea: Let AR glasses turn any table into a product-photo studio. - What it is: A Quest Pro prototype for merchants: open a virtual photo studio on your desk, arrange and light a 3D product, and capture product shots through a floating live-camera viewfinder controlled by hand pinches. - Technique: Built in Unity with Meta's Interaction SDK hand tracking in Quest Pro passthrough; the team tested several capture metaphors (instant camera, reaching into a photo, live camera preview) before choosing the live cam. - Try it: Prototype two different 'take a photo' gestures for a virtual camera in a headset or phone AR scene and have classmates rate which is faster and clearer. Twist: the photo must be taken with no buttons, only with the body. #### AI + AR Shopping Concept — Russ Maschmeyer, Shopify Spatial Commerce Team (2023) - Video: https://x.com/russ_maschmeyer/status/1640741787105984512 - Interaction: Information & UI, Voice & Sound, Tangible Objects - Platform & tech: Phone, Headset, ChatGPT, computer vision - Idea: Let AI answer shopping questions by lighting up the real world instead of a chat window. - What it is: A concept that takes the Shop app's ChatGPT-powered search into AR: in a real kitchen, the espresso machine and cups glow as the assistant responds, and a suggested lever espresso maker appears on the counter. - Technique: Likely a concept composite: a conversational AI interprets the request, object recognition picks out related items in the camera view, and glow shaders plus a placed 3D product visualise the answer. - Try it: Build a phone AR demo where a typed or spoken question ('what do I need for tea?') highlights two or three recognised objects on a table and places one suggested product. Twist: the AI may only light things up, with no text at all. #### Clueless Closet — Russ Maschmeyer, Shopify Spatial Commerce Team (2023) - Video: https://x.com/russ_maschmeyer/status/1650892732611858434 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Stable Diffusion, LoRA, ControlNet, Segment Anything - Idea: Movie-magic try-on: see yourself in any outfit combination instantly. - What it is: A real version of Cher's computerised closet from Clueless: photos of two fashionable friends and their clothes were used to teach an AI to render them wearing new outfit combinations in chosen poses and scenes. - Technique: A LoRA is trained on Stable Diffusion 1.5 for each person, face and garment, then ControlNet pose detection and Segment Anything masks guide the generation, with a lot of manual prompting and editing. - Try it: Build a webcam 'magic mirror' that segments the student's body and overlays a chosen garment image that follows their pose. Twist: let a classmate pick the outfit remotely from a phone. #### Immersive Product Story (WebXR concept) — Russ Maschmeyer, Shopify Spatial Commerce Team (2023) - Video: https://x.com/russ_maschmeyer/status/1693984617580187914 - Interaction: Portals & Worlds, Spatial Mapping - Platform & tech: Headset, Web, WebXR, Apple Vision Pro (concept) - Idea: Let a product bring its own world with it. - What it is: Starting from a tent on a web storefront, a full-size tent appears in the living room and the scene grows into an immersive camping experience, delivered through WebXR in Vision Pro's browser. - Technique: A concept for WebXR in Safari on Vision Pro: the storefront launches an immersive session that places the product at full scale and then replaces the room with a matching environment. - Try it: Make a WebXR AR scene where one product (lamp, tent or drink) changes the room's mood as lighting, sound and a surrounding environment fade in around it. Twist: the transition only starts when the user walks up to the product. #### Look, Pinch, Ask (Vision Pro concept) — Russ Maschmeyer, Shopify Spatial Commerce Team (2023) - Video: https://x.com/russ_maschmeyer/status/1696520326266634557 - Interaction: Gaze & Attention, Voice & Sound, Information & UI - Platform & tech: Headset, Apple Vision Pro (concept), conversational AI - Idea: Ask about the thing you are looking at, the moment you are curious. - What it is: A Vision Pro concept that pairs eye, hand and voice input with conversational AI: a shopper sitting on a velvet sofa looks, pinches and asks, and instant personalised product insights appear. - Technique: Likely a concept video: gaze selects the product, a pinch confirms, speech goes to a conversational AI, and the answer is shown as a floating panel next to the object. - Try it: Prototype a 'look and ask' interaction in phone AR: the object at the screen centre is selected, a tap records a question, and a short AI answer is pinned beside it. Twist: the answer must be spoken back in the object's own voice. #### Matrix Stockroom — Russ Maschmeyer, Shopify Spatial Commerce Team (2023) - Video: https://x.com/russ_maschmeyer/status/1613218237969494017 - Interaction: Voice & Sound, Information & UI, Portals & Worlds - Platform & tech: Headset, Meta Voice SDK, LLM, Handy, Blender - Idea: Show search results as an infinite room you walk through, then talk to narrow it down. - What it is: A VR concept inspired by the infinite gun rack in The Matrix: a search spawns endless shelves of products around you, voice commands narrow them down, and any item can be pulled out for AR try-on. - Technique: A concept film combining Meta's Voice SDK with a large language model as the filter; the hands were motion-captured on a Quest with the team's open-source Handy tool and rendered in Blender (the filtering pipeline is likely simulated). - Try it: Build a VR or phone-AR scene that spawns a grid of 50 objects from a search word, then add two spoken filters ('only red', 'smaller') that remove items. Twist: sort the room by a feeling ('cozy', 'loud') instead of a category. #### Product as Platform — Russ Maschmeyer, Shopify Spatial Commerce Team (2023) - Video: https://x.com/russ_maschmeyer/status/1622969682516123648 - Interaction: Tangible Objects, Gaze & Attention, Information & UI - Platform & tech: Headset, mixed reality passthrough, hand tracking, eye tracking - Idea: Every object you own could have an app living inside it. - What it is: A concept for AR glasses in which physical objects carry their own apps: pick up a film camera and its controls and tips appear anchored to it, selected by gaze-and-pinch or by virtual buttons on the object's hard surfaces. - Technique: Likely a passthrough prototype in which each product is recognised and tracked so a small UI stays attached to it, using pinch-and-gaze selection or virtual buttons backed by the object's real surfaces. - Try it: Choose one classroom object and design its 'app': anchor three AR controls to it with an image marker that only appear when the user touches or looks at it. Twist: give the object a personality so its app is quirky rather than useful. #### Pulling Products off the Web (Spatial Commerce) — Russ Maschmeyer, Shopify Spatial Commerce Team (2023) - Video: https://x.com/russ_maschmeyer/status/1691447651445940225 - Interaction: Hands & Body, Portals & Worlds - Platform & tech: Headset, Web, Apple Vision Pro (concept), visionOS, USDZ - Idea: The web page becomes a doorway that products can be pulled through. - What it is: A Vision Pro spatial-commerce concept: while browsing a store's web page, the shopper pinches a product and pulls a full-size 3D demo out of the page into the room. - Technique: A concept video likely simulating visionOS, where a 3D model embedded in the page is grabbed with a look-and-pinch gesture and dropped as a world-anchored object. - Try it: Make a web page with a 3D product that jumps into WebXR AR when the user drags it off the page. Twist: products can be pushed back into the page to 'return' them. #### Shopping with Friends (SharePlay concept) — Russ Maschmeyer, Shopify Spatial Commerce Team (2023) - Video: https://x.com/russ_maschmeyer/status/1692171419944165387 - Interaction: Shared & Social, Information & UI - Platform & tech: Headset, visionOS, SharePlay, FaceTime - Idea: Bring the fun of shopping with friends into shared AR. - What it is: A Vision Pro concept in which a friend joins via FaceTime as a floating video tile and both see the same miniature tent and campsite on the coffee table while deciding what to buy. - Technique: Likely built on visionOS SharePlay, which syncs the same 3D product scene between participants while the FaceTime call runs beside it. - Try it: Build a two-phone shared AR scene where both users see and rotate the same product and then vote on it. Twist: one user can only point and the other can only speak. #### Tracky — Russ Maschmeyer, Shopify Spatial Commerce Team (2023) - Video: https://x.com/russ_maschmeyer/status/1633899092861652994 - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Phone, Desktop, ARKit, LiDAR, Blender - Idea: Sketch AR ideas as believable videos without building the real app. - What it is: An open-source iOS app and Blender plugin: film a clip while the phone records camera pose, focal length, LiDAR depth, planes, anchors and a person mask, then import it into Blender to add AR effects that stay locked to the scene. - Technique: ARKit logs per-frame camera transforms, intrinsics, depth maps, segmentation masks and planes into a JSON-based .bren file, which the Blender add-on turns into an animated camera and aligned scene. - Try it: Record a 10-second phone clip with a camera-tracking tool (Tracky or Blender motion tracking), add one fake AR object in Blender, and render a concept video. Twist: the object must interact with something real in the shot, such as hiding behind it or bouncing on it. #### Waking Products (Vision Pro concept) — Russ Maschmeyer, Shopify Spatial Commerce Team (2023) - Video: https://x.com/russ_maschmeyer/status/1671189665867776000 - Interaction: Tangible Objects, Gaze & Attention, Information & UI - Platform & tech: Headset, Apple Vision Pro (concept), object tracking - Idea: Buying a physical product could also unlock a digital layer attached to it. - What it is: A spatial-computing concept released after Vision Pro was announced: a physical indoor herb garden on a kitchen counter is 'woken up' by the user, revealing a layer of digital functionality around the real product. - Technique: Likely a mixed-reality concept in which the product is recognised and a UI is anchored to it, triggered by gaze and pinch. - Try it: Pick a household product and design its 'awake' AR state: an image marker or object tracker opens a panel with one useful function and one playful one. Twist: the product falls asleep again when it is ignored. #### WonkaVision — Russ Maschmeyer, Shopify Spatial Commerce Team (2023) - Video: https://x.com/russ_maschmeyer/status/1631350356058726411 - Interaction: Face, Perception & Effects, Gaze & Attention - Platform & tech: Web, MediaPipe Face Mesh, WebAssembly, WebGL - Idea: Turn a flat screen into a window with nothing but a webcam. - What it is: Johnny Lee's 2007 Wii head-tracking trick revived in Chrome with a normal webcam: as you move your head, a miniature 3D storefront shifts in perspective as if behind a window, and objects appear to pop out of the screen. - Technique: MediaPipe Face Mesh iris landmarks estimate the eye's 3D position from the near-constant iris diameter (about 11.7 mm), and the scene is rendered with an off-axis perspective projection from that eye, all in the browser via WebAssembly. - Try it: Use a browser face tracker to move a three.js camera with the viewer's head so a box scene behind the screen feels deep. Twist: make one object escape through the screen when the viewer leans in. ### Shopify Spatial Commerce Team *Shopify's AR/VR and spatial-commerce prototyping lab* A rapid-prototyping crew of engineers, designers and artists at Shopify (growing out of the earlier Shopify AR/VR team) that explored AR, VR and AI for shopping and published its prototypes and open-source tools such as Tracky and Handy. #### 3D & AR on Shopify: Stroller Demo — Shopify Spatial Commerce Team, Daniel Beauchamp (Pushmatrix) (2020) - Video: https://www.youtube.com/watch?v=ntX0cOr4r08 - Interaction: Spatial Mapping, Information & UI - Platform & tech: Phone, Web, AR Quick Look, USDZ, model-viewer - Idea: Let shoppers check a bulky product's real size in the place they will use it. - What it is: A demo shown in Daniel Beauchamp's ShipIt! talk: a double stroller on a Shopify product page can be spun in 3D and then placed at real size on the street in AR. - Technique: Merchants upload a 3D model with the product; the storefront shows it in a web 3D viewer and hands a USDZ file to AR Quick Look on iPhone for true-scale placement. - Try it: Pick one bulky object (bike, stroller, instrument case), get or make its 3D model, and publish a page with a 3D viewer and an AR button; test it where the object would really be used. Twist: add a path animation showing it moving through a doorway. ### Will Pappenheimer *Artist; co-founder of Manifest.AR; professor at Pace University* Artist and founding member of the Manifest.AR collective (2010) who has made location-based AR interventions for over a decade — dancing glitch figures, falling voxels that 'paint' sites, and AR at the Venice Biennale. #### Occupy Wall Street Bufo Colony — Will Pappenheimer (2011) - Video: https://www.youtube.com/watch?v=-vVowQQbq6w - Interaction: Location & City, Information & UI - Platform & tech: Phone, Layar, GPS - Idea: Absurd creatures carry hard data into a protest space. - What it is: A colony of AR toads breaches the barricades at the NYSE, each carrying a pie chart of wealth distribution, and clumps over the Zuccotti Park drum circle around a giant 'uber toad'. - Technique: Animated toad models with attached chart billboards are geolocated at Wall Street and Zuccotti Park in a mobile AR browser, part of the Virta-Flaneurazine series. - Try it: Pick a statistic about your campus and attach it to a swarm of small AR creatures that crowd around the relevant building. Twist: the swarm grows as the number gets worse. #### Parking Lot Decorator — Will Pappenheimer (2012) - Video: https://www.youtube.com/watch?v=mdrzvbW3hbc - Interaction: Location & City, Drawing & Making - Platform & tech: Phone, Layar, GPS - Idea: Let anyone repaint the most boring corporate space. - What it is: For the ZERO1 Biennial, the huge, dull parking lots around Silicon Valley tech campuses can be decorated with fantastic colour through a phone. - Technique: Colourful patterns and objects are geolocated over large parking-lot areas in a mobile AR browser, with viewers choosing which decorations appear. - Try it: Use plane detection to let users paint giant patterns onto a parking lot or empty plaza in AR. Twist: the patterns are generated from the car colours the camera sees. #### Skywrite AR: Ceci n'est pas une pipe — Will Pappenheimer (2012) - Video: https://www.youtube.com/watch?v=-1JWsNoqzbA - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, Layar, GPS - Idea: Skywriting without a plane: a message in the sky only phones can read. - What it is: Over the Louvre and Notre Dame, Magritte's phrase 'Ceci n'est pas une pipe' is written across the sky in AR smoke letters. - Technique: Text is rendered as smoky 3D letters and placed high above landmark coordinates as geolocated points of interest in a mobile AR browser. - Try it: Write a one-line statement as AR skywriting above a campus building using sky segmentation or a far-anchored object. Twist: the letters slowly drift and dissolve like real smoke. #### Vision Funnels — Will Pappenheimer (2012) - Video: https://www.youtube.com/watch?v=kS4STKdk90s - Interaction: Perception & Effects, Portals & Worlds, Location & City - Platform & tech: Phone, Layar, iPad - Idea: A frame within a frame turns any wall into a telescope. - What it is: At LACMA, nested AR frames open like funnels in the museum's walls and plazas, pulling the viewer's gaze into tunnel-like views of the architecture. - Technique: Concentric rectangular frames are likely placed as geolocated 3D augments at fixed spots around the museum, viewed on tablets as part of the Project-O-rators series. - Try it: Build a recursive AR frame (a frame containing a render of itself) on a wall with image tracking. Twist: the funnel shows the live camera feed from behind the wall. #### Sky Petition City — Will Pappenheimer (2013) - Video: https://www.youtube.com/watch?v=5UvA9DjWRRA - Interaction: Location & City, Shared & Social, Information & UI - Platform & tech: Phone, mobile AR app, GPS - Idea: Turn online petitions into skywriting over the institutions they address. - What it is: Citizens' petitions are skywritten by a virtual plane in the air above centres of power such as the Capitol and the Pentagon. - Technique: With developer Zachary Brady, submitted petition text is rendered as a trail behind an animated plane geolocated above the target building in a mobile AR app. - Try it: Collect short requests from classmates via a form and make an AR plane write them above the administration building. Twist: only petitions with ten supporters get written. #### MyMirrorCity — Will Pappenheimer (2014) - Video: https://www.youtube.com/watch?v=H7M6ogeE1NA - Interaction: Location & City, Shared & Social, Face - Platform & tech: Phone, mobile AR app, GPS - Idea: Selfies become the new civic frescoes. - What it is: Visitors to Basel's Marktplatz decorate the square's buildings with their selfies, continuing the fresco tradition of the painted town hall. - Technique: User photos are uploaded and mapped as image panels onto geolocated building facades in a mobile AR app for Virtuale Switzerland. - Try it: Let visitors take a selfie and place it as an AR mural on a campus facade using image or plane tracking. Twist: each new selfie pushes the oldest one off the wall. #### Drawing Constellation — Will Pappenheimer (2016) - Video: https://www.youtube.com/watch?v=4oSLG7bBZBk - Interaction: Drawing & Making, Shared & Social, Location & City - Platform & tech: Phone, Layar, GPS, web server - Idea: Everyone's drawings orbit together as one communal sculpture in the air. - What it is: A participatory AR drawing installation: visitors draw on phones or tablets, and each saved drawing joins a rotating galaxy of 20 or more drawings floating over a GPS-anchored site, later reappearing at other places such as a Brooklyn park. - Technique: A modified Layar app uploads each drawing to a server that places it at the centre of an animated 3D constellation at a GPS location, pushing older drawings outward (made with Zachary Brady). - Try it: Build a WebXR page where each student draws on a 2D canvas, saves, and sees their drawing added to a slowly rotating ring of everyone's drawings anchored above a table. Twist: newest drawings orbit fastest and slow down as they age. #### Ascension of Cod — Will Pappenheimer (2017) - Video: https://www.youtube.com/watch?v=17TmoUSiKdw - Interaction: Location & City, Perception & Effects, Information & UI - Platform & tech: Phone, mobile AR app, iPad - Idea: An ecological collapse rendered as a heavenly ascension. - What it is: At Salem Maritime National Historic Site, a great school of cod rises into the sky, a monument to the fish that built New England's economy and then collapsed. - Technique: Animated flocking cod models are geolocated above the historic wharf and viewed live on iPads for Boston Cyberarts' The Augmented Landscape. - Try it: Make a flocking (boids) AR swarm of a species that has declined locally and let it rise above a campus landmark. Twist: the swarm size matches historical population data. #### Outside the Wall — Will Pappenheimer (2018) - Video: https://www.youtube.com/watch?v=PSDzOSVX9NA - Interaction: Portals & Worlds, Tangible Objects, Spatial Mapping - Platform & tech: Phone, iOS AR app, iPad - Idea: Through a tablet, the wall at home breaks open and the outside world pours in. - What it is: A mixed-reality installation of a table, chair and domestic objects viewed through an iPad, where virtual elements break out of the domestic wall into the space. - Technique: An iOS AR app registers virtual content to a physical arrangement of furniture and objects so the scene extends beyond the real wall. - Try it: Arrange a table and a few everyday objects, and use AR to 'break open' the wall and show another world. Twist: when the objects are moved, the view beyond the wall changes too. #### Pool Scratch — Will Pappenheimer (2018) - Video: https://www.youtube.com/watch?v=FvRLh9bwRd8 - Interaction: Performance, Location & City, Hands & Body - Platform & tech: Phone, motion capture, mobile AR app - Idea: Drop a game character's dance into a museum that never invited it. - What it is: An unofficial AR installation at the Guggenheim: a floating pool with a medieval Norse game character performing a motion-captured dance by Freya Björg Olafson. - Technique: A motion-capture dance is retargeted onto a game character inside a 3D pool model that is anchored in the rotunda in a mobile AR app from the ACTIVATAR show. - Try it: Record a classmate's dance with a free mocap tool (e.g. phone video to animation), retarget it onto a game avatar and place it in AR in an unexpected space. Twist: the avatar dances only when music is detected. #### CarDrop-V — Will Pappenheimer (2019) - Video: https://www.youtube.com/watch?v=4qRWk2jo0k0 - Interaction: Spatial Mapping, Perception & Effects, Location & City - Platform & tech: Phone, mobile AR, physics animation - Idea: A virtual car falls from the sky and smashes into the real ground in front of you in slow motion. - What it is: Phone viewers watch a highly detailed virtual car fall from the sky and smash into the real ground in slow motion. - Technique: Mobile AR with ground-plane detection triggers a physics-driven slow-motion drop and impact animation of a car model. - Try it: Use Reality Composer's physics to drop a large object from the air onto the ground in slow motion. Twist: the viewer taps to choose the landing spot, and it leaves a permanent crater. #### Awkward Instance — Will Pappenheimer (2021) - Video: https://www.youtube.com/watch?v=GkQRj35QfZw - Interaction: Location & City, Perception & Effects, Hands & Body - Platform & tech: Phone, mobile AR, motion capture - Idea: Mismatched human skins and motion-capture data become dancing glitch ghosts. - What it is: Site-based AR 'paranormal apparitions' — glitchy dancing human forms made from stolen skins and mismatched motion-capture — appear in outdoor places. - Technique: Rigged 3D figures are driven by deliberately mismatched motion-capture clips and placed at sites with mobile AR. - Try it: Download characters and animations from Mixamo, deliberately apply motions to characters with the wrong proportions, and place them in AR. Twist: mix motions recorded by the whole class onto a single character. #### Repairs — Will Pappenheimer (2021) - Video: https://www.youtube.com/watch?v=keEhVfejlC4 - Interaction: Tangible Objects, Performance - Platform & tech: Phone, ARKit, motion capture - Idea: Virtual bodies tending to real, broken household things. - What it is: A mixed-reality installation where domestic objects on a shelving unit are 'repaired' by AR dancers, motion-captured performances by Freya Björg Olafson viewed on iPad. - Technique: An iOS AR app anchors motion-captured dancers to physical objects on a formica shelf, likely using image or object tracking. - Try it: Bring a broken object from home, scan or image-track it, and attach a tiny AR character that performs a 'repair' ritual on it. Twist: the ritual differs for each object's story. #### Starry Interpose (with Sarah Slifer Swift) — Will Pappenheimer (2021) - Video: https://www.youtube.com/watch?v=qcuWWhc2M_I - Interaction: Performance, Location & City, Perception & Effects - Platform & tech: Phone, mobile AR - Idea: Dancers perform a mixed-reality dance together with AR artworks and sculpture poses. - What it is: A dance performance at the Manship Artists Residency in which live choreography interacts with Pappenheimer's AR artwork and poses from Paul Manship's sculptures. - Technique: Mobile AR content is placed at the performance site and filmed through devices while dancers enact sculpture poses around the virtual objects. - Try it: Pick a sculpture on campus, choreograph a dance that imitates its pose, and film it with virtual elements placed beside it in phone AR. Twist: the virtual elements on the audience's phones copy the dancers' movements. #### Colorfield Weather App Series — Will Pappenheimer (2022) - Video: https://www.youtube.com/watch?v=FedxwbR4Z_0 - Interaction: Spatial Mapping, Drawing & Making, Location & City - Platform & tech: Phone, mobile AR, voxel physics - Idea: Colored AR blocks fall from the sky and paint color-field paintings across buildings and terrain. - What it is: Virtual AR voxels fall from the sky or expand across a site, leaving trails that 'paint' colour-field paintings onto the environment's topography and architecture. - Technique: AR voxel particles with gravity and forces collide with detected surfaces and leave persistent coloured trails. - Try it: Use Unity AR Foundation or web AR to drop colored blocks that leave color on the floor plane. Twist: weather data such as temperature and wind speed decides the colors and the direction of the fall. #### Scorched Earth Policy (US Pavilion, Venice Biennale) — Will Pappenheimer (2026) - Video: https://www.youtube.com/watch?v=unvGcSgfaeM - Interaction: Location & City, Perception & Effects, Portals & Worlds - Platform & tech: Phone, geolocated AR - Idea: AR makes a national pavilion dry up, burn and fall apart before visitors' eyes. - What it is: An uninvited AR intervention at the 2026 Venice Biennale US Pavilion in which the pavilion appears to dry up and disintegrate into a burning wasteland around visitors. - Technique: Geolocated mobile AR overlays disintegration and fire effects registered to the pavilion architecture, recorded as screen captures. - Try it: Choose a campus building and overlay AR effects showing its weathering or climate damage '100 years from now'. Twist: let viewers choose between 'repair' and 'let it go' and see the two different outcomes. ### Zach Lieberman *Artist; co-founder of openFrameworks and the School for Poetic Computation (SFPC); MIT Media Lab* New York artist and educator who builds experimental drawing, animation and computer-vision tools with code. Co-creator of openFrameworks, co-founder of SFPC and YesYesNo, and runs the Future Sketches group at the MIT Media Lab. #### Drawn — Zach Lieberman (2005) - Video: https://vimeo.com/4732884 - Interaction: Drawing & Making, Hands & Body, Tangible Objects - Platform & tech: Projection, Desktop, computer vision, openFrameworks - Idea: A camera 'lifts' ink off the paper, so hand-drawn shapes come alive and can be nudged by hand. - What it is: Ink drawn by hand on paper is lifted off the page by software that alters the live video signal, so painted forms appear to come alive and can be pushed around by the hand. - Technique: Background subtraction and thresholding segment dark ink blobs from an overhead camera of the paper, converting them into physics bodies that are rendered back over the live video, with the hand tracked as a separate occluding mask. - Try it: Film a sheet of white paper from above, threshold the ink into contours in openFrameworks or p5.js, and turn them into objects that fall and can be pushed by hand. Twist: let stroke thickness decide whether an ink shape floats away or sinks. #### The EyeWriter — Zach Lieberman, Theo Watson, Chris Sugrue (2009) - Video: https://www.youtube.com/watch?v=84H-xLrLvvk - Interaction: Gaze & Attention, Drawing & Making, Projection - Platform & tech: Projection, Wearable, eye tracking, openFrameworks - Idea: Graffiti drawn with the eyes: a paralyzed graffiti artist paints by eye tracking, and his work is projected live onto city buildings. - What it is: A low-cost eye tracker that let paralyzed graffiti artist TEMPT1 draw tags with his eyes; the tags were projected live onto buildings across Los Angeles. - Technique: An IR-illuminated camera on glasses tracks the dark pupil and corneal glint, and a calibration grid maps the pupil-glint vector to screen coordinates for drawing, with dwell time used as a click. - Try it: Use WebGazer.js or MediaPipe iris landmarks to build a web page for drawing with your eyes only, using a one-second gaze dwell to put the pen down or lift it, then project the result on a wall. Twist: forbid one classmate from using their hands and have them sign their name with their eyes. #### Face projection test — Zach Lieberman, Daito Manabe / Rhizomatiks (2010) - Video: https://www.youtube.com/watch?v=o-Ke6mu-F-c - Interaction: Face, Projection - Platform & tech: Projection, openFrameworks, FaceTracker - Idea: Project graphics straight onto a moving face so the face becomes a live screen. - What it is: Real-time projection mapping onto a moving face: a face tracker drives graphics projected directly onto the performer's skin. - Technique: A face tracker on a camera co-located with the projector detects landmarks, and a camera-projector homography (calibrated once) warps the rendered graphics so they land on the moving face. - Try it: Place a projector and camera side by side, detect a face with MediaPipe Face Mesh, and use four-point calibration to project graphics onto a classmate's face (mind the brightness and eye safety). Twist: project only dynamic textures that 'run down' the face, telling a short piece about emotion. #### Night Lights — Zach Lieberman (2010) - Video: https://vimeo.com/8525186 - Interaction: Hands & Body, Projection, Location & City - Platform & tech: Projection, openFrameworks, computer vision - Idea: Turn a whole ferry building into a magnifier for the body, blowing up silhouettes and gestures five storeys high. - What it is: YesYesNo turned the five-storey Auckland Ferry Building into an interactive playground: visitors' silhouettes, hand gestures and phones controlled projections amplified across the facade. - Technique: Cameras segment visitors' silhouettes and hand gestures (background subtraction plus contour analysis), and phone input is received over the network, all mapped to graphics projection-mapped onto the building facade. - Try it: Use TouchDesigner or p5.js to enlarge people's silhouettes from a camera and project them onto classroom-building windows or a hallway wall, with a wave of the hand sending light sweeping across. Twist: add phone web control so people far away can also 'switch on the lights' on the wall. #### Chase No Face / BELL — Zach Lieberman (2011) - Video: https://vimeo.com/26649425 - Interaction: Face, Projection, Performance - Platform & tech: Projection, Kinect, FaceTracker, openFrameworks - Idea: Every graphic on the faces in this music video was projected live, with no post-production. - What it is: A music video in which graphics are projected onto singer BELL's face in real time with a hacked Kinect, face tracker and LED projector, with no post-production. - Technique: A Kinect provides depth for head pose while a 2D FaceTracker fits landmarks, and a calibrated projector renders textures onto the singer's face in real time; the camera-projector pair is registered so no post-production is needed. - Try it: Use MediaPipe Face Mesh and a projector to make a live face-projection music video for a 30-second song, shot in one continuous take. Twist: switch to a different face "skin" for each line of lyrics, with the style decided by team members drawing lots. #### Reflection Study (interactive edition) — Zach Lieberman (2015) - Video: https://vimeo.com/159142972 - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Projection, openFrameworks, computer vision - Idea: Move acrylic pieces on a light table and the software bounces simulated light off them to draw shapes and letters. - What it is: A light table where visitors move laser-cut acrylic pieces; the software simulates 2D light bouncing off the physical shapes to create typography and graphic forms. - Technique: A camera under or over the light table extracts the contours of acrylic pieces, and a 2D ray-tracing simulation bounces light rays off those polygon edges, projected back onto the table. - Try it: Shoot a few pieces of colored card from above, extract their outlines in p5.js, cast rays from a light source that bounce off the card edges, and project the result back onto the table. Twist: design a set of card shapes so the light rays end up spelling your initials. #### Hag-Seed interactive — Zach Lieberman (2016) - Video: https://vimeo.com/222236331 - Interaction: Hands & Body, Projection - Platform & tech: Projection, openFrameworks - Idea: Turn the words of a novel into a projected scene you play with your body. - What it is: An installation with Margaret Atwood for the London Literature Festival: text from her novel Hag-Seed becomes projected scenes that visitors play with using their bodies. - Technique: Camera or depth-based body segmentation gives silhouettes and motion, which are used as collision shapes and forces for animated text particles in a projected scene. - Try it: Choose a poem, turn each word into a physics particle on a projection wall, and use the body outline from MediaPipe Selfie Segmentation to push the words apart or catch them. Twist: the lines fall into order only when two people work together to form a "container" with their bodies. #### Más Que la Cara — Zach Lieberman, Molmol Kuo (2016) - Video: https://vimeo.com/211271693 - Interaction: Face, Perception & Effects - Platform & tech: Desktop, openFrameworks, face tracking, Paper.js - Idea: "More than a face": a magic mirror in public space that turns passers-by's faces into generative masks that follow their expressions. - What it is: A public installation in downtown Houston that tracks passers-by's faces and overlays dozens of generative, illustrated masks that move with their eyebrows, lips and head. - Technique: A 2D facial landmark tracker (FaceTracker-style) fits dozens of points to each face on a live camera, and vector-illustrated mask parts are skinned to eyebrow, lip and head landmarks so they deform with expression. - Try it: Draw a mask made only of geometric lines with MediaPipe Face Mesh and p5.js, with eyebrow and mouth landmarks driving the shapes. Twist: give each person a random new mask as soon as they come close, with 10 styles to "swap faces" for passers-by. #### Audio in AR space — Zach Lieberman (2017) - Video: https://vimeo.com/290238447 - Interaction: Voice & Sound, Spatial Mapping - Platform & tech: Phone, ARKit, openFrameworks - Idea: Record sound into space: make noise as you walk to leave sound shapes, then walk back to play them in reverse. - What it is: Every sound Zach makes is recorded at the phone's position and drawn as a white blob in the air; walking back through the trail replays the sounds, as if scrubbing a record through space. - Technique: ARKit world tracking logs the phone's 6-DoF pose each frame alongside microphone buffers, so each audio chunk is stored at a 3D position and replayed when the camera comes back within a distance threshold of that point. - Try it: Record audio as you walk with Unity AR Foundation or WebXR, placing a small sphere scaled by volume at the phone's position every 0.5 seconds and saving the matching audio clip, then play each clip when you approach its sphere. Twist: keep only the ambient sound from moments when you are silent, making a "path of other people's voices." #### Weird Type — Zach Lieberman, Molmol Kuo (2018) - Video: https://www.youtube.com/watch?v=UzNCGqE9MhE - Interaction: Drawing & Making, Voice & Sound - Platform & tech: Phone, ARKit, openFrameworks - Idea: Use letters as a brush to write in real space, walk around them and warp them with sound. - What it is: An iOS AR app for writing and painting with animated typography in 3D space; type can follow your path, react to sound, explode, and be walked around. - Technique: ARKit camera pose supplies a 3D path that glyph outlines are laid along, and the glyph vertices are animated by per-frame noise, time, and microphone amplitude. - Try it: In three.js WebXR or Unity, place the letters of a sentence one by one in space along the phone's path, and make them shake or burst with microphone volume. Twist: choose a sentence that only reads correctly if you walk backward from a particular spot. #### Hide yourself — Zach Lieberman, Molmol Kuo (2019) - Video: https://x.com/zachlieberman/status/1142495123407855617 - Interaction: Hands & Body, Portals & Worlds, Perception & Effects - Platform & tech: Phone, ARKit, people occlusion, openFrameworks - Idea: Use AR to hide a person from the frame, making the body vanish into the real scene. - What it is: An ARKit test with Molmol Kuo, made as ARKit 3 introduced people segmentation, playing with hiding a person inside the AR scene around them. - Technique: ARKit 3 people segmentation with depth produces a per-pixel human matte, which is used to occlude or replace the person with the reconstructed background behind them. - Try it: Use AR Foundation's Human Segmentation (or MediaPipe segmentation on the web) to replace people in the frame with a pre-shot background for an 'invisibility trick'. Twist: instead of making the person vanish completely, leave only their shadow or clothes, and tell a short story about presence. #### Weird Cuts — Zach Lieberman, Molmol Kuo (2019) - Video: https://www.youtube.com/watch?v=fU0iFi0PXxI - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Phone, ARCore, ARKit, openFrameworks - Idea: Collage camera 'paper cutouts' in space, like making a physical collage in mid-air. - What it is: An AR collage tool: you cut shapes out of the camera image and place the photographic cut-outs as floating layers in space to build assemblages. - Technique: A user-drawn polygon masks the current camera frame into a textured cut-out, which is placed as a quad at the phone's pose (or a fixed distance in front of it) using ARKit/ARCore world tracking. - Try it: Make AR scissors: circle a patch of the camera image on screen with your finger, turn it into a textured sliver that stays in mid-air, and build a spatial collage of at least five pieces. Twist: you may only cut one color or material, and must assemble a 'campus animal'. #### AR wallpaper lenses for Snap — Zach Lieberman (2020) - Video: https://x.com/zachlieberman/status/1334497448979521549 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, Lens Studio, Snapchat - Idea: AR is the new wallpaper: cover a room's walls and surfaces with generative patterns. - What it is: Sketches for Zach's Snapchat lenses that wrap generative patterns across the walls and surfaces of a room like living wallpaper. - Technique: Lens Studio world mesh / plane detection supplies wall surfaces, and a generative shader with world-space UVs tiles animated patterns across them. - Try it: Lay a breathing procedural pattern over walls using plane detection in Lens Studio or AR Foundation, with a shader that uses world coordinates as UVs. Twist: the pattern changes with the room's ambient light or sound, turning a dorm wall into mood wallpaper. #### Weird Type skate trails — Zach Lieberman, Molmol Kuo (2020) - Video: https://x.com/zachlieberman/status/1271947807709069317 - Interaction: Drawing & Making, Hands & Body - Platform & tech: Phone, ARKit, Weird Type - Idea: Someone else takes an AR text tool and finds a new trick: drawing a skateboard's path with words. - What it is: Skateboarder Christopher Ledwig uses the Weird Type app to leave a trail of text following his skateboard's motion through space (shared by Zach). - Technique: The app samples the phone's world-tracked pose each frame and emits typographic glyphs along that path, so moving the camera draws text in 3D. - Try it: Make a small tool that leaves a trail of text in the air as you move your phone, then ask classmates to use it while running, jumping or skateboarding. Twist: make the text content change with speed. #### Poem World (Snap Spectacles) — Zach Lieberman, Shantell Martin (2021) - Video: https://x.com/zachlieberman/status/1408879174429949953 - Interaction: Drawing & Making, Location & City - Platform & tech: Headset, Snap Spectacles, Lens Studio - Idea: A poem you walk into: the words are laid out in space, and reading means walking. - What it is: A short poem by Shantell Martin laid out as lines of text in space that you read by walking through them, recorded through Snap's AR Spectacles. - Technique: World-tracked text meshes are placed along a path in Lens Studio; the reading order is set by where the wearer walks rather than by scrolling. - Try it: Break a short poem into words and phrases and place them in AR along a corridor or across a sports field, so reading becomes a walking route. Twist: some words appear only when you stop. #### Spark AR sketches (retrospective) — Zach Lieberman (2024) - Video: https://x.com/zachlieberman/status/1828438368734040339 - Interaction: Face, Perception & Effects - Platform & tech: Phone, Spark AR - Idea: A look back at a series of instantly shareable face and world effects made in Spark AR. - What it is: A reel of the face and world effects Zach made for Instagram's Spark AR before the platform shut down: generative type, face distortions and drawn trails. - Technique: Face-mesh tracking and segmentation in Spark AR drive shader-based distortions and generative graphics that anyone can open instantly from a link. - Try it: Make a face effect in Lens Studio or Effect House that people understand at a glance, and send it to three people to try within 10 minutes. Twist: make the effect appear only when two people are in the frame together. ### Laan Labs (Chris Laan) *Independent computer-vision & AR studio (AirMeasure, 3d Scanner App)* Small independent lab run by Chris Laan and Jason Laan that ships computer-vision and AR apps such as AirMeasure, 3d Scanner App and AirDraw. #### ARKit 3D Point Cloud House Walk-through — Laan Labs (Chris Laan) (2017) - Video: https://www.youtube.com/watch?v=Pb4uv4FSWKI - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, ARKit - Idea: Record every feature point ARKit sees, and a walk around the house grows into a point cloud. - What it is: Logs every raw ARKit feature point while walking through a house, revealing the building as a sparse point cloud with the camera path drawn in green. - Technique: Every frame's ARKit raw feature points and camera pose are accumulated into a persistent point buffer, then rendered as a point cloud with the camera trajectory as a line. - Try it: Accumulate all feature points with AR Foundation's ARPointCloudManager while recording the camera path, then shrink the point cloud onto a table to inspect it after one lap. Twist: color the points by capture time to reveal the walking order. #### ARSpray – AR Spray Paint — Laan Labs (Chris Laan) (2017) - Video: https://www.youtube.com/watch?v=xlvaMxTfwBg - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Phone, ARKit - Idea: Your phone becomes a spray can for graffiti on real walls. - What it is: Spray-paint graffiti onto real walls and floors with the phone as a virtual spray can. - Technique: Raycasts from the phone onto detected wall and floor planes stamp paint decals (or write into a surface texture) at the hit points, with spread and opacity based on distance. - Try it: Detect walls with vertical plane detection and, while the screen is held, keep stamping round decals at the ray hit point to create a spray-paint effect. Twist: the farther you are from the wall, the wider and fainter the spray. #### AirMeasure – AR Tape Measure — Laan Labs (Chris Laan) (2017) - Video: https://www.youtube.com/watch?v=z7DYC_zbZCM - Interaction: Spatial Mapping, Information & UI - Platform & tech: Phone, ARKit - Idea: An AR tape measure that sizes up the real world with two taps. - What it is: An early AR tape measure: tap two points on real surfaces to measure distance, later extended with floor plans, trajectories and building heights. - Technique: Two raycasts against detected planes or feature points give world-space hit positions, and the Euclidean distance between them is shown as a labeled line. - Try it: Tap twice with plane raycasts to get two points, draw a line between them, and show the distance in centimeters. Twist: convert the distance into "how many bananas long" or another playful unit. #### Avocado Toast with ARKit — Laan Labs (Chris Laan) (2017) - Video: https://www.youtube.com/watch?v=oikVikS2Ge0 - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Phone, ARKit, computer vision - Idea: Recognize a real avocado and turn it into avocado toast. - What it is: A small computer-vision and ARKit test that detects a real avocado and replaces it with avocado toast. - Technique: An object detector (likely CoreML/Vision) finds the avocado in the 2D frame, and a raycast from its bounding-box center to the table plane places a 3D toast model at that spot. - Try it: Find a piece of fruit on the table with MediaPipe Object Detection or Lens Studio object recognition, and place a replacement model at its position. Twist: make the replacement the fruit's "future form," such as rotten or turned into juice. #### Drawing in 3D with ARKit — Laan Labs (Chris Laan) (2017) - Video: https://www.youtube.com/watch?v=bTHaEg-bnaU - Interaction: Drawing & Making - Platform & tech: Phone, ARKit - Idea: Use your phone as a brush to paint 3D strokes in the air of your room. - What it is: One of the first ARKit demos: a Tilt Brush-style prototype in which moving the phone leaves glowing strokes hanging in the room. - Technique: The camera pose from world tracking is sampled each frame at a point slightly in front of the lens, and these points are extruded into a line or tube mesh that stays fixed in world space. - Try it: In AR Foundation, add the point 20 cm in front of the camera to a LineRenderer while the screen is held, drawing light trails that hang in the air. Twist: make the line color change with movement speed or sound volume. #### Hiding Stuff with ARKit — Laan Labs (Chris Laan) (2017) - Video: https://www.youtube.com/watch?v=E0B-lcm0i1M - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Phone, ARKit - Idea: Cover an object with live textures of its surroundings to make it invisible. - What it is: Live textures captured from the surroundings are laid over real objects so they seem to vanish from the scene. - Technique: Camera frames of the surrounding surface are captured and projected onto proxy geometry covering the object (a diminished-reality inpainting trick), making it look like background. - Try it: First take a photo of the tabletop without the object, then project that photo onto a plane covering the object to make it "invisible." Twist: hide only half of it, as if it had been sliced off with a knife. #### Magic Floating iPhone — Laan Labs (Chris Laan) (2017) - Video: https://www.youtube.com/watch?v=1msfo48LN-o - Interaction: Perception & Effects, Tangible Objects, Portals & Worlds - Platform & tech: Phone, ARKit - Idea: Swap the real phone on the table for a virtual copy and make it levitate. - What it is: A real iPhone on the table is swapped for a virtual copy that lifts off and levitates, with particles in a second take. - Technique: A virtual iPhone model is placed exactly over the real one, while the real object is hidden by a background patch (likely a captured texture of the table), so the copy appears to lift off. - Try it: Photograph an empty part of the table and use it as a texture to cover a small real object, then place its 3D model in the same spot and let it slowly rise. Twist: make the object morph or shatter once it is in the air. #### Parallel Parking with AR — Laan Labs (Chris Laan) (2017) - Video: https://www.youtube.com/watch?v=XyrBtaeVvTs - Interaction: Location & City, Information & UI, Spatial Mapping - Platform & tech: Phone, ARKit - Idea: Measure the parking space from the curb first, then let AR guide your parallel parking. - What it is: A prototype AR guide that measures a parking spot from the sidewalk and shows whether and how the car will fit. - Technique: The user marks the corners of the parking gap by raycasting to the ground plane, and the measured length is compared to a car-sized bounding box shown in place. - Try it: Tap two points on the ground to measure the length of a gap, drop in a virtual box of a given size, and use red or green to show whether it fits. Twist: use it to check whether a piece of furniture fits into a corner of your dorm room. #### Realistic AR Brush Texture with Deep Learning — Laan Labs (Chris Laan) (2017) - Video: https://www.youtube.com/watch?v=9_2o5E1rmBI - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, ARKit, CoreML - Idea: A neural network turns photos into oil-paint brushwork and hangs them on real walls. - What it is: A neural network turns photos into painterly textures that are hung on real walls as AR paintings. - Technique: A style-transfer network (CoreML) converts a photo into a painted texture, which is applied to a framed quad placed on a detected vertical plane. - Try it: Generate a painting with an online style-transfer tool, hang it on a wall using vertical plane detection, and add a frame. Twist: photograph the wall itself, stylize it, and paste it back in place so the wall becomes a painting. #### AR Bocce Ball Measuring — Laan Labs (Chris Laan) (2018) - Video: https://www.youtube.com/watch?v=D7oKgWCopUw - Interaction: Tangible Objects, Play, Information & UI - Platform & tech: Phone, ARKit - Idea: Use AR to judge which bocce ball is closest to the target. - What it is: AirMeasure detects bocce balls on the lawn and shows which one is closest to the jack. - Technique: Ball positions are detected (likely by color segmentation or tap-to-mark) and projected onto the ground plane, then distances to the jack are computed and ranked. - Try it: Tap to mark a few real objects (such as coins) and a target point on the floor, and show which one is closest to the target. Twist: turn it into a referee for a bottle-cap flicking contest on the classroom floor. #### AR Roaches Crawling on Everything — Laan Labs (Chris Laan) (2018) - Video: https://www.youtube.com/watch?v=nnrK5ziihx8 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, 6D.ai, ARKit - Idea: Use real-time meshing to let virtual cockroaches crawl over every surface in the kitchen. - What it is: Using 6D.ai real-time meshing, virtual cockroaches scurry over and behind the actual objects in a kitchen. - Technique: 6D.ai real-time meshing provides a room mesh used both as an occlusion material and as a navigation surface, so agents crawl over and hide behind real objects. - Try it: Generate a room mesh with ARMeshManager on a LiDAR device, apply an occlusion material, and let small bugs crawl randomly along the mesh. Twist: make the bugs flee the phone's light (a ray from the screen center). #### ARKit + CoreML Foot Detection for AR Soccer — Laan Labs (Chris Laan) (2018) - Video: https://www.youtube.com/watch?v=kcsCf4SlFLs - Interaction: Hands & Body, Play - Platform & tech: Phone, ARKit, CoreML - Idea: Use a neural network to find your real foot and kick a virtual soccer ball. - What it is: A foot-segmentation neural net lets the player kick a virtual soccer ball with their real foot. - Technique: A foot-segmentation network produces a mask each frame, and the foot's estimated position is used as a kinematic collider that transfers velocity to a physics ball. - Try it: Track ankle positions with MediaPipe Pose, create a virtual ball in three.js on a web page, and kick it away when the foot gets close. Twist: turn it into a keepy-uppy counting game. #### ARKit 1.5 Vertical Planes and Meshes — Laan Labs (Chris Laan) (2018) - Video: https://www.youtube.com/watch?v=bkJtZaYoy98 - Interaction: Spatial Mapping - Platform & tech: Phone, ARKit 1.5 - Idea: Visualize the walls and irregular surface meshes that ARKit 1.5 detects in real time. - What it is: Visualises ARKit 1.5's wall detection and irregular surface meshes as they grow over a room. - Technique: ARKit 1.5 vertical plane detection and irregular plane-boundary polygons are rendered as semi-transparent meshes that update as the planes grow and merge. - Try it: Visualize horizontal and vertical planes with AR Foundation, giving each orientation its own color and showing the area. Twist: play a tone when a plane's area changes, turning the room scan into music. #### ARKit Home Robot — Laan Labs (Chris Laan) (2018) - Video: https://www.youtube.com/watch?v=NJKk2mXj6RU - Interaction: Tangible Objects, Spatial Mapping - Platform & tech: Phone, ARKit, Placenote, Romo - Idea: Build a home robot that finds its way with ARKit, from a discontinued Romo base and an iPhone. - What it is: A cheap home robot built from the discontinued Romo base and an iPhone, localised with ARKit and Placenote and driven from another phone. - Technique: The robot's iPhone runs ARKit with a Placenote persistent map for relocalization, so its pose is known in a shared map and remote commands are expressed as map-space targets. - Try it: Mount a phone on a remote-control car, run AR tracking, and send the car's position live to another phone for display. Twist: tap the floor on the other phone and have the car drive there on its own. #### ARHide — Laan Labs (Chris Laan) (2020) - Video: https://www.youtube.com/watch?v=xyeqd62AMUQ - Interaction: Portals & Worlds, Spatial Mapping - Platform & tech: Phone, ARKit, LiDAR - Idea: A LiDAR-era invisibility experiment that makes real objects vanish from view. - What it is: A LiDAR-era experiment that makes real objects in the scene disappear or turn translucent in AR. - Technique: The LiDAR scene mesh is used to cut out a real object's region, which is filled with background captured earlier or rendered with reduced alpha to make the object vanish or turn translucent. - Try it: Select a piece of furniture with the LiDAR mesh and cover it with a background texture you photographed in advance. Twist: make the object turn transparent gradually over time, as if fading away. #### Metal Particles v2 — Laan Labs (Chris Laan) (2020) - Video: https://www.youtube.com/watch?v=fH_KgOAxw6k - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, ARKit, LiDAR, Metal - Idea: Let Metal particles flow along the room surfaces scanned by LiDAR. - What it is: Metal-driven particle systems that flow over the LiDAR-scanned geometry of a real room. - Technique: GPU particles sample positions and normals from the LiDAR scene mesh (or depth map) so they flow and collide along the real room surfaces. - Try it: Emit particles from the AR mesh surface with Unity VFX Graph and let them flow over the furniture under gravity. Twist: make the particles flow in the direction the phone is moving. #### Augmented Reality 3D Gaussian Splatting in Real Time — Laan Labs (Chris Laan) (2023) - Video: https://www.youtube.com/watch?v=jfiG2tohCAo - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, ARKit, Metal, Gaussian splatting - Idea: Put moving people captured as Gaussian splats into AR space in real time. - What it is: A custom iOS Metal renderer places animated volumetric Gaussian-splat captures of people into the real world in AR. - Technique: Captured Gaussian-splat sequences are sorted and rasterized in a custom Metal renderer each frame and placed at an ARKit anchor, so volumetric people appear in the real room. - Try it: Capture a Gaussian splat of a person with Luma or Polycam and place it on the floor with three.js and WebXR (or a Unity plugin). Twist: capture several splats of the same person in different poses and switch between them in order as a stop-motion animation. #### AirDraw – Finger Paint AR for Vision Pro — Laan Labs (Chris Laan) (2024) - Video: https://www.youtube.com/watch?v=x_Vo7I8pc2s - Interaction: Drawing & Making, Hands & Body - Platform & tech: Headset, visionOS, RealityKit - Idea: Paint in the air with your finger inside Vision Pro. - What it is: Finger-painting in 3D space on Apple Vision Pro, later extended with the Logitech Muse stylus for 3D calligraphy and a voxel brush. - Technique: visionOS hand tracking gives fingertip joint positions each frame, and a pinch gesture starts appending those points to a stroke mesh in RealityKit. - Try it: Track the index fingertip on a web page with MediaPipe Hands and leave 3D lines in a three.js scene while pinching. Twist: swap the brush for voxel cubes and build blocks in mid-air. ### Niantic (John Hanke) *Game studio; makers of Ingress, Pokémon GO and Peridot* Spun out of Google in 2015 under John Hanke, Niantic built the location-based AR games Ingress and Pokémon GO and the Lightship AR platform. #### Ingress — Niantic (John Hanke) (2012) - Video: https://www.youtube.com/watch?v=92rYjlxqypM - Interaction: Location & City, Shared & Social, Play - Platform & tech: Phone, GPS, Android, iOS - Idea: Real city landmarks become portals fought over by two factions, and walking is the gameplay. - What it is: A location-based sci-fi game where two factions capture real-world landmarks as 'portals' by walking to them with a phone and link them into territory control fields. - Technique: GPS location is matched against a database of geotagged landmarks, and a proximity radius decides when a player can interact with a portal, making the map itself the AR layer. - Try it: Mark 5 "strongholds" on a campus map and use the browser Geolocation API to check whether a player is within 30 meters, capturing a stronghold on arrival. Twist: when two neighboring strongholds belong to the same side, they automatically link into territory. #### Pokémon GO — Niantic (John Hanke) (2016) - Video: https://www.youtube.com/watch?v=2sj2iQyBTQs - Interaction: Location & City, Play, Shared & Social - Platform & tech: Phone, GPS, Unity - Idea: Pokémon show up on your street corner, and you collect them by walking and using the camera. - What it is: Players walk their neighbourhood to find Pokémon that appear on the camera view over the real street, then throw Poké Balls to catch them; gyms and raids gather players at landmarks. - Technique: GPS and spawn tables place creatures near map locations, and the encounter screen overlays the creature on the camera feed using gyroscope orientation rather than full SLAM. - Try it: Use the gyroscope to control a character overlaid on the camera image that can be seen only when you face the right direction, and swipe to throw a ball and catch it. Twist: the character appears only in certain weather or at certain times. #### Pokémon GO AR+ — Niantic (John Hanke) (2017) - Video: https://www.youtube.com/watch?v=E7nwr6sA6Es - Interaction: Spatial Mapping, Play - Platform & tech: Phone, ARKit, Unity - Idea: Pokémon stand on the real ground, and you have to sneak up to catch them. - What it is: An ARKit mode where Pokémon are anchored on the ground at real scale; walking closer can startle them, so players sneak up before throwing. - Technique: ARKit plane detection anchors the creature at real scale on the ground, and the camera's distance and approach speed to the anchor drive a flee behavior. - Try it: Place a small animal on the ground with AR Foundation that runs away if the phone approaches too fast, so you can only touch it by moving in slowly. Twist: it trusts you more when you crouch down low. #### Codename: Neon — Niantic (John Hanke) (2018) - Video: https://www.youtube.com/watch?v=C0LAPeZ1Cc4 - Interaction: Shared & Social, Play, Hands & Body - Platform & tech: Phone, Niantic Real World Platform, shared AR - Idea: Several people on the same lawn see the same AR laser battle, sharing reality with low latency. - What it is: A real-time multiplayer AR laser-tag prototype: players in the same park shoot energy at each other, collect pellets and dodge shots that all players see in the same space. - Technique: Players share a low-latency multiplayer session with a common anchor, so each phone's pose and fired projectiles are synchronized in the same world coordinates. - Try it: Use Unity Netcode to let two phones fire energy balls at each other in the coordinate system of a shared image marker, scoring when a shot hits the other phone's position. Twist: energy pellets scattered on the ground disappear for both players once picked up. #### Codename: Tonehenge — Niantic (John Hanke) (2018) - Video: https://www.youtube.com/watch?v=DD-slvV3VEc - Interaction: Shared & Social, Play - Platform & tech: Phone, Niantic Real World Platform, shared AR - Idea: A few people gather around and work the same AR stone circle together to solve a puzzle. - What it is: A cooperative AR puzzle prototype in which several players standing around the same spot manipulate a shared virtual stone circle together in real time. - Technique: A shared anchor places one virtual structure for all players, and each player's input on a shared object state is replicated in real time so the puzzle needs coordinated action. - Try it: Place a shared multiplayer AR stone circle where each person can rotate only one stone, and it opens only when all are aligned. Twist: a stone can be turned only when everyone is looking at it at the same time. #### Real World Platform Occlusion Demo (Pikachu & Eevee) — Niantic (John Hanke) (2018) - Video: https://www.youtube.com/watch?v=0Y13DxRANO0 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, Deep learning depth, Niantic Real World Platform - Idea: Use deep learning so virtual Pokémon are hidden by real passers-by and objects and truly blend into reality. - What it is: A research demo by Niantic and Matrix Mill in which Pikachu and Eevee run around a plaza and correctly disappear behind real people, planters and benches using learned depth. - Technique: A neural network estimates per-pixel depth from the camera image, and virtual characters are depth-tested against it so real people and objects occlude them. - Try it: Turn on human occlusion or environment depth occlusion in AR Foundation and have a character run around your classmate. Twist: switch occlusion off on purpose and take a set of before-and-after screenshots. #### Harry Potter: Wizards Unite — Niantic (John Hanke) (2019) - Video: https://www.youtube.com/watch?v=0pThkmngqJE - Interaction: Location & City, Drawing & Making, Play - Platform & tech: Phone, ARKit, ARCore, Unity - Idea: Trace spell gestures on real streets to free magical objects scattered around the city. - What it is: A location-based AR game with Warner Bros. where players trace spell glyphs on screen to free magical 'Foundables' that appear over the real world. - Technique: Touch-path gesture recognition compares the traced stroke against template glyphs and scores accuracy and speed, with the encounter object overlaid on the camera view. - Try it: Implement simple gesture-path recognition (such as the $1 Recognizer) so that drawing the right symbol releases an object in AR. Twist: have students design three new symbols and their effects. #### Pokémon GO Snapshot — Niantic (John Hanke) (2019) - Video: https://www.youtube.com/watch?v=Htsbu7ka5rI - Interaction: Perception & Effects, Spatial Mapping, Play - Platform & tech: Phone, ARKit, ARCore - Idea: Take photos of your Pokémon in the real world, with the odd wild Pokémon photobombing. - What it is: An AR photo mode where players place their Pokémon in the real world, approach and pose it, and take pictures; wild Pokémon sometimes photobomb the shot. - Technique: A character is placed on a detected plane, and a photo mode lets the user move closer and frame the shot, with scripted random events adding a second character. - Try it: Build an AR photo mode where a 'photobomber' has a 20% chance of popping up when a picture is taken. Twist: the photobomber acts differently depending on whether a face is in the frame. #### Pokémon GO Buddy Adventure — Niantic (John Hanke) (2020) - Video: https://www.youtube.com/watch?v=W0W00MaCDBY - Interaction: Play, Spatial Mapping, Location & City - Platform & tech: Phone, ARKit, ARCore - Idea: Turn a Pokémon into an AR companion that walks with you and that you can feed, play with and photograph. - What it is: Buddy Pokémon follow the player and can be viewed in AR, where players feed them, play with them and take snapshots; the bond grows as you walk together. - Technique: An animated companion is placed on a detected plane, and touch interactions such as feeding and petting change a persistent affinity value that also grows with step count. - Try it: Make a small AR pet whose affinity rises when you tap to feed it and swipe to pet it, unlocking a new move once affinity passes a threshold. Twist: affinity also grows with the number of steps you walk each day. #### Pokémon GO Shared AR Experience — Niantic (John Hanke) (2020) - Video: https://www.youtube.com/watch?v=4PGAqkLoflY - Interaction: Shared & Social, Spatial Mapping - Platform & tech: Phone, Niantic Lightship, shared AR - Idea: Nearby players enter the same AR scene and watch their buddy Pokémon play together. - What it is: Nearby trainers join one shared AR session in which their buddy Pokémon appear together in the same real-world scene, so friends can play and take photos together. - Technique: Nearby phones localize to a common anchor (via Lightship shared AR) so each player's companion is spawned at a shared pose and seen consistently by everyone. - Try it: Align two phones with a shared image marker, have each release a pet, and make each pet appear on the other screen too and greet the other pet. Twist: when the two pets come close, they perform a combined move together. #### Pikmin Bloom — Niantic (John Hanke), Nintendo (2021) - Video: https://www.youtube.com/watch?v=ihMfEeL8PGs - Interaction: Location & City, Play - Platform & tech: Phone, ARKit, ARCore - Idea: Every step plants flowers in the city, and Pikmin follow you on your walk in AR. - What it is: A walking game made with Nintendo: Pikmin sprout from your steps, follow you through the city and plant flowers along the path you walk, visible in AR. - Technique: Pedometer and GPS trail data are drawn as a flower path on the map, and AR mode places followers on the ground plane behind the camera position. - Try it: Record the GPS track of a walk, draw it on a map as a flower path, and have a small character follow behind you in AR. Twist: the kind of flower depends on the type of place you were passing. #### Pokémon GO AR Mapping — Niantic (John Hanke) (2021) - Video: https://www.youtube.com/watch?v=naFeJxszLOs - Interaction: Location & City, Spatial Mapping, Information & UI - Platform & tech: Phone, Niantic Lightship VPS, 3D scanning - Idea: Players scan real landmarks and crowdsource a 3D world map for AR. - What it is: Players scan PokéStops and Gyms with their camera during field research tasks, producing 3D scans that build Niantic's map for persistent, location-anchored AR. - Technique: Players capture short camera scans with pose data around landmarks, which are reconstructed into 3D maps used to build a visual positioning system. - Try it: Scan a campus landmark with Polycam or Scaniverse and place the scanned model back in AR aligned with the original. Twist: have the whole class scan the same landmark and compare how different scanning paths turn out. #### Peridot — Niantic (John Hanke) (2023) - Video: https://www.youtube.com/watch?v=HFi2pJrAEdc - Interaction: Spatial Mapping, Play, Shared & Social - Platform & tech: Phone, Niantic Lightship ARDK, semantic segmentation - Idea: An AR pet that recognizes grass, water and sand and plays with the real environment. - What it is: An AR virtual pet game where each 'Dot' is a unique creature that understands the real ground, grass, water and sand around you and plays fetch or digs in them. - Technique: Semantic segmentation labels ground pixels as grass, water, sand and so on, and the pet's behavior tree picks actions such as digging or swimming based on the class under it. - Try it: Tell grass from floor with Lightship semantic segmentation or a simple color check, and make an AR pet roll on the grass and walk on the floor. Twist: when the pet is placed by water, it takes a drink. #### Hello, Dot — Niantic (John Hanke) (2024) - Video: https://www.youtube.com/watch?v=DA0PWd-x1c8 - Interaction: Spatial Mapping, Hands & Body, Play - Platform & tech: Headset, Meta Quest 3, passthrough, Unity - Idea: In a headset's passthrough view, an AR pet hatches in your room and learns your furniture. - What it is: A mixed-reality Peridot experience for Meta Quest 3 in which a Dot hatches in your room, recognises furniture and plays with you through passthrough. - Technique: Passthrough plus scene understanding labels furniture in the room, and the pet uses those labels as navigation targets and interaction affordances. - Try it: Detect two levels in AR, the tabletop and the floor, and teach a pet to jump from the floor onto the table to find you. Twist: when you set the phone down and leave it still, the pet falls asleep on the table. #### Peridot Beyond — Niantic (John Hanke) (2024) - Video: https://www.youtube.com/watch?v=wl10_rJplyc - Interaction: Shared & Social, Play, Hands & Body - Platform & tech: Headset, Snap Spectacles, Lens Studio - Idea: Wearing Spectacles, several people look after the same AR pet together in the real world. - What it is: Peridot on Snap Spectacles: several people wearing glasses see and play with the same Dots in shared multiplayer AR. - Technique: Multiple glasses join a shared session with colocalized coordinates, so the same pet entity and its state are replicated to every wearer. - Try it: Have two or three devices share one AR pet that chases the ball of whoever throws first. Twist: the pet remembers who plays with it most. #### Monster Hunter Now Palico AR — Niantic (John Hanke) (2025) - Video: https://www.youtube.com/watch?v=7C-HBHCoc70 - Interaction: Location & City, Play - Platform & tech: Phone, Niantic Lightship - Idea: Bring your hunting companion, the Palico cat, into the real world for a photo together. - What it is: In Monster Hunter Now, made with Capcom, the player's Palico cat companion can be placed in AR for photos alongside the monsters hunted in the real world. - Technique: A companion character is placed on a detected surface in a photo mode with pose presets and camera capture, turning AR into a snapshot tool. - Try it: Build an AR photo mode: place a character, choose a pose, and capture and save a screenshot with one tap. Twist: automatically add a frame related to the current location when the photo is taken. ### Frieder Weiss *Interactive video designer and 'engineer in the arts'; creator of the EyeCon motion-tracking software* Software artist who since the mid-1990s has built camera-tracking systems that turn dancers' movement into live projected light and sound, from Palindrome's EyeCon pieces to Chunky Move's Glow and Mortal Engine and stage visuals for opera, circus and pop. #### ...seine hohle Form... — Frieder Weiss (2000) - Video: https://vimeo.com/8895552 - Interaction: Voice & Sound, Hands & Body, Performance - Platform & tech: Desktop, EyeCon, Max/MSP, video tracking - Idea: A piece of music that cannot be heard unless someone dances it. - What it is: A Palindrome duet in which the music only exists while the dancers move: their positions and gestures in video-tracked zones trigger and shape real-time synthesis. - Technique: Weiss's EyeCon video-tracking software defines trigger lines and fields on the camera image, sending movement data to Butch Rovan's Max/MSP synthesis patch. - Try it: Divide a webcam image into four zones, and have movement in each zone play and bend a different sound in Max, Pure Data or Web Audio. Twist: the zones slowly drift, so the dancer must chase the music. #### Schlamp — Frieder Weiss (2003) - Video: https://www.youtube.com/watch?v=2BKTlqt4GFA - Interaction: Hands & Body, Projection, Drawing & Making - Platform & tech: Projection, EyeCon, motion sensing - Idea: Let passers-by paint with their bodies on a projected surface. - What it is: An interactive video installation made with Emily Fernandez for a virtual public square in Dresden, where visitors' movement smears and stirs projected video imagery. - Technique: EyeCon motion sensing measures movement in the camera image and uses it to distort and blend video layers in real time. - Try it: Compute frame differences from a webcam and use them as a mask that reveals a hidden video beneath a live one. Twist: the revealed areas heal back after ten seconds. #### Shadows — Frieder Weiss (2003) - Video: https://www.youtube.com/watch?v=gm_doxsdqG4 - Interaction: Perception & Effects, Projection, Hands & Body - Platform & tech: Projection, EyeCon, video delay, projector - Idea: Your shadow stops obeying you. - What it is: A shadow play for dancer Emily Fernandez in which her shadow is projected by the computer, so it can lag, speed up, mirror, change colour or detach from her. - Technique: A camera captures the dancer's silhouette and the software re-projects it as a shadow with digital delays, time stretching, translation and mirroring. - Try it: Capture a silhouette with a webcam, store the last few seconds in a buffer, and project the delayed shadow next to the real one. Twist: the shadow copies you only when you are not looking at it. #### Glow (Chunky Move) — Frieder Weiss (2006) - Video: https://www.youtube.com/watch?v=C4He543_a80 - Interaction: Hands & Body, Projection, Performance - Platform & tech: Projection, EyeCon, infrared camera, projector - Idea: Light that behaves like a living skin, growing out of the body's silhouette. - What it is: A 27-minute solo for Chunky Move in which a dancer lies and writhes on a white floor while projections react to her body, outlining it, stretching light from her limbs and hunting her with shadows. - Technique: An overhead infrared camera segments the dancer's silhouette and Weiss's EyeCon-derived software draws contours, particles and fields from it, projected back onto the floor from above with minimal latency. - Try it: Mount a phone or webcam above a white sheet, segment the person lying on it, and project an outline that grows lines from their extremities. Twist: the light must slowly lag behind, like a memory of the pose. #### Mortal Engine (Chunky Move) — Frieder Weiss (2008) - Video: https://www.youtube.com/watch?v=sbjOMualLVs - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, infrared camera, projector, laser - Idea: Darkness as the default, with the body only existing where the computer chooses to light it. - What it is: A full-length piece for six dancers on a steep, dark stage where projections, lasers and sound respond to their movement, turning bodies into shifting, shimmering shapes that appear and dissolve. - Technique: Infrared tracking of the dancers drives a projection-based lighting system that paints only their silhouettes and extends them with graphics, combined with Robin Fox's lasers and Ben Frost's reactive score. - Try it: In a dark room, use a projector as the only light and a webcam silhouette mask so light falls only on the performer. Twist: make the light expand into the empty space whenever the performer stops moving. #### Kylie Minogue — Get Outta My Way — Frieder Weiss (2010) - Video: https://www.youtube.com/watch?v=BHGaW8lBlSk - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, projector, camera tracking - Idea: Pop choreography shot live inside a responsive light set instead of adding effects in post. - What it is: A music video shot at Pinewood Studios where Kylie Minogue and dancers perform inside huge interactive projections that ripple, stretch and light up in response to their bodies. - Technique: Two stages with large projector rigs ran Weiss's real-time tracking visuals, reacting to the performers while an ARRI Alexa filmed the result in camera. - Try it: Film a 30-second phone music video where a projected pattern reacts to the dancer via webcam tracking, with no effects added afterwards. Twist: the projection only reacts to the beat when the dancer is on it. #### DUMMY lab — Frieder Weiss (2011) - Video: https://www.youtube.com/watch?v=FnbvSRylnPs - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, camera tracking, projector, live music - Idea: Acrobatics against a wall of light that answers every fall and lift. - What it is: A circus show by Eike von Stuckenbrok and Markus Pabst where an acrobat performs with a live soundtrack by Reecode inside Weiss's reactive video installation. - Technique: Real-time camera tracking of the acrobat feeds generative video projected behind and onto him, likely synchronised with the live electronic music. - Try it: Track a jumper with a webcam and project a trail that records the height of every jump on the wall. Twist: the trail becomes a staircase the performer tries to climb. #### Du, Liebe — Frieder Weiss (2011) - Video: https://www.youtube.com/watch?v=nZojuHerXR4 - Interaction: Projection, Performance, Hands & Body - Platform & tech: Projection, projection mapping, camera tracking - Idea: Projection mapping that follows people instead of buildings. - What it is: A chamber dance opera with dancer Eva Maria Christ and a countertenor, staged inside Weiss's projected visual stage that frames and reshapes the performers. - Technique: Mapped projection onto the set and performers, likely combined with camera tracking so graphics stay registered to the moving dancer. - Try it: Map a projector onto a white box and a person standing next to it, then use pose tracking so a projected frame follows the person's torso. Twist: the frame refuses to follow when the singer's voice is loud. #### Flow — Frieder Weiss (2011) - Video: https://vimeo.com/41397711 - Interaction: Projection, Hands & Body, Shared & Social - Platform & tech: Projection, floor projection, camera tracking, fluid simulation - Idea: The floor behaves like water that remembers every step. - What it is: An interactive floor-projection installation at the Cinedans Festival in Amsterdam where people walking and dancing across the floor set flowing visual fields in motion. - Technique: An overhead camera tracks bodies on the floor and injects their motion into a fluid-like simulation projected from above. - Try it: Point a projector and webcam at the floor, use frame differences as forces in a 2D fluid simulation, and let people walk through it. Twist: two people moving in sync create a calm zone. #### Butterfly under Glass — Frieder Weiss (2015) - Video: https://www.youtube.com/watch?v=GiSOWS9XQmM - Interaction: Performance, Projection, Voice & Sound - Platform & tech: Projection, interactive video, projector - Idea: A dancer pinned like a specimen inside living, reactive images. - What it is: A music-dance-theatre piece for PODIUM Esslingen in which dancer Laurie Young moves inside an interactive video scenography by Weiss, alongside live soprano and ensemble. - Technique: Likely camera tracking of the dancer drives projected imagery around and on her body in real time, as in Weiss's other stage work. - Try it: Project a grid of butterfly shapes onto a wall, and use webcam pose tracking so they freeze when a performer's hand comes near. Twist: they flutter again only when the music gets quiet. #### Dürer's Dog (Ballett Nürnberg) — Frieder Weiss (2017) - Video: https://www.youtube.com/watch?v=wopUITjQCVo - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, digital scenography, projector - Idea: A painter's etchings come alive as a stage that moves with the dancers. - What it is: A ballet by Goyo Montero about Albrecht Dürer for Staatstheater Nürnberg, with Weiss's digital scenography projecting engraving-like imagery that shifts with the ensemble. - Technique: Large-scale stage projection of generated, print-like graphics, likely partly driven by live tracking of the dancers. - Try it: Take a public-domain engraving, split it into line layers, and project them so a dancer's position reveals or erases lines. Twist: the dancer can only reveal the drawing by standing still. ### Kitasenju Design (Takayuki Watanabe) *Programmer & digital artist; real-time CG studio* One-person studio of programmer-artist Takayuki Watanabe that experiments with real-time visual expression and ships AR effect apps like MEISAI and SUPER SLIT SCAN. #### JIDO-RHYTHM Selfie Music Video — Kitasenju Design (Takayuki Watanabe) (2018) - Video: https://www.youtube.com/watch?v=aMnTfzUd0z8 - Interaction: Face, Voice & Sound, Perception & Effects - Platform & tech: Phone, ARKit, face tracking - Idea: A selfie music video where your tracked face warps to the beat. - What it is: A selfie music-video app that deforms and animates your tracked face in time with the music. - Technique: ARKit face tracking provides a face mesh whose vertices are displaced by audio-analysis values (beat and amplitude), so the face deforms in time with music. - Try it: Use Lens Studio or MediaPipe Face Mesh to build a filter that stretches the face to the music's beat. Twist: drive the deformation with your own humming instead of background music. #### ARKit3 Body Tracking Effects — Kitasenju Design (Takayuki Watanabe) (2019) - Video: https://www.youtube.com/watch?v=T5KMWPw47i0 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, ARKit 3, Unity - Idea: Hang generative graphics on the body skeleton that ARKit 3 tracks. - What it is: Generative graphics attached to a tracked human skeleton follow a dancer's movements. - Technique: ARKit 3 body tracking provides a 3D skeleton, and generative meshes and trails are attached to joint transforms so they follow the dancer. - Try it: Track full-body joints with MediaPipe Pose and attach colorful trails to the wrists and ankles. Twist: each joint triggers the sound of a different instrument. #### AR_Glitch — Kitasenju Design (Takayuki Watanabe) (2019) - Video: https://www.youtube.com/watch?v=RENlFmMSk5w - Interaction: Perception & Effects - Platform & tech: Phone, ARKit, Unity - Idea: Make the real world tear apart in generative glitches across space. - What it is: A generative glitch effect that tears and displaces the live camera image around the space. - Technique: A full-screen shader slices the camera image into blocks and offsets them with noise-driven displacement and RGB channel splitting, likely modulated by device motion. - Try it: Write a glitch shader in Unity or three.js that offsets camera image blocks and splits RGB channels. Twist: the harder the phone shakes, the stronger the glitch. #### Particles Face — Kitasenju Design (Takayuki Watanabe) (2019) - Video: https://www.youtube.com/watch?v=cKKZsk5yi98 - Interaction: Face, Perception & Effects - Platform & tech: Phone, ARKit, Unity, compute shader - Idea: Break a tracked face apart into a flowing cloud of particles. - What it is: The tracked face dissolves into a flowing compute-shader particle cloud. - Technique: Face-mesh vertices are sampled as spawn points for a compute-shader particle system, and curl noise advects the particles away so the face appears to dissolve. - Try it: Use MediaPipe Face Mesh vertices as particle emitters and let the particles drift away on noise in three.js. Twist: the whole face dissolves when the mouth opens and gathers again when it closes. #### Pixelation with Human Segmentation — Kitasenju Design (Takayuki Watanabe) (2019) - Video: https://www.youtube.com/watch?v=efjXnjwfAYY - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, ARKit 3, Unity - Idea: Cut people out of the frame and turn them into flowing pixel mosaics. - What it is: People are cut out of the scene with ARKit3 segmentation and re-rendered as moving pixel mosaics. - Technique: ARKit 3 person segmentation outputs a human stencil mask, and a shader pixelates only the masked region of the camera image. - Try it: Cut out people with MediaPipe Selfie Segmentation and pixelate only the person region. Twist: the pixel block size changes with the person's distance from the camera. #### $100 Wallpaper Using ARKit — Kitasenju Design (Takayuki Watanabe) (2020) - Video: https://www.youtube.com/watch?v=XHHp4QnZ8qY - Interaction: Spatial Mapping, Portals & Worlds - Platform & tech: Phone, ARKit, LiDAR - Idea: Give a real room new virtual wallpaper with LiDAR. - What it is: Re-skins the walls of a real room with virtual wallpaper using LiDAR scene geometry. - Technique: Wall-classified LiDAR mesh faces receive a triplanar or world-space UV texture, with people occlusion so the wallpaper sits behind people in the room. - Try it: Use vertical plane detection to find walls, apply a repeating wallpaper pattern, and turn on people occlusion. Twist: the wallpaper pattern slowly flows or changes color over time. #### AR Drawing — Kitasenju Design (Takayuki Watanabe) (2020) - Video: https://www.youtube.com/watch?v=X7-xd3Csz3g - Interaction: Drawing & Making - Platform & tech: Phone, ARKit, Unity - Idea: Draw stylized 3D lines in the air with your phone. - What it is: Strokes drawn with the phone hang in space as stylised 3D lines. - Technique: Camera positions sampled while touching are converted into ribbon or tube meshes with stylized shading, anchored in world space. - Try it: Record the phone's movement path with AR Foundation to generate a ribbon mesh and add a cartoon outline shader. Twist: finished lines slowly sway in a virtual wind. #### Fast Plane Detection and Classification with LiDAR — Kitasenju Design (Takayuki Watanabe) (2020) - Video: https://www.youtube.com/watch?v=hMQSgJR3bIs - Interaction: Spatial Mapping - Platform & tech: Phone, ARKit 3.5, LiDAR - Idea: Visualize LiDAR instantly recognizing floors, walls, ceilings and furniture. - What it is: Visualises the LiDAR iPad Pro instantly classifying floor, walls, ceiling and furniture as coloured surfaces. - Technique: ARKit scene reconstruction on LiDAR provides per-face semantic classification (floor, wall, ceiling, table, seat), which is visualized by coloring mesh faces by class. - Try it: Read mesh classifications with AR Foundation's ARMeshManager and give floors, walls and tables different colors. Twist: place a character that sits down only on seat-class surfaces. #### MEISAI – AR Filter App — Kitasenju Design (Takayuki Watanabe) (2020) - Video: https://www.youtube.com/watch?v=xivbKi_somg - Interaction: Perception & Effects, Hands & Body - Platform & tech: Phone, ARKit, Unity - Idea: A generative AR filter app that rewrites people and space in real time, like camouflage. - What it is: An iOS AR effect app offering camouflage-like generative filters that reshape people and space in real time. - Technique: Person segmentation and depth masks drive full-screen shaders that re-texture people and space with generative camouflage patterns in real time. - Try it: Use person segmentation to replace the person's area with texture sampled from the surrounding background, making a camouflage invisibility filter. Twist: you stay invisible only while standing still and reappear as soon as you move. #### Minecraft Filter — Kitasenju Design (Takayuki Watanabe) (2020) - Video: https://www.youtube.com/watch?v=-W8hKtByxEw - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, ARKit 4, LiDAR, Unity - Idea: Render the real room around you in Minecraft-style blocks. - What it is: A LiDAR filter that renders the surrounding room as blocky Minecraft-style geometry. - Technique: Scene depth or mesh is voxelized and each voxel is textured by sampling the camera color at its projected position, giving a blocky Minecraft-like rendering. - Try it: Voxelize the room mesh and color each block with the matching color from the camera image to make a block-style room. Twist: only a spherical zone in front of the phone turns into blocks. #### Physics Simulation with ARKit 3.5 — Kitasenju Design (Takayuki Watanabe) (2020) - Video: https://www.youtube.com/watch?v=40v1iKu4Sgk - Interaction: Spatial Mapping, Play - Platform & tech: Phone, ARKit 3.5, LiDAR, Unity - Idea: Virtual objects collide and pile up realistically on LiDAR-scanned furniture. - What it is: Virtual objects bounce and pile up on the LiDAR-scanned furniture and floor. - Technique: The LiDAR scene mesh is used as a MeshCollider, so rigidbodies spawned in Unity bounce off and pile up on real furniture. - Try it: Add a MeshCollider to the AR mesh and tap the screen to throw balls from the camera position so they land on the real sofa. Twist: you score only when you hit a particular piece of furniture. #### Realtime Boxelization with ARKit4 and LiDAR — Kitasenju Design (Takayuki Watanabe) (2020) - Video: https://www.youtube.com/watch?v=A1l5fc1FAwE - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, ARKit 4, LiDAR, Unity - Idea: Voxelize the whole room, people included, into blocks in real time. - What it is: The whole room, including people, is converted in real time into a field of voxel boxes. - Technique: The depth map is unprojected to world points each frame and quantized into a voxel grid, which is rendered as GPU-instanced cubes. - Try it: Unproject an iPhone depth map into points, snap them to a 5 cm grid, and render them as blocks with GPU instancing. Twist: make the block size change with the music volume. #### Unity-chan and Occlusion — Kitasenju Design (Takayuki Watanabe) (2020) - Video: https://www.youtube.com/watch?v=8Uj2dfQS8FU - Interaction: Spatial Mapping, Performance - Platform & tech: Phone, ARKit, Unity - Idea: A virtual character is correctly hidden behind real people and furniture. - What it is: A dancing Unity-chan character is correctly hidden by real people and furniture using ARKit depth occlusion. - Technique: ARKit people occlusion and LiDAR environment depth are used in AR Foundation's AROcclusionManager, so the depth test hides the character behind real people and furniture. - Try it: Turn on AROcclusionManager in AR Foundation so a dancing character is correctly hidden behind people and chairs. Twist: when the character is blocked, it pokes its head out to peek. #### Virtual Flowers Using ARKit — Kitasenju Design (Takayuki Watanabe) (2020) - Video: https://www.youtube.com/watch?v=8B76A9r89ZM - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, ARKit, LiDAR - Idea: Generative flowers grow and bloom along the room's surfaces. - What it is: Generative flowers sprout and bloom across the scanned surfaces of a room. - Technique: Points are sampled on the scanned mesh surface with their normals, and procedural flower instances grow along those normals with animated scale. - Try it: Randomly sample points on an AR mesh or plane and grow flowers along the normals that get bigger over time. Twist: flowers grow only where the phone camera has been aimed for more than three seconds. #### Drawing in the Air — Kitasenju Design (Takayuki Watanabe) (2021) - Video: https://www.youtube.com/watch?v=CEEqp9RKoLU - Interaction: Drawing & Making - Platform & tech: Phone, ARKit, Unity - Idea: Lines drawn in the air become 3D ribbons that stay in the room. - What it is: Lines drawn in mid-air settle into the room as persistent 3D ribbons. - Technique: Strokes are recorded as world-space points from the camera pose and converted into ribbon meshes that persist at their anchors (likely with physics-based settling). - Try it: Draw ribbons in the air with AR Foundation and, once you let go, let them drift down gently under gravity onto the floor plane. Twist: after landing, the ribbons turn into flowers or words. #### SUPER SLIT SCAN — Kitasenju Design (Takayuki Watanabe) (2022) - Video: https://www.youtube.com/watch?v=y3HyKMhGeTI - Interaction: Perception & Effects, Hands & Body - Platform & tech: Phone, Projection, Unity, 3D texture shader - Idea: Spread time across the frame with slit-scan and turn people into warped sculptures of space and time. - What it is: A slit-scan camera app that smears time across the frame, turning moving people into stretched, warped sculptures. - Technique: Recent camera frames are stored in a 3D texture (ring buffer), and a shader samples each pixel row or column from a different time slice to produce slit-scan distortion. - Try it: Store the last 120 camera frames in p5.js or three.js and take each pixel row from a different frame to create a time-scan effect. Twist: distribute the time offset by distance from the center of the frame instead. #### Kohkoku: Reality and Illusion AR — Kitasenju Design (Takayuki Watanabe) (2023) - Video: https://www.youtube.com/watch?v=_2GHZNGRqwI - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Phone, AR, image tracking - Idea: An AR piece for the 'reality and illusion' issue of Kohkoku magazine that makes the printed pages move on a phone. - What it is: AR artwork for the Japanese magazine Kohkoku's 'reality and illusion' issue, animating printed pages through the phone. - Technique: Printed magazine pages are registered as tracking images, and animations or 3D elements are rendered on the tracked page pose. - Try it: Use 8th Wall or MindAR to recognize a poster you designed, and make its elements lift off the paper and move. Twist: when the poster is flipped, reveal a hidden second layer of content. ### Roelof Knol *Visual artist and creative developer (interactive light and projection installations)* Amsterdam-based visual artist and developer, for years building interactive museum exhibits at the studio YiPP. His minimalist projection-mapped pieces make lines, circles and light respond to paper, balls, magnets, pendulums, visitors and even cats, and have been shown at Nxt Museum Amsterdam, SMoCA Scottsdale and Signal Festival Prague. #### Connected — Roelof Knol (2020) - Video: https://www.youtube.com/watch?v=ArFcUEoxKfo - Interaction: Projection, Shared & Social, Hands & Body - Platform & tech: Projection, projection mapping, people tracking, spatial audio - Idea: Draw the invisible lines between strangers on the floor. - What it is: An immersive room at Nxt Museum made with sound artist Marc Mahfoud: light projected on the floor links the visitors standing in it, turning each person's personal space into a shared space. Here it is the set of Benny Benassi and Bloom Twins' music video DayDream. - Technique: Overhead tracking (likely depth or infrared cameras) finds each visitor's position and a ceiling projector draws lines and shapes between them in real time, with a spatial soundscape. - Try it: Mount a webcam and a cheap projector overhead, track people as blobs with OpenCV or TouchDesigner, and project a line between every pair of people. Twist: lines only appear between people who are standing still. #### Interactive projection mapping on paper — Roelof Knol (2021) - Video: https://x.com/CurieuxExplorer/status/1439277743212937224 - Interaction: Projection, Tangible Objects, Drawing & Making - Platform & tech: Projection, projection mapping, camera tracking, Unity - Idea: A printed card becomes the source that the projected drawing grows out of. - What it is: A hand slides a small printed card across a dark table; projected lines stream out from the card's pattern and keep flowing into new patterns as it moves. - Technique: A camera tracks the card's position and rotation (likely via its printed pattern or contour) and a calibrated projector draws generative lines registered to the card in real time. - Try it: Calibrate a webcam to a projector over a table, track an ArUco-marked card with OpenCV, and project lines that radiate from its edges. Twist: two cards brought close together merge their line fields. #### Realtime interactive football wall — Roelof Knol (2021) - Video: https://x.com/Rainmaker1973/status/1535625398670614529 - Interaction: Projection, Play, Tangible Objects - Platform & tech: Projection, projection mapping, camera tracking - Idea: Any wall becomes a playable surface when the projector knows where the ball lands. - What it is: A football is kicked against a wall covered with a projected grid of rectangles; the pattern reacts in real time wherever the ball hits. - Technique: A camera or depth sensor detects the ball's impact point on the wall, which is mapped into projector coordinates to trigger changes in the generative grid (likely). - Try it: Project a grid onto a wall, detect ball hits with a webcam using frame differencing, and light up the hit cell. Twist: turn it into a two-player game where each player 'paints' cells in their own colour. #### Circle intersections: interactive magnets on whiteboard — Roelof Knol (2022) - Video: https://x.com/CurieuxExplorer/status/1526756672999288832 - Interaction: Projection, Tangible Objects, Information & UI - Platform & tech: Projection, projection mapping, camera tracking, computational geometry - Idea: Play with geometry by hand: magnets are the control points of a live diagram. - What it is: Magnets placed on a whiteboard each get a projected circle of light; where the circles overlap, their intersections are drawn, so moving a magnet reshapes the whole geometric composition. - Technique: A camera detects the magnets' positions on the board, circles are computed around each one, their intersection points are solved geometrically, and the result is projected back onto the board. - Try it: Track coloured magnets or bottle caps on a whiteboard with a webcam and project a circle around each, drawing a dot at every intersection. Twist: map the number of intersections to a musical chord. #### Depth lines on paper — Roelof Knol (2022) - Video: https://x.com/CurieuxExplorer/status/1544719441790992385 - Interaction: Projection, Tangible Objects, Perception & Effects - Platform & tech: Projection, projection mapping, camera tracking, Unity - Idea: A plain card becomes a portal into the table. - What it is: Small paper cards on a dark table each show a projected tunnel of nested glowing squares, making the cards look like holes in the table; sliding a card by hand drags its tunnel along. - Technique: Card positions are tracked by a camera and nested rectangles are projected with a perspective offset toward a vanishing point, creating a depth illusion on each card. - Try it: Track paper rectangles on a table with a webcam and project nested shrinking squares onto each to fake a hole. Twist: drop a small virtual ball that falls into whichever hole you slide under it. #### Dots on paper — Roelof Knol (2022) - Video: https://x.com/WevolverApp/status/1597712101006577664 - Interaction: Projection, Drawing & Making, Tangible Objects - Platform & tech: Projection, projection mapping, blob detection, audio-reactive - Idea: A single sheet of paper is enough as a screen, a controller and a stage. - What it is: A plain sheet of paper on a wooden table becomes a live canvas: dots on the paper drive projected graphics that animate in sync with music, framed exactly by the sheet's edges. - Technique: The sheet is detected and tracked by a camera, dots on it are found by blob detection, and generative graphics are projection-mapped precisely onto the paper (likely). - Try it: Project onto an A4 sheet whose corners are tracked by a webcam, and make dots drawn with a marker become sources of animated ripples. Twist: erasing a dot makes its ripples collapse inward. #### Real-time perspective bounce detection — Roelof Knol (2022) - Video: https://x.com/Rainmaker1973/status/1531553569995952129 - Interaction: Projection, Tangible Objects, Perception & Effects - Platform & tech: Projection, projection mapping, anamorphic perspective, camera tracking - Idea: A bounce is enough input to make a flat table feel deep. - What it is: A small ball bounced on a table inside a projected rectangle: each bounce is detected in real time and the projected frame reacts with a perspective effect, as if the surface had depth. - Technique: Bounce moments are detected from the ball's tracked height or sound (likely), and the projected image is re-rendered with an anamorphic perspective for the viewer's position. - Try it: Detect a ball bounce with a phone microphone or webcam and trigger a projected ripple at the bounce point. Twist: render the ripple anamorphically so it looks like a hole from one fixed viewpoint. #### Two projectors + augmented reality effects — Roelof Knol (2022) - Video: https://x.com/WevolverApp/status/1491350500222152704 - Interaction: Projection, Voice & Sound, Perception & Effects - Platform & tech: Projection, Phone, projection mapping, AR, audio-reactive - Idea: Stack projection and screen-based AR so light on the table and virtual content share one space. - What it is: Two projectors map audio-reactive rings and circles of light onto a coffee table, combined with augmented reality effects in the same scene so that projected and virtual layers read as one. - Technique: Two calibrated projectors cover the table surface with audio-reactive graphics while a camera-based AR view renders matching 3D elements registered to the same table (likely). - Try it: Project an audio-reactive circle onto a table, then build a phone AR scene with an image target on the table that adds a 3D element synced to the same music. Twist: the AR element should seem to cast a shadow into the projection. #### Voronoi cells drawn by a pendulum — Roelof Knol (2022) - Video: https://x.com/CurieuxExplorer/status/1553404644566507521 - Interaction: Projection, Tangible Objects, Perception & Effects - Platform & tech: Projection, projection mapping, Voronoi, Unity - Idea: Let a pendulum, not a person, conduct a generative drawing. - What it is: A pendulum ball swings over a coffee table covered in projected Voronoi cells; the cell pattern is continuously redrawn around the moving ball. - Technique: The pendulum's position is tracked from above and used as a moving seed point in a real-time Voronoi diagram that is projected back onto the table. - Try it: Hang a pendulum over a table, track it with a webcam, and project a Voronoi diagram whose seeds include the pendulum and a few fixed points. Twist: the pendulum's speed controls line thickness. #### Circle intersections: interactive pendulum projection — Roelof Knol (2023) - Video: https://x.com/CurieuxExplorer/status/1621037555729960960 - Interaction: Projection, Tangible Objects, Perception & Effects - Platform & tech: Projection, projection mapping, camera tracking - Idea: Physics as a drawing tool: gravity animates the geometry. - What it is: A pendulum swings over a projected frame on a dark carpet; circles drawn around the pendulum and fixed points intersect and shift as it swings, producing a constantly changing geometric figure. - Technique: The pendulum bob is tracked from above and circle intersections between it and fixed centres are computed each frame and projected onto the floor. - Try it: Track a hanging pendulum with a webcam and project circles around it and two fixed points, highlighting their intersections. Twist: leave a fading trail of past intersections to draw a pattern over time. #### Light Particles: projection mapping + LED + AR — Roelof Knol (2023) - Video: https://x.com/WevolverApp/status/1610939456692883456 - Interaction: Projection, Perception & Effects, Spatial Mapping - Platform & tech: Projection, Phone, projection mapping, LED, AR, particles - Idea: Make one particle system flow across projector, LEDs and AR without a seam. - What it is: Swarms of bright light particles are projected onto a living-room floor and continue into LED light and an AR layer, blurring which light is physical and which is virtual. - Technique: A single real-time particle simulation is rendered to a floor projector, an LED fixture and a camera-based AR view that share one calibrated coordinate system (likely). - Try it: Run one particle system in Unity or TouchDesigner that renders to a projector on the floor and to a phone AR view anchored to the same spot. Twist: particles that leave the projection area continue only in AR. #### Voronoi Depth — Roelof Knol (2023) - Video: https://x.com/CurieuxExplorer/status/1716746012939931835 - Interaction: Projection, Perception & Effects, Tangible Objects - Platform & tech: Projection, projection mapping, anamorphic perspective, Voronoi - Idea: Fake depth in the floor, then let real objects disturb it. - What it is: A projected rectangle on a living-room floor opens into what looks like a recessed box whose Voronoi-patterned walls react to what is placed or moved inside it. - Technique: An anamorphic projection renders a box interior for a fixed viewpoint, while tracked objects act as seeds for a Voronoi pattern mapped onto the box walls (likely). - Try it: Project an anamorphic 'hole' onto the floor that looks correct from one marked spot, then add a simple reaction when a foot enters it. Twist: the hole gets deeper the longer someone stands in it. #### the space in between — Roelof Knol (2023) - Video: https://www.youtube.com/watch?v=Ypd0NXoVUi0 - Interaction: Projection, Hands & Body, Voice & Sound - Platform & tech: Projection, projection mapping, people tracking, sound - Idea: Make the empty space between people and walls the thing that reacts. - What it is: An immersive installation at Scottsdale Museum of Contemporary Art: in a dark gallery, minimal lines and planes of projected light respond to visitors' movements with sound, turning the room into a playful audiovisual space. - Technique: Visitors are tracked in the gallery (likely with depth or infrared cameras) and their positions drive projected geometry and sound in real time. - Try it: In a darkened classroom, project a single line across the floor that bends toward the nearest person tracked by a webcam. Twist: when two people approach each other, the line breaks into two. #### Projected fish school for cats — Roelof Knol (2024) - Video: https://x.com/Rainmaker1973/status/1772239372521263467 - Interaction: Projection, Play, Hands & Body - Platform & tech: Projection, projection mapping, boids, camera tracking - Idea: AR for an audience that cannot read screens: a cat. - What it is: A school of small fish is projected onto the floor and scatters away from a cat that chases it, turning a projection into interactive play for pets. - Technique: A camera tracks the cat's position and a flocking (boids) simulation makes the projected fish flee from it in real time. - Try it: Project a boids swarm onto the floor and make it flee from the largest moving blob seen by a webcam (a person's foot or a pet). Twist: the swarm slowly learns to follow instead of flee. #### light lines — Roelof Knol (2024) - Video: https://x.com/orbithm/status/1812187426590200184 - Interaction: Projection, Tangible Objects, Perception & Effects - Platform & tech: Projection, projection mapping, camera tracking - Idea: Real objects bend the light around them. - What it is: Dense parallel lines of warm light are projected across a table and react to the objects placed among them, such as a candle, so the stripes seem to flow around real things. - Technique: Object positions are detected from above and used to displace a field of projected line segments in real time (likely). - Try it: Project a field of parallel lines onto a table and use webcam blob detection to push the lines away from any object placed on it. Twist: the lines slowly remember where objects used to be. #### walking in circles — Roelof Knol (2024) - Video: https://x.com/genmediaclub/status/1866604059240853525 - Interaction: Projection, Hands & Body, Voice & Sound - Platform & tech: Projection, projection mapping, people tracking - Idea: Your walking path rearranges the architecture of light around you. - What it is: A room-scale installation in which white bars and squares projected on the floor and walls shift around a person walking in circles, set to an ambient soundtrack. - Technique: A person tracked in the room (likely by depth or overhead camera) drives the layout of projected geometry mapped across floor and walls. - Try it: Map a projector to a floor and wall corner and make projected bars rotate to face a tracked walker. Twist: walking clockwise and counter-clockwise produce different sounds. #### Future whiteboard: marker strokes that come alive — Roelof Knol (2026) - Video: https://x.com/tokufxug/status/2072491697557361021 - Interaction: Projection, Drawing & Making, Tangible Objects - Platform & tech: Projection, projection mapping, camera tracking - Idea: Handwriting becomes the trigger for light. - What it is: On a whiteboard, lines drawn with a marker are detected in real time and projected light effects follow the pen and bring the strokes to life. - Technique: A camera watches the board and the pen tip, new strokes are detected as they appear, and a calibrated projector overlays light effects that follow them (likely). - Try it: Point a webcam and projector at a whiteboard, detect new marker strokes by frame differencing, and spawn projected sparks along them. Twist: sparks travel along the strokes and jump between lines that cross. ### Sander Veenhof *AR artist; co-organiser of 'We AR in MoMA'* Dutch artist who has staged uninvited AR interventions since 2010 (MoMA, Venice Biennale) and now prototypes playful social experiments for AR glasses. #### de-surveillance — Sander Veenhof (2009) - Video: https://www.youtube.com/watch?v=SBxdBHO9Ozo - Interaction: Perception & Effects, Tangible Objects, Hands & Body - Platform & tech: Projection, Desktop, chroma key, surveillance camera - Idea: A chroma-key hack that turns the colour blue into an invisibility cloak. - What it is: A police surveillance camera in Rotterdam appears to malfunction: anyone hiding behind a blue object becomes invisible on its screen. - Technique: A public screen shows the camera feed with blue pixels keyed out and replaced by a clean background plate, so blue objects and whoever holds them disappear. - Try it: Build a webcam chroma-key with a captured background plate so a coloured cloth becomes an invisibility cloak on a public screen. Twist: the cloak makes you invisible only to one of two cameras. #### BiggAR — Sander Veenhof (2010) - Video: https://www.youtube.com/watch?v=0bJJuphti-E - Interaction: Shared & Social, Location & City, Play - Platform & tech: Phone, Layar, GPS - Idea: A single sculpture covering the planet that everyone shares and anyone can change. - What it is: The world's biggest interactive sculpture: 7,463,185,678 virtual blocks encapsulate the whole Earth, and anyone can change the colour of every block worldwide with one tap in the Layar app. - Technique: A server keeps one global colour state and the Layar browser generates the nearby blocks procedurally around each viewer's GPS position. - Try it: Build a shared AR grid of blocks around each user with a tiny server (e.g. Firebase) storing one global colour that any user can change. Twist: the colour decays back to grey unless changed every minute. #### We AR in MoMA — Sander Veenhof, Mark Skwarek (2010) - Video: https://www.youtube.com/watch?v=b9T2LVM7ynM - Interaction: Location & City, Portals & Worlds - Platform & tech: Phone, Layar, GPS - Idea: Show up uninvited and stage an AR exhibition inside MoMA. - What it is: An uninvited AR exhibition inside MoMA New York: artworks by Manifest.AR members were geolocated into the galleries and visible only via the Layar app. - Technique: Geolocated Layar layers placed virtual works at GPS coordinates inside MoMA, exploiting the fact that the location-based layer could not be controlled by the institution. - Try it: Stage an 'uninvited' WebAR exhibition in a formal gallery at your school, with each person placing one work and spreading it via QR codes. Twist: every work must respond to a real piece on display in that gallery. #### #occupymoney — Sander Veenhof (2011) - Video: https://www.youtube.com/watch?v=QUc0z5zL5sc - Interaction: Tangible Objects, Information & UI, Shared & Social - Platform & tech: Phone, Layar Vision, Twitter API - Idea: Deliver a protest message straight into the core of the financial system: the banknote. - What it is: Tweet with #occupy1dollar and your message appears on every one-dollar bill in the world when viewed through Layar Vision. - Technique: Layar Vision image recognition tracks the one-dollar bill design and overlays the latest tweets fetched with the hashtag. - Try it: Use image tracking on a common object everyone carries (a banknote, student card, coffee cup) to show messages submitted through a form. Twist: each viewer sees a different random message. #### Battling Pavilions — Sander Veenhof (2011) - Video: https://www.youtube.com/watch?v=PWyA9xB4ybQ - Interaction: Play, Location & City, Shared & Social - Platform & tech: Phone, Layar, GPS - Idea: Turn the competition between national pavilions into an actual game. - What it is: At the Venice Biennale, virtual pavilions extend the national pavilions into AR, and visitors anywhere can play a game that makes the pavilions fight for space. - Technique: Pavilion blocks are geolocated in the Giardini in Layar, and remote and local players' actions update a shared server state that changes which pavilion occupies each spot. - Try it: Turn a campus quad into an AR territory game where teams claim tiles by standing on them with a phone. Twist: remote players online can also claim tiles. #### Virtual Traffic Light — Sander Veenhof (2011) - Video: https://www.youtube.com/watch?v=4cnPFApUyKA - Interaction: Location & City, Information & UI, Shared & Social - Platform & tech: Phone, Layar, GPS - Idea: Test whether a purely virtual sign can change real behaviour. - What it is: The Dutch island of Terschelling, which has no traffic lights, gets its first one: a virtual light in AR that actually tries to regulate traffic. - Technique: A traffic-light model with a timed state cycle is geolocated at a crossing in a mobile AR browser, shared by everyone viewing it. - Try it: Place a virtual sign (queue marker, stop line, quiet zone) in a busy campus spot with shared AR state, and observe whether people obey it. Twist: let the crowd vote on the sign's rule. #### Zoetrope meets AR — Sander Veenhof (2011) - Video: https://www.youtube.com/watch?v=9K1JBsXQG34 - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Phone, marker tracking, zoetrope - Idea: Merge a 2,000-year-old animation device with AR. - What it is: A spinning physical zoetrope is combined with augmented reality so the animation appears as 3D figures moving around the real drum. - Technique: Marker or image tracking follows frames on the rotating drum and swaps in a sequence of 3D models, so the camera sees stroboscopic 3D animation. - Try it: Print 12 image markers around a lazy Susan, map a 12-frame 3D animation onto them and spin it in front of a phone. Twist: use the phone's frame rate as the shutter. #### infiltr.AR — Sander Veenhof (2011) - Video: https://www.youtube.com/watch?v=zai5IVT9QR4 - Interaction: Location & City, Information & UI - Platform & tech: Phone, Layar, Google Maps - Idea: Use geo-AR to place art where no artist is allowed to go, and measure who notices. - What it is: Two AR augments are placed inside the Oval Office and the Pentagon; their views are tracked live on a map to test whether anyone inside ever sees them. - Technique: Augments are geolocated at the coordinates of the two buildings in the Layar browser, and each view request is logged server-side and plotted on a Google Map. - Try it: Place an AR object inside a room on campus that students cannot enter, and log every view with a simple analytics endpoint. Twist: the object changes after its first view from inside. #### Global Augmented Reality Dance — Sander Veenhof (2012) - Video: https://www.youtube.com/watch?v=8RJSRqElfYo - Interaction: Performance, Shared & Social, Hands & Body - Platform & tech: Phone, motion sensors, web app - Idea: The awkward gestures of AR users become a synchronized global dance. - What it is: The typical arm-waving of people looking at AR through phones is turned into a choreography that people all over the world perform at exactly the same moment. - Technique: A web or AR app gives timed on-screen cues synchronized to a global clock so that users worldwide move their phones in the same pattern at the same time. - Try it: Choreograph a 30-second phone dance using motion sensors, with a synced start time, and perform it simultaneously with a partner class elsewhere. Twist: score how in-sync everyone was from gyro data. #### Outdoor Game of Life — Sander Veenhof (2012) - Video: https://www.youtube.com/watch?v=aPGbav1aHH8 - Interaction: Play, Location & City, Shared & Social - Platform & tech: Phone, Layar, GPS - Idea: A cellular automaton living on a real plaza. - What it is: In front of the Albuquerque Museum, block organisms following Conway's Game of Life battle for the virtual space, visible through phones. - Technique: A server steps a Game of Life grid and publishes live cells as geolocated blocks in the Layar browser over the museum forecourt. - Try it: Run Conway's Game of Life on a grid anchored to a floor with plane detection, and let users tap to seed cells. Twist: two teams' colours compete for territory. #### Watch Your Privacy — Sander Veenhof (2014) - Video: https://www.youtube.com/watch?v=7Pbmv5LjWPY - Interaction: Information & UI, Location & City - Platform & tech: Wearable, Google Glass, OpenStreetMap - Idea: Turn the watchers into the watched. - What it is: A Google Glass app shows surveillance cameras around the wearer, plus the live positions of other Glass users. - Technique: OpenStreetMap's surveillance-camera data and shared GPS positions of Glass users are drawn as a real-time location overlay in the head-worn display. - Try it: Map the cameras on your campus into a small dataset and build a phone-AR overlay that marks them with viewing cones. Twist: add a 'blind-spot route' that avoids every cone. #### Tango Teleport — Sander Veenhof (2018) - Video: https://www.youtube.com/watch?v=FZ7We6Sz05I - Interaction: Portals & Worlds, Spatial Mapping, Hands & Body - Platform & tech: Phone, Google Tango, depth sensing - Idea: Depth lets you keep the person and swap the world. - What it is: Using a depth-sensing Tango phone, a person is cut out of reality and 'teleported' into a 360-degree AR environment. - Technique: Tango's depth camera segments pixels closer than a threshold, keeping the foreground person and replacing everything behind with a 360 panorama. - Try it: Use LiDAR depth (ARKit) or people occlusion to keep a classmate and replace the background with a 360 photo of another place. Twist: the destination changes when they jump. #### Distant Hand — Sander Veenhof (2021) - Video: https://www.youtube.com/watch?v=ZdudbnDh7tk - Interaction: Hands & Body, Perception & Effects - Platform & tech: Headset, hand tracking, Snap Spectacles - Idea: Stretch out a long AR hand that can reach faraway things. - What it is: A hand-tracked AR prototype that extends your hand far into the space in front of you to touch distant objects. - Technique: Hand tracking provides the wrist and finger joints, and a non-linear mapping extends the virtual hand far along the arm's ray so it can touch distant objects (akin to the Go-Go interaction technique). - Try it: Use MediaPipe Hands or HoloKit hand tracking to build a virtual hand that grows non-linearly as you extend your arm, and touch distant virtual objects with it. Twist: make the long hand leave touch marks on real objects it passes. #### Multi-person AR fashion — Sander Veenhof (2021) - Video: https://www.youtube.com/watch?v=pYutRPaBYFM - Interaction: Hands & Body, Shared & Social - Platform & tech: Phone, body tracking, Lens Studio - Idea: Several people wear AR digital fashion at the same time. - What it is: Real-time AR garments dress several people at once, exploring shared digital fashion on the street. - Technique: Lens Studio multi-body tracking attaches garment meshes or cloth simulations to several detected skeletons in the same camera frame. - Try it: Use Lens Studio multi-body tracking to dress everyone in the frame in one matching digital outfit. Twist: let the distance between two people set the color or pattern of the clothes, blending when they come close. #### Subway PONG — Sander Veenhof (2021) - Video: https://www.youtube.com/watch?v=n8FF1kxm11o - Interaction: Play, Shared & Social - Platform & tech: Headset, Snap Spectacles - Idea: Play Pong with AR glasses in a subway car. - What it is: A playful AR glasses experiment: a game of Pong played across a real subway carriage with fellow passengers. - Technique: Likely Spectacles world tracking with a shared or co-located session places a Pong field across the carriage, with head or hand pose controlling the paddle. - Try it: Use two phones in the same AR space to build a room-spanning Pong game, with each phone's position acting as the paddle. Twist: set the court in a public place such as an elevator or corridor, with rules that require strangers to cooperate. #### Urban Lasergame AR — Sander Veenhof (2021) - Video: https://www.youtube.com/watch?v=8wDKckP55tM - Interaction: Play, Location & City, Shared & Social - Platform & tech: Phone, AR - Idea: Turn city streets into an AR laser-tag battlefield. - What it is: A location-based AR laser tag game played in city streets. - Technique: Likely GPS positioning plus camera-based person detection (or shared AR sessions) registers hits when a player aims the phone at another player. - Try it: Build a simple laser-tag game with WebAR and person detection, where aiming at a person in the frame and tapping counts as a hit. Twist: a hit does not knock the other player out but leaves an AR decoration on them. #### AI style transfer on live AR camera — Sander Veenhof (2023) - Video: https://www.youtube.com/watch?v=XU8D91ljJ9A - Interaction: Perception & Effects - Platform & tech: Headset, style transfer, AR glasses - Idea: AI turns the world in front of you into a painting in real time. - What it is: Live AI style transfer applied to an AR glasses camera, turning the surroundings into a painting in real time. - Technique: Camera frames from AR glasses are passed through a neural style-transfer model (on-device or remote) and the stylised frames are displayed back with low latency. - Try it: Use style transfer in ml5.js or Lens Studio to render the phone camera feed in a painter's style in real time. Twist: stylize only the people or only the background, and compare what each choice means. ### Tamiko Thiel *Media artist; co-founder of Manifest.AR* Pioneering VR/AR artist (and designer of the Connection Machine CM-1/CM-2) who uses geolocated AR to insert political and ecological 'what-ifs' into specific sites, from the Venice Biennale to the Whitney. #### Art Critic Face Matrix (We AR in MoMA) — Tamiko Thiel (2010) - Video: https://www.youtube.com/watch?v=OlgjnO0asjM - Interaction: Location & City, Portals & Worlds - Platform & tech: Phone, Layar - Idea: Put a matrix of critics' faces inside MoMA without permission. - What it is: Part of the uninvited AR show in MoMA: a matrix of critics' faces floats in the galleries, visible only through the Layar browser. - Technique: Layar geolocated POIs position billboarded 2D/3D content by GPS coordinates inside the museum, so anyone with the app sees it without the institution's permission. - Try it: Use location-based AR.js or 8th Wall to place a set of unauthorized virtual faces or comments in an 'authority' space at school (the principal's door, the library). Twist: draw the faces' content from classmates' real opinions of that space. #### Carnation Rain (Largo do Carmo) — Tamiko Thiel (2011) - Video: https://www.youtube.com/watch?v=VxBEr_bq_0k - Interaction: Location & City - Platform & tech: Phone, Layar, GPS - Idea: An AR rain of carnations falls over the square where the revolution took place. - What it is: An AR rain of red carnations over Lisbon's Largo do Carmo, where the 1974 Carnation Revolution ended. - Technique: A GPS-anchored particle-like object (animated falling flowers) is placed over the historic square, using location as the carrier of historical meaning. - Try it: Pick a campus or city location tied to a historical event and use Unity AR to make objects related to that event fall from the sky. Twist: use a particle system so the fallen objects slowly pile up on the ground into a symbol. #### Invisible Istanbul: Captured (for #RSF_RWB) — Tamiko Thiel (2011) - Video: https://www.youtube.com/watch?v=72u8gwJvcQs - Interaction: Information & UI, Portals & Worlds, Location & City - Platform & tech: Phone, Layar, Twitter API - Idea: Put the viewer inside a bubble of censored speech. - What it is: Tweets from Reporters Without Borders, with every meaningful word censored, are animated on the inside of a sphere that surrounds the viewer in Istanbul. - Technique: Censored tweet text is mapped as animated textures onto the inner surface of a sphere geolocated around the viewer in a mobile AR browser, for ISEA2011 at the Istanbul Biennale. - Try it: Build a WebXR or phone-AR sphere around the viewer covered with live text from a feed, with keywords blacked out. Twist: blacked-out words reveal themselves when the viewer stands still. #### Jasmine Rain (birdcage) — Tamiko Thiel (2011) - Video: https://www.youtube.com/watch?v=AzRVOqtq8M0 - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, Layar, GPS - Idea: One symbolic object installed simultaneously in a museum and in sites of revolution. - What it is: A birdcage of falling jasmine blossoms, a symbol of the Arab Spring, is placed uninvited in MoMA's atrium, and also in Tahrir Square, Tiananmen Square and other sites. - Technique: The same animated 3D sculpture is published at multiple geolocations in a mobile AR browser, so it exists in several cities at once. - Try it: Create one small AR sculpture and publish it at three distant locations with friends in other cities, then compare photos. Twist: each location changes its colour based on local news. #### Newtown Creek (oil spill) — Tamiko Thiel (2011) - Video: https://www.youtube.com/watch?v=i_G8jm5qPS8 - Interaction: Location & City, Information & UI - Platform & tech: Phone, Layar, GPS - Idea: Bring an invisible underground oil spill up to the surface of the river. - What it is: Geolocated AR on Brooklyn's Newtown Creek visualising the huge hidden oil spill beneath it. - Technique: Geolocated AR places a large semi-transparent model of the underground oil plume at its real coordinates, making hidden environmental data spatially visible. - Try it: Find a local environmental dataset (groundwater, noise, drainage network) and use AR to visualize it on the ground at its real location. Twist: scale the visualization 1:1 with the real data values to feel the size of 'invisible quantities'. #### Reign of Gold — Tamiko Thiel (2011) - Video: https://www.youtube.com/watch?v=qRJd8QzLpx8 - Interaction: Location & City, Perception & Effects, Information & UI - Platform & tech: Phone, Layar, GPS - Idea: Make the abstract flow of money a literal downpour around you. - What it is: In front of the New York Stock Exchange, a rain of US gold eagle coins surrounds the viewer and their object of protest, part of the AR Occupy Wall Street project. - Technique: A particle-like field of animated coin models is geolocated around the viewer's position in a mobile AR browser, so it can be carried to any protest site. - Try it: Make a location-independent AR particle rain of a symbolic object (coins, receipts, leaves) around the viewer. Twist: the rain rate follows a live stock or climate index. #### Shades of Absence: Public Voids — Tamiko Thiel (2011) - Video: https://www.youtube.com/watch?v=cykHsfW1NQs - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, Layar, GPS - Idea: Place golden silhouettes of censored artists in Venice's public spaces. - What it is: Geolocated AR at the Venice Biennale placing golden silhouettes of censored artists in public squares. - Technique: GPS-anchored Layar content places flat golden silhouette billboards in public squares, relying on coarse geolocation and compass heading. - Try it: Pick a theme about absence (forgotten people, demolished buildings) and use WebAR to place golden silhouettes at the matching locations. Twist: a silhouette is visible only when nobody is standing in that spot. #### All Hail Damien Hirst — Tamiko Thiel (2012) - Video: https://www.youtube.com/watch?v=On417tdUxFA - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, Layar, GPS - Idea: A pun turned into weather: art-market money literally hailing on the star artist. - What it is: During Damien Hirst's Tate Modern retrospective, a hail of Diamond Jubilee gold sovereigns pours down on the museum and on Hirst's own sculpture Hymn outside. - Technique: Animated gold coin models are geolocated as a falling field above and around Tate Modern in a mobile AR browser. - Try it: Pick a pun about a place or person and build a phone-AR 'weather' effect that makes it literal. Twist: the effect intensifies when more visitors are counted. #### Transformation: mARping Lehel — Tamiko Thiel (2012) - Video: https://www.youtube.com/watch?v=VbbOzKc2n98 - Interaction: Location & City, Information & UI - Platform & tech: Phone, Layar, GPS - Idea: Map the invisible history and possible future of a neighbourhood onto its streets. - What it is: In Munich's Lehel neighbourhood, AR waterwheels turn where old streams once crossed the streets, alongside other visions of a sustainable city 30 years from now. - Technique: Historical stream maps are used to geolocate animated 3D waterwheels and future-city objects in a mobile AR browser across the district. - Try it: Find an old map of your campus area, locate a lost stream, path or building, and place an AR marker of it at the exact spot. Twist: add a second layer showing what could be there in 2050. #### Waterlily Invasion — Tamiko Thiel (2014) - Video: https://www.youtube.com/watch?v=B1wVL_YSodE - Interaction: Gaze & Attention, Location & City, Perception & Effects - Platform & tech: Phone, Layar, GPS - Idea: The more you look, the bigger it grows. - What it is: Mutated AR water lilies that feed on the mediated human gaze grow and engulf viewers who keep looking at them through their phones. - Technique: A geolocated plant model scales up over time while it stays in the camera view, likely using the AR browser's focus or distance events. - Try it: Make an AR plant that grows only while it is in the centre of the camera view and shrinks when ignored. Twist: it spreads to a new spot when two people look at it at once. #### Gardens of the Anthropocene — Tamiko Thiel (2016) - Video: https://www.youtube.com/watch?v=lQWPHmLB_OY - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, Layar, Unity - Idea: Plants mutated by climate change grow in a real harbor. - What it is: AR plants mutated by climate change (such as giant red algae) grow over real harbours and gardens. - Technique: Geolocated and plane-anchored 3D plant models (built in Unity, shown via Layar) are placed over real harbours and gardens at plausible scale. - Try it: Design a "mutated version" of a real plant on campus as it might look under the climate of 2100, and place it next to the original in AR. Twist: add a time slider so the plant gradually mutates from now into the future. #### I am Sound — Tamiko Thiel (2016) - Video: https://www.youtube.com/watch?v=d66v3GFFEtU - Interaction: Face, Voice & Sound, Projection - Platform & tech: Projection, face detection, metallophone, projector - Idea: Your face becomes a musical score played on the screen that shows it. - What it is: Your face, captured by a camera, is projected as a fractured cubist portrait on a curtain of metal plates that is also a 12-tone metallophone, which plays a composition generated from your face. - Technique: Face detection extracts features that are mapped to notes struck on real metal plates, while the video is projected onto those same plates so the vibration shakes the image. - Try it: Use a webcam face-tracking sketch (p5.js or TouchDesigner) to turn facial landmarks into notes and project the face onto a surface. Twist: two faces together play a duet. #### Unexpected Growth — Tamiko Thiel (2018) - Video: https://vimeo.com/298681675 - Interaction: Spatial Mapping, Perception & Effects, Location & City - Platform & tech: Phone, ARKit, Unity - Idea: A future ecosystem built from our garbage, growing on a museum terrace. - What it is: Commissioned by the Whitney Museum, a coral-reef-like growth made of plastic trash spreads over the museum's terrace, as if the sea had risen and the reef had adapted to our waste. - Technique: A mobile AR app anchors procedurally varied plastic-coral models to the terrace, and the growth likely changes with the time of day. - Try it: Scan a few pieces of real packaging trash, turn them into 3D models and build an AR 'reef' that grows across a floor with plane detection. Twist: the reef grows faster when it rains. #### Evolution of Fish — Tamiko Thiel (2019) - Video: https://www.youtube.com/watch?v=Wi9RCNHJ6RQ - Interaction: Perception & Effects - Platform & tech: Phone, Unity, ARKit - Idea: Fish evolve into plastic-bag creatures in AR. - What it is: An AR artwork of fish evolving into plastic-bag creatures; here the artist plays with video feedback in the AR view. - Technique: An ARKit plane-anchored animated model shows fish morphing into plastic forms (blend shapes or model swaps), and screen recording of a screen within the AR view produces video feedback. - Try it: Use Unity AR to animate a fish gradually morphing into a plastic bag (via blend shapes or model swaps) and place it next to a fish tank or water cooler. Twist: point the phone at another screen showing the AR view to create a video feedback loop. #### Enter the Plastocene — Tamiko Thiel (2021) - Video: https://www.youtube.com/watch?v=6p6eYXCsV9g - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, Unity, ARKit - Idea: The Plastocene, where plastic creatures swim through the gallery. - What it is: An AR/VR installation where plastic waste creatures swim through the exhibition space around visitors, shown at MEET Milan. - Technique: ARKit world tracking places flocking creatures (boids-style steering) that swim through the exhibition volume around visitors. - Try it: Use a simple boids algorithm in Unity to make a school of plastic-bottle creatures swim around the room and avoid viewers, treating the camera position as a repulsion point. Twist: tie the number of creatures to the amount of plastic waste the class produces that day. #### ReWildAR — Tamiko Thiel (2021) - Video: https://vimeo.com/655628879 - Interaction: Spatial Mapping, Location & City, Perception & Effects - Platform & tech: Phone, ARKit, Unity - Idea: Let nature reclaim a monument to industry. - What it is: In the rotunda of the Smithsonian Arts and Industries Building, AR plants overgrow the historic hall, a meditation on invasive species and rewilding. - Technique: Plant models are likely anchored to the rotunda's floor and columns with plane and image tracking in a mobile AR app, spreading over the architecture. - Try it: Pick one invasive plant in your region and build an AR vine that climbs detected walls and floors in a hallway. Twist: users can 'weed' it by tapping, but it grows back somewhere else. #### Waldwandel / Forest Flux — Tamiko Thiel (2023) - Video: https://www.youtube.com/watch?v=EKYzlzC2WXU - Interaction: Projection, Perception & Effects, Information & UI - Platform & tech: Projection, Phone, Unity, livestream compositing - Idea: Watch a forest's future unfold as a live mixed-reality projection. - What it is: An immersive AR livestream installation, made with /p, in which Bavarian forest trees transform to show how climate change will reshape the forest, shown at Kunsthalle Munich and Chiostro del Bramante, Rome. - Technique: An AR scene of evolving tree models is composited over a live camera feed and projected in the gallery as a livestream, based on forestry data on future species. - Try it: Film a campus tree with a phone, composite an AR layer that ages or replaces it with a species expected in 2080, and project the live result. Twist: the audience's movement speeds up time. ### Acute Art *AR/VR art production platform (director Daniel Birnbaum)* Production studio and app that commissions blue-chip artists (Olafur Eliasson, KAWS, Marina Abramović, Precious Okoyomon and others) to make AR sculptures placeable anywhere through a phone. #### KAWS: Expanded Holiday — Acute Art (2020) - Video: https://www.youtube.com/watch?v=YI8bG4KAB_w - Interaction: Location & City, Spatial Mapping - Platform & tech: Phone, Acute Art app - Idea: Giant AR toy sculptures appear worldwide at the same time, and you can buy one to take home. - What it is: KAWS's giant COMPANION figure released as AR sculptures placed simultaneously in cities worldwide and in users' homes, sold as limited AR editions. - Technique: Geolocated AR placement plus user-side plane detection: the same large sculpture model is anchored at fixed city coordinates and can also be placed on any detected ground plane at home. - Try it: Make a giant AR sculpture of a character you designed and have it appear at three GPS locations on campus at once, for example with location-based AR.js. Twist: give the sculpture a different pose at each location, hinting that it moves around the city. #### Olafur Eliasson: WUNDERKAMMER — Acute Art (2020) - Video: https://vimeo.com/418385263 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, Acute Art app, ARKit/ARCore - Idea: During the pandemic, AR brought the sun, clouds, rainbows and other natural phenomena into people's homes. - What it is: A collection of AR objects by Olafur Eliasson (a sun, rain cloud, rainbow, puffin, compass) placed in the viewer's home during lockdown; the AR sun even charges a virtual Little Sun lamp. - Technique: Plane detection in the Acute Art app (ARKit/ARCore) places animated models of natural phenomena at real scale in the home, with simple interaction logic such as sun exposure charging a virtual lamp. - Try it: Build a natural phenomenon at home with AR Foundation: put a raining cloud or a small sun in the living room and give it a cause-and-effect link with another virtual object. Twist: the phenomenon appears only when the real weather is the opposite, such as sunshine indoors while it rains outside. #### Unreal City — Acute Art (2020) - Video: https://www.youtube.com/watch?v=-gO3oz5UfUY - Interaction: Location & City - Platform & tech: Phone, Acute Art app - Idea: A trail of AR sculptures along the Thames. - What it is: An AR public art walk along London's Thames with 36 works by artists such as Olafur Eliasson, Cao Fei and Tomás Saraceno, curated with Dazed. - Technique: GPS/geolocated anchors (refined with local plane detection) place a sequence of sculptures along a walking route, with the app acting as a map-guided exhibition. - Try it: Have each student in the class make one AR work, install them by GPS position along a campus walkway, and link them together with WebAR. Twist: design a final work for the exhibition that can be seen only after walking the whole route. #### The Looking Glass: Darren Bader — Acute Art (2021) - Video: https://www.youtube.com/watch?v=Q4aCI3Dol1o - Interaction: Location & City, Perception & Effects, Play - Platform & tech: Phone, Acute Art app, mobile AR - Idea: Place absurd, funny AR sculptures in a city plaza to ask what counts as art. - What it is: Darren Bader's surreal AR sculptures on The Shed's plaza use humour to question art and contemporary life, visible through the Acute Art app. - Technique: Mobile AR anchors absurd 3D object combinations at fixed plaza locations via the Acute Art app. - Try it: Combine two unrelated everyday objects into an AR sculpture, place it on campus and give it a title. Twist: let viewers vote for the most absurd combination and swap it in next week. #### The Looking Glass: Precious Okoyomon — Acute Art (2021) - Video: https://www.youtube.com/watch?v=AG_NXEvKxBU - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, Acute Art app, geolocated AR - Idea: Summon soft creatures about nature and identity into a city plaza with AR. - What it is: On The Shed's public plaza and the High Line, Precious Okoyomon's AR works — creatures and nature-based forms — appear through the Acute Art app as part of the group show The Looking Glass. - Technique: Site-specific AR placement in the Acute Art app anchors animated 3D sculptures to geolocated spots on the plaza. - Try it: Design an AR creature for a campus plaza that expresses the return of nature, and decide the exact patch of ground where it alone appears. Twist: it changes color with the time of day. #### The Looking Glass: Tomás Saraceno (spiders in AR) — Acute Art (2021) - Video: https://www.youtube.com/watch?v=2UDvzuecX2Q - Interaction: Spatial Mapping, Location & City, Perception & Effects - Platform & tech: Phone, Acute Art app, plane detection - Idea: Let AR spiders crawl through the city to rethink how we relate to other species. - What it is: Tomás Saraceno's AR spiders crawl across The Shed's plaza, part of his project to make spiders and their webs part of our lives and rethink our relationship with other species. - Technique: Scanned/animated spider models are placed with plane detection in the Acute Art app so they crawl on real ground surfaces. - Try it: Use Reality Composer or Adobe Aero to place an AR insect that crawls along the ground, and write one line describing its point of view. Twist: the insect scurries away when a viewer's phone comes close. ### Aidan Wolf *AR developer; co-founder of Dream Park* Long-time Snap AR developer who made DoodleLens (doodles into AR), multiplayer Snapchat lenses and Spectacles games, and now builds 'downloadable' mixed-reality theme parks with Dream Park. #### DoodleLens — Aidan Wolf (2019) - Video: https://x.com/Aidan_Wolf/status/1124997833159929856 - Interaction: Drawing & Making, Tangible Objects - Platform & tech: Phone, Unity, ARKit, ARCore - Idea: Snap a doodle on paper and it peels off the page as a living AR sticker. - What it is: Point your phone at a doodle on paper and it is lifted off the page into AR, inspired by an old camcorder chroma feature; later updates animate doodles like flipbooks and turn them into 3D models. - Technique: The camera image is thresholded to extract dark ink strokes into a transparent texture, which is placed on a billboard anchored in AR space. - Try it: Use p5.js or Lens Studio to cut a doodle on white paper into a transparent texture and place it in AR. Twist: four doodles captured in a row form a frame-by-frame animation. #### DoodleLens flipbook animation — Aidan Wolf (2019) - Video: https://x.com/Aidan_Wolf/status/1130730833445314560 - Interaction: Drawing & Making - Platform & tech: Phone, Unity, ARKit, ARCore - Idea: Turn a stack of doodles into a flipbook animation standing in the real world. - What it is: A series of doodles captured one after another plays back as a looping flipbook animation standing in the real world. - Technique: Each captured doodle becomes a frame in a sprite sequence that cycles on an AR-anchored quad. - Try it: Have each student draw one frame and combine them into a whole-class frame-by-frame animation placed in the AR classroom. Twist: playback speeds up as viewers get closer. #### Doodles into 3D models — Aidan Wolf (2019) - Video: https://x.com/Aidan_Wolf/status/1168518755648974849 - Interaction: Drawing & Making - Platform & tech: Phone, Unity, ARKit - Idea: Extrude a flat doodle into a 3D object you can walk around. - What it is: A flat drawing is extruded into a chunky 3D object that can be placed and walked around in AR. - Technique: The extracted doodle outline is triangulated and extruded into a mesh with the drawing as its front texture (likely contour extrusion). - Try it: Extrude a hand-drawn outline SVG into 3D in Blender or three.js and bring it into AR. Twist: the extrusion depth depends on how dark the drawing's colors are. #### Project Play (Magic Leap) — Aidan Wolf (2019) - Video: https://x.com/Aidan_Wolf/status/1182082317155618816 - Interaction: Drawing & Making, Hands & Body - Platform & tech: Headset, Magic Leap One, Unity - Idea: Draw, animate and even program directly inside a headset. - What it is: A throwback to his Magic Leap app where you could draw, animate and even program objects directly in mixed reality. - Technique: Controller-driven 3D drawing strokes become objects that can be keyframed and given simple behaviors in Unity on Magic Leap. - Try it: Draw 3D lines in Quest or phone AR, then give a line a 'follow me' behavior. Twist: play a sound when two lines touch. #### Copy and paste into AR — Aidan Wolf (2020) - Video: https://x.com/Aidan_Wolf/status/1252117262166822912 - Interaction: Drawing & Making, Information & UI - Platform & tech: Phone, Unity, ARKit - Idea: Copy an image and paste it straight into real space. - What it is: Images and art copied on the phone are pasted straight into the surrounding space as AR objects, with no editing step. - Technique: Clipboard images are background-removed and spawned as textured quads at the AR raycast hit point. - Try it: Build web AR with 8th Wall or WebXR that lets you pick an image from the photo album and stick it on a wall as a virtual poster. Twist: posted images can be seen and commented on by other classmates. #### Hot Potato multiplayer lens — Aidan Wolf (2021) - Video: https://x.com/Aidan_Wolf/status/1395445443333476352 - Interaction: Shared & Social, Play - Platform & tech: Phone, Lens Studio, Connected Lenses - Idea: Pass an exploding virtual potato among friends. - What it is: Friends pass a ticking virtual hot potato between their phones in a co-located Snapchat lens until it explodes on someone. - Technique: A connected lens synchronizes ownership of a shared AR object and a countdown timer across participants. - Try it: Build a multiplayer pass-the-bomb AR game with Lens Studio or Unity Netcode. Twist: you can only throw the bomb by pointing at the other person's phone. #### PONK connected lens — Aidan Wolf (2021) - Video: https://x.com/Aidan_Wolf/status/1402742233451798529 - Interaction: Shared & Social, Play - Platform & tech: Phone, Lens Studio, Connected Lenses - Idea: Two people play AR Pong in the same real space. - What it is: Two friends play classic Pong in AR, each hitting the ball from their own phone in a shared Snapchat lens. - Technique: Snap Connected Lenses sync game state and a shared coordinate frame between devices so both players see the same ball and paddles. - Try it: Build a two-player AR mini-game with Lens Studio Connected Lens or Photon. Twist: players control their paddles with the position of their faces. #### Minecraft remade in AR — Aidan Wolf (2023) - Video: https://x.com/Aidan_Wolf/status/1625580832042758144 - Interaction: Play, Spatial Mapping, Drawing & Making - Platform & tech: Phone, Niantic Lightship, Unity - Idea: Dig, fight and build on the real ground just like Steve. - What it is: He rebuilt Minecraft mechanics in AR so you can dig into, fight on and build blocks over the real ground around you. - Technique: Real-world meshing aligns a voxel grid to the ground so placed and dug blocks stick to real surfaces (likely Lightship meshing). - Try it: Build a block-placing construction toy on a ground mesh with AR Foundation. Twist: blocks cast shadows in the direction of the real sunlight. #### Touch Grass lens — Aidan Wolf (2023) - Video: https://x.com/Aidan_Wolf/status/1701565816385179877 - Interaction: Location & City, Hands & Body, Play - Platform & tech: Phone, Lens Studio, ML segmentation - Idea: It checks that you really touched grass and hands you proof that you went outside. - What it is: A tongue-in-cheek AR lens detects when you actually touch grass and rewards you, so you can prove online that you go outside. - Technique: Hand tracking combined with a ground/grass segmentation or classification model checks that the hand overlaps real grass (likely Lens Studio ML). - Try it: Train a grass/not-grass classifier with Teachable Machine and award a badge when the phone sees a hand on grass. Twist: switch to recognizing leaves, stones and other natural objects on campus to fill a collection guide. #### Eternal rat prism AR pet — Aidan Wolf (2024) - Video: https://x.com/Aidan_Wolf/status/1784698481518747859 - Interaction: Play, Perception & Effects - Platform & tech: Phone, Lens Studio - Idea: Turn the 'spinning rat in a prism' meme into an AR pet that follows you everywhere. - What it is: An internet meme, the spinning rat in a prism, becomes a Snapchat AR pet that follows you around wherever you go. - Technique: The pet is re-positioned each frame to trail the camera with smoothing so it appears to follow the user around. - Try it: Turn a class in-joke into an AR pet that follows you around (Lens Studio or Reality Composer). Twist: if you stop for too long, it falls asleep. #### Hand-frame photos left in space — Aidan Wolf (2024) - Video: https://x.com/Aidan_Wolf/status/1838941247148278146 - Interaction: Hands & Body, Shared & Social, Location & City - Platform & tech: Headset, Snap Spectacles, Lens Studio - Idea: Frame a shot with your fingers to take a photo, and the photo stays where it was taken for others to find. - What it is: At a Snap AR hackathon his team built a way to take a photo by framing a scene with your fingers and leave that photo floating in place for others to find. - Technique: Hand tracking detects a two-hand frame gesture to crop the camera view, and the image is saved as a persistent anchored object (likely via cloud anchors). - Try it: Use MediaPipe gesture recognition to detect a two-hand viewfinder and crop the camera image. Twist: pin each photo to a map at the spot where it was taken, for a class-wide treasure hunt. #### Parking-lot kart race — Aidan Wolf (2024) - Video: https://x.com/Aidan_Wolf/status/1762908892311511228 - Interaction: Play, Location & City, Spatial Mapping - Platform & tech: Phone, Headset, AR, Unity - Idea: Turn any parking lot into a Mario Kart track. - What it is: A mixed-reality game turns any empty parking lot into a Mario Kart-style race track with virtual boosts and items, playable at Two Bit Circus. - Technique: A track, item boxes and boost pads are generated to fit the detected open ground and anchored in world space for players moving through it. - Try it: Draw a virtual track and boost strips on a basketball court in AR and have classmates race while pushing chairs. Twist: anyone who hits a virtual banana peel must spin around once on the spot. #### Spectacles walk RPG — Aidan Wolf (2024) - Video: https://x.com/Aidan_Wolf/status/1838276874860138706 - Interaction: Play, Location & City - Platform & tech: Headset, Snap Spectacles, Lens Studio - Idea: Put on the glasses and an everyday walk becomes a monster-fighting, level-up adventure. - What it is: A Snap Spectacles game made in a weekend turns an everyday walk into an RPG adventure, with monsters and loot appearing along the real path. - Technique: World tracking on Spectacles places encounters at intervals along the walked path, and hand tracking is used to fight. - Try it: Make a 'walk-to-school RPG' in phone AR: a monster spawns every 50 steps and you defeat it with hand gestures. Twist: the type of monster depends on the weather. #### Tony Hawk Pro Skater in AR — Aidan Wolf (2024) - Video: https://x.com/Aidan_Wolf/status/1752447360867950799 - Interaction: Play, Location & City, Hands & Body - Platform & tech: Phone, AR, Unity - Idea: Overlay a skateboarding game's scores and collectibles on a real skatepark. - What it is: At Venice Beach Skatepark, skaters see AR score popups, trick names and collectibles over their real runs, remaking the video game on real concrete. - Technique: Collectibles and score UI are anchored in the real park while the skater's motion is tracked relative to them (likely body/camera tracking). - Try it: Place a line of virtual coins on the sports field in AR, and score classmates as they run or cycle through them. Twist: passing through three coins in a row triggers a 'combo' effect. #### Dream Park street access point — Aidan Wolf (2025) - Video: https://x.com/Aidan_Wolf/status/1919248277792244155 - Interaction: Location & City, Play, Shared & Social - Platform & tech: Phone, Headset, Meta Quest 3, location-based AR - Idea: Install an 'entrance' on a street, and the whole street becomes a downloadable mixed-reality theme park. - What it is: A public Dream Park access point on Santa Monica's Third Street Promenade turns the whole street into a mixed-reality game for passers-by. - Technique: Game content is authored to a specific mapped location so devices localize against it and see the same attractions in the real street. - Try it: Design an 'AR theme park' route across campus with three playable AR mini-game stops (Lens Studio or 8th Wall). Twist: the last attraction opens only when three people are there together. #### DreamPark arcade vending machine — Aidan Wolf (2026) - Video: https://x.com/Aidan_Wolf/status/2028538481803428094 - Interaction: Location & City, Play - Platform & tech: Headset, Phone, mixed reality - Idea: A 'vending machine' turns an entire arcade into a mixed-reality playground. - What it is: A DreamPark 'vending machine' installed at Dave & Buster's Hollywood turns the entire arcade floor into a mixed-reality game. - Technique: A fixed physical kiosk serves as the localization anchor and entry point for location-specific MR content laid over the arcade floor. - Try it: Design a physical 'entrance' (a poster or a cardboard machine) that, once scanned, starts an AR game covering the whole classroom. Twist: the entrance sells a different 'item' each day. ### Mark Skwarek *Artist; NYU Mobile AR Lab director; Manifest.AR co-founder* AR activist-artist behind unauthorised AR interventions such as 'The Leak in Your Hometown', which turned every BP logo into a gushing oil pipe during the 2010 spill. #### AR Sun Project — Mark Skwarek, Damon Loren Baker (2010) - Video: https://www.youtube.com/watch?v=zk2VaYlKebc - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, Layar, GPS, compass - Idea: Replace a missing piece of nature with a synced virtual stand-in. - What it is: A virtual sun hangs in the sky of Dumbo, Brooklyn, tracking the real sun's path so that polluted or overshadowed places can still see a sun. - Technique: The sun's azimuth and elevation are computed from time and location, and a glowing sphere is placed at that direction in a sensor-based mobile AR browser. - Try it: Use a sun-position formula and phone compass/gyro (WebXR or AR Foundation) to place a virtual sun exactly where the real one is. Twist: show the sun as it would be at this spot on your birthday. #### The Leak in Your Hometown — Mark Skwarek (2010) - Video: https://www.youtube.com/watch?v=V6-BbqANr04 - Interaction: Tangible Objects, Perception & Effects, Location & City - Platform & tech: Phone, marker tracking, junaio - Idea: Point at any BP logo and AR crude oil gushes out of it. - What it is: Point a phone at any BP logo and an AR broken pipe gushes oil out of it, bringing the Deepwater Horizon spill to your own street. - Technique: Image (logo) marker tracking in junaio detects any BP logo and anchors an animated broken pipe with an oil particle effect to it. - Try it: Pick a brand logo you want to criticize and use MindAR or 8th Wall image tracking to make it 'leak' something that stands for its problem. Twist: let the leak grow the longer viewers point at it until it floods the whole screen. #### Warcraft ARmor — Mark Skwarek (2010) - Video: https://www.youtube.com/watch?v=1i2Qxlln_B0 - Interaction: Hands & Body, Play - Platform & tech: Desktop, marker tracking - Idea: Put game armor on a real person. - What it is: A marker-based AR demo that dresses the user's body in World of Warcraft-style armour on screen. - Technique: Printed fiducial markers worn on the body are tracked by a webcam, and armour pieces are rendered at each marker's pose. - Try it: Stick a few printed markers on your arms and chest and use MindAR to 'dress' your body in armor or clothing you designed. Twist: replace the armor with gear that stands for some everyday pressure (homework, notifications). #### erasAR: Statue of Liberty erased — Mark Skwarek (2010) - Video: https://www.youtube.com/watch?v=afaggs_RJ7U - Interaction: Portals & Worlds, Location & City, Perception & Effects - Platform & tech: Phone, Layar, GPS - Idea: Diminished reality as protest: erase a national monument instead of adding one. - What it is: Through a phone pointed at Liberty Island, the Statue of Liberty is gone and only her empty pedestal remains. - Technique: A geolocated 3D patch of sky and empty pedestal is rendered over the statue's GPS position in a mobile AR browser, likely Layar, hiding the real monument behind a matched backdrop. - Try it: Pick a landmark visible from campus, photograph the empty sky behind it and build a phone-AR overlay that hides it from one viewpoint. Twist: let viewers vote on which landmark disappears next. #### Occupation Forces — Mark Skwarek (2011) - Video: https://www.youtube.com/watch?v=EpB8rngt6iM - Interaction: Location & City, Play - Platform & tech: Phone, Layar, iPhone - Idea: Borrow alien-invasion myth to make viewers feel what occupation of their own streets would be like. - What it is: Aliens occupy a Boston park and other sites: viewers with phones discover invading ships and troops that mimic military occupation abroad. - Technique: Animated 3D alien craft and troops are placed as site-specific geolocated augments viewed on iPhone, building a narrative across several locations. - Try it: Stage a three-stop AR 'invasion' trail on campus with location-based markers, each stop revealing more of a story. Twist: the invaders react to how many viewers are present. #### The Augmented Reality Korean Unification Project — Mark Skwarek (2011) - Video: https://www.youtube.com/watch?v=In8Zrk6tGpI - Interaction: Location & City, Portals & Worlds - Platform & tech: Phone, Layar, GPS - Idea: Use AR to preview a political future that is physically impossible today. - What it is: At the Korean Demilitarized Zone, phones erase the border fortifications and show the land returned to nature, uniting North and South in AR. - Technique: Geolocated 3D landscape augments are anchored at DMZ viewpoints in a mobile AR browser so the fences and watchtowers are covered by virtual terrain and greenery. - Try it: Find a fence, wall or barrier on campus and make a phone-AR scene that replaces it with what the space could become if it were removed. Twist: show both sides' wishes at once. #### The Island of Hope (Venice Biennale AR Intervention) — Mark Skwarek (2011) - Video: https://www.youtube.com/watch?v=QvllOmeEGtw - Interaction: Location & City, Information & UI, Shared & Social - Platform & tech: Phone, Layar, Twitter API - Idea: A live social feed becomes physical-looking objects raining over a real city. - What it is: An uninvited AR island in Venice reads the day's most hopeful #hope tweets and answers the best ones with objects that erupt from its volcano and settle over the city. - Technique: A server script searches Twitter for #hope, picks tweets and spawns matching 3D objects as geolocated points of interest in a mobile AR browser around the Giardini and Venice. - Try it: Build a phone-AR scene where messages typed into a shared form become 3D objects that float up from one spot on campus. Twist: only the most upvoted wish is granted each hour. #### US - Iraq War Memorial — Mark Skwarek, John Craig Freeman (2011) - Video: https://www.youtube.com/watch?v=-H3p7uCio4U - Interaction: Location & City, Information & UI - Platform & tech: Phone, Layar, GPS - Idea: An uninvited memorial that anyone can visit with a phone. - What it is: A geolocated AR memorial lists and visualizes the dead of the Iraq war, placed in public space where no official monument exists. - Technique: Casualty data is turned into 3D markers or text objects and published as geolocated points of interest in a mobile AR browser at symbolic sites. - Try it: Take a public dataset about a hidden loss (trees cut, species lost, closed shops) and place one AR marker per item on a campus lawn. Twist: markers fade unless someone visits them. #### protestAR / #arOCCUPYWALLSTREET — Mark Skwarek (2011) - Video: https://www.youtube.com/watch?v=kw9fpt4JPII - Interaction: Location & City, Shared & Social - Platform & tech: Phone, Layar, GPS - Idea: Virtual protesters can occupy space that real bodies are banned from. - What it is: Occupy protesters barred from Wall Street are placed back in front of the New York Stock Exchange as virtual marchers and signs that police barricades cannot stop. - Technique: Protesters' avatars and signs are uploaded as geolocated 3D points of interest on the Layar AR browser at the NYSE coordinates, so anyone at Wall Street sees them through the camera. - Try it: Let each student design a virtual placard and pin it with location-based WebXR or a Lens at a place on campus they cannot normally enter. Twist: the placards only appear when two or more phones are present. #### 2012 Whitney Biennial AR Intervention (Flood) — Mark Skwarek (2012) - Video: https://www.youtube.com/watch?v=UfqUgvtcGNY - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, Layar, GPS - Idea: Show a future climate disaster at full scale where people already are. - What it is: Around the Whitney Museum, a monumental virtual flood rises; cars are slowly carried off to sea and the museum lobby fills with water. - Technique: A large animated water plane and floating car models are geolocated around the museum in a mobile AR browser so the waterline appears at street level. - Try it: Use plane detection to place a rising water surface in a room or courtyard at a projected sea-level height. Twist: tie the height to real projections for your city in 2100. #### Eyebeam mixed-reality performance — Mark Skwarek, Alan Sondheim, Foofwa d'Imobilité (2012) - Video: https://www.youtube.com/watch?v=MstwSLL6RZM - Interaction: Performance, Projection, Hands & Body - Platform & tech: Phone, Projection, Second Life, Layar - Idea: Stack three realities, a virtual world, a live dancer and an AR layer, in one room. - What it is: At Eyebeam, Alan Sondheim projects Second Life onto the wall, Foofwa d'Imobilité dances in the room, Azure Carter sings, and Mark Skwarek adds AR that viewers see on their phones. - Technique: A projected Second Life session and a live dancer share the space while geolocated or marker-based AR objects are added on smartphones as a third layer. - Try it: Run a 10-minute performance where one student dances, one projects a game world and one places AR objects that the audience sees on phones. Twist: the dancer can only see the AR layer. #### The Augmented Reality Human Body — Mark Skwarek (2012) - Video: https://www.youtube.com/watch?v=ioyE3SXsEsQ - Interaction: Hands & Body, Information & UI - Platform & tech: Phone, biofeedback sensors, marker tracking - Idea: Biofeedback made visible: see inside your own body and calm it by watching. - What it is: Viewers see their own lungs and heart through the phone in real time; as they slow their breathing, the virtual lungs slow down too. - Technique: Breathing and heart-rate sensor data drive animated organ models registered to the viewer's torso in mobile AR, built by Skwarek's NYU Polytechnic AR class. - Try it: Use a phone camera heart-rate app or a cheap pulse sensor to drive an AR heart placed on a body-tracked chest (Lens Studio or AR Foundation). Twist: two people's hearts try to sync. #### The Bottomless Pit — Mark Skwarek, Will Pappenheimer (2012) - Video: https://www.youtube.com/watch?v=LuWh7DNtc9U - Interaction: Portals & Worlds, Location & City, Shared & Social - Platform & tech: Phone, Layar, GPS - Idea: A playful impossible hole links two opposite points on the planet. - What it is: At the ZERO1 Biennial a virtual hole to the other side of the Earth opens in San Jose; the public drops objects through it to reach the antipode. - Technique: A geolocated 3D pit is anchored in a mobile AR browser, with a paired augment placed at the antipodal coordinates where dropped objects likely reappear. - Try it: Build a floor portal with plane detection that shows a live or recorded view from a partner location far away. Twist: objects thrown in on one phone appear on the other. #### MonstAR Island — Mark Skwarek (2013) - Video: https://www.youtube.com/watch?v=A_WvPrCOPcg - Interaction: Hands & Body, Location & City, Play - Platform & tech: Phone, Semblance AR - Idea: A place that transforms anyone who enters it. - What it is: At the Dumbo Arts Festival, visitors on an 'island' are turned into monsters when seen through smartphones. - Technique: Monster models are overlaid on people within a geofenced area using the Semblance AR app, likely by anchoring avatars to viewer positions rather than full body tracking. - Try it: Make a body-tracking Lens or AR Foundation scene that turns anyone standing inside a taped square into a creature. Twist: the creature changes with how many people share the square. ### MIT Tangible Media Group — Hiroshi Ishii *Research group at the MIT Media Lab led by Hiroshi Ishii (Tangible Bits, Radical Atoms)* Hiroshi Ishii's Tangible Media Group has pursued 'Tangible Bits' and 'Radical Atoms': interfaces where digital information takes physical, shape-changing form. Daniel Leithinger, Sean Follmer and others built the inFORM and TRANSFORM pin displays and combined them with projection and see-through AR. #### AR-Jig — MIT Tangible Media Group — Hiroshi Ishii (2007) - Video: https://vimeo.com/48765576 - Interaction: Tangible Objects, Drawing & Making - Platform & tech: Desktop, tangible tool, AR display - Idea: Use a row of pushable physical pins as a hand tool to scrape out a digital surface directly. - What it is: A handheld pin-array 'jig' whose physical profile controls a projected/AR 3D curve, letting designers sculpt digital surfaces with a tangible tool. - Technique: A handheld tool with an array of linear pins whose displacements are sensed is tracked in space, and the pin profile defines control points that deform a displayed 3D spline surface. - Try it: Use a row of sliding paper strips or straws (with the phone camera reading their heights) as a physical 'curve ruler', turn the profile into a live spline in three.js, and revolve it into a vase. Twist: drag the ruler along the table to sweep out a whole surface. #### T(ether) — MIT Tangible Media Group — Hiroshi Ishii (2012) - Video: https://vimeo.com/42173010 - Interaction: Shared & Social, Drawing & Making, Hands & Body - Platform & tech: Phone, Vicon motion tracking, iPad - Idea: Several people use tablets as windows and shape and animate 3D objects by hand in a shared space above a table. - What it is: Spatially-aware tablets act as windows into a shared 3D scene above a table; several users sculpt and animate virtual objects with gestures behind the screens. - Technique: Vicon motion capture tracks each iPad and the users' gloved hands, so each tablet renders a correct perspective view into a shared 3D scene and hand positions behind the screen manipulate virtual objects. - Try it: Use AR Foundation shared anchors so two or three phones see the same virtual sculpture, each acting as a "window" from a different angle, and pinch vertices by tapping and dragging. Twist: each phone can only see or edit part of the sculpture, so finishing it takes teamwork. #### Sublimate — MIT Tangible Media Group — Hiroshi Ishii (2013) - Video: https://vimeo.com/63284274 - Interaction: Tangible Objects, Perception & Effects, Information & UI - Platform & tech: Projection, Headset, shape display, see-through AR, stereo - Idea: Let data change phase like matter, between solid physical form and gaseous hologram. - What it is: Combines a shape display with see-through stereoscopic AR so data can switch between physical form (pins) and virtual 'gas' form floating above the table. - Technique: A shape display is registered with a head-tracked stereoscopic see-through display (half-silvered mirror or tracked tablet), so virtual geometry is rendered in the same coordinate space as the pins and can transition between physical and virtual. - Try it: In AR, treat a set of physical blocks (or LEGO) as "solid" data and show the rest as semi-transparent virtual particles above them; when a data point is tapped, it "solidifies" from virtual into a shape you must build with physical blocks. Twist: do the reverse, so a physical piece evaporates into virtual gas when taken away. #### inFORM — MIT Tangible Media Group — Hiroshi Ishii (2013) - Video: https://vimeo.com/79179138 - Interaction: Tangible Objects, Hands & Body, Shared & Social - Platform & tech: Desktop, Projection, shape display, Kinect, projector - Idea: Turn screen pixels into physical pixels that rise and fall, so a faraway hand can really push a ball on your table. - What it is: Daniel Leithinger and Sean Follmer's 30x30 motorized pin display renders 3D content and remote users' hands physically; objects on the table can be moved by the surface itself. - Technique: An array of 900 motorized pins is actuated to render a height map, a Kinect captures remote hands and objects as depth images that are downsampled to pin heights, and a projector colors the pin tops. - Try it: Convert the shape of a hand into a 10x10 height map with a camera or depth camera (or MediaPipe hand keypoints), render it live as an array of pillars in three.js, and project the colors onto a table. Twist: build a 3x3 physical version (servos pushing wooden sticks) that shows only the single most important piece of information. #### Physical Telepresence — MIT Tangible Media Group — Hiroshi Ishii (2014) - Video: https://vimeo.com/108402837 - Interaction: Shared & Social, Tangible Objects, Hands & Body - Platform & tech: Desktop, Projection, shape display, Kinect - Idea: In remote collaboration, the other person's hands appear as physical shapes on your table and can actually push things. - What it is: Remote participants' hands and objects are captured in 3D and rendered physically on a shape display, so someone far away can push and manipulate objects on your table. - Technique: A Kinect depth stream of the remote space is converted to a height map that drives the local shape display in real time, so remote hands physically push objects on the pins. - Try it: Build a remote "push" with two computers' cameras and MediaPipe Hands: the hand position on side A controls a virtual pusher projected on side B's table, which shoves detected paper squares (or virtual objects) there. Twist: transmit only the hand's "shadow", and discuss what anonymous remote touch feels like. #### TRANSFORM — MIT Tangible Media Group — Hiroshi Ishii (2014) - Video: https://vimeo.com/98880732 - Interaction: Tangible Objects, Perception & Effects, Hands & Body - Platform & tech: Desktop, shape display, Kinect - Idea: A living table that heaves like ocean waves and carries objects around by itself. - What it is: Three large shape displays form dynamic furniture shown at Milan Design Week: the surface flows like water, reacts to bodies and carries objects around. - Technique: Three shape displays run choreographed height-map animations and physics-style wave simulations, with Kinect tracking of visitors' bodies modulating the motion. - Try it: Build a "fluid tabletop" of hundreds of pillars in TouchDesigner or three.js, use a camera to detect people so ripples spread from wherever they stand, and project it onto a real table. Twist: have the table "deliver" a virtual object to one particular person standing beside it. #### Kinetic Blocks — MIT Tangible Media Group — Hiroshi Ishii (2015) - Video: https://vimeo.com/141631681 - Interaction: Tangible Objects, Play - Platform & tech: Desktop, shape display - Idea: The tabletop builds with blocks by itself, turning digital commands into physical assembly. - What it is: A shape display that stacks, sorts and assembles passive blocks by itself, and can replay a user's remote block-building gestures. - Technique: The shape display lifts, tilts and pushes passive blocks with sequenced pin motions (like a programmable conveyor), with tracked block positions and recorded gestures driving the manipulation plans. - Try it: Project "move arrows" onto a table and have a classmate act as the actuator, pushing blocks as the arrows show, while a program computes the next step from the block positions detected by a camera. Twist: record one classmate's building process, then have the system "play it back" to guide another person in copying it. #### Materiable — MIT Tangible Media Group — Hiroshi Ishii (2016) - Video: https://vimeo.com/165798784 - Interaction: Tangible Objects, Perception & Effects, Hands & Body - Platform & tech: Desktop, Projection, shape display, projection - Idea: The same pin surface turns into jelly, sand or rubber under your hand. - What it is: A shape display renders material properties such as elasticity, viscosity and flexibility, so touching the pins feels like pressing jelly, sand or rubber. - Technique: Force sensing through the pins' motor current detects how hard a pin is pressed, and a per-pin physics model (spring, damper, fluid coupling) sets the pin response to simulate elasticity or viscosity, with projection showing the material. - Try it: Use touchscreen pressure or press duration as input to render a block of virtual material (jelly, sand, rubber) on an AR plane, and make it respond differently to presses using spring, damping and diffusion models, with vibration feedback. Twist: invent a material that does not exist in reality. #### Mediate: A Spatial Tangible Interface for Mixed Reality — MIT Tangible Media Group — Hiroshi Ishii (2018) - Video: https://vimeo.com/285036249 - Interaction: Tangible Objects, Hands & Body - Platform & tech: Headset, Desktop, shape display, mixed reality - Idea: Give virtual objects in mixed reality a touchable, shape-changing physical stand-in. - What it is: A shape-changing tangible interface that physically represents virtual content inside mixed reality, bridging headset holograms and touchable form. - Technique: A shape-changing tangible device is spatially registered to the MR headset so that its physical form is actuated to match the virtual object the user is reaching for, providing passive haptic proxy feedback. - Try it: Register a physical box as the 'stand-in' for a virtual object in HoloKit or phone AR, and when the virtual object changes shape, prompt a classmate to adjust the box by hand (or push it open with a servo) so what the hand feels matches what the eye sees. Twist: deliberately mismatch what is seen and felt, and test how big a difference people notice. ### Olafur Eliasson *Artist; Studio Olafur Eliasson* Icelandic-Danish artist whose light, colour and perception installations (The Weather Project) extended into AR with Wunderkammer and the children's climate app Earth Speakr. #### Beauty — Olafur Eliasson (1993) - Video: https://www.youtube.com/watch?v=vQ0Eq097RKA - Interaction: Perception & Effects, Hands & Body, Gaze & Attention - Platform & tech: Projection, water mist, spotlight, perforated hose - Idea: The rainbow exists only between light, droplets and your eye: effects that depend on viewer position are personal, and AR can make every visitor's version different. - What it is: In a dark room a curtain of fine mist falls from the ceiling and a single spotlight shines through it; a rainbow appears only when you stand at the right angle and shifts as you move. - Technique: A perforated hose releases a sheet of droplets, and a lamp placed behind the viewer refracts through them at about 42 degrees, so each eye sees its own bow. - Try it: Make an AR mist curtain in Lens Studio or AR Foundation whose rainbow is computed from the angle between a virtual light and the phone, so it only appears from certain spots. Twist: use two phones side by side to show that each sees a different arc. #### Room for one colour — Olafur Eliasson (1997) - Video: https://www.youtube.com/watch?v=hd077pa-5CI - Interaction: Perception & Effects, Shared & Social - Platform & tech: Projection, mono-frequency lamps, white room - Idea: Changing the light, not the objects, changes the whole world at once; a global colour grade in AR can transform a room more strongly than any added 3D model. - What it is: A white room is lit only by yellow mono-frequency lamps; visitors, their clothes and skin all turn shades of grey and yellow, as if colour had been switched off. - Technique: Low-pressure sodium lamps emit a single wavelength of yellow light, so the eye cannot distinguish other hues and perceives the scene in yellow and black. - Try it: Write a WebXR or Lens Studio shader that maps the passthrough camera to one hue and brightness only, and have students try to sort coloured objects while wearing it. Twist: let one object keep its true colour as the only exception. #### La situazione antispettiva — Olafur Eliasson (2003) - Video: https://www.youtube.com/watch?v=d8yLRu2QwDk - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Desktop, mirrors, kaleidoscope structure - Idea: Mirrors placed at exact angles turn one view into an infinite pattern; AR can use the live camera image as raw material for geometric multiplication. - What it is: A mirror-lined structure at the Venice Biennale folds the surrounding garden and visitors into kaleidoscopic repeats, so the outside world appears multiplied from within. - Technique: Mirror panels are joined at fixed angles (likely a kaleidoscopic polyhedron) so that reflections of the surroundings repeat in a regular tiling. - Try it: Build a Lens Studio or WebXR effect that slices the live camera feed into a triangle and repeats it in a kaleidoscope around a point anchored in space. Twist: make the number of mirror segments depend on how close the user walks. #### The weather project — Olafur Eliasson (2003) - Video: https://www.youtube.com/watch?v=_1Vgeose43g - Interaction: Perception & Effects, Shared & Social, Gaze & Attention - Platform & tech: Projection, mono-frequency lamps, mirror ceiling, haze - Idea: Half an object plus a mirror makes a whole one, and a mirrored ceiling turns the audience into the show: an AR scene can add the viewers themselves to the spectacle. - What it is: A giant semicircle of yellow lamps reflected in a mirrored ceiling forms a sun in Tate Modern's Turbine Hall; visitors lie on the floor in the mist and look for themselves in the ceiling. - Technique: Hundreds of single-frequency sodium-like lamps behind a semicircular screen emit narrow yellow light that flattens all other colours, while a mirror foil ceiling doubles the hall and a haze machine adds atmosphere. - Try it: Place a half-disc sun at the end of a corridor in AR and mirror the whole scene (including a live camera feed of the people) onto a virtual ceiling. Twist: switch the scene to a single yellow wavelength so every colour in the passthrough turns grey. #### Round rainbow — Olafur Eliasson (2005) - Video: https://www.youtube.com/watch?v=BuYH6nLMR-c - Interaction: Perception & Effects, Projection - Platform & tech: Projection, acrylic ring, spotlight, motor - Idea: One small rotating object can paint a whole room with light; in AR, a single animated emitter can be enough to occupy and redefine a space. - What it is: A slowly turning acrylic ring hangs in a dark room and a spotlight shines on it; the ring throws a circular rainbow and moving light onto the walls around the visitor. - Technique: A precisely cut acrylic ring refracts a beam into a spectrum, and a motor rotates it so the projected rainbow sweeps around the walls. - Try it: Place a virtual rotating prism ring in AR and use the room mesh to project its rainbow caustics onto the real walls, updating as it turns. Twist: let the user's phone flashlight act as the light source. #### The New York City Waterfalls — Olafur Eliasson (2008) - Video: https://www.youtube.com/watch?v=6wUwV0eDDQI - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, scaffolding, pumped river water, harbour - Idea: Showing the scaffold behind the illusion lets people see both the effect and how it is made, a stance many AR works could adopt. - What it is: Four artificial waterfalls, 27 to 37 metres tall, built from exposed scaffolding, poured river water back into New York's East River and harbour for three months. - Technique: Pumps lifted river water to the top of each scaffold tower and let it fall in a sheet back into the river. - Try it: Add an AR waterfall pouring off the edge of a real building or bridge, rendering the pump and scaffold as a wireframe. Twist: the flow volume follows real-time river gauge data. #### Your atmospheric colour atlas — Olafur Eliasson (2009) - Video: https://www.youtube.com/watch?v=lRSY61HVlBM - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Projection, fog, fluorescent lights, DMX control - Idea: A room can be a colour map you walk through: AR can assign colour or sound to positions on a floor grid so that location becomes the controller. - What it is: A room filled with thick fog is lit from the ceiling by coloured lights arranged in a grid; as you walk, the fog around you shifts from one colour to another and other visitors appear and vanish. - Technique: Fluorescent tubes in red, green and blue are mounted in a ceiling grid, and dense fog blends their light into gradients across the room. - Try it: Anchor an invisible 4 x 4 colour grid to the classroom floor with AR Foundation and fill the view with a fog tinted by the student's current cell, blending between neighbours. Twist: add a soft tone for each cell so the room can be played like an instrument. #### Din blinde passager (Your blind passenger) — Olafur Eliasson (2010) - Video: https://www.youtube.com/watch?v=wYxxNA_WTs8 - Interaction: Perception & Effects, Spatial Mapping, Hands & Body - Platform & tech: Projection, fog, fluorescent lighting, tunnel - Idea: When sight is reduced, the body navigates by sound and touch; AR does not have to add more to see, it can also take visibility away to make people feel space. - What it is: Visitors walk through a 90-metre tunnel filled with dense fog; they can see only about a metre and a half ahead, and the colour of the light changes from one zone to the next. - Technique: A haze system fills a narrow corridor with fog so dense that light scatters in every direction, and coloured fluorescent tubes set zones of yellow, white and blue (likely). - Try it: Use AR Foundation's depth or mesh data to render a fog that hides everything farther than 1.5 m from the phone, and ask students to cross the room with it. Twist: change the fog colour depending on which part of the room they are in. #### Your uncertain shadow (colour) — Olafur Eliasson (2010) - Video: https://www.youtube.com/watch?v=PeBH6fTQNSc - Interaction: Hands & Body, Perception & Effects, Projection - Platform & tech: Projection, coloured lamps, additive colour mixing - Idea: Five colored lamps split your shadow into a string of rainbows. - What it is: Five coloured lamps shine on a white wall; each visitor casts a row of overlapping coloured shadows that move with them, splitting the body into a spectrum. - Technique: Coloured HMI lamps placed side by side produce additive-colour light; blocking one lamp leaves its complementary colour, so each shadow is a different hue. - Try it: Shine three phone lights in red, green and blue (or use colored cellophane) on a white wall and study the colors of the shadows. Twist: use dance moves to make the colored shadows spell out a letter. #### Your rainbow panorama — Olafur Eliasson (2011) - Video: https://www.youtube.com/watch?v=ZsMCfOW0SRA - Interaction: Location & City, Perception & Effects, Gaze & Attention - Platform & tech: Desktop, coloured glass, circular walkway, rooftop - Idea: Where you stand decides what filter you see: in AR, mapping a colour or effect to compass direction turns walking into the interface. - What it is: A 150-metre circular walkway of glass in every colour of the spectrum sits on top of the ARoS museum in Aarhus; walking around it tints the city view in changing hues. - Technique: Coloured laminated glass panels are arranged in spectral order around a ring so that hue changes with the viewer's bearing. - Try it: Build a phone AR filter that tints the camera view according to compass heading, so that turning in a full circle sweeps through the spectrum. Twist: let two students stand opposite each other and see each other in complementary colours. #### Contact — Olafur Eliasson (2014) - Video: https://www.youtube.com/watch?v=NXXuyotR8us - Interaction: Perception & Effects, Hands & Body, Portals & Worlds - Platform & tech: Projection, mirrors, light line, shadow - Idea: A single horizon line plus a mirror is enough to create a world beyond the wall; AR portals can work with equally minimal cues. - What it is: At Fondation Louis Vuitton a line of light on the floor becomes a horizon: mirrors, a moving shadow and darkness make it seem like visitors are walking on the edge of a planet. - Technique: A low slit of light is doubled by floor-to-ceiling mirrors into an endless horizon, and a slowly rotating lamp casts visitors' shadows across it (likely). - Try it: Draw a glowing horizon line across a real wall in AR, render a mirrored copy of the room beyond it, and let the students' tracked bodies cast virtual shadows along the line. Twist: slowly tilt the horizon so the room feels like it is rolling. #### Ice Watch — Olafur Eliasson (2014) - Video: https://www.youtube.com/watch?v=qd-JRGBKSXA - Interaction: Location & City, Tangible Objects, Information & UI - Platform & tech: Desktop, Greenlandic glacial ice, public square - Idea: Touching melting ice makes climate data physical and felt, a lesson for data AR: let people feel the number in their hands. - What it is: With geologist Minik Rosing, Eliasson placed twelve blocks of free-floating Greenland ice in a clock formation on public squares (Copenhagen, Paris during COP21, London) and let them melt while people touched them. - Technique: Ice that had calved from the Greenland ice sheet was harvested from a fjord, shipped in refrigerated containers and arranged like a clock face. - Try it: Place twelve AR ice blocks in a clock ring on a plaza and melt them in real time according to the day's temperature. Twist: each block shows how much of Greenland's ice is lost in the second it takes to walk around the ring. #### Riverbed — Olafur Eliasson (2014) - Video: https://www.youtube.com/watch?v=s_xA_8ps_Co - Interaction: Spatial Mapping, Hands & Body, Perception & Effects - Platform & tech: Desktop, Icelandic stones, water, museum rooms - Idea: Replacing a museum's floor with a landscape you have to climb over changes how the body moves, a strong case for full-room AR replacement. - What it is: At the Louisiana Museum in Denmark, a whole wing was filled with stones and gravel from Iceland and a stream running through it, so visitors walk a riverbed inside the museum. - Technique: Tonnes of rock were brought in and shaped over the gallery floors, with a pumped stream circulating through the rooms. - Try it: Use ARKit scene reconstruction to replace a corridor floor with an AR riverbed of rocks and flowing water that respects real walls and doorways. Twist: water flows downhill according to the phone's measured floor slope. #### Earth Speakr — Olafur Eliasson (2020) - Video: https://www.youtube.com/watch?v=oneExExNwZw - Interaction: Face, Voice & Sound, Location & City - Platform & tech: Phone, face tracking, mobile AR - Idea: Children lend their faces and voices to trees and rivers so the Earth can speak. - What it is: An AR app for children in which kids record their voice and facial expressions and 'lend' them to places and objects in the world — trees, rivers, buildings — to speak for the planet. - Technique: Face tracking captures the child's expressions and voice, which are mapped onto AR 'faces' placed on real objects detected through the phone camera. - Try it: Use AR face puppeteering, for example a Snap Lens or Reality Composer, to put your expressions on a tree on campus and record one line of what the tree wants to say. Twist: the whole class's trees form a conversation. ### Ryo Suzuki — Programmable Reality Lab *Assistant Professor, University of Colorado Boulder (previously University of Calgary); directs the Programmable Reality Lab* Ryo Suzuki builds AR tools that make the real world dynamic and programmable: sketches that respond to physical motion, textbooks whose diagrams come alive, and robots and physical objects bound to virtual content. He did his PhD with Daniel Leithinger and Mark Gross at CU Boulder. #### RealitySketch — Ryo Suzuki — Programmable Reality Lab, Rubaiat Habib Kazi (2020) - Video: https://www.youtube.com/watch?v=L0p-BNU9rXU - Interaction: Drawing & Making, Information & UI, Tangible Objects - Platform & tech: Phone, ARKit, computer vision - Idea: Lines and angles you sketch stick to real moving objects and become live physics visualizations. - What it is: Sketch lines, angles and graphs on a phone's AR view of a moving object (a pendulum, a rolling ball) and the drawings bind to it, updating live to visualize physics. - Technique: ARKit tracks the device and a detected plane, the moving object is tracked in the camera image by color, and sketched elements are bound to the object's projected 3D position so lines, angles and plots update each frame. - Try it: In phone AR, use color tracking to mark a swinging pendant or a rolling ball, let the user draw a line connected to it, and update the line's length and angle live while plotting a graph beside it. Twist: use the tool to measure some periodic motion around you that you are curious about. #### RealityTalk — Ryo Suzuki — Programmable Reality Lab, Rubaiat Habib Kazi (2022) - Video: https://www.youtube.com/watch?v=vfIMeICV-7c - Interaction: Voice & Sound, Hands & Body, Performance - Platform & tech: Phone, Desktop, speech recognition, hand tracking - Idea: As you talk, keywords turn into AR text and images floating around you that you can grab to tell your story. - What it is: While a presenter speaks, keywords are recognized and turned into AR text, images and graphics that the presenter can grab and place with hand gestures around their body. - Technique: Streaming speech recognition extracts keywords, which are turned into text or images and placed in the camera view, while hand tracking lets the presenter grab and reposition them in the space around their body. - Try it: Use the Web Speech API to detect keywords a speaker says, stick matching text or emoji into the camera view, and move them with a MediaPipe Hands pinch gesture. Twist: give a 60-second AR talk in which every visual element must be summoned by your words. #### Sketched Reality — Ryo Suzuki — Programmable Reality Lab (2022) - Video: https://www.youtube.com/watch?v=xy-IeVgoEpY - Interaction: Drawing & Making, Tangible Objects, Play - Platform & tech: Phone, ARKit, tabletop robots - Idea: A drawn spring can bounce a real little robot, and the robot can push drawn things too. - What it is: Sketches in AR interact bidirectionally with small tabletop robots: a drawn ramp makes the robot move, and a robot bumping a drawn spring makes the sketch bounce. - Technique: Small tabletop robots are tracked in the AR view and sketches are converted into physics bodies, so the simulation both sends motor commands to the robots and updates the sketch from the robots' measured positions. - Try it: Use phone AR to recognize a small marker-tagged car (or a toy a classmate pushes by hand), draw a ramp or wall on screen, and make the virtual line bounce when the car hits it. Twist: let the drawn objects command the real car, for example a micro:bit car. #### Augmented Math — Ryo Suzuki — Programmable Reality Lab (2023) - Video: https://www.youtube.com/watch?v=Zv6JQ5T-qn0 - Interaction: Information & UI, Tangible Objects - Platform & tech: Phone, OCR, AR - Idea: Point at a printed math textbook and its formulas and graphs turn into live explanations you can drag and explore. - What it is: Point a tablet at a static math textbook and formulas and figures become explorable: sliders, animated graphs and linked highlights appear on the page. - Technique: The page is captured and processed with OCR and math-formula recognition to extract equations and figures, which are then turned into parametric interactive visualizations registered to the page through image tracking. - Try it: Photograph a function formula from a textbook page (such as y = a·sin(bx)), extract its parameters with OCR or by typing them in, and overlay draggable sliders and an animated curve on the page with 8th Wall or AR Foundation image tracking. Twist: pick the textbook page you found hardest to understand back then and rework it. #### ChameleonControl — Ryo Suzuki — Programmable Reality Lab (2023) - Video: https://www.youtube.com/watch?v=VOe3fETd3sk - Interaction: Shared & Social, Hands & Body - Platform & tech: Headset, HoloLens, hand tracking - Idea: Use MR gestures to remotely 'steer' a human stand-in who carries out actions on site for you. - What it is: A remote operator guides a real human 'surrogate' through mixed reality gestures, turning a person into a teleoperated body for remote tasks. - Technique: The remote operator's hand gestures are captured and rendered in the surrogate's HoloLens as overlaid ghost hands, which the surrogate aligns their own hands with to replicate the motion. - Try it: Have one classmate wear HoloKit or watch phone AR while a remote classmate's MediaPipe hand keypoints stream in live as 'ghost hands', and have the local classmate copy the ghost hands to finish an origami or block-building task. Twist: the ghost hands give only half the cues, and see how the collaboration changes. #### HoloBots — Ryo Suzuki — Programmable Reality Lab (2023) - Video: https://www.youtube.com/watch?v=KSBPtiXy8Hg - Interaction: Shared & Social, Tangible Objects, Hands & Body - Platform & tech: Headset, HoloLens, Sony toio robots - Idea: A remote person's hologram gets a small robot body that can actually push things on your desk. - What it is: A remote user's hologram in HoloLens is paired with small mobile robots, so the remote person can physically push objects and touch hands through the robots. - Technique: The remote user's tracked hand motions in HoloLens are mapped onto the positions of Sony toio robots, which localize themselves on a position-encoded mat, and the hologram is rendered co-located with the robots. - Try it: Use a toio, a micro:bit car or a cardboard box pushed by a classmate as the physical stand-in for a 'remote hand', let the remote side drag its position on a web page, and overlay the other person's virtual hand on the object with local phone AR. Twist: have the stand-in perform social gestures such as a hug or a high five. #### RealityCanvas — Ryo Suzuki — Programmable Reality Lab (2023) - Video: https://www.youtube.com/watch?v=HVOgH1quDsc - Interaction: Drawing & Making, Perception & Effects, Hands & Body - Platform & tech: Phone, ARKit, body tracking - Idea: Add comic-style speed lines and smoke to the real world, moving live with real people and objects. - What it is: Scribble animation effects such as motion lines, smoke and speed lines onto real moving people and objects in AR, and they follow and react to the motion like a live cartoon. - Technique: ARKit body tracking and object tracking give per-frame positions of moving targets, and hand-drawn strokes are attached to those points and animated with rules (trail, follow, emit) to produce cartoon effects. - Try it: Track a classmate's wrist or ankle with MediaPipe Pose, let the user draw a speed line or a puff of smoke on screen, and bind it to the joint so it plays with the motion. Twist: add comic sound-effect lettering to everyday objects in motion (a door opening, water pouring). #### Teachable Reality — Ryo Suzuki — Programmable Reality Lab (2023) - Video: https://www.youtube.com/watch?v=JssiyfrhIJw - Interaction: Tangible Objects, Play - Platform & tech: Phone, interactive machine teaching, ARKit - Idea: Demonstrate with any object at hand a few times and it becomes a physical controller for AR effects. - What it is: Train an on-device model by demonstrating with everyday objects (a lid open or closed, a toy moved) and bind those states to AR effects, prototyping tangible AR without code. - Technique: An on-device image classifier is trained from a few user demonstrations of object states (interactive machine teaching), and state changes detected in the camera feed trigger AR effects bound to the tracked scene. - Try it: Train two states with Teachable Machine, such as cup lid open/closed or book open/shut, and link the state change to an effect in web AR, like steam rising when the lid opens. Twist: turn an everyday object with no electronics into a game controller. #### Augmented Physics — Ryo Suzuki — Programmable Reality Lab, Rubaiat Habib Kazi (2024) - Video: https://www.youtube.com/watch?v=HUgYeA3BKfk - Interaction: Information & UI, Drawing & Making - Platform & tech: Phone, segmentation, physics simulation - Idea: Photograph a physics diagram in a textbook and it comes alive as a simulation you can tune. - What it is: Photograph a static physics textbook diagram and it becomes an interactive simulation in place: pendulums swing, pulleys move and parameters can be changed. - Technique: Diagram elements in a textbook image are segmented (e.g. with SAM), recognized as physics primitives such as pendulums, pulleys or ramps, and reconstructed as a 2D physics simulation overlaid on the original figure. - Try it: Photograph a diagram from a physics textbook (an incline, a pendulum or a pulley), cut out each part by hand or with a segmentation tool, rebuild it as a moving simulation in Matter.js and overlay it on the original figure. Twist: let readers change a parameter that cannot be changed in real life (such as gravity pointing up). #### RealityEffects — Ryo Suzuki — Programmable Reality Lab (2024) - Video: https://www.youtube.com/watch?v=5x45I-eXqBk - Interaction: Perception & Effects, Information & UI - Platform & tech: Headset, Desktop, volumetric video - Idea: Add annotations and effects that follow the people and objects in a 3D video. - What it is: Volumetric videos are augmented with object-centric annotations and dynamic visual effects that follow people and objects through the 3D recording. - Technique: Objects and people are tracked across frames of a volumetric (RGB-D) recording, and annotation or particle effects are attached to those 3D tracks so they follow the subject when the recording is replayed from any viewpoint. - Try it: Record a video with depth using Record3D or Polycam, and during playback in Unity or three.js attach trails, labels and explosion particles to a person or a ball. Twist: use the effects to recut an ordinary everyday clip as a sports broadcast. #### Guided Reality — Ryo Suzuki — Programmable Reality Lab (2025) - Video: https://www.youtube.com/watch?v=n6jMzQ6Z2Ic - Interaction: Information & UI, Tangible Objects - Platform & tech: Headset, LLM, vision models, mixed reality - Idea: An LLM automatically generates AR instructions pinned to the real objects you need to handle. - What it is: LLMs and vision models generate step-by-step AR task guidance that highlights the exact real objects to touch, turning any instruction into visual in-situ help. - Technique: An LLM decomposes an instruction into steps, and a vision-language or detection model grounds each step to the relevant objects in the camera view, whose image locations are highlighted in MR. - Try it: Photograph a tabletop with a phone and send the photo plus a task instruction ('make a cup of pour-over coffee') to a multimodal LLM, asking it to return box coordinates for the object in each step, then highlight them one by one on screen. Twist: have the system give a hint when you make a mistake instead of only listing steps. #### InSituTale — Ryo Suzuki — Programmable Reality Lab (2025) - Video: https://www.youtube.com/watch?v=Wn-sOu-sK60 - Interaction: Information & UI, Tangible Objects, Performance - Platform & tech: Phone, Headset, AR, object tracking - Idea: Arranging real objects on a table drives how an AR data story unfolds. - What it is: Physical objects on a table act as tangible handles for augmented data stories: moving and arranging objects drives AR charts and narration. - Technique: Tabletop objects are tracked by the camera (markers or object detection), and their positions and arrangement are mapped to parameters of AR charts and narration segments, making physical layout the story controller. - Try it: Put ArUco or image markers on three cups, track their positions with phone AR, and map the distances between cups to bar heights or to the playback progress of a data story. Twist: use these objects to tell a data story about your own daily routine or spending. #### RealitySummary — Ryo Suzuki — Programmable Reality Lab (2025) - Video: https://www.youtube.com/watch?v=JxtctjERJBU - Interaction: Information & UI, Tangible Objects - Platform & tech: Headset, OCR, LLM, mixed reality headset - Idea: Printed paper gets an AI sidekick that writes in the margins of space. - What it is: A mixed-reality reading assistant that always sees the page you are reading and, on demand, places summaries, timelines, comparison tables and answers next to the physical document. - Technique: An always-on headset camera feeds OCR, and an LLM produces summaries and answers that are rendered as spatial visualizations anchored beside the tracked document. - Try it: Point a phone at a printed article, run OCR (Apple Vision, Google ML Kit or Tesseract.js), ask an LLM for a three-bullet summary and a timeline, and pin both next to the page with image tracking. Twist: have the AI generate one question you should be able to answer after reading the page. #### Video2MR — Ryo Suzuki — Programmable Reality Lab (2025) - Video: https://www.youtube.com/watch?v=f1L-2US25PU - Interaction: Hands & Body, Information & UI - Platform & tech: Headset, HoloLens 2, Azure Kinect, DeepMotion, AI motion capture - Idea: Lift the person out of a YouTube video and make them your 3D trainer. - What it is: Turns any online sports or exercise video into a mixed-reality coach: the person in the 2D video becomes a 3D avatar in your room, with highlighted posture differences, motion trails, speed control and repositioning for other viewpoints. - Technique: AI motion capture (DeepMotion) extracts 3D motion from 2D video to drive an avatar on HoloLens 2, while Azure Kinect tracks the user to compare postures and trigger augmentations. - Try it: Take a ten-second dance or yoga clip, extract 3D motion with a free AI mocap tool (DeepMotion, Rokoko Video or MediaPipe), retarget it to a Mixamo character and stand it on the floor next to you in phone AR. Twist: slow the avatar down whenever your own pose, tracked with MediaPipe, drifts too far from it. ### Botao 'Amber' Hu *Designer-researcher; founder of Reality Design Lab; inventor of HoloKit* Creative technologist and experiential futures designer who invented HoloKit, an open-source stereo iPhone AR headset, and leads Reality Design Lab's research on embodied, social mixed reality. #### HoloKit — Botao 'Amber' Hu (2017) - Video: https://www.youtube.com/watch?v=Imy8vwO-4mo - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Headset, Phone, HoloKit, ARKit, Unity - Idea: An open-source stereo optical see-through AR headset made from cardboard and an iPhone. - What it is: An open-source cardboard headset that turns an iPhone into a stereoscopic optical see-through AR wearable, with ARKit tracking and holograms shown in both eyes. - Technique: An iPhone renders a stereo image pair that a mirror and lens reflect into each eye as an optical see-through overlay, while ARKit visual-inertial tracking places the holograms in the world. - Try it: Convert an existing phone AR scene to stereo mode with the HoloKit Unity SDK, and compare the sense of depth in mono and stereo. Twist: make a small game where you can only judge distance with stereo vision. #### City of Sparkles — Botao 'Amber' Hu (2022) - Video: https://vimeo.com/777136892 - Interaction: Information & UI, Location & City, Portals & Worlds - Platform & tech: Headset, Apple Vision Pro, VR, geotagged Twitter data, photogrammetry - Idea: How might an AI perceive our city? Let the audience see it through the AI's eyes. - What it is: An immersive piece in which the audience embodies Zoe, an AI life form, wandering a sparkling New York built from four years of geotagged Twitter posts and reading scattered fragments of human memory. - Technique: Geotagged social-media posts are spatialised as sparkles sampled from a photogrammetry model of the real city, and the viewer navigates them as the AI protagonist. - Try it: Collect 100 geotagged posts or photos around your campus, place them as glowing points on a map-scaled WebXR model, and read them as an AI character looking for meaning. Twist: the AI can only read fragments one word at a time. #### MOFA: The Duel — Botao 'Amber' Hu (2023) - Video: https://www.youtube.com/watch?v=mCHEdItEx2s - Interaction: Shared & Social, Hands & Body, Play - Platform & tech: Headset, Phone, HoloKit X, ARKit, Unity - Idea: A same-room magic duel in which players wearing HoloKit cast spells with gestures while spectators join in on their phones. - What it is: A co-located magic duel in which players wearing HoloKit headsets cast spells with hand gestures while spectators on phones watch and join; CHI 2023 best interactivity demo. - Technique: Hand-pose recognition on the iPhone camera classifies spell gestures, and headsets and spectator phones share one coordinate system (likely via marker-based colocalization) so spells travel between real bodies. - Try it: Use MediaPipe or Apple Vision hand-pose recognition to tell two gestures apart, one casting a fireball and one a shield, and sync the two devices over the network. Twist: let spectators cheer for one side on their phones to power up that side's spells. #### EchoVision — Botao 'Amber' Hu (2024) - Video: https://www.youtube.com/watch?v=0JyHmEApctg - Interaction: Voice & Sound, Spatial Mapping, Perception & Effects - Platform & tech: Headset, Phone, HoloKit, iPhone LiDAR, Unity - Idea: Your voice triggers LiDAR echo visuals, so you 'hear' space the way a bat does. - What it is: A handheld mixed-reality mask that lets people experience bat echolocation: your voice triggers LiDAR-based 'echoes' that briefly reveal the surrounding space as a visualized sound wave. - Technique: Microphone amplitude triggers a pulse, and a shader reveals the LiDAR scene mesh as an expanding spherical wavefront from the user that fades out, mimicking echolocation. - Try it: Use AR Foundation mesh reconstruction and a distance-based spreading shader so that the room mesh briefly lights up like a sound wave whenever a sound is detected. Twist: the higher the pitch, the faster the wave travels. #### MOFA: The Ghost — Botao 'Amber' Hu (2024) - Video: https://www.youtube.com/watch?v=7bCZprvgpFU - Interaction: Shared & Social, Hands & Body, Play - Platform & tech: Headset, Phone, HoloKit X, Unity - Idea: An asymmetric ghost-hunting game: a 'ghost' visible only through devices chases players across a public space. - What it is: An asymmetric social exertion game in spontaneous co-located MR: an invisible ghost player hunts others who can only see it through their devices, played in public spaces. - Technique: All players share a colocalized coordinate frame, and the ghost's tracked pose is rendered only on the hunters' devices, creating asymmetric visibility in the same physical space. - Try it: Make a two-player asymmetric game in which one person's position can only be seen through the other person's phone AR. Twist: every so often the 'invisible' player briefly appears and makes a sound. #### FeltSight — Botao 'Amber' Hu (2025) - Video: https://www.youtube.com/watch?v=7Pq6s3VnD0A - Interaction: Hands & Body, Perception & Effects, Spatial Mapping - Platform & tech: Headset, Wearable, mixed reality, haptics - Idea: Borrowing the star-nosed mole's tactile navigation, you sense space through your fingertips rather than your eyes. - What it is: A mixed-reality haptic experience inspired by the tactile navigation of animals like the star-nosed mole: you feel your way through space via fingertip haptics instead of sight. - Technique: Distance from the fingertip to scanned geometry is mapped to vibration intensity on a wearable actuator, while visual feedback is reduced so touch becomes the main sense. - Try it: Measure the distance to objects in front of the camera with phone LiDAR or depth estimation, map it to vibration strength, and feel your way across the classroom with your eyes closed. Twist: replace the vibration rhythm with the tactile 'language' of different animals. #### TouchPort / Allow me into your dream — Botao 'Amber' Hu (2026) - Video: https://www.youtube.com/watch?v=uMSXnUxnxd4 - Interaction: Shared & Social, Hands & Body - Platform & tech: Headset, mixed reality, hand tracking - Idea: A handshake and a pull temporarily join two people's separate mixed realities into one shared space. - What it is: A mixed-reality sharing protocol in which a handshake and pull signals intent, negotiates consent and opens a temporary shared layer between two people's separate mixed realities. - Technique: Hand tracking detects a handshake-and-pull gesture between two users, which triggers a consent handshake over the network and merges their separate AR layers into a shared session. - Try it: Design a two-person phone AR flow in which each person can see the private objects the other has placed in space only after both make a certain gesture at the same time. Twist: the shared layer vanishes the moment either person lets go. ### Daniel Rozin *Artist; professor at NYU ITP* Israeli-American artist who builds 'mechanical mirrors': grids of wood, trash, pegs, steel or toy penguins moved by motors to reflect the viewer captured by a hidden camera. #### Wooden Mirror — Daniel Rozin (1999) - Video: https://www.youtube.com/watch?v=1ZPJ0U_kpNg - Interaction: Hands & Body, Tangible Objects, Perception & Effects - Platform & tech: Desktop, video camera, servo motors, wood tiles - Idea: Piece together your live mirror image from turning wooden tiles. - What it is: 830 small wooden tiles tilt with individual motors so the light they catch forms a live, wooden reflection of whoever stands in front. - Technique: A hidden camera image is downsampled to the tile grid and each pixel's brightness sets a servo angle that changes how much light the wooden tile reflects. - Try it: Build a 'mechanical mirror' from an 8x8 grid of servos and card tiles, with angles controlled by a low-resolution camera image. Twist: switch to a material of your choice (buttons, bottle caps) and compare how 'pixel-like' different materials feel. #### Trash Mirror — Daniel Rozin (2002) - Video: https://www.youtube.com/watch?v=R0dLo3HB4P8 - Interaction: Hands & Body, Tangible Objects, Perception & Effects - Platform & tech: Desktop, video camera, motors - Idea: Make a mirror that reflects you out of scraps of street trash. - What it is: Irregular pieces of trash collected from the streets become motorised pixels that tilt to reflect the viewer's image. - Technique: Each irregular trash piece sits on a motor mapped to a region of the camera image, rotating to present a lighter or darker face. - Try it: Collect scrap paper from the classroom to make irregular 'pixels', and build a small prototype with a few servos and a camera. Twist: map each scrap to a region matching its real shape in the image. #### Shiny Balls Mirror — Daniel Rozin (2003) - Video: https://vimeo.com/57244184 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, camera, motors, chrome tubes - Idea: A mirror made of tiny mirrors creates two reflections at once, a playful layering an AR designer can reuse for picture-in-picture views. - What it is: 921 hexagonal chrome tubes each hold a shiny ball; motors push the tubes in and out so the balls catch more or less light and form a reflection of the viewer, who also sees themselves in every ball. - Technique: A camera image is reduced to the tube grid and motors set each tube's depth; recessed balls look dark and extended balls reflect the room light. - Try it: Build an AR grid of chrome spheres with real-time environment reflections, and push each sphere forward according to the brightness of the user's camera image. Twist: have each sphere show a different moment from the last minute. #### Time Scan — Daniel Rozin (2004) - Video: https://vimeo.com/130223120 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, camera, custom software, slit-scan - Idea: Slit-scan turns time into a spatial axis, one of the simplest ways to make motion visible in a live AR camera effect. - What it is: A screen shows the viewer through a slit-scan effect: each column of the image comes from a slightly different moment, so moving bodies stretch and bend in time. - Technique: A buffer of recent video frames is kept, and each column of the output takes pixels from a frame further back in time. - Try it: Write a slit-scan camera shader for a phone that stores the last 120 frames and samples them by screen column, then point it at dancers. Twist: map time to distance from the phone using LiDAR depth instead of columns. #### Peg Mirror — Daniel Rozin (2007) - Video: https://www.youtube.com/watch?v=dghosA-zI6k - Interaction: Hands & Body, Tangible Objects, Perception & Effects - Platform & tech: Desktop, video camera, motors, wood - Idea: Wooden pegs turn their angled faces to build a portrait out of shadows. - What it is: 650 cylindrical wooden pegs with angled cut faces rotate so their shading forms the viewer's portrait in a circular composition. - Technique: Each peg's cut face catches different light depending on rotation; a camera image sets each peg's angle via motors. - Try it: Mount angled-cut wooden blocks or paper tubes on servos, compare how light and dark they look at different angles, then build a 3x3 prototype controlled by a camera. Twist: turn it into an 'audio mirror' that controls sound instead. #### Weave Mirror — Daniel Rozin (2007) - Video: https://www.youtube.com/watch?v=ushJnQfjbF0 - Interaction: Hands & Body, Tangible Objects, Perception & Effects - Platform & tech: Desktop, video camera, motors - Idea: Weave your image into being, strand by strand. - What it is: 768 C-shaped laminated prints rotate like woven strands to form a basket-like live reflection of the viewer. - Technique: Camera pixels drive motors that rotate each gradient-printed C-shape, exposing darker or lighter portions of the print. - Try it: Print a black-to-white gradient strip, wrap it around a rotating cylinder, and use camera brightness to control the rotation angle. Twist: use a color gradient instead of black and white to make a color mirror. #### Mirrors Mirror — Daniel Rozin (2008) - Video: https://www.youtube.com/watch?v=oKum2u7oLwc - Interaction: Hands & Body, Tangible Objects, Perception & Effects - Platform & tech: Desktop, video camera, motors, steel mirrors - Idea: Countless small mirrors reflect the room's light and dark to piece together your image. - What it is: 768 small polished steel mirrors tilt so each reflects a lighter or darker part of the room, composing the viewer's image out of reflections of the space. - Technique: A camera image controls motors that aim each steel tile toward bright or dark areas of the surrounding room. - Try it: Mount a few small mirror pieces on servos and make them reflect bright light from a window or a dark wall into the viewer's eyes to form a pattern. Twist: use a laser pointer to check each mirror's reflection direction. #### Rust Mirror — Daniel Rozin (2010) - Video: https://vimeo.com/9256258 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, camera, motors, rusted steel - Idea: Combining a live reflection with a slow environmental effect (rain) makes a mirror feel like a place with weather — an idea for layering ambient effects on AR selfies. - What it is: Hundreds of rusted steel strips rotate to reflect the viewer as a dark, grainy silhouette, and a projection of rain makes the image ripple over time. - Technique: A camera and computer vision set the rotation of rusted steel tiles, while a simulated rain layer modulates the image so drops appear to fall across the reflection. - Try it: Create an AR mirror filter made of rusty metal tiles that tilt to show the user, with virtual raindrops that disturb the tiles as they fall. Twist: let the rain get heavier the longer the person stands still. #### Angles Mirror — Daniel Rozin (2013) - Video: https://vimeo.com/61823984 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, camera, motors, custom software - Idea: Using orientation instead of colour as the pixel value shows that angle alone can carry an image — a useful idea for AR fields of arrows, grass or fins. - What it is: A field of small white rods rotates to different angles so that the shadows and highlights they make form a moving image of the viewer. - Technique: A camera image is sampled into a grid and each rod's motor rotates it so that its angle to the light creates the needed brightness. - Try it: Fill an AR plane with thin rods that rotate to match the brightness of the live camera image of the viewer, lit by one virtual light. Twist: move the virtual light and watch the portrait invert. #### Penguins Mirror — Daniel Rozin (2015) - Video: https://www.youtube.com/watch?v=QlrnjjfLkTI - Interaction: Hands & Body, Tangible Objects, Play - Platform & tech: Desktop, video camera, turntable motors - Idea: 450 toy penguins turn around, their black backs and white bellies forming your silhouette. - What it is: 450 stuffed toy penguins on turntables rotate between their white bellies and black backs to form a silhouette of whoever stands before them. - Technique: A camera extracts the viewer's silhouette and motorised turntables turn each penguin to show its white or black side as a binary pixel. - Try it: Mount a few small objects that are black on one side and white on the other (paper cups, toys) on turntables, and flip them with a camera silhouette. Twist: add a delay so the "pixels" flip one after another like a stadium wave. #### PomPom Mirror — Daniel Rozin (2015) - Video: https://vimeo.com/128375543 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Kinect, motors, faux fur - Idea: Rendering a live body with a soft, unexpected material makes the reflection funny and tender — AR body effects gain character from the choice of 'pixel' material. - What it is: Hundreds of black and beige faux-fur pompoms are pushed forward or back by motors, forming a soft, fuzzy silhouette of the person standing in front. - Technique: A depth camera segments the viewer's silhouette and 928 pompoms on motor-driven rods move forward or backward to show light or dark fur in the matching grid cells. - Try it: Use body segmentation on a phone to drive a grid of AR fur balls anchored on a real wall that push out where the person's silhouette is. Twist: swap fur for another material the class votes on (sponges, leaves, coins). #### Fabric Mirror — Daniel Rozin (2019) - Video: https://vimeo.com/317577759 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, depth camera, motors, fabric - Idea: Even cloth can become a display when each strip has two faces, which suggests AR textiles that flip to show hidden content. - What it is: Strips of fabric twist and turn on motors to reveal their light or dark side, forming a soft woven silhouette of the viewer. - Technique: Motion tracking of the viewer drives motors that rotate two-toned fabric strips, likely using a depth camera to separate the body from the background. - Try it: Hang a virtual curtain of two-coloured ribbons in a real doorway in AR and flip the ribbons to draw the silhouette of whoever walks through. Twist: make the ribbons remember the last person and slowly fade to the next. #### CMY Shadows Mirror — Daniel Rozin (2021) - Video: https://vimeo.com/1198095563 - Interaction: Hands & Body, Perception & Effects, Projection - Platform & tech: Desktop, camera, motors, coloured lights - Idea: Overlapping cyan, magenta and yellow shadows paint a colored mirror image of you. - What it is: Rotating elements cast cyan, magenta and yellow shadows that mix on the wall into a coloured reflection of the viewer. - Technique: Motors rotate elements in front of coloured light sources so overlapping shadows subtractively mix into an image driven by the camera feed (likely). - Try it: Light an object with three phone lights in red, green and blue and observe how the colored shadows mix. Twist: rotate blocking vanes with servos to map camera brightness into colored shadows. ### Greg Madison *XR interaction designer; former UX/interaction designer at Unity Labs* Interaction designer and futurist who spent years at Unity Labs prototyping how real objects, surfaces and bodies can become interfaces in mixed reality. #### Concept: AR cereal box — Greg Madison (2017) - Video: https://www.youtube.com/watch?v=HgouAZ5Jfd8 - Interaction: Tangible Objects, Information & UI - Platform & tech: Headset, Unity, AR concept - Idea: A breakfast cereal box becomes an AR interface that pops out content. - What it is: A 2017 Unity Labs concept from the 'future of mixed reality' series in which a cereal box becomes an interactive AR object with content attached to the packaging. - Technique: The packaging is recognized as a tracked image or object target, and 3D content is parented to the box so interactions such as opening or tilting drive animations. - Try it: Use a real package as an image target and overlay a small game or character on the box. Twist: when you tilt the box, the AR cereal pours out like liquid. #### Tangible Mixed Reality 01: Oculus Touch & haptics — Greg Madison (2017) - Video: https://www.youtube.com/watch?v=wOUdvLVRJpk - Interaction: Tangible Objects, Hands & Body - Platform & tech: Headset, Oculus Rift, Oculus Touch, Unity - Idea: Real objects give virtual interfaces passive touch feedback. - What it is: First of a series in which Oculus Touch controllers track real objects so they can be mirrored inside VR; here real surfaces give passive haptic feedback to a virtual interface. - Technique: A tracked controller is mounted on a real object to provide its 6DoF pose, and a virtual panel is registered to the object's surface so touching the virtual UI hits real material (passive haptics). - Try it: Stick an image marker on a piece of cardboard, overlay a set of virtual buttons on its surface in AR, and highlight a button when a finger taps the cardboard. Twist: make the surface from different materials (felt, foil) and compare which feels most like a button. #### Tangible Mixed Reality 02: Oculus Touch & launchpad — Greg Madison (2017) - Video: https://www.youtube.com/watch?v=7xVFERWilj4 - Interaction: Tangible Objects, Voice & Sound - Platform & tech: Headset, Oculus Rift, Oculus Touch, MIDI, Unity - Idea: A real MIDI pad controller becomes a spatial visual instrument in mixed reality. - What it is: A physical launchpad controller is tracked and mirrored in VR so its real pads can be pressed while wearing a headset, with virtual feedback attached. - Technique: The physical launchpad is registered in the virtual scene by a rigidly attached tracker, and incoming MIDI note events trigger visual effects at each pad's virtual position. - Try it: Connect a MIDI keyboard or a virtual drum pad on a phone to Unity, so each key press bursts particles at the matching spot in AR. Twist: make the particles fly toward a fixed target in the room and leave traces. #### Tangible Mixed Reality 03: Oculus Touch & synthesizer — Greg Madison (2017) - Video: https://www.youtube.com/watch?v=PGrJeCXyKrI - Interaction: Tangible Objects, Voice & Sound - Platform & tech: Headset, Oculus Rift, Oculus Touch, Unity - Idea: Turn the knobs of a real synthesizer to shape sound visuals in space. - What it is: A real hardware synthesizer is registered in the virtual scene so you can play and tweak it while immersed, with virtual visuals attached to the instrument. - Technique: A hardware synth is aligned to a virtual replica through a tracked offset, so knob and key positions match and virtual visualizations stay attached to the physical instrument. - Try it: Make an image marker for a real instrument (such as a ukulele) and overlay AR waveforms around its body that vibrate with the sound. Twist: detect pitch with the microphone and show a different color for each pitch. #### Tangible Mixed Reality 04: Oculus Touch & keyboard — Greg Madison (2017) - Video: https://www.youtube.com/watch?v=i5iLZ3ykfBA - Interaction: Tangible Objects, Information & UI - Platform & tech: Headset, Oculus Rift, Oculus Touch, Unity - Idea: Type on a real keyboard in VR, with virtual labels laid over the physical keys. - What it is: A physical computer keyboard is tracked so you can type for real inside VR, with virtual labels and panels attached to the real keys. - Technique: A tracker fixed to the keyboard gives its pose, and a matching virtual keyboard model with per-key labels is overlaid so the user can type on real keys while immersed. - Try it: Locate a real keyboard with an image marker and overlay AR labels on commonly used shortcuts. Twist: when a key is pressed, a line of that key's "inner monologue" floats above the keyboard. #### Tangible Mixed Reality 05: Oculus Touch & board game — Greg Madison (2017) - Video: https://www.youtube.com/watch?v=DpFZme_mOoU - Interaction: Tangible Objects, Play - Platform & tech: Headset, Oculus Rift, Oculus Touch, Unity - Idea: Virtual pieces and animations on a real board game let a digital layer redefine an ordinary object. - What it is: A real board game is tracked and augmented with a virtual layer, pitching mixed reality as a way to 'uberize' ordinary physical objects the way MP3 did music. - Technique: The board's pose is tracked with a controller or marker, and a virtual layer (animations, rules, scores) is rendered in board-local coordinates so it moves with the physical board. - Try it: Pick a simple board game (such as tic-tac-toe), recognize the board with image tracking, and overlay animations and scores on the squares. Twist: add a hidden rule that only AR can reveal. #### Tangible Mixed Reality 06: Oculus Touch & ROBOT — Greg Madison (2017) - Video: https://www.youtube.com/watch?v=qESmM_MRNZ8 - Interaction: Tangible Objects, Play - Platform & tech: Headset, Oculus Rift, Oculus Touch, robot, Unity - Idea: A real little robot wears a virtual body and interface, merging machine and character. - What it is: A small physical robot is tracked with an Oculus Touch controller and mirrored in the virtual scene, blending a real moving machine with a virtual character. - Technique: A tracker rides on a small robot so its live pose drives a virtual character or effect overlay, blending the physical motion with a digital skin. - Try it: Stick an image marker on a remote-controlled car and replace it with an animal character in AR. Twist: when the car bumps into something, the AR character makes an exaggerated face. #### Tangible UI: hand tracking on flat surfaces — Greg Madison (2019) - Video: https://www.youtube.com/watch?v=Vr9y7hG52ME - Interaction: Hands & Body, Tangible Objects, Spatial Mapping - Platform & tech: Headset, Oculus Quest, hand tracking, Unity - Idea: Use hand tracking to turn any real tabletop into a giant touch screen. - What it is: Using early Quest hand tracking, any real table becomes a giant touch screen: virtual buttons and sliders are laid flat on the desk and pressed with your fingers. - Technique: A table plane is calibrated, and fingertip positions from hand tracking are tested against virtual buttons laid flat on it, with a small distance threshold standing in for a touch event. - Try it: Combine MediaPipe Hands with tabletop plane detection to lay a set of virtual sliders on the desk, using the finger's height above the surface to decide whether it is pressed. Twist: let different fingers control different functions. #### Audio Augmented Reality with face-tracking — Greg Madison (2020) - Video: https://www.youtube.com/watch?v=_7rUphgizxk - Interaction: Voice & Sound, Face, Gaze & Attention - Platform & tech: Phone, ARKit face tracking, spatial audio, Unity - Idea: Use face tracking to get head direction and build spatial AR made only of sound. - What it is: An audio-first AR prototype that uses face tracking to follow your head and render spatial sound anchored in the room around you. - Technique: Front-camera face tracking gives the head pose, which updates the audio listener orientation so sound sources stay world-anchored while the user turns their head. - Try it: Use ARKit face tracking to get head rotation and drive the orientation of the AudioListener in Unity, so a sound stays fixed in one direction in the room. Twist: make a sound treasure hunt you can play with your eyes closed. #### Augmented Keyboard — Greg Madison (2022) - Video: https://www.youtube.com/watch?v=-q-g4qNV5u8 - Interaction: Tangible Objects, Information & UI - Platform & tech: Headset, Meta Quest 2, Tracked Keyboard SDK - Idea: Pair a physical keyboard with one smart virtual screen instead of a wall of virtual monitors. - What it is: A tracked physical keyboard on Quest 2 gets one smart virtual screen and context-aware overlays instead of a wall of virtual monitors. - Technique: A tracked-keyboard SDK provides the keyboard pose, and a single virtual screen plus context overlays are placed relative to it instead of fixed in the room. - Try it: Hang a small AR screen above a real keyboard that shows only the line you are currently typing. Twist: change the screen color according to the mood of the text as you type. #### Playing with realities in ShapeXR — Greg Madison (2022) - Video: https://www.youtube.com/watch?v=opiiwqz6Hgo - Interaction: Portals & Worlds, Spatial Mapping - Platform & tech: Headset, Meta Quest 2, ShapeXR, passthrough - Idea: Overlay a room's digital twin on the real room and switch freely between the two. - What it is: Digital-twin experiments in ShapeXR where a virtual copy of his room lines up with the real one, letting him fade between passthrough and virtual versions of the same space. - Technique: A digital-twin model of the room is aligned to the real room at 1:1, and a crossfade between passthrough and the twin lets the user compare or transition between the two. - Try it: Scan a corner of the classroom with Polycam, align it with the real space in AR, and use a slider to fade between the real and scanned versions. Twist: secretly change one detail in the scan and have classmates find it. #### Unity Slices: Table — Greg Madison (2022) - Video: https://www.youtube.com/watch?v=FPCz0UY0UrA - Interaction: Portals & Worlds, Tangible Objects, Play - Platform & tech: Headset, Meta Quest 2, Passthrough API, Unity - Idea: Play board games on one real table while sliding continuously between reality and the virtual. - What it is: A Unity Labs demo on Quest 2 passthrough that slides a real table along the reality spectrum, from pure passthrough to fully virtual, while you play chess and other tabletop games on it. - Technique: A virtual table is registered to the real one, and a blend parameter crossfades passthrough and virtual rendering with a mask or opacity ramp, moving the same object along the reality-virtuality spectrum. - Try it: Align a same-sized virtual tabletop on a real table and use a slider to replace materials step by step from fully real to fully virtual. Twist: play the same mini-game at every step and record how it feels different. #### MELODIX•R — Greg Madison (2024) - Video: https://www.youtube.com/watch?v=pYgHzTdJPHw - Interaction: Voice & Sound, Hands & Body, Performance - Platform & tech: Headset, Meta Quest 3, hand tracking - Idea: Design fiction: a mixed-reality music experience set in your real room. - What it is: An XR design-fiction prototype on Quest 3 imagining a mixed-reality music experience placed in your real room. - Technique: Hand-tracked gestures likely act on room-anchored musical objects, so reaching, pinching and placing notes in space shapes the music. - Try it: Place a few sound-making AR balls in the room and grab them with hand gestures, moving them to different heights to change their pitch. Twist: change the timbre when a ball is placed on real furniture. ### Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) *Art and design collective (interactive installations)* Paris collective founded in 2007 by Gobelins graduates including Béatrice Lartigue, Nicolas Guichard, Cyril Diagne, Juliette Champain, Pierre Thirion and Tobias Muthesius. Its installations turn swings, pebbles, post-it notes, strings of light and whole forests into interfaces, and have been shown at MuDA Zurich, Signal Film & Media, KADOC Leuven and Kunstsilo. #### Moc — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2010) - Video: https://vimeo.com/11989814 - Interaction: Voice & Sound, Drawing & Making - Platform & tech: Projection, microphone, FFT, projector - Idea: Your breath makes a tree grow. - What it is: A projected landscape scrolls past. When a visitor whistles into a microphone, a tree grows in real time following the sound; when they speak, fantastic animals appear in the forest. - Technique: The app extracts the dominant frequency of the incoming sound and maps it to the branching angle of a growing tree, pausing growth in silence. - Try it: Build a p5.js sketch that grows a recursive tree only while the microphone hears a whistle, with pitch controlling branch angle. Twist: make it an AR scene where the tree grows from the table you whistle at. #### Appel d'Air — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2012) - Video: https://vimeo.com/74689338 - Interaction: Tangible Objects, Voice & Sound, Play - Platform & tech: Projection, sensors, industrial fan, Arduino - Idea: A small breath gets a big answer. - What it is: Toddlers blow on a tiny paper windmill, and an industrial fan across the room answers with a gust of wind as long and strong as their breath. - Technique: A sensor on the paper windmill measures the speed and duration of its spin, and a controller drives the industrial fan in proportion. - Try it: Attach a cheap anemometer or a DC motor used as a generator to a paper windmill and map its signal to a USB fan or a projected storm. Twist: the answer comes back from a different corner every time. #### Loup-garou — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2012) - Video: https://vimeo.com/119942169 - Interaction: Hands & Body, Perception & Effects, Projection - Platform & tech: Projection, camera, body tracking, projector - Idea: Stand in front of the moon and become a werewolf. - What it is: In a dark, eerie space, anyone who stands in front of a projected full moon sees their silhouette turn into a werewolf that copies their pose. - Technique: A camera detects body positions and silhouettes, and the software swaps the silhouette for werewolf animations triggered by pose. - Try it: Use a pose-tracking model (MediaPipe or ml5.js) to replace a student's shadow with an animated creature when they step into a marked circle. Twist: the transformation only happens when two people stand close together. #### Sirènes Sylvestres — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2012) - Video: https://vimeo.com/74054426 - Interaction: Location & City, Hands & Body, Perception & Effects - Platform & tech: Projection, motion sensors, LED, Arduino - Idea: Lights that wake up as you come near and lead you into the dark. - What it is: A night walk through the woods where small lights hidden in the trees switch on as walkers approach, luring them step by step off the lit path and deeper into the forest. - Technique: Autonomous light and motion-sensor modules are placed along a path in the forest; each module lights up when someone enters its range, forming a progressive trail. - Try it: Place five phones or PIR-sensor lamps along a school corridor or garden path so each lights up when someone passes and guides them to a hidden spot. Twist: the lights switch off behind you so you cannot go back. #### Starfield — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2012) - Video: https://vimeo.com/36892768 - Interaction: Hands & Body, Tangible Objects, Projection - Platform & tech: Projection, Kinect, openFrameworks, projector - Idea: A playground swing becomes a spaceship. - What it is: A visitor sits on a real swing in front of a projection. As they swing back and forth, they fly through a galaxy of stars whose motion follows the rhythm of their body. - Technique: A Kinect behind the swing tracks its position, and an openFrameworks app maps the swing's arc to the camera's forward speed through a particle starfield, with an optional anaglyph 3D mode. - Try it: Track a swinging object (a pendulum, a chair on a rope or a student on a swing) with a webcam or phone accelerometer and drive a starfield in p5.js or Unity with its speed. Twist: the stars only move while two people swing in sync. #### Stop-iT — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2012) - Video: https://vimeo.com/37536157 - Interaction: Tangible Objects, Voice & Sound, Drawing & Making - Platform & tech: Projection, openFrameworks, camera, MIDI - Idea: A wall of post-it notes is a music sequencer. - What it is: Visitors stick coloured post-it notes on a wall to compose music. Each colour is an instrument, the height of a note sets its pitch, and the melody loops while people rearrange it. - Technique: A camera analyses the wall in real time, detects each note's colour and position, and an openFrameworks app turns the grid into MIDI loops. - Try it: Point a phone or webcam at a sheet of paper, detect coloured stickers with simple colour thresholding, and play a step sequence with Tone.js. Twist: use food on a plate instead of stickers. #### Galets Magiques — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2013) - Video: https://vimeo.com/64186902 - Interaction: Tangible Objects, Projection, Play - Platform & tech: Projection, sensors, projector - Idea: Shake a real pebble to reveal the animals hiding under it. - What it is: Children pick up and shake pebbles lying on a projected seashore. Each shake sends out a ripple, and fish and sea creatures hidden under the pebble swim out. - Technique: Physical pebbles likely carry motion sensors (or are tracked by a camera) so the software knows which one was shaken, and a top-down projector draws a ripple and animated creatures around that pebble's position. - Try it: Put three objects on a table under a projector or in a Lens Studio / AR Foundation scene, detect when one is lifted or moved, and spawn creatures from that spot. Twist: creatures only come out if the object is put back gently. #### Portée/ — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2014) - Video: https://vimeo.com/118226187 - Interaction: Tangible Objects, Voice & Sound - Platform & tech: Projection, electroluminescent wire, Disklavier, sensors - Idea: Unfold a music score into space and let people play it by touching the lines. - What it is: Sixteen glowing electroluminescent wires are stretched across a room like the staves of a score. When visitors touch and pluck a wire, a real Disklavier grand piano plays the note assigned to it, so the room becomes a walk-in instrument. - Technique: Vibration or touch sensors on each wire send events to a computer that triggers MIDI notes on a self-playing piano, while the wires themselves light up (sensor type likely capacitive or piezo). - Try it: Stretch five strings or ribbons across the classroom, attach a piezo or capacitive sensor to each with an Arduino or Makey Makey, and map each string to a note in a browser synth. Twist: each string plays a different student's recorded voice instead of a note. #### Empreintes — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2015) - Video: https://vimeo.com/152416235 - Interaction: Hands & Body, Drawing & Making, Projection - Platform & tech: Projection, depth sensor, projector - Idea: Make a gesture tangible by printing it into the air. - What it is: Visitors move their hands in front of a projection and leave sculptural traces of their gestures. The prints hang in space for a while, then slowly warp and fade. - Technique: A depth sensor tracks the hands and the software sweeps their silhouettes through time into 3D forms that are rendered and projected, then deformed over time. - Try it: Use MediaPipe hand tracking in the browser to record a hand's outline every frame and stack the outlines into a 3D ribbon you can orbit around. Twist: the print only stays if the hand moves very slowly. #### Narcisse — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2018) - Video: https://vimeo.com/273004699 - Interaction: Hands & Body, Face, Perception & Effects - Platform & tech: Projection, depth sensor, projector, screen - Idea: A mirror that slowly stops being you. - What it is: A screen acts as a mirror, but the reflection of the visitor is gradually altered and deformed, asking at what point we stop recognising ourselves. - Technique: A depth sensor captures the visitor's body and the software re-renders it as a live mirror image with progressive distortions (the exact deformations are likely generative). - Try it: Build a webcam mirror in p5.js or TouchDesigner that becomes more distorted the longer a person stands still. Twist: the distortion copies the face of the previous visitor. #### Nebula — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2018) - Video: https://vimeo.com/318206534 - Interaction: Hands & Body, Tangible Objects, Perception & Effects - Platform & tech: Projection, Arduino, fans, proximity sensors - Idea: Your presence is the wind that reshapes a landscape. - What it is: A long dark room is covered with drifts of white paper particles. Nine fans switch on as visitors approach, so every step blows the particles into new landscapes that the next visitor inherits. - Technique: Proximity sensors read by Arduino boards switch fans on and off depending on where visitors move, and the system pauses to self-regulate once enough particles pile up. - Try it: Build a tabletop version: place a small USB fan and a distance sensor at each side of a tray of confetti, and blow when a hand comes near. Twist: the fans blow away from the hand, so you can never reach the confetti. #### Wood Wide Web — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2023) - Video: https://vimeo.com/940813772 - Interaction: Information & UI, Perception & Effects - Platform & tech: Phone, image tracking, generative graphics - Idea: Reveal the hidden network that connects trees on top of a printed image. - What it is: Prints of forest imagery come alive in augmented reality: through a device, generative networks grow across the images like the fungal web that links tree roots underground. - Technique: Printed images act as AR image targets, and generative growth algorithms inspired by mycelium are rendered on top of them in real time (details of the app likely custom). - Try it: Print a photo of a tree, use it as an image target in Lens Studio or AR Foundation, and grow branching lines between the roots with an L-system. Twist: the network grows faster when two phones look at the print at the same time. #### Growing — Lab212 (Béatrice Lartigue, Nicolas Guichard et al.) (2025) - Video: https://vimeo.com/1159696549 - Interaction: Hands & Body, Projection, Location & City - Platform & tech: Projection, object detection, projectors, IR lights - Idea: Let passers-by feed or disturb a spreading organism. - What it is: A 55-metre-wide projection shows invasive species spreading across a landscape. Real-time tracking detects people and boats, and the growth reacts to them. - Technique: An object-detection model classifies people and boats in a camera feed under infrared light, and their positions steer a generative growth simulation projected across a huge surface. - Try it: Run a YOLO or COCO-SSD detector in the browser on a window view and let each detected person or car seed a growing pattern on screen. Twist: the pattern only grows where nobody has walked for a while. ### Paris AR Studio (Snap) *Snap's in-house augmented reality studio for culture, museums and music* Opened by Snap in 2022, the studio co-creates Lenses and Spectacles experiences with institutions and artists such as the Louvre, Versailles, Centre Pompidou, JR, Daft Punk and the Royal Philharmonic Orchestra. #### 8 Mars 8 Femmes — Paris AR Studio (Snap) (2023) - Video: https://www.youtube.com/watch?v=pXp_CRjJp1I - Interaction: Location & City, Information & UI - Platform & tech: Phone, Snapchat Lenses, Lens Studio - Idea: Put forgotten women back into the city as life-size AR portraits. - What it is: For International Women's Day 2023, AR Lenses bring eight women from French history, including Simone Veil, Simone de Beauvoir, Joséphine Baker and Olympe de Gouges, to life as illustrated figures with their stories, likely placed at locations around Paris. - Technique: Snapchat Lenses likely combine location-based placement around Paris with illustrated 2D/3D characters and short narrated stories. - Try it: Choose five people connected to your school or neighbourhood and place an AR portrait of each at a meaningful spot with 8th Wall or Lens Studio. Twist: each portrait only speaks when the visitor stands where that person once stood. #### Daft Punk: Memories Unlocked — Paris AR Studio (Snap) (2023) - Video: https://www.youtube.com/watch?v=B--2KoWqBJc - Interaction: Location & City, Voice & Sound, Tangible Objects - Platform & tech: Phone, Lens Studio, Snapchat, Image Tracking - Idea: Turn an album release into a worldwide AR treasure hunt where the cover art itself is a key. - What it is: For the 10th anniversary of Random Access Memories, fans hunt AR treasures across ten global locations to unlock a music video, scan the 2013 album cover to hear a hidden track, and see billboards come alive. - Technique: Snap Lenses combine Landmarker location triggers, image-marker tracking of the album cover and billboards, and spatial audio playback. - Try it: Design a three-stop AR scavenger hunt around your building where scanning posters unlocks pieces of one song. Twist: the final stem only plays when all three pieces have been found by different people. #### Le Louvre: Egypt Augmented — Paris AR Studio (Snap) (2023) - Video: https://www.youtube.com/watch?v=tWwvSAxCbSo - Interaction: Location & City, Information & UI, Portals & Worlds - Platform & tech: Phone, Lens Studio, Custom Landmarker, Snapchat - Idea: Use AR to show a monument as it looked on the day it was made, right on top of how it looks now. - What it is: Snapchat Lenses made with the Louvre bring ancient Egypt back to life: pointing the phone at the Obelisk on Place de la Concorde and at Egyptian antiquities restores their original gilding, colours and surroundings. - Technique: Custom Landmarker / custom location tracking built from 3D scans locks restored models onto the real obelisk and artefacts in Lens Studio. - Try it: Scan a statue, doorway or old building on campus with a phone and build a Lens or AR scene that restores a missing part or its original colour. Twist: let a slider move the viewer through three different centuries. #### The Fool Tour (Jain) — Paris AR Studio (Snap) (2023) - Video: https://www.youtube.com/watch?v=d4xsSAjNt70 - Interaction: Performance, Face, Voice & Sound - Platform & tech: Phone, Lens Studio, Snapchat - Idea: Give a concert crowd a shared AR layer that only exists during the show. - What it is: Two Lenses for singer Jain's live show: the 'Jain Machine' deals anyone a playful prediction of their future, and a second Lens opens a cosmic world that fans see during the concert itself. - Technique: A face-tracked fortune-telling Lens plus a world Lens timed to the live set, which likely anchors large-scale cosmic visuals to the stage direction. - Try it: Create a Lens for a school performance that shows a different AR sky during each song of a three-song set. Twist: the audience's cheering volume decides how intense the sky becomes. #### Augmented Burtonesque (The World of Tim Burton) — Paris AR Studio (Snap) (2024) - Video: https://www.youtube.com/watch?v=CmBZcQVAWkE - Interaction: Location & City, Portals & Worlds, Perception & Effects - Platform & tech: Phone, Lens Studio, Snapchat - Idea: Extend an artist's exhibition beyond the gallery walls into the shared public space of the museum. - What it is: In the atrium of London's Design Museum, visitors raise their phones to open a space-themed Burtonesque fantasy world that extends Tim Burton's exhibition into the building itself. - Technique: A location-anchored world Lens (likely a custom Landmarker of the atrium) places large animated Burton-style characters and sets at architectural scale. - Try it: Pick a lobby or stairwell and design an AR 'takeover' in the visual style of an artist you admire, sized to the real architecture. Twist: the takeover only appears when the space is almost empty. #### Paris 1924 — Paris AR Studio (Snap) (2024) - Video: https://www.youtube.com/watch?v=XqI0etsPbKo - Interaction: Location & City, Portals & Worlds - Platform & tech: Phone, Lens Studio, City Landmarker, Camera Kit - Idea: Time-travel the street you are standing on back exactly one hundred years. - What it is: Marking 100 years since Paris last hosted the Olympics, fans in the city see the streets around them transform into 1924 Paris, while fans elsewhere are transported to the 1924 Yves-du-Manoir stadium. - Technique: Two modes of one Lens: City/Landmarker tracking re-skins real Paris buildings on site, and a portal scene reconstructs the stadium for remote users. - Try it: Find an old photo of a spot on campus and build an AR view that overlays the historical scene aligned to the present. Twist: add a portal for people who are not on site. #### Un banquet augmenté (Château de Chantilly) — Paris AR Studio (Snap) (2024) - Video: https://www.youtube.com/watch?v=OAARNyLOYtA - Interaction: Tangible Objects, Information & UI, Spatial Mapping - Platform & tech: Phone, Lens Studio, Snapchat - Idea: Serve a historical feast on a real table so visitors learn food history by looking at their plate. - What it is: At the Château de Chantilly, visitors use a Lens to discover 19th-century court cooking as dishes and banquet scenes appear on the castle's real tables. - Technique: World and surface tracking place animated 3D dishes and table settings, likely recreated from historical recipes with a chef partner, at true scale on the château's tables. - Try it: Build an AR placemat that serves a dish from your family's or city's history, with labels for each ingredient's origin. Twist: the dish changes era every time someone takes a bite of real food. #### Dansez Versailles — Paris AR Studio (Snap) (2025) - Video: https://www.youtube.com/watch?v=-E68tunQe0E - Interaction: Hands & Body, Location & City, Performance - Platform & tech: Phone, Lens Studio, Camera Kit, Body Tracking - Idea: Let visitors physically perform history in the exact place it happened. - What it is: Four Lenses placed in the gardens of the Palace of Versailles dress visitors in Baroque costume and put them into historical court dances, with wigs and hats on the front camera and full-body dancers on the rear camera. - Technique: Face tracking dresses the selfie view while full-body tracking and garment/body-mesh templates restyle the whole figure, triggered at specific garden locations inside the palace app via Camera Kit. - Try it: Build a body-tracked Lens that teaches one historical or folk dance step by showing a ghost dancer to copy and a costume that appears only when you match the pose. Twist: two people must dance mirrored to unlock the full costume. #### Seven Wonders of the Ancient World — Paris AR Studio (Snap) (2025) - Video: https://www.youtube.com/watch?v=HZ0sseutYoE - Interaction: Information & UI, Spatial Mapping - Platform & tech: Phone, Snapchat Lenses, Lens Studio - Idea: Let a TV documentary spill its monuments into the viewer's room. - What it is: An 'augmented documentary series' made with Sandstone Global: Snapchat Lenses accompany the TV programme and rebuild the Seven Wonders, from the pyramids to the Colossus, as 3D models viewers can explore around them. - Technique: Snapchat world Lenses likely place detailed reconstructions of each wonder on the floor or table, paired with each episode of the programme. - Try it: Pick a documentary topic and build a companion AR scene (Lens Studio or model-viewer) that places one reconstructed object at true scale in the classroom. Twist: the object is revealed layer by layer as the documentary narration progresses. #### Spectacular: The Art of Jonathan Yeo in Augmented Reality — Paris AR Studio (Snap) (2025) - Video: https://www.youtube.com/watch?v=kEoDvmKSR5E - Interaction: Gaze & Attention, Perception & Effects, Tangible Objects - Platform & tech: Headset, Snap Spectacles, Lens Studio, Image Tracking - Idea: Paintings that know you are looking and answer back. - What it is: The first AR art exhibition on Snap Spectacles: as visitors approach Jonathan Yeo's royal, celebrity and self-portraits, the paintings are recognised and transform and respond around the frame. - Technique: Snap trained recognition models on each painting so Spectacles detect and track the canvas, then play spatial animations registered to it. - Try it: Choose one portrait and build an image-tracked AR layer where the subject reacts differently depending on how close the viewer stands. Twist: the portrait remembers the last visitor and greets the next one about them. #### The Colours of Music (Royal Philharmonic Orchestra) — Paris AR Studio (Snap) (2025) - Video: https://www.youtube.com/watch?v=QDbd0_9HcmI - Interaction: Voice & Sound, Perception & Effects, Performance - Platform & tech: Phone, Lens Studio, SnapML, Audio Analysis - Idea: Let everyone, including people who are hard of hearing, see music the way synaesthetes do. - What it is: Scanning a musician or instrument with this Lens turns the notes being played into evolving 3D colours and shapes around the performer, inspired by chromesthesia. - Technique: Machine learning detects instruments and analyses live audio pitch and timbre, mapping them to colour, form and motion of spatial particles anchored near the player. - Try it: Build an AR visualiser that maps pitch to colour and loudness to size for sounds from one real instrument in class. Twist: each classmate designs their own colour mapping, and the class compares whose feels most 'right'. #### Echoes (JR, La Caverne du Pont Neuf) — Paris AR Studio (Snap) (2026) - Video: https://www.youtube.com/watch?v=9BntUCcWbWQ - Interaction: Hands & Body, Location & City, Perception & Effects - Platform & tech: Phone, Headset, Lens Studio, Snap Spectacles, Body Tracking - Idea: Layer an invisible, living cave over a physical monumental artwork so the installation keeps changing as people walk through it. - What it is: Inside JR's 120-metre artificial cave wrapped around the Pont Neuf, visitors on phones or Spectacles see bats leaving sculptural light trails, echoes of the bodies around them, and a story told through a rift in the rock. - Technique: Three Lenses run on both Snapchat and Spectacles, using body tracking to create delayed 'echo' copies of visitors and world tracking to anchor bats and story scenes in the cave. - Try it: Prototype a Lens that leaves a fading ghost copy of anyone who walks through a corridor, delayed by three seconds. Twist: ghosts from different people merge when their paths cross. #### The Incredible Unknowns of the Louvre — Paris AR Studio (Snap) (2026) - Video: https://www.youtube.com/watch?v=F0HUDsG6lCI - Interaction: Information & UI, Tangible Objects, Gaze & Attention - Platform & tech: Phone, Lens Studio, Snapchat - Idea: Make visitors stop at the works everyone walks past by giving each one an AR secret. - What it is: Six overlooked Louvre masterpieces, including the Code of Hammurabi and the Kore of Samos, get a QR-code Lens that restores their lost shapes and colours and animates hidden details live on camera. - Technique: Each Lens tracks a specific artwork (likely 3D object or image tracking from museum scans) and overlays curator-validated reconstructions built from historical archives. - Try it: Pick the most ignored object in a local museum or school display and build an AR layer that reveals one fact nobody knows about it. Twist: the reveal only triggers if the viewer stands still for ten seconds. ### Pedro Lopes *Associate Professor, University of Chicago; directs the Human Computer Integration Lab* Builds interfaces that actuate the user's own body - electrical muscle stimulation, chemical and thermal haptics - to give mixed reality physical weight and to let objects 'tell' your hands how to use them. #### Affordance++ — Pedro Lopes (2015) - Video: https://www.youtube.com/watch?v=Gz4dphzBb6I - Interaction: Hands & Body, Tangible Objects - Platform & tech: Wearable, electrical muscle stimulation, object tracking - Idea: Objects use electrical stimulation to teach your hand directly how to use them. - What it is: Objects 'tell' the user how to use them by actuating the user's own hand with EMS: grabbing a spray can makes you shake it, touching a hot mug makes your hand pull back. - Technique: Detects which object the hand approaches and plays a pre-recorded EMS pattern on forearm muscles that enacts the object's intended movement. - Try it: Stick QR codes on a few objects in the classroom, recognize them in phone AR, and overlay "animated usage hints" (shaking, rotating arrows) on the hand. Twist: express the hints as phone vibration rhythms instead of images. #### Impacto — Pedro Lopes (2015) - Video: https://www.youtube.com/watch?v=k5e4mXQLq54 - Interaction: Hands & Body, Play - Platform & tech: Wearable, Headset, EMS, solenoid, VR - Idea: Make a punch in VR actually land on you. - What it is: A wearable combining a tactile solenoid tap with EMS muscle contraction so virtual boxing punches and soccer headers feel like real impacts. - Technique: Pairs a solenoid for the skin-level tap with EMS that jerks the limb backwards, together simulating the momentum of an impact. - Try it: Make a phone AR boxing game where, whenever you get hit, a partner taps your arm with a foam stick as "human haptics". Twist: compare how players dodge with and without the touch feedback. #### Force Feedback for Mixed Reality via EMS — Pedro Lopes (2018) - Video: https://www.youtube.com/watch?v=qHRn05Kmzew - Interaction: Hands & Body, Spatial Mapping, Play - Platform & tech: Headset, Wearable, HoloLens, electrical muscle stimulation - Idea: Press a virtual button or push a virtual wall in HoloLens and your arm really feels the resistance. - What it is: On HoloLens, virtual buttons, walls and heavy objects push back: EMS on the user's arm muscles creates resistance so the user feels them without holding any device. - Technique: Detects hand contact with virtual objects and stimulates antagonist muscles to create counter-force while leaving the hands free to touch real objects. - Try it: Place a virtual wall in phone AR, and when the phone passes through it, make the phone vibrate strongly and turn red to signal 'resistance'. Twist: tie a resistance band to your arm to simulate the real resistance of pushing a wall. #### Altering Perceived Softness of Real Rigid Objects — Pedro Lopes (2021) - Video: https://www.youtube.com/watch?v=I2BBV0JZ0ww - Interaction: Hands & Body, Tangible Objects, Perception & Effects - Platform & tech: Wearable, fingerpad restriction, haptic illusion - Idea: Change how the fingerpad deforms and a hard table feels soft. - What it is: A thin device restricts how the fingerpad deforms, so pressing a hard real object feels softer or harder than it is, useful for MR props. - Technique: Constrains lateral fingerpad spread with a rigid ring and nail mount, changing the tactile cue the brain uses to judge compliance. - Try it: Have classmates press the same sponge or tabletop with a bare finger and with a finger tightly wrapped in tape, and rate the difference in softness. Twist: add a soft or hard visual cue in phone AR and see whether it strengthens the illusion. #### Touch&Fold — Pedro Lopes (2021) - Video: https://www.youtube.com/watch?v=yg8BOKhs4XM - Interaction: Hands & Body, Tangible Objects - Platform & tech: Wearable, Headset, foldable haptic actuator, mixed reality - Idea: A fingertip haptic device folds out when needed and tucks away when not, so you can feel both virtual and real things. - What it is: A nail-mounted actuator unfolds a small pad onto the fingertip only when touching virtual objects, and folds away so the finger can feel real objects normally. - Technique: Mounts a servo-driven folding tactor on the fingernail that rotates onto the pad for virtual contact and retracts for real-world touch. - Try it: Make a paper finger-flap device: when you touch a virtual object in phone AR, a partner flips the paper down onto your fingerpad, and flips it up when you touch a real object. Twist: swap the paper for different materials to stand for different virtual objects. #### FeetThrough — Pedro Lopes (2023) - Video: https://www.youtube.com/watch?v=lMln8FFJ4KA - Interaction: Hands & Body, Location & City - Platform & tech: Wearable, electrotactile stimulation, insole - Idea: Your soles feel the virtual ground without losing touch with the real one. - What it is: An electrotactile insole renders virtual textures and bumps under the feet while its thin design still lets the wearer feel the real ground. - Technique: Uses thin electrode arrays on the foot's arch and sides to deliver electrotactile patterns without covering the sole's pressure-sensing areas. - Try it: Design a walking route with phone GPS or AR 'underfoot cues', where the phone in your pocket vibrates when you reach key points. Twist: use vibration patterns to simulate different ground surfaces (grass, sand). #### Stick&Slip — Pedro Lopes (2024) - Video: https://www.youtube.com/watch?v=UxqNf1BSIoo - Interaction: Hands & Body, Tangible Objects, Perception & Effects - Platform & tech: Wearable, liquid dispensing, friction modulation - Idea: Coat your fingertip with a thin layer of liquid, and any surface feels slippery or rough. - What it is: Tiny liquid coatings are dispensed onto the fingertip to change friction on the fly, making any real surface feel sticky or slippery in mixed reality. - Technique: Pumps small amounts of lubricating or friction-raising liquids onto the fingerpad and removes them, altering finger-surface friction without instrumenting the surface. - Try it: Prepare three fingertip coatings (hand cream, talcum powder and water), have classmates touch the same sheet of paper and describe how it feels, and pair this with phone AR that changes the paper's texture to ice or sand. Twist: run a blind test to see whether people can guess the coating. #### Generative Muscle Stimulation — Pedro Lopes (2026) - Video: https://www.youtube.com/watch?v=pJM2Z8mmwAw - Source code: https://github.com/humancomputerintegration/EmbodiedAI - Interaction: Hands & Body, Tangible Objects - Platform & tech: Wearable, electrical muscle stimulation, multimodal LLM, computer vision - Idea: Instead of answering in words, the AI answers by moving your body. - What it is: An AI that helps through your muscles: it looks at your pose and surroundings, understands a request such as opening a pill bottle, and generates electrical-muscle-stimulation instructions that physically move your hand and arm to do it. - Technique: Computer vision and a multimodal LLM propose movement steps from the user's context, which are constrained by a muscle-stimulation knowledge base and joint limits before being sent to an EMS device. - Try it: Without EMS hardware, prototype the same loop with visual guidance: send a photo of a task (open a jar, pour water) to a multimodal LLM that may only answer with a sequence of body actions from a fixed list (rotate wrist, grip, lift), and render each step as an AR ghost hand. Twist: make the AI refuse unsafe or impossible motions and discuss who is in control. ### Steven Feiner — Columbia Computer Graphics & User Interfaces Lab *Professor of Computer Science, Columbia University; director of the CGUI Lab* Steven Feiner built KARMA (1993) and the Touring Machine (1997), the first outdoor mobile AR system, and has led decades of research on AR task guidance, maintenance and 3D interaction with students such as Steve Henderson and Carmine Elvezio. #### ARMAR: AR for Maintenance and Repair — Steven Feiner — Columbia Computer Graphics & User Interfaces Lab (2009) - Video: https://www.youtube.com/watch?v=LaJxNw4-GXo - Interaction: Information & UI, Tangible Objects - Platform & tech: Headset, head-worn display, optical tracking - Idea: Inside an armored vehicle turret, head-worn AR arrows point straight at the next bolt to turn. - What it is: Steve Henderson and Steven Feiner guide a mechanic inside an armored personnel carrier turret with head-worn AR arrows, labels and animated tools, speeding up task localization. - Technique: An optically tracked head-worn display renders arrows, labels and animated tool models registered to the vehicle interior, guiding attention to the next task location and demonstrating the procedure in place. - Try it: In phone AR, use an image target to locate a printer or bicycle and build arrow, label and animated-wrench guidance for a five-step repair procedure. Twist: when the target part is out of view, show a direction hint at the edge of the screen. #### Augmented Reality Marble Game — Steven Feiner — Columbia Computer Graphics & User Interfaces Lab (2009) - Video: https://www.youtube.com/watch?v=6AKgH4On65A - Interaction: Play, Tangible Objects - Platform & tech: Headset, marker tracking, physics - Idea: Tilt a real board in your hands to steer virtual marbles through an AR maze. - What it is: A tilting physical board controls virtual marbles rolling through AR mazes seen through a head-worn display. - Technique: The tracked board's orientation relative to gravity is computed from the marker pose, and that tilt is fed as a gravity vector into a physics engine that rolls virtual marbles through a maze registered to the board. - Try it: In AR, use a printed maze as an image target, read its tilt relative to the table, and drive the Unity physics engine to roll a virtual marble. Twist: have two people hold the same board and clear the level together. #### Shake Menus — Steven Feiner — Columbia Computer Graphics & User Interfaces Lab (2009) - Video: https://www.youtube.com/watch?v=DE7AMVtDo9U - Interaction: Tangible Objects, Hands & Body - Platform & tech: Headset, marker tracking - Idea: Shake the object in your hand and a menu pops up around it. - What it is: Shake a tangible AR marker to pop up a radial menu around it; menus are anchored to the object in your hand. - Technique: The tracked marker's pose history is analyzed for a rapid back-and-forth motion, and when a shake is detected a radial menu is instantiated in the marker's coordinate frame so it moves with the object. - Try it: Detect a 'shake' with an image-tracked card or the phone accelerometer, pop up a ring-shaped AR menu around the card, and tilt the card to pick an option. Twist: swap in other physical gestures (flip, tap) and compare which feels most natural. #### ARmonica — Steven Feiner — Columbia Computer Graphics & User Interfaces Lab (2010) - Video: https://www.youtube.com/watch?v=IT7ao3izQQI - Interaction: Voice & Sound, Shared & Social, Play - Platform & tech: Headset, AR, physics, spatial audio - Idea: Several people place virtual objects in space and let them bounce and collide to make music. - What it is: A collaborative sonic AR environment where users place and bounce virtual objects off physical bars to trigger musical notes together. - Technique: Physical bars are tracked and modeled as colliders in a physics simulation, and virtual objects bouncing off them trigger pitched sound events rendered with spatial audio for all participants. - Try it: Detect a tabletop in AR, mark a few real pencils or rulers as 'keys' (placing matching virtual colliders by hand), and make virtual balls play different pitches when they bounce off them. Twist: two people each control a launch point and cooperate to play a melody. #### AR in the Psychomotor Phase of a Procedural Task — Steven Feiner — Columbia Computer Graphics & User Interfaces Lab (2011) - Video: https://www.youtube.com/watch?v=2eSlMSJ65Kc - Interaction: Information & UI, Tangible Objects, Hands & Body - Platform & tech: Headset, head-worn display, tracking - Idea: AR animation demonstrates how your hands should move, right on the real part. - What it is: Head-worn AR shows animated guidance for the hands-on part of an assembly task (aligning and inserting parts) directly on the physical engine combustor. - Technique: The combustor parts are tracked in 6DOF, and animated overlays compare the current part pose with the target pose, showing alignment cues (arrows, rotation hints) until the part is correctly inserted. - Try it: Use image tracking on a LEGO baseplate and a part with a sticker, and show a ghost of the target position in AR that turns greener as the part gets closer. Twist: remove the visual hints and guide the alignment with sound alone. #### SnapAR — Steven Feiner — Columbia Computer Graphics & User Interfaces Lab (2012) - Video: https://www.youtube.com/watch?v=rU5aJLOuiDI - Interaction: Information & UI, Spatial Mapping - Platform & tech: Phone, handheld AR - Idea: Take a few AR snapshots of a scene, then jump between viewpoints without walking around. - What it is: Take virtual snapshots of a scene from different viewpoints on a handheld AR device and switch between them instantly instead of walking around. - Technique: The handheld stores tracked camera images with their poses as snapshots, and the user switches the displayed view to a stored snapshot (with live AR overlays re-rendered for that pose) instead of physically walking to it. - Try it: Implement a "save viewpoint" button in AR Foundation that stores the current camera image and pose, so tapping a thumbnail later switches to that viewpoint to view the virtual objects. Twist: use these snapshots to make a multi-view "AR comparison sheet" that explains a piece of work. #### AR Task Guidance for ISS Stowage — Steven Feiner — Columbia Computer Graphics & User Interfaces Lab (2021) - Video: https://www.youtube.com/watch?v=BvddkxWVxso - Interaction: Information & UI, Spatial Mapping - Platform & tech: Headset, HoloLens - Idea: On the space station's cramped racks, AR points out where each item belongs. - What it is: Carmine Elvezio and colleagues guide astronauts' stowage operations with head-worn AR, highlighting where each item goes in cramped space-station racks. - Technique: A HoloLens is registered to the rack layout, and the application highlights the target stowage location with world-locked outlines and path cues while tracking task progress item by item. - Try it: Build an item list for a dorm cabinet or lab drawers and show a highlight box in phone AR for which slot each item goes in, one item at a time. Twist: reverse it into a find-things mode and test how much faster it is than a text-only list. ### visiophone (Rodrigo Carvalho) *Audiovisual artist and researcher; co-founder of Boris Chimp 504 and Openfield* Rodrigo Carvalho, working as visiophone, builds real-time systems where dancers, visitors and sound drive particles, lasers and light, from Kinect dance pieces in Barcelona and Austin to laser instruments and light installations with Openfield and the audiovisual duo Boris Chimp 504. He also made early prototypes at Motion Bank's first Choreographic Coding Lab. #### Interzone — visiophone (Rodrigo Carvalho) (2009) - Video: https://vimeo.com/9827907 - Interaction: Location & City, Projection, Perception & Effects - Platform & tech: Projection, Arduino, Processing, infrared sensors - Idea: Turn a corridor you pass through every day into a zone that notices you. - What it is: An audiovisual installation for transit spaces such as tunnels and stairways, which represents a post-apocalyptic zone where space-time anomalies appear as projections and sounds follow people passing through. - Technique: Six infrared sensors connected to an Arduino detect a visitor's position and send it to Processing, which triggers projected effects and sound loops. - Try it: Line a corridor with three cheap distance sensors on an Arduino and trigger different projected glitches and sounds as a person walks through. Twist: make the effects lag one step behind, as if the corridor remembers. #### Reactive Stage — visiophone (Rodrigo Carvalho) (2010) - Video: https://vimeo.com/12171899 - Interaction: Projection, Performance, Hands & Body - Platform & tech: Projection, Quartz Composer, camera tracking - Idea: Layer projections on sheer fabric to put a dancer inside the image. - What it is: A mutating graphic shape follows a dancer's position and size and changes colour and complexity with the music, projected onto three layers of transparent fabric so she seems to move inside it. - Technique: A camera tracking system sends the dancer's position and size to a Quartz Composer patch, which also analyses audio, and the output is projected through three stacked scrims for depth. - Try it: Hang two or three layers of tulle one metre apart, project one tracked shape onto all of them, and have a dancer move between the layers. Twist: give each layer a different delay so the shape trails through depth. #### Dancing With Swarming Particles — visiophone (Rodrigo Carvalho) (2011) - Video: https://vimeo.com/21052774 - Interaction: Hands & Body, Performance, Perception & Effects - Platform & tech: Projection, Kinect, Unity, OSCeleton - Idea: Your body is a magnet that gathers a swarm into your own shape. - What it is: A performer faces a projected avatar made of flocking particles that drift aimlessly until the energy of her movement pulls them together into a copy of her body; afterwards visitors try their own swarm. - Technique: A Kinect skeleton sent over OSC by OSCeleton drives a Unity3D flocking system, whose attraction to the joints grows with the performer's movement energy. - Try it: Build a p5.js boids sketch attracted to MediaPipe body landmarks, with attraction scaled by how fast the person moves, and project it life-size. Twist: when two people stand close, let their swarms swap bodies. #### Drawing with the Body — visiophone (Rodrigo Carvalho) (2011) - Video: https://vimeo.com/19142510 - Interaction: Drawing & Making, Hands & Body - Platform & tech: Desktop, Kinect, Unity, OpenNI, OSC - Idea: Every joint of the body is a brush that paints in the air. - What it is: Early Kinect experiments in which hands, feet and head become brushes that leave coloured strokes hanging in 3D space around the dancer. - Technique: Kinect skeleton tracking via OpenNI/NITE and OSCeleton streams joint positions over OSC into Unity3D, which leaves trails behind chosen joints. - Try it: In AR Foundation or a WebXR sketch, attach a trail renderer to body-tracked wrists and ankles so a short dance leaves a 3D drawing you can walk around afterwards. Twist: fade each stroke at the speed it was drawn. #### Dance Jockey — visiophone (Rodrigo Carvalho) (2012) - Video: https://vimeo.com/43462444 - Interaction: Hands & Body, Performance, Voice & Sound - Platform & tech: Projection, Kinect, Quartz Composer - Idea: The dancer is the DJ and VJ at once. - What it is: An audiovisual dance performance with musician Yago de Quay in which a Kinect reads the dancer's position and arm movements to mix, move and distort both the music and the projected footage. - Technique: Kinect skeleton data are mapped to video parameters such as position, opacity and distortion in Carvalho's visual system, and to music parameters on the sound side. - Try it: Map three body features from MediaPipe Pose (arm height, hip position, hand distance) to three parameters of a music loop and a projected video, then perform a one-minute set. Twist: hand one parameter to an audience member's phone. #### Breakdown — visiophone (Rodrigo Carvalho) (2014) - Video: https://vimeo.com/95846156 - Interaction: Hands & Body, Performance, Play - Platform & tech: Projection, Kinect, physics simulation - Idea: Perform against a world whose physics keep changing. - What it is: A dancer inhabits a projected two-dimensional world whose physical rules are constantly changed by an unseen entity; he adapts, argues with the forces and finally breaks the world's rules. - Technique: Two Kinect cameras capture the dancers' movements and gestures, which interact with a simulated physics world whose gravity and forces are rewritten live. - Try it: Project a Matter.js world onto a wall where a tracked body acts as a collider, and have a partner change gravity and friction live during a one-minute improvisation. Twist: let the dancer earn control of the physics by holding still. #### Warning: A Wearable Electronic Dress Prototype — visiophone (Rodrigo Carvalho) (2014) - Video: https://vimeo.com/95096865 - Interaction: Hands & Body, Tangible Objects, Performance - Platform & tech: Wearable, Arduino, servos, ultrasonic sensor, conductive paint - Idea: A costume that reacts to personal space like an animal. - What it is: A stage costume whose servo-driven collar, inspired by the frill-necked lizard and the Elizabethan ruff, flares when someone comes close, while stripes of conductive paint on the body let the performer play the music by touch. - Technique: An ultrasonic sensor measures proximity and drives four servos in the collar, and six conductive-paint stripes act as touch inputs sending MIDI. - Try it: Prototype a wearable with an Arduino, one distance sensor and one servo that opens a paper collar when someone enters your personal space. Twist: make it close again only if the visitor speaks softly. #### Δ∞ [Infinite Delta] — visiophone (Rodrigo Carvalho) (2016) - Video: https://vimeo.com/176050475 - Interaction: Tangible Objects, Hands & Body, Voice & Sound - Platform & tech: Projection, Arduino, Max, servos - Idea: An architectural surface that reshapes itself around the people near it. - What it is: A structure of triangular planes moved by servomotors reacts to the audience's movement, to sound and to generative sequences, changing its shape like a breathing creature. - Technique: Max and Arduino drive servomotors on each triangle, with audience movement and audio as inputs and audio-reactive visuals on the side (Boris Chimp 504 with Alma d'Arame). - Try it: Build a small kinetic panel of three cardboard triangles on servos that tilt toward the nearest person using a webcam and simple blob tracking. Twist: when nobody is there, let it slowly 'breathe' on its own. #### Close Encounters — visiophone (Rodrigo Carvalho) (2017) - Video: https://vimeo.com/319699135 - Interaction: Location & City, Voice & Sound, Play - Platform & tech: Projection, sensors, LED - Idea: Walking up to a piece of street furniture opens a conversation with aliens. - What it is: A bandstand in a town square becomes a spaceship: when someone approaches, light and sound establish contact with mysterious beings supposedly abducted from Earth decades ago. - Technique: Presence sensors detect approaching visitors and trigger light sequences and sound narratives on the bandstand, by Boris Chimp 504 (Miguel Neto sound, Carvalho lights and interaction). - Try it: Choose an object on campus (a bench, a bus stop) and write a 30-second story that plays through a hidden speaker and light when a proximity sensor or phone geofence detects someone near. Twist: the story changes depending on how many people approach together. #### Friction — visiophone (Rodrigo Carvalho) (2018) - Video: https://vimeo.com/312071200 - Interaction: Hands & Body, Voice & Sound, Perception & Effects - Platform & tech: Projection, laser, Leap Motion - Idea: Play light and sound in the air between your hands. - What it is: An audiovisual instrument for the public: two facing laser beams draw three-dimensional shapes between two frames, and visitors bend the shapes and the sound with their hands. - Technique: Hand gestures, likely read by a Leap Motion sensor, control laser drawing parameters and a sound engine in real time. - Try it: Build a gesture instrument where a phone's hand tracking controls a projected line drawing and a synth; hang a scrim or use haze so the lines read in space. Twist: require two players' hands to stretch one shape. #### Lightflow (performance) — visiophone (Rodrigo Carvalho) (2018) - Video: https://vimeo.com/312068065 - Interaction: Hands & Body, Performance, Voice & Sound - Platform & tech: Projection, Leap Motion, LED, DMX - Idea: A landscape's water flow becomes a light instrument a dancer can play. - What it is: LED tubes reinterpret the water paths of old watermills in a gallery; in the performance version a dancer uses hand gestures to play the light and the field-recorded sound. - Technique: A Leap Motion gesture interface sends hand data to software that drives DMX-controlled LED tubes and layers of on-site sound recordings. - Try it: Record water sounds near your school, lay out a line of addressable LEDs along the floor, and map a phone's hand tracking to the speed of a light pulse and the sound mix. Twist: let the pulse only move downhill, like real water. #### AR Experiments (Boris Chimp 504 print) — visiophone (Rodrigo Carvalho) (2019) - Video: https://vimeo.com/315226706 - Interaction: Voice & Sound, Tangible Objects - Platform & tech: Phone, Vuforia, Unity - Idea: A printed poster becomes a speaker whose sound you can see. - What it is: Pointing a phone at a Boris Chimp 504 art print plays a track and grows a live sound visualisation on top of the printed image. - Technique: Vuforia image tracking in Unity anchors the scene to the print, and the SimpleSpectrum asset turns the audio spectrum into 3D bars. - Try it: Make an image-tracked AR poster in AR Foundation or 8th Wall that plays a song and draws its spectrum as 3D shapes rising from the print. Twist: make the visualisation only appear when two phones look at the poster together. #### Augmented Painting — visiophone (Rodrigo Carvalho) (2019) - Video: https://vimeo.com/317035732 - Interaction: Projection, Perception & Effects, Tangible Objects - Platform & tech: Projection, Processing, Fisica, MadMapper, Ableton Live - Idea: A still painting becomes a physics world for moving light. - What it is: A projector maps animated particles onto a painted canvas by Ivo Teixeira, so the physical painting's shapes collide with, catch and scatter virtual particles in time with sound. - Technique: Particles simulated in Processing with the Fisica physics library use the painting's shapes as colliders, and MadMapper aligns the projection to the canvas. - Try it: Pick a poster or a student drawing, trace its main shapes as colliders in a p5.js + Matter.js sketch, and projection-map falling particles onto it. Twist: let a microphone make the particles heavier when the room is loud. ### John Craig Freeman *Artist; professor at Emerson College; Manifest.AR founding member* Public artist who uses geolocated AR to build memorials and stories in real landscapes, most famously marking migrant deaths along the US-Mexico border. #### Orators, Rostrums and Propaganda Stands — John Craig Freeman (2011) - Video: https://www.youtube.com/watch?v=vlUhVIBjLZY - Interaction: Location & City, Voice & Sound - Platform & tech: Phone, Layar, GPS - Idea: Place AR podiums in public squares that anyone can speak from. - What it is: Virtual orators' podiums, inspired by early Soviet agitprop, are placed in public squares via AR for anyone to 'speak' from. - Technique: Geolocated AR podium models are placed in public squares, with the user physically stepping into the virtual rostrum; likely GPS positioning with compass-based orientation. - Try it: Place an AR podium in a campus square: stepping onto it starts a recording, and each speech stays on the podium for later visitors to hear. Twist: the podium only plays back the previous recording that took the opposite position to the current speaker. #### Tiananmen SquARed — John Craig Freeman (2011) - Video: https://www.youtube.com/watch?v=s7JSUAJXvUY - Interaction: Location & City, Portals & Worlds - Platform & tech: Phone, Layar, GPS - Idea: Re-install a censored image at the exact coordinates where it cannot be shown. - What it is: The Goddess of Democracy statue and Tank Man from the 1989 Tiananmen protests reappear in AR at the ICA Boston, Union Square and Wall Street, and virtually in Tiananmen Square itself. - Technique: 3D models of the statue and the tank-and-man scene are published as geolocated points of interest in a mobile AR browser, placed at multiple coordinates at once, credited to the collective 4Gentlemen. - Try it: Pick an image that was removed from a public place and restore it there in phone AR at its original scale. Twist: make it visible only on an anniversary date. #### Water wARs — John Craig Freeman (2011) - Video: https://www.youtube.com/watch?v=hI-ChNSjnkM - Interaction: Location & City, Portals & Worlds - Platform & tech: Phone, Layar, GPS - Idea: Squat the most exclusive art venue with shelters for those it excludes. - What it is: In front of the Venice Biennale's main pavilion, virtual shanty pavilions appear for undocumented artists, squatters and future water-war refugees. - Technique: Low-poly shelter models are geolocated in the Giardini through a mobile AR browser, as part of Manifest.AR's uninvited 2011 intervention. - Try it: Design an AR 'unofficial pavilion' for a group excluded from a campus space and place it at the entrance of that space. Twist: visitors can add a room to it. #### Border Memorial: Frontera de los Muertos — John Craig Freeman (2012) - Video: https://www.youtube.com/watch?v=E-xFY4zS2DQ - Interaction: Location & City - Platform & tech: Phone, Layar, GPS - Idea: Place AR memorials of the dead at the exact spots where migrants died. - What it is: Geolocated AR skeletal effigies (inspired by Día de los Muertos) mark the exact places where migrants died crossing the US-Mexico border. - Technique: GPS-anchored Layar objects place skeletal effigies at the coordinates of recorded migrant deaths, turning a dataset into a geolocated memorial. - Try it: Take a public place-based dataset (traffic accident sites, shops that have disappeared) and put a small memorial at each data point in WebAR. Twist: generate each memorial's form from the specific details of its data point. #### EEG AR: Things We Have Lost — John Craig Freeman (2012) - Video: https://www.youtube.com/watch?v=mfNXrBIyvYo - Interaction: Gaze & Attention, Location & City, Information & UI - Platform & tech: Phone, Wearable, EEG headset, Layar - Idea: Summon something you lost into the world with the power of your attention. - What it is: People are asked what they have lost; wearing an EEG headset, they concentrate to make the lost object appear in AR at the site, from Liverpool to LACMA's MacArthur Park. - Technique: A consumer EEG headset streams an attention value; above a threshold, a pre-modelled 3D object from the interviews is spawned at the geolocation in a mobile AR app. - Try it: Collect three 'lost things' from classmates, model them, and make them appear in phone AR only after a concentration timer (hold still, stare, or breathe slowly). Twist: use a real EEG or heart-rate sensor. #### Hans RichtAR — John Craig Freeman, Will Pappenheimer (2013) - Video: https://www.youtube.com/watch?v=B2xaxwvOEp0 - Interaction: Perception & Effects, Location & City - Platform & tech: Phone, Layar, iPad - Idea: Let an abstract film escape the screen and move through the museum. - What it is: In LACMA's Hans Richter exhibition, iPads show floating abstract forms inspired by Richter's avant-garde films moving through the gallery around the real works. - Technique: Animated geometric shapes, likely modelled on Rhythmus 21, are placed in the gallery as geolocated or image-anchored augments viewed on tablets. - Try it: Choose a short abstract animation and rebuild its shapes as 3D objects that fly around a real room with plane detection. Twist: sync their rhythm to music playing in the room. #### SFMOMA AR — John Craig Freeman, Will Pappenheimer (2013) - Video: https://www.youtube.com/watch?v=uWHUORR206s - Interaction: Location & City, Portals & Worlds - Platform & tech: Phone, mobile AR app, GPS - Idea: Preview a building that does not exist yet as a playful ghost on its site. - What it is: An 'app-arition' by John Craig Freeman and Will Pappenheimer reimagines SFMOMA's future expansion as animated architecture and art that visitors see on the site with a phone. - Technique: Animated 3D architectural fragments and artworks are geolocated around the museum construction site in a mobile AR app with interactive triggers. - Try it: Pick a construction site or empty lot and build a phone-AR proposal of what should be built there. Twist: the proposal grows a new part each time someone taps it. #### School Shootings eMorial — John Craig Freeman (2013) - Video: https://www.youtube.com/watch?v=lIOIW9_xHi0 - Interaction: Location & City, Information & UI - Platform & tech: Phone, Layar, GPS - Idea: A memorial placed where the policy decisions are made. - What it is: Made with Greg Ulmer for Manifest.AR at the Corcoran, a virtual memorial for victims of US school shootings appears in AR in Washington, D.C., in sight of the Capitol. - Technique: A 3D memorial composed from victim data is geolocated near the National Mall and viewed on smartphones through a mobile AR browser. - Try it: Design an AR memorial for an issue your class cares about and place it facing the decision-making building on campus. Twist: add one element per new news report. #### 14th & AR — John Craig Freeman (2014) - Video: https://www.youtube.com/watch?v=rVzIEmdBdQs - Interaction: Location & City, Perception & Effects, Tangible Objects - Platform & tech: Phone, photogrammetry, Layar - Idea: Treat overlooked street furniture as monuments. - What it is: For Art in Odd Places on 14th Street, New York, ordinary street furniture such as a telephone booth, fire hydrant and newsstand is doubled or transformed in AR. - Technique: Photogrammetric scans of the street objects are geolocated back at or near their original spots in a mobile AR app, likely with animated distortions. - Try it: Scan a boring campus object with photogrammetry and re-place it in AR next to the original, but giant, melting or multiplied. Twist: passers-by vote which version stays. #### Portal to an Alternative Reality — John Craig Freeman (2017) - Video: https://www.youtube.com/watch?v=bZ9fOWJPE2k - Interaction: Portals & Worlds, Location & City - Platform & tech: Phone, Headset, photogrammetry, Unity - Idea: A transit stop becomes a doorway to another city. - What it is: At Boston's Chinatown subway station, a portal opens onto photogrammetric scenes of Wuhan and Saint Petersburg, linking immigrant neighbourhoods to distant cities. - Technique: Photogrammetry scans of real streets are placed behind a portal plane in a mobile AR app so viewers step through a subway entrance into a 3D replica. - Try it: Scan a place that matters to a classmate's family with a phone photogrammetry app and put it behind an AR doorway on campus. Twist: add their recorded voice as the portal's guide. #### Climate Change Migration Stories AR — John Craig Freeman (2023) - Video: https://www.youtube.com/watch?v=wBvqwbcMUJ8 - Interaction: Location & City, Voice & Sound - Platform & tech: Phone, volumetric capture, AR - Idea: Climate migrants appear as life-size AR figures and tell their own stories. - What it is: Life-size volumetric AR portraits of people displaced by climate change appear in public space and tell their own stories. - Technique: Volumetric video captures of real people are anchored life-size in public space, with ground-plane detection placing them at the viewer's scale. - Try it: Record a classmate on a phone telling a story about migrating or moving house, key out the background, and turn it into a life-size AR video billboard at the matching location. Twist: the story starts only when a viewer comes within 2 meters. ### Kiyoshi Kiyokawa *Professor, Nara Institute of Science and Technology (NAIST); head of the Cybernetics and Reality Engineering Laboratory (CARE Lab)* Pioneer of occlusion-capable optical see-through displays (ELMO) and long-time ISMAR leader. His CARE Lab explores 'reality modulation': changing what you eat, see and feel with video see-through headsets. #### Somen-to-Ramen Taste Modulation — Kiyoshi Kiyokawa (2018) - Video: https://www.youtube.com/watch?v=z53xW_BIaZE - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Headset, video see-through, cross-modal perception, 3D overlay - Idea: Change what food looks like and you partly change what it tastes like. - What it is: Diners eating plain somen noodles see them rendered as rich ramen in their headset, and many report that the taste itself shifts toward ramen. - Technique: Overlays a 3D model of a different dish onto the tracked bowl in video see-through AR, exploiting cross-modal interaction between vision and taste. - Try it: Run a blind taste test: students taste plain crackers while looking through a phone AR filter that recolours them as chocolate or cheese, then rate flavour. Twist: add matching sound effects and see if crunch changes the rating. #### Visual Noise Reduction HMD — Kiyoshi Kiyokawa (2019) - Video: https://www.youtube.com/watch?v=vIKYG9NIeIU - Interaction: Perception & Effects, Gaze & Attention, Portals & Worlds - Platform & tech: Headset, video see-through, saliency reduction, image filtering - Idea: Apply noise cancellation to vision instead of sound. - What it is: Like noise-cancelling headphones for the eyes: a video see-through headset greys out and blurs moving distractions around your work area so you can concentrate. - Technique: Keeps a user-specified work area untouched and applies grayscale and blur in real time to the surrounding video to lower its visual saliency. - Try it: Make a phone camera filter that keeps a central rectangle sharp and colourful while blurring and desaturating the rest, then read a page in a busy room with and without it. Twist: auto-detect motion and blur only moving regions. #### ModularHMD — Kiyoshi Kiyokawa (2021) - Video: https://www.youtube.com/watch?v=xMjJayBQeFs - Interaction: Portals & Worlds, Tangible Objects, Shared & Social - Platform & tech: Headset, modular HMD, peripheral display - Idea: Make the headset's walls physically removable so reality can be let in piece by piece. - What it is: A VR headset with three removable side displays: pop a module off to see the real person next to you or your phone, snap it back for full immersion. - Technique: Mounts three removable display modules in the periphery of a commercial HMD cowl, reconfigured by hand to switch between immersion and peripheral real-world awareness. - Try it: Build a cardboard viewer with three flaps that fold open to reveal the real room, then play a phone VR game and note when you open each flap. Twist: let a partner open the flaps for you as a social signal. #### Third-Person Perspective Avatar Embodiment in AR — Kiyoshi Kiyokawa (2022) - Video: https://www.youtube.com/watch?v=RBcTPqbDxhs - Interaction: Hands & Body, Perception & Effects - Platform & tech: Headset, body tracking, avatar, AR HMD - Idea: Wearing a stronger body in AR can make you physically stronger (the Proteus effect). - What it is: Users see themselves from behind as a muscular avatar standing in the real room and moving with their body; after controlling it, their actual physical performance improves. - Technique: Tracks the user's body and drives a full-body avatar rendered from a third-person viewpoint in an AR headset. - Try it: Film a classmate from behind with a phone running a body-tracking AR effect that swaps them for a superhero, show them the live feed while they do squats, and count reps against a normal-mirror round. Twist: try a tired, elderly avatar instead. #### Ukemochi — Kiyoshi Kiyokawa (2022) - Video: https://www.youtube.com/watch?v=dFAB1Uxz3zU - Interaction: Portals & Worlds, Tangible Objects, Hands & Body - Platform & tech: Headset, video see-through, semantic segmentation, social VR - Idea: Let only the food cross the boundary between the real and the virtual world. - What it is: A person in a social VR world can eat real food: the headset cuts out just the plate, food and hands from the passthrough camera and pastes them seamlessly into the virtual scene. - Technique: Segments food, dishes and hands from the video see-through feed with a neural network and composites that region into the rendered VR environment in real time. - Try it: Build a phone AR portal that shows a virtual scene everywhere except a colour-keyed placemat area where the real camera image shows through, then eat a snack through it. Twist: swap roles and show only people's hands from reality. #### AR CO2 Visualization System — Kiyoshi Kiyokawa (2023) - Video: https://www.youtube.com/watch?v=zsOv6N_jAXU - Interaction: Information & UI, Spatial Mapping - Platform & tech: Headset, CO2 sensors, volumetric visualization, AR HMD - Idea: See the air you share: invisible CO2 becomes a coloured cloud in the room. - What it is: Carbon dioxide levels in a room are shown as a 3D colour field in an AR headset, combining fixed sensors with a mobile sensor carried around, so stale corners become visible. - Technique: Interpolates readings from fixed and mobile CO2 sensors into a 3D concentration volume registered to the room and renders it as coloured voxels in AR. - Try it: Carry a phone around the classroom logging noise level or light level at marked spots, then place coloured AR spheres at those spots to reveal the field. Twist: repeat after opening the windows and compare. #### Palm-Through Interaction — Kiyoshi Kiyokawa (2023) - Video: https://www.youtube.com/watch?v=BSMTqQMACiY - Interaction: Hands & Body, Tangible Objects, Information & UI - Platform & tech: Headset, hand tracking, IoT, AR HMD - Idea: Your palm becomes a remote touchscreen aimed at whatever you look at through it. - What it is: Wearing an AR headset, you hold up your palm so a distant lamp or fan appears through it, then tap your own palm like a phone screen to control that device. - Technique: Tracks the hand in the AR headset, casts a ray through the palm to select the IoT device behind it, and maps touches on the palm to that device's UI, giving natural haptic feedback. - Try it: Prototype with a phone: when the camera sees a printed marker on a lamp, show that lamp's controls; students then 'press' the buttons drawn on their own palm while a partner switches the lamp. Twist: use the back of the hand for a second device. ### Ruofei Du *Interactive perception and graphics lead at Google XR* Researcher at Google who leads work on depth-based AR interaction (DepthLab, which shipped in the ARCore Depth API), AI-driven captions and agents for XR, and the open-source XR Blocks framework. Before Google he built mixed-reality social platforms such as Geollery at the University of Maryland. #### HoloFire (57fire) — Ruofei Du (2014) - Video: https://www.youtube.com/watch?v=4cvCBN_ARlY - Interaction: Perception & Effects, Hands & Body, Tangible Objects - Platform & tech: Desktop, thermal camera, Pepper's ghost - Idea: Power a virtual fire with the real heat of your hand. - What it is: A thermal camera watches a touch pad; the warmer your touch, the higher a fire burns inside a Pepper's-ghost holographic display. - Technique: A thermal camera measures the heat footprint left on a surface and maps its intensity to the height of a flame animation reflected in a Pepper's-ghost pyramid. - Try it: Build a phone Pepper's-ghost pyramid from clear plastic and drive a flame animation from microphone loudness (blowing on it). Twist: blowing harder should put the fire out. #### Geollery — Ruofei Du (2019) - Video: https://www.youtube.com/watch?v=Fpfw0COYxoU - Interaction: Location & City, Shared & Social, Information & UI - Platform & tech: Web, Headset, WebGL, OpenStreetMap, street view - Idea: Social media placed back where it happened, in a walkable 3D copy of the city. - What it is: A web-based mirrored world that rebuilds city streets from street-view and map data and fills them with geotagged social media posts that people can walk up to and chat around as avatars. - Technique: A progressive pipeline extrudes buildings from OpenStreetMap footprints, textures them with street-view panoramas and streams geotagged posts as billboards, all rendered with WebGL. - Try it: Collect ten geotagged photos from your campus and place them as floating cards at their real locations in a location-based AR app. Twist: let each card fade the longer ago it was taken. #### DepthLab — Ruofei Du (2020) - Video: https://www.youtube.com/watch?v=knWhKdrAg18 - Interaction: Spatial Mapping, Play, Perception & Effects - Platform & tech: Phone, ARCore Depth API, Unity - Idea: Once a phone knows depth, occlusion, physics and relighting become cheap building blocks for play. - What it is: A library of dozens of phone-AR interactions built only from a depth map: virtual objects hide behind real furniture, splat paint on walls, cast shadows, bounce on the floor and get lit by virtual lights. - Technique: Monocular depth from the ARCore Depth API is used per pixel for occlusion, converted to a mesh for collisions and to surface normals for lighting and splats. - Try it: Use AR occlusion and mesh collisions to make a ball that rolls under your real table and hides behind the chair. Twist: let the ball leave a paint trail on every real surface it touches. #### Visual Captions — Ruofei Du, Alex Olwal (2023) - Video: https://www.youtube.com/watch?v=Dv4lsS-f8Bc - Interaction: Voice & Sound, Information & UI, Shared & Social - Platform & tech: Web, Desktop, large language model, speech recognition - Idea: Live visuals that follow the conversation, like captions but in pictures. - What it is: During a video call or in-person chat, the system listens to the conversation and suggests images, maps or emoji that illustrate what is being said, which speakers can show with one click. - Technique: A fine-tuned large language model predicts from the last spoken sentence what visual would help and what kind, and image search fills it in with the user choosing how proactive it is. - Try it: Build a web page that transcribes speech and, for every noun it hears, shows a matching emoji floating next to the speaker in a phone AR view. Twist: only show images when two people say the same word. #### Human I/O — Ruofei Du (2024) - Video: https://www.youtube.com/watch?v=vHBCLixDEYs - Interaction: Hands & Body, Gaze & Attention, Voice & Sound - Platform & tech: Headset, egocentric vision, LLM, chain-of-thought prompting, audio recognition - Idea: Before deciding how to talk to you, the interface first asks which of your senses and limbs are free. - What it is: An egocentric AI system that detects when your eyes, hands, ears or voice are busy, for example while carrying groceries or sitting in a loud café, so XR interfaces can adapt their input and output to what you can do right now. - Technique: Egocentric video, audio and pose are analyzed by vision and audio models and reasoned over by an LLM with chain-of-thought prompting to estimate how available each human channel is. - Try it: Film five one-minute first-person clips (cooking, carrying boxes, a noisy street, a meeting, riding as a passenger), send key frames and a transcript to a multimodal LLM asking whether the user's hands, eyes, ears and voice are available, and design how your AR app should switch between voice, gaze and touch in each case. Twist: make the app say aloud why it just switched modes. #### Sensible Agent — Ruofei Du (2025) - Video: https://www.youtube.com/watch?v=iYayksCTYGI - Interaction: Gaze & Attention, Voice & Sound, Information & UI - Platform & tech: Headset, Wearable, multimodal LLM, head gestures, WebXR - Idea: A good assistant reads the social situation before it speaks. - What it is: A proactive AR assistant that decides not only what help to offer but how, choosing between a glanceable icon, audio or a question you answer with a nod or a gaze, depending on whether your hands are busy or you are in a conversation. - Technique: A multimodal model reads egocentric video and audio to infer context, then a policy selects both the assistance and the least disruptive input and output modality, such as head gestures or gaze dwell. - Try it: Design three delivery modes (icon, whisper, nod question) for one AR reminder, and act out when each is appropriate in class. Twist: the assistant must stay silent if two people are talking. #### Thing2Reality — Ruofei Du (2025) - Video: https://www.youtube.com/watch?v=ZClLOeu6y5A - Interaction: Shared & Social, Tangible Objects, Drawing & Making - Platform & tech: Headset, multiview diffusion, 3D Gaussian splatting, XR communication - Idea: Anything you can show on camera can instantly become a shared 3D object. - What it is: During an XR call, grab a 2D image or a physical object from the video feed and it materializes as a 3D object between the participants, generated as multiview renderings or 3D Gaussians that everyone can move and point at. - Technique: Segmented 2D content is lifted to 3D with image-to-multiview diffusion and 3D Gaussian reconstruction, then shared in the immersive call space. - Try it: Photograph a small object, remove the background, send it to an image-to-3D service (TripoSR, Meshy or Luma) and drop the result into a shared AR scene that two phones can see, for example with Lens Studio Connected Lenses. Twist: generate the 3D object from a child's drawing instead of a photo. #### XR Blocks and Vibe Coding XR — Ruofei Du (2025) - Video: https://www.youtube.com/watch?v=nknCzIxHHzw - Interaction: Hands & Body, Spatial Mapping, Information & UI - Platform & tech: Web, Headset, WebXR, three.js, Gemini - Idea: Make XR prototyping as quick as describing the idea in a sentence. - What it is: An open-source WebXR framework that packages hands, depth, physics and AI agents into high-level blocks, so that a single text prompt to Gemini can generate a working interactive XR prototype. - Technique: The framework abstracts sensor input and scene understanding into a small script API, which an LLM can target reliably when it writes the app code. - Try it: In pairs, write a one-sentence XR idea, have an LLM generate a WebXR scene from it and iterate three times in class. Twist: swap prompts with another pair and remix. #### AgentHands — Ruofei Du (2026) - Video: https://www.youtube.com/watch?v=O4Gma0XqRgM - Interaction: Hands & Body, Voice & Sound, Spatial Mapping - Platform & tech: Headset, LLM, hand animation, XR - Idea: Give a talking AI hands, so it can point instead of describe. - What it is: An AI agent in XR gets a pair of virtual hands that point at, trace and hold up real and virtual objects while it talks, so its explanations are grounded in the space around you. - Technique: An LLM plans speech together with gesture tokens that reference scene objects, and a motion generator turns them into hand animations aligned to object positions. - Try it: In phone AR, add a floating virtual hand to a chatbot that points to a named object in the room when you ask 'where is my ...?'. Twist: let the hand act out the size of an object. ### Stefanie Zollmann *Professor of Computer Science, University of Otago; co-leads the Visual Computing Otago group* Began with X-ray visualization and drone flight planning in AR at TU Graz, then built AR for stadium spectators, historical rephotography and sports coaching in Otago. #### FlyAR — Stefanie Zollmann (2014) - Video: https://www.youtube.com/watch?v=C2Er9RUgh8E - Interaction: Location & City, Information & UI, Spatial Mapping - Platform & tech: Phone, drone, GPS, tablet AR - Idea: Plan where a drone will fly by drawing in the actual sky, not on a 2D map. - What it is: A drone operator plans and watches a flight path on a tablet held up to the sky: planned waypoints and the live drone position appear directly over the real landscape and buildings. - Technique: Registers the planned flight path and live telemetry of a micro aerial vehicle to the environment using GPS/INS plus a 3D model, rendered with depth cues on a handheld AR device. - Try it: Use a phone AR app to place a chain of floating waypoints around the schoolyard, then have a classmate walk the path as the 'drone' while you check for collisions. Twist: add a no-fly zone that turns red when crossed. #### Indirect AR Browser for GIS Data — Stefanie Zollmann (2018) - Video: https://www.youtube.com/watch?v=0WiskPTmUQg - Interaction: Location & City, Information & UI - Platform & tech: Phone, indirect AR, street-level imagery, GIS - Idea: When live tracking is unreliable, augment a stored photo of the scene instead of the camera feed. - What it is: Pointing a phone at a street shows a pre-captured street-level panorama aligned to the view, with buildings and landmarks labelled perfectly because the labels live on the stored image. - Technique: Replaces the live camera with crowd-sourced street-level imagery oriented by the phone's sensors and overlays geospatial data registered to that imagery. - Try it: Take a 360 photo of the school entrance, add labels on it in an editor, then show it on a phone that rotates with your gaze while standing at the same spot. Twist: switch between today's photo and an old photo of the same place. #### ARSpectator: MR Prototypes for Augmented Sports — Stefanie Zollmann (2019) - Video: https://www.youtube.com/watch?v=LOskhPgXV2c - Interaction: Information & UI, Location & City, Performance - Platform & tech: Phone, mobile AR, localization, sports visualization - Idea: Bring broadcast-style graphics to the fans actually sitting in the stadium. - What it is: Spectators in a rugby stadium hold up a phone and see player names, statistics and game information placed on the real pitch, prototyped first in the lab with a scale model. - Technique: Localizes the phone against a prebuilt stadium model and overlays situated infographics, with lab prototypes simulating the field to test visualisations before live games. - Try it: Print a mini football pitch, track it with a phone AR image target, and overlay live 'stats' for toy players moved by classmates. Twist: show different stats to home and away fans. #### Is that me? AR Mirror — Stefanie Zollmann (2019) - Video: https://www.youtube.com/watch?v=Y65DPFWu5y8 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, depth camera, 3D capture, AR mirror - Idea: An AR mirror can test how far your reflection can change before it stops being you. - What it is: People look into an AR mirror that shows a captured 3D version of their own body moving in real time, and researchers test whether they feel that reconstructed body is really theirs. - Technique: Captures the user with depth cameras into a live 3D representation and renders it in an AR mirror setup to measure ownership, agency and self-location. - Try it: Use a phone's front camera with a body-tracking effect as a mirror, gradually delaying or distorting the image, and ask classmates when it stops feeling like them. Twist: swap two students' bodies in the mirror. #### XRSpectator — Stefanie Zollmann (2021) - Video: https://www.youtube.com/watch?v=TR27sWmCMwk - Interaction: Information & UI, Performance, Portals & Worlds - Platform & tech: Headset, 360 video, indirect AR, VR HMD - Idea: Recreate the in-stadium AR experience at home by augmenting a 360 video instead of reality. - What it is: Fans who cannot attend a match put on a VR headset and sit inside a 360 recording of the stadium, with the same AR stats and replays floating over the field as if they were there. - Technique: Wraps a 360 video of the live stadium around the user and registers the ARSpectator overlays to it, turning on-site AR into indirect AR on a VR HMD. - Try it: Film a school match with a 360 camera or phone panorama, then add floating labels to the panorama and watch it in a cardboard viewer. Twist: let viewers switch seats between two recording positions. #### ARephotography — Stefanie Zollmann (2023) - Video: https://www.youtube.com/watch?v=nGZBKTKC1us - Interaction: Location & City, Portals & Worlds, Perception & Effects - Platform & tech: Phone, machine learning, 3D reconstruction, rephotography - Idea: Turn an old photograph into a 3D time window placed on the same spot today. - What it is: Standing in front of a campus building, a phone shows how it looked in a historical photograph, with the old facade rebuilt as a textured 3D model sitting exactly over today's walls. - Technique: Uses a machine-learning pipeline to turn a single historical photo into a textured 3D model and registers it over the current building in mobile AR. - Try it: Find an old photo of your school, stand where it was taken, and overlay it semi-transparently on the live phone camera to align past and present. Twist: add a slider that wipes between the eras. #### Hybrid XRSpectator — Stefanie Zollmann (2024) - Video: https://www.youtube.com/watch?v=N3THcfiXN_w - Interaction: Information & UI, Performance, Portals & Worlds - Platform & tech: Headset, Phone, tabletop AR, 360 video, indirect AR - Idea: Zoom between a toy-sized stadium on your table and being inside the crowd. - What it is: At home, a fan watches a miniature stadium on the coffee table in AR, then dives into a surrounding 360 view of the real stadium with integrated stats and replays. - Technique: Combines a tabletop AR stadium model with a transition into an indirect AR mode built on 360 video of the match. - Try it: Build a tabletop AR mini-arena from a paper model and add a button that switches to a 360 photo taken from inside the real venue. Twist: let a tap on a tiny player open that player's first-person view. #### Train Me — Stefanie Zollmann (2024) - Video: https://www.youtube.com/watch?v=xWKT-_ijJj0 - Interaction: Hands & Body, Information & UI, Performance - Platform & tech: Headset, Phone, motion capture, 3D reconstruction, AR HMD - Idea: Record a coach once and replay them life-size wherever the athlete trains. - What it is: A coach records a movement with a phone or egocentric camera, and the trainee later replays it as a life-size 3D figure in an AR headset to copy it on the spot. - Technique: Captures sports movements with mobile or egocentric cameras, reconstructs them in 3D and visualises the instruction in an AR head-mounted display. - Try it: Film a classmate's basketball free throw with a phone, place the video as a life-size AR billboard on the court, and have others mimic it side by side. Twist: slow the replay to half speed. ### Universal Everything (Matt Pyke) *Digital art and design studio founded by Matt Pyke* Matt Pyke's studio makes digital lifeforms, crowds and walking figures for galleries, brands and architecture, increasingly as camera-driven mirrors and AR apps that let audiences see themselves transformed. #### 1000 Hands — Universal Everything (Matt Pyke) (2013) - Video: https://www.youtube.com/watch?v=yECE20Cq0mE - Interaction: Drawing & Making, Shared & Social, Projection - Platform & tech: Projection, iPad, real-time 3D - Idea: Drawings by a thousand visitors grow into 3D forms and join one shared projected landscape. - What it is: Visitors contribute line drawings through a mobile app; each drawing is grown into a 3D form that joins a collective, evolving projected landscape in the gallery. - Technique: Line drawings from a mobile app are sent to a server, converted into procedurally extruded or grown 3D geometry, and added to a shared real-time scene that is projected in the gallery. - Try it: Collect the whole class's hand-drawn lines with a web form, extrude each line in three.js into a slowly growing 3D form, and place them all in one projected landscape. Twist: let each form's growth direction depend on the mood option its drawer picked at the time. #### Future You — Universal Everything (Matt Pyke) (2019) - Video: https://vimeo.com/718890921 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Projection, depth camera, real-time generative - Idea: A motion-capture mirror shows 'future you', and the more you move, the stranger it gets. - What it is: A motion-capture mirror that turns each viewer into a synthetic robotic figure; the harder you move, the stranger its 47,000 possible responses become. - Technique: A depth camera tracks the viewer's skeleton and a real-time engine retargets it to a generative robotic figure whose form parameters change with movement speed and energy. - Try it: Use MediaPipe Pose to drive a 'robot double' built from geometric shapes, whose form parameters (size, count, color) grow more outlandish the more intensely you move. Twist: give the double a one-second 'memory' that gradually mixes in your past movements. #### Super You (AR app) — Universal Everything (Matt Pyke) (2020) - Video: https://www.youtube.com/watch?v=-zPQczdMCVc - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, ARKit body tracking - Idea: Point your phone at yourself and become a flowing digital being. - What it is: An iPhone AR app that tracks your body and transforms you into a flowing digital being in real time. - Technique: ARKit body tracking (or person segmentation) provides a skeleton and silhouette each frame, which drive particle or ribbon systems that replace the user's body in the camera view. - Try it: Use Lens Studio or ARKit body tracking to replace your whole body with particles or ribbons that flow with your joints. Twist: assign each body part a different material, such as smoke, water or text, to make a digital body that expresses your personality. #### HyperSpace AR — Universal Everything (Matt Pyke) (2021) - Video: https://www.youtube.com/watch?v=1KuwkTdab0o - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, LiDAR, ARKit - Idea: Scan the room with LiDAR and let generative forms grow along real surfaces. - What it is: A LiDAR-based AR app for iPhone 12 / iPad Pro that fills your room with generative forms that respond to the scanned walls and furniture. - Technique: LiDAR scene reconstruction yields a room mesh, and generative geometry is spawned and grown along the mesh normals with occlusion so forms appear to emerge from walls and furniture. - Try it: Use AR Foundation meshing (or plane detection) to grow vines or crystals along walls and tabletops with occlusion turned on. Twist: growth happens only on surfaces you stare at for a long time. #### Infinity — Universal Everything (Matt Pyke) (2021) - Video: https://vimeo.com/523660693 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Projection, real-time generative, camera tracking - Idea: The arrival of viewers changes an endless crowd of walkers. - What it is: A generative installation of endlessly evolving digital beings; the interactive version lets visitors' presence and movement influence the figures on screen. - Technique: A crowd of procedurally animated walkers is rendered in real time, and camera-based presence tracking (likely blob or pose detection) modulates their speed, density or appearance. - Try it: Make a line of endlessly walking abstract figures in p5.js or TouchDesigner, and use a camera to detect how many viewers there are and where they stand to change the rhythm of the line. Twist: after viewers leave, the crowd keeps some trace of them. #### Chameleon — Universal Everything (Matt Pyke) (2024) - Video: https://www.youtube.com/watch?v=EkGLZ_Ck_-w - Interaction: Hands & Body, Perception & Effects, Location & City - Platform & tech: Projection, computer vision, real-time generative - Idea: A digital chameleon creature on the street that imitates you. - What it is: A camera-driven digital being that mimics passers-by's movement and appearance, shown on screens in East London streets, malls and later Venice and Beijing. - Technique: Camera-based computer vision (pose estimation plus colour sampling) lets a generative creature copy the passer-by's movement and clothing colours on a public screen. - Try it: Use MediaPipe Pose and camera color sampling to make an abstract creature that mimics a viewer's pose and takes on the colors of their clothes. Twist: let the creature imitate only the stillest person in the frame. #### Hydrogen Wave — Universal Everything (Matt Pyke) (2024) - Video: https://www.youtube.com/watch?v=M2SOCYnWIiU - Interaction: Hands & Body, Projection - Platform & tech: Projection, depth camera, real-time particles - Idea: Push waves of 'hydrogen' particles with your body. - What it is: An interactive space at Hyundai Motorstudio Senayan Park, Jakarta, where visitors' movements send waves through a large projected field of animated particles themed on hydrogen energy. - Technique: Depth cameras track visitors' positions and velocities, which inject forces into a GPU particle simulation projected across the floor and walls. - Try it: Drive a projected particle field in TouchDesigner with a Kinect or camera optical flow, so people's movement pushes out ripples. Twist: make the particles stand for an invisible substance you care about (air pollution, Wi-Fi signals). #### Look Up - Augmented Reality Architecture — Universal Everything (Matt Pyke) (2024) - Video: https://www.youtube.com/watch?v=e5ELC-FcJXs - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, AR - Idea: AR sculptures sprout from building facades and invite people to look up at the city. - What it is: AR experiments that attach digital sculptures and animated beings to building facades, inviting people to look up at architecture. - Technique: Likely geospatial or VPS anchoring (or image tracking of the facade) places animated sculptures on building faces, with scale set so they read correctly from street level. - Try it: Pick a campus building and place a moving sculpture on its facade with 8th Wall image tracking or ARCore Geospatial. Twist: make the sculpture appear only when the viewing angle tilts up past 45 degrees, forcing viewers to actually look up. #### Prototype: Someone - a random mirror — Universal Everything (Matt Pyke) (2024) - Video: https://www.youtube.com/watch?v=OS3rX7QU468 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, pose tracking, generative - Idea: The mirror always shows 'someone else'. - What it is: A mirror prototype that reflects the viewer's pose but replaces them with a randomly generated other person. - Technique: Pose estimation extracts the viewer's skeleton, which drives a randomly generated human avatar (or a generative image model conditioned on pose) shown as the reflection. - Try it: Use MediaPipe Pose to drive a 3D human model picked at random from an asset library, and present it as a 'mirror'. Twist: switch to a new person each time someone enters the frame, and make that person look the exact opposite of the viewer. #### Ultrasound VR — Universal Everything (Matt Pyke) (2026) - Video: https://www.youtube.com/watch?v=95_WMsapSuA - Interaction: Drawing & Making, Voice & Sound, Spatial Mapping - Platform & tech: Headset, Apple Vision Pro, Spatial Audio, LiDAR, hand tracking - Idea: Draw in the air to compose music: every stroke becomes a sound sculpture that plays from where you drew it. - What it is: An interactive music ecosystem for Apple Vision Pro: by drawing in the space around you, you place audio-reactive sound sculptures and layer them into compositions, with Spatial Audio putting each sound where its sculpture stands. - Technique: Hand tracking captures drawing gestures that spawn generative, audio-reactive meshes; each one is bound to a looping sound layer rendered with Spatial Audio at its 3D position, and LiDAR scene understanding lets the sculptures sit on real surfaces. - Try it: Build a WebXR or AR Foundation scene where each tap or drawn stroke places a shape that loops one sound sample from its position (use positional audio), so walking around the room remixes the piece. Twist: make each shape's size pulse with its own sound's volume. #### Ultrasound VR — virtual embellishment of surfaces — Universal Everything (Matt Pyke) (2026) - Video: https://vimeo.com/1200725642 - Interaction: Spatial Mapping, Voice & Sound, Perception & Effects - Platform & tech: Headset, Apple Vision Pro, LiDAR, Spatial Audio, audio-reactive - Idea: Let music grow on the walls: sound becomes living ornament on real surfaces. - What it is: A test from Ultrasound VR in which audio-reactive sound sculptures attach to real walls and floors, embellishing the room's surfaces and pulsing with the music. - Technique: LiDAR scene reconstruction on Apple Vision Pro gives wall and floor planes; generative meshes are anchored to them and deformed by the audio spectrum in real time. - Try it: Detect vertical and horizontal planes with AR Foundation or WebXR, grow a pattern on each surface, and drive its scale or colour with an FFT of the microphone or a music track. Twist: give walls and floors different frequency bands, so the room becomes an equalizer. ### ZERO10 *AR fashion-tech studio: digital fashion app, AR try-on, AR Mirrors and AR storefronts* Fashion-tech company led by CEO George Yashin. It launched an iPhone app for wearing digital-only garments in AR in 2021, then turned the same real-time body tracking and cloth simulation into AR Mirrors and AR storefronts for Coach, Tommy Hilfiger, UGG, JD Sports and Disney. #### ZERO10 app: digital fashion AR try-on — ZERO10 (2021) - Video: https://x.com/zero10_ar/status/1451190331257487375 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, iOS, body tracking, cloth simulation - Idea: Clothes that exist only as software, worn on your real body through the camera. - What it is: The launch film of the ZERO10 iPhone app: people on the street wear digital-only garments that follow their body in real time through the phone camera, ready to be recorded and shared. - Technique: Full-body tracking and segmentation estimate the wearer's pose and silhouette each frame, and a real-time cloth simulation drapes the 3D garment over that body with occlusion by arms and hair (likely). - Try it: Use Lens Studio or Effect House body tracking to attach a simple 3D garment (a cape or oversized collar) to a person's upper body, then film a 15-second street 'outfit post'. Twist: the garment reacts to walking speed, getting bigger the faster you move. #### Barragán wearable NFT drop with visual effects — ZERO10 (2022) - Video: https://x.com/zero10_ar/status/1487497836031791105 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, iOS, body tracking, shaders, NFT - Idea: A garment can do what fabric cannot: burn, flow or turn transparent while you wear it. - What it is: A drop with the Mexican label Barragán in which each digital garment carries its own effect, such as a shirt that burns up, a dress with an animated oil-slick print and an X-rayed pocket vest, all worn in AR through the app. - Technique: Garments are authored as 3D assets with animated shaders and particle effects, then attached to the tracked body in the try-on engine so the effect plays on the wearer in real time. - Try it: Design one AR garment whose material changes over time (burning, melting, freezing) using a shader graph and body tracking in Lens Studio or Effect House. Twist: the effect triggers only when the wearer strikes a specific pose. #### Edward Crutchley AR showpiece — ZERO10 (2022) - Video: https://x.com/zero10_ar/status/1505944348600504326 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, body tracking, cloth simulation, digital fashion - Idea: Couture that ignores gravity and fabric cost, worn in an everyday room. - What it is: British designer Edward Crutchley's first augmented reality showpiece: a sculptural couture look with a shimmering, wing-like bodice that is worn on a real model inside an ordinary apartment. - Technique: A high-poly couture garment is simulated and rendered on the tracked body in real time, with lighting estimated from the camera image so the piece sits in the room (likely). - Try it: Sketch an 'impossible' accessory (floating wings, orbiting jewels, a halo) and build it as a body-tracked AR effect in Lens Studio or AR Foundation, then shoot it in a plain room. Twist: the accessory must interact with one real object in the room. #### ZERO10 x Crosby Studios store with digital-only clothing — ZERO10 (2022) - Video: https://x.com/zero10_ar/status/1580208212518662144 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, AR Mirror, body tracking, cloth simulation - Idea: A real shop for unreal clothes: the fitting room becomes a mirror screen. - What it is: A physical store in New York designed with Crosby Studios that sells clothes that exist only digitally: a visitor stands in front of a tall screen in a green-tiled room and sees herself wearing the digital jacket. - Technique: A camera above a portrait display tracks the shopper's body and the try-on engine renders the selected digital garment on the live video, turning the screen into an AR mirror. - Try it: Turn a laptop and an external monitor rotated to portrait into an AR mirror with MediaPipe Pose and three.js that dresses visitors in one digital garment. Twist: the mirror only shows the garment when two people stand in front of it together. #### Coach Tabby AR Storefront and AR Mirror — ZERO10 (2023) - Video: https://x.com/zero10_ar/status/1655677506454982656 - Interaction: Hands & Body, Location & City, Perception & Effects - Platform & tech: Desktop, AR Mirror, AR Storefront, body tracking - Idea: The shop window watches the street and dresses whoever stops. - What it is: For Coach's Tabby bag campaign, the SoHo store window at 143 Prince Street in New York became an AR Storefront: passers-by on the sidewalk see themselves on a pink screen in the window holding and wearing digital Tabby looks. - Technique: A camera behind the glass tracks people on the sidewalk and renders digital bags and outfits on them in real time on a window-facing display; a second AR Mirror inside continues the experience. - Try it: Point a webcam and a monitor out of a classroom window or doorway and build a 'street mirror' that adds one playful AR accessory to anyone who stops for three seconds. Twist: the accessory grows the longer they stay. #### Generative AI Virtual Try-on research — ZERO10 (2023) - Video: https://x.com/zero10_ar/status/1729535657444286935 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, Web, generative AI, diffusion model, virtual try-on - Idea: Swap real-time 3D for generative images: the model imagines you in the outfit. - What it is: A research demo in which a single photo of a person is used to generate realistic images of them wearing different garments, aimed at seeing how clothes look on you before buying online. - Technique: A diffusion-based image model is conditioned on the person's photo, pose and the garment image to synthesize a new photo of the person wearing it (likely). - Try it: Compare two try-on methods for the same garment: a body-tracked 3D AR filter and an open-source image try-on model, then have classmates rate realism and fun. Twist: feed the AI version an 'impossible' garment and see what it invents. #### Giant digital Tabby in New York — ZERO10 (2023) - Video: https://x.com/zero10_ar/status/1657058686798315521 - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, world tracking, 3D compositing - Idea: Scale is the joke: a handbag the size of a building. - What it is: A giant pink Coach Tabby bag hangs between New York buildings in AR, promoting a giveaway in which winners could own the AR bag in the ZERO10 app. - Technique: A 3D bag model is anchored in world space and rendered at architectural scale over camera footage of the street, most likely as a camera-tracked composite for the campaign video. - Try it: Use world tracking in AR Foundation or 8th Wall to place an everyday object (a mug, a sneaker) at building scale outside, then record a short vertical video. Twist: the giant object must look like it is interacting with the architecture, for example hanging from a ledge. #### JD Sports AR Mirror in a sports arena — ZERO10 (2023) - Video: https://x.com/zero10_ar/status/1688549906351271936 - Interaction: Hands & Body, Information & UI - Platform & tech: Desktop, AR Mirror, body tracking, e-commerce - Idea: Try-on and checkout collapse into one mirror at the stadium. - What it is: For a JD Sports campaign developed by the agency Rumfoords, ZERO10's AR Mirror came to a sports arena: fans try on new sportswear virtually and can start a purchase directly from the mirror screen. - Technique: The AR Mirror combines body tracking and cloth simulation with an on-screen commerce flow, so a selected digital garment links to the matching product for purchase. - Try it: Build a body-tracked AR mirror that dresses players in a team jersey and shows a QR code linking to a (mock) product page when they hold a pose for two seconds. Twist: the jersey number updates to match how many jumps the person just did. #### Tommy Hilfiger AR Mirror — ZERO10 (2023) - Video: https://x.com/zero10_ar/status/1639326863662149634 - Interaction: Hands & Body, Information & UI - Platform & tech: Desktop, AR Mirror, body tracking, cloth simulation - Idea: The shop mirror that lets you wear the whole collection in seconds. - What it is: An AR Mirror installed in Tommy Hilfiger's London store: shoppers walk up to a framed screen and see themselves wearing the digital version of the new collection, with extra visual effects. - Technique: A camera-and-screen kiosk runs real-time pose tracking, segmentation and cloth simulation to overlay digital twins of physical garments on the shopper, with a touch interface to switch looks. - Try it: Build a browser AR mirror with MediaPipe Pose that swaps between three outfits by a raised-hand gesture instead of touch. Twist: each outfit changes the background of the room too. #### Future outfits for Zendaya's Mugler robot suit — ZERO10 (2024) - Video: https://x.com/zero10_ar/status/1759918269698596941 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, body tracking, video AR, digital fashion - Idea: Remix a viral fashion moment by re-dressing the same footage with AR. - What it is: Riffing on Zendaya's red-carpet entrance in Mugler's 1995 robot suit, ZERO10 dressed the same footage in several of its own retro-futuristic digital designs and asked viewers to pick a favourite. - Technique: Body tracking runs on existing video rather than a live camera, so digital garments are fitted frame by frame onto the celebrity footage and cut into a comparison edit. - Try it: Pick a short public-domain or self-shot runway clip and run a body-tracking AR effect on it in Effect House or Lens Studio (video input mode) to produce three alternative outfits. Twist: each outfit must represent a different decade. #### Magic Mirrors for cinemas and theme parks — ZERO10 (2024) - Video: https://x.com/zero10_ar/status/1792550092722806832 - Interaction: Hands & Body, Portals & Worlds, Play - Platform & tech: Desktop, AR Mirror, body tracking, segmentation - Idea: Step into the poster: the mirror swaps you into the movie's world. - What it is: ZERO10 Magic Mirrors in entertainment venues turn visitors into their favourite characters, for example standing in an underwater world as an Aquaman-style superhero, and invite them to share the result. - Technique: Full-body tracking drives a character costume while background segmentation replaces the venue with a themed environment, all rendered live on a large portrait screen. - Try it: Build a webcam 'poster mirror' with MediaPipe Selfie Segmentation and Pose that replaces the background with a scene of your choice and adds one character accessory. Twist: the scene reacts when the visitor raises both arms. #### What others see / What we see — ZERO10 (2024) - Video: https://x.com/zero10_ar/status/1767216132061331485 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, body tracking, video AR - Idea: AR as a second way of seeing the same moment: what others see vs what we see. - What it is: A split-screen joke after the 2024 Oscars: the top shows what everyone saw on stage, and the bottom shows what ZERO10 'sees', with the presenter dressed in AR as a boxy car-robot costume. - Technique: The same broadcast clip is tracked for body pose and a rigid 3D costume is attached to the torso and limbs, then stacked against the original for comparison. - Try it: Record a 10-second clip of a classmate, then build a split-screen video where the lower half shows them in a body-tracked AR costume that changes their role (robot, knight, mascot). Twist: the costume must reveal something the top half hides. ### Antony Gormley *Sculptor* British sculptor who casts his own body in iron and steel and places the figures on beaches, rooftops, mountains and salt lakes, from Another Place to the Angel of the North. #### Field for the British Isles — Antony Gormley (1993) - Video: https://www.youtube.com/watch?v=ZP-2wUaXnjU - Interaction: Gaze & Attention, Shared & Social, Perception & Effects - Platform & tech: Desktop, hand-formed terracotta, community workshop - Idea: Tens of thousands of eyes looking back at you reverse who is watching whom, a strong idea for AR crowds that react to gaze. - What it is: About 40,000 small clay figures, each shaped by hand by families in St Helens and pressed with two eye holes, fill a room from wall to wall and all look up at the visitor in the doorway. - Technique: Community volunteers each formed figures from a fist-sized lump of clay with two pencil-made eyes; the only rule was that they face the viewer. - Try it: Ask every student to model one tiny creature in a 3D app and combine them into a crowd of hundreds in AR, all rotating to face the phone. Twist: the crowd only looks at you when you stand still. #### Another Place — Antony Gormley (1997) - Video: https://www.youtube.com/watch?v=MN1M3p14c8M - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, cast iron, body casts, tidal beach - Idea: Many identical figures facing one horizon turn a whole beach into a stage: the same trick an AR scene can use by spreading copies of one anchor across a site. - What it is: One hundred cast-iron figures of the artist's body stand spread over three kilometres of Crosby Beach near Liverpool, all staring out to sea; the tide slowly covers and reveals them. - Technique: Life-size cast-iron casts of the artist's body are fixed on foundations at different distances from the shore, so the tide and the viewer's distance vary how much of each is visible. - Try it: Place 20 copies of one scanned human figure across a school field in WebXR or AR Foundation, all facing the same compass direction, and walk among them. Twist: add a rising virtual water plane that hides the figures one by one. #### Angel of the North — Antony Gormley (1998) - Video: https://www.youtube.com/watch?v=i79ZI_KMzjw - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, weathering steel, hilltop landmark - Idea: A single giant figure on a hill becomes a landmark people navigate by, a reminder that the scale and silhouette of an AR landmark matter more than detail. - What it is: A 20-metre steel angel with 54-metre wings stands on a former colliery hill beside the A1 motorway at Gateshead, seen by about 90,000 drivers a day. - Technique: The body-form is built from ribbed weathering steel sections anchored 20 metres into the hill so it can withstand winds of over 160 km/h. - Try it: Place a 20-metre virtual figure on the tallest point visible from your campus in AR and test from how far away its silhouette is still readable. Twist: let its wings open only when the viewer is moving fast (walking vs. driving speed). #### Inside Australia — Antony Gormley (2003) - Video: https://www.youtube.com/watch?v=VeIDvX23HYs - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, 3D body scans, iron-molybdenum alloy, salt lake - Idea: Scanning a whole community and placing their slimmed bodies in the desert makes a town's population visible as a landscape, something AR could do with volunteers' scans. - What it is: Fifty-one thin black figures, based on body scans of the residents of the tiny town of Menzies, stand scattered over ten square kilometres of the dry salt lake Lake Ballard in Western Australia. - Technique: Laser scans of each resident were digitally reduced to a thin core, then cast in an alloy that the salt and sun slowly rust. - Try it: Scan five classmates with Polycam, scale each model's thickness down to 30% in Blender, and place them far apart on a field in AR. Twist: tag each figure with its owner's spoken sentence that plays when you reach it. #### Time Horizon — Antony Gormley (2006) - Video: https://www.youtube.com/watch?v=3ncox_YTRyU - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, cast iron, surveyed altitude, parkland - Idea: Keeping every figure on one absolute height makes the uneven ground visible, a way to use AR objects as measuring devices for terrain. - What it is: One hundred iron body casts are placed across a park (first in Calabria, later at Houghton Hall in Norfolk), all at the same height above sea level, so some are buried to the neck and others stand on columns. - Technique: Figures are set on plinths or sunk into pits so that all their heads share one surveyed altitude regardless of the terrain beneath. - Try it: In AR Foundation, place ten figures on a sloping lawn and lock all of their heads to one world-space height, filling the gap below with a column or hiding them in the ground. Twist: let viewers drag the shared height up and down live. #### Event Horizon — Antony Gormley (2007) - Video: https://www.youtube.com/watch?v=d6Y9-tgSUsQ - Interaction: Location & City, Gaze & Attention, Perception & Effects - Platform & tech: Desktop, cast iron and fibreglass figures, rooftops, city skyline - Idea: A figure on a skyline makes people look up and scan the city, showing that one well-placed AR character can redirect a crowd's attention. - What it is: Thirty-one body-form figures are placed on rooftops and pavements across a city (London, New York, São Paulo, Hong Kong), so people suddenly notice a lone man standing on the edge of a building. - Technique: Identical figures are mounted on rooftops at varied distances from a central viewpoint so that the eye links them into a constellation across the skyline. - Try it: Anchor five life-size figures with geospatial AR (ARCore Geospatial API or Niantic Lightship) on rooftops visible from one plaza, and ask classmates to find them all. Twist: make each figure turn to face the nearest viewer. #### 6 Times — Antony Gormley (2010) - Video: https://www.youtube.com/watch?v=9sNoyjInDJs - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, cast iron, river course, urban walk - Idea: Placing a figure at intervals along a river turns a walk into a narrative sequence, a structure location-based AR tours can copy. - What it is: Six life-size iron figures stand along the Water of Leith in Edinburgh, from the Gallery of Modern Art to the sea, four of them in the river itself, forming a walk through the city. - Technique: Identical figures are fixed on concrete footings in the riverbed and on the banks at irregular intervals along a four-mile route. - Try it: Design a six-stop AR walk along a street or stream with 8th Wall or Lens Studio, placing one figure per stop, each a little closer to water. Twist: each figure points toward the next one so the walk needs no map. #### Exposure — Antony Gormley (2010) - Video: https://www.youtube.com/watch?v=I3MG5UIi0hs - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, galvanised steel lattice, dyke, polder landscape - Idea: The figure reads as a body from far away and as pure structure up close, a lesson in designing AR objects for two viewing distances. - What it is: A 26-metre crouching figure made of an open steel lattice sits on a dyke at Lelystad in Flevoland, on land reclaimed from the sea; from close up it dissolves into a web of beams. - Technique: A digitised body pose was converted into a network of 14,000 steel members of varying length, so the lattice resolves into a figure only at a distance. - Try it: Convert a scanned body into a point-and-line wireframe (Blender geometry nodes) and place it outdoors in AR at 20 metres tall. Twist: fade the lines out as the viewer walks closer so the body disappears when you reach it. #### Horizon Field — Antony Gormley (2010) - Video: https://www.youtube.com/watch?v=RT7_-XyDHEA - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, cast iron, altitude survey, alpine landscape - Idea: All figures sit on one invisible contour line, so the landscape itself draws a horizontal line through the mountains: an AR layer can reveal a hidden datum in the same way. - What it is: One hundred cast-iron figures stand across 150 square kilometres of the Vorarlberg Alps, each at exactly 2,039 metres above sea level, so hikers meet them on slopes, ridges and near huts. - Technique: Each figure was placed by survey at the same altitude, turning a contour line from the map into a visible, walkable line of bodies. - Try it: Use a phone's barometer or GPS altitude in a WebXR page to place a virtual marker whenever the user is at one chosen height on a hill or staircase. Twist: have the whole class walk and leave markers, then view the shared contour line together. #### LAND — Antony Gormley (2015) - Video: https://www.youtube.com/watch?v=AEJpwbKRVBY - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, cast iron, heritage sites, coastal placement - Idea: Spreading five figures across a country connects distant sites into one work, a model for multi-location AR campaigns. - What it is: For the 50th anniversary of the Landmark Trust, five iron figures in different poses were placed at five historic sites around the UK coast and waterways, such as a Martello tower and a sea-washed bay. - Technique: Each cast-iron body was posed in response to its site (standing, crouching, looking) and installed for one year at a place where visitors could stay overnight. - Try it: Create five AR figures, each tied by geolocation to a different building on campus, with a pose that answers that building. Twist: the fifth figure only appears once a user has visited the other four. #### Another Time (Margate) — Antony Gormley (2017) - Video: https://www.youtube.com/watch?v=3fUZA7ylRJk - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, cast iron, tidal site - Idea: One figure in the sea becomes a clock for the tide, showing how an AR object can visualise a slow natural rhythm by how much of it is visible. - What it is: A single iron figure stands on the tidal rocks in front of Turner Contemporary in Margate, submerged at high tide and fully revealed at low tide. - Technique: A solid iron cast is bolted to the rock foreshore at a height chosen so that the tide range crosses the whole body. - Try it: Fetch live tide data from a public API and drive a virtual water plane around an AR figure placed on a beach or a campus lawn. Twist: show where the water will be in six hours as a ghost line on the figure. ### Chris O'Shea *Artist and creative technologist* British artist and designer making playful camera-based installations for public screens, museums and hospitals; long-time openFrameworks contributor. #### Out of Bounds — Chris O'Shea (2007) - Video: https://vimeo.com/1333176 - Interaction: Tangible Objects, Projection, Portals & Worlds - Platform & tech: Projection, computer vision, tracked torch - Idea: Shine an 'X-ray flashlight' at the wall to see the hidden space behind it. - What it is: An 'x-ray torch' at the Design Museum London: shining a handheld light on the wall projects what lies behind it, as if boring through the museum's walls. - Technique: A camera tracks the position of the handheld torch's light spot on the wall, and the projector reveals a hidden image layer only inside a circular mask at that position. - Try it: Use a camera to track a flashlight's spot on the wall and project a 'behind the wall' image (pipes, mice, the classroom next door) only inside the spot. Twist: the world behind the wall slowly notices it is being watched and starts to dodge the light. #### Cybrid Landscape — Chris O'Shea (2008) - Video: https://vimeo.com/1330818 - Interaction: Spatial Mapping, Information & UI, Hands & Body - Platform & tech: Desktop, computer vision, 3D terrain - Idea: Footsteps in a real building erode a virtual landscape. - What it is: A virtual 3D landscape represents a building's atrium; as people walk through the real space, their paths wear away the virtual terrain, leaving a visible history of movement over the day. - Technique: Overhead cameras track people in the atrium, and each tracked position lowers the corresponding vertices of a 3D terrain mesh. - Try it: Track classmates' positions in a room from above and let each step dig into a virtual sand terrain shown in AR on the floor. Twist: the terrain slowly heals, so only popular paths stay visible. #### Traces — Chris O'Shea (2008) - Video: https://vimeo.com/1821234 - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, computer vision - Idea: People's movements leave light trails in the projection, like long-exposure car lights. - What it is: A responsive projection with Moritz Waldemeyer that turns visitors' movements into long-exposure-like light trails inspired by car tail-light photography. - Technique: Background subtraction extracts moving bright regions or silhouettes each frame, which are accumulated into a slowly decaying frame buffer to emulate long-exposure light trails. - Try it: Use frame accumulation with decay (feedback) in p5.js or TouchDesigner so people's movements leave long-exposure light trails, and project them. Twist: the trails record only the slowest movements, and fast ones are wiped out. #### Beacon — Chris O'Shea (2009) - Video: https://vimeo.com/1025054392 - Interaction: Gaze & Attention, Hands & Body, Perception & Effects - Platform & tech: Projection, openFrameworks, thermal camera - Idea: A field of lights that notices you and turns to look. - What it is: Visitors walk through a grid of modified emergency beacon lights that turn and brighten to follow them, filling the dark space with sweeping, responsive beams. - Technique: Thermal cameras track people, and a bespoke control system (openFrameworks) drives custom hardware inside each beacon to set reflector rotation and lamp brightness (with Cinimod Studio). - Try it: Line up several phones or small lamps on servos and make them all point toward the person tracked by a webcam. Twist: when two people are present, the lights split their attention unevenly. #### Hand from Above — Chris O'Shea (2009) - Video: https://vimeo.com/7042266 - Interaction: Hands & Body, Location & City, Play - Platform & tech: Desktop, openFrameworks, computer vision - Idea: A giant hand reaches into the live feed on a square's big screen to tickle, stretch and even lift away passers-by. - What it is: On BBC Big Screens in Liverpool and beyond, a giant hand reaches into the live camera feed of the square to tickle, stretch, flick and remove unsuspecting passers-by. - Technique: Background subtraction and blob tracking find pedestrians in the live square feed, and a pre-animated giant hand is composited to target a chosen person, warping or removing their pixels. - Try it: Use MediaPipe to detect people in a CCTV-style view and have an animated giant hand reach down from the top of the frame to 'poke' one of them, stretching or squashing them. Twist: the hand only picks people looking down at their phones, making a small prank about attention. #### Body Swap — Chris O'Shea (2011) - Video: https://vimeo.com/20745353 - Interaction: Hands & Body, Shared & Social, Play - Platform & tech: Desktop, Kinect, openFrameworks - Idea: Swap bodies with the person beside you: you control their on-screen figure while they control yours. - What it is: Two people stand side by side and each controls the other's body on screen like a puppet, making each other dance, jump and look silly. - Technique: Kinect skeleton tracking of two players drives two on-screen puppets, with each player's joint data mapped to the other player's avatar. - Try it: Track two people at once with MediaPipe Pose and render A's movements on B's silhouette puppet and vice versa. Twist: swap only half the body, for example you control the other person's legs and they control your arms, to complete a shared task. #### Dash Dodge Dive — Chris O'Shea (2011) - Video: https://vimeo.com/227226458 - Interaction: Hands & Body, Shared & Social, Play - Platform & tech: Desktop, computer vision, networked big screens - Idea: Big screens in squares across the UK are linked so crowds battle other cities with their bodies. - What it is: Crowd games for the London 2012 Olympics big screens: people in public squares across the UK compete against each other using body movements seen by the screens' cameras. - Technique: Big-screen cameras use motion detection (frame differencing or blob tracking) to measure crowd movement per zone, and scores are synchronised over a network between sites. - Try it: Split the camera image into left and right zones in p5.js, measure the amount of motion in each as the two teams' "strength", and play a projected tug-of-war. Twist: two classrooms compete over the network, and the winners get to leave a message on the other side's screen. #### Little Magic Stories — Chris O'Shea (2011) - Video: https://vimeo.com/20196781 - Interaction: Performance, Hands & Body, Drawing & Making - Platform & tech: Projection, Kinect, openFrameworks - Idea: Children act on a small stage while the characters and sets they drew come alive through projection. - What it is: Children perform stories on a small stage with drawn props and costumes; Kinect tracking and projection bring their imagined characters and scenery to life in front of an audience. - Technique: Kinect skeleton and depth tracking follow the children on stage, and colored props are recognised to trigger projected characters and scenery tied to their positions. - Try it: Give each group a few drawn paper props, use color detection to trigger projected backgrounds and characters, use MediaPipe Pose to make characters follow the performers, and stage a 3-minute story. Twist: the loudness of the audience's applause picks the scene for the next act. #### Woodland Wiggle — Chris O'Shea (2013) - Video: https://vimeo.com/59349284 - Interaction: Hands & Body, Play - Platform & tech: Desktop, Kinect, openFrameworks - Idea: A wall-sized interactive fairy-tale forest in a hospital, where children paint, make music and call up weather with their bodies. - What it is: A room-sized interactive TV at the Royal London Hospital where children step into a storybook woodland to paint, play music and summon weather with animated animals. - Technique: Kinect body tracking maps children's positions and gestures to triggers in a 2D animated scene (painting, playing instruments, calling weather), with large-scale display for full-body play. - Try it: Design a motion-sensing wall for a children's waiting area: use MediaPipe Pose so reaching out, jumping and crouching trigger painting, drumming and snow. Twist: every action must be possible while sitting or lying down, for children with limited mobility. #### Wilderness Wiggle — Chris O'Shea (2016) - Video: https://vimeo.com/271539058 - Interaction: Hands & Body, Play - Platform & tech: Desktop, depth camera - Idea: A motion-sensing wilderness play wall in an Alaskan hospital where animals and landscape answer children's movements. - What it is: A body-controlled Alaskan wilderness play wall for children at the Alaska Native Medical Center, where movement makes animals and landscape respond. - Technique: A depth camera tracks body silhouettes and movement, which are mapped to responsive animal and landscape animations on a play wall. - Try it: Pick a natural landscape from your hometown and build a motion-sensing wall with a camera and p5.js where movement makes local animals appear or hide. Twist: ask a classmate from a different place to supply animals and sounds so the wall tells the story of their place. ### Christo and Jeanne-Claude *Environmental artists* Artist couple who wrapped buildings and coastlines in fabric and ran fences, curtains, gates and floating piers across landscapes for two or three weeks at a time, financing every project themselves. #### Wrapped Coast — Christo and Jeanne-Claude (1969) - Video: https://www.youtube.com/watch?v=lU5oBJ0zoV8 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, erosion-control fabric, rope, coastal cliffs - Idea: Wrapping hides detail and reveals form: a single material laid over a landscape is the physical version of an AR mesh shader. - What it is: At Little Bay in Sydney, about 2.5 kilometres of rocky coastline and cliffs were wrapped in 92,900 square metres of fabric tied with rope for ten weeks. - Technique: Synthetic fabric was draped over the cliffs and held with 56 kilometres of rope by volunteers and climbers. - Try it: Use LiDAR mesh reconstruction to cover a real rock, bench or car with a draped fabric material in AR, with rope lines following the surface. Twist: let the wind (random noise) ripple the fabric. #### Valley Curtain — Christo and Jeanne-Claude (1972) - Video: https://www.youtube.com/watch?v=nDpkNFcmEWQ - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, nylon curtain, steel cables, valley span - Idea: A thin coloured plane across a valley shows how one flat element can re-scale a whole landscape, and how weather is part of the design. - What it is: An orange nylon curtain 400 metres wide was hung across Rifle Gap in the Colorado Rockies; it stood for only 28 hours before a gale forced its removal. - Technique: The curtain hung from steel cables anchored in the mountainsides, with openings to let the wind pass (the Maysles brothers documented its making). - Try it: Span a huge orange AR plane between two real landmarks (buildings or trees) using geospatial anchors and watch it from far away. Twist: tie its survival to live wind-speed data so it tears when the wind is strong. #### Running Fence — Christo and Jeanne-Claude (1976) - Video: https://www.youtube.com/watch?v=nBVpgN4JAsE - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, white nylon panels, steel poles, rolling hills - Idea: A line that follows the land for 40 kilometres makes the topography readable, a strong idea for AR lines drawn on terrain data. - What it is: A white fabric fence 5.5 metres high ran for 39.4 kilometres across the hills of Sonoma and Marin counties in California and into the Pacific Ocean, for two weeks. - Technique: 2,050 fabric panels hung from steel cables between poles followed the ground's curves, installed with the ranchers' permission. - Try it: Use a digital elevation model and a geospatial AR SDK to draw a white fence line along a real ridge that you can see from a viewpoint. Twist: extend the line into the sea or a lake and let it sink. #### Surrounded Islands — Christo and Jeanne-Claude (1983) - Video: https://www.youtube.com/watch?v=nfDGK_WSFro - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, floating pink polypropylene, bay islands - Idea: Outlining existing things with a bright halo makes them visible as a set, a basic AR highlighting technique at landscape scale. - What it is: Eleven islands in Miami's Biscayne Bay were each surrounded by a floating skirt of pink fabric extending 61 metres into the water, for two weeks. - Technique: Sewn pink panels were attached to floating booms anchored around each island's shoreline and unfurled on the same day. - Try it: Detect objects on a table or in a courtyard with segmentation and draw a pink offset outline around each on the ground plane in AR. Twist: the outlines grow when two objects are close, merging into one shape. #### The Umbrellas, Japan–USA — Christo and Jeanne-Claude (1991) - Video: https://www.youtube.com/watch?v=S_2kQyHgBPg - Interaction: Location & City, Shared & Social, Perception & Effects - Platform & tech: Desktop, octagonal umbrellas, two-country installation - Idea: Two simultaneous sites in two countries, one colour each, show how a shared event can link distant places, a model for synchronous multi-site AR. - What it is: 1,340 blue umbrellas opened in a valley in Ibaraki, Japan, and 1,760 yellow umbrellas in the hills north of Los Angeles at the same time, each six metres tall. - Technique: Steel-framed umbrellas were set in bases across private and public land and opened on the same morning in both countries. - Try it: Build a shared AR session (Niantic Lightship or Photon) where students in two classrooms each place coloured umbrellas and see the other room's umbrellas appear as a mirrored layer. Twist: umbrellas only open when both sites are online. #### Wrapped Reichstag — Christo and Jeanne-Claude (1995) - Video: https://www.youtube.com/watch?v=4zYKa6xmbjQ - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, aluminium-coated fabric, blue rope, parliament building - Idea: Hiding a loaded symbol makes people look at it again; an AR 'wrap' filter on a monument can do the same. - What it is: After 24 years of lobbying, the German parliament building in Berlin was wrapped in silver fabric and blue rope for two weeks, drawing about five million visitors. - Technique: About 100,000 square metres of fabric and 15.6 kilometres of rope were fitted over steel frames that protected the building's sculptures. - Try it: Use Lens Studio City Landmarker or a scanned building model to wrap a famous local building in silver cloth in AR. Twist: let people pull the rope to unwrap it together. #### The Gates, Central Park — Christo and Jeanne-Claude (2005) - Video: https://www.youtube.com/watch?v=CcSkyeXdHS4 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, saffron nylon panels, vinyl frames, park paths - Idea: Repeating one element along every path turns a park's circulation into a visible drawing, a strong pattern for AR wayfinding. - What it is: 7,503 saffron-coloured gates with hanging fabric lined 37 kilometres of footpaths in New York's Central Park for 16 days in February. - Technique: Gates of different widths were made to fit each path and stood on steel bases, so no holes were drilled in the park. - Try it: Generate a repeated saffron gate every five metres along a path traced from GPS or a map in an AR walk. Twist: gates only unfurl in front of the walker and fold up behind them. #### The Floating Piers — Christo and Jeanne-Claude (2016) - Video: https://www.youtube.com/watch?v=h9KMY970tXk - Interaction: Location & City, Hands & Body, Perception & Effects - Platform & tech: Desktop, floating polyethylene cubes, yellow fabric, lake - Idea: Walking on water is felt through the body (the piers move under your feet), a reminder that AR can borrow real physical sensations. - What it is: Three kilometres of walkways covered in yellow fabric floated on Lake Iseo in Italy, letting 1.2 million people walk on water between the shore and two islands for 16 days. - Technique: 220,000 floating polyethylene cubes formed a modular dock covered with shimmering fabric that changed colour when wet. - Try it: In AR, lay a glowing path across a real pond or a car park and add subtle camera sway when the user walks on it. Twist: the path only extends one step ahead of the walker. #### The London Mastaba — Christo and Jeanne-Claude (2018) - Video: https://www.youtube.com/watch?v=P2vno5rLldE - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, painted oil barrels, floating platform, lake - Idea: A single abstract volume on water, doubled by its reflection, shows how simple geometry plus reflection can feel monumental in AR. - What it is: A 20-metre-high trapezoidal stack of 7,506 painted oil barrels floated on the Serpentine lake in Hyde Park for three months, its colours reflected in the water. - Technique: Barrels were fixed onto a scaffold on a floating platform anchored with 32 anchors to the lakebed. - Try it: Place a large AR block on a real water surface with a planar reflection shader (Unity URP) so it doubles in the water. Twist: build it from hundreds of small instanced barrels that visitors can recolour. #### L'Arc de Triomphe, Wrapped — Christo and Jeanne-Claude (2021) - Video: https://www.youtube.com/watch?v=NUXIn9Fj7GI - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, recyclable silver-blue fabric, red rope, monument - Idea: A familiar monument turned into a soft, shimmering object is exactly what a city-scale AR material swap feels like. - What it is: Realised after Christo's death, the Arc de Triomphe in Paris was wrapped in 25,000 square metres of silver-blue fabric and 3,000 metres of red rope for 16 days. - Technique: Recyclable fabric was lowered from the top over steel frames that protected the reliefs, and tied with red polypropylene rope. - Try it: Take a photogrammetry model of a local monument and, in AR, swap its material for silver fabric with red rope lines. Twist: the fabric shimmers more when the wind (device motion) is stronger. #### The Gates in augmented reality (Bloomberg Connects) — Christo and Jeanne-Claude (2025) - Video: https://www.youtube.com/watch?v=ncGYO75uATA - Interaction: Location & City, Information & UI - Platform & tech: Phone, Bloomberg Connects app, geospatial AR, archival reconstruction - Idea: A vanished temporary artwork can return in AR at its original site, turning an archive into a walkable experience. - What it is: For the 20th anniversary of The Gates, an AR experience in the Bloomberg Connects app puts the saffron gates back along Central Park's paths on visitors' phones. - Technique: Likely geospatial AR anchoring of 3D gate models to their documented positions along the park paths. - Try it: Choose a past event or vanished structure on your campus, find an archive photo, and re-place it in AR at its exact original spot. Twist: add a 'then/now' slider that crossfades the old scene with the live view. ### Eyal Ofek *Computer vision and XR researcher (formerly Microsoft Research; now University of Birmingham)* Worked on the first real-time time-of-flight camera (ZCam), then spent many years at Microsoft Research on projector-camera systems, AR layout, haptic VR controllers and walking-based VR, co-authoring IllumiRoom and RoomAlive with Hrvoje Benko and Andy Wilson. #### FLARE: Fast Layout for AR — Eyal Ofek (2014) - Video: https://www.youtube.com/watch?v=8QecVs68vgA - Interaction: Spatial Mapping, Information & UI - Platform & tech: Headset, Desktop, Kinect, constraint solver - Idea: Describe AR content with constraints about the room, not coordinates, so it adapts to any space. - What it is: AR applications declare rules (a game board on a table, a screen on a wall, menus within reach) and the system automatically lays out their content to fit whatever room it runs in. - Technique: A Kinect scan is segmented into planes, and a declarative constraint solver places application objects on suitable surfaces in real time. - Try it: Write three layout rules for an AR app (e.g. 'lamp on highest surface') and implement them with AR plane detection, then test in three different rooms. Twist: add a rule that breaks ties randomly. #### GlassHands — Eyal Ofek (2016) - Video: https://www.youtube.com/watch?v=kXCq8mwjhyY - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, front camera, reflection, hand tracking - Idea: Use the user's eyeglasses as a mirror to extend the camera's view. - What it is: A phone's front camera sees the user's hands above and around the phone reflected in their own eyeglasses, turning the space around the device into an input area. - Technique: The front camera detects the reflection in the lenses of the user's glasses, rectifies it using the lens curvature and tracks hands in the reflected image. - Try it: Place a small mirror at an angle near a laptop webcam to see the desk, and use hand tracking on the reflection to control a slider. Twist: use a spoon as the mirror. #### CLAW — Eyal Ofek, Microsoft Research — Hrvoje Benko & Andy Wilson (2018) - Video: https://www.youtube.com/watch?v=iFEcR5gGojs - Interaction: Hands & Body, Tangible Objects, Play - Platform & tech: Headset, servo, haptics, VR controller - Idea: One controller, three haptic personalities, chosen by how you hold it. - What it is: A handheld VR controller with a motorised index-finger arm: it pushes back when you grasp a virtual object, resists when you touch a virtual surface and acts as a trigger when you shoot. - Technique: A servo-driven arm with a force sensor sits under the index finger, and grip sensing switches between grasp, touch and trigger modes to render force feedback. - Try it: Tape a servo to a phone case so a lever pushes against your finger when a virtual object in phone AR is touched. Twist: make soft objects push back gently and hard ones firmly. #### DreamWalker — Eyal Ofek, Microsoft Research — Hrvoje Benko & Andy Wilson (2019) - Video: https://www.youtube.com/watch?v=1dAVH8a80BE - Interaction: Portals & Worlds, Location & City, Spatial Mapping - Platform & tech: Headset, GPS, RGB-D, VR - Idea: Replace the world while keeping its walkable shape. - What it is: A person walks a real route of several hundred metres through city streets while wearing VR and sees a completely different virtual world, with obstacles and pedestrians safely mapped into it. - Technique: GPS, inside-out tracking and RGB-D cameras fuse to keep the user on a pre-planned route, and a virtual path is warped to match it while detected obstacles appear as virtual objects. - Try it: Walk a short corridor route with a phone in a headset showing only a virtual forest path mapped to it, with a spotter. Twist: turn each real door into a virtual gate. #### SeeingVR — Eyal Ofek, Microsoft Research — Hrvoje Benko & Andy Wilson (2019) - Video: https://www.youtube.com/watch?v=GIpoUKSr5vc - Interaction: Perception & Effects, Information & UI, Voice & Sound - Platform & tech: Headset, Unity, shaders, text-to-speech - Idea: Accessibility filters for virtual worlds, applied like lenses over any app. - What it is: A toolkit of 14 visual and audio aids, such as magnification, edge enhancement, depth colouring and object descriptions, that can be added to any Unity VR app to make it usable for people with low vision. - Technique: Post-processing shaders and scene queries in a Unity plug-in add magnifiers, contrast and edge effects, depth-coded colour and text-to-speech labels without changing the app. - Try it: Add three low-vision aids (edge outline, magnifier, spoken label) to a phone AR scene and test them with blurred glasses. Twist: design an aid for AR that does not exist in the toolkit. #### VRoamer — Eyal Ofek, Microsoft Research — Hrvoje Benko & Andy Wilson (2019) - Video: https://www.youtube.com/watch?v=jeWelz-Yl94 - Interaction: Play, Spatial Mapping, Portals & Worlds - Platform & tech: Headset, depth camera, procedural generation, VR - Idea: The building you walk through is the level designer. - What it is: A VR game generates its levels on the fly as the player walks through a large, unknown real building: corridors become dungeon passages and rooms become arenas placed where there is actually space. - Technique: Depth cameras detect walkable space ahead in real time, and pre-authored room and corridor modules are selected and fitted to it procedurally. - Try it: Map a school hallway with AR planes and place a 'dungeon tile' on each free floor patch as you walk. Twist: doors you pass become level exits. #### MagicPen — Eyal Ofek (2026) - Video: https://www.youtube.com/watch?v=stWjWZemYww - Interaction: Drawing & Making, Tangible Objects, Hands & Body - Platform & tech: Headset, haptics, telescoping stylus, VR - Idea: Fake a surface by moving the tool, not the world. - What it is: A stylus that physically grows and shrinks, so its tip always rests on the virtual surface you are drawing on in VR, giving you something solid to press against while sculpting. - Technique: A motorised telescoping shaft adjusts the pen's length in real time based on the distance to the virtual surface, braced against a physical desk so pressure is grounded. - Try it: Draw on a virtual sphere floating above a table in phone AR while physically pressing a pencil on the table, and compare with drawing in mid-air. Twist: map pencil pressure to line thickness. ### Golan Levin *Artist; Professor of Electronic Art, Carnegie Mellon University; director of the Frank-Ratchye STUDIO for Creative Inquiry* Artist and educator whose installations and performances explore abstract communication, the voice, the hand and the gaze through real-time computer vision and projection. #### Messa di Voce (installation) — Golan Levin, Zach Lieberman (2005) - Video: https://vimeo.com/221802940 - Interaction: Voice & Sound, Hands & Body, Projection - Platform & tech: Projection, computer vision, audio analysis - Idea: Speech and song grow visible shapes out of your mouth that you can push with your body. - What it is: Installation version of the 2003 performance: visitors' voices are visualized as projected bubbles and shapes that emerge from their mouths and can be pushed with their bodies. - Technique: Camera tracking locates each visitor's head/mouth position, and real-time audio analysis (amplitude, pitch, spectral features) generates projected shapes emitted from that point, which then collide with body silhouettes. - Try it: Use MediaPipe to find the mouth, analyze volume and pitch with p5.sound, and blow bubbles of different shapes out of the mouth, projected on a wall. Twist: design one shape for each vowel (a, e, i, o, u) and have everyone 'write' with their voice. #### Scrapple — Golan Levin (2005) - Video: https://vimeo.com/227633208 - Interaction: Tangible Objects, Voice & Sound, Projection - Platform & tech: Projection, computer vision - Idea: Whatever you toss on the table becomes a score that plays live and gets marked up by projection. - What it is: Everyday objects placed on a table are read as marks on a live musical score; the scanned table plays them as sound while projection augments the objects with graphics. - Technique: An overhead camera segments objects on the table (background subtraction and blob detection), treating the x-axis as time and the y-axis as pitch like a spectrogram score, with a scanning playhead that sonifies blobs and projection highlighting them. - Try it: Film the table from above, detect object blobs in p5.js with the horizontal axis as time and the vertical axis as pitch, and play and highlight them by projection as a playhead sweeps across. Twist: compose using only kitchen items and play the class a 30-second 'breakfast tune'. #### Footfalls — Golan Levin, Zach Lieberman (2006) - Video: https://vimeo.com/227566535 - Interaction: Voice & Sound, Hands & Body, Projection - Platform & tech: Projection, computer vision, microphones - Idea: The harder you stomp, the more virtual objects fall from the sky, and you can catch them with your shadow and throw them back. - What it is: Stomping on the floor makes cascades of bouncy virtual objects fall from a six-metre projection; visitors catch and throw them with their silhouettes. - Technique: Floor-mounted contact microphones detect stomp onsets and amplitude, spawning physics objects, while camera-based silhouette extraction provides collision boundaries for catching them. - Try it: Use a laptop microphone to detect volume peaks from stomps or claps, dropping a batch of objects from the top of the projected wall at each peak, and let people's outlines catch them. Twist: different strengths drop different things, such as petals for a light stomp and rocks for a hard one. #### Ghost Pole Propagator — Golan Levin (2007) - Video: https://vimeo.com/222999706 - Interaction: Hands & Body, Projection - Platform & tech: Projection, computer vision, skeletonization - Idea: Distill a person's silhouette into skeleton-like lines projected on the wall. - What it is: Visitors' silhouettes are reduced to skeletal centerline 'ghost poles' projected large on a wall, echoing their posture as calligraphic stick figures. - Technique: Binary silhouettes are thinned by morphological skeletonization into centerline pixels, which are vectorized into poles and rendered as calligraphic strokes. - Try it: Use OpenCV skeletonization or MediaPipe Pose connections to turn a person's pose into calligraphic strokes projected on a wall, keeping afterimages from the last few seconds. Twist: have the class pose one after another to assemble a 'body alphabet' of your own invention. #### Interstitial Fragment Processor — Golan Levin (2007) - Video: https://vimeo.com/86071976 - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, computer vision - Idea: Turn the gaps between people's shadows into solid things that fall, bounce and make sounds. - What it is: The negative spaces between and within visitors' shadows are cut out and dropped as elastic red and blue objects that fall and collide with sound. - Technique: Silhouettes from background subtraction are analysed for enclosed holes (contour hierarchy / connected components), and those negative-space regions become rigid bodies in a 2D physics engine. - Try it: Cut out people's shadows from a camera image, find the holes enclosed inside them (like the gap between arm and body with hands on hips), and turn those holes into colored blocks that fall, collide and make sounds. Twist: everyone must cooperate to enclose a given shape (a heart, a star) with their bodies to release it. #### Reface [Portrait Sequencer] — Golan Levin, Zach Lieberman (2007) - Video: https://www.youtube.com/watch?v=9dh2ghiwJW0 - Interaction: Face, Shared & Social, Perception & Effects - Platform & tech: Desktop, face tracking, openFrameworks - Idea: A magic mirror like an exquisite-corpse game: it mixes different visitors' eyes, brows and mouths into new faces live, and a blink switches them. - What it is: A face-tracking mirror that records slices of visitors' eyes, brows and mouths and remixes them into endless 'exquisite corpse' composite faces; blinking advances the combinations. - Technique: Face tracking aligns each visitor's face to a canonical frame, horizontal strips (eyes, nose, mouth) are recorded as video slices, and blink detection triggers recombination of strips from different people. - Try it: Align faces with MediaPipe Face Mesh, save three horizontal strips (eyes, nose, mouth) separately, and let each blink randomly assemble a 'composite face' from the whole class. Twist: let emotion set the splicing rule, such as using only mouths caught while everyone is smiling. #### Ghost Projection for Richard III — Golan Levin (2010) - Video: https://vimeo.com/11013624 - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, openFrameworks, computer vision - Idea: Project static noise only onto the 'ghost' actors on stage, turning them into phantoms. - What it is: Stage media for a Carnegie Mellon production of Richard III: computer vision tracks actors playing ghosts and projects static noise only onto their moving bodies. - Technique: Stage cameras (likely IR with background subtraction) segment the actors' bodies, and the resulting matte, warped through a camera-projector calibration, restricts projected video noise to their silhouettes. - Try it: Use MediaPipe body segmentation and a projector to cast TV static only onto a classmate, keeping the background completely black. Twist: write a one-minute 'ghost monologue' and let the static grow and fade with the loudness of the lines. #### Eyeshine — Golan Levin, Kyle McDonald (2011) - Video: https://vimeo.com/29356492 - Interaction: Gaze & Attention, Perception & Effects - Platform & tech: Desktop, infrared camera, computer vision - Idea: Catch the red-eye glint in visitors' eyes so they see their own eyes glowing like a night animal's. - What it is: An installation that captures, records and replays the red-eye retroreflections from observers' eyes, revealing a normally invisible property of the human retina. - Technique: An infrared light source placed coaxially with an IR camera produces retroreflection from the retina (the red-eye effect), which is detected as bright pupil blobs and recorded and replayed. - Try it: In a dark room, photograph people's eyes with the phone flash held right next to the lens (or use an IR camera), isolate the pupil glints in p5.js and save them, then arrange them into a 'wall of eyeshine'. Twist: collect glints only from people who hold eye contact for more than 3 seconds, and discuss the ethics of 'being seen'. #### Augmented Hand Series — Golan Levin, Chris Sugrue, Kyle McDonald (2014) - Video: https://vimeo.com/111951283 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Leap Motion, openFrameworks - Idea: Put your hand in a box and watch it on screen grow an extra finger or longer fingers, live and uncanny. - What it is: Visitors insert a hand into a box and see it transformed in real time on screen: extra or missing fingers, lengthened digits, wobbling knuckles and other uncanny variations. - Technique: A Leap Motion (or camera) tracks hand joints, and an image-warping mesh is deformed per joint to add, remove, or stretch fingers on the live hand video. - Try it: Get the 21 keypoints from MediaPipe Hands and apply mesh warping to the hand video in p5.js to lengthen fingers or add one more. Twist: design a hand that "moves on its own", with one finger quietly wandering off while you hold still. #### ARKit re-code of Jeffrey Shaw's Golden Calf — Golan Levin (2017) - Video: https://vimeo.com/269478756 - Interaction: Spatial Mapping, Tangible Objects - Platform & tech: Phone, ARKit, Unity - Idea: A 1994 classic remade in phone AR: a golden calf on an empty plinth, visible only through the phone. - What it is: A speed-project with Claire Hentschker that re-creates Shaw's 1994 Golden Calf for ARKit: a phone reveals a virtual golden calf standing on an empty plinth. - Technique: ARKit horizontal plane detection (or an image marker on the plinth) anchors a reflective 3D model, and the environment probe reflects the real room on its gold surface. - Try it: Use Unity AR Foundation to "rewrite" a classic new-media artwork as phone AR by placing a metallic model with environment reflections on an empty stand. Twist: swap in an overlooked object from around you and "enshrine" it on the empty plinth. ### JR *Photographer and street artist* French artist who pastes giant black-and-white photographs onto buildings, walls, rooftops and landscapes, and builds huge scaffolded figures and anamorphic collages. #### Face 2 Face — JR (2007) - Video: https://www.youtube.com/watch?v=4u_G0G6Jog4 - Interaction: Face, Location & City, Shared & Social - Platform & tech: Desktop, paste-ups, separation wall, portrait pairs - Idea: Putting two faces side by side that people expect to see apart creates empathy, a pairing strategy AR face filters rarely use. - What it is: With Marco, JR pasted pairs of huge portraits of Israelis and Palestinians doing the same job, pulling funny faces side by side, on both sides of the separation wall and in eight cities. - Technique: Portraits taken with a wide-angle lens close up are printed large and pasted in pairs. - Try it: Build an AR wall where two users in different places each add a funny-face portrait that is pasted side by side. Twist: pair people only when they have the same job or hobby. #### Women Are Heroes — JR (2008) - Video: https://www.youtube.com/watch?v=ji2rv5yw5Hw - Interaction: Location & City, Face, Shared & Social - Platform & tech: Desktop, large-format paste-ups, favela, vinyl on rooftops - Idea: Pasting local faces on the architecture they live in gives a community authorship of the view, a model for participatory AR portraits. - What it is: In the favelas of Rio, the slums of Kibera and other places, JR pasted huge portraits of local women onto houses, stairways, trains and rooftops, so the community's faces cover the landscape. - Technique: Portraits of women's eyes and faces are printed in strips and pasted across many surfaces so they read as one image from a distance. - Try it: Take portraits of people on your street and in AR map their faces across the facades of the buildings they live or work in, split over several surfaces. Twist: the face assembles only from one viewpoint. #### Inside Out Project — JR (2011) - Video: https://www.youtube.com/watch?v=NgagcqxMB5U - Interaction: Face, Shared & Social, Location & City - Platform & tech: Desktop, photo booth truck, paste-up posters, participatory - Idea: A simple, repeatable format that anyone can deploy turns an artwork into a global platform, the dream of every social AR template. - What it is: Funded by the TED Prize, Inside Out invites any community to send portraits that JR's team prints as posters, which people paste together on walls in their own town; over 500,000 people have taken part. - Technique: A photo-booth truck prints large black-and-white portraits on the spot; participants paste them with wheat paste. - Try it: Build a WebAR 'photo booth' that turns a selfie into a black-and-white poster and pins it to a shared wall anchor where every visitor's poster accumulates. Twist: the wall shows only faces of people who are nearby right now. #### Giants, Rio 2016 — JR (2016) - Video: https://www.youtube.com/watch?v=HsILNmNMu-w - Interaction: Location & City, Hands & Body, Perception & Effects - Platform & tech: Desktop, scaffolding, photograph, Olympic city - Idea: A body frozen in mid-action at the scale of a building makes the city part of the movement, a trick for AR motion stills. - What it is: During the Rio Olympics, JR installed giant photographs of athletes on scaffolds across the city, such as a high jumper appearing to leap over a building. - Technique: Cut-out photographic prints were mounted on scaffolding and positioned so the athlete appears to interact with real buildings. - Try it: Capture a classmate mid-jump with body segmentation and place the cut-out at building scale so they appear to leap over a real rooftop in AR. Twist: shoot a sequence and play it as a slow flip-book as the viewer walks along the street. #### JR au Louvre — JR (2016) - Video: https://www.youtube.com/watch?v=M9VkFxEwINY - Interaction: Location & City, Perception & Effects, Portals & Worlds - Platform & tech: Desktop, anamorphic photo collage, glass pyramid - Idea: Printing the background onto the object makes it vanish from one viewpoint: the physical version of diminished reality in AR. - What it is: JR covered I. M. Pei's glass pyramid at the Louvre with a black-and-white photograph of the palace behind it, so from one point in the courtyard the pyramid seems to disappear. - Technique: A photograph of the facade behind the pyramid was printed in strips and pasted on the glass, aligned to one viewing position. - Try it: Photograph the wall behind a real object (a pillar, a sign) and in AR wrap the object with that photo so it disappears from one point. Twist: reveal the object again with a slow wipe when the user steps off the point. #### Giants: Kikito — JR (2017) - Video: https://www.youtube.com/watch?v=M_LMgzMnG7Q - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, scaffolding, photograph, border wall - Idea: A child's gaze across a wall turns a political barrier into a human moment, showing how an image aligned with a real structure can change its meaning. - What it is: A 20-metre photograph of a one-year-old boy, Kikito, peers over the US–Mexico border fence in Tecate, mounted on scaffolding on the Mexican side so he appears to look into the United States. - Technique: A cut-out photographic print was mounted on a scaffold structure behind the fence and aligned so the child appears to lean on it. - Try it: Place a giant AR cut-out photo of a person behind a real wall or fence so that they appear to look over it at passers-by. Twist: the person's eyes follow whoever walks past. #### Tehachapi — JR (2019) - Video: https://www.youtube.com/watch?v=WEkarbrmsIw - Interaction: Location & City, Shared & Social, Face - Platform & tech: Desktop, ground paste-up, prison yard, aerial photograph - Idea: An image made for the view from above, in a place people cannot leave, lets people be seen by the world; AR can give voice to spaces that are otherwise closed. - What it is: In the yard of a maximum-security prison in Tehachapi, California, JR and inmates pasted a huge collage of prisoners' and guards' portraits on the ground, visible from the sky. - Technique: Portrait strips were pasted over the yard's ground by participants following a grid, then photographed from a drone. - Try it: Design a large ground collage of portraits in AR for a place with restricted access (a courtyard, a rooftop) and share only the aerial view publicly. Twist: each portrait speaks a sentence when you stand on it. #### The Secret of the Great Pyramid — JR (2019) - Video: https://www.youtube.com/watch?v=rsnpm1_IXbw - Interaction: Location & City, Shared & Social, Perception & Effects - Platform & tech: Desktop, anamorphic paper collage, volunteers, courtyard - Idea: An anamorphic ground image that reveals depth from one viewpoint is the analogue ancestor of AR depth illusions, and it dissolved as people walked on it. - What it is: For the pyramid's 30th anniversary, 400 volunteers pasted 2,000 paper strips across the Louvre's Napoleon courtyard; from above, the pyramid seems to rise out of a deep rocky quarry. - Technique: A photograph of rock was distorted to match the viewing angle from above and printed on paper strips glued to the paving. - Try it: Create an anamorphic 'hole' image for a courtyard floor with a projection camera in Blender, print or display it in AR, and test the view from a balcony. Twist: let people's footsteps erase the image as they walk on it. #### Eiffel Tower anamorphosis (Trocadéro) — JR (2021) - Video: https://www.youtube.com/watch?v=ptUCuYhYL1c - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, anamorphic ground collage, Trocadéro - Idea: A single viewpoint turns a flat print into a cliff that swallows a monument; AR can use the same 'magic spot' to guide where people stand. - What it is: On the Trocadéro esplanade in Paris, a giant ground collage makes the Eiffel Tower appear to stand on the edge of a rocky cliff when photographed from the right spot. - Technique: Rock photographs were distorted for one camera position and pasted in strips on the paving in front of the tower. - Try it: Pick a landmark visible from campus and design an AR ground illusion that makes it look like it sits on the edge of a chasm from one marked spot. Twist: the chasm grows until someone steps into the frame. #### La Ferita (The Wound) — JR (2021) - Video: https://www.youtube.com/watch?v=P-FWZ5-tyws - Interaction: Location & City, Portals & Worlds, Perception & Effects - Platform & tech: Desktop, anamorphic collage, Renaissance facade - Idea: A fake tear in a real wall that shows the interior is exactly an AR portal, done in paper. - What it is: On the facade of Palazzo Strozzi in Florence, a 28-metre collage appears to tear open the stone to reveal the courtyard, the library and Renaissance halls inside, a comment on museums closed during the pandemic. - Technique: Photographs of the interior were composed into an anamorphic image aligned to the street view and pasted over the facade. - Try it: In AR, cut a jagged opening in a real wall with a depth mask and show a photo sphere of what is actually behind it (the room on the other side). Twist: the tear widens the longer you stand in front of it. #### La Caverne du Pont Neuf — JR (2026) - Video: https://www.youtube.com/watch?v=RngYQyEu54k - Interaction: Location & City, Portals & Worlds, Perception & Effects - Platform & tech: Desktop, inflatable structure, printed rock surface, bridge - Idea: Turning a bridge into a cave is world replacement at city scale, and pairing it with a phone lens shows how physical and AR layers can tell one story. - What it is: Paris's oldest bridge, the Pont Neuf, is covered by an inflated structure printed with rock, so the bridge seems to have vanished into a cave; the AR companion is listed separately as Echoes (Paris AR Studio). - Technique: A large inflatable envelope printed with rock imagery is fitted over the bridge and walkway (timelapse footage by AP). - Try it: Scan a small bridge or passage on campus and in AR replace its surfaces with a cave material, adding dripping sounds inside. Twist: the cave grows stalactites the longer the bridge is empty. ### Matthew Hallberg *AR developer and YouTuber; Snap Spectacles developer* Unity AR developer whose YouTube channel mixes tutorials with ambitious builds, from ARKit Minecraft and Portal to Iron Man HUDs and Snap Spectacles games. #### ARKit Pokémon Go with Unity GPS and Mapbox — Matthew Hallberg (2017) - Video: https://www.youtube.com/watch?v=u8JaKspgWvE - Interaction: Location & City, Play - Platform & tech: Phone, ARKit, Unity, Mapbox - Idea: ARKit plus GPS maps put Pokémon that actually stand on the street. - What it is: A Pokémon Go-style demo that combines ARKit tracking with Mapbox GPS so creatures stand at real-world locations. - Technique: Mapbox converts creature GPS coordinates into positions relative to the device's GPS location and compass heading, which are then placed in the ARKit world frame. - Try it: Use AR plus GPS to place a virtual creature at a set coordinate on campus that can only be seen and tapped to catch when you walk nearby. Twist: the creature appears only at certain times of day or in certain weather. #### Apple ARKit Augmented Reality Minecraft — Matthew Hallberg (2017) - Video: https://www.youtube.com/watch?v=qFGx9QcE5Gk - Interaction: Drawing & Making, Play, Spatial Mapping - Platform & tech: Phone, ARKit, Unity - Idea: Minecraft blocks built at real size on the real ground. - What it is: Place and break Minecraft blocks at life size on real ground, one of the most shared ARKit demos of summer 2017. - Technique: Raycasts onto detected planes and existing cube faces snap new blocks to a grid, and tapping a block removes it, reproducing Minecraft's voxel build logic in world space. - Try it: Set up a 10 cm grid on the floor plane where tapping places a block and long-pressing removes one, and let new blocks stack on the faces of existing ones. Twist: give each block type a musical note so building makes a tune. #### ARKit Portal to the Upside Down — Matthew Hallberg (2018) - Video: https://www.youtube.com/watch?v=Z5AmqMuNi08 - Interaction: Portals & Worlds - Platform & tech: Phone, ARKit, Unity - Idea: Open a portal in the living room to the Upside Down from Stranger Things. - What it is: A walk-through doorway in the living room that opens into the dark 'Upside Down' world from Stranger Things. - Technique: A stencil-buffer portal renders the alternate world only through the door frame, and the stencil test flips when the camera crosses the portal plane so the user is inside the other world. - Try it: Place a door with AR Foundation, render a dark, foggy world inside it with a stencil, and switch to full-screen rendering once you walk through. Twist: make the world behind the door the streets of your hometown. #### Memes + AR with Watson — Matthew Hallberg (2018) - Video: https://www.youtube.com/watch?v=7iN4ENlrx4A - Interaction: Voice & Sound, Spatial Mapping - Platform & tech: Phone, Vuforia, IBM Watson, Unity - Idea: Say a phrase and the matching meme appears in your room. - What it is: Say a phrase aloud and the matching meme image is searched and dropped into the room on a detected ground plane. - Technique: Speech-to-text (IBM Watson) extracts a phrase, an image search API returns a matching image, and the image is spawned as a textured quad on a detected ground plane. - Try it: Recognize a spoken phrase with the Web Speech API, find a matching image in a local library by keyword, and spawn it as an image card on the floor in WebXR. Twist: the image size follows how loudly you speak. #### Valve's Portal in Unity with ARKit — Matthew Hallberg (2018) - Video: https://www.youtube.com/watch?v=28CjsJaw_Mo - Interaction: Portals & Worlds, Spatial Mapping, Play - Platform & tech: Phone, ARKit 1.5, Unity - Idea: Shoot the blue and orange portals from Portal onto real walls. - What it is: Uses ARKit 1.5 vertical-plane detection to shoot linked blue and orange portals onto real walls and see through them. - Technique: Portals placed on detected vertical planes use render textures from paired virtual cameras whose pose mirrors the viewer's relative pose to the other portal. - Try it: Place a portal on each of two walls, each showing the RenderTexture of a virtual camera behind the other portal, and send a virtual ball in one portal and out the other. Twist: each side of the portal has a different time of day (day on one side, night on the other). #### AR Iron Man HUD — Matthew Hallberg (2019) - Video: https://www.youtube.com/watch?v=IKzqNpWC9WI - Interaction: Voice & Sound, Information & UI, Tangible Objects - Platform & tech: Headset, Phone, Vuforia, Unity, Alexa - Idea: Build your own Iron Man JARVIS headset from a phone and a cheap helmet. - What it is: A phone-in-headset JARVIS HUD that tracks a real phone, listens to voice commands and wires in Alexa skills. - Technique: A phone-in-headset stereo view renders a HUD overlay, while Vuforia tracks a second physical phone as a target and voice commands are routed through Alexa skills. - Try it: Build a head-locked HUD in HoloKit stereo mode that shows the time and the commands picked up by speech recognition. Twist: when you stare at a real object on the desk, the HUD pops up its 'scan info'. #### An Augmented Reality Web Browser — Matthew Hallberg (2019) - Video: https://www.youtube.com/watch?v=lMLSACaaOjU - Interaction: Information & UI, Spatial Mapping - Platform & tech: Phone, Unity, ARCore - Idea: Float web page windows in real space, an early spatial-computing browser. - What it is: Floating, resizable web pages placed around physical space, an early take on spatial-computing windows on Android. - Technique: Web pages are rendered to textures by an embedded browser plugin and mapped onto world-anchored quads in ARCore, with raycast input translated into page coordinates for clicks and resizing. - Try it: Place three floating panels in a WebXR scene, each showing a web page screenshot or iframe texture that can be dragged and scaled. Twist: the farther a panel is from you, the simpler its content gets (only the title remains). #### Real Life Doodles (Creator Cam) — Matthew Hallberg (2019) - Video: https://www.youtube.com/watch?v=_N2R94K9g5k - Interaction: Drawing & Making - Platform & tech: Phone, Unity, ARFoundation - Idea: Turn quick doodles into AR objects in real space. - What it is: An AR app built in a week-long devlog that turns flat doodles into objects living in the real world. - Technique: A photographed doodle is segmented from the white paper (thresholding or background removal) and applied as a cutout texture to a billboard or extruded mesh placed with plane detection. - Try it: Photograph a doodle on paper with a phone, remove the white background, place it on a tabletop as a texture and make it bounce gently. Twist: give the doodle a simple skeleton so it can walk. #### AR Photoshop Plugin — Matthew Hallberg (2021) - Video: https://www.youtube.com/watch?v=20fbXJ8fayM - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, Desktop, Unity, Vuforia, WebSockets - Idea: Pull Photoshop layers out of the monitor and into AR. - What it is: A Photoshop plugin streams each layer of a document to a phone so the layers can be pulled out of the monitor and viewed in AR. - Technique: A Photoshop plugin exports each layer as an image over WebSockets, and the phone stacks them as quads offset in depth on the monitor (tracked by Vuforia), letting the user pull layers apart in space. - Try it: Export each layer of a layered poster as a PNG, stack them by depth on the image-tracked poster, and use a pinch gesture to spread the layers apart. Twist: make the layer spacing change automatically with the phone's distance from the poster. #### AR Blasters on Snap Spectacles — Matthew Hallberg (2025) - Video: https://www.youtube.com/watch?v=WMnvuzjSSxU - Interaction: Play, Tangible Objects - Platform & tech: Headset, Phone, Snap Spectacles, Lens Studio - Idea: A phone becomes a blaster controller for AR shooting in Spectacles. - What it is: A phone becomes a blaster controller over BLE for a shooting game on Snap Spectacles AR glasses. - Technique: The phone's IMU orientation and button presses are sent over BLE to Spectacles, where they drive the aim direction of a raycast blaster in the glasses' world-tracked scene. - Try it: Stream the phone's gyroscope orientation over WebSocket to another device running WebXR and use it as the aiming direction to shoot down virtual targets. Twist: swap the phone for a real object you swing (such as a broom) with the phone strapped to it. #### Physical Pokémon Cards in AR — Matthew Hallberg (2026) - Video: https://www.youtube.com/watch?v=vv7b1DwOTsQ - Interaction: Tangible Objects, Play - Platform & tech: Headset, Snap Spectacles - Idea: Recognize physical Pokémon cards and let the Pokémon jump out onto the table. - What it is: Real Pokémon trading cards are recognised on Spectacles and their creatures appear on top of the table. - Technique: Card artwork is recognized as image targets (or by a classifier) on Spectacles, and the matching 3D creature is spawned on the card's tracked pose on the table. - Try it: Use image tracking in Lens Studio or AR Foundation to recognize three homemade cards, each spawning its own 3D creature. Twist: when two cards come close, their creatures start to battle or interact. ### Palindrome (Robert Wechsler) *Interactive dance company led by choreographer Robert Wechsler; co-developers of EyeCon and MotionComposer* Founded in New York in 1982 and based in Germany since 1991, Palindrome made some of the first dance pieces in which video motion tracking, skin contact, muscle and eye signals play the music and light, mostly with engineer Frieder Weiss. Since 2012 its work has centred on MotionComposer, a camera device that turns any movement into music for people of all abilities. #### Heisenberg's Uncertainty Principle — Palindrome (Robert Wechsler), Frieder Weiss (2001) - Video: https://www.youtube.com/watch?v=Xgnw39zrkCc - Interaction: Projection, Performance, Perception & Effects - Platform & tech: Projection, projection screen, video effects - Idea: Use projection on a see-through screen to make the dancers' position uncertain. - What it is: A 25-minute piece in which dancers perform behind and in front of a semi-transparent screen while projected words and video effects slide over their bodies, blurring where the dancers are. - Technique: A semi-transparent projection screen at the front of the stage carries live and prepared video effects designed by Frieder Weiss, layering images over the dancers behind it. - Try it: Hang a tulle or mesh sheet, project text onto it, and have a performer move behind and in front of it so the words land on and miss their body. Twist: track the performer and make the words avoid them. #### Brother/Sister (Blinde Liebe) — Palindrome (Robert Wechsler), Frieder Weiss (2002) - Video: https://www.youtube.com/watch?v=g8aWaOn5eNo - Interaction: Performance, Voice & Sound, Hands & Body - Platform & tech: Projection, EyeCon, Kalypso, Max/MSP - Idea: Let the dancers' bodies play part of an opera score in real time. - What it is: An evening-length interactive opera about two teenagers who committed a murder, with two dancers, two actors, five musicians, projected interview footage and dancers whose tracked movement shapes the electronic sound. - Technique: EyeCon analyses the camera image of each dancer and sends motion data to Max/MSP, which triggers and modulates electroacoustic material, while Kalypso processes projected video. - Try it: Stage a two-minute scene where one actor speaks text and a dancer's tracked motion (webcam + Pose detection) controls a drone sound underneath; rehearse until movement and words argue with each other. Twist: freeze the sound whenever the dancer stops, so silence becomes a line of dialogue. #### Maibaum (Baila Mi Ritmo) — Palindrome (Robert Wechsler), Frieder Weiss (2002) - Video: https://www.youtube.com/watch?v=_pHoyecfO8k - Interaction: Performance, Voice & Sound, Tangible Objects - Platform & tech: Projection, EyeCon - Idea: Make the costume itself part of the music, and let the dancers play it with their movement. - What it is: A dark Rite of Spring for six dancers in loud costumes made of kite material: recordings of the rustling costumes are processed and played back under the control of the dancers' tracked movement, with cello and bass live. - Technique: EyeCon tracks movement in defined stage zones and scrubs and filters samples recorded from the costumes, so the sound of fabric is heard both acoustically and electronically. - Try it: Record the sound of one garment (a raincoat, a tracksuit) and let a webcam's motion amount control the playback speed and filter of that recording while the wearer dances. Twist: swap garments between dancers so each moves to someone else's clothes. #### VorOrt — Palindrome (Robert Wechsler) (2002) - Video: https://www.youtube.com/watch?v=9tFnMzH5UZg - Interaction: Location & City, Performance, Portals & Worlds - Platform & tech: Projection, Steadicam, wireless video - Idea: Stitch three kinds of presence, recorded, live on screen and live in the room, into one continuous walk. - What it is: The piece starts on a live video feed of the dancer walking through the public square of the host city; she knocks on the theatre's back door and enters the stage where the audience has been watching her. - Technique: A Steadicam operator follows the dancer outside and transmits the signal wirelessly to the theatre projector until she physically arrives; the year is likely. - Try it: Livestream a classmate walking from outside the building to your classroom on a phone video call projected on the wall, and time their entrance so the screen and the door open together. Twist: add an AR label on the feed that follows them in and stays on their real body. #### Touching (e-touching) — Palindrome (Robert Wechsler), Frieder Weiss (2004) - Video: https://www.youtube.com/watch?v=FfjU-lbHdvo - Interaction: Hands & Body, Voice & Sound, Shared & Social - Platform & tech: Wearable, EyeCon, custom electronics, Max/MSP - Idea: Make touch between two people audible, because even the smallest contact carries meaning. - What it is: Two dancers wrestle, lean and press their hands together, and every moment of skin-to-skin contact becomes audible: the harder or longer they touch, the more the sound changes. - Technique: Custom electronics detect the tiny current that flows between two bodies when their skin touches and feed it into EyeCon's external-sensor input, which maps contact to sound. - Try it: Build a touch-to-sound duet with a Makey Makey or a capacitive-touch board: each partner holds one lead, and skin contact triggers a sound in a web page or Pure Data. Twist: map contact duration to pitch so a held hand becomes a slow glissando. #### Talking Bodies — Palindrome (Robert Wechsler), Frieder Weiss (2005) - Video: https://www.youtube.com/watch?v=4YKeRrsx9BM - Interaction: Hands & Body, Voice & Sound, Projection - Platform & tech: Projection, EyeCon, Kalypso, Max/MSP - Idea: Treat the moving body as a speaker that produces both sound and text. - What it is: A solo evening made with composer Dan Hosken in which dancer Emily Fernandez's movement makes the body speak: words are projected large beside her and sounds follow her gestures as a camera tracks her. - Technique: A video camera feeds EyeCon, which measures motion in zones around the dancer and triggers speech sounds and projected words, with Kalypso processing the live video. - Try it: Use a webcam and p5.js body tracking to project a word next to a performer whenever they move into a new zone of the stage, building a sentence from their path. Twist: let movement speed decide the font size, so a leap shouts. #### A Human Conversation — Palindrome (Robert Wechsler) (2006) - Video: https://www.youtube.com/watch?v=WFgEO5G2jsE - Interaction: Hands & Body, Voice & Sound, Shared & Social - Platform & tech: Projection, EyeCon, Max/MSP, touch sensors - Idea: Let a silent conversation of gestures be heard as music. - What it is: A piece for five dancers built from sign language, gesture and touch, in which the performers' conversation of hands and bodies is turned into music by composer Dan Hosken's interactive score. - Technique: Camera-based motion tracking with EyeCon and Palindrome's skin-contact sensors send gesture and touch data to a music computer, which plays and shapes the score live. - Try it: Teach pairs of students five signs, track their hands with MediaPipe in the browser, and map each recognised sign to a sound layer so a short signed dialogue becomes a duet. Twist: only play sound when both partners sign at the same time. #### Jenseits der Schatten — Palindrome (Robert Wechsler), Frieder Weiss (2006) - Video: https://www.youtube.com/watch?v=LQAGrlC6xnY - Interaction: Projection, Performance, Perception & Effects - Platform & tech: Projection, EyeCon, Kalypso - Idea: Stage Plato's cave literally: the audience watches bodies and the shadows the system makes of them. - What it is: An opera by Vladimir Tarnopolsky on Plato's cave allegory, directed by Robert Wechsler for Bonn Opera, in which singers and dancers move on stepped platforms while interactive projections and shadows answer their movement. - Technique: Camera tracking by Frieder Weiss and Diana Wildschut drives projected shadow and light imagery on the set in real time, likely using EyeCon and Kalypso. - Try it: Build a shadow cave with a projector and a webcam: record each visitor's silhouette and replay it a few seconds later beside them, so they meet their delayed shadow. Twist: let the shadow slowly drift away from the body when the person stands still. #### Eye Movement — Palindrome (Robert Wechsler) (2008) - Video: https://www.youtube.com/watch?v=bFnqHgmNPwE - Interaction: Gaze & Attention, Face, Voice & Sound - Platform & tech: Desktop, EyeCon, video tracking - Idea: Turn the smallest, fastest movement of the body, the eyes, into an instrument. - What it is: Robert Wechsler sits in close-up while his eye movements and blinks play piano notes and melodies and, in the piece Oklo, switch the stage lights. - Technique: A camera close on the face tracks changes in the eye region (and, in other versions, electrodes pick up eye muscle signals), and each change triggers a sound or a lighting cue; year and exact setup are likely. - Try it: Use a face-tracking library (MediaPipe Face Mesh) to detect blinks and gaze direction and map them to notes on a scale and to the colour of a lamp. Twist: make a piece that only works with your eyes closed, playing the moment you open them. #### Jeu de modes — Palindrome (Robert Wechsler) (2014) - Video: https://vimeo.com/108061408 - Interaction: Hands & Body, Voice & Sound, Performance - Platform & tech: Projection, motion tracking, Pure Data - Idea: Map the scale of a gesture, small, conversational or explosive, to a mode of listening. - What it is: An interactive dance trio by Andreas Bergsland and Robert Wechsler exploring gesture from tiny finger movements to explosive swipes, each heard as a different kind of sound. - Technique: Camera motion tracking separates movement into size and energy ranges, and each range drives a different acousmatic sound behaviour inspired by Francois Bayle's categories such as rub, pull and push. - Try it: Track a performer's hand speed with MediaPipe and define three bands (tiny, medium, large), each with its own sound palette; perform a one-minute phrase that crosses all three. Twist: let two dancers share the tracker so the sound switches mode when they touch. #### MotionComposer — Palindrome (Robert Wechsler) (2016) - Video: https://www.youtube.com/watch?v=4aDj7Ma_HkE - Interaction: Hands & Body, Voice & Sound - Platform & tech: Desktop, depth camera, Pure Data, Csound, Linux - Idea: Everyone is a dancer and a musician if the instrument listens to whatever movement they have. - What it is: Robert Wechsler demonstrates a camera device that turns any movement, from a wave of the arm to a blink, into music, designed so that people with very different abilities can dance and compose together. - Technique: Stereo and depth cameras track position, size and gesture of one or more people, and Palindrome-designed music environments in Pure Data and Csound map small and large movements to different sonic behaviour. - Try it: Design a movement-to-music web app with MediaPipe Pose that works equally for someone standing and someone seated, and test it with both. Twist: map only the smallest detected movement to melody, so stillness-with-intention becomes the main instrument. ### Shunichi Kasahara *Researcher, Sony Computer Science Laboratories; leads Cybernetic Humanity Studio (OIST)* Sony CSL researcher who builds perception-bending systems: first-person view sharing (JackIn), out-of-body telepresence, EMS-accelerated reactions and real-time face morphing. Earlier co-created exTouch and Second Surface at the MIT Media Lab. #### exTouch — Shunichi Kasahara, MIT Tangible Media Group — Hiroshi Ishii, Valentin Heun (2013) - Video: https://vimeo.com/57514726 - Interaction: Tangible Objects, Hands & Body - Platform & tech: Phone, tablet AR, marker tracking, actuated robot - Idea: Drag an object in the phone's AR view and the real robot follows, turning the AR screen into a physical remote control. - What it is: Point a tablet at a small robot and drag its on-screen AR image with your finger; the real robot drives and rotates to match the gesture in physical space. - Technique: Tracks the robot's pose with the tablet camera and converts touch-drag gestures on its AR image into motor commands in the robot's own coordinate frame. - Try it: Use phone AR to recognize an image marker on a remote-control car, and drag a virtual arrow on screen to drive the real car (an Arduino or Bluetooth car works). Twist: let two classmates control the same car at once, each with a phone from a different angle. #### JackIn Eye — Shunichi Kasahara (2014) - Video: https://vimeo.com/88523949 - Interaction: Shared & Social, Gaze & Attention, Perception & Effects - Platform & tech: Wearable, Headset, head-mounted camera, SLAM, image stitching - Idea: A remote person slips into someone's first-person view and can even step out of the body to watch from behind. - What it is: A remote viewer 'jacks into' a walking person's head-mounted camera; the system stitches past frames into an out-of-body third-person view so the viewer can look around beyond the wearer's gaze. - Technique: Uses SLAM on the first-person video to register frames into a growing 3D panorama, then renders a virtual camera placed behind and above the wearer. - Try it: Livestream one classmate's first-person view from a phone to another classmate wearing Cardboard, who gives spoken directions to complete a find-the-object task. Twist: the viewer can only see a feed delayed by 5 seconds. #### JackIn Head — Shunichi Kasahara (2015) - Video: https://vimeo.com/129263445 - Interaction: Shared & Social, Gaze & Attention - Platform & tech: Wearable, Headset, omnidirectional camera, image stabilization, live streaming - Idea: Borrow the panoramic camera on someone else's head as your own eyes, and look around independently from afar. - What it is: A head-worn omnidirectional camera streams stabilized 360-degree video so remote viewers can look anywhere around the wearer, independent of where the wearer turns their head. - Technique: Estimates head rotation from the spherical video and counter-rotates each frame so the 360 stream stays world-stable for remote HMD viewers. - Try it: Strap a 360 camera to a hat and livestream it to a classmate's phone VR viewer, who then directs the wearer's walking with hand signals. Twist: let three people "possess" the same person at once and vote on which way to go. #### Parallel Eyes — Shunichi Kasahara (2016) - Video: https://vimeo.com/175105689 - Interaction: Shared & Social, Perception & Effects, Play - Platform & tech: Headset, Wearable, HMD, head-mounted cameras, video mixing - Idea: Four people see through each other's eyes at once and play hide-and-seek with shared vision. - What it is: Four people wear HMDs showing a split screen of everyone's first-person views at once, then try to play hide-and-seek and coordinate while seeing through all bodies simultaneously. - Technique: Streams each participant's head-mounted camera into a shared tiled video layout shown in every HMD, a collaboration with YCAM. - Try it: Have four classmates join a video call with their phones' front or rear cameras, show all feeds side by side, and play a find-the-person game wearing phone viewers. Twist: randomly swap the position of your feed with someone else's. #### HeadLight — Shunichi Kasahara (2018) - Video: https://vimeo.com/263377583 - Interaction: Projection, Gaze & Attention - Platform & tech: Wearable, Projection, pico projector, fish-eye lens, geometric correction - Idea: Wear a projector on your head and the whole space in front of you becomes a screen. - What it is: A head-mounted projector with a fish-eye lens throws a very wide image onto the surroundings in front of the wearer, augmenting whatever the wearer faces without glasses. - Technique: Pre-distorts the image to compensate for the fish-eye conversion lens and aligns it with head orientation so graphics land on the walls the wearer looks at. - Try it: Tape a phone projector (or a phone plus a mirror) to a helmet so the projected content appears on the wall wherever you look. Twist: project 'hidden markers' that appear only when you turn your head. #### Fragment Shadow — Shunichi Kasahara (2019) - Video: https://vimeo.com/322671485 - Interaction: Projection, Perception & Effects, Hands & Body - Platform & tech: Projection, multi-projector calibration, color calibration - Idea: Several projectors split one person's shadow into multiple colored shadows. - What it is: Visitors' real shadows are split into several colored, offset shadows by precisely calibrated multiple projectors, so one body casts a fragmented, prismatic shadow. - Technique: Calibrates several projectors' geometry and color so their overlapping light sums to white, making each body occlusion reveal a different colored shadow. - Try it: Shine three red, green and blue LED lights (or three phone flashlights with colored cellophane) at a white wall from different angles and let students make colored shadows with their bodies. Twist: program the lights to flash in turn with the music. #### Preemptive Action — Shunichi Kasahara, Pedro Lopes (2019) - Video: https://vimeo.com/315085705 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Wearable, electrical muscle stimulation, high-speed camera - Idea: Electrical stimulation moves your hand before you react, yet you still feel you did it yourself. - What it is: Electrical muscle stimulation fires the user's arm a few milliseconds before they would react to a stimulus, speeding reaction time while the user still feels they moved by themselves. - Technique: Triggers EMS at a tuned latency just before the user's natural reaction, finding the window where the action is faster yet still judged as self-caused. - Try it: Make a phone reaction game where you slap the table when a signal appears, and have a partner behind you give an early cue by tapping your shoulder. Twist: compare whether you feel 'I did it' with and without the cue. #### Stealth Updates — Shunichi Kasahara (2021) - Video: https://www.youtube.com/watch?v=96xfUsgCk1U - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Desktop, Headset, image morphing, change blindness - Idea: Use change blindness to alter an image right before your eyes without you noticing. - What it is: Visual content changes in front of the viewer without being noticed, by morphing images gradually and timing changes to exploit change blindness. - Technique: Computes a smooth image morph between old and new content and plays it slowly enough to stay below the threshold of noticed change. - Try it: Use phone AR to slowly morph an element of a poster into another image over 30 seconds, have classmates stare at it and report whether they notice the change. Twist: switch quickly when the viewer blinks or turns their head. #### Parallel Ping-Pong — Shunichi Kasahara (2022) - Video: https://www.youtube.com/watch?v=q1XAmaMdEiE - Interaction: Hands & Body, Play, Shared & Social - Platform & tech: Headset, VR telepresence, robot arms, perspective switching - Idea: One person controls two bodies to play two ping-pong matches at once. - What it is: One user in VR controls two robot arms playing table tennis at two tables simultaneously, switching perspective between bodies to hit both balls. - Technique: Streams two robot-mounted camera views into an HMD and maps the user's single arm motion to whichever robot body is currently selected or blended. - Try it: Film two tabletop mini-games with two phones, show the feeds side by side on one screen, and have one classmate use both hands to remotely direct two stand-in classmates at once. Twist: swap the two feeds roughly every 10 seconds. #### Morphing Identity — Shunichi Kasahara (2023) - Video: https://www.youtube.com/watch?v=rfmIuaV9Ny0 - Interaction: Face, Perception & Effects, Shared & Social - Platform & tech: Desktop, real-time face morphing, face landmark tracking - Idea: Slowly morph your face into the face of the person opposite you, live. - What it is: Two people face a screen where their live faces gradually morph into each other in real time, exploring where 'self' ends and 'other' begins. - Technique: Tracks facial landmarks on both live video feeds and warps and cross-dissolves the textures with a controllable blend ratio. - Try it: Make a filter in Snap Lens Studio or Effect House that blends your face with your deskmate's face according to a slider. Twist: set the blend ratio by how loudly each person speaks. #### ShadowClones — Shunichi Kasahara (2023) - Video: https://www.youtube.com/watch?v=UGXvYPcW3Jc - Interaction: Hands & Body, Perception & Effects - Platform & tech: Headset, VR, multiple perspectives, body tracking - Idea: One person splits into several shadow-like bodies and coordinates them across multiple viewpoints. - What it is: A single user controls several virtual bodies seen from multiple viewpoints; shadow-like cues keep track of where each body is so they can coordinate actions across perspectives. - Technique: Renders several avatar copies driven by one tracked body and overlays shadow projections of each clone to preserve body-space coordination across views. - Try it: Place three virtual clones in phone AR that copy your movements using body skeleton tracking, and pose so that all three clones hit a target at the same moment. Twist: give each clone a different motion delay. ### Studio DRIFT *Artist duo Lonneke Gordijn and Ralph Nauta* Dutch studio known for kinetic and drone light works; made one of the first gallery mixed-reality artworks for HoloLens. #### Fragile Future — Studio DRIFT (2005) - Video: https://www.youtube.com/watch?v=Jg8YH32s2Mo - Interaction: Tangible Objects, Voice & Sound, Perception & Effects - Platform & tech: Projection, dandelion seeds, LEDs, bronze circuits - Idea: Pairing the most fragile natural object with technology makes both feel precious; AR designers can let real, delicate things carry the digital layer. - What it is: Real dandelion seed heads, glued seed by seed onto tiny LEDs, form a glowing wall of lights connected by phosphor-bronze circuits. - Technique: Dandelion seeds are picked and glued by hand onto LEDs, which sit on three-dimensional bronze circuit frames that carry the current. - Try it: Scan a real dried flower or seed head with a phone, place it in AR, and give each seed a tiny light that detaches and floats away when you blow on the microphone. Twist: the seeds that fly away settle and glow on real surfaces nearby. #### Shylight — Studio DRIFT (2014) - Video: https://www.youtube.com/watch?v=LE9NPhGEUF4 - Interaction: Gaze & Attention, Perception & Effects - Platform & tech: Projection, silk, motors, LEDs - Idea: An object that opens and closes feels like it breathes and has moods; AR objects gain life from simple cycles of growth and retreat. - What it is: Silk lamps descend from a high ceiling and bloom open like flowers, then close and shrink back upward, performing a slow choreography. - Technique: Motorised cables lower each lamp while an internal mechanism unfolds layered silk skirts, driven by programmed sequences. - Try it: Hang three AR lamps from a real ceiling that descend and bloom when nobody looks at them and shyly close when the camera points at them. Twist: make one lamp brave so it keeps blooming under attention. #### Flylight — Studio DRIFT (2015) - Video: https://www.youtube.com/watch?v=oeJMBVEYweA - Interaction: Hands & Body, Spatial Mapping, Perception & Effects - Platform & tech: Projection, glass tubes, LEDs, sensors - Idea: Light alone can behave like a flock that notices you; AR responses can be as simple as brightness waves moving away from a person. - What it is: A flock of glass tube lights hangs in a dark room and flickers in waves like birds; when visitors come close, the lights react and move away from them. - Technique: Hundreds of glass tubes with LEDs are controlled by flocking software, and sensors track visitors so the pattern shifts away from them. - Try it: Place a grid of 100 AR light tubes over a room and run a flocking brightness pattern across them that moves away from the phone's position. Twist: when the viewer stands still for ten seconds, the flock slowly returns to them. #### In 20 Steps — Studio DRIFT (2016) - Video: https://www.youtube.com/watch?v=CFjtJBGlKM8 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Projection, glass rods, motors, motion capture - Idea: Many simple moving parts can draw a single motion through time; AR can break a movement into sampled positions to make it visible and study it. - What it is: Hanging glass rods move up and down in a wave, together tracing the shape of a bird's wing beat slowed into twenty steps. - Technique: Rows of glass rods on individually motorised cables are raised and lowered to positions sampled from a recorded wing motion. - Try it: Record a classmate's arm wave with body tracking, then play it back in AR as a curtain of 40 hanging rods whose heights follow the motion in 20 steps. Twist: freeze the curtain at any moment with a tap. #### Concrete Storm — Studio DRIFT (2017) - Video: https://www.youtube.com/watch?v=Qurs9QSz_KA - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Headset, HoloLens - Idea: A forest of virtual concrete pillars in HoloLens that shifts with your steps. - What it is: Commissioned by Microsoft and Artsy, a mixed-reality artwork on HoloLens where virtual concrete pillars appear in a real room and shift as the viewer walks between them. - Technique: HoloLens spatial mapping and head tracking place world-locked virtual pillars in the room, which animate and shift as the viewer's position changes. - Try it: Use HoloKit or phone AR to place a forest of virtual pillars in a room that sink or rise based on the viewer's distance. Twist: let the pillars' material slowly change from concrete to another material, hinting at another possible city. #### Drifter — Studio DRIFT (2017) - Video: https://www.youtube.com/watch?v=2QJ-Zn-yXs4 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, helium, concrete replica, drone mechanics - Idea: Taking weight away from a heavy object breaks our expectations more than any effect; AR can make the heaviest real-looking thing float. - What it is: A full-size concrete block appears to float and drift slowly through a gallery, rising and turning above visitors' heads. - Technique: The block is a lightweight replica filled with helium, likely steered by small internal motors and a control system so it follows a slow path. - Try it: Model a photoreal concrete block, anchor it in a real room at head height and let it drift slowly, with correct shadows on the floor and occlusion behind pillars. Twist: when two viewers stand under it, it sinks toward them as if tired. #### Franchise Freedom — Studio DRIFT (2017) - Video: https://www.youtube.com/watch?v=wCyNjt8TABk - Interaction: Location & City, Perception & Effects, Hands & Body - Platform & tech: Projection, drones, flocking algorithm, LEDs - Idea: Simple local rules produce a sky that feels alive; AR swarms should follow flocking rules rather than scripted paths to feel natural. - What it is: Hundreds of drones carrying lights fly as a flock over cities and deserts, swirling and splitting like starlings in a murmuration against the night sky. - Technique: Custom software, likely based on flocking algorithms derived from starling research, generates the drones' paths, which are then flown with positioning and synchronised LEDs. - Try it: Implement a Boids flock of 200 glowing points in AR above an open space and let the viewer's phone position act as a predator the flock avoids. Twist: add a second phone as a second predator and watch the flock split. #### Meadow — Studio DRIFT (2017) - Video: https://www.youtube.com/watch?v=jhUIFldl7h8 - Interaction: Spatial Mapping, Hands & Body, Perception & Effects - Platform & tech: Projection, silk, motors, choreography software - Idea: Flipping a natural scene upside down and putting it overhead gives visitors a new relation to it; AR scenes can be placed on ceilings, not just floors. - What it is: Large silk flowers hang upside down from the ceiling and slowly open and close in waves, like a field of blossoms reacting to the day. - Technique: Motorised mechanisms raise, lower and open each silk flower in coordinated sequences driven by software. - Try it: Detect the ceiling plane and grow an upside-down AR meadow on it whose flowers open in a wave that follows the viewer across the room. Twist: open only the flowers directly above people who are lying on the floor. #### Tree of Ténéré — Studio DRIFT (2017) - Video: https://www.youtube.com/watch?v=bFz6i3GqeBY - Interaction: Location & City, Shared & Social, Perception & Effects - Platform & tech: Projection, steel, LEDs, lighting control - Idea: One large, lit landmark in an empty landscape becomes a meeting place; AR can put a single glowing anchor in open space for people to gather around. - What it is: At Burning Man, a huge tree made of steel and 25,000 LED leaves glows in the desert, its light pulsing and changing colour at night. - Technique: A steel trunk carries thousands of LED leaves on branches, controlled by a lighting system that plays programmed and likely sound-reactive patterns. - Try it: Plant one large AR tree in a field that glows brighter as more phones gather within ten metres of it. Twist: each person's leaf colour is theirs, and the tree shows who has visited today. #### Social Sacrifice — Studio DRIFT (2022) - Video: https://www.youtube.com/watch?v=s9RAZ0PG6T8 - Interaction: Hands & Body, Spatial Mapping, Perception & Effects - Platform & tech: Projection, indoor drones, local positioning, swarm software - Idea: Being inside the swarm feels different from watching it; AR swarms can be placed around the viewer's body so they move through the group. - What it is: Inside a dark space at the Venice Biennale, a school of glowing drone 'fish' circles overhead and scatters when a predator shape approaches, surrounding visitors with the swarm. - Technique: Indoor drones with lights are flown with a local positioning system, following behaviour-based software that models a fish school reacting to a predator. - Try it: Build a headset or phone AR school of 150 fish that orbit the viewer at head height and part around their hand when they reach out. Twist: pick one student as the predator and let the school flee from their phone. #### Electric Sky — Studio DRIFT (2025) - Video: https://www.youtube.com/watch?v=uApCaBR8Gl0 - Interaction: Location & City, Voice & Sound, Perception & Effects - Platform & tech: Projection, drones, LEDs, generative software - Idea: A swarm can imitate weather instead of pictures; AR sky content does not have to show images, it can behave like natural phenomena. - What it is: At LUMA Arles, a swarm of lit drones performs over the landscape, moving between cloud-like masses and lines that evoke weather and electricity in the sky. - Technique: Drone trajectories are generated by DRIFT's software to mimic natural motion and flown as a light performance, likely with a soundtrack timeline. - Try it: Create an AR 'weather' swarm that drifts like a cloud, then snaps into lightning-like lines when the phone detects a loud sound. Twist: use real cloud cover from a weather API to decide how dense the swarm is. ### TU Graz Handheld AR — Daniel Wagner & Dieter Schmalstieg *Dieter Schmalstieg (professor, TU Graz; Studierstube) and Daniel Wagner (Christian Doppler Lab for Handheld AR; later Qualcomm Vuforia, DAQRI, Snap Spectacles)* Schmalstieg's Studierstube group and Wagner's Christian Doppler Laboratory for Handheld Augmented Reality ran AR on PDAs and feature phones years before smartphones, including the Invisible Train (2004) and the first real-time natural-feature tracking on a mobile phone (2008), which fed into Qualcomm's Vuforia. #### Expedition Schatzsuche (Museum AR Quest) — TU Graz Handheld AR — Daniel Wagner & Dieter Schmalstieg (2007) - Video: https://www.youtube.com/watch?v=cwSsu1myyV8 - Interaction: Play, Location & City, Information & UI - Platform & tech: Phone, Studierstube ES, PDA - Idea: Kids hunt for treasure in a museum with PDAs, and virtual characters and puzzles appear beside the exhibits. - What it is: A handheld AR treasure hunt in a Vienna museum: children explore exhibits with PDAs that reveal virtual characters and puzzles on markers. - Technique: Handheld PDAs track printed fiducial markers placed next to exhibits and overlay virtual characters and puzzle clues, with a game state machine sequencing the quest across the museum. - Try it: Put five image markers around a campus or teaching building and use 8th Wall or AR.js to build a linked treasure hunt: each marker shows a character and a clue pointing to the next spot. Twist: the clues can only be solved using details of real exhibits or the surroundings. #### Cows vs Aliens — TU Graz Handheld AR — Daniel Wagner & Dieter Schmalstieg (2008) - Video: https://www.youtube.com/watch?v=qVDw21itJF8 - Interaction: Play, Shared & Social, Tangible Objects - Platform & tech: Phone, Studierstube ES, marker tracking - Idea: A multiplayer AR board game played by walking around a table, where moving yourself is the gameplay. - What it is: A multiplayer handheld AR board game where players move around a tabletop map, making physical mobility and social interaction the core mechanic. - Technique: Each handheld device tracks the shared tabletop map with fiducial markers, so all players see the same registered game world, and players must physically move around the table to reach different areas. - Try it: Use a printed A3 map as an image target and build a two-player phone AR board game where some targets can only be seen and tapped from the other side of the table. Twist: make the two players' views block each other, forcing them to move their bodies and talk. #### Natural Feature Tracking at 30Hz on a Mobile Phone — TU Graz Handheld AR — Daniel Wagner & Dieter Schmalstieg (2008) - Video: https://www.youtube.com/watch?v=pEo9dbPlm6U - Interaction: Tangible Objects, Spatial Mapping - Platform & tech: Phone, SIFT, Ferns, Studierstube Tracker - Idea: The first live tracking of ordinary pictures on a phone, turning any poster into an AR stage. - What it is: The first real-time 6DOF tracking of ordinary printed images on a mobile phone, using modified SIFT and Ferns, so AR content can sit on any poster or book. - Technique: Keypoints detected in the camera frame are matched to a pre-trained database of the target image using simplified SIFT descriptors and Ferns classifiers, and the resulting correspondences give a robust 6DOF pose that is then refined by frame-to-frame patch tracking. - Try it: Implement ORB feature matching plus homography estimation with OpenCV (Python or OpenCV.js) to paste an image onto a poster in the phone camera view. Twist: test which poster patterns are hardest to track, and design a 'tracking-friendly' poster. #### Studierstube ES — TU Graz Handheld AR — Daniel Wagner & Dieter Schmalstieg (2009) - Video: https://www.youtube.com/watch?v=GUy-rmPrDYE - Interaction: Play, Tangible Objects - Platform & tech: Phone, Studierstube ES, Windows Mobile - Idea: Multiplayer AR games and virtual characters already running on phones in 2009. - What it is: Tech demo of TU Graz's handheld AR framework running multiplayer AR games, virtual characters and marker/natural feature tracking on 2009-era phones. - Technique: A lightweight fixed-point tracking library estimates 6DOF phone pose from square fiducial markers or natural features in real time on 2009-era mobile CPUs, and a scene-graph renderer draws characters over the camera image, with networking for multiplayer. - Try it: Use marker tracking in AR.js or 8th Wall to place an animated character on a printed card, and let two phones see the same character state in sync through a web page. Twist: cap the frame rate at 15fps and halve the resolution to feel the constraints of 2009, then simplify the design to match. #### Real-time Panoramic Mapping and Tracking on Phones — TU Graz Handheld AR — Daniel Wagner & Dieter Schmalstieg (2010) - Video: https://www.youtube.com/watch?v=xadks8UVwlA - Interaction: Spatial Mapping, Location & City - Platform & tech: Phone, panoramic SLAM - Idea: Turn the phone around once to build a panorama map, then track direction steadily against it. - What it is: The phone builds a 360-degree panorama of its surroundings as you turn and simultaneously tracks orientation against it for outdoor AR. - Technique: Assuming pure rotation, each camera frame is registered against a cylindrical panorama map being built incrementally, so the phone simultaneously extends the map and estimates its 3DOF orientation from it. - Try it: Use the phone gyroscope (DeviceOrientation API) to build a 'turn on the spot' panoramic AR web page: place virtual labels in all 360 degrees and calibrate north against the real scene. Twist: hide a single easter egg in one direction and lead the user to turn around and find it. #### Hedgehog Labeling — TU Graz Handheld AR — Daniel Wagner & Dieter Schmalstieg (2014) - Video: https://www.youtube.com/watch?v=XUQu4XAc2MI - Interaction: Information & UI - Platform & tech: Phone, Headset, view management - Idea: Arrange AR labels in 3D like hedgehog spines so they don't collide when you change viewpoint. - What it is: Markus Tatzgern, Denis Kalkofen, Raphael Grasset and Dieter Schmalstieg arrange external AR labels in 3D like hedgehog spines so they stay readable and stable as you move. - Technique: Labels are placed along 3D poles radiating from their anchors and their positions are optimized in 3D space (not per frame in screen space), which reduces overlap and avoids label jumping as the viewpoint changes. - Try it: Label 10 parts of an object (such as a bicycle model) in AR, implement both a "screen-space avoidance" layout and a "fixed 3D hedgehog" radial layout, and walk around the object to compare their stability. Twist: make the number of labels grow or shrink with the user's distance. ### Abhishek Singh *Indie AR developer & inventor (Peeqo, Super Mario AR); founder of Rara* NYC-based developer and former NYU ITP resident researcher who recreates pop-culture games and films as life-size AR experiences and builds playful gadgets like the GIF robot Peeqo. #### Laser Cat AR — Abhishek Singh (2017) - Video: https://www.youtube.com/watch?v=LYszXfkwYeQ - Interaction: Play, Spatial Mapping - Platform & tech: Phone, ARKit - Idea: The only thing to do is tease a virtual kitten with a laser pointer. - What it is: A minimalist AR game whose only goal is to entertain a virtual kitten on your floor with a laser pointer. - Technique: A raycast from the screen center onto the detected floor plane gives the laser dot position, and the cat's navigation logic steers toward that point. - Try it: Cast a ray from the screen center to the floor plane to get a red dot, and make a virtual kitten (or a simple capsule) chase it. Twist: make the pet afraid of the phone and run away when it comes close. #### Super Mario Bros Recreated as Life-Size AR Game — Abhishek Singh (2017) - Video: https://www.youtube.com/watch?v=QN95nNDtxjo - Interaction: Play, Location & City, Hands & Body - Platform & tech: Headset, HoloLens, Unity - Idea: The first level of Super Mario, rebuilt life-size in Central Park, with the player as Mario. - What it is: He rebuilt World 1-1 of Super Mario Bros at life size and played it in first person in Central Park, hitting blocks and stomping Goombas through a HoloLens. - Technique: A life-size level built in Unity is anchored to a world origin on HoloLens so inside-out SLAM tracking lets the player walk through it, with collisions between the head position and block colliders triggering hits. - Try it: Place a row of life-size question blocks on the ground with AR Foundation, and pop out a coin and add to the score when the head (camera) hits a block. Twist: replace the level with a custom map of a hallway on your campus. #### A Universal Remote Control Built Using AR — Abhishek Singh (2018) - Video: https://www.youtube.com/watch?v=Ia_v3Qz5KGA - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, ARKit, Raspberry Pi - Idea: Tap whatever appliance you see: a universal remote built with AR. - What it is: Tapping on real lamps, fans and TVs through the phone camera turns them on and off, using ARKit anchors wired to a Raspberry Pi. - Technique: Anchors placed on real appliances are raycast-tested on tap, and each hit sends an HTTP/MQTT request to a Raspberry Pi that switches the corresponding relay. - Try it: Place an AR anchor on a desk lamp and, when it is tapped, send an HTTP request so an ESP32 or smart plug switches the light. Twist: switch it on by staring at the lamp for two seconds instead (a ray from the screen center). #### An Augmented Reality Rube Goldberg Machine — Abhishek Singh (2018) - Video: https://www.youtube.com/watch?v=1tGVbuYEukM - Interaction: Spatial Mapping, Tangible Objects, Play - Platform & tech: Phone, ARKit, Unity - Idea: A Rube Goldberg chain reaction that runs across real and virtual objects. - What it is: A chain-reaction machine that mixes real household objects with virtual dominoes, ramps and balls, playable as a physics puzzle with a reset button. - Technique: Detected planes and hand-placed invisible colliders stand in for real objects, so Unity rigidbody physics lets virtual balls and dominoes react to real furniture. - Try it: Set up virtual dominoes and a ramp on a tabletop plane, add a hand-placed invisible collider to a real book, and let a ball roll down and knock over the whole row. Twist: have the chain reaction end by triggering a real sound or phone vibration. #### Street Fighter II: Real World Warrior Edition — Abhishek Singh (2018) - Video: https://www.youtube.com/watch?v=NpvZWFd0uRg - Interaction: Play, Shared & Social, Location & City - Platform & tech: Phone, ARKit, Unity - Idea: Turn Street Fighter into a two-player, life-size AR fight that actually takes to the streets. - What it is: A multiplayer AR remake of Street Fighter II in which two players on phones control full-size fighters battling on real streets. - Technique: Two phones share one AR coordinate space (likely via ARKit collaborative sessions or a shared anchor plus network sync) so both players see the same full-size fighters, whose inputs are replicated over the network. - Try it: Use Unity Netcode and AR Foundation so two phones share one anchor and each controls a block character trying to push the other out of a ring. Twist: attack with real body movements (shaking the phone) instead of buttons. #### The Ring Brought to Life in AR — Abhishek Singh (2018) - Video: https://www.youtube.com/watch?v=g_WBVi-bu9Q - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Phone, ARKit, Unity - Idea: Make Sadako from The Ring really crawl out of the TV at home. - What it is: Re-enacts the famous horror scene in which Samara crawls out of the TV and into the living room, anchored to a real television. - Technique: A rigged character animation is aligned to a real TV (likely by manual placement or image tracking of the screen), with an occlusion mask on the TV frame so the figure appears to emerge from inside the screen. - Try it: Use image tracking to recognize a picture on a laptop screen and have a character 'crawl out' of the screen onto the desk, with an occlusion material on the screen bezel hiding the part still inside. Twist: make something emerge from a fridge door or a drawer instead. #### Wile E. Coyote vs the Roadrunner in AR — Abhishek Singh (2018) - Video: https://www.youtube.com/watch?v=K4kLIhZzwCk - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, ARKit, Unity - Idea: Stage the cartoon chase between Wile E. Coyote and the Road Runner in a real room. - What it is: Looney Tunes characters chase each other around a real room, with Wile E. Coyote's cartoon traps set against actual furniture. - Technique: Characters follow scripted paths laid out on detected floor planes, with occlusion geometry roughly matched to real furniture so they can run behind it. - Try it: Draw a path on the floor plane, have two cartoon characters chase each other along it, and place an occluder at the sofa so they disappear behind it. Twist: have students record the path by walking a loop around the room with a phone. #### Bringing Tattoos to Life Using AR — Abhishek Singh (2019) - Video: https://www.youtube.com/watch?v=ojnAHSCNFLk - Interaction: Hands & Body, Tangible Objects, Perception & Effects - Platform & tech: Phone, ARKit, image tracking - Idea: Point at a tattoo and it comes alive and tells the story behind it. - What it is: Pointing the phone at people's tattoos animates them and plays the personal story behind each design. - Technique: Each tattoo is registered as an ARKit reference image, and when detected an animated overlay and audio clip are played on the tracked image anchor. - Try it: Use AR Foundation image tracking to recognize three hand-drawn stickers, playing the matching animation and a voice recording when they are stuck on an arm. Twist: each sticker tells a story from its owner's own life. #### Musical Chairs Against the Avengers in AR — Abhishek Singh (2019) - Video: https://www.youtube.com/watch?v=mqd_6gdXgpk - Interaction: Play, Hands & Body, Spatial Mapping - Platform & tech: Phone, ARKit, Unity - Idea: Fight life-size Avengers for real chairs. - What it is: A real game of musical chairs in which the other players are life-size virtual Avengers competing for real chairs. - Technique: Virtual characters are animated toward chair positions registered as anchors, with game logic comparing the player's camera position to those anchors when the music stops. - Try it: Mark the positions of three real chairs in a room, have virtual characters race for seats when the music stops, and check whether the player (the camera) reaches an empty chair first. Twist: race for a real rug or doorway instead of chairs. #### Bringing AR to the Streets of NYC — Abhishek Singh (2022) - Video: https://www.youtube.com/watch?v=zZeNe7Cs7Y0 - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, ARKit - Idea: Drop AR spectacles straight into the crowds on New York streets. - What it is: A short street intervention in which virtual objects appear among passers-by on New York sidewalks. - Technique: Objects are placed on detected ground planes in world-tracked space outdoors, relying on SLAM stability while real passers-by move through the camera view (likely with people occlusion). - Try it: Place a group of giant virtual objects on a campus sidewalk with people occlusion turned on, and film passers-by walking past. Twist: make the objects appear only when someone walks by. ### Cai Guo-Qiang (蔡国强) *Artist; gunpowder drawings and explosion events* Chinese artist from Quanzhou who works with gunpowder, fireworks and daytime smoke, from explosion drawings on paper to sky-scale events such as Sky Ladder and the Beijing 2008 Olympic footprints. #### Transient Rainbow — Cai Guo-Qiang (蔡国强) (2002) - Video: https://www.youtube.com/watch?v=goH0BJEYvM8 - Interaction: Location & City, Perception & Effects, Shared & Social - Platform & tech: Projection, fireworks, barges - Idea: A familiar sky shape made by unexpected means holds attention precisely because it disappears; short, one-time AR events can be more memorable than persistent content. - What it is: For MoMA's temporary move to Queens, a 300-metre rainbow of fireworks arched over the East River for about 15 seconds. - Technique: About a thousand multicolour shells were launched from barges along the river in a timed arc so their bursts formed a rainbow shape. - Try it: Make a WebXR rainbow that can be triggered only once per day at a fixed geolocation and lasts 15 seconds, then leaves a faint trace in everyone's app gallery. Twist: the rainbow only appears if at least three people point their phones at the spot at the same time. #### Footprints of History — Cai Guo-Qiang (蔡国强) (2008) - Video: https://www.youtube.com/watch?v=kiRyECW2UXk - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, fireworks, timed firing - Idea: A repeated mark that moves across a city makes an invisible giant walk; AR can use a sequence of simple anchored marks to suggest a presence far larger than the screen. - What it is: During the Beijing 2008 Olympic opening, 29 giant footprints made of fireworks stepped one after another along the city's central axis toward the National Stadium. - Technique: Fireworks shaped as footprints were fired from positions spaced along the axis and timed in sequence, so each step appears after the previous one. - Try it: Place a trail of glowing footprints across your campus with geospatial anchors, each one lighting up two seconds after the last so the invisible walker leads viewers to a meeting point. Twist: make the footprints change size so the walker seems to grow as it approaches. #### Black Ceremony — Cai Guo-Qiang (蔡国强) (2011) - Video: https://www.youtube.com/watch?v=8DZgNNnhvUM - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, daytime fireworks, coloured smoke - Idea: Fireworks do not need darkness: dark smoke shapes read best against a bright sky, a reminder that daylight AR content should use dark, high-contrast silhouettes instead of glow. - What it is: In the Qatari desert, daytime fireworks release black and coloured smoke in shapes such as a rainbow, a circle and a pyramid, drawing a silent ceremony against the bright sky. - Technique: Daytime shells loaded with black and dyed smoke compositions are fired in timed patterns so the smoke clouds form geometric figures that drift and fade. - Try it: Build a daytime AR sky piece in Lens Studio or WebXR where dark particle clouds burst into a circle, then a rainbow, then fade with the wind direction; test it outdoors at noon. Twist: compare a glowing version and a black-smoke version and vote which one reads better in sunlight. #### Gunpowder drawings — Cai Guo-Qiang (蔡国强) (2012) - Video: https://www.youtube.com/watch?v=c-QIj7E6CR8 - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, gunpowder, paper, stencils - Idea: The drawing is the record of a few seconds of energy; AR can likewise leave a permanent trace of a brief action, turning a moment into a mark in space. - What it is: In his studio, Cai lays paper stencils and trails of gunpowder on large sheets, covers them with cardboard and ignites them; the explosion leaves burnt drawings of trees, figures and landscapes. - Technique: Gunpowder of different grain sizes is spread through stencils, weighted with boards and ignited, so the pressure of the burn controls how dark and sharp the marks become. - Try it: Build a phone AR 'explosion brush': students draw a path on a real wall with their finger, then trigger a burn that leaves a scorched decal texture where the path was. Twist: the scorch mark spreads further where the drawn line was thicker, like more powder. #### Mystery Circle — Cai Guo-Qiang (蔡国强) (2012) - Video: https://www.youtube.com/watch?v=kAECemP1xWY - Interaction: Spatial Mapping, Location & City, Perception & Effects - Platform & tech: Projection, rockets, fireworks, facade frame - Idea: A building facade can act as a launch surface, not just a screen; AR content that emerges from and returns to real architecture feels more rooted than content floating in front of it. - What it is: For the opening of his exhibition at MOCA Los Angeles, about 40,000 rockets and fireworks mounted on the museum facade fired in sequence, drawing crop-circle patterns in light and smoke. - Technique: Rockets were fixed to a patterned frame on the facade and fired in timed rings so the trails and smoke formed spirals and circles. - Try it: Detect a building wall with plane detection and launch AR rockets from a circular pattern on it, whose trails leave a spiral that stays on the facade for a minute. Twist: let the pattern be drawn by the audience on a tablet before launch. #### One Night Stand — Cai Guo-Qiang (蔡国强) (2013) - Video: https://www.youtube.com/watch?v=8vf0mJHR01k - Interaction: Location & City, Performance, Shared & Social - Platform & tech: Projection, fireworks, river barges - Idea: A city river is a ready-made stage with a long axis and a reflecting floor; AR events can use existing urban lines like rivers and bridges to organise the audience's view. - What it is: During Nuit Blanche in Paris, fireworks along the Seine in front of the Musée d'Orsay stage an explosive love story, with shapes rising from the river and the bridges. - Technique: Fireworks are placed on barges and along the banks and fired in a choreographed sequence so the reflections on the water double every burst. - Try it: Pick a river, pond or fountain and create an AR show that uses the water plane: every burst you place above it spawns a mirrored copy on the surface. Twist: let two phones control two lovers' fireworks that must meet in the middle of the water. #### Elegy: Explosion Event for the Opening of The Ninth Wave — Cai Guo-Qiang (蔡国强) (2014) - Video: https://www.youtube.com/watch?v=G5O7VbNmfWE - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, daytime fireworks, smoke shells - Idea: An effect in the sky can carry grief instead of celebration; AR designers can use pace, colour and fading to set an emotional tone, not only to impress. - What it is: Over the Huangpu River in Shanghai, daytime fireworks release waves of black, grey and white smoke, a mourning piece for the opening of the exhibition The Ninth Wave. - Technique: Daytime smoke shells in monochrome tones are fired in slow, successive layers above the river so each wave drifts before the next one arrives. - Try it: Design a 90-second AR sky elegy over a real street or river: grey smoke layers appear one by one, each staying longer than the last, with no sound. Twist: let each viewer tap once to release a layer for someone they miss. #### Sky Ladder — Cai Guo-Qiang (蔡国强) (2015) - Video: https://www.youtube.com/watch?v=4t2jLOcbZh0 - Interaction: Location & City, Perception & Effects, Gaze & Attention - Platform & tech: Projection, fireworks, fuses, helium balloon - Idea: A single vertical line that climbs away from the viewer turns the whole sky into a destination; in AR, one tall anchored object can give an empty horizon a direction and a story. - What it is: A 500-metre ladder of fuses and fireworks, lifted by a giant helium balloon, burns upward from a fishing village harbour into the dawn sky for about two and a half minutes. - Technique: Quick-burning fuses and gold fireworks are fixed to a ladder frame hung from a helium balloon and ignited from the bottom so the flame travels upward in sequence. - Try it: Anchor a 300-metre virtual ladder at a real outdoor spot with a geospatial anchor and animate a flame that climbs it rung by rung over 60 seconds, so students must tilt their phones upward to follow it. Twist: let the burning speed follow the loudness of the crowd's cheering. #### Sleepwalking in the Forbidden City — Cai Guo-Qiang (蔡国强) (2021) - Video: https://www.youtube.com/watch?v=BIBFvH-Whqs - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Headset, VR, HTC VIVE, particle simulation - Idea: Virtual space lets an artist do the fireworks that are forbidden in reality; AR and VR can stage impossible events in sensitive places without touching them. - What it is: In this VR work made with HTC VIVE Arts, the viewer floats through a dreamlike Forbidden City at night while fireworks and gunpowder explosions bloom around the palace roofs. - Technique: Likely a real-time VR scene built from 3D scans and models of the palace, with particle-based firework simulations and a guided floating camera path. - Try it: Scan a small landmark on campus with a photogrammetry app, place the model in a WebXR scene and let viewers trigger fireworks from its roof that would never be allowed in real life. Twist: switch to passthrough AR so the virtual fireworks explode above the real landmark. #### When the Sky Blooms with Sakura — Cai Guo-Qiang (蔡国强) (2023) - Video: https://www.youtube.com/watch?v=I2uIi0GT8Qg - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, daytime fireworks, coloured smoke - Idea: The same effect means something different in a place with memory; site-specific AR gains weight when its imagery answers the history of the exact location. - What it is: On Yotsukura Beach in Iwaki, a coast struck by the 2011 tsunami, daytime fireworks release pink and white smoke blossoms that open over the sea like cherry trees. - Technique: Daytime shells with pink, white and green smoke compositions are fired in layered clusters so the smoke forms blossom shapes and branches. - Try it: Choose a place on campus with a story (an old tree, a closed building) and grow an AR smoke tree above it whose blossom colour comes from that story; show it to someone who knows the place. Twist: blossoms fall and gather on the ground only where people stand still. ### Christian Sandor *Professor, Université Paris-Saclay; formerly led the Magic Vision Lab (UniSA), NAIST and the XR Lab at City University of Hong Kong* Trained at TU Munich, he built the Magic Vision Lab in Adelaide (BurnAR, AR X-ray, space-distorting AR), then co-directed AR research at NAIST and CityU Hong Kong. He calls the goal 'indistinguishable AR'. #### Visuo-Haptic Augmented Reality — Christian Sandor (2007) - Video: https://www.youtube.com/watch?v=63Cn6Te3TgU - Interaction: Tangible Objects, Drawing & Making, Hands & Body - Platform & tech: Headset, haptic device, video see-through HMD - Idea: See and touch the same virtual object in the same place. - What it is: Wearing a video see-through headset, a user paints on a virtual object while a haptic stylus lets them feel its surface exactly where they see it. - Technique: Co-locates a PHANToM-style haptic device with graphics rendered in a video see-through HMD, which outperformed half-mirror setups in the study. - Try it: Anchor a virtual pot in phone AR on top of a real cup, then paint the pot on screen while your other hand feels the cup. Twist: offset the virtual pot by a few centimetres and note when touch and sight stop matching. #### ARWeather — Christian Sandor (2008) - Video: https://www.youtube.com/watch?v=8bkMiXiYUnk - Interaction: Perception & Effects, Location & City - Platform & tech: Headset, Wearable, Tinmith, particle system, GPS - Idea: Change the weather around one person, wherever they walk. - What it is: A person wearing an outdoor AR backpack walks freely through simulated rain, snow or hail falling over the real campus. - Technique: Renders particle-based precipitation in world coordinates on the Tinmith wearable system, tracked by GPS and head orientation sensors. - Try it: Build a phone AR effect that makes snow fall only inside a two-metre circle around a placed anchor, then walk in and out of it outdoors. Twist: add sound that grows louder as you enter. #### Space-Distorting AR — Christian Sandor (2009) - Video: https://www.youtube.com/watch?v=LoD6vhR3Ff4 - Interaction: Location & City, Perception & Effects, Information & UI - Platform & tech: Headset, mobile mixed reality, space distortion - Idea: Bend space in AR to reveal what is behind or beside you. - What it is: To find points of interest hidden behind buildings or outside the view, the mobile AR scene bends like a fisheye or folds space so distant or occluded places swing into sight. - Technique: Applies egocentric radial distortion and melting-style deformations to a 3D model of the surroundings and its POIs in mobile mixed reality. - Try it: Take a phone 360 photo of the schoolyard, map it into a tiny-planet view, and mark three hidden spots so classmates can find them faster. Twist: compare with a normal flat map. #### AR X-Ray Vision Based on Visual Saliency — Christian Sandor (2010) - Video: https://www.youtube.com/watch?v=du72tgJxrPY - Interaction: Portals & Worlds, Perception & Effects, Spatial Mapping - Platform & tech: Headset, visual saliency, X-ray visualization - Idea: See through walls, but keep just enough of the wall so your brain believes the depth. - What it is: A user looks at a building and sees through its wall into the hidden room behind, while the most noticeable parts of the wall, like edges and signs, stay visible for context. - Technique: Computes a visual saliency map of the occluding surface and uses it to decide which occluder pixels to keep over the rendered hidden content. - Try it: Photograph a door and the room behind it, then build a phone AR peephole that blends the hidden room over the live door while keeping the door's edges. Twist: let the hole follow a flashlight app's beam. #### BurnAR — Christian Sandor (2012) - Video: https://www.youtube.com/watch?v=1o_t1NnSYp4 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Headset, video see-through HMD, hand segmentation, particle effects - Idea: Convincing AR on your own body can trigger real bodily sensations. - What it is: Looking at their own hands through a headset, users see flames and smoke rising from their skin; some report an involuntary feeling of heat although nothing is warm. - Technique: Segments the user's hands in a video see-through HMD and overlays real-time volumetric fire and smoke that follow the hand motion. - Try it: Apply a phone hand-tracking AR effect that puts fire or ice on the hands, then ask classmates to rate warmth or cold on a scale while blindfolded partners compare. Twist: add a matching crackling sound and see if ratings rise. #### ClonAR — Christian Sandor (2012) - Video: https://www.youtube.com/watch?v=Jb2LPB_IPOs - Interaction: Drawing & Making, Tangible Objects, Spatial Mapping - Platform & tech: Headset, Desktop, KinectFusion, haptics, signed distance fields - Idea: Scan, touch and remix a real object in AR in one continuous loop. - What it is: A real object is scanned with a Kinect, and its clone appears in AR where the user can feel and sculpt it with a haptic pen before sending it off to be printed. - Technique: Uses KinectFusion signed distance fields directly for both haptic rendering and GPU raymarched visuals, skipping mesh conversion. - Try it: Scan a small object with a phone 3D-scan app, place the copy next to the original in AR, and use a sculpting app to redesign it. Twist: print or build the redesigned version from clay and compare. #### AR Yōkai — Christian Sandor, Hirokazu Kato (2017) - Video: https://www.youtube.com/watch?v=sE8V2RdES4A - Interaction: Location & City, Play, Spatial Mapping - Platform & tech: Headset, HoloLens, 3D scanning, situated storytelling - Idea: Let historical paintings haunt the very building they belong to. - What it is: Visitors wearing HoloLens walk through Shunkoin, an old Kyoto temple, and meet nine yōkai spirits from classical Japanese paintings, placed into the scanned rooms as part of a shared ghost story. - Technique: 3D scans the temple interior to situate animated yōkai models with occlusion and shared placement across multiple HoloLens users. - Try it: Pick a local legend, model or collect one creature, and hide it with phone AR in the part of school where the story would happen; guide a small group through it. Twist: make the creature appear only when the lights are dimmed. ### Jason Orlosky *Associate Professor, Augusta University; director of the AR/VR lab; runs the JEOresearch channel* Did his PhD with Kiyoshi Kiyokawa at Osaka University on head-mounted text and vision augmentation (Fisheye Vision, ModulAR, Halo Content). Now builds eye-tracked AR tools, open-source eye trackers and accessibility demos. #### Towards Intelligent View Management — Jason Orlosky, Kiyoshi Kiyokawa (2013) - Video: https://www.youtube.com/watch?v=T8SzuLh0eYg - Interaction: Information & UI, Gaze & Attention - Platform & tech: Headset, video see-through HMD, view management - Idea: Learn where people want text to sit in their view before letting software place it. - What it is: While walking through streets with a video see-through display, users drag text panels where they prefer them; the recordings reveal clusters of 'safe' places that an automatic system can learn. - Technique: Records manual placements of text overlays in mobile video see-through AR and clusters them against image features such as dark, uniform regions to derive placement rules. - Try it: Walk a route with a phone camera app and a sticky note on the screen; at ten stops move the note to where it bothers you least and screenshot it, then map the patterns as a class. Twist: repeat while talking to a partner and see where the note moves. #### Fisheye Vision — Jason Orlosky, Kiyoshi Kiyokawa (2014) - Video: https://www.youtube.com/watch?v=Twowop1Xf2E - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Headset, Oculus Rift, fisheye camera, video see-through - Idea: Give people a wider-than-human field of view and let them keep a normal center. - What it is: A modified Oculus Rift with a fisheye camera gives the wearer about 180 degrees of vision while keeping the center undistorted, plus a 'Matrix mode' that shows the world in green code. - Technique: Warps a fisheye video feed so the central region keeps natural magnification while the periphery is compressed into the display edges of a video see-through Rift. - Try it: Clip a cheap fisheye lens onto a phone in a cardboard viewer and play catch or navigate a hallway for five minutes. Twist: compress only the left and right edges and test whether people notice someone approaching from the side sooner. #### Halo Content — Jason Orlosky, Kiyoshi Kiyokawa (2015) - Video: https://www.youtube.com/watch?v=9Vbs1oLpk38 - Interaction: Face, Information & UI, Shared & Social - Platform & tech: Headset, face detection, view management, HMD - Idea: AR notifications should politely step aside when a real person appears. - What it is: Emails, messages and notifications in a head-worn display drift out of the way whenever a person walks into view, forming a halo around their face instead of covering it. - Technique: Detects faces and bodies in the camera feed and moves overlay elements with a simple force-directed layout so they orbit around, rather than occlude, the conversation partner. - Try it: Use a phone face-tracking AR app to attach three floating notification cards that orbit a detected face without covering it, then hold a two-minute conversation through the phone. Twist: make urgent cards come closer and quiet ones fade out. #### ModulAR — Jason Orlosky, Kiyoshi Kiyokawa (2015) - Video: https://www.youtube.com/watch?v=nrCALSZQhzk - Interaction: Gaze & Attention, Perception & Effects - Platform & tech: Headset, eye tracking, video see-through HMD, camera modules - Idea: Vision superpowers as plug-in lenses you switch on with your eyes. - What it is: A video see-through headset with swappable camera-lens modules lets the wearer switch between telescopic zoom and wide-angle vision just by where they look, using built-in eye tracking. - Technique: Combines interchangeable camera-lens modules on an HMD with eye tracking, so gaze gestures trigger on-demand zoom or field-of-view expansion in the video feed. - Try it: Build a phone 'binocular' mode in a cardboard viewer that zooms the camera when you tilt your head up and returns to normal when you tilt down; test it on a board at the back of the room. Twist: trigger the zoom by blinking twice using front-camera face tracking. #### Augmented Reality Language Learning — Jason Orlosky (2019) - Video: https://www.youtube.com/watch?v=4SkkWc396J4 - Interaction: Information & UI, Spatial Mapping - Platform & tech: Headset, object detection, HoloLens, semantic labeling - Idea: Turn your own home into a vocabulary flashcard deck. - What it is: Walking through a room with a headset, the learner sees everyday objects automatically recognised and labelled with their foreign-language words, anchored in place. - Technique: Runs object detection on camera frames and registers each detected label to 3D positions in the headset's spatial map so labels stay attached to objects. - Try it: Use a phone object-detection or image-tracking AR tool to label ten classroom objects in a second language, then quiz a partner by hiding the labels. Twist: show only the first letter and reveal the word when the learner says it aloud. #### Controlling Air Conditioning via Augmented Reality — Jason Orlosky (2021) - Video: https://www.youtube.com/watch?v=i3OctJhaXKI - Interaction: Information & UI, Hands & Body, Spatial Mapping - Platform & tech: Headset, Phone, flow visualization, gesture control, IoT - Idea: Make invisible airflow visible, then let people grab and steer it. - What it is: The invisible airflow of a directional air conditioner is drawn as streams in AR, and the user steers it with touch gestures to aim cool air where people sit. - Technique: Likely simulates the AC's flow field from vane angles and renders it as animated particle streams registered to the unit, sending gesture edits back as control commands. - Try it: Hang ribbons in front of a desk fan, film them, then build a phone AR scene that draws matching animated particle streams from the fan; let classmates 'drag' the stream to aim it. Twist: visualise Wi-Fi strength or sound instead of air. #### Eye-tracked Swipe Keyboard — Jason Orlosky (2023) - Video: https://www.youtube.com/watch?v=I4Oj4WavKO8 - Interaction: Gaze & Attention, Drawing & Making - Platform & tech: Headset, Varjo XR-3, eye tracking, video see-through - Idea: Swipe-typing, but the finger is your gaze. - What it is: A keyboard floats in video see-through AR and the user types whole words by sweeping their gaze across the letters, like swiping on a phone, with no hands at all. - Technique: Records the gaze path over a virtual keyboard on a Varjo XR-3 and matches it against word templates with approximate path matching. - Try it: Print a large keyboard, have one student trace words with their eyes while a partner films their face with a phone, then try to decode the words from the video. Twist: design a shorter layout that makes gaze paths less ambiguous. #### HazARdSnap (AR Cyclocomputers) — Jason Orlosky, Kiyoshi Kiyokawa (2023) - Video: https://www.youtube.com/watch?v=34BXp95wq3g - Interaction: Gaze & Attention, Information & UI, Location & City - Platform & tech: Headset, eye tracking, object tracking, HMD - Idea: Attach your dashboard to the thing you should be watching anyway. - What it is: A cyclist glances at the rider ahead and speed and route data snap onto that rider's back, so the eyes never leave the road and a sudden stop is still visible. - Technique: Uses eye gaze in a head-mounted display to decide when to show data and anchors it to the tracked cyclist in front instead of a fixed screen position. - Try it: Build a phone AR prototype that pins a speed readout to a person or object tracked in the camera view, then walk behind a classmate and compare with reading a phone held at waist height. Twist: show the data only while you look at the target for more than one second. #### EyeShadows — Jason Orlosky (2024) - Video: https://www.youtube.com/watch?v=WoGK_Xf9cdU - Interaction: Gaze & Attention, Information & UI - Platform & tech: Headset, eye tracking, Unity, open source - Idea: Pull a copy of what you looked at into your periphery so selecting by eye is easy. - What it is: Glance at an object and a 'shadow' copy of it appears in the periphery; looking at the copy selects it or opens its settings, making eye-only interaction fast and less error-prone. - Technique: Spawns peripheral virtual copies of gazed objects and uses a gaze transition onto the copy as a confirm action, avoiding accidental 'Midas touch' selections. - Try it: Using phone face or eye tracking in a web AR demo, show a small copy of a gazed icon in a screen corner and select it by looking there for half a second. Twist: let the copy offer three options placed in different corners. #### Head-tracked Subtitles — Jason Orlosky (2024) - Video: https://www.youtube.com/watch?v=Oc91ouDpgio - Interaction: Voice & Sound, Information & UI, Face - Platform & tech: Headset, Unity, Vosk, Llama 3, HTC Vive Pro - Idea: Caption the real world the way films are captioned. - What it is: Everything people around you say appears as live subtitles in a video see-through headset, following your head so conversation stays readable in noise or for hard-of-hearing users. - Technique: Transcribes speech locally with Vosk, tidies it with the Llama 3 language model, and renders the text in Unity on a modified video see-through HTC Vive Pro. - Try it: Use a phone's live-caption or speech API in a web AR page to float captions under a detected face, then test it in a noisy hallway. Twist: colour each speaker differently and add live translation. ### Keiichi Matsuda *Designer and filmmaker; founder of Liquid City* Architect-trained designer and filmmaker whose speculative AR films (Augmented City, HYPER-REALITY, Merger) became the reference images of what an overlaid city could feel like. He later led design at Leap Motion and founded the XR/AI studio Liquid City. #### Augmented (hyper)Reality: Domestic Robocop — Keiichi Matsuda (2010) - Video: https://www.youtube.com/watch?v=fSfKlCmYcLc - Interaction: Information & UI, Perception & Effects - Platform & tech: Headset, After Effects, design fiction - Idea: Turn every surface at home into ads and interfaces to rehearse a life drowning in information. - What it is: A first-person walk through a kitchen where every surface is covered in AR labels, recipes, ads and reward points; the film imagines how an 'augmented' home would be monetised. - Technique: Design-fiction compositing: a handheld first-person shot is camera-tracked in post (After Effects / 3D tracker) so UI labels are pinned to kitchen surfaces; a live version can be approximated with AR plane and image anchors. - Try it: Use AR Foundation plane detection or 8th Wall to cover the tables and fridge door of a kitchen or dorm with labels, ads and points pop-ups, and shoot a 30-second first-person video. Twist: every label demands that the user 'pay' or 'watch an ad' before it can be closed. #### Augmented City 3D — Keiichi Matsuda (2010) - Video: https://www.youtube.com/watch?v=3TL80ScTLlM - Interaction: Location & City, Information & UI, Portals & Worlds - Platform & tech: Headset, 3D film, design fiction - Idea: The city becomes subscribable, customizable AR layers, so everyone sees a different city. - What it is: A stereoscopic film of a walk through a city whose AR layers can be customised, shared and hijacked, questioning who designs the augmented urban space. - Technique: Stereoscopic 3D film with camera-tracked CG layers composited onto street footage; in a live build this maps to geolocated or VPS anchors with per-user layer sets. - Try it: Choose a campus route and build two different sets of AR layers in WebXR or Unity (say a 'commercial' version and a 'community' version), switched with a button. Twist: add a 'hijacked' layer that another classmate remotely stuffs with content in your city. #### CELL (with James Alliban) — Keiichi Matsuda, James Alliban (2011) - Video: https://www.youtube.com/watch?v=lfzNF7igHzY - Interaction: Hands & Body, Information & UI - Platform & tech: Projection, Kinect, openFrameworks - Idea: A digital mirror that tracks bodies and slaps online identity tags on every passerby in real time. - What it is: An interactive installation where a depth camera tracks visitors and attaches floating identity tags drawn from social media to their bodies on a large mirror-screen. - Technique: A Kinect depth camera segments and tracks each visitor's skeleton, and openFrameworks renders text tags that follow the tracked head/torso positions on a mirror-like screen. - Try it: Track several people in a webcam feed with MediaPipe Pose on a web page and stick randomly scraped online tags (hashtags, nicknames, ad slogans) above each person's head. Twist: make the tags change with a person's stance or how long they stay, and discuss "how algorithms label you". #### HYPER-REALITY — Keiichi Matsuda (2016) - Video: https://www.youtube.com/watch?v=YJg02ivYzSs - Interaction: Information & UI, Perception & Effects, Location & City - Platform & tech: Headset, design fiction, 3D compositing - Idea: The dizziness and fragility of a reality turned into games and ads once AR takes over your whole view. - What it is: A POV short in Medellín where a woman's view is saturated with AR ads, game points and assistant UI; when her identity is hacked the overlay collapses. - Technique: POV footage camera-tracked and composited with dense 3D UI layers; can be approximated live with phone world tracking plus a large number of world- and screen-space UI elements that collapse on an event. - Try it: Fill a hallway in phone AR with ads, game points and assistant prompts, and shoot a first-person video. Twist: design a moment of "identity theft" when every interface crashes at once or turns into a stranger's interface. #### Merger — Keiichi Matsuda (2018) - Video: https://www.youtube.com/watch?v=SqW2dEkiD-Y - Interaction: Information & UI, Hands & Body - Platform & tech: Headset, 360 video, design fiction - Idea: A future office worker, surrounded by holographic work screens, fights off the fear of being replaced by AI with hand gestures. - What it is: A 360-degree film of an office worker whose entire workspace is a floating AR interface, controlled by hand and voice, as she fights to stay relevant against automation. - Technique: 360-degree rendered film of a head-centred UI sphere; a live equivalent uses a headset with hand tracking and floating panels arranged on a body-anchored cylinder. - Try it: Build a multi-window work interface that surrounds you in HoloKit or WebXR, switching tasks with gestures or pinches. Twist: keep adding windows over time until the user must 'give up' some tasks to see what is in front of them. #### Reality Channels (Liquid City) — Keiichi Matsuda (2022) - Video: https://www.youtube.com/watch?v=0tyswdePMmA - Interaction: Portals & Worlds, Location & City - Platform & tech: Phone, Headset, Unity, Niantic Lightship - Idea: Switch the city's AR skins like changing TV channels. - What it is: A prototype of the 'real-world metaverse' in which users switch between channels that re-skin the same street with different AR layers, characters and information. - Technique: Shared world-scale AR using Niantic Lightship meshing/VPS so several content layers are anchored to the same street and swapped at runtime like channels. - Try it: Make three AR channels on the same tabletop or wall (for example fairy tale, surveillance and advertising) and switch between them with a swipe. Twist: let two phones see different channels and discuss the different realities in a shared space. #### Agents (Liquid City) — Keiichi Matsuda (2023) - Video: https://www.youtube.com/watch?v=bkKv2AHpn8E - Interaction: Information & UI, Voice & Sound - Platform & tech: Headset, design fiction, AI - Idea: Give AI assistants a body as characters living in AR glasses, and think about how people relate to agents. - What it is: A short film where AI agents appear as characters in AR glasses and negotiate a person's day, asking what kind of relationship we want with AI in mixed reality. - Technique: Design-fiction film where AI agents are rendered as characters composited into glasses POV; a prototype would pair an LLM dialogue loop with world-anchored animated avatars. - Try it: Design two or three 'AI agent characters' for your day, place them in your room with phone AR, and have an LLM generate dialogue in which they compete for your attention. Twist: let one character turn down the other characters' requests on your behalf. #### Life Stories × World Labs: Night Lake — Keiichi Matsuda (2025) - Video: https://www.youtube.com/watch?v=w5fp-otQhQU - Interaction: Portals & Worlds, Voice & Sound - Platform & tech: Headset, Web, World Labs Marble, Gaussian splatting, LLM, Life Stories - Idea: A spoken memory becomes a place other people can walk through. - What it is: A true memory collected by Liquid City's AI interviewer is turned by their AI engine into descriptions, then images, and finally an explorable 3D Gaussian-splat world made with World Labs' Marble. - Technique: An LLM pipeline rewrites the interview into scene descriptions, an image model renders them, and a world model (Marble) lifts the images into navigable splats. - Try it: Interview a classmate for two minutes about a place they miss, turn the transcript into a scene prompt, generate an image and then a 3D world, and view it in WebXR together. Twist: the interviewee must guide the others through it without seeing it first. #### Project Jade - Spatial Agents (Liquid City) — Keiichi Matsuda (2025) - Video: https://www.youtube.com/watch?v=ZwyGUZIoa8Q - Interaction: Voice & Sound, Spatial Mapping, Information & UI - Platform & tech: Headset, Snap Spectacles, LLM - Idea: An AI companion that sees what you see and answers you out in space. - What it is: A spatial AI agent for AR glasses that sees what the wearer sees and responds with embodied, placed UI, shown at Snap Lens Fest 2025. - Technique: Egocentric camera frames are sent to a multimodal LLM, and responses are rendered as UI panels anchored near the referenced object using world tracking and raycasts. - Try it: Build a phone AR prototype that sends the current view to a vision-language model and pins its answer as a label next to the matching object in the frame. Twist: let the agent reply only with icons and gestures, never text, and test whether nonverbal spatial answers can be understood. #### Wisp World (Liquid City) — Keiichi Matsuda (2025) - Video: https://www.youtube.com/watch?v=CtP9FERiL0w - Interaction: Voice & Sound, Play, Spatial Mapping - Platform & tech: Headset, Snap Spectacles, Apple Vision Pro, LLM - Idea: An AI and AR game in which you go on adventures with a talking little spirit in your real room. - What it is: An AR/AI game for Snap Spectacles and Vision Pro where the player talks with a small spirit character, Wisp, who explores and comments on the real room. - Technique: Headset scene understanding (meshing/semantic labels) plus a camera-to-LLM loop lets a small animated character navigate real surfaces and speak about recognised objects. - Try it: Place a small character in Unity AR that moves along detected planes, and use a photo plus a vision-language model to have it comment on an object in the room. Twist: give the character a fixed personality (timid or arrogant) that shows in both its comments and its walking path. ### Kyle McDonald *Artist working with code; openFrameworks contributor* Media artist working with computer vision, machine learning and faces; co-created FaceOSC tools, real-time face substitution and many collaborative installations. #### Face Substitution — Kyle McDonald, Arturo Castro (2011) - Video: https://vimeo.com/29348533 - Interaction: Face, Perception & Effects - Platform & tech: Desktop, openFrameworks, FaceTracker - Idea: The earliest live face swap: your face on the webcam is seamlessly replaced with someone else's. - What it is: Widely cited as the first realistic real-time face swap: another person's face is fitted and blended onto yours live from a webcam. - Technique: FaceTracker fits a deformable landmark mesh to both the source and live face, the source texture is warped onto the live mesh, and color-transfer plus Poisson-style blending hides the seam. - Try it: Use the triangle mesh from MediaPipe Face Mesh to map the face of a figure from a famous painting onto your own, apply simple color matching, and swap faces in real time. Twist: swap only half the face to make a mirror that is 'half you, half your ancestor'. #### Scramble Suit — Kyle McDonald, Arturo Castro (2011) - Video: https://vimeo.com/29391633 - Interaction: Face, Perception & Effects - Platform & tech: Desktop, openFrameworks, FaceTracker - Idea: A 'scramble suit' from A Scanner Darkly: your face keeps being swapped for different people. - What it is: Inspired by Philip K. Dick's A Scanner Darkly: the live face is continuously re-substituted with many different faces so the wearer becomes impossible to describe. - Technique: The same FaceTracker mesh-warp-and-blend pipeline as Face Substitution, cycling through a large database of source faces every few frames so identity never stabilises. - Try it: Collect 20 front-facing photos from the class and use MediaPipe Face Mesh to give the person on camera a new face every half second, making an 'indescribable mask'. Twist: speed up the swapping as more people gather nearby, and discuss surveillance and anonymity. #### Light Leaks — Kyle McDonald, Jonas Jongejan (2013) - Video: https://vimeo.com/66167082 - Interaction: Projection, Perception & Effects, Spatial Mapping - Platform & tech: Projection, openFrameworks, structured light - Idea: Dozens of disco balls reflect projector light and turn a whole room into a space of drifting light points. - What it is: With Jonas Jongejan: dozens of mirror balls reflect light from projectors calibrated with computer vision, filling a whole room with moving points of light. - Technique: Structured-light (Gray code) scanning maps each projector pixel to where its reflection lands in the room via the mirror balls, and the content is authored in that reflected space so light points move coherently. - Try it: Aim a projector at a disco ball or a few mirror pieces, light it region by region in TouchDesigner, note where each region's light spot lands in the room, and then compose an animation of wandering light points. Twist: make the light points "dance" on the wall to the rhythm of a song. #### Sharing Faces — Kyle McDonald (2013) - Video: https://vimeo.com/96549043 - Interaction: Face, Shared & Social - Platform & tech: Desktop, openFrameworks, face tracking - Idea: A cross-border mirror: you see the face of someone in another country with your same expression and pose. - What it is: Linked mirror installations in Anyang, Korea and Yamaguchi, Japan: your expression and pose are matched in real time with photos of someone who once stood at the other site. - Technique: Face tracking extracts head pose and expression parameters, which are used as a nearest-neighbour query into a growing database of photos captured at the remote site. - Try it: Use MediaPipe to extract head orientation and how wide the mouth is open, then find and display in real time the most similar pose from a photo library another group of classmates shot in advance. Twist: link two classrooms so you can only "reply" to the other side with their past photos. #### Highsight — Kyle McDonald (2015) - Video: https://vimeo.com/144061990 - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Headset, Oculus Rift, live camera - Idea: Put on a headset and see through a real camera on a wire as it plunges through a model and into the crowd. - What it is: A 'virtual bungee jump through a real space': harnessed visitors wear an Oculus Rift showing a live camera on a wire as it drops through a diorama and into the crowd around them. - Technique: A live camera on a cable-guided rig streams low-latency video into a head-mounted display, so the viewer's vision is displaced into a moving remote viewpoint within the same physical space. - Try it: Tie a phone to a rope or pole as a "flying camera", stream its video over WebRTC to a classmate wearing HoloKit, and let them watch themselves from the ceiling. Twist: the viewer can only use this viewpoint to find an object hidden somewhere in the room. ### Lucas Rizzotto *XR creator, inventor and YouTuber (Lucas Builds the Future); founder of Where Thoughts Go* Brazilian-born immersive designer and inventor behind the 'Lucas Builds the Future' channel and AR House in LA, known for absurd yet insightful AR, VR and AI builds. #### 1000 Michael Scotts Screaming at You in AR — Lucas Rizzotto (2017) - Video: https://www.youtube.com/watch?v=1tbBosFQkM4 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, ARKit, Unity - Idea: A thousand Michael Scotts from The Office fill your room and scream at you. - What it is: An absurdist phone AR piece that fills the room with a crowd of screaming Michael Scotts. - Technique: A large number of animated character instances (GPU instancing or LOD) are scattered on detected planes with randomized timing, and spatialized audio fills the room. - Try it: Spawn 100 copies of the same character on the floor plane, each playing animations and spatial sounds at random. Twist: have every character turn its head to stare at the phone. #### CyberSnake – Holographic Snake — Lucas Rizzotto (2017) - Video: https://www.youtube.com/watch?v=1SprJQz_pGU - Interaction: Play, Hands & Body, Spatial Mapping - Platform & tech: Headset, HoloLens, Unity - Idea: You are the snake: walk around your home eating burgers while dodging your own tail. - What it is: The classic Snake game re-imagined in HoloLens: you are the snake, hunting burgers around your home while avoiding your own growing tail. - Technique: The snake head follows the headset's gaze direction at a fixed speed, body segments trail its recorded path, and a spatialized audio source on each target lets players find food outside the field of view. - Try it: Use the orientation of a phone or HoloKit to steer a snake moving around the room, with food located only by 3D spatial audio. Twist: turn off the visuals and play a round with sound alone. #### MyLab – Holographic Chemistry App — Lucas Rizzotto (2017) - Video: https://www.youtube.com/watch?v=MbOoeTlDG_w - Interaction: Information & UI, Hands & Body - Platform & tech: Headset, HoloLens, Unity - Idea: Learn chemistry with holographic atoms and molecules in HoloLens. - What it is: A HoloLens chemistry education app in which students build and inspect atoms and molecules as holograms in the room. - Technique: Atoms and molecules are 3D models in world space on HoloLens, manipulated with gaze-and-air-tap or hand gestures, and bonding rules are computed in the app logic. - Try it: Place a few atom balls on a table with AR Foundation, and let them bond automatically by rule into a water molecule when dragged close together. Twist: swap in other "building units" from your own field, such as musical notes or circuit components. #### Holograms That React to Music in Your Room — Lucas Rizzotto (2018) - Video: https://www.youtube.com/watch?v=PryOpVb1c_M - Interaction: Voice & Sound, Drawing & Making, Perception & Effects - Platform & tech: Headset, HoloLens, Unity - Idea: Fill your room with holographic shapes that pulse to the music. - What it is: Paint audio-reactive holograms all over a room that pulse and morph with whatever music is playing (with Stephen Hodgson). - Technique: Painted stroke meshes have shader parameters (scale, emission, displacement) driven by real-time FFT bands of the playing music. - Try it: Draw lines in the air to create 3D strokes, then use the music's FFT bands to control stroke thickness and brightness. Twist: bind each stroke to a different band, so bass makes strokes thicker and treble makes them brighter. #### A Futuristic Portal to My Best Friend — Lucas Rizzotto (2020) - Video: https://www.youtube.com/watch?v=713T4x89kMk - Interaction: Portals & Worlds, Shared & Social - Platform & tech: Headset, Unity, mixed reality - Idea: During quarantine, build a mixed-reality portal into a friend's home and live as virtual roommates. - What it is: A quarantine project: a mixed-reality 'portal' that opens a window from his apartment into his best friend's space so they can hang out as virtual roommates. - Technique: A wall-anchored portal renders a live volumetric or depth-video stream of the remote room behind the wall plane, using a stencil mask so the remote space appears to sit behind the real wall. - Try it: Put an AR window on a wall that plays a live video call with a friend, using depth or background separation to add a bit of depth. Twist: the window opens only when both of you stand in front of your walls at the same time. #### A Working Marauder's Map – Harry Potter AR — Lucas Rizzotto (2020) - Video: https://www.youtube.com/watch?v=A3v6dLlylU8 - Interaction: Tangible Objects, Location & City, Information & UI - Platform & tech: Phone, image tracking, GPS, Unity - Idea: Make a Marauder's Map that really shows where everyone in the castle is right now. - What it is: In a rented 'mini Hogwarts' castle in England, a physical parchment viewed through the phone shows everyone's live GPS position as walking footprints with name labels, like Harry Potter's Marauder's Map. - Technique: The parchment is tracked as an image target, and each participant's GPS coordinates are converted into map-space positions on it, where footprint animations and name labels are drawn. - Try it: Use image tracking to recognize a hand-drawn campus map and show the live GPS positions of a few classmates as walking footprints. Twist: the map shows only the people who are moving. #### I Turned My Girlfriend into a Musical Instrument — Lucas Rizzotto (2021) - Video: https://www.youtube.com/watch?v=R2KZ34OZlwk - Interaction: Hands & Body, Voice & Sound, Performance - Platform & tech: Phone, Lens Studio, Unity - Idea: Body tracking turns each of his girlfriend's movements into a playable instrument. - What it is: Body-tracked AR turns a dancer's limbs into triggers that play live music as she moves. - Technique: Body tracking gives joint positions, and when limbs enter defined trigger zones or exceed a speed threshold, notes are played and visual effects spawn. - Try it: Mark out several zones in the frame with MediaPipe Pose so that a hand or foot entering each zone plays a different note. Twist: two people play together, one on melody and one on drums. #### Live Music with My Clothes in AR — Lucas Rizzotto (2022) - Video: https://www.youtube.com/watch?v=Vpr1i3-Ekl4 - Interaction: Hands & Body, Voice & Sound, Performance - Platform & tech: Phone, Lens Studio, Unity - Idea: Tap different parts of your clothes to play music live in AR. - What it is: A live performance in which touching different parts of his outfit triggers AR instruments and visuals. - Technique: Body or surface tracking locates regions of the outfit, and hand-to-region proximity triggers instrument samples and AR visuals (likely via body segmentation plus hand tracking). - Try it: Use MediaPipe Pose to turn your chest, shoulders and knees into keys that play different instrument sounds when your hand touches them. Twist: key positions are set by colored stickers on your clothes. #### Pillow – Mixed Reality in Bed — Lucas Rizzotto (2023) - Video: https://www.youtube.com/watch?v=jGaPFRT25is - Interaction: Spatial Mapping, Play - Platform & tech: Headset, Meta Quest 3, Unity - Idea: A mixed-reality app designed for lying in bed. - What it is: A Quest mixed-reality app designed to be used lying in bed, with games, journaling and meditation projected over the ceiling and room. - Technique: The ceiling is identified from room setup or plane classification, and UI and game content are placed on it so they face a user lying down, with input adapted to a supine posture. - Try it: Detect the ceiling (a downward-facing plane) with AR Foundation and display a breathing-meditation animation on it to use while lying down. Twist: control the animation with your breathing rhythm, picked up by the phone microphone. #### Turning My Ceiling into a Planetarium — Lucas Rizzotto (2024) - Video: https://www.youtube.com/watch?v=yjGF76sgi20 - Interaction: Portals & Worlds, Spatial Mapping - Platform & tech: Headset, Meta Quest 3 - Idea: Use mixed reality to turn a bedroom ceiling into a planetarium. - What it is: Mixed reality replaces the bedroom ceiling with a live star field and planets. - Technique: The ceiling plane from the Quest room model is masked out of passthrough and replaced with a rendered sky dome, so stars appear through the ceiling while the walls stay real. - Try it: Detect the ceiling plane and replace it with a star-field dome using a mask, then look up at it through a phone. Twist: use the phone's compass so the constellations match their real directions. ### MIT Fluid Interfaces — Pattie Maes & Pranav Mistry *Research group at the MIT Media Lab led by Pattie Maes; Pranav Mistry was its best-known PhD student* Pattie Maes' Fluid Interfaces group builds wearable and augmented interfaces that blend digital information into everyday life. Pranav Mistry's SixthSense (2009) turned a pocket projector, a camera and colored finger caps into a gestural wearable computer and became one of the most-watched TED demos ever. #### Quickies: Intelligent Sticky Notes — MIT Fluid Interfaces — Pattie Maes & Pranav Mistry (2008) - Video: https://www.youtube.com/watch?v=EO0zbH6DSbw - Interaction: Tangible Objects, Information & UI - Platform & tech: Desktop, digital pen, RFID - Idea: Give the humblest paper sticky note a digital memory so it can be searched and can remind you on its own. - What it is: Handwritten paper sticky notes are captured with a digital pen and RFID so they become searchable, can remind you, and can be found again by the computer. - Technique: A digital pen (Anoto-style dot-pattern paper) captures the handwriting of each note and an RFID tag in the note identifies it, so handwriting recognition turns paper notes into searchable, locatable records. - Try it: Photograph handwritten sticky notes with a phone, run OCR (such as Tesseract.js or the system's text recognition) to save them as a searchable list, and give each note a QR code or NFC tag that jumps to its digital record when scanned. Twist: have a sticky note 'remind' you on its own at a certain time or place. #### SixthSense (WUW – Wear Ur World) — MIT Fluid Interfaces — Pattie Maes & Pranav Mistry (2009) - Video: https://vimeo.com/17567999 - Interaction: Hands & Body, Projection, Information & UI - Platform & tech: Wearable, Projection, pico projector, webcam, color marker tracking - Idea: Hang a projector and camera on your chest so the whole world, even your own palm, becomes a gesture-driven screen. - What it is: A pendant with a pocket projector, mirror and camera projects interfaces onto walls, paper and hands; colored finger caps are tracked so you can frame a photo with your fingers, dial a phone on your palm or read live data projected onto a newspaper. - Technique: A webcam tracks colored finger caps by color thresholding and blob tracking, and a pico projector aimed via a mirror renders UI onto whatever surface is in front of the wearer, with the camera and projector roughly co-aligned so gestures map to projected content. - Try it: Use a phone's front camera with MediaPipe Hands (or colored tape on your fingertips for color tracking) to recognize three gestures such as a two-hand 'viewfinder frame', and use a small projector to cast the matching interface onto the desk or your palm. Twist: keep only one gesture feature you would really use every day, and argue why it deserves to be 'worn on the body'. #### WUW / SixthSense gesture demo reel — MIT Fluid Interfaces — Pattie Maes & Pranav Mistry (2009) - Video: https://www.youtube.com/watch?v=ZfV4R4x2SK0 - Interaction: Hands & Body, Projection, Drawing & Making - Platform & tech: Wearable, Projection, pico projector, webcam - Idea: Draw a circle with your finger to see the time, pick up a book to see its rating: information appears on objects as you gesture. - What it is: Pranav Mistry's own demo reel of the wearable gestural interface: drawing in the air, projected watches on the wrist, augmented boarding passes and books. - Technique: Color-marker fingertip tracking from a chest-mounted webcam drives a gesture vocabulary (air drawing, wrist tap for a watch), while a pico projector renders the response in place on the body or on printed objects. - Try it: Use MediaPipe Hands to build 'air drawing' in a web page: pinch your index finger to start drawing, open your hand to stop, and project the stroke on a wall or overlay it on the camera view. Twist: give the stroke a physical property (it falls, fades or gets blown away by the wind). #### LuminAR — MIT Fluid Interfaces — Pattie Maes & Pranav Mistry (2010) - Video: https://vimeo.com/52899256 - Interaction: Projection, Tangible Objects, Hands & Body - Platform & tech: Projection, pico projector, robotic arm, depth camera - Idea: Swap a desk lamp for a projector robot that turns its own head, so any spot on the desk can become a touch interface. - What it is: Natan Linder's robotic desk lamp whose 'bulb' is a projector-camera: the lamp arm moves on its own to project interactive surfaces onto the desk, follow objects and scan documents. - Technique: A projector-camera pair is mounted as the head of a motorized multi-joint lamp arm, and computer vision on the camera (with depth sensing) chooses where to point the arm, projecting interfaces onto the desk and scanning objects under it. - Try it: Mount a small projector and a phone on a desk-lamp arm, use the camera to detect paper or hands on the desk, and project buttons and hints next to that object. Twist: give the lamp a 'personality', such as bowing its head when you leave or leaning in when it sees a new object (demo with servos or by posing it by hand). #### AfterMath: Seeing the Future through AR — MIT Fluid Interfaces — Pattie Maes & Pranav Mistry (2015) - Video: https://vimeo.com/130279501 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, AR, physics simulation - Idea: Use AR to rehearse the physical world's next second, so you see the consequences before they happen. - What it is: Point a tablet at a physical scene and AR predicts and shows what will happen next, e.g. where objects will fall or roll after a physical action. - Technique: The tablet tracks the scene with AR, reconstructs or fits the physical objects as rigid bodies, and runs a physics simulation forward in time to render predicted trajectories as ghost overlays. - Try it: Detect a tabletop plane in Unity + AR Foundation, place a virtual ball, and use the physics engine to preview its path after a push, showing "the next 2 seconds" as semi-transparent ghost trails. Twist: show several possible future branches and let the user pick the one they want. #### Invisibilia: Revealing Invisible Data — MIT Fluid Interfaces — Pattie Maes & Pranav Mistry (2015) - Video: https://vimeo.com/141926009 - Interaction: Information & UI, Perception & Effects - Platform & tech: Phone, AR, IoT - Idea: Make invisible wireless signals and data flows visible in the room. - What it is: An AR view that visualizes the invisible signals around connected devices, such as Wi-Fi fields and data flows, as overlays in the room. - Technique: Device positions are registered in AR (markers or known anchors) and network data such as Wi-Fi signal strength or packet activity is sampled and rendered as volumetric fields or particle flows around each device. - Try it: Place anchors at the router, phone and speaker with AR Foundation or WebXR, read the Wi-Fi signal strength measured by the phone (or use simulated data), and render it as particle clouds and data streamlines. Twist: visualize another invisible thing, such as noise, temperature or CO2 concentration. #### HoloARt: Painting with Holograms — MIT Fluid Interfaces — Pattie Maes & Pranav Mistry (2016) - Video: https://vimeo.com/202652308 - Interaction: Drawing & Making, Spatial Mapping, Hands & Body - Platform & tech: Headset, HoloLens, Unity - Idea: Fling holographic paint with your hands in a real room, and it sticks to the real walls. - What it is: Judith Amores and Pattie Maes let users paint with hand gestures in HoloLens, splashing holographic paint that sticks to and drips on real surfaces. - Technique: HoloLens spatial mapping provides a mesh of the room, and hand gestures spawn paint particles whose collisions with the mesh leave decals and drip simulations on real surfaces. - Try it: Use AR Foundation Meshing (on a LiDAR iPhone) or plane detection so tapping the screen fires paint balls that leave decals on real surfaces and slowly drip down. Twist: control the paint's color or thickness with sound volume or the phone's tilt. #### Memoro — MIT Fluid Interfaces — Pattie Maes & Pranav Mistry (2024) - Video: https://www.youtube.com/watch?v=fOmvnHcGcXA - Interaction: Voice & Sound, Information & UI, Shared & Social - Platform & tech: Wearable, LLM, semantic search, wearable audio - Idea: A memory prosthesis that speaks up only with the one fact you are missing. - What it is: A wearable audio memory assistant: during a conversation it infers what you are trying to remember and whispers a concise hint, either when you ask or proactively without being asked. - Technique: Captured conversations become searchable memories; an LLM infers the current memory need from context, runs semantic search and returns a minimal audio suggestion in query and queryless modes. - Try it: Collect a week of class notes as text, embed them in a small vector store, and build a phone app that listens to a live discussion, detects hesitation phrases ('what was it called…') and speaks back one retrieved fact. Twist: show the fact as an AR sticky note on the person who originally said it. ### Mar Gonzalez-Franco *Research scientist leading the Blended Interaction Research & Devices lab at Google (formerly Microsoft Research)* Neuroscientist and computer scientist who studied embodiment and avatars in VR, then led mixed-reality research at Microsoft Research before moving to Google. Her team explores AI agents, cross-device input and interaction with real objects in XR. #### Mise-Unseen — Mar Gonzalez-Franco, Eyal Ofek, Microsoft Research — Hrvoje Benko & Andy Wilson (2019) - Video: https://www.youtube.com/watch?v=oNSt92DRkJA - Interaction: Gaze & Attention, Perception & Effects - Platform & tech: Headset, eye tracking, VR - Idea: Change the world in plain sight by timing edits to the gaps in attention. - What it is: In VR, objects are rearranged right inside the user's field of view without them noticing, because changes are applied only where eye tracking shows they are not looking or not paying attention. - Technique: Gaze tracking builds models of attention, intention and spatial memory, and the system schedules scene changes in the periphery or during low-attention moments. - Try it: In a phone AR scene, change the colour of one virtual object only when it is off-screen, and ask classmates whether they notice. Twist: try doing it while they are reading a label. #### MRTransformer — Mar Gonzalez-Franco, Eyal Ofek (2024) - Video: https://www.youtube.com/watch?v=FG9NVsp_cYw - Interaction: Shared & Social, Hands & Body, Gaze & Attention - Platform & tech: Headset, avatars, inverse kinematics, mixed reality - Idea: Translate body language between mismatched spaces, not just positions. - What it is: Two people in rooms with different layouts collaborate in MR as avatars; the system rewrites where each avatar looks and points so the gestures still land on the right objects in the other person's room. - Technique: The avatar's gaze and pointing targets are retargeted semantically to the equivalent objects in the remote room, and the motion is adjusted with inverse kinematics. - Try it: Two students in different rooms each place the same three objects in different spots; show each other's pointing as an AR arrow that snaps to the matching object locally. Twist: add a shrug gesture meaning 'not here'. #### XR-Objects: Augmented Object Intelligence — Mar Gonzalez-Franco, Ruofei Du (2024) - Video: https://www.youtube.com/watch?v=4DjPuf-oyq8 - Interaction: Tangible Objects, Information & UI, Spatial Mapping - Platform & tech: Phone, Headset, object detection, multimodal LLM, ARCore - Idea: Treat every physical object as if it were a digital one you can click on. - What it is: Every real object in view gets its own small AR context menu: tap a pot of honey to ask for recipes, compare two bottles, or set a timer on a kettle, powered by object detection and a multimodal LLM. - Technique: Real-time object detection anchors semantic menus to each object in 3D, and a multimodal LLM answers queries using a cropped image of the selected object as context. - Try it: In phone AR, detect three kitchen objects and attach a 'what can I do with this?' button to each that queries an LLM. Twist: allow a question that involves two objects at once. #### Beyond the Phone — Mar Gonzalez-Franco, Ruofei Du (2025) - Video: https://www.youtube.com/watch?v=T9kSXqevIRo - Interaction: Information & UI, Hands & Body, Tangible Objects - Platform & tech: Headset, Phone, Android XR, cross-device - Idea: The phone and the headset are one interface, and content can flow between them. - What it is: While wearing an XR headset, content on your phone can lift off the screen: maps expand into 3D, photos spread around you and the phone becomes a controller, with smooth transitions between views. - Technique: The phone's pose is tracked by the headset, and apps define multiple views (on-screen, above-screen, world-anchored) with animated transitions triggered by gestures or context. - Try it: With phone AR, make a photo on a tablet lift up into 3D when the phone is held above it. Twist: pour the photo back by tilting the phone. #### EmBARDiment — Mar Gonzalez-Franco (2025) - Video: https://www.youtube.com/watch?v=Af2lxWTFiz4 - Interaction: Gaze & Attention, Voice & Sound, Information & UI - Platform & tech: Headset, eye tracking, LLM, avatar - Idea: Your gaze history is the prompt. - What it is: An embodied AI agent in XR watches which windows and text you have been looking at and uses that as context, so you can ask 'summarise this' or 'what did I just read?' without copying anything. - Technique: Eye tracking builds an attention-weighted memory of on-screen text, which is passed to an LLM agent represented by an avatar alongside the spoken request. - Try it: Log which of five AR text panels a user looks at longest (use head direction as a proxy) and feed only that text to a chatbot when they ask a question. Twist: let the agent say which panel it used. #### Reality Proxy — Mar Gonzalez-Franco (2025) - Video: https://www.youtube.com/watch?v=F2ul_68PrD0 - Interaction: Hands & Body, Spatial Mapping, Information & UI - Platform & tech: Headset, Quest, scene understanding, AI segmentation - Idea: Interact with a stand-in instead of the hard-to-reach real thing. - What it is: In MR, crowded or distant real objects (books on a high shelf, buildings across a city) are pulled toward you as small abstract proxies that you can select, group and filter with your hands. - Technique: Scene understanding and an AI model segment objects and attach semantic attributes, which are mapped to proxies laid out near the hand while staying linked to the real objects. - Try it: Photograph a bookshelf, detect each book spine and show small AR cards of them on the table in front of you; selecting a card highlights the real book. Twist: sort the cards by colour. #### Revisiting Put-That-There — Mar Gonzalez-Franco (2025) - Video: https://www.youtube.com/watch?v=iyBkON2r9QI - Interaction: Voice & Sound, Hands & Body, Gaze & Attention - Platform & tech: Headset, LLM, speech recognition, ray pointing - Idea: Speech plus pointing is the oldest multimodal idea, and LLMs finally make its ambiguity solvable. - What it is: Bolt's 1980 'put that there' is rebuilt for XR with an LLM: you say 'move that over there' while looking and pointing, and windows rearrange themselves according to context. - Technique: Gaze and ray-pointing targets at the moment of each deictic word ('that', 'there') are passed with the transcript and window layout to an LLM, which outputs the layout change. - Try it: Build a phone AR scene where saying 'that' selects the object at the screen centre and 'there' drops it where you point next. Twist: add 'those' for multiple objects. #### Navig-AI-tion — Mar Gonzalez-Franco (2026) - Video: https://www.youtube.com/watch?v=CZt26nSjfao - Interaction: Voice & Sound, Location & City - Platform & tech: Wearable, Phone, spatial audio, vision-language model - Idea: Navigation you can hear, grounded in what is actually around you. - What it is: Walking directions are given as spatial audio cues combined with an AI that describes nearby landmarks from the camera, so you navigate by sound and context rather than by staring at a map. - Technique: A vision-language model describes landmarks from the egocentric camera, and turn directions are rendered as spatialised sound sources placed in the direction to walk. - Try it: Guide a blindfolded partner across a courtyard using only a spatial-audio beacon placed in AR on the target. Twist: add one spoken landmark hint per 20 metres. #### World Mouse — Mar Gonzalez-Franco (2026) - Video: https://www.youtube.com/watch?v=XfF-vadP4lM - Interaction: Spatial Mapping, Information & UI, Hands & Body - Platform & tech: Headset, Desktop, mixed reality, scene mesh - Idea: Extend the humble mouse cursor into the physical world. - What it is: A cursor that travels seamlessly from the desktop screen onto real surfaces and objects in the room, so you can point at a lamp or a wall with the mouse as precisely as at an icon. - Technique: The headset's scene mesh gives surfaces for the cursor to slide on, and mouse deltas are projected onto that mesh with depth-aware transitions off the monitor's edge. - Try it: In phone AR, drive a cursor with a second phone's touchpad and let it slide over detected planes in the room to select virtual stickers. Twist: the cursor leaves a visible trail on real surfaces. ### SPACE10 (IKEA's research and design lab) *Research and design lab funded by IKEA (2015–2023); producer of the Everyday Experiments series* IKEA's independent research lab in Copenhagen explored future living through food, architecture and technology until it closed in 2023. Its Everyday Experiments platform commissioned studios such as FIELD, Bakken & Bæck, Random Studio, Yuri Suzuki and Nicole He to prototype small spatial-computing ideas for the home. #### Everyday Experiments — SPACE10 (IKEA's research and design lab) (2020) - Video: https://www.youtube.com/watch?v=yh-Hae7mQpE - Interaction: Spatial Mapping, Play, Tangible Objects - Platform & tech: Phone, ARKit, LiDAR, machine learning - Idea: Ask many small 'what if' questions about everyday home life and answer each with a quick prototype. - What it is: The launch film of IKEA and SPACE10's collection of 18 spatial-computing experiments for the home, from rooms that rearrange themselves in AR to virtual forts built from your furniture. - Technique: Each studio built a short phone prototype using ARKit, LiDAR, object detection or machine learning, documented as a video and a one-page idea. - Try it: Run a mini 'Everyday Experiments' sprint: each team picks one household chore and prototypes a 60-second phone AR idea that makes it more playful. Twist: the idea must work without any text. #### Room Shuffle — SPACE10 (IKEA's research and design lab) (2020) - Video: https://www.youtube.com/watch?v=gZiOY2WSngw - Interaction: Spatial Mapping, Tangible Objects - Platform & tech: Phone, LiDAR, object detection, ARKit - Idea: See the untapped potential of the space you already have. - What it is: A prototype from Everyday Experiments, built by Bakken & Bæck, that scans a room, recognises the furniture and proposes new arrangements, shuffling your real chairs and tables into alternative layouts in AR. - Technique: Object detection and spatial mapping estimate furniture footprints, and a layout algorithm generates responsive arrangements rendered as virtual replicas in place. - Try it: Scan three pieces of furniture with a LiDAR phone and let a script randomly rearrange their AR copies in the room. Twist: the layout must keep a clear path to the door. #### Everyday Experiments: Collaborative Futures — SPACE10 (IKEA's research and design lab) (2021) - Video: https://www.youtube.com/watch?v=qmy3fw9DP9E - Interaction: Information & UI, Voice & Sound, Spatial Mapping - Platform & tech: Phone, AR, spatial audio, AI - Idea: Use spatial computing to make invisible domestic issues (data, noise, mood) tangible. - What it is: A film about the second round of Everyday Experiments, in which more than twenty studios tackled privacy, trust and well-being at home with AR, spatial audio and AI. - Technique: A curated commissioning model: each partner studio received a theme and produced a working prototype and film, published together on an open platform. - Try it: Pick one invisible thing in your home (Wi-Fi, noise, air) and prototype a phone AR view that makes it visible for 30 seconds. Twist: make it visible only to one person in the room. #### Sound Bubbles — SPACE10 (IKEA's research and design lab) (2021) - Video: https://www.youtube.com/watch?v=sRpaBbGd4Oc - Interaction: Voice & Sound, Spatial Mapping, Shared & Social - Platform & tech: Phone, spatial audio, IoT - Idea: Use sound to create silence. - What it is: An Everyday Experiment by sound designer Yuri Suzuki: spatial audio creates zones of privacy in a shared home, bubbles of silence where one person can take a call while others keep making noise. - Technique: Likely directional speakers and noise masking, coordinated through IoT devices, shape the sound field around a listener's position. - Try it: Place a virtual 'quiet bubble' in phone AR; when a person stands inside it, lower the volume of music playing from other phones in the room. Twist: bubbles can be passed from person to person. #### Studio — SPACE10 (IKEA's research and design lab) (2023) - Video: https://www.youtube.com/watch?v=D2lLneWvQ4c - Interaction: Spatial Mapping, Information & UI - Platform & tech: Phone, LiDAR, ARKit, RoomPlan - Idea: Turn the phone into a democratic interior design tool for the room you are standing in. - What it is: SPACE10's own Everyday Experiments app: a LiDAR scan turns your room into a 3D plan you can measure, restyle with furniture, relight with virtual lamps that switch on and off, and share. - Technique: The iPhone LiDAR sensor captures depth to build floor plans and surface meshes, and dynamic AR light sources are rendered with photorealistic lighting on the scanned geometry. - Try it: Scan your dorm room, place one virtual lamp and toggle it on and off in AR to judge where light is needed. Twist: let a classmate redesign your room remotely from the scan. ### Satoshi Hattori *iOS and visionOS engineer* Tokyo engineer who ran a public '100 days AR challenge' in 2021, posting one ARKit, Metal or Unity effect per day with the source on GitHub, and later did the same for visionOS. #### Audio Reactive FaceMap (AR 100 Days, Day 52) — Satoshi Hattori (2021) - Video: https://x.com/shmdevelop/status/1412363580406517765 - Source code: https://github.com/satoshi0212/AR_100Days - Interaction: Face, Voice & Sound - Platform & tech: Phone, ARKit, Face Tracking, AudioQueue, Swift - Idea: Let the face become a VU meter. - What it is: The user's tracked face mesh changes color with the loudness of the sound around them, pulsing with speech and music. - Technique: ARKit face tracking provides the face geometry, and an AudioQueue input measures volume each frame to drive the mesh material color. - Try it: Use AR Foundation face tracking and the microphone to map loudness to a color ramp on the face mesh. Twist: map pitch instead of loudness so singing paints the face. #### Floor Pit (AR 100 Days, Day 45) — Satoshi Hattori (2021) - Video: https://x.com/shmdevelop/status/1368503261515247617 - Source code: https://github.com/satoshi0212/AR_100Days - Interaction: Portals & Worlds, Spatial Mapping - Platform & tech: Phone, ARKit, SceneKit, Swift - Idea: Cut the camera image into pieces and let reality itself fall apart. - What it is: A section of the real floor cracks into tiles that fall away, opening a pit under the user's feet. - Technique: The floor region of the camera image is likely projected as a texture onto a grid of tiles placed on the detected plane (following a Qiita article he credits); the tiles then fall, revealing a virtual pit beneath. - Try it: On a detected floor plane, texture a grid of cubes with the camera image and drop them one by one when the user steps close. Twist: reveal a different world under each tile. #### Fogged Line World (AR 100 Days, Day 35) — Satoshi Hattori (2021) - Video: https://x.com/shmdevelop/status/1359887965984464897 - Source code: https://github.com/satoshi0212/AR_100Days - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, ARKit, LiDAR, Metal, Swift - Idea: Use depth as a style switch: reality near, drawing far. - What it is: Near the phone the room looks normal, but a few meters away it dissolves into outlines, as if the world were being drawn as a wireframe in the distance. - Technique: LiDAR scene depth is read per pixel in a shader; beyond a distance threshold the camera image is replaced by an edge-detected line rendering, blended through a fog-like gradient. - Try it: Use the AR Foundation environment depth texture to cross-fade between the camera image and a line-art filter at a chosen distance. Twist: let the threshold follow the player so the drawn world closes in over time. #### LiDAR Radar (AR 100 Days, Day 21) — Satoshi Hattori (2021) - Video: https://x.com/shmdevelop/status/1353238818824970241 - Source code: https://github.com/satoshi0212/AR_100Days - Interaction: Spatial Mapping, Information & UI - Platform & tech: Phone, ARKit, LiDAR, RealityKit, Metal - Idea: Show people what the device sees, then let them see it from above. - What it is: The LiDAR scan of the room is overlaid on the camera view as glowing dots, and a double tap flips to a top-down radar map of everything scanned so far. - Technique: LiDAR depth points are accumulated and drawn with Metal as an overlay in RealityKit, then re-projected into an orthographic top view for the radar mode. - Try it: Accumulate AR Foundation depth points into a point cloud and render a minimap corner view of the scanned room. Twist: add a sweeping radar line that only reveals points as it passes. #### Neon Human (AR 100 Days, Day 33) — Satoshi Hattori (2021) - Video: https://x.com/shmdevelop/status/1358940972499959808 - Source code: https://github.com/satoshi0212/AR_100Days - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, ARKit, People Occlusion, Metal, Swift - Idea: Draw light along the edge of the human silhouette to turn anyone into a music-video character. - What it is: People in the camera view are traced with bright neon outlines while the background stays dark, like an animated music-video look applied live. - Technique: ARKit people occlusion (ARMatteGenerator) provides a person matte; a Metal shader likely detects its edges and draws glowing lines over a darkened camera image. - Try it: Take the human stencil texture from AR Foundation, run an edge-detect pass and composite neon lines over the camera feed. Twist: animate the line color with the beat of a song. #### Particle and LiDAR Scan (AR 100 Days, Day 25) — Satoshi Hattori (2021) - Video: https://x.com/shmdevelop/status/1354909729928044545 - Source code: https://github.com/satoshi0212/AR_100Days - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, ARKit, LiDAR, RealityKit, Swift - Idea: Once particles can feel real geometry, every object becomes a landscape for them. - What it is: Particles fall onto a crumpled cloth on a table and slide along its real folds, because they collide with the LiDAR-scanned shape instead of a flat plane. - Technique: The ARKit scene reconstruction mesh from LiDAR is used as a collision shape for a particle system, so particles bounce and slide along the scanned surfaces. - Try it: Use AR Foundation meshing as a VFX Graph or Shuriken collider and pour sand-like particles over objects on a desk. Twist: color each particle by the camera pixel it lands on. #### Pink Smog (AR 100 Days, Day 37) — Satoshi Hattori (2021) - Video: https://x.com/shmdevelop/status/1361336054523961349 - Source code: https://github.com/satoshi0212/AR_100Days - Interaction: Spatial Mapping, Location & City - Platform & tech: Phone, ARKit, LiDAR, Metal, Swift - Idea: Real depth makes fake weather believable: the fog thickens exactly where the world gets farther away. - What it is: A thick pink fog fills the street seen through the phone; objects fade into it with distance, so a car suddenly emerging from the haze feels dangerous. - Technique: Per-pixel LiDAR depth drives an exponential fog blend over the camera image in a Metal shader, tinted pink as a homage to a reported pink smog event. - Try it: Build depth fog with AR Foundation environment depth and give it a color that tells a story (smoke, pollen, toxic haze). Twist: let the fog clear in a circle around the user's hand. #### Real World Splatoon (AR 100 Days, Day 39) — Satoshi Hattori (2021) - Video: https://x.com/shmdevelop/status/1361732615988932612 - Source code: https://github.com/satoshi0212/AR_100Days - Interaction: Play, Spatial Mapping - Platform & tech: Phone, ARKit, LiDAR, SceneKit, Metal - Idea: Turn the scanned room into a paintable game surface. - What it is: The user fires paint balls around the room and they splat onto the real walls, furniture and floor, following each surface's shape like the game Splatoon. - Technique: Projectiles collide with the live LiDAR scene mesh, and paint is likely written onto the reconstructed geometry rebuilt as custom SCNGeometry (which he studied the day before), so the paint wraps around real objects. - Try it: Reproduce the effect with AR Foundation meshing and decal projectors: tap to throw a ball, and leave a splat where it hits. Twist: two phones, two colors, and a timer to see who covers more of the room. #### Space Distortion (AR 100 Days, Day 2) — Satoshi Hattori (2021) - Video: https://x.com/shmdevelop/status/1345363102141845504 - Source code: https://github.com/satoshi0212/AR_100Days - Interaction: Perception & Effects - Platform & tech: Phone, ARKit, Metal, Swift - Idea: Bend the camera image locally so reality looks like it has a lens in it. - What it is: A patch of the room seen through the phone warps and ripples as if the air itself were bending. - Technique: Likely a Metal shader that offsets the UV coordinates of the captured ARKit camera image inside a region anchored in the scene, producing a refraction-like distortion. - Try it: Anchor a sphere in AR and distort only the camera pixels behind it with an animated noise offset. Twist: make the distortion strength follow microphone volume. #### Space Voxels (AR 100 Days, Day 20) — Satoshi Hattori (2021) - Video: https://x.com/shmdevelop/status/1352835578509709312 - Source code: https://github.com/satoshi0212/AR_100Days - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Phone, ARKit, LiDAR, RealityKit, Swift - Idea: Paint the room into voxels that borrow their color from reality. - What it is: Where the user drags a finger across the screen, cubes appear on the real surfaces, each colored with the camera color of the spot it covers, turning the room into voxels. - Technique: Touch points are raycast against the LiDAR scene reconstruction, and a voxel is placed at each hit with its color sampled from the captured camera image at that pixel. - Try it: Build the voxel brush with AR Foundation meshing on a LiDAR phone and let the class fill a corner of the room together. Twist: snap voxels to a coarse grid so the space turns into a Minecraft version of itself. ### Scott Snibbe *Interactive artist; founder of Snibbe Studio and Snibbe Interactive* Media artist whose late-1990s/2000s 'Screen Series' turned projector light and a viewer's shadow into an interactive medium; he later produced Björk's Biophilia app and led Snibbe Interactive's museum installations. #### Boundary Functions — Scott Snibbe (1998) - Video: https://www.youtube.com/watch?v=5wA3lKcDrlM - Interaction: Hands & Body, Projection, Shared & Social - Platform & tech: Projection, overhead camera tracking, Voronoi diagram, floor projection - Idea: Draw invisible 'personal space' on the floor as Voronoi lines, so the more people there are, the less space each gets. - What it is: An overhead projector draws lines on the floor between everyone standing in the room, dividing it into personal Voronoi cells that grow, shrink and snap as people move. - Technique: An overhead camera tracks each person's floor position and a Voronoi diagram of those points is computed in real time and projected back onto the floor from above. - Try it: Use an overhead camera and projector (or a phone filming from above with p5.js) to track where classmates stand, and compute and project Voronoi boundaries in real time. Twist: color each cell by its area, turning redder as it shrinks. #### Deep Walls — Scott Snibbe (2002) - Video: https://www.youtube.com/watch?v=X7h9ckxlxtc - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, silhouette segmentation, video loop recording, projector - Idea: Record passing shadows and keep them looping forever in 'shadow cupboards' on the wall. - What it is: A projected wall of sixteen small cupboards: when someone walks into the beam their shadow is recorded, then replayed forever in one of the cupboards as a looping shadow-film of past visitors. - Technique: A camera segments the viewer's silhouette against the projection, records it while someone is in frame, and drops the clip into a grid cell that loops once the space is empty. - Try it: Use a camera to cut out people's silhouettes, record them as loops a few seconds long, and assemble them into a projected grid. Twist: each cell keeps only one kind of action (such as waving or jumping), with a simple rule deciding which cell a clip goes into. #### Shadow (Screen Series) — Scott Snibbe (2002) - Video: https://www.youtube.com/watch?v=pdxYv-_70-s - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, camera silhouette, time delay, projector - Idea: You have left, but your shadow stays behind and repeats what you just did. - What it is: A plain rectangle of projected light; viewers cast ordinary shadows, but after they step away their shadow keeps moving on the screen, repeating their gestures a moment later. - Technique: A camera captures the viewer's shadow silhouette and the system re-projects the delayed silhouette sequence onto the same screen once the live shadow disappears. - Try it: Use background differencing on a webcam feed to get a person's shadow, and project it back onto the wall with a 3-second delay. Twist: make the delayed shadow gradually fade and play in reverse. #### Compliant — Scott Snibbe (2003) - Video: https://www.youtube.com/watch?v=PSdvN7c25mI - Interaction: Hands & Body, Projection, Play - Platform & tech: Projection, soft-body simulation, silhouette collision, projector - Idea: Make the screen behave like a rubber sheet that people's shadows push and squash. - What it is: A soft rectangle of projected light behaves like a rubber sheet: visitors' shadows push, dent and chase it around the wall. - Technique: The shadow silhouette from a camera is used as a collision boundary for a spring-mesh (soft-body) simulation of the projected rectangle. - Try it: Build a spring-mesh square in p5.js, use the camera silhouette of a person as a collider to push it, and project the result. Twist: make the square emit a tone that changes pitch with its deformation when squeezed. #### Cause and Effect — Scott Snibbe (2004) - Video: https://www.youtube.com/watch?v=Db5i2Nz1jHk - Interaction: Hands & Body, Projection, Shared & Social - Platform & tech: Projection, silhouette recording, projector - Idea: Your movements are recorded and jostle with the shadows of earlier visitors, forming a chain of cause and effect across time. - What it is: Viewers' silhouettes are recorded as they bump into the shadow-recordings of previous visitors; replays show only the recorded movements, so reactions look like autonomous gestures. - Technique: Silhouette clips recorded from a camera are placed on sliding tiles whose positions are displaced by live shadows, chaining recordings of different visitors together. - Try it: Record a silhouette clip of each classmate, line them up in a row, and let the next person's shadow push the earlier clips along. Twist: turn the order of pushes into a replayable 'domino' timeline. #### Make Like a Tree — Scott Snibbe (2005) - Video: https://www.youtube.com/watch?v=CPFF3-di2PU - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, silhouette recording, layered compositing - Idea: Your shadow becomes a ghost in the forest, wandering off between the trees. - What it is: A projected forest in which each visitor's recorded shadow returns as a ghostly figure walking between the trees and fading into the distance. - Technique: Recorded silhouettes are re-inserted into a layered forest image at different depths and scales, with depth-dependent fading to suggest distance. - Try it: Take a landscape photo with several layers of depth and insert classmates' recorded silhouettes into different layers, scaled by distance. Twist: shadows grow more transparent the farther they are and slowly 'walk' deeper into the picture over time. #### Shadow Bag — Scott Snibbe (2005) - Video: https://www.youtube.com/watch?v=ybt4DJ7ptpQ - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, silhouette buffer, projector - Idea: Your shadow stops obeying: sometimes it follows you, sometimes it walks off on its own. - What it is: The projector captures a viewer's shadow and re-projects it with unpredictable variations: sometimes a true shadow, sometimes a disembodied copy that runs toward or away from them. - Technique: A camera silhouette buffer is replayed with randomly chosen transforms (delay, offset, scale, direction) chosen per interaction. - Try it: Build a shadow projection program that randomly switches among four modes: normal, delayed, mirrored and running away. Twist: have classmates guess which rule is active. #### Visceral Cinema: Chien — Scott Snibbe (2005) - Video: https://www.youtube.com/watch?v=xgxkUH6PrIE - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, silhouette compositing, projector - Idea: Visitors' shadows walk into a surrealist film and take over the lead role. - What it is: Scenes from Buñuel and Dalí's Un Chien Andalou are projected; viewers' shadows enter the film and take over the role of the character pulling a grand piano. - Technique: Live camera silhouettes are composited into pre-segmented film footage so that the shadow triggers and drives key moments of the scene. - Try it: Pick a classic film clip, use camera silhouettes to 'put' classmates into the frame and trigger the next shot. Twist: the scene continues only when a shadow strikes the same pose as the original character. #### Central Mosaic — Scott Snibbe (2007) - Video: https://www.youtube.com/watch?v=ZdgX4PUyfvw - Interaction: Hands & Body, Projection, Shared & Social - Platform & tech: Projection, silhouette recording, fractal layout - Idea: Each passer-by's shadow takes the center until newcomers push it to the edge of a fractal mosaic. - What it is: Shadows of passers-by are recorded and replayed as silhouettes that shrink and move outward into a fractal mosaic as newer recordings take the centre. - Technique: Silhouette recordings are laid out in a recursive, self-similar tiling where each new clip pushes older clips one level outward and smaller. - Try it: Arrange recorded silhouette clips in a recursive quad grid, with new clips entering the center and old ones shrinking outward. Twist: color each layer by recording time to form 'growth rings of time'. #### Falling Girl (with Annie Loui) — Scott Snibbe (2008) - Video: https://www.youtube.com/watch?v=7rmd44OwMgk - Interaction: Hands & Body, Projection, Performance - Platform & tech: Projection, live compositing, projection - Idea: Visitors are placed live into a building's windows and become the people a falling girl meets over her life. - What it is: An immersive narrative projection of a girl falling slowly past skyscraper windows; cameras place the live audience inside the apartments she passes as she ages during the fall. - Technique: Room cameras capture viewers and composite their live images into window regions of a pre-animated building while the scripted falling-girl animation plays in the centre. - Try it: Draw a building with many windows, key classmates out of the camera feed live and place them in the windows, then have a character pass from top to bottom. Twist: the character makes a reaction gesture toward whoever's window it passes. ### Shahriar Shahrabi *Technical artist and graphics programmer* Iranian technical artist in Berlin who open-sources his Unity shader and compute-shader studies (fluids, Matrix code rain, procedural painting) with long blog breakdowns. #### Raymarching in Unity — Shahriar Shahrabi (2019) - Video: https://www.youtube.com/watch?v=87YvrkrymG0 - Source code: https://github.com/IRCSS/UnityRaymarching - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL, Raymarching, SDF - Idea: Mix raymarched shapes with ordinary geometry so both live in one world. - What it is: Smooth distance-field shapes blend into each other inside a Unity scene and cast shadows onto normal meshes. - Technique: A shader raymarches signed distance functions with smooth-min blending, writes depth so meshes occlude correctly, and samples Unity's shadow maps on the raymarched volumes. - Try it: Render a raymarched blob on a real table in phone AR, writing depth so the AR occlusion works. Twist: let the blob melt into the table where it touches it. #### Texture Painting on Meshes — Shahriar Shahrabi (2019) - Video: https://www.youtube.com/watch?v=GmCZZrV004A - Source code: https://github.com/IRCSS/TexturePaint - Interaction: Drawing & Making, Tangible Objects - Platform & tech: Desktop, Unity, HLSL, Render textures - Idea: Painting on an object should feel like painting on the real thing, not on a screen. - What it is: The user paints directly onto a 3D model in real time; strokes land in the model's texture and stay on the surface as it turns. - Technique: The mesh is rendered into its UV space with a shader that tests each texel's world position against the brush, writing paint into a render texture, with a dilation pass to hide seams. - Try it: Scan a real object with a LiDAR phone, place the scan over it in AR, and let students spray-paint the object through the phone. Twist: the paint drips downward over time. #### 3D Moebius Transformations — Shahriar Shahrabi (2020) - Video: https://www.youtube.com/watch?v=81XDjBiuuEI - Source code: https://github.com/IRCSS/3D-Moebius-Transformations-Vertex-shader-in-Unity-3D - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Desktop, Unity, HLSL, Vertex shader - Idea: Bend space instead of objects: send the scene to a 4D sphere, rotate it, and bring it back. - What it is: A whole 3D scene folds, rotates and turns inside out through a 4D sphere, as if space itself were being twisted. - Technique: In the vertex shader every vertex is mapped by inverse stereographic projection to a 3-sphere in 4D, rotated there, and projected back, producing Moebius transformations of the scene. - Try it: Apply the Moebius vertex shader to a virtual room placed in AR on a phone, so the walls curl around the viewer. Twist: tie the rotation angle to how far the user walks. #### Cubism Shader — Shahriar Shahrabi (2020) - Video: https://www.youtube.com/watch?v=_DwnvbPxZTM - Source code: https://github.com/IRCSS/Cubism-Shader - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL, Voronoi - Idea: Show several viewpoints at once, as Picasso did, but live. - What it is: A 3D scene is broken into Voronoi shards, each showing the view from a slightly different angle, like a Cubist painting in motion. - Technique: A post-process segments the screen with a Voronoi pattern and, per cell, samples the scene rendered from offset camera transforms, so each shard has its own perspective. - Try it: Apply the Voronoi cubism pass to the AR camera image on a phone, giving each cell a different delay or zoom. Twist: the cells grow bigger when the room gets louder. #### Fluid Simulation in Compute Shaders — Shahriar Shahrabi (2020) - Video: https://www.youtube.com/watch?v=GkrQy5JUyZk - Source code: https://github.com/IRCSS/Compute-Shaders-Fluid-Dynamic- - Interaction: Perception & Effects, Drawing & Making - Platform & tech: Desktop, Unity, Compute Shader, HLSL - Idea: A full fluid solver fits in a handful of compute-shader passes and can paint any surface with moving ink. - What it is: Coloured dye swirls and curls around a 3D model as a full Navier-Stokes fluid solver runs on the GPU, reacting to where the user injects force. - Technique: Stable-fluids style advection, divergence, pressure (Jacobi) and projection passes run as Unity compute shaders on 2D/3D render textures, and the velocity field advects a dye texture. - Try it: Port the repo's 2D solver to a phone and project it onto an AR plane found by AR Foundation, injecting dye where the user touches the floor. Twist: use the device's motion as the force so walking stirs the ink. #### Matrix VFX — Shahriar Shahrabi (2020) - Video: https://www.youtube.com/watch?v=8l7cujPLw84 - Source code: https://github.com/IRCSS/MatrixVFX - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL, Triplanar mapping - Idea: Any object can be rewritten as digital rain that follows its shape. - What it is: A 3D-scanned statue is covered in falling green Matrix code that wraps around its surface and slowly reveals the shape. - Technique: A shader uses triplanar mapping to project a procedurally animated glyph texture onto the mesh from three axes, with per-column random speeds and brightness trails. - Try it: Apply the Matrix shader to the ARKit scene mesh so the whole room turns into code rain on a LiDAR phone. Twist: only reveal the code where the user points a virtual flashlight. #### Mesh Deformation with Compute Shaders on Quest — Shahriar Shahrabi (2020) - Video: https://www.youtube.com/watch?v=IVy6T5_9r2c - Source code: https://github.com/IRCSS/Mesh-Deformation-With-Compute-Shader-Oculus-Quest-Unity - Interaction: Hands & Body, Perception & Effects - Platform & tech: Headset, Unity, Compute Shader, Oculus Quest - Idea: Make virtual surfaces soft to the touch on standalone mobile hardware. - What it is: In VR on an Oculus Quest the user pushes into a soft mesh with a controller and it dents and springs back like clay or jelly. - Technique: A compute shader displaces each vertex away from the controller with a falloff and integrates a spring back to its rest position, running on Android/Quest GPUs. - Try it: Place a deformable blob on a real table in passthrough MR or phone AR and let students poke it with a finger or controller. Twist: the blob remembers every poke and slowly heals. #### Procedural Painting with Genetic Evolution — Shahriar Shahrabi (2020) - Video: https://www.youtube.com/watch?v=--YygVe0Eu4 - Source code: https://github.com/IRCSS/Procedural-painting - Interaction: Perception & Effects, Drawing & Making - Platform & tech: Desktop, Unity, Compute Shader, Genetic algorithm - Idea: Let evolution, not a filter, decide where each brush stroke goes. - What it is: A photo is repainted stroke by stroke: thousands of brush strokes evolve on the GPU until they reproduce the portrait as an expressive painting. - Technique: A genetic algorithm implemented in compute shaders mutates populations of brush-stroke parameters, renders them, and scores each candidate against the target image to keep the fittest. - Try it: Capture a camera frame on a phone, run a small stroke-evolution loop, and hang the resulting painting as a canvas on a real wall in AR. Twist: let the painting keep evolving each time someone looks at it. #### Interactive Volumetric Fog with Fluid Dynamics (The Vast Land) — Shahriar Shahrabi (2021) - Video: https://www.youtube.com/watch?v=hMhNhgnOGN8 - Source code: https://github.com/IRCSS/Compute-Shaders-Fluid-Dynamic- - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, Unity, Compute Shader, Raymarching - Idea: Fog becomes something you can push: a fluid simulation drives a raymarched fog volume. - What it is: A stylised valley is filled with volumetric fog that parts and swirls around a moving character and flows around the terrain as a real fluid. - Technique: The compute-shader fluid solver runs on a 3D grid with arbitrary boundaries taken from the scene, and the resulting density field is raymarched as volumetric fog with lighting. - Try it: Fill a small AR room with low-resolution simulated fog on HoloKit or a phone, using the LiDAR mesh as the fluid boundary so fog flows around real furniture. Twist: let the user's hand position blow the fog away. #### Stencil Portal Halloween Scene — Shahriar Shahrabi (2021) - Video: https://www.youtube.com/watch?v=gGeP34_6d2A - Source code: https://github.com/IRCSS/Unity-Stencil-Portal - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Desktop, Unity, Stencil buffer - Idea: A doorway can hold a whole world that exists only when you look through it. - What it is: A small frame shows a spooky Halloween world inside it; as the camera moves, the hidden scene appears only through the portal opening. - Technique: The portal mask writes a value into the stencil buffer, and the hidden scene's shaders only draw where the stencil matches. - Try it: Anchor a stencil portal to a real door or picture frame with AR Foundation image tracking so a different world appears inside it. Twist: the world behind the portal changes with the time of day. ### Steve Mann *Wearable-computing pioneer; professor at the University of Toronto (EyeTap, mediated reality)* Built and wore head-mounted computers and cameras from the late 1970s onward, streamed his life from a Wearable Wireless Webcam in 1994 at MIT, and invented the EyeTap, which turns the eye itself into a camera and display. He coined 'mediated reality' and 'sousveillance' and keeps teaching wearable AR at the University of Toronto. #### WearComp and the Wearable Wireless Webcam — Steve Mann (1996) - Video: https://www.youtube.com/watch?v=fCco6FMCRmk - Interaction: Gaze & Attention, Shared & Social, Information & UI - Platform & tech: Wearable, WearComp, head-mounted display, wireless video - Idea: Wear the computer all day so that seeing, recording and sharing become one continuous act. - What it is: A 1996 TV report follows Mann around MIT wearing a head-mounted display and camera wired to a backpack computer that streams what he sees to the web and overlays text in front of his eye. - Technique: A self-built backpack computer drives a head-mounted display and a head-mounted camera, with a radio link uploading frames to a web page in near real time. - Try it: Wear a phone on a head strap for one hour and live-stream only what you look at to a shared page for classmates. Twist: let viewers send one-word messages that appear in your view. #### EyeTap Digital Eye Glass — Steve Mann (1999) - Video: https://www.youtube.com/watch?v=DiFtmrpuwNY - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Wearable, EyeTap, beam splitter, aremac - Idea: Make the eye itself the camera and the display, so the computer sees exactly what you see and can rewrite it. - What it is: Mann explains the EyeTap: a beam splitter in front of one eye sends the exact rays entering the eye to a camera and re-emits processed light back along the same path, so the computer can replace what the eye sees. - Technique: A diverter mirror places the camera's centre of projection at the eye's own centre of projection, and an aremac (reverse camera) re-draws the processed image with correct parallax. - Try it: Build a phone 'viewfinder eye': hold the phone in front of one eye and apply a live filter that changes one class of objects (text, faces, signs). Twist: close the other eye and walk a short path using only the mediated view. #### OpenVIDIA: Mediated Reality on the GPU — Steve Mann (2005) - Video: https://www.youtube.com/watch?v=CUvyi6JshZE - Interaction: Shared & Social, Information & UI, Tangible Objects - Platform & tech: Wearable, EyeTap, OpenVIDIA, GPU - Idea: Use the graphics card as a vision processor so a wearable can register overlays on the world in real time. - What it is: Filmed through EyeTap glasses, a technician looks at equipment while GPU-processed overlays are positioned live on it, and a remote expert can guide the work visually. - Technique: The OpenVIDIA library runs feature tracking and image warping as GPU shaders, so homography-based registration keeps overlays locked to surfaces at video rate. - Try it: Pair two students: one points a phone at a device (a coffee machine, a printer) while the other, remote, draws arrows that stay anchored on it in phone AR. Twist: the remote helper may only draw one stroke per step. #### Augmented/Mediated Reality Concepts — Steve Mann (2007) - Video: https://www.youtube.com/watch?v=jA7o7honFWA - Interaction: Portals & Worlds, Perception & Effects, Location & City - Platform & tech: Wearable, EyeTap, computer vision - Idea: AR can subtract as well as add: diminished reality lets you filter the world you do not want to see. - What it is: A short concept film from Mann's EyeTap lab showing a wearer whose glasses not only add information but also remove or replace things in view, such as blocking out advertising billboards in the street. - Technique: Likely tracked planar regions such as billboards are detected in the head camera feed and overwritten with other imagery before being shown to the eye. - Try it: Use phone AR image tracking to detect one poster or logo on campus and cover it with a calm texture. Twist: replace every detected logo with a short personal note from a classmate. #### HDR EyeGlass (welding helmet) — Steve Mann (2012) - Video: https://www.youtube.com/watch?v=gtTdiqDqHc8 - Interaction: Perception & Effects - Platform & tech: Wearable, HDR, EyeTap, comparametric imaging - Idea: Give human vision a dynamic range it never had, so the brightest and darkest parts of a scene are visible together. - What it is: A welding helmet with a camera and display lets the welder see both the blinding arc and the dark metal around it clearly at the same time, by fusing several exposures in real time. - Technique: Frames with alternating exposures are combined with comparametric (quantigraphic) HDR compositing on the fly and tone-mapped to the head-worn display. - Try it: Point a phone at a window scene with a bright sky and a dark room, and build a live view that blends two exposures with a slider for the mix. Twist: map brightness to sound instead of image. #### HDRchitecture — Steve Mann (2012) - Video: https://www.youtube.com/watch?v=GM517kYS6kY - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Wearable, Desktop, HDR, GPU, 3D sensing - Idea: Perceive a space's light and geometry at once, beyond what either an eye or a camera can see alone. - What it is: A SIGGRAPH Emerging Technologies demo in which extreme-dynamic-range video from a wearable rig is combined with 3D sensing, so bright light sources and dark rooms are captured and seen together in real time. - Technique: GPU-based HDR fusion of bracketed exposures runs in real time and is mapped onto depth-sensed geometry for 3D viewing. - Try it: Scan a room with a LiDAR phone twice, once with lights on and once with lights off, and toggle between the two meshes in AR. Twist: let a hand gesture paint 'daylight' onto parts of the dark scan. #### Augmediated Reality with Depth-based Surface Tracking — Steve Mann (2013) - Video: https://www.youtube.com/watch?v=9fI7pZjtz8s - Interaction: Spatial Mapping, Tangible Objects, Information & UI - Platform & tech: Wearable, depth camera, Meta, EyeTap - Idea: Any plain surface can become a screen if the glasses understand its shape. - What it is: An early Meta/EyeTap prototype in which a depth camera on the glasses tracks bare surfaces without markers, so virtual windows stay attached to a table or sheet of paper as the wearer moves. - Technique: Structure-based tracking fits planes to the depth map every frame and runs side by side with the rendering engine, so no visual features or fiducials are needed. - Try it: Use AR plane detection to pin a note app to a blank sheet of paper and let the sheet move. Twist: when the paper is folded, split the content across the two halves. #### SWIM: Sequential Wave Imprinting Machine — Steve Mann (2017) - Video: https://www.youtube.com/watch?v=wKfwufI2hrQ - Interaction: Perception & Effects, Voice & Sound, Projection - Platform & tech: Projection, SWIM, LED, lock-in amplifier - Idea: Phenomenal augmented reality: make real physical phenomena, not graphics, visible exactly where they exist. - What it is: A robotic arm sweeps a row of LEDs through space while they light up with the live signal of a sound or radio wave, so a long-exposure camera or the eye's persistence of vision sees the invisible wave frozen in the air where it really is. - Technique: A lock-in amplifier demodulates the wave relative to its source, and the LED bar's brightness at each position is driven by that signal while it moves, imprinting the standing wave in space. - Try it: Tape a phone flashlight to a stick, make it blink with the loudness of a speaker's tone, and sweep it past the speaker in a dark room while another phone takes a long exposure. Twist: sweep at different distances and compare the patterns. #### Doppler sonar vision system for the blind — Steve Mann (2019) - Video: https://www.youtube.com/watch?v=oFLBiwTSkOg - Interaction: Voice & Sound, Perception & Effects, Hands & Body - Platform & tech: Wearable, sonar, lock-in amplifier, Arduino - Idea: Hear motion: translate an invisible sensing channel directly into a sense you already have. - What it is: A lab from Mann's wearable-computing course: a small sonar unit turns the Doppler shift of moving objects into audible tones, giving a sightless wearer a sense of motion around them. - Technique: An ultrasonic transmitter and receiver feed a homemade lock-in amplifier whose output frequency equals the Doppler shift, which is played through headphones. - Try it: Use a phone's camera optical flow to turn the speed of moving things in front of you into pitch, then walk a hallway with eyes closed. Twist: map direction to left and right ears. #### Underwater AR brain-sensing eyeglass — Steve Mann (2021) - Video: https://www.youtube.com/watch?v=-DYTmwKJkow - Interaction: Perception & Effects, Hands & Body, Location & City - Platform & tech: Wearable, head-up display, EEG, Muse - Idea: Take wearable AR into water, where vision, sound and balance all behave differently. - What it is: Mann swims in Lake Ontario wearing waterproof AR eyeglasses with a brain-sensing headband, capturing and overlaying information from below the waterline. - Technique: Likely a sealed head-up display and camera paired with an EEG headband stream data to a waterproofed wearable computer. - Try it: Put a phone in a waterproof pouch, float it in a sink or pool and design an AR overlay that only makes sense underwater (depth, bubbles, refraction). Twist: make the overlay react to the water's movement. ### teamLab *Art collective (founded by Toshiyuki Inoko)* Interdisciplinary art collective building room-scale interactive projection worlds where visitors' drawings and bodies change the artwork. #### Sketch Aquarium — teamLab (2013) - Video: https://www.youtube.com/watch?v=AnAqB7LZUb8 - Interaction: Drawing & Making, Projection, Hands & Body - Platform & tech: Projection, scanner, real-time rendering - Idea: Fish drawn by children are scanned in and swim in a projected aquarium. - What it is: Children colour paper fish that are scanned and immediately swim in a giant projected aquarium, where they react when touched. - Technique: Coloured template sheets are scanned, the drawing is texture-mapped onto a pre-made 3D fish model via the template's known layout, and the fish joins a projected real-time scene with touch or depth-sensor interaction. - Try it: Design a creature coloring template, photograph it with a phone, texture it onto a three.js model, and send it into a shared projected aquarium. Twist: let each creature's behavior (fast or slow, schooling or solitary) be decided by the dominant color in its coloring. #### Sketch Town — teamLab (2014) - Video: https://www.youtube.com/watch?v=kQ_17zapssI - Interaction: Drawing & Making, Projection, Shared & Social - Platform & tech: Projection, scanner, real-time 3D - Idea: Hand-drawn cars and houses move into a shared projected city. - What it is: Hand-drawn cars, buildings and UFOs are scanned into a shared 3D projected town where they drive and fly, and can be touched. - Technique: Scanned template drawings are UV-mapped onto 3D vehicle and building models and inserted into a shared projected city, with sensors detecting touch on the projection. - Try it: Have classmates draw houses and vehicles on templates, then scan them, texture them onto simple models and put them into a projected city. Twist: every piece must interact with someone else's piece at least once (collide, pick up, connect). #### Floating Flower Garden — teamLab (2015) - Video: https://www.youtube.com/watch?v=7i-sPiuipco - Interaction: Hands & Body, Spatial Mapping, Shared & Social - Platform & tech: Desktop, living orchids, motorised winches, sensors - Idea: Content that clears space around the user instead of crowding them feels welcoming; AR clutter should retreat as the user approaches. - What it is: A room is packed with thousands of living orchids hanging to the floor; as a visitor approaches, the flowers near them rise upward, opening a dome of space that follows the person. - Technique: Motorised lines raise and lower each plant pot, driven by sensors that track visitor positions (likely overhead depth cameras). - Try it: Fill a room in phone AR with 500 hanging flowers down to the floor, and make every flower within one meter of the phone rise to the ceiling. Twist: two users close together create one larger shared dome. #### Flowers and People, Cannot be Controlled but Live Together — teamLab (2015) - Video: https://www.youtube.com/watch?v=arafX3Es6JQ - Interaction: Projection, Hands & Body, Perception & Effects - Platform & tech: Projection, real-time rendering, sensors - Idea: Projected flowers bloom or fall as visitors touch them or stand still. - What it is: Projected flowers bloom and wither across the walls through the seasons; when visitors touch or stand still, petals scatter or grow around them. - Technique: A real-time generative flower simulation runs continuously, and depth or proximity sensors detect visitors' position and stillness to trigger blooming or scattering locally. - Try it: In TouchDesigner, use a camera to detect where people are, so projected flowers grow while someone stands still and wither when they move. Twist: make the flower species change with the time of day or the season. #### The Infinite Crystal Universe — teamLab (2015) - Video: https://www.youtube.com/watch?v=3LFegfRrwtg - Interaction: Shared & Social, Perception & Effects, Hands & Body - Platform & tech: Projection, LED strings, mirrors, smartphone app - Idea: A phone can be a remote control for a physical light volume shared by a crowd; AR can hand the same shared space to many users at once. - What it is: Thousands of vertical LED strings fill a mirrored room; visitors use a phone app to throw elements such as stars or flowers into the space, which then burst through the lights around everyone. - Technique: Point lights on hanging strings form a volumetric display mirrored to infinity, and a smartphone app sends chosen elements to the rendering engine in real time. - Try it: Build a shared WebXR room where each phone can fling one element (star, leaf, firework) into a common grid of glowing points that everyone sees. Twist: when two users throw elements at the same moment, they merge into a larger burst. #### Drawing on the Water Surface Created by the Dance of Koi and People - Infinity — teamLab (2016) - Video: https://www.youtube.com/watch?v=qAebQICA-fE - Interaction: Hands & Body, Projection, Spatial Mapping - Platform & tech: Projection, projection mapping, water, body tracking - Idea: Real water and projected fish together feel more alive than either alone; AR gains presence when it plays on a real material such as water or sand. - What it is: Visitors wade knee-deep through water in a mirrored room while projected koi swim around their legs; when a koi touches a person it bursts into flowers. - Technique: Top-down projectors map the fish onto a shallow pool, and sensors track the legs of visitors so each fish's path and collisions respond in real time. - Try it: Use a phone to detect a real puddle, pond or blue floor area and spawn AR koi that swim only on it and avoid the user's feet with people occlusion. Twist: every collision leaves a flower that floats for a minute. #### Forest of Resonating Lamps - One Stroke — teamLab (2016) - Video: https://www.youtube.com/watch?v=cEMniCKnOD0 - Interaction: Gaze & Attention, Shared & Social, Perception & Effects - Platform & tech: Projection, Murano glass lamps, LEDs, proximity sensors - Idea: A single local action that ripples through a whole system shows one person's influence at room scale; AR can propagate a tap into a visible chain reaction. - What it is: Hundreds of Murano glass lamps hang in a mirrored room; when someone stands still near a lamp, it glows and its colour spreads lamp by lamp through the whole forest. - Technique: Each lamp is individually addressable and lights by proximity sensing, then triggers its nearest neighbours in turn, a spreading algorithm like a wave on a graph. - Try it: Place 100 AR lamps around the user; the lamp the user gazes at for two seconds lights, then passes its light to its three nearest neighbours every 0.3 s until all are lit. Twist: two users starting waves in different colours see where the colours meet and mix. #### Graffiti Nature — teamLab (2016) - Video: https://www.youtube.com/watch?v=OomhbW3bffs - Interaction: Drawing & Making, Projection, Play - Platform & tech: Projection, scanner, real-time rendering - Idea: Drawn animals enter a projected ecosystem where they hunt and can be stepped on. - What it is: Drawn animals and plants are scanned into a projected room-sized ecosystem where they eat each other and flee when visitors step on them. - Technique: Scanned drawings become agents in a projected ecosystem simulation (predator-prey rules), and floor-facing depth sensors detect footsteps that make creatures flee or die. - Try it: Build a simple predator-prey simulation in p5.js where each creature a classmate draws is scanned in as one individual, project it on the floor, and detect footsteps with a camera. Twist: creatures stepped "to death" become nutrients that grow new plants. #### Universe of Water Particles on a Rock where People Gather — teamLab (2018) - Video: https://www.youtube.com/watch?v=WWuDTBpPZbA - Interaction: Spatial Mapping, Hands & Body, Projection - Platform & tech: Projection, particle simulation, depth sensing, projection mapping - Idea: The body as a physical obstacle is the simplest interaction there is; AR water, smoke or crowds that flow around the user need no instruction. - What it is: A virtual waterfall pours down a large rock-like slope; people standing on it become obstacles, so the water parts around them and flowers bloom where they stay. - Technique: Water is simulated as hundreds of thousands of particles on a 3D model of the rock, and depth sensors turn visitors into collision shapes before projection mapping. - Try it: Use ARKit/AR Foundation people occlusion or LiDAR to run a particle waterfall down a real staircase that flows around any person standing on it. Twist: someone standing still for five seconds grows a ring of flowers. #### teamLab Borderless — teamLab (2018) - Video: https://www.youtube.com/watch?v=Xy6Vz3rkd1w - Interaction: Location & City, Shared & Social, Projection - Platform & tech: Projection, networked projectors, real-time simulation, sensors - Idea: Content that wanders between spaces makes a building feel alive and worth exploring; location-based AR can let characters migrate between anchors. - What it is: A museum without a map in Tokyo, where artworks move out of one room, travel along corridors and mix with works in other rooms, reacting to visitors and to each other. - Technique: Hundreds of networked projectors and computers share one simulation, so each artwork is an agent that can cross room boundaries and interact with others. - Try it: Place three AR anchors in different rooms of your school and create one creature that travels between them along corridors over five minutes, so users must follow it. Twist: when it meets a creature made by another group, both change colour. ### Alessio Grancini *AR engineer; Snap Spectacles developer relations (ex-Magic Leap, Morphosis)* Italian architect turned AR prototyper: SCI-Arc Gehry Prize thesis on a world-scale AR game, XR lead at Morphosis, prototype engineer at Magic Leap, now building and teaching Spectacles lenses at Snap. #### Games of Deletion — Alessio Grancini (2018) - Video: https://www.youtube.com/watch?v=EccdRMAOvSM - Interaction: Location & City, Shared & Social, Play - Platform & tech: Phone, Unity, ARKit, ARCore - Idea: Turn city buildings and street art into territory to erase and fight over in a multiplayer AR game. - What it is: A world-scale multiplayer mixed-reality game in which players tag and 'delete' urban art and architecture through their phones, competing over real city spaces. - Technique: Geolocated AR content is anchored to real urban sites and synced across players so each tag or deletion changes a shared world state. - Try it: Pick five buildings on campus, use AR to 'tag' or 'erase' them and record this on a shared map, with two teams competing to take them over. Twist: erased buildings actually turn transparent in AR. #### Interfaces by Morphosis AR app — Alessio Grancini (2019) - Video: https://www.youtube.com/watch?v=VCgOt4sfs1o - Interaction: Information & UI, Location & City, Tangible Objects - Platform & tech: Phone, Unity, ARKit, ARCore - Idea: Overlay moving structure and design information on a physical architectural installation with AR. - What it is: The official AR app for Morphosis's Milan Design Week installation with Dassault Systèmes overlays animated architectural geometry and information on the physical installation. - Technique: Image or object tracking registers the phone to the installation so digital geometry and sound are aligned with the built structure (likely target-based). - Try it: Make an AR guide for a sculpture or architectural model at school: scan it to reveal structural lines and the design story. Twist: let viewers 'take it apart' to see the inner structure. #### Augmented Inspection (Vision Hack) — Alessio Grancini (2024) - Video: https://www.youtube.com/watch?v=H3DuCWQXWUw - Interaction: Hands & Body, Voice & Sound, Information & UI - Platform & tech: Headset, Apple Vision Pro, Unity, OpenAI, Boston Dynamics Spot - Idea: Put on a headset and use gestures and voice to inspect and steer a robot dog remotely. - What it is: An Apple Vision Pro app lets an operator monitor and teleoperate a Boston Dynamics Spot robot with hand gestures and voice, seeing its camera feed and point cloud in space. - Technique: Hand tracking and speech on Vision Pro drive robot commands, while the robot's camera stream and point cloud are rendered as spatial panels. - Try it: Use phone AR and a remote-control car to run a 'remote inspection': watch the car's camera feed on the phone and steer it with hand gestures. Twist: automatically tag objects of a certain color when they are spotted. #### Mistral-OUI: contextual AR UI — Alessio Grancini (2024) - Video: https://www.youtube.com/watch?v=2Bcnh3w1CCg - Interaction: Information & UI, Gaze & Attention - Platform & tech: Headset, Magic Leap 2, Unity, Mistral API, moondream - Idea: Whatever you look at, AI generates a matching interface for it on the spot. - What it is: On Magic Leap 2, a vision model recognizes what the wearer is looking at and a language model generates a custom interface for that context, shown floating in view. - Technique: Camera frames are captioned by a vision-language model, and an LLM turns the description into HTML UI rendered as a panel in the headset. - Try it: Take a photo with a phone and have a multimodal LLM identify the object and generate an 'AR sticky note' UI (shown in web AR). Twist: change the UI's style to match the object's mood. #### From prompt to lens with an MCP server for Lens Studio — Alessio Grancini (2025) - Video: https://www.youtube.com/watch?v=bhWIryYyVvM - Interaction: Drawing & Making, Voice & Sound - Platform & tech: Desktop, Headset, Lens Studio AI, Model Context Protocol, Cursor, Claude - Idea: The AR editor becomes something an AI agent can operate for you. - What it is: A walkthrough of building a first Model Context Protocol server so an AI coding agent can create and edit AR lenses inside Lens Studio from plain-language prompts. - Technique: An MCP server exposes Lens Studio actions (create objects, set properties, run scripts) as tools that an LLM agent in Cursor or Claude can call in a loop. - Try it: Expose three functions of any 3D tool (add cube, set colour, move) through a tiny MCP server and ask an AI agent to build a scene from one sentence. Twist: the agent must explain each step aloud to a classmate who is building the same scene by hand. #### Remote object detection with Hugging Face on Spectacles — Alessio Grancini (2025) - Video: https://www.youtube.com/watch?v=6FuHyGiO8dc - Interaction: Tangible Objects, Information & UI - Platform & tech: Headset, Snap Spectacles, Hugging Face, Lens Studio - Idea: Borrow any open vision model from the web and see its answers on the world. - What it is: A sample lens sends the Spectacles camera view to an object-detection model hosted on Hugging Face and draws labelled boxes on real objects in the wearer's view. - Technique: Camera frames are posted to a hosted inference endpoint, and returned bounding boxes are unprojected into 3D using camera intrinsics and depth to anchor labels. - Try it: Use a phone WebAR page to send a frame to a free Hugging Face detection model and place floating labels at the detected positions using hit tests. Twist: label objects with what they might be thinking instead of their names. #### AI-transformed video call on Spectacles — Alessio Grancini (2026) - Video: https://x.com/alessiograncini/status/2079985282510631174 - Interaction: Shared & Social, Perception & Effects - Platform & tech: Headset, Snap Spectacles, Decart AI, NDK, WebRTC, agentic coding - Idea: Telepresence where the person you see is re-imagined as they talk. - What it is: A video call experiment on Spectacles in which the caller's live video is transformed by a real-time AI video model; the prototype was built in a few hours with an agentic coding loop. - Technique: A native (NDK) WebRTC stream carries the call, frames pass through a hosted video-to-video model, and the transformed stream is shown as a floating panel in the lens. - Try it: Pair two phones on a video call where one side's feed goes through a live AI filter; decide together what the filter should reveal about the speaker. Twist: the filter changes with the speaker's tone of voice. #### Real-time AI style transfer on AR glasses — Alessio Grancini (2026) - Video: https://x.com/alessiograncini/status/2082150582467387856 - Interaction: Perception & Effects - Platform & tech: Headset, Snap Spectacles, Decart AI, NDK, WebRTC - Idea: A reality filter that lives on your glasses rather than your phone screen. - What it is: An educational experiment connects Spectacles to Decart's real-time video model and projects the transformed camera feed back into the wearer's view, so the world appears restyled live. - Technique: Camera frames are streamed over WebRTC to a hosted real-time video-to-video model and the returned frames are rendered as an overlay aligned to the camera's field of view. - Try it: Stream a phone camera to a real-time diffusion API and display the result in a HoloKit or cardboard headset with a style chosen by voice. Twist: restyle only the part of the scene inside a hand-drawn frame. #### Spectacles controlling a Reachy Mini robot — Alessio Grancini (2026) - Video: https://www.youtube.com/watch?v=7-7kT0e0knQ - Source code: https://github.com/agrancini-sc/Spectacles-Reachy-Mini-MIT-2026 - Interaction: Hands & Body, Tangible Objects - Platform & tech: Headset, Snap Spectacles, Reachy Mini, Hugging Face, Lens Studio - Idea: AR glasses become the remote control and face for an AI robot on your desk. - What it is: For MIT Reality Hack 2026, Snap's developer team connected Spectacles to Hugging Face and Pollen Robotics' Reachy Mini, so AR interfaces can drive a small expressive desktop robot. - Technique: A Lens Studio lens sends commands over the network to the Reachy Mini's open-source Python SDK, whose head and antennas react to hand input and AI behaviours. - Try it: Connect a phone AR app to a cheap servo toy over WebSocket, so pointing at it in AR makes it turn toward you. Twist: give the robot an LLM personality that comments on what the AR camera sees. ### Alexander Whitley Dance Company *Choreographer and dance-technology company* British choreographer whose company builds stage works and installations around motion tracking, responsive light and live motion capture, working with Marshmallow Laser Feast, Memo Akten, Children of the Light and creative technologist Luca Biada. He is a Sadler's Wells New Wave Associate and spoke with AΦE at the Physical-Digital Digital Innovation Network event. #### The Measures Taken — Alexander Whitley Dance Company, Marshmallow Laser Feast (2014) - Video: https://vimeo.com/85073837 - Interaction: Hands & Body, Projection, Performance - Platform & tech: Projection, motion tracking, real-time graphics, projection - Idea: A stage that is drawn by the way a machine 'sees' the dancers. - What it is: Five dancers perform inside projected lines, particles and light planes that are generated from their own tracked movement in real time, so the set keeps redrawing itself around their bodies. - Technique: Motion-tracking cameras (likely Kinect-style depth sensors) capture the dancers' positions and skeletons, and Marshmallow Laser Feast's real-time graphics project responsive visuals onto the stage and scrims. - Try it: Point a webcam and a projector at a wall, run a body-tracking sketch (MediaPipe or TouchDesigner) and draw lines that connect each student's joints to the nearest wall edge while they dance. Twist: make the lines lag by two seconds, so dancers duet with their own recent past. #### Pattern Recognition — Alexander Whitley Dance Company, Memo Akten (2015) - Video: https://vimeo.com/136252155 - Interaction: Hands & Body, Performance, Perception & Effects - Platform & tech: Projection, moving lights, motion tracking, openFrameworks - Idea: The lighting rig watches, learns and becomes a dance partner. - What it is: Two dancers move beneath a rig of motorised lights that track them, follow them and gradually improvise their own responses, turning the lighting into a third performer. - Technique: Memo Akten's system tracks the dancers from above and steers a grid of moving-head lights in real time, with behaviours that remember and vary earlier movement (likely custom openFrameworks software). - Try it: Mount a phone above a dark room and use it to track one person; drive a single desk lamp on a servo, or a projected spotlight, to follow them with a delay. Twist: let the light 'lose interest' and wander off when the dancer stays still too long. #### Strange Stranger — Alexander Whitley Dance Company (2018) - Video: https://vimeo.com/354179753 - Interaction: Hands & Body, Spatial Mapping, Information & UI - Platform & tech: Projection, motion sensors, light installation - Idea: Your 'data shadow' stays behind in the room after you leave. - What it is: Visitors walk through a grid of three-metre towers where light structures appear, transform and vanish; motion sensors record their activity so every visitor leaves traces that change the space for the next. - Technique: Motion sensors capture visitor movement and feed a lighting system by Children of the Light and Luca Biada, which stores and replays these traces, seeded at the start by a recorded performance of four dancers. - Try it: Use a webcam over a corridor to log where people walk, then project a slowly fading heat map of those paths back onto the floor. Twist: project yesterday's paths in a different colour so today's visitors walk among strangers' traces. #### Anti-Body — Alexander Whitley Dance Company (2021) - Video: https://www.youtube.com/watch?v=IZpirMOBwLg - Interaction: Hands & Body, Projection, Performance - Platform & tech: Projection, motion capture, real-time graphics, projection on scrims - Idea: Live motion capture turns dancers into the light and scenery of the show. - What it is: Three dancers, physically isolated but digitally connected, move between translucent screens while streams of motion-responsive code and particle imagery grow from their bodies, making them set, lighting and characters at once. - Technique: A live motion-capture system streams the performers' skeletons into real-time visuals by Uncharted Limbo Collective, which are projected onto layered translucent screens around them. - Try it: Hang two sheets of tulle one metre apart, project a particle system driven by a webcam skeleton onto both, and have a student dance between them. Twist: let the particles on the front sheet show where the dancer was one second ago. #### Pattern Regression — Alexander Whitley Dance Company, Memo Akten (2022) - Video: https://vimeo.com/778138186 - Interaction: Hands & Body, Performance - Platform & tech: Projection, moving lights, motion tracking - Idea: A human and a lighting machine hold a kinetic conversation in a gallery. - What it is: An installation version of Pattern Recognition: a solo dancer is surrounded by a responsive system of moving lights that tracks and reacts to her in real time, and visitors watch the dialogue at close range. - Technique: The dancer's position is tracked and mapped to pan, tilt and intensity of moving-head fixtures designed by Memo Akten, running as a performance installation rather than a stage show. - Try it: Build a 'light buddy' with three phone flashlights on stands controlled over OSC from a laptop that tracks a student with a webcam. Twist: give each light a different personality: shy, curious and copycat. #### The Last Swan — Alexander Whitley Dance Company (2025) - Video: https://vimeo.com/1219574400 - Interaction: Hands & Body, Projection, Performance - Platform & tech: Projection, motion capture, real-time rendering, projection - Idea: A classical ballet becomes a projection room the audience can enter and continue. - What it is: An English National Ballet dancer performs Swan Lake inside a projection cube where watery haze and feathers ripple with her tracked movement; afterwards audience members step in and see their own presence reflected in the same environments. - Technique: Motion capture on the dancer drives real-time visuals projected onto the walls of a cube, then an audience-facing tracking system switches the same generative scenes to respond to visitors. - Try it: Turn a classroom corner into a two-wall projection 'lake' that ripples where a webcam sees people move. Twist: after one dancer's solo, replay her ripples at half opacity while the audience adds their own. #### Mirror — Alexander Whitley Dance Company (2026) - Video: https://www.youtube.com/watch?v=zjHSgImXydw - Interaction: Hands & Body, Perception & Effects, Performance - Platform & tech: Projection, motion capture, generative AI, photogrammetry - Idea: An AI mirror that starts faithful and slowly lies. - What it is: Two dancers are watched by an AI system that first mirrors their actions on screen and then distorts, intervenes in and reshapes how they appear to each other, in a duet between bodies and a machine gaze. - Technique: Live motion capture and photogrammetry-based avatars feed a generative AI pipeline that renders an altered mirror image of the dancers in real time (details of the models are likely custom). - Try it: Build a webcam 'mirror' that shows students their pose skeleton and slowly exaggerates one joint at a time. Twist: let a partner secretly choose which joint the mirror distorts, and see if the dancer notices. ### Apple (ARKit / visionOS teams) *Platform company* Apple's ARKit and visionOS teams released SwiftShot, the first multi-user ARKit sample game, and the Vision Pro showcase Encounter Dinosaurs. #### SwiftShot — Apple (ARKit / visionOS teams) (2018) - Video: https://www.youtube.com/watch?v=0pUnaAkzUOs - Interaction: Shared & Social, Play, Tangible Objects - Platform & tech: Phone, ARKit 2, world map sharing, MultipeerConnectivity - Idea: A shared AR slingshot battle between several players at the same table. - What it is: A WWDC multiplayer ARKit 2 game: players on several iPhones/iPads see the same tabletop battlefield and fling slingshot projectiles at each other's wooden blocks. - Technique: Shares an ARWorldMap between devices so all players relocalize to the same anchors, then syncs physics state over peer-to-peer networking. - Try it: Build a two-player battle with shared anchors in Reality Composer or AR Foundation: on the same tabletop, flick the phone to launch balls and knock down the opponent's blocks. Twist: a real cup on the table acts as a solid obstacle. #### Encounter Dinosaurs — Apple (ARKit / visionOS teams) (2024) - Video: https://www.youtube.com/watch?v=31VkL_iPXkM - Interaction: Portals & Worlds, Hands & Body, Gaze & Attention - Platform & tech: Headset, Apple Vision Pro, RealityKit, hand tracking - Idea: A door opens in the wall, dinosaurs lean out to interact with you, and a butterfly lands on your fingertip. - What it is: On Vision Pro a portal opens in your wall; a butterfly lands on your outstretched finger and dinosaurs step toward you, reacting when you move or reach out. - Technique: Renders a portal into a full environment anchored to a wall and drives creature behaviours from the user's head and hand positions. - Try it: Place a portal on a wall with phone AR so the character inside backs away or steps forward as the phone gets closer. Twist: when you hold out your hand (detected by hand tracking), have a virtual butterfly land on it. ### Christian Holz *Associate professor at ETH Zurich (Sensing, Interaction & Perception Lab); formerly Microsoft Research* Did his PhD with Patrick Baudisch at the Hasso Plattner Institute, spent years as a researcher at Microsoft Research Redmond working on VR haptics and mixed reality with Eyal Ofek, Andy Wilson and Mar Gonzalez-Franco, and now leads SIPLAB at ETH Zurich. #### Imaginary Phone — Christian Holz (2011) - Video: https://www.youtube.com/watch?v=aCARtauIS50 - Interaction: Hands & Body, Information & UI - Platform & tech: Wearable, depth camera, iPhone - Idea: Use your palm as a screenless phone, relying only on memory of where the icons are. - What it is: Users operate their iPhone without taking it out of their pocket by tapping and swiping on the palm of their other hand, where they remember the phone's home-screen layout; a wearable depth camera sends those touches to the real phone. - Technique: A worn depth camera tracks the finger touching the palm and maps the touch position onto the phone's icon grid, transferring spatial memory from the physical device. - Try it: Use MediaPipe Hands to detect which of nine palm zones your index finger touches and map them to phone actions (play music, next track, torch). Twist: design the imaginary interface on your forearm instead of your palm. #### Imaginary Reality Gaming: Quantum Basketball — Christian Holz (2013) - Video: https://www.youtube.com/watch?v=NNirAkibYGc - Interaction: Play, Shared & Social, Voice & Sound - Platform & tech: Wearable, fiducial markers, overhead cameras, audio - Idea: A ball game without a ball, where the ball lives only in the players' shared imagination. - What it is: Two teams play basketball in a hall with no ball at all; where the virtual ball is can only be inferred from how the other players act and from short audio cues, and power-ups change the rules mid-game. - Technique: Players wear fiducial markers tracked by overhead cameras; the game logic simulates the invisible ball and announces key events, such as receiving the ball, through audio. - Try it: Play catch with an invisible ball tracked by phones: the holder's phone vibrates and a speaker announces each pass, with no screens needed. Twist: add a power-up that makes the ball 'heavy' so the holder can only walk slowly. #### RealityCheck — Christian Holz, Eyal Ofek, Microsoft Research — Hrvoje Benko & Andy Wilson (2019) - Video: https://www.youtube.com/watch?v=IInv4mPruOE - Interaction: Portals & Worlds, Spatial Mapping, Shared & Social - Platform & tech: Headset, depth cameras, 3D reconstruction, VR - Idea: Let the real room leak into any VR game wherever you need it. - What it is: Inside existing VR games, real people, furniture and objects from the player's room are blended into the virtual world as live 3D reconstructions, so players can see a spectator, sit on the real sofa or pick up a real cup without taking off the headset. - Technique: Room-mounted depth cameras build a real-time 3D reconstruction that is composited into seven unmodified VR titles, with several blending styles (collision cues, visible spectators, stylised objects) to resolve conflicts. - Try it: In a Quest passthrough scene, cut a passthrough window around real objects you tag (a chair, a table) and tint it in the game's art style. Twist: people entering the room appear as characters from the game. #### CoolMoves — Christian Holz, Eyal Ofek, Mar Gonzalez-Franco, Microsoft Research — Hrvoje Benko & Andy Wilson (2021) - Video: https://www.youtube.com/watch?v=PO9pMAOM4N4 - Interaction: Hands & Body, Performance - Platform & tech: Headset, VR, motion capture database - Idea: Your small movements are accentuated into expressive full-body motion. - What it is: From only a VR headset and two hand controllers, the user's avatar performs stylised full-body moves such as dance and martial-arts poses that follow the user's own timing but look like a trained performer. - Technique: Head and hand trajectories are matched in real time against a motion-capture database, and similar clips are interpolated to synthesise accentuated full-body motion. - Try it: Map webcam-detected arm motion to a character that plays a kung-fu or dance clip whenever the arm speed passes a threshold. Twist: the less you move, the more the character exaggerates. #### SoundsRide — Christian Holz (2021) - Video: https://www.youtube.com/watch?v=tRqZnFnS6d4 - Interaction: Voice & Sound, Location & City - Platform & tech: Phone, GPS, audio AR, music mixing - Idea: Score a car ride like a film, timing the music's drops to the road. - What it is: While driving, the car's music is remixed live so that musical highlights land exactly on moments along the route, like the car bursting out of a tunnel or merging onto the highway. - Technique: The system predicts arrival times at route affordances (tunnel exits, highway entrances) from navigation data and rearranges and time-stretches music segments so high-contrast musical events align with them. - Try it: Write a web app that uses GPS and a list of landmarks on a walking route to fire a music drop exactly as you turn a corner or pass a gate. Twist: every walker on the route hears a different track synced to the same moments. #### TransforMR — Christian Holz (2021) - Video: https://www.youtube.com/watch?v=RsxdGwRvvEU - Interaction: Portals & Worlds, Perception & Effects, Location & City - Platform & tech: Phone, semantic segmentation, pose estimation, inpainting - Idea: Keep reality's motion but swap every object for something from another world. - What it is: Pointing a tablet at a street, passers-by are replaced by animated creatures that copy their poses, cars become other vehicles and landmarks are swapped, composing an alternate reality from whatever happens to be in view. - Technique: Semantic segmentation and 3D pose estimation find people and objects, inpainting removes them from the video, and pose-matched virtual substitutes are rendered in their place in real time with a server backend. - Try it: Use a body-tracking AR template (Lens Studio or ARKit) to replace each passer-by with a 3D character that mirrors their skeleton, and film a street corner. Twist: choose the substitute by how fast each person walks. #### Causality-preserving Asynchronous Reality — Christian Holz (2022) - Video: https://www.youtube.com/watch?v=U5dZlmQYIgs - Interaction: Perception & Effects, Shared & Social, Spatial Mapping - Platform & tech: Headset, depth cameras, volumetric capture - Idea: Let immersed users experience real-world events later, in the right causal order. - What it is: While a user works in a headset, a colleague comes in and writes on the whiteboard; the room is captured volumetrically and, when the user takes a break, only the events that matter are replayed in causal order so they can catch up. - Technique: Depth cameras reconstruct the room as a live point cloud, the system detects causal dependencies between actions and objects, and selectively plays back segments of reality in the headset. - Try it: Record a phone video of a desk while someone moves objects, then build an AR viewer that shows ghost replays of each change anchored where it happened. Twist: replay the events in reverse and see if the story still makes sense. #### Reality Rifts — Christian Holz (2023) - Video: https://www.youtube.com/watch?v=68oIgasJ0hs - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, sensors, actuators, Vive Tracker - Idea: Remove the cause but keep the effect, and people imagine the rest. - What it is: Physical contraptions are built with a key part missing, such as a water wheel that turns under a tap with no water, yet they still respond plausibly when used, so people fill in the missing part in their imagination and feel a sense of wonder. - Technique: Hidden sensors track the user's action (for example a tracked tap handle) and embedded actuators reproduce the plausible outcome by simulating the missing component's dynamics. - Try it: Build a prototype in which turning a dry tap spins a paper water wheel through a hidden servo, and let classmates try it without explanation. Twist: add a phone AR view that shows the imagined water to only some of them. #### SituationAdapt — Christian Holz (2024) - Video: https://www.youtube.com/watch?v=3C_87tDsTGw - Interaction: Spatial Mapping, Information & UI, Shared & Social - Platform & tech: Headset, vision-language model, UI optimization, mixed reality headset - Idea: Let a vision-language model act as the social and safety editor of your floating UI. - What it is: Mixed-reality windows that rearrange themselves as you move through the world: a vision-language model judges whether a panel would block a person you are talking to, a sign or an obstacle, and an optimizer moves it somewhere better. - Technique: A perception module detects objects and people, a vision-and-language model scores candidate UI placements for environmental and social suitability, and an optimization module computes the final layout. - Try it: In a Quest passthrough or phone AR app, place three virtual panels, then every few seconds send a camera frame plus the panel positions to a vision-language model asking which panel blocks something important, and move that panel. Twist: add a rule that panels must never cover a face and test it in a crowded hallway. ### Figmin XR (Overlay) *Independent team behind the mixed-reality creation app Figmin XR* Figmin XR is a HoloLens-born, now Quest/Vision Pro app where you sketch, collect and script 3D objects that live and collide with your real room. #### The Floor Is Lava! — Figmin XR (Overlay) (2023) - Video: https://www.youtube.com/watch?v=aOXpbDBwNyM - Interaction: Spatial Mapping, Play, Perception & Effects - Platform & tech: Headset, Figmin XR, Meta Quest 3, scene mesh - Idea: The living-room floor turns into lava and only the furniture is safe to stand on. - What it is: Using Figmin XR on Quest 3, the real living-room floor is covered with bubbling virtual lava while furniture remains safe islands. - Technique: Places a lava material on the detected floor plane while the scene mesh keeps real furniture visible and occluding. - Try it: Use phone AR to detect the floor and cover it with a lava material, and only let players step on safe mats marked with images that the app recognizes. Twist: the lava rises every 30 seconds. #### The Real World Is Your Canvas — Figmin XR (Overlay) (2023) - Video: https://www.youtube.com/watch?v=fA5taYjGwU8 - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Headset, Figmin XR, Meta Quest 3, spatial anchors - Idea: Paint among the furniture of a real room, and the artwork stays where you left it. - What it is: An artist paints glowing strokes and objects around a real room in Figmin XR; the art wraps around furniture and stays anchored in place. - Technique: Combines 3D brush strokes with persistent spatial anchors so drawings stay registered to the room between sessions. - Try it: Use a phone AR drawing app (such as a successor to Just a Line, or Reality Composer) to paint a 3D installation in a corner of the classroom, save the anchor, and reopen it next class. Twist: two people take turns finishing the piece on the same anchor. #### AI-generated 3D models with Luma Genie in mixed reality — Figmin XR (Overlay) (2024) - Video: https://www.youtube.com/watch?v=PFY1mk4gzjs - Interaction: Drawing & Making, Voice & Sound - Platform & tech: Headset, Figmin XR, Luma Genie, text-to-3D - Idea: Say an object and place it on your real table a minute later. - What it is: From inside Figmin XR's built-in web browser, users generate 3D models with Luma Genie from a text prompt and drop them straight into their mixed-reality scene. - Technique: A text-to-3D service (Luma Genie) returns a textured mesh that is imported through the in-app browser into the MR scene graph. - Try it: Generate three props for a tiny story with a free text-to-3D tool and stage them on a desk with WebAR. Twist: each prop must be generated from a line of dialogue spoken by another prop. #### Figmin XR: Create, Collect & Play — Figmin XR (Overlay) (2024) - Video: https://www.youtube.com/watch?v=MB8DaHEioEc - Interaction: Drawing & Making, Spatial Mapping, Play - Platform & tech: Headset, Figmin XR, Meta Quest 3, physics - Idea: Create, collect and play with your 3D works in mixed reality. - What it is: The app's trailer: users sketch, import 3D models, animate and play physics-driven creations that bounce off the real room in mixed reality. - Technique: Integrates sketching, model import, animation and a physics engine that uses the room mesh as colliders. - Try it: Import small object models that classmates each scanned into phone AR, arrange them on a table as a 'mixed-reality collection', and let them fall under physics. Twist: give each piece a short voice introduction. #### AI-powered scripting: a school of fish — Figmin XR (Overlay) (2025) - Video: https://www.youtube.com/watch?v=o_bxxVkN1Ag - Interaction: Play, Spatial Mapping - Platform & tech: Headset, Figmin XR, LLM, scripting API - Idea: Behaviours, not just objects, can be generated from words. - What it is: A school of fish swimming around a real room is fully controlled by an AI-generated script, previewing a feature where you describe a behaviour and watch it come to life. - Technique: An LLM writes a flocking (boids-style) script against Figmin's API, which moves many instanced fish while respecting the room's space. - Try it: Ask an LLM for a boids script in three.js, then run it in WebXR so 50 paper birds fly around your classroom. Twist: make the flock avoid wherever people are standing. #### AI-scripted Playable Character — Figmin XR (Overlay) (2025) - Video: https://www.youtube.com/watch?v=z1DbE2WMZRg - Interaction: Play, Spatial Mapping, Drawing & Making - Platform & tech: Headset, Figmin XR, scripting API, LLM code generation - Idea: AI writes the script that makes a little robot run and jump around your room. - What it is: A cute robot character that walks, runs, jumps and emotes around a real room, with its behaviour written entirely by AI through Figmin's scripting API. - Technique: An LLM generates Lua-style scripts against the app's API that drive character animation states and collisions with the room mesh. - Try it: Have students use ChatGPT to generate a Unity or Reality Composer behavior script that makes a small AR character patrol the table and jump when it bumps into a real cup. Twist: change the character's personality on the fly with natural language. #### ChatGPT 5 writing mixed-reality scripts in Figmin XR — Figmin XR (Overlay) (2025) - Video: https://www.youtube.com/watch?v=eY-538AArdI - Interaction: Play, Drawing & Making - Platform & tech: Headset, Figmin XR, GPT-5, scripting API - Idea: Describe a behaviour, and an object in your room starts doing it. - What it is: A classic AI coding challenge is taken into 3D: GPT-5 writes a script for Figmin XR's scripting API and the result runs as an interactive object in the user's room. - Technique: The LLM generates Lua-style code against Figmin's documented API; the code is loaded into an object and executed in the mixed-reality scene. - Try it: Give an LLM the documentation of a tiny AR scripting API (spin, move, change colour) and ask for a bouncing-ball script; run it in WebXR. Twist: the class writes prompts only, never code, and competes for the strangest physics. #### Mixed Reality RC Airplane — Figmin XR (Overlay) (2025) - Video: https://www.youtube.com/watch?v=-jNwln2vMAg - Interaction: Play, Spatial Mapping - Platform & tech: Headset, Figmin XR, scripting API, LLM - Idea: An AI-made RC plane flies around the living room and crashes into real walls. - What it is: An AI-generated model airplane, coded entirely by an LLM, flies around the living room under the player's control and collides with real walls. - Technique: Uses the room mesh as colliders and an LLM-written flight script mapping controller input to lift, thrust and roll. - Try it: Use the phone gyroscope to fly an AR paper plane around the classroom, and make it crash when it hits a detected wall. Twist: set up 'rings' through real doorframes to form a race course. #### AI Agent Vision: rebuilding a photo as an XR sketch — Figmin XR (Overlay) (2026) - Video: https://www.youtube.com/watch?v=Sbk1b6-lcK8 - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Headset, Figmin XR, Claude, Codex, scripting API - Idea: An AI agent that sketches in 3D next to you, starting from a photo. - What it is: Claude is shown a photo of a cabin and rebuilds it inside Figmin XR as a fully editable 3D sketch in mixed reality; Claude and Codex are compared on the same task. - Technique: An LLM agent with vision reads the image and calls Figmin's scripting API to place and shape primitive strokes, producing editable geometry rather than a frozen mesh. - Try it: Give an AI model a photo and ask it to describe the object as a list of 10 primitives (boxes, cylinders) with positions; place them in AR with A-Frame. Twist: ask for the same object as a child would build it. ### Google Creative Lab *Google's in-house creative team (Experiments with Google)* Google Creative Lab makes open-source experiments that show new uses of Google tech, including a series of ARCore experiments such as Just a Line, Drawalong AR, Sounds in Space and Lines of Play. #### AR Experiments (ARCore launch collection) — Google Creative Lab (2017) - Video: https://www.youtube.com/watch?v=7SwZUNDsWaM - Interaction: Drawing & Making, Portals & Worlds, Spatial Mapping - Platform & tech: Phone, ARCore, Unity, openFrameworks - Idea: A set of small open-source experiments released with ARCore that show how many creative directions AR can take. - What it is: A collection of small open-source ARCore prototypes released at launch, from drawing and physics toys to portals, showing creative uses of the new platform. - Technique: Small experiments each isolate one ARCore capability such as plane detection, hit testing, light estimation or anchors, showing how a single technique can carry a toy. - Try it: Have each group pick one basic AR capability (plane detection, light estimation, anchors, etc.) and build, within one hour, a small toy that relies on that capability alone. Twist: swap capabilities between groups and build another one. #### Just a Line — Google Creative Lab (2018) - Video: https://www.youtube.com/watch?v=IOKwGCQJVCw - Interaction: Drawing & Making, Shared & Social - Platform & tech: Phone, ARCore, ARKit, Cloud Anchors - Idea: An AR doodle app that draws just one line, and Android and iPhone users can draw together in the same space. - What it is: A minimal ARCore/ARKit app for drawing a single white line in the air; two people on Android and iPhone can draw together in the same space, and the result is a shareable video. - Technique: The line is built from successive camera or touch-ray positions in world space extruded into a ribbon mesh, and Cloud Anchors share the anchor so two phones draw together. - Try it: Use AR Foundation to connect the phone's positions as it moves into a 3D line, then record and export the screen. Twist: you get only one stroke, and lifting your finger ends it. #### Just a Line with Shantell Martin — Google Creative Lab (2018) - Video: https://www.youtube.com/watch?v=dRj5Xfjh7Dk - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Phone, ARCore - Idea: An artist carries her signature continuous line off the wall and into the air. - What it is: Artist Shantell Martin extends her signature continuous black-line drawings into the air around her using Just a Line. - Technique: The same world-space line drawing is used by an artist whose single continuous stroke becomes a spatial drawing around her body. - Try it: Invite a classmate to draw a single-stroke picture around their own body in AR and film it from several angles. Twist: export the line as a model and 3D print it. #### D.I.S.P.L.A.Y.E.D + Audio AR — Google Creative Lab (2019) - Video: https://www.youtube.com/watch?v=1kU-uDxupV0 - Interaction: Voice & Sound, Performance, Location & City - Platform & tech: Phone, audio AR, indoor positioning - Idea: As you approach a dancer, a spatial audio story belonging only to that dancer plays in your ears. - What it is: Heidi Latsky Dance and Google Creative Lab added position-based audio AR to a living-sculpture dance performance, so visitors hear dancers' voices and stories as they walk up to each performer. - Technique: Indoor positioning estimates the visitor's location, and distance to each performer's position controls the volume of their audio track. - Try it: Set up four 'sound spots' in the classroom and adjust each spot's audio volume by the phone's distance to it, creating a walkable sound map. Twist: each sound spot plays a personal story told by a classmate. #### Drawalong AR — Google Creative Lab (2019) - Video: https://www.youtube.com/watch?v=S9g1PqY19jE - Interaction: Drawing & Making, Information & UI - Platform & tech: Phone, ARCore - Idea: Turn a YouTube drawing tutorial into tracing paper floating over your real sketchbook. - What it is: Turns a YouTube drawing tutorial into virtual tracing paper: the video floats over your real sketchbook so you can trace along. Made with YouTube artist AmandaRachLee. - Technique: A video plane is anchored to a detected paper surface with adjustable transparency, so the user can trace the tutorial through the overlay. - Try it: Place a semi-transparent video or image layer on a tabletop plane, line it up with a real sheet of paper, and trace it. Twist: have the overlay appear stroke by stroke, like a teacher demonstrating. #### Lines of Play — Google Creative Lab (2020) - Video: https://www.youtube.com/watch?v=HbeEnkAE3Rg - Interaction: Spatial Mapping, Play, Drawing & Making - Platform & tech: Phone, ARCore Depth API - Idea: Set up AR dominoes on a real floor, where real objects hide them and get in their way. - What it is: Build domino chains on real floors and furniture; with the ARCore Depth API the dominoes hide behind real obstacles, collide with real objects and topple with spatial audio. - Technique: The ARCore Depth API supplies a depth map used for both occlusion and physics collision, so dominoes hide behind and bump into real objects. - Try it: Turn on environment depth, line up AR dominoes on a tabletop and make them stop when they hit a real book. Twist: each domino plays a note of a scale as it falls, forming a melody. #### Sounds in Space — Google Creative Lab (2020) - Video: https://www.youtube.com/watch?v=A9M45uRPaJ0 - Interaction: Voice & Sound, Location & City - Platform & tech: Phone, ARCore, open source - Idea: Place sounds in space to make an AR soundscape that changes as you walk. - What it is: An open-source toolkit for curating location-based audio in a room: sounds are placed in space and the phone plays a soundtrack that changes as the visitor walks. - Technique: ARCore anchors store audio sources at positions in a room, and playback volume and panning follow the phone's tracked position. - Try it: Use AR Foundation to place several invisible sound sources with 3D audio in a room so you hear different music layers as you walk around. Twist: let visitors record their own sounds and place them in the space. ### Klaus Obermaier *Media artist, director, choreographer and composer* Austrian artist who has projection-mapped moving dancers since 1998 (D.A.V.E., Vivisector) and, with Ars Electronica Futurelab, made real-time interactive body projection with Apparition (2004) and a stereoscopic Rite of Spring. #### D.A.V.E. – digital amplified video engine (with Chris Haring) — Klaus Obermaier (1998) - Video: https://www.youtube.com/watch?v=1bhNjYTQFQY - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, body projection, choreographed video - Idea: Use the dancer's own body as the screen and project impossible transformations onto it. - What it is: A solo dancer's body becomes the projection screen: morphing faces, ageing and impossible body distortions are mapped precisely onto the moving performer. - Technique: Pre-rendered video is choreographed frame-by-frame with the dancer so the projection lands on the body, relying on rehearsed timing rather than tracking. - Try it: Have a classmate in white clothes stand in front of the projector, and project pre-cut face or texture videos onto them following a rehearsed rhythm. Twist: add a small spot of light that follows them live via a camera. #### Vivisector (with Chris Haring) — Klaus Obermaier (2002) - Video: https://www.youtube.com/watch?v=Q0YEFX6Nk9k - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, projection as lighting, video masks - Idea: Light the dancers only with projection so their bodies get sliced and dissolved by light. - What it is: Four dancers are sculpted solely by projected light — no stage lighting — so their bodies appear sliced, dissolved and multiplied by the projection. - Technique: Projectors act as the only light source ('dancing light'), with video masks choreographed to reveal and fragment specific body regions. - Try it: Turn off the classroom lights and use only black-and-white projector masks to light parts of a classmate's body. Twist: track the hand with a camera so the spot of light follows it automatically. #### Apparition (with Ars Electronica Futurelab) — Klaus Obermaier (2004) - Video: https://www.youtube.com/watch?v=-wVq41Bi2yE - Interaction: Performance, Hands & Body, Projection - Platform & tech: Projection, infrared camera tracking, real-time generative graphics - Idea: Dancers' moves generate projections live on the backdrop and on their bodies, so body and image pull on each other. - What it is: Two dancers perform inside an interactive projection that is generated in real time from their movements, both on the backdrop and directly on their bodies. - Technique: Infrared camera tracking extracts the dancers' silhouettes and motion, feeding physics-based generative visuals that are projected back on the performers and the screen in real time. - Try it: Extract a dancer's outline with a camera and use it to drive particles or a spring mesh projected back onto the dancer in real time. Twist: make the image 'reject' the dancer, with patterns fleeing outward as the outline approaches. #### Le Sacre du Printemps (The Rite of Spring) — Klaus Obermaier (2006) - Video: https://www.youtube.com/watch?v=_6NQf-a5UAc - Interaction: Performance, Perception & Effects, Voice & Sound - Platform & tech: Projection, stereo cameras, stereoscopic projection, real-time rendering - Idea: Turn a dancer into a live stereoscopic 3D image so the audience watches a 'reshaped body' through 3D glasses. - What it is: A solo dancer is filmed by stereo cameras and turned in real time into a stereoscopic 3D projection that the audience watches with 3D glasses alongside a live orchestra. - Technique: Stereo camera pairs capture the dancer and real-time software transforms the image into generative stereoscopic geometry projected with polarised 3D projection. - Try it: Film a classmate dancing with two phones side by side, combine the footage into a red-cyan anaglyph and add live effects. Twist: let the music's beat control how much the body shatters. #### the concept of ... (here and now) — Klaus Obermaier (2010) - Video: https://www.youtube.com/watch?v=afbQQM6RTe8 - Interaction: Performance, Perception & Effects, Projection - Platform & tech: Projection, live camera, stereoscopic projection - Idea: Choreograph for both the audience and the camera so bodies merge into impossible shapes on screen. - What it is: A dance piece choreographed not only for the audience's viewpoint but also for camera perspectives, so dancers' bodies combine into impossible figures on the projected image. - Technique: Fixed live cameras film the stage from unusual angles and the feed is projected as a stereoscopic/composited view that only aligns from the camera's perspective. - Try it: Fix a camera shooting from above or from the side, and have two or three classmates pose on site so their bodies combine into a new shape on screen. Twist: the audience may only watch the projection, not the live scene, and must guess where everyone is really standing. #### Dancing House — Klaus Obermaier (2011) - Video: https://www.youtube.com/watch?v=su3Zrpck4P8 - Interaction: Hands & Body, Projection, Location & City - Platform & tech: Projection, projection mapping, camera body tracking - Idea: When you move, the whole old building twists and dances with you. - What it is: Interactive projection mapping on historic façades: the building bends, sways and 'dances' in response to the movements of people standing in front of it. - Technique: A camera tracks the audience's body movement and maps it onto deformation parameters of a 3D model of the façade that is projection-mapped onto the building. - Try it: Photograph and model a wall or a cardboard building, and use body pose (such as MediaPipe Pose) to bend the projected texture. Twist: when two people make opposite moves at the same time, the building "tears". #### Dancing Light / D.O.PE. — Klaus Obermaier (2015) - Video: https://www.youtube.com/watch?v=KXoN8zNc6sg - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, projector as light, masking - Idea: Make projected light itself a dance partner for the performer. - What it is: A performer dances with computer-controlled projector light that shapes, cuts and follows her body in a dark space. - Technique: A projector is used as a programmable spotlight whose masks are sequenced (and likely tracked) to sculpt the performer's silhouette in darkness. - Try it: Project moving shapes of light in a dark room and have a classmate improvise a "duet" with them. Twist: use a camera so the light shapes flee when a person gets close. #### EGO (with Stefano D'Alessio & Martina Menegon) — Klaus Obermaier (2015) - Video: https://www.youtube.com/watch?v=KzDifurF9wQ - Interaction: Hands & Body, Perception & Effects, Projection - Platform & tech: Projection, depth camera, real-time mesh distortion - Idea: Your movements warp your mirror image into an abstract shape that still looks like you. - What it is: A large projection shows the viewer's mirror image, but their movements stretch, fragment and deform it into an abstract self that still feels like 'me'. - Technique: A depth/video camera captures the viewer and a real-time mesh of their image is distorted by motion-driven parameters before projection. - Try it: Build a "funhouse mirror" with a camera and mesh deformation, so the faster you wave, the more your reflection gets stretched. Twist: add sound, giving each kind of deformation its own timbre. #### FACING LIFE [In Doc Tri Nation] — Klaus Obermaier (2022) - Video: https://www.youtube.com/watch?v=DSdO6zoP1Zc - Interaction: Face, Hands & Body, Projection - Platform & tech: Projection, Unity, camera tracking - Idea: Your face and movement drive a virtual child who is gradually disciplined by society. - What it is: A real-time rendered installation about social conditioning in which visitors' presence and faces animate child-like avatars that are gradually 'adapted' by social codes. - Technique: Unity real-time rendering driven by a camera, likely using face and body tracking to map the viewer onto the projected avatar. - Try it: Use Unity or web face tracking so classmates drive a cartoon child, then add rules that gradually restrict its expressions. Twist: the audience votes on which expression gets banned next. ### MIKIKO / ELEVENPLAY *Choreographer and stage director; founder of the dance company ELEVENPLAY* MIKIKO choreographs Perfume and BABYMETAL and leads ELEVENPLAY, whose decade-long collaboration with Rhizomatiks (Daito Manabe, Motoi Ishibashi) produced dances with drones, robot cubes, AR overlays and mixed-reality stages. #### cube (Sónar Tokyo) — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2013) - Video: https://www.youtube.com/watch?v=zBm3mJiJzh8 - Interaction: Performance, Hands & Body, Projection - Platform & tech: Projection, computer vision, LED, real-time visuals - Idea: Dancers move inside cube frames as their motion turns into live light and graphics. - What it is: ELEVENPLAY dancers perform inside and around cube frames while computer-vision tracking turns their movement into live graphics and LED light cues. - Technique: Computer-vision tracking (by Satoru Higa) of dancers drives real-time visuals and LED 'jockeying', likely projected onto and around the cube frames. - Try it: Use a camera to track the hand positions of classmates inside cardboard frames and control lines projected onto the frames in real time. Twist: each frame responds to only one person, forming an "ensemble of light". #### fly (dance with drones) — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2014) - Video: https://www.youtube.com/watch?v=HQLORg5COiU - Interaction: Performance, Tangible Objects, Spatial Mapping - Platform & tech: Projection, drones, motion capture - Idea: Make drones dance partners that fly in sync with the dancers. - What it is: Dancers perform a duet with small drones that fly precise trajectories around their bodies, as if the objects were partners. - Technique: Motion-capture tracking positions both dancers and drones in 3D space, and pre-programmed flight paths are synchronised to the choreography and music. - Try it: Program a small drone (such as a Tello) or a remote-control car to follow a path synced to music, and have classmates choreograph around it. Twist: generate the object's path from a recorded trajectory of a dancer's hand. #### 24 drones — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2015) - Video: https://www.youtube.com/watch?v=cYWvKudIIJ8 - Interaction: Performance, Spatial Mapping, Shared & Social - Platform & tech: Projection, light drones, motion capture - Idea: Twenty-four glowing drones form a dancing constellation of light. - What it is: Three dancers and 24 light-drones perform together; the drones form moving constellations of light that respond to and frame the choreography. - Technique: An indoor motion-capture system localises 24 custom light drones (designed by Motoi Ishibashi) whose light and position are programmed against the choreography. - Try it: Have 24 classmates each hold a phone flashlight and walk in the dark to coordinates given by a projection, forming shapes. Twist: generate the shapes in real time from a dancer's pose. #### shadow (drone with a spotlight) — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2015) - Video: https://www.youtube.com/watch?v=hX2TneyE41Q - Interaction: Performance, Perception & Effects, Tangible Objects - Platform & tech: Projection, light drones, motion capture - Idea: Spotlights carried by drones make the dancer's shadows part of the dance. - What it is: A single dancer is lit only by spotlights carried by drones; as the drones move, her shadows multiply, stretch and rotate on the walls. - Technique: Light-carrying drones follow programmed paths so the light source position is choreographed, controlling the size and direction of the dancer's shadows. - Try it: Turn off the lights, tie a phone flashlight to a long pole or a small cart, and move it while a classmate dances so their shadow shifts across the wall. Twist: use two light sources of different colors to make two-toned shadows. #### phosphere — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2017) - Video: https://www.youtube.com/watch?v=W7X6UqXm9eY - Interaction: Performance, Hands & Body, Projection - Platform & tech: Projection, marker tracking, 24 projectors, real-time visuals - Idea: Markers on dancers' wrists steer 24 projectors, turning gestures into geometry of light in space. - What it is: A dance installation homage to Schlemmer's Triadic Ballet: dancers wearing wrist markers trigger light from 24 projectors mounted overhead, turning gestures into three-dimensional light geometry. - Technique: Wrist-marker motion tracking feeds a real-time system that aims light from 24 truss-mounted projectors at computed 3D positions around the dancers. - Try it: Put reflective stickers or colored markers on a classmate's wrists, track them with a camera, and have a projector cast rings of light at the matching positions. Twist: automatically connect the two hands with a beam of light. #### S . P . A . C . E . — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2020) - Video: https://www.youtube.com/watch?v=bY0lMfl1rpI - Interaction: Performance, Hands & Body, Perception & Effects - Platform & tech: Desktop, multi-camera capture, real-time image processing - Idea: Break down and reassemble dancers' body data to create geometric worlds the body itself cannot form. - What it is: Made under social distancing, dancers are filmed by special camera rigs whose images are decomposed and rebuilt into geometric worlds that the body alone could not form. - Technique: Multi-camera capture and GPU image processing decompose each dancer's body data by rule sets and recompose it into real-time geometric visuals. - Try it: Get joint points with MediaPipe Pose and copy, mirror and rotate them by rule into kaleidoscope geometry. Twist: several classmates join a remote video call and their skeletons combine into a single figure. #### border 2021 — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2021) - Video: https://www.youtube.com/watch?v=rqmO7FA2vdw - Interaction: Performance, Portals & Worlds, Spatial Mapping - Platform & tech: Headset, mixed reality headsets, mobility devices, tracking - Idea: Seated on self-driving chairs, the audience wears MR headsets and watches real and virtual dancers merge. - What it is: An update of the 2015 'border' for post-COVID audiences: seated viewers ride personal mobility devices and watch real and virtual dancers merge through mixed-reality headsets, with an online version too. - Technique: Tracked mobility chairs move audience members through the stage while mixed-reality headsets render virtual dancers registered to the physical space. - Try it: Seat audience members on wheeled chairs pushed by classmates while they watch pre-placed virtual dancers in phone AR. Twist: at one moment the live dancer and the virtual dancer swap places. #### multiplex — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2021) - Video: https://www.youtube.com/watch?v=zeWtjavS7mc - Interaction: Performance, Projection, Tangible Objects - Platform & tech: Projection, Web, motion capture data, mobile robots, projection mapping - Idea: Real dancer data, projection and self-moving cubes together form a mixed-reality stage. - What it is: Motion data from five dancers, projected images and five autonomously moving robot cubes share two exhibition spaces and an online version, hybridising real and virtual dancers. - Technique: Recorded motion-capture data drives projection-mapped imagery on mobile robot cubes whose positions are tracked so the virtual dancers appear on their moving surfaces. - Try it: Put cardboard boxes on RC cars and have a projector follow the boxes to cast dancer silhouettes on them. Twist: viewers scan a QR code on their phones to switch which virtual dancer view they see. #### Syn: Unfolded Horizons of Bodily Senses — MIKIKO / ELEVENPLAY, Daito Manabe / Rhizomatiks (2023) - Video: https://www.youtube.com/watch?v=Tcj0lfyIEb8 - Interaction: Performance, Shared & Social, Portals & Worlds - Platform & tech: Headset, Projection, mixed reality, projection, stage tracking - Idea: The audience walks onto the stage and forms the show together with 24 dancers and mixed-reality imagery. - What it is: The opening show of TOKYO NODE: audiences walk onto a 1,500 m² stage where 24 dancers, a transforming space and mixed-reality experiences blur performer and spectator. - Technique: Combines live dancers with MR content, tracked stage elements and projection so the space reconfigures as visitors move through it. - Try it: Divide the classroom into zones where walking into each zone triggers different projections and music, and have classmates perform inside them. Twist: in the audience's phone AR, there is one more dancer than on the actual stage. ### Nexus Studios *Film, animation and interactive studio* Oscar-nominated animation studio whose interactive arm made some of the best-known narrative AR apps for Google, the BBC and The New Yorker. #### HotStepper — Nexus Studios (2017) - Video: https://www.youtube.com/watch?v=kR7RBUhnQmU - Interaction: Location & City, Play - Platform & tech: Phone, ARKit, Mapbox - Idea: A little AR figure walks ahead of you to show the way. - What it is: The first AR character-based wayfinding app: enter any address and an animated character walks ahead of you through the real street to lead the way. - Technique: ARKit world tracking combined with Mapbox routing and GPS lets an animated character be placed a few metres ahead of the user along the computed path. - Try it: Use AR to place a small character on the ground that walks ahead of you and leads you to a classroom along a few preset route segments. Twist: give the character a mind of its own so it sometimes takes a detour to a place it "likes." #### The New Yorker Innovators Issue AR Cover (with Christoph Niemann) — Nexus Studios (2017) - Video: https://www.youtube.com/watch?v=ZIlFhkmSZEU - Interaction: Tangible Objects, Portals & Worlds - Platform & tech: Phone, ARKit, Unity - Idea: A subway station on a magazine cover raises a whole 3D Manhattan. - What it is: Christoph Niemann's New Yorker cover of a subway platform opens in AR into a 3D Manhattan that rises out of the magazine. - Technique: Image tracking on the printed cover anchors an animated 3D city model in Unity/ARKit that rises out of the page with depth-consistent perspective. - Try it: Make AR for a postcard or poster of your own city so the pictured city rises in 3D from the paper. Twist: the risen city shows what it looked like 50 years ago or will look like 50 years from now. #### 1600: White House AR (dollar bill) — Nexus Studios (2018) - Video: https://www.youtube.com/watch?v=emiGJNa9gwg - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, ARKit, image tracking - Idea: A 3D White House grows out of a one-dollar bill. - What it is: Point a phone at a US one-dollar bill to see a 3D White House rise from it and experience a year in the life of 1600 Pennsylvania Avenue. - Technique: Image tracking recognises the one-dollar bill's engraving as a target and anchors an animated 3D White House model with time-based storytelling. - Try it: Use a banknote or ticket as an image target, make the building or pattern on it rise in 3D, and tell a short story. Twist: make the story about the banknote's own journey through circulation. #### BBC Civilisations AR — Nexus Studios (2018) - Video: https://www.youtube.com/watch?v=YkOrx5xcgss - Interaction: Information & UI, Spatial Mapping - Platform & tech: Phone, ARKit, ARCore, Unity - Idea: Bring museum artifacts into your living room and see inside them. - What it is: An AR app with BBC R&D that places over 30 museum artefacts in your room, which you can inspect, x-ray and hear narrated. - Technique: Plane detection places photogrammetry scans of artefacts at real scale, with a clipping-plane or shader cross-section for x-ray views and attached audio narration. - Try it: Scan an old object from home with phone photogrammetry (Polycam / RealityScan), display it enlarged in AR, and add a cross-section or 'x-ray' effect with voice narration. Twist: let the object itself be the narrator. #### Childish Gambino Playmoji (Google Pixel) — Nexus Studios (2018) - Video: https://www.youtube.com/watch?v=DQTwJQfLH5M - Interaction: Hands & Body, Performance - Platform & tech: Phone, ARCore, Pixel Playground - Idea: An AR celebrity who dances in the frame with you. - What it is: A dancing AR Childish Gambino character for Pixel's Playground that reacts to the user and dances alongside people in the camera view. - Technique: ARCore plane detection anchors an animated character, and person detection or pose estimation in the camera frame lets it react to and dance beside people. - Try it: Place a Mixamo-animated character next to a classmate in AR and switch dances depending on whether MediaPipe detects a raised hand. Twist: have the character learn and copy one of a person's moves, then 'teach' it to the next person. #### Back to the Moon AR — Nexus Studios (2019) - Video: https://www.youtube.com/watch?v=WX9KuX7F9y0 - Interaction: Spatial Mapping, Portals & Worlds - Platform & tech: Phone, ARCore, Google Spotlight Stories - Idea: Méliès's trip to the moon plays out again on your tabletop in AR. - What it is: An AR version of the Emmy-nominated Google Spotlight Story about Georges Méliès, letting the animated film play out on your table. - Technique: ARCore plane detection places an animated diorama on a tabletop, with the viewer's phone movement acting as the camera for a pre-authored real-time story. - Try it: Build a 30-second tabletop theater in AR (for example, a scene from a classic silent film) where the audience moves the phone as the camera. Twist: a hidden plot point is revealed only from one specific angle. #### Changdeok ARirang — Nexus Studios (2019) - Video: https://www.youtube.com/watch?v=3-CEMYrtD-M - Interaction: Location & City, Information & UI - Platform & tech: Phone, 5G, AR - Idea: AR characters tell the history of Changdeokgung Palace on site. - What it is: A location-based AR tour of Seoul's Changdeokgung palace with animated guides and historical stories anchored to the grounds. - Technique: Location-based AR with GPS plus local plane detection (and 5G streaming of heavy assets) anchors animated guides and scenes at specific palace locations. - Try it: Pick three campus spots with a story and use WebAR to place an animated character at each one telling its past, linked into a guided tour. Twist: the guides contradict each other, and visitors must decide which version is true. #### Notable Women — Nexus Studios (2019) - Video: https://www.youtube.com/watch?v=DuxXeeGg-T0 - Interaction: Tangible Objects, Face - Platform & tech: Phone, ARCore, image tracking - Idea: Use AR to print portraits of notable women onto US dollars. - What it is: With Google and former US Treasurer Rosie Rios, the app puts portraits of historic American women onto US banknotes in AR. - Technique: ARCore augmented images detect banknote designs and replace the printed portrait region with a warped, lit portrait of a historic woman. - Try it: Use image tracking to recognize a banknote or ID photo and replace the portrait with someone you think deserves remembrance but has been overlooked. Twist: add a 15-second voice clip explaining why this person belongs there. ### OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) *Digital art group pioneering motion-captured drawing, 3D projection and AI-driven artworks* Marc Downie, Shelley Eshkar and Paul Kaiser made landmark dance-and-data works with Merce Cunningham, Bill T. Jones and Trisha Brown (Hand-drawn Spaces, Ghostcatching, Biped, Loops), then took the same hand-drawn, motion-captured figures into sidewalks, building facades, interactive 3D rooms and VR, all built with their own open-source environment Field. #### Hand-drawn Spaces — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (1998) - Video: https://vimeo.com/32776116 - Interaction: Projection, Performance, Voice & Sound - Platform & tech: Projection, motion capture, multi-screen projection - Idea: Motion-captured dancers drawn by hand travel through the room between screens. - What it is: A three-screen installation made with Merce Cunningham where life-size hand-drawn dancers perform his choreography in a limitless 3D space, walking out of one screen and into another across the viewers' space. - Technique: Cunningham's choreography was motion-captured and rendered as drawn figures in a virtual space larger than the three projections, with spatial sound by Ron Kuivila tracking their positions. - Try it: Set up two projections on facing walls and animate a stick-figure dancer (from a mocap file) that exits one wall and, after a delay matching the walking time, enters the other. Twist: add a speaker path so you hear it cross the room. #### Pedestrian — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2002) - Video: https://vimeo.com/186079084 - Interaction: Projection, Location & City, Perception & Effects - Platform & tech: Projection, Field, motion capture, projector - Idea: A tiny city of walking people appears in the pavement you are standing on. - What it is: A projector mounted high above a city sidewalk throws a miniature world onto the pavement: tiny people walk, gather and cross streets that seem to float on and inside the concrete under passers-by's feet. - Technique: Real-time 3D rendering from a large library of motion-captured walking is projected straight down, so the synthetic figures merge with the texture of the ground as a trompe-l'oeil. - Try it: Mount a projector or phone projector pointing at the floor and project looping top-down footage of tiny walking figures (Mixamo characters rendered from above) scaled to 5 cm tall. Twist: make the tiny people scatter when a real foot's shadow falls on them. #### Breath — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2007) - Video: https://www.youtube.com/watch?v=A6SAIdHukGg - Interaction: Voice & Sound, Location & City - Platform & tech: Projection, Field, lightboxes, audio analysis - Idea: A building's facade breathes in time with music. - What it is: A public artwork at Lincoln Center where banners on Avery Fisher Hall's columns and a row of tall lightboxes in the colonnade brighten and dim with the ebb and flow of a live musical composition. - Technique: Software analyses a live musical composition and couples its dynamics to the lighting of the banners and lightboxes, for the month-long Mostly Mozart Festival. - Try it: Connect a microphone to a row of smart bulbs or an LED strip so the loudness of a live singer makes a hallway or window 'breathe'. Twist: make the light lead the singer by a beat, so they have to follow it. #### Point A to B — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2007) - Video: https://vimeo.com/4377157 - Interaction: Perception & Effects, Projection, Play - Platform & tech: Projection, Field, motion capture - Idea: Split one virtual space across two screens so the gap itself becomes part of the course. - What it is: A dual-screen installation inspired by parkour, in which virtual traceurs race through a vertiginous world where the two projections rarely line up and the space between them opens like a chasm. - Technique: Motion-captured parkour runs drive figures in a real-time renderer built on Field, displayed across two parallel projections that only occasionally form one panorama. - Try it: Place two tablets or projections a metre apart and animate a runner who jumps from one to the other, timing the jump so the empty gap feels dangerous. Twist: let viewers widen the gap by moving one screen. #### After Ghostcatching — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2010) - Video: https://vimeo.com/26407428 - Interaction: Hands & Body, Perception & Effects, Performance - Platform & tech: Projection, Field, motion capture, stereoscopic 3D - Idea: A dancer's captured motion becomes drawn ghosts that escape their frame. - What it is: A stereoscopic 3D reworking of the 1999 Ghostcatching, in which Bill T. Jones's motion-captured dance and voice become drawn, multiplying figures that break out of their boxes. - Technique: Jones's original motion-capture data are re-rendered in real time as stereoscopic line drawings in Field, with generative rules multiplying and trailing the figures. - Try it: Take one free mocap clip, render it as lines only (no mesh) in three.js, and add copies that follow the original with growing delay until they break a drawn box. Twist: view it in WebXR so the ghosts surround you. #### Crossings — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2010) - Video: https://vimeo.com/15550692 - Interaction: Projection, Location & City, Performance - Platform & tech: Projection, Field, projection mapping, motion capture - Idea: Let figures walk across a building's broken geometry as if it were terrain. - What it is: For Toronto's Nuit Blanche, huge drawn figures walked, climbed and crossed the angled glass-and-metal facets of Daniel Libeskind's Crystal extension of the Royal Ontario Museum all night. - Technique: Motion-captured figures rendered in Field are projection-mapped across the facade's facets with multiple projectors, in a 'site-specifiable' system that adapts to any complex architecture. - Try it: Projection-map a walking character onto two or three boxes stacked at angles in the classroom using MadMapper or a free mapping tool, so it seems to step from face to face. Twist: make the figure's gait change on each surface. #### Into the Forest — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2011) - Video: https://vimeo.com/38462922 - Interaction: Portals & Worlds, Hands & Body, Projection - Platform & tech: Projection, Field, stereoscopic 3D, camera capture - Idea: Step into a spotlight and you become a figure in a painted 3D world. - What it is: A stereoscopic 3D installation of a painterly forest where virtual children play; when a spotlight comes on, any visitor stepping into it is captured and placed inside the 3D world among the children. - Technique: A camera captures the visitor inside the lit area, and the extracted figure is rendered as a painterly element within a real-time stereoscopic scene built in Field. - Try it: Mark a spot on the floor with tape and a lamp; when someone stands there, cut them out with segmentation and composite them into a projected illustrated scene. Twist: the longer they stand still, the more painterly they become. #### Loops — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2011) - Video: https://vimeo.com/25509279 - Interaction: Hands & Body, Perception & Effects, Voice & Sound - Platform & tech: Projection, Field, motion capture, stereoscopic 3D - Idea: Portray a person by their motion, not their appearance. - What it is: An abstract portrait of Merce Cunningham built from motion capture of his solo dance for hands: the joints become nodes in a shifting cat's-cradle network while his voice, reading a boyhood diary, drives a virtual prepared piano. - Technique: Hand motion-capture data are treated as a network of nodes whose connections are rewired by autonomous agents in Field, rendered in 3D, and released as open-source choreography data. - Try it: Track a classmate's hands with MediaPipe for 30 seconds, then draw lines between all 42 landmarks whose distance is below a threshold, creating a moving web. Twist: sonify the number of connections as a piano pitch. #### Drawn Together — OpenEndedGroup (Marc Downie, Shelley Eshkar, Paul Kaiser) (2012) - Video: https://vimeo.com/37623623 - Interaction: Drawing & Making, Voice & Sound, Tangible Objects - Platform & tech: Projection, Field, stereoscopic 3D, contact microphones, AI - Idea: Draw a duet with a machine that replies in 3D above your paper. - What it is: Visitors draw with chalk on black paper at a drafting table while wearing 3D glasses and headphones; they hear their own marks amplified, and an artificial intelligence answers with its own lines projected in stereoscopic 3D above the page. - Technique: Contact microphones in the table body pick up the sound of drawing, cameras read the marks, and AI agents built in Field generate responsive strokes projected in stereo 3D onto the drawing. - Try it: Point a phone camera at paper, detect new pen strokes with simple frame differencing, and answer each one with a mirrored or rotated virtual stroke floating above the page in AR. Twist: let the machine answer only after you stop drawing, like a conversation. ### Random International (Hannes Koch & Florian Ortkrass) *Art studio* Studio founded in 2005 that makes camera-driven installations where matter reacts to the viewer's presence — mirrors that turn to stare, light that paints your reflection, rain that stops where you walk. #### Temporary Printing Machine — Random International (Hannes Koch & Florian Ortkrass) (2005) - Video: https://www.youtube.com/watch?v=YgNFpS0bxY8 - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, UV light, phosphorescent surface, moving print head - Idea: An image that fades invites people to make more — temporary AR marks are playful because they do not last. - What it is: A moving head of UV lights passes over a glow-in-the-dark surface and 'prints' images and text in light, which slowly fade away. - Technique: A row of UV LEDs is scanned across a phosphorescent surface and switched per pixel, so the charged pigment glows and fades over minutes. - Try it: Let users 'print' glowing words onto a real wall in AR by sweeping their phone across it, with the marks fading over two minutes. Twist: a mark only stays if a second person sweeps over it again. #### Audience — Random International (Hannes Koch & Florian Ortkrass), Chris O'Shea (2008) - Video: https://www.youtube.com/watch?v=JuKi35j3Dwk - Interaction: Hands & Body, Gaze & Attention, Tangible Objects - Platform & tech: Desktop, camera tracking, servo motors, mirrors - Idea: A crowd of mirrors turns its heads in unison to stare at you like an audience. - What it is: A field of small head-sized mirrors sits idle until a visitor enters; then they all turn to follow and 'look at' that person, so the viewer sees themselves reflected everywhere. - Technique: A camera tracks the chosen visitor's position and each motorised mirror computes the pan/tilt needed to face them. - Try it: Build three to five 'eyes' from servos and small mirrors, and use the face position from a camera to turn them. Twist: when two people are present, the mirrors 'split into two camps', each watching one person. #### You Fade to Light — Random International (Hannes Koch & Florian Ortkrass) (2009) - Video: https://www.youtube.com/watch?v=aIVKVoFYvZ8 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, camera, OLED panels, custom software - Idea: Turn your mirror image into light that slowly fades away. - What it is: A wall of OLED panels translates the viewer's mirror image into glowing light that lingers and fades as they move. - Technique: A camera image is downsampled to the OLED grid and each panel's brightness follows the pixel with a slow decay, producing light 'afterimages'. - Try it: Make a low-resolution pixel mirror from the camera image and add afterglow decay to each pixel. Twist: build a physical version with LED strips or paper lanterns. #### Swarm Light — Random International (Hannes Koch & Florian Ortkrass) (2010) - Video: https://www.youtube.com/watch?v=Bhda3ASVMuE - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, LED cube matrix, swarm algorithm, sound sensing - Idea: A swarm drawn in real 3D pixels feels alive in a way a screen never does — AR flocks placed in real space inherit this power. - What it is: Three cubes of hanging LEDs show points of light moving like a flock of birds, reacting to the sound of people in the room. - Technique: A 3D grid of LEDs displays a flocking simulation; microphones in the room modulate the swarm's behaviour. - Try it: Place a flock of 300 glowing points inside a wireframe cube on a real table in AR using a boids simulation, and scatter them when the microphone hears a loud sound. Twist: make the flock slowly follow whoever is quietest. #### Study of Time — Random International (Hannes Koch & Florian Ortkrass) (2011) - Video: https://www.youtube.com/watch?v=Qq9qhtbXv28 - Interaction: Information & UI, Perception & Effects - Platform & tech: Desktop, LED grid, clock logic, custom electronics - Idea: Time can be felt as a moving light rather than read as numbers — AR clocks can be ambient instead of literal. - What it is: A hanging grid of LEDs displays the passing of time as a moving shadow of light that travels across it like the hand of a clock. - Technique: A grid of LEDs is driven by a clock routine that sweeps illumination and shadow across the surface as the hours pass. - Try it: Hang a virtual light panel over a real doorway in AR where a band of light sweeps across it once per minute. Twist: the sweep speed follows how many people walked through the door in the last hour. #### Future Self (with Wayne McGregor) — Random International (Hannes Koch & Florian Ortkrass) (2012) - Video: https://www.youtube.com/watch?v=Jqn1cMY8oGM - Interaction: Hands & Body, Performance, Perception & Effects - Platform & tech: Desktop, depth cameras, volumetric LED matrix - Idea: An array of LED rods forms a "light double" that mirrors your movements. - What it is: A three-dimensional sculpture of hanging LED rods renders a light-figure that mirrors and responds to the dancers or visitors moving in front of it. - Technique: Depth cameras capture the body and drive a volumetric LED matrix so the silhouette is rendered as points of light in 3D. - Try it: Turn your body into a 3D point cloud with a Kinect or a phone depth camera and display it as glowing dots on screen. Twist: make the light double lag half a beat behind you, the opposite of a "future you". #### Rain Room — Random International (Hannes Koch & Florian Ortkrass) (2012) - Video: https://www.youtube.com/watch?v=FslABAyj2OA - Interaction: Hands & Body, Spatial Mapping, Perception & Effects - Platform & tech: Desktop, 3D tracking cameras, solenoid valves - Idea: Walk through rain that stops just for you. - What it is: A 100 m² field of continuously falling water pauses wherever a person walks, so visitors move through rain without getting wet. - Technique: 3D cameras track visitors from above and a grid of solenoid valves shuts off the water tiles directly around each person in real time. - Try it: Make "virtual rain" with an overhead camera and floor projection, so raindrops automatically avoid the area around each person. Twist: build a one-meter-square physical prototype with solenoid valves or a small water pump. #### Study for Fifteen Points — Random International (Hannes Koch & Florian Ortkrass) (2015) - Video: https://www.youtube.com/watch?v=qnkxo7CWACs - Interaction: Perception & Effects, Hands & Body - Platform & tech: Desktop, 15 LED points on robotic rods, motion capture data - Idea: The brain needs only 15 moving points to see a person (biological motion) — AR characters can be radically minimal. - What it is: Fifteen points of light on moving rods walk in the dark; with only these dots, the viewer sees a human figure walking. - Technique: Motion-capture data of a walking person drives 15 mechanical arms that move LED points through the recorded joint positions. - Try it: Place 15 glowing dots in AR driven by a motion-capture walk cycle and let viewers guess the gender, mood or age of the walker. Twist: record your own walk with body tracking and see if classmates can recognise you. #### Swarm Study / VII — Random International (Hannes Koch & Florian Ortkrass) (2015) - Video: https://www.youtube.com/watch?v=ajV8A5Y2_dE - Interaction: Perception & Effects - Platform & tech: Desktop, brass rods, LEDs, swarm algorithm - Idea: Collective motion reads as life at any scale — a few hundred AR particles following flocking rules can feel like creatures. - What it is: A grid of brass rods tipped with lights shows glowing points moving through the structure like a school of fish or flock of birds. - Technique: An array of individually addressable LEDs on rods renders a real-time swarm simulation in three dimensions. - Try it: Build a sparse 3D grid of glowing rod tips in AR and render a flocking swarm through it so points only light up where a rod exists. Twist: let the swarm avoid the viewer's phone like fish avoiding a hand. ### Tasuku Takahashi *Interactive art engineer at teamLab; VFX Graph and XR prototyper* An engineer at teamLab who spends his own time on real-time VFX in Unity: VFX Graph on the open-source Project North Star AR headset, volumetric dancers captured with Depthkit Studio, GPU fire and fluid solvers, and sci-fi hand interfaces. Almost every experiment is published on GitHub. #### Project North Star: Strange Attractors in VFX Graph — Tasuku Takahashi (2019) - Video: https://www.youtube.com/watch?v=kLG291XVf_k - Source code: https://github.com/supertask/VFXNorthStar - Interaction: Perception & Effects - Platform & tech: Headset, Unity, VFX Graph, Project North Star - Idea: Let a chaotic equation, not an animator, choreograph a swarm of light in the room. - What it is: Thomas and Lorenz strange attractors, drawn by thousands of VFX Graph particles, hang in the air in front of the wearer of a Project North Star AR headset. - Technique: In the VFX Graph update context each particle's velocity is computed from the attractor's differential equations (Thomas cyclically symmetric and Lorenz systems), so the swarm traces the attractor's shape as it moves. - Try it: Place a Lorenz-attractor particle system on a detected table with AR Foundation and view it on a phone or HoloKit. Twist: map the attractor's parameters to your distance from it, so walking closer reshapes the chaos. #### Project North Star: VFX Graph with RealSense — Tasuku Takahashi (2019) - Video: https://www.youtube.com/watch?v=oYYl_ALUDQI - Source code: https://github.com/supertask/VFXNorthStar - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Headset, Unity, VFX Graph, Intel RealSense, Project North Star, Leap Motion - Idea: Feed a live depth camera back into an AR headset so the room itself becomes particle material. - What it is: Seen through the open-source Project North Star AR headset, the room captured by an Intel RealSense depth camera is redrawn as a cloud of glowing VFX Graph particles floating over the real space. - Technique: Keijiro Takahashi's Rsvfx converts each RealSense depth and colour frame into position and colour maps, which VFX Graph samples to spawn particles; the scene is rendered for North Star's see-through reflectors with Leap Motion's North Star rig. - Try it: Stream the iPhone LiDAR depth map into a VFX Graph (AR Foundation plus a Keijiro-style depth-to-position map) and redraw the room as drifting points on the phone or in HoloKit. Twist: when you stand still, let the points slowly fall to the floor like sand. #### 3D Volumetric Fire — Tasuku Takahashi (2021) - Video: https://x.com/supertask_jp/status/1358792459543662592 - Source code: https://github.com/supertask/UnityVolumetricFire3D - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, compute shader, raymarching - Idea: Simulate fire as a real 3D gas instead of faking it with billboards. - What it is: A mesh bursts into volumetric flames that curl and rise in 3D, rendered as a glowing gas rather than flat fire sprites. - Technique: A GPU grid fluid solver in compute shaders (after GPU Gems 3's 3D fluid chapter) advects velocity, temperature and density on a voxel grid, the source mesh is voxelised to emit fuel, and the volume is raymarched for display. - Try it: Port a small voxel-grid smoke solver to a phone and anchor the volume above a real candle or cup with AR Foundation. Twist: let blowing into the microphone push the smoke sideways. #### MLS-MPM Fluid in Unity — Tasuku Takahashi (2021) - Video: https://x.com/supertask_jp/status/1416315883832561673 - Source code: https://github.com/supertask/Unity-MLS-MPM-Fluid-Test - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, compute shader, MLS-MPM - Idea: Use the same particle-grid method as film VFX to get believable liquid at interactive speed. - What it is: Thousands of particles slosh and splash as a liquid inside a box, simulated in real time on the GPU in Unity. - Technique: The Moving Least Squares Material Point Method moves particle momentum onto a background grid, updates the grid, and transfers velocities back each step in compute shaders, with lock-free atomic writes and a bitonic sort to keep neighbours together. - Try it: Run a 2D MLS-MPM (for example the 88-line mpm88 algorithm) in a compute shader and show it on a virtual glass panel standing on a real table in AR. Twist: tilt the phone to change gravity. #### VFX Graph Dragon — Tasuku Takahashi (2021) - Video: https://x.com/supertask_jp/status/1463189020964364289 - Source code: https://github.com/TranscendVFX/VFXGraphIntermediate - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Shader Graph - Idea: Build a creature out of particles, so one dragon becomes a thousand for free. - What it is: A flock of dragons made only of particle strips and scale-like quads snakes through a dark space; the number of dragons can be increased almost without limit. - Technique: Likely a head particle follows a random-walk path while trail (strip) particles and instanced scale quads spawned behind it form the body, all inside VFX Graph without skinned meshes. - Try it: Make a particle serpent that follows a random-walk path around a detected AR plane on a phone. Twist: make the serpent flee from the phone camera when you get close. #### AR VJing + Volumetric Video — Tasuku Takahashi (2022) - Video: https://x.com/supertask_jp/status/1504792152315871238 - Source code: https://github.com/supertask/4DVFXProject - Interaction: Performance, Perception & Effects - Platform & tech: Phone, Unity, VFX Graph, Depthkit Studio, volumetric video - Idea: VJ a captured human body in AR the way you would VJ a video clip. - What it is: A dancer captured as volumetric video at the Rememory studio appears as a small hologram on a desk; hitting keys on a controller switches the VFX Graph effects (leaves, rain, orbits) that wrap around her body. - Technique: Depthkit Studio exports the capture as an mp4 with packed colour and depth plus a JSON of camera metadata; Unity decodes each frame into a point cloud whose positions feed VFX Graph emitters, and live input switches between effect graphs. - Try it: Record a friend with an iPhone LiDAR volumetric app (or Record3D), play the clip back in AR Foundation as points, and bind three VFX Graph looks to three on-screen buttons. Twist: let the music's beat, not the buttons, trigger the switches. #### VFX Dancer — Tasuku Takahashi (2022) - Video: https://x.com/supertask_jp/status/1524424117948088321 - Source code: https://github.com/TranscendVFX/VFXGraphIntermediate - Interaction: Hands & Body, Performance - Platform & tech: Desktop, Unity, VFX Graph, Mixamo - Idea: Make motion visible by letting the body leave its own history behind as light. - What it is: An animated dancer sheds long glowing trails and particles from her whole body, so every turn of the choreography leaves ribbons of light in the air. - Technique: VFX Graph samples positions on the animated skinned mesh every frame and spawns particles and trail strips there, inspired by Keijiro Takahashi's Skinner; the animation is a Mixamo clip. - Try it: Drive a Mixamo or ARKit body-tracked avatar with a trail-emitting VFX Graph and view it at full scale in AR next to a real dancer. Twist: colour the trails by the speed of each limb. #### Interactive VJ with Azure Kinect — Tasuku Takahashi (2023) - Video: https://x.com/supertask_jp/status/1612166613931032577 - Source code: https://github.com/supertask/InteractiveVJ - Interaction: Hands & Body, Performance - Platform & tech: Desktop, Unity, VFX Graph, Azure Kinect - Idea: Turn your own body into the VJ source instead of a video clip. - What it is: A DIY VJ system: the performer, seen by an Azure Kinect, is outlined in glowing blue lines and surrounded by effects that are switched from an on-screen control panel. - Technique: Azure Kinect colour and depth frames are converted to point positions and a body mask that feed VFX Graph effects (outlines, particles), with parameters exposed in a runtime GUI; likely built on Keijiro-style Kinect-to-VFX helpers. - Try it: Build a phone VJ mirror: use ARKit people segmentation to outline yourself with a glowing VFX Graph edge and switch between three looks with a tap. Twist: let the room's music volume set the glow width. #### Virtual Wearable — Tasuku Takahashi (2023) - Video: https://x.com/supertask_jp/status/1635235099989143552 - Source code: https://github.com/supertask/VirtualWearable - Interaction: Hands & Body, Information & UI - Platform & tech: Desktop, Unity, VFX Graph, Ultraleap, Vuforia - Idea: Replace rectangular screens with interfaces that are worn on the body like jewellery. - What it is: Holographic rings and app icons wrap around the user's wrist and hand; turning and closing the hand calls up and selects apps in a sci-fi interface. - Technique: Ultraleap hand tracking gives the hand skeleton, VFX Graph and Shader Graph draw the circular holographic UI anchored to wrist and palm joints, and Vuforia registers the camera view. - Try it: Use a phone's hand tracking (for example MediaPipe Hands or visionOS hand anchors) to attach a ring of three glowing icons to your wrist in AR. Twist: the ring only appears when you look at your palm. ### Thad Starner *Wearable-computing pioneer; professor at Georgia Tech; technical lead on Google Glass* Has worn a head-up display and one-handed Twiddler keyboard daily since 1993, co-founded the MIT wearable computing project, and runs the Contextual Computing Group at Georgia Tech. He was a technical lead and manager on Google Glass and works on sign-language recognition, passive haptic learning and head-worn assistants. #### Remembrance Agent — Thad Starner (1996) - Video: https://www.youtube.com/watch?v=k-zThJX920w - Interaction: Information & UI, Voice & Sound - Platform & tech: Wearable, Remembrance Agent, Twiddler, head-up display - Idea: A just-in-time memory: the wearable listens to what you are doing and surfaces your own past notes without being asked. - What it is: In a 1996 interview with Alan Alda, Starner types notes on a one-handed keyboard while his head-up display quietly shows one-line summaries of old files that relate to what is being said. - Technique: A background process continuously runs text similarity search over the user's files against the last few typed lines and displays the best matches at the bottom of the HUD. - Try it: Build a phone overlay that listens to speech, transcribes keywords and shows one matching line from a class notes file. Twist: allow only one suggestion per minute so it stays unobtrusive. #### ASL recognition from a wearable camera — Thad Starner (1998) - Video: https://www.youtube.com/watch?v=XWQN68ySnGk - Interaction: Hands & Body, Information & UI - Platform & tech: Wearable, hidden Markov models, wearable camera - Idea: Point the camera at your own hands so the wearable understands the language they speak. - What it is: A cap-mounted camera looks down at the wearer's hands while they sign American Sign Language sentences, and a recogniser turns the signs into English words in real time. - Technique: Hand blobs are tracked from the downward-looking camera and fed as features to hidden Markov models trained on sentence-level ASL. - Try it: Use MediaPipe hand tracking on a phone worn on the chest to recognise five hand signs and speak them aloud. Twist: design a new five-sign vocabulary for controlling music. #### DUCK! wearable agent for Patrol — Thad Starner (1998) - Video: https://www.youtube.com/watch?v=nf5yrpCyfHA - Interaction: Play, Shared & Social, Location & City - Platform & tech: Wearable, wearable computer, gesture recognition, location beacons - Idea: Let a wearable read the game from your body so teammates get tactical information without talking. - What it is: In Patrol, a dart-gun tag game played in MIT buildings, each player wears a computer; the DUCK! agent uses location and gesture recognition to show teammates' positions on a head-up map and flash when someone is vulnerable while reloading. - Technique: Room-level location beacons and accelerometer gesture recognition detect firing and reloading, and the states are shared over the network to everyone's HUD. - Try it: Run a hide-and-seek round where each phone uses its accelerometer to detect a 'crouch' and shares it to a team map. Twist: reveal a player's position only when they move fast. #### Gesture Pendant — Thad Starner (2000) - Video: https://www.youtube.com/watch?v=9YsYo9_yq2A - Interaction: Hands & Body, Tangible Objects - Platform & tech: Wearable, infrared camera, gesture recognition - Idea: Wear the sensor instead of instrumenting the home: gestures travel with you. - What it is: A pendant with an infrared camera and LEDs hangs on the chest; waving the hand in front of it controls lights and devices in the Georgia Tech Aware Home. - Technique: Self-illuminating infrared lights the hand so a small camera can segment it against any background, and simple gesture recognition maps motions to home-automation commands. - Try it: Hang a phone around your neck with the front camera facing out and map three hand shapes to smart-light colours. Twist: make one gesture only work while you look at the lamp. #### Perceptive Workbench — Thad Starner (2000) - Video: https://www.youtube.com/watch?v=DnMW_ycz2Kc - Interaction: Projection, Tangible Objects, Shared & Social - Platform & tech: Projection, Wearable, projector, infrared cameras, shadow reconstruction - Idea: A table that sees what you put on it and where you point, linking a desk to a player in the field. - What it is: A projection table that reconstructs objects placed on it in 3D, tracks pointing gestures above it and hosts an AR game against a remote player who wears a mobile computer. - Technique: Infrared illuminators under the surface and cameras cast shadows of objects and arms; silhouettes from several lights are intersected to reconstruct shape and arm direction. - Try it: Build a projector-camera table with a cheap projector and a webcam that detects dark shapes placed on paper and draws outlines around them. Twist: link it to a phone player who sees the same shapes as obstacles. #### Dual Purpose Speech — Thad Starner (2004) - Video: https://www.youtube.com/watch?v=F9z2wLNjxV0 - Interaction: Voice & Sound, Shared & Social, Information & UI - Platform & tech: Wearable, speech recognition, head-up display - Idea: Speech that serves two listeners at once, the person and the computer, keeping interaction socially invisible. - What it is: While scheduling a meeting in conversation, the wearer phrases sentences like 'next Tuesday at three?' so that the other person hears a normal question and the wearable calendar hears a command. - Technique: A push-to-talk key marks the phrases to recognise, and a speech recogniser with a calendar grammar turns them into navigation of the head-up calendar. - Try it: Design a voice UI for AR glasses where every command is also a natural sentence to a friend, and test it in a two-person role play. Twist: add a phrase that cancels without the friend noticing. #### Mobile Music Touch — Thad Starner (2010) - Video: https://www.youtube.com/watch?v=uEdr6iY6F-w - Interaction: Hands & Body, Voice & Sound - Platform & tech: Wearable, vibration motors, glove - Idea: Passive haptic learning: the body learns a skill while your attention is elsewhere. - What it is: A fingerless glove vibrates each finger in the order of a piano melody while the wearer does other work; later they can play the tune on a keyboard without having practised. - Technique: Vibration motors on each finger are driven in time with a melody played through headphones, repeating in the background to train finger sequence memory. - Try it: Make a phone vibrate a four-note rhythm while students read, then test whether they can tap it back on the desk. Twist: map notes to spatial positions around the body instead of fingers. #### Order picking with Google Glass — Thad Starner (2015) - Video: https://www.youtube.com/watch?v=ERXsaR_ekkw - Interaction: Information & UI, Tangible Objects - Platform & tech: Wearable, Google Glass, head-up display - Idea: A glanceable head-up display beats elaborate AR when the task is repetitive and the hands are busy. - What it is: Warehouse workers see which bin to pick from and how many items to take on Google Glass; studies showed it was faster and caused fewer errors than paper lists or pick-by-light. - Technique: A simple graphic of the shelf grid highlights the target bin on the HUD and advances as each pick is confirmed; an opaque display variant performed slightly better than the transparent one. - Try it: Build a glanceable phone overlay that guides a partner to collect five objects from labelled boxes in the right order, and time it against a paper list. Twist: remove all text and use only shapes. ### Acerola (Garrett Gunnell) *Graphics programmer and YouTube educator* Garrett Gunnell makes long-form videos as Acerola in which he rebuilds a rendering effect from a game or a paper (smoke, water, fur, pixel sorting, painterly filters) and publishes the Unity or Godot project on GitHub. #### How Games Render So Much Grass — Acerola (Garrett Gunnell) (2021) - Video: https://www.youtube.com/watch?v=Y0Ko0kvwfgA - Source code: https://github.com/GarrettGunnell/Grass - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, Unity, Compute Shader, GPU instancing - Idea: A meadow is one blade drawn a million times with small differences. - What it is: Fields of millions of grass blades sway in the wind, built step by step from billboards and geometry shaders to GPU-instanced blades culled on the GPU. - Technique: A compute shader scatters blade positions over a terrain, frustum-culls them into an append buffer, and DrawMeshInstancedIndirect renders them with vertex-shader wind from scrolling noise. - Try it: Grow the repo's instanced grass on detected AR floor planes (AR Foundation) so a carpet of grass appears in the living room. Twist: bend the blades away from the phone's position as the user walks through. #### Color Quantization and Dithering — Acerola (Garrett Gunnell) (2022) - Video: https://www.youtube.com/watch?v=8wOUe32Pt-E - Source code: https://github.com/GarrettGunnell/Post-Processing - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL, Post-processing - Idea: Fewer colours plus a clever pattern read as a strong style instead of a loss. - What it is: A 3D scene is reduced to a handful of colours and ordered-dither patterns, giving it the look of an old handheld console or a printed zine. - Technique: A post-processing pass downsamples the frame, adds a Bayer-matrix threshold offset per pixel, quantizes each channel to a few levels and optionally maps the result to a palette before upscaling with point sampling. - Try it: Apply the dithering pass to the phone's AR camera background so the world looks like a Game Boy screen, keeping virtual sprites in the same palette. Twist: switch palettes by pointing the camera at different colour cards. #### Kuwahara Filter — Acerola (Garrett Gunnell) (2022) - Video: https://www.youtube.com/watch?v=LDhN-JK3U9g - Source code: https://github.com/GarrettGunnell/Post-Processing - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL, Post-processing - Idea: Pick the calmest neighbourhood around every pixel and the image paints itself. - What it is: A 3D game scene turns into a moving oil painting as the Kuwahara filter smooths flat areas into brush-like patches while keeping edges sharp. - Technique: A post-processing shader compares the variance of several sectors around each pixel and outputs the mean of the least varied one; the anisotropic version orients the sectors along the local structure tensor. - Try it: Port the anisotropic Kuwahara pass to a URP renderer feature and apply it to the AR camera feed, so the real room becomes a painting while virtual objects stay crisp. Twist: blend from photo to painting as the user walks closer. #### Counter-Strike 2 Smoke Grenades — Acerola (Garrett Gunnell) (2023) - Video: https://www.youtube.com/watch?v=ryB8hT5TMSg - Source code: https://github.com/GarrettGunnell/CS2-Smoke-Grenades - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, Unity, Compute Shader, HLSL - Idea: Smoke that knows the room: it floods the free space and can be pushed away. - What it is: A smoke grenade fills a room with volumetric smoke that flows around walls, and gunshots and explosions punch holes that slowly close again. - Technique: The level is voxelised, a compute shader flood-fills smoke through free voxels from the grenade, and the smoke is ray-marched with noise and lighting while bullet capsules carve temporary holes in the density. - Try it: Voxelise a real room from the AR Foundation LiDAR mesh on an iPhone Pro and flood-fill the repo's smoke from a point the user taps, so it pours around real furniture. Twist: let a hand swipe (tracked by the camera) blow a hole through the cloud. #### Pixel Sorting — Acerola (Garrett Gunnell) (2023) - Video: https://www.youtube.com/watch?v=HMmmBDRy-jE - Source code: https://github.com/GarrettGunnell/Pixel-Sorting - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader, HLSL - Idea: A classic still-image glitch, made fast enough to run on live video. - What it is: Bright spans of each frame melt into long streaks as pixels are sorted by brightness in real time, turning a 3D scene into moving glitch art. - Technique: A compute shader masks pixels by a luminance threshold, finds contiguous spans per row or column, and sorts each span on the GPU before compositing it back over the frame. - Try it: Run the pixel-sort pass on the passthrough camera image of a phone AR app (AR Foundation camera background + a URP renderer feature) so the real street streaks when the user taps. Twist: sort only pixels inside the people-segmentation mask. #### Shell Texturing Fur — Acerola (Garrett Gunnell) (2023) - Video: https://www.youtube.com/watch?v=9dr-tRQzij4 - Source code: https://github.com/GarrettGunnell/Shell-Texturing - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, Unity, HLSL - Idea: Stack enough thin slices and a flat surface becomes fuzzy volume. - What it is: Grass-like fur grows out of simple meshes and sways, made of many stacked transparent shells instead of individual hair geometry. - Technique: The mesh is drawn N times, each shell pushed out along the normal; a hashed noise texture decides per-strand height, and shells discard pixels above that height, with displacement for gravity and wind. - Try it: Apply the shell-fur shader to a real object scanned with the phone (Object Capture or LiDAR mesh) and place it back in AR, so a real mug becomes furry. Twist: make the fur flatten where the user touches the screen. #### Simulating the Entire Ocean — Acerola (Garrett Gunnell) (2023) - Video: https://www.youtube.com/watch?v=yPfagLeUa7k - Source code: https://github.com/GarrettGunnell/Water - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Desktop, Unity, Compute Shader, FFT - Idea: Real oceans are sums of thousands of waves, and the GPU can add them up every frame. - What it is: An endless ocean surface rolls under a moving camera, with foam, reflections and subsurface colour, built up from simple sine waves to an FFT spectrum. - Technique: A JONSWAP wave spectrum is evolved in frequency space and inverse-FFT'd in compute shaders into displacement and normal maps, which a physically based water shader renders with foam from the Jacobian. - Try it: Place a small patch of the repo's FFT ocean inside a floor portal in AR (AR Foundation plane + stencil mask) so the room floor opens onto a sea. Twist: drive wind speed from the phone microphone level. #### Rendering Millions of Particles — Acerola (Garrett Gunnell) (2024) - Video: https://www.youtube.com/watch?v=1L-x_DH3Uvg - Source code: https://github.com/GarrettGunnell/Iterated-Function-Systems - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader, HLSL - Idea: Apply a few random affine maps over and over and a fractal appears out of points. - What it is: Tens of millions of particles trace glowing 3D fractal attractors that morph smoothly from one shape to another, lit with a brute-force ambient occlusion pass. - Technique: A compute shader runs the chaos game of an iterated function system for every particle each frame, interpolating the affine transforms between two systems, and draws the points with procedural instancing plus an occlusion estimate. - Try it: Anchor a fractal cloud from the repo on a tracked image in AR and let the phone's distance to the marker morph between two attractors. Twist: seed the affine maps from the user's hand-pose landmarks. ### Alex Olwal *Research scientist and tech lead at Google (AR, wearables, interaction)* Interaction researcher who built early spatial and mobile AR systems at KTH and Columbia, then worked at the MIT Media Lab and Google on wearables, captioning glasses and interfaces hidden in everyday materials. #### ASTOR: spatial AR on an industrial lathe — Alex Olwal (2008) - Video: https://www.youtube.com/watch?v=Qfv8mpfLZrI - Interaction: Information & UI, Projection, Tangible Objects - Platform & tech: Projection, holographic optical element, autostereoscopic display - Idea: Turn the machine's safety window into an autostereoscopic AR display. - What it is: A transparent holographic window mounted in front of a CNC lathe shows 3D overlays, such as forces on the cutting tool, directly on the machining process, without glasses. - Technique: A holographic optical element redirects projected images to each eye from a transparent screen, creating an optical see-through, glasses-free 3D overlay registered to the machine. - Try it: Mount a sheet of acrylic in front of a desk object at 45° and reflect a phone screen onto it to overlay live labels (Pepper's ghost). Twist: make a label follow a moving part. #### LUMAR: hybrid 2D/3D mobile AR — Alex Olwal (2008) - Video: https://www.youtube.com/watch?v=GJU-kRLq4sQ - Interaction: Information & UI, Tangible Objects - Platform & tech: Phone, marker tracking, mobile AR - Idea: Let paper and phone work together: the static print gives context, the phone adds depth. - What it is: Pointing a phone at a printed apartment floor plan pops up a 3D model of the apartment on the phone, while the paper remains the overview: three layers of information from one printout. - Technique: 2D tracking of the phone over the display surface is combined with 3D marker-based pose estimation, switching the phone view between an overlay and a 3D viewport. - Try it: Print a campus map and use image tracking so each building pops up in 3D when the phone hovers over it. Twist: tilting the phone flattens the model back into the map. #### LightSense: mixed-reality phone over a display — Alex Olwal (2008) - Video: https://www.youtube.com/watch?v=5AovJvB9sJs - Interaction: Information & UI, Tangible Objects, Location & City - Platform & tech: Phone, Desktop, light sensing, camera tracking - Idea: The phone's own light becomes its tracking system, so the map needs no markers. - What it is: Holding a phone over a backlit Stockholm subway map shows context-sensitive information for the spot under it, turning the map into a lookup surface. - Technique: The phone's LED is tracked by a camera behind the translucent map, giving its position so the phone can show the matching content. - Try it: Put a webcam under a glass table with a paper map on it and track a phone flashlight from below to trigger content on the phone. Twist: use two phones and show the route between them. #### SurfaceFusion — Alex Olwal (2008) - Video: https://www.youtube.com/watch?v=DSZemU_4br4 - Interaction: Tangible Objects, Projection, Information & UI - Platform & tech: Desktop, Projection, Microsoft Surface, RFID, computer vision - Idea: Combine two weak sensors to make ordinary objects first-class citizens of a touch table. - What it is: Physical objects placed on a Microsoft Surface table are identified by RFID and located by computer vision, so virtual content follows and responds to real things on the tabletop. - Technique: Vision on the table's IR camera gives object position and RFID gives identity; fusing them removes the need for printed tags on every object. - Try it: Use a webcam over a table to track coloured objects and an NFC phone tap to name each one, then project labels that follow them. Twist: make objects react when they are placed near each other. #### Wearable Subtitles — Alex Olwal, Thad Starner (2020) - Video: https://www.youtube.com/watch?v=XUVBjiXlfNs - Interaction: Voice & Sound, Information & UI, Shared & Social - Platform & tech: Wearable, head-worn display, speech recognition - Idea: Captions belong in the line of sight, not in your hand. - What it is: Lightweight eyewear shows live captions of the conversation in front of the wearer, letting deaf and hard-of-hearing people keep their eyes on the speaker instead of a phone. - Technique: A low-power head-worn display is paired with a phone running speech recognition, with a hybrid design that keeps glasses light and battery life near all-day. - Try it: Make a phone-in-headset (cardboard or HoloKit) captioning view that shows live speech-to-text above the speaker's head. Twist: shrink text when the speaker is far away. #### SpeechCompass — Alex Olwal (2025) - Video: https://www.youtube.com/watch?v=crWXO5T5jaQ - Interaction: Voice & Sound, Information & UI, Shared & Social - Platform & tech: Phone, microphone array, sound localisation, speech recognition - Idea: Captions that know where each voice comes from. - What it is: A phone case with several microphones localises who is speaking in a group and shows each caption with an arrow or colour pointing toward that speaker. - Technique: Real-time multi-microphone sound localisation on a small microcontroller estimates direction of arrival, which is attached to each speech-to-text segment. - Try it: Use two phones' microphones on a table to guess which side a voice comes from and colour the captions left or right. Twist: show the captions as speech bubbles in AR above each person. ### Andy Goldsworthy *Land artist and sculptor* British artist who works with leaves, ice, stone, wood and the tide, making both ephemeral pieces that melt or wash away and permanent walls and cairns. #### Storm King Wall — Andy Goldsworthy (1998) - Video: https://www.youtube.com/watch?v=tsezrORBFj0 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, dry-stone wall, trees, pond - Idea: A line that dodges trees and dives under water shows how an anchored path can react to the site rather than ignore it. - What it is: A dry-stone wall snakes between trees in a forest at Storm King Art Center, then runs down a hill, disappears into a pond and re-emerges on the far side. - Technique: Wallers built the 690-metre wall by hand from stones of old field walls on the site, curving it around each tree. - Try it: Generate an AR line with Unity AR Foundation plane and mesh detection that winds around detected obstacles (people, chairs, trees) instead of crossing them. Twist: let the line dive under a detected floor plane and re-emerge. #### Rivers and Tides — Andy Goldsworthy (2001) - Video: https://www.youtube.com/watch?v=AT3lveJmjY8 - Interaction: Location & City, Tangible Objects, Perception & Effects - Platform & tech: Desktop, ice, leaves, stone, tide - Idea: The artwork includes its own disappearance: designing for how a piece decays or is taken by nature is as important as how it appears. - What it is: Thomas Riedelsheimer's film follows Goldsworthy building icicle spirals, leaf chains that float down a river and a stone cone that the incoming tide swallows. - Technique: He uses only found materials and their own properties (freezing, floating, balancing), working with the time of day and the tide. - Try it: Build an AR sculpture from scanned leaves or stones on a real riverbank or puddle and program it to fall apart when a timer or the user's approach triggers it. Twist: let the pieces drift along the real water direction the user traces. #### Refuges d'Art — Andy Goldsworthy (2002) - Video: https://www.youtube.com/watch?v=Bgcwm27Ttfs - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, restored buildings, stone, hiking route - Idea: Art that you reach only by walking days through a landscape makes the journey part of the work, a model for long-distance AR trails. - What it is: In the mountains of Haute-Provence, Goldsworthy restored ruined farms, chapels and sheepfolds along a 150-kilometre hiking route and placed works inside; some can be slept in. - Technique: Old stone buildings are rebuilt as shelters and each contains a stone or clay work made for that site, linked by a waymarked trail. - Try it: Design a multi-day or multi-hour AR hiking route with geolocated 'refuges' where a virtual work appears only if the user arrived on foot (checked by step count). Twist: each refuge shows traces left by earlier hikers. #### Spire — Andy Goldsworthy (2008) - Video: https://www.youtube.com/watch?v=5xtjTu7TWuY - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, cypress trunks, forest regrowth - Idea: The work is designed to be absorbed by the forest over decades, a model for AR installations with a planned lifespan. - What it is: Thirty-seven cypress trunks from the Presidio's old forest are bound into a 30-metre spire, which the newly planted trees around it will slowly grow up to hide. - Technique: Felled trunks are stacked and fixed into a tall cone at the centre of a replanting area so that surrounding growth measures time. - Try it: Anchor a tall AR spire in a park and add simulated trees that grow a little every day the app is opened, until they hide it. Twist: let every visitor plant one virtual tree that joins the forest for everyone. #### Wood Line — Andy Goldsworthy (2011) - Video: https://www.youtube.com/watch?v=97ap3DwHK1o - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, eucalyptus branches, forest floor - Idea: A single line on the ground turns a forest into a path and a drawing at once, the simplest form of guidance AR can give. - What it is: A 350-metre line of eucalyptus logs, laid end to end, snakes along the floor of a Presidio forest between the trunks, inviting people to walk on and beside it. - Technique: Branches from trees felled for forest management are laid in a continuous curving line, partly sunk into the ground. - Try it: Draw a continuous AR line on the ground with Lens Studio or WebXR hit-testing as you walk through trees, save it, and let the next visitor follow it. Twist: the line slowly sinks into the ground over a week. #### Tree Fall — Andy Goldsworthy (2013) - Video: https://www.youtube.com/watch?v=qfKnT-8uq0I - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, eucalyptus trunk, clay-covered interior, former powder magazine - Idea: Bringing a tree indoors and coating the room around it creates a portal-like space, the same effect AR world replacement aims for. - What it is: Inside a former gunpowder store in the Presidio, a eucalyptus trunk is suspended from the vaulted ceiling, and both ceiling and trunk are covered in cracked clay. - Technique: The trunk is hung from the ceiling and the surfaces are plastered with local clay that dries and cracks into a pattern. - Try it: Use LiDAR scene reconstruction (ARKit) to cover a real room's ceiling and walls with a cracked-clay material and hang a virtual tree from the ceiling. Twist: the cracks spread wherever the user looks longest. #### Earth Wall — Andy Goldsworthy (2014) - Video: https://www.youtube.com/watch?v=I051qmxvDlE - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, rammed earth, eucalyptus branches - Idea: An object half-inside a wall reads as something passing through a boundary, a classic AR occlusion effect in physical form. - What it is: A wall of rammed earth built from the Presidio's soil encloses the branches of a eucalyptus tree, which reach out through the wall's surface. - Technique: Earth from the site is compacted in layers inside formwork around the branches so they appear to grow out of the finished wall. - Try it: Use AR occlusion (ARKit people and scene occlusion) to make a virtual branch or creature appear to push out of a real wall. Twist: let it pull back in when touched. #### Leaning into the Wind — Andy Goldsworthy (2017) - Video: https://www.youtube.com/watch?v=BQYGbfVfpm0 - Interaction: Hands & Body, Location & City, Perception & Effects - Platform & tech: Desktop, body, hedges, rain shadows - Idea: A body-shaped dry patch on wet ground is a trace of presence that disappears in minutes, a direct idea for AR silhouettes left by users. - What it is: The second film by Thomas Riedelsheimer shows Goldsworthy using his own body: walking through a hedge, lying on the ground as rain starts so a dry 'rain shadow' remains, and throwing leaves. - Technique: The rain shadows are made by lying still on dry ground at the start of a shower and standing up once the ground around is wet. - Try it: Use body segmentation (Lens Studio or MediaPipe) to capture a user's silhouette and leave it as a fading shadow decal on the ground in AR. Twist: shadows fade faster when many people walk over the same spot. ### Blair MacIntyre — Georgia Tech Augmented Environments Lab *Professor at Georgia Tech (Augmented Environments Lab); later principal research scientist at Mozilla (WebXR) and JP Morgan* Blair MacIntyre (Feiner's former student) ran Georgia Tech's Augmented Environments Lab, which produced AR games and narrative experiments like AR Façade, ARhrrrr! and NerdHerder, and the Argon AR web browser. At Mozilla he helped create the WebXR Viewer and WebXR AR standards. #### The Voices of Oakland — Blair MacIntyre — Georgia Tech Augmented Environments Lab (2005) - Video: https://www.youtube.com/watch?v=mHoe2NfTV7U - Interaction: Voice & Sound, Location & City - Platform & tech: Wearable, GPS, audio AR - Idea: Walk into a cemetery and hear the people buried there tell their own stories at their graves. - What it is: An audio-AR walking tour of Atlanta's historic Oakland Cemetery where dramatized voices of the people buried there tell their stories at their graves. - Technique: GPS position triggers location-based audio segments and the narrative is structured as an interactive tour where the user chooses which story branches to hear at each grave. - Try it: Use a phone web page and the Geolocation API to set up four geofences on campus, so that arriving at a spot automatically plays a classmate's recording of 'a story that happened here'. Twist: the content branches depending on the direction you came from or how long you stay. #### AR Façade — Blair MacIntyre — Georgia Tech Augmented Environments Lab (2007) - Video: https://www.youtube.com/watch?v=WmXZW-Eg0Lg - Interaction: Voice & Sound, Portals & Worlds, Performance - Platform & tech: Headset, video see-through HMD, Wizard-of-Oz speech - Idea: Move the AI interactive drama Façade into a real apartment, where the player talks straight to life-size characters. - What it is: The AI interactive drama Façade re-staged in a physical apartment with AR: the player walks among life-size virtual characters Grace and Trip and speaks to them aloud. - Technique: A video see-through HMD with tracking renders life-size characters in a physical apartment set, while a hidden operator transcribes the player's speech (Wizard of Oz) into the Façade drama manager that drives the characters. - Try it: Place a life-size virtual character in phone AR (a Mixamo model works), and have a classmate hidden backstage (Wizard of Oz) pick the character's animations and lines based on what the player says. Twist: let the character 'see' a real object the player picks up and comment on it. #### ARf — Blair MacIntyre — Georgia Tech Augmented Environments Lab (2008) - Video: https://www.youtube.com/watch?v=_0bitKDKdg0 - Interaction: Play, Tangible Objects - Platform & tech: Phone, iPhone, marker tracking - Idea: Raise an AR virtual dog on the very first iPhone. - What it is: An AR virtual pet dog on the original iPhone that lives on a marker and can be played with by touch. - Technique: A fiducial marker is tracked by the iPhone camera to anchor an animated 3D dog, and touch input on the screen is raycast onto the dog to trigger petting and play behaviors. - Try it: Use AR Foundation or 8th Wall to put a virtual pet on a table, with two interactions (tap to pet, drag to feed) and three emotional states. Twist: have the pet react to your real surroundings (sleeping when it is dark, looking up when it hears a sound). #### BragFish — Blair MacIntyre — Georgia Tech Augmented Environments Lab (2008) - Video: https://www.youtube.com/watch?v=w3iBrj_zfTM - Interaction: Play, Shared & Social, Tangible Objects - Platform & tech: Phone, marker tracking, UMPC - Idea: A multiplayer handheld AR game where players compete at fishing around a physical 'pond'. - What it is: A competitive handheld AR fishing game played around a shared physical pond board, exploring social play with co-located phones. - Technique: Each handheld tracks a shared printed pond board with markers, rendering fish and lures registered to it, and a networked game state makes players compete co-located around the same physical board. - Try it: Use a printed pond picture as a shared image target and make a two- or three-player phone AR fishing game where each person sees only their own rod plus the shared fish. Twist: add a 'steal a fish' move that only works when you are physically close to an opponent. #### ARhrrrr! — Blair MacIntyre — Georgia Tech Augmented Environments Lab (2009) - Video: https://www.youtube.com/watch?v=cNu4CluFOcw - Interaction: Play, Tangible Objects - Platform & tech: Phone, NVIDIA Tegra, natural feature tracking - Idea: Shoot zombies on a tabletop city map, where each real Skittle you drop is a bomb. - What it is: A zombie shooter on a tabletop city map: from a helicopter view on a mobile device you shoot zombies and can drop real Skittles candies onto the map as explosives. - Technique: Natural feature tracking of a printed city map provides the camera pose on a GPU-accelerated handheld, and colored blobs (Skittles) placed on the map are detected by color segmentation and turned into explosive game objects. - Try it: Use a printed map as an image target to make an AR shooting or tower-defense game, and use color detection to recognize real colored candies or blocks that players put on the map as bombs or obstacles. Twist: let the color of a real object decide its role in the game. #### Art of Defense — Blair MacIntyre — Georgia Tech Augmented Environments Lab (2009) - Video: https://www.youtube.com/watch?v=hSUuvgklsZw - Interaction: Play, Shared & Social, Tangible Objects - Platform & tech: Phone, marker tracking - Idea: Build the map from physical tiles and play AR tower defense together. - What it is: A collaborative AR tower defense game played on a physical board with tangible tiles, shown at SIGGRAPH 2009. - Technique: Tangible tiles carry fiducial markers that are tracked by handheld cameras, and each tile's position and rotation define a tower's placement and aim in a shared tower-defense game rendered over the board. - Try it: Use several cards with ArUco markers as 'towers': phone AR recognizes each card's position and shows a tower on it while enemies attack along a virtual path. Twist: each card's rotation sets its tower's facing, and two players must split up the placement. #### Argon AR Browser — Blair MacIntyre — Georgia Tech Augmented Environments Lab (2010) - Video: https://www.youtube.com/watch?v=F_M8C2jW8PI - Interaction: Information & UI, Location & City - Platform & tech: Phone, Web, KHARMA, JavaScript, Vuforia - Idea: Write AR with web technology so the real world can be published and browsed like a web page. - What it is: Argon, the first AR web browser, renders standards-based web content (KHARMA/KML, later HTML/JavaScript) anchored to places and images, so anyone can author AR like a web page. - Technique: Argon exposed AR as a web platform: pages written in KML/KHARMA and later HTML/JavaScript declared content anchored to geolocation or image targets, and the browser supplied the tracked camera pose to render it. - Try it: Build an AR page made entirely with web technology using WebXR (three.js) or 8th Wall, anchor an HTML card to a plane or image, and share the link so classmates can open it directly. Twist: let classmates add content to your AR page just by editing a JSON file. #### NerdHerder — Blair MacIntyre — Georgia Tech Augmented Environments Lab (2012) - Video: https://www.youtube.com/watch?v=RSxImyFXSXw - Interaction: Play, Tangible Objects - Platform & tech: Phone, Qualcomm Vuforia, Unity - Idea: Lure panicking nerds back to their desks with a donut on your tabletop. - What it is: A commercial tabletop AR game built with Qualcomm: herd panicking office nerds back to their cubicles by dangling donuts on a printed game board. - Technique: Vuforia image tracking of the printed board anchors a Unity game scene, and the phone's pose relative to the board is itself the controller, since the dangled donut follows where the camera points. - Try it: Make a small game in Unity + AR Foundation that uses the phone's position as the controller: where the phone hovers above an image target decides where a virtual lure lands, guiding virtual little people to a goal area. Twist: swap the lure for sound or light, and compare which kind of guidance is more fun. ### Bruce Thomas — UniSA Wearable Computer Lab *Professor; founder of the Wearable Computer Lab at the University of South Australia* Founded the Wearable Computer Lab at the University of South Australia, where Tinmith and ARQuake (with Wayne Piekarski) took AR outdoors around 2000. The lab later became a centre for spatial augmented reality: projecting interfaces onto foam, clay and mock-up control panels for design and prototyping. #### Digital Foam — Bruce Thomas — UniSA Wearable Computer Lab (2008) - Video: https://www.youtube.com/watch?v=HgkCyjGAA6g - Interaction: Tangible Objects, Drawing & Making - Platform & tech: Desktop, Projection, conductive foam, pressure sensing - Idea: Sculpt digital shape by squeezing a soft physical one. - What it is: A ball of conductive foam senses where and how hard it is squeezed, used as an input device for sculpting 3D shapes by pressing directly on a physical form. - Technique: Conductive foam sensors change resistance under compression, and an array of them across the surface reports a pressure map that deforms the matching virtual mesh. - Try it: Build a squeeze controller with a phone's pressure-sensitive touch or a DIY velostat sensor, and dent a 3D sphere in WebXR where you press. Twist: make the dents spring back slowly like memory foam. #### Digital Airbrushing with Spatial Augmented Reality — Bruce Thomas — UniSA Wearable Computer Lab (2009) - Video: https://www.youtube.com/watch?v=0bilGEwQgPo - Interaction: Projection, Drawing & Making, Tangible Objects - Platform & tech: Projection, spatial augmented reality, projector, tracking - Idea: Paint real objects with light using the familiar gestures of airbrushing. - What it is: Holding a tracked airbrush and stencil, a designer sprays virtual paint onto a white physical object and the paint appears on it through projectors; the stencil's mask shape is virtual and can be redrawn. - Technique: Projectors are calibrated to a known object model, and tracked tool poses drive a projective-texture paint buffer that is projected back onto the object. - Try it: Point a projector at a white box, track a phone as a 'spray can' and let it paint projected colour where it points. Twist: use a cardboard cut-out as a stencil that blocks paint. #### Improving Spatial Perception for AR X-Ray Vision — Bruce Thomas — UniSA Wearable Computer Lab, Christian Sandor (2009) - Video: https://www.youtube.com/watch?v=BTPBNggldTw - Interaction: Portals & Worlds, Perception & Effects, Location & City - Platform & tech: Wearable, Tinmith, edge detection, head-mounted display - Idea: X-ray vision only works if you keep just enough of the occluder to show depth. - What it is: Walking outdoors with a head-worn display, users see through a building wall into the room behind it, with edges of the wall kept visible so the hidden space reads as behind rather than in front. - Technique: Edge detection on the video of the occluding wall is overlaid on the rendered hidden scene, providing an occlusion cue that restores correct depth ordering. - Try it: In phone AR, place a virtual room behind a real wall and compare three looks: plain overlay, edge-outline wall, and a cut hole. Twist: let the hole follow your gaze. #### Augmented Foam Sculpting — Bruce Thomas — UniSA Wearable Computer Lab (2010) - Video: https://www.youtube.com/watch?v=VPUT2l5JvT0 - Interaction: Tangible Objects, Projection, Drawing & Making - Platform & tech: Projection, spatial augmented reality, hot-wire cutter, tracking - Idea: Capture a digital model as a side effect of making a physical one. - What it is: A designer cuts a block of foam with a tracked hot-wire cutter while projected guides show where to cut, and the system records each cut to build a matching 3D model. - Technique: The foam and cutter are tracked, each cut plane is applied as a boolean subtraction to a virtual block, and projection shows cut lines and the evolving model on the foam. - Try it: Carve a potato or clay block while a phone on a tripod scans after each cut and shows the growing digital copy next to it in AR. Twist: project the next suggested cut onto the block. #### Spatial AR for Interactive Rapid Prototyping — Bruce Thomas — UniSA Wearable Computer Lab (2010) - Video: https://www.youtube.com/watch?v=6SAjX8-iBOc - Interaction: Projection, Tangible Objects, Information & UI - Platform & tech: Projection, spatial augmented reality, projector - Idea: Prototype a product's interface by projecting it onto a blank mock-up instead of manufacturing it. - What it is: Blank physical control panels get their buttons and dials projected onto them; users operate the projected controls just like real ones, and a study compared them with physical panels. - Technique: Calibrated projectors render controls onto the mock-up's known geometry, and finger or tool tracking detects presses on the projected widgets. - Try it: Project a microwave control panel onto a cardboard box, detect finger presses with a webcam and test two layouts with classmates. Twist: let the panel rearrange itself to fit the user's hand. #### Quimo — Bruce Thomas — UniSA Wearable Computer Lab (2011) - Video: https://www.youtube.com/watch?v=LqsMzRuDZOs - Interaction: Projection, Tangible Objects, Drawing & Making - Platform & tech: Projection, spatial augmented reality, malleable material - Idea: Quick mock-ups: sculpt by hand, then dress the rough shape with projected detail. - What it is: A white malleable material that designers shape with bare hands into a rough mock-up; once shaped it holds its form and becomes a projection surface for colours and details. - Technique: Quimo is a white, shape-retaining foam material with embedded tracking markers so projectors can register imagery onto its freeform surface. - Try it: Mould white clay into a toy shape, scan it with a LiDAR phone and project or AR-paint a texture onto it. Twist: re-mould it and see what breaks. #### Improving Procedural Task Performance with AR Annotations — Bruce Thomas — UniSA Wearable Computer Lab (2013) - Video: https://www.youtube.com/watch?v=GqfTy8-fEEY - Interaction: Projection, Information & UI, Tangible Objects - Platform & tech: Projection, spatial augmented reality, projector - Idea: Put the instruction exactly where the hand needs to go. - What it is: Participants press sequences of buttons on control panels while instructions are either shown on a monitor or projected directly onto the buttons to press next. - Technique: Projector-camera calibration places highlights and arrows on the physical panel, and the next step appears as each press is registered. - Try it: Project step-by-step highlights onto a board game or a set of light switches and time a partner doing a ten-step sequence against a printed manual. Twist: remove the highlight halfway and see who remembers. #### Embodied Axes — Bruce Thomas — UniSA Wearable Computer Lab (2020) - Video: https://www.youtube.com/watch?v=p19ub_pGN5U - Interaction: Tangible Objects, Information & UI, Hands & Body - Platform & tech: Headset, HoloLens, actuated sliders - Idea: Turn the axes of a 3D chart into physical objects you can grab. - What it is: Three physical, motorised sliders arranged as x, y and z axes let users select ranges in a 3D data visualisation that floats above them in an AR headset. - Technique: The tangible controller is registered to a HoloLens scene, and actuated sliders both read the user's input and move to reflect selections made elsewhere. - Try it: Mark three rulers on a table as axes, track them with image markers and show a 3D scatter plot above them in phone AR; use a clothespin on each ruler to filter. Twist: let two people filter at once. ### Chris Harrison — CMU Future Interfaces Group *Professor at Carnegie Mellon HCII; director of the Future Interfaces Group (FIGLAB)* Chris Harrison invents new ways to turn skin, walls, tables and everyday surfaces into interactive displays and sensors. Skinput and OmniTouch (built with Microsoft Research) projected interfaces onto the body, and FIGLAB keeps producing on-body and on-world AR input techniques. #### Skinput — Chris Harrison — CMU Future Interfaces Group (2010) - Video: https://www.youtube.com/watch?v=g3XPUdW9Ryg - Interaction: Hands & Body, Projection - Platform & tech: Wearable, Projection, bio-acoustic sensing, pico projector - Idea: Use your arm as a touchscreen: tap different spots on your skin to work a menu projected onto your body. - What it is: An armband of vibration sensors detects finger taps on the forearm and hand; a pico projector shows menus and a game directly on your skin. - Technique: Piezoelectric vibration sensors on an armband capture the bio-acoustic signals of taps propagating through bone and soft tissue, and an SVM classifies the tap location from spectral features. - Try it: Strap a phone to your forearm, record accelerometer vibrations from taps at different spots on the arm, train a three-location classifier with Teachable Machine (sound or custom features), and project or display a menu on the arm. Twist: move the taps onto objects outside the body, such as a tabletop or wall. #### OmniTouch — Chris Harrison — CMU Future Interfaces Group, Microsoft Research — Hrvoje Benko & Andy Wilson (2011) - Video: https://www.youtube.com/watch?v=Pz17lbjOFn8 - Interaction: Hands & Body, Projection, Spatial Mapping - Platform & tech: Wearable, Projection, depth camera, pico projector - Idea: A shoulder-mounted depth camera and projector turn your palm, a wall or a notebook into a multitouch screen on the spot. - What it is: A shoulder-worn depth camera and projector turn hands, walls, notebooks and tables into multitouch surfaces on the fly. - Technique: A shoulder-mounted depth camera finds finger-like cylinders in the depth image and determines touch by checking whether the fingertip is within a few millimeters of the surface below it, while a calibrated pico projector renders onto the detected surfaces. - Try it: Use a depth camera (RealSense, Kinect or a LiDAR iPhone) to detect touch from the depth difference between finger and table, and place a projected keyboard on a notebook or your palm. Twist: make the interface automatically appear on the largest available surface. #### LightAnchors — Chris Harrison — CMU Future Interfaces Group (2019) - Video: https://www.youtube.com/watch?v=435OSV8Hudw - Interaction: Information & UI, Tangible Objects - Platform & tech: Phone, computer vision, LED signaling - Idea: Let a device's tiny indicator light blink out data, and a live label pops up when a phone looks at it. - What it is: Blinking point lights on everyday devices broadcast data; a phone camera decodes them and anchors live AR labels on thermostats, routers and cars. - Technique: LEDs blink in a known preamble pattern at frequencies detectable by a phone camera, and the app finds bright points across frames, decodes the temporal pattern into data, and anchors labels at the light's image position. - Try it: Have an Arduino or another phone's screen blink an encoded signal (such as an 8-bit ID), detect and decode the bright spot frame by frame with the phone camera, and show the matching AR label at its position. Twist: turn the blinking data into a 'secret message' only those who scan it can see. #### EgoTouch — Chris Harrison — CMU Future Interfaces Group (2024) - Video: https://www.youtube.com/watch?v=7JwgIm1f1ss - Interaction: Hands & Body - Platform & tech: Headset, headset cameras, machine learning - Idea: Using only a headset's built-in cameras, the skin on your arm becomes a set of touch buttons. - What it is: Uses the cameras already on AR/VR headsets to detect touches on your own skin, turning palm and forearm into buttons and sliders without extra hardware. - Technique: A CNN trained on headset camera images infers from the skin's appearance and shading whether a finger is touching the palm or forearm and where, without additional sensors. - Try it: Photograph your palm with a phone camera, train two image classes ('touch' and 'hover') in Teachable Machine, and combine them with MediaPipe fingertip positions to divide the palm into four buttons. Twist: use the buttons on your palm to control a small game in HoloKit or on a web page. ### Clémence Debaig (Unwired Dance Theatre) *Dance artist, maker and creative technologist; founder of Unwired Dance Theatre* French-born, London-based dancer who worked in software before returning to dance; she builds her own wearables, haptics and motion-capture systems to let audiences influence live dancers, locally or across the internet. Her company Unwired Dance Theatre stages telematic and WebXR performances, and she was a participant at the 2022 Choreographic Coding Lab at A+E Lab. #### Automata — Clémence Debaig (Unwired Dance Theatre) (2019) - Video: https://www.youtube.com/watch?v=iND6WvW4nNk - Interaction: Hands & Body, Tangible Objects, Performance - Platform & tech: Wearable, accelerometers, servo motors, Arduino - Idea: A costume that dances back, blurring puppeteer and puppet. - What it is: A wearable kinetic costume whose parts move when the dancer moves, raising the question of who is controlling whom: the dancer the objects, or the objects the dancer. - Technique: Accelerometers on the dancer's arms are read by a microcontroller that drives servo motors moving elements of the outfit. - Try it: Attach a micro:bit to a wrist and make its tilt move a servo that flaps a paper wing on the shoulder. Twist: invert the mapping so the wing moves most when the arm is still. #### BOTHER — Clémence Debaig (Unwired Dance Theatre) (2019) - Video: https://www.youtube.com/watch?v=vUQOzzQ35OY - Interaction: Hands & Body, Shared & Social, Performance - Platform & tech: Wearable, Web, wearables, web interface, durational performance - Idea: Give strangers a remote control over a dancer and watch how they choose to use it. - What it is: In a durational dance installation, audience members use a web interface to trigger actions on devices worn by the dancer, influencing her well-being positively or negatively and watching the effect in front of them. - Technique: A web page sends audience commands over the network to microcontroller-driven wearables (likely vibration, light or moving elements) on the dancer's body. - Try it: Build a web page with two buttons, 'care' and 'bother', that trigger a vibration motor or LED on a classmate's wrist via an ESP32 while they improvise. Twist: show the audience how many people pressed each button. #### Stuck inside — Clémence Debaig (Unwired Dance Theatre) (2019) - Video: https://www.youtube.com/watch?v=B14B_qgilMU - Interaction: Projection, Perception & Effects - Platform & tech: Projection, projection mapping, ofxPiMapper, generative animation - Idea: Make a flat wall of folded paper feel like it has an inside. - What it is: A projection-mapped wall of paper pyramids shows day passing and shadows moving, until a tiny figure appears inside the pyramids, knocking and looking for a way out. - Technique: Generative animation is mapped onto each face of the folded paper pyramids (likely with ofxPiMapper on a Raspberry Pi), with fake light and shadow reinforcing the depth illusion. - Try it: Fold six paper pyramids, stick them on a board and projection-map a moving light source across their faces with MadMapper or a free mapper. Twist: add a character that seems trapped behind one face. #### STRINGS — Clémence Debaig (Unwired Dance Theatre) (2020) - Video: https://www.youtube.com/watch?v=Qv88tt4JTG4 - Interaction: Shared & Social, Hands & Body, Performance - Platform & tech: Wearable, Web, networked haptics, live stream, web interface - Idea: Dance together at a distance by letting an online crowd pull the dancer's strings. - What it is: During lockdown a dancer performs live from her living room wearing electronic devices that react to the collective actions of an online audience, so remote viewers can literally hold and move her. - Technique: Audience input from a web interface is aggregated on a server and sent to networked haptic wearables on the dancer, who improvises in response on the live stream. - Try it: Stream a performer on a video call while viewers vote with emoji on a shared web page; the majority vote vibrates a phone taped to the performer's arm or leg. Twist: require at least three people to agree before anything happens. #### Social Distancing Shirt — Clémence Debaig (Unwired Dance Theatre) (2020) - Video: https://www.youtube.com/watch?v=zKxfqXSyp3M - Interaction: Spatial Mapping, Hands & Body, Voice & Sound - Platform & tech: Wearable, ultrasonic sensors, speakers, Arduino - Idea: Clothing that defends your personal space out loud. - What it is: A shirt with sensors on four sides detects when someone comes closer than two metres and plays a spoken warning, 'You are too close', until they step back. - Technique: Four ultrasonic distance sensors (front, back and sides) are read by a microcontroller that plays prerecorded tracks through two speakers when the 2 m threshold is crossed. - Try it: Tape an ultrasonic sensor and a buzzer to a jacket with an Arduino and set a personal-space radius that beeps faster as people approach. Twist: make it greet friends differently from strangers using a phone NFC tag. #### DISCORDANCE — Clémence Debaig (Unwired Dance Theatre) (2022) - Video: https://www.youtube.com/watch?v=p_1sgQmuf-4 - Interaction: Shared & Social, Performance, Hands & Body - Platform & tech: Headset, Projection, motion capture, VR, telematics - Idea: Two cities, one virtual stage, three kinds of audience. - What it is: Dancers in London and New York, both in full motion capture, meet in the same virtual space, co-present with audience members in VR headsets and with spectators watching in a theatre. - Technique: Motion-capture data from both cities is streamed into a shared real-time VR scene, which is also projected for the theatre audience. - Try it: Pair two groups in different rooms, each streaming pose data into one shared web scene, and choreograph a duet where avatars must 'touch' hands. Twist: add a one-second delay and make the delay part of the choreography. #### Unwanted Waters — Clémence Debaig (Unwired Dance Theatre) (2022) - Video: https://www.youtube.com/watch?v=KfIHipMlrCU - Interaction: Hands & Body, Performance, Shared & Social - Platform & tech: Web, Headset, live motion capture, WebXR, OnBoardXR - Idea: A live dancer performs inside a web page that anyone can walk into. - What it is: A dancer's live motion capture drives an avatar inside a WebXR scene that online audiences join from browsers or headsets, watching the performance unfold in a shared virtual space. - Technique: A motion-capture suit streams skeleton data over the network to a WebXR scene (on the OnBoardXR platform) where it animates an avatar in real time. - Try it: Stream MediaPipe pose data from one laptop over WebSocket to a three.js or A-Frame scene where it moves a simple avatar that classmates view on their phones. Twist: let viewers change the room's lighting to affect the mood of the dance. #### Where We Meet — Clémence Debaig (Unwired Dance Theatre) (2024) - Video: https://www.youtube.com/watch?v=5wZrmtrOJOo - Interaction: Location & City, Voice & Sound, Performance - Platform & tech: Wearable, indoor location tracking, spatial audio, headphones - Idea: Walk close to someone and you hear what they are thinking. - What it is: In an audio-based dance installation about mental health, visitors wander among solo dancers wearing headphones; walking close to a dancer triggers that dancer's inner monologue. - Technique: An indoor location-tracking system follows each visitor's position and triggers audio monologues on their headphones when they enter a zone around a performer. - Try it: Use Bluetooth beacons or printed markers plus a phone web app to play a different recorded 'thought' when a listener stands near each of three performers. Twist: the thought changes if two listeners stand there together. ### Denis Kalkofen *Associate Professor, Flinders University; formerly Graz University of Technology (ICG)* Specialist in AR visualization: X-ray and ghosted views of hidden structure, compact labels, and retargeting manuals and video tutorials into 3D AR instructions. #### Interactive Focus and Context Visualization for AR — Denis Kalkofen, TU Graz Handheld AR — Daniel Wagner & Dieter Schmalstieg (2007) - Video: https://www.youtube.com/watch?v=NOkWzJG2Ru4 - Interaction: Perception & Effects, Information & UI, Portals & Worlds - Platform & tech: Desktop, Phone, X-ray visualization, edge detection, marker tracking - Idea: See inside an object like an X-ray lens while keeping the shell outline to help you judge depth. - What it is: A magic-lens style AR view that reveals hidden internal parts of a real object (e.g. a car engine) while preserving key edges of the outside for depth cues. - Technique: Renders hidden virtual structure inside a focus region and composites real-world edges extracted from the video as context to keep occlusion cues. - Try it: In phone AR, overlay a 3D model of the 'internal structure' on a real cardboard box, keep the box's outer outline with a stroke, and compare the sense of depth with and without the outline. Twist: make the see-through region follow your finger. #### Adaptive Ghosted Views for AR — Denis Kalkofen, TU Graz Handheld AR — Daniel Wagner & Dieter Schmalstieg (2013) - Video: https://www.youtube.com/watch?v=xQlbeVjx0TI - Interaction: Perception & Effects, Information & UI, Portals & Worlds - Platform & tech: Phone, Desktop, ghosting, image-based importance, X-ray AR - Idea: An adaptive ghosted see-through view: the pipes behind the wall show up and the wall stays. - What it is: Ghosted X-ray views that adapt which real-world features to keep visible, so hidden pipes or rooms behind a wall stay readable without losing the wall. - Technique: Computes importance from real-world image features and adapts the transparency of the occluder per pixel before revealing the hidden virtual layer. - Try it: Build an AR "see through the wall" effect on a wall with a poster: keep the poster's main lines semi-transparent and show virtual pipes behind it. Twist: have classmates judge how much detail is best to keep. #### Dynamic Compact Visualizations for AR — Denis Kalkofen, TU Graz Handheld AR — Daniel Wagner & Dieter Schmalstieg (2013) - Video: https://www.youtube.com/watch?v=xL1aV9C4tYY - Interaction: Information & UI, Tangible Objects - Platform & tech: Phone, Desktop, view management, exploded views, tracking - Idea: AR labels and exploded views rearrange themselves with the viewpoint and never get cluttered. - What it is: AR annotations and exploded views of a real object are laid out compactly and reorganise dynamically as the viewpoint changes, avoiding clutter. - Technique: Uses view-management optimisation to cluster and position labels and explosion diagrams in screen space each frame. - Try it: Use phone AR to label the parts of a disassembled old appliance, and have classmates design how the labels avoid overlapping as the view rotates. Twist: only 3 labels may be shown, and the rest fold away. #### Retargeting Technical Documentation to AR — Denis Kalkofen, TU Graz Handheld AR — Daniel Wagner & Dieter Schmalstieg (2015) - Video: https://www.youtube.com/watch?v=SyM931lWDvo - Interaction: Information & UI, Tangible Objects - Platform & tech: Phone, document analysis, model-based tracking - Idea: Automatically turn a paper manual into AR steps overlaid on the real object. - What it is: Printed instruction manuals are automatically converted into 3D AR instructions: the arrows and steps from paper diagrams are mapped onto the real device. - Technique: Parses manual illustrations, registers them to a 3D model of the device, and re-renders arrows and motion cues in AR on the tracked real object. - Try it: Pick one page of an IKEA manual and have students use Reality Composer to place its arrows and part animations on the real object. Twist: compare whether the paper or the AR version gets things assembled faster. #### Retargeting Video Tutorials to AR — Denis Kalkofen, TU Graz Handheld AR — Daniel Wagner & Dieter Schmalstieg (2017) - Video: https://www.youtube.com/watch?v=FWytd1m6dKk - Interaction: Information & UI, Hands & Body, Drawing & Making - Platform & tech: Phone, Desktop, video analysis, tool tracking, AR playback - Idea: Pull the tool motions out of a video tutorial and turn them into AR paths laid over your own object. - What it is: Tool paths in video tutorials (e.g. painting or sanding strokes on a surface) are extracted and replayed as 3D AR guides on the user's own object. - Technique: Tracks the tool and its surface contact in the source video, reconstructs the 3D path, and renders it as animated guides registered to the target surface. - Try it: Pick a calligraphy or line-tracing tutorial video, trace its stroke paths by hand, and overlay them in phone AR on students' own paper so they can write along. Twist: have the path speed adapt to each student's writing speed. #### Mixed Reality Light Fields for Remote Assistance — Denis Kalkofen (2020) - Video: https://www.youtube.com/watch?v=NNmt-5NMuOk - Interaction: Shared & Social, Information & UI, Spatial Mapping - Platform & tech: Phone, Desktop, light fields, remote assistance, AR annotation - Idea: Capture a light field with a phone so a remote expert can view it from any angle and place notes back into the on-site AR. - What it is: A local worker captures a light field of a workspace with a phone; the remote expert explores it with free viewpoints and places annotations that appear in the worker's AR view. - Technique: Captures many registered images to render novel views of the scene for the remote helper, then anchors their 3D annotations back in the local tracking space. - Try it: Have student A photograph a desktop fault, such as wiring, from several angles with a phone, have student B annotate the steps on the photos, and have A restore them on site as AR labels. Twist: limit B to looking at only 3 photos. ### Johnny Chung Lee *HCI researcher; Wii Remote hacker; led Project Tango at Google* Did his PhD at Carnegie Mellon on projector-based tracking, where light sensors hidden in a surface let a projector find and follow it. His 2007 Wii Remote videos (head-tracked 3D, a $50 interactive whiteboard, finger tracking) reached millions; he later worked on Kinect at Microsoft and led Project Tango at Google. #### Automatic Projector Calibration with Embedded Light Sensors — Johnny Chung Lee (2004) - Video: https://www.youtube.com/watch?v=XgrGjJUBF_I - Interaction: Projection, Spatial Mapping, Tangible Objects - Platform & tech: Projection, structured light, photosensors, projector - Idea: Let the surface tell the projector where it is. - What it is: Light sensors hidden in the corners of a screen or object read a quick sequence of black-and-white stripe patterns, so the projector instantly knows where they are and fits its image exactly onto the target, even across several projectors or onto 3D shapes. - Technique: The projector shows Gray-coded binary patterns; each embedded photosensor decodes the on/off sequence it receives into its own pixel coordinate, which is used to pre-warp the image (UIST 2004, with Paul Dietz and Ramesh Raskar). - Try it: Recreate structured-light calibration: project a sequence of stripe patterns and film them with a phone, then decode which pixel falls on a paper target to align an image to it. Twist: move the target and re-calibrate in under ten seconds. #### Moveable Interactive Projected Displays — Johnny Chung Lee (2005) - Video: https://www.youtube.com/watch?v=liMcMmaewig - Interaction: Projection, Tangible Objects - Platform & tech: Projection, structured light, photosensors, projector - Idea: A projector that tracks the very surface it projects on, with no camera at all. - What it is: A hand-held board with light sensors in its corners can be carried around under a projector, and the projected content follows and fits it in real time, turning a piece of card into a movable, interactive screen. - Technique: Small structured-light patterns are projected only around the sensors' last known positions, and are frequency-modulated so they look like flat grey patches while updating fast enough for interactive tracking. - Try it: Make a 'magic clipboard': track a board with a printed marker using a webcam and project a map that stays locked onto it as students carry it around. Twist: show a different layer when the board is tilted. #### Foldable Interactive Displays — Johnny Chung Lee (2007) - Video: https://www.youtube.com/watch?v=nhSR_6-Y5Kg - Interaction: Projection, Tangible Objects - Platform & tech: Projection, Wii Remote, infrared LEDs, projector - Idea: Pull a large display out of your pocket by projecting onto paper that folds. - What it is: Sheets of paper and card that fold, roll and fan out become displays: a projector keeps its image locked onto the moving, bending surface, tracked by a Wii Remote watching infrared LEDs on the corners. - Technique: IR LEDs at the corners and folds of the surface are tracked by the Wii Remote, and the projected image is warped per segment to match the surface geometry (published at UIST 2008). - Try it: Project a video onto a hand-held piece of card: track its four corners with coloured stickers and a webcam, and warp the image with a homography in TouchDesigner or p5.js. Twist: fold the card in half and warp each half separately. #### Head Tracking for Desktop VR Displays — Johnny Chung Lee (2007) - Video: https://www.youtube.com/watch?v=Jd3-eiid-Uw - Interaction: Gaze & Attention, Perception & Effects, Portals & Worlds - Platform & tech: Desktop, Wii Remote, infrared LEDs, C#, DirectX - Idea: Track the viewer's head instead of the scene, and a flat screen becomes a window into depth. - What it is: A Wii Remote under the TV watches two infrared LEDs on the viewer's glasses; as he moves his head, a field of targets on an ordinary screen shifts in perspective like a window, and objects seem to float in front of the glass. - Technique: The Wii Remote's infrared camera finds two LEDs on the head, estimates head position from their spacing and offset, and renders an off-axis (view-dependent) projection each frame. - Try it: Build a head-coupled 'window' on a laptop: track the face with a webcam (MediaPipe or three.js face tracking) and move an off-axis camera so a 3D box appears to sit behind the screen. Twist: make one object poke out in front of the screen plane. #### Low-Cost Multi-touch Whiteboard using the Wiimote — Johnny Chung Lee (2007) - Video: https://www.youtube.com/watch?v=5s5EvhHy7eQ - Interaction: Projection, Drawing & Making, Tangible Objects - Platform & tech: Projection, Wii Remote, infrared pen, C# - Idea: Turn any projected wall into a touch surface with a game controller and a homemade pen. - What it is: A Wii Remote pointed at a projected image tracks pens with an infrared LED in the tip, turning any wall or screen into an interactive whiteboard with up to four pens for about the cost of the controller. - Technique: The controller's IR camera sees the pen tips as bright blobs; a four-corner calibration maps camera coordinates to screen coordinates, which drive mouse events. - Try it: Make a projected drawing wall: use a webcam to track a bright LED or phone flashlight on a projected sketch in p5.js, calibrated with four corner clicks. Twist: let two pens draw in different colours. #### Tracking Fingers with the Wii Remote — Johnny Chung Lee (2007) - Video: https://www.youtube.com/watch?v=0awjPUkBXOU - Interaction: Hands & Body, Information & UI - Platform & tech: Desktop, Wii Remote, infrared LEDs, reflective tape - Idea: Mid-air gesture control from a game controller, some LEDs and reflective tape. - What it is: An infrared LED ring lights up reflective tape on the fingertips so a Wii Remote can follow them; waving fingers in the air drags and scales items on a grid, a Minority Report style interface built from a game controller. - Technique: An IR LED array next to the Wii Remote illuminates retro-reflective tape; the controller's built-in blob tracker reports up to four points at 100 Hz to a C# and DirectX program. - Try it: Prototype a two-finger mid-air zoom: put coloured tape on thumb and index finger, track both with a webcam in p5.js and map their distance to the zoom of a photo. Twist: add a pinch gesture that 'grabs' the image. #### Google Tango Constructor — Johnny Chung Lee (2017) - Video: https://www.youtube.com/watch?v=A3WBae8dYq8 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, Headset, Google Tango, 3D scanning, VR - Idea: Capture a real room with a phone and step back into it in VR. - What it is: A developer tool for Tango phones such as the Asus Zenfone AR: walk around a space to capture a textured 3D scan, then view the result in VR on the same phone. - Technique: Tango's depth sensor and visual-inertial tracking fuse depth frames into a mesh and project camera colour onto it for a textured model. - Try it: Scan a small corner of the school with a phone 3D-scanning app and place the scan in a WebXR scene as a 'memory room'. Twist: scan the same spot a week later and let viewers fade between the two. #### Google Tango Research Update — Johnny Chung Lee (2017) - Video: https://www.youtube.com/watch?v=YulYq5P3heo - Interaction: Spatial Mapping, Information & UI - Platform & tech: Phone, Google Tango, depth sensing, 3D reconstruction - Idea: Phones that do not just map a room but know what is in it. - What it is: A short montage of research built on Google Tango phones, including rooms scanned into meshes and furniture automatically split and labelled by object type. - Technique: Tango's depth camera and motion tracking build a 3D reconstruction; research layers on top likely run semantic segmentation to colour each object class. - Try it: Scan a classroom corner with a LiDAR phone app (Polycam or Scaniverse), then hand-label chairs, tables and walls in different colours in Blender. Twist: place one AR note that only appears on objects labelled 'table'. ### Jun Rekimoto *HCI researcher; professor at the University of Tokyo and deputy director of Sony CSL* Interaction researcher who built NaviCam (1995), the first handheld ID-recognition AR system, and the 2D-barcode tracking later known as CyberCode, followed by landmark work on augmented tabletops, multi-touch and telepresence at Sony CSL and the Rekimoto Lab. #### NaviCam — Jun Rekimoto (1995) - Video: https://www.youtube.com/watch?v=S6XKPEexRbU - Interaction: Information & UI, Tangible Objects - Platform & tech: Phone, color-code ID tags, video see-through, handheld LCD - Idea: Hold a screen up to the world and it reveals the information attached to things. - What it is: A palmtop screen with a small camera is held up to the world; when it sees color-coded ID tags on shelves or doors it overlays context information on the live video, like a magnifying glass for hidden data. - Technique: A camera tethered to a workstation recognizes printed color-code tags in the video stream and draws text labels next to them on a handheld LCD (video see-through). - Try it: Print five ArUco markers, stick them on classroom objects and build a phone web page that shows a different 'secret note' next to each marker when the camera sees it. Twist: make the note change depending on how close the phone is to the marker. #### 3D CyberCode — Jun Rekimoto (1996) - Video: https://www.youtube.com/watch?v=cNFwEMkLK4A - Interaction: Tangible Objects, Play - Platform & tech: Desktop, 2D matrix code, pose estimation, CyberCode - Idea: A printed code can be both an ID and a 3D anchor for virtual objects. - What it is: Printed 2D barcode cards are placed in front of a camera and each one sprouts a 3D object that stays registered to the card as it is tilted and moved, years before ARToolKit. - Technique: The system decodes a square 2D matrix code and estimates its 3D pose from the code's corner points, then renders a model on top of the camera image. - Try it: Make a deck of six marker cards with an AR library (for example AR.js or MindAR) and give each card a 3D creature; when two cards touch, spawn a third creature between them. Twist: let players trade cards so the creatures 'travel' between tables. #### TransVision — Jun Rekimoto (1996) - Video: https://www.youtube.com/watch?v=P9P4Zp8sijk - Interaction: Shared & Social, Information & UI - Platform & tech: Phone, palmtop display, video see-through, shared scene - Idea: Shared handheld AR: many small windows onto one virtual object in the middle of a real table. - What it is: Several people around a table each hold a small palmtop display and see the same virtual 3D model sitting on the real table, so they can discuss and point at it together. - Technique: Each palmtop is a video see-through display whose position is tracked (likely by magnetic sensors), and a shared scene graph keeps the model in the same place for every viewer. - Try it: Build a two-phone shared AR scene (for example with an image target or a shared anchor) where both players see the same virtual building on a desk and can move one block each. Twist: give each phone a different 'lens' that shows a different layer of the same object. #### Pick-and-Drop — Jun Rekimoto (1997) - Video: https://www.youtube.com/watch?v=rFw9aMubL-Y - Interaction: Tangible Objects, Hands & Body - Platform & tech: Desktop, pen ID, networked displays, ubiquitous computing - Idea: Treat digital data like a physical thing you can pick up with one device and drop into another. - What it is: A user taps a file on one computer with a pen, physically carries the pen to another screen and taps again to drop it there, as if data were an object you can pick up. - Technique: Each pen has a unique ID; tapping registers the pen ID and the selected object with a network server, and tapping another screen with the same pen retrieves it. - Try it: Make two phones and a laptop share a 'grab' gesture: pinch on one screen to pick an image, then tap another device within five seconds to drop it there. Twist: let the image 'spill' if you walk too slowly. #### Augmented Surfaces — Jun Rekimoto (1999) - Video: https://www.youtube.com/watch?v=r4g_fvnjVCA - Interaction: Projection, Tangible Objects, Information & UI - Platform & tech: Projection, Desktop, ceiling camera, projector, visual markers - Idea: The whole room becomes one desktop: data flows off screens onto tables, walls and objects. - What it is: Laptops placed on a projected table and wall become part of one continuous workspace: you drag a document off the laptop screen onto the table, and physical objects with tags get digital data attached to them. - Technique: Ceiling cameras recognize laptops and tagged objects and track their positions, while projectors draw the shared workspace; a 'hyperdragging' cursor crosses from the laptop into the projection. - Try it: Point a projector and webcam at a desk and let students drag a picture from a laptop browser onto the projected table, where it sticks to a paper marker. Twist: when two markers touch, merge their pictures. #### SmartSkin — Jun Rekimoto (2002) - Video: https://www.youtube.com/watch?v=waSXkJBKT1s - Interaction: Hands & Body, Projection - Platform & tech: Projection, capacitive sensing, multi-touch, tabletop projection - Idea: Make any surface feel your whole hands, not just one pointer. - What it is: A table surface senses several hands and fingers at once, even hovering just above it, so people can knead, spread and push projected shapes with both hands. - Technique: A grid of transmitter and receiver electrodes under the surface measures capacitance changes caused by nearby hands, producing a proximity image that is interpreted as touch shapes. - Try it: Use a webcam under a glass table or a phone multi-touch page to build a 'dough' simulation that you can squeeze with two hands. Twist: make the dough remember where it was touched most and glow there. #### Flying Head — Jun Rekimoto (2013) - Video: https://www.youtube.com/watch?v=9HLyZqhkNKg - Interaction: Hands & Body, Gaze & Attention - Platform & tech: Headset, UAV, motion capture, head-mounted display - Idea: Your head becomes the joystick: body motion drives a flying camera. - What it is: A person wearing a head-mounted display walks, crouches and turns their head, and a small drone copies the same motion in another space, so flying feels like moving your own head. - Technique: Motion capture tracks the user's head position and orientation and maps it one-to-one to the drone's pose, while the drone's camera feed is shown in the HMD. - Try it: Map a phone's head-tracked orientation (in a cardboard viewer) to a webcam on a pan-tilt servo or a virtual camera in another room scene. Twist: scale your movements by 10x and discuss how it changes your sense of body size. #### JackIn Space — Jun Rekimoto (2017) - Video: https://www.youtube.com/watch?v=eXHUVTp2HNo - Interaction: Shared & Social, Spatial Mapping - Platform & tech: Headset, depth cameras, point cloud, telepresence - Idea: Blend first-person and third-person telepresence in one continuous camera move. - What it is: A remote helper in VR can jump into a worker's first-person view and then smoothly fly out into a live 3D reconstruction of the room to look around on their own. - Technique: Several depth cameras reconstruct the room as a live point cloud, and the worker's head camera is placed inside it so the remote viewer can transition between the two views. - Try it: Record a 360 video walking through a room plus a phone photogrammetry scan of the same room, and build a viewer that crossfades between the walker's view and a free-flying view. Twist: add a 'rubber band' that pulls the viewer back to the walker after five seconds. ### KAWS (Brian Donnelly) *Artist and designer* Artist whose COMPANION figures moved from street art and toys into giant inflatables and, with Acute Art, some of the most widely seen AR sculptures and AR packaging. #### KAWS:HOLIDAY Seoul — KAWS (Brian Donnelly) (2018) - Video: https://www.youtube.com/watch?v=cgtF21GOTHc - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, inflatable, lake, AllRightsReserved - Idea: A giant character doing something ordinary (sunbathing) in a real place is instantly shareable, the same reason giant AR characters spread online. - What it is: The first KAWS:HOLIDAY: a 28-metre inflatable COMPANION floats on its back on Seokchon Lake in Seoul, next to the Lotte World Tower, as if sunbathing during a heatwave. - Technique: A large inflatable figure is tethered on the lake with anchors and ballast, produced with AllRightsReserved. - Try it: Place a 20-metre AR character lying on a real lake, pool or lawn, doing an everyday activity, and design the best photo spot for it. Twist: it changes its pose according to the local weather. #### KAWS:HOLIDAY Hong Kong — KAWS (Brian Donnelly) (2019) - Video: https://www.youtube.com/watch?v=eCoEU74N7r4 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, inflatable, Victoria Harbour, AllRightsReserved - Idea: A floating giant that moves between harbour locations shows how an AR character can appear in different places on different days. - What it is: A 37-metre COMPANION floats on its back in Victoria Harbour in Hong Kong, moving between Tamar Park and Tsim Sha Tsui against the skyline. - Technique: The inflatable was mounted on a floating platform and towed between sites with the approval of harbour authorities. - Try it: Make an AR giant that appears at a different geolocated spot on campus each day, with a daily clue to find it. Twist: its route follows a real ferry or bus timetable. #### KAWS:HOLIDAY Japan — KAWS (Brian Donnelly) (2019) - Video: https://www.youtube.com/watch?v=JXiE03O3sW4 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, inflatable, campsite, Mount Fuji - Idea: Pairing a giant character with an iconic natural landmark gives every photo two anchors, a composition trick for AR. - What it is: A 40-metre COMPANION lies on the grass of Fumotoppara campground with Mount Fuji behind it; visitors camp around the figure for a week. - Technique: An inflatable figure is anchored on the meadow, with camping infrastructure laid out around it. - Try it: Place an AR character so it lines up with a real landmark (mountain, tower) from a marked photo spot. Twist: the character turns its head to look at the landmark at sunset. #### KAWS:HOLIDAY Taipei — KAWS (Brian Donnelly) (2019) - Video: https://www.youtube.com/watch?v=vq8KoNSZ7uc - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, inflatable, public square, AllRightsReserved - Idea: A soft, casual giant in front of a formal monument changes how the square feels, a contrast AR creators can use deliberately. - What it is: A 36-metre seated COMPANION holding a smaller figure lies in Liberty Square in Taipei, in front of the Chiang Kai-shek Memorial Hall; the opening was organised with singer JJ Lin. - Technique: A giant inflatable is anchored on the square with ballast, lit at night. - Try it: Place a relaxed giant AR character in front of the most formal building on campus and film reactions. Twist: the character sits up whenever someone takes a photo. #### KAWS:HOLIDAY Space — KAWS (Brian Donnelly) (2020) - Video: https://www.youtube.com/watch?v=JwWDtlr2oQI - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, sounding balloon, stratosphere, camera payload - Idea: A small figure filmed against the curve of the Earth plays with scale in the opposite direction from the giants, a reminder that AR can shrink as well as enlarge. - What it is: To mark COMPANION's 20th anniversary, a small figure in an astronaut suit was carried by a sounding balloon to 41.5 kilometres into the stratosphere and filmed floating above the Earth. - Technique: A figure and cameras on a weather-balloon payload rose for about eight hours before descending by parachute. - Try it: Launch an AR figure from your hand that rises until it becomes a speck against the real sky, and track it with a compass arrow. Twist: other users in the city can see where it drifted. #### KAWS × Reese's Puffs WebAR — KAWS (Brian Donnelly), Acute Art (2021) - Video: https://www.youtube.com/watch?v=_2MlPIhdJbY - Interaction: Tangible Objects, Play - Platform & tech: Web, Phone, WebAR, ZapWorks, Unity - Idea: Scan a cereal box and play an AR game with KAWS characters on your table. - What it is: Scanning a QR code on KAWS-designed cereal boxes launches a WebAR game in which KAWS characters appear around the box. - Technique: An on-pack QR code opens a browser-based AR experience built with the ZapWorks Universal AR SDK for Unity, tracking the packaging as the anchor. - Try it: Design web AR for a snack package with MindAR or 8th Wall so that scanning the package launches a mini-game. Twist: different patterns on the back of the package unlock different levels. #### KAWS: FAMILY AR experience (AGO) — KAWS (Brian Donnelly), Acute Art (2022) - Video: https://www.youtube.com/watch?v=PlmdoymVGjk - Interaction: Location & City, Spatial Mapping, Perception & Effects - Platform & tech: Phone, Acute Art app, ARKit/ARCore - Idea: Summon giant KAWS sculptures into a museum gallery with your phone. - What it is: At the Art Gallery of Ontario, visitors open the Acute Art app to see KAWS's giant sculptures appear inside the Galleria Italia alongside the physical exhibition. - Technique: Acute Art's app uses plane detection and site-specific placement to anchor large animated 3D sculptures in the gallery space. - Try it: Design an oversized AR sculpture for a school corridor with Reality Composer or Adobe Aero, and study the impact that scale creates. Twist: the sculpture turns to look at the nearest viewer. #### KAWS:HOLIDAY Changbai Mountain — KAWS (Brian Donnelly) (2022) - Video: https://www.youtube.com/watch?v=zqGiPif83ds - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, snow sculpture, mountain resort - Idea: Making the character from the landscape's own material (snow) ties it to the season, an idea for AR objects whose material changes with weather. - What it is: For the eighth HOLIDAY, two COMPANION figures sitting together were built as a monumental snow sculpture at Changbai Mountain in Jilin, China. - Technique: The figures were carved from compacted snow on site at monumental scale. - Try it: Make an AR figure whose material switches between snow, grass and sand according to the current season or live weather at the location. Twist: it melts slowly when the temperature is above zero. ### Marshmallow Laser Feast *Immersive art studio (Robin McNicholas, Barney Steel, Ersin Han Ersin)* London studio founded in 2011 (with Memo Akten as an early co-founder) that moved from real-time projection mapping, lasers and drone shows to multi-sensory VR and mixed-reality installations about nature, breath and perception. #### Sony PlayStation Realtime Projection Mapping — Marshmallow Laser Feast, Memo Akten (2011) - Video: https://vimeo.com/34021153 - Interaction: Projection, Tangible Objects, Portals & Worlds - Platform & tech: Projection, real-time 3D tracking, dynamic projection mapping, live puppetry - Idea: Projection mapping that follows moving objects turns an ordinary room into a film set. - What it is: A living room is transformed live, in camera, into three famous film scenes: projections track moving furniture and puppets, while pyrotechnics and real objects blend with the projected worlds. - Technique: Objects and puppets are tracked in 3D in real time and the projection content is re-rendered from the projector's viewpoint every frame, so the mapping stays locked to moving surfaces. - Try it: Track a white box with an ArUco marker from the projector's camera and keep a texture projected on it while a classmate moves it around. Twist: turn the box into a 'window' onto a movie scene that stays fixed in world space. #### Meet Your Creator — Marshmallow Laser Feast, Memo Akten (2012) - Video: https://www.youtube.com/watch?v=JLAKXJG1trU - Interaction: Performance, Perception & Effects - Platform & tech: Projection, quadrotor drones, motion capture, motorized mirrors - Idea: Flying robots as moving light pixels that draw in the air of a theater. - What it is: Sixteen quadrotor drones carrying LEDs and motorized mirrors fly in formation over a theater stage, bouncing beams of light and building glowing sculptures in the air above the audience. - Technique: A motion-capture system tracks each drone and choreography software sends synchronized flight paths, LED colors and mirror angles to bounce stage lights. - Try it: Choreograph a 60-second light show for five virtual drones in a 3D web scene (three.js), timing positions and colors to a song. Twist: add one real flashlight and a mirror in the classroom and aim the beam to hit a marked target. #### Laser Forest — Marshmallow Laser Feast (2013) - Video: https://www.youtube.com/watch?v=gDZqJ4RPVYc - Interaction: Tangible Objects, Voice & Sound, Shared & Social - Platform & tech: Projection, motion sensors, lasers, haze - Idea: A walk-through instrument where every touch becomes sound and light in space. - What it is: Visitors walk through a forest of 150 tall rods; tapping or shaking a rod plays a note and fires laser light across the dark hall, so the whole crowd becomes an orchestra. - Technique: Each rod has a motion sensor that triggers synthesized sounds and laser patterns, with lasers and haze making beams visible between the rods. - Try it: Make three 'trees' from cardboard tubes with a phone or micro:bit accelerometer inside; shaking each one plays a note and flashes a projected beam on the ceiling. Twist: make trees that only sound when shaken together. #### In the Eyes of the Animal — Marshmallow Laser Feast (2015) - Video: https://vimeo.com/140057053 - Interaction: Perception & Effects, Spatial Mapping, Location & City - Platform & tech: Headset, LiDAR, point cloud, binaural audio - Idea: Swap human senses for animal senses inside the very place you are standing. - What it is: Visitors in a real forest wear headsets and see the same forest as a mosquito, dragonfly, frog or owl might, rendered from LiDAR and CT scans as flowing particles and sound. - Technique: LiDAR scans of Grizedale Forest and CT scans of animals are turned into point-cloud worlds with spatial binaural audio, experienced in headsets on site. - Try it: Scan a corner of the school garden with a phone LiDAR app, import it as a point cloud and render it the way a bee might see it (only moving things and UV-like colors). Twist: shrink the camera to insect height and speed. #### Treehugger: Wawona — Marshmallow Laser Feast (2016) - Video: https://www.youtube.com/watch?v=if0wfysmoMU - Interaction: Tangible Objects, Perception & Effects, Hands & Body - Platform & tech: Headset, VR headset, physical sculpture alignment, particle systems - Idea: A physical hug is the interface to the invisible life inside a tree. - What it is: Participants hug a physical sculpture of a giant sequoia and put their head into its knot; in the headset, they see water and nutrients flowing inside the tree, and the longer they hug, the deeper they drift into tree time. - Technique: A headset experience is aligned with a physical tree sculpture so touch matches the virtual trunk, with particle visuals and spatial audio driven by dwell time. - Try it: Wrap a pillar or tree with a cardboard 'bark' and build a phone-in-headset scene that shows flowing particles inside it, advancing only while the user keeps both hands on it. Twist: add a fan or smell cue that appears at the deepest level. #### A Colossal Wave — Marshmallow Laser Feast (2017) - Video: https://vimeo.com/244047652 - Interaction: Perception & Effects, Tangible Objects, Voice & Sound - Platform & tech: Headset, VR umbrellas, show control, physical effects - Idea: Mix a headset vision with real physical impacts so the whole room shakes with the virtual event. - What it is: Visitors stand under VR umbrellas and watch a surreal flood, while a real dropper tower, bowling ball and gong in the gallery make physical impacts and sounds synced to the virtual wave. - Technique: Umbrella-mounted displays show the virtual scene while a show-control system triggers the physical mechanisms and sound-reactive gallery lighting in sync. - Try it: Build a short phone VR scene of a wave and time three real-world cues in the room (a drum hit, a fan, a dropped object) to match it. Twist: let the person outside the headset perform the cues live. #### We Live in an Ocean of Air — Marshmallow Laser Feast (2018) - Video: https://vimeo.com/303589503 - Interaction: Shared & Social, Voice & Sound, Perception & Effects - Platform & tech: Headset, multi-user VR, breath sensor, heart-rate sensor - Idea: Visualize the breath exchange between people and trees in a shared space. - What it is: Groups of visitors wear headsets and backpacks in a room with a giant sequoia; they see each other as glowing bodies and watch their own breath flow into the tree as it breathes back. - Technique: Multi-user tracked VR with breath sensors and heart-rate input drives particle visuals, combined with scent, wind and a physical tree set. - Try it: Use a phone microphone to detect breathing and draw each breath as a particle cloud drifting toward a virtual plant in a shared web AR scene. Twist: the plant only grows when two people breathe in sync. ### Memo Akten *Artist and researcher; Assistant Professor, UC San Diego* Turkish-born artist and computer scientist working with code, computer vision and machine learning; author of the ofxMSAFluid tools widely used in the openFrameworks community. #### Webcam Piano — Memo Akten (2007) - Video: https://vimeo.com/1219327 - Interaction: Hands & Body, Voice & Sound - Platform & tech: Desktop, Projection, openFrameworks, optical flow - Idea: A webcam turns the air into an invisible piano, so waving your body plays harmonious music. - What it is: A webcam turns the space in front of a screen into an invisible grid of notes; body movement plays harmonious, classical-sounding music visualized as fluid. - Technique: Frame differencing or optical flow over a grid of screen cells detects motion per cell, and each active cell triggers a note quantised to a musical scale, visualised with a fluid simulation. - Try it: Use p5.js to split the camera image into an 8x8 grid and play the matching pentatonic note wherever there is motion. Twist: switch the scale and timbre so one specific action (such as sweeping the floor) becomes a song. #### Body Paint — Memo Akten (2008) - Video: https://vimeo.com/3576457 - Interaction: Hands & Body, Drawing & Making, Projection - Platform & tech: Projection, openFrameworks, ofxMSAFluid, computer vision - Idea: Your body is the brush: dance in front of a projected wall and paint splashes and flows with your moves. - What it is: Visitors dance in front of a wall-sized projection and their movements splash and smear fluid paint in real time, turning the body into a brush. - Technique: Optical flow from a camera provides velocity vectors that inject dye and force into a GPU 2D fluid simulation (ofxMSAFluid), projected at wall scale. - Try it: Use optical flow in TouchDesigner or p5.js to turn body movement into forces that inject paint into a fluid, and project it on a wall as a canvas. Twist: the color each person wears decides their paint color, and the class finishes with one group 'body painting'. #### Forms — Memo Akten (2012) - Video: https://www.youtube.com/watch?v=zCiTqZf170I - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, motion tracking, procedural animation, simulation - Idea: Turning an athlete's motion into a sculpture of lines and surfaces shows how AR can make the invisible physics of a body visible around the body itself. - What it is: Footage of Commonwealth Games athletes — gymnasts, divers, weightlifters — is turned into abstract animated sculptures that trace the forces and paths of their movements. - Technique: Made with Quayola: the athletes' movement is tracked from video and used to drive procedural geometry and simulations that visualise trajectories, forces and time. - Try it: Record a friend jumping or throwing with body tracking, then play the motion back in AR as sweeping ribbons and arcs anchored where it happened. Twist: show only the forces (arrows) and hide the body. #### Learning to See — Memo Akten (2017) - Video: https://vimeo.com/260612034 - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Projection, Desktop, GAN, machine learning, camera - Idea: As hands move cloth and cables on a table, AI sees them live as waves, fire or nebulae. - What it is: A camera watches a table of everyday objects - cloth, cables, keys - and neural networks trained on waves, fire, clouds, flowers or Hubble images re-render the scene live as visitors rearrange it with their hands. - Technique: An image-to-image network (pix2pix-style GAN) trained on a single domain such as waves or flowers takes the live camera image, edge-processed, as input and outputs a re-rendered frame in real time. - Try it: Use Runway or a pretrained pix2pix or style-transfer model to render the clutter on a desk seen by the camera as clouds, fire or waves in real time. Twist: take 100 photos of one place on campus to train a small model so the clutter "sees" that place. ### Moment Factory *Multimedia entertainment studio* Studio known for architectural projection mapping, illuminated night walks (Lumina) and interactive projected play spaces worldwide. #### Nine Inch Nails: Lights in the Sky — Moment Factory (2008) - Video: https://www.youtube.com/watch?v=iD7BNesOIGE - Interaction: Performance, Hands & Body, Voice & Sound - Platform & tech: Projection, LED screens, camera tracking, real-time visuals - Idea: When images respond to where the performer stands, the screen stops being a backdrop and becomes a partner; body-tracked AR should react to people, not just sit behind them. - What it is: For Nine Inch Nails' 2008 tour, layered transparent LED screens and a projection floor surrounded the band; Trent Reznor could walk behind a wall of static and 'part' it with his body, or be framed inside glowing grids. The making-of shows how the interactive visuals were built with the band's team. - Technique: Infrared cameras tracked the musicians' positions and fed real-time generative graphics shown on semi-transparent LED panels (likely custom tracking software). - Try it: Use a phone's body segmentation (Lens Studio or MediaPipe) to put a layer of noise or static in front of a performer that clears only where their body is. Twist: the static also reacts to the loudness of the music. #### Mosaïka (Parliament Hill, Ottawa) — Moment Factory (2010) - Video: https://vimeo.com/15168693 - Interaction: Projection, Location & City, Information & UI - Platform & tech: Projection, architectural projection mapping, sound and light show - Idea: A national building can hold many small personal stories; civic AR works when it layers real voices onto symbolic places. - What it is: A 30-minute sound and light show projected onto the Parliament buildings and grounds in Ottawa, telling Canada's stories through interviews with 200 Canadians from diverse backgrounds. The Gothic façade becomes a canvas for landscapes, faces and voices. - Technique: Architectural projection mapping with multiple high-lumen projectors aligned to a 3D model of the façade, cued with a surround soundtrack of recorded interviews. - Try it: Record short interviews with five people about one building on campus, then place each voice as a spatial audio source with a small visual marker at a specific window using AR Foundation or 8th Wall. Twist: the voices only play when two phones are pointed at the building together. #### Madonna Super Bowl XLVI Halftime Show — Moment Factory (2012) - Video: https://www.youtube.com/watch?v=WqseMKmd3Bo - Interaction: Performance, Projection - Platform & tech: Projection, floor projection mapping, stage design, live show - Idea: Projecting on the floor seen from above transforms a whole venue; AR designers can treat the ground plane as the biggest, most flexible canvas. - What it is: For Madonna's 2012 Super Bowl halftime show, Moment Factory turned the entire stadium field and stage into a projection surface: temples, palaces, patterns and giant typography rearranged themselves under the performers in a few minutes of live television. - Technique: The field and modular stage were covered with top-down projection mapped to a 3D model of the set and camera angles, cued to the music. - Try it: Design a floor-based AR scene with WebXR plane detection that turns a classroom floor into a changing map as a performer walks across it. Twist: the pattern is only fully readable from the balcony or a raised phone. #### Sagrada Familia - Ode à la Vie — Moment Factory (2012) - Video: https://www.youtube.com/watch?v=RS-OTtIsBKY - Interaction: Projection, Location & City, Performance - Platform & tech: Projection, projection mapping - Idea: A projection show that makes the stone walls of the Sagrada Família grow, flood and bloom. - What it is: A monumental projection-mapping show on Gaudí's Sagrada Família façade for La Mercè festival, making the stone appear to grow, flood and bloom. - Technique: Large-scale multi-projector architectural mapping with edge blending, using a detailed 3D model of the facade to render growth and flooding effects that respect its relief. - Try it: Project onto an object with relief or a complex surface (a plaster bust, a carved box) so its surface seems to grow, flood or bloom. Twist: express an emotion from the object's "inner life". #### Foresta Lumina — Moment Factory (2014) - Video: https://www.youtube.com/watch?v=AIMcZtSUiFo - Interaction: Projection, Location & City - Platform & tech: Projection, projection, interactive lighting - Idea: A story told in light along a forest trail at night. - What it is: An illuminated night walk through a forest in Coaticook, Quebec, with projection, light and sound that tell a story along the trail. - Technique: A network of projectors, lights and speakers distributed along a trail is triggered in sequence (likely show control with sensors), so a narrative unfolds as visitors walk. - Try it: Set up three "light nodes" (a phone projector, an LED strip, a Bluetooth speaker) along a campus path at night and link them into one complete short story. Twist: one node only triggers when the audience is quiet. #### AURA — Moment Factory (2017) - Video: https://www.youtube.com/watch?v=FV3XdOda3zM - Interaction: Projection, Location & City, Performance - Platform & tech: Projection, projection mapping - Idea: A church interior turned into an immersive space of light and image. - What it is: Projection mapping and light transform the interior of Montreal's Notre-Dame Basilica into an immersive audiovisual experience. - Technique: Interior architectural mapping with many projectors and programmable lighting synchronised by a show-control timeline, following the basilica's surveyed geometry. - Try it: Write a two-minute light-and-projection storyboard for an interior you know well (an auditorium, a stairwell) and rehearse it with a small projector on a paper model. Twist: drive every lighting change from changes in sound alone, with no timeline. #### Augmented Games — Moment Factory (2021) - Video: https://www.youtube.com/watch?v=jhLKSY6NB5I - Interaction: Projection, Play, Hands & Body - Platform & tech: Projection, motion tracking, projection - Idea: Turn any space into a projected game field you can run and jump on. - What it is: Motion-tracked floor and wall projection that turns any space into a physical video-game playground and sports court. - Technique: Overhead cameras or LiDAR track players' positions on the floor, and a game engine renders court graphics and game logic back through calibrated floor and wall projection. - Try it: Track people on the floor with an overhead camera and build a small game projected onto the floor in TouchDesigner or Unity, such as claiming territory or dodging blocks. Twist: the rules require at least two people to cooperate to score. #### Phish at Sphere — Moment Factory (2024) - Video: https://www.youtube.com/watch?v=aP-y5CDJBUs - Interaction: Performance, Voice & Sound, Perception & Effects - Platform & tech: Projection, Sphere LED dome, real-time visuals, live concert - Idea: An improvising band needs improvising visuals; live AR performances need real-time operators, not only pre-baked scenes. - What it is: For Phish's four-night residency at Sphere in Las Vegas, Moment Factory created a different show each night on the venue's wraparound LED interior, with visuals played live alongside the band's improvisation. - Technique: Real-time engines drive custom content on Sphere's interior display, with visual operators performing and switching scenes live (engine likely Unreal or Notch). - Try it: Build a shared AR concert scene in Unity with Netcode or a WebXR multiplayer template, where one student operates the visuals live from a laptop while others watch through phones. Twist: the operator can only use three knobs, forcing a clear visual language. ### Schnelle Bunte Bilder (with kling klang klong) *Media art and design studio (interactive rooms and projections)* Berlin studio whose team includes Sebastian Huber, Johannes Timpernagel and Robert Pohle, often working with the sound studio kling klang klong. They build body-reactive rooms, floor projections and museum installations in vvvv, from the dance exhibits of the Deutsches Hygiene-Museum to swarms of projected fish. #### SENSESCAPES — Schnelle Bunte Bilder (with kling klang klong) (2012) - Video: https://vimeo.com/50854847 - Interaction: Hands & Body, Projection, Voice & Sound - Platform & tech: Projection, vvvv, infrared cameras, 8 projectors - Idea: Stir a design epoch with your body. - What it is: A 360-degree room at the Grassi Museum in Leipzig where visitors call up graphic interpretations of design epochs from Art Nouveau to today, then shape and sound them with their movement. - Technique: Four infrared cameras track visitors, and vvvv on five PCs drives eight projectors around the room, changing graphics and sound according to people's positions. - Try it: Pick three visual styles (for example Art Nouveau curves, Bauhaus blocks, pixel art) and let each corner of the room project one of them, blending as a tracked person walks between corners. Twist: the style mixes when two people stand in different corners. #### BLINK OF AN EYE — Schnelle Bunte Bilder (with kling klang klong) (2014) - Video: https://vimeo.com/101410848 - Interaction: Hands & Body, Perception & Effects, Projection - Platform & tech: Projection, vvvv, camera, slit-scan - Idea: See your recent past moving next to you. - What it is: Even the smallest movement leaves time-delayed images on the wall: wandering video strips, shadows of your past movements or painted light collages. - Technique: A camera feed is buffered in time and re-composited as slit-scan strips, delayed shadows and painted trails before being projected at wall scale. - Try it: Build a webcam slit-scan or delay mirror in p5.js where each column of pixels shows a different moment in the last five seconds. Twist: the delay grows the closer you stand to the screen. #### EPILOG — Schnelle Bunte Bilder (with kling klang klong) (2014) - Video: https://vimeo.com/99909498 - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, vvvv, tracking cameras, haze - Idea: Move and the light around you turns into walls and tunnels. - What it is: A 25-minute room of light, sound and haze at the Deutsches Hygiene-Museum's dance exhibition: a floor projection reacts to visitors' movements and, through the haze, their movement builds three-dimensional spaces of light. - Technique: Tracking cameras follow visitors on the floor, vvvv generates patterns that respond to them, and projectors shining through haze turn the flat patterns into volumetric beams. - Try it: In a dark room with a hazer or a spray bottle of water mist, project a white circle that follows a tracked student from a ceiling projector. Twist: the circle shrinks around people who stand still. #### STEPSEQUENCER — Schnelle Bunte Bilder (with kling klang klong) (2014) - Video: https://vimeo.com/101091090 - Interaction: Voice & Sound, Hands & Body, Play - Platform & tech: Projection, vvvv, floor projection, sensors - Idea: Your dance steps compose the beat. - What it is: Instead of music that makes people dance, dancing makes the music: visitors step on a round floor-projected instrument and use three physical stations for jumping and moving, and every action adds a sound or beat. - Technique: A camera tracks footsteps on a circular floor projection laid out like a step sequencer, and sensors in the side stations trigger additional sounds, all mixed live in the software. - Try it: Tape a 4x4 grid on the floor, detect which squares are occupied with a webcam from above, and loop a drum pattern from them in Tone.js. Twist: each square plays the voice of whoever stands on it. #### MIRROR (M2) — Schnelle Bunte Bilder (with kling klang klong) (2015) - Video: https://vimeo.com/140405048 - Interaction: Hands & Body, Voice & Sound, Perception & Effects - Platform & tech: Headset, Oculus Rift, Kinect 2, vvvv, Max/MSP - Idea: See yourself dissolve into the sound you make. - What it is: A headset experiment in which the user's movement becomes sound, sound becomes light, and their mirrored body dissolves into a fluid stream of particles. - Technique: An Oculus Rift and a Kinect 2 capture the body, a Max/MSP patch turns head and limb positions plus microphone input into sound, and vvvv renders a mirrored particle body driven by that sound. - Try it: Build a WebXR or phone AR mirror where your tracked skeleton is drawn as particles whose size follows the microphone level. Twist: the particles lag behind the body by one second. #### FANTASTIC VOYAGE — Schnelle Bunte Bilder (with kling klang klong) (2017) - Video: https://vimeo.com/207717295 - Interaction: Tangible Objects, Perception & Effects, Projection - Platform & tech: Projection, tracking, generative graphics, LED - Idea: Shine a light inside an object and it comes alive. - What it is: Visitors explore the abstract interior of a sculpture with a light tool inspired by endoscopes; the generative world reacts to where they shine, and the sculpture's outside glows to mirror their journey. - Technique: A tracked handheld light acts as the controller for a real-time generative environment shown inside the sculpture, while LEDs on its shell replay the visitor's path. - Try it: Let students shine a phone flashlight into a cardboard box with a webcam inside, and grow generative shapes on a screen wherever the light hits. Twist: shapes only survive in the dark once the light moves on. #### +Panic — Schnelle Bunte Bilder (with kling klang klong) (2021) - Video: https://vimeo.com/654523297 - Interaction: Hands & Body, Projection, Play - Platform & tech: Projection, vvvv gamma, GPU swarm, tracking - Idea: A school of fish that only trusts you if you are calm. - What it is: Thousands of projected fish swim in an exhibition foyer and sense the visitors. If people move calmly, the fish become curious and approach; fast movements spread panic through the swarm. - Technique: Overhead tracking measures visitors' positions and speeds, and a GPU swarm simulation in vvvv gamma gives each fish curiosity and contagious flight behaviours, with sound linked to the swarm. - Try it: Code a boids flock in p5.js that follows the mouse or a tracked hand when it moves slowly and scatters when it moves fast. Twist: panic spreads from fish to fish with a delay. #### Amazonia – The Interactive Forest — Schnelle Bunte Bilder (with kling klang klong) (2023) - Video: https://vimeo.com/813835066 - Interaction: Hands & Body, Projection, Information & UI - Platform & tech: Projection, body tracking, projection, real-time 3D - Idea: Walk into a rainforest wall and let your movement uncover its stories. - What it is: An immersive wall projection lets children explore the Amazon rainforest and discover stories of its Indigenous cultures by moving their own bodies. - Technique: Body tracking in front of the wall (likely depth cameras) lets children push foliage aside and trigger animated story scenes in a real-time projected environment. - Try it: Project a dense leaf layer over a hidden picture and use webcam body segmentation to clear leaves wherever a child moves. Twist: the leaves grow back unless several children hold them open together. ### Sebastian Lague *Programmer and educator; creator of the Coding Adventures series* Makes the Coding Adventures videos, in which he builds fluids, ray marchers, clouds and boids from scratch in Unity and releases every project on GitHub. #### Coding Adventure: Boids — Sebastian Lague (2019) - Video: https://www.youtube.com/watch?v=bqtqltqcQhw - Source code: https://github.com/SebLague/Boids - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, Unity, Compute Shader - Idea: Separation, alignment and cohesion are enough to make a flock. - What it is: A flock of birds swirls through space, splitting around obstacles and regrouping, driven only by three local rules. - Technique: Each boid steers by separation, alignment and cohesion with neighbours computed in a compute shader, and avoids obstacles by casting rays along points of a golden-spiral sphere. - Try it: Release a flock of boids in phone AR that uses the LiDAR mesh as obstacles, so birds fly around real furniture. Twist: the flock is attracted to the phone and scatters when you shake it. #### Coding Adventure: Clouds — Sebastian Lague (2019) - Video: https://www.youtube.com/watch?v=4QOcCGI6xOU - Source code: https://github.com/SebLague/Clouds - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL, Raymarching, 3D noise - Idea: Clouds are noise that light travels through. - What it is: Soft volumetric clouds drift through the sky and light scatters through them, controlled live from 3D noise settings. - Technique: A post-process raymarches a box volume, samples tiled 3D Worley and Perlin noise for density, and accumulates light with Beer's law and a phase function. - Try it: Fill a room-sized AR box with raymarched clouds on a phone or HoloKit so students walk through them. Twist: the clouds thicken where the room is darkest in the camera image. #### Coding Adventure: Hydraulic Erosion — Sebastian Lague (2019) - Video: https://www.youtube.com/watch?v=eaXk97ujbPQ - Source code: https://github.com/SebLague/Hydraulic-Erosion - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, Unity, C#, Compute Shader - Idea: Let simulated rain, not an artist, carve the landscape. - What it is: A rough noise terrain is weathered by hundreds of thousands of raindrops until it shows realistic valleys, ridges and river channels. - Technique: Each droplet is simulated as a particle that follows the heightmap gradient, picks up and deposits sediment based on speed and capacity, and evaporates. - Try it: Place a small terrain on a real table in AR and let students 'rain' on it by holding the phone above it until rivers form. Twist: the terrain uses a LiDAR scan of a crumpled paper sheet as its start. #### Coding Adventure: Marching Cubes — Sebastian Lague (2019) - Video: https://www.youtube.com/watch?v=M3iI2l0ltbE - Source code: https://github.com/SebLague/Marching-Cubes - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: Turn a field of numbers into a surface you can carve. - What it is: An endless cave and planet terrain is generated from 3D noise and turned into smooth meshes that the player can dig into. - Technique: Density values from noise are sampled on a grid in a compute shader, and the marching-cubes lookup table turns each cell into triangles along the iso-surface. - Try it: Generate a marching-cubes blob on a real table in phone AR and let students carve it by tapping. Twist: carving adds material somewhere else so the total volume stays the same. #### Coding Adventure: Ray Marching — Sebastian Lague (2019) - Video: https://www.youtube.com/watch?v=Cp5WWtMoeKg - Source code: https://github.com/SebLague/Ray-Marching - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader, Raymarching, SDF - Idea: Describe shapes as distance formulas and you get smooth blends and infinite fractals for free. - What it is: Shapes melt into each other, a Mandelbulb fractal unfolds, and objects cut holes into one another, all rendered from distance functions. - Technique: A compute shader marches rays through signed distance functions, combining primitives with union, subtraction and smooth-min, and renders a Mandelbulb distance estimator. - Try it: Place a raymarched Mandelbulb hovering above a real table in phone AR and let pinch gestures change its power. Twist: the fractal grows more detailed the closer you walk. #### Coding Adventure: Portals — Sebastian Lague (2020) - Video: https://www.youtube.com/watch?v=cWpFZbjtSQg - Source code: https://github.com/SebLague/Portals - Interaction: Portals & Worlds, Play - Platform & tech: Desktop, Unity, Render textures - Idea: A portal is a second camera whose image is pasted exactly where the doorway is. - What it is: The player walks and throws objects through seamless portals that connect distant rooms, seeing through each one without a visible seam. - Technique: Each portal renders a linked camera into a texture using screen-space UVs, applies an oblique near-clip plane, handles recursion and slices objects that are halfway through. - Try it: Anchor two portals to two real doors in AR so the view through one shows the room behind the other. Twist: the linked room is a virtual copy of the classroom at night. #### Coding Adventure: Ant and Slime Simulations — Sebastian Lague (2021) - Video: https://www.youtube.com/watch?v=X-iSQQgOd1A - Source code: https://github.com/SebLague/Slime-Simulation - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: Simple agents that sense and deposit trails grow living networks. - What it is: Millions of agents leave glowing trails and follow each other's scent, forming slime-mould networks that branch, merge and pulse across the screen. - Technique: A compute shader moves agents that sample a trail map with three sensors and steer toward the strongest, deposit into the map, and a diffuse-and-decay pass spreads the trails (Physarum model). - Try it: Project the slime simulation onto a detected AR floor plane and use the people-occlusion mask as food, so the network grows toward people standing in the room. Twist: the slime avoids anyone who moves too fast. #### Coding Adventure: Simulating Fluids — Sebastian Lague (2023) - Video: https://www.youtube.com/watch?v=rSKMYc1CQHE - Source code: https://github.com/SebLague/Fluid-Sim - Interaction: Perception & Effects, Play - Platform & tech: Desktop, Unity, Compute Shader, SPH - Idea: Water is just many particles that try to keep a constant density. - What it is: Thousands of particles slosh, splash and settle as water in a box, first in 2D then in 3D, reacting to the mouse pushing and pulling them. - Technique: Smoothed Particle Hydrodynamics runs in compute shaders: density and pressure are computed from neighbours found through a spatial hash, and particles are integrated each frame. - Try it: Run a small 2D SPH simulation on a phone and place its container on a real table in AR, tilting the water with the phone's gravity sensor. Twist: pour the water out onto the real floor plane. ### Snap Inc. (Snapchat Lenses) *Camera company; Snapchat Lenses and Lens Studio* Snap turned face filters into a mass medium with Snapchat Lenses (2015), then added World Lenses, Lens Studio for creators, Landmarkers and Local Lenses. #### Jeff Koons x Snapchat — Snap Inc. (Snapchat Lenses) (2017) - Video: https://www.youtube.com/watch?v=d5z9-JLIuis - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, Snapchat, geolocated lenses - Idea: Jeff Koons sculptures installed in AR at famous places around the world. - What it is: Geolocated AR versions of Jeff Koons sculptures such as Balloon Dog appeared at sites like Central Park and the Champ de Mars, visible only through a Snapchat lens at those places. - Technique: Geofenced lenses become available only near specific coordinates, and the sculpture is placed with ground tracking at a fixed site. - Try it: Design an AR sculpture for a spot on campus that is visible only there, using GPS to limit where it appears. Twist: let classmates leave AR graffiti next to the sculpture, and discuss who owns public space. #### Snapchat World Lenses — Snap Inc. (Snapchat Lenses) (2017) - Video: https://www.youtube.com/watch?v=K6x44v8prFA - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, Snapchat, SLAM - Idea: Animated 3D characters dropped into the real world through the rear camera turned AR into everyone's meme. - What it is: Snapchat's rear-camera lenses place animated 3D objects, such as a rainbow-puking cloud or the Dancing Hot Dog, on the ground in the real scene that users can walk around and record. - Technique: Rear-camera SLAM with ground detection anchors an animated character to the floor, and video recording captures the composited result for sharing. - Try it: Make a World Lens with an animated character on the ground in Lens Studio and record a 10-second clip. Twist: the character imitates the movements of a real person in front of the camera, using body tracking. #### Landmarkers — Snap Inc. (Snapchat Lenses) (2019) - Video: https://www.youtube.com/watch?v=jNs9kdSAOCA - Interaction: Location & City, Spatial Mapping, Perception & Effects - Platform & tech: Phone, Lens Studio, landmark tracking - Idea: Recognize a real landmark and make the whole Eiffel Tower warp, melt or spray rainbows on camera. - What it is: Lenses that recognise famous buildings such as the Eiffel Tower or Flatiron Building and transform their real facades with animation, e.g. making the structure melt or burst into rainbows. - Technique: Landmark tracking matches the camera view to a prebuilt 3D model of a known building, so shaders and animations can be mapped precisely onto its real facade. - Try it: Use Lens Studio's Custom Landmarker or image tracking to recognize a building facade on campus and overlay a melting or grass-growing animation on it. Twist: make the animation change with the real time of day. #### Local Lenses — Snap Inc. (Snapchat Lenses) (2020) - Video: https://www.youtube.com/watch?v=4jA1RM5_WMc - Interaction: Location & City, Shared & Social, Drawing & Making - Platform & tech: Phone, Lens Studio, city-scale 3D map - Idea: People paint real buildings along a whole street together, and the AR content stays where it was put. - What it is: Persistent, shared AR that covers entire city blocks: multiple Snapchatters can paint and decorate real building facades together, and the changes stay in place for others. - Technique: A city-scale 3D map localizes users, and shared persistent state stores each paint stroke on building surfaces so later visitors see the accumulated changes. - Try it: Use persistent anchors and a simple backend such as Firebase to build a shared AR graffiti wall whose graffiti is still there the next time you open it. Twist: the graffiti slowly fades like real paint. #### Spectacles AI release: Snap 3D and LLM integrations — Snap Inc. (Snapchat Lenses) (2025) - Video: https://www.youtube.com/watch?v=qpi1JTnPwgs - Interaction: Drawing & Making, Voice & Sound, Tangible Objects - Platform & tech: Headset, Snap Spectacles, Snap 3D, OpenAI, Gemini, SnapML - Idea: AR glasses where you can ask for a 3D object and it appears. - What it is: The June 2025 Spectacles OS release adds OpenAI, Gemini and DeepSeek integrations, better speech-to-text, the Snap 3D generative API that creates 3D objects on demand, and new object-tracking samples. - Technique: Lens Studio exposes cloud LLMs and a text-to-3D endpoint (Snap 3D) to lenses, so a spoken prompt returns a mesh that is placed in the room. - Try it: Build a voice-to-3D toy: speech-to-text, then a free text-to-3D API, then place the result on a detected table in phone AR. Twist: every new object must be something the previous one would be afraid of. #### Spectacles AI-powered translation challenge — Snap Inc. (Snapchat Lenses) (2025) - Video: https://www.youtube.com/watch?v=hR4W_Qtzij4 - Interaction: Shared & Social, Voice & Sound - Platform & tech: Headset, Snap Spectacles, speech-to-text, machine translation - Idea: Subtitles for real life let strangers build something together. - What it is: Two people who speak different languages collaborate on a build: one instructs, one builds, and a Spectacles translation lens shows each spoken sentence as subtitles in the other's language. - Technique: Speech is transcribed, translated by an AI model and rendered as captions anchored near the speaker in the wearer's view. - Try it: Run a LEGO challenge where the instructor speaks one language and the builder reads live translated captions on a phone held in a headset viewer. Twist: translate into emoji only. #### Agent Center for Spectacles — Snap Inc. (Snapchat Lenses) (2026) - Video: https://www.youtube.com/watch?v=U0cA-bVxLgM - Source code: https://github.com/specs-devs/samples - Interaction: Voice & Sound, Information & UI - Platform & tech: Headset, Snap Spectacles, CLI coding agents, Lens Studio - Idea: Your coding agents follow you around as floating panels. - What it is: Agent Center connects a developer's command-line coding agents to Spectacles, so they can prompt agents in AR and keep track of their code while away from the desk. - Technique: A bridge relays prompts and agent output between CLI agents running on a computer and a Spectacles lens that shows sessions as spatial windows. - Try it: Mirror the output of a long-running script to a phone WebAR panel that floats by your door, and add a button to send one-word replies back. Twist: the panel changes colour with the agent's mood. ### 1024 Architecture (François Wunschel & Pier Schneider) *Art and architecture studio for light, mapping and stage structures* Studio founded in 2007 by François Wunschel and Pier Schneider that builds scaffold structures, stages and façades animated by projection mapping, LEDs and lasers, and co-developed ideas behind the MadMapper software. #### BOOMBOX — 1024 Architecture (François Wunschel & Pier Schneider) (2008) - Video: https://vimeo.com/15734398 - Interaction: Projection, Performance, Voice & Sound - Platform & tech: Projection, stage projection mapping, VJ software - Idea: A stage made of simple boxes becomes a musical instrument of light. - What it is: A giant stack of cubes behind a DJ becomes a mapped sculpture whose faces light up, shift and pulse with the music across a worldwide tour of electronic parties. - Technique: Pre-built cube geometry is mapped with projectors and animated live, with visuals triggered and modulated by the music. - Try it: Stack six cardboard boxes, map each face in MadMapper or a web mapper, and make a beat-reactive light show where each box is one drum sound. Twist: let the audience rearrange the boxes between songs. #### Perspective Lyrique — 1024 Architecture (François Wunschel & Pier Schneider) (2010) - Video: https://vimeo.com/18888136 - Interaction: Voice & Sound, Projection, Location & City - Platform & tech: Projection, audio analysis, real-time mesh deformation, projection mapping - Idea: Give a building a face and let the crowd's voice control its expression. - What it is: At the Fête des Lumières in Lyon, members of the public sing or shout into a microphone and the projected face on the old Célestins theatre façade opens its mouth, stretches and deforms with their voice. - Technique: An audio-analysis algorithm maps microphone volume and frequency to the deformation of a 3D mesh of the façade, rendered live and projection-mapped back onto it. - Try it: Project a simple face onto a box or wall and drive its mouth, eyes and wobble from microphone amplitude and pitch in p5.js. Twist: let two microphones control the left and right halves of the face. #### TETRA.TENNIS — 1024 Architecture (François Wunschel & Pier Schneider) (2010) - Video: https://vimeo.com/19163299 - Interaction: Projection, Perception & Effects, Play - Platform & tech: Projection, floor projection mapping, anamorphic illusion - Idea: A sports field is a ready-made canvas whose lines can come alive. - What it is: The tennis court of the Paris Masters at Bercy is mapped from above between matches, with lines, grids and 3D illusions that make the flat court fold and break apart. - Technique: Overhead projectors aligned to the court's markings play animations that use the known line layout for perspective illusions visible to the audience in the stands. - Try it: Map the lines of a basketball or badminton court (or a taped floor plan) from above and animate the lines turning into a maze. Twist: have a player walk the maze while the projection reacts to their position. #### Make the Line Dance — 1024 Architecture (François Wunschel & Pier Schneider) (2011) - Video: https://vimeo.com/21308228 - Interaction: Hands & Body, Projection, Performance - Platform & tech: Projection, Kinect, Quartz Composer, MadMapper - Idea: Projection mapping onto a moving body instead of a static building. - What it is: A dancer moves in front of a projector and a Kinect, and lines of light are mapped onto their moving body, sticking to limbs as they turn and bend. - Technique: A Kinect tracks the dancer's skeleton, Quartz Composer draws graphics along the bones, and MadMapper aligns the projector view with the camera view. - Try it: Calibrate a projector to a webcam, run MediaPipe pose tracking and project a glowing line along one arm of a volunteer. Twist: make the line leave a trail that the dancer can 'dance with' a second later. #### The Walking Cube — 1024 Architecture (François Wunschel & Pier Schneider) (2011) - Video: https://vimeo.com/131077465 - Interaction: Tangible Objects, Perception & Effects, Performance - Platform & tech: Projection, mechanical actuators, LED, show control - Idea: Combine small physical motion with light to give a simple shape a personality. - What it is: A large cube frame is shaken and tilted by mechanical actuators, and synchronized light and video make it look like a living creature walking, breathing and stumbling. - Technique: Pneumatic or motorized actuators under the legs are sequenced with lighting and video cues from the same timeline. - Try it: Put a cardboard cube on four servo 'legs' controlled by an Arduino and sync LED strips to its movement to make it look scared, happy or tired. Twist: let it react to a clap. #### TESSERACT (HyperCube) — 1024 Architecture (François Wunschel & Pier Schneider) (2013) - Video: https://vimeo.com/79702430 - Interaction: Projection, Perception & Effects, Location & City - Platform & tech: Projection, scaffold structure, layered projection, anamorphosis - Idea: Use a real 3D frame to make a fourth dimension feel visible. - What it is: A scaffolding cube in a public square is covered with projection and light so that, seen from the front, it appears to fold, rotate and unfold like a four-dimensional hypercube. - Technique: Projection surfaces stretched inside a scaffold frame receive content rendered for the main viewing angle so layered surfaces create a volumetric illusion. - Try it: Build a cube frame from sticks and stretch two layers of tulle inside, then project content that seems to rotate a cube inside a cube. Twist: find the one viewing spot where the illusion is perfect and mark it on the floor. #### 1D ARCADE — 1024 Architecture (François Wunschel & Pier Schneider) (2025) - Video: https://vimeo.com/1098205588 - Interaction: Play, Tangible Objects, Shared & Social - Platform & tech: Projection, MadMapper, LED pixel strips - Idea: A whole video game squeezed into one line of light. - What it is: Visitors play arcade and pinball-style games made from a single line of light pixels, discovering the rules without instructions by pressing buttons and moving. - Technique: Each game is coded in MadMapper and output to LED pixel strips, reading simple physical controls in real time. - Try it: Program a one-dimensional game on a 60-pixel LED strip (or a 1-pixel-tall canvas) with two buttons, such as tug-of-war or dodge the bullet. Twist: make it a two-player game where players stand at each end of a corridor. ### Autodesk Research (Fraser Anderson, Tovi Grossman, George Fitzmaurice et al.) *Corporate HCI research group* Autodesk's UI research group builds AR tools for making and learning: projected pen overlays, AR movement mirrors, assembly-tutorial authoring and two-way MR remote instruction. #### PenLight — Autodesk Research (Fraser Anderson, Tovi Grossman, George Fitzmaurice et al.) (2009) - Video: https://www.youtube.com/watch?v=9pQz-YviPOA - Interaction: Projection, Drawing & Making, Tangible Objects - Platform & tech: Projection, Wearable, digital pen, pico projector - Idea: A pen with its own projector casts digital layers onto the paper while you write. - What it is: A digital pen with a built-in mini projector overlays digital content, such as building floor-plan layers, directly onto the paper where you are writing. - Technique: Tracks the pen's position on dot-pattern paper and projects spatially registered overlays around the pen tip. - Try it: Aim a phone projector at a printed floor plan and project a 'hidden plumbing/wiring layer' from above for students to trace with a pen. Twist: replace the projector with phone AR and compare the handheld and projected experiences. #### YouMove — Autodesk Research (Fraser Anderson, Tovi Grossman, George Fitzmaurice et al.) (2013) - Video: https://www.youtube.com/watch?v=DsZ-9opi150 - Interaction: Hands & Body, Information & UI - Platform & tech: Projection, Desktop, Kinect, half-silvered mirror, skeleton tracking - Idea: An AR mirror overlays the teacher's moves on your reflection and teaches you the dance step by step. - What it is: An AR mirror records an expert's movement and then trains learners by overlaying skeleton guides and feedback on their mirror reflection, gradually removing guidance. - Technique: Uses Kinect skeleton tracking behind a large half-silvered mirror to draw recorded motion guides aligned with the user's reflection and score their accuracy. - Try it: Use a phone's front camera and body pose recognition (such as MediaPipe or Effect House) to overlay a "teacher skeleton", and have students follow a set of calisthenics and get a score. Twist: gradually fade the skeleton until it is completely hidden. #### Ivy: Spatially Situated Visual Programming — Autodesk Research (Fraser Anderson, Tovi Grossman, George Fitzmaurice et al.) (2017) - Video: https://www.youtube.com/watch?v=Gn8sBvKHHYk - Interaction: Information & UI, Spatial Mapping, Drawing & Making - Platform & tech: Headset, visual programming, IoT, HTC Vive - Idea: Program the lights and sensors in a real room by drawing wires between them. - What it is: In VR/AR, users see the sensors and actuators of a smart environment where they physically are and wire them together with visual-programming links in space. - Technique: Anchors node-graph widgets to each device's physical location and lets users draw dataflow connections between them in 3D. - Try it: Put QR codes on the classroom's lights, door and windows, show their "nodes" in phone AR, and have students draw lines to compose rules like "door opens, light turns on" (a home-automation simulator works). Twist: two people build a rule together, one drawing the input and the other drawing the output. #### HydroRing — Autodesk Research (Fraser Anderson, Tovi Grossman, George Fitzmaurice et al.) (2018) - Video: https://www.youtube.com/watch?v=ofRJ80eyEQw - Interaction: Hands & Body, Tangible Objects - Platform & tech: Wearable, Headset, liquid flow haptics, HoloLens - Idea: Use flowing liquid to create heat, cold and pressure at your fingertip without getting in the way of touching real things. - What it is: A thin ring pumps warm, cold or pulsing liquid through a latex tube on the fingertip to create pressure, vibration and temperature cues while leaving touch of real objects intact. - Technique: Drives liquid through a thin tube over the fingerpad with pumps and heaters, modulating flow and temperature to render haptic effects. - Try it: Have classmates touch a cold water bag and a warm water bag with their fingers while seeing 'ice' or 'fire' in phone AR, and discuss how realistic it feels when sight and touch match. Twist: deliberately make the visuals contradict the temperature. #### AuthAR — Autodesk Research (Fraser Anderson, Tovi Grossman, George Fitzmaurice et al.) (2019) - Video: https://www.youtube.com/watch?v=75GGfsBshvs - Interaction: Information & UI, Tangible Objects - Platform & tech: Headset, HoloLens, Kinect, tutorial authoring - Idea: Generate an AR tutorial automatically while you assemble. - What it is: While an expert assembles furniture, the system records video and 3D tracking to author an AR assembly tutorial at the same time, which learners then follow in a headset. - Technique: Combines object tracking with synchronized video capture so each assembly step becomes a spatially anchored AR instruction with clips and annotations. - Try it: Work in pairs: one person records each step on a phone while building with LEGO, and the other uses Reality Composer to place each step's video or arrows at the matching part, for a third person to follow. Twist: the tutorial may not use any text. #### Loki — Autodesk Research (Fraser Anderson, Tovi Grossman, George Fitzmaurice et al.) (2019) - Video: https://www.youtube.com/watch?v=cBN1XZXZw3Y - Interaction: Shared & Social, Hands & Body, Information & UI - Platform & tech: Headset, VR/AR telepresence, RGB-D capture, recording and playback - Idea: Two-way mixed-reality remote teaching that switches between video, 3D and virtual replicas. - What it is: A bi-directional mixed-reality telepresence system where teacher and learner can each switch between video, 3D capture and virtual-replica views to demonstrate physical tasks. - Technique: Streams RGB-D point clouds of both spaces and lets either user record, annotate and replay 3D demonstrations inside the other's environment. - Try it: Set up a remote origami class between two classrooms using a video call plus phone AR, where the teacher's hand movements are recorded as short clips pinned to the student's AR desk. Twist: students can send back their own mistakes for the teacher to annotate. ### AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) *Immersive dance company and lab; creators of AR, VR and AI-driven dance works* Dancer-choreographers Aoi Nakamura and Esteban Lecoq (both ex-Jasmin Vardimon Company) run AΦE, whose works move dance into VR (WHIST), shared AR (0AR) and AI-driven LED installations (LILITH.AEON); their A+E Lab in Chatham hosted the UK's first Choreographic Coding Lab in 2022. #### WHIST — AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) (2017) - Video: https://vimeo.com/799520099 - Interaction: Performance, Gaze & Attention, Tangible Objects - Platform & tech: Headset, VR, 360 video, spatial audio - Idea: The audience's own attention and wandering assemble a personal cut of a Freudian dance story. - What it is: A one-hour physical-theatre installation where visitors move through a set of sculptural objects wearing VR headsets; where you look and which objects you approach choose which of 76 dream-like 360° scenes of a fictional family you see. - Technique: Physical props act as trigger zones and gaze targets that branch between pre-filmed 360° dance scenes with 3D sound, likely via positional tracking of each headset. - Try it: Film three short 360° dance clips, then build a WebXR or Cardboard scene where gazing at a real object in the room for two seconds switches to the matching clip. Twist: make the branch depend on which object the viewer ignores. #### 0AR — AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) (2018) - Video: https://www.youtube.com/watch?v=Iz_MevAUnEk - Interaction: Performance, Shared & Social, Tangible Objects - Platform & tech: Phone, ARKit, Unity, motion capture, iPad - Idea: Bring the collective liveness of a theatre audience into AR by making it a shared, co-located viewing. - What it is: A set of short AR dance pieces drawn from Akram Khan and Sidi Larbi Cherkaoui's zero degrees: five audience members on connected tablets see the dancers pop up in 3D around a shared table and interact with one another in real time. - Technique: Motion-captured dance (made with Mbryonic) is played back as 3D characters anchored to a shared tracking target, with five iPads synced over a local network so all viewers see the same moment. - Try it: Record a 20-second dance with a phone mocap app, export it as an animated FBX, and place it on an image marker in AR Foundation so a small group can watch it together around one table. Twist: let each viewer's position cue a different dancer to enter. #### LILITH.AEON — AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) (2024) - Video: https://vimeo.com/925210197 - Interaction: Performance, Hands & Body, Shared & Social - Platform & tech: Projection, LED cube, AI, motion capture, Unreal Engine - Idea: A virtual being dances back at the crowd, so every visit becomes a new duet between people and an AI. - What it is: A 60-minute installation inside a monumental LED cube where Lilith, a virtual dancer reincarnated from a cryogenically frozen child, responds to how visitors move around her with AI-generated choreography. - Technique: Audience positions are tracked in the space and fed to a motion-generation model that blends motion-captured dance clips of the character on the LED walls, likely with camera or depth sensing. - Try it: Use MediaPipe in the browser to find where a viewer stands, and have a projected avatar pick from five mocap clips so it turns towards or away from them. Twist: the avatar only dances when two viewers stand still together. ### Chris Sugrue *Artist and programmer; EyeWriter team; SFPC teacher* American artist and programmer creating interactive installations and audio-visual performances that explore light, perception and the body; part of the EyeWriter team. #### Delicate Boundaries — Chris Sugrue (2007) - Video: https://vimeo.com/1007230 - Interaction: Hands & Body, Projection, Portals & Worlds - Platform & tech: Projection, computer vision, openFrameworks - Idea: Little bugs made of light crawl out of the screen and onto the hands of whoever touches it. - What it is: Small bugs made of light crawl out of a computer screen and onto the hands and arms of people who touch it. - Technique: A camera looking at the screen and arms detects hand contact and arm contours, and a projector calibrated to the same space lets agent-based bugs walk off the screen and follow those contours. - Try it: Keep a swarm of bugs on screen, and when MediaPipe Hands detects a finger touching the screen edge, have the bugs 'crawl' into imagery projected onto the arm. Twist: bugs that stay on your hand too long grow into something you did not expect. #### A Cable Plays — Chris Sugrue (2008) - Video: https://vimeo.com/12618634 - Interaction: Tangible Objects, Performance - Platform & tech: Projection, openFrameworks, computer vision - Idea: Two people stretch yarn across a game board, and new life grows between the strings in the augmented image. - What it is: With Damian Stewart: two performers pin yarn across a game board while an augmented video of the board reveals generative creatures emerging between the strings. - Technique: A camera over the board detects the yarn lines (likely via color thresholding and Hough line detection), and the resulting polygon regions spawn generative creatures in the augmented video. - Try it: Film a pegboard from above while two people take turns stretching colored yarn, use p5.js to detect the areas enclosed by the strings, and spawn small creatures inside them. Twist: turn it into a two-player game where whoever's creatures eat the other's wins. #### Base 8 — Chris Sugrue (2008) - Video: https://vimeo.com/30834797 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Pepper's ghost, computer vision - Idea: Reach into a world floating in a glass reflection, and geometry grows out from between your fingers. - What it is: A reflection through glass makes a world float in mid-air; as visitors reach in, geometric structures grow between their fingers and out of their hands. - Technique: A Pepper's ghost setup reflects a screen in angled glass to float graphics in a real volume, while a camera tracks hand positions behind the glass to grow geometry between the fingers. - Try it: Build a small Pepper's ghost box from a phone and a sheet of clear acrylic angled at 45 degrees, then use MediaPipe Hands to grow geometry between your fingers. Twist: the structures only connect when two hands (possibly from two people) come close. #### Decrypted Reflections — Chris Sugrue (2009) - Video: https://vimeo.com/7652087 - Interaction: Tangible Objects, Projection, Perception & Effects - Platform & tech: Projection, openFrameworks - Idea: Move a set of mirrors to break a projected generative image apart and bounce it all over the room. - What it is: Visitors move an analog array of mirrors to break apart and rearrange a projected generative image, scattering its fragments across the room. - Technique: A projected generative image hits an array of physically adjustable mirrors, so rotating mirrors redirects fragments of the projection around the room in an analog, non-computed way. - Try it: Project onto a row of small rotatable mirrors (or phone screens, mirror paper) so fragments bounce around the room, and let everyone turn the mirrors to reassemble the image. Twist: design a secret message that can only be read when every fragment lands on the same wall. #### Memory of Form and Matter — Chris Sugrue (2013) - Video: https://vimeo.com/163678148 - Interaction: Projection, Tangible Objects, Perception & Effects - Platform & tech: Projection, projection mapping, openFrameworks, digital fabrication - Idea: Project simulations onto a 3D-printed sculpture so the still object seems to breathe like a living thing. - What it is: Digitally fabricated sculptures are overlaid with projection-mapped simulations, so static printed structures appear to breathe and move with light. - Technique: Projection mapping registers the projector to the 3D-printed sculpture's CAD model, and simulations (flow, displacement) are rendered in model space so light follows the physical geometry. - Try it: Make a small structure by 3D printing or paper sculpture and project a breathing light-and-dark animation onto it using TouchDesigner's Kantan Mapper or projection mapping. Twist: sync its "breathing" to the noise in the room. ### Compagnie Nicole Seiler *Choreographer and video artist; dance works built on projection, video games and audio walks* Swiss choreographer Nicole Seiler makes dance pieces that pit the body against its image: projections mapped onto dancers (Pixel Babes, Shiver), video-game bodies in public space (K Two), dancers watched through city windows (Living-Room Dancers) and a geolocated smartphone audio walk (Palimpsest). #### Madame K — Compagnie Nicole Seiler (2004) - Video: https://vimeo.com/190536002 - Interaction: Projection, Performance, Face - Platform & tech: Projection, video projection - Idea: Projection as make-up: the image a body wants to show versus the body itself. - What it is: A multimedia solo where Madame K, a woman who wants to escape her imperfect body, is shaped and disguised by video projections that fall on her and around her. - Technique: Pre-edited video is projected onto the performer and set, with choreography timed so her body fits into or breaks out of the projected shapes. - Try it: Project a filmed face or outfit onto a volunteer standing in a fixed spot, and choreograph 30 seconds of fitting in and slipping out of it. Twist: the projection is of a different person. #### Dolls/Dolls Live — Compagnie Nicole Seiler (2005) - Video: https://vimeo.com/190992843 - Interaction: Projection, Perception & Effects, Performance - Platform & tech: Projection, video projection, installation - Idea: Put projected phantoms and living dancers in the same dark room and let visitors meet both. - What it is: A walk-through video installation where filmed dancing dolls and creatures appear as ghostly projections in the dark; in the live version two real dancers step into the installation alongside their projected doubles. - Technique: Superimposed video loops are projected onto objects and surfaces around the space, with live performers matching their positions and costumes. - Try it: Film a friend dancing against black, project the clip onto a hanging sheet in a dark corridor, and have the same friend appear live next to it. Twist: the live dancer copies the projection a beat late. #### Pixel Babes — Compagnie Nicole Seiler (2006) - Video: https://vimeo.com/190371062 - Interaction: Projection, Hands & Body, Performance - Platform & tech: Projection, projection mapping, video - Idea: Media images of perfect bodies are literally pasted onto real ones. - What it is: A trio in ten tableaux about the ideal body, from Jane Fonda aerobics to fashion and plastic surgery, where video images are projected directly onto the dancers' bodies. - Technique: Video is projection-mapped onto the performers' bodies and costumes, with choreography holding poses precisely inside the projected image. - Try it: Project a fitness video onto a person in white clothing and ask them to follow the moves while staying aligned with the projected body. Twist: speed the video up until the alignment breaks. #### K Two — Compagnie Nicole Seiler (2007) - Video: https://vimeo.com/165467660 - Interaction: Performance, Play, Location & City - Platform & tech: Wearable, choreography, street performance - Idea: Humans perform the limits of game animation in the street. - What it is: An improvised street performance in which two dancers move like video game characters, with glitches, loops and stiff animation cycles, as if the game had leaked into public space. - Technique: No screens: the choreography is built from the movement grammar of game characters (idle loops, clipping, respawns), turning the body itself into the rendering. - Try it: List five game animation glitches, turn each into a movement rule, and perform them in a corridor while classmates play a game soundtrack. Twist: passers-by can press an imaginary button to trigger a respawn. #### Living-Room Dancers — Compagnie Nicole Seiler (2008) - Video: https://vimeo.com/190519439 - Interaction: Location & City, Performance, Voice & Sound - Platform & tech: Wearable, binoculars, audio walk, site-specific - Idea: Turn a street of windows into a stage, with the soundtrack in each viewer's ears. - What it is: A choreographic city walk where the audience, with binoculars and MP3 players, watches amateur dancers (tap, tango, pole, electro) performing in private apartments seen through their windows. - Technique: Low-tech augmentation: timed audio tracks on personal players are synchronised with live dancers in lit windows along a guided route. - Try it: Map a short walking route, place a friend dancing in a visible window, and give walkers a geotagged audio track in a web app that starts when they arrive. Twist: the audio describes a dance that is not what they see. #### Shiver — Compagnie Nicole Seiler (2014) - Video: https://vimeo.com/92021691 - Interaction: Projection, Perception & Effects, Performance - Platform & tech: Projection, live video, projection mapping, computer animation - Idea: Your own live double, projected onto you, starts to behave suspiciously. - What it is: A film-noir and horror-flavoured dance piece where a video system projects live, animated replicas of the dancers onto their own bodies, so reality and projection slip out of sync. - Technique: Live cameras capture the dancers, the feed is stylised with computer animation and re-projected onto the bodies with deliberate offsets and delays. - Try it: Aim a projector and webcam at the same person, apply an edge or cartoon filter, and project it back onto them with a one-second delay. Twist: the projection starts moving before the person does. #### Palimpsest — Compagnie Nicole Seiler (2018) - Video: https://vimeo.com/280331092 - Interaction: Location & City, Voice & Sound, Perception & Effects - Platform & tech: Phone, GPS, audio walk, smartphone app - Idea: Audio-only AR: describe a dance precisely enough and the city starts performing it. - What it is: A smartphone audio walk of geotagged sound compositions in the language of audio description, which narrate dancers who are not there so they gradually appear in the listener's imagination in the real city. - Technique: A smartphone app triggers audio-description tracks by GPS location along a route, so each description matches what the listener is standing in front of. - Try it: Write three 30-second audio descriptions of an imaginary dancer at three campus spots and trigger them by location in a web app. Twist: one description places the dancer right behind the listener. ### Disney Research — Ivan Poupyrev, Robert Sumner & colleagues *Disney's research labs in Pittsburgh, Zurich and Los Angeles; Ivan Poupyrev led interaction research there before Google ATAP* Disney Research explored playful and magical AR: Ivan Poupyrev's REVEL and AIREAL added touch to projected and virtual content, Robert Sumner's Zurich team brought children's coloring pages to life in AR, and Magic Bench let people share a bench with a CG character without glasses. #### REVEL — Disney Research — Ivan Poupyrev, Robert Sumner & colleagues (2012) - Video: https://www.youtube.com/watch?v=L7DGq8SddEQ - Interaction: Tangible Objects, Hands & Body, Projection - Platform & tech: Projection, reverse electrovibration - Idea: Send a faint signal through the body so projected virtual objects have texture you can feel. - What it is: Olivier Bau and Ivan Poupyrev's reverse electrovibration injects a tiny signal into the user's body so that projected or AR virtual objects on real surfaces gain touchable textures. - Technique: Reverse electrovibration injects a weak time-varying electrical signal into the user's body, creating an oscillating electrostatic attraction between the sliding finger and any grounded conductive surface, which is perceived as texture and can be modulated per virtual object. - Try it: Simulate tactile textures in AR with phone vibration (the Vibration API or Unity Handheld.Vibrate): play vibrations of different frequencies and rhythms as a finger slides over different virtual objects (REVEL's electrical stimulation is not suitable to reproduce in class). Twist: have classmates close their eyes and guess the material from the vibration alone. #### AIREAL — Disney Research — Ivan Poupyrev, Robert Sumner & colleagues (2013) - Video: https://www.youtube.com/watch?v=xaFBjUJj00M - Interaction: Hands & Body, Play - Platform & tech: Projection, Desktop, air vortex, Kinect - Idea: An air-vortex cannon lets you "touch" virtual butterflies and balls in mid-air. - What it is: Rajinder Sodhi, Ivan Poupyrev, Matthew Glisson and Ali Israr use a 3D-printed vortex cannon to fire rings of air so you can feel virtual butterflies and balls in free air. - Technique: A flexible nozzle on a pan-tilt mount aims a speaker-driven air vortex ring at the tracked user's hand (via a depth camera), delivering localized tactile pulses in free air. - Try it: Make a simple air cannon from a paper cup or plastic bottle and a balloon membrane, pair it with a virtual butterfly in phone AR, and have a classmate fire an air ring by hand whenever the butterfly "touches" someone's hand. Twist: design an "invisible" AR creature that you can only feel, not see. #### Augmented Creativity — Disney Research — Ivan Poupyrev, Robert Sumner & colleagues (2015) - Video: https://www.youtube.com/watch?v=AYemmtQOf60 - Interaction: Drawing & Making, Play - Platform & tech: Phone, AR - Idea: AR turns children's coloring, building and storytelling into something alive. - What it is: A collection of AR experiences that bridge real and virtual creative play: coloring, building and storytelling with physical toys brought to life. - Technique: A set of AR applications that track physical creative artifacts (colored drawings, built toys) with image and model-based tracking, then extract color and shape from them to drive animated virtual counterparts. - Try it: Have students make a physical piece from paper, clay or LEGO, photograph it, and bring it to life in Lens Studio or Reality Composer with animation, sound or a talking face. Twist: the piece's behavior in AR must be driven by one of its physical features. #### Live Texturing of AR Characters from Colored Drawings — Disney Research — Ivan Poupyrev, Robert Sumner & colleagues (2015) - Video: https://www.youtube.com/watch?v=SWzurBQ81CM - Interaction: Drawing & Making, Tangible Objects, Play - Platform & tech: Phone, texture synthesis, Vuforia - Idea: Hold up a tablet and a child's coloring page becomes a moving 3D character wearing the child's own colors. - What it is: Children color a page in a coloring book and a tablet instantly shows the drawn character as a 3D animated figure with their own coloring, even on hidden sides. - Technique: The coloring page is tracked and rectified, the drawn colors are sampled into the 3D character's UV texture through a precomputed mapping, and occluded parts are filled by a texture synthesis lookup from visible regions. - Try it: Design a coloring line drawing and a matching 3D model in Unity, then use AR image tracking to capture the page and map its image onto the model through the UVs (a planar projection is a fine approximation). Twist: let the coloring affect the character's behavior, for example the more red, the grumpier it gets. #### Magic Bench — Disney Research — Ivan Poupyrev, Robert Sumner & colleagues (2017) - Video: https://www.youtube.com/watch?v=JnLk2_Xbr90 - Interaction: Hands & Body, Shared & Social, Play - Platform & tech: Desktop, depth camera, haptics, 3D compositing - Idea: Sit on the bench and a baby elephant on the screen sits down beside you, while the bench rumbles along. - What it is: Sit on a bench and see yourself in a big mirror-like screen next to a CG elephant or character who reacts to you, with haptic rumble in the bench, no glasses needed. - Technique: A depth camera reconstructs the bench area so a CG character can be composited into the mirror-view video with correct depth ordering and occlusion, while actuators in the bench deliver synchronized haptic effects. - Try it: Make a "magic mirror" in Unity or Lens Studio: flip the camera image horizontally on a large screen, use body segmentation so a virtual character stands behind or in front of people by depth, and sync a phone vibrating under the chair with the character's actions. Twist: have the character react differently to each person who sits down. #### Makeup Lamps — Disney Research — Ivan Poupyrev, Robert Sumner & colleagues (2017) - Video: https://www.youtube.com/watch?v=Ilgu3aFCphs - Interaction: Face, Projection, Performance - Platform & tech: Projection, high-speed projector, face tracking - Idea: High-speed projection paints moving makeup onto a performer's face in real time. - What it is: Amit Bermano and colleagues project dynamic makeup onto a performer's moving face in real time with a high-speed projector-camera setup. - Technique: A high-speed camera tracks facial landmarks and a low-latency projector, co-axially calibrated with it, renders makeup onto the face with latency short enough that the projection stays locked during motion. - Try it: Track a classmate's face with MediaPipe Face Mesh, project painted patterns onto it (or make it a filter in Lens Studio), and measure how far it drifts when the head turns quickly. Twist: make the makeup change with speech, facial expressions or musical rhythm. #### AR Poser — Disney Research — Ivan Poupyrev, Robert Sumner & colleagues (2018) - Video: https://www.youtube.com/watch?v=Ug-bzFEN8H8 - Interaction: Hands & Body, Play - Platform & tech: Phone, pose estimation - Idea: When you take a photo, a virtual character that copies your pose appears next to you. - What it is: A phone app detects your pose in a photo and inserts a Disney-style digital avatar that imitates it, posing alongside you. - Technique: 2D pose estimation extracts the user's skeleton from the photo, and the keypoints are retargeted onto a rigged character that is rendered and composited next to the user in a mirrored or complementary pose. - Try it: Read body keypoints from a group photo with MediaPipe Pose and map them onto a simple rigged character in three.js (or a 2D paper doll) so it strikes the same or a mirrored pose. Twist: have the character always do the opposite, as a prankster. ### GoSpooky *Social and AR agency (Amsterdam / New York)* Agency founded in Amsterdam that grew out of Snapchat and Instagram AR lenses and now also runs social campaigns; its in-house R&D team publishes AR prototypes on Snap Spectacles, Quest 3 and Google's Geospatial API. #### Fashion Twin — GoSpooky (2022) - Video: https://x.com/GospookyHQ/status/1593200088758640640 - Interaction: Hands & Body, Face - Platform & tech: Phone, Lens Studio, Garment Transfer, machine learning - Idea: Borrow someone else's outfit with your camera. - What it is: Point the camera at someone and 'steal' their outfit: their clothing style is transferred onto you in real time, creating an instant fashion twin. - Technique: A demo of Lens Studio 4.34's ML Garment Transfer component, which segments the upper-body garment in one image and re-renders its texture onto the tracked body. - Try it: Design a 'swap' filter with Lens Studio or Effect House body segmentation: take a photo of a classmate's shirt pattern and apply it to your own silhouette. Twist: swap only the colour, not the pattern. #### AR Lightning — GoSpooky (2023) - Video: https://x.com/GospookyHQ/status/1729877660581736694 - Interaction: Hands & Body, Spatial Mapping, Perception & Effects - Platform & tech: Phone, Lens Studio, hand tracking, LiDAR - Idea: Your hand becomes a lightning rod that strikes the real room. - What it is: Raise your hand and lightning shoots from it into the room, zapping whatever real surface you are looking at, from a fireplace to a wall. - Technique: A Snap Lens combines hand tracking, the gaze or camera direction and a LiDAR-generated world mesh, raycasting to find the hit point where the particle lightning bolt ends. - Try it: Prototype a 'magic beam' in Lens Studio or AR Foundation: raycast from the camera to the scene mesh and draw a jagged line from the hand to the hit point. Twist: scorch marks stay where the beam hits. #### City-Scale AR Bowling — GoSpooky (2023) - Video: https://x.com/GospookyHQ/status/1691788241249161658 - Interaction: Location & City, Play - Platform & tech: Phone, Google Geospatial API, ARCore, Unity - Idea: Turn a whole city block into a bowling lane. - What it is: Giant bowling pins stand on famous buildings such as the Sacré-Cœur in Paris, and a phone player rolls a huge ball down the street to knock them over. - Technique: Google's Geospatial API localises the phone with GPS plus Street View-based visual positioning, and ARCore in Unity simulates physics for pins anchored to the real buildings. - Try it: Use Geospatial anchors (ARCore or 8th Wall) to place one giant object on a campus landmark, then add one physics interaction with it. Twist: the object reacts differently at night. #### Flower Petal Controller — GoSpooky (2023) - Video: https://x.com/GospookyHQ/status/1642900586805731328 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, Lens Studio, hand tracking, VFX particles - Idea: Conduct a cloud of petals with your hand. - What it is: Hand gestures summon and steer a swirling cloud of flower petals through a living room, which scatter and gather as the fingers open and close. - Technique: A Snap Lens uses hand tracking and gesture detection to drive forces in Lens Studio's GPU VFX particle system. - Try it: Build a gesture-driven particle swarm with MediaPipe hands and three.js: open palm pushes particles away, a fist pulls them in. Twist: each gesture changes the particle colour. #### Furniture Portal — GoSpooky (2023) - Video: https://x.com/GospookyHQ/status/1663545539449978882 - Interaction: Portals & Worlds, Hands & Body, Spatial Mapping - Platform & tech: Phone, Lens Studio, physics, VFX - Idea: Open a portal and let copies of your furniture spill out. - What it is: A glowing portal opens outdoors when you gesture with hand or foot, and copies of household items (armchairs, chairs, lamps) tumble out and pile up with physics. - Technique: A Snap Lens duplicates 3D household objects (likely pre-scanned) and runs rigid-body physics with shadows and a VFX layer, triggered by hand and foot tracking. - Try it: Scan one object in class with a phone, then in an AR scene make a portal that spawns twenty physics copies of it on tap. Twist: each copy is slightly smaller than the last. #### Ghost the Most — GoSpooky (2024) - Video: https://x.com/GospookyHQ/status/1849095337610215550 - Interaction: Play, Spatial Mapping - Platform & tech: Headset, Snap Spectacles, Lens Studio - Idea: Hide treasures in a layer of fake fog that covers the real floor. - What it is: A Snap Spectacles prototype fills the office with a knee-high haunted mist between bare trees; players wander through it with see-through glasses to find and collect treasures hidden in the fog. - Technique: A Spectacles Lens anchors a volumetric-looking mist layer and props to the tracked floor; collectibles are revealed by walking close (likely distance triggers). - Try it: Build a fog treasure hunt in Lens Studio or AR Foundation: cover the floor with a semi-transparent fog plane and hide three objects that fade in only within one metre. Twist: the fog thins wherever the player has already walked. #### Mixed Reality Snowball Throwing — GoSpooky (2024) - Video: https://x.com/GospookyHQ/status/1755946491884044623 - Interaction: Spatial Mapping, Play - Platform & tech: Headset, Meta Quest 3, scene mesh, Unity physics - Idea: Use the room's own mesh as the playground for a snowball fight. - What it is: On Quest 3 in passthrough, snowballs thrown with the controllers splat against a real bed, walls and floor, bouncing and breaking apart exactly on the room's surfaces. - Technique: Quest 3 world meshing builds a detailed scene mesh that is used as a physics collider, so thrown snowballs collide and leave splats on real furniture. - Try it: Use a LiDAR phone or Quest scene mesh to make a throw-and-splat game: every hit leaves a paint mark that stays on the real wall. Twist: marks slowly melt away after a minute. ### Hart Woolery (2020CV) *Founder and CTO, 2020CV* Computer-vision developer whose company 2020CV trained its own mobile hand, face and pet tracking models to make InstaSaber, YoPuppet, SpeakingPuppy and Snap lenses. #### InstaSaber — Hart Woolery (2020CV) (2017) - Video: https://www.youtube.com/watch?v=MWd7shj59PA - Interaction: Tangible Objects, Hands & Body, Play - Platform & tech: Phone, ARKit, CoreML, computer vision - Idea: A rolled-up sheet of paper becomes a lightsaber on your phone. - What it is: Roll up a piece of paper and the iPhone turns it into a glowing lightsaber that follows the paper tube in your hand. - Technique: A custom vision model detects the paper tube's endpoints each frame and a blade mesh with glow is rendered along that line. - Try it: Use color tracking to detect a colored straw or paper tube and draw a glowing blade over the camera image (p5.js or OpenCV.js). Twist: two blades throw sparks and make a sound when they clash. #### Mobile hand tracking + ARKit — Hart Woolery (2020CV) (2017) - Video: https://www.youtube.com/watch?v=u_FTew8oTBw - Interaction: Hands & Body - Platform & tech: Phone, ARKit, CoreML - Idea: Bare-hand interaction with AR objects on an iPhone 7, back in 2017. - What it is: An early demo on an iPhone 7 combines 2020CV's own hand tracker with ARKit so a bare hand can interact with AR content. - Technique: A custom neural hand detector runs on-device alongside ARKit tracking to map the hand position into the AR scene. - Try it: Train three gestures (open hand, fist, pointing) with Teachable Machine and use them in web AR to trigger appear, grab and launch. Twist: combine gesture sequences into "spells." #### 3D hand interaction and physics on an iPhone — Hart Woolery (2020CV) (2018) - Video: https://www.youtube.com/watch?v=c_h6UBq0u70 - Interaction: Hands & Body, Spatial Mapping, Play - Platform & tech: Phone, ARKit, CoreML, TensorFlow - Idea: With just a phone camera, knock over virtual blocks on a table with your real hand. - What it is: With only the built-in camera, a tracked 3D hand pushes and knocks over virtual objects that obey physics on a real table. - Technique: A Keras-trained model estimates 3D hand pose, and colliders attached to the hand interact with physics bodies placed on ARKit planes. - Try it: Use Lens Studio hand tracking or AR Foundation with hand colliders so your hand can knock over a row of virtual dominoes. Twist: you can only 'poke' with your index finger raised. #### SpeakingPuppy — Hart Woolery (2020CV) (2018) - Video: https://www.youtube.com/watch?v=3Dzn2WbHsyA - Interaction: Face, Voice & Sound - Platform & tech: Phone, CoreML, face tracking, voice changer - Idea: Put a mask on your dog and make it talk. - What it is: An iOS app detects your dog's face, puts masks and filters on it and gives it a voice-changed talking mouth. - Technique: A custom pet-face landmark model tracks the dog's face so masks follow it, and the user's recorded voice is pitch-shifted and lip-synced to an animated mouth. - Try it: Use the Lens Studio pet template or Teachable Machine to recognize the face of a pet or stuffed toy and give it a voice-changed narration. Twist: the audience types what it says. #### Virtual Drumpad on iOS — Hart Woolery (2020CV) (2019) - Video: https://www.youtube.com/watch?v=lrl2_67GukM - Interaction: Hands & Body, Voice & Sound, Performance - Platform & tech: Phone, CoreML, hand tracking - Idea: Tap on thin air and the phone plays a drum beat. - What it is: Tapping fingers in the air in front of the phone triggers drum pads that appear over the camera view. - Technique: Fingertip tracking detects downward strikes into on-screen pad regions and triggers low-latency audio samples. - Try it: Build air drums with MediaPipe Hands and Web Audio, where a fingertip entering a square plays a sound. Twist: lay out the drum pads along the edge of the real table. #### YoPuppet — Hart Woolery (2020CV) (2019) - Video: https://www.youtube.com/watch?v=zd-LKYu5QDQ - Interaction: Hands & Body, Voice & Sound, Play - Platform & tech: Phone, CoreML, ARKit, TensorFlow - Idea: Turn your hand into an AR puppet: open and close your hand and the puppet talks. - What it is: Hand tracking maps 22 points of your hand onto a cartoon puppet's face, so opening and closing your hand works the puppet's mouth in AR. - Technique: A deep-learning hand pose model tracks keypoints in real time and retargets them to a 2D/3D puppet rig anchored over the hand. - Try it: Build a hand puppet with MediaPipe Hands, where the thumb and fingers opening and closing control a character's mouth, and add voice changing. Twist: characters on both hands can talk to each other. #### Face filter from a single photo — Hart Woolery (2020CV) (2020) - Video: https://www.youtube.com/watch?v=aYKGafPtKbM - Interaction: Face, Perception & Effects - Platform & tech: Phone, face tracking, CoreML - Idea: From just one photo, turn someone else's face into your live filter. - What it is: A face filter is generated from one photo of another person and mapped onto the user's live face so it follows expressions. - Technique: Landmarks from a single photo are used to build a textured face mesh that is warped onto the live tracked face (likely a 2D morphable mapping). - Try it: Use MediaPipe FaceMesh to turn a photo of a celebrity or a famous painting into a mask that fits your own face. Twist: the mask appears only when you smile. ### Ishikawa Watanabe Laboratory (University of Tokyo) *Lab of Masatoshi Ishikawa with Yoshihiro Watanabe, pioneers of 1,000 fps vision and high-speed projection* Masatoshi Ishikawa's lab builds high-speed vision chips and the DynaFlash 1,000 fps projector, enabling dynamic projection mapping that sticks to moving, bending and deforming surfaces with no visible lag. #### DynaFlash — Ishikawa Watanabe Laboratory (University of Tokyo) (2015) - Video: https://www.youtube.com/watch?v=L8kjdObjZpY - Interaction: Projection, Tangible Objects, Perception & Effects - Platform & tech: Projection, DynaFlash, high-speed vision - Idea: A thousand-frame projector keeps the image stuck to the paper waving in your hand. - What it is: A 1,000 fps, 3 ms latency projector keeps imagery locked to a sheet of paper as it is waved, bent and moved quickly in the hand. - Technique: A 1,000 fps projector using a DLP chip with binary frames, paired with high-speed vision tracking of the paper's markers, closes the loop in about 3 ms so the projected image stays locked to a moving, bending surface. - Try it: Use an ordinary projector and a camera to track a moving sheet of paper (with corner markers or color detection), project an image onto it in real time, and measure and record the misalignment as it is waved. Twist: use the latency itself to design a "motion trail" aesthetic. #### Dynamic Projection Mapping onto Deforming Non-Rigid Surfaces — Ishikawa Watanabe Laboratory (University of Tokyo) (2016) - Video: https://www.youtube.com/watch?v=-bh1MHuA5jU - Interaction: Projection, Tangible Objects, Perception & Effects - Platform & tech: Projection, high-speed camera, DynaFlash - Idea: However the cloth is crumpled or folded, the projected pattern bends with it as if printed on. - What it is: Images are projected onto cloth and paper as they fold and stretch, tracked at high speed so textures appear printed on the moving material. - Technique: Invisible IR ink markers on the deformable surface are tracked by a high-speed camera, the surface mesh is reconstructed from the dot grid each frame, and the texture is warped onto it and projected by a high-speed projector. - Try it: Draw a dot grid on a piece of cloth or a T-shirt, detect the grid points with a camera (OpenCV blob detection), warp a pattern to the grid, and project it back so it bends with the folds of the fabric. Twist: project a "moving print," such as swimming fish. #### DynaFlash v2 and Post Reality — Ishikawa Watanabe Laboratory (University of Tokyo) (2018) - Video: https://www.youtube.com/watch?v=QDppJ9NWtaE - Interaction: Projection, Perception & Effects - Platform & tech: Projection, DynaFlash v2 - Idea: High-speed projection makes objects look as if their material has been rewritten, so reality itself becomes the screen. - What it is: The second-generation high-speed projector demonstrates 'post reality' effects: textures, wireframes and illusions projected onto fast-moving objects and people. - Technique: A high-speed projector with 8-bit grayscale at high frame rates, combined with millisecond vision-based tracking, projects textures and wireframes onto fast-moving objects without perceptible lag, making the augmentation appear physical. - Try it: Track a handheld white object (cardboard or a ball) in TouchDesigner, project a wireframe or texture onto it, and reduce latency by shrinking the projection area and predicting motion. Twist: make the projection reveal the invisible structure 'inside' the object. #### MIDAS Projection — Ishikawa Watanabe Laboratory (University of Tokyo) (2018) - Video: https://www.youtube.com/watch?v=c40cxE-dfPg - Interaction: Projection, Perception & Effects - Platform & tech: Projection, high-speed depth sensing, DynaFlash - Idea: Without markers or models, make a moving object look like it is made of a different material. - What it is: Markerless, model-less dynamic projection mapping that makes objects appear to be made of a different material (metal, wood) while they move. - Technique: High-speed depth sensing measures the moving object's surface normals each frame, and a material shading model (e.g. metal reflections, wood grain) is computed per frame and projected so the object appears to be made of that material. - Try it: Estimate surface normals with a depth camera or a known shape (a white cube), render a metal or wood material in TouchDesigner and project it back onto the object, then rotate the object to see whether the highlights follow. Twist: make the object look like 'liquid' or 'fur'. #### VarioLight — Ishikawa Watanabe Laboratory (University of Tokyo) (2018) - Video: https://www.youtube.com/watch?v=XEseo-orRDI - Interaction: Projection, Performance, Hands & Body - Platform & tech: Projection, galvanometer mirrors, high-speed vision - Idea: High-speed steering mirrors chase a dancer across the stage, landing images precisely on a moving body. - What it is: A mirror-steered high-speed projection system tracks a dancer across a whole stage and projects imagery onto them with no drift. - Technique: Two galvanometer mirrors steer the projection beam at high speed, driven by a high-speed vision tracker, so a small projection field follows a moving performer across a large stage. - Try it: Mount a projector on a servo pan-tilt head (or just move a projected spot within the frame), track a running classmate with a camera, and keep a pattern on them at all times. Twist: the dancer's movement switches the spot's target (raising a hand sends it to another person). #### ElaMorph Projection — Ishikawa Watanabe Laboratory (University of Tokyo) (2020) - Video: https://www.youtube.com/watch?v=uWq-a52X-7g - Interaction: Projection, Perception & Effects - Platform & tech: Projection, dynamic projection mapping - Idea: Projection alone makes a hard object look as stretchy as rubber. - What it is: Projection alone makes a rigid 3D object appear to stretch and deform elastically as it moves, exploiting high-speed tracking and perception. - Technique: The rigid object's motion is tracked at high speed and projected shading and texture are warped according to a simulated elastic deformation driven by that motion, exploiting perception so the object looks as if it stretches. - Try it: Track a hand-held white paper box and give the pattern projected onto it a stretch-and-rebound deformation driven by acceleration, for example with a spring model, so viewers feel the box is soft. Twist: make a short video of a hard object turning soft and survey whether viewers are actually fooled. #### VarioLight 2: Rhythmic Gymnastics — Ishikawa Watanabe Laboratory (University of Tokyo) (2021) - Video: https://www.youtube.com/watch?v=9X66YZTb_hA - Interaction: Projection, Performance, Hands & Body - Platform & tech: Projection, high-speed vision, galvanometer mirrors - Idea: Projection chases a gymnast's ball and ribbon and colors them during the routine. - What it is: Wide-range dynamic projection follows a rhythmic gymnast's ball and ribbon, painting light effects onto the apparatus during the routine. - Technique: High-speed vision tracks the gymnastics ball and ribbon, and galvanometer mirrors steer a projector to keep light effects registered on the fast-moving apparatus throughout the routine. - Try it: Use camera color tracking on a colored ball being tossed and caught, make light effects and particles follow its path on a projector or screen, and perform a 30-second routine to music. Twist: the ball's speed or height drives changes in the music. ### Jamie Gledhill *Audio-visual artist and interaction designer* East Anglia artist and lecturer at Norwich University of the Arts whose installations use Kinect, Leap Motion and ordinary webcams for contact-free play, from body-outline projections to an interactive sculpture takeover and an AR trail of talking trees. He was lead artist of The Multitude, Collusion's two-player body-projection game. #### Mirror Noise — Jamie Gledhill (2013) - Video: https://vimeo.com/70233378 - Interaction: Hands & Body, Voice & Sound, Perception & Effects - Platform & tech: Projection, webcams, edge detection, motion detection, microphone - Idea: A mirror that answers your movement and voice with five different moods. - What it is: Five two-minute interactive portrait studies across three screens: your outline becomes neon shapes, sparks shower where you move, recent frames scramble into a time warp, a clap or shout distorts your image, and streams of data flow around you. - Technique: Three webcams feed edge and movement detection, with a microphone driving the distortion study and a frame buffer shuffling recent frames for the time warp. - Try it: Build a webcam mirror that swaps between three effects every minute: edge neon, motion sparks and a sound-triggered stretch. Twist: let a loud clap skip to the next effect. #### Electricus — Jamie Gledhill (2014) - Video: https://vimeo.com/87755772 - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, camera tracking, projection, generative sound - Idea: Your movement charges your own body with electricity on screen. - What it is: Visitors see their outline surrounded by crackling electrical energy on a large screen; the bigger they move, the more intense the electricity looks and sounds. - Technique: Camera-based motion detection measures the amount of movement around each silhouette and scales a lightning-style effect and sound to it. - Try it: In TouchDesigner or p5.js, use frame differencing to measure motion and draw jagged lines from the silhouette edge whose length follows that motion. Twist: the electricity jumps between two people when they almost touch. #### Luminescence — Jamie Gledhill (2014) - Video: https://vimeo.com/93631167 - Interaction: Hands & Body, Shared & Social, Projection - Platform & tech: Projection, Kinect, back projection, generative sound - Idea: Energy threads connect everyone who steps into the frame. - What it is: When someone steps into the active zone in front of a large back-projected screen, an outline is drawn around them with swirling manga-like marks, and a fluxing energy thread links every person and object in the zone, building the soundtrack as more threads appear. - Technique: A Kinect infrared camera segments people and objects within a set depth range, and software draws outlines, motion-driven marks and threads between the detected blobs while their count drives the sound. - Try it: Use a webcam and background subtraction in p5.js or TouchDesigner to find each person's blob and draw a line between every pair. Twist: lines snap when two people move too far apart. #### Play Table — Jamie Gledhill (2015) - Video: https://vimeo.com/142236663 - Interaction: Shared & Social, Play, Projection - Platform & tech: Projection, overhead projection, camera tracking, physics simulation, surround sound - Idea: A table where strangers play with the same projected objects from all sides. - What it is: A multi-player audiovisual table: an interactive image is projected from above onto a large tabletop, and people standing around its edge push virtual objects that collide and make sound in surround, sometimes cooperating and sometimes competing. - Technique: An overhead camera or depth sensor tracks hands at the table edge, and a physics simulation projected from above turns those contacts into collisions and spatialised sound. - Try it: Point a projector and webcam down at a table and build a p5.js physics sketch where hand shapes push balls that play notes when they hit each other. Twist: the balls only move when two hands from opposite sides touch them. #### Talking Trees of Chalkwell Park — Jamie Gledhill (2015) - Video: https://vimeo.com/149390420 - Interaction: Location & City, Voice & Sound, Information & UI - Platform & tech: Phone, mobile AR, location triggers (likely), character animation - Idea: The park's trees become storytellers with a voice and a personality. - What it is: An augmented reality trail in Southend's Chalkwell Park, commissioned by Metal: visitors find particular trees and, through their phones, meet them as characters who share the park's social history along with some frank opinions. - Technique: A mobile AR app likely uses location or image triggers at each tree to overlay an animated character and play its recorded voice. - Try it: Pick three trees on campus, write a short monologue for each from the tree's point of view, and anchor an animated face and audio to each with Lens Studio or an AR Foundation image marker. Twist: the trees disagree with each other about the same event. #### Portals for Mortals — Jamie Gledhill (2016) - Video: https://vimeo.com/180005726 - Interaction: Shared & Social, Voice & Sound, Location & City - Platform & tech: Projection, presence sensors, audio playback, public sculpture - Idea: Walk through all four doorways together to play the whole fanfare. - What it is: An interactive takeover of a public sculpture with four doorways in Peterborough's Nene Park: walking through one doorway plays one part of a fanfare recorded by locals, and only passing through all four, usually as a group, plays the full fanfare and unlocks the next one. - Technique: Sensors in each doorway detect a person passing and trigger that doorway's part of a four-part recording; a controller checks when all four have fired to advance to the next fanfare. - Try it: Tape four cheap motion sensors to four classroom doorways or chair gaps, each triggering one layer of a song, and make the full song play only when all four fire within five seconds. Twist: the order of passing changes the arrangement. ### Jun Nishida *Assistant Professor, University of Maryland; director of the Embodied Dynamics Lab* Designs wearables that let people borrow each other's bodies: EMS suits that share muscle activity, exoskeletons that shrink your hands, and head-mounted rigs that give you a child's eye height. Previously at the University of Tsukuba, Sony CSL and Pedro Lopes's lab in Chicago. #### CHILDHOOD: Egocentric Smaller-person Experience — Jun Nishida (2015) - Video: https://vimeo.com/120369920 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Headset, Wearable, video see-through HMD, stereo camera, passive exoskeleton - Idea: Lower the camera and shrink the hands, and an adult can briefly live inside a child's body. - What it is: An adult wears a head-mounted display fed by a camera at child eye height plus passive hand exoskeletons, and suddenly sees furniture, doors and people from a small child's point of view while grasping with small hands. - Technique: Streams a waist-mounted stereo camera to a video see-through HMD so eye height drops by about half, while linkage exoskeletons map the wearer's finger motion onto smaller hands. - Try it: Strap a phone at waist height, stream its camera to a second phone in a cardboard viewer, and walk around the classroom for ten minutes; list what feels dangerous, huge or unreachable. Twist: mount the camera at a dog's or a giant's height and redesign one object for that body. #### bioSync — Jun Nishida (2017) - Video: https://vimeo.com/155394019 - Interaction: Hands & Body, Shared & Social - Platform & tech: Wearable, EMG, EMS, wearable electrodes - Idea: Share muscle tension between two bodies as if it were a live video stream. - What it is: Two people wear paired EMG/EMS armbands: when one tenses or relaxes a muscle, the other person's arm stiffens or loosens in sync, so a teacher can literally pass on a movement or a tremor. - Technique: Measures muscle activity with electromyography on one wearer and replays it as electrical muscle stimulation on the partner through the same electrodes, in both directions. - Try it: Pair students and have one guide the other's wrist with a light string and a phone vibration cue whenever the leader's phone accelerometer detects a flick; practise a drum pattern this way. Twist: swap roles mid-rhythm without stopping and see who is really leading. #### HandMorph — Jun Nishida, Pedro Lopes (2020) - Video: https://www.youtube.com/watch?v=5o2wPy5hl0w - Interaction: Hands & Body, Tangible Objects, Perception & Effects - Platform & tech: Wearable, passive exoskeleton, linkage mechanism - Idea: Change your body, not the picture: a mechanical small hand makes the real world feel oversized. - What it is: A passive exoskeleton puts a child-sized mechanical hand in front of the wearer's own; grasping real toys and tools through it makes everything feel bigger, with no headset involved. - Technique: Uses four-bar linkages to transmit each finger's motion and force to an anatomically scaled smaller hand, so haptic feedback stays congruent in the real environment. - Try it: Build a cardboard 'hand extender' with straws and string that mirrors finger bends onto a smaller cardboard hand, then try picking up objects of different sizes and rate how big they feel. Twist: make the hand larger instead and redesign a door handle for it. #### DigituSync — Jun Nishida, Pedro Lopes (2022) - Video: https://www.youtube.com/watch?v=XpOKhXOXnt8 - Interaction: Hands & Body, Shared & Social - Platform & tech: Wearable, passive exoskeleton glove, linkage mechanism - Idea: Physically couple two hands so skills transfer by touch rather than by watching. - What it is: Two people wear gloves linked by adjustable mechanical bars, so when a piano teacher curls a finger the learner's finger curls too, and both still feel the real keys. - Technique: Uses passive four-bar linkages with a variable-length link to transmit finger motion and force between users and tune how strongly the leader intervenes. - Try it: Tie two students' index fingers together with elastic bands and a ruler splint, then play a simple phone piano app together; compare accuracy with watching a video tutorial. Twist: loosen the band halfway through to fade out the teacher. #### Electrical Head Actuation — Jun Nishida, Pedro Lopes (2022) - Video: https://www.youtube.com/watch?v=vqpH9gNGpts - Interaction: Hands & Body, Gaze & Attention - Platform & tech: Wearable, Headset, EMS, neck muscle stimulation, mixed reality - Idea: Instead of drawing an arrow, the interface turns your head for you. - What it is: Electrodes on the neck gently turn the wearer's head left, right, up or down, so a system can direct attention toward something in mixed reality without any arrow on screen. - Technique: Applies electrical muscle stimulation to specific neck muscles to actuate head yaw and pitch, calibrated per user for direction and strength. - Try it: Design a phone AR scene with a hidden object and compare three guidance cues: an arrow, spatial audio, and a partner gently tapping one shoulder; time how fast people find it. Twist: try the tap cue with eyes closed and describe how it feels to be steered. #### DexteriSync — Jun Nishida (2024) - Video: https://www.youtube.com/watch?v=3Jn3m8z0vac - Interaction: Hands & Body, Perception & Effects - Platform & tech: Wearable, thermal exoskeleton, water cooling - Idea: Temperature is a dial for dexterity: cool the fingers to borrow someone else's hands. - What it is: A water-cooled exoskeleton chills or warms the fingers so the wearer's dexterity drops or recovers on demand, letting a designer feel what fine tasks are like for an older person or a cold hand. - Technique: Circulates temperature-controlled water through silicone-copper tubes along each finger to change skin temperature and thus finger dexterity and thermal perception. - Try it: Have students button a shirt and type on a phone after holding an ice pack for one minute, then redesign one app screen for cold, clumsy fingers. Twist: add gloves of different thickness and chart the error rate. #### WireDrum — Jun Nishida (2026) - Video: https://www.youtube.com/watch?v=i0oJNk8W3DU - Interaction: Hands & Body, Voice & Sound, Shared & Social - Platform & tech: Wearable, EMS, EMG, biosignal sensing - Idea: Split a hard skill across several people and merge it inside one body. - What it is: Several instructors' muscle signals are routed into one visitor's arms through EMS, so a novice can play a demanding drum rhythm by sharing their body with other people. - Technique: Combines custom biosignal sensing modules, a multi-channel electrical muscle stimulator, a visual-tactile tempo indicator and software that maps instructors' muscles onto the learner's. - Try it: Split a two-hand drum pattern among three students: two tap cues on the drummer's shoulders while a phone metronome flashes, and the drummer only follows touches. Twist: record it with a phone and overlay each person's contribution in a different colour. ### Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) *Theatre company (multimedia and holographic stage)* Founded in 1983 by Michel Lemieux and Victor Pilon, 4D Art makes plays in which live actors share the stage with life-size virtual performers projected on transparent screens. Their productions include Anima, La Tempête, NORMAN and La Belle et la Bête, the Cirque du Soleil show Delirium, and the Cité Mémoire projections in Old Montreal. #### Grand Hôtel des Étrangers — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (1994) - Video: https://www.youtube.com/watch?v=JJF4dC69Kg8 - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, video projection, transparent screen, theatre - Idea: A single room shared by one real body and several projected ones is the basic unit of mixed-reality drama. - What it is: In a cramped hotel room a lone man, played by a live actor, meets virtual characters projected around him, and memories, dreams and the present blur together. It is one of 4D Art's first full plays built on the meeting of real and projected performers. - Technique: Pre-filmed actors are projected at life size onto transparent or gauze screens placed in the set, and the live actor times his movements to the recorded ones (likely rehearsed to a fixed video track). - Try it: Film a classmate saying five lines, then place the clip as a life-size, background-removed video in AR (Lens Studio or AR Foundation) in a real corner, and perform a live dialogue with it. Twist: the recorded partner sometimes answers before the live actor speaks. #### Anima — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (2002) - Video: https://www.youtube.com/watch?v=MMFtNe2Er_k - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, video projection, transparent screen, theatre - Idea: The clash between a real body and its projected double makes the audience feel the gap between body and soul; AR doubles can carry meaning, not just spectacle. - What it is: A touring company of performers is haunted by memories: on the road and in hotel rooms they illustrate excerpts of a BBC interview with anthropologist Desmond Morris, while projected bodies appear, dissolve and multiply beside the dancers. - Technique: Life-size projections on nearly invisible screens are placed downstage of the performers so projected and real bodies appear in the same depth plane. - Try it: Record a dancer's short phrase and place three delayed AR copies of it next to the live dancer with body tracking in Lens Studio or a WebXR app. Twist: each copy is delayed a little more, so the phrase becomes a canon in space. #### La Tempête (The Tempest) — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (2005) - Video: https://www.youtube.com/watch?v=rsiOM3-Oaf8 - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, holographic projection, Pepper's ghost, theatre - Idea: Magic in a story is the natural home for mixed reality: when a plot already contains spirits, virtual characters feel motivated rather than decorative. - What it is: Shakespeare's play of spirits and illusions is staged with Prospero's magic made visible: Ariel and other spirits appear as floating, translucent projections that fly, vanish and transform around the live actors. - Technique: Projections onto large transparent screens at the front of the stage create hologram-like figures (a Pepper's-ghost-style effect) that overlap with actors standing behind them. - Try it: Stage a two-minute scene from a play where one character is a spirit, played only as an AR character (WebXR or HoloKit) that the live actor can see through the headset or phone. Twist: the audience sees the spirit only through a single shared phone passed around. #### NORMAN — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (2007) - Video: https://www.youtube.com/watch?v=62OwWsqt59g - Interaction: Performance, Hands & Body, Projection - Platform & tech: Projection, video projection, animation, dance - Idea: Hand-drawn animation next to a real dancer shows that AR does not have to be photoreal; a line that reacts to a body is already magic. - What it is: A tribute to animator Norman McLaren, made with the National Film Board of Canada: dancer Peter Trosztmer performs with drawn lines, animated figures and multiplied images of himself projected around him, as if he had stepped into a McLaren film. - Technique: Pre-rendered and likely partly cued animations are projected on transparent screens and the back wall, choreographed to the dancer's positions on stage. - Try it: Draw a short loop in a hand-drawn style (Procreate or paper, scanned) and attach it to a person's hand with hand tracking in Lens Studio or MediaPipe. Twist: the drawing is left behind in space like a trail whenever the hand stops. #### La Belle et la Bête — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (2011) - Video: https://www.youtube.com/watch?v=qExGC7Lpc7g - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, holographic projection, transparent screen, theatre - Idea: Transformation is the most powerful AR moment: showing a body change in place, right next to real people, is more affecting than any new world. - What it is: A contemporary retelling of Beauty and the Beast, co-produced with Théâtre du Nouveau Monde, where the Beast's enchanted castle, spirits and transformations appear as projected apparitions around the actors. It toured to Toronto's Luminato Festival, Boston and Taipei. - Technique: Large-scale projections on transparent scrims create volumetric-looking illusions combined with live actors, lighting and film sequences. - Try it: Build a phone AR 'transformation mirror': use body or face tracking so a user's outline slowly morphs into an animal silhouette over 30 seconds. Twist: the transformation reverses only when a second person steps into the frame. #### Icare (Icarus) — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (2014) - Video: https://www.youtube.com/watch?v=loExb1r9ayw - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, holographic projection, dance, aerial - Idea: Flight is easy to fake when the surroundings move instead of the body; AR can make people feel they fly by moving the world around them. - What it is: The myth of Icarus, the boy who flew too close to the sun, is told with dancers, aerial movement and projections that fill the space around them, so the performers appear to soar through clouds and light. - Technique: Projected environments on transparent screens in front of and behind performers, combined with suspension rigs, likely create the layered flying effect. - Try it: Make a WebXR scene where clouds rush past the user when they spread their arms (tracked by the phone's front camera or a headset controller). Twist: the higher the arms go, the brighter and hotter the scene becomes, until the wings melt. #### Cité Mémoire — Lemieux Pilon 4D Art (Michel Lemieux & Victor Pilon) (2016) - Video: https://www.youtube.com/watch?v=B_OmCLGKAWY - Interaction: Projection, Location & City, Information & UI - Platform & tech: Projection, Phone, architectural projection, mobile app, audio guide - Idea: Turning the city into a walkable anthology of site-specific scenes, with the phone carrying the sound, is a blueprint for location-based AR storytelling. - What it is: A city-wide trail of about twenty projected tableaux on walls, trees and even the ground of Old Montreal, telling stories from the city's history; visitors listen to each scene through a free mobile app. The video shows the large historical tableau on the courthouse on the Champ-de-Mars. - Technique: Permanent outdoor projectors trigger scenes on a schedule, while a geolocated app synchronises dialogue and music to what is on each wall (sync method likely timecode over the network). Created by Michel Lemieux and Victor Pilon with Michel Marc Bouchard, produced by Montréal en Histoires. - Try it: Choose three spots around campus and create a short geolocated AR scene for each with 8th Wall or Niantic Lightship, each telling one moment of the campus's history. Twist: each scene shows a different decade, and the walking route is the timeline. ### Magic Leap Studios *In-house content studio of Magic Leap* Produced Magic Leap's showcase spatial experiences - the 2015 gym-whale concept, Create, Dodge, Mica and Undersea - to show content that respects real room geometry. #### Whale in the Gym (concept) — Magic Leap Studios, Greg Broadmore (Weta Workshop / Weta Gameshop) (2015) - Video: https://www.youtube.com/watch?v=LM0T6hLH15k - Interaction: Perception & Effects, Portals & Worlds, Shared & Social - Platform & tech: Headset, concept film, VFX - Idea: A whale leaps out of a school gym floor. - What it is: A humpback whale bursts out of a school gym floor in front of students and splashes back down, a concept film for shared, room-scale mixed reality. - Technique: Concept visualization (likely VFX compositing rather than live headset capture) of a large creature breaking through a real floor plane. - Try it: Use phone AR (for example Reality Composer) to make a giant animal burst out of the classroom floor, and design an occluded hole effect. Twist: let the whole class see the same animal on several phones at once. #### Dodge (developer sample) — Magic Leap Studios (2018) - Video: https://www.youtube.com/watch?v=bT5LFe-muR8 - Interaction: Spatial Mapping, Hands & Body, Play - Platform & tech: Headset, Magic Leap One, world meshing, gesture input - Idea: Rock monsters burst out of your real floor and throw rocks, and you have to actually dodge. - What it is: Rock golems rise out of your real floor and hurl boulders that you physically dodge while swatting others away with hand gestures. - Technique: Spawns characters on detected floor meshes and uses head position as the target so the player's real body movement is the core mechanic. - Try it: Make a phone AR dodging game where virtual balls fly from a point on the floor toward the phone and players must move their bodies to avoid them. Twist: two players place launch points for each other. #### Magic Leap Create — Magic Leap Studios (2018) - Video: https://www.youtube.com/watch?v=BCL_pjRsV6M - Interaction: Drawing & Making, Spatial Mapping, Play - Platform & tech: Headset, Magic Leap One, world meshing, physics - Idea: Set dinosaurs, knights and dominoes loose in your living room, where they interact with real furniture. - What it is: A mixed-reality toy box: place dinosaurs, knights, dominoes and paint strokes in your room, and they fight, topple and bounce off real furniture. - Technique: Uses the headset's world mesh as a physics collider so user-placed toys and strokes land on, bounce off and hide behind real surfaces. - Try it: Use phone AR plane detection plus physics (Reality Composer or Unity AR Foundation) to build a virtual domino run that spans a table and the floor. Twist: it must use a real object as part of the contraption. #### Mica — Magic Leap Studios (2018) - Video: https://www.youtube.com/watch?v=XNb42Lw0lBU - Interaction: Gaze & Attention, Face, Shared & Social - Platform & tech: Headset, Magic Leap One, eye tracking, digital human - Idea: A digital human who sits at your real table and meets your gaze. - What it is: A photoreal digital human sits at your real table, makes eye contact, reacts to your gaze and shares small moments without speaking. - Technique: Drives the character's gaze and expressions from the user's tracked eye direction and anchors her to detected table and chair surfaces. - Try it: Place a virtual character with phone AR whose head always turns toward the phone camera (LookAt), so classmates feel what it is like to be watched. Twist: the character moves only when you are not looking at it. #### Undersea — Magic Leap Studios (2019) - Video: https://www.youtube.com/watch?v=-i-FVyd8MoM - Interaction: Portals & Worlds, Hands & Body, Perception & Effects - Platform & tech: Headset, Magic Leap One, hand tracking, procedural animation - Idea: Bring a coral reef that responds to your hands into your room. - What it is: A living coral reef appears in your room; fish swim around your furniture and respond as you reach toward them with your hands. - Technique: Places a reef ecosystem on meshed surfaces and uses hand tracking to trigger flocking-avoidance behaviours in the fish. - Try it: Place virtual coral and a school of fish on the floor in phone AR, with the fish swimming away as the phone approaches (distance check). Twist: use the phone flashlight to draw the fish together. ### Mark Billinghurst — HIT Lab NZ & Empathic Computing Lab *Professor at the University of South Australia and University of Auckland; founder of HIT Lab NZ and the Empathic Computing Lab* Mark Billinghurst co-created ARToolKit-based collaborative AR with Hirokazu Kato and the MagicBook (2001), then founded HIT Lab NZ. His Empathic Computing Lab now builds mixed reality systems that share gaze, emotion and viewpoint between remote collaborators. #### CoVAR — Mark Billinghurst — HIT Lab NZ & Empathic Computing Lab (2017) - Video: https://www.youtube.com/watch?v=K_afCWZtExk - Interaction: Shared & Social, Gaze & Attention, Spatial Mapping - Platform & tech: Headset, HoloLens, HTC Vive, 3D reconstruction - Idea: An AR user in the real room and a VR user in its scanned copy work together in the same space. - What it is: A HoloLens user in a real room and a VR user in its 3D scan collaborate: the VR user can shrink, fly and share gaze and gestures with the AR user. - Technique: The room is pre-scanned into a 3D model that the VR user inhabits, and both users are tracked in the same coordinate system so head pose, gaze rays and hand gestures are exchanged and rendered for each other. - Try it: Scan a room with Polycam and import it into three.js as a "VR version," have one classmate browse it in first person on a computer while another uses phone AR on site, and show each person's position and gaze direction to the other over WebSocket. Twist: let the VR user shrink or enlarge themselves. #### SharedSphere — Mark Billinghurst — HIT Lab NZ & Empathic Computing Lab (2017) - Video: https://www.youtube.com/watch?v=q_giuLot76k - Interaction: Shared & Social, Gaze & Attention, Hands & Body - Platform & tech: Headset, 360 camera, HoloLens, Leap Motion - Idea: A remote expert steps into a live 360 view of the site, and their hands and gaze appear back on site in AR. - What it is: A local worker streams a live 360 panorama; the remote expert, immersed in it, sees and points with hands and gaze shown back to the local worker in AR. - Technique: A 360 camera on the local worker streams a live panorama to the remote expert's VR view, and the expert's head gaze and Leap Motion hand gestures are sent back and rendered in the local worker's AR display. - Try it: Stream the scene to a remote classmate with a 360 camera (or a phone panorama livestream), let the remote classmate tap to annotate the view, and have the local classmate see an arrow in that direction in phone AR. Twist: the remote classmate may only use gestures and cannot speak while completing a find-the-object task. #### Mini-Me — Mark Billinghurst — HIT Lab NZ & Empathic Computing Lab (2018) - Video: https://www.youtube.com/watch?v=YrdCg8zz57E - Interaction: Shared & Social, Gaze & Attention - Platform & tech: Headset, HoloLens, VR - Idea: A remote partner becomes a 'mini-me' that always stays in your view, still pointing and looking accurately. - What it is: A remote VR collaborator appears as an adaptive miniature avatar that always stays in view of the AR user, with gaze and pointing redirected to stay accurate. - Technique: The remote VR user's avatar is scaled down and repositioned to stay within the AR user's field of view, while their gaze and pointing directions are retargeted so they still point at the correct objects. - Try it: Place a small avatar in phone AR that always stays visible at the edge of the screen, and when a remote classmate taps an object, the avatar turns to point at it. Twist: give the avatar emotional gestures (nodding, looking puzzled) and test whether they can replace voice. #### Snow Dome — Mark Billinghurst — HIT Lab NZ & Empathic Computing Lab (2018) - Video: https://www.youtube.com/watch?v=s9IQt6PKCFs - Interaction: Shared & Social, Perception & Effects - Platform & tech: Headset, HoloLens, HTC Vive - Idea: A remote collaborator can shrink into a snow globe or grow into a giant looking down on everything. - What it is: The remote VR user can shrink to miniature size inside a snow-globe-like copy of the space or grow into a giant, enabling multi-scale collaboration with the AR user. - Technique: A scaled copy of the reconstructed space is rendered as a miniature (World-in-Miniature) inside a dome, and the VR user's position and scale are mapped between the miniature and full-scale coordinate frames so their avatar appears correctly to the AR user. - Try it: Shrink a scanned room model into a 'snow globe' on the table in AR, and tap a point inside it to 'teleport' your first-person view there. Twist: sync changes made inside the snow globe (placing a piece of furniture) to the full-size AR scene. #### On the Shoulder of the Giant — Mark Billinghurst — HIT Lab NZ & Empathic Computing Lab (2019) - Video: https://www.youtube.com/watch?v=UfAq2ueg_Y4 - Interaction: Shared & Social, Gaze & Attention - Platform & tech: Headset, HoloLens, VR - Idea: Put a distant partner on your shoulder so you work together from the same viewpoint. - What it is: The local AR user carries a miniature remote collaborator on their shoulder, who sees the world from that perspective and guides the work. - Technique: A miniature avatar of the remote collaborator is attached to the local user's tracked head or shoulder pose, and the remote viewpoint is rendered from that attached position so both share nearly the same perspective. - Try it: Pin a small avatar to the top-left of the screen in HoloKit or phone AR (to mimic 'on the shoulder'), and have a remote classmate watch your camera feed and give directions while the avatar points along. Twist: make the shoulder avatar a 'past you' that plays back hints you recorded earlier. #### Superman vs Giant — Mark Billinghurst — HIT Lab NZ & Empathic Computing Lab (2019) - Video: https://www.youtube.com/watch?v=jusLbwjWVcI - Interaction: Shared & Social, Perception & Effects - Platform & tech: Headset, mixed reality, VR - Idea: A remote collaborator can fly through the space like Superman or look down on it as a giant. - What it is: A multi-scale MR telepresence interface where the remote user can fly like Superman through a reconstructed space or tower over it as a giant. - Technique: The remote user can change their viewpoint scale and fly freely through a 3D reconstruction, and their avatar is rendered to the local AR user at the corresponding scale and position within the shared coordinate frame. - Try it: In Unity, give the VR/desktop side two view modes, 'giant' and 'flying', while the AR side sees the partner as a giant head or a tiny flyer, and have both sides complete a placement task together. Twist: compare which scale works best for giving instructions. #### eyemR-Vis — Mark Billinghurst — HIT Lab NZ & Empathic Computing Lab (2021) - Video: https://www.youtube.com/watch?v=D-lZk43zZcM - Interaction: Gaze & Attention, Shared & Social - Platform & tech: Headset, eye tracking, HoloLens 2 - Idea: Show two people each other's gaze so they know where the other is looking and whether they are paying attention. - What it is: Bi-directional gaze cues between remote collaborators, visualized in MR so each person knows where the other is looking and whether they are attending. - Technique: Eye trackers on both sides capture gaze direction, which is exchanged and visualized as shared gaze rays or highlights in MR, with visual states indicating when both people look at the same object. - Try it: Use MediaPipe Face Mesh iris points or WebGazer.js to roughly estimate where two people are looking on screen, show each other's gaze cursor in a shared view, and trigger an effect when both look at the same spot. Twist: use mutual gaze to solve a two-person puzzle. ### Mark Coniglio / Troika Ranch (with Dawn Stoppiello) *Interactive dance-theatre company; Mark Coniglio created the Isadora media software and the MidiDancer sensor suit* Composer-programmer Mark Coniglio and choreographer Dawn Stoppiello founded Troika Ranch in 1994, making works in which sensor suits and live camera tracking let dancers trigger and shape video, sound and light; the tool Coniglio built for this, Isadora, became a standard in live media performance. #### In Plane — Mark Coniglio / Troika Ranch (with Dawn Stoppiello) (1994) - Video: https://vimeo.com/115681934 - Interaction: Hands & Body, Performance, Projection - Platform & tech: Wearable, Projection, MidiDancer, flex sensors, MIDI, LaserDisc - Idea: The dancer's joints are the mixing desk for the whole stage. - What it is: A duet between dancer Dawn Stoppiello and her video double: sensors on her joints let her control the music, recall video images, change the lighting and slide a robotic projector along the stage. - Technique: The MidiDancer suit has eight flex sensors at elbows, wrists, hips and knees that send MIDI wirelessly to a computer controlling sound, LaserDisc video playback and a projector on a track. - Try it: Tape a phone to a forearm, read its accelerometer in a web page, and map arm bend to video playback speed and a light's brightness. Twist: the video plays backwards when the arm moves away from the body. #### Reine Rien — Mark Coniglio / Troika Ranch (with Dawn Stoppiello) (2001) - Video: https://vimeo.com/111764665 - Interaction: Hands & Body, Projection, Voice & Sound - Platform & tech: Projection, Isadora, MidiDancer, video - Idea: Dancing makes weather; stillness makes silence. - What it is: An abstract piece inspired by Midwestern flatlands where dancing brings up projected trees, water and virtual rainfall, and when the dancers stop and make small gestures the images and voices fade away. - Technique: Movement sensing (likely MidiDancer and camera tracking) drives Coniglio's Isadora patches that crossfade landscape video and sound according to the dancers' activity. - Try it: Measure overall motion from a webcam and use it to control the density of projected rain particles and a rain sound. Twist: the smallest gesture makes lightning. #### Future of Memory — Mark Coniglio / Troika Ranch (with Dawn Stoppiello) (2003) - Video: https://vimeo.com/112552170 - Interaction: Performance, Perception & Effects, Projection - Platform & tech: Projection, Isadora, live video, projector - Idea: Live video as memory: what the camera keeps is distorted each time it is replayed. - What it is: A Bessie Award-winning piece in which four characters re-live memories while live video of their bodies is captured, looped, fragmented and replayed on stage, showing how recall rewrites the past. - Technique: Live cameras feed Isadora, where movement sensing triggers recording, looping and processing of the captured footage projected behind the performers. - Try it: Build a TouchDesigner or browser tool that records five seconds of webcam video whenever a person raises both arms and replays it with more glitch each time. Twist: the audience decides which memory gets deleted. #### Surfacing — Mark Coniglio / Troika Ranch (with Dawn Stoppiello) (2004) - Video: https://vimeo.com/111712677 - Interaction: Perception & Effects, Projection, Performance - Platform & tech: Projection, Isadora, live camera, projector - Idea: Your image is confined while your body is free, so which one are you? - What it is: Onstage cameras capture the dancers and trap their images inside a three-storey video screen, where their doubles fall in slow motion while the real bodies move freely below. - Technique: Live camera feeds are processed in Isadora (time-stretching, cropping, scaling) and projected at architectural scale in sync with the choreography. - Try it: Point a webcam at a performer, crop their image into a narrow projected box, and slow it to half speed. Twist: the box shrinks every time the performer leaves the camera's view. #### 16 [R]evolutions — Mark Coniglio / Troika Ranch (with Dawn Stoppiello) (2006) - Video: https://www.youtube.com/watch?v=LJ3_AOBX6TM - Interaction: Hands & Body, Drawing & Making, Performance - Platform & tech: Projection, Isadora, camera tracking, 3D graphics - Idea: Each gesture leaves a living 3D trace that becomes part of the set. - What it is: Four characters caught between animal and intellect are tracked by live cameras, and their movements generate real-time 3D imagery and sound that grow, trail and mutate around them. - Technique: Camera tracking of body points feeds Isadora and custom 3D rendering so dancers interactively manipulate the video and audio, as explained in the company's technology clip. - Try it: Track one wrist with MediaPipe and draw a 3D ribbon in Three.js that follows it and never disappears, projected behind the dancer. Twist: ribbons from two dancers merge when they touch. #### loopdiver — Mark Coniglio / Troika Ranch (with Dawn Stoppiello) (2009) - Video: https://vimeo.com/15062917 - Interaction: Performance, Perception & Effects, Projection - Platform & tech: Projection, Isadora, video looping - Idea: Humans perform an edit made by a computer, glitches and all. - What it is: A six-minute dance was videotaped and cut by software into a 60-minute score of loops, which the dancers then re-perform live with machine precision alongside looping video, sound and light. - Technique: A custom looping tool turned the recorded performance into a timeline of repeated fragments, which became the choreographic score and cued synchronised media. - Try it: Record 20 seconds of improvisation, write a script that cuts it into random short loops, and have the dancer learn and perform the new sequence. Twist: project the loop behind the dancer so the audience can spot every mistake. #### SWARM — Mark Coniglio / Troika Ranch (with Dawn Stoppiello) (2014) - Video: https://vimeo.com/86373749 - Interaction: Shared & Social, Projection, Play - Platform & tech: Projection, Isadora, projection, camera tracking - Idea: The audience is the dance company, and the piece only progresses through collective action. - What it is: An immersive installation-performance where simple projected visual cues call the audience into action, and only when the crowd coordinates and moves together does the piece reveal its full arc. - Technique: Projected instructions and visual stimuli are driven by Isadora, likely with camera sensing of crowd position to decide when the group has completed a task. - Try it: Project three coloured circles on the floor and use an overhead webcam to count people in each; the next scene starts only when every circle holds the same number. Twist: nobody is allowed to speak. ### Mária Júdová *Media artist working across dance, VR and biodata-driven performance* Mária Júdová makes performances and immersive works where the body drives the digital layer: light that follows a dancer's heartbeat and breath in body(input);, a drone played as an instrument, VR pieces built from motion-captured dancers such as DUST, Camouflage and the Kykeon trilogy made with tanzmainz and Motion Bank. #### Composition for a drone — Mária Júdová (2014) - Video: https://vimeo.com/96177923 - Interaction: Voice & Sound, Spatial Mapping, Tangible Objects - Platform & tech: Desktop, AR.Drone, Kinect, vvvv, Pure Data - Idea: Make a flying robot into a musical instrument whose score is space. - What it is: A short music piece written for a quadcopter: as the AR.Drone flies into different areas of a room, its position generates musical sequences on top of the sound of its own engines. - Technique: A Kinect depth image processed in vvvv locates the drone inside defined bounding boxes and sends its position to Pure Data, which triggers musical patterns. - Try it: Divide the room into four zones with tape, track a toy (or a person holding it) with a webcam, and trigger a different loop in each zone to compose a piece by flying or walking a path. Twist: record the path once and replay it as a score someone else must follow. #### body(input); — Mária Júdová (2015) - Video: https://vimeo.com/133901650 - Interaction: Hands & Body, Performance, Perception & Effects - Platform & tech: Wearable, EMG, heart-rate sensor, breath sensor - Idea: Light is driven by what happens inside the body, not by its outline. - What it is: A solo in which light is generated from the dancer's own heartbeat, breathing and muscle activity; at times the light follows her body exactly, at others it takes on a life of its own. - Technique: Sensors worn on the body (heart rate, breath and EMG) send data wirelessly to software that generates the light and sound, with custom 3D-printed enclosures and PCBs. - Try it: Use a phone camera's heart-rate app or a cheap pulse sensor on an Arduino to drive the brightness of a lamp while a dancer performs; start in sync, then let the lamp slowly invent its own rhythm. Twist: invite a viewer to hold the sensor so the dancer follows a stranger's heartbeat. #### DUST — Mária Júdová (2017) - Video: https://vimeo.com/210525711 - Interaction: Performance, Hands & Body, Perception & Effects - Platform & tech: Headset, VR, motion capture, point cloud - Idea: Experience dance from the point of view of a speck of dust inside it. - What it is: A four-minute VR dance piece in which the viewer is a particle of stardust drifting through space in the immediate presence of a dancer made of points, whose movement sweeps the viewer along. - Technique: Motion-captured dance is rendered as a point cloud in a real-time engine, and the viewer's tracked head and hands interact with the particle field. - Try it: Convert a mocap clip into a particle system in Unity, place it in AR at life size, and let the phone act as a 'dust speck' that gets pushed away by the nearest joint. Twist: make the viewer's breath (microphone) scatter the dancer. #### Camouflage — Mária Júdová (2018) - Video: https://vimeo.com/285149672 - Interaction: Shared & Social, Play, Performance - Platform & tech: Headset, Projection, VR, motion tracking - Idea: Play hide-and-seek between a real dancer and a headset wearer, watched in two worlds at once. - What it is: A VR and dance installation of hide-and-seek: a participant in a headset hunts while a dancer hides, and the audience watches the chase both live in the room and in the virtual world. - Technique: The dancer is tracked in the physical space and represented as a camouflaged avatar in VR, while an external screen shows the virtual view to the watching audience. - Try it: Pair a headset player with a tracked partner (phone on the partner's chest streaming position) in an open room: the player only sees a faint virtual shape, and the audience watches both on a projection. Twist: swap roles every 60 seconds. #### Everywhen — Mária Júdová (2018) - Video: https://vimeo.com/259204752 - Interaction: Projection, Performance, Perception & Effects - Platform & tech: Projection, projection, generative visuals - Idea: Stage history repeating itself as a body caught in looping images. - What it is: An intermedia performance about recurrence in history, in which dancer Soňa Ferienčíková moves inside projected visuals and a live soundscape that loop and return. - Technique: Real-time generative visuals are projected around the dancer and likely respond to her movement, with Andrej Boleslavský and Constantine Nisidis on visual tech. - Try it: Record a performer's 10-second phrase, project it behind them on a loop, and have them perform against their own recording as it slowly degrades each cycle. Twist: let the live dancer change one gesture per loop until the past no longer matches. #### KYKEON — Mária Júdová (2020) - Video: https://vimeo.com/493291875 - Interaction: Performance, Shared & Social, Portals & Worlds - Platform & tech: Headset, VR, motion capture - Idea: Use VR to invent a shared ritual built from dancers' motion. - What it is: An immersive VR trilogy that invites the audience into a new shamanic ritual performed by virtual dancers from tanzmainz, choreographed by Taneli Törmä. - Technique: Dancers were motion-captured and rendered as stylised avatars and particle forms in a real-time VR scene, co-produced with Motion Bank. - Try it: Design a two-minute AR ritual for a group: each phone shows the same virtual circle of mocap dancers on the floor, and the group must perform one gesture together to advance the ritual. Twist: the ritual only continues if nobody speaks. #### Constellation of the Flesh — Mária Júdová (2021) - Video: https://vimeo.com/577351369 - Interaction: Performance, Hands & Body, Perception & Effects - Platform & tech: Headset, VR, motion capture - Idea: Can the dance of trance be digitised and felt from inside? - What it is: A VR experience made with Choy Ka Fai, based on a gathering of Buryat shamans, in which motion-captured trance movements become a sensory space the viewer is immersed in. - Technique: Dancers re-enacted shamanic movement in a motion-capture studio, and the data drive abstract bodies and particles in a real-time VR scene with spatial sound. - Try it: Capture a repetitive, trance-like movement with a phone mocap app and render it in AR as a slowly growing trail of particles around the viewer. Twist: let the trail only grow while the viewer moves in sync with it. ### Nintendo *Game and hardware company (3DS AR Games, Face Raiders)* Nintendo shipped AR Games and Face Raiders on every Nintendo 3DS in 2011, using AR cards and the handheld's cameras, and later co-created Mario Kart Live with Velan Studios. #### AR Games (Nintendo 3DS) — Nintendo (2011) - Video: https://www.youtube.com/watch?v=yxfs8vI-y_Q - Interaction: Tangible Objects, Play, Perception & Effects - Platform & tech: Phone, Nintendo 3DS, AR cards, stereo 3D - Idea: Put an AR card on the table and the tabletop cracks open into a lake or a dragon bursts out. - What it is: Preinstalled on every 3DS: placing a '?' AR card on a table makes a dragon burst out of it or lets you fish in a lake that opens in the table, shoot archery targets and draw graffiti. - Technique: A known printed card provides a fiducial for camera pose, and the stereo 3D display renders content emerging from the table plane around the card. - Try it: Use a printed image marker to make an AR dragon that bursts "cracking" out of the tabletop, using a stencil to cut the hole in the table. Twist: put the marker on different surfaces and have the dragon react differently. #### AR Games: Fishing — Nintendo (2011) - Video: https://www.youtube.com/watch?v=pKe2J0odq-Y - Interaction: Tangible Objects, Portals & Worlds, Play - Platform & tech: Phone, Nintendo 3DS, AR cards - Idea: The tabletop turns into a pond in AR, and moving the handheld lets you fish at your own desk. - What it is: One of the 3DS AR Games modes: the table surface around the AR card turns into a pond, and the player fishes by moving and tilting the console. - Technique: The card marks the table as a water plane using a stencil or depth mask, and tilting and moving the console are mapped to the fishing line and reel. - Try it: Make a "water surface" on the table with a shader, tilt the phone to cast, and lift it quickly to reel in. Twist: fish only bite when you stay quiet (judged by the microphone). #### Face Raiders — Nintendo (2011) - Video: https://www.youtube.com/watch?v=bh-YrWvdUCs - Interaction: Face, Play, Hands & Body - Platform & tech: Phone, Nintendo 3DS, gyroscope, face capture - Idea: Your own face becomes the enemy, bursting out of holes in your real room's walls to attack you. - What it is: The 3DS takes a photo of your face (or a friend's) and turns it into flying enemy heads that attack through holes in your real room; you turn your body to aim and shoot. - Technique: A captured face photo is texture-mapped onto enemy meshes, and the gyroscope rotates the virtual camera so enemies surround the player in a 360-degree space overlaid on the camera feed. - Try it: Take a face photo with the front camera, map it onto spheres flying around in AR, and turn your body to aim and shoot using the gyroscope. Twist: faces pull different expressions when they are hit. #### Kid Icarus: Uprising AR Cards — Nintendo (2012) - Video: https://www.youtube.com/watch?v=vF3DpChwg9Y - Interaction: Tangible Objects, Play - Platform & tech: Phone, Nintendo 3DS, AR cards - Idea: Put two physical cards on the table and the characters jump out to fight. - What it is: Collectible AR cards for Kid Icarus: Uprising; placing two cards in front of the 3DS makes the characters appear and battle each other on the table. - Technique: Two cards are detected at once as image targets, and the relative pose between them determines the characters' facing and battle stats. - Try it: Summon one character from each of two image marker cards, so the closer the cards the more excited the characters get, and they start fighting when facing each other. Twist: stacking the cards merges the two characters into one. #### Spirit Camera: The Cursed Memoir — Nintendo (2012) - Video: https://www.youtube.com/watch?v=d6s4AXx0A7Q - Interaction: Tangible Objects, Play, Perception & Effects - Platform & tech: Phone, Nintendo 3DS, AR booklet - Idea: The 3DS becomes a camera that can photograph ghosts, and the bundled diary reveals them in AR. - What it is: A Fatal Frame spin-off co-developed with Koei Tecmo: the 3DS becomes the haunted Camera Obscura, and a bundled cursed diary reveals ghosts that appear in your real room. - Technique: Pages in a printed booklet act as image targets that trigger story events, while gyroscope-based aiming lets ghosts appear around the player in the camera view. - Try it: Make a paper "cursed diary" of a few pages, where scanning each page triggers an AR ghost or a sound. Twist: the ghosts can only be seen with the lights off (when camera brightness is low). ### Ogmento (Ori Inbar, Oriel Bergig, Nate Hagbi) *AR games developer and publisher (later Flyby Media)* Co-founded by Ori Inbar in 2009, Ogmento published location-based AR games and demoed hand-sketch and SLAM tech from its Israeli R&D team; it became Flyby Media, which Apple acquired in 2016. #### Put a Spell — Ogmento (Ori Inbar, Oriel Bergig, Nate Hagbi) (2009) - Video: https://www.youtube.com/watch?v=EB45O7-6Xrg - Interaction: Tangible Objects, Information & UI, Voice & Sound - Platform & tech: Phone, iPhone, card recognition - Idea: Physical letter cards become the game controller for learning to spell. - What it is: A spelling game for children: a 3D panda appears on the iPhone screen above a table, and kids drop real letter cards into blank spaces to spell the word it asks for; the app recognizes and pronounces the letters. - Technique: Likely marker or card-art recognition identifies each letter card and its slot, while a 3D character is anchored to the table. - Try it: Print letter cards as image targets and make a phone AR scene where a creature appears only when the cards on the table spell its name. Twist: let the creature mispronounce misspelled words so children can hear the mistake. #### In-Place Sketching AR Games — Ogmento (Ori Inbar, Oriel Bergig, Nate Hagbi) (2010) - Video: https://www.youtube.com/watch?v=cuVKMHeWq9U - Interaction: Drawing & Making, Play, Tangible Objects - Platform & tech: Phone, sketch recognition, paper tracking - Idea: Drawing and playing happen on the same sheet at the same time. - What it is: Prototype games in which players draw walls, holes and characters on paper, and the camera turns the drawing into a playable AR level that updates as they keep sketching. - Technique: Sketch recognition classifies hand-drawn symbols into game elements and tracks the paper itself, based on research by Nate Hagbi, Raphael Grasset, Oriel Bergig, Mark Billinghurst and Jihad El-Sana (IEEE VR 2010). - Try it: Define three drawn symbols (circle = hole, line = wall, X = goal) and build an AR ball-rolling game whose level is read from a photo of the paper. Twist: let a second player sabotage by drawing during play. #### In-Place AR 3D Sketching of Mechanical Systems — Ogmento (Ori Inbar, Oriel Bergig, Nate Hagbi) (2011) - Video: https://www.youtube.com/watch?v=TLiS7JrpRvs - Interaction: Drawing & Making, Information & UI, Tangible Objects - Platform & tech: Phone, Desktop, sketch recognition, physics simulation - Idea: A physics textbook diagram you draw yourself becomes a live experiment. - What it is: Students sketch simple physics setups such as springs, masses and slopes on paper; the camera recognizes the diagram and runs a 3D simulation on top of it, which changes as the sketch is edited. - Technique: Sketch parsing turns diagram symbols into rigid bodies and constraints for a physics engine, rendered in 3D over the tracked paper. - Try it: Let students draw a slope and a circle on paper, detect them, and simulate a ball rolling down in AR with adjustable gravity. Twist: add a spring symbol that bounces the ball back. #### Paranormal Activity: Sanctuary — Ogmento (Ori Inbar, Oriel Bergig, Nate Hagbi) (2011) - Video: https://www.youtube.com/watch?v=9mVmw3ltFYA - Interaction: Location & City, Shared & Social, Play - Platform & tech: Phone, iPhone, GPS, compass - Idea: A haunting that spreads across real city blocks, with players as exorcists or the possessed. - What it is: A location-based multiplayer horror game tied to the film franchise: players see demons in the camera view around real places, exorcise them, and build safe-haven sanctuaries on the city map, while risking possession that turns them against allies. - Technique: GPS and compass place spirits and sanctuaries at real locations and overlay them on the camera view, with shared server state for territory and possession. - Try it: Place five geolocated 'haunted' points around campus and make a phone AR view that shows a ghost when a player is within 20 m. Twist: players who fail to banish a ghost become visible as ghosts to other players. #### Sketch-a-race — Ogmento (Ori Inbar, Oriel Bergig, Nate Hagbi) (2011) - Video: https://www.youtube.com/watch?v=b5Mq5yzUMKg - Interaction: Drawing & Making, Tangible Objects, Play - Platform & tech: Phone, iPad 2, computer vision, sketch recognition - Idea: Your own doodle is the level design. - What it is: Draw any line on paper, point an iPad at it, and the sketch becomes a race track with cars driving along it; players race using only stop-and-go controls and can switch to a first-person view. - Technique: The camera extracts the drawn stroke from the paper, converts it into a path spline and tracks the page so the 3D track stays registered in real time. - Try it: Use OpenCV or an AR image tracker to detect a dark hand-drawn line on white paper and send a small car along it in AR. Twist: let line thickness control speed. #### BubbleWhere — Ogmento (Ori Inbar, Oriel Bergig, Nate Hagbi) (2012) - Video: https://www.youtube.com/watch?v=8GcQ89x4AsQ - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Phone, circle detection, texture capture, mobile AR - Idea: Any circle in the world can become an instant AR marker. - What it is: A proof of concept in which the phone finds any round object in view, such as a plate, coaster or logo, grabs its texture on the fly, and turns it into a trackable AR bubble. - Technique: Real-time ellipse detection finds circles, rectifies and captures their texture, and then tracks the circle's pose as a new target without prior training. - Try it: Use OpenCV Hough circles on a phone camera feed to find round objects and pin a small animated creature onto each one. Twist: creatures jump between circles when two are close together. #### SLAM Computer Vision Tank: See Through Walls — Ogmento (Ori Inbar, Oriel Bergig, Nate Hagbi) (2012) - Video: https://www.youtube.com/watch?v=ZWfKWg1UuQ0 - Interaction: Portals & Worlds, Spatial Mapping, Play - Platform & tech: Phone, iPhone, SLAM, PTAM - Idea: Blast a hole in a real wall and peek into the world behind it. - What it is: At Augmented Reality Event 2012, an iPhone mapped a real wall and street with markerless SLAM, and a virtual tank blasted holes through it that revealed a virtual world behind. - Technique: Markerless monocular SLAM (credited to Robert Castle's PTAM-style work) estimates the wall plane, and depth-masked holes render the virtual scene behind it. - Try it: Detect a vertical plane and let each tap punch a circular hole that reveals a different virtual room behind the wall. Twist: holes slowly heal over unless the player keeps looking at them. ### Trixi Studios (Chip Sineni) *Chicago VFX / AR studio led by Chip Sineni* Small animation and VFX studio that built early Project Tango and ARKit experiments, including the viral a-ha 'Take On Me' AR portal. #### Star Wars Mixed Reality Test — Trixi Studios (Chip Sineni) (2015) - Video: https://www.youtube.com/watch?v=xK5w7JW0dAg - Interaction: Spatial Mapping, Play - Platform & tech: Phone, Project Tango, Unity - Idea: Star Wars droids and ships stay put in a real room while you walk around them. - What it is: Recorded live on a Project Tango tablet, Star Wars ships and droids appear in a real room and hold their place as the camera walks around them. - Technique: Tango motion tracking plus depth sensing anchors 3D models to the room so they keep position and scale while the device moves. - Try it: Put a model of your favorite movie prop into the classroom and walk around it to shoot a short "documentary". Twist: the model turns its head to look at you when you get close. #### LONELY GHOUSTS — Trixi Studios (Chip Sineni) (2016) - Video: https://www.youtube.com/watch?v=X3WtHmv0evc - Interaction: Play, Location & City - Platform & tech: Phone, Project Tango, Unity - Idea: Meet lonely little ghosts along your everyday walks. - What it is: An early Project Tango AR game in which friendly creatures appear along your real walks and wait to be met and collected. - Technique: Tango motion tracking places animated creatures in world space so they stay put as you walk around them. - Try it: Use AR Foundation to scatter five "lonely little monsters" along a campus route that greet you when you get close. Twist: once collected, they line up and follow behind you. #### PHANTOGEIST — Trixi Studios (Chip Sineni) (2016) - Video: https://www.youtube.com/watch?v=1lWZiuGCawc - Interaction: Play, Location & City, Spatial Mapping - Platform & tech: Phone, Project Tango, Unity - Idea: A first-person ghost shooter for any real space, with the game adapting to the size of the site. - What it is: A first-person shooter on a Project Tango phone spawns ghostly enemies and environments around you anywhere, from a small room to a football field, with no markers. - Technique: Tango's inside-out motion tracking gives 6DoF pose so enemies can be placed in world coordinates and the play area scales to the available physical space. - Try it: Build a small game with AR Foundation that spawns monsters on a sports field, with enemies walking in from the distance. Twist: enemies flying in the air can only be seen when you hold the phone up high. #### PHANTOGEIST Co-op 'Automatic Mixed Reality' — Trixi Studios (Chip Sineni) (2016) - Video: https://www.youtube.com/watch?v=AQdQ93XScEc - Interaction: Shared & Social, Play - Platform & tech: Phone, Project Tango, Unity - Idea: Two devices share one battle: one person plays, the other films, and mixed-reality video comes out automatically. - What it is: Two Tango devices share the same ghost battle, so one player can film the other fighting virtual enemies, producing mixed-reality footage automatically. - Technique: Both devices localize into a shared coordinate frame (likely via Tango area learning) so enemies appear in the same physical place for each player. - Try it: Build a two-player game with 8th Wall or Lens Studio shared sessions, or an ARKit collaborative session, where one person fights monsters and the other films. Twist: the camera operator's lens can "freeze" the monsters. #### Augmented Reality's A-ha Moment (Take On Me AR portal) — Trixi Studios (Chip Sineni) (2017) - Video: https://www.youtube.com/watch?v=ZBdRAdSosv4 - Interaction: Portals & Worlds, Perception & Effects, Voice & Sound - Platform & tech: Phone, ARKit, Unity - Idea: Step through a door into the pencil-sketch world of a-ha's Take On Me, where even your own hand turns into line art. - What it is: A door-sized portal in a real living room opens into the pencil-sketch world of a-ha's 'Take On Me'; stepping inside turns everything, including your own hand, into rotoscoped comic lines while real kids dance outside. - Technique: An ARKit-anchored portal uses a stencil/depth mask so the sketch world renders only through the doorway, and a line-art shader restyles the camera feed once you cross it. - Try it: Build a portal in Reality Composer or Unity that contains a hand-drawn line-art panorama. Twist: once you step through, add an outline filter to the camera feed so reality turns into a comic too. #### Gate to the Upside Down — Trixi Studios (Chip Sineni) (2017) - Video: https://www.youtube.com/watch?v=VQymxwls0m0 - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Phone, ARKit, Unity - Idea: A store aisle turns into the Upside Down from Stranger Things. - What it is: A Stranger Things-inspired ARKit test turns an ordinary pharmacy aisle into the dark, spore-filled 'Upside Down', with the real space darkened and overgrown around the camera. - Technique: The camera feed is color-graded dark while floating particles and vine meshes are anchored to ARKit-detected planes to make the real room feel replaced. - Try it: Pick a familiar corner of a classroom and turn it into "another parallel world" with a filter, floating particles and vines on the floor. Twist: the world switches back to reality when the phone gets close to a certain real object. #### Spiders with 6D.ai (Chip Sineni) — Trixi Studios (Chip Sineni) (2018) - Video: https://www.youtube.com/watch?v=gJLkbKRa8jY - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, 6D.ai SDK, Unity - Idea: Let virtual spiders crawl along real furniture and hide behind it. - What it is: Chip Sineni's creepy-crawly test for the 6D.ai SDK: spiders crawl over and hide behind real furniture thanks to a live mesh of the room. - Technique: 6D.ai builds a real-time mesh of the environment on the phone, used for occlusion and as a navigation surface so creatures crawl over real geometry. - Try it: Use a LiDAR phone and AR Foundation Meshing to make a swarm of virtual bugs crawl over tables and chairs and get occluded. Twist: when the lights come on (ambient light gets brighter), they scatter and flee. ### Valentin Heun *Creator of the Reality Editor (MIT Media Lab); VP Innovation Engineering / Reality Lab at PTC (Vuforia Spatial Toolbox)* Valentin Heun did his PhD in Pattie Maes' Fluid Interfaces group, where he built Smarter Objects and the Reality Editor, an AR 'digital screwdriver' for wiring together connected objects. At PTC he turned it into the open-source Vuforia Spatial Toolbox for industrial AR. #### Second Surface — Valentin Heun, MIT Tangible Media Group — Hiroshi Ishii (2012) - Video: https://vimeo.com/60139965 - Interaction: Drawing & Making, Shared & Social, Spatial Mapping - Platform & tech: Phone, natural feature tracking, iPad - Idea: Several people doodle and leave notes on real surfaces, and the notes stay put for others to find. - What it is: Shunichi Kasahara, Valentin Heun and Austin Lee let several people draw and leave notes on real surfaces around them through tablets; the drawings stay anchored in space for others to see. - Technique: Natural feature tracking of the surface texture gives each tablet a 6DOF pose relative to the same physical surface, so strokes stored in surface coordinates are shared over the network and rendered from every viewpoint. - Try it: Use AR Foundation shared anchors (or the same image target) so two phones see the same wall, then draw and leave messages on it that sync in real time. Twist: messages only appear at a certain time or when a certain person comes near. #### Reality Editor: Programming Smarter Objects — Valentin Heun, MIT Fluid Interfaces — Pattie Maes & Pranav Mistry (2013) - Video: https://vimeo.com/74073989 - Interaction: Tangible Objects, Information & UI, Drawing & Making - Platform & tech: Phone, Vuforia, Open Hybrid - Idea: Drag a line from one object to another on your phone screen and they are wired together. - What it is: Early Reality Editor demo: drag virtual lines between physical objects seen through a phone to connect their inputs and outputs, like patching cables in thin air. - Technique: Marker-based AR recognition overlays each object's inputs and outputs as nodes, and dragging a line between nodes on the touchscreen writes a data-flow link that the networked objects then execute. - Try it: Recognize two image cards in phone AR (one standing for a "switch" and one for a "light"), drag a line with your finger from one node to the other to connect them, and then pressing the virtual switch lights the virtual lamp. Twist: insert a "logic box" (delay, invert or count) in the middle of the connection. #### Smarter Objects — Valentin Heun, MIT Fluid Interfaces — Pattie Maes & Pranav Mistry (2013) - Video: https://vimeo.com/60138572 - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, Vuforia, Arduino, iPad - Idea: Point a tablet at an object to see and rewrite its hidden interface and behavior. - What it is: Point an iPad at a radio or lamp and a virtual control interface appears attached to it; you can reprogram what its physical knobs do and link objects together. - Technique: Vuforia image targets recognize each physical object and anchor a virtual UI to it, while an Arduino inside the object exchanges knob and actuator states with the iPad over the network so the virtual UI can remap physical controls. - Try it: Use 8th Wall image recognition or AR Foundation Image Tracking to recognize a sticker on a lamp or speaker, pop up a virtual control panel, and control an LED's brightness through an ESP32/Arduino. Twist: let what a physical knob controls be "reassigned" in the AR interface. #### Reality Editor — Valentin Heun (2015) - Video: https://vimeo.com/143709971 - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, Open Hybrid, Vuforia, Arduino - Idea: An AR "digital screwdriver": glance at your smart devices and rewire them all. - What it is: The public Reality Editor app: connected objects carry visual markers; point your phone at them to see their capabilities and wire a light switch to a lamp or a chair to a car seat by drawing connections. - Technique: Each object carries a visual marker recognized by Vuforia, its Open Hybrid server publishes I/O nodes over Wi-Fi, and drawn AR connections become routing rules between the objects' Arduino-driven inputs and outputs. - Try it: Connect a button and a light to each of two ESP32 boards, and have a phone WebAR app recognize their sticker markers so you can draw a line from the button to the light (with a WebSocket backend relaying messages). Twist: swap in non-electronic things, such as a chair being sat on, and invent an interesting "conversation between objects". #### Reality Editor 2.0: A Web Browser for the Physical World — Valentin Heun (2017) - Video: https://vimeo.com/218412997 - Interaction: Tangible Objects, Information & UI, Spatial Mapping - Platform & tech: Phone, Vuforia, JavaScript, IoT - Idea: Browse the physical world like the web, where every object is an interactive page. - What it is: The second-generation Reality Editor treats every physical object as a web page: anchored AR UIs, logic crafting and spatial search for objects in the room. - Technique: Objects are identified by image targets and each hosts HTML/JavaScript UIs served from a local node server, which the AR browser renders as anchored web views and connects through a visual logic editor. - Try it: Use WebXR or 8th Wall to make an anchored mini web panel for each of three objects in the classroom (showing status and clickable), plus a "search" button that highlights a chosen object. Twist: let classmates edit each object's page and see what it has become after an hour. #### Vuforia Spatial Toolbox – Remote Operator — Valentin Heun (2021) - Video: https://www.youtube.com/watch?v=axwfmV9wTV8 - Interaction: Tangible Objects, Spatial Mapping, Information & UI - Platform & tech: Phone, Web, Vuforia Spatial Toolbox, LiDAR - Idea: Draw paths remotely in a live 3D scene of a factory to program robots directly. - What it is: PTC Reality Lab's open-source successor to the Reality Editor: a remote operator sees a live 3D scan of a factory and programs robots and machines by drawing spatial paths. - Technique: A LiDAR scan of the space is streamed to a remote client as a 3D model, and paths drawn in that model are converted into waypoints in the shared coordinate frame and sent to the robot controller. - Try it: Scan a room with an iPhone LiDAR app such as Polycam, open the model remotely in three.js and click to draw a path, then send the path to another phone's AR view to show as arrows on the floor. Twist: have a classmate play the robot and walk along the AR path. ### Xavi Bou *Photographer and visual artist* Barcelona photographer trained in geology who, since 2012, stacks video frames of birds in flight into single images (Ornithographies) that reveal the shapes their paths draw in the sky. #### Ornithographies — Xavi Bou (2016) - Video: https://vimeo.com/561365138 - Interaction: Perception & Effects, Information & UI - Platform & tech: Desktop, digital chronophotography, high-speed video, frame stacking - Idea: Stacking time into one frame makes the invisible shape of flight visible: the basic recipe for any AR motion trail over real footage. - What it is: A short film of the Ornithographies project: flights of gulls, swallows and starlings are shown as solid ribbons and braids that hang in the sky, with a soundtrack made from the birds' own waveforms. - Technique: Bou records birds with a high-frame-rate video camera, then merges the frames of each flight into one image by keeping, per pixel, the darkest or most different value against the sky. - Try it: Build a phone camera effect (Lens Studio or a WebGL shader) that keeps a fading buffer of the last two seconds of dark pixels against the sky, and point it at birds or passing cars. Twist: colour each trail by time so the start of the flight is blue and the end is red. #### Murmurations — Xavi Bou (2020) - Video: https://www.youtube.com/watch?v=hScBionqFBA - Interaction: Perception & Effects, Information & UI - Platform & tech: Desktop, digital chronophotography, flock tracking - Idea: Tracing a flock shows that form can emerge from many local decisions without a leader, an idea for AR swarms that react to a predator (the user). - What it is: Starling flocks drawing shifting shapes over the sky are traced so the whole murmuration reads as a moving sculpture; when a peregrine falcon attacks, the traces show the flock splitting and reforming. - Technique: The Ornithographies trail effect is applied to long takes of starling murmurations so each bird leaves a short, fading line. - Try it: Program a boids flock in Unity or three.js, place it in AR above a courtyard, and draw a short trail behind every bird. Twist: make the phone position a predator the flock avoids. #### Emergence — Xavi Bou (2021) - Video: https://www.youtube.com/watch?v=iAOUMbgTkgw - Interaction: Perception & Effects, Voice & Sound - Platform & tech: Desktop, inverted footage, slow motion, slowed bird calls - Idea: Slowing and inverting reality is enough to reveal a hidden system, a cheap and strong AR filter idea. - What it is: Real footage of starling flocks is inverted and slowed down so each bird's relation to the whole becomes readable, over a soundtrack of nightingale, owl and starling calls slowed ten to twenty times. - Technique: The footage is colour-inverted and time-stretched, and bird calls are pitch-shifted down by slowing them, likely in standard video and audio editors. - Try it: Make a camera filter that inverts colours and plays the live microphone input slowed down four times, then walk through a park with it. Twist: slow the image only where something moves. #### Tous les oiseaux du monde — Xavi Bou (2022) - Video: https://vimeo.com/763921141 - Interaction: Information & UI, Perception & Effects - Platform & tech: Desktop, computer vision, bird detection, text generation - Idea: Showing the machine's bounding boxes on nature exposes what the camera 'sees', a useful critical move for AR object detection. - What it is: Made with UCLouvain, the film asks whether we could ever count all the birds in the world: a computer-vision system frames each flying bird with coordinates while a text model answers with fluent nonsense. - Technique: A detector trained on bird silhouettes draws boxes and coordinates over each bird in the footage, alongside text from a pre-ChatGPT language model. - Try it: Run a COCO object detector (for example TensorFlow.js or Lens Studio ML) on a live camera feed outdoors and display a running count of 'birds' and 'people'. Twist: let the count also invent a poetic caption for each detection. #### Entomographies — Xavi Bou (2024) - Video: https://vimeo.com/1050040741 - Interaction: Perception & Effects, Information & UI - Platform & tech: Desktop, high-speed macro video, digital chronophotography - Idea: The same trace method works at insect scale, so an AR trail effect can make the tiny motion around a flower visible. - What it is: With entomologist Adrian Smith, Bou turns the flight of insects into glowing patterns, drawing attention to their beauty and to their decline. - Technique: High-speed macro footage of insects in flight is stacked frame by frame into single trails, likely against dark or controlled backgrounds. - Try it: Point a phone with a trail-shader effect at a flowering bush for five minutes and record the pattern of bees and flies. Twist: switch the trail colour according to the flight speed so hovering and darting look different. #### One for Sorrow — Xavi Bou (2024) - Video: https://vimeo.com/941987903 - Interaction: Perception & Effects, Voice & Sound - Platform & tech: Desktop, digital chronophotography, mirror effect - Idea: Mirroring natural traces makes people read meaning into them, a way for AR to turn random motion into a message. - What it is: Made for the Birdsong Project, the film starts with plain footage of birds, then lets the traces appear and pile up, ending with mirrored swift trails that suggest symbols in the sky, like the omens read by Roman augurs. - Technique: The trail effect is revealed gradually over the footage and then mirrored left-right to create symmetric, sign-like shapes. - Try it: Add a vertical mirror to a live motion-trail camera effect so anything moving in front of the phone draws a symmetric figure. Twist: save the most symmetric figure of the day as a 'sign' and pin it in AR where it was made. #### Fluctus — Xavi Bou (2026) - Video: https://www.youtube.com/watch?v=83Pg95Bc12M - Interaction: Perception & Effects, Information & UI - Platform & tech: Desktop, overhead high-speed photography, wildlife rehabilitation - Idea: Changing the viewpoint (from below the sky to above the bird) reveals a new structure, a reminder that AR can offer impossible camera angles on real things. - What it is: Life-size, top-down images capture the instant a bird takes off after being released from a wildlife recovery centre, showing wings, colour and structure from a bird's-eye view. - Technique: A camera mounted above the release point shoots the bird's first wing strokes; sequential frames are likely composited to show the take-off motion. - Try it: Place a scanned or rigged bird in AR and let the user switch between a ground view and a locked top-down view of its take-off. Twist: leave a translucent copy of the bird at every wingbeat so the take-off becomes a sculpture. ### ART+COM Studios (Joachim Sauter) *Berlin design studio for kinetic sculpture and media installations* Founded in 1988 by Joachim Sauter and others, ART+COM builds kinetic sculptures such as Kinetic Rain and the BMW Museum's Kinetic Sculpture, where hundreds of motor-driven elements draw shapes in the air. #### Zerseher / De-viewer — ART+COM Studios (Joachim Sauter) (1992) - Video: https://vimeo.com/386256001 - Interaction: Gaze & Attention, Perception & Effects - Platform & tech: Desktop, eye tracking, digitized painting, real-time image distortion - Idea: The act of looking can destroy what you look at — gaze is a powerful and underused input for headset AR. - What it is: A framed Renaissance painting hangs on a wall; wherever the viewer looks, an eye tracker makes that part of the picture smear and dissolve. - Technique: A hidden camera tracks the viewer's eyes and maps the gaze point onto the digitized painting, where a displacement brush distorts pixels over time. - Try it: Overlay a famous painting on a real wall with a headset or HoloKit and melt the pixels where the user's head direction points. Twist: make the painting slowly heal when nobody looks at it. #### Kinetic Sculpture – The Shape of Things to Come — ART+COM Studios (Joachim Sauter) (2008) - Video: https://vimeo.com/8554267 - Interaction: Perception & Effects - Platform & tech: Desktop, stepper motors, steel wires, metal spheres, custom motion software - Idea: Hundreds of points, each moving only up and down, can draw any 3D shape — the physical ancestor of AR point clouds and voxel morphs. - What it is: 714 metal spheres hang on thin steel wires in the BMW Museum and rise and fall together, morphing from abstract waves into the outlines of car bodies. - Technique: Each sphere hangs from a computer-controlled stepper motor; software samples a 3D shape at every sphere's x/y position and sets its height. - Try it: Build a WebXR or AR Foundation grid of 20×20 floating spheres above a table that morph between a wave and a scanned object's height map. Twist: let the user's phone position become the peak of the wave. #### Mobility – Reflective Kinematronic II — ART+COM Studios (Joachim Sauter) (2010) - Video: https://vimeo.com/67559505 - Interaction: Perception & Effects, Information & UI - Platform & tech: Desktop, motorized mirrors, spotlight, choreography software - Idea: One light source plus many angled reflectors can write anywhere in a room — AR designers can think of reflection as a way to place content on distant surfaces. - What it is: A grid of small mirrors on rotating arms catches a single spotlight and throws moving patterns of light onto the surrounding walls. - Technique: Each mirror's angle is computed so that its reflected spot lands on a target point on the wall; together the spots form letters or shapes. - Try it: Cast a virtual beam from a phone-held light in AR, bounce it off virtual mirrors placed by the user, and make it land on real walls to spell a word. Twist: turn it into a two-player puzzle where each player controls half the mirrors. #### Kinetic Rain — ART+COM Studios (Joachim Sauter) (2012) - Video: https://vimeo.com/45188800 - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, 1,216 motors, copper-coated aluminium drops, motion software - Idea: A slow, synchronized swarm above people's heads turns a transit space into a place to pause — AR can use the ceiling as a stage, not just the floor. - What it is: Two fields of 608 copper-coloured raindrops each hang over the departure hall of Changi Airport and move in a 15-minute choreography, forming wings, kites and waves. - Technique: Each drop hangs on a thin cable from its own motor; a timeline of 3D keyframes is resampled per drop so that the grid reads as one flowing surface. - Try it: Anchor a grid of 200 virtual raindrops to the ceiling of your classroom with plane detection and animate them into two shapes with a looping timeline. Twist: make the drops part around the viewer's head as they walk underneath. #### Symphonie Cinétique – The Poetry of Motion — ART+COM Studios (Joachim Sauter) (2013) - Video: https://vimeo.com/70938823 - Interaction: Perception & Effects, Voice & Sound - Platform & tech: Desktop, kinetic winches, LED light elements, live orchestra - Idea: Movement itself can be the melody — AR scenes feel alive when objects move in time with sound rather than simply appearing. - What it is: A kinetic installation and concert in which suspended light bars and shapes rise, tilt and fall in a dark hall in step with a musical score. - Technique: Motorized winches lift and tilt light elements along precomposed trajectories that are synchronized to the musical timeline. - Try it: Choreograph five glowing bars in AR that rise and tilt to a 60-second piece of music you pick, anchored in a dark room. Twist: let a clap from the audience trigger the next movement. #### RGB|CMY Kinetic — ART+COM Studios (Joachim Sauter) (2015) - Video: https://vimeo.com/386253674 - Interaction: Perception & Effects, Voice & Sound - Platform & tech: Desktop, motorized mirrors, RGB spotlights, additive/subtractive colour - Idea: Colour appears where light paths cross — an AR designer can design with virtual light and shadow mixing, not just with coloured objects. - What it is: Five motor-driven discs move through beams of red, green and blue light, and their overlapping shadows and reflections mix into cyan, magenta and yellow. - Technique: Three coloured spotlights shine on reflective discs whose height and tilt are animated, so additive mixing in the reflections and subtractive mixing in the shadows change continuously. - Try it: Place three coloured virtual lights around a real table in AR and let the user drag floating discs to see coloured shadows mix on the real surface. Twist: tie the disc heights to a music track. ### Ai Weiwei (艾未未) *Artist and activist* Chinese artist whose large public installations use everyday objects in huge numbers, from 100 million porcelain seeds to refugee life jackets wrapped around a concert hall. #### Circle of Animals/Zodiac Heads — Ai Weiwei (艾未未) (2010) - Video: https://www.youtube.com/watch?v=kWNFuG4BTfU - Interaction: Location & City, Information & UI - Platform & tech: Desktop, bronze heads, Chinese zodiac, public plaza - Idea: Re-staging lost heritage objects at monumental size in a public square asks who owns them, a question AR restitution projects also face. - What it is: Twelve large bronze animal heads of the Chinese zodiac, based on the looted fountain-clock heads of the Old Summer Palace in Beijing, stand in a circle in public plazas such as the Pulitzer Fountain in New York. - Technique: Each head was modelled from the surviving and missing originals, enlarged and cast in bronze on columns. - Try it: Scan or find 3D models of museum objects removed from their original site and place them back in AR at their place of origin. Twist: show the route each object travelled as an AR line on a map. #### Sunflower Seeds — Ai Weiwei (艾未未) (2010) - Video: https://www.youtube.com/watch?v=PueYywpkJW8 - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Desktop, 100 million porcelain seeds, Jingdezhen craftspeople - Idea: A landscape made of countless individual, handmade units shows the one and the many at once, a strong idea for AR crowds and data points. - What it is: The floor of Tate Modern's Turbine Hall was covered with 100 million hand-painted porcelain sunflower seeds, made by about 1,600 craftspeople in Jingdezhen. - Technique: Each seed was shaped, fired and hand-painted with a few brushstrokes, then spread in a layer about 10 cm deep. - Try it: Fill a room floor in AR with 100,000 instanced seeds and let each user 'paint' one unique seed that is added to the pile. Twist: the user can later find their own seed with a search function. #### Forever Bicycles — Ai Weiwei (艾未未) (2013) - Video: https://www.youtube.com/watch?v=5Rj3x7vKU4Y - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, 3,144 bicycles, modular stacking - Idea: Thousands of identical parts create moiré-like patterns that change with the viewer's position, a direct recipe for instanced AR structures. - What it is: For Nuit Blanche Toronto, 3,144 identical bicycles were linked into a huge tunnel-like structure over Nathan Phillips Square that shifts into different patterns as you walk around it. - Technique: Frames of 'Forever' brand bicycles, with wheels and handles removed, are bolted together in a modular grid. - Try it: Instance 2,000 copies of one scanned object in a grid arch in AR over a walkway and walk through it to see how patterns shift. Twist: each copy rotates a few degrees according to the user's walking speed. #### Safe Passage (Konzerthaus Berlin) — Ai Weiwei (艾未未) (2016) - Video: https://www.youtube.com/watch?v=8faWQdQ_JgY - Interaction: Location & City, Information & UI - Platform & tech: Desktop, 14,000 life jackets, neoclassical facade - Idea: Covering a cultural landmark with the objects of a crisis brings a distant fact onto a local building, a strategy for AR data overlays on monuments. - What it is: The columns of Berlin's Konzerthaus were wrapped with 14,000 life jackets collected from refugees who had landed on the Greek island of Lesbos, with a rubber boat hanging in front. - Technique: Real orange life jackets were attached in dense rows to the building's columns, forming a bright facade layer. - Try it: Wrap the columns or facade of a real building in AR with a repeated object that stands for a dataset (one object per person affected). Twist: the count updates from a live public dataset. #### Good Fences Make Good Neighbors — Ai Weiwei (艾未未) (2017) - Video: https://www.youtube.com/watch?v=kOzRdfxRa90 - Interaction: Location & City, Information & UI - Platform & tech: Desktop, steel fences, cages, citywide installation - Idea: Spreading one theme over hundreds of city sites makes a whole city the exhibition, a model for citywide AR shows. - What it is: Across New York City, Ai placed more than 300 fences, cages and banners, including a golden cage at Central Park's entrance and a mirrored passage under the Washington Square Arch, about migration and borders. - Technique: Site-specific fence structures were fabricated in steel and installed in parks, bus shelters and lamp posts in all five boroughs. - Try it: Create a citywide AR show: anchor 20 small cage or fence pieces at bus stops and gates around campus, each with a short story about a boundary. Twist: some pieces can only be seen from inside a fence. #### Law of the Journey — Ai Weiwei (艾未未) (2017) - Video: https://www.youtube.com/watch?v=1D54wIG2a34 - Interaction: Information & UI, Perception & Effects - Platform & tech: Desktop, 60-metre inflatable boat, rubber figures - Idea: Scaling up a news image until you must walk along it turns a picture into a body experience, what AR can do with a photograph. - What it is: A 60-metre black inflatable boat crowded with 258 faceless rubber refugee figures fills a hall, shown at the National Gallery in Prague and the 21st Biennale of Sydney. - Technique: The boat and figures were made from the same black rubber used for refugee dinghies, fabricated in China. - Try it: Take a news photo of a crowd and rebuild it as a 1:1 AR scene of simple figures that users can walk along. Twist: every figure has a short, real testimony that plays when approached. ### Aphra Shemza *Interactive light artist* British artist who makes interactive light sculptures, often from recycled plastic and old light fittings, that respond to how close visitors stand or how much noise they make. Much of her work reinterprets the geometric abstraction of her grandfather, the painter Anwar Jalal Shemza, and she co-created the installation Post Truth and Beauty with Tim Murray-Browne. #### Composition X — Aphra Shemza (2015) - Video: https://www.youtube.com/watch?v=hmh0cZxwpNw - Interaction: Hands & Body, Shared & Social, Perception & Effects - Platform & tech: Projection, ultrasonic sensors, LED, Arduino (likely), infinity mirror - Idea: Visitors mix the colour of the sculpture together just by where they stand. - What it is: A sphere inside a pyramid inside a cube, after Kepler's model of the cosmos: ultrasonic sensors read how far each visitor stands, and several people moving around it mix the LED colours together while a mirrored centre opens into an infinite reflection. - Technique: Ultrasonic distance sensors on each side feed a microcontroller that maps each visitor's distance to the LED colour channels, while two-way mirrors create the infinity effect. - Try it: Build a cardboard lantern with three ultrasonic sensors and an addressable LED strip so each sensor controls red, green or blue by distance. Twist: the colour only turns white when all three people stand at the same distance. #### Seconds Pass — Aphra Shemza, Tim Murray-Browne (2018) - Video: https://www.youtube.com/watch?v=NhKyJ1yPlVw - Interaction: Information & UI, Perception & Effects - Platform & tech: Projection, LED, data-driven animation - Idea: A statistic becomes a heartbeat of light you cannot look away from. - What it is: A four-metre field of glowing spheres whose pulses follow live birth and death rates of children under five: four children are born every second and one dies every six seconds, commissioned by Save the Children, with lighting animation by Tim Murray-Browne. - Technique: Addressable LEDs inside plastic spheres are animated by a timing program that fires a pulse for each birth and a dimming for each death at the real global rates. - Try it: Pick a real per-second statistic and make a phone screen or LED strip pulse at exactly that rate on a table in the room. Twist: add a second rhythm for the opposite statistic and let the two beats drift against each other. #### shemza.digital #5 — Aphra Shemza (2021) - Video: https://www.youtube.com/watch?v=BAdUb-LwuXs - Interaction: Hands & Body, Tangible Objects, Perception & Effects - Platform & tech: Projection, ultrasonic sensors, LED, recycled acrylic - Idea: A painted arch becomes a real doorway that lights up to welcome you. - What it is: A walk-through archway taken from Anwar Shemza's City Wall paintings, made in recycled green acrylic: as a visitor approaches, sensors make the arch light up and invite them to pass through it. - Technique: Ultrasonic sensors measure the visitor's distance and a bespoke circuit ramps up the LEDs inside the acrylic arch as they get closer. - Try it: Turn a classroom doorway into a threshold with a distance sensor and LED strip that brightens as someone approaches. Twist: make it fade out if two people try to pass at once. #### shemza.digital #8 — Aphra Shemza (2022) - Video: https://www.youtube.com/watch?v=zthfBCeEd7g - Interaction: Drawing & Making, Shared & Social, Projection - Platform & tech: Phone, Web, Projection, web app, QR code, real-time animation - Idea: Every visitor's phone painting joins a shared, living wall in the gallery. - What it is: Visitors scan a QR code, make their own digital painting in the style of Anwar Shemza on their phone, and send it to an ever-growing archive that animates on the gallery wall; made with Stuart Batchelor and Peter Todd. - Technique: A web painting app reached by QR code saves each submission to an online archive, and a real-time animation on the gallery screen pulls in the new paintings as they arrive. - Try it: Build a p5.js drawing page limited to three shapes and five colours that posts each drawing to a shared wall projected in class. Twist: each new drawing pushes the oldest one off the wall. #### Thanet Warn(m)ing — Aphra Shemza (2024) - Video: https://www.youtube.com/watch?v=ZT3MwIFmRx4 - Interaction: Information & UI, Voice & Sound, Perception & Effects - Platform & tech: Projection, LED, spatial audio, climate data - Idea: Stand inside sixty years of local warming, felt as colour and volume. - What it is: An immersive light and sound installation that wraps the viewer in coloured light and shifting audio driven by 60 years of temperature data from the Manston weather station in Thanet: the warmer the year, the warmer the colour and the louder the sound. - Technique: Yearly temperature values are mapped to LED colour temperature and to the gain of a local field recording, played back in sequence as a timeline. - Try it: Download your city's yearly temperature record and drive a phone flashlight colour app or an LED strip plus a sound loop from it, one year per second. Twist: let visitors scrub through the years by walking along a line on the floor. ### Balasaravanan Thoravi Kumaravel *Researcher, Microsoft Research (Redmond); HCI researcher on mixed-reality collaboration* Built Loki, TutoriVR, TransceiVR and DreamStream during his PhD at UC Berkeley with Björn Hartmann, all about bridging people inside and outside headsets. At Microsoft Research he works on generative and NeRF-based spaces for remote collaboration. #### TutoriVR — Balasaravanan Thoravi Kumaravel (2019) - Video: https://www.youtube.com/watch?v=10DofnNl3Qg - Interaction: Drawing & Making, Information & UI - Platform & tech: Headset, VR, Tilt Brush, 360 video - Idea: A tutorial should live in the same space as the work, not on a flat screen outside it. - What it is: While painting in Tilt Brush, a learner opens a tutorial video that lives inside VR, with the teacher's controllers and brush strokes replayed in 3D next to their own canvas. - Technique: Pairs a recorded 360 video tutorial with logged controller poses and tool states, so steps can be replayed as spatial ghosts and navigated by timeline. - Try it: Record a short origami or LEGO build with a phone, then place the video as a floating AR panel beside the real workspace and compare with watching on a laptop. Twist: add 3D arrows at the moments hands move out of frame. #### TransceiVR — Balasaravanan Thoravi Kumaravel (2020) - Video: https://www.youtube.com/watch?v=cwnGvUfYWws - Interaction: Shared & Social, Drawing & Making - Platform & tech: Headset, Phone, VR, tablet, asymmetric collaboration - Idea: Give the people watching a headset user a real voice inside the virtual world. - What it is: A person outside VR taps and draws on a tablet view of the virtual scene, and their annotations appear in the headset user's world, while they can also rewind or look around independently. - Technique: Hooks into VR platform APIs to capture scene views and poses for third-party apps without source code, streaming them to a tablet and projecting tablet annotations back into VR. - Try it: Pair a student wearing a phone VR viewer with a partner holding a second phone that mirrors the scene; the partner guides by drawing on their screen while the viewer finds objects. Twist: forbid speaking so all help must be drawn. #### DreamStream — Balasaravanan Thoravi Kumaravel (2022) - Video: https://www.youtube.com/watch?v=TX-nTcRY7gE - Interaction: Shared & Social, Play - Platform & tech: Headset, VR streaming, spectating, multi-user - Idea: Stream the whole world, not just the player's window into it. - What it is: Instead of watching a flat Twitch stream, spectators step into the streamer's VR world, look over their shoulder, wander off to explore and chat with the player in 3D. - Technique: Streams 3D scene state and player poses from existing VR apps to remote spectators who render the environment locally with free viewpoints. - Try it: Run a tabletop AR game on one phone and let three spectator phones join the same shared AR anchor so they can walk around the table independently. Twist: give spectators one ability, like dropping a hint marker. #### BlendScape — Balasaravanan Thoravi Kumaravel (2024) - Video: https://www.youtube.com/watch?v=maqZbVyuGBA - Interaction: Shared & Social, Portals & Worlds - Platform & tech: Desktop, generative AI, video conferencing, segmentation - Idea: Let meeting participants generate the shared place they meet in. - What it is: Video-call participants type or speak a theme and their separate backgrounds are blended by AI into one shared scene, like a single cafe or classroom that everyone appears to sit in. - Technique: Composites segmented participants into AI-generated environments that blend their physical or digital backgrounds, steered with multimodal prompts and direct manipulation. - Try it: Run a class video call where each student picks a virtual background that continues a shared panorama, so together the tiles form one scene. Twist: change the scene in the middle of a story each student tells. #### SharedNeRF — Balasaravanan Thoravi Kumaravel (2024) - Video: https://www.youtube.com/watch?v=M6UB_7Cp9Lc - Interaction: Shared & Social, Spatial Mapping - Platform & tech: Headset, Desktop, NeRF, point cloud, RGB-D - Idea: Blend a slow but beautiful NeRF with a fast but rough point cloud to share a real workbench. - What it is: A remote collaborator inspects a physical prototype in photorealistic 3D from any angle, while a live point cloud shows the local person's hands and moving parts as they change. - Technique: Renders a Neural Radiance Field of the static workspace and overlays a real-time RGB-D point cloud for dynamic elements, with the remote viewpoint driven by a head-mounted camera. - Try it: Capture a desk with a phone 3D-scan app, share the model with a remote partner, then add a live phone video window for hands; have the partner guide an assembly. Twist: rescan only the part that changed. #### SpaceBlender — Balasaravanan Thoravi Kumaravel (2024) - Video: https://www.youtube.com/watch?v=wQqJHcrOado - Interaction: Shared & Social, Portals & Worlds - Platform & tech: Headset, generative AI, diffusion, depth estimation - Idea: Merge everyone's real rooms into one imagined shared room. - What it is: Two remote people each snap a photo of their room, and generative AI stitches both into one coherent 3D space where their avatars meet for a VR brainstorming session. - Technique: Estimates depth from user photos, aligns the meshes, and completes the gaps with diffusion-based inpainting guided by geometric priors and adaptive prompts. - Try it: Have two groups photograph their corners of the school, then collage them in an image editor into one 'impossible room' and view it as a 360 image in a phone viewer. Twist: let an image generator fill the seams and discuss what it invented. ### Benjamin Kuperberg *Creative technologist and tool maker (Chataigne, BenTo, NecTouch)* French creative technologist who wrote NecTouch, the Kinect multitouch layer behind Adrien M & Claire B's XYZT floor, and later the open-source show-control tools Chataigne and Blux. With the BenTo collective he builds interactive projection, LED juggling props, drone swarms and circus visuals. #### Interactive Video Mapping on Canvas — Benjamin Kuperberg (2013) - Video: https://www.youtube.com/watch?v=EsDzHx_JTcc - Interaction: Projection, Hands & Body, Perception & Effects - Platform & tech: Projection, Kinect, projection mapping - Idea: Pictures that only appear when someone is looking. - What it is: Photographs hung on a wall are revealed by projection only when people come closer, in different visual styles, and fade away again when nobody is there. - Technique: A depth camera measures visitors' distance to each frame and projection-mapped masks reveal or hide the image accordingly. - Try it: Project onto three blank sheets of paper and use a webcam to detect which one a person stands in front of, then fade in a hidden drawing there. Twist: the picture shows a different image to each person. #### Interactive particles @ La Bifurk — Benjamin Kuperberg (2013) - Video: https://www.youtube.com/watch?v=9N--dSXwP9c - Interaction: Hands & Body, Performance, Perception & Effects - Platform & tech: Projection, depth camera, DMX, particles - Idea: Turn a dancing body into drifting sand. - What it is: Party guests step into a box and dance; their bodies appear as sand-like particle figures, and when nobody is inside, geometric shapes animate to the music. - Technique: An Intel Creative Gesture depth camera captures the dancers and the software converts the depth silhouette into particles, with the render controlled over DMX by the lighting technician. - Try it: Use a webcam and body segmentation in TouchDesigner or p5.js to turn a dancing student into particles that fall when they stop moving. Twist: the particles of two dancers mix when they touch. #### Multitouch Wood Bar with NecTouch — Benjamin Kuperberg (2013) - Video: https://www.youtube.com/watch?v=T_Vm18xWYKM - Interaction: Projection, Hands & Body, Tangible Objects - Platform & tech: Projection, Kinect, TUIO, projection mapping - Idea: Any piece of furniture can become a touchscreen. - What it is: A wooden bar counter becomes a projected multitouch surface: particles follow fingers across the top, climb a vertical stick when they touch it, and spill onto a front panel. - Technique: NecTouch uses a Kinect to detect touches on arbitrary surfaces and sends them as TUIO to an Adobe AIR app with a custom surface-warping library that maps content onto three surfaces. - Try it: Project onto a desk and detect fingertips with a depth camera or a phone above it, then let virtual marbles roll toward touches and fall off the desk's edge onto the floor. Twist: objects on the desk act as obstacles. #### Realtime Glove-Controlled Drone Swarm — Benjamin Kuperberg (2017) - Video: https://www.youtube.com/watch?v=ywpK-KrGKTY - Interaction: Hands & Body, Performance, Tangible Objects - Platform & tech: Wearable, Crazyflie, Vive tracker, Specktr gloves - Idea: Juggle with flying robots using only your hands. - What it is: A juggler wearing sensor gloves conducts a small swarm of indoor drones: each hand can pick up drones, make them follow the leader, move in sync or form shapes, as a test for a circus show. - Technique: Crazyflie drones are positioned with a TDoA radio system, Specktr touch gloves and Vive trackers give hand position and gestures, and custom software maps gestures to swarm behaviours. - Try it: Simulate the idea in WebXR or Unity: track one hand with a phone or Quest and let a flock of virtual drones follow, orbit or mirror it. Twist: each finger commands a different drone. #### SpinWall — Benjamin Kuperberg (2018) - Video: https://www.youtube.com/watch?v=dUlzz_6AU3s - Interaction: Performance, Projection, Tangible Objects - Platform & tech: Projection, projection mapping, LED poi, Chataigne - Idea: Make the props in a performer's hands and the wall behind them one light show. - What it is: A poi spinner performs in front of a wall with projection mapping while the LED poi in his hands light up in sync with the projected patterns and music. - Technique: LED poi and the projection mapping are driven from the same show-control timeline (Chataigne / BenTo), so colours and patterns on the props and wall stay in sync. - Try it: Project onto a wall while a student waves a phone showing a full-screen colour, and sync both from one web page over WebSockets. Twist: the phone's motion changes the wall instead of the timeline. #### CyrWheel Visuals and Interactions Research — Benjamin Kuperberg (2022) - Video: https://www.youtube.com/watch?v=naWtD6-Jqmo - Interaction: Performance, Tangible Objects, Hands & Body - Platform & tech: Wearable, IMU, LED, Chataigne - Idea: Let a circus apparatus light itself according to how it moves. - What it is: An acrobat spins inside an LED Cyr wheel whose light responds to the wheel's own rotation, synchronised with stage lighting, live music and a 3D tracking system. - Technique: A motion sensor embedded in the wheel streams orientation to Chataigne, which drives the wheel's LEDs, Blux stage lighting and music, with position from Augmenta's Pleiades tracker. - Try it: Tape a phone to a hula hoop or a bike wheel, stream its gyroscope to a laptop, and change projected colours with the rotation speed. Twist: when it stops, the light keeps spinning on the wall. ### Camille Utterback *Interactive installation artist; MacArthur Fellow (2009)* Artist whose camera-based projections (Text Rain with Romy Achituv, the External Measures series, Liquid Time) render the moving body as a painterly, living trace. Associate professor at Stanford. #### Text Rain (with Romy Achituv) — Camille Utterback (1999) - Video: https://www.youtube.com/watch?v=GYvyuL-Mkjg - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, video camera, brightness threshold, Max/custom software - Idea: Catch the letters of a poem falling from the sky with your body. - What it is: Participants see a mirrored video of themselves while coloured letters from a poem fall like rain; the letters land on and rest on anything dark, so people catch, lift and spill lines of text with their bodies. - Technique: Each falling letter checks the brightness of the live camera pixel below it and stops falling when the pixel is darker than a threshold, so the body's silhouette becomes a physical surface. - Try it: Make 'letter rain' with a p5.js camera sketch, where each letter stops when it meets a dark pixel. Twist: use a poem you wrote yourself, and play a sound when caught letters line up into a readable word. #### Liquid Time Series — Camille Utterback (2002) - Video: https://www.youtube.com/watch?v=qSHmx45AF_k - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, video buffer, slit-scan, camera tracking - Idea: The closer you get to the screen, the further that patch of the picture slips back into the past. - What it is: A viewer's movement fragments a pre-recorded street video: the closer they step to the projection screen, the deeper that part of the image slides into the past. - Technique: A camera measures each viewer's position and distance, and a slit-scan-style buffer picks older frames for image regions nearest to the viewer. - Try it: Store a video as a frame buffer, detect a person's position with a camera, and show earlier frames in the vertical strip where they stand. Twist: use the phone's proximity sensor or sound volume to control how far back into the past it goes. #### Untitled 5 (External Measures series) — Camille Utterback (2004) - Video: https://www.youtube.com/watch?v=zOydp3DXWrk - Interaction: Hands & Body, Drawing & Making, Projection - Platform & tech: Projection, camera tracking, generative drawing - Idea: People's pauses and movements leave growing brushstrokes on the wall. - What it is: A projected 'living painting' where visitors' movements leave soft, overlapping painterly marks that respond to how long and how fast people stay. - Technique: Camera-tracked body positions and motion history drive generative brush-stroke systems that accumulate and decay over time. - Try it: Drive a generative brush with camera motion detection, leaving strokes on the projection that thicken the longer someone stays. Twist: a line connecting two people appears only when they come close to each other. #### Abundance — Camille Utterback (2007) - Video: https://www.youtube.com/watch?v=xgRFUsVVb84 - Interaction: Hands & Body, Location & City, Projection - Platform & tech: Projection, overhead camera tracking, façade projection - Idea: The paths of people crossing a plaza are drawn live as a giant projection on the building. - What it is: At night a camera on San Jose City Hall watches people walking across the plaza and projects a responsive animation of their paths onto the building's rotunda. - Technique: A roof-mounted camera tracks pedestrians from above and their trajectories drive a generative animation projected onto the cylindrical façade. - Try it: Film the sports field from high up in a teaching building and draw pedestrians' paths live as colorful flow lines projected on a wall. Twist: when two paths cross, a 'flower' grows. #### Drawing from Life — Camille Utterback (2008) - Video: https://www.youtube.com/watch?v=E4AJY959514 - Interaction: Hands & Body, Perception & Effects, Information & UI - Platform & tech: Projection, video processing, ASCII rendering - Idea: Write your live mirror image as a portrait made from the four letters of DNA. - What it is: Museum visitors see a life-size projected image of themselves composed entirely of the letters A, C, G and T, the four bases of DNA. - Technique: Live video is sampled on a grid and each cell is rendered as one of four letters whose size or brightness follows the underlying pixel value (ASCII-art style). - Try it: Use p5.js to turn the camera image into an ASCII-character portrait using only the letters A, C, G and T. Twist: switch to a word related to your theme and make the letters jitter with the volume of sound. #### Entangled — Camille Utterback (2015) - Video: https://www.youtube.com/watch?v=cmKSwen2GAw - Interaction: Hands & Body, Projection, Shared & Social - Platform & tech: Projection, video cameras, scrim projection, custom software - Idea: People on either side of a sheer screen tangle up each other's traces. - What it is: Two layered projections on translucent scrims respond to visitors from both sides, so people on opposite sides tangle each other's painterly traces. - Technique: Two cameras track people on each side of stacked scrims and generative drawing layers are projected through the semi-transparent fabric, mixing both groups' traces. - Try it: Hang a piece of sheer fabric with a projector and a camera on each side, so the movement trails of classmates on both sides overlap on the same cloth. Twist: the pattern only connects when people on the two sides line up with each other. ### Can Büyükberber *Immersive new-media artist* Turkish-born artist working with projection mapping, domes, VR and AR; Adobe's immersive artist in residence and an early ARKit experimenter. #### Unfold 01 | Projection on Print — Can Büyükberber (2015) - Video: https://www.youtube.com/watch?v=pfeEgUcj0Ds - Interaction: Projection, Perception & Effects - Platform & tech: Projection, projection mapping - Idea: A projection laid exactly over a print makes the flat image fold and breathe like a living thing. - What it is: A morphogenesis-inspired form printed on a large canvas is overlaid with an exactly aligned projection, so the flat print seems to fold and breathe. - Technique: A projector is calibrated to the printed canvas so animated light and shadow are registered on top of the matching printed geometry. - Try it: Print a black-and-white geometric image and use a projector with MadMapper or TouchDesigner to align animated light and shadow precisely onto it. Twist: make the animation follow the rhythm of the ambient music. #### Unifield | Projection on Lasercut Sculpture — Can Büyükberber (2016) - Video: https://www.youtube.com/watch?v=H8jOvMXsZgc - Interaction: Projection, Perception & Effects, Tangible Objects - Platform & tech: Projection, projection mapping, laser cutting - Idea: Projection mapping on a laser-cut sculpture makes it look like a 4D object is passing through it. - What it is: A laser-cut sculpture based on Coxeter's higher-dimensional geometry is projection-mapped so light seems to reveal a 4D object rotating through it. - Technique: Projection mapping aligns rendered animation to the fabricated sculpture's facets so virtual light and motion appear to live on the physical form. - Try it: Fold a polyhedron from cardboard and use projection mapping to light up its faces in sequence, creating the illusion of rotation. Twist: change the direction of the rotating light when viewers walk by, detected with a camera. #### Invisible exhibition at the de Young (ARKit experiment) — Can Büyükberber (2017) - Video: https://www.youtube.com/watch?v=zC3XE6r5l0U - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, ARKit, Unity - Idea: Hold an invisible exhibition in a museum that can only be seen through a phone. - What it is: With Marpi, he placed generative 3D sculptures in the de Young Museum that are only visible through an iPad, an 'invisible exhibition' blurring what is real in the galleries. - Technique: A custom ARKit/Unity app anchors animated generative meshes to detected floor planes so they appear as sculptures standing in the real gallery. - Try it: Curate an "invisible exhibition" in a school hallway where each classmate uses Reality Composer to place a work visible only through a phone. Twist: the works appear only during certain times of day. #### Metafold 05, AR-activated digital print — Can Büyükberber (2024) - Video: https://www.youtube.com/watch?v=cwlwe4YgkuM - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Phone, image tracking AR - Idea: Scan a static print with a phone and a moving 3D sculpture grows out of it. - What it is: A printed artwork from his Metafold series springs into an animated 3D sculpture when viewed through a phone, extending the static print into motion. - Technique: Image-target tracking recognizes the print and overlays a registered animated 3D version on top of it (likely via an image-marker AR app). - Try it: Use one of your own drawings as an image-recognition marker (Artivive, Adobe Aero or 8th Wall all work) and make things fly out of the picture. Twist: when viewed from the side, show the 'world behind' the picture. #### Mixed Reality vs. Reality — Can Büyükberber (2024) - Video: https://www.youtube.com/watch?v=0qEePYKqr1U - Interaction: Perception & Effects, Play, Spatial Mapping - Platform & tech: Headset, Wearable, Meta Quest 3, Ray-Ban Meta - Idea: Do the same action once in mixed reality and once in the real world, and compare how 'physical' each feels. - What it is: Side-by-side POV captures compare playing ping-pong in Quest 3 passthrough mixed reality and with Ray-Ban Meta glasses in plain reality, noting how occlusion and gravity feel different. - Technique: Quest 3 passthrough MR with scene understanding simulates the ball, while the glasses record the real throw for a direct comparison of occlusion and physics. - Try it: Have a classmate do the same action (tossing a paper ball into a basket) once in an AR version and once for real, then edit the two side by side. Twist: secretly change the gravity in the AR version and have viewers spot the difference. ### Carmine Elvezio *Researcher and developer, Columbia University Computer Graphics and User Interfaces Lab* Long-time engineer and researcher in Steven Feiner's lab, behind virtual-replica remote assistance, NASA stowage guidance, hybrid AR sheet music and urban-data tabletops for HoloLens-era AR. #### Virtual Replicas for Remote Assistance — Carmine Elvezio, Steven Feiner — Columbia Computer Graphics & User Interfaces Lab (2015) - Video: https://www.youtube.com/watch?v=eMWdgB6upeU - Interaction: Shared & Social, Tangible Objects, Information & UI - Platform & tech: Headset, remote assistance, stereo HMD, object tracking - Idea: Point and demonstrate with a copy of the real object instead of with words or arrows. - What it is: A remote expert grabs a virtual copy of a part in the local worker's room and shows exactly where and how it should go; the local worker sees the ghost replica move into place through their AR headset. - Technique: Tracks the physical objects and gives the remote VR or AR expert manipulable 3D replicas whose poses are mirrored as guidance in the local user's stereo head-worn display. - Try it: Put a printed marker on a toy part; a remote partner moves a virtual copy of it in a shared phone AR scene to show where it goes, and the local student assembles it. Twist: allow the expert only one gesture and no voice. #### Collaborative Exploration of Urban Data in VR and AR — Carmine Elvezio, Steven Feiner — Columbia Computer Graphics & User Interfaces Lab (2018) - Video: https://www.youtube.com/watch?v=fgm_plH4qxc - Interaction: Information & UI, Shared & Social, Location & City - Platform & tech: Headset, HoloLens, VR, urban visualization - Idea: Turn city data into a shared tabletop model people can walk around and point at. - What it is: Several users gather around an immersive scale model of a city block and explore live datasets such as traffic or building information together, some in VR and some in AR. - Technique: Renders a scale model of the urban environment in shared VR and AR sessions and binds live datasets to its buildings and streets. - Try it: Print a map of your neighbourhood, track it with phone AR and raise bars over each block for data the class collects, like number of trees. Twist: let two phones see the same bars and argue for one change to the street. #### Hybrid UIs for Music Exploration in AR and VR — Carmine Elvezio, Steven Feiner — Columbia Computer Graphics & User Interfaces Lab (2018) - Video: https://www.youtube.com/watch?v=-rtaJAc_0Ys - Interaction: Voice & Sound, Information & UI, Shared & Social - Platform & tech: Headset, Phone, hand tracking, multi-touch, hybrid UI - Idea: Let 2D touch screens and 3D headsets share one music collection in the same room. - What it is: People browse an online music library together: some wear AR or VR headsets and grab genres and albums floating in 3D, others flick through the same collection on touch screens. - Technique: Synchronises a 3D music-library visualisation across head-worn displays and multi-touch tablets, combining 3D hand tracking with 2D touch input. - Try it: Lay out album covers as AR cards floating around the classroom and let students on laptops add songs to a shared playlist that phone users see appear in space. Twist: place songs by tempo from floor to ceiling. #### Hybrid RTK GNSS and SLAM Outdoor AR — Carmine Elvezio, Steven Feiner — Columbia Computer Graphics & User Interfaces Lab (2019) - Video: https://www.youtube.com/watch?v=hDvQJrtHfmo - Interaction: Location & City, Spatial Mapping, Information & UI - Platform & tech: Headset, RTK GNSS, SLAM, HoloLens - Idea: Stitch small SLAM maps together with centimetre GPS to make city-scale AR. - What it is: A wearable AR user walks a wide outdoor area and sees markers for assets given only by latitude and longitude, placed accurately over the real terrain. - Technique: Uses RTK GNSS positions to georegister and chain multiple local SLAM maps from the headset's tracker into one wide-area coordinate frame. - Try it: Collect GPS coordinates of five spots around school with a phone, then place location-based AR markers at them and walk the route checking how far off each one appears. Twist: add a hidden treasure only visible within two metres. #### Manipulating 3D Anatomic Models in AR — Carmine Elvezio, Steven Feiner — Columbia Computer Graphics & User Interfaces Lab (2019) - Video: https://www.youtube.com/watch?v=eK42UEbMQvw - Interaction: Gaze & Attention, Information & UI, Hands & Body - Platform & tech: Headset, HoloLens, head gestures, medical visualization - Idea: When your hands are busy, let your head drive the 3D model. - What it is: A clinician whose hands are busy rotates and scales a 3D anatomy model in an AR headset using only head movement, compared with the usual hand-gesture approach. - Technique: Maps head orientation and small head motions to rotation and scaling of a HoloLens hologram, with a hands-free mode switch. - Try it: Build a phone AR viewer where tilting the phone or your head rotates a 3D skull while you hold a pen in each hand, then time a labelling task. Twist: use voice to switch between rotate and zoom. #### MiXR: Hybrid AR Sheet Music — Carmine Elvezio, Steven Feiner — Columbia Computer Graphics & User Interfaces Lab (2020) - Video: https://www.youtube.com/watch?v=tGW9pbQU8cY - Interaction: Performance, Voice & Sound, Information & UI - Platform & tech: Headset, Phone, AR headset, cross-device, annotation - Idea: Sheet music that floats where you need it and can be marked up by anyone in the ensemble. - What it is: A performer sees virtual sheet music floating on the music stand through an AR headset, turns pages and annotates bars from a phone and tablet without taking hands off the instrument for long. - Technique: Combines an AR headset, a smartphone and a tablet in one networked system to place, page and annotate virtual scores in the physical environment. - Try it: Place a song's lyrics or chords as an AR panel beside a real instrument and let a second phone act as a page-turn pedal. Twist: let the 'conductor' phone highlight the bar everyone should be on. ### Christian Mio Loclair / Waltz Binaire *Dancer (Prince Mio), creative coder and founder of Waltz Binaire* Former popping battle dancer turned creative technologist who built Kinect-projector dance pieces and dance-game prototypes before founding the Berlin studio Waltz Binaire, whose Pathfinder came out of the Choreographic Coding Lab. #### KINECT | PROJECTOR DANCE — Christian Mio Loclair / Waltz Binaire (2012) - Video: https://www.youtube.com/watch?v=FnulH8TrZVo - Interaction: Projection, Hands & Body - Platform & tech: Projection, Kinect, Processing, projector calibration - Idea: A cheap calibration trick that turns any wall into a dancer's abstract mirror. - What it is: A demo of a lightweight Kinect-to-projector calibration in Processing, so a dancer's 3D movement is mirrored as abstract graphics on the wall they dance in front of. - Technique: A simple calibration maps Kinect depth coordinates onto projector pixels without OpenCV, so tracked joints line up with the projected image. - Try it: Calibrate a webcam to a projector by clicking four projected corners, then draw circles on tracked wrists that land on the real wrists. Twist: make the circles repel each other. #### WiiPop — Christian Mio Loclair / Waltz Binaire (2012) - Video: https://www.youtube.com/watch?v=erMi8kxYFIE - Interaction: Play, Hands & Body, Performance - Platform & tech: Desktop, Wearable, Kinect, Wii Remote, sensor fusion - Idea: A dance game that scores creativity rather than imitation. - What it is: A dance-game prototype that rewards freestyle instead of copying: it rates a popping dancer's spacing, musicality and use of individual body parts. - Technique: Kinect tracks large spatial movement while a body-mounted Wii Remote catches fine hits, and the two streams are fused into freestyle quality metrics. - Try it: Use webcam pose tracking to score a 30-second freestyle on variety (how many joints moved) and coverage (how much floor was used). Twist: repeating a move lowers the score. #### FLOW 1 — Christian Mio Loclair / Waltz Binaire (2013) - Video: https://www.youtube.com/watch?v=ISKV1BeB3pM - Interaction: Hands & Body, Projection, Performance - Platform & tech: Projection, Kinect, Processing, projector - Idea: Give the computer access to a street dancer's private flow. - What it is: A duet of hip-hop dance and interactive light where a dancer's flow is translated into projected shapes and trails that move with him on the wall. - Technique: Kinect depth tracking drives Processing graphics projected behind the dancer, calibrated so shapes attach to his silhouette. - Try it: Project a silhouette from a webcam segmentation model and add trails to the hands and feet that fade on the beat. Twist: the trails only appear on popping hits (sudden stops). #### Pathfinder — Christian Mio Loclair / Waltz Binaire, Motion Bank / Choreographic Coding Lab (2014) - Video: https://www.youtube.com/watch?v=8Uh0aK3ATS8 - Interaction: Performance, Perception & Effects, Drawing & Making - Platform & tech: Projection, Desktop, generative design, Processing - Idea: Reverse the usual pipeline: the graphics lead and the dancers follow. - What it is: A visual language that generates choreography: animated lines and paths on screen become instructions that dancers Raphael Hillebrand, Honji Wang and Christine Joy Alpuerto Ritter read and embody. - Technique: Generative path systems made with onformative are designed as a readable notation, so each visual behaviour (split, loop, spiral) maps to a movement task. - Try it: Write a p5.js sketch that draws one evolving line, show it on a projector, and ask a volunteer to dance whatever the line does. Twist: add a second line and see whether two dancers start to relate. #### Prince Mio improvise to Interactive Sound — Christian Mio Loclair / Waltz Binaire (2014) - Video: https://www.youtube.com/watch?v=RtUEML0ncJk - Interaction: Voice & Sound, Hands & Body, Performance - Platform & tech: Desktop, Kinect v2, generative sound - Idea: The dancer does not dance to the music; the music comes out of the dance. - What it is: A spontaneous rehearsal session for POW2045 in which the sound is completely attached to the dancer's body, so every pop and wave is heard as it happens. - Technique: A Kinect v2 depth camera tracks the body and maps joint motion to a sound collage by kling klang klong, with advice from Schnelle Bunte Bilder. - Try it: Map wrist speed to a filter cutoff and sudden stops to a drum hit using webcam pose tracking and Tone.js. Twist: only the left side of the body makes sound. #### Waltz Binaire — Dance and Interactive Media (Diesel Reboot) — Christian Mio Loclair / Waltz Binaire (2014) - Video: https://www.youtube.com/watch?v=TPDrSMfbaI0 - Interaction: Performance, Projection, Shared & Social - Platform & tech: Projection, Kinect, projector - Idea: A laptop, a Kinect and a projector are enough to make spontaneous dance visible. - What it is: A rehearsal of a piece for Diesel's Reboot campaign at Schauspielhaus Frankfurt where two dancers mixing urban and contemporary movement drive interactive graphics projected around them. - Technique: A low-weight setup of Kinect, notebook and projector evaluates the dancers' movement and renders interactive graphics designed to react to improvisation. - Try it: Pack a portable kit (laptop, webcam, pocket projector) and build a sketch that responds to two people at once, then perform it in a hallway. Twist: the graphics connect the two dancers with a line that breaks if they get too far apart. ### Craig Winslow *Light artist and projection designer* Designer and light artist known for Light Capsules, a series that uses projection mapping to temporarily restore faded 'ghost sign' advertisements painted on old buildings, made during an Adobe Creative Residency. #### Projecting West — Craig Winslow (2015) - Video: https://vimeo.com/138903302 - Interaction: Projection, Location & City - Platform & tech: Projection, portable projector, projection mapping - Idea: A road trip where each stop gets its own site-specific story in light. - What it is: On a 15-day road trip from Portland, Oregon to Portland, Maine, the team makes impulsive projection installations each night on barns, motels, rocks and roadside signs. - Technique: A portable projector and generator are set up quickly and content is mapped on site to whatever surface each town offers. - Try it: In teams, walk around campus for 30 minutes, pick an overlooked surface and make a 20-second projection about it using a laptop and pico projector. Twist: the content must use only what you learned from someone passing by. #### Light Capsule 004 — Astoria, Oregon — Craig Winslow (2016) - Video: https://vimeo.com/173006252 - Interaction: Projection, Location & City, Portals & Worlds - Platform & tech: Projection, projection mapping, archival research, vector redrawing - Idea: Use light to temporarily restore a city's forgotten layer of history. - What it is: A faded hand-painted advertisement on an old building in Astoria is brought back to life at night: projection redraws the missing letters and colors exactly on top of the ghost sign. - Technique: Archival photos and on-site research are used to redraw the sign as vector artwork, which is then projection-mapped and animated over the remaining paint. - Try it: Find an old sign, faded poster or worn floor marking, photograph it, redraw what it once looked like, and project it back on site. Twist: animate the moment the sign was first painted. #### Light Capsule 013 — Detroit x Miss Van — Craig Winslow (2016) - Video: https://vimeo.com/186274660 - Interaction: Projection, Perception & Effects, Location & City - Platform & tech: Projection, projection mapping, mural augmentation - Idea: Add a time layer to a mural: light animates what paint can only freeze. - What it is: In Detroit, a painted mural by street artist Miss Van and a neighboring ghost sign are augmented together with animated light that makes the painted figures glow and move. - Technique: The mural is photographed straight-on, animation is drawn on top of the photo and projection-mapped back so light hits exactly the painted shapes. - Try it: Choose a poster or drawing on the classroom wall, trace its shapes and animate one element (hair, water, eyes) with projected light. Twist: make the animation start only when someone stands in front of it. #### Light Capsules — London Design Festival — Craig Winslow (2016) - Video: https://vimeo.com/183522909 - Interaction: Projection, Location & City - Platform & tech: Projection, projection mapping, sign painting research - Idea: A tour of the city's ghost signs, each lit up for one night. - What it is: During the London Design Festival, five ghost signs across the city are revived with projection, drawing crowds to walls they had walked past for years. - Technique: Each sign is matched with historical sources, redrawn and projection-mapped, with animations that reveal the lettering stroke by stroke. - Try it: Map three locations on a campus walking route and design a projected reveal for each that connects into one story. Twist: publish a map so visitors can find them only at a set time. #### Light Capsules x Neon Museum — Craig Winslow (2017) - Video: https://vimeo.com/207339810 - Interaction: Projection, Perception & Effects, Tangible Objects - Platform & tech: Projection, projection mapping, neon simulation - Idea: Fake the light of a dead object so that it looks alive again. - What it is: At the Neon Museum boneyard in Las Vegas, broken vintage neon signs that no longer work are made to 'light up' again with carefully mapped projection that imitates glowing tubes. - Technique: Projection content is drawn to follow each tube path of the neon letters, with glow and flicker animated to mimic real neon. - Try it: Bend a word out of white wire or string on a board, then project a glowing, flickering neon animation precisely onto it. Twist: make one letter 'broken' and flicker on a random schedule. #### Light Sketch — El Cosmico — Craig Winslow (2017) - Video: https://vimeo.com/242350117 - Interaction: Projection, Tangible Objects - Platform & tech: Projection, projection mapping, masking - Idea: When the plan fails, map whatever you have with you. - What it is: With no generator for the planned piece, the team spontaneously projection-maps their silver Airstream trailer in the Marfa desert, turning it into a glowing, pulsing object under the stars. - Technique: A quick on-site mapping uses masks traced directly over the trailer's panels and rivet lines, running simple generative animation. - Try it: Give teams 45 minutes to map any object they carried to class (a bag, a bike, a helmet) and present it in a dark corner. Twist: the animation must reveal what the object 'dreams about'. ### Daniel Beauchamp (Pushmatrix) *Spatial-computing experimentalist; Head of AR/VR at Shopify* Principal AR/VR engineer at Shopify who posts short, often viral hand-tracking and mixed-reality experiments under the name Pushmatrix. #### AR glasses superpowers for illustrators — Daniel Beauchamp (Pushmatrix) (2021) - Video: https://x.com/pushmatrix/status/1364224471259697153 - Interaction: Drawing & Making, Tangible Objects - Platform & tech: Headset, Unity, hand tracking - Idea: Imagine the superpowers AR glasses could give illustrators: drawings on paper come to life. - What it is: A prototype imagining how AR glasses could let an illustrator or animator draw and see animated overlays directly on paper and the desk. - Technique: Likely combines a tracked sheet of paper (image or plane tracking) with animated overlays rendered in-place so drawings and animation guides align with the physical page. - Try it: Use image tracking to recognize a hand-drawn sketch and overlay a moving animation on the paper. Twist: let a character in the drawing walk off the page onto the desk. #### Live Blender preview on Vision Pro — Daniel Beauchamp (Pushmatrix) (2024) - Video: https://x.com/pushmatrix/status/1767898459024040088 - Interaction: Drawing & Making, Information & UI - Platform & tech: Headset, Desktop, Apple Vision Pro, Blender, RealityKit, USDZ - Idea: Edit a model in Blender and a life-size preview, lit by your real surroundings, appears instantly in Vision Pro. - What it is: A Blender scene streams live into Apple Vision Pro; edits made on the desktop appear instantly as a full-size object lit by your real room. - Technique: Blender exports the scene (for example as USDZ) on each change and streams it to the headset, where RealityKit reloads the model and lights it with the room's environment probe. - Try it: Write a script that exports glTF every time you save in Blender, and have a phone WebAR page reload the model automatically. Twist: play a 'growing' animation when the model appears in AR. #### Spatial Vacuuming — Daniel Beauchamp (Pushmatrix) (2024) - Video: https://x.com/pushmatrix/status/1749797146961006716 - Interaction: Spatial Mapping, Tangible Objects, Information & UI - Platform & tech: Headset, Meta Quest 3, passthrough, Unity - Idea: While vacuuming, mark the cleaned floor in real time so you never miss a corner. - What it is: A Quest controller strapped to a vacuum paints the floor as you clean in passthrough, so you can see which parts of the room you have already covered. - Technique: A tracked controller on the vacuum head writes its floor position into a render texture or grid, painting a coverage map over the floor plane in passthrough. - Try it: Strap a phone to a mop or broom, track its position with AR, and paint the floor plane to show the area already swept. Twist: add AR dust monsters that slowly grow in the corners you missed. #### Vision Pro pinball with pinch haptics — Daniel Beauchamp (Pushmatrix) (2024) - Video: https://x.com/pushmatrix/status/1762494365711417371 - Interaction: Hands & Body, Play - Platform & tech: Headset, visionOS, hand tracking - Idea: Use a finger pinch as the pinball button, so the body's own sense of touch becomes the feedback. - What it is: Pinball played on Apple Vision Pro using pinch gestures for the flippers; the tactile pinch of your own fingers stands in for button feedback. - Technique: visionOS hand tracking detects thumb–index pinch events and maps each hand's pinch to a flipper, relying on the self-contact of the fingers as passive haptic feedback. - Try it: Design a small AR game that uses only a pinch gesture, relying on the feel of your fingers touching as the feedback. Twist: let the left and right hands control different mechanics. #### Pinball table in your room (WebXR) — Daniel Beauchamp (Pushmatrix) (2025) - Video: https://x.com/pushmatrix/status/1994241272815485349 - Interaction: Play, Spatial Mapping - Platform & tech: Headset, Web, WebXR, react-three/xr, three.js - Idea: Turn a web pinball table into an AR version sitting in your room with one step. - What it is: A web pinball game that also runs in AR on a headset: the table appears in your real space, wired up with about 100 lines of react-three/xr. - Technique: The existing three.js scene is wrapped in a WebXR AR session via react-three/xr, with the table anchored to a hit-tested floor or table surface and controller/hand input mapped to flippers. - Try it: Convert a small 3D web game you made before into an AR version with WebXR and play it on a real tabletop. Twist: make the real edges of the table act as walls in the game. ### David Bowen *Artist; professor at the University of Minnesota Duluth* American artist who connects robotic devices to natural systems — wind, waves, clouds, houseflies, plants — so that remote or invisible forces drive machines in the gallery. #### growth modeling device — David Bowen (2009) - Video: https://vimeo.com/20966301 - Interaction: Spatial Mapping, Information & UI, Perception & Effects - Platform & tech: Desktop, laser scanner, 3D printer, conveyor - Idea: Sampling a living thing at intervals and freezing each sample in space turns time into a line of objects — the core of any AR time-lapse trail. - What it is: Every 24 hours a laser scans an onion plant and a 3D printer builds a plastic model of it, then a conveyor moves on, leaving a row of models that record the plant's growth. - Technique: A laser line scanner captures the plant from one of three angles each day and a fused-deposition 3D printer outputs the mesh in real time, after which a conveyor advances about 17 inches. - Try it: Scan a plant, a sleeping pet or a growing pile of dishes with a LiDAR phone every 15 minutes and place the meshes side by side in AR as a physical timeline. Twist: stack them in one spot with fading transparency instead of in a row. #### tele-present wind — David Bowen (2009) - Video: https://vimeo.com/20963294 - Interaction: Location & City, Information & UI, Perception & Effects - Platform & tech: Desktop, accelerometer, servo gimbals, Arduino, networked data - Idea: Invisible wind from somewhere else becomes visible indoors by copying its motion onto many objects — a model for AR scenes driven by live remote data. - What it is: A dried plant stalk outdoors sways in the wind, and 42 stalks mounted on tilting robots inside the gallery sway in exact unison with it in real time. - Technique: An accelerometer on the outdoor stalk streams x/y tilt to an array of servo-driven gimbals, each holding a real stalk, so the gallery field mirrors the wind with a small network delay. - Try it: Stream a phone's accelerometer from a window ledge to a second phone that shows a field of virtual grass in AR, all bending with the real wind. Twist: route the data from a friend's city so the room shows someone else's weather. #### tele-present water — David Bowen (2011) - Video: https://vimeo.com/27614383 - Interaction: Location & City, Information & UI, Spatial Mapping - Platform & tech: Desktop, motors, buoy sensor data, custom software - Idea: A remote sea can be rebuilt as a physical mesh in a room; AR can do the same with a virtual surface anchored to the floor. - What it is: A large grid of wooden and wire segments hangs from motors and heaves up and down like the surface of a lake, replaying wave data recorded by a buoy on Lake Superior. - Technique: Wave intensity and acceleration from a buoy are scaled and sent to a row of motors on a ceiling grid that lift and drop the mesh points, turning a data stream into a moving topography. - Try it: Anchor a wire-mesh plane on the classroom floor in AR and animate its vertices from a public ocean-buoy data feed or a recorded CSV. Twist: let viewers stand under it and see the mesh from below like being underwater. #### fly revolver — David Bowen (2013) - Video: https://vimeo.com/71826503 - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Desktop, computer vision, robotic pan-tilt, houseflies - Idea: Handing control to an unpredictable living agent creates tension no script can — AR characters can borrow behaviour from real creatures tracked by the camera. - What it is: Houseflies live inside an acrylic sphere in front of a target; a camera tracks where they are and a robotic mount aims a revolver at that spot in real time, triggering it when a single fly is detected. - Technique: Video tracking finds the flies' positions on the target backdrop and custom software sends pan/tilt angles to a robotic device holding the (unloaded, dry-fired) revolver. - Try it: Track a moving object in the camera feed (a fish, a ball, a pet) and make a virtual spotlight or creature in AR follow it and react when it stops. Twist: let the tracked animal 'choose' which of several AR buttons to press. #### cloud piano — David Bowen (2014) - Video: https://vimeo.com/101857804 - Interaction: Voice & Sound, Location & City, Perception & Effects - Platform & tech: Desktop, camera, custom software, robotic actuators, piano - Idea: Reading the sky as a score shows that any live view can become an instrument — a direct pattern for camera-driven AR sound. - What it is: A camera looks at the sky and a robot presses the keys of a real piano according to the shapes of the clouds passing over, so the clouds seem to play music. - Technique: Custom software maps cloud edges in the live video to key positions across the keyboard and drives a row of robotic fingers that press them. - Try it: Point a phone at the sky, threshold the clouds, and let a virtual keyboard floating in AR play the notes that the cloud edges cross. Twist: aim it at a crowd or traffic instead of clouds. #### plant drone — David Bowen (2022) - Video: https://vimeo.com/709247945 - Interaction: Drawing & Making, Location & City, Perception & Effects - Platform & tech: Desktop, drone, plant sensors, LED, long-exposure photography - Idea: Letting a non-human pilot draw in the sky and capturing the path with long exposure is a ready-made recipe for AR light trails. - What it is: A drone carries a live plant whose leaf signals decide the flight path; an LED on the drone and an open camera shutter on the ground turn the plant's piloting into long-exposure drawings in the night sky. - Technique: Variable-resistance readings from each leaf are mapped to the drone's roll, pitch and altitude, while a long-exposure photograph records the LED path. - Try it: Attach a virtual light to a moving real object (a toy car, your hand, a friend walking) with AR tracking and let it leave a persistent glowing trail in the room. Twist: let a random or plant-like signal steer a virtual drone instead of a person. ### David Lindlbauer *Assistant Professor, Carnegie Mellon HCII; leads the Augmented Perception Lab* David Lindlbauer works on mixed reality that adapts to people and context, from Remixed Reality (with Andy Wilson), which edits space and time in a live 3D copy of the room, to optimization-driven MR interfaces at ETH Zurich and CMU. #### Combining Shape-Changing Interfaces and Spatial AR — David Lindlbauer (2016) - Video: https://www.youtube.com/watch?v=fWREdKL2Kus - Interaction: Projection, Tangible Objects, Perception & Effects - Platform & tech: Projection, projector, actuated objects - Idea: Projection plus a shape-changing object makes the object seem to reach past its own edges. - What it is: Projection and shape-changing objects work together so a physical object's appearance can extend and move beyond its real shape. - Technique: Actuated objects are tracked and calibrated to a projector so that projected imagery extends the object's visible form beyond its physical shape and stays registered while it moves. - Try it: Do projection mapping on a servo-rotated paper box so the projection "extends" the box into an animal whose projected legs move when the box moves. Twist: deliberately misalign the projection and the physical motion to create an uncanny effect. #### Changing the Appearance of Real-World Objects by Modifying Their Surroundings — David Lindlbauer (2017) - Video: https://www.youtube.com/watch?v=2gez_joXaiE - Interaction: Projection, Perception & Effects - Platform & tech: Projection, projector, perceptual illusions - Idea: Leave the object alone and change only the light around it, and it seems to change color. - What it is: Projection changes the environment around a real object, exploiting color and brightness perception so the object itself appears to change. - Technique: Rather than projecting onto the object, the projector alters the color and brightness of the surrounding area, exploiting simultaneous contrast and lightness constancy so the unprojected object is perceived as changing. - Try it: Put a gray paper card on a table, project backgrounds of different colors and brightness only around it, and ask viewers whether the card's color has changed. Twist: use the illusion to give a still object the feeling of a heartbeat or breathing. #### Remixed Reality — David Lindlbauer, Microsoft Research — Hrvoje Benko & Andy Wilson (2018) - Video: https://www.youtube.com/watch?v=GoSQTPfrdCc - Interaction: Portals & Worlds, Spatial Mapping, Perception & Effects - Platform & tech: Headset, Kinect array, VR HMD, 3D reconstruction - Idea: 'Edit' reality as raw footage: copy, erase and rewind the real world around you. - What it is: A live 3D reconstruction of the room is shown in a VR headset and edited in real time: copy or erase objects and people, rewind time, or change geometry of the space. - Technique: An array of calibrated Kinects produces a live 3D reconstruction of the room that is rendered in a VR headset, so any region of the point cloud can be copied, deleted, recolored, or buffered and replayed in time. - Try it: Use the live camera feed of phone AR with a segmentation model (such as MediaPipe or a web version of SAM) to select a person or object in the frame and 'erase' it (inpainting the background) or duplicate it. Twist: add a 'time slider' so you can look back at what this spot looked like 10 seconds ago. #### Context-Aware Online Adaptation of MR Interfaces — David Lindlbauer (2019) - Video: https://www.youtube.com/watch?v=SAmmTUw96Mg - Interaction: Information & UI, Gaze & Attention - Platform & tech: Headset, HoloLens, combinatorial optimization - Idea: An MR interface that seems to read you, deciding how much to show and where based on what you are doing. - What it is: An optimizer continuously decides which virtual apps to show, where and how detailed, based on the user's current task and cognitive load. - Technique: An integer linear program decides for each app whether it is shown, where it is placed and at what level of detail, given estimated cognitive load from the task and a utility model, and re-solves as context changes. - Try it: Place five virtual info panels (time, messages, weather, navigation, music) in phone AR and write a simple rule or scoring function that hides or shrinks them depending on whether the user is walking, talking or typing. Twist: let the user 'bribe' the system to keep a panel, and discuss who holds control. #### RealityReplay — David Lindlbauer (2023) - Video: https://www.youtube.com/watch?v=wCrh6PpyqUY - Interaction: Spatial Mapping, Perception & Effects, Information & UI - Platform & tech: Headset, mixed reality, change detection - Idea: Replay the changes you missed right where they happened, so you can see what just took place. - What it is: MR detects changes that happened in the room while you were not looking and replays them in place, e.g. showing who moved an object. - Technique: The MR system keeps a 3D model of the room, detects geometric and appearance changes that happened outside the user's field of view, and replays the recorded change events in place as animated reconstructions. - Try it: Film a tabletop with a phone camera fixed in a corner, detect the moment an object is moved using frame differencing, and replay that clip in AR overlaid at its original location. Twist: only tell users 'something changed here', let them guess what happened, then reveal it. #### MiniMates — David Lindlbauer (2025) - Video: https://www.youtube.com/watch?v=fukfCSvmo44 - Interaction: Shared & Social, Spatial Mapping - Platform & tech: Headset, AR, avatars - Idea: Shrink remote colleagues into tiny figures on your desk, so even a small room can host an AR meeting. - What it is: Remote meeting participants appear as miniature avatars that fit on your desk, so AR meetings work even in small physical spaces. - Technique: Remote participants are represented as scaled-down avatars, and a layout algorithm fits them onto available tabletop area while retargeting their gaze and pointing toward the corresponding local objects. - Try it: Detect a tabletop in AR Foundation, put the video faces of three remote classmates on mini avatars arranged across the table, and turn the faces toward whoever is speaking. Twist: make each avatar's size change with how long or how loudly that person speaks. ### Dilmer Valecillos *XR developer and educator (Learn XR)* XR developer who teaches AR Foundation, Quest and VFX Graph on YouTube and shares every tutorial project on GitHub. #### AR Body Tracking with Head and Hand Particles — Dilmer Valecillos (2019) - Video: https://www.youtube.com/watch?v=jxvBrfuyusU - Source code: https://github.com/dilmerv/UnityARFoundationEssentials - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, Unity, AR Foundation, ARKit - Idea: Give real people superpowers by attaching effects to their tracked joints. - What it is: On an iPhone, a person's tracked head and hands emit fire and particle effects that follow them as they move in the real room. - Technique: AR Foundation's ARKit 3 human body tracking provides joint transforms, and particle emitters are parented to the head and hand joints each frame. - Try it: Attach VFX Graph emitters to AR body-tracking joints and design a superpower for each classmate. Twist: the effect grows stronger the faster the joint moves. #### VFX Graph Sun with 2 Million Particles — Dilmer Valecillos (2019) - Video: https://www.youtube.com/watch?v=f1BHXqeokSE - Source code: https://github.com/dilmerv/UnityVFXMillionsOfParticles - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, HDRP - Idea: Millions of tiny points are enough to build a star. - What it is: A burning sun made of two million particles churns and flares in real time, built step by step in VFX Graph. - Technique: VFX Graph spawns particles on a sphere and moves them with turbulence and conform-to-sphere forces, colouring them by lifetime with an HDR gradient. - Try it: Put the particle sun on a real table in phone AR and scale it with a pinch. Twist: planets made of fewer particles orbit around the user's head. #### Hand-Tracked Forces on VFX Particles (Quest) — Dilmer Valecillos (2020) - Video: https://www.youtube.com/watch?v=EyMF2Wo1awo - Source code: https://github.com/dilmerv/OculusQuestHandTrackingPhysicsURP - Interaction: Hands & Body, Perception & Effects - Platform & tech: Headset, Unity, VFX Graph, URP, Oculus Quest - Idea: Your hands become force fields for particles. - What it is: In VR on an Oculus Quest, the user's bare hands push, pull and scatter a cloud of VFX Graph particles. - Technique: Oculus hand-tracking bone positions are passed to VFX Graph as exposed properties and used as attractors and colliders, rendered with URP on Quest. - Try it: Bring the hand-force particles into passthrough MR on Quest 3 or HoloKit so students sculpt a particle cloud floating over a real table. Twist: two people's hands tear the cloud in opposite directions. #### AI Building Blocks: multimodal LLMs in mixed reality — Dilmer Valecillos (2026) - Video: https://www.youtube.com/watch?v=Q8BFLkRYOy0 - Source code: https://github.com/dilmerv/AIBuildingBlocksDemos - Interaction: Voice & Sound, Information & UI - Platform & tech: Headset, Meta Quest 3, Meta XR SDK v83, LLM, Unity - Idea: Ask your room a question and let the model see it before answering. - What it is: Conversational AI is added to a mixed-reality app with multimodal prompts that combine speech, images and video from the headset, for context-aware answers about the user's surroundings. - Technique: Unity building blocks wrap speech-to-text, a multimodal LLM call with passthrough frames, and text-to-speech, so the reply is spoken and shown in space. - Try it: Chain speech-to-text, a vision LLM with one camera frame, and text-to-speech in a phone web page, and use it as a museum guide for objects on a shelf. Twist: the guide must answer as the object itself. #### AI Building Blocks: object detection in mixed reality — Dilmer Valecillos (2026) - Video: https://www.youtube.com/watch?v=QIa6frPMcSQ - Source code: https://github.com/dilmerv/AIBuildingBlocksDemos - Interaction: Tangible Objects, Information & UI - Platform & tech: Headset, Meta Quest 3, Meta XR SDK AI Building Blocks, Unity - Idea: Drop-in blocks make the headset name what it sees. - What it is: The first of a series on Meta's AI Building Blocks shows real objects in the room detected and labelled in mixed reality, running either on-device or in the cloud. - Technique: Passthrough camera frames go to an object-detection model (on-device or cloud), and boxes are projected into 3D using the environment depth to anchor labels. - Try it: Use Unity with a ready-made AI building block or a WebAR detection demo to label five classroom objects, then replace each label with a fun fact. Twist: only reveal a label when two people look at the same object. #### Arcade Hoops: a mixed-reality game built with AI in 30 days — Dilmer Valecillos (2026) - Video: https://www.youtube.com/watch?v=eS0f7W6Y6cE - Interaction: Play, Spatial Mapping - Platform & tech: Headset, Unity MCP, Meta Quest, agentic coding - Idea: One developer plus coding agents can ship a mixed-reality game in a month. - What it is: A walkthrough of building and shipping a full mixed-reality basketball game to the Meta Horizon Store with AI agents, including net physics, tabletop play and a feature that lets players smash their real walls and ceiling. - Technique: Unity MCP and Meta's agentic tools let an LLM create scene objects, write C# and wire up the room mesh, with the developer prompting and reviewing each feature. - Try it: In one class session, pair-program a tiny WebXR throwing game with an AI assistant, keeping a log of every prompt. Twist: swap logs with another team and try to rebuild their game from prompts alone. ### Dong Yoon Park (Yoon Park) *Mixed reality designer and developer (HoloLens, MRTK, Meta Quest)* Designer and developer who worked for years on Microsoft's Mixed Reality team on HoloLens 2 interactions and MRTK building blocks. His personal apps Typography Insight and Type In Space treat type as physical objects you walk around, and his later Quest app Cosmic XR brings the solar system into the room. #### Holographic Type Sculpture — Dong Yoon Park (Yoon Park) (2016) - Video: https://www.youtube.com/watch?v=sevWuuHYXXc - Interaction: Drawing & Making, Location & City, Spatial Mapping - Platform & tech: Headset, HoloLens, spatial anchors, Unity - Idea: An invisible artwork added to a real museum: AR as digital graffiti in the gallery. - What it is: A tall sculpture of floating words and letterforms placed with HoloLens in the atrium and rooftop sculpture garden of the Bellevue Arts Museum; it stands among the physical artworks but is visible only through the headset, and each piece of type can be moved or rotated. - Technique: Built with the author's Typography Insight app for HoloLens: 3D text objects are placed and anchored with HoloLens spatial mapping and world anchors, and recorded with Mixed Reality Capture. - Try it: Use persistent anchors in AR Foundation or WebXR to place a stack of 3D words as a 'sculpture' in a public corner of campus that only phone viewers can see. Twist: every visitor can add one word, so the sculpture grows over the week. #### Typography Insight for HoloLens — Dong Yoon Park (Yoon Park) (2016) - Video: https://www.youtube.com/watch?v=eTI6NBanGkA - Interaction: Information & UI, Spatial Mapping, Gaze & Attention - Platform & tech: Headset, HoloLens, Unity, typography - Idea: Learn typography by walking around letters as if they were objects. - What it is: A HoloLens version of the typography learning app Typography Insight, with a 'Type Playground' where people place words in the real room in different typefaces, styles and sizes and walk around them to compare letterforms. - Technique: 3D text meshes generated from font outlines are placed with gaze and air-tap, anchored to the room through HoloLens spatial mapping. - Try it: Build a phone AR app that places the same word in three typefaces side by side on a table so students can compare them from all angles. Twist: letters grow a shadow that reveals each font's stroke contrast. #### Type In Space — Dong Yoon Park (Yoon Park) (2018) - Video: https://www.youtube.com/watch?v=qJyr4C6Weck - Interaction: Drawing & Making, Spatial Mapping, Hands & Body - Platform & tech: Headset, HoloLens, MRTK, spatial mapping - Idea: Design signage where it will live, at real scale, before it is printed. - What it is: A HoloLens app for laying out and experiencing type in the physical environment: two-handed gestures scale and rotate text, and snap-to-surface places lettering on real walls and tables, for prototyping signage, book covers or logotypes. - Technique: Uses HoloLens spatial mapping for surface snapping and two-handed manipulation from the Mixed Reality Toolkit to move, scale and rotate text objects. - Try it: Make a phone AR tool that snaps a line of text onto a detected wall at true scale so students can test a door sign or poster title in place. Twist: the text reflows automatically when the wall is too narrow. #### Type In Space for HoloLens 2 — Dong Yoon Park (Yoon Park) (2019) - Video: https://www.youtube.com/watch?v=tRoDAdLhO3I - Interaction: Hands & Body, Drawing & Making, Spatial Mapping - Platform & tech: Headset, HoloLens 2, MRTK, hand tracking - Idea: Touch type directly: letters become objects you can grab with your hands. - What it is: The HoloLens 2 redesign of Type In Space: fully articulated hand tracking lets people grab, stretch and rotate words directly with their fingers and pick fonts and colours from menus that follow the hand. - Technique: Built on MRTK with HoloLens 2 articulated hand tracking for near-field direct manipulation, bounding-box handles and hand-attached menus. - Try it: With hand tracking on Quest or Vision Pro (or MediaPipe Hands on a webcam), let students pinch and stretch a floating word to set its size and weight. Twist: squeezing a word makes the typeface bolder. #### COVID-19 data visualization with HoloLens 2 — Dong Yoon Park (Yoon Park) (2020) - Video: https://www.youtube.com/watch?v=zBkNlpGHwZQ - Interaction: Information & UI, Hands & Body - Platform & tech: Headset, HoloLens 2, MRTK, Unity, data visualization - Idea: Put a global dataset on a globe you can hold in your hands. - What it is: An open-source HoloLens 2 app that shows COVID-19 case numbers as volumetric bars on a 3D globe floating in the room, which people rotate and inspect directly with their hands. - Technique: Case data from ESRI's COVID-19 feature layer is mapped to bars on a globe mesh, with MRTK hand interactions for rotating, scaling and selecting countries. - Try it: Load a small open dataset (for example, air quality by city) into a phone AR globe on a table, with bar height per city and tap-to-read labels. Twist: scrub time by walking around the globe. #### Cosmic XR: lighting on physical surfaces — Dong Yoon Park (Yoon Park) (2026) - Video: https://www.youtube.com/watch?v=mzP3TTdPhNs - Interaction: Spatial Mapping, Perception & Effects, Information & UI - Platform & tech: Headset, Meta Quest, scene understanding, passthrough, Unity - Idea: Virtual objects feel present when their light lands on real surfaces. - What it is: An experiment in the Quest app Cosmic XR in which the glowing sun and planets placed in a living room cast light and reflections onto the real walls, floor and furniture. - Technique: Quest scene understanding supplies a mesh of the room; virtual point lights are rendered onto that invisible mesh and blended over passthrough to fake illumination on physical surfaces. - Try it: In AR Foundation with meshing (LiDAR iPhone) or Quest scene mesh, add a glowing virtual object and render its light onto an invisible occlusion mesh so the real table appears lit. Twist: the light colour follows the music playing in the room. ### Dpt. *Montreal interactive studio for installations, AR and mixed-reality experiences* Montreal creative studio that started with openFrameworks projection-mapping experiments and now makes interactive installations, AR journeys and generative-AI prototypes for museums, festivals and brands. #### Kinetic projection mapping — Dpt. (2013) - Video: https://vimeo.com/83618926 - Interaction: Projection, Tangible Objects, Spatial Mapping - Platform & tech: Projection, openFrameworks, mapamok - Idea: Projection mapping that stays glued to a moving object. - What it is: Graphics are projected onto a rotating icosahedron and stay locked to its faces as it spins. - Technique: The object's rotation angle is known in software, and a modified version of Kyle McDonald's openFrameworks tool mapamok re-renders the 3D model from the projector's calibrated viewpoint every frame. - Try it: Put a white box on a lazy Susan, measure its rotation with a phone gyroscope, and keep a projected pattern on each face as it turns. Twist: let each face show a different character who looks at the viewer. #### Réflexions — Dpt. (2013) - Video: https://vimeo.com/83617391 - Interaction: Hands & Body, Voice & Sound, Perception & Effects - Platform & tech: Desktop, openFrameworks - Idea: A mirror that listens and deliberately gets you wrong. - What it is: A digital mirror shows each visitor as a digital twin that reacts to how they move and how loudly they speak, never reproducing them faithfully. - Technique: A camera and microphone feed body silhouette and sound level into generative graphics, likely built with openFrameworks. - Try it: Build a webcam mirror in p5.js or Lens Studio where your outline is drawn with particles whose size follows your voice volume. Twist: when you are silent, the mirror image slowly walks away. #### Parade — Dpt. (2014) - Video: https://vimeo.com/96615251 - Interaction: Projection, Tangible Objects, Perception & Effects - Platform & tech: Projection, openFrameworks, motion tracking - Idea: Push the lamp and the shadows start to dance. - What it is: Inside a wooden cube a work lamp hangs over two slumped porcelain vases; when a visitor pushes the lamp, the vases' shadows on the wall come alive and dance in rhythm with the swinging light. - Technique: The lamp's swing is motion-tracked, and projected shadow animations are rendered in real-time 3D so they match the position of the real light source (with ceramicist Laurent Craste). - Try it: Place a real object under a lamp, track the lamp with a phone or webcam, and project a fake shadow that follows the light but moves on its own. Twist: the shadow reacts differently when the lamp swings fast. #### When the Crowd Becomes Circus — Dpt. (2017) - Video: https://vimeo.com/231449288 - Interaction: Shared & Social, Performance, Location & City - Platform & tech: Phone, ARKit, ARCore, WebSockets - Idea: The crowd's phones become both a remote control for the show and a window onto a miniature street circus. - What it is: A festival app lets the audience influence a live circus show by shaking, tapping or shouting into their phones, and its AR mode places tiny circus performers on the street. - Technique: A real-time server collects phone sensor input from the crowd and can push cues back to the phones' screens, speakers and flash, while an ARKit/ARCore-style mode anchors animated performers on the ground. - Try it: Make a web app that turns classmates' phone shakes into a shared meter that changes a projected performer's action. Twist: at the end, the projection takes over everyone's phone flashlights for a finale. #### Human Scales (Échelles Humaines) — Dpt. (2021) - Video: https://vimeo.com/653004912 - Interaction: Location & City, Information & UI - Platform & tech: Phone, photogrammetry, mobile AR - Idea: Put giant statues of ordinary neighbours in public space, where only monuments to famous people usually stand. - What it is: An AR walk through Montreal neighbourhoods where monumental virtual statues of local unsung heroes stand in parks and streets, each with a podcast telling their story. - Technique: People were 3D-scanned, turned into stylized sculptures and anchored at geolocated sites in a mobile AR app with accompanying audio. - Try it: 3D-scan a classmate with a phone and place a giant version of them at a campus spot in AR, with a 30-second recorded story. Twist: the statue's pose must be chosen by the person it honours. #### AR with a magnifying glass — Dpt. (2024) - Video: https://vimeo.com/973282394 - Interaction: Tangible Objects, Portals & Worlds, Perception & Effects - Platform & tech: Desktop, StreamDiffusion, computer vision - Idea: A magnifying glass that reveals imaginary worlds inside ordinary things. - What it is: A real magnifying glass is moved over a table of drawings and objects; only the area inside the lens is transformed on screen by generative AI into detailed landscapes and hidden worlds. - Technique: The lens is tracked in real time and only its crop of the camera image is fed to a real-time diffusion model (StreamDiffusion-style), whose output is composited back inside the lens outline. - Try it: Track a cardboard ring with a webcam and apply a strong filter or AI style only inside the ring. Twist: what appears inside depends on the colour of the object under it. ### Dynamicland — Bret Victor & collaborators *Nonprofit research lab in Oakland/Berkeley led by Bret Victor with Luke Iannini, Toby Schachman, Josh Horowitz, Omar Rizwan, Paula Te and others* Dynamicland grew out of Bret Victor's Communications Design Group (HARC). Its Realtalk system uses ceiling cameras and projectors so that ordinary paper, cards and objects become live, programmable computing, shared by everyone in the room. #### Hypercard in the World — Dynamicland — Bret Victor & collaborators (2016) - Video: https://www.youtube.com/watch?v=uI7J3II59lc - Interaction: Projection, Tangible Objects, Shared & Social - Platform & tech: Projection, Realtalk, projectors, cameras - Idea: Turn paper cards into programs so the whole room becomes a computer people program together. - What it is: The precursor to Dynamicland at HARC/CDG: paper cards on tables and walls become live programs through projection and cameras, and people program the room together. - Technique: Ceiling-mounted cameras identify paper cards by printed color-dot codes, and projectors render each card's program output onto and around it, with programs reacting to the positions and adjacency of other cards. - Try it: Print ArUco codes on a few cards, recognize them with an overhead camera and OpenCV, and use a projector to show each card's output (a color, a line of text), triggering a combined effect when two cards come close. Twist: have classmates use the cards as "program statements" to assemble a small animation. #### La Tabla — Dynamicland — Bret Victor & collaborators (2018) - Video: https://www.youtube.com/watch?v=VTHvNasQyu8 - Interaction: Projection, Play, Tangible Objects - Platform & tech: Projection, Realtalk, projection - Idea: A 'programmable' game played on a projection table with real cards and pieces. - What it is: A game table in Dynamicland where physical game pieces and paper cards drive projected gameplay, shown at GDC's Experimental Gameplay Workshop. - Technique: Realtalk's camera recognizes game pieces and paper cards by printed dot codes, and projection renders gameplay whose rules are themselves printed programs lying on the table. - Try it: Build a board game with top-down projection and ArUco markers: when a piece moves, the projection shows its path and effects, and swapping the 'rule card' at the table's edge changes the rules. Twist: let players change the rule cards mid-game. #### Dynamicland (progress report) — Dynamicland — Bret Victor & collaborators (2020) - Video: https://www.youtube.com/watch?v=x8-7E0IT5K0 - Interaction: Projection, Tangible Objects, Shared & Social - Platform & tech: Projection, Realtalk - Idea: A computer the size of a building with no screens: code on paper, projections on tables, and people programming together around them. - What it is: A tour of the Oakland Dynamicland space: pages of code on paper, projected visualizations, tangible programming kits and collaborative projects. - Technique: Every program is a printed page identified by dot codes, a room-wide camera-projector system (Realtalk) tracks pages and objects and runs their code, so programs communicate by statements about physical positions. - Try it: Build a programmable table with a projector and a camera: print 5 marker-tagged sheets, each standing for a function such as draw a circle, turn red or move right, and apply a sheet's effect to another sheet when you point it at that sheet. Twist: design a meta-program sheet that can read the contents of other sheets. #### Biomolecular design in Realtalk — Dynamicland — Bret Victor & collaborators (2022) - Video: https://www.youtube.com/watch?v=GmY_BrwWnCA - Interaction: Projection, Information & UI, Shared & Social - Platform & tech: Projection, Realtalk - Idea: Scientists gather around a table and design protein molecules together with paper and physical objects. - What it is: Scientists design proteins together on a Realtalk table, manipulating printed pages and physical controls that drive projected molecular models. - Technique: Printed pages and physical controls such as knobs are tracked by Realtalk's camera system, and their state feeds a molecular modeling program whose output is projected onto the table for the group to manipulate together. - Try it: Show a molecule or data model with a top-down projector and use marker-tagged paper strips as parameter knobs whose rotation the camera reads, so several people can adjust different parameters at once. Twist: swap in a complex model from your own field and see whether making it physical helps the discussion. #### Improvising cellular playgrounds in Realtalk — Dynamicland — Bret Victor & collaborators (2023) - Video: https://www.youtube.com/watch?v=nT4E5HkpLjo - Interaction: Projection, Drawing & Making, Tangible Objects - Platform & tech: Projection, Realtalk - Idea: Improvise an evolving cellular-automaton playground from paper and markers. - What it is: Live-improvised cellular automata and simulations built from paper and markers on a projected table, edited by hand in real time. - Technique: A camera reads marker-drawn cells and rule pages on paper, and a projector renders cellular automaton simulations directly over the drawings, so editing the paper edits the simulation live. - Try it: Use a camera to read a black-and-white grid drawn on paper as the initial state, run Conway's Game of Life in a projection overlaid on the paper, and have erased or added cells take effect immediately. Twist: invent a new rule and write it on a rule card, so swapping cards swaps rules. #### Dynamicland intro — Dynamicland — Bret Victor & collaborators (2024) - Video: https://www.youtube.com/watch?v=5Q9r-AEzRMA - Interaction: Projection, Tangible Objects, Shared & Social - Platform & tech: Projection, Realtalk - Idea: Every program is a physical object in the room that you can see and touch. - What it is: An introduction to the current Dynamicland: communal, physical computing where every program is a visible, touchable object in the room. - Technique: A room-scale projector-camera system recognizes printed objects and runs Realtalk programs that are physically embodied on paper, making code, data and output co-located and visible to everyone in the space. - Try it: Use a projector and a camera to turn a classroom wall into a 'public program wall', where anyone can change a shared animation by sticking a marked sheet of paper on it. Twist: require all state to be visible, and ban any hidden screen or keyboard. ### Es Devlin *Artist and stage designer* British artist and stage designer known for rotating sculptures, mirror mazes and light-and-text installations for public spaces, and for stage sets for opera, the Olympics, Beyoncé and The Weeknd. #### Mirror Maze — Es Devlin (2016) - Video: https://www.youtube.com/watch?v=IMbLlN_6AD8 - Interaction: Perception & Effects, Portals & Worlds, Spatial Mapping - Platform & tech: Projection, mirrors, video, scent - Idea: A maze of mirrors hides where the real walls are, so projected content seems to float everywhere; AR can use the same confusion between real and reflected space. - What it is: In a Peckham warehouse, visitors wander a maze of mirrors where video, light and scent appear and disappear, a 'video sculpture' made for Chanel's The Fifth Sense with i-D. - Technique: Angled mirror panels form corridors that multiply the visitor and hidden screens, while scent and sound are cued along the path (exact screen layout likely). - Try it: Build a WebXR maze of virtual mirror panels in a real room, using render textures so each panel reflects the others and the user's camera. Twist: hide one 'true exit' panel that shows the real camera feed instead of a reflection. #### Please Feed the Lions — Es Devlin (2018) - Video: https://www.youtube.com/watch?v=SpZ-k9woZ14 - Interaction: Voice & Sound, Location & City, Shared & Social - Platform & tech: Projection, projection mapping, machine learning, fibreglass sculpture - Idea: Giving a monument a voice that answers the public turns a statue into a conversation; AR layers on landmarks work best when they respond to what people bring. - What it is: A fifth, fluorescent-red lion joins the four bronze lions in London's Trafalgar Square; the public types a word into it and the lion roars out a line of poetry that is also projected into its open mouth. - Technique: A text-generation model trained on 19th-century poetry, likely a Google-built system, takes each submitted word and composes a couplet that is spoken and projection-mapped onto the lion. - Try it: Anchor an AR creature to a real statue or landmark with image or location tracking; users type one word and the creature speaks a short generated poem with speech synthesis. Twist: every poem stays attached to the statue, so later visitors hear a chorus of all previous words. #### Forest of Us — Es Devlin (2021) - Video: https://www.youtube.com/watch?v=9n-hBp3th4E - Interaction: Perception & Effects, Portals & Worlds, Voice & Sound - Platform & tech: Desktop, mirrors, soundscape - Idea: One shape seen from the inside (a lung as a forest) can reframe a whole topic; AR scale shifts, putting the user inside a body or a tree, teach through the body. - What it is: At Superblue Miami, visitors walk through a mirror maze shaped like a bronchial tree, where narration and sound draw the likeness between the branches of trees and the airways of our lungs. - Technique: Mirrored walls follow a branching plan, so reflections multiply the branches, while a soundscape and spoken text guide the route. - Try it: Model a branching tube (lung or tree) in Blender, scale it to 10 m and place it in a gym or courtyard in phone AR so students walk its corridors. Twist: sync the tube's glow to the user's breathing, measured with the microphone. #### Poem Pavilion (UK Pavilion, Expo 2020 Dubai) — Es Devlin (2021) - Video: https://www.youtube.com/watch?v=91RiJ1vNMmg - Interaction: Shared & Social, Information & UI, Location & City - Platform & tech: Projection, LED facade, machine learning, timber - Idea: A building that writes with its visitors' words is a public, readable record of who came; shared AR text anchored on architecture can do the same at any scale. - What it is: The UK Pavilion at Expo 2020 Dubai is a cone of wooden slats whose front face is a giant LED screen; every visitor adds one word and a machine-learning model turns all the words into a collective poem shown on the facade. - Technique: Visitor words are collected at kiosks and fed to a language model, likely trained on a poetry corpus, whose output lines scroll across LED strips set into the slats. - Try it: Build a Lens Studio or WebXR 'poem wall' on a real school facade: each visitor enters one word, and the app recombines all words into a short poem shown as floating text on the wall. Twist: words from people standing closer to the wall appear larger. #### Come Home Again — Es Devlin (2022) - Video: https://www.youtube.com/watch?v=p5rGKOhkwIY - Interaction: Location & City, Voice & Sound, Information & UI - Platform & tech: Projection, drawn species, light, choir - Idea: Drawing hundreds of named species on one lit object makes an abstract list feel like a congregation; AR can turn data about a place into figures that surround the visitor. - What it is: In the garden of Tate Modern, a large sphere is drawn with 243 species from London's priority list of endangered animals and plants; at dusk it lights up and a choir sings from inside it. - Technique: Hand-drawn species are printed on the skin of a domed structure, lit from within and by projection at night, and paired with live choral evensong. - Try it: Collect 20 species that live within 1 km of your school, draw each on paper, and place them on a sphere in AR at a real outdoor spot; tapping a species plays its sound or name. Twist: species fade out if they are locally endangered, leaving gaps in the sphere. #### Your Voices — Es Devlin (2023) - Video: https://www.youtube.com/watch?v=_ZAm4Zvg0XE - Interaction: Voice & Sound, Shared & Social, Location & City - Platform & tech: Projection, kinetic sculpture, LED text, voice recordings - Idea: A slowly rotating object lets text be read from any side of a plaza; in AR, rotating an anchored object can replace UI for showing many messages in one place. - What it is: At Lincoln Center in New York, a tall revolving tree-like kinetic sculpture turns slowly and lights up with the words and voices of the public during the winter season. - Technique: A motorised rotating structure carries light-emitting text surfaces, and contributed voice recordings are likely cued in sync with the lighting. - Try it: Place a slowly rotating AR 'tree' in a courtyard whose branches display short messages recorded by classmates; the message facing the user plays aloud. Twist: speaking into the phone adds your voice as a new glowing branch. ### Gibson/Martelli (Ruth Gibson & Bruno Martelli) *Artist duo working with motion capture, AR and VR dance* Choreographer Ruth Gibson and VR artist Bruno Martelli have worked since 2000 with motion capture and game engines, turning dance into AR prints (MAN A), room-scale VR and participatory mocap performances (DAZZLE, PAN+TILT, nino). #### MAN A — Gibson/Martelli (Ruth Gibson & Bruno Martelli) (2014) - Video: https://vimeo.com/478965016 - Interaction: Tangible Objects, Performance, Perception & Effects - Platform & tech: Phone, image tracking, motion capture, Unity - Idea: Camouflage that hides dancers until a phone reveals them. - What it is: Large dazzle-camouflage wall prints that look flat until visitors scan them with a mobile app, revealing motion-captured dancers performing in AR on and around the pattern; shown at Selfridges and won a Lumen Prize Gold Award. - Technique: The dazzle patterns double as image-tracking markers, and the app anchors mocap-animated characters to them. - Try it: Design a black-and-white geometric poster, register it as an image target in Lens Studio or AR Foundation, and attach a Mixamo dance to it. Twist: each fragment of the pattern reveals a different body part. #### MAN A VR — Gibson/Martelli (Ruth Gibson & Bruno Martelli) (2015) - Video: https://www.youtube.com/watch?v=45BOhBSF_U4 - Interaction: Performance, Gaze & Attention, Portals & Worlds - Platform & tech: Headset, Phone, VR, motion capture, Unity - Idea: Step inside the camouflage and be surrounded by dancers you can walk among. - What it is: A room-scale VR piece that pushes the viewer into the computer space of the MAN A dancers, surrounding them with choreographed avatars in dazzle-patterned bodies. - Technique: Motion-captured choreography is played back on stylised avatars in a game engine, viewed through a room-scale headset or a customised mobile viewer. - Try it: Place three looping mocap dancers around the origin of a WebXR scene and give them a bold two-colour material. Twist: dancers only move when the viewer is not looking at them. #### DAZZLE: A Re-assembly of Bodies — Gibson/Martelli (Ruth Gibson & Bruno Martelli) (2020) - Video: https://vimeo.com/868428016 - Interaction: Shared & Social, Hands & Body, Performance - Platform & tech: Headset, motion capture, VR, Unreal Engine - Idea: A ball where everyone, audience included, dances as a live avatar. - What it is: A participatory VR dance ball inspired by the 1919 Chelsea Arts Club Dazzle Ball: visitors in zero-waste costumes are motion-captured and dance as avatars alongside live performers in mocap suits. - Technique: Live optical motion capture streams performers and visitors into a shared real-time VR world, with a virtual camera recording the event. - Try it: Stream two people's webcam poses into one shared web scene as abstract avatars, and play music so they dance together remotely. Twist: each avatar wears the other person's colours. #### PAN+TILT — Gibson/Martelli (Ruth Gibson & Bruno Martelli) (2022) - Video: https://vimeo.com/758291757 - Interaction: Hands & Body, Shared & Social, Portals & Worlds - Platform & tech: Headset, motion capture, machine learning, Unreal Engine, VR - Idea: Use a somatic dance technique as the controller for a virtual world. - What it is: A VR experience for two visitors inspired by Joan Skinner's Releasing Technique, in which motion capture and machine learning turn visitors' movement into exploration of a holographic landscape; shown at the BFI London Film Festival Expanded. - Technique: Visitors are motion-captured and their movement qualities, classified with machine learning, change how the Unreal environment responds. - Try it: Write three Skinner-style spoken prompts, train a Teachable Machine pose model on how people respond to each, and let the detected class change a WebXR landscape. Twist: the landscape only answers slow movement. #### WOW+FLUTTER — Gibson/Martelli (Ruth Gibson & Bruno Martelli) (2024) - Video: https://vimeo.com/1018110955 - Interaction: Shared & Social, Performance, Hands & Body - Platform & tech: Projection, Desktop, motion capture, Unreal Engine, Midjourney - Idea: Dancers in two cities share one body of non-human avatars. - What it is: A dual-site performance linking dancers in the North and South East of England, where AI-designed non-human avatars are animated live by the dancers via mocap and shown with video, virtual environments and live music. - Technique: Avatars generated from Midjourney prompts are modelled and rigged, then driven by live motion capture streamed between sites over a 10 Gb link into Unreal. - Try it: Generate a creature image with an AI tool, rig it with Mixamo, and drive it with a webcam pose stream from a partner in another room. Twist: each partner controls only half of the creature. #### nino — Gibson/Martelli (Ruth Gibson & Bruno Martelli) (2025) - Video: https://vimeo.com/1081511795 - Interaction: Hands & Body, Shared & Social, Performance - Platform & tech: Projection, RADiCAL, markerless motion capture, Unreal Engine - Idea: Suit-free mocap turns every visitor into a puppeteer of a dreamlike avatar. - What it is: A performance at Trafó Budapest in which dancers and then audience members animate avatars in surreal landscapes, captured without suits by an AI camera so anyone can step in and move a character. - Technique: Markerless AI motion capture (RADiCAL) streams audience poses into Unreal Engine avatars projected on large vertical screens. - Try it: Use a webcam pose model to drive a rigged character projected on a wall, and invite passers-by to take over. Twist: the avatar keeps the last visitor's habits for a few seconds after they leave. ### Ivan Poupyrev *Interaction scientist; Sony CSL, Disney Research, Google ATAP (Soli, Jacquard); CEO of Archetype AI* Built early tangible AR at Sony CSL and HIT Lab, then turned everyday matter into interfaces at Disney Research (Touché, Botanicus Interacticus, Ishin-Den-Shin) and Google ATAP (Soli radar, Jacquard textiles). #### TeslaTouch — Ivan Poupyrev, Chris Harrison — CMU Future Interfaces Group (2010) - Video: https://www.youtube.com/watch?v=MHNrQlpstJo - Interaction: Hands & Body, Tangible Objects - Platform & tech: Desktop, electrovibration, touch screen - Idea: Make the surface of a flat screen feel textured. - What it is: A touch screen that produces variable friction under a sliding finger using electrovibration, so flat images can feel rough, bumpy or sticky. - Technique: Applies a modulated high voltage to an insulated electrode so electrostatic attraction changes skin friction as the finger moves. - Try it: Use the phone vibration API to build a 'haptic texture' drawing board where a finger sliding over different colored areas triggers different vibration rhythms. Twist: have a partner close their eyes and guess the pattern by touch alone. #### Botanicus Interacticus — Ivan Poupyrev (2012) - Video: https://www.youtube.com/watch?v=17QOyr2d5-I - Interaction: Tangible Objects, Voice & Sound, Projection - Platform & tech: Projection, Desktop, swept-frequency capacitive sensing, projection - Idea: Stroke a real plant and it answers with sound, light and shadow. - What it is: Living plants become interactive: stroking, pinching or sliding along a stem is sensed through the plant itself and triggers projected visuals and sound. - Technique: Places a single electrode in the plant's soil and applies Touché-style swept-frequency sensing to locate and classify touches on the plant. - Try it: Connect a potted plant to a Makey Makey or touch sensor so touching a leaf grows virtual flowers with sound on a phone or projection. Twist: the flowers only bloom when two people touch the plant at the same time. #### Touché — Ivan Poupyrev (2012) - Video: https://www.youtube.com/watch?v=E4tYpXVTjxA - Interaction: Tangible Objects, Hands & Body - Platform & tech: Desktop, swept-frequency capacitive sensing, SVM classification - Idea: A single electrode can tell whether you pinch, grasp, or plunge your whole hand into water. - What it is: Swept-frequency capacitive sensing that recognises how you touch a doorknob, a table, water or your own body - one pinch versus full grasp - with a single electrode. - Technique: Sweeps the excitation frequency across a range and classifies the resulting capacitive profile, because different grips change the body's impedance at different frequencies. - Try it: Connect a water glass, a piece of fruit or a doorknob to a Makey Makey or an Arduino's capacitive touch pin, and train a simple classifier to tell "a touch" from "a grip". Twist: map the different grips to different effects in phone AR. #### Ishin-Den-Shin — Ivan Poupyrev (2013) - Video: https://www.youtube.com/watch?v=Iw1FhmY1sIU - Interaction: Voice & Sound, Hands & Body, Shared & Social - Platform & tech: Wearable, body-coupled audio, electrostatic modulation - Idea: Touch someone's ear with your finger and a whisper travels through your body to them. - What it is: A person speaks into a microphone and then touches someone's ear with a finger; the recorded message is heard only by the person being touched, transmitted through the body. - Technique: Converts recorded speech into a modulated inaudible high-voltage signal on the speaker's body, and the touched earlobe vibrates as a tiny speaker. - Try it: Use bone-conduction headphones or phone recordings to design a message-passing game where "only the person being touched can hear" (a tap on the shoulder triggers playback). Twist: pass the message along a human chain, letting it gradually change on the way. #### Project Jacquard — Ivan Poupyrev (2015) - Video: https://www.youtube.com/watch?v=qObSFfdfe7I - Interaction: Tangible Objects, Hands & Body - Platform & tech: Wearable, conductive yarn, capacitive textile sensing - Idea: Weave a jacket sleeve into a touchpad. - What it is: Conductive yarns woven into ordinary fabric turn a jacket sleeve into a touch surface: swipe or tap the cuff to control your phone. - Technique: Weaves a grid of conductive threads into the textile and reads capacitive changes from a snap-on tag that classifies swipes and taps. - Try it: Sew conductive tape or conductive thread onto an old glove or sleeve and connect it to a Makey Makey to switch scenes in a phone AR app. Twist: let different parts of the clothing trigger different AR "spells". #### Project Soli — Ivan Poupyrev (2015) - Video: https://www.youtube.com/watch?v=0QNiZfSsPc0 - Interaction: Hands & Body - Platform & tech: Wearable, Phone, 60 GHz radar, machine learning gestures - Idea: Rub your fingers in the air to turn an invisible dial. - What it is: A tiny radar chip senses sub-millimetre finger motions in the air, so rubbing fingers turns a virtual dial and tapping presses an invisible button. - Technique: Emits a broad radar beam and classifies the Doppler and range signature of micro-finger movements into 'virtual tool' gestures. - Try it: Use phone AR hand tracking or MediaPipe to build mid-air micro-gestures (pinch to turn a knob, two-finger tap to press a button) that control a virtual radio. Twist: the task must be done without looking at your hands. ### Jaume Plensa *Sculptor* Spanish sculptor of monumental heads made of letters, mesh or white stone, placed in plazas, parks and on the waterfront, including Chicago's Crown Fountain. #### Crown Fountain — Jaume Plensa (2004) - Video: https://www.youtube.com/watch?v=zGYTvFFgBAQ - Interaction: Face, Location & City, Shared & Social - Platform & tech: Desktop, LED towers, video portraits, water - Idea: A giant face that suddenly does something physical (spits water) creates delight and play, a good cue for AR characters that trigger real-world effects. - What it is: Two 15-metre glass-brick towers in Chicago's Millennium Park show video faces of a thousand Chicagoans; every few minutes a face purses its lips and water spouts from its mouth onto children playing in the pool. - Technique: LED screens behind glass bricks play filmed portraits; a nozzle at the mouth position is synchronised with the video to release water. - Try it: Record short face videos of classmates and place them as two huge AR screens facing each other on a plaza; when a face puckers, spawn an AR water jet between them. Twist: trigger the water with the real user's own lip pucker via face tracking. #### Dream — Jaume Plensa (2009) - Video: https://www.youtube.com/watch?v=qHuEnPjo6Rk - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, white Spanish dolomite, former colliery - Idea: A calm face on a scarred landscape changes what the place means for a whole town, showing that an AR landmark can carry collective memory. - What it is: A 20-metre white head of a nine-year-old girl with closed eyes rises from the spoil heap of the former Sutton Manor colliery in St Helens, commissioned with ex-miners through Channel 4's Big Art Project. - Technique: A digitally elongated scan of a girl's head was cast in white dolomite-and-concrete sections on a pile foundation. - Try it: Scan a classmate's face with eyes closed, stretch it vertically by 50% in Blender, and place it 15 metres tall on a hill or rooftop in AR. Twist: the eyes open only when nobody is looking directly at it. #### Echo — Jaume Plensa (2011) - Video: https://www.youtube.com/watch?v=J7nrzfkwGNQ - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, fibreglass, marble dust resin, city park - Idea: A dreaming face among skyscrapers slows the pace of a busy street, showing how an AR figure can set a mood rather than deliver information. - What it is: A 13-metre white head of a young girl with closed eyes, elongated as if seen in a distorting mirror, sat in Madison Square Park, New York, among the office towers. - Technique: A 3D scan of a girl's face is stretched digitally, then fabricated in resin and marble dust over a steel frame. - Try it: Place a large, calm AR face in the busiest spot of your campus and measure (by observation) whether people slow down near it. Twist: the face breathes slowly, synced to the average walking speed of passers-by. #### Julia — Jaume Plensa (2018) - Video: https://www.youtube.com/watch?v=4W9Q45QjeME - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, white polyester resin, marble dust, plaza plinth - Idea: Seen from the side, the thin head almost disappears; from the front it fills the square, a lesson in designing AR objects that change with angle. - What it is: In Madrid's Plaza de Colón, a 12-metre elongated white head of a girl with closed eyes stands on a plinth once meant for a statue, in front of the city's traffic. - Technique: The portrait is flattened in depth and stretched in height so its profile is very thin, then cast in resin with marble dust. - Try it: Model an AR face that is only 10 cm thick but 10 m tall, place it on a plaza, and film how it appears and disappears as you walk around it. Twist: show a different person's face on the back side. #### Behind the Walls — Jaume Plensa (2019) - Video: https://www.youtube.com/watch?v=mXyEnS7SHyo - Interaction: Location & City, Face, Perception & Effects - Platform & tech: Desktop, white resin, marble dust, hands over eyes - Idea: A giant figure that refuses to look makes viewers aware of their own looking, a strong idea for gaze-aware AR. - What it is: At Rockefeller Center in New York, a 12-metre white head of a young woman covers her eyes with her hands, a comment on our choice not to see what is around us. - Technique: A scanned head and hands are elongated and fabricated in resin with marble dust, mounted in a plaza. - Try it: Place a giant AR head that covers its eyes whenever the user points the camera straight at it and peeks when the camera looks away. Twist: invite two users, so the head only uncovers its eyes when both look away. #### Water's Soul — Jaume Plensa (2020) - Video: https://www.youtube.com/watch?v=jIRANB8uLkM - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, fibreglass, resin, riverside pier - Idea: A 'hush' gesture aimed at a whole skyline speaks to a city at once, a model for AR figures that address a distant view. - What it is: A 24-metre white head of a girl with a finger to her lips stands on a pier in Newport, Jersey City, facing Manhattan across the Hudson River, as a reminder to protect water. - Technique: A scanned figure is elongated and built in fibreglass and resin on a steel structure on the waterfront. - Try it: Place a giant AR figure on a riverbank or lake edge that makes a gesture toward the opposite shore, and add a soundscape that goes quiet when you face it. Twist: the figure's gesture changes with the live water level of the river. ### Jelle Vermandere *Game developer and YouTuber* Belgian maker who builds mixed-reality games that use real objects and his body as controllers: a real football, RC cars, a smart Rubik's cube, a heart-rate monitor. #### 4D Explorer (4D Raymarching) — Jelle Vermandere (2020) - Video: https://www.youtube.com/watch?v=nUExziADzjc - Source code: https://github.com/Jellevermandere/4D-Raymarching - Interaction: Perception & Effects, Play - Platform & tech: Desktop, Unity, Raymarching, SDF - Idea: See a 4D world by walking through its 3D slices. - What it is: The player moves through a world where objects slide in and out of a fourth dimension, growing, shrinking and vanishing as the 3D slice shifts. - Technique: Signed distance functions are defined in four dimensions and raymarched in a 3D slice whose W coordinate the player controls, with 4D rotations applied to the objects. - Try it: Place a 4D object on a real table in phone AR and map the fourth coordinate to the phone's height above the floor. Twist: two students at different heights see different shapes at the same spot. #### AR Boxing Game — Jelle Vermandere (2020) - Video: https://www.youtube.com/watch?v=AOUaIqQml4g - Interaction: Play, Hands & Body - Platform & tech: Phone, Unity, AR Foundation - Idea: Bring a sparring partner into your living room, so punching the air becomes a workout. - What it is: He places virtual opponents in his real living room with AR Foundation and punches at them, turning a fitness routine into an AR boxing match. - Technique: AR Foundation plane detection places the opponent on the floor and the phone's pose (held or strapped) is used to detect punches and dodges. - Try it: Use AR Foundation to place a punching-bag opponent on the floor that throws punches while the player dodges sideways. Twist: the higher your heart rate, the stronger the opponent. #### Controlling a Soccer Game with a Real Ball — Jelle Vermandere (2022) - Video: https://www.youtube.com/watch?v=tB6Ag6VCYFY - Interaction: Tangible Objects, Hands & Body, Play - Platform & tech: Desktop, Python, OpenCV, Unity - Idea: Kick a real ball and the match on screen follows. - What it is: A webcam tracks a real football and his body pose so kicking the actual ball drives a digital football match in Unity. - Technique: Python computer vision detects the ball position and the player's pose from a webcam and streams them to Unity, which maps them into the game world. - Try it: Use a webcam and color tracking so a real paper ball controls a pinball on screen. Twist: trigger a goal animation when the paper ball hits a particular cardboard area on the wall. #### Real Hole in the Wall Game — Jelle Vermandere (2022) - Video: https://www.youtube.com/watch?v=tFIZ23fRHxQ - Interaction: Hands & Body, Play - Platform & tech: Desktop, MoveNet, Unity, heart-rate monitor - Idea: Based on Brain Wall: shape your body to match the hole in the wall to get through. - What it is: A full-body game based on Japan's Brain Wall: a webcam reads his pose and a heart-rate monitor adds pressure as walls with body-shaped holes rush toward him. - Technique: MoveNet pose estimation extracts body keypoints from a webcam, which are compared against the silhouette of each incoming wall hole. - Try it: Build a Hole in the Wall game with Teachable Machine or MoveNet where a pose outline appears on screen and a classmate has 3 seconds to match it. Twist: two people must combine their bodies to pass one wall. #### Turning RC Cars into Pets — Jelle Vermandere (2023) - Video: https://www.youtube.com/watch?v=9FBi0qbLiio - Interaction: Tangible Objects, Play - Platform & tech: Desktop, Arduino, OpenCV, Unity - Idea: Give RC cars personalities so they wander the floor and respond to you like pets. - What it is: Overhead computer vision tracks several Arduino-controlled RC cars in real time and gives each one a personality, so they roam the floor like pets that react to him. - Technique: A camera tracks each car with CV markers, and a behavior script sends steering commands back over radio so the physical cars act on virtual rules. - Try it: Use micro:bit cars and an overhead camera for tracking, and make the cars afraid of classmates' feet so they run away. Twist: project an emotion face on the floor for each car. #### A Rubik's Cube Game with a Real Cube — Jelle Vermandere (2024) - Video: https://www.youtube.com/watch?v=7UXtkvvA0dI - Interaction: Tangible Objects, Play - Platform & tech: Desktop, GAN smart cube, Bluetooth, Unity - Idea: Twist a real Rubik's cube to control a game world. - What it is: A Bluetooth smart Rubik's cube becomes the controller: every real twist rotates and changes a puzzle world he built in Unity. - Technique: The GAN smart cube reports each face turn over Bluetooth, and Unity maps those moves to rotations of the in-game level. - Try it: Use a phone with a gyroscope or a micro:bit as the 'cube', and rotate it to tilt an on-screen maze so a ball rolls to the exit. Twist: shaking it regenerates the maze. ### Joanie Lemercier (AntiVJ) *Visual artist; co-founder of the AntiVJ label* French artist who co-founded AntiVJ and pioneered projection mapping on architecture and on drawn landscapes, later turning to environmental activism. #### AntiVJ - projection on building — Joanie Lemercier (AntiVJ) (2006) - Video: https://www.youtube.com/watch?v=L64-nqZsgjo - Interaction: Projection, Location & City - Platform & tech: Projection, projection mapping, VJ software - Idea: An early experiment projecting live visuals onto a building and playing with its structure. - What it is: An early experiment projecting live visuals onto a building and playing with its existing architecture, a seed of the AntiVJ label's mapping style. - Technique: Early manual projection mapping: live VJ software output is masked and keystoned by hand to fit building features, with visuals designed around the existing architecture. - Try it: Use one projector and VJ software (Resolume or TouchDesigner) to hand-mask an outside wall with windows, and perform a 3-minute visual set live. Twist: make each window an independent 'character' that passes graphics to the others. #### AntiVJ - Grote Kerk — Joanie Lemercier (AntiVJ) (2008) - Video: https://www.youtube.com/watch?v=OksyXOMTnG4 - Interaction: Projection, Performance - Platform & tech: Projection, projection mapping - Idea: A live projection performance inside a cathedral. - What it is: A live audiovisual projection performance on the interior of the Grote Kerk cathedral in Breda. - Technique: Interior projection mapping with multiple projectors warped onto vaults and columns, performed live and synced to audio. - Try it: Choose an indoor space with vaults or columns (a stairwell, a corridor) and perform a projection live with a piece of sound, emphasizing the height of the space. Twist: use light of only one color for the whole piece so the structure itself becomes the star. #### AntiVJ - SONGDO — Joanie Lemercier (AntiVJ) (2009) - Video: https://www.youtube.com/watch?v=8tQpubnD0LQ - Interaction: Projection, Performance, Location & City - Platform & tech: Projection, projection mapping - Idea: An audiovisual projection on the buildings of New Songdo City in South Korea. - What it is: An audiovisual projection performance on architecture in New Songdo City, South Korea. - Technique: Architectural projection mapping with content built from the building's surveyed geometry and synced to a live audio performance. - Try it: Build a 'future city' model from blocks or cardboard boxes and project an audiovisual animation of the city growing from empty ground to completion. Twist: at the end of the animation, let nature slowly 'take back' the city. #### Eyjafjallajökull — Joanie Lemercier (AntiVJ) (2011) - Video: https://www.youtube.com/watch?v=hmV1rTUlJ2s - Interaction: Projection, Perception & Effects - Platform & tech: Projection, projection mapping - Idea: Projection makes a hand-drawn line mountain look like an erupting volcano. - What it is: Light is projected onto a hand-drawn wireframe mountain so the drawing appears to erupt, shift and transform into a volcano. - Technique: A projector is aligned to a large hand-drawn wireframe on paper, and projected light and animation reveal, erase and extend the lines to create the illusion of an erupting 3D mountain. - Try it: Draw the wireframe of a mountain in black pen on white paper, align a projector to the drawing, and make light flow between the lines to simulate an eruption. Twist: use a landform from your hometown instead and add one local natural phenomenon. #### FUJI — Joanie Lemercier (AntiVJ) (2014) - Video: https://www.youtube.com/watch?v=gVuWcyXHMoI - Interaction: Projection, Perception & Effects - Platform & tech: Projection, projection mapping - Idea: Projection brings light and weather to a pencil drawing of Mount Fuji. - What it is: A 14-minute installation where projection over a large drawing of Mount Fuji creates shifting light, weather and geometric illusions. - Technique: Projection over a large pencil drawing uses precisely registered light to add clouds, sunrise and geometric overlays, with the drawing serving as a static luminance base. - Try it: Draw a simple pencil landscape and use projection to simulate how light, cloud shadows and weather change over a day. Twist: drive the weather changes with real data from that day's weather API. #### Blueprint — Joanie Lemercier (AntiVJ) (2015) - Video: https://www.youtube.com/watch?v=OozmLRPL1zQ - Interaction: Projection, Perception & Effects - Platform & tech: Projection, projection mapping - Idea: Light builds and unbuilds structures on a blueprint's line drawing. - What it is: A 15-minute audiovisual installation of projected light on a technical line drawing, exploring the blueprint as the origin of structures. - Technique: Projection mapped onto a technical line drawing builds and deconstructs structures with animated line work aligned to the printed lines. - Try it: Print a line drawing of a building or machine and project a line animation that gradually "builds" it and then "takes it apart". Twist: make the construction go wrong at one step, and let a different structure grow out of the mistake. ### Judith Amores *Researcher (MIT Media Lab PhD; Microsoft Research)* HCI researcher who built playful AR prototypes from Barcelona to MIT Media Lab and Google Creative Lab, and multi-person AR at Microsoft Research with Jaron Lanier. #### Augmented Menu — Judith Amores (2014) - Video: https://www.youtube.com/watch?v=Z5_jopA_rJk - Interaction: Voice & Sound, Information & UI, Face - Platform & tech: Phone, marker-based AR, lip sync - Idea: A virtual waiter appears on your table to walk you through today's menu. - What it is: A virtual AR waiter with automatic gestures, facial expressions and lip-sync appears at the table to present the daily menu and talk with diners. - Technique: An avatar anchored on a table marker is animated procedurally with gesture, head-look and lip-sync generated from the audio input. - Try it: Make an AR menu card that, when scanned, brings up a talking 3D character who introduces the dishes (Reality Composer + TTS). Twist: the character looks at whoever is closest to it. #### Qbox: AR Magic Toy — Judith Amores (2014) - Video: https://www.youtube.com/watch?v=t3FpKu0xYQI - Interaction: Tangible Objects, Play - Platform & tech: Phone, marker-based AR - Idea: Put a toy into a magic box and it comes alive on the screen. - What it is: Put a small toy inside a box and, through an AR interface, it comes to life, interacting with the box, the child and the world around it. - Technique: Marker tracking on the box registers an animated virtual version of the inserted toy on top of the physical one (likely marker/toy recognition). - Try it: Make a cardboard box with a recognition image, so different animations appear in AR when different toys go in (by swapping cards). Twist: shaking the box makes the character faint. #### Reality Masher: multi-person AR — Judith Amores (2015) - Video: https://www.youtube.com/watch?v=EfxMPXbvCLw - Interaction: Shared & Social, Spatial Mapping - Platform & tech: Headset, HoloLens - Idea: Several people in headsets share the same set of holograms in one room. - What it is: A Microsoft Research prototype with Jaron Lanier in which several people wearing early HoloLens headsets share and play with the same holograms in one room. - Technique: Multiple HoloLens devices share a common spatial anchor so holograms occupy the same physical location for every participant. - Try it: Use Lens Studio Connected Lens or an 8th Wall shared session so two phones see the same virtual pet and take turns feeding it. Twist: one person can only see it and the other can only hear it. #### Draw and Dance — Judith Amores, Anna Fusté (2017) - Video: https://www.youtube.com/watch?v=q_0HZ14EA_Y - Interaction: Drawing & Making, Voice & Sound, Play - Platform & tech: Phone, Unity, ARCore, Vuforia, API.AI - Idea: Draw a stick figure and it dances in the real world to the sounds and music around it. - What it is: You draw a stick figure on your phone and it becomes a dancing AR character that reacts to music and sounds around it, even perching on a Google Home speaker to dance to its output. - Technique: A 2D drawing is rigged into a stick-figure skeleton, anchored with ARCore, and animated by microphone amplitude and voice commands via API.AI. - Try it: Use Scratch, p5.js or Lens Studio to make a student-drawn character dance to microphone volume and place it in AR. Twist: two characters dance different steps when they hear different pitches. #### Invisible Highway — Judith Amores, Anna Fusté (2017) - Video: https://www.youtube.com/watch?v=9ZhqsuWF96I - Interaction: Drawing & Making, Tangible Objects, Spatial Mapping - Platform & tech: Phone, Unity, Tango, Arduino - Idea: Draw a road in AR and a real toy car drives along it. - What it is: Draw a path on the floor in AR and a real robot car follows it through the room, while the phone adds a generated highway and scenery along the route. - Technique: The drawn AR path is converted to world coordinates via Tango tracking and sent as waypoints to a robot car whose position is tracked to follow it. - Try it: Draw an AR route on the floor with a tablet, then have a micro:bit car drive along the coordinates, using an overhead camera for tracking if needed. Twist: add virtual obstacles on the road that the car must steer around. ### Ken Nakagaki *Assistant Professor, University of Chicago; directs the Actuated Experience Lab (AxLab)* Former MIT Tangible Media researcher who builds shape-changing lines, robot swarms and mechanical shells that make digital information physically move around people. #### LineFORM — Ken Nakagaki, MIT Tangible Media Group — Hiroshi Ishii (2015) - Video: https://www.youtube.com/watch?v=XB1U0oTj8lE - Interaction: Tangible Objects, Information & UI - Platform & tech: Desktop, Wearable, servo chain, shape-changing interface - Idea: A line that bends itself, both displaying information and constraining your hand. - What it is: A snake-like chain of servo modules that can bend into shapes, wrap the wrist, act as a physical graph line, or constrain the user's hand motion. - Technique: Chains serially linked servo segments and computes joint angles by inverse kinematics so the line can take on target curves or respond to touch. - Try it: Make a bendable physical line from straws and pipe cleaners, then use phone AR to show flowing data or a music waveform along it. Twist: when a classmate reshapes the line, the AR content changes with it. #### AnimaStage — Ken Nakagaki, MIT Tangible Media Group — Hiroshi Ishii (2016) - Video: https://vimeo.com/221508021 - Interaction: Tangible Objects, Performance, Drawing & Making - Platform & tech: Desktop, actuated stage, paper craft - Idea: A moving stage turns paper cutouts and handmade props into puppets. - What it is: A tabletop stage with a hidden grid of actuators animates paper crafts and 3D-printed figures placed on it, turning hand-made props into puppets. - Technique: Drives an array of linear actuators under a flexible stage surface so objects attached to it are pushed, tilted and bounced to perform scripted motions. - Try it: Build a shaking paper-puppet stage with cardboard and phone vibration (or a small motor), and overlay facial expressions and lines with phone AR. Twist: let the rhythm of the audience's clapping drive the stage's motion. #### ChainFORM — Ken Nakagaki, MIT Tangible Media Group — Hiroshi Ishii (2016) - Video: https://www.youtube.com/watch?v=Ler01uKHhcs - Interaction: Tangible Objects, Hands & Body - Platform & tech: Desktop, Wearable, modular servo chain, touch sensing, LEDs - Idea: Snap-together smart chains that can sense touch, display light and change shape. - What it is: Modular chains with built-in touch sensing, angle sensing and LEDs that can be snapped together into shape-changing strips, wearables or creatures. - Technique: Each module carries a servo, angle sensor, capacitive touch pad and LED on a shared bus, so a chain can both record and replay its own shape. - Try it: Build a prototype chain that "records its shape" from building blocks or paper links, and use phone AR to label each link with a color and value. Twist: wrap the chain around a wrist to make a wearable reminder. #### HERMITS — Ken Nakagaki, MIT Tangible Media Group — Hiroshi Ishii (2020) - Video: https://www.youtube.com/watch?v=nsi4DsiAWs8 - Interaction: Tangible Objects, Play - Platform & tech: Desktop, toio robots, mechanical shells, 3D printing - Idea: Like hermit crabs changing shells, small robots gain new functions by crawling into different shells. - What it is: Small self-propelled robots dock into interchangeable mechanical shells - a lever, a crank, a vehicle - gaining new physical interactions like hermit crabs changing shells. - Technique: Uses toio robots with position tracking to drive into passive 3D-printed shells whose gears and levers convert robot motion into new actuation or input. - Try it: Design three cardboard shells for an RC car (a bulldozer blade, a flag and a little house) and label each shell's function in phone AR. Twist: have the car automatically recognize which shell it is wearing from a QR code. #### Buoyancé — Ken Nakagaki (2025) - Video: https://www.youtube.com/watch?v=v9mNKIILt_s - Interaction: Tangible Objects, Information & UI - Platform & tech: Desktop, mobile robots, helium balloons, string reeling - Idea: Robots reel helium balloons up and down to form an interactive 3D display in mid-air. - What it is: Mobile robots reel helium balloons up and down by strings, forming floating 3D displays and interactive mid-air objects above a tabletop. - Technique: Each tabletop robot winds a tether to set balloon height while driving to set x-y position, so a swarm places floating points in 3D space. - Try it: Use balloons, string and manual raising and lowering to turn a tabletop into a mid-air bar chart, then add a label to each balloon with phone AR. Twist: when classmates pull a balloon to change its height, the data in AR updates with it. #### Shape n' Swarm — Ken Nakagaki (2025) - Video: https://www.youtube.com/watch?v=5u0M9yL7tyY - Interaction: Tangible Objects, Voice & Sound, Hands & Body - Platform & tech: Desktop, swarm robots, LLM, hand tracking - Idea: Sketch a shape with your hands and say one sentence, and a robot swarm forms it and brings it to life. - What it is: Users sculpt a shape with their hands and describe it in speech; an LLM plans how a swarm of small tabletop robots should arrange and move to represent and animate it. - Technique: Combines hand-sketched shape input with an LLM that outputs robot target positions and motion scripts for a swarm UI. - Try it: Have a classmate describe an animal by voice, let ChatGPT turn it into a set of coordinates, and have several classmates holding paper cards line up on the floor to form the shape. Twist: require the shape to move (for example, a bird flapping its wings). ### Lightform (Brett Jones & Kevin Karsch) *Projected-AR hardware/software startup; Brett Jones was lead author of IllumiRoom and RoomAlive* Lightform built the LF1/LF2 projector-camera units and Lightform Creator so anyone could scan a scene and projection-map it in minutes. #### It's Lit! Projected AR Murals — Lightform (Brett Jones & Kevin Karsch) (2018) - Video: https://www.youtube.com/watch?v=XTR-NhJ9em8 - Interaction: Projection, Drawing & Making - Platform & tech: Projection, Lightform, projection mapping - Idea: A hand-painted mural plus projected animation makes still letters light up. - What it is: Lettering artist Gemma O'Brien and Craig Winslow hand-paint a mural, then Lightform projects animated light that traces, fills and reveals the painted letters. - Technique: Scans the painted mural to extract its line work, then uses those edges as masks and paths for projected animated effects. - Try it: Have students write large letters with a black marker on white board, then use a projector and free projection-mapping software (such as MapMap) to send flowing light along the strokes. Twist: control the light's brightness with sound volume. #### Renegade Projected AR in Downtown Vegas — Lightform (Brett Jones & Kevin Karsch) (2018) - Video: https://www.youtube.com/watch?v=_RFa7f8If2c - Interaction: Projection, Location & City - Platform & tech: Projection, Lightform LF1, structured-light scanning, projection mapping - Idea: Ride a projector bike that scans and projects as it goes, turning city walls into AR murals. - What it is: A Lightform projector on a cargo bike rides through downtown Las Vegas at night, scanning building facades and instantly mapping animated graphics onto them. - Technique: The projector-camera unit projects structured-light patterns to capture a depth/edge map of the facade, then auto-aligns effects to its edges. - Try it: At night, use a phone projector or small projector to cast a pre-made animation onto a campus wall, then photograph the wall and trace its outlines in Keynote to align it. Twist: the projection must make use of a real crack or window in the wall. #### Lightform LF2 and Lightform Creator — Lightform (Brett Jones & Kevin Karsch) (2019) - Video: https://www.youtube.com/watch?v=uTTVjHoH734 - Interaction: Projection, Tangible Objects, Spatial Mapping - Platform & tech: Projection, LF2, computer vision, projection mapping - Idea: Scan a scene once and pick an effect for every object surface. - What it is: The LF2 projector-camera and Creator software scan a scene of objects and let designers select surfaces and apply animated effects in minutes. - Technique: Captures a high-resolution scan from the projector's viewpoint so every pixel of the scan corresponds to a projector pixel, removing manual calibration. - Try it: Stack a few white paper boxes on a table and use a projector with OBS/MapMap to give each face a different color animation. Twist: move one box and realign everything within one minute. #### NightBloom at the Conservatory of Flowers — Lightform (Brett Jones & Kevin Karsch) (2019) - Video: https://www.youtube.com/watch?v=45tjfOd4vzg - Interaction: Projection, Location & City, Perception & Effects - Platform & tech: Projection, Lightform LFC, projection mapping - Idea: Project onto real plants so the greenhouse 'blooms' at night. - What it is: An immersive night event at San Francisco's Conservatory of Flowers where projections wash over living plants and glasshouse structures. - Technique: Uses scan-based alignment to project organic animated textures onto irregular foliage and architecture. - Try it: Project slowly shifting colors and points of light onto a potted plant in the classroom. Twist: use the phone microphone so the light on the plant 'breathes' with people's voices. #### Project LFX: Steerable Projected Interfaces — Lightform (Brett Jones & Kevin Karsch) (2021) - Video: https://www.youtube.com/watch?v=3XIGxNP3-Mk - Interaction: Projection, Spatial Mapping, Information & UI - Platform & tech: Projection, steerable projector, pan-tilt mount, room scanning - Idea: A projector that turns its head to deliver interfaces onto any surface in the room. - What it is: A pan-and-tilt projector that moves projected interfaces around the room onto tables, walls and objects wherever they are needed. - Technique: Mounts a projector-camera on a motorized pan-tilt head and re-warps content in real time for each target surface in a scanned room model. - Try it: Put a phone projector on a rotating desk-lamp arm or turntable and turn it by hand to project a timer interface onto the desk, the wall and the door. Twist: use a servo and Arduino to make it follow a person automatically. ### Limelight (Viktor Vicsek & team) *Projection mapping and light art studio* Hungarian studio co-founded by light artist Viktor Vicsek, active in building projection since the mid-2000s, from the 2006 anniversary projection on the Hungarian Parliament to festival pieces in Toruń, Melbourne and Lille. It also makes immersive rooms, LED sculptures and mapping on miniature models and wood reliefs. #### Hungarian Parliament 1956–2006 — Limelight (Viktor Vicsek & team) (2006) - Video: https://www.youtube.com/watch?v=DoMY-vbb5NI - Interaction: Location & City, Information & UI - Platform & tech: Projection, large-format projection, Pani projectors - Idea: A building connected to a historical event is the best screen for remembering it: AR memorials can place archive images on the very place where things happened. - What it is: For the 50th anniversary of the 1956 Hungarian revolution, images were projected across about 10,000 square metres of the Parliament building in Budapest with 18 high-power Pani projectors, one of the early large architectural projections in Europe. - Technique: High-power large-format projectors (Pani, likely using slides or early digital sources) are aligned to the Parliament facade to cover it with historical imagery. - Try it: Find an archive photo taken on your campus and use image or geo anchoring (8th Wall, ARKit) to place it exactly where it was shot, so viewers can compare then and now. Twist: fade between old and new as the viewer walks closer. #### Stars (Skyway Festival, Toruń) — Limelight (Viktor Vicsek & team) (2012) - Video: https://www.youtube.com/watch?v=RjBro_MbGjo - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, projection mapping, 3D animation - Idea: Linking a building to the local story (here, astronomy in Copernicus's city) gives abstract visuals a reason to be on that exact wall; AR content also works best when it is site-specific. - What it is: A 3D mapping for the Skyway light festival in Toruń, Poland, on the facade of the Collegium Maximum: the building fills with stars, constellations and the city's own 'stars' of culture and history, a nod to Toruń's most famous son, Copernicus. - Technique: A 3D model of the facade is animated with star fields and architectural deformations and projected in alignment from the square. - Try it: Research one local story about a building on campus and make a 60-second AR scene in Adobe Aero or Lens Studio that plays on its facade. Twist: the scene only unlocks at night, using the camera's light level. #### Interconnection — Limelight (Viktor Vicsek & team) (2016) - Video: https://www.youtube.com/watch?v=a1v4W95wJnM - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, projection mapping, 3D animation - Idea: Zooming between scales on one fixed surface is an easy-to-read storytelling device; AR can zoom through scales around a real object in the same way. - What it is: The winning piece of the iMapp Bucharest 2016 contest, which took both the jury and the people's choice award. The facade moves from cells and neurons to cities and galaxies to show that the micro and macro, the individual and society, are connected. - Technique: Scale-transition animations are built on a 3D model of the Palace of the Parliament and projected with many aligned, blended projectors. - Try it: Make an AR 'powers of ten' zoom on a real object (a leaf, a coin) in Reality Composer or WebXR: pinch to fly from atoms to galaxies around it. Twist: the zoom is controlled by walking toward or away from the object. #### Talking Heads — Limelight (Viktor Vicsek & team) (2019) - Video: https://www.youtube.com/watch?v=fhL3gYWpXig - Interaction: Face, Perception & Effects - Platform & tech: Projection, LED sculpture, DMX, light programming - Idea: Faces made only of light still read as characters; an AR agent does not need a realistic face, only readable expressions and turn-taking. - What it is: Two large head sculptures by Viktor Vicsek, each covered with about 4,000 individually controllable LEDs, face each other and 'talk': their light patterns form facial expressions and emotions that answer one another. - Technique: Thousands of addressable LEDs on a head-shaped form are driven by programmed light sequences (likely DMX or pixel-mapping software) that are choreographed as a dialogue. - Try it: Make two AR faces built only from dots of light in Lens Studio or Three.js, and script a short conversation between them using expressions only. Twist: one face mirrors the viewer's expression from face tracking. #### Embryogenesis (with Gabriel Schama) — Limelight (Viktor Vicsek & team) (2020) - Video: https://www.youtube.com/watch?v=9UdrqrwuR4c - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Projection, projection mapping, layered wood sculpture, 4K projector - Idea: A detailed physical object gives projected light something to hold on to; AR effects also read best when they follow the real depth and edges of an object. - What it is: A layered, laser-cut wooden relief by Gabriel Schama is brought to life by 4K projection mapping: light flows through its intricate layers so the static sculpture seems to grow like an embryo. - Technique: Content is mapped layer by layer onto the carved wood from a digital model of the relief, projected with a single high-resolution Panasonic projector. - Try it: Scan a textured object (a carved box, a plant) with LiDAR and write a Lens Studio or Unity shader that sends light waves across its real depth layers. Twist: the waves follow the viewer's breathing via the microphone. #### Micromonumental Mapping — Limelight (Viktor Vicsek & team) (2020) - Video: https://www.youtube.com/watch?v=5y02p_JcJ2s - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Projection, projection mapping, scale model, 3D animation - Idea: A miniature model is a cheap, controllable stand-in for a building; a tabletop AR prototype is the fastest way to test a large site-specific idea. - What it is: For the Lille Video Mapping Festival, Limelight mapped a 1:40 scale 3D model of the Lille Opera House instead of the real building, so a full monumental mapping show plays on a tabletop. - Technique: A physical scale model of the opera house is lit by a single projector with content rendered from the matching 3D model and viewpoint. - Try it: Print or build a small model of a campus building, track it with an object or image target in Vuforia or Lens Studio, and play a mapping-style animation on it. Twist: move the model and let the animation follow in real time. ### Luke Hurd *AR effects creator and educator (Spark AR, Lens Studio)* Early Instagram/Spark AR effect creator and teacher whose YouTube breakdowns explained face-mesh, occlusion and segmentation tricks; later built branded AR for National Geographic/NASA JPL and Coachella. #### 'Automaton' Instagram effect — Luke Hurd (2019) - Video: https://www.youtube.com/watch?v=VvYa5yF38xQ - Interaction: Face, Perception & Effects - Platform & tech: Phone, Spark AR, face mesh, scripting - Idea: Turn your face into a mechanical automaton. - What it is: A face effect that retextures and lights the face mesh to turn the user into a mechanical automaton with layered metal parts. - Technique: A custom texture and normal maps are applied to the tracked face mesh, with additional 3D parts attached to face anchors and lit in real time. - Try it: Use face-mesh texturing in Lens Studio to turn yourself into a museum relic or a robot. Twist: the parts on your face spring open when you open your mouth. #### 'Hole in the head' occlusion effect — Luke Hurd (2019) - Video: https://www.youtube.com/watch?v=j7klMWrJFHs - Interaction: Face, Perception & Effects - Platform & tech: Phone, Spark AR, occlusion - Idea: Open a hole in your head with a tiny world hidden inside. - What it is: A face-tracked effect cuts a convincing hole into the user's head where a tiny 3D scene sits inside. - Technique: An invisible occluder mesh around the head writes depth only, so a 3D model placed inside appears to sit in a cavity in the face. - Try it: Use an occluder material to open a hole in your forehead and put a small spinning planet inside. Twist: the scene inside the hole switches when you blink. #### 'Wichita' Instagram effect — Luke Hurd (2019) - Video: https://www.youtube.com/watch?v=CvVhlCBInvI - Interaction: Face, Perception & Effects, Voice & Sound - Platform & tech: Phone, Spark AR, segmentation, scripting - Idea: Turn a selfie into the endless triangular tunnel from the 'Seven Nation Army' video. - What it is: An Instagram effect inspired by the White Stripes' 'Seven Nation Army' video splits your face and background into layered, animated kaleidoscopic tunnels. - Technique: Person segmentation separates the user from the background, and scripted layers of scaled, offset camera textures create the recursive tunnel. - Try it: Use Spark/Lens Studio or p5.js to turn the camera feed into a tunnel that recurses endlessly toward the center. Twist: make the number of recursion layers follow the music volume. #### National Geographic / NASA JPL Mars AR filter — Luke Hurd (2021) - Video: https://www.youtube.com/watch?v=PSEej76-XfI - Interaction: Portals & Worlds, Information & UI, Location & City - Platform & tech: Phone, Spark AR, Instagram - Idea: On the day Perseverance landed, the newest Mars photos became an AR portal you could walk into. - What it is: Timed with the Perseverance landing, an Instagram AR portal let 220 million followers step into a 360° panorama of Jezero Crater made from images only hours old. - Technique: A world-anchored portal masks a spherical 360° panorama so it is visible only through the door, updated with near-real-time rover imagery. - Try it: Make an AR portal from a real 360 panorama photo of your school, space or the sea floor. Twist: the scene inside the portal automatically switches to a new photo every day. #### Charlie Parker memorial Custom Landmarker — Luke Hurd (2022) - Video: https://www.youtube.com/watch?v=PdiAQvRs-50 - Interaction: Location & City, Tangible Objects, Perception & Effects - Platform & tech: Phone, Lens Studio, Custom Landmarker - Idea: Turn a real memorial statue into an AR stage, with effects fitted precisely to its shape. - What it is: Using Lens Studio's Custom Landmarker, a real Charlie Parker memorial statue is scanned so AR effects wrap precisely around the sculpture on site. - Technique: A pre-scanned 3D model of the statue is used as a tracking target so the lens localizes against the physical monument and aligns effects to it. - Try it: Pick a sculpture or door sign on campus and build an AR effect fitted to it with Lens Studio Custom Landmarker or 8th Wall. Twist: make the effect tell the history of the object. #### Coachella: A Virtual Adventure — Luke Hurd (2022) - Video: https://www.youtube.com/watch?v=RsR3wLgIQKI - Interaction: Performance, Location & City, Portals & Worlds - Platform & tech: Phone, Spark AR, Instagram - Idea: Build an AR adventure for a music festival that mixes the real and the virtual. - What it is: A case study of AR experiences built for Coachella that let festival-goers and remote fans unlock virtual worlds around the event. - Technique: Location- and marker-triggered Instagram effects deliver themed AR scenes around the festival (likely world-anchored portals and target tracking). - Try it: Make an AR treasure hunt for a school event where each of three venues reveals a virtual scene and collecting all three unlocks the finale. Twist: online viewers can join with the same effect. ### Makeability Lab — Jon E. Froehlich & Jaewook Lee *HCI lab at the University of Washington; Jaewook Lee leads its AI-powered AR research* Jon E. Froehlich's Makeability Lab builds interactive tools for accessibility, cities and making. PhD student Jaewook Lee has led a run of wearable AR systems that pair real-time vision models and LLMs with the physical world, from cooking aids for low-vision users to AI-assisted outdoor AR authoring. #### ARSports — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2024) - Video: https://www.youtube.com/watch?v=2IvhNwAAl4E - Interaction: Play, Hands & Body, Perception & Effects - Platform & tech: Headset, Oculus Quest 2, ZED Mini, instance segmentation - Idea: Let a vision model highlight the only things that matter in a fast game: the ball and the people. - What it is: A wearable AR prototype for low-vision players that outlines the ball and the other players with bright segmentation masks in near real time during basketball and tennis. - Technique: Instance-segmentation models fine-tuned on new first-person basketball and tennis datasets process ZED Mini stereo video on a Quest 2 and render high-contrast masks over the passthrough view. - Try it: Run a pretrained ball-and-person detector (YOLO or MediaPipe) on the phone camera in a web AR page and draw thick neon outlines around the ball; play catch with a partner and tune the colors until the ball is easiest to follow. Twist: leave a fading trail behind the ball so its trajectory becomes readable. #### CookAR — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2024) - Video: https://www.youtube.com/watch?v=gtJhme8em7U - Source code: https://github.com/makeabilitylab/CookAR - Interaction: Tangible Objects, Hands & Body, Information & UI - Platform & tech: Headset, stereo passthrough camera, affordance segmentation, Unity - Idea: Color the world by what is safe to grab and what can hurt you. - What it is: A head-mounted AR system for people with low vision that colors kitchen tools by affordance in real time: the handle of a knife or pan glows in a safe color, the blade or hot surface in a warning color. - Technique: A segmentation model fine-tuned on a new egocentric dataset of kitchen-tool affordances runs on stereo-camera passthrough video and draws graspable and hazardous parts as colored overlays. - Try it: Train a two-class segmentation or detection model (for example with Roboflow or Teachable Machine) on photos of mugs split into 'handle' and 'hot body', then overlay green and red masks on the live camera in a phone or Lens Studio AR app. Twist: replace the colors with sound, a soft tone near the handle and a harsh one near the hot part. #### EARLL — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2024) - Video: https://www.youtube.com/watch?v=AjdQsxkPVkU - Interaction: Tangible Objects, Hands & Body, Information & UI - Platform & tech: Headset, AR glasses, object segmentation, depth sensing - Idea: Your hands choose the lesson: touch an object and learn its name. - What it is: A language tutor for AR glasses that notices which everyday object you pick up and quietly shows its word in the language you are learning, turning chores into vocabulary practice. - Technique: Real-time segmentation and depth sensing localize objects around the user, a hand-object contact check detects manipulation, and a vocabulary prompt for that object is triggered in the glasses. - Try it: Make a phone AR app with an object detector (Lens Studio's ML object detection or TensorFlow.js COCO-SSD) that shows the Spanish word above a cup, book or chair once it has stayed in view for two seconds. Twist: the learner must say the word aloud and pass a speech-recognition check before the label appears. #### GazePointAR — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2024) - Video: https://www.youtube.com/watch?v=3DebNQ1kpe0 - Interaction: Gaze & Attention, Voice & Sound, Hands & Body - Platform & tech: Headset, HoloLens 2, eye tracking, GPT, vision-language model - Idea: Replace the pronouns in a spoken question with whatever your eyes and finger are pointing at. - What it is: A voice assistant on HoloLens 2 that knows what 'this' and 'that' mean: it combines eye gaze, pointing and conversation history, so you can look at a plant or a math problem and simply ask 'what is this?' or 'how do I solve that?'. - Technique: Gaze and hand rays from the headset select an image crop of the target, a vision model describes it, and that description replaces the pronoun before the query goes to a GPT-style LLM. - Try it: Build a phone AR 'what is this?' button: on tap, capture the camera frame, crop around the screen center, send the crop plus the spoken question to a multimodal LLM API and show the answer as a floating label on the object. Twist: answer only with a small arrow and a single word anchored on the object. #### SonifyAR — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2024) - Video: https://www.youtube.com/watch?v=2iXS2tvjwKg - Interaction: Voice & Sound, Spatial Mapping, Tangible Objects - Platform & tech: Phone, ARKit, LLM, text-to-audio, programming by demonstration - Idea: Let AI decide what an AR scene should sound like from real materials and virtual actions. - What it is: An AR sound-authoring system: when a virtual ball drops onto a real wooden table, it works out what just happened and on which material, then recommends, retrieves or generates a matching sound effect. - Technique: A programming-by-demonstration pipeline logs AR events, vision models describe the real surface involved, and an LLM turns that context into text that drives sound recommendation, retrieval, text-to-audio generation or style transfer. - Try it: In a phone AR app, drop virtual objects onto detected planes and, on each collision, send a crop of the surface to a vision model asking 'what material is this?'; then play a sound from a small library (wood, metal, fabric, water) or from a text-to-audio model. Twist: give the virtual object a material too and let the LLM describe the combined sound. #### ImaginateAR — Makeability Lab — Jon E. Froehlich & Jaewook Lee (2025) - Video: https://www.youtube.com/watch?v=G9mH1HKSPkg - Interaction: Voice & Sound, Location & City, Drawing & Making - Platform & tech: Phone, mobile AR, LLM, text-to-3D, scene graph - Idea: Speak a story into the place where you stand and let AI build it there. - What it is: A mobile outdoor AR authoring tool, made by Jaewook Lee with Niantic Spatial: say a scene such as 'a dragon enjoying a campfire' and AI generates the 3D assets and arranges them in the real park or street, ready to be adjusted by hand. - Technique: An offline scene-understanding pipeline builds an outdoor scene graph, an LLM plans which objects to create and where, and a fast text-to-3D generator produces assets that are placed on detected surfaces. - Try it: Prototype 'say it, place it' in Lens Studio or AR Foundation: send a spoken phrase to an LLM that returns a JSON list of objects and positions, map each object to a free 3D model (or a text-to-3D API such as Meshy) and place it on detected ground planes. Twist: photograph the site first and ask the LLM to build its story around what is already there. ### Masatatsu Nakamura (mattatz) *Graphics and tools programmer* Tokyo programmer who has open-sourced dozens of Unity geometry and GPU-simulation projects (voxelizer, Teddy, cellular growth, volume rendering) and now builds the browser CAD tool Nodi. #### unity-teddy — Masatatsu Nakamura (mattatz) (2016) - Video: https://www.youtube.com/watch?v=ncvcNBKO6kk - Source code: https://github.com/mattatz/unity-teddy - Interaction: Drawing & Making - Platform & tech: Desktop, Web, Unity, C# - Idea: A 2D doodle becomes a 3D object the moment the stroke closes. - What it is: The user draws a closed outline with the mouse and it instantly inflates into a soft 3D blob that can be rotated. - Technique: An implementation of Igarashi's Teddy algorithm: the outline is triangulated with constrained Delaunay, the chordal axis is extracted, and vertices are elevated along it to inflate a closed mesh. - Try it: Let users trace a shape on the phone screen and spawn the inflated mesh onto a detected AR plane, where it drops with physics. Twist: sample the camera image inside the outline as the blob's texture, so you can 'lift' a drawing off real paper. #### unity-volume-rendering — Masatatsu Nakamura (mattatz) (2017) - Video: https://x.com/mattatz/status/935704854046969856 - Source code: https://github.com/mattatz/unity-volume-rendering - Interaction: Information & UI, Perception & Effects - Platform & tech: Desktop, Unity, HLSL, Raymarching - Idea: Raymarch through a 3D texture so volumetric data becomes a glowing object you can slice. - What it is: A medical CT volume floats as a translucent 3D cloud; a slicing plane cuts into it to reveal the inner structure. - Technique: Object-space raymarching steps a ray through a Texture3D inside a cube, accumulates density with a transfer function and clips samples against a user-controlled plane. - Try it: Place the volume cube on a real table with AR Foundation and use the phone itself as the slicing plane, so moving the phone cuts through the data. Twist: load a volume the class makes from stacked photos of a fruit sliced in real life. #### CellularGrowth — Masatatsu Nakamura (mattatz) (2018) - Video: https://x.com/mattatz/status/1002190822064861184 - Source code: https://github.com/mattatz/CellularGrowth - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: Let simple push-and-divide rules for thousands of cells grow an organic form on the GPU. - What it is: A soft, brain-like mass of cells keeps dividing and pushing against its neighbours until it folds into coral and gut-like shapes. - Technique: Each cell is a particle in a compute buffer; compute shaders apply repulsion between neighbours, spring forces along edges and divide cells over time, and the result is drawn as instanced spheres or a membrane. - Try it: Port the growth kernel to AR Foundation and seed the cells where the user taps a real table, so a creature grows out of the surface. Twist: make the growth rate follow the room's light level from the camera. #### GPUVoxelParticleSystem (unity-voxel) — Masatatsu Nakamura (mattatz) (2018) - Video: https://x.com/mattatz/status/951031022342242304 - Source code: https://github.com/mattatz/unity-voxel - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader, Geometry Shader - Idea: Turn any mesh into thousands of voxel particles that can leave and return to their home position. - What it is: A 3D model is broken into glowing voxel cubes that scatter, swirl and snap back together into the original shape. - Technique: The mesh is voxelized on the GPU, each voxel becomes a particle moved by a compute shader (noise and home-position forces), and a geometry shader expands each particle into a cube. - Try it: Voxelize a scanned object and place it on a real floor with AR Foundation; when the user walks through it, the voxels scatter around their body and slowly reassemble. Twist: let voxels that fly off stick to the detected room mesh. #### GPUVoxelSkinnedMesh (unity-voxel) — Masatatsu Nakamura (mattatz) (2018) - Video: https://x.com/mattatz/status/955682645131718657 - Source code: https://github.com/mattatz/unity-voxel - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: Voxelize a moving, skinned body in real time so motion itself becomes blocky sculpture. - What it is: An animated character is re-voxelized every frame, so a running figure made of cubes flickers and shimmers while it moves. - Technique: Each frame the SkinnedMeshRenderer is baked to a mesh, a compute shader rasterizes its triangles into a voxel grid, and the filled cells are drawn as instanced cubes. - Try it: Drive the voxelizer with ARKit body tracking so a cube version of the user dances beside them on the phone screen. Twist: freeze the voxels every few seconds to leave a trail of cube statues on the floor. #### PointCloudExplorer — Masatatsu Nakamura (mattatz) (2021) - Video: https://x.com/mattatz/status/1351050543746211842 - Source code: https://github.com/mattatz/PointCloudExplorer - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, Unity, Point Cloud - Idea: Treat a real place scanned as points as a living material, not a finished model. - What it is: A walk through a city block rebuilt from millions of laser-scan points, with points that shimmer, scan and dissolve as the camera moves. - Technique: Point-cloud data is split into chunks and rendered with a custom point shader; the same code base drove the 'Virtual Hatsudai' app for NTT ICC, with scanlines, transitions and voxelization effects on the points. - Try it: Scan a corridor with an iPhone LiDAR app, import the points into Unity and replay them in AR aligned to the real corridor, so the scan and the place overlap. Twist: let the points drift away wherever a person is standing. ### Michael Nebeling *Associate Professor, University of Michigan School of Information; Information Interaction Lab* Creates rapid prototyping tools that let non-programmers make AR/VR from paper, Play-Doh and video (ProtoAR, 360proto, XRDirector). #### ProtoAR — Michael Nebeling (2018) - Video: https://www.youtube.com/watch?v=XSmxJHmlaOs - Interaction: Drawing & Making, Tangible Objects - Platform & tech: Phone, photogrammetry, ARCore, paper prototyping - Idea: Draw on paper, sculpt with clay, snap a photo, and get a working AR prototype. - What it is: Designers sketch on paper and shape Play-Doh, capture them with a phone, and immediately place the captured 2D/3D assets in a working mobile AR prototype. - Technique: Captures sketches as image layers and clay models via multi-angle photos turned into 3D, then streams them into a marker/plane-based AR viewer. - Try it: Have students draw interfaces with pen and paper and sculpt characters from clay, scan them with a phone (such as Reality Composer or Polycam), and arrange them in AR for a demo. Twist: build an AR tour prototype in 30 minutes. #### 360proto — Michael Nebeling (2019) - Video: https://www.youtube.com/watch?v=CY9xwOA5cmQ - Interaction: Drawing & Making, Portals & Worlds - Platform & tech: Phone, Headset, 360 paper templates, WebXR - Idea: Sketch a 360-degree scene on paper, snap a photo, and step inside it in a headset. - What it is: Paper sketches drawn on equirectangular 360 grids are photographed and instantly viewed as immersive VR/AR scenes, with layers for interactive prototypes. - Technique: Maps hand-drawn sketches on a distorted 360 grid template onto a sphere around the viewer, with layered sketches for AR overlays. - Try it: Print an equirectangular grid sheet, have students draw a room on it, then photograph it and browse it in a phone panorama viewer. Twist: overlay a second sketch layer of 'AR labels' on the same scene. #### MRAT: Mixed Reality Analytics Toolkit — Michael Nebeling (2020) - Video: https://www.youtube.com/watch?v=DG5pqrQPdBc - Interaction: Information & UI, Gaze & Attention, Spatial Mapping - Platform & tech: Headset, Unity, HoloLens, interaction logging - Idea: Draw where users walked and looked in MR as traces in space. - What it is: A toolkit that logs and visualises how users move, look and interact in mixed reality apps, replaying their paths and gaze as 3D traces in the room. - Technique: Instruments Unity MR apps to record head pose, gaze and input events and renders them as in-situ 3D heatmaps and trails. - Try it: Use phone AR to record where a classmate's phone moves during a treasure hunt in the classroom, then replay it in AR as colored trails. Twist: use color to show dwell time and discuss the design problems it reveals. #### XRDirector — Michael Nebeling (2020) - Video: https://www.youtube.com/watch?v=mMp_hzLp3Yc - Interaction: Shared & Social, Performance, Drawing & Making - Platform & tech: Phone, Headset, shared AR, HoloLens, role-based authoring - Idea: Split roles like on a film set: director, actors and camera rehearse together in AR. - What it is: A role-based collaborative authoring tool where one person acts as director, others as actors or camera, puppeteering virtual characters in AR together like filming a scene. - Technique: Synchronises a shared AR scene across devices and assigns each device a role with its own controls for moving characters, cameras and timelines. - Try it: Work in groups with Reality Composer or a shared AR app: one person acts as director and calls out instructions while two others each control a virtual character with a phone to rehearse a 30-second skit. Twist: at the end, the camera operator screen-records the final cut on their phone. #### XRStudio — Michael Nebeling (2021) - Video: https://www.youtube.com/watch?v=bYLXVaLTdMo - Interaction: Performance, Shared & Social, Information & UI - Platform & tech: Headset, Desktop, virtual production, mixed reality capture, live streaming - Idea: Mix a teacher's XR lecture into a live stream for students to watch. - What it is: A virtual production system for teaching in XR: the instructor's immersive session is mixed with real camera video and live-streamed so remote students can follow. - Technique: Composites the instructor's tracked XR view with external camera footage and scene cameras into multiple switchable streams. - Try it: Use OBS to combine a classmate's phone AR view with the classroom camera into a live stream while they explain a 3D model in AR. Twist: viewers change the model's color in AR through live chat comments. #### Reframe — Michael Nebeling (2024) - Video: https://www.youtube.com/watch?v=TFdgx44l6FE - Interaction: Drawing & Making, Information & UI, Shared & Social - Platform & tech: Phone, Headset, AR storyboarding, threat modeling - Idea: Act out privacy-threat scenarios with AR storyboards in real space. - What it is: An AR storyboarding tool where designers place characters and events in the real space to act out and analyse security and privacy threats of AR apps. - Technique: Provides placeable character and annotation primitives in AR plus a threat-modelling framework to structure character-driven scenarios. - Try it: Use phone AR to place virtual figures for 'user', 'bystander' and 'attacker' in the classroom, act out a scene of someone secretly filming with AR glasses, and discuss how to design protections. Twist: each group designs a privacy patch for another group's scene. ### Microsoft Research Cambridge / I3D — Shahram Izadi *Led the Interactive 3D Technologies group at Microsoft Research (KinectFusion, HoloDesk, Holoportation); now leads Android XR at Google* Shahram Izadi's teams at Microsoft Research built KinectFusion real-time 3D reconstruction, the HoloDesk situated see-through display and Holoportation, live 3D capture of people streamed into HoloLens. He later co-founded perceptiveIO and now leads AR/XR at Google. #### KinectFusion — Microsoft Research Cambridge / I3D — Shahram Izadi (2011) - Video: https://www.youtube.com/watch?v=RSh8Voanp3c - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Kinect, GPU, dense SLAM - Idea: Sweep a Kinect around once to rebuild the whole room live, and virtual particles can hit the real table. - What it is: Sweep a Kinect around a room and a dense 3D model is built in real time; virtual particles then bounce off and splash onto real surfaces. - Technique: Depth frames are aligned with ICP to estimate camera pose and fused into a GPU truncated signed distance function (TSDF) volume, whose ray-cast surface serves as a collision mesh for virtual particles. - Try it: Use AR Foundation Meshing on a LiDAR iPhone to generate a live room mesh, then scatter thousands of virtual particles and let them bounce and pile up on the real table and sofa. Twist: replace the particles with something that 'grows' (moss, vines) and spreads along the mesh. #### HoloDesk — Microsoft Research Cambridge / I3D — Shahram Izadi (2012) - Video: https://www.youtube.com/watch?v=OlSyCoqOmhU - Interaction: Hands & Body, Tangible Objects - Platform & tech: Desktop, Kinect, beam splitter, head tracking - Idea: Reach under a half-mirror and catch and toss virtual balls with your real hands. - What it is: A see-through display over a desk lets you reach under it and juggle, grasp and share virtual 3D objects with your real hands, with correct occlusion. - Technique: A beam splitter shows a display's image in the volume under it, a head-tracking camera provides view-dependent rendering, and a Kinect's depth of the hands is converted into particle proxies that physically interact with and occlude virtual objects. - Try it: Build a "Pepper's ghost" box from a tablet and a sheet of acrylic tilted at 45 degrees, and feed the hand keypoints detected by MediaPipe into a physics engine so your hand can nudge virtual cubes inside the box. Twist: put a real object in the box and make the virtual objects move around it. #### SemanticPaint — Microsoft Research Cambridge / I3D — Shahram Izadi (2015) - Video: https://www.youtube.com/watch?v=z_TcWC7yjj0 - Interaction: Spatial Mapping, Voice & Sound, Hands & Body - Platform & tech: Desktop, KinectFusion, online learning - Idea: Point and speak while scanning a room, and the system learns to recognize every piece of furniture. - What it is: While scanning a room, the user touches objects and names them by voice; the system learns and labels chairs, tables and walls in the live 3D map. - Technique: A KinectFusion-style reconstruction is labeled online: user touch plus a spoken label seeds voxels, and a streaming random-forest classifier with CRF smoothing propagates labels across the 3D map. - Try it: Scan a room with AR Foundation Meshing, let the user tap a patch of mesh and say its name (speech to text), color that region and store the label, and also show the detected plane classifications (table, wall, floor). Twist: tag objects with emotions or memories instead of names. #### Holoportation — Microsoft Research Cambridge / I3D — Shahram Izadi (2016) - Video: https://www.youtube.com/watch?v=84ngPFBWLVI - Interaction: Shared & Social, Hands & Body - Platform & tech: Headset, HoloLens, depth camera rig, Fusion4D - Idea: Teleport a faraway person live as a life-size hologram standing in front of you. - What it is: Rings of depth cameras capture people in full 3D and stream them live into HoloLens, so a remote person appears life-size in your room; conversations can be recorded and replayed at miniature scale. - Technique: Multiple RGB-D camera pods capture a person from all sides, the Fusion4D pipeline fuses them into a temporally consistent textured mesh in real time, and it is compressed and streamed to a HoloLens where it is rendered anchored in the room. - Try it: Record a volumetric video (or depth point cloud) with Record3D or Polycam on an iPhone, import it into Unity, and play it back at reduced scale on a tabletop in HoloKit or phone AR, like a "mini you" talking. Twist: make the playback pausable and viewable from behind, and record a message you want to leave for your future self. #### Mobile Holoportation — Microsoft Research Cambridge / I3D — Shahram Izadi (2017) - Video: https://www.youtube.com/watch?v=nTkFO2xNkIk - Interaction: Shared & Social - Platform & tech: Headset, HoloLens, depth cameras, LTE streaming - Idea: A person is holographically "teleported" live even from inside a moving car. - What it is: A car-mounted version of Holoportation streams a passenger's live 3D capture over a mobile network into a remote HoloLens. - Technique: Depth cameras in the car capture the passenger, and the mesh is compressed aggressively (low-resolution geometry plus video-coded textures) to fit an LTE uplink before reconstruction on the remote HoloLens. - Try it: Capture a point cloud of a person with a phone's front depth camera (or MediaPipe depth estimation), send it at very low bandwidth over WebRTC to another phone for display in AR, and test the lowest acceptable point count and frame rate. Twist: send only the skeleton and voice, and rebuild the person as a cartoon avatar on the receiving end. #### Android XR glasses live demo (TED) — Microsoft Research Cambridge / I3D — Shahram Izadi (2025) - Video: https://www.youtube.com/watch?v=gElClXpg4J0 - Interaction: Information & UI, Voice & Sound, Gaze & Attention - Platform & tech: Headset, Android XR, Gemini - Idea: Walk on stage wearing AI glasses that remember where you just put your keys. - What it is: On the TED stage Izadi demos prototype Android XR glasses with Gemini: live translation, memory of where you left objects, and contextual help overlaid in a see-through display. - Technique: The glasses stream camera frames and audio to a multimodal model (Gemini), which returns translations, object memory and answers that are rendered as text in a monocular see-through display. - Try it: Build a 'memory glasses' prototype with a phone camera and a multimodal LLM API: capture a frame every few seconds and have the model log where objects are, so that when you ask 'where are my keys?' it marks the last place it saw them in the AR view. Twist: have it remember only one thing you often lose, and discuss where the privacy boundaries should be. ### Mirza Beig *Technical artist and VFX developer for Unity* Unity VFX developer who posts shader and particle experiments almost every week and releases many of them as free open-source projects. #### GPU Fog Particles — Mirza Beig (2021) - Video: https://x.com/TheMirzaBeig/status/1471820398056677376 - Source code: https://github.com/MirzaBeig/GPU-Fog-Particles - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Shader, Particle System - Idea: Fog made purely from noise can hug the world without hard edges. - What it is: Soft atmospheric fog drifts through a scene without any textures, blending smoothly into geometry and fading as the camera moves into it. - Technique: Particle quads use a shader that samples animated 3D noise, softens intersections with scene depth, and fades near the camera instead of using sprite textures. - Try it: Spawn the fog particles on the AR floor plane with depth softening against the AR occlusion depth, so fog pools around real furniture. Twist: the fog parts where people stand. #### Anime Speed Lines — Mirza Beig (2022) - Video: https://user-images.githubusercontent.com/37354140/151656459-d99d2d36-06ed-4889-8a10-17cabd1626d8.mp4 - Source code: https://github.com/MirzaBeig/Anime-Speed-Lines - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, Post-processing - Idea: Borrow the manga shorthand for speed and surprise as a live camera effect. - What it is: Manga-style speed lines burst from the screen edges toward the centre, flickering and pulsing as if the viewer were rushing forward. - Technique: A full-screen post-process generates radial lines procedurally in polar coordinates with animated noise, masking the centre with an adjustable vignette. - Try it: Apply the speed lines to a phone's AR camera feed and trigger them when the phone moves fast. Twist: the lines point toward the nearest detected face. #### Post-Processing Scan — Mirza Beig (2022) - Video: https://x.com/TheMirzaBeig/status/1809173668456792524 - Source code: https://github.com/MirzaBeig/Post-Processing-Scan - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, Unity, URP, Post-processing - Idea: A single expanding sphere reveals the hidden shape of the world. - What it is: A glowing sonar pulse sweeps outward through a scene, lighting up every surface it passes like a 3D scanner. - Technique: A URP post-process reconstructs world position from the depth buffer and draws bands where the distance to the scan origin crosses a moving radius (a spherical SDF). - Try it: Run the scan over the AR camera's depth (LiDAR or ARCore depth) so a pulse ripples across the real room from where the user taps. Twist: the pulse speed follows the loudness of a clap. #### Chromatic Distortion Sphere — Mirza Beig (2024) - Video: https://www.youtube.com/watch?v=IkBZLo4ROU0 - Source code: https://github.com/MirzaBeig/Chromatic-Distortion-Sphere - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, Shader Graph - Idea: A sphere that refracts the world with a rainbow edge instantly reads as magic. - What it is: A floating bubble bends the scene behind it and splits the light into rainbow fringes, wobbling like a soap bubble. - Technique: The shader samples the opaque scene texture with offsets along the surface normal, separately for red, green and blue, and adds a Fresnel rim and vertex wobble. - Try it: Float the chromatic bubble in phone AR so it distorts the real camera image behind it, and let users pop it with a tap. Twist: the bubble follows the user's hand. #### Constellation Plexus Ribbons — Mirza Beig (2024) - Video: https://x.com/TheMirzaBeig/status/1840974568216813609 - Source code: https://github.com/MirzaBeig/Constellation-Plexus - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, Unity, URP, Particle System - Idea: Connect nearby points with light and a cloud of dots becomes a constellation. - What it is: Glowing ribbons connect drifting particles into constellations that follow the mouse, like star maps being drawn in the air. - Technique: Unity's particle system emits particles with trails, and a script connects neighbouring particles with ribbon lines whose opacity depends on distance (plexus effect). - Try it: Attach the plexus emitter to a tracked hand or the phone in AR so students draw constellations around the room. Twist: each student's constellation links to the others when they stand close. #### Galaxy Water — Mirza Beig (2026) - Video: https://x.com/TheMirzaBeig/status/2008528798397198550 - Source code: https://github.com/MirzaBeig/Galaxy-Water - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Desktop, Unity, URP, Shader - Idea: Water that reflects a universe instead of the sky. - What it is: A liminal platform stands in dark water that reflects a starry galaxy, with sparkling particles drifting above the surface. - Technique: A water shader combines normal-map ripples with a procedural starfield in the reflection, plus particle sparkles, in URP. - Try it: Turn a real floor into galaxy water in phone AR by placing the shader on the detected plane with people occlusion. Twist: footsteps create ripples of stars. ### Molmol Kuo *Artist; co-founder of YesYesNo; SFPC teacher* Taiwanese-born artist working with AR, body tracking and storytelling; co-founded the interactive studio YesYesNo with Zach Lieberman and was artist-in-residence for AR at Google Arts & Culture and Adobe. #### Audio Walking Tour (AR) — Molmol Kuo (2017) - Video: https://www.youtube.com/watch?v=d52-71PXlmg - Interaction: Voice & Sound, Hands & Body - Platform & tech: Phone, ARKit, openFrameworks - Idea: Walk into oral histories of border crossing through AR and your own movement. - What it is: An AR documentary piece in which border-crossing interview stories are tied to paper documents and body movement, so listeners walk and move through the stories in space. - Technique: Image tracking recognises the printed documents as anchors, and ARKit world tracking positions audio clips relative to them so that the listener's movement between spots selects and pans the stories. - Try it: Interview a classmate about a migration or moving story, split the recording into segments, and attach them to image anchors on a few paper documents or tickets (AR Foundation Image Tracking or 8th Wall), so you hear whichever segment belongs to the paper you walk up to. Twist: make one segment audible only when you crouch or bend down. ### Morten Just *Designer & indie developer (Rotato, Design Camera); former Google designer* Danish designer and indie Mac/iOS developer who prototypes interface ideas in public; his ARKit experiments led to the Design Camera and Rotato apps. #### Augmented Reality Web Pages on Physical Books — Morten Just (2018) - Video: https://www.youtube.com/watch?v=OTqkN254mrc - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, ARKit 2, image tracking - Idea: Recognize a physical book and wrap its web page information around it. - What it is: A physical book is recognised and wrapped with its live web page—reviews and details float around the cover. - Technique: The book cover is tracked as a reference image, and web content (reviews, details) is rendered as panels around the cover's tracked pose. - Try it: Recognize a book cover and show a card with its rating and a one-line review next to it. Twist: show a sentence a classmate wrote about the book instead. #### Counting Foreign Currency in AR — Morten Just (2018) - Video: https://www.youtube.com/watch?v=HneMHLBj_lQ - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, ARKit 2, image tracking - Idea: Recognize foreign banknotes on a table and show the converted amount right on each note. - What it is: Banknotes on a table are recognised and their converted value is shown on top of each note. - Technique: Each banknote design is a reference image, so detected notes get an anchor with a label showing its value converted with an exchange rate. - Try it: Use image tracking to recognize a few kinds of banknotes (or homemade token cards), show the converted amount on each, and add up the total. Twist: express the amount as how many cups of bubble tea it buys. #### Design Camera 3D + AR — Morten Just (2018) - Video: https://www.youtube.com/watch?v=ShWBcbvUGnc - Interaction: Information & UI, Spatial Mapping - Platform & tech: Phone, ARKit, SceneKit - Idea: Place a realistic iPhone showing your app design on any real table and photograph it. - What it is: Places an infinitely scalable, photoreal iPhone showing your app design on any real surface for mockup photography. - Technique: A photoreal phone model with the app design as a screen texture is placed on a detected plane, using light estimation and contact shadows for realistic mockup shots. - Try it: Use Reality Composer or AR Foundation to place a phone model showing your own design screenshots on a table, add shadows and take a photo. Twist: scale it up to the size of a building and take another shot. #### Instructions in AR — Morten Just (2018) - Video: https://www.youtube.com/watch?v=Xwm4vidjtp0 - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, ARKit 2, image tracking - Idea: Stick handwritten instructions onto the real device they explain. - What it is: Hand-written technical instructions are attached to the real device they explain, using live image tracking. - Technique: A photo of the device's panel is used as a reference image, and handwritten instruction images are attached as overlays at fixed offsets on the tracked image. - Try it: Photograph a printer or coffee machine panel as the recognition image and pin handwritten step instructions next to the matching buttons. Twist: steps appear one at a time, and each tap reveals the next. #### Kitchen Timers in Augmented Reality — Morten Just (2018) - Video: https://www.youtube.com/watch?v=3c114LvhJIA - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, ARKit 2 - Idea: Float countdown timers above the pots cooking on the stove. - What it is: Countdown timers float above real pots on the stove, rendered with motion blur, adaptive lighting and depth of field. - Technique: Timers are placed as world anchors above detected pots, and renderer features (light estimation, motion blur, depth of field) make the overlay match the camera image. - Try it: Tap on a pot to place a floating countdown that turns red and rings when time is up. Twist: run several timers at once, automatically sorted and highlighted by time remaining. #### Stenciling in Augmented Reality — Morten Just (2018) - Video: https://www.youtube.com/watch?v=KjWL_zHOysM - Interaction: Drawing & Making, Tangible Objects - Platform & tech: Phone, ARKit - Idea: Cast a reference image onto paper so anyone can trace a good drawing. - What it is: Projects a reference image onto paper through the phone so you can trace it by hand, giving instant drawing skills. - Technique: A semi-transparent reference image is anchored to the paper's plane (via plane detection or image tracking), so it stays registered while the user traces it by hand. - Try it: Use plane detection to fix a semi-transparent line drawing onto a sheet of white paper on the desk, and trace it. Twist: show the line drawing in steps, outline first, then details. ### Nam June Paik (白南准) *Artist; founder of video art* Korean-American artist (1932–2006) of Fluxus who turned television into a sculptural, playable material: magnets bending the picture, TVs growing in a garden, a Buddha watching himself on a closed circuit, robots and TV cellos in performance. #### Participation TV — Nam June Paik (白南准) (1963) - Video: https://www.youtube.com/watch?v=DimYjLdQMHc - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, television, microphone, amplifier - Idea: One of the first artworks where the audience's voice draws on a screen; sound-reactive AR effects start here. - What it is: Visitors speak or make sounds into a microphone and the signal explodes into bursts of lines and shapes on a television screen. - Technique: Audio from the microphone is fed into the television's deflection circuits, so sound modulates the electron beam and generates live abstract imagery. - Try it: Build a voice-reactive AR effect where the user's microphone input draws bursting line patterns on a detected wall, with loudness controlling size and pitch controlling colour. Twist: let two people's voices collide in the middle of the wall. #### Robot K-456 — Nam June Paik (白南准) (1964) - Video: https://www.youtube.com/watch?v=JciiqCsRhdY - Interaction: Performance, Location & City - Platform & tech: Desktop, radio control, motors, speaker - Idea: Taking a performing machine into the street makes the city its stage; location-based AR characters can perform in public space the same way. - What it is: A remote-controlled humanoid robot built from scrap walks the streets, plays a speech by John F. Kennedy and excretes beans; in a 1982 performance it was 'killed' by a car. - Technique: A radio-controlled frame with 20 channels moves the robot's limbs and plays tape audio, operated live by Paik and engineer Shuya Abe. - Try it: Place a clumsy AR robot on a geolocated street corner (ARCore Geospatial or Niantic Lightship) that recites a short speech to whoever arrives. Twist: let it 'die' dramatically if a real car passes within the camera view. #### Magnet TV — Nam June Paik (白南准) (1965) - Video: https://www.youtube.com/watch?v=44DJOtaU-b8 - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Desktop, television, electromagnet - Idea: A physical object distorts a screen image in real time; AR can let real objects act as lenses or magnets that bend virtual content around them. - What it is: A large magnet placed on top of a television bends its electron beam into swirling abstract shapes; moving the magnet changes the image. - Technique: The magnetic field deflects the cathode-ray tube's electron beam, so the broadcast image warps into patterns that follow the magnet's position. - Try it: Track a real object (image or object tracking in ARKit or Lens Studio) and use its screen position to warp the live camera feed around it like a magnet on a CRT. Twist: add a second object that attracts the distortion. #### TV Cello — Nam June Paik (白南准) (1971) - Video: https://www.youtube.com/watch?v=-9lnbIGHzUM - Interaction: Performance, Voice & Sound - Platform & tech: Desktop, television monitors, cello strings, video - Idea: An instrument that is also a screen merges performer, sound and image into one object, a model for AR instruments that show what they play. - What it is: Charlotte Moorman plays a cello made of stacked televisions; as she bows the strings, the screens show live and recorded images of herself and the performance. - Technique: Three television monitors are stacked in a cello-shaped frame with strings, combining live closed-circuit video with a playable instrument. - Try it: Make an AR instrument that appears on a real table: plucking its virtual strings (hand tracking or touch) plays notes and shows a live camera image of the player inside its body. Twist: let a second phone see and play the same instrument. #### TV Buddha — Nam June Paik (白南准) (1974) - Video: https://www.youtube.com/watch?v=MVwR1n0IZAs - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Desktop, closed-circuit video camera, television, Buddha statue - Idea: A closed loop between object, camera and screen makes watching itself the subject; AR mirror and selfie effects can be designed around who is watching whom. - What it is: A Buddha statue sits facing a television that shows a live camera image of the Buddha itself, locked in endless self-contemplation. - Technique: A closed-circuit camera feeds the statue's image straight back to a monitor in front of it, creating a real-time feedback loop without recording. - Try it: Place a virtual character in AR that watches a virtual TV showing the phone camera's live view of the character itself. Twist: when a real person steps into the character's view, the character turns to look at them instead. #### TV Garden — Nam June Paik (白南准) (1974) - Video: https://www.youtube.com/watch?v=KtWmmoYyYzI - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, television monitors, live plants, video - Idea: Screens planted among living things become part of an ecosystem; AR content feels more at home when it is distributed through the real environment instead of floating in front of it. - What it is: Dozens of televisions face up among live plants in a dark room, glowing like flowers and playing the fast-cut video Global Groove. - Technique: Monitors are laid among real plants at ground level, turning the screens into luminous blossoms in a garden landscape. - Try it: Plant 20 small AR video screens among real houseplants using plane and object detection, each playing a short clip and glowing like a flower. Twist: make each screen turn toward the nearest person. ### Olivier Ratsi *Visual artist (AntiVJ label; projection and perspective installations)* French artist and member of the AntiVJ visual label whose installations question how we perceive space. His projects Deconstruction Time Again and Echolyse use projection, LEDs and physical frames to build shapes that only become whole, or fall apart, depending on where the viewer stands, as in Onion Skin, DELTΔ, Shape and Frame Perspective. #### White Roads in the Red Matrix — Olivier Ratsi (2012) - Video: https://vimeo.com/52538265 - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Projection, LED, custom software, installation - Idea: A few lines of light can fake a deep space; AR designers can suggest depth with perspective cues alone before adding any 3D model. - What it is: Part of Ratsi's Deconstruction Time Again project: white lines of light run through a red grid in a dark space, forming and breaking perspective 'roads' that seem to recede into a depth that is not physically there. - Technique: Custom LED lines driven by software (developed by Anthony Gouvrillon, LEDs by Julien Guinard) animate perspective grids synchronised with Thomas Vaquié's sound. - Try it: Draw a perspective grid of glowing lines on a real wall in WebXR so it looks like a tunnel going through the wall. Twist: the vanishing point follows the viewer's head, so the tunnel always points at them. #### Onion Skin (Echolyse) — Olivier Ratsi (2013) - Video: https://vimeo.com/155308126 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Projection, projection mapping, 5.1 sound, physical corner - Idea: A projected image can cancel or exaggerate the real geometry it sits on; AR can flatten, fold or unfold a room corner in the same way. - What it is: The first piece of the Echolyse project: two walls meeting at a right angle are covered with projected lines and planes that, from the right position, flatten the corner into a single image, then peel it apart in layers like an onion's skin, with 5.1 sound by Thomas Vaquié. - Technique: Content designed for one viewpoint is projected onto a physical two-wall module so the corner reads as a flat plane (anamorphosis), then animated to reveal the depth again. - Try it: Use a room corner and WebXR plane detection to place an image that looks flat from one spot and breaks apart as you move. Twist: add a floor marker for the 'sweet spot' and let viewers find it without instructions. #### DELTΔ (Echolyse) — Olivier Ratsi (2015) - Video: https://vimeo.com/155655808 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Projection, projection mapping, wood structure, sound - Idea: A simple geometric frame can act as a portal once light suggests depth inside it; AR portals work best when they sit in a clear physical frame. - What it is: A site-specific installation in the Echolyse series built around the triangle of the Greek letter delta, originally a sign for a door: a wooden structure is mapped with single and multiple projections so the triangle opens and closes like a passage. Year approximate (Vimeo upload 2016). - Technique: Projection content is mapped onto a painted wood triangle structure, with light and shadow animations creating the illusion of openings and depth, synchronised to sound by Thomas Vaquié. - Try it: Build a triangular doorway from tape or cardboard and use image tracking in Lens Studio to turn its inside into an AR portal to another space. Twist: the portal only opens when the viewer is exactly centred in front of it. #### Shape — Olivier Ratsi (2017) - Video: https://vimeo.com/248975465 - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Projection, projection, fog, black polygons - Idea: An illusion that works only from one spot turns the viewer into a participant who must search; AR anchors can reward a precise viewpoint in the same way. - What it is: In a dark room filled with fog, large black polygons hang in space. A point marked on the floor shows where to stand: from there, the fragments and the light join into a perfect rectangle; step aside and the shape explodes. - Technique: Black planes are hung at different depths so that, together with projected light in haze, they line up into a rectangle only from the marked viewpoint (anamorphosis). - Try it: Scatter virtual fragments across a room in Unity AR Foundation so they align into a word only from one marked spot. Twist: the word changes each time a new viewer finds the spot. #### Fragmented Plane — Olivier Ratsi (2018) - Video: https://vimeo.com/260737007 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Projection, projection, physical planes, sound - Idea: One flat image spread over many depths shows how fragile a 2D reading of 3D space is; AR designers can play with that moment of breaking. - What it is: A large installation in which a projected plane is broken across several physical surfaces at different depths; the image reads as one flat plane from the front and as scattered fragments as the audience moves around it. Sound by Thomas Vaquié. - Technique: Projection content is mapped from a single viewpoint onto multiple surfaces placed at different distances, so it only coheres from the front (method likely anamorphic mapping). - Try it: Place several AR panels at different depths with WebXR so that together they show one photo from the doorway, then fly apart as viewers walk in. Twist: the photo is the view the viewer would see if the room were not there. #### Frame Perspective — Olivier Ratsi (2019) - Video: https://vimeo.com/314461942 - Interaction: Perception & Effects, Location & City - Platform & tech: Projection, anamorphic painting, light, urban installation - Idea: An anamorphic frame anchored to real buildings is AR without a screen: it only 'works' from the viewpoint the designer chose. - What it is: Coloured frames painted and lit across real architecture — walls, courtyards, streets — only line up into perfect rectangles from one point; from anywhere else they fall apart into broken shapes. Versions were shown in Metz, at Bristol Light Festival and in a drone-filmed version. - Technique: Rectangles are projected from the chosen viewpoint onto the architecture and then painted or lit along the projection lines, so they align only from that spot (anamorphosis). - Try it: Pick a courtyard, stand at one spot and use a phone AR app to place a floating rectangular frame that spans several walls; then walk around to see it break. Twist: mark the spot with nothing, and let people discover it by looking through the app. ### Pavlo Tkachenko *AR developer and Snap Spectacles lens creator* AR developer who builds Spectacles lenses, often with Stijn Spanhove, and won a Spectacles Community Challenge with his generative lens Bubbles. #### BPlane Adventures — Pavlo Tkachenko (2025) - Video: https://www.youtube.com/watch?v=f-3du7Fy3kA - Interaction: Gaze & Attention, Hands & Body, Play - Platform & tech: Headset, Snap Spectacles, Lens Studio - Idea: Steer a small plane by turning your head and boost it with your hand as it weaves through the space around you. - What it is: A relaxing flow game for Spectacles: you steer a small plane with head movements and use your hands to boost it through the space around you. - Technique: Head orientation steers the plane's heading, while a hand gesture triggers a speed boost, both mapped in world space around the user. - Try it: Use head orientation (HoloKit or the phone gyroscope) to fly a paper plane through AR rings in the room, opening your palm to speed up. Twist: generate ring positions automatically from the room's planes. #### Bubbles — Pavlo Tkachenko (2025) - Video: https://www.youtube.com/watch?v=TAPU-f4w7xs - Interaction: Drawing & Making, Hands & Body - Platform & tech: Headset, Snap Spectacles, Snap3D, Lens Studio - Idea: Type some words to generate 3D objects, then fill the space around you with them, each sealed in a bubble. - What it is: A Spectacles lens where you type a prompt, generate 3D models with Snap3D generative AI, and place several objects around you inside floating bubbles. - Technique: A text prompt is sent to a text-to-3D service, and the returned mesh is scaled into a transparent sphere container placed around the user via world tracking. - Try it: Generate three small objects with a text-to-3D tool and place each in a transparent bubble in AR, so an object only drops to the ground when its bubble is popped. Twist: have classmates set the prompts for each other's bubbles. ### Refik Anadol *Media artist; founder of Refik Anadol Studio* Turkish-American artist who turns architecture into 'data paintings' with large-scale projection and, later, AI-generated 'data sculptures' on building façades and immersive rooms. #### Visions of America: Amériques — Refik Anadol (2014) - Video: https://www.youtube.com/watch?v=U-9VAPC92Bw - Interaction: Performance, Hands & Body, Projection - Platform & tech: Projection, Kinect, real-time generative visuals - Idea: Let a conductor's gestures conduct the projections across an entire concert hall in real time. - What it is: Inside Walt Disney Concert Hall, projections explode across the hall in response to the gestures and body language of conductor Esa-Pekka Salonen performing Varèse's Amériques. - Technique: A depth camera (Kinect) tracks the conductor's gestures and maps them to real-time generative visuals projected onto the hall's interior surfaces. - Try it: Use MediaPipe Pose to track a classmate's "conducting" arm movements and control the bursts and direction of projected particles. Twist: have the class play a simple rhythm, with the conductor's gestures deciding which group's visuals light up. #### Infinity Room — Refik Anadol (2015) - Video: https://www.youtube.com/watch?v=p9Cj1PdmtMA - Interaction: Projection, Perception & Effects, Voice & Sound - Platform & tech: Projection, projection, mirrors, generative graphics - Idea: Projection and mirrors make a small room look endless. - What it is: A mirrored room whose walls are flooded with projected generative patterns, creating the sensation of floating in an infinite, shifting space. - Technique: Projectors cover the walls and floor while mirrored surfaces multiply the generative, sound-driven imagery to dissolve the room's boundaries. - Try it: Line a cardboard box with mirror paper, project patterns into it with a phone or small projector, and watch the infinite reflections. Twist: make the patterns change with the spectrum of the background music. #### Melting Memories — Refik Anadol (2018) - Video: https://www.youtube.com/watch?v=__l2sl5Z1GQ - Interaction: Information & UI, Hands & Body, Perception & Effects - Platform & tech: Projection, EEG, LED wall, generative graphics - Idea: Biosignals become compelling when turned into matter-like forms; AR can visualise a user's heartbeat or breath as an object in the room instead of a number. - What it is: Large LED screens show slowly shifting, sculptural forms made from EEG recordings of people's brains while they recalled memories. - Technique: EEG signals recorded during memory recall are processed into parameters that drive 3D noise fields rendered as data sculptures on LED walls. - Try it: Use a phone camera heart-rate app or the microphone to capture a user's pulse or breath, and map it to the noise of a soft 3D blob floating beside them in AR. Twist: record one minute and replay it next to the live one, so past and present calm can be compared. #### WDCH Dreams — Refik Anadol (2018) - Video: https://www.youtube.com/watch?v=PuMVVsoiLPM - Interaction: Projection, Information & UI, Location & City - Platform & tech: Projection, machine learning, projection mapping, 42 projectors - Idea: Let a concert hall use AI to 'dream' its hundred years of memories and project them onto its facade. - What it is: For the LA Phil's centenary, 45 terabytes of the orchestra's archives are processed by machine learning and projected onto the exterior of the Walt Disney Concert Hall as the building's 'dreams'. - Technique: Machine-learning models cluster and morph archival images and audio, and the results are projection-mapped with 42 projectors onto Gehry's curved stainless-steel façade. - Try it: Collect old school photos, generate a sequence with simple image clustering or style blending, and project it onto a campus wall. Twist: let passers-by vote on their phones for the theme of the next 'dream'. #### Machine Hallucination — Refik Anadol (2019) - Video: https://www.youtube.com/watch?v=x1EVhNM-uf4 - Interaction: Information & UI, Projection, Perception & Effects - Platform & tech: Projection, GAN, projection mapping, photo archive - Idea: Walking through a model's latent space makes an invisible dataset feel like a place; AR can present data as an environment you stand inside rather than a chart. - What it is: At ARTECHOUSE New York, walls and floor of a large room are filled with a moving 'dream' of the city generated by a neural network trained on millions of photographs of New York. - Technique: A GAN trained on a large photo archive of New York generates latent-space walks that are rendered and projection-mapped across walls and floor. - Try it: Collect 200 photos of your campus, train or fine-tune a small image model (or use an online GAN tool), and play the interpolation video on a floating AR panel at the real location. Twist: the interpolation step follows the user's walking speed. #### Quantum Memories — Refik Anadol (2020) - Video: https://www.youtube.com/watch?v=2Rlgphw9Jxw - Interaction: Perception & Effects, Information & UI, Projection - Platform & tech: Projection, GAN, quantum noise, LED screen - Idea: A flat screen can feel volumetric when the content is rendered for one sweet spot; AR designers can use forced perspective to fake depth on flat surfaces. - What it is: At the NGV Triennial in Melbourne, a giant screen shows waves of nature imagery that seem to push out of the wall, generated from hundreds of millions of landscape photos combined with noise from a quantum computer. - Technique: Images generated by a GAN are placed in a 3D point field whose motion uses quantum-computing noise, and rendered with forced perspective for the room. - Try it: Pin a flat AR panel to a real wall and render a point cloud behind it that only looks deep from one marked spot on the floor. Twist: when the user steps off the spot, the illusion breaks and the points spill out into the room. ### Robert Lepage / Ex Machina *Theatre director; multidisciplinary production company* Québec director Robert Lepage founded Ex Machina in 1994 to mix theatre, film, opera and new media. His productions use projection, mirrors and moving machines to turn a small set into many places, from The Far Side of the Moon to the Metropolitan Opera Ring cycle and the giant Image Mill projection on Québec's grain silos. #### The Far Side of the Moon (La Face cachée de la Lune) — Robert Lepage / Ex Machina (2000) - Video: https://www.youtube.com/watch?v=K_HI70pLYhc - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, projection, mirrors, stage machinery - Idea: One reconfigurable surface can stand for many places; AR can reuse a single real wall the same way instead of building new sets. - What it is: Robert Lepage plays two brothers in a solo show where a sliding mirrored wall and projections turn a laundromat, an apartment and zero gravity into the same small set. - Technique: A large tilting mirror reflects the performer lying on the floor so he appears to float, combined with video projection on moving panels. - Try it: Point a phone at a single blank wall and build three AR scenes (laundromat, bedroom, spacecraft) that swap when the user turns the phone. Twist: one scene flips gravity so objects fall toward the ceiling. #### The Andersen Project (Le projet Andersen) — Robert Lepage / Ex Machina (2005) - Video: https://www.youtube.com/watch?v=zMIhuuncY6I - Interaction: Performance, Projection - Platform & tech: Projection, projection, stage machinery - Idea: Small physical cut-outs in a projected world give the performer real places to enter, the way AR portals are strongest when framed by a real doorway. - What it is: A solo show based on Hans Christian Andersen's tales in which a curved projection screen and small physical openings let the performer step between Paris streets, an opera house and a peep-show booth. - Technique: Video is projected onto a concave screen with openings and moving elements that the actor passes through. - Try it: Find a real doorway and anchor an AR street scene inside it with image or plane tracking, so walking through the door changes the city. Twist: the city changes depending on which door you choose. #### Le Moulin à Images (The Image Mill) — Robert Lepage / Ex Machina (2008) - Video: https://www.youtube.com/watch?v=6JsNKXwavqY - Interaction: Projection, Location & City, Information & UI - Platform & tech: Projection, architectural projection, multi-projector, sound - Idea: An industrial ruin becomes a history book when you project the city's past onto it; site-specific AR can turn any large surface into a storytelling wall. - What it is: For Québec City's 400th anniversary, a 40-minute projection on the Bunge grain silos in the Old Port tells the city's history across a surface about 600 metres long, then one of the largest architectural projections ever made. - Technique: A bank of projectors covers the silo facade, with images and sound synchronised and broadcast to the audience across the harbour. - Try it: Choose the longest wall you can find, split its length into four decades, and anchor AR photos and sound for each decade along it so walking the wall walks through time. Twist: speed up the timeline when the viewer runs. #### Der Ring des Nibelungen (Metropolitan Opera) — Robert Lepage / Ex Machina (2010) - Video: https://www.youtube.com/watch?v=1_ssNfEXu_0 - Interaction: Performance, Projection, Spatial Mapping - Platform & tech: Projection, stage machinery, projection, interactive video - Idea: A moving physical surface plus projection gives virtual landscapes real slopes to walk on, the same pairing AR needs between content and real geometry. - What it is: Lepage's Ring cycle at the Metropolitan Opera uses 'The Machine', a set of 24 rotating planks, onto which projections of rivers, rocks and forests move as singers climb on it. - Technique: Hydraulic planks reconfigure into shapes while video, some of it responding to the singers' movements, is projected onto their surfaces. - Try it: Arrange a few tilted boards or stairs in the classroom, scan them with a LiDAR phone, and drape an AR river that flows down their real slopes. Twist: the river's direction flips when someone climbs onto a board. #### Needles and Opium (Les Aiguilles et l'opium) — Robert Lepage / Ex Machina (2013) - Video: https://www.youtube.com/watch?v=6jWBCnIHe9A - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, rotating set, projection, harness - Idea: Rotate the room and gravity looks different; AR can reorient a virtual room around the user for the same disorientation. - What it is: In this revival, performers move inside an open rotating cube whose walls receive projections of Paris hotel rooms and New York clubs, so the actor seems to float or walk on walls. - Technique: A three-sided cube rotates on its axis while projections on its inner walls keep the illusion of a fixed room, with actors on harnesses. - Try it: Build an AR room around the viewer in WebXR that slowly rotates 90 degrees while furniture stays glued to the walls. Twist: only one object obeys real gravity and falls. #### 887 — Robert Lepage / Ex Machina (2015) - Video: https://www.youtube.com/watch?v=B3mPXPkiJtk - Interaction: Performance, Projection, Tangible Objects - Platform & tech: Projection, miniature set, live video, projection - Idea: Scale jumps between a model and a life-size projection make memory spatial; AR can do this by letting a tabletop model open into a room-sized scene. - What it is: Lepage revisits his childhood apartment block at 887 Murray Avenue using a rotating miniature building, tiny figures filmed live and projections that zoom from model to full-size memory. - Technique: A detailed dollhouse-like set is filmed with small live cameras and the feed is projected at large scale behind the actor. - Try it: Make a paper model of your childhood home, anchor it in AR, and tap a window to expand that room to life size around you. Twist: add a recorded voice memory that only plays inside the life-size room. ### Ryoji Ikeda (池田亮司) *Artist and composer; data and light* Japanese artist and composer who turns raw data, sine tones and pure white light into massive immersive installations and performances such as test pattern, the transfinite and the searchlight piece spectra. #### the transfinite — Ryoji Ikeda (池田亮司) (2011) - Video: https://www.youtube.com/watch?v=q7SEYpWPczk - Interaction: Projection, Information & UI, Perception & Effects - Platform & tech: Projection, projectors, LED screens, sound - Idea: Scale changes meaning: the same data that is abstract on a laptop becomes overwhelming at 12 metres; AR can play with the size of information to make it felt. - What it is: In the Park Avenue Armory drill hall, a huge vertical screen and a floor are covered with streams of numbers, barcodes and flicker; visitors lie on the floor and walk behind the screen. - Technique: Multiple projectors display mathematically generated data visuals on a front screen and a floor, synchronised with a multichannel electronic score. - Try it: Pick a dataset (for example, every step count in the class) and place it in AR as a floor-to-ceiling wall of scrolling numbers, then shrink it to palm size with a pinch. Twist: let the wall scroll at the speed of the loudest sound in the room. #### superposition — Ryoji Ikeda (池田亮司) (2012) - Video: https://www.youtube.com/watch?v=0ivkmVDg4D0 - Interaction: Performance, Information & UI, Voice & Sound - Platform & tech: Projection, screens, live performers, real-time data - Idea: Performers can be data operators rather than dancers; in AR theatre, visible live input (typing, tapping) makes the digital layer feel caused by people. - What it is: A stage performance about quantum mechanics in which two performers operate devices, type and tap Morse code while a wall of screens shows equations, data and live camera images. - Technique: Live performer actions, cameras and signal generators are mixed with pre-computed visuals and sound on a synchronised multi-screen system. - Try it: Stage a five-minute AR performance where one student taps Morse code on a table and each tap spawns a floating symbol visible to the audience's phones. Twist: let the audience decode the message together. #### the radar — Ryoji Ikeda (池田亮司) (2012) - Video: https://www.youtube.com/watch?v=WNJpQVoU5ZE - Interaction: Location & City, Information & UI, Projection - Platform & tech: Projection, projection, astronomical data, outdoor screen - Idea: Showing the sky's hidden data under the real sky gives people a second sky; AR star maps work best when aligned with where the stars really are. - What it is: In Rio de Janeiro a large outdoor projection shows a slowly rotating radar-like map of the universe built from astronomical data, seen by crowds at night. - Technique: Astronomical survey data are rendered as a sweeping radar-style animation and projected at monumental scale outdoors (likely a single large screen). - Try it: Build a phone AR sky overlay that uses compass and gyroscope to place a rotating radar sweep over the real night sky, revealing labels of real stars as the sweep passes. Twist: let each student mark a 'discovery' that others can see. #### test pattern [100m version] — Ryoji Ikeda (池田亮司) (2013) - Video: https://www.youtube.com/watch?v=RZ-dLYmoGW4 - Interaction: Projection, Information & UI, Voice & Sound - Platform & tech: Projection, projectors, binary data, sound - Idea: Pure data rendered at human scale becomes a landscape; a floor-anchored AR layer can turn any stream (network traffic, sound) into terrain you walk on. - What it is: A 100-metre floor projection of flickering black-and-white barcodes runs at high speed in sync with pulses of sound; visitors walk and lie on the moving data. - Technique: Audio signals are converted in real time into binary barcode patterns (likely at many frames per second) and projected by a row of synchronised projectors. - Try it: Anchor a long strip on the corridor floor in AR Foundation and fill it with black-and-white bars generated from the phone microphone's spectrum in real time. Twist: speed the pattern up when more than one phone joins. #### spectra — Ryoji Ikeda (池田亮司) (2014) - Video: https://www.youtube.com/watch?v=qtgvxElYA9k - Interaction: Location & City, Perception & Effects, Voice & Sound - Platform & tech: Projection, searchlights, sound - Idea: One vertical beam can mark a place for a whole city; AR landmarks visible from far away (a pillar, a column) help people find and gather at a location. - What it is: Forty-nine searchlights in a grid near the Houses of Parliament send a single column of white light kilometres into the London night sky for the First World War centenary, with a sine-tone soundscape at ground level. - Technique: Searchlights aimed straight up form a single pillar that becomes visible where it scatters off moisture and dust in the air; speakers in the grid play pure tones. - Try it: Build a geolocated AR beam with 8th Wall or ARCore Geospatial that rises from a point on campus and can be seen from anywhere nearby, with its sound fading in as you approach. Twist: add a second beam that only appears when two people stand under the first. #### data-verse — Ryoji Ikeda (池田亮司) (2019) - Video: https://www.youtube.com/watch?v=BmEz0nTl4qw - Interaction: Information & UI, Projection, Perception & Effects - Platform & tech: Projection, projection, scientific datasets, sound - Idea: Zooming through orders of magnitude in one gesture explains scale better than any label; an AR 'powers of ten' zoom anchored in the room makes science tangible. - What it is: A large projection travels through open scientific data at every scale, from DNA and the human body to the cosmos, as a continuous flow of black-and-white diagrams, point clouds and grids. - Technique: Datasets from CERN, NASA and the Human Genome Project are visualised in custom software and edited into a single camera move across scales. - Try it: Make an AR scene where a coin on the desk is the starting point of a zoom: each pinch jumps one power of ten, from atoms up to the solar system, with real data points. Twist: tie each zoom level to a different tone so the scale can be heard. ### Saype (Guillaume Legros) *Land artist and painter* French-Swiss artist who sprays giant figures onto grass with biodegradable paint made of chalk, charcoal and milk proteins; they fade as the grass grows. #### Land Art au col des Aravis — Saype (Guillaume Legros) (2015) - Video: https://www.youtube.com/watch?v=YdlpCru_x1Q - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, biodegradable paint, mountain meadow - Idea: A picture painted on a slope is designed for the view from across the valley, an early lesson in anamorphic placement for AR. - What it is: On a meadow at the Col des Aravis in the French Alps, Saype paints a giant figure on the steep grass, readable from the road and the opposite slope. - Technique: The artist sprays biodegradable paint freehand guided by points marked on the slope, correcting for the perspective of the viewing point. - Try it: Project a drawing in AR onto a sloping lawn so that it looks correct only from one chosen viewing point across the field. Twist: show visitors the distorted version first and let them find the right spot. #### Leysin land art (world record) — Saype (Guillaume Legros) (2016) - Video: https://www.youtube.com/watch?v=W_CiGuMlhHw - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, biodegradable paint, 10,000 m² grass painting - Idea: At this size the image becomes landscape, and it disappears as the grass grows: scale and decay are both part of the work. - What it is: In Leysin, Switzerland, Saype painted an ephemeral 10,000-square-metre figure on a hillside, then a world record for the largest painting on grass. - Technique: Chalk-and-charcoal paint is sprayed across the hillside following a grid scaled up from a drawing. - Try it: Place a 100-metre AR drawing on a sports field and check where it can be read from (ground, stairs, rooftop). Twist: the drawing fades a little each day, like grass growing over it. #### Message from Future — Saype (Guillaume Legros) (2018) - Video: https://www.youtube.com/watch?v=CPZgThf1h4M - Interaction: Location & City, Information & UI, Perception & Effects - Platform & tech: Desktop, biodegradable paint, lakeside park - Idea: A message painted into a landscape can speak for a cause at the scale of a city, a strong use of large-scale location AR. - What it is: In Geneva, a 5,000-square-metre painting of a girl folding a paper boat made of a map of the world supports the rescue ship Aquarius and migrants crossing the sea. - Technique: Painted with biodegradable pigments on a lakeside lawn, sized to be read by drones and from surrounding buildings. - Try it: Paint a message for a cause as a giant AR image on a lawn and share it as a geo-anchored post that others can visit. Twist: the image reveals a second message when seen from directly above. #### Beyond Walls — Saype (Guillaume Legros) (2019) - Video: https://www.youtube.com/watch?v=GbX3m2mG644 - Interaction: Location & City, Shared & Social, Perception & Effects - Platform & tech: Desktop, biodegradable paint, grass, aerial view - Idea: One motif repeated city by city builds a world-spanning chain, a model for a location AR campaign where each site adds a link. - What it is: Started under the Eiffel Tower in 2019, Beyond Walls is a chain of giant painted hands clasping each other on grass across cities worldwide (Paris, Geneva, Berlin, Istanbul, Cape Town and more), each visible from the air. - Technique: Hands are sprayed on grass with a homemade biodegradable paint (chalk, charcoal, milk proteins), working from a grid drawn on the ground. - Try it: Create an AR 'hand' that any user can place at their location, anchored so that it links to the nearest other hand on a shared world map. Twist: the chain only glows when it forms an unbroken line across a whole city. #### Beyond Walls, Champ de Mars — Saype (Guillaume Legros) (2019) - Video: https://www.youtube.com/watch?v=xYIs1lqAAJs - Interaction: Location & City, Shared & Social - Platform & tech: Desktop, biodegradable paint, Eiffel Tower lawn - Idea: Placing the image under a famous viewpoint (the Eiffel Tower) guarantees an audience from above, a tactic for AR content anchored near lookouts. - What it is: Under the Eiffel Tower, Saype painted a 600-metre chain of joined hands on the lawns of the Champ de Mars, the first step of Beyond Walls. - Technique: Paint was sprayed over several days across the lawn from a gridded plan, readable from the tower's viewing decks. - Try it: Find the highest publicly accessible point on campus and design an AR image on the ground below that only makes sense from there. Twist: add a matching small image at ground level that points people up to the lookout. #### World in Progress — Saype (Guillaume Legros) (2020) - Video: https://www.youtube.com/watch?v=n5IkFCcqzsM - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, biodegradable paint, UN park - Idea: A picture of children drawing the world on the ground is a meta image of what location AR does: people drawing on the planet. - What it is: For the 75th anniversary of the United Nations, Saype painted two children drawing a round of people holding hands on the lawn of the Palais des Nations park in Geneva. - Technique: Painted with his biodegradable mix on the park's lawn at monumental scale. - Try it: Make a shared AR ground-drawing app where each user adds one figure to a circle of held hands on a lawn. Twist: the circle closes only when it reaches one figure per country represented in the class. ### Seb Lee-Delisle *Digital artist; creator of large-scale interactive light and laser installations* British artist and coder whose public installations let crowds launch projected fireworks, play laser games and make music with lasers; PixelPyros and his early pieces were built in openFrameworks. #### Lunar Trails — Seb Lee-Delisle (2012) - Video: https://vimeo.com/54043239 - Interaction: Play, Drawing & Making, Information & UI - Platform & tech: Desktop, drawing robot - Idea: Your path through a screen game is drawn out physically on the wall. - What it is: Visitors play the 1979 arcade game Lunar Lander on a real cabinet, and a large hanging drawing robot plots each player's flight path onto the gallery wall, building up a collective drawing. - Technique: The game's lander position is read in software and sent as coordinates to a polargraph-style hanging plotter that draws the trajectories in pen. - Try it: Record the path of a player in a simple phone game and replay it as a drawing on paper with a pen plotter, or as a persistent AR trail in the room. Twist: overlay the trails of everyone in the class and see where they all crash. #### PixelPyros — Seb Lee-Delisle (2012) - Video: https://vimeo.com/61174060 - Interaction: Hands & Body, Projection, Shared & Social - Platform & tech: Projection, openFrameworks - Idea: Wave your hand and a firework launches from that spot; the crowd choreographs the show. - What it is: A 20-metre projected fireworks display where the audience launches every rocket by waving a hand over sensors along the bottom of the screen; up to thirty people play at once. - Technique: Motion sensing along the base of the screen (likely camera-based in the original openFrameworks version) maps each hand position to a launch point, and particle fireworks are projected with high-power projectors and lasers. - Try it: Project a row of launch pads on a wall and use a webcam to detect hands passing over them; each detection fires a particle firework. Twist: fireworks launched within one second of each other combine into a bigger one. #### Laser Light Synths — Seb Lee-Delisle (2016) - Video: https://vimeo.com/314844029 - Interaction: Voice & Sound, Projection, Shared & Social - Platform & tech: Projection, lasers, custom synthesizers - Idea: Anyone can play music, and every note becomes light that fills the space. - What it is: Visitors play large illuminated touch synthesizers; every note triggers laser graphics that sweep across the building and surround the players with beams. - Technique: Custom touch-sensitive synths are locked to a musical scale so there are no wrong notes, and their input drives laser animations through laser-show software written by the artist. - Try it: Make a four-key instrument (touch pads or phone buttons) in a pentatonic scale, where each note sends a coloured beam or shape across a projected wall. Twist: two players must play the same note together to unlock a big effect. #### Laser Duck Hunt — Seb Lee-Delisle (2018) - Video: https://vimeo.com/263303651 - Interaction: Play, Tangible Objects, Projection - Platform & tech: Projection, lasers - Idea: A screen game escapes onto the wall as laser light, played with its original toy gun. - What it is: The NES game Duck Hunt is redrawn in laser light on a wall, and players shoot the ducks with a hacked Zapper gun that puffs smoke when fired. - Technique: The hacked Zapper's aim is likely tracked with a camera or photo sensor, and the game is rendered as vector laser graphics through a laser DAC. - Try it: Pick a classic screen game and rebuild one level as a projected wall game played with a physical prop tracked by a webcam or phone. Twist: the prop must also produce a real-world effect (sound, air, light) when used. #### Lightning Catchers — Seb Lee-Delisle (2018) - Video: https://vimeo.com/264245459 - Interaction: Play, Tangible Objects, Projection - Platform & tech: Projection, lasers, camera tracking - Idea: Catch virtual lightning with a real glowing stick. - What it is: Laser lightning bolts strike down onto a courtyard, and players run with glowing LED sticks to catch them before they hit the ground. - Technique: A camera likely tracks the bright LED sticks and the software aims laser-drawn bolts, detecting a catch when a stick's position meets a bolt's end point. - Try it: Track a phone flashlight or LED with a webcam and project falling objects that players must catch with the light. Twist: each catch makes the next object fall somewhere else in the room. #### Laser Light City — Seb Lee-Delisle (2020) - Video: https://vimeo.com/466618882 - Interaction: Location & City, Shared & Social, Projection - Platform & tech: Projection, Web, lasers, web - Idea: Control lasers over your city from your window, together with neighbours you cannot meet. - What it is: During the Covid lockdowns, powerful lasers on a rooftop draw patterns across Brighton's sky and buildings, and people at home can take control of the beams from their phones. - Technique: A web interface sends each participant's input to the laser-show server, which turns it into beam positions and colours for the rooftop lasers. - Try it: Build a web page that lets classmates move a shape on a shared projected wall from their own phones. Twist: the shape only moves when at least three people push in the same direction. ### Tim Murray-Browne *Artist and creative technologist making interactive body-sound installations* British artist who builds open-ended interactive installations and dance pieces where movement tracked by cameras and sensors shapes sound, light and AI imagery, from the touring Cave of Sounds to the AI self-portrait Self Absorbed. He took part in the 2022 Choreographic Coding Lab at A+E Lab and wrote about it as 'embodied thinking'. #### Cave of Sounds — Tim Murray-Browne (2013) - Video: https://vimeo.com/76453883 - Interaction: Voice & Sound, Hands & Body, Shared & Social - Platform & tech: Desktop, custom instruments, Kinect, light sensors - Idea: A circle of strange instruments turns strangers into a band without instructions. - What it is: Eight new electronic instruments stand in a circle around a glowing hub; visitors play them freely by moving hands through light, casting shadows or gesturing, and together improvise a shared piece of music. - Technique: Each instrument was built by a different Music Hackspace member from sensors such as cameras, light sensors and Kinect, all synced to a shared tempo so any gesture fits the ensemble. - Try it: Build four 'instruments' from phone sensors in web pages (tilt, shake, microphone, camera brightness) that all quantise to one shared beat, and let visitors jam. Twist: make an instrument that only sounds when two people stand close. #### This Floating World — Tim Murray-Browne (2015) - Video: https://www.youtube.com/watch?v=D1ZEzkMCNsI - Interaction: Hands & Body, Projection, Performance - Platform & tech: Projection, 3D depth camera, custom software, generative visuals - Idea: The dancer grows the landscape she dances in, like a vine shaped by a wall. - What it is: A dancer performs a solo inside projected vines and flowing forms that are shaped by her movement, while the score shifts with her body; later shown as an installation that visitors could dance in. - Technique: A 3D depth camera tracks the dancer; custom software analyses the movement and, with live input from an operator's laptop, generates the projected visuals and modulates the music. - Try it: With a webcam and TouchDesigner or p5.js, project lines that grow from the performer's silhouette edges and slowly fade. Twist: let lines grow only where the dancer has stood still for three seconds. #### Where things start from — Tim Murray-Browne (2015) - Video: https://www.youtube.com/watch?v=mhtq0pi2iYI - Interaction: Hands & Body, Voice & Sound - Platform & tech: Desktop, Kinect, sound synthesis - Idea: Reward stillness and slowness with sound instead of fast gestures. - What it is: A sketch of an interactive sound space where holding a slow, sustained movement generates noise textures, tested with Kinect recordings of a dancer. - Technique: Kinect skeleton data is analysed for sustained low-velocity motion, which is mapped to the amplitude and filtering of a noise synthesiser. - Try it: Build a web page where MediaPipe body tracking plays a drone that gets louder the slower you move. Twist: fast moves erase the sound for five seconds. #### Post-Truth and Beauty — Tim Murray-Browne, Aphra Shemza (2017) - Video: https://www.youtube.com/watch?v=tVVIV_sNlaA - Interaction: Gaze & Attention, Voice & Sound, Perception & Effects - Platform & tech: Projection, head tracking, speaker ring, LED light - Idea: Truth as something you only ever see partially, depending on where your head is. - What it is: A visitor steps into a ring of speakers; as they move their head, light patterns and spatial sound change so that each viewpoint reveals only a partial glimpse of an abstract world. - Technique: The position of the visitor's head is tracked (likely with a depth camera) and mapped to light states and to the panning of sound across a circular speaker array. - Try it: Use a laptop face tracker to change a projected abstract image and a stereo pan as the viewer leans left, right or closer. Twist: hide one 'secret' image that appears only from one exact spot. #### Self Absorbed — Tim Murray-Browne (2023) - Video: https://www.youtube.com/watch?v=JKg-6fHRT9U - Interaction: Hands & Body, Perception & Effects, Voice & Sound - Platform & tech: Projection, StyleGAN, unsupervised movement model, camera tracking - Idea: Navigate an AI's memory of your life with your body, not a mouse. - What it is: The artist moves in front of a large screen and floats through an AI-generated stream of images and sounds made from twenty years of his own photos and recordings, steering it only with his body. - Technique: An unsupervised model trained on how the artist moves encodes live body tracking into a low-dimensional vector that steers the latent space of image (StyleGAN) and sound models trained on his personal archive. - Try it: Map three body features from a webcam pose model (arm spread, height, lean) to sliders that blend between a small set of your own photos in a web page. Twist: swap the photo set for a classmate's and see if movement still feels personal. ### Tobias Langlotz *Professor, University of Otago (Human-Computer Interaction / Computational Glasses)* Former TU Graz researcher who co-built early smartphone AR authoring and browsers, now works on 'computational glasses' that correct color vision, guide gaze by modulating saliency, and render virtual objects with a real camera's quirks. #### Sketching up the World — Tobias Langlotz, Stefanie Zollmann (2011) - Video: https://www.youtube.com/watch?v=TjUwRIRzCus - Interaction: Drawing & Making, Location & City - Platform & tech: Phone, panoramic tracking, mobile AR authoring - Idea: Anyone should be able to author AR content on the spot, with just a phone. - What it is: On an early smartphone, a user sketches and places 3D content directly onto buildings in an unprepared outdoor scene, leaving annotations that others can find later. - Technique: Uses panoramic tracking with sensor fusion on the phone to register sketched and extruded content to the surroundings without a prior model. - Try it: Walk around campus with a phone AR drawing app and leave three spatial doodles that explain something about each place, then swap phones and hunt for each other's notes. Twist: allow only arrows and one word. #### PanoVC — Tobias Langlotz (2016) - Video: https://www.youtube.com/watch?v=s-Z17uOdfJc - Interaction: Shared & Social, Spatial Mapping - Platform & tech: Phone, panorama stitching, mobile telepresence - Idea: Share your whole surroundings on a call, not just where your camera points. - What it is: A video call where the caller's phone continuously builds a panorama of their surroundings, so the remote person can look around the other place freely instead of seeing a single camera frame. - Technique: Incrementally stitches and streams a panoramic map from the sender's phone camera while live frames update the part currently in view. - Try it: Take a phone panorama of the classroom, send it to a partner in another room, and keep sending live photos of the spot they ask to see; let them guide you to find an object. Twist: mark where the remote person is looking with a sticky note. #### ChromaGlasses — Tobias Langlotz, Stefanie Zollmann (2018) - Video: https://www.youtube.com/watch?v=kBe-pfc7PrY - Interaction: Perception & Effects, Information & UI - Platform & tech: Headset, optical see-through HMD, colour compensation, computational glasses - Idea: Glasses that correct not focus but colour, by editing reality in real time. - What it is: Glasses with an optical see-through display analyse the scene and shift the colours of the world, pixel by pixel, so people with colour vision deficiency can tell apart colours they normally confuse. - Technique: Captures the scene with a camera aligned to the eye, computes a compensation for the user's colour deficiency, and overlays it with pixel precision on an optical see-through HMD. - Try it: Make a phone camera filter that pushes reds toward orange and greens toward blue, then test it on colour-blindness plates viewed through a colour-blindness simulator. Twist: design a filter that helps people spot ripe fruit. #### Neural Cameras — Tobias Langlotz, Stefanie Zollmann (2021) - Video: https://www.youtube.com/watch?v=cGc8vBVbIvI - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, neural networks, camera simulation, coherent rendering - Idea: Make virtual objects look filmed by the same imperfect camera as everything else. - What it is: Virtual objects inserted into a phone video pick up the same noise, blur, colour and lens quirks as the real camera, so they stop looking pasted on. - Technique: Trains neural networks on image databases from a physical camera to jointly simulate its imaging pipeline and applies them to rendered objects before compositing. - Try it: Place a 3D model into a phone AR scene, then add grain, slight blur and a colour tint until classmates can no longer tell which object is real. Twist: match a vintage camera look instead. #### Look over there! Saliency Modulation — Tobias Langlotz, Stefanie Zollmann, Yuta Itoh (2022) - Video: https://www.youtube.com/watch?v=oBp1wZAA8io - Interaction: Gaze & Attention, Perception & Effects - Platform & tech: Headset, saliency modulation, optical see-through, visual guidance - Idea: Guide attention by editing how noticeable reality is, not by adding more graphics. - What it is: Instead of arrows or outlines, AR glasses subtly boost the target object and dampen its surroundings, so your eyes drift to the right place without any visible marker. - Technique: Computes a saliency map of the scene and applies local contrast and colour changes through the display to raise the target's saliency relative to distractors. - Try it: Build a phone camera filter that slightly desaturates everything except one tracked object, then ask classmates to name the first thing they notice. Twist: make the effect so subtle that people cannot say why they looked there. ### Torin Blankensmith *Creative technologist, real-time graphics artist, Parsons adjunct* Ex-Google Creative Lab technologist (co-built AR Wormhole) and co-creator of Shader Park; his free MediaPipe and body-tracking plugins for TouchDesigner put camera-based body interaction in thousands of students' hands. #### Echoes — Torin Blankensmith (2023) - Video: https://x.com/blankensmithing/status/1737307286572298738 - Interaction: Hands & Body, Shared & Social, Perception & Effects - Platform & tech: Projection, Desktop, TouchDesigner, MediaPipe - Idea: Every move you make summons the most similar move made by an earlier visitor. - What it is: At Art Basel Miami, each visitor's live pose is matched to the closest pose from previous visitors, so your movement is echoed by people who stood there before you. - Technique: Pose landmarks from a camera are stored per visitor, and a nearest-neighbor search retrieves the closest past pose frame to display alongside the live body. - Try it: Record a pose library of the whole class with MoveNet, then when a newcomer strikes a pose, show 'the classmate most like you' on screen. Twist: the closer the match, the sharper the image. #### MediaPipe body tracking plugin for TouchDesigner — Torin Blankensmith (2023) - Video: https://x.com/blankensmithing/status/1684227303406989312 - Interaction: Hands & Body, Face - Platform & tech: Desktop, TouchDesigner, MediaPipe - Idea: A no-barrier webcam body-tracking plugin that lets anyone make motion-driven visuals. - What it is: A free, no-setup webcam plugin tracks face, hands and body in TouchDesigner, shown driving real-time visuals over the performer's body. - Technique: MediaPipe models run inside an embedded browser in TouchDesigner and stream landmark data as channels to drive visuals. - Try it: Use this plugin (or MediaPipe on the web) to build a 'body instrument' in which different joints play different notes when they touch areas on the screen. Twist: you can only play with your eyes closed. #### Conducting a virtual choir like Jacob Collier — Torin Blankensmith (2024) - Video: https://x.com/blankensmithing/status/1769508579613040847 - Interaction: Hands & Body, Voice & Sound, Performance - Platform & tech: Desktop, TouchDesigner, MediaPipe - Idea: Conduct harmonies in the air with hand gestures, the way Jacob Collier conducts his audience as a choir. - What it is: Hand gestures tracked from a webcam conduct pitches and harmonies, imitating how Jacob Collier conducts his audience as a choir. - Technique: Hand landmark height and spread from MediaPipe are mapped to musical pitch and chord voicing, with graphics overlaid on the live camera image. - Try it: Use MediaPipe Hands and Tone.js so that hand height controls pitch and hand spread controls the chord. Twist: have two people play together as a duet. #### Immersive Memory (AlUla) — Torin Blankensmith (2024) - Video: https://x.com/blankensmithing/status/1858627004398268491 - Interaction: Location & City, Hands & Body, Projection - Platform & tech: Projection, TouchDesigner, body tracking, spatial audio - Idea: Among the ruins of an old town, visitors' movements call up images and spatial sound from the memory of the incense route. - What it is: An interactive installation across three spaces in AlUla's Old Town, a UNESCO site of the incense trade, responds to visitors with visuals and a spatial audio score. - Technique: Camera-based body tracking drives projected generative visuals in each room, tied together by a multichannel spatial audio score (likely MediaPipe/depth tracking). - Try it: Make a 'memory projection' for an old building on campus, where old photos appear as particles when someone approaches the wall. Twist: the full photo only becomes clear when several people approach at once. #### Hand-tracking watercolor — Torin Blankensmith (2025) - Video: https://x.com/blankensmithing/status/1899948380299944296 - Interaction: Drawing & Making, Hands & Body, Perception & Effects - Platform & tech: Desktop, TouchDesigner, MediaPipe - Idea: Wave your hands in the air and watercolor blooms across the picture. - What it is: Moving your hands in front of a webcam leaves blooming watercolor strokes that bleed and dry over the live image. - Technique: Fingertip positions inject color into a feedback/diffusion shader that simulates pigment spreading and edge darkening. - Try it: Make mid-air watercolor with a p5.js feedback buffer and MediaPipe fingertip tracking. Twist: when you make a fist, the paint 'dries' and stays fixed. #### I/O Brush re-creation (Genuary) — Torin Blankensmith (2025) - Video: https://x.com/blankensmithing/status/1875375318946795571 - Interaction: Drawing & Making, Hands & Body, Tangible Objects - Platform & tech: Desktop, TouchDesigner, MediaPipe - Idea: Dip your hand into reality to pick up colors and textures, then paint with them. - What it is: Re-creating MIT Tangible Media's I/O Brush, a webcam-tracked hand 'picks up' colors and textures from real objects and paints with them on screen. - Technique: MediaPipe hand tracking finds the fingertip, samples the camera pixels under it as the brush, then paints strokes into a feedback canvas. - Try it: Use p5.js and MediaPipe Hands so that touching a real object with your fingertip picks up its color, then paint in the air with it. Twist: pick up a short clip of live video instead of a color. ### Troika (Eva Rucki, Conny Freyer & Sebastien Noel) *London art collective working with perception, light and kinetic sculpture* Troika is a London collective founded in 2003 whose sculptures and installations, from the flip-dot Cloud at Heathrow to the rotating Dark Matter, play with perception, light and the gap between what we see and what we know. #### Cloud — Troika (Eva Rucki, Conny Freyer & Sebastien Noel) (2008) - Video: https://www.youtube.com/watch?v=42hgPLL8IrA - Interaction: Information & UI, Perception & Effects - Platform & tech: Desktop, 4,638 flip-dots, custom controller, airport installation - Idea: A display wrapped around a sculptural form reads as a living skin — AR content feels more physical when it follows the curvature of a real object. - What it is: A 5-metre cloud-shaped sculpture at Heathrow Terminal 5 is covered in 4,638 flip-dots that flip between silver and black, sending ripples and patterns across its skin with a soft clicking sound. - Technique: Electromagnetic flip-dots, each a disc with a black and a reflective side, are driven row by row so animations travel across the curved surface. - Try it: Scan a real object (a chair, a sculpture) with LiDAR and cover its mesh in AR with thousands of tiny flipping tiles that ripple when you tap it. Twist: add a click sound per tile and let the ripple spread only as fast as sound would. #### Falling Light — Troika (Eva Rucki, Conny Freyer & Sebastien Noel) (2010) - Video: https://www.youtube.com/watch?v=jfTp1ltayik - Interaction: Perception & Effects - Platform & tech: Desktop, crystal lenses, spotlights, motors - Idea: Caustics make light look like a liquid — animated light spots on the real floor are a cheap, convincing AR effect. - What it is: Light shines through slowly turning glass lenses, and the focused spots on the floor look like drops of light that fall, grow and splash across the ground. - Technique: Rotating lenses shift and refocus the beams from a light source above, so bright caustic spots move and pulse on the floor below. - Try it: Project virtual caustic 'raindrops' onto a detected floor plane in AR that fall from a point above and splash into rings. Twist: let the drops fall only where nobody is standing. #### Thixotropes — Troika (Eva Rucki, Conny Freyer & Sebastien Noel) (2011) - Video: https://www.youtube.com/watch?v=nYfP4qGLy94 - Interaction: Perception & Effects - Platform & tech: Desktop, motorized rotation, LED strips, persistence of vision - Idea: Motion can create volume that isn't there — a single moving line in AR can suggest a full solid. - What it is: Hanging sculptures made of LED lines spin, and at speed the lines blur into solid-looking 3D volumes that change shape as the rotation shifts. - Technique: Rotating LED lines at high speed exploits persistence of vision, so the eye integrates their paths into surfaces of revolution. - Try it: Spin a thin glowing line fast in AR with long motion trails so it sweeps out a vase-like volume above a table. Twist: let the user bend the line with a pinch gesture to reshape the volume. #### Arcades — Troika (Eva Rucki, Conny Freyer & Sebastien Noel) (2013) - Video: https://www.youtube.com/watch?v=WMzHXYQrfgQ - Interaction: Perception & Effects - Platform & tech: Projection, Fresnel lenses, haze, light beams - Idea: Light should travel in straight lines, so when it looks curved we stop and look — AR can break physical expectations in small, elegant ways. - What it is: Beams of light pass through haze and lenses and appear to bend into a row of glowing arches, so light seems to curve like architecture. - Technique: Beams are split and redirected by lenses and mirrors so that a sequence of straight segments in haze reads as a continuous arch. - Try it: Draw a row of volumetric light arches over a real hallway in AR that look like bent beams of light. Twist: let the arches straighten out when the viewer stops walking. #### The Shape of a Circle in the Mind of a Square — Troika (Eva Rucki, Conny Freyer & Sebastien Noel) (2013) - Video: https://www.youtube.com/watch?v=eEUkYyhnB4M - Interaction: Perception & Effects - Platform & tech: Desktop, kinetic sculpture, motors, rotating profiles - Idea: Rotation reveals that circle and square can be the same object — a clear demo of why AR must be judged from many viewpoints. - What it is: A black rotating sculpture turns steadily and its outline switches between a circle and a square, 'squaring the circle' in front of the viewer's eyes. - Technique: The sculpture's cross-sections are shaped so that its projected silhouette morphs between a circle and a square as it rotates about one axis. - Try it: Build a spinning AR object that looks like a circle to one user and a square to another standing 90° away, using shared anchors. Twist: let the users swap places and argue about what they saw. #### Dark Matter — Troika (Eva Rucki, Conny Freyer & Sebastien Noel) (2014) - Video: https://www.youtube.com/watch?v=17f47By0DjI - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, rotating geometric objects, motors, spotlight - Idea: A 3D object has as many identities as viewpoints — AR designers can hide a message that only resolves from one angle. - What it is: Three black geometric objects rotate slowly; from certain angles each one looks like a perfect square or circle, then collapses into an unfamiliar shape as it turns. - Technique: The objects are designed so their silhouettes are simple primitives from particular directions; rotation sweeps the viewer through these silhouettes. - Try it: Model a shape in Blender whose silhouette is a circle from the front and a square from the side, then place it in AR so visitors discover both. Twist: add a third hidden silhouette that spells a letter. ### URBANSCREEN *Artist collective and studio for architectural projection* Collective founded in 2005 that pioneered large-scale 'architectural staging' — façade projections such as 555 KUBIK and the Sydney Opera House sails — and 'augmented sculptures' where projection and custom-built forms are designed together. #### 555 KUBIK — URBANSCREEN (2009) - Video: https://www.youtube.com/watch?v=H8qcml3smAA - Interaction: Projection, Perception & Effects, Location & City - Platform & tech: Projection, 3D projection mapping, anamorphic animation - Idea: What if a house were dreaming? Make a boxy building look like it is bending and folding. - What it is: 'How it would be if a house was dreaming': the cube façade of Hamburg's Galerie der Gegenwart appears to bulge, fold and dissolve under a projected 3D animation. - Technique: A 3D model of the façade is animated with fake depth and lighting, then projected from the audience's viewpoint so the illusion of deformation holds. - Try it: Build a building from cardboard boxes, animate a deforming model of the same shape in Blender, and project it aligned from the viewer's position. Twist: change the viewing angle and discuss why the illusion disappears. #### Lighting the Sails (Sydney Opera House) — URBANSCREEN (2012) - Video: https://www.youtube.com/watch?v=snT8psrPxmA - Interaction: Projection, Perception & Effects, Location & City - Platform & tech: Projection, multi-projector mapping, 3D scan - Idea: The sails of the Sydney Opera House are taken apart, rebuilt and made to breathe in projection. - What it is: A projected homage to Jørn Utzon in which the Opera House sails appear to be unfolded, rebuilt and given a human, breathing expression. - Technique: Laser-scanned geometry of the curved shells guides multi-projector content that exaggerates their construction logic with simulated depth and light. - Try it: Use a curved sheet of paper or a lampshade as a "sail" and project a wireframe onto it that splits apart and reassembles. Twist: let the loudness of breathing control how much the "sail" billows. #### 26th Floor (Augmented Sculpture) — URBANSCREEN (2013) - Video: https://www.youtube.com/watch?v=4BUhNy1iUhc - Interaction: Projection, Perception & Effects, Tangible Objects - Platform & tech: Projection, augmented sculpture, projection mapping - Idea: Sculpture and projection are designed together, giving a white form a flowing skin of light. - What it is: An 'augmented sculpture' for the Four Seasons Hotel Beirut, where a white faceted object and its projected light skin were designed together. - Technique: A faceted physical sculpture is built from the same digital model used to author the projection, so every face receives perfectly aligned content. - Try it: Fold a polyhedron from white card, build the same model on the computer, and project different flowing colors face by face. Twist: pass a beam of "light" from face to face to form a continuous animation. #### 320° Licht — URBANSCREEN (2013) - Video: https://www.youtube.com/watch?v=X_31XyGBL7U - Interaction: Projection, Perception & Effects, Location & City - Platform & tech: Projection, multi-projector blending, generative graphics - Idea: Turn the inside of a giant gas holder into an ever-changing space of light. - What it is: Inside the 100-metre-high Gasometer Oberhausen, graphic patterns grow and change across 320 degrees of wall, turning the industrial cylinder into an immersive light space. - Technique: An array of projectors is blended across the cylindrical interior, with generative patterns designed to exploit the surface's extreme height. - Try it: Project vertically stretching patterns in a stairwell or corridor so people have to look up. Twist: make the patterns "grow" upward from wherever a viewer is standing. #### SOLANUM (Augmented Sculpture) — URBANSCREEN (2014) - Video: https://www.youtube.com/watch?v=ZSQ3N8FKyb8 - Interaction: Projection, Tangible Objects, Perception & Effects - Platform & tech: Projection, augmented sculpture, digital fabrication - Idea: A layered physical sculpture and a close-fitting virtual skin merge into one. - What it is: A site-specific augmented sculpture in the JW Marriott Houston lounge: layered physical forms and a custom-fit virtual skin melt together into one evolving object. - Technique: Layered physical slices are fabricated from the same 3D design as the projection content, enabling pixel-accurate mapping on every layer edge. - Try it: Stack several layers of foam board into a form and project scan lines whose color changes from layer to layer. Twist: let the number of people present (counted by a camera) set the speed of the scan lines. #### Crystal Cloud (Fête des Lumières) — URBANSCREEN (2016) - Video: https://www.youtube.com/watch?v=82jYUtxbsCQ - Interaction: Projection, Perception & Effects, Location & City - Platform & tech: Projection, projection mapping, 3D model - Idea: A dialogue between crystal and cloud performed on a deconstructivist building. - What it is: The Coop Himmelb(l)au Musée des Confluences in Lyon is staged as a play between 'crystal' and 'cloud' — solid facets and soft vapour projected across its deconstructivist shell. - Technique: A precise 3D model of the irregular façade lets projected content switch between hard faceted lighting and soft volumetric effects. - Try it: Wrap two paper boxes, one in foil and one in cotton, project the same animation onto both, and compare the effect on each material. Twist: design a piece of imagery that "migrates" from one material to the other. ### Yayoi Kusama (草間彌生) *Artist; Infinity Mirror Rooms* Japanese artist whose obsession with repetition and dots produced the Infinity Mirror Rooms, where mirrors multiply lights or objects into endless space, and the participatory Obliteration Room. #### Narcissus Garden — Yayoi Kusama (草間彌生) (1966) - Video: https://www.youtube.com/watch?v=HxMwouPTU7M - Interaction: Location & City, Perception & Effects, Shared & Social - Platform & tech: Desktop, mirrored stainless steel spheres, water, lawn - Idea: Many small mirrors scattered over a landscape multiply it into countless tiny worlds, a direct recipe for instanced reflective AR objects. - What it is: Hundreds of mirrored steel spheres float on a pond or lie across a lawn, each reflecting the sky, the trees, the other spheres and the viewer; first shown at the 1966 Venice Biennale, here at Inhotim in Brazil (2009). - Technique: Mass-produced polished spheres are laid out loosely on water or grass, where wind and currents move them into new clusters. - Try it: Scatter 300 instanced AR spheres with a real-time reflection probe over a lawn or pond and let the device's motion act as wind that drifts them. Twist: each sphere shows the reflection from where it was placed, not where it is now. #### Fireflies on the Water — Yayoi Kusama (草間彌生) (2002) - Video: https://www.youtube.com/watch?v=ahZh_Qpb8ik - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Projection, mirrors, water pool, light bulbs - Idea: A few points of light plus mirrors on every side are enough to make a closet feel like the night sky, so an AR scene does not need many assets, only the right reflections. - What it is: One visitor at a time steps onto a small platform in a dark mirrored room; about 150 tiny lights hang over a black pool and repeat endlessly in the mirrors and the water. - Technique: Mirror walls and a still water floor create recursive reflections of small hanging bulbs, and the one-person, short-visit rule keeps the illusion unbroken. - Try it: Build a phone AR 'mirror room' in AR Foundation: place 30 glowing points over a detected floor and render them again in a mirrored copy of the scene below the floor plane and in four virtual mirror walls. Twist: let each point fade out when the viewer's gaze rests on it, so looking makes the fireflies disappear. #### The obliteration room — Yayoi Kusama (草間彌生) (2002) - Video: https://www.youtube.com/watch?v=-xNzr-fJHQw - Interaction: Shared & Social, Drawing & Making, Spatial Mapping - Platform & tech: Desktop, stickers, white furniture, participation - Idea: Letting every visitor add one small mark turns a room into a shared, growing record; persistent multi-user AR annotations can make a space change over time the same way. - What it is: An all-white furnished apartment is handed over to visitors, who each get sheets of coloured dot stickers; over weeks the furniture, walls and floor disappear under thousands of dots. - Technique: The work is a participatory rule, not an image: white surfaces, a fixed palette of round stickers and many visitors produce an emergent pattern. - Try it: Build a shared AR room with persistent anchors (e.g. ARCore Cloud Anchors or Niantic Lightship): each user may place five coloured dots on real surfaces, and all dots stay for the next users. Twist: dots slowly fade after a week, so the room is always half obliterated and half restored. #### Aftermath of Obliteration of Eternity — Yayoi Kusama (草間彌生) (2009) - Video: https://www.youtube.com/watch?v=aUViLK-bBaY - Interaction: Perception & Effects, Hands & Body - Platform & tech: Projection, mirrors, LED lanterns, black water - Idea: Light that appears and then vanishes on a timed breath reads as life and death; an AR designer can give virtual objects a short, visible lifespan to carry emotion. - What it is: From a walkway over dark water in a mirrored room, visitors watch hundreds of gold lantern-like lights glow up and then go out, like floating lanterns drifting away. - Technique: Hanging LED lanterns fade in and out in sequence, and their reflections in mirrors and a black pool extend the flicker into an endless field. - Try it: Place 100 lantern lights over a real floor or pond with phone AR, make them brighten together and then die out one by one over 60 seconds. Twist: let the user relight one lantern by holding a finger on it, and only that lantern survives the next cycle. #### Infinity Mirrored Room – Filled with the Brilliance of Life — Yayoi Kusama (草間彌生) (2011) - Video: https://www.youtube.com/watch?v=bT7i507OnOw - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Projection, mirrors, LED lights, water - Idea: When the body is inside the reflection, the viewer becomes part of the pattern; AR that includes the user's own reflection feels far more immersive than AR the user only looks at. - What it is: Visitors walk a path over a shallow pool inside a mirror-lined room where hundreds of coloured LEDs hang at different heights and slowly change colour, shown at Tate Modern in 2021–22. - Technique: Two-way recursive mirrors on walls and ceiling plus a reflective pool multiply a few hundred programmed LED bulbs into an apparently endless field. - Try it: Use Lens Studio or WebXR to hang 200 small coloured spheres around the user, then add a mirrored duplicate of the whole cluster below the floor and let colours cycle slowly. Twist: insert a simple avatar that copies the user's head position into the mirrored copy, so the user sees themself among the lights. #### Infinity Mirrored Room – The Souls of Millions of Light Years Away — Yayoi Kusama (草間彌生) (2013) - Video: https://www.youtube.com/watch?v=7dd3ZtB4Cjs - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Projection, mirrors, LED spheres, timed lighting - Idea: A strict time limit turns a simple light loop into a precious moment; AR experiences can use scarcity of time the same way instead of running forever. - What it is: Inside a small mirrored cube, LED spheres hanging from the ceiling pulse through colours in a slow sequence while one or two visitors stand alone for under a minute. - Technique: Programmed LED globes are multiplied by mirrored walls, ceiling and floor, and a timed colour sequence gives the short visit a beginning and an end. - Try it: Make a 45-second WebXR or phone AR piece: spawn a field of glowing spheres around the user that runs one colour sequence and then dims to black and closes itself. Twist: allow each device to play it only once per hour, and show a countdown to the next visit. ### Yuta Itoh *Associate Professor, University of Tokyo (Augmented Vision Lab); formerly Tokyo Tech and TU Munich* Researches optical see-through displays that do more than add graphics: they subtract light, fix eyesight and show the future path of moving objects. #### Laplacian Vision — Yuta Itoh (2016) - Video: https://www.youtube.com/watch?v=2GD7KQOHiMs - Interaction: Perception & Effects, Information & UI, Play - Platform & tech: Headset, Projection, physics prediction, object tracking, optical see-through HMD - Idea: See where a flying object is going to be. - What it is: AR glasses or a projector show the predicted future trajectory of a thrown ball as a line in the air, helping people anticipate and catch it. - Technique: Tracks the object with cameras, fits a ballistic model and renders the extrapolated path in the user's view with latency compensation. - Try it: Detect a rolling ball in phone AR and draw its position one second ahead by parabolic or linear extrapolation. Twist: deliberately show a "wrong future" and see whether classmates are misled. #### OST Rift — Yuta Itoh (2016) - Video: https://www.youtube.com/watch?v=UatOFZuD5L0 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Headset, optical see-through HMD, latency compensation, Oculus Rift - Idea: A VR headset converted to optical see-through AR, with virtual objects pinned firmly to reality. - What it is: A consumer VR headset converted into an optical see-through AR display with careful timing so virtual objects stay locked to the real world during head motion. - Technique: Adds beam-splitter optics to a Rift and predicts head pose to compensate display latency, keeping overlays temporally consistent with the real scene. - Try it: Build simple see-through AR glasses from HoloKit or a phone plus a half-mirror, and compare how a virtual cube drifts during fast head turns. Twist: add an artificial 200 ms delay and have classmates describe how it feels. #### AdaptiVisor — Yuta Itoh (2017) - Video: https://www.youtube.com/watch?v=fi4PjGIbuu8 - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Headset, occlusive optical see-through HMD, spatial light modulator, HDR camera - Idea: Pixel-level sunglasses that dim only the glaring parts of reality. - What it is: An occlusion-capable see-through HMD selectively dims bright regions of the real world, like a sun visor per pixel, to help the eye adapt to glare. - Technique: Uses a camera to find over-bright regions and an LCD panel in the optical path to attenuate exactly those pixels of real-world light. - Try it: Use the phone camera to detect the brightest regions of the image in real time and darken them in the video pass-through, making a "smart anti-glare" filter. Twist: flip it so only the brightest regions stay and everything else goes dark. #### Occlusion Leak Compensation — Yuta Itoh (2017) - Video: https://www.youtube.com/watch?v=iqSjFHu9wsI - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Headset, occlusion-capable OST display, spatial light modulator - Idea: Make virtual objects in see-through AR truly block the reality behind them. - What it is: A see-through display that can make virtual objects truly opaque by blocking real light, and compensates the blurry 'leak' around the mask so edges look sharp. - Technique: Predicts the defocus blur of the occlusion mask and adds compensating light from the display so the combined image has crisp opaque edges. - Try it: In phone video pass-through AR, compare rendering a virtual object as a semi-transparent overlay versus fully occluding, and observe which looks more real. Twist: add shadows to the virtual object and compare again. #### Varifocal Occlusion — Yuta Itoh (2019) - Video: https://www.youtube.com/watch?v=HKhp_ac_jV0 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Headset, varifocal optics, occlusion mask - Idea: An occlusion layer that refocuses near and far, so real objects at any distance are blocked correctly. - What it is: An optical see-through HMD whose occlusion mask can be moved in focal depth, so real objects are blocked correctly whether they are near or far. - Technique: Slides the occlusion SLM along the optical axis so the mask's focal plane matches the depth of the real object being blocked. - Try it: Photograph a near object and a far object with a phone camera, overlay a virtual occluder while focused on each, and discuss why it blurs. Twist: use a manual-focus app to show the focal plane shifting. ### 1927 (Suzanne Andrade & Paul Barritt) *Theatre company combining live performance and projected animation* London theatre company founded in 2005 by writer-director Suzanne Andrade and animator Paul Barritt. Its actors perform inside hand-drawn animated films projected onto flat screens, so painted doors, animals and crowds appear to share the stage with them; The Magic Flute with Barrie Kosky has toured opera houses worldwide. #### Between the Devil and the Deep Blue Sea — 1927 (Suzanne Andrade & Paul Barritt) (2007) - Video: https://www.youtube.com/watch?v=0xIKO3--kq8 - Interaction: Performance, Projection, Drawing & Making - Platform & tech: Projection, projected animation, live music - Idea: A drawn world becomes inhabitable when a real person stands in it; hand-drawn AR can feel warmer than realistic 3D. - What it is: 1927's debut cabaret places performers against a white screen filled with Paul Barritt's hand-drawn animations, so they walk through drawn streets and silent-film fairy tales with live music. - Technique: Animation is projected onto a flat screen and the actors are placed and lit so their bodies line up with painted doors, windows and props. - Try it: Draw a street scene on paper, scan it, and place it as a flat AR backdrop that a classmate can stand in while you film them opening a drawn door. Twist: the drawn door only opens when they knock. #### The Animals and Children Took to the Streets — 1927 (Suzanne Andrade & Paul Barritt) (2010) - Video: https://www.youtube.com/watch?v=hTNeZsIRaKA - Interaction: Performance, Projection, Drawing & Making - Platform & tech: Projection, projected animation, live music - Idea: Precise alignment between a live head and a drawn body makes one character from two media; AR face and body effects rely on the same registration. - What it is: A dark comic tale set in the Bayou Mansions tenement, where three performers act, sing and play inside animated windows, staircases and crowds projected around them like a living graphic novel. - Technique: Performers stand in small cut-outs and at marked positions on a projection surface so projected animated bodies, clothes and objects fit around them. - Try it: Use face tracking in Lens Studio to attach a real face to a hand-drawn animated body that walks across the screen. Twist: the drawn body keeps walking when the real head stops, and they have to catch up. #### The Magic Flute (Die Zauberflöte, with Barrie Kosky) — 1927 (Suzanne Andrade & Paul Barritt) (2012) - Video: https://www.youtube.com/watch?v=wDTtRhxtm7E - Interaction: Performance, Projection, Drawing & Making - Platform & tech: Projection, projected animation, opera - Idea: Pinning performers to fixed points on a projected wall trades freedom for perfect registration, a useful trade-off for AR scenes that must stay aligned. - What it is: Mozart's opera at the Komische Oper Berlin, staged by Barrie Kosky with 1927: singers stand on small revolving doors high in a white wall while animated elephants, spiders and hearts surround them. - Technique: A flat wall with pivoting doors at several heights holds the singers while silent-film-style animation is projected across it. - Try it: Anchor an AR mural to a real wall and mark three spots where classmates stand, then animate characters that react to each person's pose. Twist: the animation changes when two people swap spots. #### Golem — 1927 (Suzanne Andrade & Paul Barritt) (2014) - Video: https://www.youtube.com/watch?v=1oizDDrC2i8 - Interaction: Performance, Projection, Drawing & Making - Platform & tech: Projection, projected animation, claymation, live music - Idea: A projected assistant that slowly takes control is a sharp warning for AR assistants; the form carries the critique. - What it is: A satire about technology in which a clay servant, the Golem, gradually takes over its owner's life; the actors perform inside animated cityscapes, clay characters and advertisements. - Technique: Hand-made clay animation and drawings are composited and projected on screens, with performers timed to them and live music on stage. - Try it: Build an AR assistant character that follows a user around the room and gradually covers more of their view with suggestions. Twist: the user must find a gesture that makes it shrink back. #### Roots — 1927 (Suzanne Andrade & Paul Barritt) (2019) - Video: https://www.youtube.com/watch?v=l-rx8_Hnz-k - Interaction: Performance, Projection, Drawing & Making - Platform & tech: Projection, projected animation, live music - Idea: Folk tales told through simple drawn projection show that a story needs style more than fidelity; AR storytelling can choose a handmade look too. - What it is: A show built from little-known folk tales, in which performers and musicians move among hand-drawn and cut-out animation projected across the stage. - Technique: Paul Barritt's drawn and cut-out animation is projected on screens while performers are lit and placed to share the frame. - Try it: Cut paper figures, photograph them, and animate them in AR around a real table to retell a folk tale in under two minutes. Twist: the story ending changes depending on where the viewer stands. ### 3hund (Friedrich van Schoor & Tarek Mawad) / Glowing Atoms *Light and projection artists (projection mapping in nature)* 3hund was the label of German media artists Friedrich van Schoor and Tarek Mawad, known for projecting glowing plants, mushrooms and giant spiders onto forests, beaches and building windows. Van Schoor now works with Dorothee van Schoor as Glowing Atoms; the work has been shown at Nuit Blanche Paris 2015, Ars Electronica 2017 and TEDx Sydney 2015. #### Spider Projection / Araneola — 3hund (Friedrich van Schoor & Tarek Mawad) / Glowing Atoms (2012) - Video: https://vimeo.com/37176398 - Interaction: Projection, Perception & Effects, Location & City - Platform & tech: Projection, projection mapping, scale model, video - Idea: Filming something real at miniature scale and projecting it at building scale creates a believable giant; matching scale and viewpoint matters more than polish. - What it is: Giant spiders seem to crawl inside a building in Saarbrücken, pressing against the windows from within. In fact two small spiders were filmed in a scale model of the building and the footage was projected from inside onto the window panes. - Technique: Live spiders were placed in a scale model and filmed from the street viewer's perspective, then rear-projected onto diffused windows so the scale and angle line up with the real façade. - Try it: Film a small toy or insect on a tabletop with a phone, key out the background, and place the clip as a giant video plane behind a real window using AR Foundation or Lens Studio with occlusion from the window frame. Twist: the creature reacts when a viewer steps into the camera's view. #### Bioluminescent Forest (Projections in the Forest) — 3hund (Friedrich van Schoor & Tarek Mawad) / Glowing Atoms (2014) - Video: https://vimeo.com/115082758 - Interaction: Projection, Location & City, Perception & Effects - Platform & tech: Projection, projection mapping, 3D animation - Idea: Light only the details nature already has, like a leaf's veins or a mushroom cap, and the real world looks enchanted rather than covered; AR overlays work best the same way. - What it is: After six weeks in a forest, the artists projected glowing veins, luminous plants and pulsing mushrooms onto real trees, moss and fungi at night. Filmed as a short film, the forest seems to breathe with its own bioluminescence. - Technique: Each plant and mushroom was likely photographed from the projector's position, then animated in 2D/3D so the light follows its exact outline before being projected back at night. - Try it: Scan a houseplant or a shoe with a phone (Polycam or Reality Composer), then build an AR effect in Lens Studio or AR Foundation that makes only its edges and veins glow and pulse slowly. Twist: the glow spreads to the next object only when the phone gets very close, like touching it. #### Spider Projection v.02 (Nuit Blanche 2015) — 3hund (Friedrich van Schoor & Tarek Mawad) / Glowing Atoms (2015) - Video: https://vimeo.com/152629846 - Interaction: Projection, Location & City, Perception & Effects - Platform & tech: Projection, projection mapping, scale model, video - Idea: Windows are ready-made screens with depth behind them; AR can treat every window as a portal into a hidden interior. - What it is: For Nuit Blanche in Paris, huge spiders appear trapped behind the windows of a city building, creeping across the glass above the night crowd. The second version of the piece was later shown in Le Havre and for Yves Saint Laurent's Halloween windows. - Technique: As in the 2012 version, miniature footage of real spiders was likely filmed inside a model of the façade and projected from behind onto several windows at once. - Try it: Map the windows of your school building with an image target or a WebXR plane, and place a different animated creature behind each window. Twist: creatures move from window to window as if they live inside the building. #### LUCID — 3hund (Friedrich van Schoor & Tarek Mawad) / Glowing Atoms (2016) - Video: https://vimeo.com/186269874 - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, electroluminescent wire, light installation, film - Idea: One luminous line placed in a vast landscape reframes the whole view; a single, well-placed AR primitive can do more than a crowded scene. - What it is: Simple geometric shapes made of electroluminescent wire, such as circles, triangles and lines, are placed in empty landscapes in Iceland and Austria. Each glowing figure sits in a lake, a cave or a mountain slope and turns the whole place into a quiet, surreal scene. - Technique: Battery-powered EL wire was bent over thin frames into circles and triangles and filmed at dusk with long exposures, so the lines read as pure light against the landscape. - Try it: Place a single glowing ring or triangle with WebXR or 8th Wall outdoors, anchored to the horizon line, and photograph it from three distances. Twist: the shape stays still in the world but its colour follows the time of day from the phone clock. #### Seaside — 3hund (Friedrich van Schoor & Tarek Mawad) / Glowing Atoms (2022) - Video: https://vimeo.com/785576368 - Interaction: Projection, Location & City, Tangible Objects - Platform & tech: Projection, projection mapping, film - Idea: Found objects can carry the story: a shell lit from outside becomes a small stage, and AR can pick out real objects rather than add new ones. - What it is: On a stormy North Sea beach in the Netherlands in winter, shells, sand ripples and beach grass light up with delicate projected patterns. The light moves with the rhythm of wind and waves, filmed on site at night. - Technique: A portable projector was aimed at individual shells and grass tufts on the beach, with animations likely masked to each object's silhouette. - Try it: Collect five small natural objects, scan them with a phone and create an AR effect that traces glowing patterns over each one on a table. Twist: the patterns move in time with a recorded sound of waves, so the table breathes like a beach. ### Anish Kapoor *Sculptor* British-Indian sculptor of mirror-polished and void-like forms such as Cloud Gate in Chicago, Sky Mirror and the whirlpool Descension. #### Sky Mirror — Anish Kapoor (2001) - Video: https://www.youtube.com/watch?v=Ee1nAvEZTwY - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, concave stainless steel dish, sky reflection - Idea: Bringing the sky down to eye level with a mirror is exactly what an AR portal to the sky can do on a lawn. - What it is: A six-metre concave steel mirror tilted toward the sky brings the clouds down to the ground; one side shows the sky, the other the surroundings upside down. The film shows a version installed at De Pont museum in Tilburg. - Technique: A large concave dish of polished stainless steel is tilted on a stand so its focal reflection frames the sky. - Try it: Place a circular AR portal on a lawn that shows the real sky above the user (from the phone's upward camera or a sky map). Twist: tilt the portal to show tomorrow's forecast sky. #### Cloud Gate — Anish Kapoor (2006) - Video: https://www.youtube.com/watch?v=0r_EQ4RqaAk - Interaction: Location & City, Shared & Social, Perception & Effects - Platform & tech: Desktop, polished stainless steel, seamless welding - Idea: A mirror object puts the public and the city inside the sculpture; people come to see themselves, which is also why AR mirror filters spread. - What it is: A 10-metre, 110-ton mirror-polished steel bean in Chicago's Millennium Park reflects the skyline, the clouds and the crowds walking under its arch, distorting them in its curves. - Technique: 168 stainless steel plates were welded and polished until the seams disappeared, creating a continuous curved mirror. - Try it: Place a large AR blob with a real-time environment-reflection material in a plaza and let friends gather under it to see their distorted reflection. Twist: the blob slowly morphs its shape to exaggerate whoever is closest. #### Dismemberment, Site 1 (Gibbs Farm) — Anish Kapoor (2009) - Video: https://www.youtube.com/watch?v=2CdmszOKzo0 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, red PVC membrane, steel ellipses, sculpture park - Idea: A single stretched surface across a valley shows how one bold shape can frame an entire landscape. - What it is: At Gibbs Farm in New Zealand, a red membrane 85 metres long is stretched between two steel ellipses across a valley, like a trumpet or a skin laid over the land. The video is a tour of Gibbs Farm that includes it. - Technique: A tensioned PVC fabric is stretched between two large steel rings anchored at either end of a gully. - Try it: Stretch an AR membrane between two real points (trees, lamp posts) using a cloth simulation, and colour it strongly against the landscape. Twist: let users pluck it like a drum to make a low sound. #### ArcelorMittal Orbit — Anish Kapoor (2012) - Video: https://www.youtube.com/watch?v=3xwVfgnJyeI - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, red steel lattice, observation tower, Cecil Balmond - Idea: A tower that twists around itself turns climbing into looping motion, a reference for AR landmarks you move through rather than look at. - What it is: Designed with engineer Cecil Balmond for the London 2012 Olympics, a 114-metre red steel structure loops around itself with a viewing platform and, since 2016, a spiral slide. - Technique: A continuous looping steel lattice is built around a central core; the structure's irregular curves were computed for stability. - Try it: Generate a looping AR ribbon tower with a spline tool and let users ride a camera path along it. Twist: the ribbon's loop count equals the number of people currently in the session. #### Descension — Anish Kapoor (2017) - Video: https://www.youtube.com/watch?v=p8knuUS4w-Q - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, water vortex, black dye, pumps - Idea: A hole that seems to swallow the ground is the physical version of an AR 'portal into the floor' and shows how motion sells depth. - What it is: In Brooklyn Bridge Park, a circular pit of black-dyed water spins into a continuous whirlpool about eight metres across that seems to drain into the ground. - Technique: Pumps spin black-dyed water in a round basin fast enough to form a stable vortex whose centre appears bottomless. - Try it: Use an AR depth mask to cut a round hole in a real floor and fill it with a spinning black vortex shader. Twist: objects the user drops near it get pulled in. ### Anna Fusté *Creative technologist and researcher (MIT Media Lab alumna)* Catalan creative technologist whose AR work uses paper, textiles and robots as tangible interfaces, from Google AR Experiments to MIT Media Lab research on computational thinking. #### Paper Cubes — Anna Fusté, Judith Amores (2017) - Video: https://www.youtube.com/watch?v=arOEUVO1rIw - Interaction: Tangible Objects, Play - Platform & tech: Phone, Unity, Vuforia - Idea: Paper cubes become remote controls that make AR stick figures jump, turn and stop. - What it is: Printable paper cubes become physical controllers for a crowd of AR stick figures: show a cube and the figures jump, turn or stop, and an 'AI cube' makes them smarter. - Technique: Each cube face is a Vuforia image target; the recognized face and cube pose trigger behaviors for AR characters walking over the table. - Try it: Print a few patterned paper cubes and recognize them with 8th Wall or Vuforia, with each pattern mapped to a character action. Twist: two cubes placed close together combine into a new command. #### HyperCubes — Anna Fusté (2019) - Video: https://www.youtube.com/watch?v=GnVp4hrQ7QU - Interaction: Tangible Objects, Information & UI, Play - Platform & tech: Phone, AR markers, Unity - Idea: Line up paper cubes to write a 'program', and an AR character carries it out in order. - What it is: Children arrange paper cubes printed with markers as coding blocks; the phone reads the sequence and an AR character executes it in space, teaching computational thinking. - Technique: Paper cubes act as fiducial markers whose order and spatial arrangement are read as a sequence of instructions, and the result is animated in AR. - Try it: Make a set of paper 'code blocks' (forward, turn, jump) and have a phone read their order so an AR robot walks through a maze. Twist: add a 'loop' block. #### Kinetic AR: robot motion programming in AR — Anna Fusté (2020) - Video: https://www.youtube.com/watch?v=imirzyrjOu0 - Interaction: Tangible Objects, Drawing & Making, Information & UI - Platform & tech: Phone, Vuforia Spatial Toolbox, Unity - Idea: Program a robot by drawing its path right on the floor. - What it is: Using a phone, you draw and edit a robot's motion path directly on the floor in AR and the physical robot follows it, making robot programming spatial and visual. - Technique: The AR interface tracks the robot and floor in a shared frame, and user-drawn spline paths are converted into motion commands sent to the robot. - Try it: Place a few waypoints on the floor in phone AR and have a robot vacuum or LEGO car pass through them in order. Twist: each waypoint can be set to make the robot stop and dance. ### Behnaz Farahi *Designer and artist; interactive wearables and responsive architecture* Iranian-born designer who makes 3D-printed garments and kinetic surfaces that react to gaze, facial expression, speech and brain activity using cameras, soft robotics and shape-memory alloys. #### The Living, Breathing Wall — Behnaz Farahi (2013) - Video: https://vimeo.com/81572631 - Interaction: Voice & Sound, Hands & Body, Spatial Mapping - Platform & tech: Desktop, shape-memory alloy, speech recognition, Arduino - Idea: A wall that recognises speech and answers with shape shows how architecture can listen; AR can let real walls 'listen' the same way. - What it is: A kinetic wall panel of flexible cells changes shape in response to people's speech and movement, opening and closing like a breathing skin. - Technique: Speech recognition and motion sensing drive shape-memory-alloy actuators that deform flexible panels; details of the control are likely custom Arduino code. - Try it: Detect a real wall with plane detection and cover it with AR cells that open when the microphone hears a keyword and close in silence. Twist: give each speaker's voice pitch its own region of the wall. #### Caress of the Gaze — Behnaz Farahi (2015) - Video: https://vimeo.com/152363295 - Interaction: Gaze & Attention, Hands & Body, Face - Platform & tech: Wearable, 3D printing, shape-memory alloy, eye tracking camera - Idea: Making another person's gaze visible on your body turns an invisible social signal into a physical response — the same move as gaze-driven AR effects. - What it is: A 3D-printed garment covered in quill-like spines ripples and lifts on the part of the body that an onlooker is looking at. - Technique: A small camera with gaze and face tracking estimates where the viewer is looking, and shape-memory-alloy actuators under multi-material 3D-printed spines move that region. - Try it: Use face tracking on a phone pointed at a friend to detect where on their body you are looking, and grow AR spines or feathers at that spot. Twist: invert it so the feathers flatten and hide wherever they are looked at. #### Synapse — Behnaz Farahi (2015) - Video: https://vimeo.com/139237974 - Interaction: Hands & Body, Face, Perception & Effects - Platform & tech: Wearable, EEG, 3D printing, actuators - Idea: Showing an inner state (attention) on the outside of the body is a strong idea for AR wearables and social filters. - What it is: A 3D-printed helmet whose surface opens and closes in response to the wearer's brain activity, lighting up as attention rises. - Technique: An EEG headset reads attention levels and drives actuators and lights embedded in a multi-material 3D-printed shell. - Try it: Make an AR head-halo that opens like petals when a simple input rises (heart rate from a watch, typing speed, or microphone level). Twist: show it only to other people, never to the wearer. #### Opale — Behnaz Farahi (2017) - Video: https://vimeo.com/232258166 - Interaction: Face, Hands & Body - Platform & tech: Wearable, facial expression tracking, soft robotics, pneumatics, fibre optics - Idea: Clothing that reacts to others' emotions shows a new kind of expressive interface; AR face filters can move from decorating yourself to answering others. - What it is: A garment covered in a forest of fibre-optic strands bristles, trembles or calms depending on the facial expressions of the people around the wearer. - Technique: A camera classifies nearby faces into expressions and drives pneumatic soft-robotic chambers under a silicone skin filled with fibre-optic strands. - Try it: Build a Lens Studio or ARKit effect that grows fur on the viewer's shoulders which bristles when the camera sees an angry face and softens for a smile. Twist: show the reaction on the person being filmed rather than the one filming. #### Iridescence — Behnaz Farahi (2019) - Video: https://vimeo.com/325446043 - Interaction: Face, Hands & Body, Perception & Effects - Platform & tech: Wearable, facial tracking, electromagnetic actuators, 3D printing - Idea: Flipping two-coloured elements is a cheap, readable display — an AR designer can reuse the flip as a visual language for attention and mood. - What it is: A collar of 200 flipping quills, inspired by a hummingbird's throat, changes colour and pattern in response to the movement and emotions of people nearby. - Technique: A facial-tracking camera feeds an interactive system that drives an array of electromagnetically flipped quills, producing colour waves across the collar. - Try it: Cover a real object (a chair, a jacket, a doorframe) with a grid of AR flip-tiles that turn over in a ripple toward whichever face the camera sees. Twist: map smiles to one colour and frowns to the other. ### Carsten Höller *Artist and former scientist making participatory perception experiments* Belgian-born artist trained as an entomologist, known for slides, Upside-Down Goggles and flickering light works. He made the portal walk Through for Apple's [AR]T program and the 7.8 Reduced Reality App with Acute Art, which flickers phone screen, torch and vibration at 7.8 Hz. #### Upside-Down Goggles — Carsten Höller (1994) - Video: https://www.youtube.com/watch?v=Ct3c9PzS6yE - Interaction: Perception & Effects, Hands & Body, Portals & Worlds - Platform & tech: Wearable, prism goggles - Idea: Perception is trainable: a headset or phone passthrough filter that simply flips or delays the world is already a powerful experience, no content needed. - What it is: Visitors put on heavy goggles that flip the world upside down and try to walk, pour or shake hands while their body slowly adapts. - Technique: Prism lenses invert the image on the retina, repeating classic perception experiments by George Stratton in the 1890s. - Try it: Make an upside-down passthrough in a HoloKit or Cardboard viewer by flipping the camera feed in Unity or WebXR, then time students pouring water. Twist: add a 300 ms delay instead of flipping and compare which is harder. #### Test Site — Carsten Höller (2006) - Video: https://www.youtube.com/watch?v=3xC53y2DQGc - Interaction: Hands & Body, Perception & Effects, Play - Platform & tech: Desktop, steel slides, polycarbonate - Idea: An artwork can be a body experience rather than an image: AR works gain force when they change how people move through a building, not only what they see. - What it is: Five giant spiral steel slides twist down through Tate Modern's Turbine Hall, and visitors ride them from different floors, shrieking into the hall below. - Technique: Stainless steel and transparent polycarbonate tubes form enclosed spiral slides engineered like fairground rides, with riders sliding on mats. - Try it: Design an AR 'slide' route down a real staircase with WebXR: arrows, speed lines and a sound that rises in pitch as you go down. Twist: record each run and replay classmates' ghost avatars sliding beside you. #### Golden Mirror Carousel — Carsten Höller (2014) - Video: https://www.youtube.com/watch?v=UQKF1dtvN3Y - Interaction: Perception & Effects, Hands & Body - Platform & tech: Desktop, carousel, mirror panels, gold - Idea: Slowing a familiar ride down turns fun into contemplation: in AR, changing only the speed of a known motion can make viewers notice their own body and surroundings. - What it is: A gleaming gold carousel with mirrored panels turns very slowly; riders on swinging seats see themselves and the gallery fragmented and revolving in the reflections. - Technique: A working carousel is clad in golden mirror panels and runs at a fraction of normal speed, so reflections become a slowly shifting kaleidoscope. - Try it: Anchor a slowly rotating ring of virtual mirror panels around a person in AR, each showing a delayed camera feed. Twist: tie rotation speed to the rider's breathing measured by phone motion. #### Decision (Isomeric Slides) — Carsten Höller (2015) - Video: https://www.youtube.com/watch?v=yAzyXVCelGE - Interaction: Hands & Body, Shared & Social, Play - Platform & tech: Desktop, steel slides, architecture - Idea: Choice is a material: an AR tour can split into two paths at a physical doorway, so the visitor's decision becomes part of the work. - What it is: Two intertwined tubular slides spiral out of the Hayward Gallery; visitors choose one of two doors and exits throughout the exhibition, and leave by sliding down. - Technique: Two mirror-image (isomeric) steel slides are mounted on the building exterior, paired with other decision-based pieces such as a dark corridor with two entrances. - Try it: Build a two-door AR experience at a real fork in a hallway: each branch shows a different version of the same virtual creature. Twist: at the end, show how many classmates chose each branch. #### Through — Carsten Höller, Apple (ARKit / visionOS teams) (2019) - Video: https://www.youtube.com/watch?v=SrqgNt2l91s - Interaction: Portals & Worlds, Perception & Effects, Location & City - Platform & tech: Phone, ARKit, orthographic rendering - Idea: Walk through a portal into a world that has no perspective. - What it is: On Apple's [AR]T Walk, a portal opens in the street and takes viewers into a world without perspective, where the usual depth cues of the city disappear. - Technique: An ARKit portal: a masked doorway shows a virtual interior rendered without perspective foreshortening, which replaces the camera view once the user steps through. - Try it: Build an AR portal whose inside world is rendered with an orthographic camera, then compare how it feels to walk through versus a normal perspective portal. Twist: make the inside world flip upside down, echoing Höller's Upside-Down Goggles. ### Cayden Pierce (Mentra / AugmentOS) *Founder of Mentra; builder of the open-source MentraOS (AugmentOS) smart-glasses OS* Cayden Pierce started TeamOpenSmartGlasses and Mentra, whose open-source OS runs AI apps such as live captions, conversation helpers and agents on many brands of display glasses. #### Convoscope: live definitions and insights during conversation — Cayden Pierce (Mentra / AugmentOS) (2023) - Video: https://www.youtube.com/watch?v=3n6DzuYQ_v8 - Source code: https://github.com/Mentra-Community/MentraOS - Interaction: Voice & Sound, Information & UI, Shared & Social - Platform & tech: Wearable, smart glasses, ChatGPT, speech recognition, open source - Idea: An assistant that listens along and whispers context into your eye. - What it is: An open-source smart-glasses app listens to your conversation and overlays real-time definitions, facts and insights in your view, so you can keep talking while the AI fills in the gaps. - Technique: Continuous speech recognition feeds an LLM that extracts rare terms and entities, and short cards are pushed to the glasses display. - Try it: Run live speech-to-text on a phone during a class discussion, have an LLM pick one unusual term every 30 seconds and show its definition in large text in a headset viewer. Twist: show a counter-argument instead of a definition. #### Hybrid Thinking smart glasses — Cayden Pierce (Mentra / AugmentOS) (2024) - Video: https://www.youtube.com/watch?v=DWyLFKIVrfM - Interaction: Voice & Sound, Information & UI, Shared & Social - Platform & tech: Wearable, smart glasses, LLM, proactive agents - Idea: Thinking with an AI in the middle of a real conversation, not after it. - What it is: AI smart glasses join everyday conversations and proactively surface ideas, facts and follow-up questions, a mode Cayden Pierce calls hybrid thinking. - Technique: Proactive agents monitor the transcript, decide when a contribution is useful, and send brief text to the display without being asked. - Try it: Prototype a 'silent debate partner': a phone listens to a two-person discussion and every minute shows one question the listener could ask next. Twist: the partner must support the person who is losing. #### AI smart glasses that order a Big Mac — Cayden Pierce (Mentra / AugmentOS) (2025) - Video: https://www.youtube.com/watch?v=ILG9XfiRlPM - Source code: https://github.com/Mentra-Community/MentraOS - Interaction: Voice & Sound, Location & City - Platform & tech: Wearable, AugmentOS, Browser Use, LLM agent - Idea: Glasses as the remote control for an agent that acts in the world for you. - What it is: A spoken request on AugmentOS glasses launches a web agent that autonomously browses and orders a Big Mac, reporting progress in the wearer's view. - Technique: Voice commands from the glasses app are passed to Browser Use, an LLM agent that operates a web browser, and status updates stream back to the display. - Try it: Connect a speech-to-text page to a browser-automation agent that can do one harmless task (search a library catalogue) and show each step as floating text in AR. Twist: the agent must ask permission with a nod before the final click. #### Live caption glasses for the Deaf and hard of hearing — Cayden Pierce (Mentra / AugmentOS) (2025) - Video: https://www.youtube.com/watch?v=CRyhfPrfywU - Source code: https://github.com/Mentra-Community/MentraOS - Interaction: Voice & Sound, Shared & Social - Platform & tech: Wearable, AugmentOS, speech recognition, Even Realities G1, Vuzix Z100 - Idea: Subtitles for real life, placed where you are already looking. - What it is: AugmentOS shows fast, free live captions of the people around you on several brands of display glasses, so Deaf and hard-of-hearing wearers can follow speech while looking at the speaker. - Technique: Streaming speech recognition runs on the phone or cloud and pushes rolling text to the glasses' micro-display with low latency. - Try it: Build a phone web page with live speech-to-text shown as large captions in a headset viewer, and test it in a noisy café. Twist: colour each speaker's words differently using voice direction. #### Smart glasses AI tour guide — Cayden Pierce (Mentra / AugmentOS) (2026) - Video: https://www.youtube.com/watch?v=pEm8m59k3KQ - Source code: https://github.com/Mentra-Community/MentraOS - Interaction: Location & City, Voice & Sound, Information & UI - Platform & tech: Wearable, MentraOS, Mentra Live, LLM, GPS - Idea: A personal guide for any street, built in a weekend on open glasses. - What it is: Walking around a city with an AI tour guide on MentraOS glasses that knows the history, geography and architecture of what the wearer is looking at, built over a weekend. - Technique: GPS location and camera frames are sent with a guide persona to an LLM, and short spoken or displayed explanations come back as the wearer walks. - Try it: Make a location-triggered WebAR tour of five spots on campus where a vision model describes each building from a photo, in the voice of a chosen historical figure. Twist: the guide gets the facts subtly wrong and students must catch it. ### Children of the Light (Christopher Gabriel & Arnout Hulskamp) *Light-art duo* Amsterdam duo who use light as their main material across performance, sculpture and immersive installations, from a darkened grid of reflective tubes lit only by visitors' phones to rotating halos and slowly rising lines of light. They made the kinetic light sculpture at the centre of Alexander Whitley's Overflow. #### ZEROTIME — Children of the Light (Christopher Gabriel & Arnout Hulskamp) (2014) - Video: https://vimeo.com/129778294 - Interaction: Perception & Effects, Tangible Objects, Shared & Social - Platform & tech: Phone, Projection, retroreflective material, haze, phone flashlights, generative sound - Idea: You only receive as much light as you shine into the room. - What it is: A light installation without light: visitors enter a dark, hazy room and use their phone torches to find their way through a grid of retroreflective tubes that bounce the light straight back, while a program turns the phones' light into a five-channel soundscape. - Technique: Retroreflective tubes return each phone's torch beam toward its source through haze, and light sensors in the space feed an interactive framework that generates the sound. - Try it: Darken a room, hang strips of retroreflective tape on strings and let students explore with phone torches while a light sensor on an Arduino changes a drone sound. Twist: give only one student a torch and let them guide the others. #### Reflector Suits — Children of the Light (Christopher Gabriel & Arnout Hulskamp) (2015) - Video: https://vimeo.com/149505356 - Interaction: Perception & Effects, Performance - Platform & tech: Wearable, retroreflective fabric, screen printing, performance - Idea: Two living sculptures that only light up in other people's cameras. - What it is: Unable to show a light work in a bright space without haze, the duo became the artwork: they stand still in public in patterned retroreflective suits that flare into light whenever a camera flash or headlight hits them, here on the streets of Shibuya. - Technique: Retroreflective fabric bounces light straight back to its source, so the suits look plain to the eye but glow in any photo taken with flash. - Try it: Cut shapes from retroreflective tape, stick them on clothing or a wall, and compare what people see directly with what appears in flash photos. Twist: hide a message that only shows in a flash photo. #### Warping Halos — Children of the Light (Christopher Gabriel & Arnout Hulskamp) (2017) - Video: https://vimeo.com/233282594 - Interaction: Perception & Effects, Performance, Voice & Sound - Platform & tech: Projection, kinetic light, LED ring, DMX, haze - Idea: A single spinning ring of light turns into a halo that bends the room. - What it is: A large motorised, illuminated ring hangs from the ceiling and revolves through the space, shifting colour and intensity with a soundscape so the halo seems to warp and multiply around the viewers. - Technique: An LED ring on a motorised mount is rotated on several axes and sequenced with DMX lighting and sound, so persistence of vision and haze create ghost circles. - Try it: Attach an LED hoop to a slow rotating motor or a lazy Susan in a dark room and film it with long exposure on phones. Twist: change the rotation speed with the loudness of music. #### TRANSITO — Children of the Light (Christopher Gabriel & Arnout Hulskamp) (2018) - Video: https://vimeo.com/278128603 - Interaction: Perception & Effects, Location & City - Platform & tech: Projection, LED, light sequencing, architecture - Idea: Waves of light make a corridor feel infinite and pull you along it. - What it is: A 50-metre light corridor made with the architects Space Encounters for Salone del Mobile in Milan: coloured waves of light travel along repeated frames, so the gallery seems endless and visitors feel carried forward by the light. - Technique: A row of LED frames along the corridor is sequenced as travelling waves of colour, using repetition and perspective to erase the sense of the room's length. - Try it: Line a school corridor with ten phone screens or LED strips and program a colour wave that travels at walking speed. Twist: make the wave run against the walking direction and see how people's pace changes. #### Between Light — Children of the Light (Christopher Gabriel & Arnout Hulskamp) (2019) - Video: https://vimeo.com/493404347 - Interaction: Perception & Effects, Location & City - Platform & tech: Projection, kinetic light, LED, motorised winches - Idea: A single line of light becomes a horizon that rises and sets inside a church. - What it is: In the Gothic Oude Kerk in Amsterdam, a long, gently sagging line of light hangs in the empty nave and, over a 24-minute sequence, slowly rises and falls like a sunrise and sunset. - Technique: A suspended LED line on motorised winches is choreographed in height and brightness on a slow 24-minute cycle. - Try it: In phone AR, place a thin glowing line across a real room and animate it to rise and fall over five minutes while students sit and watch. Twist: tie the height of the line to the real sun's elevation at that moment. ### Christopher Bauder / WHITEvoid *Light and kinetic artist; founder of WHITEvoid and the Kinetic Lights system* Berlin artist whose studio WHITEvoid makes kinetic light installations and audiovisual shows, often with Robert Henke (Atom, GRID, DEEP WEB) and Kangding Ray (SKALAR), and who marked the Berlin Wall with 8,000 glowing balloons in Lichtgrenze. #### ATOM (with Robert Henke) — Christopher Bauder / WHITEvoid (2007) - Video: https://www.youtube.com/watch?v=3SpEX2Scwso - Interaction: Performance, Voice & Sound, Perception & Effects - Platform & tech: Desktop, Kinetic Lights winches, helium balloons, LEDs, Ableton Live - Idea: A low-resolution grid of 64 lights is enough to make a 3D display — AR effects don't need many elements if their motion is rhythmic. - What it is: An 8×8 grid of glowing helium balloons on computer-controlled winches rises and sinks above the audience, forming shapes in sync with Robert Henke's electronic music. - Technique: Each balloon hangs on a DMX-controlled winch; height and LED colour are sequenced from the same timeline as the music. - Try it: Make an 8×8 grid of glowing spheres float above the floor in AR and drive their heights from the frequency bands of live microphone input. Twist: let two audience members each control one half of the grid with their phones. #### Lichtgrenze — Christopher Bauder / WHITEvoid (2014) - Video: https://www.youtube.com/watch?v=Sq0Ecpakna4 - Interaction: Location & City, Shared & Social - Platform & tech: Desktop, 8,000 illuminated balloons, light stands, city-scale installation - Idea: Marking a vanished boundary on the real map, then letting people release it, is location-based AR in physical form. - What it is: For the 25th anniversary of the fall of the Berlin Wall, 8,000 glowing balloons on poles traced 15 km of the former wall through the city and were released one by one into the night sky. - Technique: Balloons with LEDs were mounted on stands along the surveyed wall path; volunteer 'balloon patrons' released them in sequence at a set time. - Try it: Plot a vanished line in your town (an old wall, river or tram track) as a row of glowing geo-anchored markers in phone AR. Twist: let each visitor release one marker so the line disappears over the day. #### DEEP WEB (with Robert Henke) — Christopher Bauder / WHITEvoid (2016) - Video: https://www.youtube.com/watch?v=n2bf8L7G6WE - Interaction: Performance, Voice & Sound, Perception & Effects - Platform & tech: Projection, 175 motorized spheres, 12 laser projectors, Kinetic Lights winches - Idea: Lines only appear when light hits an object in the air — AR can use real or virtual 'anchors' so that graphics connect points in space. - What it is: In a huge dark power station, 175 motorized white spheres rise and fall while laser beams hit them and draw connecting lines, forming a moving 3D web of light to a live score. - Technique: Sphere heights are driven by winches and laser projectors are calibrated to each sphere's known 3D position, so beams can jump between them. - Try it: Hang a few real balls or paper lanterns, register their positions in AR, and draw animated laser-like lines between them from your phone. Twist: let the lines rewire themselves when someone walks through them. #### GRID (with Robert Henke) — Christopher Bauder / WHITEvoid (2017) - Video: https://www.youtube.com/watch?v=gT2Mnbs8vYs - Interaction: Performance, Voice & Sound - Platform & tech: Desktop, DMX winches, RGB LED rods, Kinetic Lights - Idea: Tilting a line of light turns it into a plane — a few controllable lines can sketch whole architectures around the viewer. - What it is: Horizontal LED rods hang in a grid on winches, lifting, tilting and changing colour to Robert Henke's music, so lines of light build planes and tunnels in space. - Technique: Each rod hangs from two winches, so the difference in cable length sets its tilt; motion and colour are sequenced with the music. - Try it: Hang ten virtual light rods from the ceiling in AR and animate their tilt to form a tunnel the viewer can walk through. Twist: bind each rod's angle to the viewer's distance from it. #### SKALAR (with Kangding Ray) — Christopher Bauder / WHITEvoid (2018) - Video: https://www.youtube.com/watch?v=8-hxsm8lDps - Interaction: Perception & Effects, Voice & Sound - Platform & tech: Desktop, kinetic mirrors, moving head lights, winches, spatial sound - Idea: Mirrors multiply a few light sources into a whole architecture — reflective virtual surfaces can multiply the reach of AR content. - What it is: Dozens of large kinetic mirrors hang in the dark Kraftwerk Berlin hall and tilt to catch beams from moving lights, bouncing them into ever-changing structures to music by Kangding Ray. - Technique: Winch-held mirrors are tilted by motors while moving-head lights aim at them; beam paths are planned from the known mirror positions. - Try it: Place three virtual mirrors in a real room and let the user aim a beam from their phone so it bounces between them onto the real walls. Twist: make the mirrors slowly tilt by themselves so the user has to chase the beam. ### Conrad Shawcross *Sculptor of machines, robots and light* British sculptor (born 1977, Royal Academician) whose mechanical sculptures, from Slow Arc Inside a Cube to the robot of The ADA Project, trace geometry, time and motion with a moving light or arm. #### Slow Arc Inside a Cube IV — Conrad Shawcross (2009) - Video: https://www.youtube.com/watch?v=nxPzEa2YSFw - Interaction: Perception & Effects - Platform & tech: Desktop, mechanical arm, single light source, steel cage cube - Idea: Moving one light inside a simple object turns the whole room into a moving drawing — AR shadows from virtual lights can transform real spaces. - What it is: A robotic arm moves a single bright light slowly inside a cage cube, casting the shadow of the cage across the whole room so the walls seem to stretch and tilt. - Technique: A mechanical arm traces a slow path (inspired by the motion of a molecule) with a lamp inside a lattice cube, projecting shifting perspective shadows. - Try it: Put a virtual point light inside a virtual lattice cube in the middle of a real room and cast its shadows onto the scanned walls with AR occlusion. Twist: let the user drag the light with their hand and watch the room warp. #### Timepiece — Conrad Shawcross (2013) - Video: https://www.youtube.com/watch?v=zkSl7KlnQm8 - Interaction: Perception & Effects - Platform & tech: Desktop, industrial robot arms, light source, clock mechanism - Idea: Turning a clock into a moving light makes time something you stand inside — AR can externalize invisible rhythms at architectural scale. - What it is: In the domed Roundhouse, a huge three-armed mechanism turns like a clock with a light at its tip, sweeping shadows across the space and marking hours, minutes and seconds. - Technique: Three nested rotating arms move at the rates of clock hands, carrying a lamp whose combined motion traces a complex path through the dome. - Try it: Anchor a room-sized virtual clock in AR whose three arms carry a light and cast shadows of real furniture over one class period. Twist: speed it up 60× when someone raises a hand. #### The ADA Project — Conrad Shawcross (2014) - Video: https://www.youtube.com/watch?v=ndSth0XSEUw - Interaction: Performance, Voice & Sound - Platform & tech: Desktop, industrial robot arm, light, commissioned music - Idea: A robot becomes a dancer when its motion follows music — AR characters don't need faces, only well-timed movement. - What it is: An industrial robot named ADA holds a light at the end of its arm and 'dances' to pieces by different composers, drawing arcs of light in a dark room. - Technique: A six-axis industrial robot is programmed with motion paths synchronized to each commissioned score, with a lamp as its end effector. - Try it: Program a virtual robotic arm in AR with a light at its tip to dance to a 30-second track, and record its light trails as a long exposure. Twist: let a classmate compose the music and swap tracks without changing the motion. #### Optic Cloak — Conrad Shawcross (2015) - Video: https://www.youtube.com/watch?v=QVjBKCZX-w4 - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, perforated aluminium panels, moiré patterns, architecture - Idea: Camouflage can be dynamic: patterns that shift with your movement make a solid building look immaterial — a lesson for diminished-reality AR. - What it is: A 49-metre energy-centre flue in Greenwich is wrapped in faceted, perforated panels whose overlapping holes create shimmering moiré patterns that make the tower's outline dissolve. - Technique: Two layers of perforated triangular panels overlap at an offset so that parallax creates moiré interference as the viewer moves, inspired by dazzle camouflage. - Try it: Wrap a real pillar or building in AR with two layers of virtual perforated patterns that shimmer with the user's movement. Twist: tune the patterns so the object seems to disappear from one sweet spot. #### The Dappled Light of the Sun — Conrad Shawcross (2015) - Video: https://www.youtube.com/watch?v=n5u1RIIi0p0 - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, weathering steel tetrahedra, fractal geometry - Idea: A structure overhead can shape the light you walk through — outdoor AR can use the sky and sun as part of the composition. - What it is: A cloud-like canopy of thousands of steel tetrahedra hovers above visitors' heads outdoors, breaking sunlight into dappled patches like a tree canopy. - Technique: Tetrahedra of weathering steel are assembled in a branching, cloud-like mass on a few supports, letting sunlight filter through the gaps. - Try it: Float a canopy of virtual tetrahedra above an outdoor plaza in phone AR and cast shadows that match the real sun direction. Twist: make the canopy slowly grow new branches each time a visitor looks up. ### Cyril Diagne *Artist, designer and ML engineer; co-founder of ClipDrop, former artist in residence at the Google Arts & Culture Lab* French creative technologist who co-founded the Lab212 collective, taught interaction design at ECAL and spent five years at the Google Arts & Culture Lab before co-founding ClipDrop (acquired by Stability AI). His 2020 AR Cut & Paste demo, which lifts real objects into Photoshop with a phone, became one of the most shared AR+ML prototypes ever. #### Super Mario Brush — Cyril Diagne (2012) - Video: https://vimeo.com/46281850 - Interaction: Projection, Drawing & Making, Play - Platform & tech: Projection, openFrameworks, computer vision, projector - Idea: A paper drawing becomes a playable video game level. - What it is: Children draw a Super Mario level on paper with coloured pads for ground, blocks and enemies; a camera reads the drawing and the projected game becomes playable on top of it. - Technique: A camera above the table segments the coloured pads by colour, converts them into tile types of a Mario-style level and projects the running game back onto the paper (made with Béatrice Lartigue for Lab212). - Try it: Build a projector-and-webcam setup where red paper becomes lava and blue paper becomes platforms in a simple p5.js platformer. Twist: let a second player change the level by moving paper while the first is playing. #### Enthusiast Overlay — Cyril Diagne (2016) - Video: https://vimeo.com/224709486 - Interaction: Tangible Objects, Perception & Effects, Drawing & Making - Platform & tech: Desktop, computer vision, real-time tracking - Idea: Scribbled cartoon characters bring ordinary hanging objects to life on a see-through screen. - What it is: Twelve tennis balls hang in front of an AR screen; on the screen each ball gets a rough, hand-drawn comic face and body that bounces with endless enthusiasm, like the internet's 'realistic doodles' drawn over video clips. - Technique: A camera behind the screen likely tracks each suspended ball and pins pre-drawn, loosely animated doodle sprites to its position in real time. - Try it: Hang a few coloured balls on strings, track them by colour with a webcam in p5.js, and draw a scribbled face that sticks to each ball on a monitor placed in front. Twist: make each character react when two balls touch. #### AR Cut & Paste — Cyril Diagne (2020) - Video: https://x.com/cyrildiagne/status/1259441154606669824 - Interaction: Tangible Objects, Information & UI, Drawing & Making - Platform & tech: Phone, Desktop, U²Net, SIFT, Photoshop, Python - Idea: Copy and paste between the physical world and your computer with a phone camera. - What it is: Point a phone at an object in a book or on the desk, tap to 'cut' it, then point the phone at a laptop screen to 'paste' it straight into Photoshop or a slide at the exact spot you are aiming at. - Technique: A U²Net salient-object model cuts the object out of the photo, and SIFT feature matching between the phone camera and a screenshot finds where the phone is pointing on the screen, then a local server pastes the cut-out there. - Try it: Build a two-step web prototype: photograph an object with a phone, remove the background with an off-the-shelf segmentation API, and send the cut-out to a laptop page where it drops at a random position. Twist: use a printed marker on the laptop bezel so the phone's aim decides where the object lands. #### Copy printed text to desktop with AR+ML — Cyril Diagne (2020) - Video: https://x.com/cyrildiagne/status/1262047009411907585 - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, Desktop, OCR, SIFT, Python - Idea: Treat printed text as something you can pick up and drop into a digital document. - What it is: The sixth of his ten AI+UX prototypes: aim the phone at a paragraph in a printed book, and the recognised text is pasted into a document on the computer where the phone points. - Technique: Likely combines on-device OCR of the camera frame with the same screen-matching trick as AR Cut & Paste to locate the target point on the desktop. - Try it: Make a phone web app that reads a printed sentence with an OCR library and posts it to a shared class web page in real time. Twist: let the sentence appear in the page's font and size of the place where the phone is aimed. #### Doodle face masks in the browser — Cyril Diagne (2020) - Video: https://x.com/cyrildiagne/status/1254372807531728896 - Interaction: Face, Drawing & Making - Platform & tech: Web, TensorFlow.js, FaceMesh, WebGL - Idea: Anyone can design a face filter by drawing on a flat map of a face. - What it is: Draw a doodle on a flat face template in the browser and it instantly becomes a mask that follows your real face in the webcam, stretching with every expression. - Technique: TensorFlow.js FaceMesh predicts 468 face landmarks; a 3D mesh with a fixed UV layout is aligned to them each frame, so a 2D drawing on the UV texture wraps onto the moving face. - Try it: Use a MediaPipe FaceMesh web template to map a student's drawing onto the face texture, then hold a 'mask parade' where everyone wears a classmate's doodle. Twist: make parts of the drawing animate when the mouth opens. ### DARKFIELD (David Rosenberg & Glen Neath) *Immersive audio theatre company; artistic directors David Rosenberg and Glen Neath* DARKFIELD builds shows inside shipping containers fitted out as a séance room, an airliner cabin or a hotel, then turns the lights off completely and tells the story through 360-degree binaural sound and physical effects. During the pandemic its DARKFIELD RADIO app moved the same trick into people's own kitchens and living rooms. #### SÉANCE — DARKFIELD (David Rosenberg & Glen Neath) (2016) - Video: https://www.youtube.com/watch?v=S3DsmGIBGbw - Interaction: Voice & Sound, Perception & Effects, Shared & Social - Platform & tech: Wearable, binaural audio, dummy-head recording, shipping container set, sensory effects - Idea: Total darkness plus binaural sound makes a group believe something has entered the room. - What it is: Twenty people sit around a table in a shipping container dressed as a Victorian séance room; the lights go out and, through headphones, a medium and unseen presences seem to move around and touch the group. - Technique: Pre-recorded binaural audio (recorded with a dummy head at each seat position) is played in sync to every headset while small physical effects such as air, vibration and touch are triggered in the dark container. - Try it: Record a 3-minute binaural 'ghost story' with a dummy head or in-ear binaural mics in your classroom, then play it to classmates blindfolded in the same seats. Twist: add one real physical cue (a draught, a tap on the table) timed to the audio. #### FLIGHT — DARKFIELD (David Rosenberg & Glen Neath) (2018) - Video: https://www.youtube.com/watch?v=sEgi87Pp3hs - Interaction: Voice & Sound, Perception & Effects, Tangible Objects - Platform & tech: Wearable, binaural audio, replica aircraft cabin, motion effects - Idea: A real seat and seatbelt plus binaural sound let two parallel worlds happen to your body. - What it is: Audiences board a 40ft shipping container built as an exact Airbus economy cabin, buckle in and, in complete darkness, hear a flight that splits into two realities: in one the plane lands, in the other it does not. - Technique: A physical replica cabin supplies touch and posture while binaural recordings, seat motion and lighting blackout supply the narrative, borrowing the many-worlds interpretation of quantum mechanics as the plot device. - Try it: Choose a familiar seat (bus, dentist chair, cinema) and design a 2-minute audio piece that makes it feel like it is moving. Twist: branch the audio so that listeners in two halves of the room get different endings. #### DOUBLE (DARKFIELD RADIO) — DARKFIELD (David Rosenberg & Glen Neath) (2020) - Video: https://www.youtube.com/watch?v=SZGvMM_xg7U - Interaction: Voice & Sound, Shared & Social, Perception & Effects - Platform & tech: Phone, binaural audio, mobile app, site-agnostic script - Idea: Your own kitchen and the person opposite you become the set of an audio play. - What it is: Two people who live together sit at their own kitchen table facing each other, put on headphones and close their eyes; a phone app then plays a story about the Capgras delusion, the fear that a loved one has been replaced by an exact copy. - Technique: A phone app delivers a timed binaural recording scripted around a standard layout (two chairs, a table), so the sounds of footsteps and voices land convincingly in the listeners' own room. - Try it: Script and record a 3-minute audio piece for two listeners at any table, including instructions such as 'reach out and touch the other person's hand'. Twist: give the two listeners slightly different tracks so they no longer trust each other. #### EULOGY — DARKFIELD (David Rosenberg & Glen Neath) (2021) - Video: https://www.youtube.com/watch?v=b-jatWxFqXw - Interaction: Voice & Sound, Perception & Effects, Play - Platform & tech: Wearable, binaural audio, speech recognition, branching narrative - Idea: A whole building exists only in sound, and your spoken answers choose which room you enter. - What it is: In a dark container each audience member is guided through an imaginary labyrinthine hotel; they answer questions aloud and the story reacts to what they say. - Technique: Binaural 360-degree audio builds the virtual architecture while speech recognition on each seat branches the narrative according to the participant's replies. - Try it: Build a voice-controlled audio walk of an imaginary building with a web speech API: at each 'door' the listener says left or right. Twist: let the narrator remember a word the listener said earlier and use it against them. #### ARCADE — DARKFIELD (David Rosenberg & Glen Neath) (2024) - Video: https://www.youtube.com/watch?v=HltDEYWOorw - Interaction: Voice & Sound, Play, Perception & Effects - Platform & tech: Wearable, binaural audio, arcade controls, branching narrative - Idea: A video game with the video removed: you become the avatar through sound. - What it is: Players sit at retro arcade cabinets in a pitch-dark container and steer an avatar through a war-torn world; there is almost nothing to see, so the game plays out as 360-degree sound and physical effects around them. - Technique: Branching choose-your-own-path audio is driven by arcade controls, with binaural sound and sensory effects replacing the screen. - Try it: Make a 5-choice audio adventure controlled by a joystick or Makey Makey, played in a dark room with headphones. Twist: add one moment where the player must physically stand up or turn around to continue. ### FIELD (FIELD.IO / FIELD.SYSTEMS) *Creative studio for art, design and technology* Studio working with code, data, moving image and spatial experiences for clients and cultural projects. For IKEA and SPACE10's Everyday Experiments it built several LiDAR and AI-driven AR prototypes, from a room that plays music to elephants that measure your furniture. #### Fort Builder — FIELD (FIELD.IO / FIELD.SYSTEMS), SPACE10 (IKEA's research and design lab) (2020) - Video: https://vimeo.com/554339068 - Interaction: Play, Spatial Mapping, Tangible Objects - Platform & tech: Phone, LiDAR, physics, ARKit - Idea: The childhood joy of building and smashing a fort, without the mess. - What it is: Digital copies of your real sofa cushions, chairs and lamps can be stacked into a fort that ignores gravity until you let go, and then it all comes crashing down without breaking anything. - Technique: Objects are captured from the room as meshes, stacked with physics suspended, then released into a rigid-body simulation that collides with the scanned room. - Try it: Stack AR copies of three scanned objects into a tower in phone AR, then tap to enable gravity. Twist: the tallest tower in class wins, but it must stand for five seconds. #### Spatial Instruments — FIELD (FIELD.IO / FIELD.SYSTEMS), SPACE10 (IKEA's research and design lab) (2020) - Video: https://vimeo.com/429281154 - Interaction: Voice & Sound, Spatial Mapping, Perception & Effects - Platform & tech: Phone, LiDAR, ARKit, generative audio - Idea: Perceive your room through sound instead of sight, so rearranging furniture becomes composing. - What it is: Moving a phone around the living room turns every object it sees into part of an ever-changing soundscape: shelves, lamps and sofas each play notes based on their shape, depth, colour and position. - Technique: LiDAR depth and object detection segment the room into objects, and each object's size, distance and colour are mapped to pitch, timbre and rhythm in a generative audio engine. - Try it: Map the heights of detected AR planes in a room to notes and play them as the phone sweeps past. Twist: moving one chair must change the melody. #### Chain of Traceability — FIELD (FIELD.IO / FIELD.SYSTEMS), SPACE10 (IKEA's research and design lab) (2021) - Video: https://www.youtube.com/watch?v=BkPHI3FM7lg - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, object recognition, blockchain, ARKit - Idea: Let objects tell their own life story, from source to second life. - What it is: Point a phone at a household object and AR reveals its history: where its materials came from, how it was made, its carbon footprint and how it could be repurposed. - Technique: Neural-network object recognition identifies the item, looks up a digital twin registered on a blockchain and renders its supply-chain data as AR layers around it. - Try it: Pick three objects in class, research where they came from and attach an AR timeline that unfolds from each when recognised. Twist: end each timeline with an idea for its next life. #### Digital Buddy — FIELD (FIELD.IO / FIELD.SYSTEMS), SPACE10 (IKEA's research and design lab) (2021) - Video: https://vimeo.com/558597213 - Interaction: Information & UI, Voice & Sound - Platform & tech: Phone, GPT-3, ARKit - Idea: A companion on your side that reads the fine print for you. - What it is: A small 3D avatar appears next to you in AR and explains long terms and conditions in plain language, for example answering whether a messaging app can read your private messages. - Technique: GPT-3 summarises and answers questions about privacy policies, and the answer is voiced by an animated AR character placed on a nearby surface. - Try it: Place a character in phone AR that reads a pasted app policy through an LLM and answers three yes/no questions. Twist: the character's mood changes with how privacy-friendly the answer is. #### Extreme Measures — FIELD (FIELD.IO / FIELD.SYSTEMS), SPACE10 (IKEA's research and design lab) (2021) - Video: https://vimeo.com/554342553 - Interaction: Spatial Mapping, Play, Perception & Effects - Platform & tech: Phone, LiDAR, ARKit - Idea: Measure space by filling it with something absurd and friendly. - What it is: Instead of a tape measure, you place a virtual elephant under the bed or in a nook and it inflates until it fills the space, giving an intuitive sense of how much room is there. - Technique: LiDAR depth defines the free volume around a placed anchor, and a character mesh scales until it collides with the surrounding geometry, reporting the resulting size. - Try it: In phone AR, place a balloon in a shelf gap and grow it until it hits the planes around it, then show its volume in 'balloons'. Twist: choose a unit that makes people laugh. ### Florentijn Hofman *Artist* Dutch artist of giant public sculptures of familiar toys and animals, above all the Rubber Duck that has floated in more than twenty harbours. #### Rubber Duck — Florentijn Hofman (2007) - Video: https://www.youtube.com/watch?v=G4-2UUz4Px4 - Interaction: Location & City, Perception & Effects, Shared & Social - Platform & tech: Desktop, inflatable, harbour, world tour - Idea: Scaling up a bath toy makes the city look like a bathtub, a perfect example of how size change alone creates a new story in AR. - What it is: A giant yellow rubber duck, up to 26 metres tall, has floated in more than twenty harbours since 2007, from Saint-Nazaire and Osaka to Hong Kong and Sydney, drawing crowds to the waterfront. - Technique: A PVC inflatable is mounted on a floating pontoon and inflated with blowers, then towed around the harbour. - Try it: Scan a small everyday object and place it at 20 metres tall in a real body of water or a plaza in AR. Twist: the object bobs to the rhythm of live wave or wind data. #### Lookout Rabbit — Florentijn Hofman (2011) - Video: https://www.youtube.com/watch?v=umn8NRidbes - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, timber structure, lookout, riverside park - Idea: A giant figure you can enter and look out of turns a sculpture into a viewpoint, an idea for AR creatures that become portals. - What it is: A 12-metre rabbit with a red dot on its side stands in Valkhof Park in Nijmegen; visitors can climb inside and look out over the river Waal. - Technique: A timber frame shaped like a rabbit contains stairs and a viewing opening at the red dot. - Try it: Place a giant AR animal with a door; when the user steps inside, switch to a view from its eyes, showing the landscape from its height. Twist: the animal's eyes follow wherever the user was looking before entering. #### Steelman — Florentijn Hofman (2011) - Video: https://www.youtube.com/watch?v=I2GhmlB4HOI - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, concrete, public housing district - Idea: A sleepy giant toy in a housing estate gives a neighbourhood a friendly mascot, a gentle model for community AR characters. - What it is: A 10-metre concrete bear holding a pillow under its arm stands in the Staalmanplein neighbourhood of Slotervaart, Amsterdam. - Technique: The bear was cast in concrete in sections over a steel frame and finished on site. - Try it: Design an AR mascot for your street that residents vote on, and place it where most neighbours pass. Twist: it yawns and goes to sleep at the neighbourhood's average bedtime. #### HippopoThames — Florentijn Hofman (2014) - Video: https://www.youtube.com/watch?v=byJIXyGlWkY - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, timber cladding, floating sculpture, river - Idea: An animal half-submerged reads as alive and moving through the real river, a good use of the water line as an AR occlusion boundary. - What it is: For the Totally Thames festival in London, a 21-metre wooden hippo floated in the Thames at Nine Elms, rising from the water as if wading through the river. - Technique: A timber-clad frame shaped like the upper half of a hippo was built on a floating barge. - Try it: Place an AR animal in a real river or pond so that only the top half shows above the waterline, using a water-plane occluder. Twist: it slowly swims upstream against the current. #### Moon Rabbit — Florentijn Hofman (2014) - Video: https://www.youtube.com/watch?v=TzN0A_UB8ww - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, inflatable, landscape art festival - Idea: Linking a giant figure to a local legend and festival gives it meaning beyond scale, a lesson for culturally specific AR. - What it is: At the Taoyuan Land Art Festival in Taiwan, a 25-metre rabbit lies on its side gazing at the sky, a nod to the Mid-Autumn legend of the rabbit on the moon. - Technique: A large fabric-covered form lies on the ground in the festival grounds, lit at night. - Try it: Choose a local legend and place its main character as a giant AR figure where the story is set. Twist: the figure appears only on the festival date or at the right moon phase. ### Freya Björg Olafson *Intermedia artist working with performance, video, motion capture and XR* Icelandic-Canadian intermedia artist whose solo performances stage the body against screens, avatars and internet footage: AVATAR (2009) on online self-display, HYPER_ (2013) with UV light and 3D projection, and MÆ – Motion Aftereffect (2019) on how XR changes the body. She joined the 2nd Choreographic Coding Lab at A+E Lab in 2025. #### AVATAR — Freya Björg Olafson (2009) - Video: https://vimeo.com/6602039 - Interaction: Performance, Projection, Face - Platform & tech: Projection, Quartz Composer, video projection, webcam - Idea: Perform the body as an internet profile picture come to life. - What it is: A solo performance on online self-display: the performer copies, mirrors and merges with projected webcam videos and internet footage of people documenting themselves, inspired by the mantra 'I post therefore I am'. - Technique: Live and prerecorded webcam imagery is processed in Quartz Composer and projected on and around the performer, so live body and screen body overlap. - Try it: Record ten seconds of a classmate's webcam selfie, project it at life size and have them perform a live duet with their own recording. Twist: slowly shift the projection off their body so the two selves drift apart. #### HYPER_ — Freya Björg Olafson (2013) - Video: https://vimeo.com/104413592 - Interaction: Projection, Perception & Effects, Performance - Platform & tech: Projection, stereoscopic 3D projection, UV light, Isadora - Idea: Make a live body and a stereoscopic image trade places. - What it is: The performer's body shifts between 2D and 3D in front of a large immersive screen: with UV light and 3D glasses for the audience, her figure seems to dissolve into projected colour and step out of it again. - Technique: Stereoscopic video programmed in Isadora is projected behind and onto the performer while UV light isolates parts of her costume, so live and projected bodies merge in depth. - Try it: Make red-cyan anaglyph video of a moving hand, project it on a white wall, and have a performer in white gloves align their hands with it while the audience wears paper 3D glasses. Twist: switch off the room light so only the gloves remain. #### CPA [Consistent Partial Attention] — Freya Björg Olafson (2016) - Video: https://vimeo.com/179255945 - Interaction: Gaze & Attention, Performance, Information & UI - Platform & tech: Projection, Headset, video score, heads-up displays, projection - Idea: A dance score delivered through screens, including one in front of the dancer's eye. - What it is: Three performers sight-read a choreographic score made of found internet videos of people dancing at home, playing on laptops, monitors, a projector and heads-up displays they wear, so the audience watches attention itself being split. - Technique: A digital video collage is distributed to multiple displays, including head-mounted displays, and the dancers follow it live without full rehearsal of the material. - Try it: Put a short dance video on a phone in a cardboard headset and have a classmate copy it live while the class watches both. Twist: play two different videos to two eyes or two phones at once. #### MÆ – Motion Aftereffect — Freya Björg Olafson (2019) - Video: https://vimeo.com/428830273 - Interaction: Hands & Body, Portals & Worlds, Performance - Platform & tech: Projection, Headset, motion capture, VR, projection - Idea: Stage the gap between the body in the headset and the body in the room. - What it is: A solo performance about how XR changes the body: guided by internet monologues about VR gameplay, out-of-body experiences and avatar customisation, the performer moves between her body, a motion-captured avatar and a projected VR scene. - Technique: The artist built a VR scene and motion-capture avatar that are projected live while she performs, combining ready-made 3D models with found audio monologues. - Try it: Have one student wear a phone VR headset in a simple WebXR scene while its view is cast to a projector, and a second student narrates what the first 'sees'. Twist: the narrator lies about the scene. #### Field of View — Freya Björg Olafson (2022) - Video: https://vimeo.com/721137520 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, webcam, frame trails - Idea: Your movement paints itself as a glitch trail. - What it is: A webcam feed of the moving artist freezes frame after frame on top of itself, so every gesture leaves a painted trail across the image, recalling old Windows screen-drag glitches. - Technique: In Unity the webcam texture is not cleared between frames, so new frames print on top of old ones and create persistent trails of movement. - Try it: In p5.js draw the webcam image without clearing the background, masked by a person-segmentation model, to create live body trails. Twist: reset the canvas whenever the performer claps. ### Geoffrey Alan Rhodes *Filmmaker and AR artist; founding member of Manifest.AR* Experimental filmmaker who turned augmented reality into a way of re-editing cinema: his 52 Card Cinema pieces, made at York University's AR Lab, map film shots onto playing cards. He co-signed the 2011 AR Art Manifesto, taught at the School of the Art Institute of Chicago and now teaches in Shanghai, where he builds AR museum and history tours. #### 52 Card Psycho — Geoffrey Alan Rhodes (2008) - Video: https://www.youtube.com/watch?v=Qgmh4e_w1oc - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Projection, Desktop, ARTag, Max/MSP - Idea: A deck of cards as a physical, playable film edit. - What it is: Fifty-two playing cards, each printed with a marker, are replaced in the viewer's view by the 52 shots of the shower scene in Hitchcock's Psycho; dealing, stacking or shuffling the cards re-edits the film. - Technique: ARTag fiducial markers on each card are tracked by a camera and each marker plays its assigned video clip in place, with Max/MSP handling playback, built at York University's AR Lab. - Try it: Cut a 20-second scene into 12 shots, assign each to a printed image marker, and let classmates arrange the cards to re-edit it in phone AR. Twist: card order also changes the soundtrack. #### Mao Wants This Dollar! — Geoffrey Alan Rhodes (2011) - Video: https://www.youtube.com/watch?v=ex-EHCLcm7I - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, Junaio, image tracking - Idea: Every banknote in your wallet becomes a political cartoon. - What it is: Point a phone at any US dollar bill and it becomes an IOU with Mao announcing that the United States of America owes China one dollar. - Technique: The Junaio AR browser's image recognition tracks the one-dollar bill design and overlays an animated IOU and portrait, made for Manifest.AR at the Boston Cyberarts Festival. - Try it: Image-track a common printed object (ticket, receipt, banknote) and overlay a short animation that rewrites its meaning. Twist: the overlay changes with today's exchange rate. #### twittAR Ideograms — Geoffrey Alan Rhodes (2012) - Video: https://www.youtube.com/watch?v=sA-wpEq3P2U - Interaction: Information & UI, Location & City - Platform & tech: Phone, Layar, Twitter API - Idea: Trending topics become cave paintings floating in the air. - What it is: At the ZERO1 Biennial, the Bay Area's changing Twitter trends are translated into floating 3D 'internet ideograms' that appear and disappear like a rebus puzzle. - Technique: A Layar channel polls local Twitter trends and maps keywords to 3D icon models that are placed as geolocated objects around the Biennial site. - Try it: Pull five trending words from any feed, map each to an emoji-like 3D model and float them over a campus plaza in AR. Twist: viewers have to guess the sentence the icons spell. #### An (AR) Allegory of the (VR) Cave — Geoffrey Alan Rhodes (2015) - Video: https://www.youtube.com/watch?v=iSLpDp-pkPQ - Interaction: Performance, Tangible Objects, Projection - Platform & tech: Projection, Phone, image tracking, projector - Idea: A lecture that stages its own argument in augmented reality. - What it is: A live lecture-performance at NYU in which the talk itself happens through AR, using Plato's cave to ask what screens, avatars and augmentation do to the self. - Technique: Image-tracked slides and props on stage trigger AR video and 3D content that the audience sees through a projected camera view. - Try it: Give a 5-minute talk where every key point is revealed by pointing a phone at a printed card, with the phone mirrored to a projector. Twist: one card reveals the audience itself. #### Chicago 00: The Eastland Disaster — Geoffrey Alan Rhodes (2016) - Video: https://www.youtube.com/watch?v=PKJLbi-vzIc - Interaction: Location & City, Information & UI, Portals & Worlds - Platform & tech: Phone, mobile AR app, GPS - Idea: Stand where history happened and see the archive photo aligned with the present. - What it is: Along the Chicago Riverwalk, archival photographs of the 1915 capsizing of the SS Eastland are superimposed on the exact spot where it happened, in a narrated AR tour with the Chicago History Museum. - Technique: Geolocated photo planes and 3D reconstructions from the museum's archive are anchored to riverwalk viewpoints in a mobile app, with a VR gallery for remote viewing. - Try it: Find an old photo of a campus location, stand where it was taken, and build a phone-AR overlay that aligns it with the live view. Twist: add a slider to crossfade between past and present. ### Hiroaki Umeda (S20) *Choreographer, dancer and visual artist* Japanese choreographer who founded the company S20 in 2000 and designs sound, light and projected images for his own solo dances. His works, from while going to a condition to Holistic Strata at YCAM, place a single body inside fields of projected lines and particles that seem to push against it. #### while going to a condition — Hiroaki Umeda (S20) (2002) - Video: https://www.youtube.com/watch?v=XQE444jISmk - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, projection, lighting, sound design - Idea: One body and one field of light is enough; AR designers can learn how little content is needed when timing and contrast are strong. - What it is: Hiroaki Umeda dances alone in front of a wall of projected light and noise, his sharp, trembling moves synchronised with flickering images and sound. - Technique: Umeda composes sound and projected visuals himself under the name S20 and times them to his choreography. - Try it: Build a Lens Studio or WebXR effect that fills the room behind a dancer with a flickering noise field that pulses with the soundtrack. Twist: the field freezes whenever the dancer holds still. #### Adapting for Distortion — Hiroaki Umeda (S20) (2008) - Video: https://www.youtube.com/watch?v=N1i3MA13v9I - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, projection, optical patterns, sound design - Idea: Projecting a pattern onto a body changes how we read its shape, a direct lesson for AR body effects and camouflage. - What it is: Projected moving stripes cover the floor and the dancer, creating optical vibration so the body seems to warp and dissolve into the pattern. - Technique: High-contrast line patterns are projected from above onto the stage and the performer, with image design by Bertrand Baudry and S20. - Try it: Use body segmentation in Lens Studio to texture a person with moving stripes that continue seamlessly into the background. Twist: the stripes bend around the body as if it were a lens. #### Holistic Strata — Hiroaki Umeda (S20) (2011) - Video: https://www.youtube.com/watch?v=2uQqxLpnueE - Interaction: Performance, Projection, Hands & Body - Platform & tech: Projection, projection, sensing, generative graphics - Idea: A particle field that behaves like a medium around the dancer shows how AR can make empty space feel thick and physical. - What it is: Commissioned by YCAM, the piece surrounds Umeda with dense layers of projected particles and lines on the floor and screen, which seem to flow around and through his body. - Technique: Sensing by YCAM engineers feeds programmed image systems by S20, Satoru Higa and Yoshito Onishi that render particle fields in real time. - Try it: Fill a room with an AR particle field in Unity AR Foundation that parts around a tracked body like water. Twist: particles remember where the body passed and slowly refill the gap. #### Intensional Particle — Hiroaki Umeda (S20) (2015) - Video: https://www.youtube.com/watch?v=jwYUo-yyXCg - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, projection, generative graphics, sound design - Idea: Treating the body as a force that bends a virtual field is an AR interaction model that needs no buttons at all. - What it is: A solo in which a huge projected particle landscape on the floor and back wall surges and reacts around Umeda's movement, as if the body were a force field. - Technique: Generative particle imagery programmed by Shoya Dozono is projected on floor and wall, likely driven in sync with the choreography and sound. - Try it: Create a WebXR floor of particles that are pushed outward by the phone's position like a magnet, then choreograph a one-minute solo with it. Twist: the push reverses into a pull on the chorus. #### Median — Hiroaki Umeda (S20) (2018) - Video: https://www.youtube.com/watch?v=cP9q7SBmjGA - Interaction: Performance, Projection, Shared & Social - Platform & tech: Projection, projection, generative graphics, choreography - Idea: A shared projected field turns several bodies into one system; multi-user AR can show relationships between people the same way. - What it is: A group piece in which several dancers move through projected fields of light and particles that connect and separate them across the stage. - Technique: Real-time generative imagery by Shoya Dozono and Gabor Papp is projected across the stage and synchronised with the choreography. - Try it: Build a shared AR session where lines of light connect every participant's phone and stretch or snap as people move apart. Twist: a line only reforms when both people face each other. ### Hirokazu Kato *Professor, NAIST; head of the Interactive Media Design Laboratory; creator of ARToolKit* Wrote ARToolKit with Mark Billinghurst in 1999, the marker tracker that seeded a generation of AR. His NAIST lab later built AR pop-up books, wearable projection surfaces and, with Christian Sandor, the HoloLens game HoloRoyale. #### AR Pop-up Book — Hirokazu Kato (2011) - Video: https://www.youtube.com/watch?v=D2PF-r_XOX0 - Interaction: Tangible Objects, Play - Platform & tech: Desktop, Phone, texture tracking, natural feature tracking - Idea: Any printed page can become a pop-up book when the picture itself is the tracker. - What it is: Opening an ordinary picture book in front of a camera makes 3D characters and scenes pop out of its pages, tracked from the printed illustrations themselves rather than black markers. - Technique: Uses natural-feature texture tracking on the page images to estimate camera pose and render 3D content registered to each page. - Try it: Pick a children's book page, set it as an image target in a phone AR tool, and make one character jump out when the page is opened. Twist: make characters from two pages talk to each other when both are visible. #### Interactive Surface with Projection-based Wearable System — Hirokazu Kato (2011) - Video: https://www.youtube.com/watch?v=RnupWPW1LeU - Interaction: Projection, Hands & Body - Platform & tech: Wearable, Projection, wearable projector, camera, touch detection - Idea: Carry your interface with you and project it onto whatever surface is nearby. - What it is: A wearable projector and camera turn nearby surfaces into touch interfaces: menus are projected onto a wall or table and the user taps them with a finger. - Technique: Likely calibrates a body-worn projector-camera pair so the camera detects fingertip contact on the projected image. - Try it: Mount a pico projector or a phone projector on a strap, project three buttons on a desk, and use a second phone's camera to detect which one a finger covers. Twist: project on your own forearm. #### Towards AR User Interfaces in 3D Media Production — Hirokazu Kato, Christian Sandor (2014) - Video: https://www.youtube.com/watch?v=7_1LXbbS43s - Interaction: Drawing & Making, Hands & Body - Platform & tech: Headset, stereo HMD, hand tracking, 3D modelling - Idea: Bring professional 3D modelling off the monitor and into the space above your desk. - What it is: A 3D artist models and animates with virtual objects floating over the real desk, using hand gestures in a headset instead of mouse and keyboard. - Technique: Likely combines a stereo HMD with 3D hand tracking to map familiar content-creation commands onto direct spatial manipulation. - Try it: Use a phone AR sculpting or drawing app to model a simple object above your desk, then time the same task in a desktop 3D tool. Twist: work in pairs where one person only moves the camera. #### HoloRoyale — Hirokazu Kato, Christian Sandor (2018) - Video: https://www.youtube.com/watch?v=8jn9clcNwf8 - Interaction: Play, Shared & Social, Spatial Mapping - Platform & tech: Headset, HoloLens, shared anchors, occlusion - Idea: A multiplayer AR game at building scale where virtual action respects real architecture. - What it is: Teams of players in HoloLens headsets battle across a large real space, with virtual robots and projectiles that hide behind, bounce off and collide with the actual walls and furniture. - Technique: Maps a large venue in advance and shares a common coordinate frame across HoloLens devices so virtual elements get occlusion and physics against the real geometry. - Try it: Design a shared phone AR capture-the-flag game for the school hall where flags hide behind real pillars, using a common anchor on the floor. Twist: add a rule that you can only shoot what the other team cannot see. ### INITI (Dan Gregor) *Interactive projection studio* Studio of visual artist Dan Gregor, a founding member of The Macula, making interactive projection works such as the laser-pointer instrument Archifon (with musician Floex), smartphone-controlled mappings and the INITI Playground projection games. #### Archifon — INITI (Dan Gregor) (2012) - Video: https://vimeo.com/37920250 - Interaction: Projection, Voice & Sound, Shared & Social - Platform & tech: Projection, laser-pointer tracking, Max/MSP, projection mapping - Idea: Play architecture like an instrument by pointing at it. - What it is: Up to ten visitors aim laser pointers at the columns, arches and statues of a deconsecrated Baroque chapel, and each architectural element they hit lights up with projection and plays a sound, turning the chapel into a giant instrument. - Technique: A camera tracks the laser dots on the mapped interior, and each surface of the 3D-mapped chapel is linked to visuals and Max/MSP sound triggers. - Try it: Point a webcam at a projected wall, detect a red laser dot with OpenCV or p5.js, and divide the wall into six zones that each play a note and flash. Twist: make zones need two lasers at once to sound a chord. #### Netykavka — INITI (Dan Gregor) (2013) - Video: https://vimeo.com/72222918 - Interaction: Projection, Hands & Body, Perception & Effects - Platform & tech: Projection, volumetric light, haze, hand detection - Idea: Make a beam of light feel like a solid object you can touch. - What it is: A cone of projected light in a hazy dark room, inspired by Anthony McCall's solid-light films, reacts when visitors put their hands into the beam. - Technique: Haze makes the projector's beam visible as a 3D volume, and a sensor (likely a depth camera) detects hands intersecting the beam to change the projected line drawing. - Try it: In a dark room with a fog machine or hair spray, project a slowly growing circle outline and use a webcam to detect a hand in the beam so the circle breaks where it is 'touched'. Twist: make two beams that can only be merged by two people. #### The Radar — INITI (Dan Gregor) (2013) - Video: https://vimeo.com/59839964 - Interaction: Shared & Social, Projection, Play - Platform & tech: Projection, Phone, smartphone web controllers, projection mapping - Idea: The crowd's phones become game controllers for a building-sized screen. - What it is: Passers-by connect their own smartphones to a local Wi-Fi network and use them as controllers to solve a projected puzzle on a building façade together. - Technique: A local web server hands each connected phone a control page and relays its inputs to the real-time mapping software, supporting several players at once. - Try it: Build a WebSocket page that lets up to five phones steer dots on a projected wall, and design a puzzle that needs all five to cooperate. Twist: give each phone a different, secret ability. #### Demonz (INITI Playground) — INITI (Dan Gregor) (2016) - Video: https://vimeo.com/166247151 - Interaction: Play, Projection, Tangible Objects - Platform & tech: Projection, motion tracking, projection mapping, interactive floor - Idea: Turn any flat surface into a giant touch screen for physical ball games. - What it is: Kids and adults throw real balls at animated monsters projected on a large floor or wall, playing a digital version of dodgeball in the real space. - Technique: Motion tracking detects where balls and bodies hit the surface and the projection-mapped game responds, so the play area can scale to any size and number of players. - Try it: Project a target game on a wall and detect soft-ball hits with a webcam by frame differencing; each hit pops a creature. Twist: make creatures dodge when players aim too long. ### Ines Alpha *3D digital makeup artist* French artist who creates 3D 'e-makeup' that grows from faces as AR filters and face-tracked videos, working with musicians, fashion brands and Prada Beauty. #### ines alpha x Katsu — Ines Alpha (2018) - Video: https://www.youtube.com/watch?v=7CY6GcYFAz8 - Interaction: Face, Perception & Effects - Platform & tech: Phone, 3D face tracking, Cinema 4D - Idea: A 3D digital makeup look tailored to an artist, growing along with the face as it moves. - What it is: A face-tracked 3D makeup look made with artist Katsu: iridescent sculptural forms grow from and cling to the face as it moves. - Technique: Sculptural meshes modeled in Cinema 4D are parented to face-tracking anchors, with iridescent shader effects that change with view angle. - Try it: Make a small sculptural object in any 3D software and attach it to a face-tracking anchor (such as the forehead or a cheekbone). Twist: the object unfolds or folds up with facial expressions. #### ines alpha x The Bulletproof Cat — Ines Alpha (2018) - Video: https://www.youtube.com/watch?v=kVcPbWZL5u0 - Interaction: Face, Perception & Effects - Platform & tech: Phone, 3D face tracking - Idea: Translucent organic shapes cling to the face and move with every expression. - What it is: Another face-tracked 3D makeup collaboration in which translucent, organic 3D shapes sit on and move with a performer's face. - Technique: Translucent organic meshes are bound to the face mesh and deform with blendshape-driven expressions, using refraction or transparency shaders. - Try it: Read expression coefficients from ARKit blendshapes and use them to drive the size or shape of a translucent form on the face. Twist: make the form glow only when the person smiles. #### Bimba y Lola x Ines Alpha filter — Ines Alpha (2019) - Video: https://www.youtube.com/watch?v=bXX0QVlD-Uc - Interaction: Face, Perception & Effects - Platform & tech: Phone, Spark AR, Instagram - Idea: Turn a fashion season's look into 3D digital makeup anyone can wear as a filter. - What it is: An Instagram face filter for the Bimba y Lola Fall/Winter 2019 collection that covers the face in Alpha's glossy sculptural 3D makeup. - Technique: Spark AR face tracking attaches glossy 3D makeup meshes to the face, with materials art-directed to match a fashion collection's palette. - Try it: Pull a color palette from a garment or poster and build a face AR filter that echoes it. Twist: the filter changes automatically to match the colors of clothing in the shot. #### Future Gloss — Ines Alpha (2019) - Video: https://www.youtube.com/watch?v=kDhMFguQFso - Interaction: Face, Perception & Effects - Platform & tech: Phone, Lens Studio, face tracking - Idea: Glossy liquid 3D makeup flows over your face in real time, free from physical limits. - What it is: Ines Alpha's first Snapchat lens: glossy, liquid 3D makeup that drips and flows over the user's face in real time. - Technique: A face mesh from face tracking drives 3D liquid geometry attached to facial landmarks, with glossy PBR materials and an environment map for reflections. - Try it: Use Lens Studio or Effect House to place a 3D liquid shape that follows the face mesh, and tune a glossy reflective material. Twist: make the liquid flow as the mouth opens and closes. #### Prada Beauty 'Rethinking Beauty' AR filter — Ines Alpha (2023) - Video: https://www.youtube.com/watch?v=GZ92cYFhddo - Interaction: Face, Perception & Effects - Platform & tech: Phone, Spark AR, Instagram - Idea: A digital makeup filter with three modes, designed for a luxury beauty brand. - What it is: As Prada Beauty's Global Creative E-Makeup Artist, Alpha designed a three-mode Instagram AR filter with FFFACE.ME that layers futuristic e-makeup on the user's face. - Technique: A multi-mode face filter switches between makeup looks via tap or picker UI, each combining face-mesh textures, attached 3D pieces and animated shaders. - Try it: Make a face filter with three modes that you switch between by tapping the screen, each mode expressing a different idea of 'future makeup'. Twist: trigger the mode switch with a blink. ### James Alliban *Media artist and AR developer* One of the first artists to make webcam AR widely shared online (2009's AR business card), later building Kinect installations and co-creating CELL with Keiichi Matsuda. #### Augmented Reality Business Card — James Alliban (2009) - Video: https://www.youtube.com/watch?v=5qoucBw60jM - Interaction: Tangible Objects, Voice & Sound - Platform & tech: Desktop, FLARToolKit, Papervision3D, Flash - Idea: Hold up a business card and a talking 3D version of its owner pops out. - What it is: Holding up his printed business card to a webcam makes a 3D video of the artist pop out of it and introduce himself. - Technique: FLARToolKit tracks a square fiducial marker printed on the card in a Flash webcam app, and a video plane of the artist is rendered at the marker's pose with Papervision3D. - Try it: Design your own business card and use MindAR to make a keyed self-introduction video pop out of it. Twist: have the video of you react to how the viewer holds the card (tilt it and you 'fall over'). #### Augmented Reality Ribbons — James Alliban (2009) - Video: https://www.youtube.com/watch?v=0K6YcVHhG1k - Interaction: Tangible Objects, Voice & Sound, Perception & Effects - Platform & tech: Desktop, FLARToolKit, Flash - Idea: AR ribbons pour out of a marker and sway to music. - What it is: Marker-tracked AR where flowing 3D ribbons spill out of a printed marker and follow its movement, reacting to music. - Technique: A tracked fiducial marker emits procedurally generated ribbon geometry whose points trail the marker's recent positions, with amplitude driven by audio analysis. - Try it: Use MindAR or AR.js to make an image marker trail a 3D ribbon that records its path, with microphone volume controlling the ribbon's width. Twist: replace the ribbon with a 'text band' made of words from your recent chats. #### Wallpaper* Augmented Reality issue — James Alliban (2009) - Video: https://www.youtube.com/watch?v=hzSHcdwFuXs - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Desktop, FLARToolKit, Flash - Idea: Make the pages of a design magazine grow 3D artworks in front of a webcam. - What it is: A set of webcam AR pieces for Wallpaper* magazine's AR issue, bringing printed pages to life with animated 3D artworks. - Technique: Printed magazine pages act as fiducial or natural-feature markers for a Flash webcam app that renders animated 3D pieces on each page. - Try it: Make one AR page for a magazine you like, using image tracking to grow one element of the layout into a 3D piece. Twist: a 'spread' piece that only appears when two pages sit side by side. #### Traces — James Alliban (2011) - Video: https://www.youtube.com/watch?v=0KvCqF-dS2U - Interaction: Hands & Body, Drawing & Making - Platform & tech: Projection, Kinect, openFrameworks - Idea: A Kinect turns visitors' movements into trails of light. - What it is: An interactive artwork using a Kinect camera that reflects visitors as trails of light, turning movement into drawings. - Technique: A Kinect depth image is thresholded into a body silhouette, and its contour or motion is accumulated over time in openFrameworks to draw fading light trails. - Try it: Use Kinect or camera body segmentation in TouchDesigner to turn people's outlines into slowly fading light trails projected on a wall. Twist: make the trails record only the 'gap' between two people rather than their bodies. ### James Turrell *Artist; Light and Space* American artist who has spent five decades shaping light and perception: Skyspaces that frame the sky through an opening in a ceiling, Ganzfeld rooms without visible edges, and Roden Crater, a naked-eye observatory built into an extinct volcano. #### Roden Crater — James Turrell (1979) - Video: https://www.youtube.com/watch?v=g0g6JFYRKxQ - Interaction: Location & City, Perception & Effects, Gaze & Attention - Platform & tech: Projection, volcanic crater, tunnels, naked-eye observatory - Idea: Architecture can act as a time-based lens on the sky; an AR experience can likewise be tied to a place and a moment, opening only when sun or moon line up. - What it is: An extinct volcano in the Arizona desert is being reshaped into a naked-eye observatory: tunnels and chambers frame the sky, the sun and the moon at precise times. - Technique: Openings, tunnels and bowl-shaped rooms are aligned to astronomical events (solstices, lunar standstills) so that sky light is framed and flattened into a vault. - Try it: Write a phone AR app that uses GPS, compass and a sun-position library to show a floating ring that only aligns with the real sun at one minute of the day at your school. Twist: make the ring open a portal to the night sky of the same place. #### The Wolfsburg Project — James Turrell (2009) - Video: https://www.youtube.com/watch?v=QWekIcZaKns - Interaction: Perception & Effects, Spatial Mapping, Gaze & Attention - Platform & tech: Projection, Ganzfeld, LED and fluorescent lighting, curved walls - Idea: Remove every edge and texture and the brain loses depth; AR designers can use uniform fields of colour to dissolve a room, or add a single object to re-anchor it. - What it is: In a huge white room at Kunstmuseum Wolfsburg, visitors walk into a field of uniform coloured light (a Ganzfeld) where walls, floor and distance disappear. - Technique: Rounded corners and a sloping floor hide all edges, and programmable colour light fills the space evenly so the eye cannot focus. - Try it: Using a HoloKit or phone headset, overlay a full-view colour fog that slowly shifts hue and fades out the passthrough image, then bring the room back gradually. Twist: leave one real object visible and ask students where they think the walls are. #### Within without — James Turrell (2010) - Video: https://www.youtube.com/watch?v=biScm3kGMdY - Interaction: Perception & Effects, Gaze & Attention, Location & City - Platform & tech: Desktop, skyspace, stupa, water moat - Idea: A long approach prepares the eyes: before showing AR content, a staged transition (darkness, water, a tunnel) makes the reveal much stronger. - What it is: At the National Gallery of Australia, a ramp leads under a pool into a basalt stupa inside a pyramid; lying in the dome, visitors see the sky through a circular opening. - Technique: Visitors walk down through water and into a dark chamber so their eyes adapt, then look up at a sharp-edged circle of sky in a domed ceiling. - Try it: Design a three-stage WebXR sequence: a dimming tunnel, a floating circular window, and finally a live sky camera texture inside that circle. Twist: keep the circle closed until the user has stayed still and looked up for ten seconds. #### Twilight Epiphany — James Turrell (2012) - Video: https://www.youtube.com/watch?v=GrD5ylEn1wQ - Interaction: Perception & Effects, Gaze & Attention, Location & City - Platform & tech: Projection, skyspace, LED lighting, roof aperture - Idea: Colour next to a window changes the colour we see through it; an AR layer placed around a real view can re-tint that view without touching it. - What it is: A grass-covered pavilion at Rice University has a square opening in its roof; at sunrise and sunset, LEDs wash the ceiling with changing colours so the sky in the opening seems to turn solid and change colour itself. - Technique: A knife-edge aperture removes any visible roof thickness, and programmed LED sequences exploit simultaneous contrast to shift the perceived colour of the sky. - Try it: Build a phone AR frame that surrounds a real window or patch of sky with a slowly changing coloured border and let students compare how the sky's colour appears to shift. Twist: drive the border colour from the actual time until sunset. #### Aten Reign — James Turrell (2013) - Video: https://www.youtube.com/watch?v=Wih1K71hYlk - Interaction: Perception & Effects, Gaze & Attention, Spatial Mapping - Platform & tech: Projection, LED lighting, fabric scrims, daylight - Idea: Layered, softly lit rings erase edges and make a building feel bottomless; AR can use nested translucent shells to change how deep a real ceiling looks. - What it is: Visitors lie on benches under the Guggenheim rotunda, which is filled with nested elliptical rings of fabric lit by slowly shifting coloured light mixed with daylight from the skylight. - Technique: Five concentric fabric cones hang in the rotunda; computer-controlled LEDs light each ring separately while natural light enters from the oculus. - Try it: Anchor five nested, glowing ellipses to the classroom ceiling in AR Foundation and cross-fade their colours over 60 seconds so that the ceiling seems to recede. Twist: sample the real room's light with the camera and blend it into the outermost ring. ### Janet Echelman *Artist; aerial net sculptures* American artist who suspends vast knotted-fibre nets between buildings; the soft forms move with the wind and are lit at night, sometimes interactively, as in Skies Painted with Unnumbered Sparks for TED 2014. #### Her Secret is Patience — Janet Echelman (2009) - Video: https://www.youtube.com/watch?v=0rwy6IS0cHo - Interaction: Location & City, Spatial Mapping, Perception & Effects - Platform & tech: Desktop, fibre netting, steel poles, lighting - Idea: A soft, light form hung over a city makes the invisible wind visible; AR sky objects can reveal real forces by moving with them. - What it is: Above a downtown park in Phoenix, a huge net sculpture shaped like a cactus bloom hangs from steel poles, rippling in the desert wind by day and glowing in shifting colours by night. - Technique: Layers of knotted high-tenacity fibre netting are suspended from three steel poles, with coloured lighting projected onto the net at night. - Try it: Hang a virtual net mesh between three real points you tap (lamp posts, trees) and simulate it as cloth moved by the phone's measured wind or by a slider. Twist: at night mode, colour the net by the noise level of the street below. #### Skies Painted with Unnumbered Sparks — Janet Echelman (2014) - Video: https://www.youtube.com/watch?v=uNQ0f_Vg3T0 - Interaction: Shared & Social, Drawing & Making, Projection - Platform & tech: Projection, fibre netting, projection mapping, web app - Idea: Phones in the crowd can paint on a shared canvas in the sky; the same model works for AR, where many small inputs add up to one large shared image. - What it is: A 745-foot net sculpture floats between buildings in Vancouver; at night visitors choreograph the projected light on it in real time with gestures on their phones. - Technique: Projectors map light onto the suspended net while a web app, made with Aaron Koblin, turns visitors' touch gestures into light trails in real time. - Try it: Build a shared WebXR sky canvas anchored above a courtyard where each phone's swipe leaves a coloured light trail that everyone else sees. Twist: trails fade faster when too many people draw at once, forcing the crowd to take turns. #### As If It Were Already Here — Janet Echelman (2015) - Video: https://www.youtube.com/watch?v=avciVzTLb9Y - Interaction: Location & City, Information & UI, Perception & Effects - Platform & tech: Desktop, fibre netting, tension cables, lighting - Idea: A sky object can hold the memory of what used to be on the ground below; AR can place a trace of a vanished street right above where it was. - What it is: Above the Rose Kennedy Greenway in Boston, a 600-foot net sculpture hangs between buildings, its bands of colour recalling the city's old traffic lanes and changing light at night. - Technique: A lightweight fibre net is tensioned by ropes to three surrounding buildings, and coloured lighting is programmed to change through the night. - Try it: Find an old map of your campus or street, and hang an AR net above today's site whose colour bands follow the old roads underneath. Twist: when the viewer looks straight up, the old map fades in on the ground. #### Earthtime 1.8 London — Janet Echelman (2016) - Video: https://www.youtube.com/watch?v=FrM1KiS6JPo - Interaction: Information & UI, Location & City, Perception & Effects - Platform & tech: Desktop, fibre netting, data modelling, lighting - Idea: Scientific data can become a form that floats over daily life; AR can lift abstract datasets into the air above the places people pass every day. - What it is: Over Oxford Circus in London, a net sculpture whose form comes from data of the 2011 Japan earthquake hangs above the crossing and glows in shifting colours at night. - Technique: The net's shape is derived from a model of the tsunami's wave propagation, knitted from fibre and suspended from surrounding buildings, with programmed lighting. - Try it: Turn a real dataset (earthquake waves, rainfall, commuting) into a 3D surface and hang it in AR above a busy crossing so passers-by see it overhead. Twist: update the surface live with today's value of that dataset. #### Earthtime 1.26 Munich — Janet Echelman (2021) - Video: https://www.youtube.com/watch?v=CYbeC1txB1g - Interaction: Information & UI, Location & City - Platform & tech: Desktop, fibre netting, lighting - Idea: A title that names an invisible change (a shorter day) gives a floating form a reason to exist; AR works benefit from linking a visible shape to an unseen fact. - What it is: In Munich, a large Earthtime net sculpture floats above a public space, its form linked to the 2010 Chile earthquake that shortened the Earth's day by 1.26 microseconds. - Technique: Fibre netting modelled on earthquake data is rigged between supports and lit with programmed colour sequences that change through the evening. - Try it: Pick an invisible number about your city (a daily CO2 value, noise level) and make an AR form above a square whose size or tension follows it, with a caption that states the fact. Twist: show last year's form beside it for comparison. ### Jean Tinguely *Artist; maker of absurd kinetic machines* Swiss artist (1925–1991) who built noisy, self-destroying and drawing machines from scrap, from the Méta-Matic drawing robots (1959) and Homage to New York (1960) to the forest giant Le Cyclop. #### Méta-Matic drawing machines — Jean Tinguely (1959) - Video: https://www.youtube.com/watch?v=VxoqVvQeil0 - Interaction: Drawing & Making, Tangible Objects - Platform & tech: Desktop, motors, scrap metal, paper, pens - Idea: A machine that draws turns an invisible process into a visible trace; AR drawing tools can make the maker, human or machine, part of the show. - What it is: Motorised scrap-metal machines that hold a pen and scribble abstract drawings on paper for visitors, parodying automatic abstract painting. - Technique: An eccentric arm driven by a motor moves a pen in unpredictable loops over a clamped sheet; visitors fed a coin and chose the pen colour. - Try it: Make an AR drawing machine that sits on a real sheet of paper and draws random linked loops on it, with the result savable as an image. Twist: let the user's heartbeat or voice control the arm speed. #### Homage to New York — Jean Tinguely (1960) - Video: https://www.youtube.com/watch?v=6dgGu2w3Qvo - Interaction: Performance, Perception & Effects - Platform & tech: Desktop, scrap machinery, motors, fire, piano - Idea: A work designed to exist only as a one-time event shows that ephemerality and failure can be the experience; AR can also stage content that happens once and is gone. - What it is: A 27-foot machine of bicycle wheels, a piano, motors and fireworks built in MoMA's sculpture garden that played, painted, smoked and destroyed itself in about 30 minutes before an audience. - Technique: Dozens of salvaged motors drove independent sub-machines with timers and triggers, some rigged to burn or break, so the sequence unfolded with deliberate chaos. - Try it: Build an AR Rube Goldberg machine on a real table that runs once when triggered, spilling, burning and collapsing with physics, and then cannot be restarted. Twist: let viewers' phones each trigger one part of the chain. #### Heureka — Jean Tinguely (1964) - Video: https://www.youtube.com/watch?v=DpbegMX2m6o - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, steel, scrap parts, motors - Idea: A machine that performs on a public schedule gathers people at a place and time; AR works in public space can use timed appearances to create shared moments. - What it is: An 8 m machine sculpture of wheels, levers and scrap parts, built for the Swiss National Exhibition and now standing by Lake Zurich, which rattles into useless motion at set times each day. - Technique: Electric motors drive dozens of interlinked wheels and arms that move without producing anything, celebrating movement for its own sake. - Try it: Anchor a large AR machine sculpture at a real outdoor spot (ARCore Geospatial) that only moves at the top of every hour and stays still otherwise. Twist: add a counter showing how many people watched it move today. #### Méta-Harmonie — Jean Tinguely (1978) - Video: https://www.youtube.com/watch?v=-Xp2jjSaRVg - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, motors, wheels, found instruments - Idea: Letting visitors see every moving part that makes a sound turns noise into music; AR sound objects are more engaging when their mechanism is visible. - What it is: Giant sound machines built from wheels, drums, pipes and found instruments that clang, whistle and bang in chaotic rhythms when switched on. - Technique: Motor-driven wheels carry beaters and strikers past percussion and wind elements, so the rotation speeds and gear ratios compose a slowly shifting polyrhythm. - Try it: Build an AR music machine on a table where each visible rotating wheel triggers a sample when its striker passes a virtual drum; users tap to add wheels at new speeds. Twist: tie one wheel to the phone's walking cadence. #### Le Cyclop — Jean Tinguely (1994) - Video: https://www.youtube.com/watch?v=-lELc3auHgM - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, steel, mirror mosaic, scrap machinery - Idea: A monumental object hidden in a forest that you enter becomes a destination; AR can hide large walk-in worlds at specific sites that reward the journey. - What it is: A 22.5 m walk-in head built in the forest of Milly-la-Forêt by Tinguely, Niki de Saint Phalle and friends over 25 years, with a mirrored face, moving machines and a ball run inside. - Technique: A steel structure clad in mirror mosaic houses dozens of scrap-metal machines, a rolling-ball circuit and artworks by other artists; it was built by hand from 1969 to 1994. - Try it: Place a giant walk-in AR head in a real park using geospatial anchors, with a portal mouth leading into a small kinetic interior. Twist: let each visitor add one moving part that stays for the next visitor. ### Jenny Holzer *Conceptual artist (text in public space, light projections, LED)* American artist who since the late 1970s has put language into public space, first as posters (Truisms), then as LED signs and, from the 1990s, as monumental xenon projections of poems and declassified documents on buildings, rivers and landscapes. She represented the US at the 1990 Venice Biennale. #### For the Guggenheim — Jenny Holzer (2008) - Video: https://www.youtube.com/watch?v=neLvdTd1tWs - Interaction: Information & UI, Perception & Effects - Platform & tech: Projection, LED sign, spiral ramp, text - Idea: Text that follows the architecture's path turns reading into walking; AR captions and messages can be laid along a route instead of floating in front of the face. - What it is: In her 2008 exhibition at the Solomon R. Guggenheim Museum in New York, a long LED text band runs up the museum's spiral ramp, so words race around the rotunda as visitors walk. Filmed by VernissageTV (museum-show documentation, not the artist's upload). - Technique: A custom continuous LED sign is installed along the parapet of the Guggenheim ramp and programmed with scrolling texts. - Try it: Lay an AR text ribbon along a real staircase or corridor with Lens Studio or 8th Wall so it scrolls as you walk up. Twist: the text only advances while the viewer is moving. #### Projection for Chicago — Jenny Holzer (2008) - Video: https://www.youtube.com/watch?v=QeOOFKzQxS0 - Interaction: Location & City, Information & UI - Platform & tech: Projection, xenon projection, text - Idea: A poem projected slowly on a building makes a whole street read together; location-based AR text can create the same shared, public reading. - What it is: As part of her PROTECT PROTECT exhibition at the MCA Chicago, Holzer projects the poems of Wisława Szymborska in huge scrolling letters onto building facades around the city, including the Lyric Opera House and Riverside Plaza. Documented in Art21's Extended Play series (not the artist's upload). - Technique: High-powered xenon projectors throw slowly scrolling text onto facades, following the xenon projection method Holzer has used since the 1990s. - Try it: Choose a short poem and place it as slow-scrolling AR text on a real building facade with geo or image anchors in 8th Wall. Twist: each viewer can add one line that everyone else sees. #### Jenny Holzer at Blenheim Palace — Jenny Holzer (2017) - Video: https://www.youtube.com/watch?v=pKIQNbuIqpE - Interaction: Location & City, Information & UI - Platform & tech: Projection, light projection, LED, text - Idea: Placing personal testimony on a monument of power creates a strong contrast; AR can layer many small human voices over an official place. - What it is: For the Blenheim Art Foundation, Holzer's texts about power, war and its aftermath — including words of veterans gathered with the Not Forgotten Association — appear as giant walls of projected light on Blenheim Palace, and in stone and LED works inside. Documented by NOWNESS. - Technique: Xenon or large-format projectors cast scrolling texts across the palace facade at night, complemented by LED signs indoors. - Try it: Collect short spoken statements from classmates about a place, transcribe them, and pin them as AR text to that place's facade with Lens Studio. Twist: the texts are only readable when two people stand together in front of it. #### Louise Bourgeois x Jenny Holzer: Projections, Basel — Jenny Holzer (2022) - Video: https://www.youtube.com/watch?v=6v_nPUaa_OY - Interaction: Location & City, Information & UI - Platform & tech: Projection, xenon projection, text - Idea: Another artist's private words become public architecture; AR can carry a museum's content out into the city around it. - What it is: Alongside the exhibition 'Louise Bourgeois x Jenny Holzer' at the Kunstmuseum Basel, Holzer projects excerpts of Bourgeois's writings onto three public buildings in Basel at night. Documented by VernissageTV. - Technique: Xenon projectors cast slowly moving lines of text onto facades and a river bank, the same technique Holzer has refined since the 1990s. - Try it: Take quotes from a book in the campus library and place them as AR text on buildings along a walking route with 8th Wall geo anchors. Twist: the route leads back to the shelf where the book stands. #### Drone text performance, Museum of Islamic Art, Doha — Jenny Holzer (2026) - Video: https://www.youtube.com/watch?v=MwJdMC7g7Ug - Interaction: Location & City, Information & UI - Platform & tech: Projection, drone swarm, text, light - Idea: Words written in the sky are read by an entire crowd at once; AR text in the sky can do the same, but for every phone pointing up. - What it is: On the eve of the first Art Basel Qatar, the lights around I. M. Pei's Museum of Islamic Art in Doha go out and a surprise work by Holzer begins: a monumental text piece and drone performance writing words in the night sky. Documented by Art Basel (press upload). - Technique: A synchronised swarm of lit drones forms letters and phrases in the sky, likely combined with projected text on the museum. - Try it: Write a short sentence in the sky above campus with WebXR or Lens Studio sky segmentation, letter by letter like drones arriving. Twist: the sentence rearranges into a new one when viewers clap. ### Jeri Ellsworth *Hardware engineer; co-founder of castAR and Tilt Five* Self-taught chip designer who built AR glasses at Valve, then spun them out as castAR (2013) and later Tilt Five, a retro-reflective tabletop AR game system. #### castAR Kickstarter — Jeri Ellsworth (2013) - Video: https://www.youtube.com/watch?v=AOI5UW9khoQ - Interaction: Projection, Tangible Objects, Play - Platform & tech: Wearable, castAR, retro-reflective projection, head tracking - Idea: Tiny projectors in the glasses bounce off a reflective board so everyone sees 3D holographic games on the table. - What it is: Glasses with tiny projectors that bounce light off a retro-reflective mat, so each wearer sees their own correct-perspective 3D holograms, such as board games, on the table. - Technique: Head-mounted projectors cast a stereo image onto a retro-reflective mat that returns light mainly to its source, so each wearer sees only their own head-tracked perspective. - Try it: Use front-camera face tracking on a phone to create head-coupled perspective, turning a tablet lying on the table into a "holographic tabletop" with depth. Twist: have two people look at the same tablet from different angles and discuss why only one sees the correct perspective. #### castAR at Maker Faire 2013 — Jeri Ellsworth (2013) - Video: https://www.youtube.com/watch?v=3jxr3UrY2mc - Interaction: Projection, Tangible Objects - Platform & tech: Wearable, castAR - Idea: A demo of the AR glasses and wand prototype that came out of Valve. - What it is: Jeri Ellsworth demonstrates the castAR glasses and wand, explaining how the technology she started at Valve tracks the head and projects stereo images onto the mat. - Technique: Head tracking relative to the mat, likely via camera-tracked markers, updates the off-axis stereo projection so holograms stay locked to the table. - Try it: Run a small experiment with cheap materials (reflective tape and a flashlight) to see how retroreflective material sends light back to its source. Twist: based on this, sketch a tabletop AR device of your own design. #### castAR SmartGlasses at AWE 2015 — Jeri Ellsworth (2015) - Video: https://www.youtube.com/watch?v=8We6Om4WaAM - Interaction: Projection, Shared & Social, Play - Platform & tech: Wearable, castAR - Idea: Several players sit around a reflective table and play a holographic board game together. - What it is: A later castAR prototype shown at AWE 2015 with tabletop games that several players see at once around the retro-reflective surface. - Technique: Because each wearer's projector image returns only to their own eyes, multiple players can share one surface with individually correct perspective views. - Try it: Design a multiplayer tabletop game where each player sees different information (for example, only you can see your own hand of cards), built with several phones running AR around the same image marker. Twist: add one public card that everyone can see. #### Tilt Five Holographic Game System — Jeri Ellsworth (2019) - Video: https://www.youtube.com/watch?v=x_71C3bMNUA - Interaction: Shared & Social, Tangible Objects, Play - Platform & tech: Wearable, Tilt Five, retro-reflective projection - Idea: A reflective game board and a pair of glasses let friends share holographic board games around a table. - What it is: The Tilt Five Kickstarter: lightweight glasses and a retro-reflective game board plus a wand controller, so friends around a table share holographic board and video games. - Technique: Glasses project stereo images onto a retro-reflective board and track head pose against it, while a tracked wand gives a ray for pointing at tabletop content. - Try it: Use an image marker as a game board in AR and select and move pieces with a ray cast from the center of the phone screen. Twist: pieces can 'stand up' taller than the board. #### Tilt Five — Gaming with Tabletop Holograms — Jeri Ellsworth (2022) - Video: https://www.youtube.com/watch?v=Fzr-IeyWj40 - Interaction: Tangible Objects, Play, Shared & Social - Platform & tech: Wearable, Tilt Five, Unity - Idea: Games rise up from the tabletop board and you move holographic pieces directly with a wand. - What it is: Shipping Tilt Five games shown through the glasses: tabletop strategy, board games and toys that rise from the board and respond to the wand. - Technique: Content is authored in board-local coordinates in Unity, so miniatures appear to rise from the board and stay registered as players move their heads. - Try it: Make a small city that grows out of the tabletop, with buildings revealing different details depending on the angle as you walk around the table. Twist: the little people in the city look toward wherever you are. ### Karolina Sobecka *Artist and researcher (gravitytrap, Flightphase)* Polish-born artist who projects responsive animals and figures into public space — from shop windows to buildings passing by a moving car — and later turned to cloud and atmosphere research. #### Wildlife — Karolina Sobecka (2006) - Video: https://vimeo.com/6400445 - Interaction: Location & City, Projection, Tangible Objects - Platform & tech: Projection, wheel sensor, mobile projection - Idea: Project a tiger from a moving car onto roadside buildings so it runs alongside the car. - What it is: A tiger is projected from a moving car onto the buildings it passes; the tiger runs alongside, speeding up and slowing down with the car and stopping when it stops. - Technique: A sensor on the wheel measures rotation and sets the playback frame rate of the tiger's run cycle, which a car-mounted projector throws onto passing façades. - Try it: Put a projector on a cart or bicycle, use an encoder or phone accelerometer to control the frame rate of a running animation, and project it along a corridor. Twist: the animal 'collapses from exhaustion' when you go too fast. #### Chase — Karolina Sobecka (2009) - Video: https://vimeo.com/6400072 - Interaction: Location & City, Projection, Play - Platform & tech: Projection, mobile projection, motion sensing - Idea: Projected characters chase a moving car along the street. - What it is: An interactive mobile projection from a moving car in which projected characters chase along the street as the vehicle travels. - Technique: Likely the same car-mounted projection rig as Wildlife, with animation playback driven by vehicle motion data. - Try it: Project from a phone projector or a small projector while walking, keeping the character's speed in sync with your walking pace. Twist: let viewers use their phones to make the character jump over obstacles on the wall. #### Sniff (with James George) — Karolina Sobecka (2009) - Video: https://vimeo.com/6400266 - Interaction: Hands & Body, Projection, Location & City - Platform & tech: Projection, openFrameworks, Unity, Blender, camera tracking - Idea: A virtual dog in a shop window follows passers-by and judges whether you are friendly or mean to it. - What it is: A projected dog in a street-facing shop window follows passers-by, reads their gestures as friendly or aggressive and tries to get them to play. - Technique: A camera tracks pedestrians outside the window with openFrameworks and feeds position and gesture speed to a Unity-animated dog with simple behaviour states. - Try it: Use a camera to detect a person's position and waving speed, and drive an on-screen virtual pet to turn its head, approach or shy away. Twist: give the pet 'memory' so it reacts differently when the same person comes back. #### All the Universe is Full of the Lives of Perfect Creatures — Karolina Sobecka (2012) - Video: https://vimeo.com/35262930 - Interaction: Face, Perception & Effects - Platform & tech: Desktop, FaceTracker, ofxFaceTracker, half-silvered mirror - Idea: Your face turns into an animal in the mirror and copies your expressions. - What it is: Looking into a mirror, the viewer sees an animal's head overlaid exactly on their own reflection, mimicking their head movement and facial expressions. - Technique: A camera behind a half-silvered mirror runs FaceTracker (ofxFaceTracker) and drives a 3D animal head rendered on a display so it aligns with the viewer's reflection. - Try it: Drive an animal head with MediaPipe Face Mesh, overlaid on the mirrored video so it blinks and opens its mouth in sync. Twist: build a real "magic mirror" with a half-silvered mirror and a monitor, and have the animal sometimes disobey. #### It's You — Karolina Sobecka (2012) - Video: https://vimeo.com/35266165 - Interaction: Hands & Body, Projection, Location & City - Platform & tech: Projection, camera presence detection, public projection - Idea: A projected crowd hides a secret, and only steps aside when you lean in. - What it is: Projected figures crowd around something hidden; when a pedestrian stands behind them as if to look over their shoulders, they step aside to reveal part of the scene, blocking it for others. - Technique: A camera detects a pedestrian's position in the interaction zone and triggers animation branches of the projected crowd that open a view at that spot. - Try it: Project a crowd of silhouettes with their backs to the audience, and use a camera to detect who steps behind them so the figures at that spot move aside and reveal a hidden image. Twist: the hidden content is a sentence left by the previous visitor. ### Ken Perlin — NYU Future Reality Lab *Professor of Computer Science at NYU; director of the Future Reality Lab; inventor of Perlin noise* Oscar-winning graphics pioneer Ken Perlin now builds shared, co-located mixed reality: Chalktalk, a sketch language whose drawings come alive as simulations, and Holojam, untethered multi-user VR/AR in a motion-captured room. #### Holojam — Ken Perlin — NYU Future Reality Lab (2015) - Video: https://www.youtube.com/watch?v=kzx5igORwk4 - Interaction: Shared & Social, Drawing & Making, Hands & Body - Platform & tech: Headset, Gear VR, OptiTrack - Idea: A group in wireless headsets becomes cartoon avatars in one room and draws in the air together. - What it is: Untethered mobile headsets and optical motion capture let a group walk around one room as avatars, drawing 3D ribbons in the air together. - Technique: OptiTrack cameras track reflective markers on each untethered mobile headset and on the hands, and positions are streamed over Wi-Fi so every headset renders all users' avatars and 3D ribbons in the shared room. - Try it: Use a shared coordinate system in HoloKit or phone AR (an image target as the common origin) so several people draw 3D ribbons in the same space with their phone positions and see each other's avatars. Twist: ribbons only appear when two people cooperate by pressing at the same time. #### Flock: A Holojam Experience — Ken Perlin — NYU Future Reality Lab (2016) - Video: https://www.youtube.com/watch?v=BZXRX-kDj5M - Interaction: Shared & Social, Perception & Effects - Platform & tech: Headset, Holojam, motion capture - Idea: A group of people become a flock of birds together in a shared space. - What it is: David Lobser's co-located Holojam piece where participants become birds and create a flock together in shared space. - Technique: Each participant's tracked position drives a bird avatar, and boids-style flocking rules (separation, alignment, cohesion) generate additional birds that respond to the participants' movements. - Try it: Implement a boids flock in three.js, use people's positions detected by a camera as "lead birds," and project it on a wall or show it in AR. Twist: make the flock be drawn to or flee from each person depending on how fast they move. #### Chalktalk AR + Holojam — Ken Perlin — NYU Future Reality Lab (2017) - Video: https://vimeo.com/224598543 - Interaction: Drawing & Making, Shared & Social - Platform & tech: Headset, Chalktalk, Holojam, motion capture - Idea: Several people in one room draw living computational graphics in the air together. - What it is: Several co-located people wearing tracked headsets draw Chalktalk objects together and see each other's living sketches in the same space. - Technique: Motion capture tracks all headsets in one room coordinate frame, and a networked server synchronizes Chalktalk sketch state so every co-located user sees the same living drawings in place. - Try it: Use AR Foundation shared anchors or WebXR with WebSocket so two phones in the same space can draw 3D lines at once and sync them in real time. Twist: one person's lines can only be "activated" by the other person, turning into moving objects when tapped. #### Chalktalk in Augmented Reality — Ken Perlin — NYU Future Reality Lab (2017) - Video: https://vimeo.com/232230096 - Interaction: Drawing & Making, Information & UI - Platform & tech: Headset, Chalktalk, JavaScript, AR headset - Idea: Draw a few strokes in the air and the sketch is recognized as a living, simulated 3D object. - What it is: Perlin draws simple sketches in the air that are recognized and come alive as 3D, simulating objects (a hypercube, a pendulum, a creature) floating in shared AR. - Technique: Freehand strokes are matched against a library of glyph templates by shape recognition, and each recognized sketch is replaced by a procedural, animated 3D object rendered in a shared AR space. - Try it: Recognize three simple hand-drawn symbols (circle, triangle, wave) on a web canvas, for example with the $1 Unistroke Recognizer, and replace them with moving 3D objects in a WebXR scene. Twist: make two recognized objects interact when they come close, such as a sun making a flower bloom. #### NYU Holodeck — Ken Perlin — NYU Future Reality Lab (2018) - Video: https://www.youtube.com/watch?v=3XBt9PJG66I - Interaction: Shared & Social, Information & UI - Platform & tech: Headset, motion capture, Chalktalk - Idea: Turn shared mixed reality into a 'holodeck' where people take classes and collaborate together. - What it is: The NSF-funded Holodeck project combines motion capture, headsets and Chalktalk into shared mixed reality for teaching and collaboration. - Technique: Room-scale optical motion capture, networked headsets and a shared scene-graph server let multiple users and Chalktalk content occupy one synchronized mixed-reality space. - Try it: Design and test a 10-minute 'shared AR mini-lesson': use a shared anchor so every phone in the class sees the same 3D model (such as a heart or a molecule), and when the speaker taps it, everyone sees the same highlight. Twist: let students raise a hand (by placing a marker in AR) to ask the speaker a question. ### Kimchi and Chips *Art studio of Mimi Son and Elliot Woods* Korean-British duo working with light fields, projection and robotic mirrors to make images that occupy real space. #### Lit Tree — Kimchi and Chips (2011) - Video: https://vimeo.com/24049819 - Interaction: Projection, Hands & Body, Spatial Mapping - Platform & tech: Projection, openFrameworks, Kinect, structured light - Idea: Use the leaves of a real tree as 3D pixels, so a visitor's hand gesture lights up inside the tree. - What it is: A small potted tree is scanned and projected onto so its leaves act as 3D pixels; visitors' hands are scanned over a plinth and reflected as volumes of light inside the tree. - Technique: Structured-light scanning with a projector and Kinect maps every projector pixel to a 3D point on the leaves, turning the tree into a sparse volumetric display that lights voxels inside a scanned hand volume. - Try it: Aim a projector at a potted plant, scan line by line in TouchDesigner to record which pixels land on leaves, then light only the leaves to draw a plane of light moving up and down. Twist: use hand height from MediaPipe to control the light plane, as if stroking the tree. #### Assembly — Kimchi and Chips (2012) - Video: https://vimeo.com/42707293 - Interaction: Projection, Spatial Mapping, Perception & Effects - Platform & tech: Projection, projection mapping, 3D scanning - Idea: 5,500 hanging blocks are painted with projected pixels, letting digital forms live in real space. - What it is: A hemisphere of 5,500 suspended white blocks is scanned and coated with pixels from external projectors, forming a habitat for digital forms inside physical space. - Technique: Projector-camera structured-light scanning locates each suspended block in 3D, so multiple projectors can address the blocks as volumetric pixels for animated forms. - Try it: Hang 30 small white cubes in a cardboard box, light them one by one with a projector and record their positions, and build a "voxel screen" that shows a swimming shape. Twist: make it from one cube per student, each showing its owner's name when it lights up. #### Line Segments Space — Kimchi and Chips (2013) - Video: https://vimeo.com/111610020 - Interaction: Projection, Spatial Mapping, Perception & Effects - Platform & tech: Projection, projection mapping, openFrameworks - Idea: A web of nylon threads in the dark, lit by precise projection into flowing 3D shapes. - What it is: A web of nylon threads spans a dark gallery; calibrated projection lights the strings to articulate dynamic imaginary forms in the physical volume. - Technique: Calibrated projection (projector pose solved against the known string geometry) lights only the pixels that fall on each nylon thread, drawing shapes along lines in physical space. - Try it: String a few dozen white threads inside a dark box, have the projector light only the pixels that fall on the threads using a mask, and make light flow along them in TouchDesigner to trace a 3D outline. Twist: arrange the threads by the strokes of a piece of text, so the light "writes" the characters in the air. #### Light Barrier — Kimchi and Chips (2014) - Video: https://www.youtube.com/watch?v=Dp7c_0v2TRw - Interaction: Projection, Perception & Effects - Platform & tech: Projection, projection, mirror arrays, haze - Idea: A mirror array bends projected light into haze to form floating shapes of light. - What it is: Projected light is reflected by 200 curved mirrors into haze, forming floating volumetric shapes that seem to exist in mid-air. - Technique: Structured-light scanning calibrates each projector pixel's ray after reflection by convex mirrors, and those rays are rendered as volumetric beams that intersect in haze to draw 3D shapes. - Try it: Shine a projector into the mist from a fog machine or humidifier, use white-line-on-black graphics to create sweeping "light sheets", and compose a one-minute light sculpture animation. Twist: let the spectrum of live music determine the shape of the beams. #### Light Barrier Third Edition — Kimchi and Chips (2016) - Video: https://vimeo.com/218354021 - Interaction: Projection, Perception & Effects - Platform & tech: Projection, projection, concave mirrors, haze - Idea: A larger Light Barrier where light condenses in fog into volumetric images and objects. - What it is: A larger edition at the Asian Culture Center in Gwangju: concave mirrors and projection materialize volumetric light objects and scenes in fog. - Technique: Concave mirror arrays focus calibrated projector rays into haze, and mapping each pixel's reflected ray lets the system draw volumetric light objects at chosen positions in the room. - Try it: Reflect a projection into fog with a concave makeup mirror and try to focus the light into a floating point of light that moves. Twist: use the point of light to tell a small "firefly" story in which it flees when viewers come close. ### Leandro Erlich *Artist; architectural illusions* Argentine artist who builds everyday architecture that behaves impossibly: a swimming pool you can stand under, a house facade people 'climb' via a giant mirror, a house hanging from a crane. #### Swimming Pool — Leandro Erlich (1999) - Video: https://www.youtube.com/watch?v=NT7gjhHq9d0 - Interaction: Perception & Effects, Portals & Worlds, Shared & Social - Platform & tech: Desktop, glass, water, architecture - Idea: Two audiences looking at each other through one surface create the story: an AR scene can give 'inside' and 'outside' viewers different roles in the same illusion. - What it is: Seen from the deck, a pool looks full of water with people standing fully clothed at the bottom; they reached it through a side door, because only a thin sheet of water sits on a sealed glass ceiling. - Technique: A glass slab about 10 cm below the rim holds a shallow layer of water, so light refraction fakes the depth of a full pool above a dry, walk-in room painted pool-blue. - Try it: Anchor a virtual rippling water plane over a real table with phone AR, and have a classmate crouch under the table while others view through the phone. Twist: add caustic light and bubbles that follow the person underneath using body tracking. #### Dalston House (Bâtiment) — Leandro Erlich (2013) - Video: https://www.youtube.com/watch?v=AFF3BAnSNn4 - Interaction: Perception & Effects, Hands & Body, Shared & Social - Platform & tech: Desktop, mirror, printed facade, scaffolding - Idea: Swap the gravity axis and let people perform for their own reflection: an AR mirror that re-orients the floor turns every visitor into an acrobat without any rendering tricks on their body. - What it is: A full-size Victorian house facade lies flat on the ground in East London; a giant tilted mirror above it reflects visitors who lie on the facade so they appear to hang from window ledges and climb walls. - Technique: A printed and built facade is laid horizontally and a large mirror angled at 45 degrees stands over it, rotating the scene by 90 degrees for anyone watching the mirror. - Try it: Build a 'gravity mirror' in Lens Studio or WebXR: film the floor with the phone camera, rotate the feed 90 degrees and composite a building facade under the people lying on it. Twist: let a second phone show the unrotated view side by side so the audience sees both the trick and the result. #### Pulled by the Roots — Leandro Erlich (2015) - Video: https://www.youtube.com/watch?v=A58zmN67Z90 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, crane, sculpted house, fibreglass - Idea: An impossible object placed in a real public space works best when it borrows a familiar machine: attach AR fantasy to real cranes, poles or bridges so the city seems to be doing it. - What it is: A construction crane in Karlsruhe's market square holds a whole townhouse in the air, with its roots dangling beneath it like a tree pulled out of the ground. - Technique: A lightweight replica of a house with sculpted roots is likely built from a steel frame and fibreglass skin so an ordinary tower crane can suspend it above the square. - Try it: Find a real hook, lamp post or crane on campus and use image or geo anchoring to hang a virtual building with roots from it in phone AR. Twist: make the roots slowly grow toward the ground the longer someone looks at it. #### The Cloud — Leandro Erlich (2016) - Video: https://www.youtube.com/watch?v=Fw3JzEJu9F4 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, printed glass, vitrine, layered image - Idea: Volume can be faked with a stack of flat layers: slicing a 3D shape into printed planes is a cheap way to make holograms or 'volumetric' AR content that reads from many angles. - What it is: A white cloud floats inside a glass vitrine; walking around it, you notice it is made of stacked glass sheets, each printed with one thin slice of the cloud. - Technique: A 3D cloud model is cut into slices and each slice is printed with ceramic ink on a glass pane; the panes are stacked at even spacing so the image fuses into a volume. - Try it: Slice a 3D model into 20 semi-transparent planes in Unity or three.js and anchor the stack in AR, then compare it with the solid mesh as you walk around. Twist: let each plane drift slightly with the phone's motion so the cloud 'breathes' when you move. #### The Building (Liberty Science Center) — Leandro Erlich (2023) - Video: https://www.youtube.com/watch?v=heWqEr-MqEI - Interaction: Hands & Body, Shared & Social, Perception & Effects - Platform & tech: Desktop, mirror, printed facade - Idea: A proven illusion becomes a play space when you design the mirror as a stage: AR selfie effects work best when the camera image is the show and the physical set is the backstage. - What it is: In a science museum in New Jersey, families crawl, dangle and pose on a brownstone facade laid on the floor, and watch themselves climbing it in a huge mirror overhead. - Technique: The Bâtiment principle is adapted for a museum: a printed facade on the floor and an angled mirror suspended above, sized so a whole family fits in the reflection. - Try it: Make a Lens Studio ground-plane lens that turns a floor photo of your classroom into a wall to climb, and invite three classmates to invent poses for it. Twist: add a virtual pigeon that lands on whoever holds still longest. ### Marco Tempest *Illusionist and techno-magician; MIT Media Lab Director's Fellow* Swiss-born magician who has combined illusion with augmented reality, projection tracking and robotics since the late 2000s, building his early AR card tricks and projection routines with Zach Lieberman in openFrameworks. #### Augmented Reality Magic 1.0 — Marco Tempest (2009) - Video: https://vimeo.com/4336830 - Interaction: Tangible Objects, Performance, Play - Platform & tech: Desktop, openFrameworks, ARToolKit, OpenCV - Idea: Marker-based AR as a close-up magic trick: the cards themselves are the markers. - What it is: Tempest lays real playing cards on a table in front of a camera; on the screen animated jokers climb out of the cards, comment on the trick and help him find the chosen card. - Technique: Each card face works as a fiducial marker tracked with ARToolKit inside an openFrameworks app written with Zach Lieberman and Theo Watson, and pre-animated characters are rendered on top of the live camera image. - Try it: Print five image markers as playing cards and build a phone AR trick in which a character appears on the card the audience picked. Twist: the character must lie once, and the performer has to catch it. #### Magic Projection (live at TEDxTokyo) — Marco Tempest (2010) - Video: https://vimeo.com/11801074 - Interaction: Projection, Tangible Objects, Performance - Platform & tech: Projection, openFrameworks, OpenCV - Idea: A handheld blank board becomes a living screen because the projection follows it. - What it is: Tempest walks on stage holding a blank white board; a tracked projector keeps an animated stick man on the board as he moves it, and the character jumps, falls and interacts with Tempest's hands. - Technique: The board is tracked in the camera image (likely with infrared markers or brightness tracking, built with Zach Lieberman in openFrameworks) and the projected animation is warped to its corners in real time. - Try it: Track a sheet of white cardboard with a webcam and keep a projected character on it while you move it around. Twist: when the board tilts past 45 degrees, the character slides off and lands on the floor projection. #### A magical tale (with augmented reality) — Marco Tempest, onformative (Cedric Kiefer & Julia Laub) (2012) - Video: https://vimeo.com/39487725 - Interaction: Performance, Tangible Objects, Projection - Platform & tech: Projection, Desktop, openFrameworks - Idea: Storytelling in which every AR effect is timed like a magic reveal. - What it is: On the TED stage Tempest tells a story about magic while a camera-projector rig augments what he holds: projected graphics land on cards and a small screen, and the live mixed image is shown to the audience. - Technique: A head-mounted or tripod camera feeds an openFrameworks application that tracks the props and composites or projects synchronized graphics in real time; the exact rig is likely custom. - Try it: Write a 90-second story and stage it with three AR reveals on handheld objects, rehearsed so each effect lands on a spoken line. Twist: one reveal must happen in the audience's view but not in the performer's. #### Nikola Tesla in Sound and Light — Marco Tempest (2012) - Video: https://vimeo.com/42402467 - Interaction: Projection, Tangible Objects, Performance - Platform & tech: Projection, openFrameworks, Arduino - Idea: A pop-up book becomes a projection-mapped stage for a live magic story. - What it is: Tempest tells the story of Nikola Tesla on a tabletop stage of pop-up paper structures; projection mapping lights up the paper, animates Tesla's inventions and syncs with sound and the performer's actions. - Technique: White pop-up paper sculptures are mapped with pre-calibrated projection from an openFrameworks show system, with Arduino-triggered cues keeping the projection in step with the performance. - Try it: Build a small pop-up scene from white paper and projection-map three animated states onto it with a phone projector or cheap projector. Twist: opening or closing a paper flap must switch the scene. #### Cyber Cards Live — Marco Tempest (2013) - Video: https://vimeo.com/63112943 - Interaction: Hands & Body, Tangible Objects, Performance - Platform & tech: Desktop, openFrameworks, OpenCV - Idea: Hand gestures over real cards trigger virtual changes that look like sleight of hand. - What it is: Four real cards lie on a table; with a wave of the hand their faces change, swap and vanish in the live HD camera image shown to the audience. - Technique: Cards are recognized with SURF feature tracking and hand movement with motion-template gradients, and new card faces are composited onto the camera feed in openFrameworks. - Try it: Use image tracking on four cards and a simple hand-motion detector so a swipe over a card swaps its face in the phone view. Twist: the swap only happens if nobody in the frame is looking at the card. ### Marlena Myles *Artist (Spirit Lake Dakota)* Dakota artist who uses AR public art, animation and projection to restore Dakota language, stories and place-knowledge to the land of the Twin Cities. #### Dakota Spirit Walk — Marlena Myles (2021) - Video: https://www.youtube.com/watch?v=nxo1UIyZR-Y - Interaction: Location & City, Information & UI, Voice & Sound - Platform & tech: Phone, mobile AR, image markers, audio narration - Idea: On a sacred trail, AR brings Dakota spirits and language back into view. - What it is: A permanent AR public-art trail at the Bruce Vento Nature Sanctuary in Saint Paul where phones reveal Dakota spirits, stories and language at sacred sites. - Technique: Image-marker or location-triggered AR (likely Adobe Aero/app) anchors animated Dakota artwork and audio narration to signs along the trail. - Try it: Design three AR stations along a campus path, each using an image marker to show a place's original name and its story. Twist: invite local elders to record the narration. #### Innerworld Prism — Marlena Myles (2021) - Video: https://www.youtube.com/watch?v=Er7oZ2n1u8s - Interaction: Projection, Location & City, Perception & Effects - Platform & tech: Projection, large-scale projection, animation - Idea: A giant projection tells a dream about letting go of the self and returning to nature. - What it is: A large-scale projection in Saint Paul commissioned by the Great Northern Festival: a dreamlike animation about losing the human ego to regain nature. - Technique: Animated Dakota-inspired illustration is projected at architectural scale onto an urban surface. - Try it: Draw a 30-second looping animation using local natural or cultural symbols and project it onto the outer wall of a school building at night. Twist: make elements of the animation crawl out through the windows in the wall. #### Augmented Reality Dakota Landmap of the Twin Cities — Marlena Myles (2022) - Video: https://www.youtube.com/watch?v=uc3mS47Xr_c - Interaction: Location & City, Information & UI, Tangible Objects - Platform & tech: Phone, Adobe Aero, image-anchored AR - Idea: A city map redrawn with Dakota place names comes alive in 3D through AR. - What it is: An illustrated map of the Twin Cities with Dakota place names that becomes a 3D animated landscape when viewed through Adobe Aero. - Technique: An Adobe Aero scene anchors layered 3D illustrations and animations to the printed map image. - Try it: Draw a map of your local area labeled with old or dialect place names, and use Adobe Aero to make the landmarks pop up in 3D. Twist: tapping each landmark plays a sound clip about it. #### Dakota Sacred Hoop Walk — Marlena Myles (2022) - Video: https://www.youtube.com/watch?v=0tWS13lHc5Q - Interaction: Location & City, Information & UI - Platform & tech: Phone, mobile AR - Idea: Walk the Sacred Hoop trail and use AR to understand the land from a Dakota perspective. - What it is: An AR exhibit that helps visitors understand nature, land and culture from a Dakota perspective through artwork revealed at stations along a walk. - Technique: Phone-based AR triggered at stations along the path overlays animated Dakota artwork and teachings. - Try it: Choose four points on campus for the four directions and use AR to show nature knowledge tied to east, south, west and north. Twist: the final animation unlocks only after you walk the full circle in order. #### Star Spirit of Creation (Dakota thoughts in Latvia) — Marlena Myles (2024) - Video: https://www.youtube.com/watch?v=0gVogFBSst8 - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, mobile AR, screen recording - Idea: Bring Dakota star spirits into a foreign landscape to connect the ancestral memories of two places. - What it is: An AR video piece made in Latvia in which Dakota star spirits and patterns appear over the people and places the artist visited. - Technique: AR animation is placed into real scenes and captured as screen recordings (likely with a mobile AR authoring tool). - Try it: Design an AR 'guardian spirit' for your hometown culture and film short clips of it appearing in places you travel to or around campus. Twist: the spirit takes a different form in each place. ### Maxin10sity *Projection mapping studio (architectural mapping, sky mapping)* Hungarian studio led by art directors László Czigány and András Sass that maps some of the largest buildings in the world, from Bucharest's Palace of the Parliament to the Bolshoi Theatre, Karlsruhe Palace and the Treasury at Petra. It has won the Circle of Light, Genius Loci and Berlin Festival of Lights competitions. #### Parallel Universes (Bolshoi Theatre) — Maxin10sity (2015) - Video: https://www.youtube.com/watch?v=2dh9Zl2vQZk - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, projection mapping, 3D animation - Idea: Showing several 'versions' of the same real building in turn is a strong way to tell time or alternate realities; AR can do this on any street corner. - What it is: A mapping on the Bolshoi Theatre in Moscow made for the Circle of Light Art Vision 2015 contest: the classical colonnade dissolves into alternative versions of itself, as if several parallel buildings were sliding through the same facade. - Technique: Animated 3D variations of the columns and pediment are rendered against a model of the building and projected in alignment, with light and shadow faked to sell depth. - Try it: Pick one building on campus and make three AR 'parallel versions' of it (ruined, future, underwater) in Reality Composer or Lens Studio, swapping them with a swipe. Twist: the version changes with the time of day on the phone clock. #### LEGACY (Schlosslichtspiele Karlsruhe) — Maxin10sity (2016) - Video: https://www.youtube.com/watch?v=VeGZ6p37now - Interaction: Location & City, Information & UI - Platform & tech: Projection, projection mapping, 3D animation, sound design - Idea: A building can reveal what is inside it; AR can likewise 'open' a museum, a machine or a wall to show hidden content in place. - What it is: A long mapping on Karlsruhe Palace, home of the Badisches Landesmuseum, for the Schlosslichtspiele festival: the facade opens like a cabinet to show the museum's collections and history to the crowd in the palace square. - Technique: Scenes of museum objects are composited into a 3D model of the palace facade so the building appears to open, then projected from towers in the square. - Try it: Choose a closed door or cabinet on campus and build an AR 'x-ray' that opens it and shows what is inside with Lens Studio or WebXR image tracking. Twist: viewers must stand in a specific spot for the doors to open. #### Virtual Sky Mapping – The Dream — Maxin10sity (2021) - Video: https://www.youtube.com/watch?v=RhbLeV2m0xE - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, virtual production, 3D animation, broadcast compositing - Idea: This is AR by broadcast: virtual content locked to a real landmark in a camera image, which is exactly what a phone AR fireworks show would be. - What it is: For Seattle's 2021 New Year's Eve, when crowds could not gather, the usual fireworks at the Space Needle were replaced by a virtual 'sky mapping' show: 3D light shapes and animations composited around the real tower in the broadcast and stream. - Technique: Tracked or fixed broadcast cameras film the Space Needle while 3D animations are rendered to match the camera position and composited in real time or in post (method likely virtual production). - Try it: Build a 'virtual fireworks' WebXR or 8th Wall scene anchored above a tall landmark near campus, with bursts timed to music. Twist: each viewer's tap launches a firework everyone else sees too. #### Re-Vision (The Treasury, Petra) — Maxin10sity (2023) - Video: https://www.youtube.com/watch?v=0VqvHN74oNA - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, projection mapping, 3D animation - Idea: On a sacred or heritage site, restraint is part of the design: light that only traces what is already there respects the place more than a new story would. - What it is: Made for the first Petra Light Festival in Jordan, this mapping plays across the rock-cut facade of the Treasury (Al-Khazneh) with abstract light that traces its carved columns and reliefs without covering its heritage with figurative content. - Technique: A detailed 3D model of the carved facade is used to generate edge-following, abstract animations that are projected in alignment with the stone. - Try it: Choose a historic detail (a carved door, a statue) and design an AR layer in Lens Studio that only outlines and animates its existing edges in light. Twist: the light slowly fades if the viewer comes too close, asking for respect. ### Mikko Haapoja *Developer; former AR/VR development manager at Shopify* Toronto developer who spent almost ten years at Shopify on 3D/AR/VR product media, the BFCM live globe and Sidekick, contributed to google/model-viewer, and joined Opendoor in 2026. #### #1scanaday — Mikko Haapoja (2021) - Video: https://x.com/MikkoH/status/1380712221240594432 - Interaction: Spatial Mapping, Tangible Objects - Platform & tech: Phone, LiDAR, photogrammetry, Polycam, 3d Scanner App, Sketchfab - Idea: A daily capture habit teaches you what reality can and cannot be scanned. - What it is: Starting on New Year 2021, Mikko posted one 3D scan a day made with iPhone LiDAR and photogrammetry apps; day 100 was this bronze lion, and every scan was published on Sketchfab. - Technique: Each object or space is captured with a LiDAR iPhone or with photo-based photogrammetry apps (3d Scanner App, Polycam, Trnio) and uploaded as a textured mesh to Sketchfab for web and AR viewing. - Try it: Scan one object a day for a week with a free phone scanner, log what failed (shiny, thin, dark), and place the best scan in AR. Twist: scan the same object with two different apps and compare. #### Four Scanner Apps, One Object — Mikko Haapoja (2021) - Video: https://x.com/MikkoH/status/1377062209164079107 - Interaction: Spatial Mapping, Tangible Objects - Platform & tech: Phone, LiDAR, photogrammetry, iPhone - Idea: Compare tools on one subject to understand what each is good at. - What it is: On day 88 of his scan diary, Mikko captured the same object with four iPhone scanner apps and showed the results side by side: LiDAR had improved on small objects, but photogrammetry still won on detail. - Technique: The same subject is captured with LiDAR-based and photogrammetry-based apps and the meshes are compared for geometry and texture quality. - Try it: In teams, scan the same classroom object with two to four capture methods and present a side-by-side AR comparison. Twist: pick the object that makes the methods disagree the most. #### Mirror Self-Portraits — Mikko Haapoja (2021) - Video: https://x.com/MikkoH/status/1347920408033562624 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, LiDAR, 3D scanning, Sketchfab - Idea: Use a scanner's failure mode as a creative effect. - What it is: Mirrors break LiDAR and photogrammetry, so Mikko turned the failure into an effect: the scanner reconstructs the reflection as a warped room behind the glass, creating trippy 3D self-portraits. - Technique: LiDAR and photogrammetry assume opaque surfaces, so a mirror's reflection is reconstructed as extra geometry behind the glass, yielding mirrored self-portrait meshes. - Try it: Deliberately scan 'unscannable' things (mirror, glass, water, a moving person) and curate the most beautiful failures into an AR glitch gallery. Twist: design a scene that only makes sense because of the scanning error. #### Scanning My Son While He Slept — Mikko Haapoja (2022) - Video: https://x.com/MikkoH/status/1534246133362577409 - Interaction: Tangible Objects, Spatial Mapping - Platform & tech: Phone, Desktop, Object Capture, RealityKit, USDZ - Idea: 3D scans as next-level family portraits and time capsules. - What it is: Mikko used Apple's Object Capture to scan his two-year-old son while he slept, imagining a future where he can put on a headset and see him at that age in perfect scale. - Technique: Photos taken around the still subject are processed by Apple's Object Capture photogrammetry API on a Mac into a textured USDZ model that can be viewed at 1:1 scale in AR. - Try it: Make a 1:1 AR time capsule: scan a meaningful object or a very still classmate and attach a short note that appears next to it in AR. Twist: design how and when the capsule may be opened in ten years. #### The Element in iOS Safari — Mikko Haapoja (2022) - Video: https://x.com/MikkoH/status/1541529590652456964 - Interaction: Information & UI, Tangible Objects - Platform & tech: Web, Phone, WebKit model element, USDZ, Safari - Idea: Treat 3D objects as first-class web content, just like images. - What it is: A quick test of WebKit's experimental HTML model element on iOS Safari: a Converse sneaker hosted on Shopify's CDN spins as a native 3D object inside an ordinary web page. - Technique: WebKit's experimental model tag embeds a USDZ file directly in HTML so the browser renders it and handles rotation without WebGL code, hinting at pages that can hand 3D straight to AR. - Try it: Build a product page that shows a 3D model with model-viewer (or the model element where supported) plus a 'view in your space' AR button. Twist: make the 3D model react to page scrolling. ### Motion Bank / Choreographic Coding Lab *Dance research project (Forsythe Company, now Hochschule Mainz) and the travelling Choreographic Coding Lab format* Motion Bank grew out of The Forsythe Company in 2010 to build digital dance scores and annotation tools; since 2013 its Choreographic Coding Labs have gathered coders and dancers in Frankfurt, Berlin, Melbourne, New York, Amsterdam, Chatham, London and elsewhere to prototype movement-driven software and art. #### Gravitacional (visiophone, CCL Frankfurt 2013) — Motion Bank / Choreographic Coding Lab (2013) - Video: https://vimeo.com/81705600 - Interaction: Hands & Body, Perception & Effects, Projection - Platform & tech: Projection, Desktop, Kinect, particle system, Processing - Idea: Turn the dancer's hands into gravity wells that sculpt a particle cloud. - What it is: A prototype from the first Choreographic Coding Lab by visiophone (Rodrigo Carvalho): a field of particles is pulled by a central gravity and by two extra forces that follow the dancer's hands. - Technique: Hand positions from a depth camera become attractor or repeller forces in a particle simulation, projected as a performative visual system. - Try it: Track both hands with MediaPipe in p5.js and let each hand attract particles while a fixed centre pulls them back. Twist: closing a hand flips its force from attract to repel. #### MOMENTUM (Schnelle Bunte Bilder & kling klang klong) — Motion Bank / Choreographic Coding Lab (2014) - Video: https://vimeo.com/112193826 - Interaction: Hands & Body, Voice & Sound, Perception & Effects - Platform & tech: Projection, Desktop, Kinect, particle system, generative sound - Idea: Your body becomes a liquid creature whose motion is also the soundtrack. - What it is: An interactive system shown through the Choreographic Coding Lab in which a person and their surroundings dissolve into a fluid particle creature, and body movement generates sound and music. - Technique: A depth camera's point cloud seeds a particle system while movement speed and position are mapped to generative sound, built by Schnelle Bunte Bilder with sound studio kling klang klong. - Try it: Use a phone's LiDAR or a webcam depth model to turn a person into points, let the points drift with velocity, and map total movement to the volume of a synth. Twist: when the person freezes, the creature slowly forgets their shape. #### Quality Visualizing Tool (Kat Sullivan & Sergio Mora-Diaz, CCL 2015) — Motion Bank / Choreographic Coding Lab (2015) - Video: https://vimeo.com/138653312 - Interaction: Hands & Body, Information & UI, Performance - Platform & tech: Desktop, Projection, Asus Xtion, Cinder, skeleton tracking - Idea: Make invisible movement qualities visible, one visual rule per quality. - What it is: A set of live programs from the 4th Choreographic Coding Lab that visualise different trackable qualities of a dancer, such as speed, expansion and flow, as graphics drawn around the body. - Technique: An Asus Xtion depth camera feeds skeleton data to Cinder sketches that compute features like velocity or body volume and draw them in real time. - Try it: Pick three movement words (sharp, heavy, expansive), compute a simple metric for each from webcam pose data, and give each its own visual layer. Twist: let a dancer perform without knowing which word is being measured. #### Choreographic Coding Lab at A+E Lab (CCL-14) — Motion Bank / Choreographic Coding Lab, AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) (2022) - Video: https://vimeo.com/759923547 - Interaction: Performance, Hands & Body, Projection - Platform & tech: Projection, Desktop, motion capture, Isadora, Piecemaker - Idea: A one-week lab format where code meets choreographic thinking, and dancers test prototypes on stage the same day. - What it is: Documentation of the UK's first Choreographic Coding Lab in Chatham, where about twenty coders and dancers spent a week prototyping mocap particles, projected visuals and Isadora patches with facilitators from Motion Bank, Troika Ranch and Studio Wayne McGregor. - Technique: Participants worked from Motion Bank mocap datasets and Piecemaker annotations, and with live mocap suits, Isadora and TouchDesigner-style tools to drive projections from movement. - Try it: Run a mini lab: pair each coder with a dancer, give them one mocap clip or webcam pose stream, and have each pair show a projected prototype within two hours. Twist: each pair must use one card from a choreographic prompt deck as its rule. ### Musion (Musion Eyeliner) *Holographic projection company (Pepper's ghost stage systems)* British company behind the Musion Eyeliner system, a modern Pepper's ghost that reflects a projected video off a thin, tensioned foil angled across the stage so that people and cartoon characters seem to stand in the room. It staged the Gorillaz 'hologram' at the 2005 MTV Europe Music Awards and life-size speeches by Prince Charles and Richard Branson. #### Gorillaz hologram (MTV Europe Music Awards) — Musion (Musion Eyeliner) (2005) - Video: https://www.youtube.com/watch?v=CRViE4N-u5Y - Interaction: Performance, Perception & Effects - Platform & tech: Projection, Musion Eyeliner, Pepper's ghost, projection foil - Idea: A virtual character sharing a real stage is the core promise of AR performance, and this shows that a flat image works if the reflection, scale and lighting match the stage. - What it is: At the 2005 MTV Europe Music Awards in Lisbon, the cartoon band Gorillaz appears to play live on stage as life-size 3D-looking characters, billed as the 'world's first 3D hologram performance'. The same system later put Gorillaz on stage with Madonna at the 2006 Grammys. - Technique: An HD projector (or LED source) is reflected by a thin, tensioned transparent foil angled at 45 degrees across the stage (Pepper's ghost), so the image appears in the stage space behind it. - Try it: Build a tabletop Pepper's ghost with a phone and a sheet of clear plastic at 45 degrees, then play an animated character so it 'stands' on a small stage with a real toy. Twist: make the virtual character and a real puppet perform a duet. #### Richard Branson hologram (Virgin Digital launch) — Musion (Musion Eyeliner) (2005) - Video: https://www.youtube.com/watch?v=x4FO6HFe6rA - Interaction: Performance, Perception & Effects - Platform & tech: Projection, Musion Eyeliner, Pepper's ghost - Idea: The designers chose to make the figure slightly transparent on purpose; AR avatars also need a visual 'this is virtual' cue so people trust what they see. - What it is: At the launch of Virgin Digital in Virgin's London Megastore, Sir Richard Branson appears 'digitally downloaded' on stage as a semi-transparent life-size figure, filmed with a degree of transparency to look like a hologram. - Technique: Pre-recorded footage of Branson shot against black is projected onto an angled Eyeliner foil so he appears to stand on the stage. - Try it: Record a classmate against a black backdrop, key the video, and place them as a slightly transparent AR presenter on a real table with Reality Composer or 8th Wall. Twist: test three levels of transparency and ask viewers which feels most honest. #### John McEnroe hologram (Wimbledon Lawn Tennis Museum) — Musion (Musion Eyeliner) (2006) - Video: https://www.youtube.com/watch?v=SfjAxPYpQck - Interaction: Information & UI, Perception & Effects - Platform & tech: Projection, Musion Eyeliner, Pepper's ghost, museum installation - Idea: A virtual guide placed in the exact room where history happened is more convincing than any screen; AR museum guides should stand in the space, not in a panel. - What it is: In the gentlemen's dressing room of the new Wimbledon Lawn Tennis Museum, a virtual John McEnroe appears among the real lockers and talks visitors through the history of the room and the championship. Year of museum opening; video uploaded 2008. - Technique: A hidden projector and angled foil (Pepper's ghost) place pre-recorded footage of McEnroe inside the real dressing-room set. - Try it: Pick a room on campus with history, record a short guide monologue, and place the guide as an AR figure sitting on a real bench with Lens Studio or Reality Composer. Twist: the guide's story changes depending on which object the visitor looks at. #### Prince Charles hologram (World Future Energy Summit) — Musion (Musion Eyeliner) (2008) - Video: https://www.youtube.com/watch?v=wphytMJ_F_A - Interaction: Performance, Shared & Social - Platform & tech: Projection, Musion Eyeliner, Pepper's ghost - Idea: A virtual presence can replace travel when it respects the stage conventions of a real speaker; this is the idea behind holographic telepresence in AR. - What it is: A life-size 3D-looking image of the Prince of Wales delivers a speech on the environment to world leaders at the World Future Energy Summit in Abu Dhabi, then vanishes, saving him the flight. - Technique: Recorded video of the speaker is projected onto a large angled foil in front of the stage so he appears standing at the lectern (Pepper's ghost). - Try it: Stage an AR 'remote guest lecture': record a two-minute talk, place the speaker at a real lectern with image anchors, and invite the class to listen. Twist: the speaker answers one pre-recorded question when a student raises a hand. #### Telstra live hologram telepresence — Musion (Musion Eyeliner) (2008) - Video: https://www.youtube.com/watch?v=P3jhFXmNUt8 - Interaction: Shared & Social, Performance - Platform & tech: Projection, Musion Eyeliner, live video link, Pepper's ghost - Idea: Live telepresence changes a projection effect into a conversation; the moment the remote person can answer, the audience treats them as present. - What it is: In an Australian first, Telstra's Chief Technology Officer Dr Hugh Bradlow is beamed live from Melbourne into a business event in Adelaide as a life-size 'hologram' who talks to the audience in real time. - Technique: A live HD video feed of the speaker filmed against black in Melbourne is sent over a network link and projected onto an Eyeliner foil on the Adelaide stage. - Try it: Connect two rooms with a video call, key out the background, and show the remote person as a life-size AR figure standing in the other room with WebRTC and WebXR. Twist: let the remote person point at real objects in the room. ### Nakata Nobuyuki (nobnak) *Graphics programmer for digital-art installations* Tokyo programmer who builds real-time graphics for large digital-art installations and has open-sourced many Unity effects since 2013: watercolour filters, caustics, light streaks, smoke blur. #### Mesh-Based Watercolor Paint — Nakata Nobuyuki (nobnak) (2014) - Video: https://vimeo.com/114418780 - Source code: https://github.com/nobnak/PaintingWithPolygons - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, Unity, C# - Idea: Simulate watercolour as growing, deforming polygons rather than pixels. - What it is: Brush strokes spread and bleed like wet watercolour pigment, with darker edges and soft overlapping layers. - Technique: Implements 'Painting with Polygons' (DiVerdi et al.): each stroke is a polygon whose vertices are displaced and subdivided step by step to imitate pigment flow, then composited with edge darkening. - Try it: Let users paint watercolour strokes on a detected AR wall with a finger and watch them bleed over a minute. Twist: make strokes bleed faster when the phone's microphone hears running water. #### Real-time Water Caustics Simulation — Nakata Nobuyuki (nobnak) (2014) - Video: https://vimeo.com/110753770 - Source code: https://github.com/nobnak/CausticsUnity - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL - Idea: Compute where light lands after passing through a moving water surface. - What it is: Rippling light patterns dance on the floor of a pool as the water surface above it moves. - Technique: Following Yuksel and Keyser's height-field caustics method, rays refracted through the water height field are projected onto the floor and their density is accumulated into a caustics texture each frame. - Try it: Project the caustics texture onto a real floor plane in AR with a light that follows the phone, so the room looks like it is under water. Twist: let taps on the screen drop ripples into the simulated surface. #### Smoke Blur Effect — Nakata Nobuyuki (nobnak) (2015) - Video: https://vimeo.com/148481861 - Source code: https://github.com/nobnak/SmokeBlur - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Image Effect - Idea: Blur an image along a flow field so it looks like it is turning into smoke. - What it is: The rendered scene melts into curling smoke: pixels are dragged along swirling flows so edges wisp away. - Technique: A post-processing pass advects the previous frame's colour through a curl-noise velocity field and blends it with the new frame, creating a feedback smoke trail. - Try it: Apply the smoke blur to the AR camera image masked by people segmentation, so only people dissolve into smoke as they move. Twist: tie the smoke strength to the loudness of the room. #### Star glow shader (Kawase Light Streak) — Nakata Nobuyuki (nobnak) (2016) - Video: https://vimeo.com/173164170 - Source code: https://github.com/nobnak/KawaseLightStreakUnity - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Image Effect - Idea: Recreate a photographic star filter in real time. - What it is: Bright highlights in the scene burst into long cross-shaped star streaks, like a camera lens with a star filter. - Technique: Following Masaki Kawase's GDC 2003 technique, bright pixels are extracted and blurred repeatedly along a few directions with growing sample offsets, then added back to the image. - Try it: Apply the streak effect to an AR scene so real lamps and virtual objects both flare with star streaks. Twist: rotate the streak directions with the phone's roll angle. #### WaterColor Filter — Nakata Nobuyuki (nobnak) (2016) - Video: https://vimeo.com/152248671 - Source code: https://github.com/nobnak/WaterColorFilter - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Image Effect - Idea: Turn any real-time scene into a moving watercolour illustration. - What it is: A 3D scene is re-rendered as a watercolour painting, with wobbling edges, darkened pigment borders and paper texture. - Technique: Tone-based diffuse shading, edge wobbling by noise-offset UVs, edge darkening from colour gradients and a paper-grain overlay are combined in an image-effect chain. - Try it: Run the filter over the AR camera feed so the real room and the virtual objects share the same watercolour style. Twist: let the paper texture be a photo of real paper the students scan. ### Ned Kahn *Environmental artist and sculptor* American artist, trained at the Exploratorium, whose façades and sculptures are skins of thousands of hinged aluminium or plastic panels, fog vortices and moving sand that reveal wind, water and light as they pass through. #### Tornado — Ned Kahn (1996) - Video: https://www.youtube.com/watch?v=wMDM0x01gy4 - Interaction: Perception & Effects, Hands & Body - Platform & tech: Desktop, fog, fans - Idea: Fog is a way to see flow; AR can use particles in the same way to show otherwise invisible currents, and invite the hand to disturb them. - What it is: A column of fog twists into a tall, stable vortex inside a frame; visitors can put their hands into it and watch the tornado bend and reform. - Technique: Fans on top pull air upward while angled vents add rotation, and a fog machine seeds the flow so the vortex becomes visible (year approximate; Kahn has built tornados since the 1980s). - Try it: Spawn a particle vortex on a real table in phone AR and let hand tracking push the particles away. Twist: let the vortex slowly drift toward the loudest sound in the room. #### Wind Veil — Ned Kahn (2000) - Video: https://www.youtube.com/watch?v=ZDPqrA4-jK0 - Interaction: Location & City, Perception & Effects, Spatial Mapping - Platform & tech: Desktop, hinged aluminium panels, wind - Idea: Making an invisible force visible across a surface is AR's core promise; Kahn does it at building scale with no electronics. - What it is: Tens of thousands of small hinged aluminium panels cover a parking-garage façade and flutter in the wind, so gusts sweep across the building as visible ripples of light. - Technique: Each panel hangs freely on a hinge; wind pressure tilts it and changes how it reflects sky and sun, so the façade becomes a real-time map of airflow (year approximate). - Try it: Cover a real building façade in WebXR with a grid of small reflective flaps that tilt with live wind data from a weather API. Twist: let the viewer blow into the microphone to add a gust. #### Technorama Facade — Ned Kahn (2002) - Video: https://www.youtube.com/watch?v=hVyo5ICl5-I - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, hinged aluminium panels, wind - Idea: Every pixel here is a physical sensor and actuator in one, a reminder that the best AR displays may be the world's own surfaces reacting. - What it is: The façade of the Swiss Science Center Technorama in Winterthur is covered with tens of thousands of small hinged aluminium plates that ripple in the wind, turning the building into a wind display. - Technique: Free-swinging aluminium plates mounted in a grid reflect the sky at varying angles, so pressure waves in the wind appear as moving patterns of light and shadow. - Try it: Map a grid of virtual plates onto a façade photo with image tracking and drive them with a fluid simulation. Twist: let the plates spell a word only when the wind stops. #### Articulated Cloud — Ned Kahn (2004) - Video: https://www.youtube.com/watch?v=hy9OrQyl1dE - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, translucent plastic squares, wind - Idea: A soft, moving skin makes architecture feel weather-sensitive; AR façades can borrow this idea of a responsive veil over a fixed structure. - What it is: The façade of the Children's Museum of Pittsburgh is a veil of thousands of translucent squares that shimmer in the wind like a cloud wrapped around the building. - Technique: Lightweight plastic tiles on hinged mounts respond individually to wind and scatter daylight, creating a large-scale analog display of turbulence. - Try it: Anchor a translucent tile veil to a real building with geospatial AR and animate each tile with Perlin-noise wind. Twist: make the veil part where people walk underneath. #### Wind Arbor — Ned Kahn (2011) - Video: https://www.youtube.com/watch?v=XZxZSHukZpo - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, hinged aluminium flaps, wind - Idea: Shadows cast inside tell visitors what is happening outside; AR can likewise let outdoor data leak indoors as light and shadow. - What it is: A tall glass façade at Marina Bay Sands in Singapore is lined with hundreds of thousands of small hinged metal flaps that flutter with the wind and cast moving shadows into the atrium. - Technique: Wind moves each flap independently and the changing pattern filters sunlight entering the building, creating a kinetic sunscreen (year approximate). - Try it: Project virtual flapping shadows onto a real floor in AR, driven by live wind speed from a weather API. Twist: tie the shadow density to the day's air quality. ### Nobumichi Asai *Technical and creative director (WOW / Asai studio)* Japanese visual artist who pioneered real-time face-tracking projection mapping, painting moving digital makeup onto performers' faces with near-zero latency. #### OMOTE — Nobumichi Asai (2014) - Video: https://www.youtube.com/watch?v=18y9RsYNLYw - Interaction: Face, Projection - Platform & tech: Projection, face tracking, projection mapping - Idea: Turn a face into a tracked projection canvas where digital makeup moves with every expression. - What it is: Real-time face tracking and projection mapping paint shifting digital makeup, cracks and masks onto a model's moving face. - Technique: Face tracking with markers or landmarks drives a 3D face mesh whose texture is projected by a calibrated projector, with latency kept low so makeup stays registered to the moving face. - Try it: Generate a face mesh with MediaPipe Face Mesh and project a "digital makeup" onto a white mask or a classmate's face (calibrate on a still mask first). Twist: make the makeup change with expression, for example cracks appear when the mouth opens. #### connected colors — Nobumichi Asai (2016) - Video: https://www.youtube.com/watch?v=nMvFwC3bo_E - Interaction: Face, Projection, Perception & Effects - Platform & tech: Projection, face tracking, projection mapping - Idea: Flower and bird patterns bloom live on a tracked face. - What it is: Face-tracked projection mapping blooms plants, birds and patterns across a performer's face in sync with her movement. - Technique: Real-time face tracking feeds a projector-camera system that renders animated patterns in the face's UV space, so plants and birds appear to grow on the skin as the head turns. - Try it: In TouchDesigner, drive a pattern-growing animation with MediaPipe face landmarks and project it onto the face of a classmate wearing a white mask. Twist: make the growth speed follow the music or the sound of breathing. #### INORI (Prayer) — Nobumichi Asai (2017) - Video: https://www.youtube.com/watch?v=9n0ZAwt23VU - Interaction: Face, Projection, Performance - Platform & tech: Projection, high-speed projector, face tracking - Idea: Tracking and projection at a thousand frames per second keep face imagery locked onto intense dance. - What it is: Dancers AyaBambi's faces are mapped with high-speed projection (1000 fps tracking) so skulls, tears and eyes follow their fast choreography. - Technique: A 1000 fps high-speed camera and a DynaFlash-type high-frame-rate projector close the tracking-to-projection loop within a few milliseconds, so imagery stays registered on fast-moving faces. - Try it: Build face-tracked projection with an ordinary projector and a webcam, measure and record the "ghost trail" caused by latency, then try to reduce it by predicting motion with velocity extrapolation. Twist: use the ghost trail itself as a visual language and make a dance that deliberately "can't keep up." #### Kacho-fugetsu — Nobumichi Asai (2017) - Video: https://www.youtube.com/watch?v=5cS7KLUPfZQ - Interaction: Face, Projection - Platform & tech: Projection, face tracking, projection mapping - Idea: Seasonal motifs of flowers, birds, wind and moon are projected onto a moving face. - What it is: Seasonal Japanese motifs of flowers, birds, wind and moon are projected onto a moving face using real-time tracking. - Technique: Likely the same face-tracked projection pipeline: tracked head pose drives a UV-mapped face model whose texture cycles through seasonal motifs, projected from a calibrated projector. - Try it: Make face textures for the four seasons with Lens Studio or phone AR face tracking, and switch seasons when the head turns. Twist: replace the four seasons with four typical kinds of weather or sounds from your own city. #### Dockyard 3D Projection Mapping / Yokohama Odyssey — Nobumichi Asai (2018) - Video: https://www.youtube.com/watch?v=vguPITbNorQ - Interaction: Projection, Location & City - Platform & tech: Projection, projection mapping - Idea: Turn a century-old stone dock into a projection stage for a seafaring epic. - What it is: Large-scale 3D projection mapping in the historic stone dry dock of Yokohama, turning the dock walls into an animated voyage. - Technique: Large-scale architectural projection mapping: the dock is surveyed into a 3D model, multiple projectors are aligned and edge-blended, and content is rendered from the model's UV layout. - Try it: Build a 'building' from a cardboard box or blocks, align projection to it in TouchDesigner / MadMapper, and make a two-minute animation about the history of a place. Twist: use only the building's own outlines as animation material. ### Novarama *Game studio; creators of Invizimals* Barcelona studio founded by Daniel Sánchez-Crespo that made the Invizimals series for PSP, PS Vita and PS3, which used a camera and marker trap card to capture creatures on the player's table. #### Invizimals — Novarama (2009) - Video: https://www.youtube.com/watch?v=JbTJw_2E-Ww - Interaction: Tangible Objects, Play - Platform & tech: Phone, PSP Go!Cam, marker tracking - Idea: Use a handheld's camera to 'catch' invisible monsters at home and battle them on a card. - What it is: A PSP game with a clip-on camera: players scan their surroundings to detect invisible creatures, then trap them on a printed marker card where they fight on the tabletop. - Technique: A printed fiducial card gives the camera pose for tabletop rendering, while a color-detection capture step (pointing the camera at certain hues) is used to spawn creatures. - Try it: Use camera color detection to 'discover' hidden creatures: a hint appears when you point at an object of a certain color, and then you summon it onto an image-marker card. Twist: different colors map to creatures with different attributes. #### Invizimals: Shadow Zone — Novarama (2010) - Video: https://www.youtube.com/watch?v=BiK2LdMeyXc - Interaction: Tangible Objects, Play - Platform & tech: Phone, PSP camera, marker tracking - Idea: The sequel adds more invisible creatures and capture challenges built around the trap card. - What it is: The sequel adds more creatures, new capture challenges and battles that play out around the camera-tracked trap card on the player's table. - Technique: Marker-based tracking on the trap card anchors creatures and battle effects, with capture challenges requiring the player to move the handheld relative to the marker. - Try it: Design a capture challenge that can only be completed by moving the phone around a marker card, for example lining up with the creature from a specific angle. Twist: the capture window gets shorter the rarer the creature is. #### Invizimals: The Lost Tribes — Novarama (2011) - Video: https://www.youtube.com/watch?v=-xlg4Xnu7UA - Interaction: Tangible Objects, Play - Platform & tech: Phone, PSP camera, marker tracking - Idea: Hunt new tribes of monsters through the camera and battle them on a card. - What it is: Third PSP entry with new tribes of Invizimals to hunt through the camera and battle on the marker trap card. - Technique: The marker card defines the arena origin, and creatures from different sets are rendered in the card's frame with turn-based battle animations. - Try it: Use two different image-marker cards to summon characters from two factions and have them fight turn-based battles on the table. Twist: use the distance between the two cards as a parameter for attack range. #### Reality Fighters — Novarama (2012) - Video: https://www.youtube.com/watch?v=AIDNJViTf4Q - Interaction: Face, Play, Tangible Objects - Platform & tech: Phone, PS Vita, AR cards, face capture - Idea: Put your own face on a fighter and brawl in AR in a real street or living room. - What it is: A PS Vita launch title in which players put their own photographed face on a fighter and brawl in AR on their real surroundings, filmed by the Vita's rear camera. - Technique: A photographed face is mapped onto a fighter's head, and the characters are rendered over the rear camera feed anchored by an AR card. - Try it: Take a photo of your face with the front camera, paste it onto the head of a tiny AR fighter, and fight a classmate's fighter on the table. Twist: the loser's face texture shows "scars". #### Invizimals: The Alliance / The Lost Kingdom — Novarama (2013) - Video: https://www.youtube.com/watch?v=s1c3QuXY95I - Interaction: Tangible Objects, Play - Platform & tech: Phone, Desktop, PS Vita, PS3, marker tracking - Idea: Catch monsters in AR on a handheld, then take them along into an adventure on the console. - What it is: PS Vita AR capture game (The Alliance) paired with a PS3 adventure (The Lost Kingdom) where creatures captured in AR travel between the handheld and the console world. - Technique: Creatures captured in the handheld AR mode are saved to a shared profile and loaded as playable characters in a separate console game, linking two devices through data. - Try it: "Catch" a character in phone AR and save it as JSON, then load it as a controllable character in a Unity scene on a computer. Twist: after leveling it up on the computer, the character's look in phone AR changes too. ### NuFormer *Projection mapping studio* Dutch studio that was among the first to tour large-scale 3D video mapping on buildings for brands and cities around 2009–2012, from Frankfurt and Madrid to Yerevan and Tokyo. It later added live motion capture, interactive window projection and sky projection to its toolkit. #### 3D Video Mapping – Volvo, Frankfurt — NuFormer (2009) - Video: https://www.youtube.com/watch?v=ar-sVuv63dM - Interaction: Projection, Location & City - Platform & tech: Projection, projection mapping, 3D animation - Idea: A facade becomes believable 3D content only when the virtual geometry is modelled from the real one; AR occlusion and anchoring work the same way. - What it is: A city building in Frankfurt appears to crack, fold and rebuild itself as 3D animation is projected exactly onto its facade for a Volvo campaign with Saatchi & Saatchi. - Technique: The building is surveyed and rebuilt as a 3D model, animation is rendered from the audience's viewpoint and projected with several aligned high-power projectors. - Try it: Scan a campus wall with a LiDAR phone, rebuild it as a simple mesh in Reality Composer or Unity, and animate its bricks falling out and snapping back in AR. Twist: the bricks only move when someone walks past the wall. #### 3D Video Mapping Interactivity Test — NuFormer (2011) - Video: https://www.youtube.com/watch?v=3HrGkyoCGmg - Interaction: Projection, Hands & Body - Platform & tech: Projection, projection mapping, body tracking - Idea: Letting the audience drive a building-sized image turns passive watching into play, which is what makes location AR memorable. - What it is: A test in which spectators in front of a mapped building control the projected content with their movements, so logos and objects on the facade react to the crowd. - Technique: Cameras or depth sensors likely track people in front of the building and send their positions to a real-time engine that renders the mapped content. - Try it: Build a WebXR or Lens Studio scene anchored to a building facade where virtual balls bounce off the wall and follow the phone holder's body position. Twist: two phones must cooperate to keep one ball alive. #### 3D Mocap Mapping (with Motek Entertainment) — NuFormer (2012) - Video: https://www.youtube.com/watch?v=hwROCMIOkZo - Interaction: Projection, Hands & Body, Performance - Platform & tech: Projection, projection mapping, motion capture, real-time rendering - Idea: A live puppeteer behind a virtual giant makes a projection feel alive; AR characters gain the same presence when a hidden human drives them. - What it is: A giant 3D character projected onto a building is driven live by a performer in a motion-capture suit, so it can talk to, wave at and play with the people standing below. - Technique: Motion capture from Motek streams skeleton data into a real-time engine that renders the character onto the pre-mapped facade. - Try it: Anchor a rigged avatar to a wall in AR and drive it live from a second phone running body tracking (ARKit or MediaPipe) over a WebSocket. Twist: the audience can only see the puppeteer's avatar, never the puppeteer. #### 3D Video Mapping – Matenadaran, Yerevan — NuFormer (2012) - Video: https://www.youtube.com/watch?v=vjL8zCveLdc - Interaction: Projection, Location & City, Information & UI - Platform & tech: Projection, projection mapping, 3D animation - Idea: The best overlay tells the story of the very building it covers, which is the strongest reason to anchor AR content to a specific place. - What it is: For Yerevan's year as UNESCO World Book Capital, the facade of the Matenadaran manuscript library is covered with animated illuminated manuscripts, letters and ornaments from Armenia's literary history. - Technique: Motifs from the library's own manuscripts are animated in 3D and mapped onto the building's model so the ornaments follow its real arches and columns. - Try it: Pick a campus building, collect three archival images about its history, and anchor them as animated AR layers that unfold from its doors and windows. Twist: each layer appears only at the spot where the photo was originally taken. #### Interactive Window Projection – Zierikzee — NuFormer (2013) - Video: https://www.youtube.com/watch?v=vXLq8LAtaE0 - Interaction: Projection, Hands & Body - Platform & tech: Projection, rear projection, people tracking - Idea: An ordinary window becomes a portal the moment it notices you, a small-scale idea that transfers directly to AR shop fronts. - What it is: A shop window in the Dutch town of Zierikzee becomes a large video screen whose animated content follows and reacts to the passers-by moving in front of it. - Technique: Rear-projection film on the glass shows the content, while a camera tracks moving people and objects to trigger the interaction. - Try it: Anchor an AR portal on a real shop window with image tracking and fill it with fish that swim toward the viewer's phone position. Twist: the fish scatter if two viewers stand close together. ### Obscura Digital *Creative technology studio (monumental projection)* San Francisco studio, with Travis Threlkel as chief creative officer, that projected onto some of the world's most recognisable buildings, including the Guggenheim, the Sydney Opera House, the Empire State Building and St Peter's Basilica. The Madison Square Garden Company acquired it in 2017 to develop what became Sphere in Las Vegas. #### CueLight interactive pool table — Obscura Digital (2009) - Video: https://www.youtube.com/watch?v=e0_wloTBxUM - Interaction: Projection, Play, Tangible Objects - Platform & tech: Projection, overhead projection, camera tracking - Idea: Augmenting an existing game with visual feedback keeps its rules and adds delight; the best AR games often start from a familiar physical game. - What it is: A pool table in a Hard Rock Hotel & Casino suite in Las Vegas has projected graphics on its felt that follow the balls: ripples, trails and effects react to every shot, turning a classic game into a light show. - Technique: An overhead camera likely tracks the balls on the felt and an overhead projector draws effects at their positions in real time. - Try it: Point a phone down at a table-top game (marbles, coins or a mini pool set) and use colour tracking in OpenCV.js or Lens Studio to draw AR trails behind each piece. Twist: trails become walls that later pieces bounce off. #### Guggenheim YouTube Play — Obscura Digital (2010) - Video: https://www.youtube.com/watch?v=hSyoEDz2MAg - Interaction: Projection, Location & City, Voice & Sound - Platform & tech: Projection, projection mapping, real-time graphics - Idea: Mapping inside and outside at once makes one building a continuous experience; AR can link an exterior view with what happens inside. - What it is: For the 2010 YouTube Play biennial awards, Obscura mapped the Guggenheim Museum's spiral exterior and, at the same time, the entire interior rotunda, surrounding the audience with real-time graphics inside Frank Lloyd Wright's building. - Technique: Seamless projection on the curved exterior plus a 360° projection on the rotunda, driven by custom real-time video graphics systems. - Try it: Make a two-part AR piece: an effect on the outside of a building that hints at something, and a second effect inside that answers it (8th Wall or AR Foundation). Twist: what visitors do inside changes what people outside see. #### YouTube Symphony Orchestra, Sydney Opera House — Obscura Digital (2011) - Video: https://www.youtube.com/watch?v=EOFuzE42z58 - Interaction: Projection, Voice & Sound, Performance - Platform & tech: Projection, audio-reactive projection, live digital painting - Idea: Visuals that react to live music tie a large audience to one shared moment; sound-reactive AR can make co-located viewers feel the same beat. - What it is: For the YouTube Symphony Orchestra grand finale on 20 March 2011, Obscura turned the Sydney Opera House 'inside out and outside in': audio-reactive graphics and live digital painting were projected on the concert hall interior and on the famous sails at the same time. - Technique: Audio from the orchestra fed real-time graphics engines, while artists painted digitally live; outputs were split to interior and exterior projector arrays. - Try it: Build a shared AR scene (Lens Studio Connected Lenses or WebXR with WebSockets) where music from one phone drives shapes that everyone sees on the same real wall. Twist: one participant can paint strokes that then pulse to the music. #### Fiat Lux: Illuminating Our Common Home — Obscura Digital (2015) - Video: https://www.youtube.com/watch?v=jUlxYv3egLo - Interaction: Projection, Location & City - Platform & tech: Projection, large-scale projection, architectural projection - Idea: Placing living nature on the most symbolic human architecture creates a strong contrast; AR overlays gain meaning from what the place already stands for. - What it is: On 8 December 2015, the façade and dome of St Peter's Basilica in Vatican City were covered with projected images of animals, oceans, forests and people, inspired by Pope Francis's encyclical on the environment. It was produced with the Oceanic Preservation Society; the video is their upload. - Technique: Multiple high-power projectors aligned to the basilica's façade and dome, playing a pre-rendered photographic sequence without the need for detailed geometric deformation. - Try it: Collect five nature photos from students, then place them in AR (WebXR or Lens Studio) as a slow slideshow on the most formal building on campus. Twist: the slideshow advances only when people nearby stay silent. #### Projecting Change: Empire State Building — Obscura Digital (2015) - Video: https://www.youtube.com/watch?v=T6FQvFJG9dc - Interaction: Projection, Location & City, Information & UI - Platform & tech: Projection, large-scale projection, projection mapping - Idea: Scaling an animal to the size of a skyscraper makes an abstract statistic unforgettable; AR can use scale shifts to make data emotional. - What it is: Manta rays 40 storeys high, a snow leopard and swimming whales were projected across the Empire State Building for the film Racing Extinction. The work was created by director Louie Psihoyos and Obscura's Travis Threlkel to draw attention to endangered species. - Technique: A bank of high-lumen projectors on a nearby rooftop covered the tower's upper floors, with imagery pre-distorted for the tall, stepped façade. - Try it: Pick an endangered animal and use 8th Wall or Lens Studio to place it at building scale on a nearby tower, with one number about its population shown beside it. Twist: the animal fades a little each time someone takes a screenshot. ### Onionlab *Creative studio (projection mapping and audiovisual installations)* Barcelona studio working across projection mapping, music videos and immersive spaces. It is known for stereoscopic 3D mapping shows such as Diplopia and Axioma and for façade projections on the Museo del Prado and Madrid's Palacio de Cibeles. #### Santa Eulalia 3D Projection Mapping — Onionlab (2011) - Video: https://www.youtube.com/watch?v=QbEkt4UBmWA - Interaction: Projection, Location & City - Platform & tech: Projection, 3D projection mapping, architecture - Idea: Treating a façade as a material that can be cut and sewn ties the illusion to the brand story; AR effects on buildings feel stronger when the building 'behaves' like the subject. - What it is: To open the new Passeig de Gràcia store of Barcelona fashion house Santa Eulalia (founded 1843), Onionlab mapped the building's façade: stone seems to fold, open like fabric and reveal tailoring patterns and sewing tools. - Technique: 3D projection mapping aligned to a model of the façade, with animations of deforming geometry rendered from the audience's central viewpoint. - Try it: Take a photo of a building front, rebuild its main planes in Blender, and make an AR effect with 8th Wall or Lens Studio City Landmarker where the façade peels open like cloth. Twist: the fabric pattern is drawn live by a student on an iPad. #### Evolució — Onionlab (2013) - Video: https://www.youtube.com/watch?v=mhQ50QFsnMc - Interaction: Projection, Perception & Effects - Platform & tech: Projection, 3D projection mapping, generative graphics - Idea: Abstract geometry that follows the real edges of a building teaches the eye where the structure is; AR can use simple lines to reveal architecture. - What it is: An abstract building projection about evolution as transformation over time, first shown at the Mapping Festival in Geneva and then at Signal Festival in Prague. Lines, grids and volumes build and break the architecture to a tightly synced soundtrack. - Technique: Content is animated on a 3D model of the façade so every line snaps to real edges, then projected with edge-blended projectors. - Try it: Use WebXR or AR Foundation to trace the edges of a room with glowing lines that grow in sequence to music. Twist: the lines evolve from straight to organic curves over one minute. #### Diplopia — Onionlab (2015) - Video: https://www.youtube.com/watch?v=OJId9dEOZ2Q - Interaction: Projection, Perception & Effects - Platform & tech: Projection, anaglyph 3D, stereoscopic projection mapping - Idea: Showing depth through two slightly different images reveals the core mechanism behind every stereo AR headset. - What it is: The first anaglyph 3D mapping made in Spain, shown at the opening of the International Mapping Festival of Girona: with red-cyan glasses, the audience sees shapes pop out of the façade. The piece is about binocular vision, how two eyes merge two images into one object. - Technique: Left- and right-eye renders offset by the interocular distance are colour-encoded in red and cyan and projected together, then decoded by paper glasses. - Try it: Render the same AR object for a left and right eye in a WebXR or three.js scene and export an anaglyph image, then view it through cheap red-cyan glasses. Twist: make an object that is only visible to one eye and see how the brain reacts. #### Axioma — Onionlab (2016) - Video: https://www.youtube.com/watch?v=q2wT_WgOHJ4 - Interaction: Projection, Perception & Effects - Platform & tech: Projection, stereoscopic 3D, projection mapping - Idea: Pure geometry floating in front of a real façade shows how depth cues alone, without story, can hold an audience; AR primitives can be the content. - What it is: A stereoscopic mapping show for Llum BCN festival in Barcelona: with glasses handed out before the show, points, lines, planes and polyhedra appear to float in front of the building, an interpretation of the axioms of geometry. - Technique: Stereo pairs of geometric animations are projected onto the façade; the screen plane is set at the wall so objects appear in front of and behind it. - Try it: In a headset or HoloKit, place a set of floating geometric primitives just in front of a real wall and animate them from point to line to plane to solid. Twist: each step only happens when two users look at the same object. #### Museo del Prado 200 years — Onionlab (2018) - Video: https://www.youtube.com/watch?v=H9cjG69TfNM - Interaction: Projection, Location & City, Performance - Platform & tech: Projection, 3D projection mapping, fireworks, aerial performance - Idea: A museum can show what is inside on its outside; AR can turn any building's walls into a preview of its contents. - What it is: For the 200th anniversary of the Museo del Prado in Madrid, Onionlab mapped the museum's façade as part of the show 'Un lugar de memoria y futuro', with an aerial performance by La Fura dels Baus and fireworks. The stone front appears to breathe, crack and fill with paintings from the collection. - Technique: Large-scale architectural projection mapping synchronised with pyrotechnics and aerial performers for a live event. - Try it: Choose a library or museum near you and build an AR layer (8th Wall or Lens Studio) that shows three objects from its collection emerging from the façade. Twist: the objects appear in the windows closest to where they are really kept inside. ### Oscar Falmer *AR developer; former Apple AR evangelist and Snap AR developer relations lead* French developer who made TweetReality, the first AR Twitter client, and a series of ARKit concept apps in 2017-2018, then spent years as an AR evangelist at Apple and Snap. #### TweetReality — Oscar Falmer (2017) - Video: https://www.youtube.com/watch?v=5eUai9bed5Y - Interaction: Information & UI, Shared & Social - Platform & tech: Phone, ARKit, Twitter API - Idea: Unroll a social feed into the room so scrolling becomes turning your body. - What it is: The first augmented-reality Twitter client: your timeline appears as floating cards arranged around you in space, which you browse by moving the phone. - Technique: ARKit world tracking lays out Twitter API content as floating 3D cards around the user, with tap-to-open, like and retweet buttons in space. - Try it: Lay out ten messages from a class chat as cards in a circle around the viewer in AR. Twist: older messages drift farther away and fade. #### AR Business Card Concept — Oscar Falmer (2018) - Video: https://www.youtube.com/watch?v=Jq98OEXJSr8 - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, ARKit 2, Image Tracking - Idea: A paper card that opens into a small interactive portfolio. - What it is: Pointing an iPhone at a printed business card makes a video, contact buttons and social links unfold around it in AR. - Technique: ARKit 2 image detection and tracking recognises the card and anchors video planes and tappable UI to its edges. - Try it: Design your own business card and build an image-tracked AR layer with three buttons that each show something about you. Twist: the layer looks different when the card is upside down. #### AR Restaurant Menu Concept — Oscar Falmer (2018) - Video: https://www.youtube.com/watch?v=E2K052WwzpM - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, ARKit 2, Image Tracking - Idea: Let diners preview the actual plate on their own table before ordering. - What it is: Scanning a restaurant menu makes 3D models of the dishes appear on the table so diners can see each plate at real size before ordering. - Technique: ARKit image tracking recognises the menu page, and tapping an item places a photogrammetry-style 3D dish on the detected table plane. - Try it: Photoscan three foods from the cafeteria and build an AR menu that places them on a table. Twist: show each dish's calories or carbon footprint as a floating label. #### Leap Motion-style hand tracker with Meta Ray-Ban glasses — Oscar Falmer (2026) - Video: https://www.youtube.com/watch?v=OPrTNGddcZ4 - Interaction: Hands & Body, Gaze & Attention - Platform & tech: Wearable, Web, Meta Ray-Ban, MediaPipe, WebSocket - Idea: Turn camera glasses into a hand controller for any screen, even one with no webcam. - What it is: Video from Meta Ray-Ban glasses is streamed to a phone and on to a laptop webpage, where hand gestures seen from the wearer's point of view pinch and rotate 3D models on screen. - Technique: The Wearables Developer Access Toolkit streams glasses video to an iPhone, which forwards frames over WebSocket to a web page running MediaPipe hand tracking. - Try it: Stream a phone camera held at eye level to a laptop and use MediaPipe hands to rotate a 3D model with a pinch. Twist: add a second gesture that switches between models. #### Meta glasses fitness coach — Oscar Falmer (2026) - Video: https://x.com/OscarFalmer/status/2018276017089990824 - Interaction: Hands & Body, Voice & Sound - Platform & tech: Wearable, Meta Wearables SDK, body tracking, Ray-Ban Meta - Idea: A coach that watches your form from your own point of view. - What it is: A proof of concept counts squats in real time with body tracking from glasses video and could add hands-free coaching such as form cues, goals and exercise variations. - Technique: Frames from the glasses camera (likely looking at a mirror or the legs) run through a pose model, and a simple state machine counts repetitions and triggers spoken cues. - Try it: Use a webcam pose model in the browser to count squats and speak encouragement at every fifth rep; show the count as large AR text. Twist: the coach gets more sarcastic as your form gets worse. ### Paul Friedlander *Kinetic light artist (former physicist)* British artist who trained in physics and has spent decades building kinetic light sculptures in which spinning strings, lit by rapidly changing coloured strobe light, form floating waves, spirals and 'cosmic' shapes. His work has appeared at the Phaeno Science Centre, light festivals in Jerusalem, Skopje and Vilnius, and music festivals such as Shambala. #### Light Harp (Shambala Festival) — Paul Friedlander (2017) - Video: https://www.youtube.com/watch?v=3cMw1ZAor54 - Interaction: Voice & Sound, Perception & Effects, Location & City - Platform & tech: Projection, spinning string, stroboscopic light, sound - Idea: Turning an instrument into light makes sound visible; AR music tools can show vibration as form instead of as a flat visualiser. - What it is: At Shambala Festival in England, a row of glowing, vibrating strings stands in the night like a giant harp of light, their coloured wave shapes shifting in front of the festival crowd. - Technique: Several strings are driven into motion and lit by synchronised coloured strobes so each string appears as a standing wave of light (link to live sound likely). - Try it: Build an AR 'light harp' in Unity AR Foundation or WebXR: five virtual strings in the room that vibrate as glowing waves and play notes when a hand passes through. Twist: two phones share the harp so two people can play together. #### Origin of String Theory (Phaeno) — Paul Friedlander (2017) - Video: https://www.youtube.com/watch?v=1j0SblgkNbQ - Interaction: Perception & Effects, Information & UI - Platform & tech: Projection, spinning string, stroboscopic light, science centre commission - Idea: A physics idea becomes understandable when you can watch it happen in 3D; AR science lessons can make abstract waves visible around a real object. - What it is: The smallest of several works commissioned by the Phaeno Science Centre in Wolfsburg, seen on test in the studio: a single glowing string traces changing wave shapes that echo the idea of vibrating strings in physics. - Technique: A motor-driven string is lit by stroboscopic light timed to its rotation so the wave pattern appears frozen or slowly drifting. - Try it: Make a WebXR science demo in which students stretch a virtual string between two real points and pluck it to see standing waves of 1, 2 or 3 nodes. Twist: link the number of nodes to the note it plays. #### Spinning Cosmos — Paul Friedlander (2017) - Video: https://www.youtube.com/watch?v=hoGE_-4kvA0 - Interaction: Perception & Effects - Platform & tech: Projection, spinning string, stroboscopic light, kinetic sculpture - Idea: A simple physical mechanism plus darkness can create a sense of scale far larger than the object; AR scenes gain the same power from a dark, reduced surrounding. - What it is: A dark room in which spinning, strobe-lit strings open into spirals, rings and nebula-like forms, like a small cosmos turning in front of the visitor. - Technique: Several strings rotated by motors at controlled speeds are lit by synchronised strobes so their moving shapes appear still or slowly morphing (exact control system likely custom). - Try it: Darken a room, then use a phone AR app (Reality Composer or Lens Studio) to place a slowly turning spiral galaxy made only of lines around a small real object. Twist: the galaxy only appears when all room lights are off. #### Light Wave Tree (Skopje) — Paul Friedlander (2019) - Video: https://www.youtube.com/watch?v=IMBguNvpoHc - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, spinning string, stroboscopic light, outdoor installation - Idea: An organic form built from pure motion reads as alive; AR plants and creatures can also be made from motion rather than from detailed models. - What it is: A kinetic light installation for the Skopje Light Art District in North Macedonia: spinning, colour-lit strings form a glowing 'tree' of waves in a public space at night. - Technique: Multiple spinning strings are arranged as branches and lit by synchronised coloured strobes to create layered wave shapes. - Try it: Grow an AR tree in Lens Studio or Unity out of animated line waves (no meshes), placed on a real spot outside. Twist: the tree grows a new branch every time someone new looks at it. #### World Lines (Colour Light Waves) — Paul Friedlander (2022) - Video: https://www.youtube.com/watch?v=4BxK173q2XI - Interaction: Perception & Effects, Voice & Sound - Platform & tech: Projection, spinning string, stroboscopic colour light, kinetic sculpture - Idea: Persistence of vision can draw a 3D shape in thin air; many 'holographic' AR effects are just this — a fast-moving line that the eye fuses into a volume. - What it is: A short film of Friedlander's 'Colour Light Waves', his own invention: a spinning string lit by fast-changing coloured light forms glowing waves and knots that float in the dark. It mixes live performances at the Bookery Gallery in London (2021, 2022) with earlier installations in Valencia, Jerusalem, Shambala and Slane Castle. - Technique: A motor spins a taut string into a standing-wave shape while stroboscopic, colour-changing light synchronised to the rotation freezes or shifts the visible pattern. - Try it: Build a WebXR or Three.js effect that draws a single glowing line moving fast along a sine wave in space, leaving a short trail so the eye sees a surface. Twist: map the wave's frequency to music from the microphone. ### Pipilotti Rist *Video and installation artist* Swiss artist famous for immersive, colour-saturated video installations. Her [AR]T Walk piece International Liquid Finger Prayer (2019) sent groups racing through city streets after a singing, shimmering AR form. #### Ever Is Over All — Pipilotti Rist (1997) - Video: https://www.youtube.com/watch?v=IDydC2EYVDo - Interaction: Projection, Gaze & Attention, Perception & Effects - Platform & tech: Projection, video projection, slow motion - Idea: Two images meeting in a corner make the viewer stitch cause and effect together; AR can split a story between two surfaces so the viewer's head turns do the editing. - What it is: Two overlapping projections fill a corner: on one side a woman in a blue dress walks down a street smashing car windows with a long flower while a policewoman salutes her; on the other, slow shots of a field of the same flowers. - Technique: Two large video projections overlap at a corner, and slow motion plus a hummed soundtrack make the violent act feel light and dreamlike. - Try it: Shoot two 30-second clips, one of an action and one of an object, and pin them in AR on two walls meeting at a corner so they overlap. Twist: the second clip plays only when the user looks at the corner line. #### Homo sapiens sapiens — Pipilotti Rist (2005) - Video: https://www.youtube.com/watch?v=2lanq0jk1mQ - Interaction: Projection, Location & City, Gaze & Attention - Platform & tech: Projection, ceiling projection, church, beds - Idea: Putting the image where paintings of heaven used to be reuses the building's own logic; AR works well when content respects where people already look in a place. - What it is: In the Venetian church of San Stae, visitors lie on cushioned beds and watch a video projected across the ceiling, showing two women in a lush paradise-like landscape. - Technique: Projectors aimed at the vaulted ceiling fill it with video, and horizontal beds place the audience in the right viewing angle. - Try it: Find a building part people already look up at (a stairwell, dome or tree canopy) and pin a looping AR video to it that reinterprets what used to be there. Twist: the video changes when two users look up at it together. #### Pour Your Body Out (7354 Cubic Meters) — Pipilotti Rist (2008) - Video: https://www.youtube.com/watch?v=89vgdELbVyQ - Interaction: Projection, Perception & Effects, Hands & Body - Platform & tech: Projection, video projection, edge blending, round sofa - Idea: Giving people a place to lie down changes how long they stay; AR experiences gain depth when they design the body's posture, not only the content. - What it is: MoMA's atrium becomes a projection room: huge, saturated videos of bodies, fruit and water fill the walls while visitors lie on a giant round sofa in the middle. - Technique: Multiple high-lumen projectors blend edge to edge across the atrium walls, and a soft circular seat invites people to recline and look up and around. - Try it: Make a lying-down AR piece: the user lies on the floor, points the phone at the ceiling, and a slow video world unfolds across the real ceiling plane. Twist: the video speed follows how still the phone is held. #### Pixel Forest — Pipilotti Rist (2016) - Video: https://www.youtube.com/watch?v=yRnDHu0Fmtk - Interaction: Perception & Effects, Spatial Mapping, Hands & Body - Platform & tech: Projection, addressable LEDs, resin, video mapping - Idea: Exploding a screen into its pixels lets people walk inside an image; volumetric AR content can be shown as sparse points that surround the body. - What it is: Thousands of LED lights, each inside a hand-made resin crystal, hang on strings through a dark room; together they play a video, but visitors walk between the pixels instead of looking at a screen. - Technique: Addressable RGB LEDs are hung in a 3D grid and driven like a very low-resolution display, with video mapped onto their positions. - Try it: Sample a short video into a 20×20×10 grid of glowing points in WebXR and let students walk through the room-sized image. Twist: points near the viewer brighten and grow, so your body leaves a trail through the picture. #### International Liquid Finger Prayer — Pipilotti Rist, Apple (ARKit / visionOS teams) (2019) - Video: https://www.youtube.com/watch?v=2Iza32cgqpg - Interaction: Location & City, Play, Voice & Sound - Platform & tech: Phone, ARKit, spatial audio - Idea: Chase a singing, glittering blob through the city. - What it is: On Apple's [AR]T Walk, a shimmering liquid form bounces, taunts and sings through the city while participants race to catch it on their phones. - Technique: Location-anchored ARKit animation with spatial audio that moves along a path, turning the walk into a chase. - Try it: Make an AR creature that flees along a path when the phone gets within two metres, and plays a sound that gets louder as you close in. Twist: it can only be caught when two players corner it together. ### SCE London Studio *Sony first-party studio (EyeToy, EyePet, Wonderbook)* Sony's London Studio pioneered camera games from EyeToy to EyePet and the Wonderbook, an AR book peripheral for PS3. #### EyePet — SCE London Studio (2009) - Video: https://www.youtube.com/watch?v=ORYSSQZTzbE - Interaction: Hands & Body, Tangible Objects, Drawing & Making - Platform & tech: Desktop, PlayStation 3, PlayStation Eye, Magic Card - Idea: An AR pet that reacts to your hand gestures and turns what you draw on paper into toys. - What it is: A virtual monkey-like pet appears on the living-room floor through the PlayStation Eye; players stroke, feed and wash it with their hands and draw toys on paper that it brings to life. - Technique: A fixed camera uses a marker card to locate the floor, and frame-differencing motion detection lets the player's hand movements near the pet count as touches. - Try it: Use a webcam and frame differencing to detect hand movement in one region of the image, and make an on-screen virtual pet react when it is 'stroked'. Twist: draw a circle on paper and have the pet play with it as a ball. #### EyePet (PlayStation Move edition) — SCE London Studio (2010) - Video: https://www.youtube.com/watch?v=TztCY2iz450 - Interaction: Tangible Objects, Hands & Body, Play - Platform & tech: Desktop, PlayStation Move, PlayStation Eye - Idea: The Move controller in your hand becomes an AR hairdryer or paintbrush for looking after your pet. - What it is: EyePet updated for the PlayStation Move controller, which becomes tracked tools such as a hairdryer, shower head or paint brush in the pet's AR world. - Technique: The Move controller's glowing ball is tracked by the camera for position and its IMU for orientation, so it can be rendered as swappable virtual tools in the pet's space. - Try it: Use a phone as a prop controller, map its orientation to AR tools such as a hairdryer or brush, and groom a pet. Twist: give each tool its own distinctive sound feedback. #### EyePet & Friends — SCE London Studio (2011) - Video: https://www.youtube.com/watch?v=XNKnuY8fvOY - Interaction: Shared & Social, Hands & Body, Play - Platform & tech: Desktop, PlayStation Move, PlayStation Eye - Idea: Two AR pets play together in your living room. - What it is: Two players can each raise a pet that play together in the living room, with new costume, drawing and mini-game modes. - Technique: Two tracked controllers and pet instances share the camera-based floor frame, letting two players' pets interact in the same augmented video. - Try it: Put two pets in the same AR view, each controlled by a different player, and have them interact based on their distance. Twist: give each pet its own personality so the two come into conflict. #### Wonderbook: Book of Spells — SCE London Studio (2012) - Video: https://www.youtube.com/watch?v=Y6ms06Ujj88 - Interaction: Tangible Objects, Hands & Body, Play - Platform & tech: Desktop, Wonderbook, PlayStation Move, PlayStation Eye - Idea: A blank physical book turns into a pop-up spellbook on screen, with the controller as your wand. - What it is: Written with J.K. Rowling's Pottermore, a blank marker book becomes a pop-up spell book on screen; players wave the Move controller as a wand to cast spells that affect the pages and room. - Technique: Each page of the book carries a distinct marker, so page identity and pose drive which pop-up scene is rendered, while the tracked Move wand gestures cast spells into it. - Try it: Put a different image marker on each page of a small booklet so turning the page pops up a matching 3D scene in AR. Twist: use the phone as a wand, waving it above the page to trigger changes. #### Wonderbook: Book of Potions — SCE London Studio (2013) - Video: https://www.youtube.com/watch?v=439opcIxxig - Interaction: Tangible Objects, Hands & Body, Play - Platform & tech: Desktop, Wonderbook, PlayStation Move - Idea: Grind and pour ingredients with a controller over a real book to brew AR potions. - What it is: A second Harry Potter Wonderbook where players brew potions in a virtual cauldron sitting on the book, grinding and pouring ingredients with the Move controller. - Technique: The book marker anchors a virtual cauldron, and controller motion patterns such as grinding and pouring are recognized from IMU data to add ingredients. - Try it: Put an AR cauldron on an image marker and add different ingredients by shaking or tilting the phone (accelerometer), with colored smoke rising when the recipe is right. Twist: the ingredients must be added in the correct order. ### SCRN *Shader developer and machine-learning tinkerer (VRChat)* A VRChat creator who implements neural networks entirely inside Unity fragment shaders: YOLOv4-tiny object detection, pix2pix, face-landmark tracking, a transformer translator and a voice-viseme classifier, plus raymarching experiments. Everything is open source. #### Pix2Pix in a Fragment Shader — SCRN (2020) - Video: https://x.com/SCRNinVR/status/1317299768301735936 - Source code: https://github.com/SCRN-VRC/Pix2Pix-in-a-Fragment-Shader - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, Headset, Unity, HLSL, pix2pix, VRChat - Idea: A GAN can be just another material on a surface. - What it is: A generative image-to-image network running entirely in a Unity shader turns simple line drawings into generated Pikachu, Mario or tree images in real time inside VRChat. - Technique: A pix2pix U-Net generator cut to a quarter of the original size is implemented layer by layer in Cg/HLSL fragment passes over render textures, with weights trained in Keras and stored in textures, so inference happens as part of rendering. - Try it: Let people doodle on a virtual canvas in AR and show a style-transferred version beside it using an on-device image-to-image model. Twist: the canvas reads a real paper drawing through the camera instead. #### 3D Face Landmarks in Shaders — SCRN (2021) - Video: https://x.com/SCRNinVR/status/1404187468153327617 - Source code: https://github.com/SCRN-VRC/3D-Face-Landmark-in-UnityCG-HLSL - Interaction: Face, Perception & Effects - Platform & tech: Desktop, Headset, Unity, HLSL, MediaPipe Facemesh, VRChat - Idea: Put the whole face-tracking pipeline on the GPU so an avatar can see its owner without plugins. - What it is: A VRChat avatar mirrors a real face in real time, including blinks, brow movements and eye direction, tracked by networks that run inside shaders. - Technique: MediaPipe's Facemesh and Iris models are rewritten as chained UnityCG/HLSL fragment passes that output 3D landmarks, which then drive blend shapes for blinking, brows and gaze. - Try it: Use ARKit face tracking on a phone to drive a particle mask whose emission follows your blinks and brow raises. Twist: the mask sheds particles only when you frown. #### Voice Recognition with Shaders — SCRN (2021) - Video: https://x.com/SCRNinVR/status/1367343416644886531 - Source code: https://github.com/SCRN-VRC/Voice-Recognition-Shader - Interaction: Voice & Sound, Shared & Social - Platform & tech: Desktop, Headset, Unity, HLSL, VRChat - Idea: Recover speech from lip-sync signals when the raw audio is out of reach. - What it is: A small robot dog attached to a VRChat avatar obeys spoken commands, recognised by a shader-based neural network that only sees the lip-sync visemes the platform exposes. - Technique: Viseme weights over time form the input sequence to a small classifier implemented in fragment shaders; its output drives the robot dog's animation, showing that visemes retain enough information to recognise words. - Try it: Map spoken keywords (via on-device speech recognition) to three VFX Graph spells that appear in front of the speaker in AR. Twist: the spell only works if two people say it together. #### YOLOv4-tiny in Fragment Shaders — SCRN (2021) - Video: https://x.com/SCRNinVR/status/1380238589238206465 - Source code: https://github.com/SCRN-VRC/YOLOv4-Tiny-in-UnityCG-HLSL - Interaction: Perception & Effects, Information & UI - Platform & tech: Desktop, Headset, Unity, HLSL, YOLOv4-tiny, VRChat - Idea: Run a neural network where scripts are not allowed: inside the shader itself. - What it is: An object detector written entirely in Unity fragment shaders draws labelled boxes around people and objects in a live video feed inside VRChat. - Technique: Each YOLOv4-tiny layer (convolutions, pooling, activations) is re-implemented as a fragment-shader pass that writes activations into render textures, chained frame by frame, with the trained weights baked into textures. - Try it: Run a tiny detector on the AR camera image (Unity Sentis or a shader port) and attach a VFX Graph burst to every detected cup. Twist: the effect changes depending on how many objects of the same class are visible. #### Raymarching with ShadowCaster — SCRN (2022) - Video: https://x.com/SCRNinVR/status/1536860644758245379 - Source code: https://github.com/SCRN-VRC/Raymarching-with-ShadowCaster - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL, raymarching, SDF - Idea: Make distance-field shapes behave like real objects by marching them in the shadow pass too. - What it is: A raymarched SDF object inside a box casts correct shadows and sits in the scene depth like a normal mesh. - Technique: The SDF is also raymarched in the ShadowCaster pass so it casts shadows, and the resulting depth is read by the other passes so the shape writes correct depth and receives shadows. - Try it: Raymarch a blobby SDF sculpture on a real table in AR and make it write depth so real-world occlusion and virtual shadows line up. Twist: let the blobs merge when two phones point at the same spot. ### Scatter / Depthkit (James George, Alexander Porter, Yasmin Elayat) *Volumetric filmmaking studio and makers of the Depthkit capture software* Grew out of the open-source RGBDToolkit that paired a depth sensor with a DSLR. Scatter makes volumetric documentaries such as CLOUDS, Zero Days VR and The Changing Same, and sells Depthkit, the capture tool behind many volumetric AR and VR pieces, including Terminal 3. #### CLOUDS — Scatter / Depthkit (James George, Alexander Porter, Yasmin Elayat) (2014) - Video: https://www.youtube.com/watch?v=KefV_ZAsOxo - Interaction: Perception & Effects, Information & UI - Platform & tech: Desktop, Headset, RGBDToolkit, Kinect, openFrameworks - Idea: Film people as data, so the documentary itself can be rearranged and explored like code. - What it is: An interactive generative documentary about code and culture: dozens of creative coders filmed as flickering depth-camera point clouds, surrounded by live visual systems, with a story engine choosing what comes next. - Technique: Interviews were shot with RGBD (a Kinect mounted on a DSLR, calibrated with the open-source RGBDToolkit), and an openFrameworks story engine sequences clips and generative scenes. - Try it: Record a classmate's 30-second answer with a LiDAR phone as a point cloud and place it in AR next to the question written in 3D. Twist: dissolve the point cloud when the answer ends. #### Zero Days VR — Scatter / Depthkit (James George, Alexander Porter, Yasmin Elayat) (2017) - Video: https://www.youtube.com/watch?v=E_NZEdeh2cA - Interaction: Information & UI, Perception & Effects, Portals & Worlds - Platform & tech: Headset, Depthkit, Unity, VR - Idea: Give an invisible event a body and a space you can stand in. - What it is: A VR documentary about the Stuxnet cyberweapon that places you inside the invisible world of computer code, travelling through data centres and centrifuges at human scale. - Technique: Volumetric interviews captured with Depthkit are combined with abstract data visualisation in Unity to create a navigable space. - Try it: Pick an invisible process (a text message being sent, Wi-Fi) and stage it in room-scale AR around the class as a journey. Twist: let the audience be the data packet. #### Depthkit Holoportation via webcam — Scatter / Depthkit (James George, Alexander Porter, Yasmin Elayat) (2021) - Video: https://www.youtube.com/watch?v=S3jLCI1S_BM - Interaction: Shared & Social, Hands & Body, Performance - Platform & tech: Headset, Phone, Depthkit, Azure Kinect, Unity - Idea: Stream a person, not a video rectangle. - What it is: A full-body volumetric person is captured with depth cameras and livestreamed as a hologram into AR and VR scenes for teleconferencing and performance. - Technique: Depthkit Studio fuses several depth sensors into a textured mesh sequence, compresses it as video and streams it to a Unity or web client that reconstructs the hologram. - Try it: Record a 10-second LiDAR point-cloud clip of a classmate and play it back life-size in phone AR in the hallway. Twist: make the hologram only visible from one side. #### The Changing Same — Scatter / Depthkit (James George, Alexander Porter, Yasmin Elayat) (2021) - Video: https://www.youtube.com/watch?v=tugi4v7S32o - Interaction: Portals & Worlds, Location & City, Performance - Platform & tech: Headset, Depthkit, Unity, VR - Idea: Time travel through one place to show that history repeats in cycles. - What it is: An episodic VR experience in which the participant travels through non-linear time and space to witness connected moments of racial injustice in the United States. - Technique: Volumetric performances captured with Depthkit are staged in Unity environments that transform between eras around the participant. - Try it: Choose one spot on campus and build a phone AR piece that shows three photos from different decades at the exact same viewpoint. Twist: make the transition happen when the viewer steps forward. #### The MetaBox — Scatter / Depthkit (James George, Alexander Porter, Yasmin Elayat) (2023) - Video: https://www.youtube.com/watch?v=AluvT6z0gbQ - Interaction: Shared & Social, Projection - Platform & tech: Projection, Headset, Depthkit, Igloo, VR - Idea: A room that turns whoever walks in into a hologram for someone far away. - What it is: Participants step into an Igloo immersive projection room and instantly appear as volumetric holograms to remote colleagues in VR, so body language crosses between the two spaces. - Technique: Depthkit volumetric capture inside the projection room streams participants into a shared VR scene, while the remote VR view is projected on the room's walls. - Try it: Set up a corner with a projector showing a remote classmate's video on the wall while your phone streams you to them. Twist: only show the remote person when both of you face the wall. ### Sila Sveta *Multimedia production studio (projection mapping, stage shows, holographic performance)* Studio founded in Moscow in 2007 that grew from festival projection mapping into large stage shows, product launches, concert tours and ceremonies worldwide. Its work combines 3D mapping, live dancers, kinetic stages and holographic screens, from the 2011 MIGZ festival to the Levitation performances and a holographic Swan Lake. #### Things are not what they seem (MIGZ Festival) — Sila Sveta (2011) - Video: https://www.youtube.com/watch?v=JkvKEtHl6XM - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, projection mapping, 3D animation - Idea: The title says it all: mapping is about making solid things look unstable, and AR can make the real world feel unsure of itself in the same way. - What it is: Sila Sveta's mapping for the MIGZ 3D Mapping Festival in Moscow's Gorky Park in September 2011: the building seems to break, shift and rebuild itself to music by Sasha Knyazev (Triangle Sun). - Technique: 3D deformation animations rendered against a model of the building are projected in alignment, with fake lighting to suggest moving volumes. - Try it: Make an AR effect in Lens Studio's world mesh or ARKit scene reconstruction that makes a real wall bulge, crack and reassemble to a beat. Twist: the wall only breaks where the viewer is looking. #### Porsche Macan: 3D Mapping & Dance Performance — Sila Sveta (2014) - Video: https://www.youtube.com/watch?v=Xadmwk2qAVE - Interaction: Performance, Hands & Body, Tangible Objects - Platform & tech: Projection, projection mapping, live dance, kinetic cube - Idea: Pairing a real dancer with a projected double creates a dialogue between body and image; AR performance can use the same trick with a virtual clone. - What it is: For the Russian premiere of the Porsche Macan at Barvikha Concert Hall, live dancers perform with their own projected copies and industrial motion graphics on a central podium and side screens; at the end the projected cube lifts to reveal the car. - Technique: Pre-recorded footage of the dancers is mapped onto a cube and screens and synchronised with a rehearsed live choreography. - Try it: Record a 30-second dance, then play it back as an AR clone next to the live dancer with body occlusion in Unity or Lens Studio, and choreograph a duet. Twist: the clone lags one beat behind, like an echo. #### Central Children's Store permanent 3D mapping — Sila Sveta (2015) - Video: https://www.youtube.com/watch?v=gzCXYFK6IfU - Interaction: Information & UI, Perception & Effects - Platform & tech: Projection, projection mapping, permanent installation, automated control - Idea: A flat wall can pretend to have depth if the animation invents it; AR on flat surfaces also works by drawing convincing fake depth. - What it is: A permanent indoor mapping in Moscow's Central Children's Store, billed as the largest of its kind: a colourful 3D animation about the history of Russia plays on a huge flat wall, and an automated system dims the store lights and starts each show. Year approximate (video uploaded 2016). - Technique: Trompe-l'oeil 3D animation is projected onto a large flat atrium wall, with a remote control system that dims the lighting and launches scheduled shows. - Try it: Pick a flat wall at school and make an AR 'window' in 8th Wall or Reality Composer that shows a deep 3D scene about the building's history. Twist: the scene starts only when the room lights are dimmed. #### Levitation 2 — Sila Sveta (2021) - Video: https://www.youtube.com/watch?v=IYvbl4KY4-A - Interaction: Performance, Perception & Effects, Hands & Body - Platform & tech: Projection, holographic screen, projection, choreography - Idea: Illusions of weightlessness depend on hiding the support and controlling the viewpoint; AR can make real people seem to levitate with the same discipline. - What it is: The second part of Sila Sveta's Levitation performance, which continues from the exact last frame of the first (seen more than 20 million times): a dancer appears to float and interact with objects and light in a space that mixes theatre, choreography and optical illusion. - Technique: A performer on a hidden rig works with projected or holographic imagery and a locked camera or audience viewpoint so the body appears to float (exact rig likely wires or a hidden platform). - Try it: Use ARKit or Lens Studio person segmentation to lift a live person off the floor on screen while hiding their feet with a virtual cloud. Twist: the person can only rise when they hold perfectly still. #### The holographic ballet Swan Lake — Sila Sveta (2026) - Video: https://www.youtube.com/watch?v=RfKX8A98s-I - Interaction: Performance, Perception & Effects - Platform & tech: Projection, holographic screens, projection, 270-degree stage - Idea: Wrapping layered holographic screens around dancers turns a flat stage into a volume; AR ballet can place swans and lakes in front of, beside and behind the dancer. - What it is: A 25-minute reinterpretation of Swan Lake as an immersive holographic ballet on a three-sided holographic set with a 270-degree view: dancers perform in a precise dialogue with layered visuals and large-scale projection. Year is the video upload year; the premiere date is not confirmed. - Technique: Transparent holographic screens on three sides are projected with layered imagery, likely combined with a back projection, to create depth planes that dancers move between. - Try it: Choreograph one minute of dance with a flock of virtual birds in Unity AR Foundation, with birds passing in front of and behind the dancer via people occlusion. Twist: the flock follows the dancer's arms like a leader. ### Skullmapping (Filip Sterckx & Antoon Verbeeck) *Projection mapping and animation studio* Belgian duo of animator Filip Sterckx and painter Antoon Verbeeck. They are known for Le Petit Chef, a tiny projected chef who cooks on the diner's plate, and for dynamic projections that let figures escape from paintings and walk through galleries and airports. #### Le Petit Chef — Skullmapping (Filip Sterckx & Antoon Verbeeck) (2015) - Video: https://www.youtube.com/watch?v=yBJEP4lsRFY - Interaction: Tangible Objects, Projection, Play - Platform & tech: Projection, projection mapping, 3D animation, ceiling projector - Idea: The best AR scale is sometimes tiny: a character sized to your plate turns an ordinary object into a stage and makes the wait for food a story. - What it is: A tiny animated chef appears on a diner's empty plate, lights a grill, cooks a steak and serves it, all projected from above onto the plate and table. It grew into a worldwide dining show in restaurants and hotels. - Technique: A ceiling projector maps a pre-rendered 3D animation onto a fixed plate and table layout, with the plate rim and cutlery positions used as the animation's boundaries. - Try it: Anchor a tiny AR chef to a real plate using image or plane tracking, and animate a 30-second 'cooking' sequence whose flames and smoke respect the plate's edge. Twist: the chef reacts when a real fork enters the plate. #### WO1 Hologram — Skullmapping (Filip Sterckx & Antoon Verbeeck) (2015) - Video: https://www.youtube.com/watch?v=yKqmxWbItck - Interaction: Location & City, Perception & Effects, Projection - Platform & tech: Projection, holographic gauze, animation, projection - Idea: Placing ghostly figures in the real site of an event gives memory a location; site-specific AR memorials work the same way. - What it is: In an underground crypt at Leuven's city cemetery, where victims of the First World War rest, translucent figures of Belgian soldiers and civilians appear to float in the dark space as a memorial. - Technique: A holographic illusion, likely a projection onto transparent gauze or a Pepper's-ghost screen in a dark crypt, shows animated figures floating in space. - Try it: Choose a place on campus with a history, research one story, and place a translucent AR figure there that appears only at that GPS location and fades as you approach. Twist: visitors can leave a spoken note that the next visitor hears. #### Gallery Invasion — Skullmapping (Filip Sterckx & Antoon Verbeeck) (2016) - Video: https://www.youtube.com/watch?v=APpw6ZKIQ3I - Interaction: Projection, Spatial Mapping, Play - Platform & tech: Projection, Mirror Head, dynamic projection, motion capture - Idea: When content can leave its frame and travel through the room, the room becomes the interface; AR can let characters jump from a poster into space the same way. - What it is: In a room hung with paintings, a character escapes from one canvas, runs along the walls, jumps between frames and knocks things over, all as a projection that moves around the room. - Technique: Dynamic projection mapping: a motorized Mirror Head (Dynamic Projection Institute) steers the projector beam around the room, synchronized with a motion-captured 3D animation mapped to each wall and painting. - Try it: Use image tracking on two posters in the classroom and make an AR character leap from one poster, walk along the detected wall and dive into the other. Twist: the character carries a colour from the first poster and repaints part of the second. #### Bruegel's Creatures — Skullmapping (Filip Sterckx & Antoon Verbeeck) (2020) - Video: https://www.youtube.com/watch?v=8hellCoFdu4 - Interaction: Projection, Perception & Effects, Tangible Objects - Platform & tech: Projection, projection mapping, 3D animation - Idea: Colouring and animating a real print in place shows how AR can add a layer to an existing artefact without replacing it. - What it is: Monsters from Bruegel's 'The Fall of the Rebel Angels' crawl out of an engraving and roam Brussels Airport, while projection adds colour to the black-and-white print; the piece ends by revealing where the original painting hangs. - Technique: Projection mapped precisely onto an engraving adds colour fills, while 3D-animated creatures continue the motion onto surrounding surfaces. - Try it: Pick a black-and-white print, track it in Lens Studio or AR Foundation, and paint its regions with animated colour, then let one creature crawl off the edge onto the table. Twist: the colour spreads only where the viewer taps. #### Rubens Cupid — Skullmapping (Filip Sterckx & Antoon Verbeeck) (2020) - Video: https://www.youtube.com/watch?v=JREE37Q5cNM - Interaction: Projection, Location & City, Gaze & Attention - Platform & tech: Projection, projection mapping, 3D animation - Idea: An artwork that leaves its frame to guide you to the real thing is a model for AR wayfinding and museum invitations. - What it is: At Brussels Airport, a cupid escapes from a Rubens painting reproduction and flies around the terminal, over travellers and luggage, before returning to the canvas and pointing to where the original hangs. - Technique: Projection mapping with a steerable projector and pre-rendered 3D animation of the cupid, mapped onto the painting and the surrounding airport architecture. - Try it: Anchor an AR character to a painting or poster on campus and have it fly off, guide the viewer down a corridor, and land at a real location such as the library. Twist: the character only continues when the viewer keeps it in view. ### The Macula *Projection-mapping collective* Czech audiovisual collective (with members such as Michal Kotek, Lukáš Duběda, Dan Gregor and Amar Mulabegović) that made some of the most-watched architectural mappings of the 2010s, from Prague's Astronomical Clock to Liverpool's Liver Building. #### The 600 Years — The Macula (2010) - Video: https://vimeo.com/15749093 - Interaction: Projection, Location & City, Perception & Effects - Platform & tech: Projection, architectural projection mapping, 3D model alignment - Idea: Make a monument narrate its own biography by pretending to take it apart. - What it is: For the 600th anniversary of Prague's Astronomical Clock, the medieval tower seems to crack, fill with gears, burn, rebuild itself and show its own history in a 10-minute mapping watched by thousands in Old Town Square. - Technique: A precise 3D model of the tower is used to render animations that align to every ledge and window, projected from multiple overlapping high-lumen projectors. - Try it: Photograph a shoebox 'building' head-on, trace its features in After Effects or p5.js, and make a 30-second mapping in which it breaks apart and reassembles. Twist: tell the building's fictional history in three chapters. #### Luminous Flux — The Macula, INITI (Dan Gregor) (2011) - Video: https://vimeo.com/26827092 - Interaction: Projection, Perception & Effects, Location & City - Platform & tech: Projection, projection mapping, edge blending - Idea: A city celebrates a building by letting light rewrite its stone. - What it is: For the Liver Building's 100th anniversary, Liverpool's waterfront landmark is covered in moving light that makes its clock towers ripple, fold and open up above the crowd. - Technique: Animations are rendered against a surveyed model of the façade and played through edge-blended projectors so the illusions of depth line up with real ornaments. - Try it: Map a staircase or doorway on campus and create one illusion that makes the real structure look soft, flexible or hollow. Twist: sync the illusion to a clock so it happens exactly on the hour. #### iMapp Bucharest 555 — The Macula, Maxin10sity (2014) - Video: https://www.youtube.com/watch?v=tcq2gG1qxwc - Interaction: Projection, Location & City, Perception & Effects - Platform & tech: Projection, large-scale projection mapping, 3D scanning - Idea: At building scale, a single illusion can make thousands of people feel the ground move. - What it is: The vast Palace of the Parliament in Bucharest is mapped for the city's 555th anniversary, with the façade dissolving into particles, mechanisms and flying fragments at enormous scale. - Technique: A large array of projectors covers one of the world's biggest façades, with content rendered to a 3D scan of the building (third-party footage of the show). - Try it: Design a mapping storyboard for the largest wall you can find, then prototype it at 1:100 on a printed photo with a pico projector. Twist: design one moment that only works from a single viewing spot. #### SIM/NEBULA — The Macula, Can Büyükberber (2015) - Video: https://vimeo.com/138894725 - Interaction: Projection, Performance, Voice & Sound - Platform & tech: Projection, interior projection mapping, live orchestra sync - Idea: Map the inside of a concert hall so the architecture performs alongside the orchestra. - What it is: Inside Prague's Rudolfinum concert hall, the Czech Philharmonic plays while the ornate hall itself is mapped with 45 minutes of futuristic visuals about a cybernetic organism being born. - Technique: Multiple artists created chapters rendered to a 3D model of the hall interior, and projectors covered walls, balconies and organ in sync with the live score. - Try it: Pick a piece of classical music and map a classroom corner so that each instrument group 'lives' on a different surface. Twist: let the visuals react to a live microphone instead of a timeline. #### Down the Rabbit Hole — The Macula (2016) - Video: https://vimeo.com/181344271 - Interaction: Projection, Portals & Worlds, Location & City - Platform & tech: Projection, projection mapping, forced perspective - Idea: A façade as a rabbit hole: the building opens onto another world. - What it is: At Blackpool's Lightpool festival, the Tower Ballroom façade becomes a portal into weightless space inspired by astronaut Tim Peake and Alice in Wonderland. - Technique: A projection-mapped narrative renders false perspective 'openings' into the wall so the audience seems to look through it into space. - Try it: Create a projected 'hole' in a flat wall using forced perspective, and animate something falling into it. Twist: make the hole follow the position of the nearest viewer using a webcam. ### Tobias Gremmler *Media artist and designer (motion visualisation, stage visuals)* German media artist who has taught in Hong Kong and works between computation, motion capture and stage design. He created stage visuals for Björk's Cornucopia tour, virtual costume-actors for Chinese opera, media scenography for theatre by Tim Yip, and the widely shared Kung Fu Motion Visualization. #### Kung Fu Motion Visualization — Tobias Gremmler (2016) - Video: https://www.youtube.com/watch?v=RwJG62tRjGU - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, motion capture, 3D visualisation - Idea: Making a movement's path visible teaches the movement; AR can leave the same traces around a real athlete or dancer for learning and spectacle. - What it is: Kung fu forms performed by the masters Wong Yiu Kau and Li Shek Lin are motion-captured and turned into flowing ribbons, particles and abstract bodies that make the path and energy of each movement visible. - Technique: Optical motion-capture data of the masters' joints is used to generate trails, ribbons and particle bodies in 3D software, each variation revealing a different aspect of the motion. - Try it: Record a classmate doing a martial arts or dance move with ARKit body tracking and draw ribbons from their wrists and ankles that stay in the air for three seconds. Twist: replay the saved ribbons so a second person can try to follow them. #### Media Scenography for Chinese Theater (with Tim Yip) — Tobias Gremmler (2018) - Video: https://www.youtube.com/watch?v=ua5EtzWgFnw - Interaction: Performance, Perception & Effects - Platform & tech: Projection, stage projection, simulation, 3D animation - Idea: Building a stage language from a few elemental materials keeps digital scenery coherent; AR worlds also feel stronger when every effect comes from one material palette. - What it is: Stage visuals for a theatre show by Oscar-winning art director Tim Yip that blends traditional and modern theatre: digital sand, bamboo, fabric, water, stone and smoke transform into landscapes, walls, warriors and dragons around the actors. Sound by Benjamin Teare. - Technique: Physical simulations of sand, cloth, water and smoke are rendered as projected scenery and morph between elements to match the drama. - Try it: Choose one material (sand or smoke) and build three AR scenes in Unity VFX Graph or Lens Studio where it becomes a wall, a creature and a landscape around a real person. Twist: the transitions are triggered by the person's gestures. #### Virtual Actors in Chinese Opera — Tobias Gremmler (2018) - Video: https://www.youtube.com/watch?v=UbbJVAts-D8 - Interaction: Performance, Hands & Body - Platform & tech: Projection, motion capture, 3D animation, stage projection - Idea: A virtual performer can be built from a tradition's costume language instead of a realistic human; AR characters can borrow cultural form rather than photorealism. - What it is: Created for a theatre piece fusing Chinese opera with new media, these virtual actors are made from the shapes, colours and motions of traditional opera costumes and dance, exploring how costume and motion can reshape a human body. - Technique: Motion-captured opera movement drives abstract 3D figures whose sleeves, ribbons and headdress shapes are simulated and rendered for projection on stage. - Try it: Design an AR character in Lens Studio or Unity whose body is made only of long flowing sleeves, driven by body tracking of a live performer. Twist: the sleeves keep moving for a moment after the performer stops, like a memory. #### Björk Cornucopia: Body Memory (stage visuals) — Tobias Gremmler (2019) - Video: https://www.youtube.com/watch?v=GaQfixl2Ss4 - Interaction: Performance, Voice & Sound - Platform & tech: Projection, 3D animation, stage projection, particle systems - Idea: Visuals that look like they grow out of the performer's body blur the line between costume and environment, which is the natural home of performance AR. - What it is: Stage visuals for 'Body Memory' in Björk's Cornucopia tour: organic, fungal and body-like forms grow and dissolve on the huge screens and curtains around the singer, extending the costumes and choreography into the stage space. - Technique: Pre-rendered and likely partly generative 3D animations of organic forms are projected onto layered stage curtains and screens and timed to the music. - Try it: Make a Lens Studio or Effect House body-tracking lens where organic tendrils grow from a singer's shoulders and hands as they sing. Twist: the tendrils grow faster with louder voice. #### Björk: Arisen My Senses (concert visuals) — Tobias Gremmler (2021) - Video: https://www.youtube.com/watch?v=r-aQX5spvyM - Interaction: Performance, Voice & Sound - Platform & tech: Projection, audio-reactive visuals, 3D animation - Idea: Reformatting the same visual system for a new shape (wide stage vs. circle) shows that good stage visuals are a system, not a single video, which is how AR effects should be designed too. - What it is: An audio-reactive excerpt of Gremmler's concert visuals for Björk's 'Arisen My Senses', transformed from a wide stage projection format into a circular arrangement of pulsing, flower-like shapes. - Technique: 3D forms are animated by audio analysis of the track (amplitude and frequency bands likely driving growth and colour) and rendered for stage projection. - Try it: Build an audio-reactive AR 'flower' in Lens Studio or WebXR that opens and pulses with music from the microphone, placed on a real table. Twist: two phones hearing the same music grow two flowers that bloom in sync. ### Tomás Saraceno *Artist and architect* Argentine artist who builds cloud-like modules, walkable nets and Aerocene, a community of solar-powered balloons that fly without fossil fuels. #### Poetic Cosmos of the Breath — Tomás Saraceno (2007) - Video: https://www.youtube.com/watch?v=qHOsO-IYpbw - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, plastic-bag balloon, solar heating - Idea: Everyday waste turned into a flying object shows how humble materials can carry big visions, an attitude useful for low-tech AR prototypes. - What it is: A large balloon sewn from plastic bags rose over a field in the UK, heated only by the sun, as an early step toward Saraceno's ideas of floating cities. - Technique: Plastic bags were taped together into a huge envelope; black bags absorb heat so the air inside becomes buoyant. - Try it: Scan crumpled plastic bags and use them as the texture of a huge AR balloon floating over the school. Twist: each student adds one 'bag' to the balloon, and it rises only once it has enough. #### Cloud Cities — Tomás Saraceno (2011) - Video: https://www.youtube.com/watch?v=-VWAeLxq8gE - Interaction: Perception & Effects, Shared & Social - Platform & tech: Desktop, modular polyhedra, mirrors, suspended structures - Idea: Clusters of connected cells suggest how an AR city could grow modularly in the air above us. - What it is: At Hamburger Bahnhof in Berlin, dozens of polyhedral modules, some walkable, some with plants or mirrors, float in the hall like a cluster of bubbles or a city in the clouds. - Technique: Geodesic and polyhedral frames are covered with transparent and reflective films and held by cables. - Try it: Generate a cluster of connected AR bubbles in a room using a Voronoi or foam algorithm and let users add a new cell by tapping. Twist: cells with more visitors inside grow larger. #### in orbit — Tomás Saraceno (2013) - Video: https://www.youtube.com/watch?v=ROqL-8h_7DM - Interaction: Hands & Body, Shared & Social, Perception & Effects - Platform & tech: Desktop, steel wire net, inflated spheres, atrium - Idea: A shared surface that transmits everyone's movement makes multiplayer presence physical, a model for shared AR spaces where others' actions are felt. - What it is: A three-level net of steel wire hangs more than 25 metres above the piazza of K21 in Düsseldorf; visitors climb in and feel every other person's movement through the net. - Technique: A tensioned net spans the glass dome of the building, with large PVC spheres suspended in it; the net's vibrations spread from each visitor. - Try it: Build a shared AR net over a room (Photon or Lightship) where each user's position bends the net and sends ripples to others. Twist: link two rooms so the net carries movements between them. #### Aerocene — Tomás Saraceno (2015) - Video: https://www.youtube.com/watch?v=FPMrSkF_7BY - Interaction: Location & City, Shared & Social, Information & UI - Platform & tech: Desktop, solar balloons, open-source community - Idea: A shared, open tool for flying without fuel turns the sky into a commons, a frame for collaborative AR in the air. - What it is: Aerocene is an open community project of balloon sculptures that float using only the heat of the sun and the air, without burning fuel or using helium; the Art21 film shows launches and Saraceno's studio. - Technique: Lightweight, dark-coloured envelopes absorb sunlight so the air inside warms and becomes lighter than the surrounding air. - Try it: Build an AR balloon whose lift is driven by live solar irradiance data for your location, and launch it together with classmates. Twist: every balloon's flight path is saved and drawn as a trail over the city for the next group. #### Fly with Aerocene Pacha — Tomás Saraceno (2020) - Video: https://www.youtube.com/watch?v=AG_UXEXg_Mk - Interaction: Location & City, Shared & Social - Platform & tech: Desktop, solar balloon, human flight, salt flats - Idea: A human lifted by the sun over a threatened landscape links a body, a message and a place, the three things a strong location AR work needs. - What it is: Over the Salinas Grandes salt flats in Argentina, Leticia Marqués made the world's most sustainable manned flight, lifted only by the sun in the Aerocene Pacha balloon, carrying the message 'Water and life are worth more than lithium'. - Technique: A large black solar balloon heats up in the sun and rises with a pilot; the flight set records for solar-only lighter-than-air flight. - Try it: Stage an AR balloon launch on a real landscape under threat (a river, a green space) that carries a user-written message readable from below. Twist: the balloon only rises on sunny days. ### Tony Oursler *Artist (video sculpture, projection on objects and faces)* American artist, born 1957, who since the early 1990s has projected videos of faces onto dolls, spheres, trees and clouds of smoke, turning inert things into talking figures he calls electronic effigies. His work is held by MoMA, SFMOMA and Tate, among others. #### Electronic effigies (talking dummies) — Tony Oursler (1994) - Video: https://www.youtube.com/watch?v=RNrwxYeHw-I - Interaction: Projection, Face, Voice & Sound - Platform & tech: Projection, video projector, cloth dummies, video - Idea: A face projected onto the plainest object is enough to create a character with a point of view; AR characters do not need detailed bodies to feel present. - What it is: Small cloth dolls and dummies lie slumped on the floor or pinned under furniture, but each has a real human face projected onto its blank head that talks, complains or pleads with visitors. The SFMOMA video shows how these 'electronic effigies' sit between sculpture and media. - Technique: Small LCD video projectors throw recorded close-ups of actors' faces onto white cloth heads, positioned so the face wraps the curved surface. - Try it: Record a 20-second talking face on a phone, then use Lens Studio or AR Foundation to paste it onto a real object such as a pillow or mug, masked to its shape. Twist: the face only talks when nobody is looking at it directly, and goes silent when the phone points straight at it. #### Colors — Tony Oursler (1995) - Video: https://www.youtube.com/watch?v=lq30GmG4_RA - Interaction: Projection, Face, Voice & Sound - Platform & tech: Projection, video projector, fabric figure, video - Idea: A long, looping monologue gives a projected character a mind; in AR, voice and persistence can matter more than animation quality. - What it is: A projected face animates a soft fabric figure, which talks obsessively about colours in Oursler's hypnotic, anxious style. The piece in the Margulies Collection in Miami runs for about an hour, as if the figure never stops thinking aloud. - Technique: A single video projector aimed at a stuffed form displays a recorded face performance with sound, the projector hidden so only the lit head is noticed. - Try it: Build a WebXR character that is just a face on a sphere and let it speak one sentence per colour it sees, sampling the camera feed. Twist: it becomes more anxious when the room is dim. #### The Influence Machine — Tony Oursler (2000) - Video: https://www.youtube.com/watch?v=OzcuLP2vkWg - Interaction: Projection, Location & City, Face - Platform & tech: Projection, video projection, smoke, outdoor projection - Idea: Projecting onto smoke and foliage makes images feel like ghosts in a place; semi-transparent, wavering AR content can read as presence rather than as a screen. - What it is: At night in a park, giant faces are projected onto trees, buildings and drifting clouds of smoke, while voices tell the history of communication and the idea that spirits live in media. First staged in New York and London in 2000, it was revived outdoors at Stockholm University in 2016. - Technique: Video projectors aimed at smoke generators and tree canopies turn moving particles into temporary screens, combined with a spatial soundtrack across the park. - Try it: Create an outdoor WebXR or 8th Wall piece that places a semi-transparent talking face in a tree, with particles drifting through it. Twist: the face is only visible from one spot on the path, so visitors must search for the right place to stand. #### template/variant/friend/stranger — Tony Oursler (2014) - Video: https://www.youtube.com/watch?v=--d7NG_DWgk - Interaction: Face, Projection, Information & UI - Platform & tech: Projection, face recognition, video projection, cut-out panels - Idea: Showing the tracking mesh itself turns an invisible technology into the subject; AR face filters can reveal their landmarks instead of hiding them. - What it is: Large flat cut-out heads are covered with the dots and lines of face-recognition software, while projected eyes and mouths move within them. Shown at Art Basel Unlimited 2016, the installation asks what a face becomes when machines read it as data. - Technique: Printed facial-landmark diagrams on shaped panels are combined with projected video features aligned to those panels. - Try it: Make a face filter in Lens Studio or MediaPipe that draws the face mesh and landmark numbers on the user's face instead of any decoration. Twist: the mesh slowly drifts off the face when the user stays still, and snaps back when they move. #### Imponderable — Tony Oursler (2015) - Video: https://www.youtube.com/watch?v=zqyEt7YtNB8 - Interaction: Perception & Effects, Projection - Platform & tech: Projection, 4D film, Pepper's ghost, archive - Idea: The history of spirit photography is a history of mixed reality; tricks that once convinced people ghosts were real are still the core of believable AR. - What it is: A 'total cinema' installation about spiritualism and early photography, built from Oursler's archive of séance images and trick photographs. Shown at MoMA in 2016, it uses Pepper's-ghost style apparitions, projected figures and physical effects such as moving seats and wind. - Technique: Pepper's-ghost reflections, a projected film and synchronised physical effects (seat movement, air) are layered in one screening room, first made at LUMA Foundation (likely 2015). - Try it: Recreate a Victorian spirit photograph in AR: use a phone and a sheet of glass or a Pepper's-ghost prism to float a translucent figure next to a real person. Twist: the ghost appears only in the captured photo, not in the live view. ### Uncharted Limbo Collective (George Adamopoulos, Eleana Polychronaki, Chris Waters) *Creative coding studio for real-time visuals and performance* Studio of creative coders and visual artists working at the meeting point of code, interactive visuals and the performing arts. They built the motion-responsive visuals for Alexander Whitley's Anti-Body and co-created Chaotic Body and PRE-FIGURES with him, and develop dome and AI co-performer projects such as Protist Reverie and Monolith. #### Chaotic Body II: Liminal Phase — Uncharted Limbo Collective (George Adamopoulos, Eleana Polychronaki, Chris Waters), Alexander Whitley Dance Company (2021) - Video: https://www.youtube.com/watch?v=sf1D3erzPMw - Interaction: Hands & Body, Performance, Perception & Effects - Platform & tech: Phone, Desktop, motion capture, Unity, particle simulation, AR - Idea: Breath dissolves the boundary between the dancers' bodies and their surroundings. - What it is: A motion-capture dance film for screen and augmented reality made with choreographer Alexander Whitley, in which the breath and movement of two dancers dissolve their bodies into flowing, chaos-theory-inspired forms. - Technique: Motion-capture data of the dancers drives real-time particle and fluid simulations in Unity whose behaviour follows chaotic attractor patterns. - Try it: Record a short movement with a phone pose-tracking app and use the joint positions to drive a particle system in Lens Studio or three.js that is placed on the floor in AR. Twist: tie particle emission to the dancer's breathing rhythm. #### Motion-Reactive Physarum — Uncharted Limbo Collective (George Adamopoulos, Eleana Polychronaki, Chris Waters) (2022) - Video: https://www.youtube.com/watch?v=Ir8NH9M7ZhI - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, Physarum simulation, GPU compute, camera input (likely) - Idea: A living slime mould that grows toward the people moving in front of it. - What it is: A real-time simulated slime mould that grows its vein-like networks in response to movement, used to augment the Blessed Art Foundation's launch party. - Technique: A GPU agent-based Physarum simulation deposits and follows trails, and a motion input, likely from a camera, adds attractant where people move. - Try it: Adapt an open-source Physarum shader in TouchDesigner or p5.js so webcam motion adds food to the trail map, and project it on a wall. Twist: stillness makes the network slowly retreat. #### PRE-FIGURES — Uncharted Limbo Collective (George Adamopoulos, Eleana Polychronaki, Chris Waters), Alexander Whitley Dance Company (2022) - Video: https://www.youtube.com/watch?v=i_Zr_9x_1zk - Interaction: Performance, Portals & Worlds, Projection - Platform & tech: Projection, Unity, Barracuda, MiDaS depth estimation, projection - Idea: Archive dancers step out of old film and share the stage with living ones. - What it is: A hybrid performance for BFI's Luminous event in which a live dance trio moves alongside dancers from the BFI national archive, whose flat film footage is reanimated as floating 3D figures on stage. - Technique: Archive footage is passed through the MiDaS depth-estimation model inside Unity (via Barracuda), turning each frame into a displaced 3D surface that is rendered and projected in real time. - Try it: Run a depth-estimation model on an old dance clip and display it as a point cloud in WebXR or phone AR next to a live classmate. Twist: let the live dancer copy the archive dancer with a two-second delay. #### Monolith — Uncharted Limbo Collective (George Adamopoulos, Eleana Polychronaki, Chris Waters) (2024) - Video: https://www.youtube.com/watch?v=Hdep8X88SWA - Interaction: Hands & Body, Performance, Perception & Effects - Platform & tech: Projection, Desktop, Unity, GVHMR, SMPL, fluid simulation - Idea: A dancer's body stirs a virtual fluid as if dancing in water. - What it is: S+T+ARTS AIR residency project on whether a digital entity can perform alongside humans: a dancer's body, reconstructed from ordinary video, stirs a 3D fluid simulation in real time as part of a 'digital performer's brain'. - Technique: GVHMR monocular human mesh recovery produces an SMPL body from video, and the per-vertex motion of that mesh is added to the velocity field of a real-time fluid simulation in Unity. - Try it: Use MediaPipe pose on a webcam and inject each wrist's velocity into a 2D fluid shader projected on the floor. Twist: use the dancer's shadow instead of the skeleton as the obstacle. #### Protist Reverie — Uncharted Limbo Collective (George Adamopoulos, Eleana Polychronaki, Chris Waters) (2025) - Video: https://www.youtube.com/watch?v=zgehFbLJuZ4 - Interaction: Voice & Sound, Projection, Performance - Platform & tech: Projection, full-dome projection, Physarum simulation, audio reactive - Idea: A digital organism lives across the whole dome and breathes with live music. - What it is: A one-off live full-dome performance at Athens Digital Arts Festival where their algorithmic slime-mould organism fills the dome and responds in real time to the improvised music of Saber Rider. - Technique: A real-time Physarum-style simulation rendered in fisheye for dome projection takes audio analysis of the live set as input to its growth and colour. - Try it: Render an audio-reactive particle sketch in a fisheye projection and project it onto an umbrella or a paper dome that students lie under. Twist: let one student's voice feed the organism while another's starves it. ### United Visual Artists (UVA) *London art practice for light, sound and responsive installations* Founded by Matt Clark in 2003 after building stage visuals for Massive Attack, UVA makes installations such as Volume, Momentum and Our Time that use light, pendulums and sound to change how people feel space and time. #### Volume — United Visual Artists (UVA) (2006) - Video: https://www.youtube.com/watch?v=vbglOn9ea-Y - Interaction: Hands & Body, Voice & Sound, Shared & Social - Platform & tech: Desktop, LED columns, infrared cameras, generative sound - Idea: People become the performers when a space answers their movement — AR in public spaces works best when crowds can 'play' it together. - What it is: A field of 46 tall light columns in the V&A garden senses visitors walking among them and answers with waves of light and sound. - Technique: Infrared cameras track visitors' positions and a generative system maps proximity to the light and sound behaviour of each column. - Try it: Place 16 virtual light columns in a courtyard with shared AR anchors so that each one glows and hums when a participant walks close to it. Twist: make columns sing louder when two people stand near the same one. #### Chorus — United Visual Artists (UVA) (2009) - Video: https://www.youtube.com/watch?v=dEqIbzMKpN0 - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, motorized pendulums, LED light, speakers - Idea: Phase is a composition tool: identical pendulums slightly out of sync produce endless patterns — useful for any repeating AR animation. - What it is: Tall pendulums, each carrying light and a speaker, swing in and out of phase in a dark space, sweeping beams and voices across the room. - Technique: Motor-driven pendulums of equal length are started with different phases and each one carries a light and a speaker, so light and sound move together. - Try it: Hang eight virtual pendulums with spot lights and spatial audio from the ceiling in AR, each with a slightly different period. Twist: let the user nudge one pendulum by walking into its path. #### Vanishing Point — United Visual Artists (UVA) (2013) - Video: https://www.youtube.com/watch?v=RAlhQZJRlTA - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Projection, laser projection, haze, perspective geometry - Idea: Perspective is a construction you can stand inside — AR designers can make the vanishing point itself a visible object. - What it is: Laser lines drawn in haze converge on a single point in the room, building and dissolving Renaissance-style perspective grids around the viewer. - Technique: Lasers project lines through haze from a fixed origin so that all lines meet at a chosen vanishing point, animated to reveal and collapse the grid. - Try it: Draw glowing perspective lines in AR that all converge on one point in a real corridor, and let them rebuild the grid as the user walks. Twist: move the vanishing point to follow the user's head. #### Momentum — United Visual Artists (UVA) (2014) - Video: https://www.youtube.com/watch?v=B5FjivaKSyA - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, 12 pendulums, light, spatial sound, Barbican Curve - Idea: Movement of light makes time tangible — you feel the rhythm of the room through shadows passing over your own body. - What it is: Twelve pendulums carrying light and sound swing along the 90-metre curved Barbican gallery, sweeping shadows and beams across visitors in the dark. - Technique: Robotic pendulums follow programmed swing patterns while their lights and speakers are sequenced so beams and sounds travel along the curve. - Try it: Place a virtual pendulum light in a dark room with AR and cast real-time shadows of the user's body onto the detected walls. Twist: let the pendulum slow down whenever the user stands still. #### Our Time — United Visual Artists (UVA) (2016) - Video: https://www.youtube.com/watch?v=lNDMno_Gids - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, motorized pendulums, light, spatial sound - Idea: Changing the tempo of a whole space changes how long a minute feels — AR can shape subjective time, not just space. - What it is: A grid of pendulums hangs from the ceiling in darkness; lights and sounds swing and change speed so the room seems to speed up, slow down and stretch time. - Technique: Motorized pendulums with lights are driven at shifting rates and phases, with sound tied to each pendulum's movement. - Try it: Fill a room with a grid of swinging virtual lights in AR and slowly change their tempo over three minutes, then ask participants how long they think they were inside. Twist: link the tempo to the room's noise level. ### Within (Wonderscope) *Immersive studio; Wonderscope AR stories for kids* Within, co-founded by Chris Milk and Aaron Koblin, made Wonderscope (2018), an iOS app of AR stories where children speak characters' lines aloud to move the story forward. #### Wonderscope — Within (Wonderscope) (2018) - Video: https://www.youtube.com/watch?v=vqN3Rjv6nlk - Interaction: Voice & Sound, Spatial Mapping, Play - Platform & tech: Phone, ARKit, speech recognition - Idea: A child reads lines aloud to move the story along, and AR characters answer right in the living room. - What it is: An iOS app of AR stories for children: characters appear life-size in the room and the child moves the story along by walking around and reading lines out loud to them. - Technique: Life-size characters are placed on the floor via plane detection, and speech recognition checks the child's spoken line against the script to advance the story. - Try it: Make a three-line AR mini story with a character standing on the floor, where the plot continues only after the child reads the right line. Twist: when the line is wrong, the character gives a funny hint. #### Clio's Cosmic Quest — Within (Wonderscope) (2019) - Video: https://www.youtube.com/watch?v=_Kxz8QLJchc - Interaction: Voice & Sound, Spatial Mapping - Platform & tech: Phone, ARKit, speech recognition - Idea: A space adventure unfolds in your own room, and you have to speak up to move it along. - What it is: A space-adventure story on Wonderscope staged life-size in the child's room; like all Wonderscope stories, the child advances it by reading characters' lines aloud. - Technique: Story scenes are staged in room-scale world coordinates, and voice-recognition gates plus walking prompts pace the narrative. - Try it: Design an AR story where viewers must walk to the other side of the room to see the next scene. Twist: viewers must say the name of a planet to 'take off'. #### Wonder's Land: Ringmaster Wanted! — Within (Wonderscope) (2019) - Video: https://www.youtube.com/watch?v=WKI1P2YUDu8 - Interaction: Voice & Sound, Performance - Platform & tech: Phone, ARKit, speech recognition - Idea: A child joins an AR circus show and takes part with their voice. - What it is: A circus-themed Wonderscope story that plays out around the child, who takes part by speaking lines aloud to the performers. - Technique: Characters are arranged around the viewer in a circle, and speech input triggers performer animations, making the child a participant. - Try it: Place several AR performers in a circle around the viewer, and when the viewer says a cue word, the matching performer takes the stage. Twist: the louder the viewer speaks, the more exaggerated the performance. #### Sinclair Snake — Within (Wonderscope) (2020) - Video: https://www.youtube.com/watch?v=MNP77K9RsjI - Interaction: Voice & Sound, Spatial Mapping - Platform & tech: Phone, ARKit - Idea: Talk with a little AR snake and move the story forward with your voice. - What it is: A Wonderscope story centred on a snake character, told in AR scenes around the child's home and advanced by speaking aloud. - Technique: An animated character rig is anchored in the room, and scene transitions are triggered by the child's spoken responses via speech recognition. - Try it: Have an AR character ask you a question and branch into two different storylines based on your spoken answer. Twist: the character remembers what you answered last time. #### Willowcrest Manor — Within (Wonderscope) (2020) - Video: https://www.youtube.com/watch?v=oEpxUGvaCFM - Interaction: Voice & Sound, Spatial Mapping, Play - Platform & tech: Phone, ARKit - Idea: A mysterious manor unfolds a detective story in a child's bedroom. - What it is: A Wonderscope mystery set in a spooky mansion that unfolds around the child's room. - Technique: Room-scale placement of set pieces and characters, likely with scripted triggers based on where the viewer looks, turns the child's room into a staged mystery. - Try it: Build a gaze-triggered AR mystery scene where a clue appears only when you look at a certain corner. Twist: once the clue appears, it moves somewhere else if you look away. ### World of Zero (Sam Wronski) *Developer and YouTube shader educator* Sam Wronski runs the World of Zero YouTube channel, live-building Unity shaders and image effects such as a revealing flashlight, an energy shield and a night-vision camera, and collects them in the open UnityVisualizations repository. #### Contour Map Shader — World of Zero (Sam Wronski) (2017) - Video: https://www.youtube.com/watch?v=AK8oV4BzrW4 - Source code: https://github.com/WorldOfZero/UnityVisualizations - Interaction: Perception & Effects, Information & UI - Platform & tech: Desktop, Unity, ShaderLab - Idea: Any surface can be read as a map by slicing it at regular heights. - What it is: A terrain is drawn as a cartographic map, with evenly spaced contour lines tracing its height. - Technique: The shader samples a heightmap, takes the fractional part of height divided by the line spacing, and draws a thin line where it crosses zero, using screen-space derivatives to keep line width constant. - Try it: Apply the contour shader to the LiDAR scene mesh from AR Foundation meshing, using world height instead of a heightmap, so the whole room is overlaid with topographic lines. Twist: animate the line spacing so a flood level rises through the room. #### Conway's Game of Life with Compute Shaders — World of Zero (Sam Wronski) (2017) - Video: https://www.youtube.com/watch?v=ItPTBSeGjdM - Source code: https://github.com/WorldOfZero/UnityVisualizations - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: Four simple rules on a grid generate endless living patterns. - What it is: Conway's Game of Life runs entirely on the GPU, with cells blinking, gliding and dying across a large texture in real time. - Technique: A compute shader reads the previous state texture, counts the eight neighbours of every cell and writes the next generation into a second render texture, swapping the two each frame. - Try it: Map the Game of Life texture onto a detected AR floor plane and let the user seed live cells by tapping or walking over the floor. Twist: seed cells from the camera image so real objects become starting patterns. #### Energy Shield Shader — World of Zero (Sam Wronski) (2017) - Video: https://www.youtube.com/watch?v=NeZcAYJdkv4 - Source code: https://github.com/WorldOfZero/UnityVisualizations - Interaction: Perception & Effects, Play - Platform & tech: Desktop, Unity, ShaderLab, C# - Idea: Every impact leaves a ripple on an invisible barrier. - What it is: A translucent sci-fi force field flares with expanding ripples wherever projectiles hit it. - Technique: A C# script sends an array of recent hit positions and times to the shader, which adds a fading ring around each hit on top of a fresnel rim glow. - Try it: Wrap a person or a real chair in an AR shield dome using AR Foundation body or plane anchors, and send a hit to the shader wherever the user taps the screen. Twist: let microphone claps trigger random impacts. #### Night Vision Camera Effect — World of Zero (Sam Wronski) (2017) - Video: https://www.youtube.com/watch?v=mElPESXcakM - Source code: https://github.com/WorldOfZero/UnityVisualizations - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, ShaderLab - Idea: Depth becomes light: what is close to the camera is what you can see. - What it is: A dark scene is re-lit as a grainy green night-vision image, inspired by the camcorder in Outlast, where nearby surfaces glow brightest. - Technique: An image effect combines the depth texture with scene albedo so brightness falls off with distance, then tints the result green and adds animated noise and a vignette. - Try it: Rebuild the effect on the AR camera feed, using the LiDAR environment depth from AR Foundation's occlusion manager as the depth input so the real room turns into night vision. Twist: add virtual creatures that are only visible in the night-vision mode. #### The Magic Revealing Flashlight Shader — World of Zero (Sam Wronski) (2017) - Video: https://www.youtube.com/watch?v=b4utgRuIekk - Source code: https://github.com/WorldOfZero/UnityVisualizations - Interaction: Perception & Effects, Play - Platform & tech: Desktop, Unity, ShaderLab - Idea: Light itself becomes the key that reveals hidden content. - What it is: A hidden message on a wall stays invisible until a flashlight beam sweeps over it; only the lit cone shows the secret texture. - Technique: The shader receives the spotlight's position, direction and cone angle as uniforms and, per pixel, outputs the hidden texture's alpha only where the surface point lies inside the cone. - Try it: Place an AR Foundation plane on a real wall, attach the reveal shader, and drive the spotlight from the phone camera's pose so the phone acts as a torch that uncovers hidden writing. Twist: hide a different message for each of two phones so players must combine their beams. ### Yuma Yanagisawa *Media artist and creative coder* Japanese artist who works with shaders, VFX Graph and data, from raindrop and ripple shaders to large projected data sculptures and AI flower pieces. #### Raindrops Shader — Yuma Yanagisawa (2017) - Video: https://www.youtube.com/watch?v=dQSLjsDAzw0 - Source code: https://github.com/yumayanagisawa/Unity-Raindrops - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL - Idea: A rainy window can be drawn entirely in one shader. - What it is: Raindrops run down a pane of glass in front of a city scene, each drop refracting and blurring the view behind it. - Technique: A fragment shader, ported from a Shadertoy approach, generates drop and trail masks procedurally in grid cells and uses them to offset and blur the sampled background. - Try it: Apply the raindrop shader to the phone's AR camera feed so the real world looks like it is behind a rainy window. Twist: wiping the screen clears the drops for a moment. #### Wave Propagation Shader (Water Ripple) — Yuma Yanagisawa (2017) - Video: https://www.youtube.com/watch?v=rK5AAb-1pgE - Source code: https://github.com/yumayanagisawa/Unity-Wave-Propagation-Water-Ripple - Interaction: Perception & Effects, Hands & Body - Platform & tech: Desktop, Unity, HLSL, Render textures - Idea: Turn any image into a pond. - What it is: Touching the surface sends circular ripples across an image, which bend the picture as the waves travel and fade. - Technique: A multi-pass shader stores wave height in ping-pong render textures, solves the wave equation each frame, and uses the height gradient to refract the image. - Try it: Put the ripple shader on the camera feed in AR and trigger ripples where people's feet are detected. Twist: ripples only appear on the floor plane, not on walls. #### Demographics of Israel: Data Sculpture — Yuma Yanagisawa (2018) - Video: https://www.youtube.com/watch?v=qop7sk6H4GA - Source code: https://github.com/yumayanagisawa/Unity-Visual-Effects-Graph-Practice - Interaction: Information & UI, Perception & Effects - Platform & tech: Projection, Desktop, Unity, VFX Graph - Idea: Let a dataset set the size of a living particle sculpture instead of drawing a chart. - What it is: Four clouds of coloured particles swell and shrink over the years, each sized by the population share of one religious group, projected at Safra Square in Jerusalem. - Technique: Four VFX Graph systems spawn particles inside spheres whose radii are driven by yearly population percentages, with turbulence for motion. - Try it: Turn a class survey into a particle sculpture placed in the school yard with AR Foundation, one cloud per answer. Twist: the clouds update live as new answers come in. #### Metaballs — Yuma Yanagisawa (2019) - Video: https://www.youtube.com/watch?v=kuLUqNNlN4g - Source code: https://github.com/yumayanagisawa/Unity-Metaballs - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL, Raymarching - Idea: Blobs that merge smoothly feel alive. - What it is: Glossy liquid blobs float, merge and split apart like drops of mercury. - Technique: Metaballs are raymarched as a smooth-min union of sphere distance functions and shaded with environment reflections. - Try it: Float metaballs above a real table in AR and let each student's tracked hand be one of the balls. Twist: blobs reflect the real room using the AR environment probe. #### Point Cloud Garden — Yuma Yanagisawa (2020) - Video: https://www.youtube.com/watch?v=J6QHswn6Zdg - Source code: https://github.com/yumayanagisawa/Unity-Point-Cloud-VFX-Graph - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, HDRP, Pcx - Idea: A scanned place becomes a living material when its points can move. - What it is: A scanned garden appears as millions of coloured points that drift, dissolve and reform as the camera glides through it. - Technique: A PLY point cloud is baked into position and colour maps (following Keijiro's Pcx) and fed to VFX Graph, which spawns a particle per texel and animates it with noise in HDRP. - Try it: Scan a corner of the campus with a LiDAR phone, bake it into point maps, and replay it as a dissolving point cloud in the same spot with AR Foundation. Twist: the points drift away in the wind direction of the day. ### Zhuoyue Lyu *HCI researcher in mixed-reality interaction design, PhD candidate at the University of Cambridge* PhD researcher in Per Ola Kristensson's group at Cambridge's Department of Engineering, trained in computer science at the University of Toronto and in education at Harvard, with earlier stints at the MIT Media Lab and Stanford. He designs and builds mixed-reality interactions that treat everyday objects, walls, tables and windows as partners in interaction. #### AIive: Interactive Visualization and Sonification of Neural Networks in Virtual Reality — Zhuoyue Lyu (2021) - Video: https://www.youtube.com/watch?v=kE-NDBdZ9AU - Source code: https://github.com/ZhuoyueLyu/sonifyAI - Interaction: Hands & Body, Voice & Sound, Information & UI - Platform & tech: Headset, Unity, VR, hand tracking - Idea: Make an invisible process tangible: a learning machine you can walk around, touch and hear getting better. - What it is: A VR piece (IEEE AIVR 2021, with Jiannan Li and Bryan Wang, Best Presentation Award) where a neural network is laid out around you as layers of glowing nodes; you grab and stretch its hyperparameters with your hands while its loss and accuracy are played back as sound during training. - Technique: A Unity VR scene with hand tracking renders each network layer as nodes in space and maps grabbed handles to hyperparameters of a live training loop, likely streaming loss and accuracy values into pitch and rhythm of synthesized sound. - Try it: Place a small neural network (for example a TensorFlow.js digit classifier) in WebXR or phone AR as a stack of spheres on your desk, and map its training loss to a tone that falls as it learns. Twist: let a hand pinch-and-drag set the learning rate and hear what happens when it is too high. #### Touching The Droid: Understanding and Improving Touch Precision With Mobile Devices in Virtual Reality — Zhuoyue Lyu (2022) - Video: https://www.youtube.com/watch?v=8Ju-UA7FCoc - Interaction: Tangible Objects, Hands & Body, Information & UI - Platform & tech: Headset, Phone, VR headset, smartphone, hand tracking - Idea: A real phone in your hand can be a precise touch surface inside a virtual world, as long as the virtual hand is quietly corrected to match where you actually touch. - What it is: An ISMAR 2022 study led by Fengyuan Zhu with Zhuoyue Lyu, Mauricio Sousa and Tovi Grossman (University of Toronto): people hold a real phone or tablet while wearing a VR headset and tap its virtual twin, first with no rendered hand at all, then with a calibration that realigns the virtual hand to the real touch. - Technique: The phone's touch events serve as ground truth to continuously estimate the offset between the tracked physical hand and its rendered model, and a dynamic calibration shifts the virtual hand to cancel it, raising touch accuracy by 43% and reducing 'screen penetration' and 'floating touch' errors. - Try it: In WebXR on a headset, render a virtual copy of a phone held in the other hand and log where your fingertip appears versus where the phone registers the tap. Twist: hide the virtual hand completely and compare accuracy with and without it. #### Clo(o)k: Human-Time Interactions Through a Clock That "Looks" — Zhuoyue Lyu (2023) - Video: https://www.youtube.com/watch?v=A3jYe0NNDAk - Source code: https://github.com/ZhuoyueLyu/Arduino/tree/esp32 - Interaction: Gaze & Attention, Tangible Objects, Shared & Social - Platform & tech: Desktop, ESP32-CAM, OpenCV, Arduino, stepper motors - Idea: An everyday object that notices your attention can bend the time it shows: time rushes when ignored and stops for a conversation. - What it is: A hand-built wall clock with a camera at its centre: it keeps normal time while you look at it but speeds up when you look away, pauses when it sees people talking, and when two people in London and Hangzhou look at their Clo(o)ks at the same moment, each shows the other's time. Swappable rings turn it into a planetary model, a face or kinetic art (CHI 2025 Best Video Showcase). - Technique: An ESP32-CAM streams video to a laptop running OpenCV face detection, and the face count returned over serial drives two stepper motors (one via a custom-milled D11C board) that turn the hand rings. - Try it: Build a web or phone-AR clock anchored on a real wall whose hands run at normal speed only while the front camera detects your face, and fast-forward when you look away. Twist: let two classmates' clocks sync over the network so they only stop when both are looking. #### Objestures: Everyday Objects Meet Mid-Air Gestures for Expressive Interaction — Zhuoyue Lyu (2026) - Video: https://www.youtube.com/watch?v=p79r1l3LeQE - Interaction: Tangible Objects, Hands & Body, Voice & Sound - Platform & tech: Headset, Meta Quest 3, Unity, hand tracking - Idea: Any object within reach can become a controller without sensors: a cup is a dial, a book edge is a slider, a plush toy is a pressure pad. - What it is: A CHI 2026 design space (with Per Ola Kristensson) that lets each hand either handle an ordinary object or gesture in mid-air: squeeze a stuffed toy while sliding a finger along a book edge to play music, rotate a cup to change brush size while drawing, move a cup lid carrying a model building while a fist plays the sun to watch its shadow move, or steer a spaceship game along a window frame. - Technique: Uses only the Quest 3 built-in hand tracking in passthrough (Unity): after a thumbs-up to register an object's position, the system reads five interaction types (binary, linear, rotational, nonlinear, free) from finger and knuckle motion on the uninstrumented object, smoothed with an adaptive moving average. - Try it: In a headset or phone-AR hand-tracking app, register a real mug and map its rotation, read from the tracked hand, to the pitch of a synth tone; add a mid-air pinch with the other hand to trigger notes. Twist: remap the same mug to something that contradicts its meaning and discuss why it feels wrong. #### Unbounded: Object–Boundary Interaction in Mixed Reality — Zhuoyue Lyu (2026) - Video: https://www.youtube.com/watch?v=FL3hmRgtYUM - Interaction: Spatial Mapping, Hands & Body, Portals & Worlds - Platform & tech: Headset, Meta Quest 3, Unity, Meta XR SDK - Idea: A boundary is not just a limit: every wall, tabletop and window can let things bounce, pass through, hide, emerge or be written on. - What it is: Eight passthrough mixed-reality prototypes (CHI 2026, with Per Ola Kristensson) in which walls, tables, shelves, windows and mirrors become interactive boundaries: a ball falls through the tabletop on its last bounce, a document is fed through a shelf and shredded, a virtual marker writes TODO on the wall, an acoustic partition slides out of the windowsill, and a painting, a hidden wall drawer and a magic mirror turn a room into an art exhibit. - Technique: Built in Unity 6 with the Meta XR All-in-One SDK on Meta Quest 3 passthrough; the headset's scene understanding supplies wall, table and window planes, and scripted objects switch between colliding with a plane, passing through it or spawning from it, with occlusion hiding what is 'behind' the surface. - Try it: Use AR Foundation plane detection on a phone to make a virtual ball that bounces on a real table three times and then sinks through it, revealing a small hidden world under the tabletop through an occlusion mask. Twist: pick one of the five motions (rebound, reemerge, vanish, emerge, along) and apply it to a wall instead. ### Zimoun *Sound sculptor and installation artist* Swiss artist who builds large architectural installations from hundreds of identical cheap parts (dc-motors, cotton balls, cardboard boxes, wires, sticks) whose collective rattling becomes a dense, living sound field; each title simply lists its materials. #### Untitled Sound Objects (with Pe Lang) — Zimoun (2008) - Video: https://www.youtube.com/watch?v=DMPfed58xb0 - Interaction: Voice & Sound, Tangible Objects, Perception & Effects - Platform & tech: Desktop, dc-motors, found materials - Idea: Small, framed sound objects show that an AR piece can be intimate and wall-sized rather than room-sized. - What it is: A series of small wall objects by Pe Lang and Zimoun in which motors make strings, paper and wires rustle and buzz, each box its own tiny mechanical sound machine. - Technique: Each object pairs a simple motor with one material so that the sound and the motion are the same event, visible and audible at once. - Try it: Build three framed virtual boxes on a real wall with image tracking, each animating one material (string, paper, wire) with a matching sound. Twist: let visitors tap a box to change its material. #### 275 prepared dc-motors, filler wire 1.0mm — Zimoun (2011) - Video: https://www.youtube.com/watch?v=sGkSrYx6mZE - Interaction: Voice & Sound, Spatial Mapping, Perception & Effects - Platform & tech: Desktop, dc-motors, filler wire - Idea: Covering an architectural surface with one repeated gesture reframes the whole room; AR can do the same by instancing one tiny behaviour across a scanned mesh. - What it is: Hundreds of thin wires sprout from motors mounted across a room's walls and spin, whirring and scratching, so the whole space shivers like a field of insects. - Technique: Each motor spins a short length of 1 mm welding filler wire that scrapes the wall; the material list is the entire system (year from the exhibition, likely 2011). - Try it: Use LiDAR scene reconstruction to get the room mesh, instance 500 short spinning wires normal to the surfaces, and add a faint scratch sound to each. Twist: let wires near the viewer's hand spin faster. #### 329 prepared dc-motors, cotton balls, toluene tank — Zimoun (2013) - Video: https://www.youtube.com/watch?v=8jOBgFJSXxg - Interaction: Voice & Sound, Spatial Mapping, Perception & Effects - Platform & tech: Desktop, dc-motors, cotton balls, wire - Idea: The space is the instrument: an AR designer learns that simple agents placed along a real surface can make the architecture itself sound and feel alive. - What it is: Inside a huge empty steel toluene tank, 329 small motors swing cotton balls on wires against the curved wall; the tapping multiplies into a roaring, echoing rain you stand inside. - Technique: Each dc-motor carries a short wire with a cotton ball that randomly strikes the tank wall; the large reverberant volume blends hundreds of slightly different rhythms into one texture. - Try it: Scan a real wall with phone AR plane detection, scatter 100 virtual tappers along it that play a spatialised click with random timing, and walk through the resulting sound field. Twist: let the tapping rate follow how close the viewer stands. #### 192 prepared dc-motors, wooden sticks 2.4m — Zimoun (2016) - Video: https://www.youtube.com/watch?v=oLpwK6mhhXs - Interaction: Voice & Sound, Spatial Mapping, Perception & Effects - Platform & tech: Desktop, dc-motors, wooden sticks - Idea: A forest of long lines makes a room's height and depth legible; AR designers can use vertical elements the same way to reveal space. - What it is: Long wooden sticks hang from motors in a dense grid and knock against each other and the floor, creating a forest of clattering, swaying lines you can walk around. - Technique: Motors at the top of each 2.4 m stick create a small eccentric wobble that propagates down the stick, so the tips tap the floor in unpredictable patterns. - Try it: Hang 200 virtual sticks from the ceiling of a scanned room in AR Foundation, drive each with a small random pendulum motion and a knock sound on floor contact. Twist: let the sticks part like curtains as the viewer walks through. #### 658 prepared dc-motors, cotton balls, cardboard boxes 70x70x70cm — Zimoun (2017) - Video: https://www.youtube.com/watch?v=YF1SmQcJrM8 - Interaction: Voice & Sound, Spatial Mapping, Perception & Effects - Platform & tech: Desktop, dc-motors, cotton balls, cardboard boxes - Idea: A grid of identical cells becomes a landscape of sound when each cell is slightly out of sync; in AR, repetition plus tiny randomness reads as life. - What it is: A wall of stacked cardboard boxes, each with motors inside swinging cotton balls, fills a hall with a soft, crackling drizzle that changes as you walk along it. - Technique: Motors inside each box strike the cardboard from within, turning every box into a resonator; stacking them into a wall creates a spatial array of sound sources. - Try it: Stack a 10×10 grid of virtual boxes against a real wall in WebXR, give each one a looping rattle sound with random phase, and record a walk past it with spatial audio. Twist: let each box you look at fall silent. ### enra (Nobuyuki Hanabusa) *Performance group combining dance and projected motion graphics* Japanese performance group directed by Nobuyuki Hanabusa in which dancers, gymnasts and jugglers play in exact sync with motion graphics projected on the screen behind and around them. Pieces such as pleiades and FUMA-KAI became some of the most viewed stage-projection videos online. #### primitive — enra (Nobuyuki Hanabusa) (2012) - Video: https://www.youtube.com/watch?v=IALr6M2NXsE - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, projection, motion graphics, choreography - Idea: Virtual objects feel solid when the body reacts at the exact right frame; timing, not tracking, sells the contact. - What it is: Five performers move in front of a projection screen and appear to strike, carry and ride the geometric shapes and lines that move across it. - Technique: The motion graphics are pre-rendered to music and the performers rehearse their moves to hit each cue precisely, with no live tracking. - Try it: Record a short AR animation of a ball crossing the room, then rehearse a classmate to 'catch' and 'throw' it on camera until it looks real. Twist: film it from a second angle and see whether the illusion breaks. #### FUMA-KAI — enra (Nobuyuki Hanabusa) (2013) - Video: https://www.youtube.com/watch?v=HhJeNgjIKVw - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, projection, motion graphics, choreography - Idea: Graphics that seem to come from a gesture make the performer look powerful; AR effects should appear to start from the user's hand. - What it is: Dancers and a juggler interact with projected wind, light trails and particles, as if their gestures generate storms of graphics around them. - Technique: Pre-rendered projected animation is choreographed so trails start exactly where the performers' hands are at each moment. - Try it: Use hand tracking in Lens Studio or WebXR to emit particle trails from each fingertip, then choreograph a 30-second duet with a partner. Twist: one dancer's trails push the other's away. #### pleiades — enra (Nobuyuki Hanabusa) (2013) - Video: https://www.youtube.com/watch?v=0813gcZ1Uw8 - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, projection, motion graphics, choreography - Idea: Two bodies and one shared cosmos show how a virtual layer can unite performers; shared AR works when everyone sees the same object respond. - What it is: Two dancers, Saya Watatani and Maki Yokoyama, move with projected stars, planets and orbits so precisely that they seem to spin constellations around themselves. - Technique: Motion graphics by Nobuyuki Hanabusa and Seiya Ishii are pre-rendered and the choreography is built frame by frame around them. - Try it: Anchor a shared AR planet system between two people using a multiplayer AR session and choreograph them passing a moon to each other. Twist: the orbit speeds up when they move apart. #### Torque starter — enra (Nobuyuki Hanabusa) (2014) - Video: https://www.youtube.com/watch?v=3JdT4fDi4iI - Interaction: Performance, Projection, Play - Platform & tech: Projection, projection, motion graphics, juggling - Idea: Mixing a real prop with virtual consequences is more convincing than a fully virtual object; AR shines when a real thing triggers the effect. - What it is: Performer Yusaku Mochizuki juggles and spins real props while projected gears, sparks and machine parts react to every throw and catch. - Technique: Pre-rendered animation is timed to a rehearsed juggling routine so projected effects start at each catch. - Try it: Track a real ball with object or colour tracking on a phone and spawn AR sparks and gears whenever it is caught. Twist: a missed catch makes the whole machine stall. #### Cubee — enra (Nobuyuki Hanabusa) (2019) - Video: https://www.youtube.com/watch?v=lW6mEJMA-mA - Interaction: Performance, Projection, Perception & Effects - Platform & tech: Projection, projection, motion graphics, choreography - Idea: A flat screen can suggest depth when the body pretends to enter it, a cheap trick for making AR portals feel deep. - What it is: Dancer Maki Yokoyama appears to build, open and step through projected cubes, turning flat graphics into rooms she can enter. - Technique: Perspective-drawn cube animation is projected behind the dancer and her movement is choreographed to match the implied depth. - Try it: Place an AR cube portal on the floor and film a classmate stepping 'into' it, using people occlusion so the cube edges pass in front of and behind them. Twist: the cube changes into a different room each time someone enters. ### hecomi *Unity engineer and graphics blogger* Japanese Unity engineer behind uRaymarching, UnityFurURP and uLipSync, who documents each experiment in depth on his blog tips.hecomi.com. #### Kinect V2 Depth into the G-Buffer — hecomi (2016) - Video: https://www.youtube.com/watch?v=Nl3lxlz0qME - Source code: https://github.com/hecomi/UnityKinectV2DeferredRendering - Interaction: Hands & Body, Spatial Mapping - Platform & tech: Desktop, Unity, Kinect V2, Deferred rendering - Idea: Treat the depth camera image as part of the renderer, not as a texture. - What it is: A live Kinect depth image becomes solid, lit geometry in a Unity scene, receiving virtual lights and shadows like any other object. - Technique: A shader writes the Kinect V2 depth and reconstructed normals directly into Unity's deferred G-buffer, so the real person is lit by Unity lights. - Try it: Do the same with the LiDAR depth on an iPhone: write it into the depth buffer and relight the real room with a moving virtual lamp. Twist: the virtual light casts coloured shadows of real people. #### Screen Space Collision GPU Particles — hecomi (2016) - Video: https://www.youtube.com/watch?v=G-4k8Zur7zY - Source code: https://github.com/hecomi/UnityPseudoInstancedGPUParticles - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, Unity, HLSL, Deferred rendering - Idea: If you can see a surface, particles can bounce off it: use the depth buffer as the collider. - What it is: Hundreds of thousands of GPU particles pour over a scene and bounce off every visible surface without any physics colliders. - Technique: Particle positions update in a shader that projects each particle into screen space, compares its depth with the G-buffer depth and normal, and reflects its velocity on contact. - Try it: Use the AR depth texture (LiDAR or ARCore Depth) as the collision surface so virtual snow settles on real furniture. Twist: particles stick and pile up where they land. #### uRaymarching — hecomi (2016) - Video: https://www.youtube.com/watch?v=AppyVflAagc - Source code: https://github.com/hecomi/uRaymarching - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL, Raymarching, SDF - Idea: Write only a distance function and get a full, lit shader that behaves like any other object. - What it is: Raymarched distance-field shapes appear in a normal Unity scene with correct lighting, shadows and intersections with ordinary meshes. - Technique: An editor tool generates deferred/forward shaders from a template around a user-written distance function, outputting depth and G-buffer data so raymarched objects are lit and shadowed by Unity. - Try it: Generate a raymarched sculpture with uRaymarching and place it on a real plinth in phone AR, writing depth for occlusion. Twist: the distance function morphs with the time of day. #### Water Surface Simulation with CustomRenderTexture — hecomi (2017) - Video: https://www.youtube.com/watch?v=jclxfdS3a3w - Source code: https://github.com/hecomi/UnityWaterSurface - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, Unity, CustomRenderTexture, HLSL - Idea: A ripple simulation fits in a texture that updates itself every frame. - What it is: Objects dropped onto a water plane send out ripples that spread, reflect off the edges and interfere with each other. - Technique: A CustomRenderTexture runs the 2D wave equation in a shader, reading the previous two frames, adding impulses where objects hit, and feeding the height map into the water shader's normals. - Try it: Turn a real tabletop into water in phone AR and create ripples wherever a tracked finger or object touches it. Twist: ripples also start when a sound is detected. #### UnityFurURP (Shell / Fin Fur Shader) — hecomi (2021) - Video: https://www.youtube.com/watch?v=Hab3dcumtXU - Source code: https://github.com/hecomi/UnityFurURP - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, Unity, URP, HLSL - Idea: Stack many transparent copies of a surface and it turns into fur. - What it is: Soft animal fur grows on a model and sways as it moves, rendered in real time with layered shells and fins. - Technique: The URP shader extrudes the mesh along its normals into many shell layers (and fin quads at silhouettes), cutting each layer with a noise mask and moving it with gravity and wind. - Try it: Grow fur on a real object in AR by applying the shell shader to a LiDAR scan placed over it, so a mug or chair becomes furry on screen. Twist: the fur bristles when someone moves the phone quickly. ### onformative (Cedric Kiefer & Julia Laub) *Studio for digital art and design* Berlin studio founded by Cedric Kiefer and Julia Laub; Laub co-authored the book Generative Gestaltung. Its generative and interactive installations range from perception experiments and responsive sculptures to augmented stage magic and choreography tools such as Pathfinder. #### Fragments of RGB — onformative (Cedric Kiefer & Julia Laub) (2010) - Video: https://vimeo.com/21235126 - Interaction: Perception & Effects, Gaze & Attention, Projection - Platform & tech: Projection, camera tracking, projection, Processing - Idea: A screen that falls apart into pixels when you get close. - What it is: What looks like a classic LED screen is simulated with projected light points; as viewers approach, the pixels change, scatter and dissolve with their movement and point of view. - Technique: A camera tracks the viewer's position and a projection simulates an LED pixel grid whose pixels are displaced and dissolved according to distance and angle. - Try it: Show a video as a grid of large dots in p5.js and use the webcam's face size to scatter the dots as the viewer leans in. Twist: the image only becomes readable from one exact spot. #### NikeFuel Station — onformative (Cedric Kiefer & Julia Laub) (2012) - Video: https://vimeo.com/44338220 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Kinect, generative graphics - Idea: Your body rebuilt as a cloud of glowing pixels. - What it is: The centrepiece of a London pop-up store turns anyone standing in front of it into a moving multi-pixel avatar that mirrors their body and energy. - Technique: A Kinect captures the visitor's body and depth, and generative software renders it as a pixel-particle avatar on large screens (collaboration with AKQA London). - Try it: Turn a webcam body segmentation into a grid of squares whose brightness follows how fast each body part moves. Twist: the avatar leaves a pixel trail that shows your whole day's activity. #### Kinect / Leap controlled realtime creature — onformative (Cedric Kiefer & Julia Laub) (2013) - Video: https://vimeo.com/71086950 - Interaction: Hands & Body, Play - Platform & tech: Desktop, Kinect, Leap Motion, Processing - Idea: Puppeteer a creature with your bare hands. - What it is: A work-in-progress digital creature that moves and reacts in real time to a person's hands and body, controlled with a Kinect and a Leap Motion. - Technique: Kinect body data and Leap Motion finger data drive the limbs and behaviour of a procedurally generated creature built with Processing 2.0. - Try it: Use MediaPipe hand tracking to drive a simple tentacle creature in p5.js, with each finger controlling one limb. Twist: place it in AR so it crawls on your real desk. #### ANIMA iki — onformative (Cedric Kiefer & Julia Laub) (2015) - Video: https://vimeo.com/128767230 - Interaction: Voice & Sound, Hands & Body, Projection - Platform & tech: Projection, projection mapping, sensors, generative sound - Idea: A glowing orb that behaves like a shy living being. - What it is: A two-metre glowing sphere hangs in a dark room; shimmering patterns flow across it and tones change as it senses visitors' movements and sounds, like a creature getting to know them. - Technique: Sensors read visitors' movement and sound, and generative visuals projected onto the sphere plus an interactive soundscape express the entity's changing moods. - Try it: Project a p5.js noise pattern onto a white balloon or paper lamp and make it calm, curious or startled depending on microphone level and webcam motion. Twist: the orb remembers the loudest person and turns away from them. ### Alexander Calder *Artist; inventor of the mobile* American sculptor (1898–1976) who invented the mobile, balanced abstract shapes that move with air and touch, and who performed his wire Cirque Calder by hand. #### Cirque Calder (Calder's Circus) — Alexander Calder (1926) - Video: https://www.youtube.com/watch?v=t6jwnu8Izy0 - Interaction: Performance, Tangible Objects - Platform & tech: Desktop, wire, cork, fabric, found objects - Idea: A tabletop world of tiny performers animated by hand is still one of the most charming formats for AR: small scale, close attention and visible manipulation. - What it is: A miniature circus of wire acrobats, animals and props that Calder performed by hand for friends, making tightrope walkers balance and a sword swallower swallow. - Technique: Wire figures with simple joints are moved with strings, levers and the artist's hands, while he provides sound effects and narration live. - Try it: Build a tabletop AR circus with three wire-style characters that users animate by dragging (touch) or pinching (hand tracking), with sound effects. Twist: record one person's performance and replay it for the next viewer. #### Two Spheres — Alexander Calder (1931) - Video: https://www.youtube.com/watch?v=pqWi5HDB72Q - Interaction: Perception & Effects - Platform & tech: Desktop, wood, wire, motor - Idea: Two dots moving in space are enough to suggest orbits and relationships; minimal AR motion can carry a whole story. - What it is: One of Calder's early motorised works in which two small spheres move on wire arms in a slow, looping choreography against a flat ground. - Technique: A hidden motor and cranks drive the wire arms so the spheres trace varied paths, anticipating later motor-driven kinetic art. - Try it: Place two small AR spheres in front of a real wall and animate them on linked orbits with visible fading trails. Twist: let one sphere follow the phone while the other keeps its own orbit. #### Triple Gong — Alexander Calder (1948) - Video: https://www.youtube.com/watch?v=l0kybG2OabA - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, brass, sheet metal, wire - Idea: Sound triggered by chance collisions makes a moving object feel alive and unpredictable; AR physics objects can generate sound the same way. - What it is: A mobile whose swinging elements occasionally strike small brass gongs, so the sculpture adds chance sounds to its slow movement. - Technique: The balance of the arms lets hanging beaters swing into brass discs only when the motion is large enough, making sound an emergent result of movement. - Try it: Build an AR sound mobile whose parts play a note when they collide, hung in the room and set moving by the user's breath or hand. Twist: tune each gong to a different note so the room plays a random melody. #### Black Mobile with Hole — Alexander Calder (1954) - Video: https://www.youtube.com/watch?v=Nj-V1rUM75k - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, sheet metal, wire, paint - Idea: A few balanced parts moving at different speeds create endless, never-repeating compositions; ambient AR objects can rely on simple physics instead of animation. - What it is: A hanging mobile of black metal shapes on balanced wire arms that turn slowly and independently in the air currents of the room. - Technique: Each arm is balanced at a single pivot point so that tiny air currents set every level rotating at its own rate, producing compound motion. - Try it: Hang an AR mobile from a detected ceiling using hinge joints and let the phone's movement create a breeze that sets it turning. Twist: let users add or remove shapes and watch the balance change. ### Anna Zhilyaeva (Anna Dream Brush) *VR painter and live performer* Russian-born, Paris-based artist who paints three-dimensional 'volumist' works in Tilt Brush and Open Brush, often filmed in mixed reality or performed live, including a VR interpretation of Delacroix at the Louvre. #### Medusa (Tilt Brush mixed reality) — Anna Zhilyaeva (Anna Dream Brush) (2017) - Video: https://www.youtube.com/watch?v=rX9G5MOdwSc - Interaction: Drawing & Making, Hands & Body - Platform & tech: Headset, Tilt Brush, green screen, mixed-reality capture - Idea: Painting becomes sculpture when every brushstroke hangs in space. - What it is: Filmed in mixed reality, the artist spends two hours painting a life-size Medusa with living snake hair, and the camera circles the sculpture-like painting as she works inside it. - Technique: Tilt Brush strokes are recorded in VR and composited with green-screen footage of the painter from a tracked camera. - Try it: Paint a life-size figure in 3D in 30 minutes, then record a slow orbit around it that reveals something not visible from the front. Twist: paint only with light-emitting brushes in a dark scene. #### Rendez-vous in Mixed Reality — Anna Zhilyaeva (Anna Dream Brush) (2017) - Video: https://www.youtube.com/watch?v=Uxc7Um_Lk5c - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Headset, Tilt Brush, mixed-reality capture - Idea: A real pet meeting a painted one reveals the magic and absurdity of mixed reality. - What it is: In a mixed-reality video, the artist paints a glowing cat in the air of her living room while her real cat Mimi watches and tries to catch the virtual one. - Technique: The painting is made in Tilt Brush on a tracked VR headset, and a separately tracked external camera composites the 3D strokes over real room footage. - Try it: Use Open Brush or an AR drawing app to paint a character that interacts with a real object or person in the frame, and film it from a second camera. Twist: have a real person or pet 'respond' to the drawing on cue. #### Live VR Painting at the Louvre — Anna Zhilyaeva (Anna Dream Brush) (2018) - Video: https://www.youtube.com/watch?v=Zs3n07Clw7A - Interaction: Drawing & Making, Performance - Platform & tech: Headset, Tilt Brush, HTC Vive, live streaming - Idea: Step inside a famous painting and rebuild it as a space. - What it is: In a live performance at the Louvre, the artist paints a three-dimensional interpretation of Delacroix's 'Liberty Leading the People' in VR while visitors watch the strokes appear on screens. - Technique: A tracked VR headset with Tilt Brush is mirrored to large displays so the audience sees the 3D painting grow in real time. - Try it: Pick a classic painting and rebuild it as a 3D scene in an XR drawing tool, deciding what lies behind each figure. Twist: present it live while a partner narrates the history of the original. #### You Are Strange — Anna Zhilyaeva (Anna Dream Brush) (2020) - Video: https://www.youtube.com/watch?v=vXLUabAA75c - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Headset, Tilt Brush, mixed-reality capture - Idea: Let a drawn daydream share the room with you. - What it is: A short mixed-reality poem in which a dreamy painted figure and floating words appear around the artist in a real room, as if her imagination were leaking into the space. - Technique: The 3D painting is created in VR and composited over footage of the real room from a tracked camera. - Try it: Write a four-line poem and paint each line as a 3D element placed around a real person in a room using an AR drawing app. Twist: film it so the words are only readable from one camera position. ### Anthony McCall *Artist; solid-light works* British-born artist whose 'solid light' films, starting with Line Describing a Cone (1973), project slowly moving lines through haze so that the beam itself becomes a walk-in sculpture. #### Line Describing a Cone — Anthony McCall (1973) - Video: https://www.youtube.com/watch?v=1-HWsxPnNNY - Interaction: Projection, Perception & Effects, Hands & Body - Platform & tech: Projection, 16mm film projector, haze - Idea: The projector beam, not the screen, is the artwork: an AR designer can treat the volume between device and surface as a place to put content. - What it is: A single white dot on a 16mm film slowly draws a circle over 30 minutes; in the hazy room the projector beam becomes a hollow cone of light that visitors walk around and cut with their hands. - Technique: A 2D animation of a growing arc is projected through haze so that particles scatter the light and reveal the 3D surface swept by the beam. - Try it: Build a WebXR scene where a virtual projector on the wall sends a translucent cone toward the floor whose rim grows along a circle over two minutes; let the cone brighten where a tracked hand intersects it. Twist: give every student's phone its own cone so that several light volumes cross in the room. #### You and I, Horizontal II — Anthony McCall (2006) - Video: https://www.youtube.com/watch?v=HgzcbQlwT6w - Interaction: Projection, Perception & Effects, Hands & Body - Platform & tech: Projection, digital projector, haze machine, computer animation - Idea: A slowly changing line on a flat plane becomes a moving architecture once it has depth: animate 2D shapes over time and they read as spatial rooms in AR. - What it is: A digital projector throws slowly morphing curved lines across a dark, hazy room; the light forms soft walls and folding membranes that visitors step through. - Technique: Computer-generated ellipses and travelling waves are projected horizontally through haze, so each line becomes a thin plane of light at eye level. - Try it: Extrude an animated 2D line (a sine wave that slowly morphs into an ellipse) into a translucent sheet anchored at chest height in AR Foundation, and ask classmates to walk through it. Twist: when two people stand on opposite sides, bend the sheet into a curve that connects them. #### Crossing — Anthony McCall (2016) - Video: https://www.youtube.com/watch?v=yNbitJwJzDs - Interaction: Projection, Perception & Effects, Voice & Sound - Platform & tech: Projection, digital projectors, haze, sound - Idea: Projecting down instead of across turns light into standing figures: vertical volumes feel like companions, a cue for sizing AR presences to the human body. - What it is: Two vertical cones of light fall from ceiling projectors onto the floor; their outlines slowly cross and separate while a soundtrack plays, and visitors stand inside the beams. - Technique: Ceiling-mounted projectors draw animated line drawings on the floor; haze makes the beams visible as vertical surfaces, and the sound is timed to the movement (likely a four-channel score). - Try it: Place two person-height virtual light cones in the classroom with WebXR, animate their floor outlines so they slowly cross, and add spatial audio to each. Twist: let the cones drift toward whichever student has been still the longest. #### Solid Light (Tate Modern) — Anthony McCall (2024) - Video: https://www.youtube.com/watch?v=1OVzjGO5iS4 - Interaction: Projection, Perception & Effects, Shared & Social - Platform & tech: Projection, digital projectors, haze, sound - Idea: When light has volume, the audience becomes part of the image: other people's bodies interrupting the beam are content, which is how shared AR scenes should feel. - What it is: A retrospective of McCall's solid-light works fills darkened galleries at Tate Modern with horizontal and vertical light volumes that crowds explore together. - Technique: Several rooms each hold one projector-and-haze piece, spaced so that beams never overlap and each volume can be circled. - Try it: Make a shared AR session (Lens Studio Connected Lenses or a WebXR multiplayer demo) with one large light cone where each participant's phone position cuts a visible shadow slice into the volume. Twist: record the slices over five minutes and replay them as a ghost crowd. ### Arcade (Arcade Ltd) *Spatial design and AR studio ('The Spatial Agency')* Arcade was a digital practice founded by architects, technologists, game developers and strategists that used AR and VR to connect people to places through play. Its 2019 projects included A Vixen's Tale for Welsh National Opera, an AR guide for the SEA LIFE London Aquarium, a Madame Tussauds app that animated wax figures, and an AR literary trail for the Roald Dahl Museum. #### A Vixen's Tale — Arcade (Arcade Ltd), AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) (2019) - Video: https://www.youtube.com/watch?v=ebI643cFWfk - Interaction: Tangible Objects, Play, Performance - Platform & tech: Phone, image tracking, face filter, physical set, mobile AR - Idea: A physical pop-up-book set and AR together turn an opera into a walk-through game. - What it is: Visitors walk through five wooden arches styled like a pop-up book and hold up a phone to follow an animated fox through gamified scenes from Janáček's opera The Cunning Little Vixen; a face filter lets them become the vixen. - Technique: Physical arches act as image targets and staging for phone AR vignettes that unlock chapters of the story with music from the opera, plus a face-tracking filter. - Try it: Pick a song or story with five scenes, build five cardboard 'arches' with printed markers, and make one AR vignette per arch in Lens Studio or 8th Wall. Twist: the final arch only unlocks if the visitor has collected an item in each earlier scene. #### Madame Tussauds Fame Cam — Arcade (Arcade Ltd) (2019) - Video: https://www.youtube.com/watch?v=pVP-1MSKMmo - Interaction: Face, Tangible Objects - Platform & tech: Phone, object tracking, face tracking, mobile AR - Idea: A wax figure that moves only when you look at it through your phone. - What it is: At Madame Tussauds London, visitors point a phone at 25 celebrity wax figures and watch them come to life, react to the visitor's facial expressions or transform the scene around them. - Technique: Object tracking on each wax figure anchors animated overlays, while front-camera face tracking lets the visitor's expressions trigger the figure's reactions. - Try it: Choose a statue or poster on campus and make a phone AR effect that makes it blink or speak when viewers smile at it. Twist: the statue only reacts to the second person who looks at it. #### Marvellous Missenden — Arcade (Arcade Ltd) (2019) - Video: https://www.youtube.com/watch?v=kN0hb8Upp2I - Interaction: Location & City, Information & UI - Platform & tech: Phone, ARKit, geolocation, mobile AR - Idea: Show readers the exact spot that inspired a line of a story. - What it is: A proof-of-concept AR trail through the village of Great Missenden that places Roald Dahl quotations over the buildings and landscape that inspired them. - Technique: An iOS ARKit app with geolocated points of interest superimposes text and illustrations on the village scenery at each stop of the trail. - Try it: Pick a book set in your city, find three real places mentioned in it, and make a location-based AR trail that places the quote in the sky above each place. Twist: let visitors add their own sentence at each stop for the next reader. #### SEA LIFE AR Chatbot (Roxy) — Arcade (Arcade Ltd) (2019) - Video: https://www.youtube.com/watch?v=zR406oTNwYM - Interaction: Information & UI, Play, Voice & Sound - Platform & tech: Phone, mobile AR, chatbot, Unity - Idea: An AR companion that talks with children about the real animals in front of them. - What it is: Children at the SEA LIFE London Aquarium chat with Roxy, an AR character in a little craft, who guides them through challenges about the animals in the tanks until they become Junior Rangers. - Technique: A mobile AR app combines a placed 3D character, a scripted chatbot dialogue and collectible cards triggered at each tank. - Try it: Design an AR guide character for a museum room that asks visitors one question per object and rewards correct looks with a collectible. Twist: the guide only answers questions whispered to it. ### Arthur Ganson *Kinetic sculptor and engineer* American sculptor whose small hand-built machines of wire and gears perform poetic gestures: a wishbone that walks, a chair that explodes and reassembles, a gear train that ends in concrete; many are on long-term view at the MIT Museum. #### Machine with Wishbone — Arthur Ganson (1988) - Video: https://www.youtube.com/watch?v=4pZXoayEL78 - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, wire, gears, wishbone - Idea: Giving a found object a gait gives it a character; AR can animate everyday objects with minimal, well-chosen motion. - What it is: A tiny wire machine drags a chicken wishbone along a circular track, so the bone seems to walk forward with a strange, earnest gait. - Technique: A crank mechanism pushes and lifts the wishbone in a stepping cycle, and the whole machine rotates slowly so the walk becomes an endless loop. - Try it: Scan an everyday object (fork, clothes peg) with photogrammetry and give it a two-step walk cycle across a real table in AR. Twist: let it stop and turn toward whoever speaks. #### Margot's Cat — Arthur Ganson (1991) - Video: https://www.youtube.com/watch?v=a6aicIcQJvc - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, wire, gears, miniature chair - Idea: Timing and anticipation make inanimate objects feel intentional; AR animations should borrow the rhythm of a real creature rather than smooth tweens (year approximate). - What it is: A tiny chair is repeatedly flung upward and caught by a delicate wire mechanism, so it seems to leap and land again like a playful cat. - Technique: A cam-driven arm stores and releases energy, launching the chair in a ballistic arc before a second arm catches it, repeated as a loop. - Try it: Make a small virtual chair on a real desk jump along a physics arc every few seconds in phone AR, with squash-and-stretch on landing. Twist: let it jump only when the viewer looks away. #### Machine with Concrete — Arthur Ganson (1992) - Video: https://www.youtube.com/watch?v=5q-BH-tvxEg - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, gears, motor, concrete - Idea: Motion can be made to disappear into deep time; an AR piece can show a process whose end lies far beyond a human life. - What it is: A motor spins the first gear fast; through a chain of twelve 1:50 gear reductions the last gear is embedded in a block of concrete, since it would take over two trillion years to turn once. - Technique: Twelve worm-style reductions of 50:1 each multiply to 50^12, so the output speed is effectively zero and the concrete never feels it. - Try it: Anchor a virtual gear train to a real wall in AR that ends inside a real object, with each gear's real speed computed and displayed. Twist: display how many years until the last gear completes one turn, counting down live. #### Cory's Yellow Chair — Arthur Ganson (1997) - Video: https://www.youtube.com/watch?v=fFG-Lk9c2CI - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, wire, gears, chair parts - Idea: The exploded view is a powerful AR move: taking a real object apart in space and reassembling it explains it and delights at once. - What it is: A small yellow chair continuously bursts apart into its separate pieces, which float outward and then come back together into a whole chair again. - Technique: Each chair part is held on its own wire arm driven by a shared cam system, so all pieces move out and back in coordinated radial paths. - Try it: Scan a real chair, split the model into parts in Reality Composer or Unity, and animate an exploded view that loops around the real chair in AR. Twist: let the viewer's distance control how far the parts fly apart. ### Asobo Studio *Game developer; HoloLens launch-title studio (later HoloForge Interactive)* Bordeaux game studio that built the most ambitious HoloLens launch titles and later spun out HoloForge Interactive for holographic work. #### Fragments — Asobo Studio (2016) - Video: https://www.youtube.com/watch?v=m6Wndguve8U - Interaction: Spatial Mapping, Portals & Worlds, Play - Platform & tech: Headset, HoloLens, spatial mapping - Idea: Your real home becomes a crime scene, with virtual characters sitting on your sofa. - What it is: A HoloLens crime thriller that maps your home and stages the investigation inside it: life-size characters sit on your real furniture and clues and memory reconstructions appear in your room. - Technique: Spatial mapping classifies seats, tables and floors in the home, and the story system places characters and clues on matching surfaces, so each room produces a different staging. - Try it: Detect the horizontal surfaces in a room and place the clues of a "crime scene" on the highest and lowest ones. Twist: have an AR character sit on a real chair and describe the case. #### Young Conker — Asobo Studio (2016) - Video: https://www.youtube.com/watch?v=U91aOkLGD6w - Interaction: Spatial Mapping, Play - Platform & tech: Headset, HoloLens, spatial mapping - Idea: Scan the room and automatically turn furniture and floors into platformer levels. - What it is: A HoloLens platformer that scans your room and procedurally turns furniture and floors into levels for the squirrel Conker to run across. - Technique: The room mesh is analyzed for walkable horizontal surfaces, and a procedural level generator places platforms, collectibles and paths across real furniture. - Try it: Find all horizontal surfaces in a room with plane detection and automatically generate jump paths and coins between them. Twist: make the character slip off the edges of real objects. #### HoloForge Lab: HoloParticles tennis — Asobo Studio (2021) - Video: https://www.youtube.com/watch?v=CCW7XMOyR2Q - Interaction: Hands & Body, Perception & Effects, Play - Platform & tech: Headset, HoloLens 2 - Idea: Play a game of holographic tennis with holographic particles and hand tracking. - What it is: A HoloForge Lab experiment with holographic particle effects on HoloLens 2, ending in a mock holographic tennis rally with hand-tracked swings. - Technique: GPU particle effects are world-anchored, and a hand-tracked palm or racket collider transfers velocity to a ball whose trajectory is simulated with simple physics. - Try it: Use a phone as a racket that sends an AR ball flying based on acceleration when you swing, leaving a particle trail along its path. Twist: make the ball bounce back after hitting a real wall. ### BARTKRESA studio (Bart Kresa) *Projection designer; founder of BARTKRESA studio* Projection designer whose Los Angeles studio maps architecture, sculptures and live events, from Game of Thrones premieres on Frank Gehry's Walt Disney Concert Hall to the 360° projected sculpture Shogyo Mujo, shown at Burning Man, Adobe MAX and SIGGRAPH and later installed at AREA15 in Las Vegas. #### Space Shuttle Endeavour Projection Mapping — BARTKRESA studio (Bart Kresa) (2013) - Video: https://www.youtube.com/watch?v=_pTiq2nrvDc - Interaction: Projection, Tangible Objects - Platform & tech: Projection, projection mapping, museum - Idea: A real historic object is more moving than a replica; AR in museums should augment the artefact itself rather than a screen next to it. - What it is: At the California Science Center in Los Angeles, the retired Space Shuttle Endeavour itself becomes the projection surface: light, colour, texture and animation move across its hull while the shape of the orbiter stays recognisable. - Technique: Projectors positioned around the orbiter are mapped to its 3D geometry; the event date is likely 2012–2013, soon after the shuttle arrived at the museum. - Try it: Pick a large object in your school, such as a piano, a statue or a bike, and create an AR layer that shows how it works inside, anchored with an object or image target. Twist: the layer tells the object's own life story in first person. #### Shogyo Mujo (with Josh Harker) — BARTKRESA studio (Bart Kresa) (2014) - Video: https://www.youtube.com/watch?v=yD99kirMs48 - Interaction: Projection, Perception & Effects - Platform & tech: Projection, 360° projection mapping, sculpture - Idea: A neutral white form is the perfect AR target: one physical shape can carry infinite materials and stories. - What it is: A large white skull sculpture designed with artist Josh Harker is wrapped in 360° projection that makes it crack, bloom, burn and change skin. Made for Burning Man 2014, it was shown at Adobe MAX 2014 and SIGGRAPH 2015 Emerging Technologies before travelling to AREA15. - Technique: Several projectors surround the sculpture and are calibrated to its 3D model, so textures wrap seamlessly all the way around. - Try it: 3D-print or buy a small white object, scan it, and use Vuforia Model Targets or Lens Studio object tracking to apply changing AR materials to it. Twist: the material changes each time a viewer walks a quarter turn around it. #### Game of Thrones Premiere, Walt Disney Concert Hall — BARTKRESA studio (Bart Kresa) (2017) - Video: https://www.youtube.com/watch?v=-TEylnercdA - Interaction: Projection, Location & City - Platform & tech: Projection, architectural projection mapping - Idea: Free-form architecture forces content to follow curves instead of rectangles; AR on curved surfaces must be designed for the surface, not just placed on it. - What it is: For the 2017 Game of Thrones season premiere in Los Angeles, Frank Gehry's curved steel Walt Disney Concert Hall was covered with projected ice, fire, dragons and banners from the series. - Technique: Projection mapping on a highly curved façade requires an accurate 3D model and careful projector placement to avoid distortion across the steel sails. - Try it: Scan a curved object such as a car, a bottle or a column with a LiDAR phone, then map an animated texture that flows along its curves in Reality Composer or Unity. Twist: the texture flows faster where the curvature is sharper. #### SVIATOVID — BARTKRESA studio (Bart Kresa) (2019) - Video: https://www.youtube.com/watch?v=G8eV_Qy7nGQ - Interaction: Projection, Perception & Effects - Platform & tech: Projection, 360° projection mapping, sculpture - Idea: A sculpture with a face on every side rewards walking around it; AR objects should give each viewing direction its own reason to look. - What it is: A 15-foot, four-faced sculpture inspired by the Slavic deity Sviatovid, presented at ISE 2019. Projection turns each face into stone, metal, fire or pattern, so the figure seems to look in every direction at once. - Technique: A 360° array of projectors is aligned to a 3D model of the sculpture, with content authored per face and blended at the edges. - Try it: Place a four-sided AR totem in a room (WebXR or AR Foundation) where each side shows a different face and element. Twist: the face you are not looking at changes while your back is turned. ### Benoit Arbelot *Creative engineer and digital artist* French creative engineer who builds his own fluid and Physarum (slime-mould) simulation frameworks in Unity and uses them for short real-time films such as Lights of Nibel and Coriolis. #### 3D Physarum Transport Networks — Benoit Arbelot (2019) - Video: https://vimeo.com/379589358 - Source code: https://github.com/Barbelot/Physarum3D - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, Unity, Compute Shader - Idea: Slime-mould logic in 3D: simple agents following their own trails build networks. - What it is: Millions of agents crawl through a 3D volume, leaving trails that they then follow, so glowing vein-like networks grow and reorganise on their own. - Technique: Following Jeff Jones's Physarum model, compute shaders move agents that sense a 3D trail texture ahead of them, turn toward the strongest trail, deposit into it, and the trail volume is diffused and decayed each frame before volume rendering. - Try it: Seed the Physarum agents from a LiDAR point cloud of a real room in AR so glowing networks grow along the furniture edges. Twist: let the user place food sources by tapping, and watch the network reroute. #### Chimera: Particle Advection — Benoit Arbelot (2020) - Video: https://x.com/BenoitArbelot/status/1297605129114390534 - Source code: https://github.com/Barbelot/Chimera - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, Unity, VFX Graph, HDRP - Idea: Let a fluid, not a noise function, decide where particles go. - What it is: VFX Graph particles are carried along by a live 2D fluid simulation, curling into vortices wherever the fluid is stirred. - Technique: The fluid's velocity is stored in the R and G channels of a render texture, which VFX Graph samples each frame to set particle velocity. - Try it: Lay the fluid texture over an AR table top and emit particles from a tracked real object as it slides across. Twist: make the particles sample the fluid only inside the camera's people-occlusion mask. #### Lights of Nibel — Benoit Arbelot (2020) - Video: https://vimeo.com/458749435 - Source code: https://github.com/Barbelot/Chimera - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, HDRP, Custom Render Texture - Idea: Light as a liquid that creatures stir as they pass. - What it is: A short real-time film inspired by the Ori games: spirits of light drift through a dark forest, stirring glowing fluid that swirls and fades behind them. - Technique: Chimera, his 2D fluid framework, solves a fast vorticity-confined fluid in a custom render texture; emitters inject velocity and colour, and the velocity field advects dye and VFX Graph particles. - Try it: Map Chimera's fluid texture onto a detected AR wall and let virtual fireflies (VFX Graph particles) stir it, then let the user's finger become another emitter. Twist: use the room's real light level to set how bright the fluid glows. #### Coriolis — Benoit Arbelot (2021) - Video: https://vimeo.com/543236276 - Source code: https://github.com/Barbelot/Physarum3D - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: The same growth rule reads as cells, rivers or galaxies depending on the scale. - What it is: A real-time journey through different scales of time and space, where Physarum networks and fluid currents form nebula-like structures that grow and dissolve. - Technique: His Physarum and fluid simulation frameworks run together in Unity: agent trails form the structure while the fluid field advects and distorts them. - Try it: Place a Physarum-plus-fluid volume above a table in AR and let the phone's zoom (distance to the volume) change the simulation scale. Twist: freeze the growth when the user holds still, so the piece rewards patience. ### Caitlin Fisher *Digital-literature author; director of the Augmented Reality Lab, York University* Electronic-literature writer (the award-winning hypertext novel These Waves of Girls) and York University professor who founded one of the first art-focused augmented reality labs. Her lab built SnapDragonAR and a string of tabletop AR poems, pop-up books and history projects. #### AR Pop-up Book (Future Stories) — Caitlin Fisher (2010) - Video: https://www.youtube.com/watch?v=4wXfk1Dcd5I - Interaction: Tangible Objects, Drawing & Making - Platform & tech: Desktop, Web, FLARToolkit, Flash - Idea: Anyone can make a pop-up book whose pages come alive on screen. - What it is: A do-it-yourself augmented reality pop-up book from York University's AR Lab: each page's marker makes images and video pop up in 3D, released as a kit anyone can fill with their own pictures. - Technique: A reusable FLARToolkit Flash template reads a playlist file, so each printed marker page shows the image or movie listed for it in a webcam view. - Try it: Make a four-page zine where each page is an image target that pops up a 3D scene in WebXR or Lens Studio. Twist: the last page shows the reader's own face. #### Circle — Caitlin Fisher (2011) - Video: https://www.youtube.com/watch?v=xdjRLfERsas - Interaction: Tangible Objects, Voice & Sound, Performance - Platform & tech: Desktop, Phone, SnapDragonAR, Vuforia - Idea: Reading becomes walking around a miniature stage. - What it is: A tabletop AR theatre in which small scenes and voices appear on a circle of markers, letting the reader move around the table to assemble a story; shortlisted for the UK New Media Writing Award. - Technique: SnapDragonAR, the York AR Lab's marker-tracking tool, maps video and sound to markers arranged in a ring; a later version was rebuilt for iPad with Vuforia. - Try it: Place eight image markers in a circle on a table and attach a short scene to each, so walking around tells a story. Twist: the ending depends on which direction the reader walks. #### Requiem — Caitlin Fisher (2011) - Video: https://www.youtube.com/watch?v=B6xu4UI4GNE - Interaction: Tangible Objects, Voice & Sound - Platform & tech: Desktop, Phone, FLARToolkit, Unity, Vuforia - Idea: A poem you hold in your hand and turn to read. - What it is: An augmented reality poem: holding a marked card releases imagery and spoken sound that unfold over it, one fragment of her tabletop AR novel Cardamom of the Dead. - Technique: FLARToolkit tracks a black-and-white marker card and superimposes animated imagery and sound, later rebuilt in Unity with Vuforia. - Try it: Write a six-line poem, give each line its own printed card, and use image tracking to make each card release an image and a recorded voice. Twist: the poem changes meaning depending on card order. #### Augmented Reality Freedom Stories — Caitlin Fisher (2013) - Video: https://www.youtube.com/watch?v=ZXGcupvYnPY - Interaction: Tangible Objects, Information & UI, Voice & Sound - Platform & tech: Phone, Vuforia, Unity - Idea: Handle a replica document and hear the person it belonged to. - What it is: Seldom-told African Canadian histories of the Underground Railroad, including Harriet Tubman's journeys, come alive as AR documents and voices over hand-crafted objects made with community partners. - Technique: Primary sources are printed onto physical artefacts used as image targets, and a tablet app overlays video, audio and animated text on them. - Try it: Pick a local historical figure, print three of their documents as image targets, and record a voice reading each one in AR. Twist: the voice speaks only when the document is held upright. ### Carlos Cruz-Diez *Artist; researcher of colour as an event* Venezuelan artist (1923–2019) who treated colour as something that happens in real time and space, through series such as Physichromie, Transchromie, Chromointerference and the walk-in colour rooms of Chromosaturation. #### Physichromie — Carlos Cruz-Diez (1959) - Video: https://www.youtube.com/watch?v=NIMnBoQcsbo - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Desktop, painted lamellae, acrylic, relief - Idea: The colour you see depends on where you stand, a physical version of view-dependent materials that AR can exploit to reward walking around an object. - What it is: Reliefs of thin vertical coloured blades; as the viewer walks past, reflected colour between the blades creates hues that are not painted anywhere and change with position. - Technique: Coloured lamellae set perpendicular to the surface reflect light onto each other, so additive 'ambient' colours appear in the gaps and shift with viewing angle. - Try it: Make a WebXR or AR Foundation wall panel of thin fins whose shader colour depends on the angle between the camera and each fin, so the panel changes as you walk by. Twist: hide a word that only appears from one position. #### Chromointerference (Chromointerferent environment) — Carlos Cruz-Diez (1964) - Video: https://www.youtube.com/watch?v=Fma9IV8ojug - Interaction: Projection, Perception & Effects, Hands & Body - Platform & tech: Projection, projected line patterns, animation - Idea: Projected patterns that slide over walls and people make the body part of the display, showing how AR overlays can dress people as well as surfaces. - What it is: Moving patterns of fine coloured lines are projected over a room and the visitors in it; where the patterns overlap, new colours and moiré shapes appear on walls and bodies. - Technique: Two layers of coloured line patterns move against each other; their optical interference generates colours and shapes that are not present in either layer alone. - Try it: Use body segmentation (ARKit people occlusion or Lens Studio) to project moving coloured stripes onto a person and the wall behind them at different speeds, creating moiré. Twist: make the stripe frequency follow the music playing in the room. #### Chromosaturation — Carlos Cruz-Diez (1965) - Video: https://www.youtube.com/watch?v=RDN_yN3rwMA - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Projection, fluorescent tubes, coloured light, white rooms - Idea: Colour can be an environment you walk through instead of a surface you look at; AR colour grading of the whole passthrough view can do the same. - What it is: Three connected white rooms flooded with pure red, green and blue light; walking between them, visitors see colour saturate their vision and mix at the thresholds. - Technique: Monochrome light fills each chamber so the eye adapts and loses its usual colour references; the boundaries between rooms become zones of additive colour mixing. - Try it: Write a passthrough shader (Quest, Vision Pro or a phone camera filter) that floods the view with one pure colour per room-scale zone, blending at invisible thresholds. Twist: tie the colour to the user's walking speed. #### Transchromie — Carlos Cruz-Diez (1965) - Video: https://www.youtube.com/watch?v=FvZ60oj30WQ - Interaction: Perception & Effects - Platform & tech: Desktop, coloured transparent strips, acrylic - Idea: A physical filter that recolours the real world depending on where you stand is the direct ancestor of AR colour filters. - What it is: Rows of transparent coloured strips stand in front of the world; looking through them, the landscape and people behind are recoloured and the colours mix as you move. - Technique: Overlapping transparent coloured sheets act as subtractive filters, so the colour of the scene behind depends on how many strips the line of sight crosses. - Try it: Build a passthrough AR 'window' of virtual coloured strips that subtractively tint the camera image behind them, placed in a real doorway. Twist: let the strips slide when two people stand on opposite sides. ### Codrin Mihail (Kodrin) *Technical artist and interactive developer* Montreal technical artist working across Unity, Houdini and Unreal who open-sourced the URP-PSX retro shader kit, a compute-shader flow field and an HDRP VFX Graph workshop. #### GPU-FlowField — Codrin Mihail (Kodrin) (2020) - Video: https://x.com/CodrinMihail/status/1279055017094205440 - Source code: https://github.com/Kodrin/GPU-FlowField - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, Compute Shader - Idea: Let a noise-based vector field steer a huge particle swarm. - What it is: Hundreds of thousands of particles stream through a 3D flow field, forming swirling ribbons of light. - Technique: A compute shader fills a 3D grid of direction vectors in URP; each particle reads the nearest grid point every frame to steer its velocity, and the particles are drawn procedurally from a structured buffer. - Try it: Fill a real room with the flow field in AR and let the phone act as an attractor the particles swirl around. Twist: bend the field around surfaces from the AR room mesh. #### URP-PSX — Codrin Mihail (Kodrin) (2020) - Video: https://x.com/CodrinMihail/status/1254786582021697537 - Source code: https://github.com/Kodrin/URP-PSX - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, Shader Graph - Idea: Deliberately degrade rendering to evoke a specific era of graphics. - What it is: A modern Unity scene rendered to look like a PlayStation 1 game: wobbling vertices, pixelated textures, reduced colours and fog. - Technique: Shader Graph materials snap vertex positions to a low-resolution grid in clip space and use affine texture mapping, while URP post effects add colour-depth reduction, dithering and pixelation. - Try it: Render AR objects with the PSX look and apply the same pixelation to the camera feed, so the whole real room looks like a 1996 game level. Twist: let the resolution drop further the farther objects are from the phone. #### Ghost of Tsushima lanterns (VFX Graph) — Codrin Mihail (Kodrin) (2021) - Video: https://x.com/CodrinMihail/status/1358992850009088000 - Source code: https://github.com/Kodrin/GoTLanterns-VFXGraph - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, Unity, VFX Graph, Shader Graph, HDRP - Idea: Recreate a memorable game atmosphere entirely on the GPU. - What it is: Hundreds of glowing water lanterns bob and drift on a dark surface and part around the player, recreated from the duel scenes of Ghost of Tsushima. - Technique: A fully commented HDRP VFX Graph spawns lantern meshes as particles on the water plane, animates bobbing and drift with noise, pushes them away from the player position, and a Shader Graph material adds emissive flicker. - Try it: Float the lanterns over a real floor or pond in AR and let them drift away from the viewer's feet. Twist: let each lantern light up when someone speaks near the phone. #### HDRP VFX Graph Workshop (VFX-Essentials) — Codrin Mihail (Kodrin) (2021) - Video: https://x.com/CodrinMihail/status/1366480535862837256 - Source code: https://github.com/Kodrin/VFX-Essentials - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, HDRP - Idea: Teach VFX Graph by starting from waves: position particles with simple math before anything else. - What it is: Grids of particles ripple in sine waves, rings and interference patterns — the first building blocks of a VFX Graph course. - Technique: Each sample drives particle positions in VFX Graph with sine, distance and time operators on a grid, then layers in noise, SDFs and textures in later sections. - Try it: Rebuild the wave sample with AR Foundation + URP VFX Graph on a phone, laying the wave grid on the floor so it ripples out from the user. Twist: set the wave centre to wherever the user last tapped. ### Collusion *Art and technology commissioning organisation (ART // TECH // PLAY)* Cambridge-based producer that commissions interactive and projection-based public artworks, often co-created with young people in King's Lynn and the East of England, and runs the ART // TECH // PLAY artist network where AΦE spoke about their VR and AR work. #### The Intergalactic Hanseatic League — Collusion (2021) - Video: https://www.youtube.com/watch?v=iz8NmgJvhd0 - Interaction: Projection, Location & City, Information & UI - Platform & tech: Projection, projection mapping - Idea: Historic buildings receive messages from the future about the town's climate. - What it is: A month-long sci-fi time-travel adventure across King's Lynn town centre and online, with projected transmissions on historic buildings telling a climate story co-created with local children. - Technique: Large-format projection mapping on town-centre facades plays episodic 'transmissions', linked to online content and school workshops in which children wrote the story. - Try it: Write a one-minute 'message from 2100' about your street and project it onto the school building after dark with a borrowed projector. Twist: the message must refer to a real detail of the wall it is projected on. #### The Multitude — Collusion, Jamie Gledhill (2021) - Video: https://www.youtube.com/watch?v=NywjgxG6JrM - Interaction: Hands & Body, Projection, Play - Platform & tech: Projection, body tracking, projection, game engine - Idea: Your projected body is the game controller in a quest to save nature. - What it is: A 20-minute playable story for two people who see projections of their own bodies and must use them to win over four elemental spirits of nature, some of whom are not impressed with humanity. - Technique: A depth or body-tracking camera turns the two players' silhouettes into projected characters that trigger story events when they reach, pose or cooperate (developed by Jamie Gledhill; likely Kinect with a game engine). - Try it: Make a two-player shadow game: project each player's webcam silhouette in a different colour and ask them to overlap to 'unlock' a sound. Twist: one spirit only responds if the players move very slowly. #### Sound Mirrors — Collusion (2023) - Video: https://www.youtube.com/watch?v=U5ol3hcQY4w - Interaction: Voice & Sound, Projection, Location & City - Platform & tech: Projection, sound sculpture, projection - Idea: A town listens to its young people through a giant ear and a projected image. - What it is: A large sound sculpture and projection in King's Lynn that played back young people's reflections on their identity and their town, inviting passers-by to stop and listen. - Technique: A physical sound-mirror structure focuses recorded voices toward listeners while projected visuals on nearby surfaces show the words and imagery made in workshops. - Try it: Record classmates answering 'what does this place sound like?', then play the clips from a parabolic umbrella so only one person at a time can hear them. Twist: project the transcript only when someone is standing at the listening spot. #### Walking With Pride: The Colours of History — Collusion (2025) - Video: https://www.youtube.com/watch?v=K0AKnAfEPsA - Interaction: Location & City, Information & UI - Platform & tech: Phone, WebAR, location-based AR - Idea: Hidden histories reappear exactly where they happened when you raise your phone. - What it is: A town-wide augmented reality trail and exhibition in King's Lynn that reveals the town's LGBTQ+ histories at the places where they happened, co-created with local young creatives. - Technique: Location-anchored phone AR places illustrated figures and stories at sites around the town centre, supported by printed trail maps and a gallery exhibition (likely WebAR image or geo anchors). - Try it: Research one forgotten story from your neighbourhood, draw its main character and anchor it with WebAR at the real location. Twist: the character must point toward the next stop of the trail. ### Craig Walsh *Artist (site-specific projection on trees, landscapes and architecture)* Australian artist who projects video into found sites: faces of local residents onto trees (Humannature, Monuments), creatures into city drainage channels (Incursion), and community stories onto buildings (Digital Odyssey). His work has toured to Havana, Taipei, Toronto and San Jose. #### Humannature (Adelaide) — Craig Walsh (2009) - Video: https://vimeo.com/8420194 - Interaction: Face, Location & City, Projection - Platform & tech: Projection, video projection, trees as screens - Idea: A face on an organic, uneven surface feels alive because the surface breaks and animates it; AR faces on real objects gain presence from the material they sit on. - What it is: At night, the huge faces of local residents are projected onto the canopies of park trees; the faces blink, look around and slowly change expression, so the trees seem to be watching the people below. - Technique: High-lumen projectors aimed at tree canopies play long-take portrait videos, with the leaves acting as a textured, moving screen. - Try it: Film a 30-second close-up of a classmate slowly looking around, then in phone AR place it on a real tree or bush using a vertical anchor and a leaf-shaped mask. Twist: the face turns to look at whoever holds the phone. #### HOME (Alice Springs) — Craig Walsh (2010) - Video: https://vimeo.com/35550168 - Interaction: Location & City, Projection, Shared & Social - Platform & tech: Projection, video projection, community video - Idea: The people of a place are the best content for that place; location-based AR can make residents visible on their own streets. - What it is: Part of the Digital Odyssey project, portraits and stories of Alice Springs residents are projected onto local buildings and landscape features at night, made in collaboration with the community. - Technique: Community-recorded portrait videos are projected site-specifically onto walls and natural features from mobile projection rigs. - Try it: Interview three people who live or work near a spot on campus, and geo-anchor their short video portraits to the walls they chose using location-based AR. Twist: the portrait speaks only when the viewer stands where that person usually stands. #### Incursion, 25.033°N / 121.633°E (Taipei) — Craig Walsh (2010) - Video: https://vimeo.com/34827533 - Interaction: Location & City, Projection, Spatial Mapping - Platform & tech: Projection, 4-channel projection, site-specific video - Idea: Overlooked infrastructure becomes a stage when something unexpected appears in it at the right scale, a strong idea for city-scale AR. - What it is: A four-channel projection in a concrete drainage channel of Taipei's Zhonggang river corridor shows giant creatures and figures moving through the waterway in a 14-minute loop, as if the infrastructure were inhabited. - Technique: Four synchronized projectors map a looping video onto the walls and surfaces of an urban drainage channel. - Try it: Find a neglected corner of campus (a drain, stairwell, loading dock) and place a large animated creature there in phone AR that moves along the real geometry. Twist: the creature hides when more than one phone looks at it. #### Monuments (Virginia Tech Drillfield) — Craig Walsh (2022) - Video: https://www.youtube.com/watch?v=82wypNISZpw - Interaction: Face, Location & City, Projection - Platform & tech: Projection, video projection, trees as screens - Idea: Replacing the statue with a living face on a tree asks who deserves a monument; AR can place alternative monuments in public space the same way. - What it is: Large-scale faces of Virginia Tech community members are projected onto three trees on the campus Drillfield at night, turning the trees into living monuments to people rather than to statues. - Technique: Projectors on the lawn throw portrait videos onto tree canopies, using the foliage as a three-dimensional, breathing screen. - Try it: Pick one statue or plaque on campus and geo-anchor an AR 'counter-monument' next to it: a life-size portrait video of a person the community would honour. Twist: visitors vote with a tap, and the most-voted portrait grows larger. ### Danlin Huang *Designer-researcher; China Academy of Art School of Design Innovation* Designs speculative mixed-reality rituals and tools - shared mushroom-mask MR, body-posture oracle glyphs, AR origami tutoring - often with Botao Amber Hu. #### Dialogue with Great Souls — Danlin Huang (2024) - Video: https://www.youtube.com/watch?v=DtI2JtVNGIU - Interaction: Portals & Worlds, Location & City, Shared & Social - Platform & tech: Phone, WebAR, portal rendering - Idea: Set down an AR 'Anywhere Door' on the street and walk through it to talk with ancient sages. - What it is: A street exhibition on Daxue Road, Shanghai with Soul App: scan a code, place an AR 'Anywhere Door' anywhere and step through into a scene to converse with eight historical sages. - Technique: A phone AR portal with stencil masking hides the inner scene until the user walks through the door plane, anchored at user-chosen positions along the street. - Try it: Build an Anywhere Door with phone AR that hides a scene of a historical figure of your choice and a voice clip you can only hear after walking through. Twist: the door can only be placed at campus locations connected to that figure. #### FungiSync — Danlin Huang, Botao 'Amber' Hu (2024) - Video: https://www.youtube.com/watch?v=BzvtihUKAaw - Interaction: Shared & Social, Performance, Hands & Body - Platform & tech: Headset, Phone, HoloKit, co-located multiplayer AR, hand tracking - Idea: Put on a mushroom mask and trade the way you see the world through a handshake. - What it is: A mixed-reality participatory ritual: participants wear mushroom-decorated MR masks and exchange 'worldviews' through secret handshakes, blending each other's visual filters like a mycorrhizal network. - Technique: Likely co-located phone-based MR headsets share an anchor and detect handshake gestures to swap and mix each participant's visual shader. - Try it: Have each group of classmates make a phone AR filter that represents their own 'worldview', and swap filters when two people high-five (detected by proximity or a QR scan). Twist: when three or more people meet, their filters blend. #### Body Oracle Translator — Danlin Huang, Botao 'Amber' Hu (2025) - Video: https://www.youtube.com/watch?v=GXeLSfZnXAQ - Interaction: Hands & Body, Information & UI, Perception & Effects - Platform & tech: Phone, pose estimation, generative AI glyphs, handheld MR - Idea: Translate body postures into AI-generated 'oracle-bone script' in real time. - What it is: A handheld MR device translates people's body postures in real time into speculative AI-generated hieroglyphs inspired by Chinese oracle-bone script. - Technique: Classifies body poses with a pose-estimation model and maps them to a pre-generated glyph set rendered as AR overlays next to the person. - Try it: Use phone body-pose recognition (Effect House or MediaPipe) to map five poses to five invented pictographs and overlay them next to the person when taking a photo. Twist: string a sequence of poses into a 'body sentence'. #### Foldiverse — Danlin Huang (2025) - Video: https://www.youtube.com/watch?v=DW8Xlm51S4A - Interaction: Tangible Objects, Information & UI, Voice & Sound - Platform & tech: Headset, Phone, mixed reality guidance, voice assistant - Idea: An AR origami teacher guides a child through folding, step by step. - What it is: An MR origami trainer for autistic children: a virtual 3D origami model and a voice assistant called FoFo guide each fold step-by-step over the real paper. - Technique: Shows animated 3D fold steps registered near the child's paper with voice prompts from an assistant, advancing when a step is complete. - Try it: Play a step-by-step 3D origami animation on a tabletop with phone AR, using a voice button to go to the next step. Twist: have the phone recognize the paper's shape to decide automatically whether the current step is done. ### David Rokeby *Artist; pioneer of interactive sound and vision installations* Canadian artist who since the early 1980s has built camera-based systems that answer body movement with sound (Very Nervous System) or look at objects and speak about them (The Giver of Names). #### Very Nervous System — David Rokeby (1986) - Video: https://www.youtube.com/watch?v=qdvyuvfKVU0 - Interaction: Hands & Body, Voice & Sound, Performance - Platform & tech: Desktop, video camera, custom image processing, synthesiser - Idea: The whole body as the interface, with no visible device, is the root of camera-based interaction; AR sound can use the same idea with body tracking. - What it is: In an empty room, a person's movements are watched by video cameras and turned instantly into music, so the body plays the space like an instrument. - Technique: Custom image-processing hardware and software measure motion in regions of the camera image and map speed and location to synthesiser parameters in real time. - Try it: Divide the room into invisible AR zones and use body tracking so that moving fast or slow in each zone triggers different sounds. Twist: add a delay so the room 'answers' a second later, like an echo of your body. #### The Giver of Names — David Rokeby (1991) - Video: https://www.youtube.com/watch?v=sO9RggYz24Q - Interaction: Tangible Objects, Information & UI, Voice & Sound - Platform & tech: Desktop, video camera, computer vision, speech synthesis - Idea: A machine that names objects badly but beautifully shows how recognition can be poetic; AR object labels do not have to be literal. - What it is: Visitors place any object on a pedestal; a computer looks at it with a camera and speaks aloud a strange, poetic sentence about what it thinks it sees. - Technique: The camera image is analysed for colour, shape and texture, and these features are fed into a network of word associations that generates grammatical sentences. - Try it: Point a phone at objects on a table and use an image-recognition model plus a language model to float a short poem above each item in AR. Twist: force the poem to avoid the object's real name. #### Watch — David Rokeby (1995) - Video: https://www.youtube.com/watch?v=LCDRZWNU7Ck - Interaction: Perception & Effects, Location & City - Platform & tech: Projection, video camera, real-time video processing - Idea: Splitting a live view into 'what moves' and 'what stays' is a simple filter that reveals time; AR reality filters can do this on the camera feed. - What it is: Two projections of a street: one shows only what moves (still things disappear), the other shows only what stays still (moving people blur into ghosts). - Technique: Real-time video processing uses frame differencing for the motion image and a long running average for the stillness image. - Try it: Write a camera shader with two modes — frame difference (only motion) and long-exposure average (only stillness) — and toggle it by tilting the phone. Twist: mix them so moving people leave colour while the city stays grey. #### Seen — David Rokeby (2002) - Video: https://www.youtube.com/watch?v=0Ai49w7QPPY - Interaction: Location & City, Perception & Effects, Information & UI - Platform & tech: Projection, video, motion analysis - Idea: Letting crowds draw their own paths across a square shows the hidden choreography of a place — a classic AR trace effect waiting to be anchored on site. - What it is: Made for the Venice Biennale, video of St Mark's Square is processed so that moving people leave trails and patterns of their paths across the piazza. - Technique: Motion analysis over time accumulates the positions of moving pixels into trails that are layered back onto the video. - Try it: Film a busy square with a phone on a tripod, extract people's paths with a person detector, and replay the paths as glowing lines anchored to the real ground in AR. Twist: colour each path by walking speed. ### Don Allen Stevenson III *XR creator, educator and former DreamWorks specialist trainer* Former DreamWorks Animation trainer who became a full-time XR creator, making wearable AR fashion, full-body Lenses and mixed-reality sketches, and now teaching creative AI tools. #### Doc Ock AR Digital Fashion — Don Allen Stevenson III (2021) - Video: https://www.youtube.com/watch?v=zSeiswhKAjo - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, Lens Studio, Cinema 4D, Substance 3D Painter - Idea: Wear a movie character's signature prosthetic as digital fashion. - What it is: A wearable Snapchat filter that gives the user Doctor Octopus's animated mechanical tentacles, which move around the body as if attached to the spine. - Technique: Tentacles modelled and animated in Cinema 4D, textured in Substance 3D Painter, and attached to Lens Studio body tracking. - Try it: Design an extra limb (wing, tail or arm) as a body-tracked AR wearable that reacts to how fast you move. Twist: the limb must help you do something the real body cannot. #### Lil Nas X Met Gala Armour Filter (in 4 hours) — Don Allen Stevenson III (2021) - Video: https://www.youtube.com/watch?v=2MfLSKx-6Bo - Interaction: Hands & Body, Drawing & Making - Platform & tech: Phone, Gravity Sketch, Lens Studio, Cinema 4D, Substance 3D Painter - Idea: Treat AR fashion as real-time commentary: react to a pop-culture moment the same day. - What it is: Inspired by Lil Nas X's golden armour at the Met Gala, Don Allen III sculpts and publishes a wearable AR version on Snapchat within about four hours, showing every step. - Technique: Armour sculpted in Gravity Sketch VR, refined in Cinema 4D and Substance 3D Painter, then skinned to the body mesh in Lens Studio. - Try it: Pick a news or pop-culture moment from this week and make a wearable AR piece about it in under three hours. Twist: it must be readable without any text or logos. #### Rift — Don Allen Stevenson III, Paige Piskin (2021) - Video: https://www.youtube.com/watch?v=0rChMGXYRmY - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, Lens Studio, Body Tracking, Cinema 4D - Idea: Two creators co-design one Lens that restyles the entire body, not just the face. - What it is: A full-body Snapchat Lens co-created by Don Allen Stevenson III and Paige Piskin: crystal shards burst from the wearer's shoulders and chest, the eyes glow white, and the scene shifts into a psychedelic colour palette. - Technique: Lens Studio full-body tracking and 3D body mesh attach rigged costume pieces and VFX to the user's skeleton. - Try it: In pairs, design a full-body Lens where one partner makes the costume and the other makes the environment effect, then merge them. Twist: the costume must change when the wearer raises both arms. #### Sandworm (Meta Quest 3 + Apple Vision Pro) — Don Allen Stevenson III (2024) - Video: https://www.youtube.com/watch?v=LIJpe5evVjE - Interaction: Spatial Mapping, Location & City, Perception & Effects - Platform & tech: Headset, Meta Quest 3, Apple Vision Pro - Idea: Make giant fiction erupt from the actual terrain around you, not from a flat floor. - What it is: A short mixed-reality sketch in which a Dune-style sandworm rises out of real, uneven outdoor ground, built by combining a Meta Quest 3 and an Apple Vision Pro. - Technique: The Quest 3 is likely used to scan and place the creature on uneven terrain, while the Vision Pro views or records the passthrough composite. - Try it: Scan an outdoor patch of ground with a phone's LiDAR or photogrammetry and make a creature emerge from it that follows the real bumps. Twist: the creature must hide again when someone walks toward it. ### Doug Aitken *Artist and filmmaker* American artist whose mirrored house (Mirage), mirrored hot-air balloon (New Horizon) and underwater pavilions dissolve architecture into the landscape it reflects. #### Underwater Pavilions — Doug Aitken (2016) - Video: https://www.youtube.com/watch?v=gzGgzf361ZE - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, mirrored geodesic sculptures, sea floor, diving - Idea: A sculpture you can only visit by entering another element (water) shows how an AR piece can require a special trip or state to be seen. - What it is: Off Catalina Island, three geodesic sculptures with mirrored and rough surfaces are anchored under the sea; divers swim through them as the reflections of fish and water shift. - Technique: Faceted structures of mirror and textured panels were moored to the sea floor at depths reachable by recreational divers. - Try it: Make an AR piece that only appears when the phone is inside a clear bag in a pool or bathtub (detected by camera blur or a colour cue). Twist: the piece reflects any fish or swimmers passing by. #### Mirage — Doug Aitken (2017) - Video: https://www.youtube.com/watch?v=Gn3BUue9nEc - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, mirrored panels, ranch-style house, desert - Idea: A mirrored object is only its surroundings, reframed: a perfect physical reference for environment-reflective AR materials and 'invisible' objects. - What it is: For Desert X near Palm Springs, a single-storey ranch house is covered inside and out with mirrors, so it reflects the desert and mountains and seems to vanish into them. - Technique: A timber frame house shape is clad with mirror panels on every surface, including the interior, so views multiply inside. - Try it: Place an AR house with a real-time environment-probe mirror material (ARKit environment texturing) in an open landscape and see how it disappears. Twist: open its door into a mirrored interior that reflects the user infinitely. #### Mirage Gstaad — Doug Aitken (2019) - Video: https://www.youtube.com/watch?v=wdJP-sIGgS4 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, mirrored house, alpine meadow, snow - Idea: The same object in a new landscape becomes a new work, which shows why AR assets should be designed to take on the character of each site. - What it is: For Elevation 1049 in Gstaad, Switzerland, the mirrored house moves to a snowy Alpine meadow, where it reflects mountains, snow and sky and changes with every season. - Technique: The mirror-clad structure was rebuilt on a mountain site and left through winter and summer so the reflected landscape transforms it. - Try it: Use the same mirrored AR object in three different outdoor places (park, street, rooftop) and record how each site changes it. Twist: let the object keep a faint memory texture of the last place it was placed. #### New Horizon — Doug Aitken (2019) - Video: https://www.youtube.com/watch?v=u3C7cAa6Q8A - Interaction: Location & City, Shared & Social, Perception & Effects - Platform & tech: Desktop, mirrored hot-air balloon, travelling events - Idea: A reflective object in the sky becomes a gathering point for a community, showing how a shared AR landmark can structure an event. - What it is: A 30-metre mirrored hot-air balloon travelled across Massachusetts, rising at dawn and dusk to reflect the sky and land, with talks and music at each stop. - Technique: A hot-air balloon envelope made of reflective fabric flies tethered and in free flight, lit at night from within. - Try it: Anchor a large reflective AR sphere in the sky above a school event and let everybody's phone show the same sphere with synchronised lighting. Twist: the sphere rises higher as more people look at it. ### Dumb Type *Artist collective (performance, installation, projection)* Founded in 1984 by students of Kyoto City University of Arts, including Teiji Furuhashi and Shiro Takatani, with sound by Ryoji Ikeda from the 1990s. Their performances pH, S/N, OR and memorandum stage bodies inside scanning light, projected text and data, and represented Japan at the Venice Biennale in 2022. #### pH — Dumb Type (1990) - Video: https://www.youtube.com/watch?v=kjWm4zm94lE - Interaction: Performance, Hands & Body, Projection - Platform & tech: Projection, motorized truss, floor projection, strobe - Idea: A machine that sweeps the space turns people into data being read; AR interfaces can also 'scan' users, and designers can make that sensing visible and physical. - What it is: Performers move on a white floor while a large motorized truss slowly scans back and forth above them, forcing them to duck, jump or lie flat; projected images and lines sweep the floor like a copier or scanner. - Technique: A moving overhead truss carrying lights and projectors travels across the stage on a timed cue sequence, with floor projection and strobes synchronized to the sound. - Try it: Make a WebXR or Lens Studio scene where a virtual scanning bar sweeps across the real floor and people must dodge it; detect bodies with segmentation and flash red when they touch it. Twist: whoever is hit gets copied and left behind as a frozen silhouette. #### Lovers (Teiji Furuhashi) — Dumb Type (1994) - Video: https://www.youtube.com/watch?v=ZAaYEZN7EwI - Interaction: Projection, Gaze & Attention, Hands & Body - Platform & tech: Projection, video projectors, rotating projection unit, sensors - Idea: Life-size figures that walk around you on the walls make a room feel inhabited; scale and eye contact matter more than realism when placing virtual people. - What it is: In a dark square room, life-size naked figures are projected onto the four walls; they walk, run, turn toward the visitor and embrace, and some respond to the visitor's presence. Text such as 'Love is everywhere' drifts across them. - Technique: Computer-controlled video projectors on a rotating central unit move the projected figures around the four walls, with sensors that likely trigger some figures to approach the visitor. - Try it: Capture three classmates walking in a volumetric or green-screen video, then place them life-size in phone AR so they circle the viewer along the room's walls. Twist: when the viewer looks at one of them, that figure stops and turns to face them. #### OR — Dumb Type (1997) - Video: https://www.youtube.com/watch?v=2ZMYPXvR_TU - Interaction: Performance, Perception & Effects - Platform & tech: Projection, video projection, strobe, white cyclorama - Idea: Flashes of light can make a real person appear and disappear, a stage-scale version of the fades and occlusion an AR designer uses to add or remove things from reality. - What it is: On a curved, blindingly white stage, performers appear and vanish in bursts of light, projection and sound by Ryoji Ikeda; the piece reflects on the thin line between life and death, as in an operating room. - Technique: A white curved cyclorama acts as both floor and screen; high-intensity strobes, video projection and precisely cued sound hide and reveal performers. - Try it: Use phone AR people occlusion to make a classmate 'fade out' of the camera view when a white flash covers the screen, and fade back in with a delay. Twist: each disappearance leaves a lingering glowing outline in the spot. #### memorandum — Dumb Type (1999) - Video: https://www.youtube.com/watch?v=H9MlE1eXvGM - Interaction: Performance, Information & UI, Projection - Platform & tech: Projection, scrim projection, video, lighting - Idea: A semi-transparent layer between the viewer and the performer is exactly an AR display; memorandum shows how text and image on that layer can rewrite the person behind it. - What it is: Performers move behind and in front of a translucent screen that carries projected images, text and flickering data; they seem to be stored, erased and replayed like memories. - Technique: A large scrim in front of the stage takes front projection while backlight reveals performers behind it, so projected content and live bodies overlap in one plane. - Try it: Hang a sheet of tulle or tracing paper, project text onto it and have a classmate stand behind it lit from behind; then recreate the layer in phone AR with a transparent text plane that follows the person. Twist: the text shows the person's own spoken words transcribed live. ### Eadweard Muybridge *Photographer; pioneer of motion photography (1830–1904)* English-born photographer who used banks of tripwire-triggered cameras to split a galloping horse, athletes and animals into sequences of still frames, then projected them as moving images with his zoopraxiscope. #### The Horse in Motion (Sallie Gardner at a Gallop) — Eadweard Muybridge (1878) - Video: https://www.youtube.com/watch?v=1X9UmOps-ag - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, wet-plate cameras, tripwire shutters - Idea: Space the moments of a movement along a line and people can walk through time: lay out a motion as a row of frozen poses in AR and let the viewer's position choose the frame. - What it is: Twelve cameras along a Palo Alto racetrack fire in turn as the horse breaks their tripwires, catching the gallop in a row of frames that shows all four hooves off the ground; played in sequence, the frames move. - Technique: A row of cameras with fast shutters is triggered by tripwires as the subject passes, so camera position encodes time. - Try it: Record a 2-second jump with a phone, extract 12 frames, cut out the person and place the cut-outs as billboards in a line on the floor in WebXR or Lens Studio. Twist: scrub the animation by walking along the row instead of pressing play. #### Zoopraxiscope — Eadweard Muybridge (1879) - Video: https://www.youtube.com/watch?v=_eGcxp-TeFA - Interaction: Perception & Effects, Tangible Objects, Projection - Platform & tech: Desktop, glass disc, projector, shutter - Idea: Motion is an illusion built from still frames plus a shutter: an AR designer can make any looping animation feel physical by tying it to a real rotating object. - What it is: A spinning glass disc painted with figures traced from Muybridge's photographs is projected through a counter-rotating shutter, so the horses and athletes appear to run on the screen; shown here as a working replica. - Technique: Sequential images on a rotating disc are projected through a synchronised slotted shutter, so each frame flashes briefly in place. - Try it: Track a spinning record or lazy Susan with image tracking in Lens Studio or AR Foundation and attach a 12-frame loop that advances with the measured rotation angle. Twist: make the animation run backwards when the disc is spun the other way. #### The Attitudes of Animals in Motion: Athletes — Eadweard Muybridge (1881) - Video: https://www.youtube.com/watch?v=RReKrWokC7Q - Interaction: Perception & Effects, Hands & Body, Information & UI - Platform & tech: Desktop, multiple cameras, gridded backdrop - Idea: A measuring grid behind a moving body turns a gesture into data you can read — the same role a visible AR floor grid can play when teaching movement. - What it is: Plates of athletes running, jumping and somersaulting against a numbered grid, each movement broken into a dozen poses; here animated back into motion from the original photographs. - Technique: Multiple cameras photograph the subject in front of a lined backdrop so positions can be compared frame to frame. - Try it: Build a WebXR scene with a 1 m grid on the floor and record a friend's body pose with a phone pose-tracking model, then replay the skeleton as 10 translucent snapshots over the grid. Twist: color each snapshot by speed so fast phases glow. #### Animal Locomotion — Eadweard Muybridge (1887) - Video: https://www.youtube.com/watch?v=07x7KhuwwFE - Interaction: Perception & Effects, Hands & Body - Platform & tech: Desktop, multi-camera rig, clockwork timer - Idea: Shooting one motion from several sides at once is volumetric capture in its earliest form — the idea behind today's AR volumetric people. - What it is: The University of Pennsylvania series of 781 plates showing people and animals walking, carrying, climbing and flying, shot from several angles at once; the frames are animated in this video. - Technique: Banks of cameras at the side, front and back are fired by an electrical clockwork timer so each phase is seen from multiple viewpoints. - Try it: Film one short action from three phones placed at 0, 90 and 180 degrees, then place the three clips as synced vertical panels around a point in AR so walking around the point switches views. Twist: replace the panels with a single Gaussian-splat or photogrammetry capture and compare. ### Erik Nordeus (Habrador) *Developer, technical artist and tutorial author* Swedish engineer who publishes Unity tutorials and open-source simulations under the name Habrador, from fluid solvers and tornadoes to ray-marched and parallax shaders. #### Tornado Simulator — Erik Nordeus (Habrador) (2016) - Video: https://www.youtube.com/watch?v=IlhdLQ5NU5E - Source code: https://github.com/Habrador/Unity-Tornado-Simulator - Interaction: Play, Perception & Effects - Platform & tech: Desktop, Unity, C#, Particle System - Idea: A tornado is a few forces: pull in, lift up, spin around. - What it is: A twister sweeps across a small town, sucking up cars, cows and debris and flinging them around its funnel. - Technique: Rigidbodies inside the funnel get a radial pull toward the axis, an upward lift and a tangential force, while particle systems draw the funnel and dust. - Try it: Drop a miniature tornado onto an AR table top and let it pick up virtual objects the user has placed there. Twist: make it suck up a real object's scanned copy from the LiDAR mesh. #### Parallax Occlusion Mapping Hole — Erik Nordeus (Habrador) (2017) - Video: https://www.youtube.com/watch?v=nvok7temQuI - Source code: https://github.com/Habrador/Unity-Advanced-Shaders-Tutorial - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Desktop, Unity, HLSL - Idea: A flat surface can fake depth by shifting its texture with the view angle. - What it is: A deep, mysterious hole appears in a flat ground plane and stays convincing as the camera moves, although the surface is a single flat quad. - Technique: Parallax occlusion mapping steps along the view ray through a height map in the fragment shader and offsets the texture coordinates to where the ray hits, faking a deep cavity. - Try it: Place the hole shader on an AR floor plane so a pit opens in the real floor and holds up as the user walks around it. Twist: animate the height map so something climbs out. #### Death Star Ray Marching Shader — Erik Nordeus (Habrador) (2022) - Video: https://www.youtube.com/watch?v=_MgfKdxa6lE - Source code: https://github.com/Habrador/Unity-Advanced-Shaders-Tutorial - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL, Ray marching - Idea: Two spheres and a subtraction make an iconic shape without a single polygon. - What it is: A Death Star floats in space, rendered not from a mesh but by a shader that marches rays into a signed distance field of a sphere with a carved dish. - Technique: A fragment shader on a cube ray-marches a signed distance field built from a sphere minus a smaller sphere, computes normals from the SDF gradient and shades the result. - Try it: Put the ray-marched SDF inside a cube anchored in AR so it looks like a solid object hovering in the room, then add a second SDF that blends with it smoothly. Twist: let the user carve the shape by tapping to subtract spheres. #### Eulerian Fluid Simulator — Erik Nordeus (Habrador) (2023) - Video: https://www.youtube.com/watch?v=6Jw1CsTOkDg - Source code: https://github.com/Habrador/Ten-Minute-Physics-Unity - Interaction: Perception & Effects, Drawing & Making - Platform & tech: Desktop, Unity, C# - Idea: A fluid is just a grid of velocities kept divergence-free. - What it is: Smoke-like streams flow around an obstacle the user drags through a wind tunnel, shedding vortices, in a grid fluid simulator written from scratch in Unity. - Technique: A staggered-grid Eulerian solver (after Matthias Müller's Ten Minute Physics) applies Gauss-Seidel projection for incompressibility and semi-Lagrangian advection of velocity and smoke, all in C#. - Try it: Port the solver to a compute shader and project it onto an AR floor plane, using the tracked phone position as the moving obstacle so walking leaves a wake. Twist: add real walls from the LiDAR mesh as solid cells. ### EsProgram *Unity engineer* Tokyo Unity engineer behind InkPainter, a widely used open-source texture-painting library, and its companion water-wave shaders. #### InkPainter — EsProgram (2016) - Video: https://www.youtube.com/watch?v=TWGK5UQ6KsU - Source code: https://github.com/EsProgram/InkPainter - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Desktop, Unity, C#, RenderTexture - Idea: Paint directly onto any mesh at runtime, like spraying ink on real objects. - What it is: The user splashes coloured ink onto 3D objects with the mouse; the paint wraps around curved surfaces and stays on them. - Technique: A raycast hit gives the UV coordinate on the mesh, and a shader blits a brush texture into the object's RenderTexture at that UV, with blending modes and normal/height painting. - Try it: Use AR Foundation meshing so the user can spray virtual ink onto the real room mesh with a tap. Twist: make two phones share the paint so a class can paint the same wall together. #### InkPainter with Google Tango — EsProgram (2017) - Video: https://www.youtube.com/watch?v=dA-rf3lkRg0 - Source code: https://github.com/EsProgram/InkPainter - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Phone, Unity, Google Tango, C# - Idea: Graffiti on the real world by painting onto its scanned mesh. - What it is: Holding a Tango phone, the user sprays ink onto a scanned room; the paint lands on real walls and furniture in the camera view. - Technique: Google Tango's depth sensing reconstructs the room as a mesh; InkPainter paints into that mesh's texture from phone raycasts and the mesh is rendered invisibly except for the paint. - Try it: Re-create this with AR Foundation meshing on a LiDAR iPhone: hide the room mesh and show only the ink. Twist: let the ink drip down vertical surfaces over time. #### WaveShaderDemo (water surface shader) — EsProgram (2018) - Video: https://www.youtube.com/watch?v=HBlMCAyFIkA - Source code: https://github.com/EsProgram/WaveShaderDemo - Interaction: Perception & Effects, Drawing & Making - Platform & tech: Desktop, Unity, HLSL, RenderTexture - Idea: A minimal, readable wave simulation that makes any plane react like water. - What it is: A calm water surface in Unity where the cursor leaves trails of ripples that spread and fade across the pond. - Technique: Two RenderTextures ping-pong a discrete wave equation in a fragment shader, and the result is sampled as a height/normal map by the surface shader. - Try it: Attach the water shader to an AR tabletop and let fingers dragged across the screen draw ripples on the table. Twist: turn the ripples into sound by sampling the height under a virtual microphone point. #### WaveformProvider — EsProgram (2018) - Video: https://www.youtube.com/watch?v=3RgRCOm0NAQ - Source code: https://github.com/EsProgram/WaveformProvider - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, Unity, HLSL, RenderTexture - Idea: Store the water surface as a texture and let a wave equation run on it. - What it is: Objects dropped into a pool send expanding ripples across the water, which interfere and bounce off the edges. - Technique: A shader solves the 2D wave equation on a height RenderTexture each frame; InkPainter writes impulses where objects touch the surface, and the height map drives normals for the water material. - Try it: Lay a virtual water surface over a real floor in AR and create ripples wherever the phone's position is projected onto it. Twist: make real objects detected by the camera leave ripples as they move. ### Fabin Rasheed *Artist, designer and technologist (Nurecas)* Indian artist-designer who prototyped ARKit sculpting, multiplayer AR and AR drone control in 2017-18, presented an Adobe MAX Sneak, and now makes generative art. #### Face substitution tests — Fabin Rasheed (2013) - Video: https://www.youtube.com/watch?v=Ae9XSa0XONw - Interaction: Face, Perception & Effects - Platform & tech: Desktop, openFrameworks, face tracking - Idea: Paste another face onto yours in real time. - What it is: Early openFrameworks tests swap faces in live video, pasting another face onto the tracked face with matching position and expression. - Technique: A face tracker fits a mesh to the live face and texture-maps a second face onto it with color blending (likely based on Kyle McDonald and Arturo Castro's openFrameworks FaceSubstitution code). - Try it: Use MediaPipe FaceMesh to paste the face from a famous painting onto your own so it follows your expressions. Twist: two classmates swap faces. #### MakerStudio: sculpting and painting in ARKit — Fabin Rasheed (2017) - Video: https://www.youtube.com/watch?v=HIUVwuRiPes - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Phone, ARKit, Apple Pencil - Idea: Sculpt and paint with an Apple Pencil on an iPad, and the piece stands right in your room. - What it is: With an Apple Pencil on an iPad, he sculpts and paints a Halloween creature that stands in the room, using ARKit to walk around and refine it. - Technique: Pencil strokes deform and paint a mesh while ARKit tracking lets the artist view and edit the sculpture from any angle in real space. - Try it: Put a scanned model of your own clay sculpture into AR with Reality Composer or Adobe Aero and paint it on your phone. Twist: classmates gather around the model and each add one stroke. #### Multiplayer Paintball in ARKit — Fabin Rasheed (2017) - Video: https://www.youtube.com/watch?v=T0oyKUhkLHM - Interaction: Shared & Social, Play - Platform & tech: Phone, ARKit, Unity, networking - Idea: Two phones share one AR space for a paintball fight. - What it is: Two iPhones share an ARKit scene and players shoot paintballs that splat on virtual targets and each other. - Technique: Devices align their ARKit world origins (likely via a shared starting marker) and sync projectile events over the network. - Try it: Build a multiplayer paint-the-wall game with 8th Wall or Lens Studio shared sessions, where the paint stays on real walls. Twist: after one minute, the side that has painted the larger area wins. #### Aquila: flying a drone with an AR eagle — Fabin Rasheed (2018) - Video: https://www.youtube.com/watch?v=S8ztOm7znGQ - Interaction: Voice & Sound, Tangible Objects, Play - Platform & tech: Phone, ARKit, ARCore, drone SDK - Idea: Summon an AR eagle by voice, and it takes a real drone into the air. - What it is: A voice-controlled AR eagle, inspired by Assassin's Creed Origins, acts as the interface to launch and fly a real drone. - Technique: Speech commands trigger an animated AR companion whose actions are mapped to drone SDK commands for takeoff and flight. - Try it: Use speech recognition and an AR character to control a remote-control toy (or a Tello drone): say 'take off' and the character flies first, then the toy moves. Twist: the character copies the direction of your hand gestures. ### Felice Varini *Artist; anamorphic paintings* Swiss painter who paints fragments of circles, ellipses and lines across buildings, rooms and whole towns so that the shapes align into one perfect figure only from a single viewpoint. #### Across the Buildings — Felice Varini (2013) - Video: https://www.youtube.com/watch?v=HTDRz9A89lw - Interaction: Spatial Mapping, Perception & Effects, Location & City - Platform & tech: Desktop, adhesive film, projector, architecture - Idea: Graphics that fold over real architecture reveal its shape: an AR overlay that is designed flat from one view shows off the depth of the building from every other view. - What it is: In Granary Square at King's Cross, bold coloured bands wrap the heritage buildings; from street level they look like random shapes, from one point they align into a flat geometric figure. - Technique: Shapes are projected from a chosen viewpoint onto the building surfaces and executed as temporary painted or adhesive bands that follow every corner and ledge. - Try it: Project a flat logo from the phone's current pose onto the scanned room mesh (LiDAR) so it sticks to every surface, then walk sideways to see it break. Twist: let the viewer freeze a new projection point by tapping, layering several broken logos. #### Felice Varini at the Grand Palais — Felice Varini (2013) - Video: https://www.youtube.com/watch?v=-DjKShvvrjQ - Interaction: Perception & Effects, Gaze & Attention, Spatial Mapping - Platform & tech: Desktop, paint, projector - Idea: Searching for the viewpoint is itself the interaction: an AR piece can reward people for moving their body to one pose instead of tapping a button. - What it is: Inside the Grand Palais in Paris, Varini's new anamorphic paintings stretch over walls, stairs and ceilings of the galleries and resolve into circles and lines when you find the right spot. - Technique: Anamorphic painting: the figure is projected from a fixed eye point and painted onto whatever surfaces it falls on, so it is only coherent from that point. - Try it: Hide a virtual shape across a hallway that only resolves from one pose, and show a faint 'hot-cold' hum as the phone approaches the right position. Twist: after it resolves, the shape follows the viewer for ten seconds before breaking apart. #### Trois ellipses ouvertes en désordre — Felice Varini (2014) - Video: https://www.youtube.com/watch?v=9QFJvZVmBgs - Interaction: Location & City, Perception & Effects, Play - Platform & tech: Desktop, paint, projection, rooftops - Idea: A work can have two lives, fragments on the ground and a whole image from above: AR city games can hide pieces in the street and reveal the full shape only from a high viewpoint. - What it is: Across the roofs and facades of 99 buildings in Hasselt, three open ellipses become whole only from the hotel sky lounge; walking the streets, people meet puzzling orange fragments. - Technique: The ellipses are likely drawn by projecting from the lounge viewpoint and painting the resulting shapes onto rooftops and walls, so each fragment is a precise piece of one perspective projection. - Try it: Use geospatial anchors (ARCore Geospatial or Niantic) to scatter fragments of one shape across three streets, visible as a whole only from a rooftop or bridge. Twist: collect fragments on foot first; each collected piece lights up in the rooftop view. #### Cercles concentriques excentriques (Carcassonne) — Felice Varini (2018) - Video: https://www.youtube.com/watch?v=Iybcb-pKLQY - Interaction: Location & City, Perception & Effects, Gaze & Attention - Platform & tech: Desktop, aluminium foil, paint, projector - Idea: One vantage point turns a city into a single image: AR designers can mark a 'sweet spot' where scattered anchored fragments line up, and make the walk toward it part of the experience. - What it is: Yellow bands painted across the walls and towers of the medieval city of Carcassonne snap into perfect concentric circles from one spot in front of the gate, and break into scattered shards from anywhere else. - Technique: A projector placed at the viewpoint throws the circle outlines onto the architecture at night, and the traced shapes are covered with thin aluminium foil painted yellow (removable). - Try it: Place five AR fragments on different surfaces of a courtyard so they form one circle only from a marked spot; test with WebXR hit-testing. Twist: when the viewer reaches the spot, let the circle spin once and then shatter again. ### Gamgie (Clément Rignault) *Visual artist; real-time particles for circus, dance and mapping (Oxipital)* French visual artist who performs and designs real-time particle visuals with the Oxipital engine (Unity and TouchDesigner), for circus companies tracked with Augmenta, museum projection pieces and a video-mapped show deep inside the Gouffre de Padirac cave. #### Propagason — Gamgie (Clément Rignault) (2019) - Video: https://vimeo.com/362786908 - Interaction: Voice & Sound, Projection, Shared & Social - Platform & tech: Projection, TouchDesigner, audio analysis, particles - Idea: Send your voice into space and watch it travel. - What it is: An interactive installation that asks visitors to speak or sing their wishes into the sky; each voice becomes a ring of light that travels outward through space, joining a collective picture of everyone's sounds. - Technique: Microphone input is analysed in real time and drives the size and brightness of expanding particle rings, likely built with Gamgie's Oxipital/TouchDesigner pipeline. - Try it: Make a 'voice to the sky' projection in p5.js: each sound above a threshold spawns a ring whose size follows loudness and whose colour follows pitch. Twist: rings from two voices at once merge into one. #### Morphose (Cie Ultreia) — Gamgie (Clément Rignault) (2023) - Video: https://www.youtube.com/watch?v=nBenQHOFNK8 - Interaction: Projection, Perception & Effects - Platform & tech: Projection, TouchDesigner, Resolume, Oxipital, projection mapping - Idea: Let old paintings spill beyond their frames onto the room. - What it is: In a museum about to close for two years, nine projectors bring 21 paintings, the ceiling and the walls to life, extending and transforming the canvases around the visitors. - Technique: Nine projectors and four computers map content from TouchDesigner, Resolume, Chataigne and Oxipital onto the paintings, with some backgrounds expanded by DALL-E outpainting. - Try it: Pick one painting (or a printed reproduction on the wall) and project an animated extension of it beyond the frame. Twist: the extension changes when a visitor stands in front of the painting. #### Voyage au Centre de la Terre (Padirac cave mapping) — Gamgie (Clément Rignault) (2024) - Video: https://www.youtube.com/watch?v=Lsv5H5XheLk - Interaction: Projection, Location & City, Performance - Platform & tech: Projection, Oxipital, TouchDesigner, Unity, Chataigne - Idea: Use a real cave as the screen and the stage for a journey to the centre of the Earth. - What it is: Deep inside the Gouffre de Padirac cave in France, an immersive theatre play ends with video mapping on the rock walls as the character Queen Siggi reawakens her powers. - Technique: Particle and light content generated with Oxipital (Unity, TouchDesigner) is projection-mapped onto irregular cave rock and triggered through a custom web cue interface and Chataigne show control. - Try it: Project-map a short scene onto a rough, non-flat surface (crumpled paper, stones, a corner of the school) using MadMapper or TouchDesigner. Twist: let a performer trigger the next cue by stepping into a pool of light. #### La Tournoyante x Gamgie — Gamgie (Clément Rignault) (2025) - Video: https://www.youtube.com/watch?v=XBd41JohXq8 - Interaction: Performance, Hands & Body, Projection - Platform & tech: Projection, Augmenta, Oxipital, TouchDesigner, Unity - Idea: Projected particles that dance with acrobats as a live partner. - What it is: In an improvised session with the circus company La Tournoyante, acrobats spin and move on a theatre stage while projected particle fields follow, swirl around and react to their bodies in real time. - Technique: Augmenta LiDAR tracking sends performers' positions to the Oxipital particle engine (Unity/TouchDesigner), which Gamgie plays live like an instrument. - Try it: Track one dancer from above with a webcam or Kinect in TouchDesigner and project particles that swirl around their position on the floor. Twist: the particles slowly remember and redraw the dancer's path. ### Gene Kogan *Artist and programmer; creator of ml4a and the Kinect Projector Toolkit* Artist and educator working with machine learning and interactive projection; his openFrameworks Kinect Projector Toolkit aligns a projector with a depth camera, and his ml4a project teaches machine learning to artists. #### Projection + Dance — Gene Kogan (2013) - Video: https://vimeo.com/81914893 - Interaction: Hands & Body, Projection, Performance - Platform & tech: Projection, openFrameworks, Kinect, Kinect Projector Toolkit - Idea: Projection that knows where the dancer's body is in 3D. - What it is: At a workshop in Ho Chi Minh City, real-time visuals are projected onto and around improvising dancers, following their bodies across the stage. - Technique: His openFrameworks Kinect Projector Toolkit calibrates a projector to a depth camera so depth-camera points can be mapped directly to projector pixels. - Try it: Calibrate a projector to a depth camera (or use a webcam and four-point homography) and project a trail that follows a performer's hand. Twist: the trail stays on the floor after the dancer leaves. #### Silhouettes — Gene Kogan (2014) - Video: https://vimeo.com/113887700 - Interaction: Hands & Body, Projection, Drawing & Making - Platform & tech: Projection, openFrameworks, Kinect, ofxFluid - Idea: Paint walls with your shadow. - What it is: People walking past a wall see their shadows painted onto it in swirling fluid colours that linger after they move on. - Technique: A Kinect calibrated to the projector with his toolkit feeds body contours into a fluid simulation (ofxFluid) that is projected back aligned with the real shadow. - Try it: Use webcam segmentation to inject colour into a fluid simulation wherever a person's shadow falls on a projected wall. Twist: each visitor gets their own colour that mixes with others. #### Story of Light (Star Stomp) — Gene Kogan (2015) - Video: https://vimeo.com/120653546 - Interaction: Voice & Sound, Hands & Body, Projection - Platform & tech: Projection, Kinect, piezo microphones, openFrameworks - Idea: Stomp on the floor to make both sound and light. - What it is: An interactive floor where stomping makes strange sounds from 350 hidden contact microphones while projected graphics bloom around people's feet. - Technique: Piezo contact microphones under the floor produce the sound, and a ceiling Kinect and projector track feet and project graphics around them (with Alec Schachner). - Try it: Tape contact microphones or a phone under a board and trigger projected ripples at the spot where someone stomps (found with a webcam). Twist: louder stomps make ripples that travel to the other players. #### Cubist Mirror — Gene Kogan (2016) - Video: https://vimeo.com/167910860 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, neural style transfer, openFrameworks - Idea: A mirror that reflects you as a Cubist painting. - What it is: A screen shows the room and the people in front of it re-painted live in the style of a Cubist painting. - Technique: Real-time neural style transfer, a network trained on a Cubist painting, is applied frame by frame to the webcam feed. - Try it: Build a webcam or phone camera mirror that applies a real-time style model (for example ml5.js style transfer) chosen from three artworks. Twist: the style switches to whichever painting best matches the colours you are wearing. ### Glen Rhodes *Game developer & technologist* Game developer who published some of the very first Unity + ARKit experiments in June 2017, days after the iOS 11 beta, testing scale, occlusion and placement. #### ARKit Real-World Geometry Occlusion — Glen Rhodes (2017) - Video: https://www.youtube.com/watch?v=CQFkTvEcfpk - Interaction: Spatial Mapping, Portals & Worlds - Platform & tech: Phone, ARKit, Unity - Idea: Depth masks on real walls let virtual objects hide behind them. - What it is: Depth masks mapped onto real walls at runtime let physical architecture hide virtual objects behind it. - Technique: Invisible occluder meshes that write only to the depth buffer are placed on detected wall planes at runtime, so real architecture hides virtual objects behind it. - Try it: Place a depth-only occlusion material on a vertical plane so a virtual character disappears when it walks behind the wall. Twist: cut a small window in the wall so the character can only be seen through it. #### Experiment: Building a City in ARKit — Glen Rhodes (2017) - Video: https://www.youtube.com/watch?v=Wk_gZTUMBh4 - Interaction: Drawing & Making, Spatial Mapping, Play - Platform & tech: Phone, ARKit, Unity - Idea: Place buildings one by one on the sofa to build a miniature city. - What it is: Choose buildings from a menu and place them to grow a miniature town on the sofa. - Technique: Building prefabs chosen from a UI menu are placed with raycasts onto a detected surface, with grid snapping for a tidy miniature city layout. - Try it: Make a grid on the tabletop plane and pick buildings, trees and roads from a menu to assemble a small town. Twist: placed buildings come to life on their own, for example with little cars appearing on the roads. #### The Armada Arrives — Glen Rhodes (2017) - Video: https://www.youtube.com/watch?v=FEQo-13ShSg - Interaction: Perception & Effects, Location & City - Platform & tech: Phone, ARKit, Unity - Idea: A giant fleet hovers in the sky above your house. - What it is: A fleet of enormous spaceships hovers above his house, testing how very large virtual objects feel in ARKit. - Technique: Very large models are placed far from the camera in world space with correct scale, atmospheric fog and shadows, so world tracking conveys their size above the house. - Try it: Place a spaceship hundreds of meters long hovering in the air outdoors, and add fog and shadows so it looks far and big enough. Twist: the ship's shadow moves across the ground over time. #### Titanic AR Demo — Glen Rhodes (2017) - Video: https://www.youtube.com/watch?v=HtnC1dZr_cI - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, ARKit, Unity - Idea: A full-size Titanic slowly sails away across a real lake. - What it is: At a real lake, a full-size Titanic appears on the water and sails off into the distance. - Technique: A full-scale ship model is placed at a distance on the water with a horizon-aligned anchor, and animated motion makes it sail away while tracking holds the world frame. - Try it: Place a life-size ship on a sports field or by a lake and let it slowly sail away. Twist: the ship moves only when your back is turned, so you notice its new position when you turn around. ### Google ATAP (Advanced Technology and Projects) *Google's hardware research group (Project Tango, Spotlight Stories, Soli, Jacquard)* Google's small, deadline-driven skunkworks, started from Motorola. Project Tango, led by Johnny Lee, put depth sensing and motion tracking into phones in 2014 and paved the way for ARCore; Spotlight Stories turned the phone into a window onto 360° animated films; Soli brought radar gestures into products. #### Project Tango — Google ATAP (Advanced Technology and Projects) (2014) - Video: https://www.youtube.com/watch?v=Qe10ExwzCqk - Interaction: Spatial Mapping, Location & City, Play - Platform & tech: Phone, Project Tango, depth sensor, visual-inertial odometry - Idea: Give a phone a human-scale understanding of space and motion. - What it is: Google ATAP introduces a phone prototype with depth sensors and motion tracking that maps rooms in 3D as you walk, hinting at indoor navigation, AR games and measuring spaces. - Technique: A fisheye camera, a depth sensor and an IMU feed visual-inertial odometry and area learning that produce a live 6DoF pose and a 3D point cloud of the surroundings. - Try it: Walk a hallway with an AR app that drops a breadcrumb every metre, then ask a partner to follow the crumbs back. Twist: hide one crumb that only appears when you crouch. #### Project Tango: NASA SPHERES — Google ATAP (Advanced Technology and Projects) (2014) - Video: https://www.youtube.com/watch?v=Vc5YyLl1Ksg - Interaction: Spatial Mapping, Tangible Objects - Platform & tech: Phone, Project Tango, robotics - Idea: The same spatial-awareness chip that plays AR games can guide a robot in zero gravity. - What it is: A Tango phone is mounted on SPHERES, NASA's floating robots inside the International Space Station, giving them the ability to see and map the module to navigate on their own. - Technique: Tango's visual-inertial tracking and depth mapping replace external beacons, feeding pose estimates to the free-flying robot's control system. - Try it: Mount a phone on a toy car or robot and use AR world tracking to make it return to a marked start point. Twist: let a second phone see the robot's path as an AR trail. #### Spotlight Stories — Google ATAP (Advanced Technology and Projects) (2014) - Video: https://www.youtube.com/watch?v=urC4Bz1vPU4 - Interaction: Gaze & Attention, Portals & Worlds, Performance - Platform & tech: Phone, real-time rendering, orientation sensors - Idea: Let the viewer hold the camera, and let the story wait for their attention. - What it is: Animated short films made for the phone: you hold the phone up like a window and turn around to follow the story unfolding all around you, and the story waits for you to look. - Technique: Phone orientation sensors drive the virtual camera in a real-time rendered 360° scene, with story beats triggered when the viewer looks at the right place. - Try it: Make a 360° phone scene with three story beats placed around the viewer, each triggered only when they look at it. Twist: the story's ending depends on which beat they saw last. #### Nonverbal Interactions with Soli Radar — Google ATAP (Advanced Technology and Projects) (2022) - Video: https://www.youtube.com/watch?v=r-eh2K4HCzI - Interaction: Gaze & Attention, Hands & Body - Platform & tech: Desktop, Soli, radar, machine learning - Idea: Devices that respond to social cues like approach and turning away, the way people do. - What it is: Devices use radar to read body language: a screen wakes as you approach, pauses a video when you turn away, and responds to leaning in or glancing, without cameras. - Technique: Soli millimetre-wave radar senses distance, orientation and motion of the body, and machine learning classifies movements such as approach, pass, turn and glance. - Try it: Use a laptop webcam's face detection to pause a video when you look away and resume when you turn back. Twist: make it respond differently to two people. ### Greg Broadmore (Weta Workshop / Weta Gameshop) *Concept artist; creator of Dr. Grordbort; studio director of Weta Gameshop* Weta Workshop concept artist who created the retro-sci-fi Dr. Grordbort universe and led Weta Gameshop's multi-year collaboration with Magic Leap. #### Just another day in the office at Magic Leap — Greg Broadmore (Weta Workshop / Weta Gameshop) (2015) - Video: https://www.youtube.com/watch?v=LHIJrhKkCos - Interaction: Spatial Mapping, Play, Portals & Worlds - Platform & tech: Headset, Magic Leap, Weta Workshop - Idea: An ordinary office day is interrupted by robots bursting through the walls. - What it is: Early Magic Leap concept footage made with Weta: a worker checks email on a holographic screen, then a robot invasion breaks out and he fights it with a ray gun around the real office. - Technique: Spatial meshing of the office lets robots occlude behind and collide with real desks and walls, and a tracked controller acts as the ray gun for raycast hits. - Try it: Build a phone AR scene where a small robot peeks out from behind a table, using occlusion to hide it behind real objects. Twist: when tapped, the robot flees behind a different real object. #### Dr. Grordbort's Invaders — Greg Broadmore (Weta Workshop / Weta Gameshop) (2018) - Video: https://www.youtube.com/watch?v=45ZHNq9_7eY - Interaction: Spatial Mapping, Portals & Worlds, Play - Platform & tech: Headset, Magic Leap One, Unreal Engine - Idea: Portals open in your walls and retro sci-fi robots storm your living room. - What it is: A Magic Leap One launch game in which portals open in your real walls and robots from Dr. Grordbort's universe pour into your living room while you fight back with a ray gun. - Technique: Wall surfaces from spatial mapping host portal decals with stencil masking, so robots appear to climb out of the real wall and then navigate the room mesh. - Try it: Detect a wall and 'blast' a stencil hole in it, with enemies crawling out of the hole. Twist: the hole grows bigger the more you shoot, finally revealing another world behind the wall. #### Grordbattle (multiplayer) — Greg Broadmore (Weta Workshop / Weta Gameshop) (2019) - Video: https://www.youtube.com/watch?v=y4VTxpDNeWk - Interaction: Shared & Social, Play, Spatial Mapping - Platform & tech: Headset, Magic Leap One - Idea: Several players in the same room wear Magic Leap and fight with ray guns. - What it is: A co-located multiplayer Magic Leap duel from Weta Gameshop: players in the same real room see each other's avatars and trade ray-gun fire around the furniture. - Technique: Co-located players share a common coordinate frame (shared spatial anchor or map), and each player's head and weapon poses are networked so avatars and shots line up with real bodies. - Try it: Use Unity Netcode and an image marker to align two phones' coordinate systems so two people can shoot AR light balls at each other in the same room. Twist: real furniture blocks the shots. ### Gregory Barsamian *Sculptor of stroboscopic 3D animation* American sculptor who since the 1980s has built large 3D zoetropes: dozens of sculpted objects spin on armatures while a synchronised strobe fuses them into dreamlike, morphing animations in physical space. #### Feral Fount — Gregory Barsamian (1996) - Video: https://www.youtube.com/watch?v=Ixg44H4EVeo - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, sculpted objects, armature, strobe light - Idea: Physical morphing in front of your eyes is more uncanny than screen animation; AR transformations of real objects can aim for the same dream logic. - What it is: A spinning 3D zoetrope in which sculpted drops fall from a tap and transform into other forms, animated in real space by a synchronised strobe. - Technique: Dozens of sculpted stages of a transformation are mounted around a rotating armature; a strobe flashes once per stage so the eye fuses them into continuous motion. - Try it: Model eight stages of a real object turning into something else, place them around a virtual turntable next to the real object, and flash one at a time. Twist: start the morph only when the viewer touches the real object. #### Juggler — Gregory Barsamian (1997) - Video: https://www.youtube.com/watch?v=koA9YxMZF9Q - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, sculpted objects, armature, strobe light - Idea: Darkness plus a timed flash hides the mechanism and leaves only the illusion; AR benefits from similarly controlling what the viewer can and cannot see. - What it is: In a dark room a spinning sculpture comes alive under strobe light: objects arc through the air in a loop, mutating as they fly, like a juggling act in a dream. - Technique: Sculpted keyframes spin on a large armature at a speed matched to the strobe frequency, so each flash reveals the next frame of the loop. - Try it: Make objects fly around the viewer in an arc in a dark room using HoloKit, each flashing into view once per second as a slightly different shape. Twist: sync the flash to the viewer's heartbeat or footsteps. #### Die Falle — Gregory Barsamian (1998) - Video: https://www.youtube.com/watch?v=-rAExrwhzag - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, sculpted objects, armature, strobe light - Idea: Showing an illusion in a science context invites people to explain what they see; AR can pair wonder with a reveal of how the trick works. - What it is: A large stroboscopic sculpture shown in Science Gallery Dublin's ILLUSION exhibition, in which sculpted figures cycle through a looping, nightmarish transformation. - Technique: Persistence of vision: a rotating armature of sculpted stages is lit by brief flashes so each stage is seen only at one position (year approximate). - Try it: Build a phone AR zoetrope with 12 models on a ring and add a toggle that switches between 'strobe' view and 'continuous' view. Twist: let viewers find the rotation speed that makes the animation lock in. #### Blue Shirt — Gregory Barsamian (2013) - Video: https://www.youtube.com/watch?v=mZoTY-CP8BA - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, sculpted objects, armature, strobe light - Idea: Everyday objects in impossible motion are the essence of good AR surprise: familiar enough to believe, strange enough to remember. - What it is: A 3D strobe animation shown at Wood Street Galleries in Pittsburgh in which a blue shirt and other everyday forms fold and transform in a continuous loop. - Technique: A motor-driven armature carries sculpted versions of the shirt at successive stages while a strobe tuned to the rotation freezes each stage into one fluid animation. - Try it: Scan a real shirt, rig it to fold itself in AR on a real chair, and render it at 8 fps like a strobe. Twist: let the folding speed up whenever someone laughs. ### Hellicar & Lewis (Pete Hellicar & Joel Gethin Lewis) *Creative studio* London studio (active c. 2008–2015) that built camera- and Kinect-driven installations with openFrameworks, from mirror-like feedback walls to Somantics, an interactive app for autistic children. #### Feedback (with Todd Vanderlin) — Hellicar & Lewis (Pete Hellicar & Joel Gethin Lewis) (2010) - Video: https://vimeo.com/19598568 - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, openFrameworks, Kinect, video feedback - Idea: Feed people's movements back into themselves as layer upon layer of visual echoes. - What it is: A large interactive wall for Roundhouse CircusFest that turns performers' and visitors' movements into layered visual feedback; a later version used Kinect for 3D interaction. - Technique: Camera (later Kinect depth) input is composited with its own previous frames to create a video-feedback trail driven by the body. - Try it: Overlay the camera image on the previous frame with a slight scale and rotation to project a 'visual echo'. Twist: use hand height to control the echo's rotation angle. #### White Heat — Hellicar & Lewis (Pete Hellicar & Joel Gethin Lewis) (2010) - Video: https://vimeo.com/15633768 - Interaction: Hands & Body, Projection, Tangible Objects - Platform & tech: Projection, openFrameworks, boids, camera tracking - Idea: A swarm of rule-driven light creatures gathers, scatters and evolves around visitors and books. - What it is: A flock of projected 'light-beings', inspired by the Game of Life and Reynolds' boids, reacts to visitors and to a collection of artists' books and evolves over the show. - Technique: openFrameworks runs boid flocking plus cellular-automaton rules, using camera-detected visitors as attractors or obstacles. - Try it: Write a boids flock in p5.js and project it onto a table, treating people detected by the camera as attractors. Twist: place a book on the table and have the flock avoid it or gather around it. #### Divide By Zero — Hellicar & Lewis (Pete Hellicar & Joel Gethin Lewis) (2011) - Video: https://vimeo.com/19487686 - Interaction: Performance, Hands & Body, Perception & Effects - Platform & tech: Desktop, camera tracking, generative graphics - Idea: A dance under hypnotic guidance turns unconscious movement into a visible interface. - What it is: A dance film in which a performer's movements, guided by a hypnotherapist, drive interactive graphics that connect unconscious content with digital interfaces. - Technique: Camera tracking of the dancer drives real-time generative graphics composited around her body (likely openFrameworks). - Try it: Have a classmate close their eyes and improvise movement while listening to a guided script, as a camera draws their motion trail as lines. Twist: afterward, let them watch the playback and write down their reading of the "subconscious graphics". #### Inside - Out — Hellicar & Lewis (Pete Hellicar & Joel Gethin Lewis) (2011) - Video: https://vimeo.com/30795891 - Interaction: Shared & Social, Projection, Hands & Body - Platform & tech: Projection, live video, projection, openFrameworks - Idea: Project people from one space live into another so both sides end up watching each other. - What it is: A shop installation where cameras and projectors transport the activity of one space into another, creating surprising feedback loops between viewers and participants. - Technique: Live camera feeds from one room are processed and projected into the other (and back), forming a two-way video feedback loop. - Try it: Set up two laptops at opposite ends of a hallway, video-call between them and project each feed onto the wall so people on both sides can play across the wall. Twist: show only the other side's silhouettes, mirrored left to right. ### Hugues Bruyère *Partner and innovation lead at Dpt. (Montreal)* Hugues Bruyère leads R&D at the Montreal studio Dpt., experimenting with mixed reality, diffusion models, Gaussian splats and museum installations. #### Mixed Reality + Diffusion with the Quest Passthrough Camera API — Hugues Bruyère (2025) - Video: https://x.com/smallfly/status/1901403937321750862 - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Headset, Meta Quest 3, Passthrough Camera API, Stable Diffusion - Idea: Paint over the real room with a prompt, from inside the headset. - What it is: A mixed-reality prototype reads the Quest passthrough camera and replaces parts of the room with diffusion-generated imagery that stays in place as the wearer looks around. - Technique: Passthrough camera frames are sent to a fast diffusion model (image-to-image), and the generated images are re-projected onto the room surfaces from the capture pose. - Try it: Capture a photo of a wall in phone AR, restyle it with a fast image-to-image model and paste it back as a textured plane on the same wall. Twist: restyle the wall only where your hand shines a virtual flashlight. #### The prompt engineer and the director — Hugues Bruyère (2025) - Video: https://x.com/smallfly/status/1911543395694027201 - Interaction: Shared & Social, Perception & Effects, Voice & Sound - Platform & tech: Headset, Meta Quest 3, Passthrough Camera API, SDXL Turbo - Idea: Generative AR works best as a conversation between two people. - What it is: A father types prompts while his son directs what the mixed-reality diffusion system should turn their living room into, making the prototype a family game. - Technique: The same passthrough-plus-SDXL Turbo pipeline runs with a second person giving directions, so prompts evolve from a child's instructions. - Try it: In pairs, one student describes a transformation of the room aloud and the other must translate it into prompts for a live restyling demo. Twist: the director may only use sounds, not words. #### Art Morph (Hands-on Hallucinations) at the Cleveland Museum of Art — Hugues Bruyère (2026) - Video: https://x.com/smallfly/status/2089105114183381042 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Projection, Desktop, diffusion, camera, museum installation - Idea: Your own hands become the brush that turns you into a museum painting. - What it is: In the new ArtLens gallery at the Cleveland Museum of Art, visitors use their hands in front of a camera and see themselves morphed into the style of artworks from the collection. - Technique: Likely a camera feed with hand or body segmentation drives an image-to-image diffusion model conditioned on artwork styles from the collection. - Try it: Set up a webcam and projector, pick three public-domain paintings as style references, and let visitors morph into them with a fast image-to-image model. Twist: the painting chosen depends on the pose the visitor holds. #### Carveout: object understanding inside Gaussian splats — Hugues Bruyère (2026) - Video: https://x.com/smallfly/status/2076061670322094401 - Interaction: Spatial Mapping, Information & UI - Platform & tech: Desktop, Web, Gaussian splatting, segmentation, open-vocabulary labels - Idea: A scan of a real place becomes a box of named, movable objects. - What it is: Carveout segments a captured Gaussian-splat scene down to individual Gaussians and extracts labelled 3D objects, such as the furniture of a local barbershop, as separate pieces. - Technique: 2D segmentation masks from several views are lifted onto the Gaussians, and each cluster receives a label, likely from an open-vocabulary vision model. - Try it: Scan a desk with a free splat app, segment one object in two photos with a free segmentation tool, and hide or move only that object in a web splat viewer. Twist: swap objects between two students' scans. ### Jason deCaires Taylor *Sculptor* British sculptor who sinks cast human figures to the sea floor, where they become artificial reefs slowly overgrown by coral and algae. #### The Silent Evolution (MUSA) — Jason deCaires Taylor (2010) - Video: https://www.youtube.com/watch?v=oip5M3IJ4bI - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, pH-neutral marine cement, sea floor, artificial reef - Idea: A crowd that nature slowly transforms makes time and ecology visible, a model for AR figures that age with their environment. - What it is: Over 400 life-size figures cast from people of a local fishing village stand on the sea floor off Cancún, Mexico, in the Museo Subacuático de Arte, slowly covered by coral and algae. - Technique: Figures are cast in marine-grade cement that encourages coral growth and sunk onto the sea floor to divert divers from natural reefs. - Try it: Scan ten people and place them as an AR crowd in a pond or pool, then add a 'growth' shader that covers them with coral textures a little more each day. Twist: the growth speed follows real sea temperature data. #### Ocean Atlas — Jason deCaires Taylor (2014) - Video: https://www.youtube.com/watch?v=k81odhXg2Lw - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, 60-ton sculpture, sea floor to surface - Idea: A figure that carries the boundary between two worlds (water and air) is a strong image for AR works that sit on a surface between realities. - What it is: Off Nassau in the Bahamas, a 5-metre, 60-ton figure of a local girl kneels on the sea floor and holds up the surface of the ocean on her back. - Technique: The sculpture was cast in sections of pH-neutral cement and assembled on the sea floor by divers and a crane barge. - Try it: Place an AR figure under a real table or water surface so it appears to hold it up, using the table plane as an occlusion boundary. Twist: when the user places objects on the table, the figure strains more. #### Museo Atlántico — Jason deCaires Taylor (2016) - Video: https://www.youtube.com/watch?v=Xhxckg-_kdI - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, underwater museum, Lanzarote - Idea: A procession of figures walking toward a threshold turns the sea floor into a narrative scene, a pattern for AR storytelling with frozen crowds. - What it is: Europe's first underwater museum off Lanzarote holds works such as The Rubicon, a line of figures walking toward a gate, and a raft of refugees, 12 metres below the surface. - Technique: Figures cast from local people are fixed to the sandy floor in groups and arranged as scenes. - Try it: Arrange a frozen AR procession of scanned people walking toward a real doorway on campus. Twist: the procession advances one step each time a real person walks through the door. #### Coral Greenhouse — Jason deCaires Taylor (2019) - Video: https://www.youtube.com/watch?v=oSDIfIg5Fko - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, stainless steel structure, Great Barrier Reef - Idea: A human-made structure designed to be colonised by life flips the relation between architecture and nature, a thought for AR spaces that host real processes. - What it is: At the Museum of Underwater Art on the Great Barrier Reef, a steel greenhouse on the sea floor contains figures of students tending coral, designed as a nursery for reef life. - Technique: A marine-grade stainless steel and cement frame was sunk at John Brewer Reef, with planting trays for coral fragments. - Try it: Place an AR greenhouse frame over a real garden bed or pot plant and log real observations (height, leaves) as notes attached to it. Twist: the virtual frame fills in with glass panels as the real plant grows. ### Jens Grubert *Professor of human-computer interaction at Coburg University; mixed-reality researcher* Did his PhD at TU Graz on mobile augmented reality in public spaces, then became a professor at Coburg University, where he researches interaction across multiple displays and mixed reality for mobile knowledge work, often with Eyal Ofek. #### Playing it Real: Magic Lens Games in a Public Space — Jens Grubert (2012) - Video: https://www.youtube.com/watch?v=l0YrG9e_omI - Interaction: Play, Location & City - Platform & tech: Phone, natural feature tracking - Idea: Turn a public poster into a game board you play through your phone. - What it is: Players hunt cartoon monsters on a large printed poster in a public place, either looking through the phone as a magic lens over the poster or using a static peephole view on screen, and the study compares how people choose to play. - Technique: Natural-feature tracking registers the phone to the printed poster so game sprites appear in place, compared with an untracked peephole mode that scrolls the same content on screen. - Try it: Print an A1 poster and build an image-tracking AR game where creatures hide in parts of the illustration and must be tapped through the phone. Twist: the creatures flee toward whichever part of the poster nobody is looking at. #### Towards User-Perspective AR for Public Displays — Jens Grubert (2014) - Video: https://www.youtube.com/watch?v=z0nVgk1OxSc - Interaction: Portals & Worlds, Gaze & Attention, Information & UI - Platform & tech: Phone, natural feature tracking, screencast - Idea: Make the handheld truly see-through, from your eyes rather than the camera's. - What it is: A tablet held in front of a public screen, such as a ski-resort map, shows that screen's live content as if the tablet were clear glass, rendered from the user's eye position rather than the camera's, with extra information layered on top. - Technique: The tablet decodes a QR code to receive the display's screencast over the network, tracks the display with natural-feature tracking, and re-renders the content with a user-perspective projection. - Try it: Build a phone AR app that tracks a poster and shows it through the phone like a window, using front-camera face tracking to match the rendering to the viewer's eye position. Twist: tilt the phone to peek behind the poster into a hidden layer. #### MultiFi — Jens Grubert (2015) - Video: https://www.youtube.com/watch?v=nf7K7-nB5os - Interaction: Information & UI, Hands & Body - Platform & tech: Headset, Wearable, Phone, smartwatch, HMD, cross-device - Idea: Treat glasses, watch and phone as one display with different levels of detail. - What it is: A smartwatch, a phone and a head-mounted display act as one interface: the glasses show a large, low-resolution overview around the body, the watch or phone held in view shows the same content in sharp detail, and the forearm serves as a clipboard for widgets. - Technique: The devices are tracked relative to the body, so the HMD renders the context layer and hands off detail to whichever touch display lines up with the user's gaze. - Try it: With a HoloKit or phone headset plus a second phone, show a campus map floating around you in the headset and the detail card of whatever building the handheld points at. Twist: the handheld becomes a magnifier that sharpens only what is behind it. #### HeadPhones — Jens Grubert (2018) - Video: https://www.youtube.com/watch?v=l98_YozIWYc - Interaction: Gaze & Attention, Information & UI - Platform & tech: Phone, face tracking, cross-device - Idea: Use the viewer's head as the shared reference that stitches many phones into one screen. - What it is: Several ordinary phones lying on a table join into one large display simply by seeing the user's head with their front cameras, so a map spreads across all of them and the phones can be moved around freely. - Technique: Each phone's front camera tracks the user's head; because every device sees the same head, their relative poses can be solved into one coordinate system over a local network, without markers or external cameras. - Try it: Line up four phones, estimate each one's offset from the viewer with face tracking, and draw one continuous image across them in a web page. Twist: the image becomes a window into a 3D scene that shifts as you move your head. ### Jeremy Bailey *Artist, 'Famous New Media Artist'* Canadian performance artist who has used webcam tracking and AR since the mid-2000s to satirise the art world and tech industry with deadpan software demos. #### Video Terraform Dance Party — Jeremy Bailey (2008) - Video: https://www.youtube.com/watch?v=nWfAJyIfmn4 - Interaction: Hands & Body, Drawing & Making, Performance - Platform & tech: Desktop, webcam tracking - Idea: Shape terrain on screen with dance moves. - What it is: A webcam performance where the artist's body movements sculpt a 3D landscape around him live on screen. - Technique: Webcam-based motion tracking (likely colour or frame-difference tracking) maps body positions to height-field deformations of a 3D terrain rendered live around the performer. - Try it: Use MediaPipe Pose so the height of your hands pushes up or presses down a three.js terrain mesh, and 'dance out' a mountain on screen. Twist: present the dance as a deadpan product demo for 'terrain design software'. #### Nail Art Museum — Jeremy Bailey (2014) - Video: https://www.youtube.com/watch?v=40pSU5ZM784 - Interaction: Hands & Body, Tangible Objects - Platform & tech: Phone, marker tracking - Idea: Shrink a museum and move it onto your fingernails. - What it is: An AR 'museum' that places tiny artworks and gallery spaces on the artist's fingernails, parodying the art market. - Technique: Small printed markers on the fingernails are tracked by the camera, and miniature gallery models are rendered on each nail's pose. - Try it: Stick small image markers on your nails or fingertips (or use fingertip positions from MediaPipe Hands) and place a mini museum with artworks on them. Twist: write an over-the-top opening speech for this museum and record it as a video. #### Dropping an Ai Weiwei Urn in AR — Jeremy Bailey (2015) - Video: https://www.youtube.com/watch?v=JJhF4j8XkKk - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, AR - Idea: Re-enact Ai Weiwei dropping the urn, in AR. - What it is: The artist re-enacts Ai Weiwei's famous urn-dropping photo with a virtual urn in AR. - Technique: A virtual urn is anchored to the tracked hands or a plane and animated with physics to fall and shatter, re-staging a famous photographic sequence in AR. - Try it: Pick a famous performance-art photograph, re-enact the key object's action with phone AR and a physics engine, and shoot it as a triptych. Twist: replace the dropped object with a virtual version of your own "sacred object". #### ARt — Jeremy Bailey (2019) - Video: https://www.youtube.com/watch?v=w0Gl5WR3A6o - Interaction: Hands & Body, Performance - Platform & tech: Phone, AR - Idea: Demo the absurdity of AR art with a perfectly straight face. - What it is: A deadpan performance-demo (for TEDxTheWrong) launching 'the world's first platform for augmented reality art', satirising AR hype with face- and body-tracked artworks. - Technique: Face and body tracking attach artworks to the performer in real time, framed as a mock product launch; the technique is ordinary on purpose and the satire carries the piece. - Try it: Make an 'art platform' filter in Lens Studio or Effect House, then record a deadpan two-minute product launch demo. Twist: the demo must include one clearly absurd feature presented as crucial. ### Jim Denevan *Land artist* American artist who draws enormous geometric patterns with a stick or rake on beaches, deserts and frozen lakes, including a nine-square-mile drawing on Lake Baikal. #### Lake Baikal — Jim Denevan (2010) - Video: https://www.youtube.com/watch?v=3CkmjXuZpsw - Interaction: Location & City, Drawing & Making - Platform & tech: Desktop, frozen lake, vehicles, snow - Idea: The drawing's scale is so large that only aircraft can see it, raising the question of who an AR work is for when its full view is private. - What it is: With a small crew over two weeks, Denevan carved a pattern of circles spanning nine square miles into the snow on the frozen surface of Lake Baikal in Siberia, then the world's largest artwork. - Technique: Circles were laid out with ropes and marked by dragging tools behind vehicles and on foot across the snow-covered ice. - Try it: Plan a drawing larger than your campus on a map, then place segments of it in AR at real GPS points so each walker sees only a piece. Twist: combine all walkers' screenshots into one aerial view at the end. #### St. Catharines — Jim Denevan (2013) - Video: https://www.youtube.com/watch?v=aU7mrOdypgY - Interaction: Location & City, Drawing & Making - Platform & tech: Desktop, field drawing, aerial film - Idea: Nested circles made one by one show how a big pattern can be built step by step by a single person, like incremental AR drawing. - What it is: Over a week in St. Catharines, Ontario, Denevan drew a large land artwork of nested circles on open ground, filmed from the air by Peter Hinson. - Technique: Circles are marked by walking around a fixed centre with a rope and cutting or raking the ground. - Try it: Use a phone as a 'rope': anchor a centre point in AR and draw a circle on the ground as you walk while keeping the distance constant. Twist: every new circle must touch two earlier ones. #### Sand drawings — Jim Denevan (2014) - Video: https://www.youtube.com/watch?v=Trdv3F7lcWI - Interaction: Location & City, Drawing & Making, Perception & Effects - Platform & tech: Desktop, stick, rake, beach, tide - Idea: Drawing at the scale of walking and seeing at the scale of flying is exactly the gap AR can close with a live overhead view. - What it is: Walking with a stick on the beach at low tide, Denevan draws huge circles and geometric patterns that can only be seen fully from the cliffs or the air before the tide erases them. - Technique: He lays out circles by walking at a fixed distance from a centre and repeats forms with measured paces, using only a stick. - Try it: Walk a pattern on a field while a WebXR app records the path and shows it as a line on a live mini-map. Twist: add rules (for example, circles must touch) and draw as a team. #### Arcosanti — Jim Denevan (2018) - Video: https://www.youtube.com/watch?v=671Ur90JlhQ - Interaction: Location & City, Drawing & Making - Platform & tech: Desktop, desert ground, rake, aerial film - Idea: An ephemeral drawing beside a utopian architecture shows how a temporary layer can talk to a permanent site, as AR often does. - What it is: A short film shows Denevan drawing a vast geometric pattern into the desert ground near Arcosanti, the experimental town in Arizona, seen from a drone. - Technique: Lines are raked into the desert surface by walking with fixed stride lengths and cords from centre points. - Try it: Draw an AR geometric pattern around a notable building on campus that relates to its floor plan. Twist: make the pattern visible only in the drone-view mode of the app. ### Jonas Jongejan *Creative technologist, Google Creative Lab* Danish creative technologist who built early ARCore experiments at Google Creative Lab and wrote the openFrameworks addon ofxARCore. #### AR Wormhole — Jonas Jongejan, Google Creative Lab (2017) - Video: https://www.youtube.com/watch?v=Xo0_B3pNNnA - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Phone, ARCore, Unity, 360 camera - Idea: Step through a wormhole and look around the world as it was ten seconds ago. - What it is: A thought experiment about stepping into the past: a live 360 camera streams to the phone with a 10-second delay, and walking through a portal lets you look around the moment that just happened. - Technique: A fixed 360 camera streams equirectangular video with a 10-second buffer, and an AR portal renders that delayed sphere so that looking through it shows the same place in the past from the phone's orientation. - Try it: Build a portal in Unity AR Foundation and map onto it a 360 video shot earlier at the same spot, so walking in shows you "this place in the past." Twist: make the time gap adjustable, such as "this place one year ago," to tell how the place has changed. #### ARCore Drawing (ofxARCore) — Jonas Jongejan (2017) - Video: https://www.youtube.com/watch?v=oNHX0jKjtoo - Interaction: Drawing & Making - Platform & tech: Phone, ARCore, openFrameworks - Idea: An open-source example of drawing 3D lines in the air with openFrameworks and ARCore. - What it is: A simple demo for drawing 3D lines in space on an Android phone, made to show how openFrameworks and ARCore combine; the code became the ofxARCore addon. - Technique: Camera pose from ARCore is exposed to openFrameworks, and positions sampled while touching are connected into 3D polylines rendered in world space. - Try it: Skip Unity and build an air-drawing tool with three.js and WebXR instead, then compare the development experience of the two toolchains. Twist: make the line color change with drawing speed. ### Jorge Guevara *Dancer, choreographer and researcher of dance in VR* Dancer and choreographer who has spent years improvising with Naoto Hieda in the live-coded Best Practices in Contemporary Dance series. As a fellow of the Akademie für Theater und Digitalität in Dortmund and a researcher at Motion Bank, he studies how an audience can still feel a dancing body when the dance is mediated by VR headsets. #### Paper Classifier (openFrameworks + Wekinator) — Jorge Guevara (2019) - Video: https://www.youtube.com/watch?v=y2tAz2yMLz4 - Interaction: Tangible Objects, Hands & Body, Voice & Sound - Platform & tech: Desktop, openFrameworks, Wekinator, OSC, computer vision - Idea: A plain piece of paper becomes a five-button controller once the camera learns its shapes. - What it is: Holding a sheet of paper in front of a webcam in five different ways triggers five different particle-system visuals, each with its own song. - Technique: Webcam features are sent over OSC to Wekinator, which is trained on five classes of paper poses; the recognised class is sent back to an openFrameworks particle system that switches visuals and music. - Try it: Train Teachable Machine on three ways of holding a notebook and use each class to change a p5.js scene and sound. Twist: make the scene react only when two students hold their papers the same way. #### Becoming Performance in VR — Jorge Guevara (2023) - Video: https://www.youtube.com/watch?v=aKd1FepHuiQ - Interaction: Performance, Hands & Body, Shared & Social - Platform & tech: Headset, Projection, VR headsets, motion capture (likely), real-time 3D - Idea: Ask what a headset audience needs in order to feel a dancer who is only there as data. - What it is: The second research cycle of his Dance in VR fellowship at the Akademie für Theater und Digitalität: a studio showing where a dancer performs while spectators follow the dance through VR headsets, testing whether they can still feel the presence of the digital body. - Technique: The dancer's movement is likely captured and streamed into a shared real-time VR scene viewed on standalone headsets, with the same scene projected for the rest of the room. - Try it: Stream a phone-tracked performer's position into a shared WebXR scene as a simple glowing figure and let classmates watch it in headsets while the real performer moves in the same room. Twist: hide the performer behind a curtain and compare how present the avatar feels. #### Graphical Representation of Dance in VR (Motion Bank) — Jorge Guevara (2023) - Video: https://www.youtube.com/watch?v=DeNW3NMeh-Q - Interaction: Drawing & Making, Performance, Hands & Body - Platform & tech: Headset, VR headset, 3D drawing, dance notation - Idea: A dance score you can step inside instead of read. - What it is: Applied research at Motion Bank in which dancers wear VR headsets to create and move through graphic, three-dimensional versions of dance scores, testing how choreographic notation could exist in space rather than on paper. - Technique: Dancers likely use a VR drawing tool to record their movement paths as 3D strokes, which then remain in the room as a spatial score for others to follow. - Try it: In a WebXR or phone AR drawing app, have one student trace a 20-second phrase with the controller, then ask another to dance it only by following the strokes. Twist: colour each stroke by speed. #### Between Information and Noise — Jorge Guevara (2024) - Video: https://www.youtube.com/watch?v=QfJdhuRkY_s - Interaction: Performance, Drawing & Making, Projection - Platform & tech: Headset, Projection, VR headset, 360 video, projection - Idea: Let the audience sit inside the dance and see both the body and its virtual trace at once. - What it is: A daily practice on dance and VR, recorded in 360° at the Akademie für Theater und Digitalität: Jorge dances among a seated audience while the virtual layer of the piece appears as projected light lines, and each session is shared as a 360 video. - Technique: Likely a tracked VR headset or controllers feed his movement into a real-time 3D scene that is projected into the room, while a 360 camera records both layers from the centre. - Try it: Put a 360 camera in the middle of a circle of classmates and have a dancer draw light lines in a VR painting app that are projected on the wall behind them. Twist: rotate who wears the headset every minute. ### Judy Chicago *Artist; feminist art pioneer, smoke sculptures* American artist best known for The Dinner Party; since the late 1960s her Atmospheres use coloured smoke and fireworks to soften and colour landscapes, a feminine answer to male-dominated land art. #### Atmospheres — Judy Chicago (1970) - Video: https://www.youtube.com/watch?v=Re8isNQ5S6M - Interaction: Location & City, Perception & Effects, Shared & Social - Platform & tech: Projection, smoke flares, coloured smoke - Idea: Colour released into the air changes the mood of a whole landscape for a few minutes; AR can tint the atmosphere around viewers instead of adding objects to it. - What it is: In the late 1960s and early 1970s, Chicago set off coloured smoke flares in deserts, beaches and streets, softening the landscape with drifting pinks, purples and greens; this archival talk recalls three Atmospheres from April 1970. - Technique: Commercial coloured smoke flares were placed on the ground, on bodies and on structures and lit together so the smoke mixed and drifted with the wind. - Try it: Create an AR 'atmosphere' filter that fills the space around the viewer with volumetric coloured fog emitted from points they tap on the ground, drifting with a wind direction read from a weather API. Twist: each person gets one colour, and the fog only turns purple where two people's clouds meet. #### A Butterfly for Pomona — Judy Chicago (2012) - Video: https://www.youtube.com/watch?v=vqnNHP9SNxk - Interaction: Location & City, Spatial Mapping, Perception & Effects - Platform & tech: Projection, fireworks, road flares - Idea: A drawing made on the ground for people watching from above uses the stands as the viewpoint; AR ground drawings should be designed for where the audience actually stands. - What it is: On the Pomona College football field, fireworks and road flares form a giant burning butterfly on the ground that flares, smokes and slowly burns out as the audience watches from the stands. - Technique: Road flares and ground fireworks are laid out on the field along the lines of a butterfly drawing and lit in sequence so the image appears and then collapses into smoke. - Try it: Lay out a 20-metre AR butterfly of flames on a sports field or plaza with plane detection, designed to be read from a balcony or stairs; its wings burn out one segment at a time. Twist: viewers on the ground see only abstract fire, and the butterfly resolves only from the upper viewpoint. #### A Purple Poem for Miami — Judy Chicago (2019) - Video: https://www.youtube.com/watch?v=FV9PijQOEds - Interaction: Location & City, Perception & Effects, Performance - Platform & tech: Projection, smoke flares, pyrotechnics - Idea: A single dominant colour can title a whole event; limiting an AR palette to one hue makes a site-wide effect feel authored rather than noisy. - What it is: On a Miami beach, a smoke performance unfolds in purple and other colours, with plumes rising from the sand and structures toward the sea for about ten minutes. - Technique: Coloured smoke flares and pyrotechnic devices placed along the beach are fired in a timed score so the purple clouds build and move with the sea breeze. - Try it: Write a one-colour AR 'poem': place five smoke emitters on a beach or lawn that each release a purple cloud with one word of a short poem hidden inside, revealed when you walk into it. Twist: the words dissolve when two people stand in the same cloud. #### Forever de Young — Judy Chicago (2021) - Video: https://www.youtube.com/watch?v=CeScvBuGGP0 - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, coloured smoke, pyrotechnics - Idea: Colour can dress an entire building for a few minutes; AR can temporarily re-skin a landmark with a transient layer rather than a permanent overlay. - What it is: For her retrospective in San Francisco, Chicago wraps the de Young museum's tower and grounds in rolling clouds of coloured smoke that rise in rainbow layers. - Technique: Coloured smoke devices are mounted at several heights on the tower and landscape and fired in a colour sequence so the plumes stack like a rainbow. - Try it: Anchor an AR effect to a real tower or tall building so coloured smoke pours from its windows level by level, from red at the bottom to violet at the top, over two minutes. Twist: each floor's colour is chosen live by people inside that floor. ### Julien Kipp (julienkay) *XR and neural-rendering developer* German Unity developer who ports research renderers to Unity, including MobileNeRF, SNeRG and MERF viewers, Google's layered light-field video and Sentis packages for depth estimation and diffusion. #### Layered Light Field Video on Quest — Julien Kipp (julienkay) (2020) - Video: https://x.com/julien_kaye/status/1279487242608742401 - Source code: https://github.com/julienkay/LightfieldVideoUnity - Interaction: Perception & Effects - Platform & tech: Headset, Unity, Oculus Quest, Light Field Video - Idea: Volumetric video can be a stack of video-textured shells rather than a point cloud. - What it is: Google's immersive light-field videos play on an Oculus Quest in Unity: a waterfall or a dog in a room shows real parallax as the viewer moves their head. - Technique: The player reads the layered-mesh representation (a set of concentric meshes with per-frame geometry) and maps the decoded video atlas onto them, keeping geometry and video frames in sync. - Try it: Play one of Google's sample light-field clips in passthrough on a Quest, cropped to a sphere floating in the room. Twist: let the viewer reach in and shrink the sphere to a snow-globe. #### MobileNeRF in Unity — Julien Kipp (julienkay) (2022) - Video: https://x.com/julien_kaye/status/1556257212829417474 - Source code: https://github.com/julienkay/MobileNeRF-Unity-Viewer - Interaction: Perception & Effects - Platform & tech: Desktop, Headset, Unity, MobileNeRF, HLSL - Idea: A neural capture becomes a normal game asset when the network is baked into a shader. - What it is: Scenes captured as MobileNeRF neural radiance fields play in Unity as ordinary meshes with a small neural shader, so view-dependent reflections render in real time. - Technique: The port imports MobileNeRF's textured polygon meshes and converts the tiny view-dependent MLP into generated shader code, so rasterization plus a per-pixel network evaluation reproduces the radiance field. - Try it: Load a MobileNeRF scene in the viewer and place it at tabletop scale in AR Foundation as a miniature diorama. Twist: dissolve between the neural scene and the real table as the user walks around it. #### Genesis: 3D Worlds from AI Skyboxes — Julien Kipp (julienkay) (2023) - Video: https://x.com/julien_kaye/status/1626694253437939712 - Source code: https://github.com/julienkay/com.doji.genesis - Interaction: Portals & Worlds - Platform & tech: Desktop, Unity, Blockade Labs Skybox, Depth Estimation - Idea: Give a flat AI panorama depth and it becomes a stage set you can step into. - What it is: A panorama generated from a text prompt is turned into a walkable 3D world in Unity by estimating its depth and extruding the sphere around the viewer. - Technique: An equirectangular Skybox Lab panorama is passed through a monocular depth network, and the depth map displaces a sphere mesh so parallax appears when the camera moves. - Try it: Generate a panorama of an imagined place, run it through Genesis and open it as a portal in AR that the user can lean into. Twist: generate two panoramas and blend them by depth. #### SNeRG in Unity — Julien Kipp (julienkay) (2023) - Video: https://x.com/julien_kaye/status/1632880010942992386 - Source code: https://github.com/julienkay/SNeRG-Unity-Viewer - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Desktop, Headset, Unity, SNeRG, HLSL - Idea: Raymarch a neural capture like a volume texture and it becomes a place you can stand in. - What it is: A Sparse Neural Radiance Grid scene is raymarched inside Unity, including in stereo on a PC VR headset, so a captured neural scene can be walked around. - Technique: The viewer loads SNeRG's sparse 3D texture atlas and a small deferred MLP; a raymarching shader accumulates density and features, then evaluates the network per pixel, with manual depth testing for single-pass stereo. - Try it: Place a SNeRG scene inside a portal frame in AR so the neural room is only visible through the doorway. Twist: fade the grid in voxel by voxel as the user approaches. ### Julio Le Parc *Artist; light and kinetic art, co-founder of GRAV* Argentine artist (born 1928) and co-founder of GRAV (Groupe de Recherche d'Art Visuel), who since the late 1950s has built mobiles of reflective plates, moving-light rooms and games that turn spectators into participants. #### Lumière en mouvement — Julio Le Parc (1962) - Video: https://www.youtube.com/watch?v=NcwQ8Q087v4 - Interaction: Perception & Effects - Platform & tech: Projection, polished metal strips, spotlights, dark room - Idea: Light that moves over walls and people makes the whole room, including the visitor, part of the image, which is the promise of room-scale AR. - What it is: In a dark room, curved polished strips hang and turn in air currents, catching spotlights and throwing slow ribbons of light over the walls and visitors. - Technique: Suspended reflective strips rotate freely in air currents; each reflects a spotlight into a moving line of light on the surrounding surfaces. - Try it: Build a phone AR scene with virtual mirrors hanging in the room that reflect a virtual spotlight onto detected walls, with the mirrors swinging from the user's breath (microphone). Twist: project the light bands onto the user's own body with people occlusion. #### Continuel mobile, lumière (A) — Julio Le Parc (1963) - Video: https://www.youtube.com/watch?v=47Q_QDZadxY - Interaction: Perception & Effects - Platform & tech: Projection, mirrored squares, nylon thread, light - Idea: Many tiny reflectors driven by air produce endless variation with no programming, a model for particle effects that feel physical rather than scripted. - What it is: Hundreds of small mirrored squares hang on threads in front of a light source, turning with the slightest air movement and scattering flickering points of light. - Technique: A grid of small reflective plates suspended on individual threads rotates independently in ambient air, so their reflections form a constantly shifting field. - Try it: Create an AR mobile of 200 small reflective squares hanging from a detected ceiling, each with a slight random rotation driven by a noise field, reflecting the real camera environment map. Twist: let a hand wave (hand tracking) push a gust through the grid. #### Continuel-lumière plafond — Julio Le Parc (1963) - Video: https://www.youtube.com/watch?v=tH5wCntSfVc - Interaction: Perception & Effects - Platform & tech: Projection, reflective plates, motors, spotlights - Idea: A few moving mirrors can turn any ceiling into a live, ever-changing display; AR can reuse a room's surfaces the same way instead of floating screens. - What it is: Light bounced off slowly moving reflective elements drifts across a ceiling as shifting bands and curves that never repeat. - Technique: Fixed spotlights hit motorised or freely turning reflectors, which project moving caustic-like bands onto the ceiling surface. - Try it: Detect the ceiling with ARKit or WebXR plane detection and render drifting light bands on it that respond to the phone's tilt as if the phone were the mirror. Twist: let several phones each control one band and compose the ceiling together. #### Sphère (Esfera) — Julio Le Parc (2001) - Video: https://www.youtube.com/watch?v=O2WIqtDg11U - Interaction: Perception & Effects - Platform & tech: Desktop, acrylic plates, nylon thread, steel frame - Idea: A volume built from suspended particles is both an object and a cloud, a good way to present AR objects that should feel light and permeable. - What it is: A large sphere made of hundreds of small coloured or mirrored plates suspended on threads; it reads as a solid ball from afar and breaks into flickering particles up close. - Technique: Plates hung at calculated lengths inside a spherical frame define the sphere's surface as a sampled point cloud that shimmers as each plate turns. - Try it: Build an AR sphere from 500 floating quads sampled on its surface, each slowly rotating and reflecting the environment, and anchor it in a real space. Twist: let the sphere scatter into particles when the phone comes closer than 1 m and re-form when it backs away. ### Kat Sullivan *Creative technologist, dancer and educator working with motion capture and game engines* A dancer turned creative technologist, Kat Sullivan builds tools that read and transform the moving body, from a Kinect system that labels body language to live motion capture that multiplies one ballerina into a whole corps in Unreal Engine. She contributed a quality-visualising tool to the Choreographic Coding Lab and teaches 3D and mocap pipelines. #### TORN — Kat Sullivan (2015) - Video: https://vimeo.com/137311875 - Interaction: Hands & Body, Perception & Effects, Performance - Platform & tech: Desktop, Kinect, DepthKit, DSLR - Idea: Film a dancer as a 3D volume you can tear apart and fly around. - What it is: A dance film in which performer Caitlin Sikora appears as a volumetric point-cloud figure that fractures, rotates and tears apart in a virtual space. - Technique: A Kinect paired with a DSLR through the open-source DepthKit toolkit captures RGBD video, which is rendered as manipulable 3D geometry. - Try it: Capture a 15-second clip with an iPhone LiDAR app (Record3D or Polycam video), import the point cloud into Unity, and place it life-size in AR so classmates can walk through the dancer. Twist: split the figure along the beat of the music. #### Dancing with the Kinect — Kat Sullivan (2016) - Video: https://vimeo.com/157173015 - Interaction: Projection, Hands & Body, Perception & Effects - Platform & tech: Projection, Kinect - Idea: Your silhouette leaves coloured echoes on the wall. - What it is: Two dancers move in front of a wall where their outlines are redrawn as glowing, colour-shifting contour lines that linger and overlap behind them. - Technique: Kinect depth images are thresholded into body contours, and successive contours are drawn with fading colour to leave trails; the software is likely Processing or Max. - Try it: Use a webcam with background subtraction or MediaPipe Selfie Segmentation to draw each frame's outline in a new colour and keep the last 30 outlines on screen, projected full-size. Twist: only keep outlines when the dancer freezes. #### Parsing our Silent Language — Kat Sullivan (2016) - Video: https://vimeo.com/165629185 - Interaction: Hands & Body, Information & UI, Shared & Social - Platform & tech: Projection, Kinect, machine learning - Idea: Make the unspoken signals of body language visible as captions. - What it is: A Kinect watches two people talking and labels their body language live, projecting words such as 'Interested' or 'Disinterested' over each of them. - Technique: Kinect depth and skeleton data are compared against posture features learned from recorded actors, and the classified state is shown as text on the depth image or projection. - Try it: Train a tiny pose classifier (Teachable Machine) on three postures, such as open, closed and leaning in, and caption each person live in a phone AR view. Twist: let the caption lie on purpose and ask pairs to act against it. #### Kat & Cassie Make A Ballet — Kat Sullivan (2020) - Video: https://www.youtube.com/watch?v=cSspDP7H5tg - Interaction: Hands & Body, Performance, Perception & Effects - Platform & tech: Desktop, motion capture, Unreal Engine - Idea: One dancer, captured live, becomes an entire corps de ballet. - What it is: American Ballet Theatre principal Cassandra Trenary dances alone in a motion-capture suit, and her movement is multiplied in a game engine into a whole ensemble of virtual dancers that repeat, delay and mirror her. - Technique: Optical motion capture streams Trenary's skeleton into Unreal Engine, where copies of her avatar are offset in time and space to build choreography with many bodies from one. - Try it: Record a 20-second phrase with a phone mocap app, import it into Unity or Blender, and arrange eight copies in a circle with 0.25-second delays to make a canon. Twist: place the ensemble in AR on the classroom floor and dance with it. ### Kodai Takao *Computer graphics engineer specialising in physics simulation and live performance* Tokyo graphics engineer who implements fluid, snow and flocking simulations on the GPU in Unity and builds real-time visuals for live shows. #### GPU Accelerated 2D Wave Simulation — Kodai Takao (2017) - Video: https://www.youtube.com/watch?v=XqwTrwWG-_4 - Source code: https://github.com/kodai100/Unity_Waves - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, Unity, Compute Shader - Idea: A few lines of physics on the GPU turn a surface into a living membrane. - What it is: Waves spread from points of impact over a surface, bouncing off walls and interfering into complex patterns. - Technique: The 2D wave equation is integrated with finite differences in a compute shader over a height grid, and the grid drives a displaced mesh. - Try it: Lay the wave membrane over a real table in AR and trigger waves where students' fingers touch the screen. Twist: add walls by placing real books on the table, detected as obstacles. #### Position Based Fluid — Kodai Takao (2017) - Video: https://www.youtube.com/watch?v=N-XVl2Jip3M - Source code: https://github.com/kodai100/Unity_PositionBasedFluid - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: Fluids stay stable if you correct positions instead of forces. - What it is: A block of water collapses and sloshes inside a box, simulated with tens of thousands of particles in real time. - Technique: Position Based Fluids is solved in Unity compute shaders: particles are sorted into a grid for neighbour search and iteratively projected to satisfy a density constraint. - Try it: Run a small PBF simulation in a virtual glass on a real table in phone AR, tilting it with the phone's gyroscope. Twist: tapping the table sends a shock wave through the water. #### Unity FLIP Fluid — Kodai Takao (2017) - Video: https://www.youtube.com/watch?v=JRTQ6Kgi_Wk - Source code: https://github.com/kodai100/Unity_FLIPFluid - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, Unity, Compute Shader - Idea: Combine particles and grids to get detailed yet stable water. - What it is: Splashing liquid is simulated with a hybrid of particles and a grid, keeping fine splashes while staying stable. - Technique: The FLIP method transfers particle velocities to a MAC grid, solves pressure there, and transfers the velocity change back to the particles (implemented on the GPU in Unity). - Try it: Place a FLIP water tank in AR and let students pour virtual water from one real cup-shaped marker into another. Twist: the water changes colour when the two streams mix. #### GPU Instanced Flocking on URP — Kodai Takao (2021) - Video: https://www.youtube.com/watch?v=FXcFn8QM8dk - Source code: https://github.com/kodai100/Unity_BoidsSimulationOnURP - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, Compute Shader, GPU instancing - Idea: A school of thousands only needs local rules and the GPU. - What it is: Thousands of fish-like agents school and swirl together in real time in the Universal Render Pipeline. - Technique: Boids rules are evaluated in a compute shader and the agents are drawn with GPU instancing (DrawMeshInstancedIndirect) in URP. - Try it: Release a school of fish in phone AR that swims around the room and flees from the camera. Twist: the fish follow the loudest sound source in the room. ### Krzysztof Wodiczko *Artist; public-projection pioneer; professor at Harvard GSD* Polish-born artist who since the early 1980s has projected images onto monuments and façades in more than eighty cities, later animating statues with the faces, hands and voices of veterans, migrants and workers. #### Hirshhorn Museum, Washington, D.C. — Krzysztof Wodiczko (1988) - Video: https://www.youtube.com/watch?v=XYih-aS6JK8 - Interaction: Projection, Location & City - Platform & tech: Projection, slide projection, architectural projection - Idea: Project a body onto a building so the architecture speaks about the power around it. - What it is: Giant images of hands holding a gun, candles and a bundle of microphones are projected around the drum-shaped Hirshhorn Museum, turning the building into a political body on the National Mall. - Technique: Large-format slide projectors throw still images onto the curved façade, with the images composed to wrap around the building's cylinder. - Try it: Pick a campus building, photograph its façade, and design three still projection images (drawn over the photo) that make the building 'say' something about what happens inside. Twist: only use hands. #### Tijuana Projection — Krzysztof Wodiczko (2001) - Video: https://www.youtube.com/watch?v=dI_85KyAFWk - Interaction: Projection, Face, Voice & Sound - Platform & tech: Projection, Wearable, head-mounted camera, live video projection - Idea: A wearable camera turns a monument-sized building into the live face of one person. - What it is: Women who work in Tijuana's maquiladora factories wear a head-mounted camera and microphone, and their faces and testimonies are projected live onto the huge sphere of the city's Omnimax theater. - Technique: A lightweight headset with a camera fixed on the wearer's face streams video and audio to projectors aimed at the 60-foot dome, mixed with prerecorded segments. - Try it: Mount a phone on a selfie-stick headband pointing at your face, stream it to a projector and project your talking face onto a round object (a balloon or dome) in class. Twist: let a classmate interview you live while the audience only sees the projected face. #### Abraham Lincoln: War Veteran Projection — Krzysztof Wodiczko (2012) - Video: https://www.youtube.com/watch?v=81nxZhbqv-Y - Interaction: Projection, Face, Location & City - Platform & tech: Projection, head-mounted camera, projection mapping onto statue - Idea: Let living people borrow a monument's body to tell the stories it never told. - What it is: For a month, the bronze Abraham Lincoln statue in New York's Union Square speaks with the faces, gestures and voices of veterans of the Vietnam, Iraq and Afghanistan wars, projected precisely onto the statue. - Technique: Veterans were filmed with a camera fixed to their head so the face stays still, then the footage was warped to match the statue's face and hands and projected at night. - Try it: Film a classmate's face with a phone rigidly attached to their head, then use MadMapper or a projection-mapping web tool to fit the video onto a bust, mask or plaster head. Twist: let the object answer questions from the audience. #### Queen Victoria — Krzysztof Wodiczko (2014) - Video: https://www.youtube.com/watch?v=oRmT13QlY78 - Interaction: Projection, Face, Location & City - Platform & tech: Projection, dual-camera recording, projection mapping - Idea: A colonial monument becomes a microphone for the community living around it. - What it is: The statue of Queen Victoria in a Kitchener park takes on the face and hands of seven local residents, including refugees, who tell their life stories through her. - Technique: Two cameras recorded each speaker, one head-mounted for the face and one for the torso, and the merged video was aligned to the statue's face and hands in projection. - Try it: Choose a local statue or portrait, collect three short voice stories from people who live nearby and map their faces onto a printed or 3D-printed copy. Twist: switch stories when someone steps closer. ### Kuflex (Igor Tatarnikov & Denis Perevalov) *Interactive media-art lab for projection rooms and body-tracking installations* Lab of artist Igor Tatarnikov and programmer Denis Perevalov (author of books on openFrameworks) that builds depth-camera projection rooms, body-projection pieces and interactive walls, with tracking and rendering in openFrameworks. #### Quantum Space — Kuflex (Igor Tatarnikov & Denis Perevalov) (2015) - Video: https://vimeo.com/120944206 - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, openFrameworks, depth camera, Max - Idea: Walk into a room and your body dissolves into quanta of light. - What it is: A room whose walls are fully covered with interactive projection: as visitors move, their bodies dissolve into streams of light particles across all the walls. - Technique: Six depth cameras and six projectors driven by three computers track everyone in the room; tracking and rendering modules are written in openFrameworks, with Max for automation. - Try it: Project a particle system on one wall and emit particles from a webcam silhouette of whoever stands in front of it. Twist: particles from two people who touch hands change colour. #### Eyemote — Kuflex (Igor Tatarnikov & Denis Perevalov) (2017) - Video: https://vimeo.com/245804445 - Interaction: Gaze & Attention, Tangible Objects - Platform & tech: Desktop, Projection, Tobii EyeX, Arduino - Idea: Turn things on with a glance. - What it is: A visitor sits at a stand with household devices and switches on a lamp, a TV or a kettle just by looking at it, with projected animation and sound confirming each action. - Technique: An affordable Tobii EyeX eye tracker is calibrated to the user, and gaze dwell on each device's region triggers relays and projected feedback. - Try it: Use head direction in AR (or a webcam gaze estimate) to switch a real LED or a virtual object on when you look at it for two seconds. Twist: looking away turns it off, so two people must cooperate to keep everything on. #### Portal — Kuflex (Igor Tatarnikov & Denis Perevalov) (2017) - Video: https://vimeo.com/246672516 - Interaction: Hands & Body, Projection, Portals & Worlds - Platform & tech: Projection, depth camera, mirrors - Idea: Step inside a kaleidoscope and become its pattern. - What it is: A visitor stands in a mirrored room while kaleidoscopic projections react to their movement, multiplying their body into an endless symmetric pattern. - Technique: Movement is likely tracked with a depth camera and drives symmetric generative projection, multiplied optically by mirrors on the walls and floor. - Try it: Tape two mirrors at 60 degrees, point a phone camera into them and make an app that paints your movement into the reflected pattern. Twist: the pattern freezes whenever you stop moving. #### Symbiosis — Kuflex (Igor Tatarnikov & Denis Perevalov) (2017) - Video: https://vimeo.com/226068670 - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, openFrameworks, depth camera - Idea: Your body becomes the core of a projected creature. - What it is: A person stands in front of a dark wall and a projected creature grows around and over their body, with wings, antennae and patterns extending from their silhouette as they move. - Technique: A depth camera captures the visitor's silhouette and pose, and generative graphics are projected back aligned to the body, likely with openFrameworks. - Try it: Use body tracking (webcam pose estimation or Lens Studio) to attach generated wings and horns to a person and project them back at life size. Twist: the creature's shape depends on how still the person stands. ### Len Lye *Artist; kinetic sculptor and experimental filmmaker* New Zealand-born artist (1901–1980) who painted directly on film and built 'tangible motion sculptures' of flexing steel that whip, sing and shimmer; many were realised full size after his death by the Len Lye Foundation. #### Blade — Len Lye (1959) - Video: https://www.youtube.com/watch?v=MAjXg3RkUxs - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, stainless steel blade, motor, cork ball - Idea: Pushing a simple material to its limit makes an event; AR effects can build tension by slowly increasing energy until something rings or snaps. - What it is: A tall, thin steel blade is flexed back and forth by a motor until it vibrates wildly, flashing light and striking a ball to make ringing tones. - Technique: A motor drives the base of a vertical steel strip at a frequency close to its resonance, creating large oscillations and percussive contact. - Try it: Build an AR blade whose vibration amplitude grows the longer the user holds the phone still, until it rings loudly and resets. Twist: let two blades resonate together when phones are close. #### Fountain — Len Lye (1959) - Video: https://www.youtube.com/watch?v=5L9gvFmZJVY - Interaction: Perception & Effects - Platform & tech: Desktop, stainless steel rods, motor - Idea: Rigid material moving like liquid surprises the eye; AR effects can play with material expectations by giving solid things fluid motion. - What it is: A spray of hundreds of thin stainless-steel rods rising from a base; a motor rotates the base so the rods sway and shimmer like water. - Technique: A slow, oscillating motor at the base sets the flexible rods into whipping, wave-like motion, with light glinting along their length. - Try it: Grow an AR fountain of 300 thin metallic rods from a real floor spot, driven by a slow oscillation so they sway like water and reflect the environment. Twist: let the user's voice make the fountain 'spray' higher. #### Flip and Two Twisters (Trilogy) — Len Lye (1977) - Video: https://www.youtube.com/watch?v=A8TXkUSxB5Y - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, stainless steel strips, motors, programmed control - Idea: Motion and sound from the same material feel inseparable; AR objects gain presence when their sound comes directly from how they move. - What it is: Three suspended stainless-steel bands are twisted and flipped by motors into violent, singing loops and whips of reflected light. - Technique: Programmed motors twist long steel strips beyond their resting shape so they release energy as looping waves, producing roaring and ringing sounds. - Try it: Model an AR steel ribbon with a simple spring chain hung in the room; when the user twists their phone, the ribbon whips and sounds with pitch tied to its speed. Twist: add three ribbons that answer each other. #### Wind Wand — Len Lye (2000) - Video: https://www.youtube.com/watch?v=HH2gjPPi0XQ - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, carbon fibre, LED light, wind - Idea: A single flexible line on a coastline makes the invisible wind visible to the whole city; AR can visualise invisible forces with the simplest form. - What it is: A 48 m flexible carbon-fibre pole on the New Plymouth waterfront, topped with a light, that bends and sways in the wind and glows at night. - Technique: A hollow carbon-fibre tube with a red glass ball and light at the tip flexes elastically, amplifying the wind into large, slow sweeps. - Try it: Place a 50 m AR wand at a real waterfront or park with a geospatial anchor and bend it with live wind data, glowing at night. Twist: let a crowd of phones pushing together bend it further. ### Lichtfaktor *Light-painting and light-graffiti collective* Cologne collective (Marcel Panne, Jan Leonardo Wöllert and friends) who since 2006 have made light-graffiti photographs and stop-motion films drawn frame by frame with flashlights, plus interactive light-painting setups for festivals. #### Light Graffiti in Cologne (Tracks feature) — Lichtfaktor (2008) - Video: https://www.youtube.com/watch?v=swPAKxTh-aM - Interaction: Drawing & Making, Location & City - Platform & tech: Desktop, long exposure, LED flashlights - Idea: Graffiti made of light leaves no mark on the wall; AR street art can be just as temporary and still tied to a specific corner. - What it is: Arte's Tracks follows Lichtfaktor writing glowing letters and figures on Cologne streets with flashlights during long exposures, alongside other new graffiti forms. - Technique: Several painters draw with handheld LEDs in front of a camera on a tripod set to a long exposure. - Try it: Make a geo-anchored light-graffiti app with persistent anchors (ARCore Geospatial or Niantic Lightship) where a stroke drawn in the air stays at that street corner for others to find. Twist: each tag fades a little every time someone views it. #### The Very Angry Caterpillar — Lichtfaktor (2008) - Video: https://www.youtube.com/watch?v=BpK5QuSUcLo - Interaction: Drawing & Making, Spatial Mapping, Play - Platform & tech: Desktop, long exposure, stop motion, LED lights - Idea: Stop-motion light drawing gives a character a real path through a real room — a low-tech prototype for AR characters that walk on real surfaces. - What it is: A light-painted caterpillar crawls, eats and grows across walls and floors, animated from hundreds of long exposures made with the BBC children's show Blue Peter. - Technique: Each frame is a separate long exposure in which the painters redraw the character slightly further along its path; the frames are sequenced into video. - Try it: Record a hand-drawn 3D stroke creature in an AR drawing tool, then script it to crawl along a detected plane and climb a wall using plane detection. Twist: the creature grows one segment every time it reaches a real object. #### dENiZEN — Light Painting Video — Lichtfaktor (2011) - Video: https://www.youtube.com/watch?v=RAe30VG5CI4 - Interaction: Drawing & Making, Shared & Social, Location & City - Platform & tech: Desktop, long exposure, stop motion, LED lights - Idea: Many people drawing together in one frame produces a density no single artist could reach; shared AR drawing sessions can build the same collective scenes. - What it is: A music video painted over four cold nights in Cologne: dozens of friends draw flowing shapes, faces and words in light around streets and buildings, frame by frame. - Technique: A large crew performs coordinated light strokes during each long exposure and repeats them with small changes for stop-motion. - Try it: Set up a multi-user AR drawing room (e.g. Unity + Netcode or 8th Wall shared sessions) where four students draw simultaneously around one anchor for 60 seconds. Twist: each student may only draw in their assigned color and height band. #### Luma Paint — Interactive Light Graffiti — Lichtfaktor (2016) - Video: https://www.youtube.com/watch?v=_c7eTvKNdYU - Interaction: Drawing & Making, Hands & Body, Shared & Social - Platform & tech: Desktop, camera, real-time long exposure, light pens - Idea: Turning a slow photographic technique into instant feedback makes it a public game: live accumulation of traces is a strong AR interaction on its own. - What it is: At the Luminale festival visitors pick up light pens and draw in front of a camera; a live long-exposure picture builds up on a large screen so everyone sees their strokes appear instantly. - Technique: A camera feed is processed in real time with a max-brightness accumulation buffer, likely in custom software, so light strokes persist on the display. - Try it: Write a WebAR or Lens Studio effect that keeps the brightest value of each pixel over time (a live long exposure) and invite people to wave phone flashlights in front of it. Twist: the accumulated image slowly drifts upward like smoke. ### Lukas Moro *Interaction designer and creative technologist* Lukas Moro prototypes small, tactile interfaces where mixed reality headsets, paper and real-time AI meet. #### Interfacing AI through physical paper — Lukas Moro (2024) - Video: https://x.com/lukas_moro/status/1838207092471050645 - Interaction: Drawing & Making, Voice & Sound, Information & UI - Platform & tech: Headset, Meta Quest 3, OCR, LLM - Idea: Paper stays the interface; the headset quietly turns it into a conversation with AI. - What it is: Highlighting and commenting on a printed page with a real pen and voice is synchronised to a digital copy, so the AI knows exactly which passage the reader is talking about. - Technique: The headset camera detects the page and highlighter marks, OCR maps them to the digital text, and spoken comments are attached to those passages for an LLM. - Try it: Print a one-page text, highlight three lines, photograph the page and have a vision model return the highlighted lines plus a summary pinned next to the page in AR. Twist: the AI argues with your highlights. #### Math Notes on real paper — Lukas Moro (2024) - Video: https://x.com/lukas_moro/status/1829487148078412019 - Interaction: Drawing & Making, Information & UI - Platform & tech: Headset, Meta Quest 3, passthrough camera, OCR, LLM - Idea: Could mixed reality bring a renaissance of paper as a productivity tool? - What it is: Handwritten equations on physical paper are read by the headset and solved, with answers appearing on the page like Apple's Math Notes, but for pen and paper. - Technique: Camera frames are sent to handwriting OCR and an LLM that evaluates the expressions; results are rendered as handwriting-style text aligned to the page. - Try it: Photograph a handwritten sum with a phone, ask a vision model to solve it, and draw the answer in AR right after the equals sign. Twist: the AI writes the answer in the same handwriting as the student. #### Motion as input for real-time image generation — Lukas Moro (2024) - Video: https://x.com/lukas_moro/status/1842953164498100617 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Projection, Kinect, StreamDiffusion - Idea: Dance the prompt instead of typing it. - What it is: A person's movement captured by a Kinect drives a real-time image generator, so the picture morphs continuously as they move. - Technique: The Kinect depth or silhouette image is used as the init image (or control) for StreamDiffusion, producing new frames at interactive rates. - Try it: Use a webcam body-segmentation mask as the input to a fast image-to-image model and project the output on a wall behind the dancer. Twist: freeze the prompt word only when the dancer stops moving. #### Real-time diffusion as a muse while painting on paper — Lukas Moro (2024) - Video: https://x.com/lukas_moro/status/1847299759603699906 - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Headset, Meta Quest 3, StreamDiffusion - Idea: Keep the pencil; let the AI dream alongside your drawing. - What it is: While the user paints on a real sheet of paper, a Quest 3 shows a live diffusion image next to it that reinterprets every stroke, acting as a muse rather than a replacement. - Technique: A view of the paper is streamed to StreamDiffusion (image-to-image at low steps), and the result is shown as a floating panel beside the physical page. - Try it: Point a phone at a sketchbook, send the image to a fast image-to-image model every second, and show the result in a headset viewer next to the page. Twist: the AI only sees what you erased. ### Maarten Baas *Dutch designer; author of the Real Time clock series* Dutch designer (born 1978) whose Real Time series films people drawing, sweeping or painting the hands of a clock in real time for twelve hours, turning a clock face into a performance. #### Grandfather Clock (Real Time) — Maarten Baas (2009) - Video: https://www.youtube.com/watch?v=aYD-CDMhnmI - Interaction: Performance, Perception & Effects - Platform & tech: Desktop, 12-hour video, screen inside clock case, performer - Idea: A screen inside a familiar object makes it feel inhabited — AR can place 'someone inside' everyday objects. - What it is: A grandfather clock whose face is a screen showing a man inside who erases and redraws the hands with a marker every minute, as if he lived inside the clock. - Technique: A 12-hour video of the performer drawing the hands behind glass is played on a display hidden in a traditional clock case, synced to the real time. - Try it: Track a real picture frame or box with image tracking and play a short loop of a tiny person living inside it. Twist: the tiny person notices the viewer and waves. #### Sweepers' Clock — Maarten Baas (2009) - Video: https://www.youtube.com/watch?v=0eBVFLlKO80 - Interaction: Performance, Perception & Effects - Platform & tech: Desktop, 12-hour video, performers, brooms - Idea: Time shown by a human performance feels warm and absurd — AR clocks, timers and progress bars could be performed rather than drawn. - What it is: Two street sweepers push lines of rubbish across the floor for twelve hours, constantly re-forming the hour and minute hands of a clock in real time. - Technique: A continuous 12-hour video is filmed in one take per segment and played back in sync with the real time of day. - Try it: Record a volumetric or green-screen clip of yourself moving two objects as clock hands for one minute, and play it in AR on a real table in sync with the clock. Twist: let the performer react when the user taps the table. #### Schiphol Clock — Maarten Baas (2016) - Video: https://www.youtube.com/watch?v=e_3KY2gWDwg - Interaction: Performance, Location & City - Platform & tech: Desktop, 12-hour video, large transparent clock, airport - Idea: An endless small human task in a public place gives travellers a moment of empathy — AR in transit spaces can use tiny human stories. - What it is: A huge clock at Amsterdam Schiphol Airport appears to hold a man in overalls who paints and wipes the clock hands from inside, minute by minute, all day. - Technique: A 12-hour video recorded in real time is shown on a screen inside a transparent clock housing, synced to local time. - Try it: Create a location-anchored AR clock at your school entrance where a recorded classmate 'paints' the time every minute. Twist: swap the performer each hour so a different student lives in the clock. #### Schiphol People's Clock — Maarten Baas (2024) - Video: https://www.youtube.com/watch?v=WqxXsbquzOE - Interaction: Shared & Social, Performance - Platform & tech: Desktop, video, hundreds of performers, airport display - Idea: Letting many people 'be' the interface makes it communal — AR experiences can be built from crowd-contributed recordings. - What it is: A new clock at Schiphol in which the hands are formed by many different people, one after another, so the time is shown by a changing crowd of performers. - Technique: Likely a sequence of recorded clips of many participants posing as hands, edited and played back in sync with real time. - Try it: Collect short phone videos of classmates posing as clock hands and assemble them into an AR clock that plays each clip at the matching minute. Twist: let visitors record themselves and join the clock. ### Marion Tränkle / NOW *Light artist and scenographer; co-founder of the performance collective NOW* Marion Tränkle designs light and scenography for performances that turn body signals into space, working with dancer-researcher Roos van Berkel and composer Ivo Bol as NOW. Their pieces map performers' EEG to lasers ((UN)FOCUSSED), score eye blinks (BLINK), measure bouncing bodies on a trampoline and light up a former panopticon prison. #### (UN)FOCUSSED — Marion Tränkle / NOW (2016) - Video: https://vimeo.com/174162011 - Interaction: Hands & Body, Perception & Effects, Performance - Platform & tech: Wearable, Projection, EEG, laser, sound - Idea: Put two brains side by side as laser light and see when they synchronise. - What it is: Two performers wear EEG headsets and their brainwaves directly drive laser drawings, light and sound around them, revealing how closely their brains synchronise. - Technique: Hacked consumer EEG headsets stream brain-frequency bands that Alberto Novello maps directly to laser galvanometers and a sound composition by Ivo Bol. - Try it: Use a Muse or other consumer EEG band (or a heart-rate sensor as a stand-in) on two volunteers and draw each signal as a line on a projected oscilloscope, then ask them to try to synchronise. Twist: turn on a light only when both signals match. #### See Me Now? — Marion Tränkle / NOW (2018) - Video: https://vimeo.com/308889681 - Interaction: Location & City, Gaze & Attention, Performance - Platform & tech: Projection, light design, sound composition - Idea: Use light to flip who watches whom in a building made for surveillance. - What it is: A site-specific performance in the former Panopticon prison in Haarlem, where sound, light and six performers shift attention between the central surveillance dome and the 224 surrounding cells. - Technique: A light composition by Tränkle and a sound composition by Ivo Bol are cued across the cells and the dome, directing the audience's gaze from the centre to the periphery. - Try it: Pick a space with a clear centre (atrium, stairwell) and design a five-minute light-and-sound score with phone flashlights that moves the audience's attention from the centre to the edges. Twist: let the audience hold the lights and decide where to look. #### BLINK — Marion Tränkle / NOW (2019) - Video: https://www.youtube.com/watch?v=vEc037P5tiI - Interaction: Gaze & Attention, Face, Performance - Platform & tech: Projection, eye tracking, light, sound - Idea: Blinking, which we never notice, becomes the score of a performance. - What it is: A performance for three people that uses eye blinking as a performative tool: conscious and unconscious blinks, read as indicators of mood, generate a score of light and sound. - Technique: Blinks are detected per performer, likely with eye-tracking cameras or EOG electrodes, and each blink triggers light and sound events in a shared score. - Try it: Detect blinks with MediaPipe Face Mesh for three students sitting in a row and flash a lamp and play a click for each one, then compose a one-minute piece only with blinks. Twist: forbid blinking for 20 seconds and let the room hear the release. #### Bounce (elasticity test) — Marion Tränkle / NOW (2019) - Video: https://www.youtube.com/watch?v=uiQmSlH7bEo - Interaction: Hands & Body, Tangible Objects, Voice & Sound - Platform & tech: Projection, distance sensors, light, sound - Idea: Measure how a body bounces and let the whole room bounce with it. - What it is: Dancer Roos van Berkel jumps on a trampoline whose sensors measure the elastic quality of her body, and the data turn the studio into a living, elastic room of sound and light. - Technique: Distance sensors designed by physicist Tom Bergman track the trampoline surface with high precision, and the data drive sound by Ivo Bol and light by Tränkle. - Try it: Place a phone under a mini-trampoline or on a springy board and use its accelerometer (via a web app) to map each bounce to a light flash and a pitch. Twist: map the time spent in the air, not the impact. ### Mark Dawson *Software engineer; author of 'ARKit by Example'* Engineer whose 2017 'ARKit by Example' blog series and demos taught many early ARKit developers, later followed by LiDAR meshing experiments on the 2020 iPad Pro. #### ARKit Light Estimation Demo — Mark Dawson (2017) - Video: https://www.youtube.com/watch?v=7Kk6iVr5ULo - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, ARKit - Idea: Virtual objects brighten and dim with the real lights in the room. - What it is: A virtual cube brightens and dims as he switches real room lights on and off, showing ARKit light estimation. - Technique: ARKit light estimation returns ambient intensity and color temperature each frame, which are applied to the scene's lights so virtual objects match the room lighting. - Try it: Turn on light estimation in AR Foundation, feed the estimated values into the scene lights, and switch the room lights on and off to watch the virtual object change. Twist: the darker the room, the brighter the virtual object glows, like a night light. #### ARKit Virtual Van Gogh Bedroom Tour — Mark Dawson (2017) - Video: https://www.youtube.com/watch?v=lvjVgt_ce5Q - Interaction: Portals & Worlds, Spatial Mapping - Platform & tech: Phone, ARKit, SceneKit - Idea: Van Gogh's Bedroom in Arles moved into your kitchen at full size so you can walk inside. - What it is: A life-size 3D model of Van Gogh's painting 'Bedroom in Arles' is placed in his kitchen to walk around in. - Technique: A 1:1 3D reconstruction of the painted room is placed on the detected floor plane, so the viewer walks through it using world tracking. - Try it: Build a simple 3D room in Blender based on a painting and place it at real scale in the classroom in AR so people can walk in. Twist: pick a surreal painting of a place that could never exist. #### 3D Scene Reconstruction with iPad Pro LiDAR — Mark Dawson (2020) - Video: https://www.youtube.com/watch?v=6JMYkHC69OA - Interaction: Spatial Mapping - Platform & tech: Phone, ARKit, LiDAR, Metal - Idea: Rebuild a room's mesh live with iPad LiDAR. - What it is: Scans a room live with the iPad Pro LiDAR and renders the growing mesh with shading and grid lines. - Technique: ARKit scene reconstruction mesh anchors are rendered in Metal with a normal-based shading and a world-space grid shader, updating as the mesh grows. - Try it: Show the scanned room mesh with the AR Foundation mesh manager and render it with a world-space grid-line shader. Twist: color the grid lines as a gradient by the order in which areas were scanned. #### iPad Pro LiDAR Matrix Simulation — Mark Dawson (2020) - Video: https://www.youtube.com/watch?v=bGgSpHBy_5g - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, ARKit, LiDAR, Metal - Idea: Cover the scanned room with the green code rain from The Matrix. - What it is: The scanned room is texture-mapped with falling green Matrix code, turning his house into the digital rain. - Technique: The LiDAR mesh is shaded with a world-space projected shader that scrolls glyph textures downward over time, producing Matrix-style digital rain on every surface. - Try it: Write a shader for the AR mesh that scrolls a glyph texture downward along the world-space Y axis. Twist: replace the glyphs with the letters of your name or lines of a poem. ### Michael Naimark *Media artist and researcher (projection, immersive film, VR)* Naimark worked at MIT's Architecture Machine Group (Aspen Moviemap), Atari Research, Apple and Interval Research, and later taught at NYU, USC and MIT Media Lab. His projection works from 1979 on, such as Displacements and Talking Head Projection, are early cases of projecting images back onto the real shapes they came from. #### Displacements — Michael Naimark (1980) - Video: https://www.youtube.com/watch?v=bMDr_CFFgWE - Interaction: Projection, Spatial Mapping, Perception & Effects - Platform & tech: Projection, 16mm film, rotating turntable, white-painted room - Idea: Projecting a scene back onto its own shape makes it look real; anything that does not match the surface becomes a ghost, a lesson in how registration decides what feels solid in AR. - What it is: A furnished American living room is filmed with two performers by a slowly rotating camera; then the whole room is spray-painted white and the film is projected back onto it from the same spot. The furniture looks solid and colourful again, while the people appear as ghosts. - Technique: A 16mm camera on a turntable records the room; the camera is replaced by a projector on the same turntable, so camera and projector share one viewpoint and the image lands exactly on the whitened objects. - Try it: Scan a small corner of the classroom with a LiDAR phone, paint or cover the real objects white, then project or overlay the captured texture back onto them in AR from the capture position. Twist: record a moving person in the scan so only they appear as a ghost. #### Talking Head Projection — Michael Naimark (1980) - Video: https://www.youtube.com/watch?v=uo8CPZdw0d4 - Interaction: Face, Projection, Shared & Social - Platform & tech: Projection, super-8 film, pan-tilt gimbal, face-shaped screen - Idea: Presence comes from matching the shape and the motion of the surface to the image; an AR avatar needs a body that moves with it, not just a flat video. - What it is: A filmed face is projected onto a face-shaped screen mounted on a pan-tilt gimbal. The screen turns and nods exactly as the real person's head moved during filming, so the projected head looks solid and present in the room. - Technique: Head orientation was recorded during super-8 filming and later drove a motorized face-shaped screen while the film was projected onto it, an extension of Disney's Haunted Mansion head projection (MIT Architecture Machine Group, with Nicholas Negroponte and Chris Schmandt). - Try it: Record a classmate's face with ARKit face tracking, then in AR place their video on a 3D head mesh that replays the recorded head rotation in the room. Twist: drive the head's rotation live from a second phone so it becomes a telepresence puppet. #### EAT — Michael Naimark (1989) - Video: https://www.youtube.com/watch?v=KJD381TlEIs - Interaction: Tangible Objects, Projection, Play - Platform & tech: Projection, video projector, videodisc, Mac II, live performer - Idea: A familiar ritual (ordering, lifting the cloche) frames a virtual object so well that people accept it instantly; AR works best when it borrows an everyday script. - What it is: A visitor sits alone at a formally set dinner table and orders from a live waiter; when the plate cover is lifted, the 'food' is a video projected onto the empty plate. A large red button labelled EAT is on the table. - Technique: A ceiling video projector aimed at the plate plays clips from a videodisc selected by a Mac II computer according to the order, with a live performer as waiter. - Try it: Set a real plate on a table, build a phone AR menu, and place an animated 3D dish on the plate when a paper menu item is tapped with image tracking. Twist: the dish slowly disappears bite by bite each time someone presses a big on-screen EAT button. #### Be Now Here — Michael Naimark (1995) - Video: https://www.youtube.com/watch?v=J2-VQFPYftM - Interaction: Portals & Worlds, Location & City, Perception & Effects - Platform & tech: Projection, stereoscopic 35mm, rotating floor, 3D glasses - Idea: Moving the viewer's body with the image (the rotating floor) creates a strong sense of place, a reminder that AR presence comes from the body, not only the eyes. - What it is: Visitors wear 3D glasses and stand on a 16-foot rotating floor in front of a stereoscopic projection of public plazas in endangered World Heritage cities such as Jerusalem, Dubrovnik, Timbuktu and Angkor. As the floor turns, the panorama turns with them, as if the place were live. - Technique: Two 35mm cameras on a rotating tripod shot stereo panoramas; in the installation a stereoscopic projector and four-channel audio are synchronized with a rotating platform, and a pedestal lets visitors choose place and time. - Try it: Capture a 360° photo of a campus plaza, open it in WebXR as a portal, and let classmates enter it while standing on an office chair that someone slowly turns in sync with the panorama. Twist: offer two times of day and let the viewer switch by stepping forward. ### Miguel Chevalier *Digital artist* Mexican-French pioneer of computer art since the 1980s whose generative, interactive floor and vault projections (Magic Carpets, Fractal Flowers, Digital Supernova) respond to visitors walking through them. #### Fractal Flowers — Miguel Chevalier (2008) - Video: https://vimeo.com/796298809 - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, generative L-systems, camera tracking - Idea: A fractal garden that bends and blooms as you come close. - What it is: A virtual garden of generative fractal flowers grows, blooms and dies on a projection, bending and reacting as visitors move in front of it. - Technique: L-system-like generative plant growth runs in real time and a camera tracks visitors' positions to bend or trigger blooms near them. - Try it: Write a recursive fractal flower in p5.js and make the flowers bend toward a person when the camera detects them approaching. Twist: the color of each visitor's clothing becomes the color of the new flowers. #### Magic Carpets — Miguel Chevalier (2014) - Video: https://www.youtube.com/watch?v=LvNhZrWVByM - Interaction: Hands & Body, Projection, Location & City - Platform & tech: Projection, floor projection, overhead tracking, generative graphics - Idea: Turn the floor of an old building into a magic carpet that changes with every step. - What it is: Generative carpets of light are projected across the floor of a historic courtyard; patterns ripple and reorganise wherever visitors walk. - Technique: Overhead cameras track visitors (Voxels tracking software) and generative pattern software reacts locally to their positions on the floor projection. - Try it: Project a geometric-pattern "carpet" onto the floor and use an overhead camera to detect footsteps and spawn ripples. Twist: the line between two people generates new pattern motifs. #### Pixel Waves — Miguel Chevalier (2015) - Video: https://vimeo.com/1208277695 - Interaction: Hands & Body, Projection, Voice & Sound - Platform & tech: Projection, wave simulation, overhead tracking - Idea: Walk through a sea of pixel light whose waves move with you. - What it is: Waves of pixel-like light roll across the floor and respond to people walking through them like a digital sea. - Technique: Floor projection of a height-field wave simulation, perturbed at the positions of people detected by an overhead tracking system (Voxels track). - Try it: Build a 2D wave-equation simulation on the web and project it onto the floor, dropping a "pebble" wherever the camera detects a person. Twist: let the volume of sound change the color of the waves. #### Digital Supernova — Miguel Chevalier (2019) - Video: https://www.youtube.com/watch?v=ZbSWZL_YZ3w - Interaction: Projection, Perception & Effects, Location & City - Platform & tech: Projection, generative graphics, vault projection mapping - Idea: Turn a cathedral vault into a sky of constantly exploding digital supernovae. - What it is: Generative geometric 'supernovae' are projected onto the Gothic vaults of Rodez Cathedral, turning the ceiling into a slowly exploding digital sky. - Technique: Real-time generative geometry is warped to the cathedral's vault surfaces using mapped projection and edge blending. - Try it: Lie on the floor and project rotating generative geometry onto the ceiling for everyone to look up at. Twist: use a classmate's heartbeat (measured with a phone) to set the rhythm of the geometry's bursts. ### Ming Wai Chan (cinight) *Graphics developer* Copenhagen-based graphics developer who keeps MinimalCompute, a widely used set of minimal Unity compute-shader scenes, plus custom SRP and shader samples, and posts short demo videos of each. #### Sparkle Reflection Effect — Ming Wai Chan (cinight) (2020) - Video: https://www.youtube.com/watch?v=Yh8PZ77jdSc - Source code: https://github.com/cinight/MinimalCompute - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader, DrawProceduralIndirect - Idea: Let the image itself decide where the sparkles go. - What it is: Tiny star-shaped glints pop up only on the brightest highlights of the image, like light catching on glitter or wet surfaces. - Technique: A compute shader scans the frame for pixels above a brightness threshold, appends them to a buffer, and draws star sprites there with an indirect draw call, so no data returns to the CPU. - Try it: Run the sparkle pass on the AR camera image so real highlights (windows, jewellery, water) twinkle on the phone. Twist: tint each sparkle with the colour sampled from the pixel underneath. #### Drawing with Epicycles (DFT) — Ming Wai Chan (cinight) (2021) - Video: https://www.youtube.com/watch?v=3YWKfmuZ1aw - Source code: https://github.com/cinight/MinimalCompute - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader, DFT - Idea: Any drawing can be rebuilt from rotating circles. - What it is: Chains of spinning circles redraw a picture line by line, each circle rotating at its own frequency, and the tip paints into a texture on the GPU. - Technique: A discrete Fourier transform turns a traced path into circle frequencies and phases; the epicycle tip positions go through a StructuredBuffer to a compute shader that paints them into a render texture. - Try it: Let users trace a shape in the air with the phone in AR, then replay it as spinning epicycles hovering where it was drawn. Twist: drop the highest frequencies so the drawing becomes smoother and more abstract. #### GPU Fluid 3D — Ming Wai Chan (cinight) (2023) - Video: https://www.youtube.com/watch?v=1Yi4Wek994M - Source code: https://github.com/cinight/MinimalCompute - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: A GPU fluid needs only a few kernels: advect, project, repeat. - What it is: Coloured smoke swirls inside a 3D volume as a compute-shader fluid solver runs and a moving source keeps injecting density and force. - Technique: The MinimalCompute 01_4_Fluid_3D scene extends the GPU Gems stable-fluids solver to 3D render textures (advection, divergence, Jacobi pressure, projection) and ray-marches the density. - Try it: Put the 3D fluid box in AR around a real object and inject smoke from the phone's position so the user can blow smoke at it. Twist: use the microphone to control injection strength. #### VFX Graph as Mesh Deformer — Ming Wai Chan (cinight) (2023) - Video: https://www.youtube.com/watch?v=dmoOpK7DHRY - Source code: https://github.com/cinight/MinimalCompute - Interaction: Perception & Effects, Face - Platform & tech: Desktop, Unity, VFX Graph, GraphicsBuffer - Idea: Treat every vertex as a particle and all of VFX Graph's forces become mesh deformers. - What it is: A mesh ripples and deforms as if it were liquid, because each of its vertices is driven by a VFX Graph particle. - Technique: VFX Graph writes particle positions into a GraphicsBuffer indexed like the mesh's vertices, and a shader reads that buffer to displace the mesh. - Try it: Use the trick on the AR face mesh (ARKit face tracking) so a turbulence field makes the user's face melt and reform. Twist: attract the vertices toward the user's hand position. ### Moritz Waldemeyer *Designer and engineer of light, wearables and interactive installations* German-born London designer who engineered Hussein Chalayan's video and laser dresses, stage costumes for U2, OK Go and Rihanna, and interactive light sculptures for brands and galleries. #### Video Dress (for Hussein Chalayan) — Moritz Waldemeyer (2007) - Video: https://www.youtube.com/watch?v=NqBdxiWMJEc - Interaction: Hands & Body, Perception & Effects - Platform & tech: Wearable, LED matrix, diffusing fabric - Idea: Clothing as a display: the body carries its own moving image. - What it is: A dress embedded with thousands of LEDs plays soft, low-resolution video of sea and flowers across its surface, so the garment becomes a moving screen worn on the body. - Technique: LED matrices sewn under a diffusing fabric layer are driven as a flexible low-resolution display playing prerecorded video. - Try it: Tape an LED strip or a phone flashlight array under a piece of translucent fabric and design a 10-second 'video' that suits the garment. Twist: make the pattern react to the wearer's movement with a phone accelerometer. #### Snowflake Sculpture, Time Square Seoul — Moritz Waldemeyer (2011) - Video: https://www.youtube.com/watch?v=dxBVrMfJtuM - Interaction: Perception & Effects, Location & City, Gaze & Attention - Platform & tech: Projection, anamorphosis, suspended sculpture - Idea: An anamorphic sculpture rewards the one person who finds the right spot. - What it is: A huge constellation of suspended snowflake shapes fills a shopping mall atrium; from most balconies it looks abstract, but from one hidden vantage point the pieces line up into a secret image. - Technique: Each element is placed in 3D along lines of sight from a chosen viewpoint so that their projections align into one picture only from there (anamorphosis). - Try it: Hang ten paper shapes at different depths in a room so that from one marked spot they form a letter or face, then build the same illusion in an AR scene. Twist: make two different images visible from two different spots. #### Journey of Colour — Moritz Waldemeyer (2019) - Video: https://www.youtube.com/watch?v=kiTv0UH4rRU - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Projection, camera color sensing, LED wall - Idea: Use a physical sample as a remote control for the color of a whole space. - What it is: Visitors hold fabric and paint samples from the design center's showrooms up to a reader, and a long LED wall instantly washes into that exact color. - Technique: A camera reads the color of the held sample and maps it to the RGB output of the LEDs on the wall, with animated transitions. - Try it: Build a web page that samples the color of any object held in front of the phone camera and floods a projector or second screen with it. Twist: mix the colors of the last three objects into a gradient. #### The Mora Constellation — Moritz Waldemeyer (2026) - Video: https://www.youtube.com/watch?v=N9BuisYqWZM - Interaction: Hands & Body, Perception & Effects - Platform & tech: Projection, presence sensing, addressable LEDs - Idea: A light field that notices your presence the way a candle does. - What it is: In Palazzo Mora during the Venice Biennale, a field of slender glowing elements hangs in space and flickers more intensely as visitors walk by, like candle flames disturbed by a passing breath. - Technique: Presence sensors (likely motion or depth sensors) modulate the brightness and flicker of individually controlled light elements in real time. - Try it: Hang twelve LEDs at different heights and use a webcam's motion detection so the ones nearest a moving person flicker like candles. Twist: make them calm down only when the room is silent. ### Mustafa Doga Dogan *HCI researcher; Research Scientist at Adobe Research (PhD from MIT CSAIL)* Did his PhD in Stefanie Mueller's HCI Engineering Group at MIT CSAIL on invisible tags and fabrication-aware interfaces, worked with Google on XR-Objects, was a visiting postdoc at the University of Tokyo, and is now a research scientist at Adobe Research in Basel. #### InfraredTags — Mustafa Doga Dogan (2022) - Video: https://www.youtube.com/watch?v=-8NWIsyY-2E - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, 3D printing, infrared camera, ArUco - Idea: Hide AR markers inside objects so the object itself quietly becomes the marker. - What it is: QR codes and ArUco markers are 3D printed inside everyday objects such as a mug, a router or a speaker, invisible to the eye but readable by an infrared camera on a phone, so pointing the phone at an object brings up its controls or Wi-Fi password. - Technique: Objects are printed with an infrared-transmitting filament and the tag's bits are air gaps inside the shell, which show up at a different intensity in a near-infrared image. - Try it: Glue a paper marker under a thin layer of translucent plastic, light it from behind, and build a phone AR app that reveals a control panel when the hidden marker is detected. Twist: hide a different secret message inside each classmate's object. #### BrightMarker — Mustafa Doga Dogan (2023) - Video: https://www.youtube.com/watch?v=M5YNR6cVIiM - Interaction: Tangible Objects, Hands & Body - Platform & tech: Phone, Headset, 3D printing, fluorescent filament, infrared camera - Idea: Objects carry glowing markers that only a special camera can see, turning them into trackable props. - What it is: Markers made of infrared-fluorescent filament are printed into coloured objects; a small imaging module on a phone or headset sees only the glowing markers, turning objects into tracked props, such as a speaker whose knobs set the AR volume or a paddle used as a VR haptic prop. - Technique: An infrared-fluorescent filament shifts incoming light to a longer wavelength, and an optical filter on the camera passes only that emission, so the marker stays high-contrast whatever the object's surface colour. - Try it: Stick UV-reactive or retroreflective tape patterns on household objects and track them with a phone camera and flashlight to drive AR sliders and buttons. Twist: the same object controls different things depending on which face is up. #### StandARone — Mustafa Doga Dogan (2023) - Video: https://www.youtube.com/watch?v=LGzQMHaABmg - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, infrared watermark, inkjet printing - Idea: The paper document itself carries its AR experience, hidden in the ink. - What it is: A printed restaurant menu carries its own AR app: an invisible infrared watermark in the paper encodes the AR content, so a phone scanning the page shows interactive dishes and personal options without downloading anything. - Technique: The AR scene description is encoded into a machine-readable watermark printed with IR-absorbing inks on an inkjet printer and decoded by a phone with an infrared camera. - Try it: Design a poster whose AR layer is encoded in a very faint QR code (light yellow on white) and write a web AR page that builds the scene from the code's text alone. Twist: two posters scanned together combine into a third scene. #### XR-penter — Mustafa Doga Dogan (2025) - Video: https://www.youtube.com/watch?v=F5o5gKcjsQI - Interaction: Spatial Mapping, Tangible Objects, Drawing & Making - Platform & tech: Headset, mixed reality headset, scene mesh - Idea: Design furniture in place from the scrap wood you actually have. - What it is: A casual woodworker registers a pile of scrap boards in a mixed-reality headset, designs a chair from exactly those pieces at full scale in the room, then sees cut lines overlaid on the real boards before sawing and drilling. - Technique: The headset records each scrap board's dimensions, aligns the design frame to the floor using the scene mesh, and overlays cut positions on the physical boards. - Try it: Measure five cardboard pieces, load them as virtual blocks in a Quest or phone AR scene, and have students build a full-size stool from only those blocks. Twist: the stool must fit one particular corner of the classroom. ### NAKED, INC. (Ryotaro Muramatsu) *Creative company for projection mapping and immersive events* Tokyo creative company founded in 1997 and led by Ryotaro Muramatsu, known for large projection mapping shows on Tokyo Station, Tokyo Tower and historic shrines and castles. Its touring events FLOWERS BY NAKED and TOKYO ART CITY by NAKED turn projection into walk-in, participatory exhibitions. #### TOKYO TOWER CITY LIGHT FANTASIA – Spring Concerto — NAKED, INC. (Ryotaro Muramatsu) (2016) - Video: https://www.youtube.com/watch?v=TwAxq5i648Q - Interaction: Projection, Portals & Worlds, Location & City - Platform & tech: Projection, projection mapping, window projection - Idea: A window is already a see-through display; drawing on it over a real skyline is exactly the optical idea of AR glasses. - What it is: On the observation deck of Tokyo Tower, animations are projected onto the large windows so that cherry blossoms and light float over the real night view of Tokyo below. - Technique: Projectors inside the deck likely throw images onto a semi-transparent film on the glass, so the city stays visible through the animation. - Try it: Stand at a window with a view, anchor AR content to the skyline with world tracking, and let virtual petals drift over real buildings. Twist: each building lights up when a petal lands on it. #### TOKYO ART CITY by NAKED — NAKED, INC. (Ryotaro Muramatsu) (2017) - Video: https://www.youtube.com/watch?v=PECqN-vX_J4 - Interaction: Projection, Tangible Objects, Location & City - Platform & tech: Projection, projection mapping, architectural model - Idea: Projecting onto a miniature city gives you a god's-eye view of a place, the same tabletop viewpoint that makes AR maps and city models so readable. - What it is: A large white scale model of Tokyo, including the Tokyo Station building that NAKED once mapped at full size, is covered with projected light, traffic, particles and seasons that visitors watch from around the table. - Technique: White 3D-printed or milled building models are mapped with several overhead projectors so each block receives its own animation. - Try it: Place a white cardboard model of your campus on a table and use plane or object tracking to overlay AR traffic, weather and day-night cycles on it. Twist: tap one building to shrink yourself into its street view. #### FLOWERS BY NAKED 2019 — NAKED, INC. (Ryotaro Muramatsu) (2019) - Video: https://www.youtube.com/watch?v=KAjNzRf9Y-k - Interaction: Projection, Hands & Body, Shared & Social - Platform & tech: Projection, projection mapping, interactive projection, real flowers - Idea: Mixing real flowers with projected ones blurs which is which, the core pleasure of a good AR blend. - What it is: An immersive flower exhibition in Nihonbashi, Tokyo, where real flower installations and projected blossoms fill a hall, and visitors' movements make digital petals bloom and scatter. - Technique: Floor and wall projections are driven by sensors that detect visitors, likely depth cameras, and layered over physical flower arrangements. - Try it: Put a real bouquet on a table and use Lens Studio or WebXR to grow AR flowers out of it that bloom when a hand comes close. Twist: petals fall faster the more people crowd around. #### NAKED Yorumode 2022 Heian Jingu — NAKED, INC. (Ryotaro Muramatsu) (2022) - Video: https://www.youtube.com/watch?v=TEHFyH44eBs - Interaction: Projection, Location & City, Tangible Objects - Platform & tech: Projection, projection mapping, interactive lanterns, lighting - Idea: Giving each visitor a handheld light makes them a participant in the show, a model for AR where the phone itself is the lantern. - What it is: A night visit to Kyoto's Heian Shrine where visitors carry glowing NAKED lanterns through the grounds while projections and light art animate the historic buildings around them. - Technique: Projection mapping on the shrine buildings is combined with lanterns that likely change colour through wireless control during the walk. - Try it: Design a WebXR night walk where the phone acts as a lantern and reveals hidden AR creatures only inside its light cone. Twist: when two lanterns meet, a shared creature appears between them. ### NONOTAK (Noemi Schipfer & Takami Nakamoto) *Audiovisual light and projection duo* Studio of illustrator Noemi Schipfer and architect-musician Takami Nakamoto that builds immersive light and sound architectures from projection on layered translucent screens, kinetic light and live performance. Their performance SHIRO opened the Blackbox Whitecube XR symposium at Hellerau where AΦE showed 0AR. #### DAYDREAM — NONOTAK (Noemi Schipfer & Takami Nakamoto) (2013) - Video: https://www.youtube.com/watch?v=JZQmg-jP9CQ - Interaction: Projection, Perception & Effects - Platform & tech: Projection, projection on scrims, moiré - Idea: Two layers of projected lines are enough to make a solid room feel liquid. - What it is: An installation where visitors stand inside a tunnel of two translucent screens on which projected lines move with sound, creating the illusion that the room is stretching, rotating or dissolving. - Technique: Two projectors cover a front and a rear scrim; phase-shifted line animations on each layer create moiré and parallax depth cues that bend perceived space, synced to an original soundtrack. - Try it: Project two slightly different stripe animations onto two layers of fabric and walk between them with the lights off. Twist: let a phone's gyroscope control the phase shift so one visitor 'tilts' the room for everyone. #### SHIRO — NONOTAK (Noemi Schipfer & Takami Nakamoto) (2015) - Video: https://www.youtube.com/watch?v=DcGHcsyuXuE - Interaction: Projection, Perception & Effects, Performance - Platform & tech: Projection, projection on scrims, audio-reactive visuals - Idea: Performers disappear inside a room made only of projected light. - What it is: An audiovisual live show in which the two artists perform inside a structure of translucent screens while projected white lines and strobes pulse with the music, making the space seem to expand, collapse and flip. - Technique: Synchronised projectors map generative line patterns onto front and back layers of semi-transparent scrim around the performers, with visuals driven by the live sound (custom software). - Try it: Hang a sheet of tulle in front of a wall, project the same moving line pattern on both with a slight offset, and let a student stand between them. Twist: tie the line speed to a microphone so claps make the space jump. #### VERSUS — NONOTAK (Noemi Schipfer & Takami Nakamoto) (2015) - Video: https://www.youtube.com/watch?v=OrkI6WW2bIo - Interaction: Projection, Perception & Effects - Platform & tech: Projection, dome projection, spatial audio - Idea: A whole dome becomes one moving optical illusion. - What it is: An immersive audiovisual piece made for the Satosphere, a 360-degree projection dome, where monochrome geometric light patterns sweep over and around the audience lying and standing below. - Technique: Generative line and grid patterns are rendered in dome-master format and mapped across the dome's projectors, synchronised with spatial sound. - Try it: Build a mini dome with an umbrella lined with white paper and a phone projector, then design a pattern that makes viewers feel they are rising. Twist: the pattern must work only when viewed from lying on the floor. #### NARCISSE — NONOTAK (Noemi Schipfer & Takami Nakamoto) (2025) - Video: https://www.youtube.com/watch?v=rXw_K5YlPOU - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Projection, kinetic light, LED, reflection - Idea: Light that watches its own reflection move. - What it is: A kinetic light installation where a ceiling of motorised light panels tilts and glows in sequence, and its movements are doubled in a reflective surface below, like a self-regarding light creature. - Technique: Arrays of servo-driven LED panels are choreographed from a timeline synced to sound, and a mirror-like floor surface reflects them to double the space (reflection setup likely varies per venue). - Try it: Mount five phone screens on hinges showing white fields, animate their brightness to music and film their reflection in a tray of black water. Twist: film only the reflection and ask viewers which side is real. ### Nancy Baker Cahill *Artist; founder of the 4th Wall AR app* Artist who builds monumental drawn AR sculptures and sited public artworks (Desert X, Liberty Bell) through her free 4th Wall app. #### 4th Wall (app and Contested Landscapes) — Nancy Baker Cahill (2018) - Video: https://www.youtube.com/watch?v=i9cFUDshxMQ - Interaction: Location & City, Drawing & Making - Platform & tech: Phone, 4th Wall app, Unity - Idea: Use a free app to make giant hand-drawn sculptures appear anywhere. - What it is: Monumental drawn sculptures by the artist appear in galleries and public sites through her free 4th Wall AR app. - Technique: Hand drawings are converted into animated 3D line or ribbon geometry in Unity and placed at monumental scale via plane detection or geolocated anchors in the 4th Wall app. - Try it: Scan one of your own line drawings, extrude it into a 3D line sculpture in Unity or three.js, and place it on the sports field at 10-meter scale. Twist: have the lines slowly 'draw' themselves so viewers see the process rather than the finished piece. #### Revolutions / Margin of Error (Desert X) — Nancy Baker Cahill (2019) - Video: https://www.youtube.com/watch?v=rb-9x1pH_4M - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, 4th Wall app - Idea: A giant AR sculpture of tangled lines rising over the desert. - What it is: Geolocated AR sculptures in the Coachella Valley desert for Desert X, including a writhing mass of drawn lines rising over a landfill site. - Technique: GPS-anchored animated line sculptures are placed at specific desert sites, with site choice (a landfill) carrying the meaning; local plane detection fixes the base on the ground. - Try it: Find a campus corner you think is overlooked (a trash station, a power room) and use AR to place a writhing, rising mass of lines above it. Twist: drive the lines' color and motion with the spot's live noise level or temperature. #### Liberty Bell — Nancy Baker Cahill (2020) - Video: https://www.youtube.com/watch?v=_SrpZZv1aNU - Interaction: Location & City, Voice & Sound - Platform & tech: Phone, 4th Wall app - Idea: Ring an AR Liberty Bell in the sky over historic American sites. - What it is: A giant AR Liberty Bell rings over historic US sites (Philadelphia, Selma, Washington) as a call to civic action. - Technique: Geolocated AR places a large animated bell model with spatial audio at historic sites, so the sound source is positioned in the sky above the viewer. - Try it: Design an AR sound object hanging in the air for a place tied to a public issue, and use Unity spatial audio so the sound changes with distance. Twist: the object only rings when several people aim at it together. #### Siren Songs — Nancy Baker Cahill (2021) - Video: https://www.youtube.com/watch?v=mzfqln_s0cE - Interaction: Location & City, Voice & Sound - Platform & tech: Phone, 4th Wall app - Idea: Singing AR sound sculptures among the Alps. - What it is: An animated, immersive AR public artwork for Elevation 1049 in Gstaad and St. Moritz: drawn forms float in the Alpine landscape with a sung soundtrack. - Technique: Geolocated, large-scale animated drawings are anchored in open landscape with a synced spatial soundtrack, using GPS and compass for placement at distance. - Try it: Choose an open landscape and use WebAR to place floating lines that rise and fall with a song, recording your own humming as the soundtrack. Twist: generate the shape of the lines in real time from the sound spectrum. ### Nancy Holt *Land artist* American artist (1938–2014) whose concrete tunnels, rings and pipes frame the sun, stars and horizon at precise moments such as the solstices. #### Sun Tunnels — Nancy Holt (1976) - Video: https://www.youtube.com/watch?v=f8OLToLVYCw - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, concrete tunnels, solstice alignment, star holes - Idea: The tunnels frame the sky at precise moments and places, the same logic as an AR experience that only resolves at one time and position. - What it is: Four concrete tunnels, each 5.5 metres long, lie in an open X in the Great Basin Desert of Utah, aligned with sunrise and sunset on the solstices; holes in their walls form four constellations of light inside. - Technique: Tunnels were cast with holes sized and placed to match the stars of Draco, Perseus, Columba and Capricorn, and oriented by solar calculation. - Try it: Compute today's sunset azimuth for your location and place an AR tube on a field that frames the sun exactly at sunset. Twist: punch holes in the tube that match the constellation overhead tonight. #### Dark Star Park — Nancy Holt (1984) - Video: https://www.youtube.com/watch?v=K-t5FDC-J1M - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, gunite spheres, steel poles, shadow alignment - Idea: A shadow that fits its outline once a year is a clock, an idea AR can replicate by matching virtual shadows to the real sun. - What it is: In Rosslyn, Virginia, large dark spheres, pools and poles are placed so that every 1 August at 9:32 a.m. their shadows line up exactly with outlines drawn on the ground, marking the date the land was bought in 1860. - Technique: Positions of spheres, poles and ground outlines were calculated from the sun's altitude and azimuth at the anniversary time. - Try it: Place AR spheres on a plaza and draw outlines on the ground where their shadows will fall at a chosen date and time, using a solar-position library. Twist: let each visitor set their own anniversary. #### Solar Rotary — Nancy Holt (1995) - Video: https://www.youtube.com/watch?v=5kO2k3SPtuc - Interaction: Location & City, Perception & Effects, Information & UI - Platform & tech: Desktop, aluminium sun disc, shadow plaques, campus plaza - Idea: Using the sun as a pointer that visits dated plaques turns a plaza into a calendar, a direct model for time-based AR annotations. - What it is: At the University of South Florida, a ring with a circular hole on tall poles casts a spot of light that, on specific dates, falls on plaques marking events in local history; on the summer solstice it lands on a central seat. - Technique: The aperture's position was calculated so the sun spot's path hits each plaque on its anniversary date. - Try it: Compute where the midday sun spot from a real opening (a window, a gate) falls through the year and pin AR notes on those floor points with the dates of campus events. Twist: the notes light up only when the real sun spot touches them. #### Up and Under — Nancy Holt (1998) - Video: https://www.youtube.com/watch?v=iaIY1ZCMIkc - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, sand quarry, concrete tunnels, grass-covered mound - Idea: Tunnels through a mound are viewfinders that tell you exactly where to stand and look, like AR markers that reward one viewpoint. - What it is: In a former sand quarry in Nokia, Finland, a long grass-covered earth mound is pierced by concrete tunnels that frame views of the sky, water and each other. - Technique: Concrete pipes were laid through a shaped earthwork and grassed over; their axes are aligned with the landscape and cardinal points. - Try it: Place three AR tubes through a real hedge or building gap so each frames a different target (a tree, a window, the sky). Twist: show a hidden message only when the user stands inside a tube's axis. ### Nathan Gitter *Designer & engineer; human interface designer at Apple* Designer-engineer who shared playful ARKit and TrueDepth prototypes in 2017–2018, wrote the widely read 'What I Learned Making 5 ARKit Prototypes', and later joined Apple's human interface team. #### Rainbrow – Eyebrow-Controlled Game — Nathan Gitter (2017) - Video: https://www.youtube.com/watch?v=9k_9BKA3w_Q - Interaction: Face, Play - Platform & tech: Phone, ARKit, TrueDepth - Idea: An iPhone X game where raising your eyebrows moves the character up and down. - What it is: An iPhone X game controlled with your face: raise your eyebrows to fly up, lower them to go down, using TrueDepth face tracking. - Technique: ARKit face tracking on the TrueDepth camera exposes the browInnerUp/browDown blend-shape coefficients, which are mapped directly to the character's vertical position. - Try it: Read eyebrow height from ARKit face blendshapes (or MediaPipe Face Landmarker) to steer a little bird up and down past obstacles. Twist: control it with how wide your mouth opens or by raising one eyebrow instead. #### Bringing Newspapers to Life with ARKit — Nathan Gitter (2018) - Video: https://x.com/nathangitter/status/1010599486303981570 - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, ARKit 2, image tracking - Idea: Point at a newspaper and the weather maps and sports highlights start moving on the page. - What it is: Pointing the phone at a printed newspaper plays animated weather maps and sports highlights on top of the page. - Technique: Newspaper photos are ARKit 2 reference images, and a video plane is placed exactly over each detected image so the printed picture appears to play. - Try it: Use image tracking to recognize a photo in a newspaper and cover it with a video of the same size so the photo comes alive. Twist: the video is a 'news follow-up' you shot yourselves. #### Exploring Paintings in AR — Nathan Gitter (2018) - Video: https://x.com/nathangitter/status/1012328934581719041 - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, ARKit 2, image tracking - Idea: Point at a famous painting to recognize its figures, and tap one to hear its story. - What it is: Looking at a painting identifies its key figures, which can be tapped to learn more about them. - Technique: The painting is tracked as a reference image, and tappable hotspots are placed at known 2D coordinates on the image plane, opening info panels when hit. - Try it: Recognize a printed copy of a famous painting and place tappable hotspots on several figures that open an introduction. Twist: when a figure is tapped, make it speak a line. #### Time Travel in AR — Nathan Gitter (2018) - Video: https://x.com/nathangitter/status/1020733723183124480 - Interaction: Location & City, Portals & Worlds - Platform & tech: Phone, ARKit 2, image tracking - Idea: In front of a graffiti wall painted over every day, use AR to see what it used to look like. - What it is: In a public graffiti park whose walls change daily, the phone shows earlier versions of the artwork in place, letting you see into the past. - Technique: Wall sections are tracked with image recognition (likely against earlier photos), and archived photos of past graffiti are overlaid at the detected image pose. - Try it: Photograph a campus wall as it looks today to use as the recognition image, then overlay old photos when it is scanned later, with a slider to switch between dates. Twist: let classmates leave graffiti on the wall that can only be seen in AR. ### Nicolas Schöffer *Artist; father of cybernetic art* Hungarian-born French artist (1912–1992) who built 'spatiodynamic' sculptures of polished metal, motors and light, including CYSP 1 (1956), the first sculpture to react autonomously to sound, light and movement, and a 52 m cybernetic light tower in Liège. #### CYSP 1 — Nicolas Schöffer (1956) - Video: https://www.youtube.com/watch?v=u8nxktU1R6E - Interaction: Hands & Body, Voice & Sound, Perception & Effects - Platform & tech: Desktop, photoelectric cells, microphone, electronic brain, motors - Idea: The first sculpture with its own senses and behaviour; AR objects become characters as soon as they react to light, sound or people. - What it is: A tower of polished metal plates on a wheeled base that turns, rolls and spins its plates in reaction to light, colour and sound around it; it performed with Maurice Béjart's dancers. - Technique: Photoelectric cells and a microphone feed an analogue electronic 'brain' built with Philips, which drives motors for the base and each rotating plate. - Try it: Build an AR sculpture in Unity or Lens Studio that rotates its plates toward bright light (camera light estimation) and spins faster with loud sounds. Twist: make it dance with a real person tracked by body tracking. #### Tour Lumière Cybernétique (Liège) — Nicolas Schöffer (1961) - Video: https://www.youtube.com/watch?v=y1nHehuu7Jw - Interaction: Location & City, Perception & Effects, Information & UI - Platform & tech: Projection, steel tower, rotating mirrors, lights, sensors - Idea: A landmark that reads its environment and answers with light makes a city site feel alive; geolocated AR landmarks can react to live weather or city data the same way. - What it is: A 52 m steel tower beside the Palais des Congrès in Liège with rotating mirrors and lights, designed to react to the weather, the city and the river and to play light onto the building's glass facade. - Technique: Sensors measuring temperature, humidity, light and sound feed a control system that drives the tower's rotating mirrors and lamps (restored and reactivated in 2016). - Try it: Anchor a tall AR light tower at a real outdoor location and drive its rotating lights from a live weather API (wind speed, temperature). Twist: let passers-by add their own 'signal' by shaking their phones. #### Lumino — Nicolas Schöffer (1968) - Video: https://www.youtube.com/watch?v=F2qpfmguwNs - Interaction: Perception & Effects - Platform & tech: Desktop, motors, coloured filters, light bulbs, diffusing screen - Idea: A kinetic light artwork made small enough for a living room; ambient AR pieces can likewise live quietly on a shelf instead of demanding attention. - What it is: A small box with a round frosted screen on which slowly rotating coloured filters and reflectors produce endlessly changing abstract light patterns; it was sold as a domestic light sculpture by Philips. - Technique: Rotating discs of coloured gel and reflective material behind a diffusing screen mix and blur light into continuously shifting forms. - Try it: Design an AR 'Lumino' that sits on a real shelf as a small glowing disc showing slow generative colour patterns, visible only in dim light (use light estimation). Twist: sync its pattern speed to the room's ambient noise. #### Chronos 10 — Nicolas Schöffer (1969) - Video: https://www.youtube.com/watch?v=jhWNVgya--E - Interaction: Perception & Effects, Projection - Platform & tech: Projection, polished steel, motors, coloured lights, screen - Idea: A sculpture and its projected shadow play form one work; AR objects can be designed together with the light and shadow they cast on real surfaces. - What it is: A rotating sculpture of polished steel blades and mirrors, lit by coloured lamps, throws moving reflections and shadows onto a screen beside it. - Technique: Motorised reflective elements rotate under programmed coloured lights, and the resulting reflections and shadows are captured on a translucent screen. - Try it: Create an AR kinetic sculpture on a table whose virtual lights cast colourful moving shadows onto the real wall behind it (shadow-receiving plane). Twist: let the user move the light with their phone as a torch. ### Nils Völker *Berlin artist making kinetic installations from inflatable bags* German artist (born 1979) who builds large grids of plastic bags that inflate and deflate in computed waves, and machines that draw light patterns. #### One Hundred and Eight — Nils Völker (2010) - Video: https://www.youtube.com/watch?v=1BfHY5sFGJM - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, 108 plastic bags, cooling fans, Arduino, camera tracking - Idea: Cheap, soft material with good timing feels alive — AR effects gain life from rhythm and reaction, not detail. - What it is: A wall of 108 plastic bags inflates and deflates in rolling waves; when a viewer approaches, the bags near them pull back as if breathing away. - Technique: Each bag is inflated by a small fan controlled by a microcontroller; a camera detects the viewer and triggers local deflation waves. - Try it: Cover a real wall in AR with a grid of soft virtual balloons that breathe in a wave and shrink away from the user's hand. Twist: make them breathe in sync with the user's own breathing picked up by the microphone. #### Variations on Pi (Light Drawing Machine) — Nils Völker (2010) - Video: https://www.youtube.com/watch?v=C5koC8cYe4Y - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, drawing machine, LED, long-exposure photography - Idea: A number sequence becomes a visible trace when motion is recorded over time — AR trails can reveal data as drawings. - What it is: A small machine moves a light along paths computed from the digits of pi, and long exposures turn its motion into glowing drawings. - Technique: A pen-plotter-like mechanism steers an LED along paths generated from successive digits of pi, photographed with long exposure. - Try it: Program a virtual light point in AR that moves along a path generated from the digits of pi or a friend's phone number and leaves a permanent glowing trail. Twist: two sequences draw at once and their lines interfere. #### Thirty Six — Nils Völker (2012) - Video: https://www.youtube.com/watch?v=GW39zLyr1Jc - Interaction: Perception & Effects - Platform & tech: Desktop, 36 inflatable bags, fans, microcontroller - Idea: A tiny grid is enough to read waves — sequencing matters more than resolution when designing AR motion. - What it is: Thirty-six silver bags on a wall swell and collapse in computed sequences, creating ripples that travel across the grid. - Technique: Fans driven by a microcontroller inflate each bag according to a pre-programmed wave pattern with phase offsets per position. - Try it: Animate a 6×6 grid of inflatable virtual cushions on a table in AR using a ripple function with phase offsets. Twist: let the user tap one cushion to start a new ripple from there. #### Sixty Eight — Nils Völker (2015) - Video: https://www.youtube.com/watch?v=l5R2BBSuCXI - Interaction: Perception & Effects - Platform & tech: Desktop, 68 inflatable elements, fans, microcontroller - Idea: Slow, organic motion on a hard surface makes architecture feel alive — AR can animate walls with subtle breathing instead of loud effects. - What it is: A large installation of 68 inflatable elements rises and falls in slow waves across a wall, like a surface breathing. - Technique: Individually controlled fans inflate and deflate each element following layered wave functions. - Try it: Make a real wall 'breathe' in AR with a displacement shader driven by overlapping sine waves. Twist: slow the breathing down as the room gets quieter. ### Norman McLaren *Animator and filmmaker, National Film Board of Canada (1914–1987)* Scottish-Canadian animator at the NFB who painted directly on film, invented pixilation with live actors, and used optical printing to multiply dancers into trails of motion. #### Neighbours — Norman McLaren (1952) - Video: https://www.youtube.com/watch?v=e_aSowDUUaY - Interaction: Perception & Effects, Hands & Body, Performance - Platform & tech: Desktop, 35mm film, pixilation, stop motion - Idea: Sampling a real body at the wrong rate makes it move like a cartoon; AR frame-dropping and pose-snapping can give real people superpowers. - What it is: Two men fight over a flower in an Oscar-winning anti-war film shot as pixilation: live actors are photographed one frame at a time so they glide, float and move impossibly. - Technique: Pixilation: actors hold poses for single frames while the camera advances, creating motion that could not be filmed continuously. - Try it: Build a phone AR 'pixilation booth' that captures a person segmentation mask only when they jump, and plays the captures back so they appear to float across the floor. Twist: two people take turns so their clips interleave into one fight. #### Pas de deux — Norman McLaren (1968) - Video: https://www.youtube.com/watch?v=WopqmACy5XI - Interaction: Performance, Hands & Body, Perception & Effects - Platform & tech: Desktop, optical printer, 35mm film - Idea: Delayed copies of a body turn movement into form; the same echo effect on a live AR body track makes dance visible as sculpture. - What it is: Two dancers lit from the side against black are printed on top of themselves with frame delays, so every movement unfolds into a fan of luminous echoes. - Technique: Optical printing: the same film strip is exposed repeatedly with offsets of several frames, stacking time-shifted images of the dancers. - Try it: Create a body-segmentation effect in Lens Studio or WebAR that draws the last 8 masks of a dancer with delays of 3 frames each and decreasing opacity. Twist: map the delay length to the music's tempo. #### Ballet Adagio — Norman McLaren (1972) - Video: https://www.youtube.com/watch?v=aTlPYCz7UfY - Interaction: Performance, Hands & Body, Perception & Effects - Platform & tech: Desktop, high-speed camera, 35mm film - Idea: Slowing time exposes the mechanics of skill; AR replays of real performers at slow speed make an effective teaching layer. - What it is: A classical pas de deux filmed at high speed and played back in slow motion, so the lifts and balances unfold with a clarity the live audience never sees. - Technique: High-speed cinematography played back at normal speed, stretching each second of movement into several. - Try it: Record a dancer or athlete volumetrically with a depth-capable phone app and place the capture life-size in AR at 1/4 speed so students can walk around a lift. Twist: add a floor marker that shows where the center of mass is at each moment. #### Narcissus — Norman McLaren (1983) - Video: https://www.youtube.com/watch?v=YN9Iu0z2waI - Interaction: Performance, Hands & Body, Perception & Effects - Platform & tech: Desktop, optical printer, 35mm film, slow motion - Idea: Dancing with your own copy is a compelling interaction: a delayed or mirrored AR clone of the user makes a partner out of oneself. - What it is: McLaren's last film retells the Narcissus myth as a ballet, using mirror images, multiple exposures and slow motion so the dancer duets with his own reflection. - Technique: Mirror printing, multiple exposure and slow-motion cinematography composited on an optical printer. - Try it: Capture a person with body tracking and spawn a mirrored clone that repeats their moves 2 seconds later beside them in AR. Twist: the clone slowly stops copying and starts improvising. ### Onix-Systems *Software studio; Unity AR/VR development team* Ukrainian software company founded in 2000 whose Unity team built early HoloKit X co-located duel demos plus object-recognition and indoor-navigation AR prototypes. #### Spy Car (AR object recognition demo) — Onix-Systems (2019) - Video: https://www.youtube.com/watch?v=MEfE7v7twsg - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, Vuforia, VisionLib, Unity, Android - Idea: A toy becomes its own interactive manual once the camera recognizes its 3D shape. - What it is: Pointing an Android phone at a toy spy car recognizes the physical object and layers labels, gadget descriptions and animations onto its real body. - Technique: Vuforia or VisionLib model-target tracking matches the toy against its 3D model, so annotations stay registered to the object as it is turned. - Try it: Scan a small physical object as an object target and attach three tappable hotspots that explain its hidden parts. Twist: reveal an animated X-ray view of the inside when the object is turned upside down. #### Dragon Duel (HoloKit X demo) — Onix-Systems (2021) - Video: https://www.youtube.com/watch?v=jwX5b-6chVI - Interaction: Hands & Body, Play, Spatial Mapping - Platform & tech: Headset, Phone, HoloKit X, ARKit, Unity - Idea: A boss fight staged in your own room, with you moving your whole body to dodge. - What it is: A player in a HoloKit X headset battles a dragon that flies around the real room, dodging its fire and casting spells back at it. - Technique: ARKit world tracking renders a stereo view on the iPhone inside the HoloKit X, and the dragon's flight path is likely scripted around the user's tracked head position. - Try it: Script a flying creature that circles the player's head position in AR and periodically fires a slow projectile the player must physically sidestep. Twist: the creature lands on the largest detected horizontal surface when it gets tired. #### Duel of Wizards (HoloKit X demo) — Onix-Systems (2021) - Video: https://www.youtube.com/watch?v=xr0T2z9WPJc - Interaction: Shared & Social, Play, Hands & Body - Platform & tech: Headset, Phone, HoloKit X, ARKit Collaboration, Multipeer Connectivity, Unity - Idea: A wand duel where the opponent is a real person across the room, not a computer character. - What it is: Two players wearing HoloKit X face each other in the same room and aim their controllers like wands; spells fly across the real space and hit the opponent's shared-anchored avatar. - Technique: Players join an ARKit Collaboration session over Apple Multipeer Connectivity to share one coordinate frame, while game logic syncs over UNET. - Try it: Build a two-phone AR duel where both devices share an anchor and a tap fires a projectile from one phone's position toward the other's. Twist: make projectiles bounce off real walls detected by plane detection. #### Supermarket AR Navigation (QR anchor) — Onix-Systems (2023) - Video: https://www.youtube.com/watch?v=AtCkMlrbiHk - Interaction: Information & UI, Location & City, Spatial Mapping - Platform & tech: Phone, Unity, AR Foundation, QR code anchors - Idea: A printed QR code works as a cheap indoor GPS fix that re-anchors an AR route inside a store. - What it is: A shopper scans a QR code on a supermarket shelf and an AR path appears on the floor, guiding them aisle by aisle to the product on their list. - Technique: The QR code gives a known pose that aligns a pre-authored store map with the AR session, after which world tracking keeps the floor arrows in place. - Try it: Map a corridor of your school with three QR codes as anchors and draw an AR arrow path from the entrance to a classroom. Twist: when two users' routes cross, show each of them the other's arrow in a different colour. ### Pablo Valbuena *Artist working with light, space and time* Spanish artist whose 2007 'Augmented Sculpture' at Ars Electronica fused projected light with physical geometry, a founding work of art projection mapping. #### Augmented Sculpture v1.2 — Pablo Valbuena (2007) - Video: https://www.youtube.com/watch?v=5nzhV0x3_qM - Interaction: Projection, Perception & Effects - Platform & tech: Projection, projection mapping - Idea: Precise projection makes still white blocks warp through space and time. - What it is: A white geometric sculpture is overlaid with precisely mapped projected light, so edges, planes and shadows appear to move, fold and rebuild the physical form. - Technique: A projector is calibrated to a white polyhedral sculpture whose exact 3D model is known, so rendered light and shadow on each face can simulate edges and planes moving over the static form. - Try it: Fold two or three geometric solids from white cardboard, align the projection face by face in MadMapper or TouchDesigner, and use only black and white lines to make the blocks look like they fold and rebuild. Twist: make one face 'go missing' by projecting the background color onto it, and see whether it disappears. #### Augmented Space (The Hague City Hall) — Pablo Valbuena (2008) - Video: https://www.youtube.com/watch?v=eGzBayXZOJw - Interaction: Projection, Location & City - Platform & tech: Projection, projection mapping - Idea: Project along a building's own structural lines to bring a city hall facade to life. - What it is: Architectural projection on The Hague city hall that traces and animates the building's own structural lines. - Technique: Architectural projection mapping from a surveyed facade model: content is authored along the building's own structural lines so animation traces, splits and redraws the real geometry. - Try it: Photograph a classroom wall or door frame head-on, trace all its structural lines, and in the projection make the lines light up one by one, shift out of place and return. Twist: move only one line and keep everything else still, to see how much perception a minimal change can shift. #### Para-Site (Mattress Factory) — Pablo Valbuena (2011) - Video: https://www.youtube.com/watch?v=h4XQAOH7liE - Interaction: Projection, Perception & Effects - Platform & tech: Projection, projection mapping - Idea: Light projections that live like parasites on the corners of a building. - What it is: Site-specific video projection on the architecture of the Mattress Factory, with light that seems to crawl along corners and surfaces. - Technique: Site-specific projection mapping onto corners and edges of a room, with a keystone- or mesh-warped projection so thin light lines crawl precisely along architectural edges. - Try it: Project thin lines of light that slowly crawl along the edges of one corner of your classroom, fitted precisely with mesh warp. Twist: make the light behave like a 'parasite' that shies away when viewers approach (detected with a camera). #### re-flex (Z33) — Pablo Valbuena (2012) - Video: https://www.youtube.com/watch?v=ixrj2kFhGik - Interaction: Projection, Perception & Effects - Platform & tech: Projection, projection mapping - Idea: An installation where light reveals and warps the geometry of a room. - What it is: A projected light installation for Z33 that reveals and distorts the geometry of the room, discussed by the artist. - Technique: Projected light lines calibrated to the room's geometry reveal or contradict its perspective, relying on anamorphic alignment from chosen viewpoints. - Try it: Project a set of lines in a small room that only join into a complete figure from one position (anamorphic perspective) and look like fragments from anywhere else. Twist: mark a wrong viewing spot on the floor and let the audience find the right one themselves. ### Pablo Ventura *Choreographer working with robots, motion tracking and algorithmic dance* Spanish-born choreographer Pablo Ventura ran Ventura Dance Company in Zurich, staging dancers alongside industrial robots (ZONE, with robotlab), commissioning the hanging robot Kubic from Louis-Philippe Demers, building a computer-aided Choreography Machine and using motion tracking to let dancers play sound and leave traces in space. #### ZONE (Kuka robot choreography) — Pablo Ventura (2001) - Video: https://www.youtube.com/watch?v=PD1i6rItAfA - Interaction: Performance, Tangible Objects - Platform & tech: Projection, Kuka robot, robotics - Idea: Choreograph an industrial robot as a dancer next to humans. - What it is: In the dance work ZONE, a large industrial Kuka robot arm made by robotlab performs a choreography by Pablo Ventura alongside the company's dancers. - Technique: The Kuka arm's motion paths were programmed as dance phrases in collaboration with robotlab and timed against live dancers and music. - Try it: Program a cheap robot arm or a phone on a motorised selfie stick with a 30-second phrase, then choreograph a human duet partner around it. Twist: let the human copy the robot's timing but never its shape. #### Kubic's Cube — Pablo Ventura (2006) - Video: https://www.youtube.com/watch?v=xBki9nYeZ9Q - Interaction: Performance, Tangible Objects, Gaze & Attention - Platform & tech: Projection, robotics, LED, neon - Idea: A robot becomes the only dancer, and the audience walks around it like a sculpture. - What it is: A robot called Kubic hangs in the centre of a cube of neon and LED lights and performs a live choreography to music by Francisco López, while the public walks freely around it and sees it from every side. - Technique: A suspended robot built by Louis-Philippe Demers is operated live by Ventura, mixing static poses with chance-based interactive movement patterns that make it seem alive. - Try it: Hang a servo-driven object (a lamp or cardboard head) from the ceiling, program three poses and a random 'breathing' motion, and let visitors walk around it. Twist: make it turn toward whoever moves last. #### Choreophony — Pablo Ventura (2013) - Video: https://www.youtube.com/watch?v=EtQ31OHI60o - Interaction: Hands & Body, Voice & Sound, Performance - Platform & tech: Projection, motion tracking, audio samples - Idea: Choreography that writes its own polyphonic soundtrack. - What it is: A lecture-performance where a dancer's movements trigger and shape live audio samples, so the audience can follow the causal and chance interplay between movement, space and sound. - Technique: Camera-based motion tracking maps the dancer's position and movement quality to several sample layers at once, producing an interactive polyphonic score. - Try it: Map four stage zones and two movement speeds to eight short samples with a webcam and MediaPipe, then perform the same phrase twice in different places to hear two different pieces. Twist: explain the mapping to the audience first, then break it. #### Heliopolis — Pablo Ventura (2014) - Video: https://vimeo.com/109680105 - Interaction: Projection, Location & City, Performance - Platform & tech: Projection, motion tracking, light sculpture - Idea: A city that records every citizen's path as a visible trace. - What it is: A dance-media performance-installation about a fictional city, where a light sculpture hovers like an obelisk and the dancing citizens are continuously 'scanned' and tracked, leaving trails of their presence as they interact with the city's sounds. - Technique: Motion tracking of the dancers drives projected traces and sound triggers, combined with a video screen that follows a person's day in the city. - Try it: Track students walking across a projected floor map with an overhead webcam and leave a fading line behind each of them, then play a short sound when two paths cross. Twist: make the trails permanent and show the 'surveillance map' at the end. ### Paige Piskin *AR filter creator (Snap Lens Studio, Spark AR)* Official Snap Lens Creator whose makeup, fashion and cartoon filters have been used billions of times; she also teaches lens making. #### Lil Cartoon Bratz Doll — Paige Piskin, Snap Inc. (Snapchat Lenses) (2021) - Video: https://www.youtube.com/watch?v=UIexjgFSXjc - Interaction: Face, Perception & Effects - Platform & tech: Phone, Lens Studio, machine learning - Idea: Turn your face into a 90s fashion-doll cartoon in real time. - What it is: A Snapchat lens that turns the user's face into a big-eyed, glossy-lipped 90s fashion-doll cartoon, inspired by Piskin's favourite childhood toy. - Technique: A machine-learning face transformation (likely a GAN-based style model) restyles the face into a cartoon while keeping head pose and expression. - Try it: Use Lens Studio's ML stylization or face-distortion tools to turn a face into the style of a toy from your childhood. Twist: write a short note explaining which features of the original face you kept. #### Coachella 'Luminous' filter — Paige Piskin (2024) - Video: https://www.youtube.com/watch?v=Wca9ejwmY0c - Interaction: Face, Perception & Effects - Platform & tech: Phone, Spark AR, Instagram - Idea: A festival-themed glowing makeup filter that makes your face shine like stage lights. - What it is: A glowing, light-reactive face effect created for Coachella on Instagram, with behind-the-scenes on the makeup design and technical build. - Technique: Face-attached emissive materials with bloom are modulated by camera luminance or light estimation, so the glow reacts to the surrounding lighting. - Try it: Make a face filter whose glow changes with ambient brightness, getting brighter in the dark and fainter in bright light. Twist: drive the glow with microphone volume instead of brightness. #### Coachella 'MetaButterfly' filter — Paige Piskin (2024) - Video: https://www.youtube.com/watch?v=cLaf9RX2IdU - Interaction: Face, Perception & Effects - Platform & tech: Phone, Spark AR, Instagram - Idea: An AR filter of iridescent butterfly makeup and wings designed for a music festival. - What it is: An Instagram AR effect for a Coachella campaign with butterfly-themed makeup effects on the face; Piskin walks through the inspiration, makeup and tech. - Technique: Face-tracked butterfly meshes and makeup textures are anchored to facial landmarks, with wing animations driven by time or facial movement. - Try it: Place a few AR butterflies that rest along the eyebrows or cheekbones and fly away when the mouth opens. Twist: after flying off, the butterflies land on real objects in the frame. ### Pell Ensemble (Rebecca Evans) *Digital dance company led by choreographer Rebecca Evans* Cross-disciplinary dance company directed by choreographer Rebecca Evans that makes digital dance works for stage, streets and classrooms, from an interactive audience-data game to a walking AR trail and dance-and-code workshops with UCL and Studio Wayne McGregor. Evans spoke alongside AΦE at the Digital Bodies + Virtual Spaces symposium and the Digital Innovation Network. #### Leytonstone TrailblazARs — Pell Ensemble (Rebecca Evans) (2019) - Video: https://www.youtube.com/watch?v=Qsexe2mxU54 - Interaction: Location & City, Performance, Information & UI - Platform & tech: Phone, Blippar, image tracking - Idea: Young residents hide their own dances in the history of their street. - What it is: A walking AR app along Leytonstone High Road where phones reveal dance, music and stories about the street's history, all created by local schoolgirls with Pell Ensemble. - Technique: Students filmed performances and built the trail in Blippar, which triggers video and audio overlays when phones recognise images or locations on the high street. - Try it: Pick five spots around school, film a ten-second dance at each that tells one fact about the place, and attach them as AR markers with a WebAR tool. Twist: the dance at the last stop must quote a move from each earlier stop. #### Upload/Unplug — Pell Ensemble (Rebecca Evans) (2019) - Video: https://www.youtube.com/watch?v=knGN34-HFDs - Interaction: Shared & Social, Performance, Information & UI - Platform & tech: Phone, Projection, audience voting, tablets - Idea: Your data literally choreographs a person in front of you. - What it is: A live gaming performance where the audience's choices on devices become data that is uploaded to shape David, a new human being, whose dance changes with what the room decides. - Technique: Audience members answer prompts on tablets or phones; the votes are aggregated live and trigger different choreographic sections and screen content (likely a custom web voting system). - Try it: Build a quick web poll with three buttons and map each result to a projected instruction card for a dancer. Twist: reveal at the end what personal data each choice would have 'cost' the audience. #### Move: Build: Code — Pell Ensemble (Rebecca Evans) (2020) - Video: https://www.youtube.com/watch?v=8elvMUiUDM0 - Interaction: Hands & Body, Tangible Objects, Performance - Platform & tech: Wearable, micro:bit, robotics - Idea: Kids learn to code by teaching a robot to answer their dance. - What it is: In a three-day summer workshop, 12–14-year-olds build and code a robot that responds to their own choreographed dance movements, ending in a performance with the robots. - Technique: Students wear or hold micro:bit-style sensor boards whose accelerometer readings are sent by radio to a small robot that reacts with movement and light (likely micro:bit). - Try it: Tape a micro:bit to a wrist and send its tilt to a second micro:bit on a wheeled robot that drives when the dancer spins. Twist: make the robot copy the dance only after a four-count delay, like a canon. ### Philip Beesley / Living Architecture Systems Group *Architect and artist; professor at the University of Waterloo* Canadian architect whose 'living architecture' installations are dense canopies of acrylic fronds, sensors, shape-memory actuators, lights and chemistry that breathe and ripple as visitors move through them. #### Hylozoic Ground — Philip Beesley / Living Architecture Systems Group (2010) - Video: https://www.youtube.com/watch?v=jgL2ppDmNtA - Interaction: Hands & Body, Spatial Mapping, Perception & Effects - Platform & tech: Desktop, shape-memory alloy, proximity sensors, microcontrollers, acrylic - Idea: Thousands of tiny local reactions add up to a room that feels alive; in AR, many small agents reacting to nearby people beat one big scripted animation. - What it is: At the Venice Architecture Biennale, a forest-like canopy of thousands of acrylic fronds, feathers and glass vessels ripples, breathes and glows as visitors walk beneath it. - Technique: Capacitive and infrared proximity sensors feed a distributed network of microcontrollers that pull shape-memory-alloy wires, moving fronds and triggering lights in waves that spread outward. - Try it: Fill a ceiling in AR with hundreds of simple fronds that each bend toward the nearest tracked hand and pass the motion on to their neighbours. Twist: add a slow 'breathing' rhythm that pauses when nobody is present. #### Sibyl — Philip Beesley / Living Architecture Systems Group (2012) - Video: https://www.youtube.com/watch?v=jTc14p5ognM - Interaction: Hands & Body, Spatial Mapping, Perception & Effects - Platform & tech: Desktop, LEDs, shape-memory alloy, sensors, laser-cut mesh - Idea: A suspended form that trembles when you look up at it shows how overhead space — often ignored in AR — can host responsive content. - What it is: For the Sydney Biennale, a suspended cloud of lace-like meshes, glass flasks and lights shivers and pulses in response to visitors below. - Technique: A mesh of laser-cut components is threaded with sensors, LEDs and shape-memory actuators controlled by a networked set of microcontrollers. - Try it: Anchor a floating AR lattice above the viewer's head and make its nodes flicker and sag toward whoever is directly below. Twist: let two viewers' positions create interference patterns in the lattice. #### Meander — Philip Beesley / Living Architecture Systems Group (2020) - Video: https://www.youtube.com/watch?v=nWijIywLUnw - Interaction: Hands & Body, Voice & Sound, Perception & Effects - Platform & tech: Desktop, sensors, machine learning, LEDs, sound - Idea: A responsive space with its own slowly changing mood feels more alive than one that reacts identically every time — a lesson for persistent AR worlds. - What it is: A long hanging landscape in Cambridge, Ontario whose fronds, lights and sound respond to visitors with behaviours that learn and drift over time. - Technique: Sensor data from the crowd feeds behaviour software (reportedly with machine-learning components) that sets light, sound and actuator patterns across a distributed network. - Try it: Make an AR installation whose reaction to visitors (colour, speed, sound) slowly drifts based on how many people visited in the last hour, stored in a simple server. Twist: let it become 'sleepy' after crowds and 'curious' after silence. #### Grove — Philip Beesley / Living Architecture Systems Group (2021) - Video: https://www.youtube.com/watch?v=f-wQeHNae48 - Interaction: Hands & Body, Voice & Sound, Spatial Mapping - Platform & tech: Desktop, sensors, LEDs, multichannel sound, 3D printing - Idea: Surrounding a visitor with many soft, reacting elements creates immersion without any screen — a reminder that AR can build space with layers, not just objects. - What it is: At the 2021 Venice Architecture Biennale, a dense grove of hanging fronds, lights and speakers forms a walk-in environment that murmurs and glows around visitors. - Technique: Networked sensors, lights and a multichannel sound system are embedded in a suspended structure of 3D-printed and laser-cut components. - Try it: Build a walk-in AR grove of translucent hanging strands with spatial audio sources that brighten and whisper as the viewer passes. Twist: let each strand hold a short voice recording left by a previous visitor. ### Philippe Parreno *Artist; exhibitions as automated systems* French artist who treats exhibitions as scores: lights, screens, blinds, sound and floating objects are run by software or living systems, as in Anywhen at Tate Modern's Turbine Hall. #### Marquee — Philippe Parreno (2013) - Video: https://www.youtube.com/watch?v=h8pMX6jO2Z4 - Interaction: Perception & Effects, Shared & Social - Platform & tech: Projection, light bulbs, neon, acrylic - Idea: A familiar signal with its content removed makes people notice the signal itself; AR UI can be exhibited as an empty frame to make users feel anticipation. - What it is: A white cinema marquee hangs in the gallery with no title on it, its bulbs and neon tubes flickering in patterns as if announcing a film that never arrives. - Technique: Incandescent bulbs, neon and acrylic panels are sequenced by a controller to reproduce the rhythm of classic theatre marquees. - Try it: Model a blank marquee in Blender, bring it into Lens Studio and hang it over a real classroom door with blinking bulbs but no text. Twist: text appears only when a second user points their phone at the same marquee. #### H {N)Y P N(Y} OSIS — Philippe Parreno (2015) - Video: https://www.youtube.com/watch?v=Opa4J9VXReo - Interaction: Gaze & Attention, Performance, Perception & Effects - Platform & tech: Projection, marquees, player piano, rotating bleachers - Idea: Moving the audience itself (rotating the seats) is a strong way to direct attention; AR can guide a viewer's body instead of adding arrows. - What it is: In the vast drill hall of New York's Park Avenue Armory, glowing marquees, film screens, a self-playing piano and rotating bleachers carrying the audience are switched on and off in a choreographed sequence. - Technique: A computer-controlled score cues lights, films, sound and mechanised seating, so the hall behaves like one automaton. - Try it: Create a phone AR tour in which the user stands on a marked spot and virtual light boxes and screens around them switch on in a set order, each drawing the gaze to a new direction. Twist: only one element may be lit at a time, forcing the user to turn their whole body. #### Anywhen — Philippe Parreno (2016) - Video: https://www.youtube.com/watch?v=M1RWxQaM5mc - Interaction: Performance, Information & UI, Shared & Social - Platform & tech: Projection, motorised screens, lighting, bioreactor - Idea: An exhibition can be a timeline, not a set of objects; AR experiences can choreograph when things appear, not only where. - What it is: In Tate Modern's Turbine Hall, screens rise and fall, lights dim, fish balloons drift and sounds move around the space, all in a sequence partly steered by live microorganisms in a lab on site. - Technique: A central control system triggers motorised screens, lighting rigs and speakers, with some cues driven by sensor readings from a bioreactor of microorganisms (link details likely). - Try it: Write a 10-minute AR 'score' for a hall: at set times, a floating screen, a light cone and a sound source appear, move and vanish, synced for every phone by the clock. Twist: let one cue be triggered by live data such as the room's noise level or the weather outside. #### My Room Is Another Fish Bowl — Philippe Parreno (2018) - Video: https://www.youtube.com/watch?v=B21g3MFnpUI - Interaction: Spatial Mapping, Hands & Body, Perception & Effects - Platform & tech: Desktop, helium balloons, mylar, air currents - Idea: Objects that drift at eye level and react to air turn a room into water without any screen; AR creatures feel present when they avoid or bump into the body. - What it is: Dozens of helium-filled mylar fish float at head height through a gallery, drifting with the air currents and gently bumping into visitors. - Technique: Balloons are weighted to neutral buoyancy so tiny drafts from visitors and ventilation steer them (weighting method likely small counterweights). - Try it: Fill a room with 20 AR fish that hover at the user's eye height and drift away from the phone as if pushed by air. Twist: use the microphone so blowing on the phone makes the nearest fish swim off. ### Przemyslaw Zaworski *Graphics programmer* Polish programmer who has published hundreds of small Unity shaders and compute-shader demos since 2016, many of them ShaderToy ports, each with a short YouTube video and MIT-licensed source. #### ShaderToy to Unity: Fluid Dynamics — Przemyslaw Zaworski (2017) - Video: https://www.youtube.com/watch?v=8mBqEpj_3tk - Source code: https://github.com/przemyslawzaworski/Unity3D-CG-programming - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: A self-feeding texture is enough to make convincing, never-repeating fluid. - What it is: Colourful fluid swirls endlessly across a texture as Florian Berger's ShaderToy fluid runs inside Unity. - Technique: A compute shader reads the previous frame's texture, advects colour along a rotational velocity field, and writes to a second render texture; the two buffers swap every frame to accumulate the flow. - Try it: Put the fluid texture on an AR plane under a real object and inject colour at the object's position each frame so it seems to stir paint on the table. Twist: use the phone's motion as the stirring force. #### See-Through Objects Shader — Przemyslaw Zaworski (2018) - Video: https://www.youtube.com/watch?v=1lw2hrVuPYk - Source code: https://github.com/przemyslawzaworski/Unity3D-CG-programming - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Desktop, Unity, ShaderLab - Idea: A mask decides where a solid object stops being solid. - What it is: Parts of an object turn see-through according to a mask, revealing what is behind, with optional desaturation and grain filters on the visible surface. - Technique: An unlit shader samples a colour map and a mask texture and discards or blends pixels where the mask is dark, with keyword variants for desaturation and grain. - Try it: Build an AR x-ray: cover a real wall with an occlusion plane that uses a mask from the phone's touch point, so the user 'cuts' holes to see virtual pipes behind the wall. Twist: make the hole follow the user's gaze on a headset. #### ShaderToy to Unity: Particle Collision — Przemyslaw Zaworski (2018) - Video: https://www.youtube.com/watch?v=gUhTbheuoyo - Source code: https://github.com/przemyslawzaworski/Unity3D-CG-programming - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: Particle physics can live entirely in a GPU buffer. - What it is: Thousands of particles move and collide with one another in real time, ported from a ShaderToy sketch into Unity. - Technique: A compute shader stores each particle's position and velocity in a structured buffer, integrates motion with the frame delta and resolves collisions, then a shader draws the buffer as points. - Try it: Spawn the particle buffer above a detected AR table plane and use the plane height as the collision floor so particles pile up on the real table. Twist: use the LiDAR mesh so particles also bounce off real objects. #### Pencil Effect — Przemyslaw Zaworski (2026) - Video: https://www.youtube.com/watch?v=Km6WS_T9ER0 - Source code: https://github.com/przemyslawzaworski/Unity3D-CG-programming - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL - Idea: Turn any live image into a moving sketchbook page. - What it is: A rendered 3D scene is redrawn in real time as a hand-sketched pencil drawing with hatching strokes and paper grain. - Technique: A single full-screen pixel shader detects edges in the source image and layers hash-based noise strokes whose density follows luminance, likely with a scale factor to control stroke size. - Try it: Run the pencil shader as a post effect on the passthrough camera image of an AR Foundation app so the real world looks hand-drawn while virtual objects stay colored. Twist: fade from sketch to photo wherever the user points the phone. ### Random Studio *Experience design studio* Amsterdam studio founded by Daan Lucas that builds interactive installations, spatial scenography and retail experiences for cultural institutions and brands. It also made Hidden Characters and Point & Repair for SPACE10's Everyday Experiments and runs an innovation studio for self-initiated prototypes. #### Conduct The Orchestra — Random Studio (2013) - Video: https://vimeo.com/67575564 - Interaction: Hands & Body, Voice & Sound, Performance - Platform & tech: Desktop, motion tracking, audio time-stretching - Idea: Anyone can be the conductor of a great orchestra. - What it is: At Amsterdam's Concertgebouw, members of the public conduct their own orchestra: their arm movements set the tempo and dynamics of a recorded performance. - Technique: Motion tracking reads the conducting gesture and its speed and size are mapped to playback tempo and volume of the orchestral recording. - Try it: Track a hand with MediaPipe and map its up-down speed to the playback rate of a song in the browser. Twist: split the orchestra into sections that follow different students. #### SUN — Random Studio (2017) - Video: https://vimeo.com/212766197 - Interaction: Tangible Objects, Play, Portals & Worlds - Platform & tech: Projection, tracking, real-time rendering, projection - Idea: Bounce a ball to move the sun. - What it is: Made with artist Philip Schuette: a giant bouncy ball controls a projected sunset, so bouncing and pushing the ball makes the sun rise and fall over a hyper-real horizon. - Technique: The ball's position and bounces are tracked (likely with a camera or depth sensor) and mapped to the height and colour of a rendered sun in a real-time landscape. - Try it: Track a coloured ball with a webcam in p5.js and map its height to the sun's position and sky colour in a simple gradient landscape. Twist: the moon is controlled by a second ball held by another student. #### BODY — Random Studio (2018) - Video: https://vimeo.com/314018273 - Interaction: Hands & Body, Information & UI, Play - Platform & tech: Desktop, pose estimation, image search - Idea: Use your whole body as a search term. - What it is: A playful search engine for Cinekid Festival, made with Philip Schuette: strike a pose and it is reflected back as images from the internet that match your body's shape, cycling as you move. - Technique: Pose estimation turns the visitor's body into a query, and a pre-indexed library of web images is matched by pose or silhouette and shown in real time. - Try it: Collect 50 photos of people or statues, extract their poses with MediaPipe, and show the closest match to a student's live pose. Twist: search a collection of objects instead of people. #### A Moving Sanctuary — Random Studio (2026) - Video: https://vimeo.com/1196639577 - Interaction: Hands & Body, Perception & Effects, Voice & Sound - Platform & tech: Projection, thermal camera, computer vision, AI - Idea: A room that breathes with you. - What it is: A cocoon-like pod made for Lexus at Milan Design Week: visitors lie down inside and their breathing and small body movements steer the light and sound around them. - Technique: Thermal sensing and computer vision detect breathing, a custom-trained AI model classifies inhale and exhale in real time, and the output drives a continuous choreography of light and sound. - Try it: Use a phone accelerometer on the chest or a webcam to detect breathing and slowly pulse a projected light or LED strip in the same rhythm. Twist: the light gently leads the breath by breathing a little slower than you. ### Rebecca Horn *Artist; performance, body extensions and kinetic sculpture* German artist (1944–2024) who began with wearable 'body extensions' that stretched the senses and moved on to feathered, motorised sculptures and room-sized machines that seem to breathe, paint and play. #### Finger Gloves and body extensions — Rebecca Horn (1972) - Video: https://www.youtube.com/watch?v=6uEkq3IBIf0 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Wearable, fabric, balsa wood, wearable sculpture - Idea: Extending a finger by a metre changes how a whole room feels; AR tools that lengthen reach or add body parts do the same, and the body should feel them first. - What it is: Horn's early wearable extensions, such as metre-long finger gloves that let her touch a wall across the room and a unicorn horn strapped to the head, stretch the body's reach and change how the wearer feels space. - Technique: Lightweight rigid extensions strapped to the fingers or head shift the wearer's proprioception and make every contact with a surface a remote, amplified gesture. - Try it: Build a hand-tracked AR prototype (Quest passthrough, Vision Pro or Lens Studio hand tracking) that adds 1 m virtual fingers to the user's hand and lets them brush real walls with light traces. Twist: make the fingers heavy and lagging so they sway behind the hand. #### Pencil Mask (Bleistiftmaske) — Rebecca Horn (1972) - Video: https://www.youtube.com/watch?v=Eh9JH7daSbg - Interaction: Face, Drawing & Making - Platform & tech: Wearable, fabric straps, pencils, performance - Idea: The face becomes a drawing tool and every small head movement leaves a mark, showing that head pose alone is expressive enough to draw with. - What it is: A mask of fabric straps studded with pencils covers the face; as Horn rocks her head in front of a wall, the pencils leave a dense record of her movement. - Technique: Rigid pencils fixed to a head harness translate the head's position and rotation directly into strokes on a nearby wall, a physical head-pose recorder. - Try it: Make a face-tracked AR filter where virtual pencils sprout from the face and draw on the nearest detected wall plane as the user moves their head. Twist: let two people draw on the same wall and see whose head lines cross. #### Kleines Federrad (Small Feather Wheel) — Rebecca Horn (1982) - Video: https://www.youtube.com/watch?v=ZOeO6vB1uxU - Interaction: Perception & Effects - Platform & tech: Desktop, feathers, motor, metal - Idea: A single slow opening gesture, borrowed from an animal, is enough to make a machine feel alive; AR objects can borrow the same body language. - What it is: A small motorised wheel of feathers opens and turns slowly, fanning out like a bird's display and folding shut again. - Technique: A small motor drives a hinged wheel of real feathers through an open–rotate–close cycle, so soft natural material carries a mechanical rhythm. - Try it: Design an AR 'feather wheel' that sits on a table and opens like a fan when the user leans in close (distance from camera), then folds shut when they back away. Twist: make it open only for quiet voices, using the microphone level. #### Concert for Anarchy — Rebecca Horn (1990) - Video: https://www.youtube.com/watch?v=l1y1d7AcNNY - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, grand piano, motors, ceiling suspension - Idea: Hanging a familiar object upside down and letting it erupt on a timer turns a museum room into suspense; AR objects gain drama from long stillness and sudden, rare events. - What it is: A grand piano hangs upside down from the ceiling; every few minutes its keyboard bursts out and its keys spill forward with a jarring chord before it slowly closes again. - Technique: Motors push the keyboard and keys outward and retract them on a cycle, while the inverted suspension makes an ordinary instrument feel unstable and alive. - Try it: Anchor an upside-down virtual piano (or other furniture) to the real ceiling with ARKit or WebXR plane detection and make it burst open with sound at random intervals. Twist: make it erupt only when nobody is looking at it (use camera direction). ### Red Paper Heart *Brooklyn art and technology studio for interactive installations* Brooklyn studio that makes interactive installations with sensors, projection and light, often using openFrameworks, for museums and brands. #### PLAYGROUND — Red Paper Heart (2013) - Video: https://vimeo.com/61893280 - Interaction: Voice & Sound, Hands & Body, Projection - Platform & tech: Projection, openFrameworks, Kinect, projection mapping - Idea: Walk into the music and see the room answer every sound. - What it is: A small shed is turned into a black-and-white projected world that reacts to music and to visitors' movements, and whoever holds the music controller conducts the whole room. - Technique: Five projectors map the interior walls, and a Kinect tracks movement while audio analysis of the playing track drives the geometric visuals (with Fake Love and Aramique). - Try it: Project audio-reactive line drawings onto the corner of a room and let the position of the nearest person change the style. Twist: the room goes silent and still when nobody is inside. #### Hidden Stories — Red Paper Heart (2015) - Video: https://vimeo.com/132444721 - Interaction: Voice & Sound, Tangible Objects - Platform & tech: Wearable, sensors, audio - Idea: Wallpaper you can eavesdrop on, like pressing a glass to the wall. - What it is: Visitors press a cup-shaped listening device against a patterned wallpaper; each shape in the pattern plays a real person's story about an everyday object. - Technique: Sensors hidden behind each wallpaper shape detect the listening cup, which plays the matching audio clip. - Try it: Put image markers or NFC tags behind a printed pattern and play a recorded story when a phone or cup touches each shape. Twist: two stories play only when two people listen at the same time. #### Moving Stained Glass — Red Paper Heart (2016) - Video: https://vimeo.com/151795207 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, transparent LCD, generative graphics - Idea: Stained glass that moves, lit only by the sun. - What it is: Sunlight shines through a window whose coloured panes are generated and animated by a display, casting moving stained-glass patterns across the room. - Technique: Likely a transparent LCD without a backlight acts as a filter for natural sunlight, showing a generative art system that casts real coloured light. - Try it: Put a coloured transparent sheet (or an old phone screen with the backlight removed, if possible) in a sunny window and design patterns for the shadows it casts at different hours. Twist: the pattern tells the time. #### The Reading Light — Red Paper Heart (2022) - Video: https://vimeo.com/759623238 - Interaction: Projection, Tangible Objects, Perception & Effects - Platform & tech: Projection, motion tracking, projection - Idea: Words that exist only where the light shines. - What it is: A motion-tracked pendant lamp is swung past a textured wall, and digital letters appear only where its light falls, as if the words were part of the room. - Technique: The lamp's position is tracked and a projector likely renders text masked to the lamp's virtual light cone, so it appears to be revealed by the real light. - Try it: Track a flashlight with a webcam and project a hidden poem that is only visible inside the flashlight's circle. Twist: the poem changes depending on how long the light rests on each line. ### Resolution Games *XR game studio (Angry Birds FPS, Demeo, Spatial Ops)* Stockholm studio founded by Tommy Palm that has shipped mixed-reality games on Magic Leap, Quest and Vision Pro since 2018. #### Angry Birds AR: Isle of Pigs — Resolution Games (2019) - Video: https://www.youtube.com/watch?v=NWDptXnBhsM - Interaction: Spatial Mapping, Play - Platform & tech: Phone, ARKit, ARCore, Unity - Idea: Build Angry Birds fortresses on your living-room floor and walk around them to find the weak spots. - What it is: Pig fortresses are built on the player's floor at room scale; players walk around them to find weak points and then launch birds from a slingshot. - Technique: Destructible physics structures are built at room scale on the floor plane, and a slingshot is aimed from the camera position so the player walks to find angles. - Try it: Place an AR block tower on the floor and launch balls from a slingshot at the phone's position, nudging players to walk to the side to find weak spots. Twist: the tower's shape is set automatically by the amount of open floor in the real room. #### Angry Birds FPS: First Person Slingshot — Resolution Games (2019) - Video: https://www.youtube.com/watch?v=xvcpH97zCoQ - Interaction: Spatial Mapping, Play - Platform & tech: Headset, Magic Leap One - Idea: Build pig forts on your real tables and floor, then pull back a slingshot by hand to knock them down. - What it is: Angry Birds rebuilt for Magic Leap One: pig fortresses are built on your real tables and floor, and you pull back a hand-held slingshot to knock them down. - Technique: Structures are spawned on detected tables and floors, and a slingshot mechanic maps the controller pull-back vector to launch velocity for physics projectiles. - Try it: Stack an AR block tower on a table and fire a ball at it, with launch power set by how far you pull the phone back. Twist: use real objects as cover that hides some of the blocks. #### Demeo (Mixed Reality mode) — Resolution Games (2023) - Video: https://www.youtube.com/watch?v=gxWzjgoNHl8 - Interaction: Shared & Social, Play, Tangible Objects - Platform & tech: Headset, Meta Quest 3, passthrough - Idea: Put a dungeon board game on a real table and gather around it with friends. - What it is: The tabletop dungeon crawler moved into passthrough: the game board sits on your real table and friends gather around it as avatars. - Technique: The game board is anchored to a detected table plane, and remote players appear as networked avatars around the same board. - Try it: Place a small AR game board on a table, sync a remote classmate's moves over the network, and represent them with a simple avatar. Twist: resize the board automatically according to the number of players. #### Spatial Ops — Resolution Games (2024) - Video: https://www.youtube.com/watch?v=cK6mspbNkR0 - Interaction: Shared & Social, Spatial Mapping, Play - Platform & tech: Headset, Meta Quest 3, shared spatial anchors - Idea: Turn your home or office into a mixed-reality shooting arena for players in the same space. - What it is: A mixed-reality FPS in which players lay out cover and objectives across a real space and fight co-located or remote opponents through their homes and offices. - Technique: Players place virtual cover and objectives on a shared anchor map of the real space, and co-located players are synchronized through shared spatial anchors. - Try it: Set up a few virtual cover blocks in the classroom with AR so that a second phone joining the session sees the same layout. Twist: the cover slowly 'erodes' over time. ### Reuben Margolin *Kinetic sculptor* American kinetic sculptor who builds hanging wave sculptures from wood, strings, pulleys, bicycle wheels and glass, in which camshafts add sine waves together so that thousands of points ripple like water or a caterpillar's back. #### Square Wave — Reuben Margolin (2005) - Video: https://www.youtube.com/watch?v=PvcZ2DffaYA - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, wood, strings, pulleys, camshafts - Idea: Two simple sine waves added together already produce a surface that feels alive; AR motion can be built from the same small set of summed waves. - What it is: A square grid of suspended wooden blocks rises and falls as two perpendicular waves pass through it, forming a rippling surface that constantly crosses and recombines. - Technique: Two rows of camshafts each generate a wave; strings running over pulleys sum the two displacements at every grid point, so each block's height is x-wave plus y-wave. - Try it: Place a 20×20 grid of floating cubes over a real table in phone AR and set each height to sin(x+t)+sin(y+t). Twist: let the viewer's phone position set the phase of one of the waves. #### Magic Wave — Reuben Margolin (2007) - Video: https://www.youtube.com/watch?v=4ZjAQkkqtZI - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, wood, strings, bicycle wheels, motor - Idea: Seeing the mechanism beside the effect teaches AR designers to expose the 'engine' of an effect as part of the experience. - What it is: Hundreds of hanging points driven by a stack of turning wheels trace an undulating sheet that looks like ocean swell frozen in wood and string; the video shows how it is built. - Technique: Rotating eccentric wheels act as wave generators and a network of strings mechanically adds their motions at each hanging point (year approximate). - Try it: Build a WebXR scene where three virtual wheels hover beside a hanging mesh and visibly drive it through connecting lines. Twist: let viewers grab a wheel and change its speed. #### Nebula — Reuben Margolin (2015) - Video: https://www.youtube.com/watch?v=RDQsp9pPnjM - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, glass, cables, motors, camshafts - Idea: A volume of points moving together reads as one creature; AR particle clouds gain presence when they move as coherent waves rather than noise. - What it is: A vast cloud of hanging glass pieces fills an atrium and slowly swells and folds in waves, catching light like a nebula breathing above the lobby. - Technique: Motor-driven camshafts above the ceiling feed summed wave motions down thousands of cables to the glass elements, turning a lobby into a 3D wave field. - Try it: Fill a real room with 2,000 virtual glass shards in phone AR and move them with two summed 3D waves; add a refractive shader. Twist: let the cloud slowly lean toward the brightest window. #### Caterpillar with Green Stripe — Reuben Margolin (2018) - Video: https://www.youtube.com/watch?v=RF0o9Em1HnY - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, wood, cams, crank - Idea: A creature's liveliness can come from a single traveling wave; AR characters can feel alive with nothing more than a phase-shifted loop. - What it is: A segmented wooden caterpillar undulates along its length as a crank turns, capturing the wave that runs down a real caterpillar's body. - Technique: A cam per segment, each offset by a fixed angle on a shared shaft, lifts the segments in sequence so the wave travels along the body. - Try it: Build a 12-segment creature crawling across a real floor in Lens Studio, lifting each segment with sin(t - k*i). Twist: make the wave reverse when the camera comes close. ### Robert Smithson *Land artist* American artist (1938–1973) and founding figure of Land Art, best known for Spiral Jetty, a 457-metre coil of basalt in Utah's Great Salt Lake. #### Partially Buried Woodshed — Robert Smithson (1970) - Video: https://www.youtube.com/watch?v=9KXbUVekRg4 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, woodshed, earth, entropy - Idea: An artwork that is meant to decay turns entropy into content, a useful stance for AR works that degrade on purpose. - What it is: At Kent State University, Smithson had a backhoe pile earth onto a woodshed until its central beam cracked, then left it to decay; only traces remain today. - Technique: Twenty truckloads of earth were dumped on the roof of an existing shed until it broke, fixing a moment of collapse. - Try it: Anchor a small AR building at a spot on campus and program it to lose one element every day the site is visited. Twist: publish a timeline slider so visitors can scrub back to how it looked on day one. #### Spiral Jetty — Robert Smithson (1970) - Video: https://www.youtube.com/watch?v=xWd_YGHjWKM - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, basalt rock, earth, salt lake - Idea: A shape that you can walk along but only fully see from the air shows how AR content can reward both the ground view and the overview. - What it is: A 457-metre coil of black basalt and earth spirals counter-clockwise into the red-tinted water of Utah's Great Salt Lake; it was submerged for decades and is now crusted white with salt. - Technique: Dump trucks and a front loader pushed 6,650 tons of rock and earth into the shallow lake along a staked spiral path. - Try it: Walk a spiral on a field while a WebXR app records your GPS trace, then view the path as a raised AR ribbon from above using a 'drone view' toggle. Twist: raise and lower a virtual water level that hides and reveals the spiral. #### Broken Circle/Spiral Hill — Robert Smithson (1971) - Video: https://www.youtube.com/watch?v=7uW4BLGhhcs - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, sand quarry, earthwork, glacial boulder - Idea: Pairing a flat figure (the circle) with a climbable one (the hill) gives two viewpoints on the same work, a classic AR composition pattern. - What it is: In a sand quarry in Emmen, Netherlands, a half-land, half-water circle faces a small hill with a white-sand path spiralling to its top; a huge glacial boulder sits at the circle's centre. - Technique: The circle was dug and built up in the quarry's water and sand, and the hill's path was shaped with white sand and topsoil. - Try it: Place a flat AR ring on the ground and a small AR hill beside it that users can 'climb' by walking up real stairs to look down on the ring. Twist: the ring only closes into a full circle when seen from the top. #### Amarillo Ramp — Robert Smithson (1973) - Video: https://www.youtube.com/watch?v=jvC8TYCqlug - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, earth, stone, ranch lake - Idea: A ramp that curls up and stops invites the body to walk into the sky, a strong lead-in shape for AR content that continues where the ground ends. - What it is: A curved earth ramp rising from the ground and ending mid-air sits in a dry lake on a Texas ranch; Smithson died in a plane crash while surveying it, and Nancy Holt and friends completed it. - Technique: Earth and red shale were piled into a rising, curving ramp that forms an open circle about 46 metres across. - Try it: Stand at the end of a real ramp or staircase and extend it with an AR ramp that continues into the air. Twist: the virtual ramp grows only while someone is walking on the real one. ### Robert Xiao *Assistant Professor, University of British Columbia; former PhD student at CMU FIGLAB* Robert Xiao built many of FIGLAB's projected-AR systems (WorldKit, Desktopography, LumiWatch) and MRTouch with Microsoft Research, which added touch on real surfaces to HoloLens. #### WorldKit — Robert Xiao, Chris Harrison — CMU Future Interfaces Group (2013) - Video: https://www.youtube.com/watch?v=YkEd_wHXMwE - Interaction: Projection, Drawing & Making, Hands & Body - Platform & tech: Projection, Kinect, projector - Idea: Swipe your hand over any table or wall and a touchable button or slider appears there. - What it is: Paint an interface onto any surface with a sweep of the hand: buttons, sliders and displays are projected where you draw them and become touch-enabled. - Technique: A calibrated Kinect and projector let the user paint a region with a hand; the depth map within that region is used as a touch-sensing surface and the painted region becomes a projected widget. - Try it: With a projector and a camera, draw a rectangle on the wall with your hand, and have the system generate a projected button or slider there that switches the lights or music when touched. Twist: have classmates "set up" a projection-only workstation in the classroom. #### Desktopography — Robert Xiao, Chris Harrison — CMU Future Interfaces Group (2017) - Video: https://www.youtube.com/watch?v=L5mCxfjk6hc - Interaction: Projection, Spatial Mapping, Tangible Objects - Platform & tech: Projection, depth camera, projector - Idea: Windows projected on a desk move out of the way of cups and papers and share the space with the clutter. - What it is: Projected apps on a real desk dodge, resize and move around coffee cups, papers and laptops so digital and physical clutter can coexist. - Technique: A depth camera segments physical objects on the desk and a layout optimizer moves and resizes projected windows to avoid occlusion by them while keeping them near their last positions. - Try it: Detect the outlines of objects on a desk with an overhead camera (OpenCV foreground segmentation), and make projected sticky-note windows bounce away or shrink when they hit a cup or book. Twist: let a window "attach" to an object so it follows when the object is moved away. #### LumiWatch — Robert Xiao, Chris Harrison — CMU Future Interfaces Group (2018) - Video: https://www.youtube.com/watch?v=VJNMrulWJ3k - Interaction: Hands & Body, Projection - Platform & tech: Wearable, laser projector, time-of-flight sensors - Idea: The first projection watch that really works, putting a touch screen right on your forearm. - What it is: The first fully functional projection smartwatch: a laser projector and depth sensing put a touchscreen on your forearm. - Technique: A small laser-scanning projector in the watch body projects onto the forearm with keystone correction, and a row of time-of-flight sensors along the watch edge measures finger position along the arm for 1D/2D touch. - Try it: Use a phone as the 'watch': detect your finger's position on your arm with the front camera and MediaPipe, map it to a slider that controls volume, and show feedback with a small projector or the screen. Twist: design an interaction that only works on the arm and cannot be done on a phone screen. #### MRTouch — Robert Xiao, Microsoft Research — Hrvoje Benko & Andy Wilson (2018) - Video: https://www.youtube.com/watch?v=ATNKTJthfC4 - Interaction: Hands & Body, Spatial Mapping - Platform & tech: Headset, HoloLens, depth sensing - Idea: Stick HoloLens holographic interfaces onto real walls and tap them directly with a finger. - What it is: Adds touch input on real walls and tables to HoloLens: users drag out virtual screens on any surface and tap them directly. - Technique: HoloLens depth frames are segmented into planar surfaces, fingertips are detected in depth, and a touch is registered when the fingertip is within a small distance of the plane, letting users drag out virtual screens on real surfaces. - Try it: Detect a wall plane in AR Foundation and use AR hand tracking or MediaPipe fingertip positions to tell whether a finger is close to it; when it is, let the user drag out a rectangular screen and tap on it. Twist: place a virtual screen related to each object's function on a door, a window and a blackboard. ### Romain Tardy *Visual artist; former core member of the AntiVJ label* French artist who co-shaped AntiVJ's architectural projection mapping (Déshérence, O (Omicron)) and now builds site-specific light and projection installations that treat buildings and landscapes as narrative surfaces. #### Déshérence (AntiVJ) — Romain Tardy (2009) - Video: https://vimeo.com/12622615 - Interaction: Projection, Perception & Effects, Location & City - Platform & tech: Projection, projection mapping, 3D façade model - Idea: Make a building facade peel, crumble and reassemble in projection. - What it is: An audiovisual installation in Enghien-les-Bains where projection mapping makes a building façade appear to crack, peel and reassemble in sync with music by Squeaky Lobster. - Technique: Façade geometry is measured and modelled, then animations are rendered to that model and projected from a calibrated position so light aligns with each architectural edge. - Try it: Photograph a classroom wall with a door and windows head-on, animate a 'peeling' effect along their outlines in After Effects or a web page, and align it with a projector. Twist: let the drum beats of a piece of music trigger the order of the peeling. #### O (Omicron) (with Thomas Vaquié) — Romain Tardy (2012) - Video: https://vimeo.com/41486619 - Interaction: Projection, Perception & Effects, Location & City - Platform & tech: Projection, multi-projector mapping, dome projection - Idea: Use projection to reveal the structural beauty of a century-old concrete dome. - What it is: A permanent 360° projection on the concrete dome of Wrocław's Centennial Hall that reveals and reinterprets the ribbed structure of the 1913 building. - Technique: Multiple calibrated projectors cover the interior dome; content is authored on a 3D model of the rib structure so lines trace the real architecture. - Try it: Use an umbrella or a paper dome as the "dome" and project glowing line animations along its ribs. Twist: make the lines "grow" along the structure to simulate the building being constructed. #### The Ark (with Squeaky Lobster) — Romain Tardy (2013) - Video: https://vimeo.com/70131252 - Interaction: Projection, Location & City, Perception & Effects - Platform & tech: Projection, projection mapping on organic forms - Idea: Project onto a real grove of cacti and turn the plants into a glowing archive. - What it is: At night in the Oaxaca ethnobotanical garden, projections and sound make a grove of giant cacti appear to hold and transmit a luminous archive of the plants' history. - Technique: Projectors are aligned to the silhouettes of real cacti so scanned outlines and patterns wrap organic, non-planar surfaces. - Try it: Shine a projector on a potted plant or branches and make a mask animation that follows the plant's outline. Twist: use a microphone so the plant lights up with patterns when it "hears" sound. #### Future Ruins — Romain Tardy (2015) - Video: https://vimeo.com/151385179 - Interaction: Projection, Perception & Effects - Platform & tech: Projection, projection mapping, custom software - Idea: Use light and structure to imagine how future people will see today's buildings as ruins. - What it is: An installation for La Nuit des Images at the Musée de l'Elysée in which built structures and light imagine architecture as ruins seen from the future. - Technique: Custom software (Hand Coded) drives projection or light onto purpose-built physical structures, likely using mapped geometry to animate their surfaces. - Try it: Build a "ruin" out of cardboard, photograph it, and make an aligned projection that shows it sometimes whole and sometimes collapsing. Twist: let the audience choose between its state "100 years later" and "1,000 years later". ### Rubaiat Habib Kazi *Senior Research Scientist, Adobe Research* Rubaiat Habib Kazi studies sketching and animation as dynamic, expressive media (Kitty, Draco, Kinetic Texture). In AR he co-created RealitySketch, RealityTalk and Augmented Physics with Ryo Suzuki, and Pronto for sketch-based AR video prototyping. #### Pronto — Rubaiat Habib Kazi (2020) - Video: https://www.youtube.com/watch?v=CXdgTMKpP_o - Interaction: Drawing & Making, Hands & Body - Platform & tech: Phone, ARKit, sketching - Idea: Sketch on video and act it out yourself to build an AR interaction prototype in minutes. - What it is: Germán Leiva, Cuong Nguyen, Rubaiat Habib Kazi and Paul Asente let designers prototype AR by sketching on video and acting out interactions, with sketches anchored in 3D. - Technique: ARKit records a video with camera poses, and sketches drawn on video frames are lifted into 3D using the tracked camera and detected planes so they stay anchored as the recorded scene is replayed. - Try it: Record an AR video with Reality Composer or Adobe Aero and quickly prototype an AR app's interaction in it with hand-drawn sketches and acting it out, completing two rounds of iteration in 30 minutes. Twist: the prototype may use only hand-drawn lines and no 3D models at all. ### Sean Follmer *Associate Professor of Mechanical Engineering, Stanford University; director of the SHAPE Lab* Co-created inFORM at the MIT Media Lab, then founded Stanford's SHAPE Lab, which builds mobile shape displays, tabletop swarm robots and, more recently, body-anchored audio augmentation. #### Zooids — Sean Follmer (2016) - Video: https://www.youtube.com/watch?v=ahHjSY_GLDg - Interaction: Tangible Objects, Information & UI - Platform & tech: Projection, Desktop, swarm robots, optical tracking, swarm UI - Idea: A display made of many tiny robots that you can touch, rearrange and hold. - What it is: Dozens of small tabletop robots act like physical pixels, arranging themselves into charts, shapes and handles that users can push, drag and grab. - Technique: Tracks each robot with a projector-based optical system and drives them with collision-free motion planning so the swarm forms target layouts and responds to touch. - Try it: Have ten students act as 'pixels' on a gridded floor and follow phone instructions to form a bar chart of class data, then let a volunteer push one to change the data. Twist: use projection to show each pixel's target spot. #### shapeShift — Sean Follmer (2017) - Video: https://www.youtube.com/watch?v=Z6LSsJoDdtY - Interaction: Tangible Objects, Hands & Body - Platform & tech: Desktop, Headset, shape display, robot, haptics - Idea: Put a shape display on wheels so physical pixels can follow virtual objects around. - What it is: A compact pin-based shape display rides on a small robot across the table, rendering moving shapes you can feel and, paired with a headset, standing in physically for virtual objects. - Technique: Moves a 7 mm pitch actuated pin array on an omnidirectional robot base and synchronises its position and height map with tracked virtual content in a head-mounted display. - Try it: Stick a printed marker on a toy car or robot and anchor a virtual object to it in phone AR, so touching the real toy lets you 'touch' the virtual one. Twist: swap the virtual object while keeping the same physical proxy and discuss which feels right. #### Audio Augmentation of Manual Interactions for Mindfulness — Sean Follmer (2025) - Video: https://www.youtube.com/watch?v=4UD07rFtuTw - Interaction: Voice & Sound, Hands & Body, Tangible Objects - Platform & tech: Wearable, motion sensing, sonification, audio AR - Idea: Add sound to everyday hand actions so people slow down and notice them. - What it is: Ordinary actions like pouring water or folding cloth are paired with gentle, responsive sounds, turning chores into a mindful audio experience. - Technique: Senses hand and object manipulation and maps motion features to real-time generated audio feedback delivered to the user. - Try it: Use a phone's accelerometer in a web page to play a soft tone whose pitch follows how slowly you pour a glass of water, then try the chore with and without it. Twist: map a different sound to washing hands. #### Audio Personas — Sean Follmer (2025) - Video: https://www.youtube.com/watch?v=l8lis-JPIzA - Interaction: Voice & Sound, Shared & Social, Hands & Body - Platform & tech: Wearable, spatial audio, body tracking, headphones - Idea: Give people an audible aura that others hear in space, a sound-based profile picture. - What it is: Each person in a room carries a personal sound, a short audio cue anchored to their body, so as you walk around you hear who is nearby and what they are like before you even look. - Technique: Tracks people's positions and renders body-anchored spatial audio cues to each listener through headphones, studied for how they shape social perception. - Try it: Give each student a phone playing a chosen looped sound in a pocket and walk around blindfolded partners who guess who is approaching. Twist: let the sound change with the person's mood or walking speed. ### Simon East (Curvor / Flowfal) *Mobile interaction designer; creator of the Flowfal motion-to-music system* Co-founder of the Leeds-area mobile studio Curvor, where he develops Flowfal, a low-latency system that turns the movement of phones, smartwatches and wands into control of Ableton Live, Max and TouchDesigner. He took part in the 2022 Choreographic Coding Lab at A+E Lab with Rebecca Evans of Pell Ensemble, and Flowfal now drives her motion-activated dance scores. #### Flowfal — Simon East (Curvor / Flowfal) (2023) - Video: https://www.youtube.com/watch?v=HZXXwKR3wF4 - Interaction: Hands & Body, Voice & Sound, Performance - Platform & tech: Wearable, Phone, smartwatch, Ableton Live, Max/MSP, low-latency Wi-Fi - Idea: Any phone or watch you already wear becomes a gesture instrument. - What it is: At the Sónar+D exhibition in Barcelona, a visitor dances with a smartwatch on her wrist and her arm movements bend and trigger an Ableton sound in real time. - Technique: An app streams the device's orientation and acceleration over low-latency Wi-Fi to plug-ins in Ableton Live or Max, where each movement is mapped to parameters, MIDI notes or controllers. - Try it: Use a phone sensor-to-OSC app to send tilt data to a browser synth and let students dance a filter sweep. Twist: two phones, one controlling pitch and one controlling volume, so a duet is needed to play a melody. #### Audience-Controlled Tracks at Totally Wired EMOM — Simon East (Curvor / Flowfal) (2024) - Video: https://www.youtube.com/watch?v=XTYXh4qTujE - Interaction: Shared & Social, Voice & Sound, Hands & Body - Platform & tech: Phone, Wearable, Flowfal, Ableton Live, smartphones - Idea: The crowd, not the performer, holds the controls. - What it is: At an electronic music open mic in Chorlton, musician Thunkist hands Flowfal devices to people in the pub so the audience's movements shape the track being played. - Technique: Several phones or watches running Flowfal stream motion to Ableton Live, where each device is mapped to a different layer or effect of the track. - Try it: Give four students phones that each control one layer of a web loop by shaking them, and let the room build the track together. Twist: a layer only plays while its phone is being passed from hand to hand. #### Pell Ensemble x handmadesound x Thunkist: Magnitogorsk R&D — Simon East (Curvor / Flowfal), Pell Ensemble (Rebecca Evans) (2026) - Video: https://www.youtube.com/watch?v=an9eBq1ZzA0 - Interaction: Voice & Sound, Performance, Hands & Body - Platform & tech: Wearable, smartwatch, Ableton Live, surround sound, 360 video - Idea: The dancer's arms place each sound in the room around the audience. - What it is: Proof of concept for a spatial-sound, 360° film dance work about the industrial history of Magnitogorsk: with a smartwatch on each wrist, dancer Rebecca Evans triggers the music and moves it around a surround-sound space. - Technique: Flowfal watch data is mapped in Ableton Live to clip triggers and to the pan and position of surround channels, and the result is rendered binaurally for the 360 video. - Try it: Set up four speakers in the room corners and map a phone's compass heading to which speaker a loop plays from, so a dancer can throw sound around the class. Twist: add a second dancer whose sound always plays from the opposite corner. #### Smartwatch Theremin — Simon East (Curvor / Flowfal) (2026) - Video: https://www.youtube.com/watch?v=-j-9hsr9UcI - Interaction: Hands & Body, Voice & Sound - Platform & tech: Wearable, smartwatch, Max/MSP, Flowfal - Idea: Play an invisible theremin with two wristwatches. - What it is: Two ordinary Android watches approximate a theremin inside a Max patch: raising the right arm controls volume, rolling the left hand sets pitch and lifting the left arm adds vibrato. - Technique: Flowfal streams roll and pitch angles from each watch into a standalone Max patch, where arm angle maps to gain and wrist roll maps to oscillator frequency. - Try it: Strap two phones to students' forearms and build a Web Audio theremin where one arm's angle is pitch and the other's is volume. Twist: snap the pitch to a pentatonic scale so the class can play a round together. ### Thomas Van Bouwel *Solo indie developer (Cubism, Laser Dance); former architect* Belgian architect-turned-solo game developer behind the hand-tracked puzzle game Cubism and the mixed-reality game Laser Dance. #### Cubism (Mixed Reality mode) — Thomas Van Bouwel (2023) - Video: https://www.youtube.com/watch?v=BrqmjOLzXNo - Interaction: Hands & Body, Play - Platform & tech: Headset, Meta Quest 3, hand tracking, passthrough - Idea: Assemble a floating colored block puzzle with your bare hands above a real table. - What it is: The hand-tracked block puzzle Cubism rebuilt for passthrough: colored polycube pieces float above your real table and you assemble them with your bare hands. - Technique: Hand-tracking pinch grabs drive rigidbody pieces that snap to a voxel grid when released near a valid slot, while passthrough keeps the real table visible as the work surface. - Try it: Make a 3×3×3 grid and drag three polycube pieces into it, snapping each to the nearest grid point when it gets close. Twist: once the cube is complete, it turns into a character that moves. #### Laser Dance — Thomas Van Bouwel (2024) - Video: https://www.youtube.com/watch?v=Dx2JZ_9_lbc - Interaction: Spatial Mapping, Hands & Body, Play - Platform & tech: Headset, Meta Quest 3, Scene API, Unity - Idea: Turn the real walls and furniture of your living room into a laser maze you dodge through with your whole body. - What it is: A Quest 3 mixed-reality game that scans your living room and strings laser beams between your real walls and furniture; you dance, dodge and crawl through the beams to reach buttons. - Technique: The Scene API supplies labeled room geometry (walls, floor, furniture boxes), and laser segments are generated between sampled points on those surfaces, with body collision checked against the headset and hand poses. - Try it: Get wall and floor planes with AR Foundation plane detection, generate a few random laser lines between two walls, and make the phone vibrate and fail when it touches a laser. Twist: make the lasers move to the beat of the music. ### Tin & Ed (Tin Nguyen & Edward Cutting) *Artist duo (AR, installation, interactive media)* Australian duo Tin Nguyen and Edward Cutting, now based in New York and alumni of the New Museum's NEW INC. Their AR apps, inflatable sculptures and body-tracked screens imagine hybrid life forms and deep time, shown at MoMA, the Getty, the Barbican and ArtScience Museum. #### Kaleidoscopic Home AR — Tin & Ed (Tin Nguyen & Edward Cutting) (2021) - Video: https://vimeo.com/747340367 - Interaction: Spatial Mapping, Play, Hands & Body - Platform & tech: Phone, ARKit, object recognition, generative forms - Idea: Grow playground sculptures out of your furniture to get you moving. - What it is: An Everyday Experiment for SPACE10 and IKEA: an AR app turns your home into a playground where colourful sculptural forms grow out of tables, chairs, floors and walls to nudge you into moving. - Technique: Object recognition and plane detection find furniture and surfaces, and generative forms are anchored to them so they appear to emerge from the real room. - Try it: Use AR Foundation or Lens Studio plane detection to spawn a twisting tube from every detected surface and ask a classmate to follow it with their body. Twist: the tube grows toward the person who has been sitting longest. #### Kernel — Tin & Ed (Tin Nguyen & Edward Cutting) (2021) - Video: https://vimeo.com/908430724 - Interaction: Tangible Objects, Location & City, Perception & Effects - Platform & tech: Phone, object tracking, inflatable sculpture - Idea: A physical sculpture that sprouts virtual life when you look through a phone. - What it is: A large inflatable sculpture shaped like a seed pod stands in a garden; through an AR app it digitally germinates, sprouting new virtual life forms from the tips of its buds. - Technique: Generative patterns at the tip of each bud act as AR tracking features on the physical sculpture, so virtual growths are registered precisely to the inflatable. - Try it: Make a small paper or clay sculpture with printed patterns, scan it as an object target in Vuforia or Lens Studio, and let virtual sprouts grow from it. Twist: the sprouts are different for each viewer's birthday month. #### Life Forces — Tin & Ed (Tin Nguyen & Edward Cutting) (2021) - Video: https://vimeo.com/906183149 - Interaction: Hands & Body, Perception & Effects, Shared & Social - Platform & tech: Desktop, iPad Pro, body tracking, real-time 3D - Idea: Your body becomes pollen, fungus or rock in a living diorama. - What it is: Three LED screens show endlessly scrolling landscapes of moss, fungi and flowers. Viewers' bodies are tracked so they can roleplay as pollen and spores, feed as slime mould or sway as rock formations. - Technique: An iPad Pro under each screen runs live body tracking, and a real-time 3D scene built from scanned plants responds to the viewer's pose and movement. - Try it: Use ARKit body tracking or MediaPipe pose to map a student's arms to swaying stems in a p5.js or Unity garden. Twist: each classmate controls a different species and the garden only thrives when all move together. #### Deep Field — Tin & Ed (Tin Nguyen & Edward Cutting) (2023) - Video: https://vimeo.com/925554115 - Interaction: Location & City, Voice & Sound, Perception & Effects - Platform & tech: Phone, ARKit, procedural modelling, spatial audio - Idea: A phone reveals a speculative garden of plants that do not exist yet. - What it is: An interactive AR artwork for museum gardens and public spaces: through a phone, imagined plants that glow, shapeshift or sprout teeth grow around visitors, accompanied by recordings of animals and insects that are now gone. - Technique: Procedurally generated and hand-sculpted 3D plants are placed in the surroundings with markerless phone AR, while a soundscape from The Listening Planet's field recordings plays around them. - Try it: Design one imaginary plant from two real species, model it simply in Reality Composer, Lens Studio or WebXR, and place it in the school garden with a matching sound. Twist: the plant only blooms when someone stays still next to it. ### Tomás García *Animator, character designer and AI artist* Argentine animator (ex-PepperMelon, MTV redesign) whose June 2017 ARKit tests of a moon landing in his kitchen and a Falcon 9 landing in a pool went viral in the first weeks of ARKit. #### ARKit + Unity + Motion Capture: robot dancing — Tomás García (2017) - Video: https://www.youtube.com/watch?v=1G2YbQuQHps - Interaction: Performance, Hands & Body - Platform & tech: Phone, ARKit, Unity, motion capture - Idea: Motion-capture data makes a robot dance in your living room. - What it is: A robot character performs motion-captured dance moves in his living room, showing how real mocap data makes an AR character feel alive. - Technique: A humanoid rig driven by recorded motion-capture clips is placed on an ARKit floor plane so the dance happens at human scale. - Try it: Download a dance clip from Mixamo, bind it to a character, and place the character in your classroom in AR. Twist: use the beat picked up by the phone microphone to control playback speed. #### Falcon 9 landing in a swimming pool — Tomás García (2017) - Video: https://www.youtube.com/watch?v=NodGjd3C0SQ - Interaction: Spatial Mapping, Location & City - Platform & tech: Phone, ARKit, Unity - Idea: A Falcon 9 rocket lands on a drone ship floating in your own swimming pool. - What it is: A SpaceX Falcon 9 booster descends with fire and smoke and lands on a drone ship floating in his real swimming pool. - Technique: An ARKit plane on the water surface anchors a Unity-animated rocket, drone ship and particle exhaust at a deliberate miniature scale. - Try it: Find a real surface on campus (a pond, a playground, some steps) and restage a piece of science news on it as a miniature AR reenactment. Twist: viewers can launch it with a "countdown" hand gesture. #### Moon Landing @ Living Room — Tomás García (2017) - Video: https://www.youtube.com/watch?v=HBEQpBdoaj0 - Interaction: Spatial Mapping - Platform & tech: Phone, ARKit, Unity - Idea: A moon landing covers the living room floor as a real dog walks past the astronauts. - What it is: A larger version of the moon-landing test fills the living room floor, with a real dog wandering past the out-of-scale astronauts. - Technique: The same ARKit floor anchor is used with a scaled-up scene so real objects and pets walk through the virtual set. - Try it: Make the same AR scene in three sizes (palm, tabletop, room) and compare how each feels. Twist: have the scene make way for the positions of real pets or classmates. #### Moon Landing @ my Kitchen — Tomás García (2017) - Video: https://www.youtube.com/watch?v=6OV2mBbNtVk - Interaction: Spatial Mapping, Information & UI - Platform & tech: Phone, ARKit, Unity - Idea: The Apollo moon landing happens on a kitchen counter. - What it is: The Apollo lunar module lands on his kitchen counter and astronauts step out, captured in real time with one of the first ARKit builds. - Technique: ARKit horizontal plane detection anchors an animated Unity scene to the countertop and keeps it stable as the phone moves. - Try it: Turn a historical event into a tabletop-sized AR animation (Reality Composer is enough) and place it on a desk. Twist: show narration subtitles when a viewer comes close. ### Wayne Piekarski *Creator of the Tinmith outdoor AR backpack and co-creator of ARQuake (Wearable Computer Lab, University of South Australia); later developer advocate at Google* With Bruce Thomas at the University of South Australia he built Tinmith (1998-2006), a backpack computer with a head-mounted display and pinch gloves, used for ARQuake and for modelling buildings outdoors in AR. #### ARQuake — Wayne Piekarski (2000) - Video: https://www.youtube.com/watch?v=No7QF0MwSjg - Interaction: Play, Location & City - Platform & tech: Headset, Wearable, Tinmith, Quake engine, GPS - Idea: Turn a real campus into a first-person shooter level. - What it is: A player with a backpack computer, head-mounted display and toy gun walks across the university campus shooting Quake monsters that appear among the real buildings. - Technique: Modifies the Quake engine to render only monsters and items in see-through view, driven by GPS, a head tracker and a campus model for occlusion. - Try it: Design a location-based phone AR game where virtual creatures spawn at three GPS points on campus and players must reach them to catch them. Twist: let creatures hide behind real walls using a simple occlusion box. #### Tinmith-Metro Outdoor AR Modelling — Wayne Piekarski (2001) - Video: https://www.youtube.com/watch?v=7YcycATEs0k - Interaction: Drawing & Making, Spatial Mapping, Location & City - Platform & tech: Headset, Wearable, Tinmith, pinch gloves, GPS - Idea: Model architecture by standing in front of it and grabbing its shape with your hands. - What it is: Standing outside a building, the user wears pinch gloves and 'captures' its walls by pointing and pinching, building a 3D model of the real building on the spot. - Technique: Uses working-planes projected from the tracked head and gloves to intersect infinite planes into solid building geometry, all on a wearable backpack computer. - Try it: Use a phone AR measuring or plane-detection app to trace the walls of a room and export a floor plan, then compare with a tape-measure drawing. Twist: model a building outside using only photos taken from the pavement. #### Tinmith Glove User Interface — Wayne Piekarski (2002) - Video: https://www.youtube.com/watch?v=W2MVc-ajD4U - Interaction: Hands & Body, Information & UI - Platform & tech: Headset, Wearable, Tinmith, pinch gloves, fiducial markers - Idea: Your fingers become the menu when there is no mouse outdoors. - What it is: Menus are mapped onto the fingers of pinch gloves: the user presses thumb to finger to pick commands while walking outdoors in AR, and points with tracked thumbs to place objects. - Technique: Uses conductive pinch gloves for discrete menu selection and fiducial markers on the thumbs for 3D cursor tracking by the head-mounted camera. - Try it: Write four commands on your fingertips, and build a phone app where a partner triggers the matching AR action when you tap thumb to finger. Twist: design a two-level menu using both hands. #### Hand of God — Wayne Piekarski (2006) - Video: https://www.youtube.com/watch?v=tVlRwCtWKNM - Interaction: Shared & Social, Location & City, Hands & Body - Platform & tech: Headset, Desktop, Tinmith, tabletop capture, 3D video - Idea: Remote guidance delivered as a giant hand reaching down from the sky. - What it is: An indoor commander points at a map on a tabletop, and the outdoor AR user sees a giant 3D hand descend from the sky to show where to go or what to pick up. - Technique: Captures the indoor user's hand and props on the tabletop as 3D video and renders them geolocated and scaled up in the outdoor user's Tinmith view. - Try it: Have an indoor student point at a printed campus map on a video call, then show a giant hand image anchored above the pointed spot in the outdoor partner's phone AR. Twist: let the indoor student drop small toy props that appear outdoors. ### Yuri Suzuki *Sound artist and designer; Pentagram partner from 2018* Japanese sound artist and designer based in London, who joined Pentagram as a partner in 2018. His horn sculptures, AI choirs and playful instruments invite the public to speak, sing and listen in shared space, and he made Sound Bubbles for SPACE10's Everyday Experiments. #### Garden of Russolo — Yuri Suzuki (2013) - Video: https://vimeo.com/75251985 - Interaction: Voice & Sound, Tangible Objects, Location & City - Platform & tech: Projection, microphones, audio filters, horns - Idea: Speak into a box and hear your voice turned into noise music. - What it is: Phonograph-like wooden boxes in the V&A's garden let visitors speak or sing into them and hear their own voice returned as a muffled, twisted atmospheric noise, a tribute to Futurist composer Luigi Russolo. - Technique: Each box captures the voice with a microphone and re-plays it through filters and amplification that distort and reinterpret the input, following Suzuki's earlier white noise machines. - Try it: Build a web page that records one second of voice and loops it back through a random chain of Web Audio filters, then place phones in cardboard horns around the school. Twist: each horn replays the voice of the previous visitor. #### AR Music Kit — Yuri Suzuki (2016) - Video: https://www.youtube.com/watch?v=6uUSVdQcafE - Interaction: Tangible Objects, Voice & Sound - Platform & tech: Phone, marker tracking, iOS, audio samples - Idea: Play music by showing paper to a phone camera. - What it is: A few printed paper cards turn a phone into a DIY instrument: when the camera sees a patterned code, the device plays the matching note, so you compose by arranging and moving paper. - Technique: Printed fiducial markers are recognised by the phone's camera and each code is mapped to a sound sample. - Try it: Print ArUco or image markers, recognise them in a WebAR page, and play a different sample for each marker seen. Twist: the order of markers from left to right becomes a melody. #### The Welcome Chorus — Yuri Suzuki (2019) - Video: https://www.youtube.com/watch?v=pB1TBwACzsE - Interaction: Voice & Sound, Shared & Social, Location & City - Platform & tech: Projection, AI text generation, speech synthesis, horn speakers - Idea: A public choir of horns that learns new lyrics from passers-by. - What it is: Twelve horns, one for each district of Kent, sing lyrics generated live by an AI trained on words contributed by local people; visitors speak or sing into a conductor sculpture and their words enter the chorus. - Technique: A site-specific text generator trained on community lyrics feeds a singing synthesis system routed to twelve horn speakers, and a microphone in the conductor sculpture adds visitors' input to the model. - Try it: Collect one sentence about the school from every student, generate new lines with a small text model or a random remix script, and play them from phones hidden in paper horns around the room. Twist: each horn only sings words from one class. #### Sonic Bloom — Yuri Suzuki (2021) - Video: https://vimeo.com/610583719 - Interaction: Voice & Sound, Shared & Social, Location & City - Platform & tech: Projection, acoustic horns, public sculpture - Idea: Horns that let strangers talk to each other across a street. - What it is: A cluster of colourful horn-shaped structures in a London street captures and carries voices, so strangers can talk and listen to each other across the sculpture while city and nature sounds mix in. - Technique: Acoustic horns and pipes physically channel sound between listening points, likely combined with amplified ambient recordings. - Try it: Build two cardboard or plastic tube horns connected by a garden hose so two students can whisper across a room, then map the best listening spots. Twist: add a third horn that only plays birdsong. ### fuqunaga *Game and interactive-art programmer* Japanese interactive-art programmer who open-sources compute-shader tools for Unity such as GpuTrail, VatBaker and RosettaUI. #### Conway's Game of Life on the GPU — fuqunaga (2017) - Video: https://www.youtube.com/watch?v=Oxo22vWgzKc - Source code: https://github.com/fuqunaga/GpuLifeGame - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: Four rules on a grid produce endless emergent life. - What it is: A huge Game of Life grid evolves in real time, with gliders and patterns rippling across the screen. - Technique: Each cell's state is updated in parallel in a compute shader from its eight neighbours and rendered as a texture. - Try it: Map a Game of Life grid onto a real wall in AR and seed live cells where the camera sees dark pixels. Twist: the simulation speed follows the room's noise level. #### GpuTrail: 100,000 Trails — fuqunaga (2017) - Video: https://www.youtube.com/watch?v=sS2MYj6LceY - Source code: https://github.com/fuqunaga/GpuTrail - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: Trails, usually expensive, can be fully GPU-driven at massive scale. - What it is: One hundred thousand glowing trails stream through space at once, each following its own particle. - Technique: Trail node positions are stored in compute buffers and updated per frame in a compute shader, then drawn as camera-facing strips with frustum culling and LOD. - Try it: Attach GPU trails to tracked hands or the phone in AR so movement leaves a long trace of light around the room. Twist: trails of different students attract each other. #### PortalGate — fuqunaga (2018) - Video: https://www.youtube.com/watch?v=jzud9m-NgnA - Source code: https://github.com/fuqunaga/PortalGate - Interaction: Portals & Worlds, Play - Platform & tech: Desktop, Unity, Render textures - Idea: Two linked holes in walls rewrite how space connects. - What it is: A functional clone of Valve's Portal: the player shoots two portals on walls and walks or drops objects through them seamlessly. - Technique: Each portal renders a linked camera view into a texture on the opening, and objects crossing the plane are cloned and teleported with their velocity rotated. - Try it: Place two portals on real walls in AR so a ball rolled into one comes out of the other. Twist: the second portal is in a different classmate's phone. #### VFXGraphSandbox: Runtime Point Cache — fuqunaga (2019) - Video: https://x.com/fuqunaga/status/1209023053016924161 - Source code: https://github.com/fuqunaga/VFXGraphSandbox - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph - Idea: Generate the point cache live instead of baking it, so any animated mesh can become particles. - What it is: A character model is turned into particles at runtime: VFX Graph samples its surface every frame, so the particle body follows the animation. - Technique: A script samples points on the skinned mesh each frame and writes them into textures bound to VFX Graph as a runtime point cache; a second hack reads vertex animation textures (VAT). - Try it: Sample the AR body-tracking skeleton mesh at runtime and emit VFX Graph particles from it, so a classmate dissolves into particles on the phone. Twist: the particles stay behind where the person was a second ago. ### Étienne-Jules Marey *Physiologist and inventor of chronophotography (1830–1904)* French physiologist who built the chronophotographic gun, fixed-plate chronophotography, bronze and plaster zoetrope sculptures of birds in flight, and smoke machines that made air currents visible. #### Chronophotography on a Fixed Plate — Étienne-Jules Marey (1882) - Video: https://www.youtube.com/watch?v=Q02SAf_eUmU - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, chronophotographic camera, rotating shutter - Idea: Reduce the body to a few bright lines and a whole movement fits in one image: stick-figure trails are often clearer than full avatars in AR. - What it is: Athletes in black suits with white lines and dots run, jump and fence in front of a dark shed; a rotating shutter records many phases on one plate, so the body becomes a fan of overlapping lines. - Technique: A slotted rotating disc exposes one photographic plate at regular intervals while the subject, dressed to reflect only along limbs, moves against black. - Try it: Use a phone body-tracking API (ARKit body tracking or MediaPipe in WebXR) to draw only the limb segments of a moving person and keep every 5th frame on screen as a fading echo. Twist: let the viewer freeze the whole fan in space and walk around it. #### Flight of a Gull (zoetrope sculptures) — Étienne-Jules Marey (1887) - Video: https://www.youtube.com/watch?v=0wXNrersYT0 - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, plaster, bronze, zoetrope - Idea: Motion can be stored as a ring of physical keyframes: an AR designer can animate by swapping solid sculptures rather than deforming one mesh. - What it is: Plaster and bronze gulls modelled from chronophotographs of a single wingbeat sit in a ring inside a zoetrope; when it spins behind slits, the sculpted bird flies in 3D. - Technique: Each phase of the wingbeat is sculpted as a separate object and viewed through a spinning slotted drum, which flashes each object in turn. - Try it: Model or photogrammetry-scan 8 poses of a hand opening and closing, place them in a ring in AR, and show only one at a time at 12 fps as the ring rotates. Twist: slow the rotation until viewers see the individual sculptures again. #### Chronophotographic Films — Étienne-Jules Marey (1890) - Video: https://www.youtube.com/watch?v=BIKwns3y2_w - Interaction: Perception & Effects, Hands & Body - Platform & tech: Desktop, chronophotographic camera, film strip - Idea: Slow motion reveals rules the eye cannot see, like a cat righting itself mid-air: AR replays at a fraction of real speed can teach physical skills. - What it is: Short films from Marey's Station Physiologique: horses, cats falling and turning in the air, fencers, athletes and birds, recorded on moving film strips with his chronophotographic camera. - Technique: A camera with an intermittent film transport records dozens of frames per second on paper or celluloid strips. - Try it: Capture a friend throwing a ball with the phone's slow-motion mode and place the clip life-size in AR where it happened, playing at 1/8 speed. Twist: add a draggable timeline ring on the floor to scrub the throw by walking. #### Smoke Machine (Mouvements de l'air) — Étienne-Jules Marey (1901) - Video: https://www.youtube.com/watch?v=95XOEN4f8EE - Interaction: Perception & Effects, Spatial Mapping, Information & UI - Platform & tech: Desktop, smoke, wind tunnel, camera - Idea: Invisible fields become readable when you seed them with visible tracers: AR can show wind, WiFi or sound as streams of particles bending around real objects. - What it is: Parallel threads of smoke flow through a glass chamber and bend around obstacles of different shapes, making the invisible air current visible and photographable. - Technique: Evenly spaced smoke streams are fed into a laminar air flow in front of a black background and photographed as they deflect around models. - Try it: Scan a desk with LiDAR and run a simple 2D flow field in Unity or three.js so particle streamlines bend around the scanned objects. Twist: let the user blow into the microphone to change the flow speed. ### 8th Wall (Erik Murphy-Chutorian) *WebAR platform, acquired by Niantic in 2022* 8th Wall, founded by Erik Murphy-Chutorian in 2016, made markerless AR run in the mobile browser without an app and powered thousands of WebAR campaigns. #### Microsoft Mixed Reality Capture holograms in WebAR — 8th Wall (Erik Murphy-Chutorian) (2020) - Video: https://www.youtube.com/watch?v=LH_eViGIohQ - Interaction: Performance, Spatial Mapping - Platform & tech: Web, Phone, 8th Wall, volumetric video - Idea: Life-size holographic video of real people appears right in front of you in the phone browser. - What it is: Volumetric video of real performers captured in Microsoft's Mixed Reality Capture Studios plays as life-size holograms in the phone browser through 8th Wall. - Technique: Volumetric video (a sequence of textured meshes) is streamed and played in sync with audio, and anchored at life size on a detected ground plane in the browser. - Try it: Record a video of a classmate with phone background removal and place it on the AR floor as a camera-facing flat hologram person. Twist: make the hologram turn its head to look at you as you approach. #### Lightship VPS for Web — 8th Wall (Erik Murphy-Chutorian), Niantic (John Hanke) (2022) - Video: https://www.youtube.com/watch?v=laK-QxFLELw - Interaction: Location & City, Spatial Mapping - Platform & tech: Web, Phone, 8th Wall, Niantic Lightship VPS, WebAR - Idea: Anchor AR content precisely to real places right in the browser, with no app to download. - What it is: 8th Wall and Niantic brought centimetre-level visual positioning to the mobile browser, so WebAR content can be anchored to specific real-world locations without an app. - Technique: Browser camera frames are matched against a prebuilt map of a location to return a 6DoF pose, letting WebAR content be anchored to a specific place. - Try it: Simulate location-based positioning in web AR with an image marker: after scanning a poster by the door, content appears fixed on the door frame. Twist: each time it opens, show a sentence left by the previous visitor. #### 8th Wall Sizzle Reel 2023 — 8th Wall (Erik Murphy-Chutorian) (2023) - Video: https://www.youtube.com/watch?v=jHDtBc6fG44 - Interaction: Portals & Worlds, Face, Spatial Mapping - Platform & tech: Web, Phone, 8th Wall, WebAR - Idea: A reel of web AR pieces that open from a link, with no app required. - What it is: A showcase of WebAR projects built by agencies and creators on 8th Wall: face effects, portals, world tracking and image targets that open from a link in the phone browser. - Technique: Browser-based SLAM, face tracking and image targets run in a mobile web page, so each experience opens from a link or QR code without an app install. - Try it: Choose a WebAR template (face, image or world tracking), build a small piece in two hours that opens from a QR code, and share it on the spot. Twist: the piece must relate to the city where the class takes place. ### Agnes Denes *Conceptual and environmental artist* Hungarian-born American pioneer of ecological art who planted two acres of wheat on a Manhattan landfill in 1982 and a mountain of 11,000 trees in Finland. #### Wheatfield – A Confrontation — Agnes Denes (1982) - Video: https://www.youtube.com/watch?v=zxYjGy3csTU - Interaction: Location & City, Information & UI, Perception & Effects - Platform & tech: Desktop, wheat, landfill, Lower Manhattan - Idea: A living field in the wrong place is a sharp confrontation of values, a template for AR that plants nature where it 'should not' be. - What it is: On a landfill two blocks from Wall Street, Denes planted and harvested two acres of golden wheat against the backdrop of the World Trade Center towers. - Technique: About 200 truckloads of soil were brought onto the rubble; wheat was sown by hand, irrigated and harvested four months later. - Try it: Cover a car park or a financial-district plaza with a waving AR wheat field that responds to device tilt as wind. Twist: the field's height tracks a live commodity price for wheat. #### Tree Mountain – A Living Time Capsule — Agnes Denes (1996) - Video: https://www.youtube.com/watch?v=nmVFGwNeWcc - Interaction: Location & City, Shared & Social, Information & UI - Platform & tech: Desktop, 11,000 trees, gravel pit, mathematical planting pattern - Idea: One person, one tree, one position in a pattern: a shared artwork that lasts centuries, a model for persistent multi-user AR. - What it is: In Ylöjärvi, Finland, 11,000 fir trees were planted by 11,000 people on a man-made mountain in a pattern derived from the golden ratio; each planter was given custodianship of a tree for 400 years. - Technique: Tree positions follow a spiral pattern inspired by pineapple and sunflower phyllotaxis, laid out on a reshaped gravel pit. - Try it: Lay out a Fibonacci spiral of AR saplings on a lawn and let each classmate claim one with their name, stored in a shared database. Twist: saplings grow only on days their owner visits. #### The Living Pyramid — Agnes Denes (2025) - Video: https://www.youtube.com/watch?v=gTMUGHzXK-o - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, planted pyramid, desert plants, terraced earth - Idea: A monumental shape made of living plants changes with the seasons, showing how an AR form can have a growing skin. - What it is: At Desert X 2025 in the Coachella Valley, a terraced earth pyramid is planted with native desert flowers and grasses, a new version of Denes's living pyramids first grown in New York and Luxembourg. - Technique: An earth pyramid is built in stepped terraces, each planted with species chosen to bloom at different times. - Try it: Place an AR pyramid of terraces on a plaza and let its planting change by month using a list of local native plants. Twist: visitors water a terrace with a tap gesture, and it blooms early. ### Alex Chinneck *Sculptor* British artist who makes buildings appear to melt, slide, float or unzip, using real bricks and engineering so the illusion holds in daylight. #### From the knees of my nose to the belly of my toes — Alex Chinneck (2013) - Video: https://www.youtube.com/watch?v=v4Uxx5F_RI8 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, brick facade, structural engineering - Idea: Deforming a real facade with real materials is exactly what AR 'melt' filters do digitally, and it shows how much credibility depends on material detail. - What it is: The whole brick facade of a derelict house in Margate appears to slide off the building and slump into the front garden, while the house stays intact behind it. - Technique: A new brick skin was built on a hidden steel frame in a sliding curve, with real windows and doors bent to follow it. - Try it: Use a facade-tracking or scanned-building AR setup to make a real house front slide down into the garden with a physics animation. Twist: let the user push it back up by swiping. #### Take my lightning but don't steal my thunder — Alex Chinneck (2014) - Video: https://www.youtube.com/watch?v=9C-HTHAUfk0 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, hidden steel support, stone replica building - Idea: A levitating building is a classic AR demo made physical; the hidden support shows how illusions depend on concealed anchors. - What it is: In Covent Garden Piazza, the top half of a classical stone building appears to float in the air, cut cleanly from its base, with no visible support. - Technique: The upper section is built on a single hidden steel support concealed within the gap and the building's structure. - Try it: Scan a small building or kiosk and make its top half lift off and float in AR, leaving a clean cut. Twist: the gap widens as more people stand underneath. #### A pound of flesh for 50p — Alex Chinneck (2016) - Video: https://www.youtube.com/watch?v=5X144p9qA1w - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, wax bricks, heating, melting house - Idea: A building that changes over weeks makes time the material, a model for AR structures that decay slowly across visits. - What it is: A house built from 7,000 wax bricks slowly melts over a month in London, its walls and roof sagging until the structure collapses. - Technique: Bricks cast in wax are laid like normal masonry and heated from inside so the house deforms gradually. - Try it: Build an AR house on campus that melts a little each day using a vertex-displacement shader driven by the day count. Twist: speed up the melting on hotter days using live temperature. ### Andres Amador *Sand artist* American artist who rakes vast geometric and organic patterns into beaches at low tide, knowing the sea will erase them within hours. #### Mandala I — Andres Amador (2012) - Video: https://www.youtube.com/watch?v=AYDsFNkG1vE - Interaction: Location & City, Drawing & Making - Platform & tech: Desktop, rake, beach, radial symmetry - Idea: Radial symmetry lets a human-scale gesture add up to a readable image from above, a principle for AR ground patterns. - What it is: A time-lapse shows Amador raking a large radially symmetric mandala into the sand, seen from above as the pattern grows ring by ring. - Technique: He marks a centre, lays out rings with a rope and repeats the same motif around each ring. - Try it: Build an AR mandala tool: anything the user draws on the ground is repeated eight times around a centre point. Twist: the symmetry count increases with each new ring. #### Earthscapes — Andres Amador (2014) - Video: https://www.youtube.com/watch?v=T_tIG5mo1DM - Interaction: Location & City, Drawing & Making, Perception & Effects - Platform & tech: Desktop, rake, beach, low tide - Idea: Working inside a fixed window of time (between tides) is a design constraint that AR events can borrow. - What it is: At low tide on Californian beaches, Amador rakes huge fractal and organic patterns into the wet sand in a few hours; the rising tide erases them. - Technique: He shades areas by raking the top layer of wet sand, creating contrast between smooth and textured surfaces, following a rough plan. - Try it: Create an AR drawing tool that works only during a set daily window (for example, the next low tide from a tide API), and erases everything when it closes. Twist: draw with texture instead of colour. #### Guided group beach mandala — Andres Amador (2016) - Video: https://www.youtube.com/watch?v=tVxKGuW9U4E - Interaction: Location & City, Shared & Social, Drawing & Making - Platform & tech: Desktop, rakes, group workshop, beach - Idea: Splitting a big image into simple local tasks lets strangers co-create, a model for multi-user AR drawing sessions. - What it is: Amador leads a group of participants who rake a large mandala together on the beach, each person responsible for part of the pattern. - Technique: Each participant gets a sector and a simple instruction; the radial layout keeps everyone's parts aligned. - Try it: Run a shared AR drawing session where each student is assigned one sector of a ground mandala and can only draw inside it. Twist: sectors rotate every five minutes so everyone finishes someone else's work. ### Andrew Hart *iOS developer; creator of ARKit+CoreLocation; founder of Dent Reality / Hyper* iOS developer who open-sourced ARKit+CoreLocation in 2017, pioneering phone AR navigation, then founded Dent Reality (now Hyper) for indoor AR wayfinding. #### ARKit + CoreLocation — Andrew Hart (2017) - Video: https://x.com/AndrewProjDent/status/886916872683343872 - Interaction: Location & City, Information & UI - Platform & tech: Phone, ARKit, CoreLocation - Idea: GPS landmark labels pinned firmly onto the real street view. - What it is: Floating labels pinned to real-world landmarks by GPS stay locked in place as he walks through London, combining ARKit tracking with CoreLocation. - Technique: GPS coordinates of landmarks are converted to bearings and distances from the device's location and heading, then placed as labels in ARKit's gravity-and-heading aligned world frame. - Try it: Use AR plus GPS to place floating text labels at the coordinates of several campus buildings, so the labels stay fixed toward each building as you walk. Twist: have each label show a personal memory of yours from inside that building. #### ARKit + CoreLocation Part 2: AR Turn-by-Turn Navigation — Andrew Hart (2017) - Video: https://x.com/AndrewProjDent/status/888380207962443777 - Interaction: Location & City, Information & UI - Platform & tech: Phone, ARKit, CoreLocation - Idea: Navigation arrows drawn straight onto the real street for the first time. - What it is: AR arrows and route lines are drawn on the actual street, leading the walker turn by turn to the destination. - Technique: Route waypoints from a directions API are converted from GPS to local AR coordinates and drawn as arrows and lines on the ground plane, re-aligned as location updates arrive. - Try it: Get a few points of a walking route from a map API and use AR to draw arrows on the ground that lead to the next intersection. Twist: make the route pass through the quietest places on campus instead of taking the shortest path. #### Dent Reality Indoor AR — Andrew Hart (2018) - Video: https://x.com/AndrewProjDent/status/1023959459838738432 - Interaction: Location & City, Information & UI - Platform & tech: Phone, ARKit - Idea: Use AR to find your way to products inside malls and other indoor spaces. - What it is: An indoor AR wayfinding prototype that guides shoppers to products and locations inside buildings with overlays anchored to the space. - Technique: Indoor positioning likely relies on a pre-built visual map or image anchors for relocalization, with a pathfinding graph producing routes drawn as anchored AR overlays. - Try it: Stick a few image markers along a classroom-building corridor as positioning points, and after scanning show arrows leading to a given classroom. Twist: the destination is a hidden 'treasure'. ### Angela He (zephyo) *Game developer and artist* Artist-programmer who makes small narrative games and shares short Unity visual experiments (rain, anime skies, hair physics) with full source. #### Anime-inspired scene (2D-Unity-Experiments) — Angela He (zephyo) (2020) - Video: https://x.com/zephybite/status/1310626494935257088 - Source code: https://github.com/zephyo/2D-Unity-Experiments - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Desktop, Unity, Shader Graph, RenderTexture - Idea: Composite flat illustration layers with shaders until they feel like a living anime background. - What it is: A layered anime-style landscape with drifting clouds, glowing sky and parallax, composited live in Unity from flat layers. - Technique: Separate cameras render layers to RenderTextures, and Shader Graph materials combine them with scrolling noise, gradient skies and bloom to animate a static illustration. - Try it: Place the layered scene inside an AR portal on a wall so the anime sky moves with parallax as the phone moves. Twist: tint the sky layer with the real sky colour sampled from the camera. #### Dynamic hair and petals (2D-Unity-Experiments) — Angela He (zephyo) (2020) - Video: https://x.com/zephybite/status/1305539495912144897 - Source code: https://github.com/zephyo/2D-Unity-Experiments - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, 2D Animation, Physics 2D - Idea: Give flat 2D art physical secondary motion with a chain of joints. - What it is: A 2D character's long hair sways and bounces as she moves while cherry petals fall around her. - Technique: Hair strands are rigged with Unity's 2D Animation bones and connected with HingeJoint2D chains so physics drives the bones, while petals are a particle system. - Try it: Stand a 2D paper-doll character on an AR plane and let the phone's motion swing her hair through the joint chain. Twist: blow into the microphone to create wind that lifts the hair and petals. #### Rain shader (2D-Unity-Experiments) — Angela He (zephyo) (2020) - Video: https://x.com/zephybite/status/1308833244712927236 - Source code: https://github.com/zephyo/2D-Unity-Experiments - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Shader Graph, RenderTexture - Idea: Make the screen itself feel like wet glass between viewer and world. - What it is: Rain streaks down a window in front of a softly blurred scene; drops gather, slide and leave clear trails in the fogged glass. - Technique: A full-screen shader (based on a Shadertoy raindrop effect) renders the scene to a RenderTexture, then offsets and blurs its UVs with procedural drop normals so the drops refract the image behind them. - Try it: Apply the rain shader to the AR camera feed so the real street looks like it is seen through a rainy window. Twist: wipe a clear patch with a finger swipe that slowly fogs up again. ### Anrick Bregman (Studio ANRK) *AR/VR director; founder of Studio ANRK* Director and official Snap Lens creator whose Studio ANRK makes AR apps, Lenses, VR films and installations, from Bosch's painted creatures to restored Egyptian artefacts. #### Bosch AR — Anrick Bregman (Studio ANRK) (2020) - Video: https://www.youtube.com/watch?v=iznOf-dOIbw - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, ARKit, Unity - Idea: Let 500-year-old painted monsters walk out of the canvas and into your street. - What it is: An iOS app that lets you pick one of five strange creatures from Hieronymus Bosch's Garden of Earthly Delights and release it, animated, into the world around you. - Technique: Characters redesigned in 3D from the painting, rigged and animated, then placed with ARKit plane detection in a Unity app. - Try it: Choose a painting in the public domain, model one small character from it, and animate it walking on a real table in AR. Twist: the character reacts when it reaches the table edge. #### Volumetric people with superpowers (Studio ANRK x Volograms) — Anrick Bregman (Studio ANRK) (2021) - Video: https://www.youtube.com/watch?v=ku8aMvC6bbA - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, Lens Studio, Volograms, Volumetric Video - Idea: Put a real, filmed human into social AR and then make them superhuman. - What it is: Test Snapchat Lenses that drop volumetric captures of real people into the scene and give them superpowers, such as bursts of energy around their bodies. - Technique: Volograms turns single-camera video into volumetric 3D sequences that are imported into Lens Studio and combined with particle VFX. - Try it: Capture a classmate as a short volumetric clip (with a volumetric app or photogrammetry turntable) and place them in AR with one magical effect. Twist: the effect is triggered by the viewer's voice. #### Project Revival — Anrick Bregman (Studio ANRK), Nexus Studios (2024) - Video: https://www.youtube.com/watch?v=nr9tFXYf0Iw - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, Instagram AR, Photogrammetry, LiDAR - Idea: Stand in front of a broken artefact and see it whole again, exactly where it sits. - What it is: At the Egyptian Museum and the National Museum of Egyptian Civilization in Cairo, visitors scan a QR code next to an artefact, such as Thutmose IV's chariot, to see it restored to its original state in Instagram AR. - Technique: Artefacts were LiDAR-scanned and photogrammetry-captured, manually restored in 3D with museum experts, then delivered as Instagram AR effects aligned to the objects. - Try it: Photograph a chipped mug or broken object from 30 angles, build a 3D model, repair it digitally, and overlay the repaired version on the real object. Twist: show the break happening in reverse as an animation. ### Anthony Howe *Kinetic wind sculptor* American sculptor who machines large stainless-steel wind sculptures whose hundreds of cups and blades rotate into shimmering, hypnotic 3D patterns; he designed the kinetic cauldron sculpture for the Rio 2016 Olympic Games. #### In Cloud Light III — Anthony Howe (2012) - Video: https://www.youtube.com/watch?v=JTWqo5H-aig - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, stainless steel, bearings, wind - Idea: Reflection plus rotation make a sculpture borrow colour from the sky; AR materials that reflect the real environment gain the same site-specific shimmer. - What it is: A tall wind sculpture of stacked, spoke-mounted stainless discs rotates in the breeze, each tier turning at its own pace so the form constantly shimmers and reconfigures. - Technique: Each tier is balanced on its own bearing and sized so that the same wind drives it at a different speed, producing phasing patterns (year approximate). - Try it: Place a stack of five reflective rings in a real park with geospatial AR, use environment probes for real reflections, and spin each at a slightly different rate. Twist: link the spin to the live local wind speed. #### Di-Octo — Anthony Howe (2014) - Video: https://www.youtube.com/watch?v=UvgyHLAxgDc - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, stainless steel, bearings, wind - Idea: Counter-rotating layers create depth and moiré; AR designers can use layered opposite motions to make a virtual object feel volumetric and alive. - What it is: Two stacked stainless-steel spheres of spoon-like cups turn in opposite directions in the wind, producing a mesmerising interference pattern that seems to fold in on itself. - Technique: Precision-machined cups catch wind on nested rotating frames; because each frame spins at its own rate, the sculpture never repeats the same configuration. - Try it: Build two nested rings of reflective cups in phone AR spinning in opposite directions at speeds driven by the phone's microphone wind noise. Twist: let the viewer's walking speed set the rotation. #### Rio 2016 Olympic Cauldron — Anthony Howe (2016) - Video: https://www.youtube.com/watch?v=xpZmwQD_-Wk - Interaction: Performance, Perception & Effects - Platform & tech: Desktop, stainless steel, motors, fire - Idea: A small real flame amplified by a moving reflective frame shows how AR can enlarge a modest real event rather than replace it. - What it is: For the Rio 2016 Olympic opening ceremony, a small cauldron flame was framed by a huge kinetic sun of reflective spheres and plates that spun and pulsed around it. - Technique: A motorised rotating structure of polished stainless elements reflects and multiplies the flame's light, turning a small fire into a stadium-scale sun. - Try it: Detect a candle or phone flashlight with the camera, then surround it in AR with a rotating ring of mirrors that reflect its glow. Twist: let the ring grow as more people gather around. ### Asad J. Malik (1RIC / Jadu) *AR director and founder of Jadu* Pakistani-born director whose studio 1RIC made volumetric AR narratives for HoloLens and Magic Leap that premiered at Tribeca and Sundance. He then founded Jadu, a multiplayer AR fighting game with holographic characters. #### Terminal 3 — Asad J. Malik (1RIC / Jadu) (2018) - Video: https://www.youtube.com/watch?v=8B70uYZFO3Y - Interaction: Performance, Voice & Sound, Shared & Social - Platform & tech: Headset, HoloLens, Depthkit - Idea: Make the viewer hold power over a stranger in their own space, and let empathy literally fill in the person. - What it is: In HoloLens, you play a US airport border officer interrogating life-size volumetric Muslim travellers; as their stories emerge, their bodies fill in from wireframe to full hologram, and you decide whether to let them in. - Technique: Interviews captured volumetrically with Depthkit are played back in HoloLens, and the visual resolution of each traveller increases with the dialogue choices made. - Try it: Record a classmate's short story as a LiDAR point cloud and reveal it in phone AR only as the listener asks more questions. Twist: the listener must decide something at the end. #### A Jester's Tale — Asad J. Malik (1RIC / Jadu) (2019) - Video: https://www.youtube.com/watch?v=63N6YuniA6g - Interaction: Performance, Spatial Mapping, Portals & Worlds - Platform & tech: Headset, Magic Leap, volumetric video - Idea: Let a fairy tale invade your room until you are inside it. - What it is: A Magic Leap narrative that premiered at Sundance: you come home cold and tired for a bedtime story, and a psychologically unsettling children's fable spills into the physical room around you. - Technique: Magic Leap spatial mapping anchors volumetric and animated characters to the room's furniture, with interactions that respond to the viewer's position. - Try it: Write a three-scene bedtime story for phone AR where each scene is anchored to a different piece of real furniture. Twist: the last scene happens behind the viewer. #### Jadu — Asad J. Malik (1RIC / Jadu) (2024) - Video: https://www.youtube.com/watch?v=Am17U_jG2Ds - Interaction: Play, Shared & Social, Spatial Mapping - Platform & tech: Phone, ARKit, ARCore, multiplayer - Idea: Bring the fighting-game arena into your real room. - What it is: A multiplayer AR fighting game in which holographic fighters battle on your floor or desk, controlled from the phone, with players able to walk around and film the fight. - Technique: Motion-captured and volumetric characters are placed on detected planes, with networked multiplayer synchronising both fighters' positions in each player's AR view. - Try it: Build a two-player phone AR game where each player's character stands on the same shared table and can push the other off the edge. Twist: the table's real edge is the ring boundary. ### BREAKFAST (Andrew Zolty) *Brooklyn studio for kinetic, data-driven flip-disc and Brixel artworks* Studio founded by Andrew Zolty that invents kinetic media such as flip-discs and rotating Brixels, making walls that mirror passers-by or visualize live wind, weather and ocean data. #### Echo (Flip-Discs) — BREAKFAST (Andrew Zolty) (2016) - Video: https://www.youtube.com/watch?v=3kYKPIh3TCk - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, flip-discs, depth camera, custom software - Idea: A low-resolution mirror with sound is more magical than a perfect one — AR body effects can be coarse if they are tactile and audible. - What it is: A wall of flip-discs clatters to mirror the silhouette of a person moving in front of it, leaving a fading echo of their motion. - Technique: A camera or depth sensor segments the viewer; the silhouette is downsampled to the flip-disc grid, with a decay so past frames linger. - Try it: Build a Lens Studio or AR Foundation effect that turns the camera's person segmentation into a grid of flipping discs with a click sound per flip. Twist: make the discs lag behind so users see their own recent past. #### Brixel Mirror — BREAKFAST (Andrew Zolty) (2019) - Video: https://www.youtube.com/watch?v=Z-5cVpWhp30 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Brixels (rotating mirrored bricks), camera, motors - Idea: A mirror made of real mirrors, each one choosing where to reflect, blends physical and computed reflection — like AR on a real surface. - What it is: Hundreds of mirrored rotating bricks turn to reflect or hide light, forming a moving image of the viewer standing in front. - Technique: Each Brixel is a motor-driven rotating element with mirrored faces; the camera image is mapped to rotation angles per brick. - Try it: Overlay a grid of virtual mirrored tiles on a real wall in AR that turn to reflect the environment map where the viewer's body is. Twist: let each tile hold the reflection of a different moment in time. #### World Skies — BREAKFAST (Andrew Zolty) (2021) - Video: https://www.youtube.com/watch?v=JUkegoJ8nx8 - Interaction: Information & UI, Location & City - Platform & tech: Desktop, flip-discs, live sky data, custom software - Idea: Bringing a faraway sky into a room creates a window, not a screen — AR portals can use live data from real places. - What it is: A flip-disc artwork that animates the colour and motion of skies from cities around the world, updating from live data. - Technique: Likely live weather and sky imagery from multiple cities are processed into patterns rendered on a large flip-disc display. - Try it: Open an AR window on your wall that shows the live sky of a city chosen by the user, using a weather API for cloud cover and time of day. Twist: the window slowly turns to a different city each time someone walks past. ### Beam'Art (Benjamin Petit & Antoine Vanel) *French visual-performance duo for interactive projection mapping* Duo of Benjamin Petit and Antoine Vanel who built their own openFrameworks and OpenNI tools for depth-camera projection: augmented dance floors on building façades and real-time mapping onto moving dancers. #### Augmented Dance Floor (ADF') — Beam'Art (Benjamin Petit & Antoine Vanel) (2011) - Video: https://vimeo.com/26668124 - Interaction: Hands & Body, Projection, Performance - Platform & tech: Projection, openFrameworks, Kinect, OpenNI - Idea: The dancing crowd sees itself as light on the building in front of it. - What it is: At an outdoor music festival in Geneva, the crowd dancing in front of the Palais Eynard is captured in 3D and projected onto the building's façade as glowing point clouds inside a virtual grid room. - Technique: Kinect depth data processed with OpenNI and openFrameworks is rendered as point clouds and 3D optical flow, then projection-mapped onto the façade. - Try it: Capture a group with a depth camera or phone LiDAR and project their point cloud large on a wall behind them. Twist: the point cloud lags a few seconds, so the crowd dances with its past self. #### Interactive Body Projection Mapping (Hypermetrop) — Beam'Art (Benjamin Petit & Antoine Vanel) (2011) - Video: https://vimeo.com/34609484 - Interaction: Hands & Body, Projection, Performance - Platform & tech: Projection, openFrameworks, Kinect, OpenNI - Idea: Clothe moving dancers in live projected light. - What it is: During a residency with hip-hop dancers, audio-reactive textures are projected precisely onto the dancers' moving bodies while they perform. - Technique: A Kinect calibrated to the projector segments the dancers' bodies, and openFrameworks renders audio-reactive graphics masked to their silhouettes. - Try it: Calibrate a depth camera or webcam segmentation to a projector and project a pattern only onto a moving person. Twist: the pattern changes each time the dancer claps. #### Augmented Silhouette — Beam'Art (Benjamin Petit & Antoine Vanel) (2012) - Video: https://vimeo.com/39887510 - Interaction: Hands & Body, Shared & Social, Projection - Platform & tech: Projection, openFrameworks, Kinect, OpenNI - Idea: Your silhouette becomes a light body that throws particles. - What it is: Visitors see their glowing silhouettes on a wall, and colourful particles pour out of their hands and bounce between people as they move. - Technique: OpenNI user tracking extracts silhouettes and hand joints from a Kinect, and openFrameworks renders particles emitted from the hands. - Try it: Use webcam body segmentation and hand tracking to draw a glowing silhouette that emits particles from the hands. Twist: particles thrown by one person stick to anyone else they touch. ### Bilawal Sidhu *Creator and technologist; former Google Maps / AR product manager* Bilawal Sidhu is a creator and former Google product manager for 3D maps and AR who experiments with reality capture, splats and multimodal AI. #### Running AR videos through NeRF and Gaussian splatting — Bilawal Sidhu (2023) - Video: https://x.com/bilawalsidhu/status/1694086197029503466 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, Desktop, AR, NeRF, Gaussian splatting - Idea: Bake AR into reality: once reconstructed, the virtual and the real are the same material. - What it is: Phone videos that already contain AR content are fed to NeRF and Gaussian-splatting pipelines, so the virtual objects become part of a reconstructed 3D scene that can be flown through. - Technique: Screen recordings of AR sessions are treated as multi-view captures; structure-from-motion recovers camera poses and a radiance field or splat model is trained on the composited frames. - Try it: Record a slow orbit around an AR object on a table, train a Gaussian splat from the video with a free app, and view the result in a web viewer. Twist: place a second AR object inside the splat and record again, stacking layers of fake reality. #### An LLM turns a home scan into an editable scene with a portal — Bilawal Sidhu (2026) - Video: https://x.com/bilawalsidhu/status/2096810261730505075 - Interaction: Portals & Worlds, Voice & Sound - Platform & tech: Desktop, Headset, multimodal LLM, 3D scan, inverse graphics - Idea: Multimodal LLMs may have crossed the chasm on inverse graphics: scans become worlds you can direct. - What it is: A 3D scan of his parents' old home is given to a multimodal LLM, which rebuilds it as an editable scene; he then turns down gravity and opens a portal inside it. - Technique: The model interprets the scan (likely images or mesh renders) and emits structured scene code with separate objects and physics, which can be edited by further prompts. - Try it: Photograph a room from four angles and ask a multimodal LLM to output a three.js scene with boxes for each object; load it in WebXR and change one physical law. Twist: the rebuilt room is how you remember it, not how it is. #### IronSight: 4D replays from Ray-Ban Meta clips — Bilawal Sidhu (2026) - Video: https://x.com/bilawalsidhu/status/2074536788853665831 - Interaction: Information & UI, Spatial Mapping - Platform & tech: Wearable, Desktop, Ray-Ban Meta, 4D reconstruction, AR view - Idea: First-person clips become replayable spaces you can re-enter from any angle. - What it is: Clips recorded on Ray-Ban Meta glasses at a shooting range are reconstructed into 4D scenes that can be replayed from any angle, including an AR view that sees targets through walls. - Technique: Likely a feed-forward 4D reconstruction model estimates camera motion, depth and dynamic geometry from the glasses video, which is rendered as a navigable replay. - Try it: Record a 10-second first-person clip of a simple action (throwing a ball), reconstruct it with a free video-to-3D tool and replay it in AR at the same spot from a new angle. Twist: replay it from the ball's point of view. ### Bordos.ArtWorks (László Zsolt Bordos) *Projection mapping artist* Hungarian artist László Zsolt Bordos, a pioneer of European architectural mapping since the late 2000s. His works include a stereoscopic 3D mapping on the Museum of Art and History at Mapping Festival Geneva 2012, Escape at Signal Festival Prague 2015 and generative light pieces such as Lux Formae. #### Stereoscopic 3D mapping, Museum of Art and History (Mapping Festival Geneva) — Bordos.ArtWorks (László Zsolt Bordos) (2012) - Video: https://www.youtube.com/watch?v=n36cpSmKt1c - Interaction: Projection, Perception & Effects, Location & City - Platform & tech: Projection, stereoscopic projection, 3D glasses, architectural mapping - Idea: Adding stereo depth to a building projection makes content leave the surface and occupy the air in front of it, which is what a head-worn AR display does by default. - What it is: The audience wears 3D glasses while the museum facade seems to push out toward them and fold open in true stereoscopic depth, ending in an interactive epilogue; the soundtrack is by Alva Noto. - Technique: Two projector sets with polarizing or shutter filters project left- and right-eye images mapped to the facade, viewed through 3D glasses (Bordos with Ivo Kovacs, Daniel Szalko and Andras Miklos Balogh). - Try it: Model a simple box facade, then in WebXR or HoloKit render geometry that grows out of the real wall toward the viewer in stereo. Twist: compare the same scene on a flat phone screen and in stereo and list what only stereo can do. #### ESCAPE (St. Ludmila Church, Signal Festival Prague) — Bordos.ArtWorks (László Zsolt Bordos) (2015) - Video: https://www.youtube.com/watch?v=ih8vLl75v7U - Interaction: Projection, Location & City, Perception & Effects - Platform & tech: Projection, architectural projection mapping, 3D animation - Idea: Treating architecture as something that can break and release what is inside it is a strong dramatic arc for any AR piece anchored on a building. - What it is: The neo-Gothic facade of St. Ludmila Church in Prague appears to crack, breathe and let figures and abstract forms escape from its arches, set to music by Jóhann Jóhannsson. - Technique: Architectural projection mapping from a 3D model of the church, animating fake depth, lighting and crumbling geometry aligned to the real stonework. - Try it: Scan a building facade with a LiDAR phone and make a short AR sequence where cracks spread along its real edges and a creature escapes from a window. Twist: the cracks follow the viewer's gaze. #### LUX FORMAE (Solid Light Festival, Rome) — Bordos.ArtWorks (László Zsolt Bordos) (2018) - Video: https://www.youtube.com/watch?v=wQynpuovlIc - Interaction: Projection, Voice & Sound, Location & City - Platform & tech: Projection, generative visuals, DMX lighting, projection mapping - Idea: Pure geometric light that follows the building's own lines can be more powerful than illustration; AR overlays can simply trace and emphasise real structure. - What it is: Generative light patterns and architectural lighting move across the arches of a Roman building in time with electronic sound, tracing and reshaping its geometry instead of telling a story. - Technique: Real-time generative visuals are mapped onto the facade and combined with DMX-controlled lighting, cued to the soundtrack by Ondřej Skála. - Try it: In AR, detect the edges of a real doorway or arch (manually outline them if needed) and animate glowing lines that travel along those edges to music. Twist: the line speed follows the loudness of the room. ### Brilliant Labs *Maker of open-source AI glasses (Monocle, Frame, Halo)* Brilliant Labs makes lightweight, open-source AR glasses and clip-ons whose assistant Noa sees, listens and answers in a small display. #### arGPT fact-checking on the Monocle — Brilliant Labs (2023) - Video: https://www.youtube.com/watch?v=VyHFm3OrDPo - Interaction: Voice & Sound, Information & UI - Platform & tech: Wearable, Brilliant Labs Monocle, ChatGPT, iOS - Idea: A fact-checker that sits on your glasses during a conversation. - What it is: The Monocle clip-on display runs arGPT, an always-on ChatGPT assistant that can translate, act as a tour guide or fact-check what someone just said, showing the answer in a tiny eyepiece. - Technique: Audio captured by the Monocle is transcribed on a paired phone, sent to ChatGPT with a fact-checking prompt, and the short answer is streamed back to the micro-display. - Try it: Record a 20-second claim, transcribe it and ask an LLM to rate it true, false or unclear, showing the verdict in a headset viewer. Twist: fact-check a classmate's story about their weekend. #### Frame: open-source AI glasses — Brilliant Labs (2024) - Video: https://www.youtube.com/watch?v=OS6GMsYyXdo - Source code: https://github.com/brilliantlabsAR/frame-codebase - Interaction: Voice & Sound, Information & UI - Platform & tech: Wearable, Brilliant Labs Frame, Noa, multimodal AI, Lua, open source - Idea: Open hardware so anyone can build their own seeing, talking glasses. - What it is: Frame is a lightweight pair of open-source AR glasses with a camera, microphone and display, whose assistant Noa answers questions about what you see and hear. - Technique: The glasses run a small Lua runtime and send camera and audio over Bluetooth to a phone, which queries multimodal models and returns text to the display. - Try it: Using any camera-equipped glasses or a phone on a strap, build 'what am I looking at?' with a free vision API and show a one-line answer. Twist: answer as a poem of exactly seven syllables. #### Halo and Noa: glasses that remember what you forgot — Brilliant Labs (2025) - Video: https://www.youtube.com/watch?v=Xp06a3n4LZU - Source code: https://github.com/brilliantlabsAR/halo-firmware - Interaction: Voice & Sound, Information & UI, Location & City - Platform & tech: Wearable, Brilliant Labs Halo, Noa, long-term memory - Idea: Glasses as a memory that notices what slipped through the cracks. - What it is: While running errands, Halo glasses with the Noa assistant remind the wearer of the small things they meant to buy, drawing on what it heard and saw earlier. - Technique: Noa keeps a searchable memory of past conversations and views, and matches it against the current location or scene to trigger a reminder. - Try it: Keep a spoken shopping list in a notes app, then use phone geolocation to show the relevant item as floating AR text when you enter a shop. Twist: the reminder only appears if you are looking at the right shelf. ### Cao Fei *Multimedia artist working across video, virtual worlds, VR and AR* Chinese artist whose work on automation, labour and virtual life runs from the Second Life city RMB City (2007) to the VR piece The Eternal Wave and several AR commissions. She created the AR-enhanced BMW Art Car #18 and the Trade Eden walk for Apple's [AR]T program. #### BMW Art Car #18 — Cao Fei (2017) - Video: https://www.youtube.com/watch?v=i39mDMyn5-8 - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Phone, Headset, AR app, object tracking, VR film - Idea: The real artwork on the car is invisible until you look through a phone. - What it is: A plain carbon-black BMW M6 GT3 that looks almost unpainted; seen through a phone app, trails of coloured light flow over the car's body, and a VR film extends the story of its future driver. - Technique: A marker-less object-recognition AR app recognises the car and overlays animated light trails aligned to its body, while the physical car stays uncoloured. - Try it: Pick a plain object (a white box, a bike) and build an image- or object-tracking AR scene that paints animated light lines only visible through the phone. Twist: the lines move faster the faster the phone moves around the object. #### Trade Eden — Cao Fei, Apple (ARKit / visionOS teams) (2019) - Video: https://www.youtube.com/watch?v=g3IJT00M3CY - Interaction: Location & City, Play - Platform & tech: Phone, ARKit, iPhone - Idea: A playful automated factory in the street pokes fun at consumer goods and useless machines. - What it is: An AR stop on Apple's [AR]T Walk: a part fun-house, part automated factory appears in the street, where packaged goods travel through useless machines and the viewer tests their dexterity to catch them. - Technique: Built with ARKit and anchored to specific city locations on a guided group walk; visitors used iPhones provided by Apple (likely with location-triggered scenes). - Try it: Design an AR 'factory' in Reality Composer or Lens Studio where tapping sends a box down a chain of pointless machines. Twist: the machine must end by delivering the box back to the player unchanged. #### The Eternal Wave — Cao Fei, Acute Art (2020) - Video: https://www.youtube.com/watch?v=DdvJsIVgMx8 - Interaction: Portals & Worlds, Spatial Mapping - Platform & tech: Headset, Phone, VR, HTC Vive, Unity - Idea: A real stage set becomes the doorway into a VR time-travel story. - What it is: Visitors in a headset start in a replica of the Hongxia Theatre kitchen from her film Nova and step through portals that jump across time and space, into early Chinese computer factories and the theatre's surroundings; an AR version later brought the kitchen to phones. - Technique: Room-scale VR that matches a physical set, with portal transitions between scanned and modelled scenes; the AR follow-up lets users tap furniture to trigger changes. - Try it: Build a WebXR or AR Foundation scene where a real doorway in the classroom opens into a different era. Twist: an everyday object in the room must be the key that opens the portal. ### Chico MacMurtrie / Amorphic Robot Works *Artist; robotic and inflatable sculpture* American artist who leads Amorphic Robot Works, a collective building pneumatic robots and giant inflatable sculptures that breathe, unfold and move like bodies. #### Sixteen Birds — Chico MacMurtrie / Amorphic Robot Works (2006) - Video: https://www.youtube.com/watch?v=3xxTv3-DWUI - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, pneumatics, cables, inflatables - Idea: Moving creatures along fixed paths overhead fills a big volume with life using very simple motion — a cheap trick for AR flocks. - What it is: Sixteen inflatable birds glide along cables through a large public space, flapping their soft wings as air is pumped into them. - Technique: Inflatable bird bodies are driven along overhead cables while valves pulse air into the wings to create flapping. - Try it: Draw a few spline paths across the ceiling of a real atrium in AR and send soft, flapping birds along them in a loop. Twist: let a raised hand call the nearest bird down to land on it. #### Inflatable Architectural Body — Chico MacMurtrie / Amorphic Robot Works (2007) - Video: https://www.youtube.com/watch?v=vqXJpuglpa0 - Interaction: Hands & Body, Spatial Mapping, Perception & Effects - Platform & tech: Desktop, pneumatics, fabric, valves, microcontrollers - Idea: Soft inflation makes huge forms feel gentle and alive; AR objects can gain the same presence with slow breathing growth instead of instant appearance. - What it is: A large inflatable structure of fabric tubes rises, bends and arches as air is pumped in and out, turning a building-sized form into a slowly moving body. - Technique: Computer-controlled valves and blowers inflate sealed fabric chambers in sequence so the structure unfolds and moves like a muscle. - Try it: Grow a giant AR inflatable arch out of a real doorway that swells in sequence, tube by tube, when someone walks under it. Twist: make it deflate and slump if people crowd it. #### Border Crossers — Chico MacMurtrie / Amorphic Robot Works (2010) - Video: https://www.youtube.com/watch?v=8yeyn_8PSPU - Interaction: Location & City, Shared & Social - Platform & tech: Desktop, pneumatics, inflatable fabric - Idea: A figure that physically reaches across a real line gives the line meaning; AR can place bodies that bridge real boundaries on site. - What it is: Giant inflatable figures slowly rise and reach across a border line, bridging the space between two sides with soft, arching bodies. - Technique: Sequentially inflated fabric chambers and pneumatic control let very large figures stand, bend and lean without a rigid frame. - Try it: Anchor two AR figures on either side of a real line (a street, a fence, a crack in the pavement) that grow toward each other and touch when two phones point at them together. Twist: they only meet if both users stand still. ### Chiharu Shiota (塩田千春) *Artist; thread installations* Japanese artist who weaves kilometres of red or black thread through rooms, around keys, shoes, boats or beds, turning memories into dense walk-in webs. #### The Key in the Hand — Chiharu Shiota (塩田千春) (2015) - Video: https://www.youtube.com/watch?v=u_M40SwNw0w - Interaction: Shared & Social, Spatial Mapping, Perception & Effects - Platform & tech: Desktop, red wool, keys, boats - Idea: Thousands of personal objects hung in one web make memory visible as a volume; AR can let a crowd's contributions hang in a shared space the same way. - What it is: In the Japan Pavilion at the Venice Biennale, a canopy of red yarn fills the room and holds about 50,000 old keys collected from the public, dangling above two wooden boats. - Technique: Hand-knotted red wool lines run from ceiling to floor, each carrying keys, so the density of thread and objects builds a walk-under cloud. - Try it: Build a WebXR or phone AR room where each participant uploads a photo of one key or small object; the app hangs it on a red line from the ceiling at their standing position. Twist: the line color changes with how long ago the object was added, so old memories fade to pink. #### Uncertain Journey — Chiharu Shiota (塩田千春) (2016) - Video: https://www.youtube.com/watch?v=6S3RCLaPhMg - Interaction: Spatial Mapping, Perception & Effects, Hands & Body - Platform & tech: Desktop, red wool, metal boat frames - Idea: Thousands of thin lines in one color can turn an empty room into a solid-feeling volume; in AR, line density alone can define a space without any walls. - What it is: Red yarn spun from the gallery ceiling wraps around several bare metal boat frames, turning the whole white space into a dense red web the visitor walks through. - Technique: Red wool is tied point by point from ceiling to the boat frames, building up a web whose density reads as color and shadow. - Try it: Write a Three.js/WebXR script that draws 2,000 random red lines between the detected ceiling height and a few anchor objects you place in a room. Twist: lines within a meter of the viewer loosen and sag, so walking carves a path through the web. #### A Walk through the Line — Chiharu Shiota (塩田千春) (2017) - Video: https://www.youtube.com/watch?v=5UmbVYwyiqQ - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Desktop, thread, site-specific installation - Idea: A line drawing can be scaled up until the viewer is inside it; AR drawing tools become stronger when strokes are big enough to walk through rather than look at. - What it is: For Pafos 2017, European Capital of Culture, Shiota spent days knotting threads through a building until visitors walked inside a drawing in the air. - Technique: Thread is tied between walls, ceiling and objects by hand, and the timelapse shows how thousands of individual knots accumulate into a volumetric drawing (details of this installation likely). - Try it: Use a 3D drawing app (Open Brush, or a Lens Studio drawing template) to draw lines in a room for one hour with a team, only in one color, until the space feels filled. Twist: record the build as a timelapse and play it back in AR so newcomers watch the web grow around them. ### Chris Drury *Land artist* British artist who builds stone and turf 'cloud chambers', camera obscuras that project the sky onto the floor, and earthworks based on patterns of weather, water and the body. #### Installations at Montalvo Arts Center — Chris Drury (2009) - Video: https://www.youtube.com/watch?v=sj1dSbK_Npg - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, stone chamber, woven branches, forest - Idea: Echoing a pattern you find in nature (veins, rivers, whirlpools) at a new scale makes the hidden structure visible, a strategy for AR generative forms. - What it is: At Montalvo Arts Center in California, Drury builds works that mirror natural patterns, including a woven structure up in the trees, and explains how his work follows the hidden patterns of nature. - Technique: He weaves branches and builds chambers on site, likely referencing drawn patterns of water flow and blood vessels. - Try it: Photograph a leaf's veins, trace them into a 3D path, and grow the pattern at tree scale between real trunks in AR. Twist: make the branches pulse like blood flow when touched. #### Carbon Sink: What Goes Around, Comes Around — Chris Drury (2011) - Video: https://www.youtube.com/watch?v=tHgpIlfjyWM - Interaction: Location & City, Information & UI, Perception & Effects - Platform & tech: Desktop, beetle-killed pine, coal, campus lawn - Idea: A simple spiral of two materials tells a cause-and-effect story; AR data art can be this direct and this local. - What it is: At the University of Wyoming, a 11-metre spiral of beetle-killed pine logs and lumps of coal made a direct link between fossil fuels, a warming climate and dying forests; it was removed after protests from the coal industry. - Technique: Logs from local forests killed by pine beetles were arranged in a whirlpool with coal at its centre. - Try it: Build an AR spiral on campus from two scanned materials that stand for a local cause and effect (for example, cars and dead trees). Twist: tie the spiral's size to today's local air-quality index. #### Horizon Line Chamber — Chris Drury (2015) - Video: https://www.youtube.com/watch?v=bo748C3iz2U - Interaction: Location & City, Projection, Perception & Effects - Platform & tech: Projection, camera obscura, stone chamber, Morecambe Bay - Idea: A dark room that shows the outside world live is the ancestor of AR passthrough: it reframes a familiar view by changing how it is delivered. - What it is: A stone chamber on the shore of Morecambe Bay works as a camera obscura: inside the dark room, a lens projects the moving sea, sky and horizon upside down onto the walls. - Technique: A lens and mirror set into the roof or wall project the exterior view onto the interior surfaces; no electricity is used. - Try it: Build a cardboard camera obscura, then build the digital version: a phone app that flips the live camera upside down and projects it onto a wall with a pico projector. Twist: delay the image by one minute so people meet their past selves. ### Dan Flavin *Artist; fluorescent light* American minimalist (1933–1996) who used standard coloured fluorescent tubes as his only material, placing them in corners, corridors and a Milan church to flood architecture with coloured light. #### untitled (to Donna) II — Dan Flavin (1971) - Video: https://www.youtube.com/watch?v=c6uz8fy1sM4 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Projection, fluorescent tubes, corner - Idea: A corner is the most ignored part of a room; placing light there changes the whole geometry, a reminder to use corners and edges as AR anchors. - What it is: A square of fluorescent tubes set diagonally across a room corner glows yellow toward the viewer and pink and blue into the corner behind it, making the corner itself dissolve into coloured light. - Technique: Tubes facing front and back emit different colours, so the direct light and the reflected light in the corner mix into two distinct fields. - Try it: Detect a real room corner with AR Foundation plane intersection and place a virtual square of light across it, lighting the corner behind with one colour and the viewer's side with another. Twist: let the colours swap when the viewer walks to the side. #### untitled (Dia:Bridgehampton) — Dan Flavin (1983) - Video: https://www.youtube.com/watch?v=8bPqlQEPUO0 - Interaction: Perception & Effects, Location & City - Platform & tech: Projection, fluorescent tubes, former church building - Idea: Off-the-shelf lights arranged with care become art; AR designers can make strong work from default primitives (a line, a quad) instead of custom assets. - What it is: In a former firehouse and church in Bridgehampton, New York, nine works made of standard coloured fluorescent tubes line a long gallery, soaking the walls in coloured light. - Technique: Commercial fluorescent fixtures in standard lengths and colours are mounted in repeated configurations so that their glow, not the tubes, colours the room. - Try it: Using only unlit primitive cylinders and emissive colour in AR, build a series of five light 'corners' along a corridor and bake their glow onto the real walls. Twist: allow only three colours in the whole series. #### Santa Maria Annunciata in Chiesa Rossa — Dan Flavin (1997) - Video: https://www.youtube.com/watch?v=45Ax3XGczBo - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, fluorescent tubes, church architecture - Idea: Light can follow the story of a building (entrance, nave, altar); AR experiences in architecture can be sequenced along the route people already walk. - What it is: Flavin's last work lights the nave, transept and apse of a Milan church with green, blue, pink, gold and ultraviolet fluorescent light, turning the building into a sequence of colour spaces. - Technique: Fluorescent fixtures are hidden along the upper edges of the nave and transepts, washing each architectural zone in a different colour. - Try it: Map the route from your school entrance to the main hall and assign each section an AR colour wash that fades in as a visitor enters it. Twist: end the route with an ultraviolet-looking zone that reveals hidden text. ### Dan Miller *XR developer advocate; former Unity AR evangelist* San Francisco XR developer who spent years at Unity showing how AR Foundation works, and who open-sourced small samples that put URP post-processing, Shader Graph and VFX Graph on top of the phone camera. #### AR Foundation + VFX Graph — Dan Miller (2019) - Video: https://x.com/DanMillerDev/status/1186744214392049664 - Source code: https://github.com/DanMillerDev/ARFoundation_VFX - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, Unity, AR Foundation, ARKit, VFX Graph, URP - Idea: Any VFX Graph effect becomes an AR effect once the render pipeline and the camera background agree. - What it is: A glowing VFX Graph particle system floats on a real table, seen through the phone, showing that Unity's GPU particle tool can run on top of the AR camera feed. - Technique: The project configures the Universal Render Pipeline with the AR Foundation background renderer feature, places a slimmed-down Unity Logo VFX Graph on a detected plane, and checks for Metal/Vulkan support at runtime. - Try it: Use the repo as a template: swap the sample graph for your own particle effect and spawn it where the user taps a detected plane. Twist: make the particles collide with the AR plane so they pour off the table edge. #### AR Foundation Post Processing — Dan Miller (2019) - Video: https://x.com/DanMillerDev/status/1180163735668707328 - Source code: https://github.com/DanMillerDev/ARFoundation_PostProcessing - Interaction: Perception & Effects - Platform & tech: Phone, Unity, AR Foundation, ARKit, URP Post Processing - Idea: Make virtual objects sit in the camera image by giving them the same lens as the real camera. - What it is: The live AR camera view gets cinematic post-processing: depth of field that follows the phone camera's real focus, film grain and motion blur on both the room and the virtual objects. - Technique: A URP post-processing volume runs over the AR Foundation camera background, and a script reads ARKit camera intrinsics each frame to drive the depth-of-field focus distance, grain and motion blur. - Try it: Clone the repo, build it to a phone and place one glossy object on a table; tune depth of field so the object blurs exactly like the real background when you refocus. Twist: animate the grain and color grade to the time of day read from the device clock. #### AR Foundation Demos: AR Shaders — Dan Miller (2020) - Video: https://x.com/DanMillerDev/status/1294377908555800576 - Source code: https://github.com/Unity-Technologies/arfoundation-demos - Interaction: Perception & Effects - Platform & tech: Phone, Unity, AR Foundation, Shader Graph, URP, Unity MARS - Idea: Shaders that ground and reveal virtual content are what make AR look real, and they can be shared as a library. - What it is: A reel of shaders made for mobile AR: a wireframe over the room's LiDAR mesh, soft blurred shadows that ground objects on real tables, LiDAR depth fog and camera-grain-matched materials. The shaders were written by Arturo Nereu and Landon Townsend; Dan Miller put them into AR. - Technique: URP Shader Graph shaders with custom HLSL nodes: barycentric vertex colors draw a wireframe on the ARKit mesh, stochastic contact shadows render on transparent planes, the ARKit background shader mixes in scene fog from LiDAR depth, and ARKit's camera-grain texture is applied to virtual materials. - Try it: Import the AR shaders from arfoundation-demos, show the LiDAR mesh as a wireframe while scanning and ground each placed object with the blurred shadow. Twist: fade the wireframe out wherever the user has already placed content. ### Darren Pearson (DARIUSTWIN) *Light-painting artist and animator* Los Angeles artist who draws glowing skeletons, dinosaurs and characters in the air with handheld lights and strings hundreds of long exposures into light-painting stop-motion films. #### Kill the Lights — Darren Pearson (DARIUSTWIN) (2017) - Video: https://www.youtube.com/watch?v=HaWYh7YLazc - Interaction: Drawing & Making, Hands & Body, Location & City - Platform & tech: Desktop, long exposure, stop motion, handheld LEDs - Idea: A character drawn only as outlines can live convincingly in a real landscape; AR characters do not need full shading to feel present. - What it is: Glowing skeletons drawn in light dance, ride and wander through night landscapes in a stop-motion film built from long-exposure photographs. - Technique: Each frame is one long exposure in which the artist redraws the skeleton with handheld lights in a slightly new pose. - Try it: Draw a wireframe skeleton in an AR drawing tool, rig it with a phone body-tracking skeleton so it copies a friend's dance, and anchor it outdoors at night. Twist: leave a faint trail of past poses behind each bone. #### She Lights The Night — Darren Pearson (DARIUSTWIN) (2018) - Video: https://www.youtube.com/watch?v=iOU2kjWp9jg - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Desktop, long exposure, stop motion, handheld LEDs - Idea: A light-painted character can light up its own surroundings; in AR, virtual characters that cast light on real surfaces feel far more grounded. - What it is: A glowing girl carries light through a dark world, animated from 1,012 hand-drawn light-painting photographs. - Technique: Frame-by-frame long exposures drawn with handheld lights, sequenced into a short film. - Try it: Place a glowing AR character in a dark room using a real-time light estimation and a point light attached to the character so it lights the scanned mesh around it. Twist: the room's real lamps dim when the character gets close. #### Fiat Lux — Darren Pearson (DARIUSTWIN) (2022) - Video: https://www.youtube.com/watch?v=RF7GNqIQNn0 - Interaction: Drawing & Making, Location & City - Platform & tech: Desktop, long exposure, stop motion, handheld LEDs - Idea: Story beats can be staged in real locations using only drawn light; AR stories can likewise lean on sparse, glowing marks rather than heavy 3D assets. - What it is: A narrative light-painting short in 11 scenes made from 686 long-exposure photographs straight out of the camera, telling a story entirely with drawn light. - Technique: Hundreds of in-camera light paintings are shot as a stop-motion sequence, without compositing. - Try it: Storyboard a three-scene AR story told only with glowing line drawings anchored at three spots in a park, each scene triggered when the viewer arrives. Twist: the drawings are traced by the viewer first, then animate on their own. ### Ernesto Neto *Artist; soft walk-in sculptures* Brazilian artist who builds huge crocheted and stretch-fabric environments, often filled with spices or scented seeds, that visitors walk through, lie in and touch. #### Léviathan Thot — Ernesto Neto (2006) - Video: https://www.youtube.com/watch?v=zd-VE7eOLoQ - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, lycra, sand, spices - Idea: A soft form hung in a hard, historic space changes how the whole building feels; AR overlays can soften or re-body architecture instead of just labeling it. - What it is: Under the dome of the Panthéon in Paris, huge stretchy white lycra forms hang like the limbs of a giant creature, weighted with sand and spice-filled pods that bulge toward the visitors below. - Technique: Tensioned lycra membranes are anchored to the building and shaped by the weight of filled pockets, so gravity sculpts the forms. - Try it: Build an AR piece that hangs a soft-body cloth creature from the ceiling of a real hall, using a cloth simulation in Unity AR Foundation pinned to detected ceiling points. Twist: add heavy 'drops' when the user taps, and watch the cloth sag toward them. #### GaiaMotherTree — Ernesto Neto (2018) - Video: https://www.youtube.com/watch?v=eKhyy3qEYms - Interaction: Location & City, Voice & Sound, Perception & Effects - Platform & tech: Desktop, crocheted cotton, spices, cushions - Idea: A room-within-a-room placed in a busy public hall creates a pause zone; AR can carve a calm, soft space inside a crowded place without moving anything physical. - What it is: In the main hall of Zurich's central station, a 20-metre hand-knotted cotton tree hangs from the roof; commuters walk inside it, sit on cushions and join talks and rituals. - Technique: Colourful cotton strips are crocheted and knotted by hand into a net canopy suspended from the station roof, with hanging drops holding spices and seeds. - Try it: Design a location-based AR 'tree tent' for a busy public space: when the user stands at the anchor, a knotted canopy grows around them and city noise is replaced by soft sound. Twist: the canopy only grows while the user stays still for ten seconds. #### SunForceOceanLife — Ernesto Neto (2023) - Video: https://www.youtube.com/watch?v=vT4zn_y6Kiw - Interaction: Hands & Body, Shared & Social, Perception & Effects - Platform & tech: Desktop, crocheted cotton, tension structure - Idea: When the floor itself gives way, visitors feel a space with their whole body; AR can borrow this by tying virtual space to balance, steps and posture, not only to sight. - What it is: At the Museum of Fine Arts, Houston, visitors take off their shoes and climb into a large crocheted, walkable structure whose soft floor moves under their feet. - Technique: Hand-crocheted nets of cotton are tensioned from the gallery architecture to make a hammock-like floor and canopy (layout and materials likely similar to his earlier walk-in works). - Try it: Make a phone AR walk where the virtual floor appears to sag and ripple under each footstep detected by the phone's accelerometer. Twist: two users in the same space make shared ripples that meet and interfere. ### George Rickey *Artist; kinetic sculptor of wind-driven blades* American sculptor (1907–2002) whose stainless-steel blades, lines and planes, balanced on precise bearings, swing and gyrate slowly in the slightest wind. #### Wild Carrot II — George Rickey (1987) - Video: https://www.youtube.com/watch?v=GqzjY_l8_2c - Interaction: Perception & Effects - Platform & tech: Desktop, stainless steel, bearings - Idea: Abstract steel that moves like a plant borrows the behaviour of nature without imitating its look; AR plants can do the same with motion alone. - What it is: A cluster of small stainless-steel blades on a branching stem that trembles and turns in the breeze like a plant in a field. - Technique: Many small balanced elements on a branching armature each respond to the wind separately, so the whole reads as a living cluster. - Try it: Grow a field of abstract AR 'plants' made of balanced blades on a real lawn, swaying with a noise-driven wind that gusts when the user walks through. Twist: make the plants turn away from the user like a shy herd. #### Two Lines Oblique Gyratory II — George Rickey (1989) - Video: https://www.youtube.com/watch?v=Hih0BDuKGik - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, stainless steel, bearings - Idea: Two lines moving in wind are enough to animate a whole landscape; AR objects outdoors can be kept minimal and let the environment drive them. - What it is: Two long stainless-steel blades balanced on bearings sweep and cross each other slowly in the wind, drawing arcs against the sky. - Technique: Counterweighted blades on precision bearings respond to the lightest wind while their geometry keeps the blades from colliding. - Try it: Anchor two tall AR blades in an outdoor space and drive their rotation from live wind data (a weather API) or the phone microphone's wind noise. Twist: draw the blade tips' paths as fading lines in the sky. #### Annular Eclipse — George Rickey (2000) - Video: https://www.youtube.com/watch?v=dUIJmM2psKY - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, stainless steel, bearings - Idea: Rare alignments are rewarding moments; AR works can design for the chance moment when parts line up from the viewer's point of view. - What it is: Stainless-steel rings balanced on arms rotate independently in the wind on Park Avenue, occasionally aligning to frame each other like an eclipse. - Technique: Each ring sits on its own bearing with balanced weights, so independent wind-driven rotations occasionally produce concentric alignments. - Try it: Create an AR sculpture of two rings that rotate independently and glow brightly only when they align from the user's current viewpoint. Twist: make alignment rarer the longer the user stands still. ### Georges Rousse *Artist and photographer; anamorphic interventions* French artist who paints shapes across the walls, floors and debris of abandoned buildings so that they read as a flat, floating figure from the camera position, then photographs the result. #### Bending Space: the Durham Project — Georges Rousse (2011) - Video: https://www.youtube.com/watch?v=cnwTgXymWf0 - Interaction: Location & City, Perception & Effects, Spatial Mapping - Platform & tech: Desktop, paint, plywood, photography - Idea: Buildings about to change are perfect hosts for temporary images: AR can give abandoned or transitional spaces a last visible layer before they disappear. - What it is: In old tobacco warehouses of Durham, North Carolina, Rousse and local volunteers paint and build shapes that, from the camera's point of view, float in the empty industrial rooms. - Technique: Anamorphic painting and cut-out constructions are aligned through a fixed camera; the photograph, not the room, is the finished work. - Try it: Pick a disused room on campus, scan it with LiDAR, and place one anamorphic AR shape that resolves from the doorway; record a 30-second video walking in. Twist: let students leave short voice notes about the room that play when the shape resolves. #### Art Project in Miyagi — Georges Rousse (2013) - Video: https://www.youtube.com/watch?v=vSQtrKtdYEg - Interaction: Location & City, Perception & Effects, Shared & Social - Platform & tech: Desktop, paint, photography, community workshop - Idea: A single aligned shape can hold a community's memory of a place: AR memorial layers work when locals help decide where the one meaningful viewpoint is. - What it is: After the 2011 tsunami, Rousse works with residents in Miyagi, Japan, painting geometric shapes inside damaged buildings that come together as one figure in his photographs. - Technique: The same camera-aligned anamorphic process is carried out with volunteers, likely inside buildings awaiting demolition, and the photographs are exhibited locally. - Try it: Interview a staff member about a room that matters to them, then place one AR shape and a short text that only align from the spot they chose. Twist: save the anchor so next year's class can find it again. #### Musée de l'Homme in situ — Georges Rousse (2015) - Video: https://www.youtube.com/watch?v=CB2ZIcahpaU - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, paint, large-format camera, photography - Idea: The camera is the only true viewer: in AR the phone is also a single eye, so a shape painted for one lens can feel more magical than one that works from everywhere. - What it is: A time-lapse shows Rousse painting a shape across the floors, walls and columns of the Musée de l'Homme in Paris; only through his camera does it become a clean flat figure floating in the room. - Technique: Rousse fixes a large-format camera, marks the shape through the ground glass onto the architecture, paints the pieces, and then photographs the finished image from the same point. - Try it: Paint a real shape in masking tape across a corner of the classroom, aligned through a phone on a tripod, then add an AR layer that only appears when the phone returns to the tripod pose. Twist: let the AR layer animate the taped shape into 3D when aligned. ### Gianni Colombo *Artist; kinetic and programmed-art pioneer, member of Gruppo T* Italian artist (1937–1993) of Milan's Gruppo T who built pulsating reliefs and 'programmed environments' such as Spazio elastico (1967), a dark room of fluorescent elastic cords that slowly stretch and shift under UV light. #### Strutturazione pulsante (Pulsating Structure) — Gianni Colombo (1959) - Video: https://www.youtube.com/watch?v=vb_p3SxIBak - Interaction: Perception & Effects - Platform & tech: Desktop, polystyrene blocks, motors, relief - Idea: A surface that breathes by moving small pieces is a physical pixel display; AR can make real walls appear to pulse with the same displacement idea. - What it is: A white relief of small blocks that are pushed in and out by hidden motors, so the surface slowly pulses and changes its pattern of light and shadow. - Technique: Motor-driven cams push individual blocks back and forth by a few centimetres, so the relief's shading pattern shifts continuously. - Try it: Overlay a detected real wall with an AR grid of blocks that push out and in like a breathing surface, driven by a noise field or the user's breath. Twist: make the blocks push out where the user's shadow falls. #### Spazio elastico (Elastic Space) — Gianni Colombo (1967) - Video: https://www.youtube.com/watch?v=gA24n_wGuss - Interaction: Perception & Effects, Spatial Mapping, Hands & Body - Platform & tech: Projection, fluorescent elastic cords, UV light, motors - Idea: Distorting the grid that defines a space makes the space itself feel elastic; AR can warp the perceived room with glowing guide lines instead of objects. - What it is: A dark room crossed by a grid of fluorescent elastic cords glowing under ultraviolet light; motors slowly stretch the cords so the grid warps and the room's geometry seems to breathe. - Technique: Fluorescent elastic cords form a 3D grid in a black room; motors pull individual cords so the lattice deforms slowly while UV light makes only the lines visible. - Try it: Fill a dim real room with a glowing AR 3D grid (WebXR or Unity) and let one grid node slowly follow the user so the lattice bends around them. Twist: dim the passthrough so only the grid is visible, like a UV room. #### Topoestesia (itinerario programmato) — Gianni Colombo (1970) - Video: https://www.youtube.com/watch?v=nkeF48-8Vto - Interaction: Spatial Mapping, Hands & Body, Perception & Effects - Platform & tech: Desktop, architectural environment, inclined floors, light - Idea: Changing the ground under the feet changes perception more than any image; AR experiences can guide walking paths and pacing to shape how a space is felt. - What it is: A walk-through environment of tilted floors, steps and narrow passages that throws visitors off balance and makes them feel space through their bodies rather than their eyes. - Technique: A sequence of spaces with inclined planes and irregular steps is programmed as an itinerary, so the body's balance and movement become the medium. - Try it: Design a WebXR walk through a real corridor where AR floor markings, tilted virtual horizons and narrowing walls make the walker slow down and sway. Twist: make the virtual horizon tilt a little more with every step. ### Harold "Doc" Edgerton *Electrical engineer, MIT professor, father of stroboscopic photography (1903–1990)* MIT engineer who developed the electronic stroboscope and microsecond flash, freezing milk drops, bullets and athletes and layering multiple flashes into a single frame. #### Quicker 'n a Wink — Harold "Doc" Edgerton (1940) - Video: https://www.youtube.com/watch?v=gspK_Bi0aoQ - Interaction: Perception & Effects - Platform & tech: Desktop, stroboscope, high-speed camera - Idea: A strobe light lets a still eye see a fast world; AR effects that flash or sample the camera feed can reveal hidden rhythms in ordinary motion. - What it is: Academy Award-winning MGM short in which Edgerton's stroboscope freezes and slows down everyday actions — a golf swing, a drop of water, a spinning fan — revealing motions too fast for the eye. - Technique: A gas-discharge stroboscope flashes at a precise rate so a rotating or repeating motion appears frozen or slowed. - Try it: Build a Lens Studio or WebAR filter that shows the camera image only every Nth frame and holds it, with a slider for N, then point it at a fan, a swing or a dripping tap. Twist: tap a beat on screen to set the strobe rate by rhythm. #### Milk Drop Coronet — Harold "Doc" Edgerton (1957) - Video: https://www.youtube.com/watch?v=zOGmwd7kauE - Interaction: Perception & Effects - Platform & tech: Desktop, microsecond flash, sound trigger - Idea: Freezing one exact instant turns ordinary events into sculptures: AR can let people pause the world at a chosen moment and walk around it. - What it is: A single drop of milk hitting a plate is frozen by a microsecond flash at the instant it forms a perfect crown of droplets. - Technique: A sound or light trigger fires an electronic flash lasting a few microseconds, so a normal camera with an open shutter records a sharp instant. - Try it: Capture a splash with the phone's 240 fps mode, pick the most sculptural frame, turn it into a depth-based 3D relief with a depth-estimation model and place it on a real table in AR. Twist: let the viewer tap to advance the frozen splash by one frame at a time. #### Bullet through Apple — Harold "Doc" Edgerton (1964) - Video: https://www.youtube.com/watch?v=22J7-ypzkYk - Interaction: Perception & Effects - Platform & tech: Desktop, microsecond flash, sound trigger - Idea: Violent, invisible events become calm and readable when frozen; AR replays can slow dangerous physics to a speed where people can study them safely. - What it is: A .30 caliber bullet is caught leaving an apple that is exploding on both sides, frozen by a flash lasting about a third of a millionth of a second. - Technique: The sound of the gunshot triggers a very short electronic flash in a dark room while the camera shutter is already open. - Try it: Simulate a projectile passing through a virtual fruit in Unity with a mesh-fracture asset, then place the frozen mid-impact moment in AR at table scale with a time slider. Twist: let the viewer set the exact microsecond to freeze by pinching the slider. ### Harshini J. Karunaratne *Interdisciplinary artist and creative technologist* Artist trained in interactive media at NYU Abu Dhabi who works with projection mapping, gesture sensors, dance and live visuals for stage, media facades and installations. At the 2026 CCL at the Royal Ballet and Opera she built a transmedial Hades from fragments of the RBO archive. #### CHOREO — Harshini J. Karunaratne (2015) - Video: https://vimeo.com/191660123 - Interaction: Play, Performance, Hands & Body - Platform & tech: Desktop, Projection, Isadora, recorded dance clips, keyboard input - Idea: Play a recorded dancer like a video game character. - What it is: A dancer's recorded movements become a game: participants press keys to rearrange and trigger fragments of the performance, turning spectators into co-choreographers. - Technique: Clips of a dancer are cut into short phrases in Isadora, and each key press switches or layers phrases on a projected display. - Try it: Film a classmate doing eight short moves, map them to keyboard keys in a web page and let others compose a dance by typing. Twist: the typed word decides the sequence. #### Luminoscope — Harshini J. Karunaratne (2016) - Video: https://vimeo.com/178385157 - Interaction: Projection, Hands & Body, Tangible Objects - Platform & tech: Projection, Leap Motion, MadMapper, Modul8 - Idea: Found junk plus projection plus a hand sensor becomes a responsive sculpture. - What it is: A sculpture made from an old umbrella and cardboard boxes becomes a glowing object when projection-mapped visuals are wrapped around it and changed by the visitor's hand gestures in the air. - Technique: Visuals made in Modul8 are mapped onto the sculpture with MadMapper, and a Leap Motion (via Geco MIDI) turns hand gestures into MIDI controls. - Try it: Projection-map a cardboard box sculpture and use a webcam hand tracker to change colours as visitors wave over it. Twist: each face of the box responds to a different hand. #### Mudança de Dança — Harshini J. Karunaratne (2017) - Video: https://vimeo.com/248122645 - Interaction: Location & City, Performance, Perception & Effects - Platform & tech: Projection, media facade, LED building, dance video - Idea: Scale a dancer up to the size of a building. - What it is: A dance piece made for the pixel media facade of the FIESP building on São Paulo's Paulista Avenue, where a dancer's abstracted body moves across an entire skyscraper at FILE LED SHOW 2017. - Technique: Dance footage choreographed by Kai-Wen Yang was processed into a low-resolution, high-contrast animation suited to the facade's coarse LED grid. - Try it: Downsample a dance clip to a 20×30 pixel grid and play it on a wall of paper windows lit from behind, or on a building photo in AR. Twist: let the dancer only move in the lit windows. ### Hironori Sugino (sugi-cho) *Freelance creative programmer* Tokyo freelance programmer who builds GPU particle, projection-mapping and depth-camera visuals for installations and VJ sets, and co-authored the Unity Graphics Programming books. #### Screen Space Particle Collision — Hironori Sugino (sugi-cho) (2016) - Video: https://www.youtube.com/watch?v=E81EVRG0SlU - Source code: https://github.com/sugi-cho/Unity-GPU-Particle - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, Unity, HLSL, GPU particles - Idea: Use what the camera sees as the physics world for particles. - What it is: A fountain of GPU particles lands on a 3D scene and slides and bounces over its surfaces, computed entirely on the GPU. - Technique: A GPU particle system reads the camera's depth and normal buffers, detects when a particle passes behind a visible surface, and reflects its velocity along the surface normal. - Try it: Feed the AR camera's depth into the particle shader so glitter pours from the ceiling and bounces off real people and tables. Twist: the particles take their colour from the camera pixel where they bounce. #### Projection Spray (VR Spray Drawing) — Hironori Sugino (sugi-cho) (2017) - Video: https://www.youtube.com/watch?v=TTv6YPWNLxY - Source code: https://github.com/sugi-cho/ProjectionSpray - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Headset, Unity, HLSL, SteamVR - Idea: A spray can is just a projector that writes into the texture of whatever it hits. - What it is: In VR the user sprays paint from a virtual can; colour lands on the surfaces of 3D objects exactly where the spray cone hits them. - Technique: The spray is modelled as a projector: a depth map from the can's view tests visibility, and each object's UV-space texture is updated where the projection hits. - Try it: Spray-paint a LiDAR scan of the classroom in AR using the phone as the can, with the paint projected from the phone's pose. Twist: each student's paint colour is picked from the real object they point at. #### Depth Camera Voxel Particles — Hironori Sugino (sugi-cho) (2018) - Video: https://www.youtube.com/watch?v=r6qXq082iH4 - Source code: https://github.com/sugi-cho/RealSenseVisualize - Interaction: Hands & Body, Performance - Platform & tech: Desktop, Projection, Unity, Intel RealSense, GPU instancing - Idea: Rebuild a live body from cubes that can break away. - What it is: A person captured by a RealSense depth camera is rebuilt as a cloud of glowing voxel cubes that move with them and scatter. - Technique: Depth frames from an Intel RealSense are converted to 3D points in Unity, and instanced cubes are placed on a voxel grid at those points, with GPU animation for scattering (likely the same pipeline used for his VJ Voxel Depth VR set). - Try it: Turn LiDAR depth from an iPhone into voxel cubes in AR so a classmate standing in front becomes a block sculpture. Twist: blocks fall to the real floor when the person stops moving. ### Holosphere *Spatial computing studio (VR, AR, projection)* Birmingham XR studio that builds VR, AR and real-time projection projects, from training simulations to live art. For Birmingham Royal Ballet it created a motion-capture-driven 35-metre backdrop, and it experiments with pose tracking on building facades and real-time mocap imagery in museums. #### Waterhall Mocap Performance — Holosphere (2024) - Video: https://www.youtube.com/watch?v=_p_85sK5is4 - Interaction: Hands & Body, Drawing & Making, Performance - Platform & tech: Projection, motion capture, real-time graphics - Idea: A dancer's limbs leave painted trails that hang in the air behind them. - What it is: For the reopening of the Waterhall at Birmingham Museum and Art Gallery, Birmingham Royal Ballet dancers Reina Fuchigami and Shuailun Wu perform while their movement paints real-time motion-capture imagery on a screen; visitors could also try it with their own bodies. - Technique: Real-time motion capture tracks the dancers' joints, and the joint paths are rendered as brush-like ribbons in a game engine and shown live on a large screen. - Try it: Use a webcam and MediaPipe to draw fading ribbons from a dancer's wrists and ankles on a projected screen, then invite the audience to try. Twist: each visitor's ribbons stay on screen until the next person replaces them. #### Pose Tracking R&D on the Aston Webb Building — Holosphere (2025) - Video: https://www.youtube.com/watch?v=HBz0KWN4GJ4 - Interaction: Hands & Body, Projection, Location & City - Platform & tech: Projection, pose estimation, projection mapping, 3D scan - Idea: Wave your arms and a whole building moves with you. - What it is: Pose-tracking tests that use the University of Birmingham's Aston Webb Building as a canvas: people on the ground wave their arms and the projection on the facade reacts to their movement. - Technique: A camera with real-time pose estimation tracks a person's skeleton, and the joint data drives graphics that are mapped onto the scanned building facade. - Try it: Project onto the school building or a large wall and use MediaPipe pose in TouchDesigner so a student's arm angle opens and closes projected windows. Twist: the facade only reacts when three people strike the same pose. #### University of Birmingham 125th Anniversary Projection Show — Holosphere (2025) - Video: https://www.youtube.com/watch?v=wt2GkcPV9DY - Interaction: Projection, Portals & Worlds, Location & City - Platform & tech: Projection, projection mapping, drone photogrammetry, digital twin - Idea: A building's facade opens to show what is inside it. - What it is: A 13-minute projection-mapped show on the Aston Webb Building that seems to open the facade to reveal interior spaces such as the Great Hall and its stained glass, built on drone scans of the building and campus. - Technique: Drone photogrammetry creates a digital twin of the facade used to pre-visualise the show, and scanned interiors are rendered in perspective so they appear to sit behind the wall. - Try it: Photograph a classroom wall, model it quickly in Blender, and project a fake cutaway revealing the room behind it. Twist: reveal what the wall looked like fifty years ago. ### Humans since 1982 (Per Emanuelsson & Bastian Bischoff) *Swedish-German art studio for kinetic clock installations* Studio founded in 2008 by Per Emanuelsson and Bastian Bischoff, known for ClockClock and A Million Times: grids of analogue clock hands that dance in unison and then settle to spell the current time. #### ClockClock — Humans since 1982 (Per Emanuelsson & Bastian Bischoff) (2010) - Video: https://www.youtube.com/watch?v=lMdloSwiNbQ - Interaction: Information & UI, Perception & Effects - Platform & tech: Desktop, 24 analogue clocks, stepper motors, custom firmware - Idea: A familiar object used in an unfamiliar grid becomes a new display — AR can recombine everyday objects as pixels. - What it is: Twenty-four analogue clocks arranged in a grid rotate their hands in choreographed sweeps, then stop so that the hands together form the digits of the current time. - Technique: Each clock has two independently driven hands on stepper motors; firmware animates transitions and then aligns hand angles into segment-style digits. - Try it: Build a 6×4 grid of virtual clock faces on a real wall in AR whose hands swirl and then spell the current time or the user's initials. Twist: let the hands point at the viewer as they walk past. #### A Million Times — Humans since 1982 (Per Emanuelsson & Bastian Bischoff) (2013) - Video: https://www.youtube.com/watch?v=XdaKTnqotbE - Interaction: Information & UI, Perception & Effects - Platform & tech: Desktop, analogue clock modules, motors, choreography software - Idea: Many identical rotating parts can move like a flock — emergent patterns from simple rules are an efficient way to animate AR grids. - What it is: A large wall of analogue clocks turns its hands into flowing patterns of waves, spirals and flocks before settling to show the time. - Technique: Clock hands are driven individually and animated as a vector field sampled at each clock position, so the whole grid reads as flowing motion. - Try it: Animate a grid of virtual clock hands in AR as a vector field that flows around the user's hand position. Twist: stop the flow and spell the time whenever the user holds still for three seconds. #### A Million Times at Changi — Humans since 1982 (Per Emanuelsson & Bastian Bischoff) (2014) - Video: https://www.youtube.com/watch?v=d2dUlZjuGj0 - Interaction: Information & UI, Location & City - Platform & tech: Desktop, 1,008 clocks, motors, airport installation - Idea: Scale turns a clock into architecture — the same grid logic works on a phone screen or a building-sized AR overlay. - What it is: A monumental wall of 1,008 clocks in Singapore Changi Airport performs choreographed patterns and time displays for passing travellers. - Technique: Over a thousand clock modules with two motors each are networked and driven by a central choreography engine. - Try it: Cover a building facade in phone AR with a huge grid of clocks that ripple when a person enters the frame. Twist: show each traveller's home time zone when they point the phone at themselves. ### Instituto Stocos (Muriel Romero, Pablo Palacio & Daniel Bisig) *Interactive dance research group: choreographer, composer and AI/software artist* Spanish-Swiss collective of choreographer Muriel Romero, composer Pablo Palacio and AI researcher Daniel Bisig that builds pieces where sensors read the dancer's body and drive swarm simulations, synthesised voice, robotic lights or a self-playing piano. Bisig, now at Zurich University of the Arts and Motion Bank, co-facilitated the Choreographic Coding Labs at A+E Lab in 2022, 2025 and at the Royal Ballet and Opera in 2026. #### Stocos — Instituto Stocos (Muriel Romero, Pablo Palacio & Daniel Bisig) (2011) - Video: https://www.youtube.com/watch?v=MRTGNMYyGUY - Interaction: Hands & Body, Projection, Voice & Sound - Platform & tech: Projection, swarm simulation, motion tracking, generative music - Idea: Let dancers and a simulated swarm choreograph each other on the same floor. - What it is: Two dancers move inside a floor projection of an artificial swarm; the simulated agents, the music and the light react to their movement and in turn push the dancers into new patterns. - Technique: Body tracking feeds the dancers' positions into a swarm simulation whose agents are projected from above, and the same agent data drives stochastic sound synthesis and lighting. - Try it: Project a p5.js boids flock onto the floor with a cheap projector and use a webcam blob tracker so the flock avoids or follows a person walking through it. Twist: give each person a different 'smell' so the flock prefers one of them. #### Piano&Dancer — Instituto Stocos (Muriel Romero, Pablo Palacio & Daniel Bisig) (2021) - Video: https://www.youtube.com/watch?v=_3gEitmSdis - Interaction: Hands & Body, Voice & Sound, Performance - Platform & tech: Projection, mechanical player piano, motion sensing, gesture analysis - Idea: A dancer plays a piano from across the room with her body alone. - What it is: On stage are a dancer and a piano but no pianist: as the dancer moves, the piano keys press themselves and the music answers her gestures without her ever touching the instrument. - Technique: Sensors on the dancer are analysed for gesture features that an algorithm translates into MIDI notes for a self-playing mechanical piano. - Try it: Use a webcam pose model (ml5.js / MediaPipe) to turn wrist height and speed into MIDI notes played on a laptop piano sound. Twist: only let slow movements make sound. #### Embodied Machine — Instituto Stocos (Muriel Romero, Pablo Palacio & Daniel Bisig) (2022) - Video: https://www.youtube.com/watch?v=ntp-rhbwV9c - Interaction: Hands & Body, Performance, Voice & Sound - Platform & tech: Projection, robotic moving lights, motion sensors, voice synthesis - Idea: Turn stage lights into limbs that extend the dancer's gestures. - What it is: A dancer steers a forest of robotic light beams with her body: as she reaches and turns, the motorised lights swing to follow and illuminate her, while her movement also becomes synthetic vocal sound. - Technique: Wearable motion sensors are analysed for movement qualities and mapped to pan, tilt and intensity of DMX moving-head lights and to a voice synthesiser (likely custom Max or C++ software). - Try it: Map a phone's orientation (via a WebSocket page) to the direction of a small servo-mounted torch so a performer 'points' light around a dark room. Twist: add a delay so the light follows like a lazy shadow. ### Isaac Cohen (Cabbibo) *Artist & creative coder; VR/AR experience designer* Artist-coder known as Cabbibo who makes joyful, tactile interactive worlds in WebGL, VR and AR, such as Blarp! and ARQUA!. #### ARQUA! — Isaac Cohen (Cabbibo) (2017) - Video: https://www.youtube.com/watch?v=Xp-Bcs5fLlc - Interaction: Drawing & Making, Hands & Body, Perception & Effects - Platform & tech: Phone, ARKit, Unity - Idea: Use your body as the controller to build a rainbow fantasy aquarium in your room. - What it is: An ARKit launch app for building a rainbow fantasy aquarium in your room—placing kelp, fish schools and crystal rods by moving your body as the controller (with Viacom NEXT). - Technique: Objects are spawned at or along the phone's camera pose as the user moves, so body movement becomes the placement controller, with flocking behaviors for fish schools. - Try it: Hold the screen and walk around the room to drop kelp and fish schools along your path, and make the fish swim with a simple Boids algorithm. Twist: the school scatters when the phone gets close. #### IMMATERIA: Forms On Forms — Isaac Cohen (Cabbibo) (2020) - Video: https://www.youtube.com/watch?v=GC5f3d3uanM - Source code: https://github.com/cabbibo/IMMATERIA - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: Chain simple compute-shader 'lifeforms' so one simulation lives on the surface of another. - What it is: Shapes made of compute-shader particles grow on the surfaces of other shapes, layering soft forms on forms in a Unity scene. - Technique: IMMATERIA wraps compute buffers ('forms') and kernels ('lifeforms') in reusable Unity components; bindings pass one form's vertices into another's simulation so particles are spawned and constrained on its surface. - Try it: Use the room mesh from AR Foundation as the first 'form' and grow a second particle form over the real furniture. Twist: let the grown form slowly retreat from wherever the phone points. #### IMMATERIA: My First Gooey Mesh — Isaac Cohen (Cabbibo) (2020) - Video: https://www.youtube.com/watch?v=H_08R0Mzq54 - Source code: https://github.com/cabbibo/IMMATERIA - Interaction: Perception & Effects, Play - Platform & tech: Desktop, Unity, Compute Shader - Idea: Make any mesh gooey by simulating its vertices as springy particles on the GPU. - What it is: A soft mesh wobbles and jiggles like jelly, its vertices pulled back to shape by springs after being disturbed. - Technique: Mesh vertices are copied into a compute buffer, a kernel applies spring forces toward their rest positions plus noise and damping, and the deformed positions are read back in the vertex shader. - Try it: Make a gooey virtual object that sits on a real table in AR and wobbles whenever the user bumps the phone toward it. Twist: let the goo sag and drip over the table edge under gravity. ### Jason Bruges Studio *Interactive art and architecture studio* London studio founded by Jason Bruges in 2001 that makes responsive light, kinetic and robotic installations for public spaces, hospitals and architecture. #### Wind to Light — Jason Bruges Studio (2009) - Video: https://www.youtube.com/watch?v=lwLozMxaMv8 - Interaction: Location & City, Information & UI, Perception & Effects - Platform & tech: Desktop, micro wind turbines, LEDs - Idea: Wiring each light to its own local sensor makes an invisible field (wind) readable at a glance; AR can overlay the same kind of field map on a real place. - What it is: A field of small wind turbines each powers its own light, so gusts sweeping across the site show up as waves of brightness. - Technique: Each micro turbine drives an LED directly or via a small controller, so brightness is a live measure of local wind speed; the exact electronics are a guess. - Try it: Scatter virtual lamps across a park or courtyard in AR and set each one's brightness from a shared wind or noise signal with a small delay per position, so gusts appear to travel. Twist: use the phone microphone so voices light up the field. #### Mimosa — Jason Bruges Studio (2010) - Video: https://www.youtube.com/watch?v=srYK8pEYnMc - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, OLED, motors, sensors - Idea: Borrowing a plant's reflex (the mimosa folds when touched) gives a responsive surface an instantly readable personality — useful for AR objects that should feel alive. - What it is: A wall of mechanical OLED 'flowers' opens, closes and glows as visitors move in front of it, like a plant reacting to touch. - Technique: Likely camera or proximity sensing drives small motors that open petal-shaped OLED panels and set their brightness according to how close people are. - Try it: Place a bed of virtual flowers on a real wall in AR that fold shut when the phone or a tracked hand comes near and slowly reopen when it leaves. Twist: make them remember visitors and close faster for people who approach quickly. #### The Nature Trail — Jason Bruges Studio (2012) - Video: https://www.youtube.com/watch?v=8j2RegeSwYM - Interaction: Location & City, Hands & Body, Play - Platform & tech: Desktop, LEDs, motion sensors, laser-cut panels - Idea: Content that appears only when someone passes and walks with them turns a stressful route into a companion — a direct template for location-based AR wayfinding. - What it is: Along a hospital corridor at Great Ormond Street, LED animals hidden behind a patterned wall appear and walk beside children as they are wheeled to surgery. - Technique: Motion sensors along the corridor trigger LED animations embedded behind laser-cut wall panels, so animals reveal themselves in sequence at the pace of the passer-by. - Try it: Build an AR walk along a real corridor where small animals peek out from anchored spots and follow the user at walking pace. Twist: make the animals shy — they only come out if the user walks slowly and quietly. ### Jaume Sanchez Elias *Creative technologist (WebGL, WebXR)* Web developer and 'pixel burner' behind clicktorelease.com and many three.js experiments and tools, now working on WebGL and WebXR interfaces at DeepMind. He made Transcendence for the KIKK 2018 AR exhibit and keeps publishing face, hand and depth experiments in the browser. #### Depth Player (Android Lens Blur) — Jaume Sanchez Elias (2014) - Video: https://www.youtube.com/watch?v=eabttaSmLPE - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Web, three.js, depth maps, WebGL - Idea: Every phone photo secretly holds a 3D scene; pull it out. - What it is: A web player that pulls the hidden depth map out of photos taken with Android's Lens Blur camera mode and shows them as 3D reliefs you can orbit in the browser, such as a window seen from inside a room. - Technique: A JavaScript extractor reads the depth map stored in the Lens Blur JPEG's XMP metadata, and three.js displaces a mesh or point cloud by that depth. - Try it: Take portrait-mode photos on a phone, extract the depth map (or estimate it with a depth model), and display each as a displaced mesh in three.js that tilts with the mouse or phone gyroscope. Twist: animate a slow 'breathing' of the depth. #### Gnarls Barkley 'Crazy' + FaceMesh — Jaume Sanchez Elias (2020) - Video: https://www.youtube.com/watch?v=R-NEWO9Rbog - Interaction: Face, Voice & Sound, Perception & Effects - Platform & tech: Web, TensorFlow.js FaceMesh, three.js, WebGL - Idea: Turn your face into a living Rorschach inkblot. - What it is: A browser toy that recreates the Rorschach-inkblot look of the 'Crazy' music video on your own face: a live webcam face mesh is rendered as a symmetrical, shifting ink blot while the song plays. - Technique: TensorFlow.js FaceMesh tracks 468 face landmarks from the webcam; three.js turns them into a mesh shaded with a mirrored, noise-driven ink shader. - Try it: Pick a music video with a strong visual style and rebuild it as a face filter with MediaPipe Face Mesh and three.js (or Effect House). Twist: the effect reacts to the loudness of the song. #### WebXR Hand Trails — Jaume Sanchez Elias (2020) - Video: https://x.com/thespite/status/1300429595989217280 - Interaction: Hands & Body, Drawing & Making - Platform & tech: Headset, Web, WebXR Hand Input, three.js - Idea: Every fingertip paints a ribbon in the air. - What it is: In a WebXR session with hand tracking, every fingertip drags its own glowing ribbon through space, so waving your hands leaves a bundle of flowing trails around you. - Technique: The WebXR Hand Input API provides joint poses each frame; three.js extrudes camera-facing ribbon geometry along the recent positions of each fingertip. - Try it: Record fingertip positions with WebXR hands (Quest browser) or MediaPipe on a webcam and draw fading ribbons from them in three.js. Twist: ribbons play a note whenever two of them cross. ### Jayson Haebich *Light and laser artist* Artist who writes his own openFrameworks laser software to draw beams through haze and make them respond to people tracked by a depth camera. #### Laser interaction prototype — Jayson Haebich (2012) - Video: https://vimeo.com/44904580 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Projection, openFrameworks, lasers, Kinect - Idea: Laser beams you can touch. - What it is: Laser beams cut through haze to form planes of light, and they bend and react when a person reaches into them. - Technique: Custom openFrameworks software sends vectors to the laser via a DAC, and a Kinect calibrated to the laser's 3D field detects when a hand intersects a beam. - Try it: In a hazy room (or with a projector and a fog of dust), project thin lines through the air and use a webcam to detect when a hand blocks one, which then changes it. Twist: each line plays a note when touched. #### Elastic Light — Jayson Haebich (2014) - Video: https://vimeo.com/103230160 - Interaction: Hands & Body, Play, Perception & Effects - Platform & tech: Projection, openFrameworks, lasers, depth camera - Idea: A gesture interface made of pure light in mid-air. - What it is: A series of simple puzzles made from laser lines in the air: visitors rotate, stretch and warp the beams with their hands to complete each shape. - Technique: Laser vectors are drawn through haze by openFrameworks software, and depth-camera hand tracking maps gestures onto beam transformations. - Try it: Design three AR puzzles where a floating line must be rotated or stretched by hand tracking to match a target outline. Twist: the line resists like a rubber band and snaps back if you let go too early. #### The Diffusion of Light — Jayson Haebich (2015) - Video: https://vimeo.com/127403161 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Projection, lasers - Idea: One beam of white light turned into a room full of colour by broken glass. - What it is: A white laser beam sweeps across a large pile of broken glass and mirrors, splitting into coloured reflections that move across the whole room. - Technique: Programmed laser paths are aimed across the glass pile so refraction and reflection scatter the light around the space. - Try it: Shine a moving phone flashlight or projector spot across a pile of clear objects and film how the room changes; then script the path to 'play' the reflections. Twist: add one coloured object and make it the star of the piece. ### Jennifer Townley *Mechanical sculptor* Dutch artist who designs precise mechanical sculptures of gears, spirals and repeated parts whose slightly different rotation speeds produce slowly shifting, hypnotic patterns, as well as drawing machines. #### Cubes — Jennifer Townley (2013) - Video: https://www.youtube.com/watch?v=PQnHcoyhBs0 - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, gears, cubes, motor - Idea: Slow phase drift between identical parts creates a narrative arc of order, chaos and order again, a simple structure for time-based AR pieces. - What it is: A column of cubes turns slowly, each at a slightly different speed, so the stack twists into ever-changing spirals and then realigns into a straight tower. - Technique: A gear train gives each cube a marginally different rotation rate, so the stack only returns to alignment after a long common period (year approximate). - Try it: Stack 20 virtual cubes on a real shelf in AR, rotating each at 1 + i/100 speed, and time how long until they realign. Twist: make the tower chime when it becomes straight again. #### Asinas — Jennifer Townley (2015) - Video: https://www.youtube.com/watch?v=P6EvXt-C9LI - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, gears, aluminium, motor - Idea: Apparent motion can emerge from two rotations; AR designers can create flowing effects without moving objects, just by phasing their parts. - What it is: Two interlocking spirals of white segments rotate at slightly different speeds, so a wave appears to travel through the sculpture while nothing actually moves forward. - Technique: A single motor drives two concentric spiral assemblies through gears with a slightly different ratio, so their relative phase drifts and creates travelling interference. - Try it: Build two nested virtual helices in phone AR rotating at speeds 1.00 and 1.03 and watch the moiré wave appear. Twist: let the viewer's tilt change the ratio. #### Phaser — Jennifer Townley (2019) - Video: https://www.youtube.com/watch?v=vjtY-Sykf_Q - Interaction: Drawing & Making, Perception & Effects, Tangible Objects - Platform & tech: Desktop, drawing machine, gears, pen - Idea: The drawing is a record of motion over time; AR can leave persistent trails that turn movement into a lasting artefact. - What it is: A drawing machine whose linked rotating arms guide a pen to trace dense, evolving geometric patterns on paper, recording the machine's motion as a drawing. - Technique: Two or more rotating arms with slightly different periods combine like a harmonograph, so the pen path slowly precesses into a complex figure (year approximate). - Try it: Attach a virtual pen to two rotating virtual arms on a real table in phone AR and let it draw a persistent line. Twist: let viewers nudge one arm's speed with a slider and compare drawings. ### Jesús Rafael Soto *Artist; kinetic and op art pioneer* Venezuelan artist (1923–2005) of the Paris kinetic movement, known for 'Vibrations' that shimmer as you move and for 'Penetrables', rooms of hanging threads or tubes that visitors walk through. #### Vibration (Blue and Black) — Jesús Rafael Soto (1966) - Video: https://www.youtube.com/watch?v=n8eaoKmo9TY - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Desktop, painted wood, metal rods, fine-line ground - Idea: Motion parallax between a near layer and a fine patterned layer creates a shimmering effect that only exists while the viewer moves, a cheap trick for making AR overlays feel alive. - What it is: Thin metal rods hang in front of a panel of fine black and white lines; as the viewer moves, the rods seem to dissolve and vibrate against the striped ground. - Technique: The fine line spacing of the background is close to the width of the rods, so small parallax shifts produce moiré-like interference that reads as vibration. - Try it: Place a virtual striped panel on a real wall and hang a few virtual rods 20 cm in front of it in AR, then tune the line spacing until walking past makes them shimmer. Twist: use the phone's gyroscope to add the same effect for a seated viewer. #### Penetrable BBL Bleu — Jesús Rafael Soto (1999) - Video: https://www.youtube.com/watch?v=7zW2y9tqJBo - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, PVC tubes, steel frame - Idea: The artwork is completed by bodies moving through it; in AR, letting virtual strands part and swing around the user makes presence physical. - What it is: A cube of hanging blue tubes that visitors push through; the tubes brush, swing and click around the body, and the moving crowd makes the whole block ripple. - Technique: Dense rows of flexible tubes hang from a ceiling grid so that touch and movement propagate as visible waves through the volume. - Try it: Create a Lens Studio or Unity AR 'curtain' of blue strands with simple spring physics that part around the camera as the user walks through. Twist: record each visitor's path as a slow ripple that stays visible for the next person. #### Houston Penetrable — Jesús Rafael Soto (2014) - Video: https://www.youtube.com/watch?v=976Ghk1Ue4E - Interaction: Hands & Body, Perception & Effects, Spatial Mapping - Platform & tech: Desktop, PVC tubes, hand-painted ink, suspension grid - Idea: A volume that is solid from afar and dissolves when you enter it: AR can hide shapes in point grids that only resolve from certain distances or angles. - What it is: Some 24,000 hanging PVC tubes fill a hall; parts of them are painted yellow so that from a distance a floating ellipsoid appears, and it dissolves into rain-like strands once visitors walk inside. - Technique: The hidden shape is painted segment by segment onto thousands of vertical tubes, so the ellipsoid reads as a volume only when the tubes are seen together from outside. - Try it: Build a WebXR or AR Foundation scene of a few thousand thin vertical lines anchored in a room, colouring the segments that fall inside a virtual sphere so the sphere appears from afar and disappears as the user walks in. Twist: make the lines sway and chime when the phone passes through them. ### Jiabao Li *Artist; assistant professor at UT Austin* Chinese-born artist and designer whose work uses AR, projection and performance to let people perceive the world through bats, glaciers and plankton. #### Nocturnal Fugue (with Matt McCorkle & Botao Hu) — Jiabao Li, Botao 'Amber' Hu (2024) - Video: https://www.youtube.com/watch?v=npzhJdvyLew - Interaction: Voice & Sound, Location & City, Perception & Effects - Platform & tech: Headset, Phone, HoloKit, spatial audio, mixed reality - Idea: On a night under a bat bridge, use sound and MR headsets to experience how bats perceive the world. - What it is: A multisensory night journey beginning under Austin's Congress Bat Bridge, combining bat stories, sound and the EchoVision mixed-reality headsets to experience the bats' world. - Technique: Location-based sound composition is combined with HoloKit-based EchoVision headsets that visualise the participant's voice as echoes in the dark. - Try it: Design a nighttime 'sound walk' route across campus that plays animal sounds on a phone at each stop and shows the sound waves in AR. Twist: participants must use their own voices to 'echolocate' the next stop. #### EchoVision (Ars Electronica Deep Space 8K edition) — Jiabao Li, Botao 'Amber' Hu (2025) - Video: https://www.youtube.com/watch?v=W0TSj-LilLw - Interaction: Voice & Sound, Projection, Perception & Effects - Platform & tech: Projection, audio analysis, Deep Space 8K projection - Idea: On a giant wall and floor, visitors' calls turn into bat-like echolocation waves. - What it is: A projection edition of EchoVision in Ars Electronica's Deep Space 8K: visitors' voices become visualised echolocation waves across the giant wall and floor, alongside AI-composed bat songs. - Technique: Microphone input is analysed for pitch and loudness and rendered as expanding echo waves in the 8K wall-and-floor projection, likely with floor tracking of visitors. - Try it: Project onto a classroom floor and use a microphone to turn shouts into ripples spreading from the center that bounce off 'obstacles' (cardboard boxes placed on the floor). Twist: only low-pitched sounds can reveal distant obstacles. #### The Long Fall: A descent into the Ocean's Living Memory — Jiabao Li (2025) - Video: https://www.youtube.com/watch?v=fe-Xd3gxcsU - Interaction: Projection, Performance, Information & UI - Platform & tech: Projection, immersive projection, microscopy, spatial audio - Idea: Sink down with the plankton and see how the ocean remembers carbon. - What it is: A 15-minute audiovisual performance (with Manu Prakash and others) that takes the audience on an immersive descent with plankton, projected across wall and floor. - Technique: Microscopy footage and simulations of plankton are composed into a live wall-and-floor projection performance with spatial sound. - Try it: Shoot water droplets or plant cells with a phone macro lens and project the footage onto the floor as a visual 'descent' narrative. Twist: when viewers walk into different depth zones (marked on the floor), different sounds play. ### John Cleater *Architect and AR artist (Cleater Studio)* Architect who began using the Layar AR browser in 2011 to float animated 'Sky Pavilions' over the High Line, Venice and his own garden, and who curated Peeling Layers of Space Out of Thin Air, an open-field show of AR architecture at Omi International Arts Center. #### Peeling Layers of Space Out of Thin Air: AR by Architects — John Cleater (2011) - Video: https://www.youtube.com/watch?v=ieA_J02l1Sc - Interaction: Location & City, Shared & Social - Platform & tech: Phone, Layar, GPS - Idea: An exhibition of buildings with no materials, open for a century. - What it is: A group exhibition of AR architecture scattered over a 30-acre field at Omi International Arts Center, with a stated run time of 100 years, visited with phones. - Technique: Architects' 3D models are geolocated as Layar points of interest across the field, so visitors walk the landscape to discover each virtual structure. - Try it: Curate a class AR exhibition in one outdoor space where each student places one virtual structure with location anchors. Twist: set an expiry date after which each piece disappears. #### Sky Pavilions — John Cleater (2011) - Video: https://www.youtube.com/watch?v=7HJt6DBB8uE - Interaction: Location & City, Performance - Platform & tech: Phone, Layar, PorPOISe - Idea: Architecture that flies in, lands and leaves like weather. - What it is: Virtual 'cloudburst' pavilions filled with nonsensical and practical guidance float over the High Line in New York, land in gardens and appear in Venice; in one live performance the band Dewanatron grapples with them. - Technique: Animated 3D pavilions (about 3,000 polygons with animated textures) are served to the Layar browser through PorPOISe and placed as geolocated points of interest with tap-triggered actions. - Try it: Design a small AR pavilion that 'lands' on a campus lawn with an animation, and add one tap interaction that gives advice. Twist: it takes off again when too many people gather. #### AR Tornado — John Cleater (2015) - Video: https://www.youtube.com/watch?v=hAVpCTmMuEY - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, mobile AR, GPS - Idea: Bring a disaster into a peaceful landscape at true scale. - What it is: On a hilltop overlooking the Catskill Mountains in Columbia County, New York, a virtual tornado spins across an otherwise beautiful day. - Technique: An animated tornado model is likely geolocated in the landscape so it stays fixed to a spot on the horizon as the viewer pans the phone. - Try it: Build a large-scale AR weather event (tornado, storm cloud, aurora) anchored far away on the horizon with a compass-based placement. Twist: it moves closer when the real wind picks up. ### Jonathan Thorpe (Sonosthesia) *XR developer; software architect at Opuscope; author of Sonosthesia* An XR and mobile developer since 2013 and software architect at Opuscope in Paris. Through his open-source Sonosthesia packages for Unity he turns audio descriptors, MIDI and hand gestures into signals that drive shaders, VFX Graph and mesh deformation. #### Pollen VFX Composition — Jonathan Thorpe (Sonosthesia) (2024) - Video: https://www.youtube.com/watch?v=MOeGDgMatwg - Source code: https://github.com/jbat100/sonosthesia-unity-demo-deform - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Sonosthesia - Idea: Grow an organic form whose opening and shedding are timed by sound. - What it is: Petal-like ribbons open and turn like a flower while clouds of pollen particles drift and flare with the rhythm of the music. - Technique: Spectral band energies from the track drive procedural mesh deformation of the petals and VFX Graph spawn rates for the pollen through Sonosthesia's mapping packages; it runs at about 260 fps on an M3 Pro. - Try it: Place an AR flower on a real plant pot and let the phone's microphone decide when it opens and releases pollen. Twist: the flower closes again when the room goes quiet. #### Sonosthesia: GPU Sound Visualisation — Jonathan Thorpe (Sonosthesia) (2024) - Video: https://x.com/johnnyfrenchy/status/1752349779223330997 - Source code: https://github.com/jbat100/sonosthesia-unity-demo-deform - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, Unity, Shader Graph, Sonosthesia - Idea: Break sound into a few descriptor streams and wire each one to a visual parameter. - What it is: A glowing ring of light wobbles, swells and changes colour in real time with the music, prototyped as a sound-controlled object for XR. - Technique: Sonosthesia extracts energy in several spectral bands from the audio source and streams these descriptors into Shader Graph properties that displace and colour the mesh every frame. - Try it: Split the phone microphone into bass, mid and treble bands and map each to one property of a Shader Graph ring floating in AR. Twist: make the ring react only to the voice of the person looking at it. #### Spark VFX Composition — Jonathan Thorpe (Sonosthesia) (2024) - Video: https://www.youtube.com/watch?v=_2siOZ5pIWQ - Source code: https://github.com/jbat100/sonosthesia-unity-demo-deform - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Shader Graph, Sonosthesia - Idea: Let each part of the music own one part of the effect: the core, the sparks, the glow. - What it is: A deforming sphere throws out branching electric sparks that crackle and pulse in sync with a piece of music. - Technique: Audio descriptors from a Timeline-played track are routed through Sonosthesia signal packages to mesh deformation on the sphere and to VFX Graph properties that spawn and shape the spark strips; it runs at about 98 fps on an M1 Pro. - Try it: Anchor a sparking orb to a real speaker in AR and drive its sparks from the phone microphone's onsets. Twist: sparks jump towards whoever is standing closest. ### Joon Moon (Joon Y. Moon / 문준용) *Media artist; creator of the Augmented Shadow series* South Korean media artist and engineer whose Augmented Shadow series (since 2010) mixes real objects and light with computer-generated shadows; the first piece was shown in MoMA's 2011 Talk to Me exhibition. #### Augmented Shadow — Joon Moon (Joon Y. Moon / 문준용) (2010) - Video: https://www.youtube.com/watch?v=0arZMuPK58w - Interaction: Tangible Objects, Projection, Perception & Effects - Platform & tech: Projection, openFrameworks, projector, tabletop tracking - Idea: Fake the shadows of real objects so a hidden story can live inside them. - What it is: A tabletop where visitors move real white blocks under a light; the shadows they cast are computer-generated and grow little houses, trees and characters that live only in the shadow world. - Technique: Tracked tabletop objects drive an openFrameworks renderer that projects computed shadow shapes from a virtual light, so shadow and object stay aligned as blocks move. - Try it: Track two or three paper boxes on a table with a webcam and project fake shadows from a virtual lamp with a projector; add one shadow that behaves differently from its box. Twist: let the shadows walk away when nobody touches the boxes. #### Hello, Shadow! — Joon Moon (Joon Y. Moon / 문준용) (2019) - Video: https://www.youtube.com/watch?v=RbwEf1QGA8U - Interaction: Tangible Objects, Projection, Perception & Effects - Platform & tech: Projection, projector, tracked lamp, real-time graphics - Idea: The light in your hand becomes a lens that reveals shadows the object does not actually cast. - What it is: Visitors turn a handheld lighting device around a sculpture; the shadow on the wall is computer-generated and changes into unexpected shapes depending on the angle of the light. - Technique: The pose of the handheld light is tracked and a shadow is rendered from that virtual light position onto the wall, likely with a projector aligned to the real lamp's frustum. - Try it: Use a phone flashlight as a 'lamp', track its angle with the phone's gyroscope, and render a shadow on a projected wall that morphs between two silhouettes as the angle changes. Twist: hide a secret shape that only appears at one exact angle. #### Augmented Shadow: Chasing Stars in Shadow — Joon Moon (Joon Y. Moon / 문준용) (2022) - Video: https://vimeo.com/782602226 - Interaction: Projection, Tangible Objects, Play - Platform & tech: Projection, projector, tracked lights, real-time graphics - Idea: Scale a tabletop shadow trick into a room-sized story that visitors light up themselves. - What it is: An immersive room where visitors carry lanterns among white objects; shadow children cross between 2D shadows and 3D objects, fishing and chasing stars as the lights move. - Technique: Several tracked handheld lights act as virtual light sources; projectors render consistent synthetic shadows and animated characters on walls and floors in real time. - Try it: In a dark corner, give two students tracked flashlights and project a shared shadow scene where a character runs toward whichever light is closer. Twist: the character only becomes 3D (a physical cut-out) when both lights meet. ### Julian Triveri (anaglyph) *Solo developer of Lasertag for Quest* Independent developer who builds Lasertag, a co-located mixed-reality laser tag game for Quest 3 that uses real-time depth to map rooms. #### Lasertag: light VFX test — Julian Triveri (anaglyph) (2024) - Video: https://www.youtube.com/watch?v=yxDhCg-q0JU - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Headset, Meta Quest 3, Depth API - Idea: A lighting test in which virtual lasers light up the real walls of a room. - What it is: A test of laser shots that light up the real room: virtual beams cast light onto scanned walls and furniture in passthrough. - Technique: Virtual dynamic lights attached to laser beams illuminate an invisible scanned-room mesh with a shadow-receiving or additive-lighting material, so light appears to fall on real surfaces in passthrough. - Try it: Give the scanned room mesh a transparent material that still receives light, so a virtual flashlight casts a spot of light on real walls. Twist: change the color of the light spot with the volume of your voice. #### Lasertag — Julian Triveri (anaglyph) (2025) - Video: https://www.youtube.com/watch?v=T1PIcuOYmXU - Interaction: Shared & Social, Spatial Mapping, Play - Platform & tech: Headset, Meta Quest 3, Depth API, Unity - Idea: Turn any space into a same-room multiplayer laser-tag arena where real walls block the beams. - What it is: A mixed-reality laser tag game that scans your space and turns it into an arena for friends in the same room; beams stop at real walls and furniture. - Technique: The Depth API provides real-time depth, so laser raycasts are truncated at the reconstructed surfaces and real walls and furniture block shots. - Try it: Use AR Foundation occlusion (depth) so that AR beams are cut off at real objects, and build a prototype where players duel from behind a cabinet. Twist: real surfaces that get hit keep a glowing spot for a few seconds. #### Lasertag: synced room scanning — Julian Triveri (anaglyph) (2025) - Video: https://www.youtube.com/watch?v=oJ_fFONAEQ4 - Interaction: Spatial Mapping, Shared & Social - Platform & tech: Headset, Meta Quest 3, Depth API - Idea: Several headsets scan the same room together in real time, building one shared model. - What it is: Demo of multiple Quest headsets building one shared depth reconstruction of a room in real time, so all players fight inside the same scanned geometry. - Technique: Each headset fuses depth frames into a shared voxel or mesh reconstruction in a common anchor frame, and updates are merged over the network so all players see the same geometry. - Try it: Have two LiDAR iPhones each scan half of a room, align them with a shared image marker, and merge the results into one mesh. Twist: color the scanned mesh in two colors to show who scanned which part. ### Koki Ibukuro (asus4) *Freelance creative technologist* Japanese creative technologist in Berlin who maintains the TensorFlow Lite and ONNX Runtime plugins for Unity and builds AR experiments that run neural networks on the phone camera in real time. #### MediaPipe FaceMesh on Unity — Koki Ibukuro (asus4) (2020) - Video: https://www.youtube.com/watch?v=ctFZjSzx6Vc - Source code: https://github.com/asus4/tf-lite-unity-sample - Interaction: Face - Platform & tech: Phone, Desktop, Unity, TensorFlow Lite, MediaPipe - Idea: A face filter engine you own: run the face model yourself instead of relying on a platform's lens tools. - What it is: A dense face mesh from MediaPipe tracks a person's face in the camera image in real time inside Unity, ready to be textured, lit or distorted. - Technique: BlazeFace detects the face and the MediaPipe Face Mesh TensorFlow Lite model regresses 468 landmarks, which the tf-lite-unity-sample project turns into a Unity mesh every frame. - Try it: Take the Face Mesh sample and paint the mesh with an animated shader (war paint, glowing veins, a glitch). Twist: detect a mouth-open gesture from the landmarks and emit particles from the lips. #### World Ensemble — Koki Ibukuro (asus4) (2023) - Video: https://www.youtube.com/watch?v=adra9QppRO0 - Source code: https://github.com/asus4/WorldEnsemble - Interaction: Voice & Sound, Location & City - Platform & tech: Phone, Unity, AR Foundation, ARCore Geospatial, Photorealistic 3D Tiles - Idea: Use the geometry of the city as a sequencer: every building you tag becomes a voice in the song. - What it is: Walking through a city with a phone, the user taps buildings to place instruments on them and taps the sky to switch the music and visual effects, so the street becomes a playable score. - Technique: ARCore Geospatial and Photorealistic 3D Tiles localize the phone and provide building geometry; taps likely raycast against the tiles to anchor instruments, and the visual effects switch with the music (AR Foundation Replay is used to test walks in the editor). - Try it: Record an AR Foundation replay of a walk around campus and map three buildings to three loops that start when you look at them. Twist: make the visual effect on each facade follow that building's loop. #### Myaku Myaku AR — Koki Ibukuro (asus4) (2025) - Video: https://github.com/user-attachments/assets/03961874-421f-41bc-a1d8-f9e0e07b4fe4 - Source code: https://github.com/asus4/MyakuMyakuAR - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Phone, Unity, AR Foundation, ONNX Runtime, YOLO11 - Idea: Segment real objects and re-skin them as a character, so the world itself becomes the costume. - What it is: Point the phone at a vase of flowers and it turns into a wobbling, glossy red-and-blue blob covered in eyes, the look of the Expo 2025 mascot Myaku-Myaku, while the rest of the room stays real. - Technique: A YOLO11 segmentation model runs on-device (likely through his own ONNX Runtime plugin for Unity), and the object masks, likely combined with AR depth, are covered by a noisy, glossy blob shader with eyes. - Try it: Run a segmentation model on the AR camera and replace one object class (cups, plants, chairs) with a material of your choice. Twist: make the material react to the object's label, so every class gets its own creature. ### Lauren Lee McCarthy *Artist; creator of p5.js; Professor, UCLA Design Media Arts* Artist examining social relationships amid surveillance, automation and algorithmic living; creator of the p5.js library and former SFPC teacher. #### us+ — Lauren Lee McCarthy, Kyle McDonald (2013) - Video: https://vimeo.com/81903116 - Interaction: Face, Shared & Social, Information & UI - Platform & tech: Desktop, Web, face tracking, speech analysis, Google Hangouts - Idea: An AI "social coach" in video calls that reads your face and words and nudges you on screen in real time. - What it is: A Google Hangout app that analyzes speech and facial expressions during a video call and overlays live feedback and automatic interventions to 'improve' the conversation. - Technique: Webcam face tracking and speech-to-text with sentiment/word-count analysis run during the video call, and rule-based triggers overlay text prompts and auto-mute when thresholds are crossed. - Try it: Build a video-chat "coach" web page with the Web Speech API and MediaPipe that tracks who talks most and who is smiling, and pops up prompts. Twist: add one excessive automatic intervention rule (for example, you get muted if you say "I" too often) and experience a conversation managed by an algorithm. #### Unlearning Language — Lauren Lee McCarthy, Kyle McDonald (2022) - Video: https://vimeo.com/780628671 - Interaction: Hands & Body, Voice & Sound, Shared & Social - Platform & tech: Projection, machine learning, computer vision - Idea: In a room watched constantly by AI, people have to invent new ways of communicating that machines cannot read. - What it is: An installation and performance where an AI tracks participants' speech, gestures and expressions and responds with light and sound, pushing them to invent ways of communicating that algorithms cannot detect. - Technique: Machine-learning classifiers (pose estimation, facial expression, speech recognition) detect recognisable human signals, and detection events drive light and sound feedback that participants try to evade. - Try it: Use MediaPipe and Teachable Machine to build a system that recognizes nods, waves and smiles and flashes an alarm light when it does, and have your group invent a way to communicate that it cannot detect. Twist: after one minute of undetected communication, teach your secret code to another group. #### Voice In My Head — Lauren Lee McCarthy, Kyle McDonald (2023) - Video: https://www.youtube.com/watch?v=B2-dV8IrhWo - Interaction: Voice & Sound, Shared & Social - Platform & tech: Wearable, LLM, speech recognition, earpiece - Idea: An AI inner voice in your earpiece whispers live advice while you talk to people. - What it is: Participants wear an earpiece through which an AI voice listens to their conversations and whispers real-time guidance, replacing their inner monologue as they move through social situations. - Technique: Speech recognition transcribes the wearer's surroundings, an LLM generates short suggestions conditioned on a personality profile, and text-to-speech delivers them through an earpiece with low latency. - Try it: Use speech recognition in a phone web page plus an LLM API to summarize a conversation into one suggestion, and have TTS read it quietly into your earphones. Twist: give the voice a personality that is the exact opposite of yours, chat with a classmate for 5 minutes while wearing it, then reflect. ### Leap Motion Design Research (Barrett Fox, Martin Schubert, Keiichi Matsuda et al.) *Hand-tracking interaction design team at Leap Motion / Ultraleap* Leap Motion's small design research group published a stream of hand-interaction experiments and built Project North Star, an open-source wide-field AR headset. #### Practice Makes Perfect (North Star) — Leap Motion Design Research (Barrett Fox, Martin Schubert, Keiichi Matsuda et al.) (2018) - Video: https://www.youtube.com/watch?v=6RV0jGXzjTs - Interaction: Hands & Body, Play, Information & UI - Platform & tech: Headset, Project North Star, Leap Motion hand tracking, object tracking - Idea: A real table and paddle, a virtual ball and an AI opponent, plus predicted trajectories to help you practice. - What it is: On the Project North Star headset a player uses a real table and paddle; a hand gesture summons a virtual ball, an AI opponent returns it, and predicted flight paths and light trails coach the player's form. - Technique: Tracks the paddle and hands, simulates the virtual ball with parabolic equations of motion, and renders predicted trajectories with the headset's low latency and wide field of view. - Try it: Mark a 'best landing spot' target zone on a table in phone AR and score when classmates hit it with a paper ball. Twist: the target moves after each hit and shows the path of the previous throw. #### Project North Star: Desk UI — Leap Motion Design Research (Barrett Fox, Martin Schubert, Keiichi Matsuda et al.) (2018) - Video: https://www.youtube.com/watch?v=6dB1IRg3Qls - Interaction: Hands & Body, Information & UI, Spatial Mapping - Platform & tech: Headset, Project North Star, Leap Motion, Unity - Idea: Attach holographic interfaces to a real desk and touch them like physical objects. - What it is: A North Star prototype that attaches holographic interfaces to a real desk, with virtual panels and widgets that you touch and manipulate as if they were physical desk objects. - Technique: UI panels are anchored to a calibrated desk surface and respond to fingertip proximity and press depth, so virtual widgets behave like physical objects on the desk. - Try it: Place a small AR drawer on the desk that reveals today's to-do list when pulled open with a finger. Twist: turn each to-do item into a paper ball you can pick up and throw into a trash can. #### Project North Star: Exploring Augmented Reality — Leap Motion Design Research (Barrett Fox, Martin Schubert, Keiichi Matsuda et al.) (2018) - Video: https://www.youtube.com/watch?v=7m6J8W6Ib4w - Interaction: Hands & Body, Information & UI - Platform & tech: Headset, Project North Star, Leap Motion, Unity - Idea: Use an open-source wide-field AR headset to explore an interaction language for handling holograms with bare hands. - What it is: Footage shot through the North Star AR headset showing bare-hand interactions: virtual objects, menus and effects you grab, pinch and swipe with tracked hands. - Technique: Leap Motion hand tracking provides articulated hand skeletons, and physics-based grab and pinch detection lets users manipulate virtual objects without controllers in an optical see-through display. - Try it: Use MediaPipe Hands or HoloKit gesture recognition to make an AR cube you can pick up with a pinch and throw with a wave. Twist: a grabbed object leaves a trail that follows the speed of your hand. ### Lee Vermeulen *Game developer (Alientrap, Modbox); XR physics prototyper* Co-founder of the indie studio Alientrap and creator of the VR/MR sandbox Modbox, he posts short passthrough-AR prototypes that combine hand tracking with liquid physics, portals and mirrors. #### AR liquid physics puzzle game prototype — Lee Vermeulen (2023) - Video: https://www.youtube.com/watch?v=hL9ylHVagIM - Interaction: Hands & Body, Play, Spatial Mapping - Platform & tech: Headset, Unity, Varjo XR-3, Ultraleap, Zibra Liquids - Idea: A classic marble-run puzzle becomes tactile when the liquid shares your real desk and hands. - What it is: In passthrough AR on a desk, the player builds ramps and blocks with bare hands to guide a stream of simulated liquid into a bucket. - Technique: A GPU fluid solver (ZibraAI) runs in Unity with Ultraleap hand colliders, rendered over Varjo XR-3 passthrough so the liquid appears on the real table. - Try it: Build a Quest passthrough or phone AR scene where particles fall from a spout and must reach a cup; let players place three planes to redirect them. Twist: the planes can only be placed on real surfaces detected by plane tracking. #### Portals in the liquid physics AR puzzle — Lee Vermeulen (2023) - Video: https://www.youtube.com/watch?v=WzWgDNzf5lQ - Interaction: Hands & Body, Portals & Worlds, Play - Platform & tech: Headset, Unity, Varjo XR-3, Ultraleap, Zibra Liquids - Idea: Portals make space itself a puzzle piece in your own room. - What it is: Floating portal pairs are added to the AR liquid game; water poured into one ring pours out of another across the room, and hands can reach through the portals too. - Technique: Render-to-texture portal cameras (based on Tom Goethals' VR portal code) teleport fluid particles and hand colliders between linked transforms. - Try it: Create two linked portal rings in AR Foundation and teleport any ball that enters one to the other with preserved velocity. Twist: place one portal on the ceiling so gravity keeps the ball looping. #### Sculpting Zero G Liquid in AR — Lee Vermeulen (2023) - Video: https://www.youtube.com/watch?v=UTbsPqY5v1E - Interaction: Hands & Body, Drawing & Making - Platform & tech: Headset, Unity, Varjo XR-3, Ultraleap, Zibra Effects, compute shader - Idea: Turn a fluid simulation into a sculpting material by making it forget to flow. - What it is: Floating blobs of weightless liquid hang in the room and can be pushed and shaped with the hands, then stay where they are left like soft clay. - Technique: A custom compute shader damps particle velocity in a zero-gravity fluid sim so hand-tracked pushes leave persistent shapes. - Try it: Spawn a cloud of particles in AR with no gravity and strong damping; let the hand (or phone position) push particles away within a radius. Twist: slowly restore gravity over one minute so the sculpture melts. ### Leo Villareal *Artist; LED light sculpture* American artist who writes custom software to animate thousands of LEDs in never-repeating sequences, most famously on the San Francisco Bay Bridge in The Bay Lights. #### Multiverse — Leo Villareal (2008) - Video: https://www.youtube.com/watch?v=_TMobatWhok - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Projection, LEDs, tunnel, custom software - Idea: Light moving faster or slower than the viewer changes the sense of speed; AR motion cues in a corridor can make people feel they are accelerating or floating. - What it is: In the underground walkway between two buildings of the National Gallery of Art in Washington, about 41,000 LEDs run along the ceiling and walls, sending pulses of light that race past people on the moving walkway. - Technique: Programmable LED strips run in channels along the tunnel, and custom sequencing software produces random, never-repeating light movements. - Try it: Line a real corridor with virtual light strips in AR Foundation that move in or against the user's walking direction at speeds tied to their pace. Twist: make the lights race ahead to guide a newcomer to a room. #### Buckyball — Leo Villareal (2012) - Video: https://www.youtube.com/watch?v=gyDJArirU7I - Interaction: Perception & Effects, Location & City - Platform & tech: Projection, LED tubes, geodesic sphere, custom software - Idea: A molecule enlarged to the size of a pavilion becomes a gathering place; AR can blow up tiny scientific structures into public sculptures that people walk around. - What it is: Two nested geodesic spheres made of LED tubes stand in Madison Square Park; the tubes change colour in endless sequences that glow over the lawn at night. - Technique: Each edge of the two spheres is an LED tube, and software maps generative colour sequences onto the geometry of the carbon-60 molecule. - Try it: Place a room-sized AR model of a molecule (C60, caffeine or DNA) in the schoolyard and animate colour waves along its bonds. Twist: let students walk inside it and trigger a sound when they touch a node. #### The Bay Lights — Leo Villareal (2013) - Video: https://www.youtube.com/watch?v=dcTkhDMyIk4 - Interaction: Location & City, Perception & Effects, Information & UI - Platform & tech: Projection, LEDs, custom software, bridge cables - Idea: An existing structure's own geometry can be the display; AR at city scale works best when it follows the lines of real infrastructure. - What it is: About 25,000 white LEDs strung along the vertical cables of the San Francisco–Oakland Bay Bridge play endless non-repeating patterns of light across the water. - Technique: Individually addressable LEDs are attached to each suspender cable and driven by generative software whose patterns are likely inspired by wind, water and traffic. - Try it: Pick a real railing, fence or row of columns and attach AR light points along its lines with plane or image tracking, then animate slow generative waves through them. Twist: make passing people push the waves in their walking direction. ### Lily Hassioti *Sound and installation artist* Greek artist trained in computational arts at Goldsmiths who builds responsive sound installations out of plants, motors, metal sheets and thermal cameras. With Samuele Albani she made Passaggi // Presenze, where people passing through doors and moving through a room compose the sound together. #### Evolution of the Garden — Lily Hassioti (2019) - Video: https://www.youtube.com/watch?v=V6MjBPvO5gU - Interaction: Tangible Objects, Voice & Sound - Platform & tech: Projection, Makey Makey, analogue circuits, plants, speakers - Idea: A garden you play by touching its leaves. - What it is: A sound installation and performance at Athens Digital Arts Festival, made with Dimitra Kousteridou: touching potted plants wired into analogue and digital circuits adds voices to a growing soundscape. - Technique: Plants are connected to a Makey Makey and small analogue and digital circuits, so the body's touch closes a circuit that triggers or modulates sound through a mixer and speakers. - Try it: Wire three houseplants to a Makey Makey and map each to a sound loop in a browser sampler. Twist: a sound only plays while two people touch different plants and hold hands. #### Instrument for Dissonance — Lily Hassioti (2020) - Video: https://www.youtube.com/watch?v=7j4NHuRxjGU - Interaction: Hands & Body, Voice & Sound, Tangible Objects - Platform & tech: Projection, motors, metal sheets, presence sensor (likely) - Idea: A self-playing machine whose rhythm is interrupted by your presence. - What it is: Motors swing ropes against suspended metal sheets in anxious, repeating loops; when a person enters, their position in the room shortens the loops and resets the rhythm. - Technique: A presence or distance sensor (likely) reads where the visitor stands and changes the timing of the motors that swing the ropes onto the metal sheets. - Try it: Hang tin lids from a string, tap them with a servo on an Arduino in a steady loop, and let a distance sensor speed up the loop as someone approaches. Twist: the machine stops entirely if the visitor stands perfectly still. #### Thermal Improvisations — Lily Hassioti (2023) - Video: https://www.youtube.com/watch?v=A5ROdOCii0Y - Interaction: Perception & Effects, Voice & Sound, Projection - Platform & tech: Projection, thermal camera, projection, live sound - Idea: Play the room with body heat that only a thermal camera can see. - What it is: An audiovisual performance at Goldsmiths that explores the thermal landscape of a space: a performer moves in front of a thermal camera whose heat image is projected, and the image shapes the improvised sound. - Technique: A thermal camera feed is projected live and its temperature values are likely analysed in software to modulate sound in real time. - Try it: Clip a cheap phone thermal camera to a projector and project the live heat image; place a warm mug and ask students to leave handprints on the wall and watch them fade. Twist: map the hottest point's position to the pitch of a tone. ### Lusion (Edan Kwan & team) *Real-time 3D and WebXR studio co-founded by creative technologist Edan Kwan* Bristol studio known for award-winning WebGL sites and R&D; co-founder Edan Kwan also made the generative piece Bilibala Holala for the KIKK 2018 AR exhibit. Its AR work ranges from browser-based WebXR product placement to the Quest mixed-reality showcase Spatial Fusion with PHORIA and Meta. #### Real-time Raytracing x AR (Apollo 11) — Lusion (Edan Kwan & team) (2019) - Video: https://vimeo.com/348855347 - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Desktop, Unity HDRP, NVIDIA RTX, raytracing - Idea: Let objects from the film on your TV float out into your living room. - What it is: A tribute to the 50th anniversary of the Apollo 11 landing: while moon-landing footage plays on a living-room TV, raytraced moon rocks float around the screen and through the real room, as if they had escaped the footage. - Technique: Unity's HDRP with NVIDIA RTX raytracing renders reflective rocks composited over tracked camera footage (ARKit/ARCore-style tracking); likely an offline-tracked real-time render rather than a phone app. - Try it: Make something escape from a screen: play a short clip on a monitor and, with an image target on the monitor frame, spawn AR objects that fly out when the clip reaches a certain moment. Twist: the objects fly back when the video is paused. #### WebXR Sneakers Demo — Lusion (Edan Kwan & team) (2020) - Video: https://vimeo.com/413649522 - Interaction: Spatial Mapping, Information & UI - Platform & tech: Web, Phone, WebXR, three.js, hit-test - Idea: Let a web page put its product on your floor. - What it is: A mock shoe-shop web page where tapping a button places a photoreal sneaker on the floor in AR straight from the mobile browser, without an app or plug-in. - Technique: The WebXR Device API's immersive-ar session with hit-testing places a three.js-rendered model on detected surfaces in Chrome on Android. - Try it: Build a one-page WebXR 'gallery shop' with three.js or model-viewer where each student's 3D object can be placed on a real table. Twist: let the object change when placed on the floor instead of a table. #### Spatial Fusion — Lusion (Edan Kwan & team) (2022) - Video: https://www.youtube.com/watch?v=RikFV2EY-1Y - Interaction: Portals & Worlds, Spatial Mapping, Play - Platform & tech: Headset, Web, WebXR, three.js, Meta Quest Pro, plane detection - Idea: Open a portal in your own room straight from a web page. - What it is: A browser-based mixed-reality experience for Quest 2 and Quest Pro: a tear opens a portal in your room, and you use fusion energy to repair a gravity-defying spacecraft that appears among your real walls and furniture. - Technique: WebXR immersive-ar sessions with plane detection, spatial anchors and colour passthrough, built in three.js by PHORIA and Lusion with Meta; audio by Zelig. - Try it: Use WebXR plane detection (or AR Foundation) to find the largest wall and cut a portal into it that shows a different world. Twist: the portal only opens when two people stand in front of it. ### László Moholy-Nagy *Artist; Bauhaus teacher and pioneer of light art* Hungarian artist (1895–1946) and Bauhaus teacher who built the Light Prop for an Electric Stage (1930), a motorised machine of metal and glass that throws moving shadows and reflections around a room, and filmed it as Ein Lichtspiel Schwarz Weiss Grau. #### Ein Lichtspiel Schwarz Weiss Grau — László Moholy-Nagy (1930) - Video: https://www.youtube.com/watch?v=sRssg9hSYAI - Interaction: Perception & Effects - Platform & tech: Projection, 16 mm film, Light Prop - Idea: Filming a light machine with overlays and multiple exposures reveals motion as layered traces, which is how AR can show time and movement in a single frame. - What it is: A short abstract film of the Light Prop in motion, in which reflections, shadows and multiple exposures turn the machine into a flowing play of black, white and grey. - Technique: Close-ups, negative images and multiple exposures of the rotating machine layer successive moments of light over each other on film. - Try it: Make an AR camera filter that layers the last 30 frames of the live camera with decreasing opacity so moving lights leave multi-exposure trails. Twist: convert it to black, white and grey only and film a desk lamp spinning. #### Light Prop for an Electric Stage (Light-Space Modulator) — László Moholy-Nagy (1930) - Video: https://www.youtube.com/watch?v=QHdK19meZTk - Interaction: Projection, Perception & Effects - Platform & tech: Projection, metal, glass, motor, light bulbs - Idea: The object matters less than the light it throws into the room; in AR, the effect an object has on the space around it can be more important than the object itself. - What it is: A motorised construction of perforated metal, glass and rods that rotates under coloured lamps, throwing moving shadows and reflections across a darkened room. - Technique: Rotating perforated discs, a glass spiral and a sliding ball are lit by programmed coloured lamps, and their moving shadows form the actual artwork. - Try it: Build a virtual kinetic machine in AR that casts real-time shadows and light patterns onto the detected walls and floor, with the machine itself nearly invisible. Twist: let users only see the machine when they walk behind it. #### Room of the Present (Raum der Gegenwart) — László Moholy-Nagy (1930) - Video: https://www.youtube.com/watch?v=OZXl5Jfg07U - Interaction: Perception & Effects, Information & UI - Platform & tech: Projection, projection, photography, Light Prop replica, architecture - Idea: A room designed as a showcase for new media, where the space itself is part of the exhibit; AR exhibitions can treat the room as the interface. - What it is: A room Moholy-Nagy designed in 1930 to show new media (film, photography, a working Light Prop and architecture) together; it was realised for the first time in 2009 and shown at the Guggenheim. - Technique: The environment combines projections, moving displays, photographs and a replica Light Prop in an architectural layout meant to be experienced as a whole. - Try it: Design an AR 'room of the present' in a real room: pin a video on one wall, a 3D model on the table and a light machine on the ceiling, all linked to one timeline. Twist: let the room rearrange itself when a second visitor arrives. ### Matt DesLauriers *Generative artist and creative coder (canvas-sketch)* Canadian artist now in the UK who works with code and generative systems, author of the open-source canvas-sketch toolkit. Beyond prints and web works he builds public installations such as LUMOS in Toronto and TEMPEST on Reykjavik's Harpa facade, and made Quantum Spyglass for the KIKK 2018 AR exhibit. #### LUMOS — Matt DesLauriers (2018) - Video: https://vimeo.com/261406016 - Interaction: Hands & Body, Location & City, Perception & Effects - Platform & tech: Projection, thermal camera, LED, generative code - Idea: A winter beacon that turns warm when it feels your body heat. - What it is: A tall geometric light sculpture in a Toronto park glows cold blue when alone; as people approach, a thermal camera senses their body heat and the light inside shifts to warm reds, oranges and yellows. - Technique: A thermal imaging camera detects warm bodies near the sculpture, and generative light code maps the detected heat to the colour of LEDs inside (with Jean-Michel Gariepy and Steven Mengin). - Try it: Build a lamp that reacts to presence: use a webcam person detector (or a cheap PIR/thermal sensor with Arduino) to fade a projected or LED colour from blue to warm as people come closer. Twist: it only turns fully warm when two people stand together. #### AR Technical Demo: Virtual Plants — Matt DesLauriers (2019) - Video: https://vimeo.com/319348489 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, ARKit, procedural geometry - Idea: Grow a generative plant on your desk. - What it is: A short iOS 12 AR test in which a delicate stem of purple and white flowers grows and sways in real time on a wooden table next to a laptop. - Technique: Procedurally generated plant geometry is animated in real time and anchored to a detected horizontal plane with ARKit on iOS 12 (with Yi-Wen Lin). - Try it: Write an L-system plant in three.js and place it on a table with WebXR hit-test so it grows over ten seconds. Twist: each tap on the table plants a different random species. #### TEMPEST — Matt DesLauriers (2019) - Video: https://vimeo.com/317006348 - Interaction: Projection, Voice & Sound, Location & City - Platform & tech: Projection, LED facade, generative code, Web Audio - Idea: One button lets a passer-by throw lightning across a whole building. - What it is: On the LED facade of Reykjavik's Harpa concert hall, visitors push a button to set off blinding lightning flashes and loud thunder across the building, placing the audience inside a storm. - Technique: A real-time generative storm is rendered to the facade's low-resolution LED grid, and a physical button triggers lightning and synchronised thunder audio. - Try it: Map a p5.js storm onto a classroom window or wall with a projector and wire one big button (Makey Makey or keyboard key) that triggers lightning and thunder. Twist: the storm grows calmer the longer nobody presses it. ### Max Weisel *Co-founder of Normal (Normcore, Nock)* Designer-engineer who co-founded Normal, maker of the Normcore multiplayer framework and the VR sport Nock, and prototypes cross-device AR/VR interaction. #### Mixed Reality Test — Max Weisel (2018) - Video: https://www.youtube.com/watch?v=Um7qgKIlOSI - Interaction: Shared & Social, Portals & Worlds - Platform & tech: Phone, Headset, ARKit, Normcore, Unity - Idea: Let iPhone AR and a VR headset share the same multiplayer space. - What it is: A small demo mixing VR and AR: an ARKit iPhone and a VR headset share the same Normcore session so the phone sees the VR player's world placed in the real room. - Technique: A networked session (Normcore) shares object and player transforms, and the AR phone aligns its world origin to the VR play space so VR content appears in the real room. - Try it: Have a Unity scene on a computer and a phone AR app share the same set of object positions, so objects moved on the computer show up on the phone's tabletop in real time. Twist: the phone can push objects back the other way. #### iPhone AR in VR — Max Weisel (2019) - Video: https://www.youtube.com/watch?v=VsYw8WR81kw - Interaction: Shared & Social, Portals & Worlds - Platform & tech: Phone, Headset, ARKit, Normcore, Unity - Idea: Phone users join and watch a multiplayer VR space through AR. - What it is: A prototype that lets iOS users join and spectate a multiplayer VR space through ARKit, seeing the headset players' avatars in their own room. - Technique: An ARKit phone joins a multiplayer VR room as a spectator client, rendering the synchronized avatar transforms relative to an origin placed on a detected floor. - Try it: Build a spectator mode: one device plays a game while another phone uses AR to show the player's moves as a small figure on the table. Twist: spectators can drop props on the table that affect the game. #### Augmented Reality Gestures — Max Weisel (2020) - Video: https://www.youtube.com/watch?v=rPDislY6Yew - Interaction: Hands & Body, Information & UI - Platform & tech: Phone, ARKit, Unity - Idea: A set of 3D multi-touch gestures designed for phone AR. - What it is: A prototype of multi-touch gestures for navigating 3D AR content on a touch screen, with 3D pan, pinch and zoom mapped to the scene. - Technique: Multi-touch input is mapped to 3D transforms: one-finger drag pans along a plane, two-finger pinch scales, and twist rotates, with the mapping relative to the camera view. - Try it: Implement one-finger panning and two-finger scaling and rotation for a model in phone AR, and ask classmates to test which gesture is hardest to understand. Twist: design a new three-finger gesture. ### Maya Lin *Artist and architect* American artist and architect who designed the Vietnam Veterans Memorial as a student and makes earthworks and environmental memorials such as Wavefield and Ghost Forest. #### Vietnam Veterans Memorial — Maya Lin (1982) - Video: https://www.youtube.com/watch?v=wuxjTxxQUTs - Interaction: Location & City, Information & UI, Perception & Effects - Platform & tech: Desktop, polished black granite, engraved names, earth cut - Idea: Reflection puts the living among the names, and the wall grows taller as you walk in, a model for data AR that uses the viewer's body and path. - What it is: Two black granite walls cut into the ground of the National Mall in Washington meet in a V, engraved with the names of more than 58,000 Americans killed in the Vietnam War; visitors see their own reflection among the names. - Technique: The wall is set into a cut in the earth that deepens toward the apex; names are listed chronologically by date of death. - Try it: Build an AR wall that rises from the ground as the user walks along it and lists a dataset (for example, species lost in your region) in chronological order, with a mirror material. Twist: the names nearest the user's reflection become readable aloud. #### Storm King Wavefield — Maya Lin (2009) - Video: https://www.youtube.com/watch?v=JW0Cbrlyhcg - Interaction: Location & City, Hands & Body, Perception & Effects - Platform & tech: Desktop, earthwork, grass, gravel pit - Idea: A frozen water pattern made of earth lets people 'swim' with their bodies, a model for AR terrain that users walk through rather than look at. - What it is: On a former gravel pit at Storm King Art Center, seven rows of grass-covered earth waves, up to 4.5 metres tall, roll across 4.4 hectares; walking into them, you lose sight of everything but the next wave. - Technique: Earth was shaped into sine-like waves based on the pattern of water waves, then grassed over. - Try it: Generate an AR field of sine-wave ridges 1–2 metres high on a flat lawn with a height shader, and walk through it. Twist: let the waves slowly travel in the direction of the real wind. #### Ghost Forest — Maya Lin (2021) - Video: https://www.youtube.com/watch?v=0DHpGPwHm2I - Interaction: Location & City, Voice & Sound, Information & UI - Platform & tech: Desktop, dead Atlantic white cedars, soundscape - Idea: Bringing the evidence of climate change into a busy park makes a distant event physical, a direct brief for climate-data AR. - What it is: Forty-nine dead Atlantic white cedars, killed by saltwater from rising seas in New Jersey, stand in a grove in Madison Square Park, with a soundscape of animal calls from species once native to Manhattan. - Technique: Salvaged trees were transported and set upright in the park; an audio piece plays through a phone link for visitors. - Try it: Place a grove of grey AR trees in a school courtyard and attach sounds of species that lived there 200 years ago (from local records). Twist: the grove grows by one dead tree for every centimetre of projected sea-level rise. ### Meta Company (Meron Gribetz) *AR headset startup behind the Meta 1 and Meta 2 (2012–2019)* Founded by Meron Gribetz in 2012 with Steve Mann as chief scientist, Meta made the Meta 1 and Meta 2 see-through headsets with depth sensing for bare-hand interaction and promoted 'neuroscience-inspired' interfaces where you reach out and touch holograms. The company closed in 2019 and is unrelated to Facebook's later rename to Meta. #### Meta 1: Planets and 3D Printing — Meta Company (Meron Gribetz) (2014) - Video: https://www.youtube.com/watch?v=aypM1qaWPck - Interaction: Hands & Body, Drawing & Making, Tangible Objects - Platform & tech: Headset, Meta 1, Unity, 3D printing - Idea: Shape a hologram with your hands, then print it into a real object. - What it is: Through-the-glasses footage of the Meta 1 developer kit: the wearer holds a burning sun and planets in their palms, then sculpts a 3D object in mid-air with bare hands and sends it to a real 3D printer. - Technique: A depth camera on the glasses tracks hand and finger positions for 'ZeroUI' gestures in Unity, and the sculpted mesh is exported for 3D printing. - Try it: Build a hand-tracked AR scene where pinching and pulling deforms a sphere, then export it as an STL file. Twist: print the class's sculptures and place them next to their AR versions. #### Journey to the Center of the Natural Machine — Meta Company (Meron Gribetz) (2017) - Video: https://www.youtube.com/watch?v=8X-TdvDwBLk - Interaction: Shared & Social, Hands & Body, Information & UI - Platform & tech: Headset, Meta 2, Unity - Idea: Two people share one hologram face to face across a table, instead of each disappearing into their own world. - What it is: An AR piece at the Sundance Film Festival's New Frontier: two visitors sit face to face wearing Meta 2 headsets and watch a holographic human brain grow and evolve on the table between them, reaching out together to touch it. - Technique: Two Meta 2 headsets run a synchronised scene anchored to a physical table, with hand interaction so both visitors can touch the same virtual brain. - Try it: Make a two-phone shared AR scene anchored to a table (image marker or an ARKit collaborative session) where a virtual object only grows while both players touch it. Twist: each player sees a different stage of the object's evolution. ### Meta Reality Labs (Quest mixed reality) *Platform team behind Quest Presence Platform showcases* Built the Presence Platform showcases The World Beyond and First Encounters that teach Quest users what passthrough MR can do. #### The World Beyond — Meta Reality Labs (Quest mixed reality), Thomas Van Bouwel (2022) - Video: https://www.youtube.com/watch?v=SfQnR6hYeJM - Interaction: Portals & Worlds, Spatial Mapping, Hands & Body - Platform & tech: Headset, Meta Quest, Scene API, passthrough, voice SDK - Idea: Use a flashlight to dissolve real walls into an alien world, with a pet that listens to you. - What it is: A Presence Platform showcase where you shine a flashlight to dissolve your real walls into an alien world and play with Oppy, a pet that responds to hand gestures and voice. - Technique: Uses the Scene model to know wall planes, then masks passthrough with a shader where the virtual flashlight hits, revealing the VR world behind. - Try it: Use phone AR to detect a wall and cut a round hole where the phone points to show another world, using a mask shader or a portal component. Twist: the hole only opens when you say a magic word. #### First Encounters — Meta Reality Labs (Quest mixed reality), Thomas Van Bouwel (2023) - Video: https://www.youtube.com/watch?v=t4jcAbfDoPI - Interaction: Portals & Worlds, Spatial Mapping, Play - Platform & tech: Headset, Meta Quest 3, Scene API, passthrough - Idea: Blast through the walls of your home and see alien space outside. - What it is: Fluffy aliens attack after your ship crashes into your room; you shoot holes through your real walls to see space outside and chase aliens through them. - Technique: Scans the room into a scene mesh, then replaces passthrough with a virtual exterior wherever the player's shots break wall segments. - Try it: Detect a wall with phone AR, and wherever it is tapped, put a 'hole' that reveals a space panorama. Twist: the more holes there are, the more 'aliens' enter the room. #### Meta Ray-Ban Display live demo (Connect 2025) — Meta Reality Labs (Quest mixed reality) (2025) - Video: https://www.youtube.com/watch?v=gZ9IsB72nVk - Interaction: Hands & Body, Information & UI, Voice & Sound - Platform & tech: Wearable, Meta Ray-Ban Display, Meta Neural Band, Meta AI, EMG - Idea: An AI assistant you see, not just hear, answering in the corner of your eye. - What it is: At Connect 2025, Mark Zuckerberg demonstrates Meta Ray-Ban Display glasses with a small in-lens display and an EMG wristband: live AI answers, translation, messages and navigation appear in view, controlled by tiny finger gestures. - Technique: A monocular waveguide display shows Meta AI responses grounded in the glasses' camera and microphone, while a surface-EMG wristband decodes finger gestures as input. - Try it: Prototype a 'corner of the eye' assistant: a phone in a headset viewer shows one line of AI answer in the top corner while the camera stays live, and one keyboard key acts as the pinch. Twist: limit every answer to five words. ### Meta Reality Labs Research — AI assistants for AR glasses *Research teams at Meta Reality Labs studying interaction with contextual AI on AR glasses* Researchers at Meta Reality Labs Research, including Tanya Jonker's group, study how always-on AR glasses should use AI: when to explain it (XAIR), which actions to offer (OmniActions) and when to interrupt (ProMemAssist). Much of the work is led by PhD interns such as Xuhai Xu, Jiahao Nick Li and Kevin Pu. #### XAIR — Meta Reality Labs Research — AI assistants for AR glasses (2023) - Video: https://www.youtube.com/watch?v=yugkzrBbTCs - Interaction: Information & UI, Gaze & Attention - Platform & tech: Headset, explainable AI, design framework, AR headset prototype - Idea: Treat 'why is the AI showing me this?' as a first-class AR design question. - What it is: A design framework, with headset prototypes, for when and how AR assistants should explain their AI decisions, such as why a recipe step or a plant-watering reminder appears, and how much detail to show. - Technique: Built from a literature review, a survey of more than 500 users and expert workshops, XAIR splits explanation design into when to explain, what to explain and how to present it, and was validated with designers and a real-time AR prototype. - Try it: Take an AI-driven AR feature you have built (a label, a recommendation, a reminder) and design three explanation levels: none, a one-word reason, and a tap-to-expand panel; test them with classmates and note when the explanation helps or annoys. Twist: show the explanation only when the user's gaze lingers on the AI output. #### OmniActions — Meta Reality Labs Research — AI assistants for AR glasses (2024) - Video: https://www.youtube.com/watch?v=DZ8CXS2iS0o - Interaction: Information & UI, Voice & Sound, Tangible Objects - Platform & tech: Headset, LLM, chain-of-thought prompting, multimodal sensing - Idea: Turn anything you perceive into a ready-made next action. - What it is: A pervasive-AR concept that predicts which digital action you may want to take on whatever you are seeing or hearing, such as saving a poster's date to your calendar or sharing a song, and offers it as a one-tap shortcut. - Technique: A diary study produced a taxonomy of follow-up actions; an LLM with chain-of-thought prompting reads multimodal sensory input and predicts the most likely action categories. - Try it: Build a phone camera app with three buttons the AI fills in: point at a poster, menu or product, send the frame to a multimodal LLM that returns the three most useful actions as JSON (label and app), and render them as AR chips next to the object. Twist: have the AI also predict one action the user should not take. #### ProMemAssist — Meta Reality Labs Research — AI assistants for AR glasses (2025) - Video: https://www.youtube.com/watch?v=4fd5p7sR6Mo - Interaction: Gaze & Attention, Information & UI, Voice & Sound - Platform & tech: Headset, smart glasses, working-memory model, LLM, multimodal sensing - Idea: An assistant that knows not only what to say but when your mind has room for it. - What it is: Smart glasses that model your working memory in real time and time their help to it: they hold back a reminder while you are juggling numbers and deliver it at a natural break instead of interrupting. - Technique: Following cognitive theories of working memory, multimodal sensor signals are encoded as memory items subject to displacement and interference, and a timing predictor weighs the value of help against the cost of interruption. - Try it: Prototype an interruption-aware notifier: log what the user is doing (typing speed, speech, head motion from a phone or headset) and queue AI-generated reminders until a simple 'busy score' drops below a threshold. Twist: show queued reminders as small objects piling up in the corner of view. ### Metaio (Thomas Alt & Peter Meier) *AR software company; makers of the Metaio SDK and junaio browser* Munich company founded in 2003 whose tracking SDK, Creator tools and junaio AR browser powered much pre-ARKit phone and kiosk AR; Apple acquired it in 2015. #### LEGO Digital Box kiosk — Metaio (Thomas Alt & Peter Meier), The LEGO Group (Hidden Side) (2009) - Video: https://www.youtube.com/watch?v=CNkilCYnmoY - Interaction: Tangible Objects, Information & UI - Platform & tech: Desktop, Metaio Unifeye, kiosk camera - Idea: A closed package previews its own contents in 3D before purchase. - What it is: In LEGO stores, a child holds a sealed LEGO box up to a screen-and-camera kiosk and sees the finished model built and animated on top of the box. - Technique: The kiosk camera recognizes printed box art as a natural-feature target and renders the assembled 3D set on top of it, mirrored on a large screen. - Try it: Use a cereal or product box as an image target and show an animated 3D preview of what could be inside. Twist: reveal a different surprise when the box is shaken. #### Augmented City — Metaio (Thomas Alt & Peter Meier) (2012) - Video: https://www.youtube.com/watch?v=Sp1X0Q_GzZk - Interaction: Location & City, Information & UI - Platform & tech: Phone, Wearable, Metaio SDK, junaio, 3D tracking - Idea: Every surface of a city is a potential screen once the camera recognizes it. - What it is: A vision video in which a phone and glasses recognize buildings, shop fronts and objects in a city block and layer navigation, offers and interactive content directly onto them. - Technique: Metaio's markerless 3D tracking and junaio channels recognize building facades and objects from edge and feature models to anchor content without GPS drift. - Try it: Photograph three storefronts or building entrances near campus, turn them into image targets, and attach one useful piece of information to each. Twist: make the information change with the time of day. #### Thermal Touch — Metaio (Thomas Alt & Peter Meier) (2014) - Video: https://www.youtube.com/watch?v=K2XL0qnu4Z4 - Interaction: Hands & Body, Tangible Objects, Information & UI - Platform & tech: Wearable, Desktop, thermal camera, Metaio SDK, tablet PC - Idea: The heat your fingertip leaves behind is the click. - What it is: A prototype for smart glasses in which any surface, such as a poster, wall or car, becomes a touchscreen: an infrared camera sees the warm spot a finger leaves and triggers the AR content there. - Technique: A thermal camera and a standard camera are calibrated together; residual heat from a touch is detected in the infrared image and mapped onto the tracked surface's AR coordinates. - Try it: Fake Thermal Touch with a normal camera: track a printed poster, detect a finger dwell on it via hand tracking, and leave a slowly fading glow at each touch point. Twist: make the glow fade at a rate based on how long the finger stayed. ### Michael Heizer *Land artist* American artist who cuts and moves the desert at monumental scale: Double Negative, Levitated Mass and City, a mile-long complex he built in Nevada over five decades. #### Double Negative — Michael Heizer (1969) - Video: https://www.youtube.com/watch?v=1U0Q6MZmw3c - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, excavation, mesa edge - Idea: A sculpture made of removed material shows that absence can be the object, a direct idea for diminished-reality AR. - What it is: Two trenches, each about 9 metres wide and 15 metres deep, are cut into facing edges of the Mormon Mesa in Nevada; the sculpture is the empty space between them. - Technique: About 244,000 tons of rhyolite and sandstone were blasted and bulldozed out of the mesa edge. - Try it: Use a depth mask to cut a deep virtual trench into a real floor or lawn in AR so users seem to look down into it. Twist: align two trenches so they read as one line only from a single spot. #### Levitated Mass — Michael Heizer (2012) - Video: https://www.youtube.com/watch?v=MD_6az-OI_A - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, 340-ton granite boulder, concrete slot - Idea: Walking under something heavy that seems to float creates real bodily tension, a cue AR can borrow with carefully scaled overhead objects. - What it is: A 340-ton granite boulder sits on two ledges above a 139-metre concrete slot at LACMA in Los Angeles; visitors walk down the ramp and pass underneath it. - Technique: The boulder was moved 170 kilometres over 11 nights on a custom transporter and bolted onto steel shelves over the slot. - Try it: Place a photoreal AR boulder over a real corridor or doorway so people must walk under it, and add a faint rumble sound as they pass. Twist: the boulder rises a little higher for each person who walks under it. #### City — Michael Heizer (2022) - Video: https://www.youtube.com/watch?v=iWCThfpaxN8 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, compacted earth, concrete, desert valley - Idea: A walkable abstract architecture in the desert shows how pure geometry at huge scale can feel like a lost civilisation, a mood AR world-builders chase. - What it is: Begun in 1970 and opened in 2022, City is a complex of compacted-earth mounds, ramps and concrete forms about 2.4 kilometres long in a remote Nevada valley; only six visitors a day are admitted. - Technique: Earth from the site was excavated, compacted and shaped with heavy machinery into mounds and depressions, edged with concrete. - Try it: Model a large geometric complex (ramps, mounds, sunken courts) in Blender and place it at 1:1 scale on a sports field in AR. Twist: allow only one viewer at a time, so each visit feels solitary. ### Microsoft HoloLens team (Alex Kipman, Kudo Tsunoda & Microsoft Studios) *Platform team and first-party studio for HoloLens* Shipped the first self-contained holographic computer and its launch apps - RoboRaid, HoloStudio and the Minecraft tabletop demo. #### Minecraft for HoloLens (E3 demo) — Microsoft HoloLens team (Alex Kipman, Kudo Tsunoda & Microsoft Studios), Mojang Studios (Minecraft Earth) (2015) - Video: https://www.youtube.com/watch?v=FNoHNTaP_Gk - Interaction: Spatial Mapping, Play, Voice & Sound - Platform & tech: Headset, HoloLens, spatial mapping, voice commands - Idea: Put an entire Minecraft world on a real coffee table and control it by voice and gesture. - What it is: On stage at E3, a player sees a Minecraft world built on a real coffee table, zooms into it, opens it like a dollhouse and commands it by voice. - Technique: Anchors a scaled game world to a detected table surface and combines gaze, air-tap and voice for world manipulation. - Try it: Place a small block world on a table with phone AR and trigger "explode" or "rain" with voice commands (such as iOS Shortcuts or Web Speech). Twist: let the audience see the same world on another phone. #### HoloStudio — Microsoft HoloLens team (Alex Kipman, Kudo Tsunoda & Microsoft Studios) (2016) - Video: https://www.youtube.com/watch?v=BRIJG0x_We8 - Interaction: Drawing & Making, Hands & Body - Platform & tech: Headset, HoloLens, gesture modelling, 3D printing - Idea: Build 3D models in the air with gestures, then print them as real objects. - What it is: Build 3D models in 3D with holographic tools modelled on real ones, then send them to a 3D printer to become physical objects. - Technique: Provides a toolbelt of primitive shapes and holographic tools manipulated with gaze and air-tap, exporting printable meshes. - Try it: Assemble a tabletop sculpture in phone AR (e.g., Reality Composer), export screenshots, and make a physical version from cardboard or clay. Twist: allow only three basic shapes. #### RoboRaid — Microsoft HoloLens team (Alex Kipman, Kudo Tsunoda & Microsoft Studios), Asobo Studio (2016) - Video: https://www.youtube.com/watch?v=Hf9qkURqtbM - Interaction: Spatial Mapping, Play, Gaze & Attention - Platform & tech: Headset, HoloLens, spatial mapping, spatial sound - Idea: Alien robots break in through the real walls of your home. - What it is: Alien robots burst through your actual walls, leaving cracks, and you shoot them with gaze and air-tap while physically dodging projectiles. - Technique: Uses the spatial-mapping mesh to pick wall spawn points and render crack decals, with spatial audio signalling enemies behind the player. - Try it: Detect vertical walls in phone AR and generate "broken hole" decals with enemies crawling out of them, which players shoot by tapping a crosshair. Twist: signal enemy positions with sound alone. ### Naoko Tosa *Media artist; professor at Kyoto University* Japanese media-art pioneer who, from the early 1990s, built interactive characters that read human emotion (Neuro-Baby), installations that reveal the unconscious feelings between two people (Unconscious Flow) and 'cultural computing' systems based on Zen. She is now best known for Sound of Ikebana, high-speed films of fluid shaped by sound. #### Neuro-Baby — Naoko Tosa (1993) - Video: https://www.youtube.com/watch?v=etEIbMZ6uUY - Interaction: Voice & Sound, Face - Platform & tech: Desktop, neural network, voice analysis - Idea: A virtual being that responds to how you feel, not what you say. - What it is: A computer-generated baby face on screen listens to the tone of your voice and responds with its own emotional expressions and sounds. - Technique: A neural network classifies emotion from the prosody of the user's voice and drives the facial animation of a 3D baby character. - Try it: Build a character that reacts to voice loudness and pitch (p5.js sound input or Lens Studio audio) with different facial expressions. Twist: it mirrors the opposite emotion. #### Unconscious Flow — Naoko Tosa (1999) - Video: https://www.youtube.com/watch?v=eF4DsTolxJM - Interaction: Hands & Body, Shared & Social, Perception & Effects - Platform & tech: Projection, Desktop, heart-rate sensors, CG - Idea: Show the hidden emotional relationship between two people. - What it is: Two people hold sensors while two virtual mermaids represent them; the mermaids' behaviour reveals whether the pair are truly relaxed or tense with each other, whatever their faces say. - Technique: Heart-rate sensors measure each participant's arousal and synchrony, and the pair of values is mapped to how the CG mermaids approach, avoid or dance with each other. - Try it: Give two people pulse sensors (or phone camera heart-rate apps) and map their heart-rate difference to the distance between two AR creatures. Twist: the creatures merge when hearts sync. #### ZENetic Computer — Naoko Tosa (2004) - Video: https://www.youtube.com/watch?v=RYeT75F7ezQ - Interaction: Tangible Objects, Perception & Effects, Drawing & Making - Platform & tech: Projection, Desktop, story engine, tangible interface - Idea: A computer that asks you questions instead of answering them. - What it is: The user arranges elements in a virtual ink-wash landscape, and the system answers with Zen-style riddles and stories built from Buddhist concepts, inviting reflection rather than tasks. - Technique: A tangible or pointer interface places symbolic objects in a sansui landscape, and a story engine maps their arrangement to the five skandhas to select koan-like dialogues. - Try it: Design an AR sand tray where placing three physical tokens produces a short, strange question generated from their combination. Twist: the question is spoken, never shown. ### Nathan Shafer *AR artist, comics author and educator* Alaskan artist who uses geolocated and marker-based AR to tell Alaskan stories: vanished glacier edges, domed cities that were never built, and Indigenous-language portraits made with schools. He later moved into AR comics and games with Native and autistic characters. #### Exit Glacier Augmented Reality Terminus Project — Nathan Shafer (2012) - Video: https://www.youtube.com/watch?v=PQac4zosOVw - Interaction: Location & City, Information & UI, Perception & Effects - Platform & tech: Phone, Layar, GPS - Idea: Show climate change on site by restoring what has melted. - What it is: At Exit Glacier in Kenai Fjords, Alaska, visitors see where the glacier's edge stood in past decades as AR ice rising from the now-empty valley. - Technique: Historical terminus positions from park survey markers are rebuilt as 3D ice models and geolocated along the trail in the Layar AR browser. - Try it: Find historical data for something that has shrunk or moved near you (a shoreline, a tree line, a river) and place its past outline in phone AR at full scale. Twist: add a year slider. #### Seward's Success (as seen in Indie Alaska) — Nathan Shafer (2014) - Video: https://www.youtube.com/watch?v=8Elu87SfL98 - Interaction: Location & City, Portals & Worlds - Platform & tech: Phone, Layar, GPS - Idea: Build the city that was never built, where it was supposed to be. - What it is: A documentary on Shafer's AR work, centred on Seward's Success, a futuristic domed city planned for Anchorage in the 1960s but never built, which he rebuilds in AR on its intended site. - Technique: Architectural drawings of the unbuilt dome are modelled in 3D and geolocated at the planned site so phones show it on the skyline. - Try it: Find an unbuilt or demolished building plan for your city and place it in phone AR at its intended site and scale. Twist: let viewers step inside through a portal. #### Languages of the Face — Nathan Shafer (2018) - Video: https://www.youtube.com/watch?v=GBNJ2EQKRwU - Interaction: Face, Information & UI, Tangible Objects - Platform & tech: Phone, image tracking - Idea: A portrait that teaches an endangered language. - What it is: An augmented portrait made with the Sitka School District: two self-portraits, with and without glasses, reveal the names of facial features in Deg Xinag and Łingít when viewed through AR. - Technique: The printed portraits act as image targets, and an AR app overlays labels and likely audio pronunciations anchored to each facial feature. - Try it: Draw a self-portrait, use it as an image target, and label each feature in a heritage or minority language with recorded pronunciation. Twist: switch to face tracking so the labels follow a live face. ### Neil Mendoza *Artist making robotic and interactive sculptures from everyday objects* Artist and openFrameworks developer who began with AR and big-screen pieces in Bristol (The Sancho Plan) and now builds humorous machines that watch, avoid or talk back to their viewers. #### Big Screen Augmented Reality (with The Sancho Plan) — Neil Mendoza (2010) - Video: https://vimeo.com/10607151 - Interaction: Shared & Social, Tangible Objects, Voice & Sound - Platform & tech: Projection, openFrameworks, marker tracking - Idea: Public AR on a giant screen, where holding up a board brings a character to life for the whole square. - What it is: In Bristol's Millennium Square, people hold up white boards to the music and the BBC Big Screen shows animated characters and instruments standing on the boards in the live image. - Technique: A camera above the square tracks the boards as markers, and characters are composited onto them and shown on the big screen (likely openFrameworks; The Sancho Plan with Neil Mendoza and Pervasive Media Studio). - Try it: Film the class from above with a projector or big screen showing the feed, and make paper markers that each spawn a musical character when held up. Twist: characters only play when their markers are close together. #### I Spy — Neil Mendoza (2012) - Video: https://vimeo.com/55122295 - Interaction: Gaze & Attention, Hands & Body - Platform & tech: Desktop, openFrameworks, Kinect, servos, Android - Idea: Our gadgets finally stare back at us. - What it is: A hanging mobile of four tablets showing eyes and mouths turns to follow visitors as they walk through the room, as if the gadgets were watching them. - Technique: Two Kinects track people in the space and an openFrameworks app computes the servo angles of the mobile's arms so each tablet faces the nearest viewer. - Try it: Place a phone showing an animated eye in the room and use body tracking so the eye always looks at the nearest person. Twist: with two people present, the eye looks at whoever is not looking at it. #### Antivanity Mirror — Neil Mendoza (2020) - Video: https://vimeo.com/398041909 - Interaction: Face, Gaze & Attention, Perception & Effects - Platform & tech: Desktop, face tracking, servos - Idea: A mirror that refuses to let you look at yourself. - What it is: A small robotic mirror turns its face away whenever someone tries to look into it, so you can never see yourself. - Technique: A camera detects the viewer's face and servos rotate the mirror away from it in real time. - Try it: Make a phone app that shows the front camera but always pans or blurs the image away from any detected face. Twist: it only lets you see yourself when you close your eyes. ### Oliver Kreylos *Researcher at UC Davis (KeckCAVES, W. M. Keck Center for Active Visualization in the Earth Sciences); creator of the Vrui VR toolkit* Oliver Kreylos built the open-source Augmented Reality Sandbox, where a Kinect scans real sand and a projector paints live topography and flowing water onto it, now installed in hundreds of museums and classrooms. He also pioneered Kinect-based 3D video capture for immersive telepresence. #### 3D Video Capture with Kinect — Oliver Kreylos (2010) - Video: https://www.youtube.com/watch?v=7QrnwoO1-8A - Interaction: Hands & Body, Shared & Social - Platform & tech: Desktop, Kinect, Vrui - Idea: Days after the Kinect came out, he captured a person as live 3D video you could view from any angle. - What it is: Days after Kinect's release, Kreylos showed live 3D video of himself captured by a Kinect and viewed from any angle, an early step toward volumetric telepresence. - Technique: The Kinect's depth image is back-projected into a 3D point cloud using its camera intrinsics and textured with the calibrated RGB image, so the capture can be rendered from any virtual viewpoint. - Try it: Capture a live point cloud of yourself with a LiDAR iPhone or Record3D, orbit it from any angle in three.js, and place a virtual object beside it. Twist: stylize the point cloud (voxels, wireframe or particles) and shoot a 30-second short. #### Augmented Reality Sandbox — Oliver Kreylos (2012) - Video: https://www.youtube.com/watch?v=j9JXtTj0mzE - Interaction: Projection, Tangible Objects, Spatial Mapping - Platform & tech: Projection, Kinect, projector, Vrui - Idea: Pile up sand by hand while projection draws contour lines live; hold out a hand to make it rain down the dunes. - What it is: Shape real sand with your hands; a Kinect reads the terrain and a projector paints elevation colors and contour lines onto it, and holding out a hand makes virtual rain that flows downhill. - Technique: A Kinect above the sandbox measures the sand's height map, which is filtered and used to color an elevation map with contour lines projected back onto the sand, while a shallow-water simulation runs rain across the terrain. - Try it: Measure height with a depth camera (or approximate it with a tray of colored clay and an RGB camera), generate contour lines and elevation colors in TouchDesigner, and project them back onto the sand or clay. Twist: replace the "rain" with something else that spreads, such as crowds, fire or sound. #### 3D Video Capture With Three Kinects — Oliver Kreylos (2014) - Video: https://www.youtube.com/watch?v=Ghgbycqb92c - Interaction: Shared & Social, Hands & Body - Platform & tech: Headset, Desktop, Kinect, Vrui, Oculus Rift - Idea: Three Kinects combine into a full 3D person placed in a virtual space for remote collaboration. - What it is: Three calibrated Kinects capture a person in full 3D and merge them with virtual environments for immersive tele-collaboration. - Technique: Three Kinects are extrinsically calibrated to one coordinate frame (e.g. by a shared calibration target), and their point clouds are merged into a single 3D representation rendered in a VR environment. - Try it: Capture depth data of the same person from two angles with two phones (or scan separately with Polycam), then align and merge them by hand in CloudCompare or three.js using shared reference points. Twist: compare how much "presence" viewers feel with one viewpoint versus two. ### Owen (owenyuwono) *Developer and founder of Pixel8 Labs* Indonesian developer who in 2026 open-sourced a run of three.js + WebGPU prototypes: an FFT ocean (Poseidon), an SPH fluid (Tiamat), procedural grass (Gaia) and trees (Dryad). #### Dryad — Owen (owenyuwono) (2026) - Video: https://x.com/owenyuwono/status/2096193851648258351 - Source code: https://github.com/owenyuwono/dryad - Interaction: Perception & Effects, Location & City - Platform & tech: Web, three.js, WebGPU - Idea: A seed number is enough to grow a unique tree, down to each leaf. - What it is: Procedural trees grow from a seed in the browser, with every leaf rendered as its own card so whole canopies move volumetrically in the wind. - Technique: Branches are generated by rules seeded from a number (and, per the repo, planet physics), and leaves are drawn as GPU-instanced cards animated by wind in three.js / WebGPU. - Try it: Grow a Dryad tree from a WebXR hit-test point on the floor, using the date or the user's name as the seed so everyone plants a different tree. Twist: let the tree grow a little each time the user returns to the same spot. #### Poseidon — Owen (owenyuwono) (2026) - Video: https://x.com/owenyuwono/status/2064260229207142583 - Source code: https://github.com/owenyuwono/poseidon - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Web, three.js, WebGPU, TSL - Idea: A film-grade ocean technique, now a web page. - What it is: A photoreal open ocean with breaking crests, foam and golden-hour light runs entirely in the browser. - Technique: Three cascades of Tessendorf inverse FFT run as WebGPU compute shaders (three.js TSL) to produce displacement and normals, shaded with measured water optics, subsurface scattering and foam. - Try it: Put a small patch of Poseidon inside a WebXR hit-test portal on the floor so the phone shows a sea below the room. Twist: sync the sun direction to the real time of day. #### Tiamat — Owen (owenyuwono) (2026) - Video: https://x.com/owenyuwono/status/2063441468690411593 - Source code: https://github.com/owenyuwono/tiamat - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Web, Phone, three.js, WebGPU, SPH - Idea: Tilt the phone and the water inside it moves like water in a glass. - What it is: Tens of thousands of water particles slosh inside the browser window, and on a phone the water follows the device's tilt through the gyroscope. - Technique: An SPH (smoothed-particle hydrodynamics) solver runs 20k–100k particles in WebGPU compute shaders with the gravity vector taken from the device orientation, and the surface is ray-marched. - Try it: Pour Tiamat's fluid into a virtual glass anchored on a real table in WebXR and feed the phone's motion into the solver, so the water sloshes as the user walks around the glass. Twist: let the user tap to drop objects that splash. ### Pierrick Sorin *Video artist (optical theatres, Pepper's ghost, live compositing on stage)* French video artist known for self-filmed burlesque shorts and, since the 1990s, for 'théâtres optiques': small boxes where a filmed tiny Sorin appears among real objects through a Pepper's-ghost glass. He also designs video for opera, such as Rossini's La Pietra del paragone at the Théâtre du Châtelet. #### Titre variable n°1 (théâtre optique) — Pierrick Sorin (1999) - Video: https://www.youtube.com/watch?v=O11LIyjcxiw - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, Pepper's ghost, video monitor, glass, miniature set - Idea: A tilted sheet of glass is the cheapest optical see-through display: it teaches how to place a virtual figure among real objects and how lighting decides whether it looks present. - What it is: Inside a small box, a tiny filmed Pierrick Sorin appears to stand among real objects and act out a clumsy gag. The figure is a video reflection on a tilted glass, so it floats in the same space as the physical props. - Technique: Pepper's ghost: a hidden monitor below or above reflects in a 45° glass pane, superimposing the filmed performer (shot on black) onto the lit miniature set behind the glass. - Try it: Build a shoebox Pepper's-ghost theatre with a phone screen and a sheet of acetate, then stage a 15-second clip of yourself (filmed on black) interacting with a real object inside the box. Twist: port the same scene to a HoloKit or phone AR and compare which feels more present. #### La Pietra del paragone (Rossini, Théâtre du Châtelet) — Pierrick Sorin (2007) - Video: https://www.youtube.com/watch?v=CzjS3MCjBzI - Interaction: Performance, Portals & Worlds - Platform & tech: Projection, live chroma key, miniature sets, video cameras - Idea: Showing the real stage and the composited image side by side turns the seam into the show, a model for AR performances where the audience watches both the performer and the augmented feed. - What it is: Singers perform in front of a blue screen on an almost empty stage, while cameras film them and miniature sets on tables; a large screen above shows them live, composited into a tiny painted world. The audience sees both the trick and the illusion at once. - Technique: Live chroma key: stage cameras film singers against blue, other cameras film scale models, and a real-time keyer composites them on a big screen (staging by Giorgio Barberio Corsetti, video by Sorin). - Try it: Film a classmate on a green cloth with one phone while a second phone films a shoebox diorama, and composite them live in OBS or Lens Studio on a projector. Twist: let the performer pick up a real prop that becomes giant in the diorama. #### Optical theatre for 'From Kalila wa Dimna to La Fontaine' (Louvre Abu Dhabi) — Pierrick Sorin (2024) - Video: https://www.youtube.com/watch?v=8IjXXyl3GwA - Interaction: Perception & Effects, Tangible Objects, Information & UI - Platform & tech: Desktop, Pepper's ghost, video, vitrine - Idea: A museum vitrine can host a living scene without a headset; the same framing (a box, a window, a lit set) helps AR content feel placed and intentional. - What it is: For an exhibition on animal fables at the Louvre Abu Dhabi, Sorin built an optical theatre in which filmed actors and animals appear as small floating figures inside a miniature stage, acting out a fable among physical scenery. - Technique: A larger Pepper's-ghost vitrine: hidden screens reflect in angled glass over a lit physical set, with actors shot on black so only their figures appear. - Try it: Pick a short fable and stage it in AR on a real bookshelf: use image tracking on a book cover to anchor two tiny animated animals that act out the story. Twist: the moral appears as text only when the viewer leans closer. ### Pol Bury *Artist; sculptor of slow kinetic movement* Belgian artist (1922–2005) who made movement almost imperceptible: balls and rods that creep on hidden motors, and polished steel fountains whose spheres tip and pour water in slow rhythms. #### Sculptures à cordes — Pol Bury (1974) - Video: https://www.youtube.com/watch?v=DYzYSDz8vo0 - Interaction: Perception & Effects - Platform & tech: Desktop, wood, nylon cords, hidden motors - Idea: Motion at the edge of perception creates unease and attention; AR can use nearly imperceptible change to reward patient looking. - What it is: Groups of wooden elements hung on cords that move so slowly, driven by hidden motors, that viewers are not sure anything is moving until they look away and back. - Technique: Hidden low-speed motors pull cords that move the elements a few millimetres per second, below the threshold where motion is obvious. - Try it: Place an AR cluster of wooden rods on a shelf that moves only when the user is not looking at it (camera direction), so it has changed each time they look back. Twist: make it move toward the user a little each time. #### Fontaine (Fondation Maeght) — Pol Bury (1978) - Video: https://www.youtube.com/watch?v=8vl6onFHh7o - Interaction: Perception & Effects - Platform & tech: Desktop, stainless steel, water, hydraulics - Idea: Movement driven by slowly accumulating weight creates anticipation; AR can build suspense with gradual build-ups that release in a single gesture. - What it is: Slender stainless-steel tubes in a pool fill with water until they tip, pour and slowly rise again, in an endless, unhurried rhythm. - Technique: Each pivoting tube fills with water until its centre of mass passes the pivot, then tips to empty and returns, like a shishi-odoshi garden device. - Try it: Build an AR shishi-odoshi fountain on a real table that fills with virtual water at the rate the user speaks and tips with a sound when full. Twist: chain several so one tipping fills the next. #### Fontaines (Sphérades), Palais-Royal — Pol Bury (1985) - Video: https://www.youtube.com/watch?v=wVyVdvV1XsA - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, polished stainless steel spheres, water - Idea: Reflective, slowly moving objects blend into their surroundings by mirroring them; AR objects with real-time environment reflections sit more naturally in a place. - What it is: Two fountains of polished steel spheres in the Palais-Royal courtyard slowly tilt and turn under flowing water, reflecting the arcades and sky. - Technique: Water flow shifts the weight of hollow steel spheres so they rock and rotate slowly, while their mirror surfaces continuously reflect the historic courtyard. - Try it: Place a cluster of mirror spheres in a real courtyard with AR environment probes so they reflect the actual surroundings, rocking slowly as a virtual stream flows. Twist: let the spheres reflect a different season than the real one. ### Raffaello D'Andrea / Verity *Roboticist; ETH Zurich professor and founder of Verity* Canadian-Italian-Swiss engineer who co-founded Kiva Systems and founded Verity; his flying machine arena at ETH Zurich produced drone-built architecture, the Cirque du Soleil short SPARKED and indoor drone swarms for Metallica and Broadway. #### Flight Assembled Architecture (with Gramazio Kohler) — Raffaello D'Andrea / Verity (2011) - Video: https://www.youtube.com/watch?v=uhfBB5mm2HM - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, quadcopters, motion capture, foam bricks - Idea: Flying machines can build in places people cannot reach; AR can preview such aerial construction by showing each flight path and each placed piece in sequence. - What it is: Small quadcopters pick up foam bricks one by one and stack 1,500 of them into a curved six-metre tower inside a gallery, the first building assembled by flying robots. - Technique: A motion-capture system tracks the quadcopters indoors while software plans each brick's pick-up and placement path from a digital architectural model. - Try it: Program an AR swarm of virtual drones that fetch blocks from a real table and stack them into a tower on the floor, one block per second, with visible flight lines. Twist: let a student change the tower's design mid-build and watch the drones re-plan. #### SPARKED (with Cirque du Soleil) — Raffaello D'Andrea / Verity (2014) - Video: https://www.youtube.com/watch?v=6C8OJsHfmpI - Interaction: Performance, Hands & Body - Platform & tech: Projection, quadcopters, motion capture, choreography - Idea: A flying lampshade becomes a character through timing and motion alone; AR objects can gain personality from how they move toward and away from people. - What it is: In a short film made with Cirque du Soleil, ten quadcopters dressed as lampshades dance around an electrician in a workshop, becoming characters with moods of their own. - Technique: Indoor quadcopters are tracked by motion capture and follow choreographed trajectories synchronised with the performer and music, with lightweight lampshades as costumes. - Try it: Create three AR lamps that float around a student and react to them: one shy (flees), one curious (follows), one sleepy (sinks when ignored). Twist: remove all faces and colours and see if people still name their moods. #### Metallica WorldWired Tour indoor drone swarm — Raffaello D'Andrea / Verity (2017) - Video: https://www.youtube.com/watch?v=A4TBvBfPNVg - Interaction: Performance, Voice & Sound, Shared & Social - Platform & tech: Projection, indoor drones, local positioning - Idea: Points of light above an audience create a shared focus without screens; AR concerts can place a few floating lights overhead instead of a flat virtual display. - What it is: During Metallica's WorldWired Tour, a swarm of small glowing drones hovers and dances above the arena stage, forming moving constellations over the band. - Technique: Verity's indoor drones use a local positioning system instead of GPS, flying autonomous, redundant formations timed to the show's cues. - Try it: Stage a short AR concert moment: 20 glowing orbs gather above a performer's head during the chorus and scatter on the drum hit, triggered by audio beat detection. Twist: sync the orbs across all phones in the room so everyone sees the same swarm. ### Ralfonso *Kinetic, light and interactive sculptor* Swiss-born artist Ralf Alfonso (Ralfonso) creates wind-driven stainless-steel kinetic sculptures, light sculptures and interactive murals for public spaces worldwide, and co-founded the Kinetic Art Organization (KAO). #### AD INFINITUM — Ralfonso (2008) - Video: https://www.youtube.com/watch?v=ioOzYWPKNyk - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, stainless steel, bearings, wind - Idea: A form that reads differently from every angle rewards walking around it, the key behaviour to design for in world-anchored AR. - What it is: A tall outdoor wind sculpture whose looping stainless-steel forms rotate in the breeze, drawing an endless figure that shifts as you walk around it. - Technique: Balanced stainless-steel elements on low-friction bearings turn in light wind, and their curved loops overlap into changing silhouettes (year approximate). - Try it: Place a looping virtual sculpture in an open square with geospatial AR, rotating slowly on two axes. Twist: reveal a hidden symbol that only aligns from one walking position. #### SENSE #1 — Ralfonso (2012) - Video: https://www.youtube.com/watch?v=bCaWkHyEwUE - Interaction: Hands & Body, Information & UI, Perception & Effects - Platform & tech: Desktop, LED, sensors, sculpture - Idea: Combining ambient information (time) with response to presence makes a public piece useful and playful at once, a good pattern for civic AR. - What it is: An interactive light mural and sculpture that doubles as a clock, lighting up in response to people passing and to the time of day. - Technique: Presence sensors trigger light sequences while a clock logic shifts the base pattern through the day (details not documented; likely motion sensors). - Try it: Anchor an AR clock to a real wall whose hands are made of light particles that scatter when someone walks by. Twist: let the particles reassemble a little slower each hour. #### UNION 3 — Ralfonso (2014) - Video: https://www.youtube.com/watch?v=Mhv469IURPk - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, stainless steel, LED, wind - Idea: Designing for both day and night states is essential in outdoor AR, where lighting conditions change what content reads well. - What it is: A wind-driven kinetic sculpture in Qingdao, China, that combines turning stainless-steel forms with coloured light, so it changes from a daytime sculpture into a night-time light work. - Technique: Wind turns the steel elements while integrated LEDs colour them after dark, adding a second, programmable layer to the physical motion (year approximate). - Try it: Build an AR sculpture that switches between a reflective day material and a glowing night material using the device's ambient light estimate. Twist: let the colour follow the city's live temperature. ### Raymond Lo *Computer-vision engineer; co-founder and former CTO of Meta (Spaceglasses); student of Steve Mann* Studied with Steve Mann at the University of Toronto, where he built EyeTap prototypes and GPU-based HDR vision for welding glasses, co-founded the AR headset company Meta in 2012, and later worked on AI computer vision, including OpenVINO at Intel. #### EyeTap X: Depth Vision — Raymond Lo (2013) - Video: https://www.youtube.com/watch?v=KyZ2XUsRJXs - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Wearable, EyeTap, PrimeSense, Android - Idea: See distance as colour, even in the dark. - What it is: A home-built EyeTap X wearable overlays a colour heat-map of depth onto the wearer's see-through view in real time, so near and far surfaces glow in different colours and the room stays readable even in darkness. - Technique: A PrimeSense depth sensor feeds a quad-core Android board that colour-maps each depth value and draws it on a see-through display at about 25-30 fps. - Try it: On a LiDAR iPhone, render the depth map as a false-colour overlay in a HoloKit or phone AR view and walk a dim corridor with it. Twist: turn depth into sound pitch instead of colour. #### EyeTap X: Interface Cube — Raymond Lo (2013) - Video: https://www.youtube.com/watch?v=_CpL36SLk-E - Interaction: Hands & Body - Platform & tech: Wearable, EyeTap, PrimeSense, OpenGL ES - Idea: Grab a hologram with your bare hand, no controller needed. - What it is: Through the EyeTap X glass, a coloured virtual cube floats in front of the wearer; the depth camera detects the hand grasping it, and the cube follows the hand as it moves. - Technique: Depth-map gesture recognition segments the nearest hand blob, detects an open or closed grasp and moves a 3D overlay rendered with OpenGL ES to the tracked position. - Try it: With MediaPipe Hands in the browser or Quest hand tracking, make a cube that can only be picked up with a closed fist and is dropped when the hand opens. Twist: the cube slips out if you grab it too fast. #### Early Meta Headset Prototype Demo — Raymond Lo, Meta Company (Meron Gribetz) (2014) - Video: https://www.youtube.com/watch?v=nCyyhU8Ux18 - Interaction: Hands & Body, Information & UI - Platform & tech: Headset, Meta headset, depth camera - Idea: Replace the desk monitor with windows you can grab and place anywhere in your room. - What it is: Filmed through the lens of an early Meta headset prototype: the wearer grabs a floating Meta logo, pulls up a row of app icons, opens virtual monitors and browser windows in the living room and pokes a glowing sphere. - Technique: A see-through headset with a depth camera likely tracks the hand as a point cloud and uses simple grab gestures to move windows anchored in the room. - Try it: In WebXR passthrough, let users pinch to spawn a browser panel and throw it onto a wall where it sticks. Twist: panels nobody has looked at for a minute slowly fade and drift away. ### Richard Long *Land artist* British artist who since 1967 has made art by walking: lines trodden into grass, stone circles and text works that record routes across the world. #### A Line Made by Walking — Richard Long (1967) - Video: https://www.youtube.com/watch?v=5eVaZRaQWRQ - Interaction: Location & City, Hands & Body, Drawing & Making - Platform & tech: Desktop, walking, grass, photograph - Idea: The body's repeated path becomes the drawing: the purest form of a location-based trace that AR can record and replay. - What it is: As a student, Long walked back and forth along a straight line in a Wiltshire field until the flattened grass caught the light, then photographed it; the photograph is the work. - Technique: Repeated walking pressed the grass flat along one axis so the line appeared by reflected light, fixed in one photograph. - Try it: Walk the same straight line 20 times with a WebXR or AR Foundation app that thickens and brightens an AR line each time you pass. Twist: invite others to walk it too, and let the line fade if nobody walks for a day. #### Stones and Flies: Richard Long in the Sahara — Richard Long (1988) - Video: https://www.youtube.com/watch?v=hB_EAlSc7uE - Interaction: Location & City, Tangible Objects, Perception & Effects - Platform & tech: Desktop, stones, walking, desert - Idea: Simple geometric marks (line, circle) instantly read as human in a vast landscape, a lesson in how little an AR mark needs. - What it is: Philip Haas's film follows Long across the Sahara as he walks, arranges stones into lines and circles, and draws in the sand, leaving marks the wind will erase. - Technique: Long collects stones on the spot and places them in geometric figures, recording each walk with photographs and text. - Try it: Collect ten real stones, scan them with Polycam, and arrange copies in AR into a 20-metre circle on a field. Twist: add one new stone to the circle each time a visitor arrives, until it closes. #### Richard Long at The Hepworth Wakefield — Richard Long (2012) - Video: https://www.youtube.com/watch?v=tarLr9BL45w - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Desktop, stone, mud, gallery floor - Idea: Bringing a walk's materials indoors as a precise floor pattern shows how an outdoor trace can be translated into a room-scale layout, as AR can do. - What it is: Behind the scenes of Long's exhibition at The Hepworth Wakefield: stones laid out in large floor works and mud applied by hand directly to the walls. - Technique: Stones are sorted and placed by hand following a simple geometry, and mud is applied to walls with the hands in rhythmic strokes. - Try it: Record a walk outdoors as a GPS track, then in a classroom lay it out at room scale in AR as a line of stones on the floor. Twist: add muddy hand-prints on the real walls in AR at the points where you stopped. ### Rob Mulholland *Sculptor* Scottish sculptor whose mirrored human figures stand in forests and lochs, reflecting their surroundings until they almost disappear. #### Vestige — Rob Mulholland (2012) - Video: https://www.youtube.com/watch?v=gf4BOH9vod4 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, mirror-polished stainless steel, forest - Idea: Figures made only of reflections are the physical form of an 'invisible' AR character that is noticed only when it moves or catches light. - What it is: Six life-size mirrored human figures stand among the trees at the David Marshall Lodge in Aberfoyle, Scotland, reflecting the forest so they almost vanish. - Technique: Flat human silhouettes are cut from mirror-polished stainless steel and placed at varied angles so they reflect the surrounding trees. - Try it: Place AR human silhouettes with a real-time reflection material in a wooded area and let users try to find them. Twist: figures slowly change position when nobody is looking. #### Skytower — Rob Mulholland (2013) - Video: https://www.youtube.com/watch?v=QVUayqFIOIw - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, woven steel rod, public sculpture - Idea: Steel drawn like wind-blown willow shows how a form can record a force, a starting point for AR sculptures shaped by real wind data. - What it is: A six-metre public sculpture in Airdrie, Scotland, made of 2,000 metres of steel rod shaped like willow and woven together, as if the wind had scattered and reshaped the structure. - Technique: Over 2,000 metres of 12 mm steel rod were bent, interwoven and secured with more than 6,000 welds. - Try it: Generate an AR tower of curved lines whose lean and spread follow the live wind direction and speed at your location. Twist: record a whole day and freeze the tower as a sculpture of that day's wind. #### Still — Rob Mulholland (2014) - Video: https://www.youtube.com/watch?v=3VsdvqXvJYI - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, mirror-polished stainless steel, loch - Idea: A figure standing in water that only shows water is a clean example of camouflage through reflection, useful for subtle AR presence. - What it is: A mirrored man stands in the water of Loch Earn near St Fillians, reflecting the loch, the sky and the hills; the local community reinstalled it in 2023. - Technique: A stainless-steel figure polished to a mirror finish is fixed on a base under the water surface. - Try it: Place a mirrored AR figure standing in a real pond or fountain with a screen-space reflection shader and film it at dawn and dusk. Twist: its reflection shows a different time of day than the real scene. ### Roberto Coviello *XR and AI engineer* Zurich-based XR engineer who maintains QuestCameraKit, an open collection of Unity samples for the Meta Quest Passthrough Camera API: color sampling, object detection, camera shaders and streaming. #### Decart XR: speak a prompt and transform the world — Roberto Coviello (2025) - Video: https://x.com/xrdevrob/status/1974535441811677627 - Source code: https://github.com/xrdevrob/QuestCameraKit - Interaction: Voice & Sound, Perception & Effects, Portals & Worlds - Platform & tech: Headset, Meta Quest 3, QuestCameraKit, Decart AI, speech-to-text - Idea: Say a world and step into it without leaving your room. - What it is: Built on QuestCameraKit, Decart XR lets the wearer speak a prompt and watch a real-time AI video model transform the passthrough view of their surroundings. - Technique: Passthrough frames stream to Decart's real-time video-to-video model with a spoken prompt (speech-to-text), and the transformed stream replaces the passthrough view. - Try it: Wire speech-to-text to a real-time restyling API on a phone in a HoloKit or cardboard viewer, and hold a 'world-building' round where each student speaks one prompt. Twist: prompts must describe a smell or a sound, not a look. #### QuestCameraKit: passthrough camera with object detection and more — Roberto Coviello (2025) - Video: https://www.youtube.com/watch?v=1z3pcMJbnRA - Source code: https://github.com/xrdevrob/QuestCameraKit - Interaction: Tangible Objects, Spatial Mapping, Information & UI - Platform & tech: Headset, Meta Quest 3, Passthrough Camera API, Unity Sentis, YOLO, OpenAI - Idea: Once the headset camera is open, every AI model becomes a mixed-reality feature. - What it is: On the day Quest opened its passthrough cameras, a set of open-source samples showed a colour picker, on-device object detection, QR code tracking, frosted-glass shaders and camera-to-LLM questions. - Technique: Camera frames from the Passthrough Camera API are fed to Unity Sentis (YOLO) on device or to cloud models, and results are projected into the room with depth. - Try it: Clone an open-source camera kit (or use a phone WebAR demo) and change the detector's output: instead of boxes, spawn a small animal on every detected cup. Twist: animals only appear on objects older than you. ### Samuele Albani *Artist and researcher in interactive sound and social choreography* Italian artist and computational-arts researcher (Goldsmiths) who designs collaborative instruments played by the distance, touch and poses of two or more people, using wearables and camera-based pose detection. At the 2026 CCL at the Royal Ballet and Opera he replaced one of the two players of his instrument distanze with an AI agent. #### CON/TOUCH #2 — Samuele Albani (2023) - Video: https://vimeo.com/862691895 - Interaction: Shared & Social, Tangible Objects, Voice & Sound - Platform & tech: Wearable, wireless wearable sensors, pressure sensing, synthesiser - Idea: An instrument that can only be played by touching another person. - What it is: Two people wear wireless sensors on their shoulders; touching and pressing each other's shoulders generates and modulates sound, making a musical instrument out of social touch. - Technique: Wireless wearable sensors measure haptic gestures on the shoulders and send them to a software synthesiser that maps them to frequency and timbre. - Try it: Sew conductive fabric pads onto two shoulder straps and connect them to a Makey Makey or micro:bit so touch triggers notes. Twist: sound only plays while both people are touching at once. #### Passaggi // Presenze — Samuele Albani (2023) - Video: https://vimeo.com/862699079 - Interaction: Spatial Mapping, Shared & Social, Voice & Sound - Platform & tech: Projection, laser proximity sensors, camera tracking, generative sound - Idea: Doorways and footsteps become the score of a room. - What it is: A site-specific installation made with Lily Hassioti: every time someone crosses a doorway the note of a synthesiser changes and their image is archived, while a camera maps where people stand so the crowd composes sound and visuals together. - Technique: Laser proximity sensors at the doors detect passages, and a machine-learning camera tracker maps visitor positions to sound parameters and projected visuals. - Try it: Put a phone camera over a classroom door and use a simple motion detector to change a chord every time someone walks through. Twist: record a snapshot of each passer and project the growing collage. #### distanze — Samuele Albani (2023) - Video: https://vimeo.com/812529965 - Interaction: Shared & Social, Hands & Body, Voice & Sound - Platform & tech: Desktop, pose detection, machine learning, software synthesiser - Idea: Play music with the space between two bodies. - What it is: A musical instrument for two players: a camera tracks both bodies and the distances between their noses, hips and hands control a synthesiser, so empty space becomes an invisible interface for music. - Technique: A machine-learning pose model analyses the camera image, and distances between chosen body keypoints of the two players are mapped to synthesiser parameters. - Try it: With ml5.js pose detection, map the distance between two people's wrists to pitch and between their heads to volume. Twist: add a third person whose presence changes the scale. ### Shinji Ohmaki (大巻伸嗣) *Installation artist working with air, fabric, bubbles and light* Japanese artist (born 1971) who fills spaces with slowly floating fabric (Liminal Air Space-Time), streams of soap bubbles (Memorial Rebirth) and rotating lights, making air and time visible. #### Liminal Air Space-Time — Shinji Ohmaki (大巻伸嗣) (2014) - Video: https://www.youtube.com/watch?v=UqXWbbkRiJ8 - Interaction: Perception & Effects - Platform & tech: Desktop, large silk fabric, fans, air flow - Idea: Air is invisible until something shows it — AR can visualize flow, wind and breath through a single responsive surface. - What it is: A vast sheet of thin fabric floats and ripples above visitors in a large hall, lifted by fans so it rises, sinks and breathes like a living surface. - Technique: Fans placed around the space create controlled air currents that keep a large, very light fabric sheet aloft and moving in slow waves. - Try it: Simulate a giant cloth floating above the floor in AR that billows when the user blows into the phone microphone. Twist: let several phones in the room each act as a 'fan' from different directions. #### Gravity and Grace — Shinji Ohmaki (大巻伸嗣) (2016) - Video: https://www.youtube.com/watch?v=g46PZLXk01A - Interaction: Perception & Effects - Platform & tech: Desktop, perforated metal vessel, rotating light source - Idea: A single light inside a patterned shell can fill every surface of a room, including people — AR light projection should land on bodies too. - What it is: A huge perforated metal vase with an intense light inside slowly rises and falls in a dark room, throwing patterns of flowers and figures across the walls, floor and visitors. - Technique: An extremely bright lamp inside a pierced metal vessel acts as a pattern projector; moving the light vertically changes the scale of the shadows. - Try it: Place a virtual lantern with a cut-out pattern in the centre of a real room and project its light patterns onto walls and people using body segmentation. Twist: let the lantern pattern change based on who stands closest. #### Memorial Rebirth — Shinji Ohmaki (大巻伸嗣) (2017) - Video: https://www.youtube.com/watch?v=SykNfKqt9kk - Interaction: Location & City, Shared & Social - Platform & tech: Desktop, bubble machines, soap bubbles, public space - Idea: Something that fills the air and vanishes brings strangers together — ephemeral, city-scale AR events can do the same. - What it is: Dozens of machines release a flood of soap bubbles over a neighbourhood street or park, turning an ordinary place into a shared, fleeting celebration. - Technique: Arrays of bubble machines are placed around a site and run in sequence so that waves of bubbles drift with the wind across the crowd. - Try it: Launch a shared AR bubble storm in a park where every participant's phone releases bubbles that everyone else can see and pop. Twist: popped bubbles leave a short sound so the park 'rings'. ### Shohei Mori *Mixed-reality researcher working on diminished reality and pseudo-weight illusions* Trained in Japan at Ritsumeikan and Keio universities and later a researcher at TU Graz, he builds diminished-reality systems that erase or see through real objects, and with Ritsumeikan's Reality Media lab studies how visually editing an object changes how heavy it feels. #### Work Area Visualization by Multi-View Diminished Reality — Shohei Mori (2017) - Video: https://www.youtube.com/watch?v=hcMjn_mD3uI - Interaction: Portals & Worlds, Hands & Body, Information & UI - Platform & tech: Desktop, diminished reality, multi-camera - Idea: See through your own hands to the work they are hiding. - What it is: Several cameras watch a workbench from different angles; the system removes the worker's hands and tools from the main view by filling them in from the other cameras, so the whole work piece stays visible while it is being assembled. - Technique: Multi-view diminished reality: the region occluded by hands in the main view is replaced with pixels from other calibrated cameras, warped into the main viewpoint. - Try it: Film a desk with two phones on stands and blend the second view into the first wherever a hand is detected (a hand-tracking mask), so hands turn semi-transparent. Twist: make only the tool vanish so it seems to move by itself. #### Exploring Pseudo-Weight in AR Extended Displays — Shohei Mori (2022) - Video: https://www.youtube.com/watch?v=5VCDReGcd40 - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Headset, Phone, video see-through HMD, pseudo-haptics - Idea: A virtual screen attached to a real phone changes how heavy the phone feels. - What it is: A user holds a phone whose screen is extended by a virtual panel shown in a headset; placing the panel farther from the hand makes the same phone feel lighter, and making the panel lag behind makes it feel heavier. - Technique: A video see-through headset tracks the phone and renders the virtual extension at a chosen offset and latency; the weight illusion comes purely from visual layout and delay (pseudo-haptics). - Try it: Anchor a virtual panel to a tracked phone or box in WebXR or AR Foundation, add sliders for its offset and follow delay, and ask classmates to rate how heavy the object feels. Twist: make the panel wobble like jelly and see whether the object feels soft. #### Perceived Weight of Mediated Reality Sticks — Shohei Mori (2025) - Video: https://www.youtube.com/watch?v=hN08_fS7oTk - Interaction: Perception & Effects, Tangible Objects, Portals & Worlds - Platform & tech: Headset, diminished reality, video see-through HMD - Idea: Change only how a stick looks, and your hand reports a different weight. - What it is: Participants swing a real wooden stick while a headset shows it longer (AR extension), shorter (diminished reality) or with a section cut out of the middle; the longer the stick looks, the lighter it feels, while a stick with a visible gap still feels like one piece. - Technique: A tracked physical stick is visually lengthened with rendered geometry, or shortened and cut with diminished-reality inpainting, in a video see-through headset; users then judge its weight and centre of gravity by wielding it. - Try it: Track a cardboard tube in AR (an image marker on its end) and render virtual extensions of different lengths; have classmates swing it and rank which version feels heaviest. Twist: keep the length but render the tube as a feather or a hammer. ### Studio Roosegaarde (Daan Roosegaarde) *Design lab creating interactive landscapes of light* Daan Roosegaarde's studio makes 'techno-poetic' public works; DUNE, an interactive landscape of light fibres first installed along Rotterdam's Maastunnel, brightens as people walk and make sound. #### DUNE — Studio Roosegaarde (Daan Roosegaarde) (2010) - Video: https://www.youtube.com/watch?v=nf-q5zs8HgE - Interaction: Hands & Body, Location & City, Voice & Sound - Platform & tech: Projection, light fibres, motion sensors, microphones - Idea: Augment a public path with nature-like light that notices the people using it. - What it is: A long landscape of thousands of light fibres along the Maastunnel in Rotterdam that brightens and pulses as pedestrians walk past and make sound. - Technique: Motion and sound sensors along the installation control LEDs at the fibre tips section by section, creating a local wave of light around each passer-by. - Try it: Line a corridor with an addressable LED strip and an ultrasonic sensor every metre so light follows walkers. Twist: whispering makes it glow softer, shouting makes it hide. #### Van Gogh Path — Studio Roosegaarde (Daan Roosegaarde) (2014) - Video: https://www.youtube.com/watch?v=f68cdc27HWg - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, glow-in-the-dark stones, solar LEDs, bicycle path - Idea: A path that becomes a painting when you ride it at night ties art to movement and route, a model for AR guidance lines with character. - What it is: In Nuenen, where Van Gogh lived, a one-kilometre bicycle path is inlaid with thousands of stones that charge in daylight and glow at night in swirls inspired by The Starry Night. - Technique: Glow-in-the-dark stones and solar-powered LEDs are set into the path surface in a swirl pattern. - Try it: Draw a glowing AR path that swirls along a real footpath and brightens under each walker's feet. Twist: the pattern is generated from a famous painting chosen by the user. #### GROW — Studio Roosegaarde (Daan Roosegaarde) (2021) - Video: https://www.youtube.com/watch?v=BId_104fAZI - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, red and blue light recipes, agricultural field - Idea: Lighting a working landscape shows the hidden value of a place people normally drive past, a strong brief for AR in agriculture. - What it is: A 20,000-square-metre field of leeks in the Netherlands is lit at night with moving red and blue light, a recipe that can boost plant growth and reduce pesticide use, making farming visible as art. - Technique: Linear lights sweep over the crop rows in patterns based on research into light recipes for plant growth. - Try it: In AR, sweep coloured light bars across a real lawn or garden and label each plant bed with its growth data. Twist: the light speeds up where plants need more care. ### Takis *Artist; sculptor of magnetism, light and sound* Greek artist Panayiotis Vassilakis (1925–2019) who made invisible forces visible: objects floating on magnetic fields, blinking 'Signaux' (signals) on tall rods, and musical sculptures played by magnets. #### Magnetic sculptures (Télésculptures) — Takis (1960) - Video: https://www.youtube.com/watch?v=Q5ktkR-xSoM - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Desktop, electromagnets, metal objects, wire - Idea: Showing an invisible force through its effect on objects is exactly how AR can make fields, signals or data feel physical. - What it is: Metal objects hang on wires or float in the air, pulled and pushed by electromagnets so that they tremble, swing and hover as if held by invisible hands. - Technique: Electromagnets switched on and off at intervals attract and release suspended metal elements, keeping them in unstable floating balance. - Try it: Make an AR scene where small virtual metal objects float and tremble around a real object (tracked with image or object tracking) acting as a magnet. Twist: show the field lines only when the user taps and holds. #### Musical sculptures (Musicales) — Takis (1966) - Video: https://www.youtube.com/watch?v=jBK_m68TVf0 - Interaction: Voice & Sound, Tangible Objects - Platform & tech: Desktop, electromagnets, needles, guitar strings, amplifier - Idea: An instrument played by magnetism instead of hands makes invisible forces audible; AR can sonify invisible data or fields in the same spirit. - What it is: Single strings stretched over wooden panels are struck by needles swinging under electromagnets, producing random amplified tones in the gallery. - Technique: An electromagnet pulses at irregular intervals, pulling a suspended needle against an amplified string so that magnetism plays the instrument. - Try it: Anchor AR strings to a real wall and have invisible 'magnetic' pulses, driven by the phone's magnetometer, pluck them. Twist: bring a real magnet near the phone to play them directly. #### Signaux lumineux (Bassin Takis, La Défense) — Takis (1990) - Video: https://www.youtube.com/watch?v=Z6WjEcBXGPs - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, steel rods, coloured lights, water basin - Idea: Blinking lights at different heights in a field read as a language of signals; AR wayfinding and data layers can borrow this calm, distributed signalling. - What it is: Forty-nine tall flexible steel masts topped with coloured blinking lights stand in a reflecting pool at La Défense, swaying slightly like signals at a railway yard or harbour. - Technique: Tall slender masts flex in the wind while lamps at their tips blink in varied rhythms, doubled by reflections in the water basin. - Try it: Plant a field of AR signal masts on a real square, each blinking a rhythm tied to live data (bus arrivals, air quality). Twist: let users claim a mast and set its rhythm as a message for friends. ### Tatsuo Miyajima (宮島達男) *Artist; LED counters* Japanese artist whose LED digits count from 1 to 9 (never 0) at different speeds, representing lives and time, from the Roppongi wall Counter Void to the building-sized projection Time Waterfall. #### Counter Void — Tatsuo Miyajima (宮島達男) (2003) - Video: https://www.youtube.com/watch?v=HSO-5P1i71k - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, neon, digital counters, glass wall - Idea: Numbers counting at different speeds can speak about life and loss; a public AR piece can carry memory by being switched on only on certain days. - What it is: Six giant digital counters, each over three metres tall, glow on a glass wall in Roppongi, Tokyo, counting down from 9 to 1 at different speeds and never showing zero; after the 2011 earthquake it was switched off and is relit each March. - Technique: Neon-lit seven-segment numerals behind a glass facade are driven by controllers that set a different counting rhythm for each digit. - Try it: Place a row of AR counters on a school wall with 8th Wall image targets, each counting down at a speed chosen by a different student and skipping zero. Twist: make the counters appear only on one date of the year that the class chooses. #### Sea of Time – TOHOKU — Tatsuo Miyajima (宮島達男) (2017) - Video: https://www.youtube.com/watch?v=I7YyVE1touE - Interaction: Shared & Social, Location & City, Perception & Effects - Platform & tech: Projection, LED counters, water, community participation - Idea: Letting each participant set one parameter of their own light makes a shared memorial personal; multi-user AR can collect such small choices into one collective piece. - What it is: A long-term project in which people from the areas hit by the 2011 tsunami each set the speed of one LED counter; thousands of counters are to glow under water as a memorial. - Technique: Individual LED counter units, each programmed with the speed chosen by one participant, are placed in a shallow pool of water (likely in stages over several years). - Try it: Build a shared WebXR pond on the floor where each student sets the speed and colour of one glowing counter and drops it into the water. Twist: let counters fade when their author leaves the room and reappear when they return. #### Time Waterfall — Tatsuo Miyajima (宮島達男) (2017) - Video: https://www.youtube.com/watch?v=A-JMlkH7_BA - Interaction: Location & City, Projection, Perception & Effects - Platform & tech: Projection, LED facade, digital numbers - Idea: Turning a skyscraper into a clock of falling numbers shows how city-scale surfaces can hold one simple idea; geo-AR can do the same on any tall building. - What it is: Commissioned by Art Basel, streams of numbers fall down the 490-metre facade of Hong Kong's International Commerce Centre like a waterfall, visible across Victoria Harbour. - Technique: The tower's LED lighting facade displays animated digits flowing downward, each column counting at its own pace. - Try it: Use ARCore Geospatial or a building image target to pour falling digits down the tallest building visible from school, with each column seeded by a classmate's birthday. Twist: freeze the waterfall whenever someone stands still and looks at it for five seconds. ### Teehan+Lax Labs *R&D lab of the Toronto design agency Teehan+Lax* The experimental arm of the digital agency Teehan+Lax (2010-2014), which published short openFrameworks and hardware prototypes about new interfaces, from touching distant screens through a phone to physical ASCII art. #### Touch Vision Interface — Teehan+Lax Labs (2011) - Video: https://vimeo.com/28792538 - Interaction: Information & UI, Shared & Social, Projection - Platform & tech: Phone, Projection, openFrameworks, iOS - Idea: Touch a far-away screen through your phone's camera view. - What it is: A user points a phone camera at a distant screen or projection and draws or drags on the phone's live camera view; the change appears on the real surface as if touched directly. - Technique: The phone image is matched to the target display, and touch coordinates are transformed through that homography and sent over the network to the openFrameworks app running the screen. - Try it: Use image tracking on a projected wall so that tapping it in a phone AR view places a sticker at the same spot on the real wall. Twist: several phones draw on the same wall at once. #### Painting with a Digital Brush — Teehan+Lax Labs (2012) - Video: https://vimeo.com/46636045 - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Wearable, LED strip, long-exposure photography - Idea: Free ASCII art from the screen and paint it into the air with light. - What it is: An ASCII-art image is painted into physical space in long-exposure photographs: a light stick moved across the room displays characters that build up the picture in mid-air. - Technique: The image is converted to characters and played column by column on an LED strip whose position is synchronized with the camera's long exposure. - Try it: Use a phone screen as a brush: show one slice of an image at a time while walking in front of a long-exposure camera. Twist: paint a message that is only readable from one position in the room. #### D.I.G.I.T. — Teehan+Lax Labs (2013) - Video: https://vimeo.com/79332227 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, seven-segment displays, camera - Idea: A mirror made of calculator digits. - What it is: A wall of hundreds of seven-segment displays acts as a mirror: the camera image of the person in front is drawn with digital number segments. - Technique: The camera feed is downsampled and each cell's brightness pattern is matched to the closest seven-segment character, then sent to driver boards. - Try it: Make a webcam mirror that redraws the image with only characters from a limited set (digits, emoji or letters of your name). Twist: the set of allowed characters changes with the time of day. ### Theo Jansen *Kinetic artist* Dutch artist who since 1990 has evolved Strandbeests, wind-powered walking creatures made of PVC tubes that roam the beaches of the Netherlands. #### Strandbeest evolution (archive) — Theo Jansen (2013) - Video: https://www.youtube.com/watch?v=MYGJ9jrbpvg - Interaction: Location & City, Play, Perception & Effects - Platform & tech: Desktop, PVC tubes, Jansen linkage, wind power - Idea: A machine that walks by the wind makes an invisible force legible as a creature, a good model for AR creatures driven by real sensor data. - What it is: A compilation of Strandbeests from 1990 onward walking along Dutch beaches, powered only by wind, from early fragile models to large beasts that store air in plastic bottles. - Technique: Each leg is a Jansen linkage of PVC tubes whose proportions (found by a genetic algorithm) turn rotation into a smooth walking step. - Try it: Simulate a Jansen linkage in Unity or p5.js, place the walking beast on a beach or lawn in AR and drive its speed with live wind data. Twist: it stops and 'anchors' itself when the wind gets too strong. #### Strandbeest Evolution 2021 — Theo Jansen (2021) - Video: https://www.youtube.com/watch?v=C97kMKwZ2-g - Interaction: Location & City, Play - Platform & tech: Desktop, PVC tubes, wind power, annual iteration - Idea: Treating each version as a species with a life cycle turns iteration into narrative, a way to present AR prototypes as a lineage. - What it is: Jansen's annual report: every spring he takes a new beast to the beach, tests it against wind, sand and water all summer, and in autumn declares it extinct and moves it to the 'bone yard'. - Technique: New leg mechanisms, sails and air-storage systems are built each year and tested on the beach with live wind. - Try it: Show all versions of your semester's AR prototype side by side as a 'museum of extinct species' in one AR scene. Twist: each prototype has a label with its birth date, cause of extinction and descendant. #### Strandbeest Evolution 2025 — Theo Jansen (2025) - Video: https://www.youtube.com/watch?v=ANhA94ZqnEQ - Interaction: Location & City, Play - Platform & tech: Desktop, PVC tubes, wind power - Idea: Thirty-five years of one idea show that depth comes from iterating on a single system, a lesson for long-running AR projects. - What it is: The 2025 update on the Strandbeest family, which has been evolving since 1990, shows the newest beast walking on the beach and its experiments with wind and water. - Technique: Continuation of the PVC linkage system with new adaptations tested on the beach. - Try it: Give an AR creature three 'genes' (leg length, speed, sail size) and let the class breed a new generation each week by combining the fittest. Twist: fitness is measured by how far it walks in the real wind outside. ### Theo Triantafyllidis *Artist working with games, simulation and mixed reality* Greek-born artist whose live simulations and mixed-reality performances (Anti-Gone, All My Confusing Pets) blur performer, avatar and game engine. #### Studio Visit (mobile portal) — Theo Triantafyllidis (2018) - Video: https://www.youtube.com/watch?v=zCc_qfNEAeM - Interaction: Portals & Worlds - Platform & tech: Phone, Unity, ARKit - Idea: A phone AR portal leads into the artist's bizarre studio. - What it is: A site-specific mixed-reality installation: a mobile AR portal opens from the gallery into a surreal virtual version of the artist's studio. - Technique: An AR portal uses a stencil-buffer mask on a doorway plane so the virtual studio is only visible through the opening, and the scene is registered to the gallery wall. - Try it: Use a stencil shader in Unity AR to build a portal that leads to an exaggerated fictional version of your desk. Twist: let a few objects 'leak' out of the world behind the door onto the real side. #### All My Confusing Pets — Theo Triantafyllidis (2019) - Video: https://www.youtube.com/watch?v=WqdZMLGUfnE - Interaction: Performance, Play - Platform & tech: Headset, Unity, mixed reality - Idea: A mixed-reality performance where the artist shares the stage with a pack of chaotic virtual pets. - What it is: A mixed-reality performance where the artist interacts with glitchy virtual pets composited into the real space. - Technique: Mixed-reality compositing of glitchy AI-driven creatures in Unity into the real space, with the performer's tracked pose used for interaction and collision. - Try it: Place a few virtual pets with conflicting behavior rules (chase you, avoid you, copy you) in HoloKit or phone AR, and record a performance of you trying to look after them. Twist: make the pets' glitches (jitter, clipping) part of their personalities. #### Anti-Gone — Theo Triantafyllidis (2020) - Video: https://www.youtube.com/watch?v=qMvq9pYd2PI - Interaction: Performance, Hands & Body - Platform & tech: Projection, motion capture, Unity - Idea: A swamp theater where a motion-captured performer drives a virtual creature in real time. - What it is: A live mixed-reality performance (Sundance 2020) where a motion-captured performer drives an ork-like avatar through a simulated swamp in real time. - Technique: Real-time motion capture streams the performer's skeleton into Unity, retargeted to a game avatar in a simulated environment and projected for a live audience. - Try it: Use MediaPipe Pose or a phone motion-capture app to drive a strange creature in Unity in real time, and project it on a wall to perform a 3-minute monologue live. Twist: let the audience vote on their phones to change the weather in the creature's environment. ### Thomas Dambo *Recycle artist* Danish artist who builds giant trolls from scrap wood and hides them in forests around the world, each with a story and a treasure-map trail. #### Forest Giants (Bernheim Forest) — Thomas Dambo (2019) - Video: https://www.youtube.com/watch?v=Db0x_ptA8FU - Interaction: Location & City, Play, Tangible Objects - Platform & tech: Desktop, recycled wood, forest trail - Idea: Characters hidden along trails turn a walk into a search, the physical version of an AR scavenger hunt. - What it is: Three giant trolls built from scrap wood (a mother and two children) sit and lie along trails in Bernheim Forest, Kentucky, found by hikers exploring the woods. - Technique: Pallets and scrap wood are cut and screwed onto a timber frame, with faces sculpted from branches, built with local volunteers. - Try it: Hide three AR characters in a park with Niantic Lightship or 8th Wall, each with a riddle pointing to the next. Twist: each character only appears when the user is quiet (low microphone level). #### Long Leif, the biggest troll — Thomas Dambo (2024) - Video: https://www.youtube.com/watch?v=U5Kx20pRwkY - Interaction: Location & City, Play - Platform & tech: Desktop, recycled wood, birdhouses, Minnesota - Idea: A network of characters with a story across a region shows how to design a multi-site AR world with one mythology. - What it is: In Minnesota, Dambo builds Long Leif, his biggest troll yet, as part of a trail of nine sculptures and 700 birdhouses made from recycled materials. - Technique: Large timber frames are clad with reclaimed wood; birdhouses from recycled materials are placed along the trail. - Try it: Write a short myth for your neighbourhood and place its characters as AR figures at five real locations, with a map. Twist: add AR birdhouses that real birds' sounds can 'move into' when recorded nearby. #### Rose Wonders (Burning Man) — Thomas Dambo (2025) - Video: https://www.youtube.com/watch?v=_YrbYB2PtWs - Interaction: Location & City, Shared & Social - Platform & tech: Desktop, recycled wood, desert, Burning Man - Idea: A giant character in an empty desert becomes a gathering point for a temporary city, a model for AR landmarks at festivals. - What it is: At Burning Man 2025 in the Nevada desert, Dambo's biggest sculpture yet, Rose Wonders, is a giant wooden troll figure in the playa, introduced with a poem. - Technique: A large wooden structure of recycled timber was prefabricated and assembled on the playa. - Try it: Design an AR landmark character for a school festival that everyone can find from any point on the site and that recites a short poem when approached. Twist: add one line to the poem for every visitor. ### Timo Arnall *Designer and filmmaker; Immaterials project* Designer and filmmaker who, with Jørn Knutsen and Einar Sneve Martinussen at the Oslo School of Architecture and Design, made the 'Immaterials' films that visualise RFID fields and WiFi signal strength with long-exposure light. #### Immaterials: the ghost in the field — Timo Arnall (2009) - Video: https://vimeo.com/7022707 - Interaction: Information & UI, Perception & Effects, Spatial Mapping - Platform & tech: Desktop, RFID, LED, long exposure, stop motion - Idea: Invisible technical fields have shapes: AR can show the real reach of sensors, beacons and NFC so people understand where interactions work. - What it is: An RFID reader with an LED that lights up whenever it detects a tag is moved around the tag in steps and photographed with long exposures, revealing the invisible 3D shape of the reading field. - Technique: An LED on the reader is switched on only when a tag is read; stepping the reader through space during long exposures and animating the frames builds a volumetric map. - Try it: Walk a phone around a room while logging Bluetooth signal strength from a beacon, then render each sample as a colored sphere at the phone's AR position. Twist: animate the spheres to pulse at the beacon's advertising rate. #### Immaterials: Light painting WiFi — Timo Arnall (2011) - Video: https://www.youtube.com/watch?v=cxdjfOkPu-E - Interaction: Information & UI, Location & City, Perception & Effects - Platform & tech: Desktop, WiFi, LED bar, long exposure, Arduino - Idea: Walking a sensor through the city and drawing its readings in place is situated data visualization — the core promise of AR data overlays. - What it is: A 4 m rod of 80 LEDs shows the WiFi signal strength as a bar graph while it is walked through Oslo streets; long exposures turn the readings into glowing walls of data in the city. - Technique: A measuring rod with an LED bar graph displays live RSSI readings, photographed with long exposures as it moves along a path. - Try it: Log WiFi or cellular signal strength with GPS while walking a campus route, then display it as a vertical bar wall along the route in AR. Twist: let two groups log different networks and show both walls side by side. #### Robot Readable World — Timo Arnall (2012) - Video: https://www.youtube.com/watch?v=7DWsMzyX9so - Interaction: Perception & Effects, Spatial Mapping, Information & UI - Platform & tech: Desktop, computer vision, found footage - Idea: The machine's view is itself a visual language; exposing tracking points and meshes can be a deliberate AR aesthetic instead of a debug mode. - What it is: A short film assembled from computer-vision research footage: tracking boxes, feature points, depth maps and labels show how machines see streets, faces and objects. - Technique: Found footage from computer-vision demos, with overlays of feature detection, segmentation and tracking, edited into a single film. - Try it: Build an AR filter that renders only the debug layers of AR tracking — feature points, plane meshes, depth — as the main visual style, and film a walk through a building with it. Twist: let the mesh age and fade so the machine's memory of the room becomes visible. ### Timoni West *Designer; former head of XR research at Unity Labs (EditorXR, Carte Blanche, Unity MARS)* Led Unity Labs' XR research team, which prototyped authoring VR inside VR (EditorVR, Carte Blanche) and then Unity MARS, a tool for building AR apps that adapt to any real room. She speaks widely on designing tools for spatial computing. #### Carte Blanche — Timoni West (2016) - Video: https://www.youtube.com/watch?v=bKR0eEKTiBM - Interaction: Tangible Objects, Drawing & Making, Play - Platform & tech: Headset, Unity, VR - Idea: Make world-building feel like a card game played on a tabletop. - What it is: A Unity Labs concept video for VR authoring aimed at non-programmers: users deal virtual cards to spawn objects, arrange a scene on a table-sized stage and give simple commands to a helper character. - Technique: Likely a prototype where assets are represented as physical-feeling cards in a hand-tracked deck, with a miniature 'world in miniature' view for placement. - Try it: Make a deck of printed image-marker cards; laying a card on the table in phone AR spawns its object in a miniature scene. Twist: stacking two cards combines their objects. #### EditorVR — Timoni West (2016) - Video: https://www.youtube.com/watch?v=FzjxRi5J4XI - Interaction: Drawing & Making, Hands & Body, Information & UI - Platform & tech: Headset, Unity, HTC Vive, Oculus Rift - Idea: Build spatial content where it will be experienced, with your hands, instead of on a flat monitor. - What it is: On stage at the Vision VR/AR Summit, West puts on a headset and builds a Unity scene from inside VR, grabbing assets from floating panels and placing them with tracked controllers. - Technique: The Unity editor's scene hierarchy, inspector and asset browser are re-exposed as 3D workspaces attached to the controllers, editing the same scene data as the desktop editor. - Try it: Design and paper-prototype a three-panel VR editing tool for arranging furniture, then build one panel in WebXR. Twist: every tool must be operable with one hand. #### Unity MARS — Timoni West (2020) - Video: https://www.youtube.com/watch?v=825WgLYUqvg - Interaction: Spatial Mapping, Information & UI - Platform & tech: Phone, Headset, Unity MARS, AR Foundation - Idea: Author AR against rules about reality rather than a single known room. - What it is: An authoring tool for AR apps where content is placed by conditions (on a table at least 1 m wide, next to a face, on the floor near a wall) and tested in simulated rooms before being deployed to real ones. - Technique: Proxies with semantic and size conditions are matched at runtime against planes, images and faces reported by AR Foundation, and a simulation view replays recorded environments in the editor. - Try it: Write an AR scene whose three objects are placed only by rules (largest table, nearest wall, floor near a door) and test it in two rooms. Twist: add a fallback when no rule is satisfied. ### Tonchidot (Takahito Iguchi) *Startup; Sekai Camera AR app* Tonchidot, led by Takahito Iguchi, showed Sekai Camera at TechCrunch50 in 2008: an iPhone app that let people leave floating 'air tags' of text, photos and sound in places. #### Sekai Camera (TechCrunch50 demo) — Tonchidot (Takahito Iguchi) (2008) - Video: https://www.youtube.com/watch?v=KgTwSXK_5dg - Interaction: Location & City, Shared & Social, Information & UI - Platform & tech: Phone, iPhone, GPS, compass - Idea: Let people leave words, photos and sounds at real places like floating bubbles. - What it is: The TechCrunch50 demo of Sekai Camera: point an iPhone at a street or shop and see floating 'air tags' left by other people, with notes, photos and voice memos attached to places. - Technique: GPS position and compass heading determine which geotagged notes lie in the camera's field of view, and tags are drawn as floating labels at their bearing. - Try it: Use the phone's GPS and compass to build 'floating tags' in a web page that show the messages left in whichever direction you face. Twist: scale tag font size by distance so faraway messages blur. #### Sekai Camera science museum navigation — Tonchidot (Takahito Iguchi) (2009) - Video: https://www.youtube.com/watch?v=1De9YCeKz_A - Interaction: Information & UI, Location & City - Platform & tech: Phone, iPhone, Sekai Camera - Idea: Guide visitors through a science museum with air tags. - What it is: Sekai Camera air tags used as a guide inside a science museum, pointing visitors to exhibits through the camera view. - Technique: Location-tagged labels pointing to exhibits are overlaid on the camera view based on device position and heading, working as indoor wayfinding. - Try it: Place a few QR codes or image markers around a teaching building so that scanning one shows an AR arrow pointing to the next spot. Twist: the arrow's style is decided by the previous visitor's rating. #### Sekai Camera for iPad — Tonchidot (Takahito Iguchi) (2010) - Video: https://www.youtube.com/watch?v=YGwyhEK8mV8 - Interaction: Location & City, Information & UI - Platform & tech: Phone, iPad, Sekai Camera - Idea: Turn the iPad into a large window for seeing the 'air tags' floating around the city. - What it is: Sekai Camera on the iPad: the larger screen becomes a window onto the air-tag layer around the user. - Technique: The same geotag overlay approach is applied on a larger screen, with the device treated as a handheld window onto the tag layer. - Try it: Compare the same AR tag interface on a phone and a tablet, and adjust tag density and font size to suit the larger screen. Twist: design an interaction that only makes sense on the big screen. ### Toshio Iwai (岩井俊雄) *Media artist and game designer* Japanese media artist whose early Time Stratum zoetropes used the flicker of TV monitors as a strobe; he later created Morphovision with NHK, the piano performance with Ryuichi Sakamoto, Electroplankton and the Tenori-on instrument. #### Time Stratum II (時間層II) — Toshio Iwai (岩井俊雄) (1985) - Video: https://www.youtube.com/watch?v=iHHQ9SRBn1s - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, zoetrope, video monitor flicker, paper cut-outs - Idea: Any light source with a rhythm can be a strobe; AR designers learn that timing, not hardware, is what creates animation. - What it is: Paper figures arranged on a rotating disc come alive as an animation because they are lit only by the flicker of video monitors, shown here at the Taro Okamoto Museum of Art. - Technique: Iwai synchronises the rotation of a zoetrope disc with the flicker of TV screens, so the screens act as a stroboscope that freezes each figure in turn (year approximate). - Try it: Print a zoetrope disc, spin it on a turntable, and film it with a phone camera at a matched shutter speed; then add AR figures that match the disc's frames. Twist: use a screen flicker from another phone as the only light. #### Music Plays Images x Images Play Music (with Ryuichi Sakamoto) — Toshio Iwai (岩井俊雄) (1996) - Video: https://www.youtube.com/watch?v=QRHwP82WiK4 - Interaction: Voice & Sound, Performance, Projection - Platform & tech: Desktop, Disklavier, projection, MIDI - Idea: Placing imagery exactly over the source of sound fuses the two senses; AR music visuals work best when they sit on the instrument itself. - What it is: Ryuichi Sakamoto plays a self-playing grand piano while images projected onto a scrim above the keys both respond to and trigger the notes, so music and image perform together. - Technique: MIDI from the Disklavier piano drives projected graphics, and graphic events can send MIDI back to make the piano's keys move by themselves (Prix Ars Electronica Golden Nica 1997). - Try it: Track a real keyboard with image tracking and spawn AR particles above each key from Web MIDI input. Twist: let particles that fall back onto a key play that note. #### Morphovision — Toshio Iwai (岩井俊雄) (2005) - Video: https://www.youtube.com/watch?v=GbXeybKgIyY - Interaction: Perception & Effects, Projection, Tangible Objects - Platform & tech: Desktop, rotating model, line-scanned light, high-speed projector - Idea: By controlling which slice of a real object is lit and when, you can deform reality itself; this is diminished and altered reality done with light. - What it is: A small model house spins rapidly while a projector scans it line by line with light, so the solid house appears to bend, melt and twist like rubber. - Technique: Developed with NHK Science & Technology Research Laboratories: a high-speed projector illuminates the rotating model with a moving slit of light, so each horizontal slice is seen at a different rotation angle. - Try it: Rotate a scanned real object in AR and offset each horizontal slice's rotation by its height using a vertex shader. Twist: let the viewer's phone tilt control how much the object twists. ### Ugo Rondinone *Artist* Swiss artist who stacks huge boulders into colourful totems, most famously Seven Magic Mountains in the Nevada desert, and places stone giants in city plazas. #### Human Nature — Ugo Rondinone (2013) - Video: https://www.youtube.com/watch?v=Jlk2SVAtdAk - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, bluestone, stacked figures, city plaza - Idea: Crude, ancient-looking figures in a polished business plaza create strong contrast, a design cue for AR figures that do not blend in. - What it is: Nine stone giants, 5 to 6 metres tall and made of stacked bluestone blocks, stood in Rockefeller Plaza in New York among the office towers. - Technique: Rough-hewn bluestone slabs were stacked to form head, body and legs and fixed with internal steel. - Try it: Build rough stacked-stone AR giants from three to five blocks each and place them among glass buildings on campus. Twist: the giants slowly rotate to face whoever is closest. #### Seven Magic Mountains — Ugo Rondinone (2016) - Video: https://www.youtube.com/watch?v=GDqbobusO80 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, painted limestone boulders, desert - Idea: Bright artificial colour against a natural landscape reads instantly as 'placed', the look many AR objects have, used here on purpose. - What it is: Seven towers of stacked limestone boulders, painted in fluorescent colours and up to 11 metres tall, stand in the Nevada desert south of Las Vegas. - Technique: Locally quarried boulders were stacked on steel pins and painted with bright, weather-resistant paint. - Try it: Stack scanned rocks into seven AR totems in a desert-like space and paint each stone a flat bright colour. Twist: let visitors restack one stone per visit. #### Liverpool Mountain — Ugo Rondinone (2018) - Video: https://www.youtube.com/watch?v=Y81OJiom72Y - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, painted boulders, waterfront - Idea: The same object placed in a new setting (desert to harbour) changes its meaning, which AR creators can test instantly by relocating content. - What it is: A stack of brightly painted boulders stands on the Liverpool waterfront outside Tate Liverpool, bringing the desert totem idea into a port city. - Technique: Boulders were stacked on a steel core and painted in solid bright colours. - Try it: Place the same bright AR totem in three places (a car park, a garden, a lobby) and interview passers-by about what it means in each. Twist: change one colour of the totem to match each site's dominant colour. ### Unity Visual Effect Graph team (Thomas Iché, Julien Fryer, Vlad Neykov and the Unity Demo Team) *Unity Technologies team that builds VFX Graph and its showcase samples* The Unity team behind the Visual Effect Graph, who publish the Spaceship Demo, the VFX Graph sample scenes and demo-team tools such as Mesh-to-SDF as open projects. #### Spaceship Demo — Unity Visual Effect Graph team (Thomas Iché, Julien Fryer, Vlad Neykov and the Unity Demo Team) (2018) - Video: https://www.youtube.com/watch?v=rqMcPZoEc3U - Source code: https://github.com/Unity-Technologies/SpaceshipDemo - Interaction: Perception & Effects, Play - Platform & tech: Desktop, Unity, VFX Graph, HDRP - Idea: Show everything a GPU particle graph can do inside one playable space. - What it is: A first-person walk through a sci-fi spaceship full of sparks, holographic tables, smoke and a glowing reactor core, all made with VFX Graph. - Technique: HDRP scenes use VFX Graph systems with GPU simulation, signed distance field collisions, point caches and lit particles, triggered through Timeline and gameplay events. - Try it: Pick one effect from the Spaceship Demo (the holo-table pin screen) and rebuild it at tabletop scale in phone AR over a real desk. Twist: the pin screen shows the LiDAR depth of the room. #### Mesh-to-SDF — Unity Visual Effect Graph team (Thomas Iché, Julien Fryer, Vlad Neykov and the Unity Demo Team) (2022) - Video: https://user-images.githubusercontent.com/6276154/201238758-cc87ec4a-a65a-4cb2-b3cc-123576ab9ea2.mov - Source code: https://github.com/Unity-Technologies/com.unity.demoteam.mesh-to-sdf - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Compute Shader, SDF - Idea: Give a moving body a distance field every frame and effects can feel its shape. - What it is: An animated character is wrapped in a live signed distance field, so particles and smoke flow around and cling to its moving body. - Technique: A compute-shader generator splats the skinned mesh into a 3D texture and flood-fills distances every frame (Unity Demo Team), which VFX Graph samples for collision and attraction. - Try it: Generate an SDF from the AR body-tracking mesh and let VFX Graph smoke wrap around a classmate on the phone screen. Twist: the smoke is repelled by one hand and attracted by the other. #### VFX Graph Smoke Portal Sample — Unity Visual Effect Graph team (Thomas Iché, Julien Fryer, Vlad Neykov and the Unity Demo Team) (2023) - Video: https://www.youtube.com/watch?v=57cKxN3XdEY - Source code: https://github.com/Unity-Technologies/VisualEffectGraph-Samples - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, HDRP, Six-way lighting - Idea: Smoke that is lit like real smoke makes a portal feel physical. - What it is: A swirling ring of lit smoke and embers forms a portal, with the smoke catching light from the fire and the scene around it. - Technique: Flipbook smoke particles use six-way lighting (baked light maps from six directions) so they respond to scene lights, combined with VFX Graph spawning along a ring and distortion. - Try it: Place the smoke portal on a real doorway with AR Foundation and light it with AR light estimation so the smoke matches the room. Twist: the portal only opens when two people stand on either side. ### Velan Studios (Karthik & Guha Bala) *Game studio; Mario Kart Live and Hot Wheels Rift Rally* Founded by brothers Karthik and Guha Bala, Velan Studios builds mixed-reality games that pair real RC cars with camera-based AR tracks. #### Mario Kart Live: Home Circuit — Velan Studios (Karthik & Guha Bala), Nintendo (2020) - Video: https://www.youtube.com/watch?v=f2mCqUSDCJE - Interaction: Tangible Objects, Spatial Mapping, Play - Platform & tech: Desktop, Nintendo Switch, RC car camera, marker gates - Idea: A real RC kart with a camera turns your home into a Mario Kart track. - What it is: A real RC kart with a camera drives around cardboard gates in your home; on the Switch, the camera feed becomes a Mario Kart track with items, rivals and weather overlaid on your rooms. - Technique: Fiducial markers on cardboard gates localize the RC car's onboard camera, so the track, items and rivals are rendered from the kart's viewpoint over its live video. - Try it: Mount a phone on a toy car, lay a few image markers on the floor as gates, and show speed-boost items in AR when the car passes them. Twist: arrange the gates into a track that is visible only in AR. #### Mario Kart Live: Home Circuit — developer interview — Velan Studios (Karthik & Guha Bala) (2020) - Video: https://www.youtube.com/watch?v=A7O7jzRfZ4k - Interaction: Tangible Objects, Spatial Mapping - Platform & tech: Desktop, Nintendo Switch, computer vision - Idea: The developers explain how they fused a real car with a virtual track. - What it is: Velan Studios explains how they built the kart, the gate-based track system and the latency-sensitive AR rendering that makes a living room feel like a race course. - Technique: The gates both define the track path and correct camera pose drift, and rendering is tuned to minimize latency between the car's video and its overlays. - Try it: Measure the latency from camera frame to overlay in your own AR prototype, for example with high-speed video, and try one way to reduce it. Twist: deliberately increase the latency and find the threshold where users start to feel uncomfortable. #### Hot Wheels: Rift Rally — Velan Studios (Karthik & Guha Bala) (2023) - Video: https://www.youtube.com/watch?v=b0Q0fYl0pH0 - Interaction: Tangible Objects, Spatial Mapping, Play - Platform & tech: Phone, Desktop, RC car camera, iOS, PlayStation - Idea: Race a smart Hot Wheels car with a camera around your home while the screen overlays the track and rivals. - What it is: A smart RC car with a camera, controlled from a phone or console, races through gates at home while the screen overlays a Hot Wheels track, other racers and rift effects. - Technique: A camera car streams video to a phone or console that tracks gate markers for localization and overlays track geometry and effects in the car's perspective. - Try it: Watch the camera feed of a second phone (mounted on a small car) remotely from your phone and overlay AR track elements on it. Twist: add a 'rift' effect that appears only after the car passes a certain picture. ### Walter De Maria *Sculptor and land artist* American artist (1935–2013) known for The Lightning Field, 400 steel poles on a grid in the New Mexico desert, and for rooms filled with earth. #### The Lightning Field — Walter De Maria (1977) - Video: https://www.youtube.com/watch?v=iozjJgjOsUg - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, stainless steel poles, grid, desert plateau - Idea: A strict grid laid over wild land makes both light and terrain readable; AR grids can do the same for invisible fields. - What it is: Four hundred stainless-steel poles stand in a grid one mile by one kilometre in the high desert of New Mexico, their tips forming a perfectly level plane; visitors stay overnight to watch the light change (and occasionally lightning). - Technique: Poles of varying height (average about 6 metres) were surveyed so their pointed tips align to one horizontal plane over uneven ground. - Try it: Lay out a 10×10 grid of thin AR poles on an uneven field with tips locked to one world height, and watch it at sunset. Twist: make poles glow brighter where live weather data reports higher humidity or wind. #### The New York Earth Room — Walter De Maria (1977) - Video: https://www.youtube.com/watch?v=krF9DEH327w - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, soil, SoHo loft - Idea: Filling a whole room with one natural material replaces the interior with a landscape, the physical version of AR room replacement. - What it is: A SoHo loft in New York is filled with 127 cubic metres of dark soil, 56 centimetres deep; a caretaker waters and rakes it, and visitors look in from the doorway. - Technique: Soil is spread evenly across the floor to a fixed depth behind a glass barrier and maintained weekly. - Try it: Use ARKit scene reconstruction to fill a real room's floor with an AR soil layer 50 cm deep that correctly hides furniture legs. Twist: let footprints appear in the soil where people walk. #### The Vertical Earth Kilometer — Walter De Maria (1977) - Video: https://www.youtube.com/watch?v=oCwu3H0m35o - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, brass rod, drilled shaft, public square - Idea: An artwork almost entirely invisible and known only by its description shows that AR can point to things we can never see, such as depth. - What it is: For documenta 6 in Kassel, a brass rod one kilometre long was sunk vertically into the ground of Friedrichsplatz; all that can be seen is its end, a small circle flush with a sandstone plate. - Technique: A shaft was drilled for 79 days and 167 brass segments were screwed together and lowered into it. - Try it: Mark a spot on the ground and, in AR, show what lies directly beneath it to one kilometre depth (soil layers, pipes, bedrock) from open geological data. Twist: show nothing but a small circle until the user kneels down. ### Yannick Jacquet (Legoman) *Media artist and art director; former AntiVJ member* Visual artist known as Legoman in the AntiVJ label; his mapped installations mix physical print, mechanism and projection so that drawn machines and images appear to move on paper and walls. #### Intangible States (with Stray Dogs) — Yannick Jacquet (Legoman) (2010) - Video: https://www.youtube.com/watch?v=cgU7F42vgj4 - Interaction: Projection, Performance, Voice & Sound - Platform & tech: Projection, live projection mapping, VJ software - Idea: Project an urban drifter's dreams onto a stage built from a pile of boxes. - What it is: A live audiovisual performance where video by Legoman is projection-mapped onto a stage set of stacked boxes and screens, evoking the dream states of an urban nomad. - Technique: Live VJ software maps video onto each face of stacked boxes using masks and corner-pinning, performed in sync with live electronic music. - Try it: Stack cardboard boxes into a stage and use free mapping software (such as MapMapper or a Resolume trial) to put different videos on each face. Twist: switch each box's image live in response to the music. #### Mécaniques Discursives (with Fred Penelle) — Yannick Jacquet (Legoman) (2011) - Video: https://www.youtube.com/watch?v=3URvRzYkSGo - Interaction: Projection, Drawing & Making, Tangible Objects - Platform & tech: Projection, projection mapping on print, woodcut - Idea: Projection makes the printed machine on the wall actually turn. - What it is: A wall of woodcut prints depicting a strange machine is brought to life by precise projection: gears turn, creatures move and light flows through the printed mechanism. - Technique: Video is drawn to match the exact contours of hand-printed woodcuts on the wall and projected in registration so animation appears to live inside the paper. - Try it: Hand-draw a black-and-white 'machine' and pin it to the wall, photograph it, animate each part rotating on the computer, and project it back in alignment. Twist: the machine changes speed when the audience claps. #### Remote Memories (with Laurent Delforge / Before Tigers) — Yannick Jacquet (Legoman) (2016) - Video: https://www.youtube.com/watch?v=v23ZJmWdmo4 - Interaction: Projection, Perception & Effects, Voice & Sound - Platform & tech: Projection, projection mapping, polyptych - Idea: Fragments of memory pieced into a panoramic projection you have to take in slowly. - What it is: A panoramic polyptych of projection-mapped panels and sound that slowly assembles a pictorial landscape of memories, also performed live. - Technique: Video is mapped across several physical panels of different sizes and depths so a single composition spans the polyptych. - Try it: Arrange five whiteboards of different sizes into a polyptych and project an animation of old photos across all the panels. Twist: each panel stands for a fragment of memory told by a different person. ### Yeast Culture *Visual collective (live concert visuals and installations)* British collective that creates live visuals for classical concerts, bands and gallery installations and also directs concert films. Projects include visuals for Ludovico Einaudi at the Royal Albert Hall, Nitin Sawhney and Bernstein's Mass at the Royal Festival Hall, and a live motion-capture performance with cellist Matthew Barley and Centroid. #### Live Motion Capture with Matthew Barley (with Centroid) — Yeast Culture (2009) - Video: https://www.youtube.com/watch?v=BWXJtxHLZGA - Interaction: Performance, Hands & Body, Voice & Sound - Platform & tech: Projection, motion capture, real-time graphics, projection - Idea: A musician's gesture is already choreography; capturing it lets the instrument play the visuals as well as the sound, as body-tracked AR can do. - What it is: At Kings Place in London, cellist Matthew Barley performs while wearing motion-capture markers; his bowing and body movement drive projected 3D figures and abstract forms in real time, made with motion-capture experts Centroid. - Technique: An optical motion-capture system streams skeleton data into a real-time 3D engine whose output is projected behind the performer. - Try it: Use MediaPipe Pose in the browser to track a musician's arms and drive an AR ribbon that follows the bow or drumstick. Twist: the ribbon's thickness follows the loudness of the instrument. #### Ludovico Einaudi at the Royal Albert Hall — Yeast Culture (2009) - Video: https://www.youtube.com/watch?v=laOTwxhmsyA - Interaction: Performance, Voice & Sound, Projection - Platform & tech: Projection, concert visuals, projection, live film direction - Idea: Quiet music needs quiet images; AR for a performance should match tempo and restraint, not compete with the performer. - What it is: Yeast Culture created the visuals and directed the film of Ludovico Einaudi's 2009 concert at the Royal Albert Hall, where slow, painterly images float above and around the pianist and ensemble. - Technique: Pre-produced and live-mixed video is projected onto large screens in the hall, cued to the set list, and captured for a concert film. - Try it: Choose a slow piece of music and design an AR layer (WebXR or Lens Studio) of drifting shapes around a live player that moves at the tempo of the music. Twist: the shapes stop moving whenever the music rests. #### Bernstein's Mass, Royal Festival Hall — Yeast Culture (2010) - Video: https://www.youtube.com/watch?v=XXnUAMvwSps - Interaction: Performance, Projection - Platform & tech: Projection, concert visuals, projection - Idea: With a huge cast on stage, images must frame people rather than cover them; AR in crowded spaces should leave room for the humans. - What it is: Yeast Culture designed the projected visuals for a staging of Leonard Bernstein's Mass at the Royal Festival Hall in London in summer 2010, framing the large choir and orchestra with shifting imagery. - Technique: Large-format projection above and behind the performers, cued to the score (likely operated live by the visual team). - Try it: Film a class choir or group and design an AR frame (Lens Studio or WebXR) that places imagery only in the empty space around people, using person segmentation. Twist: the imagery grows as more people join the frame. ### Youssef Afella *VFX and technical artist; graphics programmer* A technical artist and graphics programmer who releases compact Unity URP rendering projects: an infinite GPU-instanced grass field, 2D global illumination with Radiance Cascades, and a glass shader that fakes refraction without ray tracing. #### Infinite Grass Field — Youssef Afella (2024) - Video: https://x.com/Youssef_Afella/status/1831049793629724836 - Source code: https://github.com/Youssef-Afella/UnityURP-InfiniteGrass - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, Unity, URP, compute shader, GPU instancing - Idea: Grow grass only where the camera can see it, so a field can be infinite at a fixed cost. - What it is: An endless meadow of grass blades sways in the wind around the camera; wherever you fly, fresh grass is already there. - Technique: A compute shader lays out blade positions on a world-space grid that follows the camera, culls them against the view frustum, and draws the survivors with GPU instancing; blades bend with scrolling wind noise (based on NiloCat's instanced grass example). - Try it: Cover a detected AR floor with GPU-instanced grass on a phone and let the grass bend away from your feet using the camera position. Twist: grass only grows where the LiDAR mesh says the floor is empty. #### Radiance Cascades 2D GI — Youssef Afella (2025) - Video: https://x.com/Youssef_Afella/status/1896237865484636513 - Source code: https://github.com/Youssef-Afella/UnityURP-RadianceCascades2DGI - Interaction: Perception & Effects, Play - Platform & tech: Desktop, Unity, URP, Radiance Cascades - Idea: Compute light the way it really travels, but cheaply, by sharing rays between nearby points. - What it is: Coloured light from glowing shapes spreads, bounces and casts soft shadows across a 2D scene in real time, with lights of any shape. - Technique: Radiance Cascades stores light in several cascades of probes: near cascades are dense with short rays, far ones sparse with long rays; the cascades are merged from far to near to give full 2D global illumination each frame. - Try it: Project a Radiance Cascades light pass onto a real tabletop in AR, treating paper shapes seen by the camera as occluders. Twist: players place virtual lamps to light a path to a goal. #### Fake Real Glass — Youssef Afella (2026) - Video: https://x.com/Youssef_Afella/status/2020632755399213068 - Source code: https://github.com/Youssef-Afella/UnityURP-FakeRealGlass - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, shader - Idea: Glass only has to look right from the camera, so fake the bending of light in screen space. - What it is: Glass objects bend and tint the scene behind them with convincing refraction and edge highlights, without any ray tracing. - Technique: The shader reads the opaque scene colour behind the object and offsets the lookup using surface normals and an estimated thickness, adding Fresnel reflection and edge darkening instead of tracing rays. - Try it: Put a fake-glass sculpture on a real table in AR and refract the live camera image through it (the AR background is the scene colour). Twist: let the glass slowly melt into a puddle when you tap it. ### Zadok Ben-David *Sculptor* Yemen-born, London-based sculptor known for fields of thousands of tiny etched-steel figures and flowers whose look changes completely as you walk around them. #### Evolution and Theory — Zadok Ben-David (2008) - Video: https://www.youtube.com/watch?v=os4bvw8n7zU - Interaction: Perception & Effects - Platform & tech: Desktop, painted aluminium, sand, miniature figures - Idea: A crowd of tiny strange figures on sand becomes a world to lean over and explore, like a tabletop AR diorama. - What it is: Hundreds of small aluminium figures inspired by Victorian science illustrations (hybrids of humans, animals and machines) stand in a field of sand, installed piece by piece in a time-lapse. - Technique: Figures are cut from aluminium sheet, hand-painted and stood in a bed of sand in a loose grid. - Try it: Place 200 tiny AR figures made from public-domain engravings on a table or sandbox and let users walk around them with a phone. Twist: figures slowly swap body parts with their neighbours over time. #### Blackfield — Zadok Ben-David (2010) - Video: https://www.youtube.com/watch?v=F7h5blRk-V8 - Interaction: Perception & Effects - Platform & tech: Desktop, etched stainless steel flowers, two-sided painting - Idea: The same field tells two opposite stories depending on where you stand, a direct model for viewpoint-dependent AR. - What it is: Thousands of tiny steel flowers are planted in white sand; from one side they are all black and burnt-looking, but when you walk around they turn into bright colours. - Technique: Each etched steel flower is painted black on one side and in colour on the other, and all are oriented the same way. - Try it: Build an AR field of flat flowers that are black when seen from the north and colourful when seen from the south, using a view-direction shader. Twist: the colours on the bright side come from photos visitors take. #### People I Saw But Never Met — Zadok Ben-David (2025) - Video: https://www.youtube.com/watch?v=SS3J5j-XAfc - Interaction: Shared & Social, Perception & Effects - Platform & tech: Desktop, etched steel silhouettes, crowd - Idea: A crowd of strangers at a small scale lets you walk among everyone at once, like an AR visualisation of a population. - What it is: Over 9,000 small steel silhouettes of people from 20 countries, drawn from photographs of strangers, stand together on the ground in Vila Nova de Cerveira, Portugal. - Technique: Silhouettes are traced from photos of passers-by, cut from steel and stood upright in a large grid on the floor. - Try it: Collect silhouettes of 100 people with body segmentation, turn them into small flat AR figures and stand them on a courtyard floor. Twist: each figure turns toward the visitor who resembles it most in height. ### Zbigniew Rybczyński *Filmmaker and video pioneer* Polish filmmaker and Academy Award winner whose 'Tango' layers 36 looping characters in one room, and who pioneered HD video compositing and slit-scan time effects in 'Steps' and 'The Fourth Dimension'. #### Tango — Zbigniew Rybczyński (1980) - Video: https://www.youtube.com/watch?v=WcySR3RmenE - Interaction: Perception & Effects, Spatial Mapping, Shared & Social - Platform & tech: Desktop, optical printer, hand-cut mattes, 35mm film - Idea: Many looping recordings can share one real room if their paths are choreographed; AR spaces can hold layers of past visitors without clutter. - What it is: In one small room, 36 characters from different stages of life enter, loop through their actions and leave, all overlapping in time yet never colliding; the film won the 1983 Oscar for animated short. - Technique: Each actor was filmed separately and composited through thousands of hand-cut mattes and multiple exposures on an optical printer. - Try it: Record 5 people separately walking short loops through the same corner of a room with phone volumetric or segmentation capture, then replay all loops together in AR at their original positions. Twist: add a 6th loop live — the viewer — and see who collides. #### Steps — Zbigniew Rybczyński (1987) - Video: https://www.youtube.com/watch?v=vBaH03PrOhE - Interaction: Portals & Worlds, Perception & Effects, Location & City - Platform & tech: Desktop, HD video, chroma key, compositing - Idea: Placing living people inside historic footage is the ancestor of AR history walks; the friction between the two times is the content. - What it is: A group of American tourists is keyed into the Odessa Steps sequence of Eisenstein's Battleship Potemkin, walking among the historic footage in an early HD video composite. - Technique: Blue-screen video keying and early high-definition compositing combine new actors with the 1925 film footage. - Try it: Find a historic photo of a place on campus, align it as an AR overlay at the exact spot, and let visitors photograph themselves standing inside it. Twist: the historic people slowly turn to look at the visitor. #### The Fourth Dimension — Zbigniew Rybczyński (1988) - Video: https://www.youtube.com/watch?v=LlVh0TCDDgU - Interaction: Perception & Effects - Platform & tech: Desktop, video, slit-scan, scanline delay - Idea: Delaying each row of the image makes time bend bodies; slit-scan is one of the simplest and strongest time-warp effects for AR cameras. - What it is: People, objects and landscapes twist and ripple because each horizontal line of the image is delayed by one more frame than the line above, turning time into a visible spatial distortion. - Technique: Slit-scan by scanlines: frames are rebuilt so each of about 480 lines comes from a progressively later frame. - Try it: Implement a time-displacement shader in Lens Studio or three.js that samples a ring buffer of camera frames with a delay proportional to screen y, then film a friend spinning. Twist: drive the delay by a depth map so near objects warp more than far ones. ### Zelia ZZ Tan *Choreographer, dancer and hybrid-performance artist; co-founder of TechDanceLab* Hong Kong choreographer and former City Contemporary Dance Company dancer who makes hybrid performances in which dancers share the stage with motion-captured avatars, projections and AR. She co-founded TechDanceLab, studied Interactive Media for Performance at CalArts and led the 'Choreographing the In-Between' hybrid dance workshop and a collaborative lab at A+E Lab in 2025. #### Accelerating Dimension — Zelia ZZ Tan (2022) - Video: https://www.youtube.com/watch?v=yMc8d2ATxNs - Interaction: Performance, Play, Hands & Body - Platform & tech: Projection, Phone, motion capture, projection, AR - Idea: A dance show structured like a game that the audience co-plays. - What it is: Dancers act as game players who invite the audience into a multisensory adventure mixing live improvisation, motion capture, projections and AR, moving between physical and virtual dimensions. - Technique: Motion-capture suits drive projected avatars while AR views on audience devices add virtual layers to the room, with game-like choices shaping the sequence (details likely vary by version). - Try it: Design a three-level 'dance game': each level is a movement task, and the audience votes with phones which projected world the dancers enter next. Twist: add one AR object that only phone viewers can see and the dancers must react to by listening to them. #### Sensing Adjacency — Zelia ZZ Tan (2022) - Video: https://www.youtube.com/watch?v=drXmMBy0a_o - Interaction: Hands & Body, Performance, Perception & Effects - Platform & tech: Projection, IMU motion capture, avatars - Idea: Choreograph the gap between a body and its imperfect digital double. - What it is: An experimental piece where a dancer in a motion-capture suit performs next to her projected avatar, sensing how information is lost and distorted between her body and its digital double. - Technique: An inertial (IMU) motion-capture suit streams rotations to a hub that animates an avatar in a virtual scene; the team tuned hip height and ground contact live and turned tracking errors into choreography. - Try it: Record a short phrase with phone pose tracking, play it back on an avatar and ask the dancer to learn the avatar's 'wrong' version. Twist: perform the original and the glitched version side by side. #### REfract — Zelia ZZ Tan (2023) - Video: https://www.youtube.com/watch?v=PsYzQYleWsk - Interaction: Hands & Body, Performance, Projection - Platform & tech: Projection, motion capture, game engine, avatars - Idea: A dancer and her avatars share one body and argue over who leads. - What it is: A dancer shares the stage with avatars in a projected digital desert full of floating letters; her motion-captured body drives the avatars, which gradually evolve and negotiate agency with her. - Technique: Motion-capture data from the dancer is streamed into a game engine that renders the avatars and desert live on a large screen behind her (likely Unity or Unreal with an inertial or AI mocap system). - Try it: Drive a free avatar in a browser with webcam pose tracking and project it behind a dancer. Twist: every 30 seconds give the avatar one movement the dancer must copy instead of the other way round. ### Zenka (Zenka Caro) *AR artist* Artist who from 2012 made AR street art, murals, postcards and linoleum prints that came alive through Aurasma on phones. #### Space Mural (Aurasma) — Zenka (Zenka Caro) (2013) - Video: https://www.youtube.com/watch?v=qAtHjySUoHM - Interaction: Perception & Effects, Location & City - Platform & tech: Phone, Aurasma, image tracking - Idea: A space mural turns into moving images through a phone camera. - What it is: A painted space-themed mural that triggers animated AR content through Aurasma when viewed with a phone. - Technique: An image recognition service matches the mural and triggers an overlay animation registered to the painted surface. - Try it: Draw a simple mural on a whiteboard and use image tracking to make its elements fly out. Twist: invite classmates to add one stroke to the mural and see whether recognition still works. #### Augmented Reality Street Art — Zenka (Zenka Caro) (2014) - Video: https://www.youtube.com/watch?v=oVaoTytQPfY - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, Aurasma, image tracking - Idea: Street paste-ups start moving inside your phone. - What it is: Wheat-pasted street art by Zenka that, seen through an AR app, animates on the wall. - Technique: The printed artwork is registered as an image target, and a video or animation layer is aligned to it so the static piece appears to move. - Try it: Set one of your own posters as an image target in 8th Wall or Lens Studio, so scanning it plays an animated version. Twist: the animation only keeps going while someone stands in front of the poster. #### Augmented Reality Linoleum Prints — Zenka (Zenka Caro) (2016) - Video: https://www.youtube.com/watch?v=POl-PJie6Qw - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Phone, image tracking - Idea: Traditional printmaking meets AR, and hand-carved prints come alive through the lens. - What it is: Hand-carved linoleum prints that come alive with animation when scanned, joining traditional printmaking with AR. - Technique: High-contrast hand-carved prints serve as image targets with strong feature points, and animations are aligned to the printed forms. - Try it: Make a high-contrast pattern with a rubber stamp or by hand, test how well it is recognized as an image marker, and add a matching animation. Twist: print several slightly different copies, each triggering a different animation. ### Zhou Zhou *Media artist and PhD researcher in human-machine improvisation* Chinese media artist and PhD researcher between the UAL Creative Computing Institute and London Contemporary Dance School. His dance works use motion capture, live neuroimaging and AI pipelines to let performers and audiences reshape the digital space around them; at the 2026 CCL at the Royal Ballet and Opera he built We Say, We See, a voice-to-image installation. #### Inner Room (installation) — Zhou Zhou (2023) - Video: https://www.youtube.com/watch?v=sa7peuzAn6w - Interaction: Hands & Body, Voice & Sound, Portals & Worlds - Platform & tech: Desktop, hand tracking, spatial sound, real-time visuals - Idea: Your hand sculpts a room and its sound at the same time. - What it is: An immersive installation where the visitor's hand shapes both sound and space: moving it through the virtual room changes sound effects, textures and the architecture of the scene. - Technique: Hand tracking (likely Leap Motion) maps position and gesture to parameters of a real-time 3D scene and its audio. - Try it: In a web page, use MediaPipe Hands so the distance between thumb and index finger scales a 3D room and pitches a drone. Twist: open palm freezes the room in place. #### Inner Room (dance) — Zhou Zhou (2026) - Video: https://www.youtube.com/watch?v=oZopGxgotVo - Interaction: Hands & Body, Projection, Performance - Platform & tech: Projection, real-time motion capture, generative visuals - Idea: The dancer's body becomes the weather of the room. - What it is: A ritual-like immersive dance where real-time motion capture of the performer transports the audience into a projected digital realm that shifts between friction and calm with her movement. - Technique: Motion-capture data from the dancer drives real-time generative visuals (likely in TouchDesigner or Unreal) projected around the audience. - Try it: Map a performer's average speed from a webcam pose model to the colour and turbulence of a full-wall projected particle field. Twist: let an audience member's movement add counter-turbulence. #### Me See I — Zhou Zhou (2026) - Video: https://www.youtube.com/watch?v=0nqbl4EtzI0 - Interaction: Perception & Effects, Performance, Projection - Platform & tech: Projection, Wearable, live neuroimaging, brain-computer interface, generative visuals - Idea: Let the performer's brain, not only his body, move the stage. - What it is: A dancer guides the audience through a landscape of memory and the subconscious, while his live brain activity is visualised and animates the digital environment around him. - Technique: A wearable neuroimaging headset (likely EEG or fNIRS) streams brain signals that are mapped to parameters of projected generative visuals. - Try it: Use a consumer EEG band (or a heart-rate sensor as a stand-in) to change the density of a projected forest while a performer alternates between calm and active tasks. Twist: hide the signal from the performer and let the audience guess what drives it. ### Adam Magyar *Photographer and video artist* Hungarian artist who adapts industrial line-scan and high-speed cameras to record city crowds: 'Urban Flow' slit-scan panoramas of pedestrians and 'Stainless' ultra-slow films of commuters on passing subway platforms. #### Urban Flow — Adam Magyar (2006) - Video: https://www.youtube.com/watch?v=erkk1aPiI1k - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, line-scan camera, custom software - Idea: A camera that records only one line turns time into the horizontal axis; AR slit-scan filters can make the passage of people readable at a glance. - What it is: Slit-scan panoramas of pedestrians crossing one line on a street: each person is recorded as they pass, so the image is a row of people from different moments, discussed here in a short documentary. - Technique: An industrial line-scan camera records a single column of pixels thousands of times per second, stacking them into an image where x is time (year of first works approximate). - Try it: Build a WebAR slit-scan: take the center column of the camera feed each frame and append it to a growing panorama displayed on a virtual wall in AR. Twist: let the user rotate the slit to horizontal and see what changes. #### Stainless — Shinjuku — Adam Magyar (2010) - Video: https://www.youtube.com/watch?v=hS5NBUG3BYs - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, high-speed camera, custom software - Idea: Changing the time scale of a crowd turns strangers into portraits; AR can slow a busy place down so people notice the individuals in it. - What it is: A camera on a moving subway train records the waiting crowd on Shinjuku station's platform at a very high frame rate; played back, the commuters seem almost frozen, like living statues sliding past. - Technique: A custom rig built around an industrial high-speed camera, mounted in the train, records hundreds of frames per second so seconds stretch into minutes. - Try it: Film a busy corridor at 240 fps from a slowly moving cart, then place the clip at 1/10 speed as a window in AR on the same wall it was filmed along. Twist: faces become sharp only when the viewer stops walking. ### Amon Tobin — ISAM Live (with V Squared Labs, Vita Motus, Leviathan, Blasthaus) *Electronic musician; ISAM Live audiovisual show* Brazilian electronic musician on Ninja Tune whose 2011 ISAM Live show placed him inside a sculpture of stacked cubes that became a projection-mapped stage. The show was designed with V Squared Labs, Vita Motus Design, Leviathan and Blasthaus and returned as ISAM Live 2.0 in 2012–2013. #### ISAM Live — Amon Tobin — ISAM Live (with V Squared Labs, Vita Motus, Leviathan, Blasthaus) (2011) - Video: https://www.youtube.com/watch?v=WLrt7-kIgIM - Interaction: Projection, Performance, Voice & Sound - Platform & tech: Projection, projection mapping, stage sculpture, live audiovisual - Idea: When the whole set is one mapped object, the performer becomes part of the image; AR stages can wrap virtual worlds around a real body the same way. - What it is: Amon Tobin performs his album ISAM from inside a large sculpture of stacked white cubes, which projection mapping turns into machines, spaceships, collapsing walls and landscapes in sync with the music. - Technique: A cube structure designed by Vita Motus is mapped by V Squared Labs and Leviathan, with visuals cued to the music and the performer sitting inside a central cube. - Try it: Stack a few white boxes around a desk, track them in AR with image or object tracking, and play music while AR machinery unfolds across the boxes. Twist: the performer's hand gestures trigger each collapse. #### ISAM Live 2.0 — Amon Tobin — ISAM Live (with V Squared Labs, Vita Motus, Leviathan, Blasthaus) (2012) - Video: https://www.youtube.com/watch?v=ba_4oOpn9Qw - Interaction: Projection, Performance, Voice & Sound - Platform & tech: Projection, projection mapping, stage sculpture, live audiovisual - Idea: Showing and hiding the performer through the mapped surface plays with what is real behind the image, a key tension in AR occlusion. - What it is: The revised ISAM show, performed at Coachella and the Sydney Opera House, uses the same cube sculpture with new mapped visuals, including a transparent layer that reveals Tobin inside. - Technique: Updated projection content on the existing cube structure, with lighting inside the central cube likely used to make the surface appear transparent at moments. - Try it: Build an AR scene where a virtual wall covers a classmate and use people occlusion to let their silhouette punch through at chosen beats. Twist: the audience's claps decide when the wall opens. ### Andrei Iurin (NullTale) *Indie developer and Unity VFX tool maker* Indie developer publishing as NullTale, who releases URP post-processing and lighting tools such as VolFx, GiLight2D, AsciiFx and PixelationFx. #### GiLight2D: Bad Apple!! in 2D Ray Tracing — Andrei Iurin (NullTale) (2023) - Video: https://www.youtube.com/watch?v=fNq0HUg6L8o - Source code: https://github.com/NullTale/GiLight2D - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, Jump flooding, 2D ray tracing - Idea: Treat every bright pixel as a light and let rays find the shadows. - What it is: The Bad Apple!! silhouette animation is relit with 2D ray-traced global illumination, so every shape glows, casts soft shadows and bounces light. - Technique: A URP render feature builds a distance field with jump flooding and casts many rays per pixel through it to gather emissive light, producing 2D global illumination. - Try it: Run 2D GI on a top-down AR floor projection so glowing virtual objects cast soft light around students' real shadows. Twist: students' silhouettes from the people-occlusion mask become the light blockers. #### VolFx — Andrei Iurin (NullTale) (2024) - Video: https://www.youtube.com/watch?v=0Byz2CEw-y8 - Source code: https://github.com/NullTale/VolFx - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, Render Feature - Idea: Post-processing does not have to apply to the whole screen; it can be a brush for chosen layers. - What it is: Scenes flip between dreamy, glitchy and painterly looks: blur, colour maps, flow and dithering are layered only onto chosen objects while the rest stays untouched. - Technique: A URP renderer feature renders selected layers into their own buffers and runs a stack of custom full-screen passes (blur, flow, colour map, dither) on them before compositing. - Try it: Use layer-selective post effects in phone AR so only the virtual creature is painted in watercolour while the camera feed stays real. Twist: switch the style when the creature is touched. ### Andrew Mendez *Researcher, engineer and early Lens Studio creator* Cornell Tech Connective Media graduate who was one of Snap's first featured Lens Studio creators in 2018, mixing computer vision research with playful Lenses such as MathArt and Smile Rater. #### MathArt Lens — Andrew Mendez (2018) - Video: https://www.youtube.com/watch?v=Hb7EyIwqFaM - Interaction: Information & UI, Perception & Effects - Platform & tech: Phone, Lens Studio, Snapchat - Idea: Turn an equation into a sculpture you can stand next to. - What it is: An early Snapchat world Lens that places a large, colourful parametric math sculpture in the room, which the user can walk around through the camera. - Technique: A parametric surface mesh (likely generated from a mathematical function) is placed with Lens Studio world tracking and ground detection. - Try it: Generate a 3D surface from a math function in Python or Blender and place it at human scale in AR. Twist: let a slider change one parameter so the sculpture morphs live. #### Realtime Image Segmentation on iOS — Andrew Mendez (2018) - Video: https://www.youtube.com/watch?v=zowNhRghLqo - Interaction: Perception & Effects, Hands & Body - Platform & tech: Phone, Core ML, Keras, MobileNet - Idea: Let the phone know which pixels are people and which are things, the basis for occlusion and background effects. - What it is: A 2018 demo of a semantic segmentation network running live on an iPhone 6s camera feed, colouring people and objects pixel by pixel. - Technique: A MobileNet-based FCN-8s model trained in Keras on Pascal VOC is converted with coremltools and run on each camera frame (about 100-200 ms). - Try it: Run a pretrained person-segmentation model in the browser and use the mask to replace everything except people with a painting. Twist: invert it so only people become the painting. ### Andrew Roth *Technical lead at York University's Augmented Reality Lab* Andrew Roth handled Max/MSP programming and technical planning for 52 Card Psycho and other projects of Caitlin Fisher's AR Lab at York University, where marker-based AR was used for cinema, poetry and tabletop stories. On his own channel he documented lab experiments such as a tracked-cube poetry machine and an AR puzzle maker. #### Augmented Reality Puzzle — Andrew Roth (2008) - Video: https://www.youtube.com/watch?v=iVv9Q8M1tTQ - Interaction: Tangible Objects, Play - Platform & tech: Desktop, ARTag, Max/MSP, Jitter - Idea: A jigsaw whose pieces are plain cards and whose picture only exists in AR. - What it is: A physical-digital puzzle made with the tools behind 52 Card Psycho: shuffled marker cards each show a fragment of an image or clip, and the player has to arrange them until the picture comes together on screen. - Technique: A Max/MSP 'puzzle maker' slices a clip into tiles assigned to ARTag markers, and the tracker renders each tile on its card so the correct layout reveals the image. - Try it: Split a photo into nine tiles, assign each to a printed image marker in an AR web app, and let groups race to rebuild the picture on a table. Twist: make the tiles video frames so the correct order also plays the clip. #### Multi-Sequential Poetry — Andrew Roth (2008) - Video: https://www.youtube.com/watch?v=wcpkgQ5azwU - Interaction: Tangible Objects, Spatial Mapping, Information & UI - Platform & tech: Desktop, DART, marker tracking, gyroscope - Idea: Walk through a room to assemble a poem that never reads the same way twice. - What it is: A York University AR Lab piece based on Raymond Queneau's and Christian Bök's combinatorial poems: a visitor holds a tracked cube in a room, and poem lines and media clips appear on it as they enter different zones. - Technique: A tracker combining sonic and gyroscopic positioning locates the user, and the Designer's Augmented Reality Toolkit (DART) triggers text and media clips on the tracked cube per zone. - Try it: Print a cube with six image markers, and in an AR web app show a different line from a list of poem fragments on each face depending on which part of the room the phone is in. Twist: two readers with two cubes must stand together to complete a stanza. ### Andy Duboc *Creative technologist* Montreal-based creative coder whose open Unity shader tests (hologram, frosted glass, Shader Graph sandbox, reaction-diffusion) became standard references, and who now makes real-time installations. #### Reaction-Diffusion: Coral — Andy Duboc (2018) - Video: https://x.com/andyduboc/status/1069928765277593601 - Source code: https://github.com/andydbc/unity-reaction-diffusion - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: Two chemicals and two rates are enough to grow coral. - What it is: Coral-like patterns grow and branch across a surface as two simulated chemicals react and diffuse in real time. - Technique: A Gray-Scott reaction-diffusion model runs in a compute shader over a pair of ping-pong textures, and the concentration drives colour and likely height for the coral look. - Try it: Grow the reaction-diffusion texture on an AR plane or the LiDAR mesh, seeding it where the user taps so coral spreads over the real room. Twist: let the feed rate follow the ambient light level. #### Webcam Effect with Shader Graph — Andy Duboc (2020) - Video: https://x.com/andyduboc/status/1279416830478684160 - Source code: https://github.com/andydbc/WebcamEffect - Interaction: Perception & Effects, Face - Platform & tech: Desktop, Unity, Shader Graph - Idea: The camera feed is just another texture you can shade. - What it is: A live webcam image is fed into Shader Graph and turned into a stylised, animated effect on screen. - Technique: A script captures a WebCamTexture and assigns it to a material, and Shader Graph nodes then distort, posterise or displace the texture; an accompanying article explains the setup. - Try it: Replace the WebCamTexture with the AR Foundation camera background texture and build the same Shader Graph effect as a filter on the phone's view of the world. Twist: apply the effect only inside the human-segmentation mask. ### Anhong Guo — Human-AI Lab (University of Michigan) *Assistant Professor of Computer Science & Engineering at the University of Michigan* Anhong Guo directs the Human-AI Lab, which builds AI systems for accessibility and physical-world interaction. His group's projects include WorldScribe, live context-aware visual descriptions for blind users, and Rubikon, an AR tutor that reconfigures a physical task. #### WorldScribe — Anhong Guo — Human-AI Lab (University of Michigan) (2024) - Video: https://www.youtube.com/watch?v=9dRExJzyqxw - Source code: https://github.com/HumanAILab/worldscribe - Interaction: Voice & Sound, Information & UI - Platform & tech: Phone, vision-language models, object detection, text-to-speech, sound recognition - Idea: A narrator for the world that decides what to say, how much, and when to stay quiet. - What it is: Live, context-aware visual descriptions for blind users: sweep a phone around and it speaks short, prioritized descriptions of what matters for your goal, adding detail when the scene is still and pausing when someone starts to talk. - Technique: A cascade of fast object detectors and slower vision-language models produces descriptions at several levels of detail, ranked by semantic similarity to the user's stated intent, while sound recognition adjusts the volume and pauses speech. - Try it: Build a web page that grabs a camera frame every few seconds, sends it with the prompt 'I am looking for my keys; describe only what helps, in under ten words' to a multimodal LLM, and reads the answer aloud with text-to-speech. Twist: change the intent (cooking, crossing a street, finding a friend) and compare how the narration changes. #### Rubikon — Anhong Guo — Human-AI Lab (University of Michigan) (2025) - Video: https://www.youtube.com/watch?v=4Jes_0nd3kY - Interaction: Tangible Objects, Play, Information & UI - Platform & tech: Desktop, intelligent tutoring system, cube tracking, AR display - Idea: Keep the real object in your hands but let AI rewrite its state, so every practice round targets your weak spot. - What it is: An AR tutor for learning the Rubik's Cube: you hold and twist a real cube while the display shows an AR cube set to exactly the tricky state you need to practice, generated from your weak spots. - Technique: The physical cube's moves are tracked and mirrored onto an AR-rendered cube whose starting configuration is generated by an intelligent tutoring model of the learner's skill components. - Try it: Pick a physical puzzle (a tangram, a knot, a guitar chord shape) and build a phone AR overlay that shows a target state; log the learner's errors and ask an LLM to choose the next target that practices the most frequent mistake. Twist: let the AI show a slightly wrong target on purpose and ask the learner to spot the error. ### Ann Veronica Janssens *Artist; light, fog and perception* Belgian artist who fills rooms with dense coloured mist so that visitors lose sight of walls and each other and move inside colour itself. #### Blue, Red and Yellow — Ann Veronica Janssens (2001) - Video: https://www.youtube.com/watch?v=B5tvvN5-CCA - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Projection, artificial fog, LEDs, colour filters - Idea: Three primary lights and fog are enough to erase architecture; AR designers can create a strong sense of place with atmosphere instead of objects. - What it is: A walk-in pavilion whose ceiling is covered with LED light filtered through red, blue and yellow is filled with fog; inside, colour seems to become a material and distances vanish. - Technique: LED panels behind primary colour filters light a closed room of artificial fog, so the fog scatters coloured light evenly in all directions. - Try it: Divide a real room into three zones with AR Foundation and render fog in red, blue and yellow for each, mixing where zones overlap. Twist: turn the fog off in steps and ask people to guess where the real walls are. #### yellowbluepink — Ann Veronica Janssens (2015) - Video: https://www.youtube.com/watch?v=qxVIZAHFs6M - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Projection, artificial mist, coloured light - Idea: Colour can be something you are inside rather than something you look at; volumetric AR effects around the viewer shift the experience from watching to being immersed. - What it is: A gallery at Wellcome Collection is filled with dense coloured mist; visitors walk into yellow, blue and pink light and lose sight of walls, floor and each other. - Technique: Haze machines saturate the room with fine mist, and coloured lamps light it in zones so that colour appears to hang in the air itself. - Try it: Create a volumetric colour fog in Unity with AR Foundation that is densest around the viewer and changes from yellow to blue to pink as they walk across the room. Twist: when two phones are close, mix their colours. ### Anne Horel *Digital artist; official Snap Lens creator and TikTok AR ambassador* Paris-based artist who moved from viral GIFs and Vines into face filters in 2019, and since 2022 folds generative AI into pop, kitsch and surreal AR effects with over 130 million views. #### Instagram AR Filters (reel) — Anne Horel (2020) - Video: https://www.youtube.com/watch?v=MZp15N8FcP8 - Interaction: Face, Perception & Effects - Platform & tech: Phone, Spark AR, Instagram - Idea: Treat a face filter as a tiny wearable artwork in one consistent personal style. - What it is: A reel of Anne Horel's face filters: pop, kitsch and surreal effects that stick pizza, glitter, emoji-like creatures and GIF-style loops onto the face and head. - Technique: Spark AR face tracking attaches 2D sprite loops and 3D props to face landmarks, with segmentation and colour effects for the background. - Try it: Design three face filters that all share one invented visual style, using only hand-drawn 2D loops attached to face points. Twist: each filter must express a different mood without using text. #### Creating Lenses with GenAI Tools — Anne Horel (2023) - Video: https://www.youtube.com/watch?v=YUGlBTneq0M - Interaction: Face, Perception & Effects - Platform & tech: Phone, Lens Studio, Generative AI, Snapchat - Idea: Use generative AI as a sketchbook that feeds straight into a live AR effect. - What it is: Anne Horel presents her workflow of generating textures, characters and backgrounds with generative AI tools and turning them into playful Snapchat Lenses. - Technique: Images made with text-to-image tools are cleaned up and mapped as face textures, 2D layers and environment art inside Lens Studio. - Try it: Generate five images with an AI tool from one prompt theme and build a face Lens that cycles through them when the user opens their mouth. Twist: the prompt must describe a feeling, not an object. ### Ars Electronica Futurelab (Spaxels) *Art-and-technology lab; drone swarm pioneers* The research lab of Ars Electronica in Linz; its Spaxels project (from 2012) flew some of the first choreographed swarms of lit quadcopters, later scaled up with Intel for the 2015 Drone 100 record. #### Spaxels — Ars Electronica Futurelab (Spaxels) (2013) - Video: https://www.youtube.com/watch?v=COdkA5-ideA - Interaction: Play, Location & City, Shared & Social - Platform & tech: Projection, quadcopters, LEDs, swarm control - Idea: Each drone is a pixel in 3D space, so the sky becomes a display you can walk under; AR designers can think of floating points as a low-resolution volumetric screen. - What it is: Dozens of LED-lit quadcopters fly in formation over Linz, playing a giant game of Pong in the night sky and morphing into a flying bird. - Technique: A ground station sends precomputed flight paths and colours to each GPS-guided quadcopter, which carries bright LEDs, so the swarm forms shapes and animations. - Try it: Build a 'spaxel' display in AR: 50 glowing spheres anchored in the sky above a courtyard that fly into formations (a ball, a paddle, a bird) and play Pong controlled by two phones. Twist: limit the swarm to 12 points and see which shapes are still readable. #### Drone 100 — Ars Electronica Futurelab (Spaxels) (2015) - Video: https://www.youtube.com/watch?v=7cegKFOW5fM - Interaction: Location & City, Perception & Effects - Platform & tech: Projection, drones, LEDs, animation software - Idea: Going from ten points to a hundred changes what shapes can be drawn; AR designers should test at what density a cloud of points starts to read as an image. - What it is: One hundred LED drones fly together in the night sky, forming shapes and animations and setting a world record for the most drones airborne at once. - Technique: Intel drones with LED payloads follow choreographed trajectories from animation software, synchronised by a central system and GPS timing. - Try it: Export a short 3D animation as point positions and replay it in AR as 100 glowing drones over a field, then remove points until the shape breaks. Twist: add a slow, 'realistic' flight between shapes to see how transitions change the story. ### Arthur Bouffard *AR creative developer and technical artist; Snap Lens Ambassador* Award-winning Lens developer known for physics-heavy, VFX-polished Lenses and city-scale Landmarker experiments; he joined Snap's Paris AR Studio as a technical artist in 2026. #### SURF-AR — Arthur Bouffard (2022) - Video: https://www.youtube.com/watch?v=ENRJpk_CtJ0 - Interaction: Hands & Body, Information & UI - Platform & tech: Phone, Lens Studio, Body Tracking, 3D Body Mesh - Idea: Let learners walk around a 3D replay of their own body to see what a coach would see. - What it is: A fitness Lens that teaches anyone to surf on dry land in four steps, guiding each pose and capturing a 3D version of the learner so they can inspect their own form from any angle. - Technique: Full-body tracking and a 3D body mesh record the user's pose at each step, then place the captured figure in world space for review. - Try it: Build a body-tracked Lens that teaches one sport or yoga pose in three steps and freezes a 3D snapshot of the student at each step. Twist: overlay the teacher's snapshot in a different colour for comparison. #### Furniture Madness — Arthur Bouffard (2023) - Video: https://www.youtube.com/watch?v=OBGeIARuRs8 - Interaction: Spatial Mapping, Hands & Body, Play - Platform & tech: Phone, Lens Studio, Physics, VFX Graph - Idea: A live Lens that looks like a finished VFX shot by combining physics, particles and hand control. - What it is: A glowing portal opens on the floor and spews sofas, chairs and tables that tumble and pile up around the user with physics, polished enough to look like a post-produced VFX shot. - Technique: Lens Studio physics bodies collide with the ground plane while VFX particles and hand tracking trigger and steer the spawning furniture (likely). - Try it: Make a Lens where a hand gesture opens a portal that spills 20 physics objects into the room. Twist: the objects must be things from the room itself, scanned or modelled by students. ### Birmingham Royal Ballet (Virtual Stage) *Ballet company with an immersive-technology programme* One of the UK's leading ballet companies, which under Carlos Acosta launched the Virtual Stage programme to explore AR, VR, motion capture and real-time graphics with partners such as Holosphere, including city-wide AR ballet trails. Its creative digital producer Tom Rogers joined AΦE on the Physical-Digital panel at Sadler's Wells. #### 'Still Life' at the Penguin Café AR — Birmingham Royal Ballet (Virtual Stage) (2017) - Video: https://www.youtube.com/watch?v=vPWOjh_b7o0 - Interaction: Performance, Spatial Mapping, Play - Platform & tech: Phone, markerless AR, 3D animation - Idea: Take the ballet's animals out of the theatre and let them dance in your kitchen. - What it is: A phone app places the animal dancers from the ballet 'Still Life' at the Penguin Café, such as the waiter penguin, into your room, where they dance on the floor next to you and can be photographed. - Technique: Animated 3D characters of the ballet's costumes are anchored on detected floor planes in a markerless phone AR app, with a camera button for sharing (likely ARCore/ARKit through a branded app). - Try it: Record one short dance with phone pose tracking, retarget it onto a free animal model and place it in AR on the classroom floor. Twist: make the animal only dance when two phones look at it at once. #### Virtual Stage x Holosphere — Birmingham Royal Ballet (Virtual Stage) (2023) - Video: https://www.youtube.com/watch?v=I0y7QCaBFm4 - Interaction: Hands & Body, Performance, Projection - Platform & tech: Projection, motion capture, real-time rendering, LED wall - Idea: A ballet dancer paints a 35-metre wall with his body in real time. - What it is: A BRB2 dancer performs in front of a 35-metre by 3-metre immersive backdrop, and his motion-captured body drives the real-time artwork around him as he dances. - Technique: Low-latency VR-grade motion tracking feeds a distributed real-time rendering system of the kind used for virtual production, displayed across a giant LED or projection backdrop by Holosphere. - Try it: Split a long wall into three projector zones and let one dancer's tracked hands 'push' colour from one zone to the next. Twist: add a second dancer whose movement erases what the first one paints. ### Brad Dwyer *Developer & founder (Hatchlings, Magic Sudoku; co-founder of Roboflow)* Des Moines developer whose studio Hatchlings shipped Magic Sudoku and Nose Zone at the ARKit and iPhone X launches before he co-founded the computer-vision company Roboflow. #### Magic Sudoku — Brad Dwyer (2017) - Video: https://x.com/braddwyer/status/910030265006923776 - Interaction: Tangible Objects, Information & UI, Play - Platform & tech: Phone, ARKit, CoreML, Vision - Idea: Point at a Sudoku on paper and the answers are written right into the empty squares. - What it is: Point the phone at a printed Sudoku and the solution appears written into the empty squares on the paper, using Vision, CoreML and ARKit. - Technique: Vision detects the Sudoku grid rectangle, a CoreML digit classifier reads the cells, a solver fills in the answers, and the digits are rendered on the paper's plane via ARKit. - Try it: Recognize an arithmetic problem on paper with OCR (Tesseract.js) and show the result at the answer position in AR. Twist: instead of giving the answer, give a one-step hint. #### ARKit 2 Eye Transform Test — Brad Dwyer (2018) - Video: https://x.com/braddwyer/status/1003778802747691008 - Interaction: Gaze & Attention, Face, Play - Platform & tech: Phone, ARKit 2, TrueDepth - Idea: Test ARKit 2 eye tracking as a new control scheme for a game you play with your face. - What it is: A quick test of ARKit 2's per-eye transforms, drawing where each eye is looking as input for his face-controlled game Nose Zone. - Technique: ARKit 2 leftEyeTransform and rightEyeTransform are read each frame and rays are cast from each eye along its forward axis to visualize gaze direction. - Try it: Use ARKit face tracking to cast two rays from the eye positions and show where you are looking. Twist: use the direction of your nose tip instead of your eyes as the control, and compare their accuracy. ### CJT (CJT-Jackton) *Shader developer* Graphics developer who publishes compact URP shader studies (crystal caustics, iridescence, anime crystal, knitwear) and an early mixed-reality theater system for HoloLens and Magic Leap. #### Crystal Caustics — CJT (CJT-Jackton) (2019) - Video: https://www.youtube.com/watch?v=-mrXdcObRNk - Source code: https://github.com/CJT-Jackton/Crystal-Caustics - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HDRP, HLSL - Idea: Bake the expensive physics once, then replay it cheaply as light. - What it is: A rotating crystal throws sharp, sparkling caustic light patterns onto the surfaces around it in real time. - Technique: Caustics are pre-rendered with photon mapping into cubemaps for a set of light directions around a bounding icosahedron, and at runtime the point-light cookie blends the nearest cubemaps using barycentric weights of the light direction. - Try it: Place a virtual gem on a real table with AR Foundation and project its caustic cookie onto the detected plane, using the AR light estimation direction as the light. Twist: let the user rotate the gem with a pinch so the caustics dance. #### Mixed Reality Theater — CJT (CJT-Jackton) (2019) - Video: https://www.youtube.com/watch?v=EEx642qwkxE - Source code: https://github.com/CJT-Jackton/Mixed-Reality-Theater - Interaction: Performance, Shared & Social - Platform & tech: Headset, Unity, Magic Leap, HoloLens - Idea: Stage effects no longer need to be physical: every seat sees them through AR glasses. - What it is: During a live theater performance, audience members wearing AR glasses see virtual stage effects appear around the actors, triggered by the director. - Technique: All headsets calibrate to a shared origin by scanning one tracked image, the director places effects that are stored as anchors on a server, and the server broadcasts cues so every device plays the effect at the same place and time. - Try it: Build a two-phone AR scene with image-target calibration and trigger a particle burst on both devices from a laptop cue. Twist: tie the cue to the stage lighting desk or a music beat. ### Chelly Jin *New media and performance artist working with movement, code and biosensors* Korean-American artist (MFA Media Arts, UCLA) who combines expressive programming, biosensors, projection and contact improvisation in performances and installations about belonging and healing. Her piece do.we.touch was made at the 2022 Choreographic Coding Lab at A+E Lab. #### Hypnagogia — Chelly Jin (2018) - Video: https://vimeo.com/692077812 - Interaction: Perception & Effects, Voice & Sound, Performance - Platform & tech: Wearable, Muse EEG headband, Processing, Arduino - Idea: Show the conscious brain as light while the voice reveals the unconscious. - What it is: An installation-performance about the state between waking and dreaming: the artist wears an EEG headband whose brainwaves play an orchestrated light sequence while she reads her recorded dreams aloud. - Technique: A Muse EEG headband streams brainwave data to Processing, which drives lights through Arduino Firmata. - Try it: Use a breathing or heart-rate sensor to dim and brighten a string of LEDs while a performer reads a dream diary aloud. Twist: the lights react to the audience's collective noise instead. #### do.we.touch — Chelly Jin (2022) - Video: https://vimeo.com/870878739 - Interaction: Hands & Body, Projection, Performance - Platform & tech: Projection, Wearable, pulse sensors, Orbbec Astra, Isadora, DMX - Idea: Let heartbeats light up the space between two people who are negotiating touch. - What it is: Created at the Choreographic Coding Lab at A+E Lab, this contact-improvisation duet explores consent and intimacy: performers wear pulse sensors whose heartbeats trigger stage light and projection while depth-camera tracking follows their closeness. - Technique: Arduino pulse sensors and Orbbec Astra movement tracking feed Isadora, which drives DMX lighting and projection. - Try it: Stream two phone-camera heart-rate readings (or clip-on sensors) to a projection that brightens as the pair's pulses converge. Twist: projection only appears when the partners are within arm's length. ### Chelsi Alise Cocking *New media artist, product designer and researcher* American new media artist and MIT Media Lab alumna whose installations use computer vision to make movement and faces visible in new ways; she has also designed products at Dropbox and Adobe. She took part in the 2022 Choreographic Coding Lab at A+E Lab with dancers Raianna Brown and Lenataa Goka. #### Illuminate — Chelsi Alise Cocking (2023) - Video: https://vimeo.com/850306904 - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, computer vision, custom visualization software, projection - Idea: What would it be like if we could see our movement? - What it is: In an open space, a person's movement leaves glowing trails that appear in front of them in real time, as if the usually invisible wake of their motion were hanging in the air. - Technique: Camera-based body tracking feeds custom interactive visualization software that renders persistent motion trails onto a large projection or screen. - Try it: With MediaPipe Pose in the browser, draw fading ribbons from both wrists and project them life-size. Twist: each ribbon keeps the colour of the person who drew it so groups can 'sign' the space together. #### Photorythms — Chelsi Alise Cocking (2023) - Video: https://vimeo.com/838381946 - Interaction: Face, Perception & Effects - Platform & tech: Desktop, face detection, computer vision, generative graphics - Idea: Augment a portrait with rhythms that follow the face, not filters that hide it. - What it is: A computational portrait series and public installation: visitors sit in front of a camera and see their face detected and augmented live with rhythmic generative forms, making portraits more expressive than a photo. - Technique: Face detection finds facial landmarks in a live camera image, and generative forms are drawn from those points in real time. - Try it: Build a p5.js face-mesh sketch that draws concentric rhythmic lines from each eye and the mouth. Twist: the rhythm speeds up when the sitter smiles. ### Chris Fraser *Artist; light and camera obscura installations* American artist who builds dark rooms and structures where narrow slits and lenses turn daylight or lamps into moving planes and projections that visitors walk through. #### In Passing — Chris Fraser (2013) - Video: https://www.youtube.com/watch?v=A9lJ3vhTztk - Interaction: Perception & Effects, Hands & Body, Projection - Platform & tech: Projection, red, green and blue lamps, apertures, corridor - Idea: A slit turns a lamp into a drawing tool in space; in AR, thin light planes that respond to walking can guide people through a building. - What it is: A corridor built around the edge of a gallery in Portland has narrow slits and openings; red, green and blue lights in the centre pass through them to form planes and bands of coloured light that change as visitors walk. - Technique: Three coloured lights at the centre of the room shine through cut apertures, projecting sheets of light and coloured shadows into the corridor, like a set of camera obscuras. - Try it: Place three virtual coloured lights in the middle of the room in AR and add a virtual wall with slits so that thin coloured light planes cross the real floor. Twist: let students move the slits with a tap to design their own light path. #### Atmosphere — Chris Fraser (2015) - Video: https://www.youtube.com/watch?v=StJYBpOYqhc - Interaction: Perception & Effects, Hands & Body, Projection - Platform & tech: Projection, steel structure, glass tubes, light - Idea: Putting the light source at the centre and people around it makes every visitor's shadow part of the image; AR effects that use the viewer's silhouette invite participation. - What it is: A tall circular steel structure fills a museum gallery; inside, a central column of glass tubes projects light onto the walls and casts the silhouettes of visitors who walk up a ramp around it. - Technique: A vertical array of glass tubes refracts a central light outward onto the curved inner wall, turning visitors' shadows into moving projections (likely a single light source). - Try it: Use body segmentation in Lens Studio or AR Foundation to cast a virtual shadow of each person onto a curved virtual wall around a glowing column in the room. Twist: stretch the shadows longer the closer someone walks to the column. ### Claire Hentschker *Artist working with photogrammetry, VR and found footage* Carnegie Mellon-trained artist who extracts 3D space from film frames and found videos with photogrammetry, as in Shining360 and Dead Mall Chunks, and co-created an ARKit re-code of Jeffrey Shaw's Golden Calf with Golan Levin. #### Dead Mall Chunks — Claire Hentschker (2016) - Video: https://www.youtube.com/watch?v=QL7JgrXcJHw - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, photogrammetry, point cloud, found footage - Idea: Use other people's videos to archive places that are disappearing. - What it is: Point clouds reconstructed from found videos of abandoned shopping malls float as ghostly fragments of storefronts, escalators and arcades. - Technique: Frames from walk-through videos on YouTube are processed with photogrammetry into point clouds that are then collected and rendered in 3D. - Try it: Find a public walk-through video of a place that no longer exists, reconstruct a fragment of it as a point cloud and place it in AR on your desk. Twist: add sound from the original video that plays only when you lean in. #### Shining360 — Claire Hentschker (2016) - Video: https://www.youtube.com/watch?v=AupAFblRwgY - Interaction: Spatial Mapping, Perception & Effects, Portals & Worlds - Platform & tech: Headset, Web, photogrammetry, 360 video, VR - Idea: Every film shot hides a 3D space that can be reconstructed and walked through. - What it is: A 30-minute 360-degree film that reconstructs the spaces of Kubrick's The Shining as fragmented 3D models, letting viewers look around the hotel while the camera follows the original shots. - Technique: The film was split into frames, photogrammetry computed depth and 3D fragments from each scene, and the fragments were stitched along the original camera path for VR. - Try it: Take a 10-second tracking shot from a film or your own phone, extract frames and run them through a photogrammetry tool, then fly a virtual camera through the result. Twist: look at what the original camera never showed. ### Climax Studios *Game studio; ARise, Towers for Tango* UK developer that made early phone AR games for Google Tango and ARKit, including the perspective puzzle game ARise. #### ARise — Climax Studios (2017) - Video: https://www.youtube.com/watch?v=ZIF2w-wULcw - Interaction: Spatial Mapping, Play, Perception & Effects - Platform & tech: Phone, ARKit, Unity - Idea: Solve puzzles without touching the screen by moving around the table until the paths line up. - What it is: A puzzle game where the player never touches the hero: you physically move and tilt your phone around a tabletop world until the paths line up from the right perspective. - Technique: Perspective-alignment puzzles check the camera's viewing angle relative to the diorama, and when disconnected paths line up in screen space the level treats them as connected. - Try it: Place a set of misaligned AR blocks on a table that form a single path only when seen from one particular angle. Twist: design a level that needs two angles to line up in sequence. #### Towers for Tango — Climax Studios (2017) - Video: https://www.youtube.com/watch?v=IStjntHonUo - Interaction: Spatial Mapping, Play - Platform & tech: Phone, Google Tango, Unity - Idea: On a depth-sensing phone, a tower-defense battlefield is laid across real floors and furniture. - What it is: A tower-defence game for Google's Tango depth-sensing phones in which the battlefield and towers are placed on real floors and furniture. - Technique: Tango's depth sensor builds a room mesh, and towers and enemy paths are placed on real floors and furniture surfaces from that mesh. - Try it: Reconstruct a mesh with a LiDAR iPhone and lay the enemies' route on the real floor so it steers around furniture. Twist: players can move real chairs to change the route. ### Creature *Small XR studio (Starship Home)* Small independent studio behind Starship Home, a narrative mixed-reality game for Quest 3. #### Starship Home — Creature (2024) - Video: https://www.youtube.com/watch?v=JC2dYTI5zoA - Interaction: Portals & Worlds, Spatial Mapping, Hands & Body - Platform & tech: Headset, Meta Quest 3, Scene API - Idea: Convert your own room into a starship cabin whose walls become windows onto space. - What it is: A Quest 3 story game that converts your room into a starship: real walls get replaced by windows onto space, and you grow alien plants and pilot the ship with your hands. - Technique: Scene API wall planes are replaced with window shaders that render space behind them, while furniture remains in passthrough, turning the room into a spacecraft interior. - Try it: Detect a wall and render it as a porthole that looks out onto the stars, keeping the rest of the view real. Twist: change the view outside the porthole when the real lights in the room are switched on or off. ### Cyan (Cyanilux) *Shader artist and tutorial writer* Hobbyist Unity shader artist who posts stylised Shader Graph effects (water, fire, forcefields, CRT) and writes in-depth breakdowns and URP tools on cyanilux.com. #### Grass Geometry Shader (URP) — Cyan (Cyanilux) (2020) - Video: https://x.com/Cyanilux/status/1255870774558232582 - Source code: https://github.com/Cyanilux/URP_GrassGeometryShader - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Desktop, Unity, URP, Geometry Shader - Idea: Grow a whole meadow from a plane by letting the GPU emit the blades. - What it is: A field of grass blades sways in the wind, generated entirely on the GPU from a flat mesh. - Technique: A hand-written URP geometry shader emits a few triangles per vertex as a blade, randomises height and bend per blade, and bends them with a scrolling wind texture (based on Roystan's tutorial). - Try it: Grow the grass on a detected AR floor plane and push the blades away from the phone's projected position so walking parts the grass. Twist: make the grass grow only where the camera sees green. #### Retro CRT Shader — Cyan (Cyanilux) (2020) - Video: https://x.com/Cyanilux/status/1301866507669057536 - Source code: https://github.com/Cyanilux/URP_RetroCRTShader - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, Shader Graph - Idea: Turn a clean render into a warm, flickering memory of old screens. - What it is: A 3D scene seen through an old CRT monitor: curved glass, scanlines, colour bleeding, static noise and rolling distortion. - Technique: A Shader Graph applied to a camera RenderTexture warps UVs with a barrel distortion, multiplies scanline and RGB-mask patterns, and adds noise-driven static and vertical roll. - Try it: Map the CRT shader onto a virtual TV placed in the room with AR, showing the live camera feed from the phone's other lens. Twist: increase the static the farther the viewer stands from the TV. ### Cybersaur Arts (Josh Garner) *Community digital-arts company; projection and visualisation artist Josh Garner* Cybersaur Arts runs projection-mapping, lighting and interactive visual projects with young people and communities in Medway and Kent, often for Electric Medway and The Historic Dockyard Chatham. Workshops teach TouchDesigner, Resolume, projection mapping and movement sensors, and the results end up on building facades and festival nights. #### Gateway To Our Past — Cybersaur Arts (Josh Garner) (2021) - Video: https://www.youtube.com/watch?v=sF1wmDix_dk - Interaction: Location & City, Information & UI - Platform & tech: Phone, QR codes, mobile web, animation - Idea: Street walls become doorways to the history of the place you are standing in. - What it is: Code plaques on local walls invite passers-by to scan them with a phone and see artwork inspired by the history of that exact spot. - Technique: Likely QR codes linking to web-based animated artworks tied to each location, forming a walkable heritage trail. - Try it: Research three historic facts about places around your school, make a short animation for each, and put up QR posters at the exact spots. Twist: make the artwork only unlock when the phone's GPS is within 20 metres. #### Bioluminescence — Cybersaur Arts (Josh Garner) (2024) - Video: https://www.youtube.com/watch?v=V9e2UwIVXME - Interaction: Projection, Drawing & Making, Location & City - Platform & tech: Projection, projection mapping, TouchDesigner, Resolume, Procreate - Idea: Everyone's hand-drawn deep-sea creature ends up glowing on a heritage building. - What it is: Glowing sea creatures drawn by workshop participants swim across the historic buildings of Chatham Dockyard at night as a large projected artwork for Electric Medway's Nocturnal festival event. - Technique: Participants draw creatures in Procreate, which are animated and composited in TouchDesigner and Resolume and then projection-mapped onto the facades with Projection Mapper (formerly Dynamapper). - Try it: Have each student draw a glowing sea creature on a tablet, animate it with a simple wiggle, and project the whole school of creatures onto a classroom wall or the outside of the building at dusk. Twist: add a webcam so the creatures scatter when someone walks past. ### Damien Rompapas *XR researcher and game developer; founder of BEER Labs* Did his PhD at NAIST with Christian Sandor and Hirokazu Kato, where he led the large-scale AR game HoloRoyale and co-created AR Yōkai, then built the open-source Project Esky software for DIY headsets and founded BEER Labs to make XR games and research prototypes. #### 3DColAR — Damien Rompapas (2022) - Video: https://www.youtube.com/watch?v=PGOm-8Vf3Po - Interaction: Drawing & Making, Hands & Body - Platform & tech: Headset, hand tracking, Unity - Idea: Pick colours by reaching into a floating colour cube and paint holograms with your finger. - What it is: In a head-worn AR display, users pick colours from 3D colour spaces such as an RGB cube floating in the air, then paint a virtual 3D figure with a virtual pen they can grasp or directly with their fingertip. - Technique: Mid-air hand tracking drives both 2D sliders and 3D colour-space widgets (RGB cube, HSV shapes), and surface painting writes colour into the model's texture where the tracked pen tip or fingertip touches it. - Try it: Build a WebXR or Quest scene with a floating RGB cube: pinch inside it to pick a colour, then touch a 3D model to paint it. Twist: sample colours from real objects in the passthrough view instead of the cube. #### B-Handy — Damien Rompapas (2022) - Video: https://www.youtube.com/watch?v=AZOsmsZHv3U - Interaction: Hands & Body, Information & UI, Spatial Mapping - Platform & tech: Headset, hand tracking, Unity - Idea: Your hand becomes a ruler that copies itself across space. - What it is: Wearing an AR headset, a user measures objects and distances in hand-lengths: the system tracks the hand and lays virtual copies of it end to end in space, so body-based measuring no longer relies on counting in your head. - Technique: Hand tracking on an optical see-through headset records the hand's pose and size, then places virtual duplicates of the hand mesh at successive positions to visualise the measurement. - Try it: In a hand-tracked AR app (Quest passthrough or Lens Studio), let users stamp copies of their hand along a table edge and show the total in hands and centimetres. Twist: measure with your foot, or your whole body lying down. ### Daniel Ilett *Shader educator and indie developer* UK developer and PhD researcher who teaches Unity shaders on YouTube and his blog, recreating effects from games such as Antichamber and Portal. #### Fully Functional Portals in URP — Daniel Ilett (2021) - Video: https://www.youtube.com/watch?v=PkGjYig8avo - Source code: https://github.com/daniel-ilett/portals-urp - Interaction: Portals & Worlds, Play - Platform & tech: Desktop, Unity, URP, Render textures - Idea: Recreate the magic of Portal with render textures in a modern pipeline. - What it is: The player walks through seamless portals between rooms, seeing the connected space through each opening in real time. - Technique: Linked portal cameras render into textures sampled in screen space on the portal surface, with oblique clipping and teleport logic, adapted for URP. - Try it: Build an AR doorway that shows a virtual copy of another room, and let students step through it to swap worlds. Twist: the other side shows the same room one hour earlier (a recorded scan). #### Impossible Geometry with Stencil Shaders — Daniel Ilett (2022) - Video: https://www.youtube.com/watch?v=EzM8LGzMjmc - Source code: https://github.com/daniel-ilett/shaders-impossible-geom - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Desktop, Unity, URP, Stencil buffer - Idea: Space can be bigger on the inside if each window only shows its own world. - What it is: A cube shows a different world on each face, and a corridor leads to rooms that could not fit inside it, like in the game Antichamber. - Technique: Each window writes a unique stencil reference, and each hidden world's materials render only where their stencil value matches, in URP. - Try it: Anchor a stencil cube to an image marker in AR so each face shows a different miniature world. Twist: rotating the real marker changes which world is on top. ### Daniel Palacios *Artist; kinetic and sound installations* Spanish artist who builds machines that make invisible processes visible: a spinning rope that turns the movement of visitors into waves and sound, and a laser that engraves growth rings as they happen. #### Waves — Daniel Palacios (2006) - Video: https://vimeo.com/12075151 - Interaction: Hands & Body, Voice & Sound, Perception & Effects - Platform & tech: Desktop, motors, rope, camera tracking - Idea: One line that goes from calm to chaos as a crowd grows teaches an AR designer to map audience activity to a single, readable visual parameter. - What it is: A long rope spun by motors at both ends hangs in a dark room; when nobody is there it is a still line, and as more people move in front of it, it whips into waves and chaotic shapes that also hum as they cut the air. - Technique: Cameras measure how much motion is in front of the piece and set the speed of two motors that rotate a rope; the rope's standing-wave shape and sound follow from physics, not from rendered graphics. - Try it: Build a WebXR scene with one glowing line strung between two anchors in the room, and drive its wave amplitude and frequency from the phone camera's frame-difference motion score. Twist: add a second person's phone and let the line only go wild when both people move at once. #### Whatever happened, Happened — Daniel Palacios (2011) - Video: https://vimeo.com/27441259 - Interaction: Perception & Effects, Information & UI - Platform & tech: Desktop, laser engraver, wood, generative growth model - Idea: Change we never see happen can be made visible by drawing its trace step by step — AR can do the same by recording and replaying slow processes where they happened. - What it is: A machine slowly engraves concentric growth rings into a slice of virgin wood with a laser, so the viewer watches years of tree growth being written as a physical record. - Technique: A laser engraver is driven by a growth model that adds one ring at a time to the wood surface, so the image accumulates physically instead of being shown as a finished graphic. - Try it: Record a plant, a shadow or a pile of objects with a phone every few minutes for an hour, then place the stacked outlines as rings around the real object in AR. Twist: let viewers scrub time by walking closer or farther away. ### Deniz Bicer *Creative coder and Unity developer* Creative coder who explores Physarum (slime-mould) simulations, flow agents and pixel sorting with Unity compute shaders and later published an interactive explainer of the Physarum algorithm. #### Physarum with Vector Fields — Deniz Bicer (2019) - Video: https://x.com/ojelibalon/status/1168248320529174535 - Source code: https://github.com/DenizBicer/Physarum - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader - Idea: Add one outside force to an emergent system and it starts drawing with intent. - What it is: Slime-mould agents spread their glowing trails across the screen, but an embedded vector field bends their paths into swirling, combed patterns. - Technique: A Unity compute shader implements Jeff Jones's Physarum model (sense, rotate, move, deposit, diffuse, decay); the vector-field variant adds a flow term to each agent's heading (the variant is likely an extension of the published repo). - Try it: Run the Physarum simulation on a texture laid over an AR floor and derive the vector field from the phone's movement, so walking combs the slime. Twist: use the room's LiDAR depth edges as walls the agents cannot cross. #### Dithered Hand as Physarum Stimulus — Deniz Bicer (2022) - Video: https://x.com/ojelibalon/status/1477660120200974338 - Source code: https://github.com/DenizBicer/Physarum - Interaction: Perception & Effects, Hands & Body - Platform & tech: Desktop, Unity, Compute Shader - Idea: An image becomes food: the slime draws a picture by eating it. - What it is: A dithered image of a hand is fed to a Physarum simulation, and the agents gather along its dots until a living, vein-like hand emerges (Genuary 2022, day 2). - Technique: The dithered hand texture is added to the trail map as a constant attractant, so agents that sense the trail concentrate on the image's dark dots (likely built on the published Physarum repo). - Try it: Feed the live people-segmentation mask from AR Foundation (or a hand mask from MediaPipe) into the Physarum trail map so slime grows over the user's real hand on screen. Twist: make the mask repel instead of attract, so the slime flees the body. ### Dennis Hlynsky *Video artist and educator* American artist and RISD professor who processes ordinary video of birds and insects so each animal leaves a trail, revealing flocking patterns. #### Bird Watching (small brains series) — Dennis Hlynsky (2009) - Video: https://www.youtube.com/watch?v=BA5iOn_toVQ - Interaction: Perception & Effects, Information & UI - Platform & tech: Desktop, frame accumulation, After Effects, consumer video - Idea: A cheap camera plus frame accumulation reveals hidden patterns of motion: the simplest version of an AR trail effect. - What it is: Ordinary footage of birds at a feeder and in flocks is processed so each bird leaves a trail, turning a backyard into swirling drawings of flight. - Technique: Clips from a Flip camera are unpacked into frames and processed in After Effects with echo and tonal effects so moving birds leave trails. - Try it: Write a WebGL camera shader that blends each new frame with the last 60 frames using a 'darken' blend, and film birds, traffic or people. Twist: make trails longer for faster movers. #### Micromigrations — Dennis Hlynsky (2014) - Video: https://www.youtube.com/watch?v=ML6n2t6uy1Q - Interaction: Perception & Effects, Information & UI - Platform & tech: Desktop, path tracing, After Effects, insects and birds - Idea: The same trace method works at many scales, from insects to flocks, so a single AR effect can reveal many kinds of motion in nature. - What it is: An Atlantic short on Hlynsky's work, in which the paths of flies, starlings and crows are traced digitally so their hidden patterns of movement become ethereal illustrations. - Technique: Video frames are composited so each animal's position over time accumulates into a line, with contrast adjusted to isolate them. - Try it: Use a phone camera trail effect to record five different moving things outdoors (insects, leaves, birds, people, cars) and compare their trace shapes. Twist: guess the source from the trace alone. ### Directive Games *Game studio; The Machines (ARKit launch title)* Directive Games built The Machines, a tabletop multiplayer strategy game in Unreal Engine that Apple showed on stage when ARKit launched in 2017. #### The Machines — Directive Games (2017) - Video: https://www.youtube.com/watch?v=xTNlSazLMGE - Interaction: Play, Shared & Social - Platform & tech: Phone, ARKit, Unreal Engine - Idea: Two phones share a miniature battlefield on one table for a real-time battle. - What it is: The released game: 1v1 real-time multiplayer battles on a real table, where each phone is a moving camera over a shared miniature battlefield. - Technique: Two phones share one tabletop anchor and a synchronized game state, with each phone acting as a free-moving camera over the miniature battlefield. - Try it: Use network sync to build a two-player tabletop AR battle where each player sees the same map from their own angle. Twist: each player can only see the area near their own units (fog of war). #### The Machines (Apple keynote reveal) — Directive Games (2017) - Video: https://www.youtube.com/watch?v=KBfEgFI90YA - Interaction: Play, Shared & Social, Spatial Mapping - Platform & tech: Phone, ARKit, Unreal Engine - Idea: Walk around the table to survey the battlefield, where moving your body is itself a tactic. - What it is: Shown at Apple's iPhone 8 event: a tabletop strategy battle in which players walk around the table to change the view and find openings in the enemy base. - Technique: ARKit world tracking places a detailed battlefield on the table, and the player's physical movement around it becomes the camera control, revealing new sightlines. - Try it: Put an AR maze castle on a table whose hidden entrance is visible only from a certain angle. Twist: the entrance appears only when two people stand on opposite sides at the same time. ### Dom Scott *Creative technologist and light artist* Maker of sound-reactive LED installations and Burning Man art-car electronics who co-developed the free, GPU-accelerated MediaPipe plugin for TouchDesigner with Torin Blankensmith. #### MediaPipe TouchDesigner: Face-tracked Texture — Dom Scott (2024) - Video: https://www.youtube.com/watch?v=_m4zNzKxkxg - Interaction: Face, Projection - Platform & tech: Desktop, Projection, TouchDesigner, MediaPipe - Idea: Build your own face filter engine inside a visual programming tool instead of an app. - What it is: A demo of the MediaPipe plugin for TouchDesigner that pins a 2D image onto a moving face in the webcam feed, following head rotation and position in real time. - Technique: MediaPipe Face Landmarker runs on the GPU inside TouchDesigner and outputs a transformation matrix that drives a textured SOP aligned to the face. - Try it: Use the free MediaPipe TouchDesigner plugin to attach a hand-drawn mask to your face and project the result on a wall. Twist: the mask changes drawing when you tilt your head. #### Talk to the Mushrooms (Mushroom Music) — Dom Scott (2024) - Video: https://www.youtube.com/watch?v=VN7wOe7Wvxw - Interaction: Voice & Sound, Tangible Objects, Location & City - Platform & tech: Desktop, LED, Microcontroller, Audio Analysis - Idea: Augment a real forest floor with light that listens, without any screen at all. - What it is: A patch of 100 glowing LED mushrooms on a woodland floor lights up and dances in response to visitors' voices or music. - Technique: Microphone input is analysed for volume and frequency and mapped to addressable LEDs inside handmade mushroom caps over Wi-Fi-synced controllers (likely). - Try it: Build five paper lanterns with LEDs that react to classmates' voices, each lantern listening to a different pitch range. Twist: the lanterns only glow when two people sing in harmony. ### Edward Ihnatowicz *Artist; pioneer of robotic and cybernetic sculpture* Polish-born British sculptor (1926–1988) whose Senster (1970), a 4 m hydraulic creature at Philips' Evoluon, turned toward quiet sounds and slow movement, and whose SAM (1968) leaned toward voices at Cybernetic Serendipity. #### SAM (Sound Activated Mobile) — Edward Ihnatowicz (1968) - Video: https://www.youtube.com/watch?v=8b52qpyV__g - Interaction: Voice & Sound, Gaze & Attention - Platform & tech: Desktop, microphones, hydraulic pistons, fibreglass petal - Idea: Turning to face a voice is the simplest signal of attention; AR objects that orient to the speaker feel social with almost no other behaviour. - What it is: A flower-like sculpture on a spine of aluminium vertebrae that turned its petal-shaped head toward whoever was talking at the Cybernetic Serendipity exhibition in London. - Technique: Microphones in the head compare sound levels and drive small hydraulic pistons in the neck until the head points at the loudest steady voice. - Try it: Create an AR flower on a table that turns its head toward whichever of two phones is producing sound (shared AR session). Twist: have it lose interest and droop if nobody talks for 20 seconds. #### The Senster — Edward Ihnatowicz (1970) - Video: https://www.youtube.com/watch?v=1jDt5unArNk - Interaction: Voice & Sound, Hands & Body, Gaze & Attention - Platform & tech: Desktop, hydraulics, microphones, Doppler radar, Philips P9201 computer - Idea: Shy behaviour, not spectacle, made people stay with it; AR characters that approach gentle attention and retreat from noise feel far more alive. - What it is: A 4 m tall hydraulic creature like a lobster claw on a long neck, which turned toward quiet sounds and slow movements of visitors and recoiled from loud noise or sudden gestures. - Technique: Four microphones located sounds and a Doppler radar detected motion; a Philips minicomputer moved six hydraulic joints to approach or retreat, with smooth damped motion. - Try it: Build a shy AR creature with a long neck that leans toward the phone when the user speaks softly and pulls back when they are loud or move the phone fast (accelerometer). Twist: give it memory, so it trusts repeat visitors more. ### Eyez *Real-time graphics artist* Real-time CG artist (formerly in Tokyo) who explores compute-shader simulations such as physarum, cellular automata and boids, and built a LiDAR-driven VFX Graph toolkit for iPad AR. #### ARVolumeVFX — Eyez Matrix — Eyez (2021) - Video: https://x.com/EyezCG/status/1460742859661869059 - Source code: https://github.com/EyezLee/ARVolumeVFX - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, Unity, VFX Graph, AR Foundation, ARKit, LiDAR - Idea: Rebuild the real room out of particles, live, from the device's depth sensor. - What it is: On an iPad, the room captured by LiDAR turns into streams of green particles that trace every wall, chair and surface around the viewer. - Technique: A LidarDataProcessor reads AR Foundation's environment mesh and depth, and a VFX property binder passes the mesh vertices to VFX Graph, where a subgraph spawns particles on the environment-mesh positions. - Try it: Build the toolkit's example scene on a LiDAR iPhone or iPad and replace the green 'matrix' look with your own particle style. Twist: make particles fall off surfaces like rain whenever the user claps. #### ARVolumeVFX — Human particles — Eyez (2021) - Video: https://user-images.githubusercontent.com/18374192/142727769-b8595802-7458-47a5-8802-a7c10fb85871.mp4 - Source code: https://github.com/EyezLee/ARVolumeVFX - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, Unity, VFX Graph, AR Foundation, ARKit, LiDAR - Idea: Use the AR human stencil to keep particles only inside a person's body. - What it is: A person in front of the iPad is filled with a cloud of particles that follows their silhouette while the background stays real. - Technique: Human depth and stencil textures from ARKit people occlusion are bound into VFX Graph; a 'Human Froxel' subgraph places particles in the person's volume and a 'Kill Nonhuman' block removes any particle outside the stencil. - Try it: Point the phone at a dancer and use the human stencil to turn them into a particle body; record a short clip. Twist: let particles that leave the silhouette keep flying so fast moves shed a trail. ### Fabio Lattanzi Antinori *Artist working with data sculpture, public art and AI* Italian-born, London-based artist who turns live data streams such as the price of search keywords into public sculptures, LED signs, karaoke machines and generative choirs. At the 2026 CCL at the Royal Ballet and Opera he trained a language model on AI platforms' Terms and Conditions and had synthetic voices sing them. #### The Cost of Your Words — Fabio Lattanzi Antinori (2020) - Video: https://vimeo.com/501403867 - Interaction: Information & UI, Location & City - Platform & tech: Projection, Google Ads data, LED display, public sculpture - Idea: Show passers-by what their words are worth to an algorithm. - What it is: A public sculpture outside the Royal Society of Sculptors advertises itself through Google Ads and displays, on its LED surface, the live price that words cost to reach people online. - Technique: The sculpture's ad account data (keywords and cost per click) is fetched from Google Ads and shown on an integrated LED display. - Try it: Make a phone AR label or a paper sign that shows the live search-trend score of a word written on a classroom object. Twist: let visitors pick the word and see its value change. #### The Keywords Karaoke — Fabio Lattanzi Antinori (2024) - Video: https://vimeo.com/952736898 - Interaction: Voice & Sound, Location & City, Information & UI - Platform & tech: Desktop, AI song generation, geo-localised keywords, karaoke machine - Idea: Sing your neighbourhood's search history. - What it is: A karaoke machine plays songs whose lyrics and music are generated by AI from the search keywords of the local area, inviting visitors to sing the data of their neighbourhood. - Technique: Geo-localised keyword data is fed to AI models that generate lyrics and music, played back with karaoke-style on-screen lyrics. - Try it: Collect ten words students see on the walk to school, have a text model write a chorus and perform it karaoke-style in class. Twist: each street gets its own verse. ### Florencia Raffa *AR artist; Snap Lens Network creator and TikTok Effect House ambassador* Argentine illustrator turned AR artist who makes Lenses and effects for Snapchat, Instagram and TikTok, and now prototypes mixed-reality utility apps for Meta Quest 3. #### XRchitects — Florencia Raffa (2024) - Video: https://www.youtube.com/watch?v=lGugvHPHUxQ - Interaction: Spatial Mapping, Information & UI - Platform & tech: Headset, Meta Quest, Mixed Reality, Scene API - Idea: Sketch renovation plans on the real walls without leaving a mark. - What it is: A mixed-reality app for planning a home's new electrical layout by placing virtual sockets, switches and cable runs directly on the real walls instead of drawing on them. - Technique: Meta Quest scene understanding detects wall planes so users can anchor and snap electrical components and wire paths to them (likely). - Try it: Build an AR tool that lets a student place three virtual shelves on a real wall and see their measurements. Twist: the tool warns when a shelf would cover a light switch. #### Boulderverse — Florencia Raffa (2025) - Video: https://www.youtube.com/watch?v=MAyJQm34PI4 - Interaction: Spatial Mapping, Play, Hands & Body - Platform & tech: Headset, Meta Quest 3, Mixed Reality - Idea: Turn any climbing wall into an endlessly re-settable game board. - What it is: A Meta Quest 3 prototype for bouldering that overlays routes, holds and challenges on a climbing wall in mixed reality. - Technique: Quest 3 passthrough and spatial anchors align virtual holds and route markers to a physical wall (likely). - Try it: Design an AR 'floor is lava' course by placing virtual stepping stones on a real floor, then time classmates crossing it. Twist: the stones slowly move while you walk. ### Google Android XR & DeepMind (Project Astra) *Google team building Android XR glasses and the Project Astra multimodal assistant* Google's Android XR group, led by Shahram Izadi, and Google DeepMind's Project Astra team build glasses and headsets where Gemini sees and hears the world with you. #### Project Astra: a universal AI assistant that sees — Google Android XR & DeepMind (Project Astra) (2024) - Video: https://www.youtube.com/watch?v=hIIlJt8JERI - Interaction: Voice & Sound, Information & UI, Location & City - Platform & tech: Phone, Wearable, Project Astra, Gemini 2.0, prototype glasses - Idea: A single assistant that lives in your camera and keeps the context of your day. - What it is: Google DeepMind's research prototype assistant uses a phone camera or prototype glasses to understand the world live, speak several languages, use tools such as Search and Maps, and remember what it has seen. - Technique: A low-latency multimodal model streams video and native audio, caches recent visual context as memory, and calls external tools before answering. - Try it: With a free multimodal live API, walk students through a neighbourhood and let the assistant narrate the street's history; log what it gets wrong. Twist: give the assistant a secret goal it must steer you toward. #### Android XR glasses live demo with Gemini (Google I/O 2025) — Google Android XR & DeepMind (Project Astra) (2025) - Video: https://www.youtube.com/watch?v=7nv1snJRCEI - Interaction: Voice & Sound, Information & UI, Location & City - Platform & tech: Wearable, Headset, Android XR, Gemini, display glasses - Idea: Glasses with an assistant that remembers what you saw a minute ago. - What it is: On stage at Google I/O 2025, prototype Android XR glasses with Gemini remember where things were seen, give turn-by-turn directions, send messages and translate a conversation live, all shown in a small in-lens display. - Technique: Gemini Live receives the glasses' camera and audio stream, keeps short-term visual memory, and returns spoken answers plus minimal overlays in a monocular display. - Try it: Build a 'where did I leave it?' prototype: a phone camera logs a frame every five seconds with a caption from a vision model; later you ask by voice and get the last matching frame. Twist: the assistant can only answer by pointing, never with words. ### Groupe F *Pyrotechnic art company* French collective of pyrotechnicians and artists founded in 1990 that stages fireworks, light and drones as open-air theatre, including the Eiffel Tower shows for Paris National Day and the Paris 2024 Olympics. #### Pleins feux sur la Tour Eiffel — Groupe F (2018) - Video: https://www.youtube.com/watch?v=vr0qFE9DCVE - Interaction: Location & City, Performance, Spatial Mapping - Platform & tech: Projection, fireworks, pyrotechnic control - Idea: Firing from the landmark instead of behind it makes the building a performer; AR effects gain force when they emit from the architecture's own structure. - What it is: On Bastille Day, fireworks launched from the Eiffel Tower itself wrap the structure in light, cascades and colour, making the tower appear to burn and sparkle from within. - Technique: Groupe F places pyrotechnic devices along the tower's girders and levels and fires them from a computer-controlled score synchronised with music. - Try it: Scan or model a tower-like structure on campus (a clock tower, a crane) and attach AR sparks to its edges so light runs up the frame to music. Twist: let the structure 'breathe' by switching sparks on floor by floor with the bass. #### Drones and fireworks at the Eiffel Tower — Groupe F (2024) - Video: https://www.youtube.com/watch?v=6VaMZJaD_Xo - Interaction: Location & City, Performance - Platform & tech: Projection, drones, fireworks, show control - Idea: Drones draw precise images while fireworks give raw energy; AR design can pair a readable, precise layer with a looser particle layer for impact. - What it is: For Paris's national day in 2024, drones draw figures and symbols in the sky beside the Eiffel Tower while fireworks burst from the tower, mixing drawn images with explosions. - Technique: A drone swarm flies pre-programmed formations timed to the music while pyrotechnics on the tower are fired from the same show-control timeline. - Try it: Build an AR sky scene with two layers above a real landmark: a crisp line drawing made of points and a chaotic firework particle system, sharing one timeline. Twist: let the audience choose which layer wins in the finale. ### Halfbrick Studios *Game studio; creator of Fruit Ninja* Australian studio whose swipe-to-slice Fruit Ninja moved from phone touchscreens to Kinect body tracking and then to hand-tracked play in the Apple Vision Pro living room. #### Fruit Ninja Kinect — Halfbrick Studios (2011) - Video: https://www.youtube.com/watch?v=xYbj9ZNqFIQ - Interaction: Hands & Body, Play - Platform & tech: Desktop, Kinect, Xbox 360 - Idea: Your full-body silhouette is the blade. - What it is: Players stand in front of the TV and see their own silhouette among flying fruit; any fast sweep of an arm or leg slices the fruit, so whole-body flailing becomes the controller. - Technique: Kinect depth sensing segments the player's silhouette and skeleton; fast-moving limb joints generate slicing trails that hit fruit. - Try it: Use a webcam with MediaPipe Pose to draw the player's silhouette and make wrists and ankles leave slicing trails that cut falling shapes. Twist: allow two players and give points only when both slice the same object at once. #### Super Fruit Ninja (Apple Vision Pro) — Halfbrick Studios (2024) - Video: https://www.youtube.com/watch?v=3WhgGopEE44 - Interaction: Hands & Body, Play, Spatial Mapping - Platform & tech: Headset, Apple Vision Pro, visionOS, hand tracking - Idea: The oldest touchscreen swipe becomes a full-arm slice through the air of your own room. - What it is: Fruit flies up around the player's real room, and they slice it with their bare hands; juice splatters on the surroundings and a sensei character guides minigames in the space. - Technique: visionOS hand tracking provides hand joint velocities that are tested against fruit colliders, with splatter decals placed in the passthrough space. - Try it: Spawn objects that arc through the room and slice them with a tracked hand, scoring only swings above a velocity threshold. Twist: let sliced halves stick to real walls and stay there for the rest of the session. ### Hiroaki Oishi (irishoak) *VJ and interactive programmer* Tokyo VJ and programmer who performs at club events such as BRDG's CHANNEL series and publishes Unity post-processing, GPU fluid and TouchDesigner experiments. #### Fluid2D Blur Image Effect — Hiroaki Oishi (irishoak) (2016) - Video: https://vimeo.com/150463085 - Source code: https://github.com/hiroakioishi/UnityFluid2DBlurImageEffect - Interaction: Perception & Effects, Drawing & Making - Platform & tech: Desktop, Unity, Image Effect, GPU Fluid - Idea: Use a fluid simulation as a brush that distorts the screen image. - What it is: A GPU fluid simulation smears and swirls the rendered image, so moving the mouse stirs the picture like ink in water. - Technique: A 2D Navier-Stokes solver (ported from haxiomic's GPU Fluid Experiments) runs on render textures, and its velocity field offsets and blurs the camera image in a post-processing pass. - Try it: Feed the AR camera image into the effect and let finger drags stir the real world like liquid paint. Twist: drive the fluid with optical flow so people walking past stir the image themselves. #### TouchDesignerPoliceAudioReactive — Hiroaki Oishi (irishoak) (2017) - Video: https://vimeo.com/245430053 - Source code: https://github.com/hiroakioishi/TDPoliceAudioReactive - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, TouchDesigner - Idea: Make an audio-reactive piece whose subject is a typo everyone makes. - What it is: An audio-reactive video piece made for the TouchDesigner Advent Calendar 2017 that playfully 'polices' how the name TouchDesigner is written, with graphics pulsing to the music. - Technique: A TouchDesigner network analyses the audio and maps its levels to typography and graphic elements (details in the linked Qiita article); the project file is shared on GitHub. - Try it: Rebuild the idea in AR: float a word that people often misspell in the room and make its letters jump to music from the phone's microphone. Twist: letters only snap into the correct spelling when the room gets loud. ### Hiroshi Sugimoto (杉本博司) *Photographer and architect* Japanese artist whose 'Theaters' expose one photograph for the full length of a film, leaving a glowing white screen, and whose 'Seascapes' repeat one horizon of sea and sky across the world. #### Theaters — Hiroshi Sugimoto (杉本博司) (1976) - Video: https://www.youtube.com/watch?v=ZspPvi-Dg_k - Interaction: Perception & Effects - Platform & tech: Desktop, large-format camera, long exposure - Idea: Adding up an entire film yields only light: an AR piece can show the sum of everything that happened in a place rather than any single moment. - What it is: Sugimoto opens his large-format camera's shutter when a film starts in a cinema or drive-in and closes it when the film ends; the screen becomes a pure white rectangle lighting the empty auditorium. Here he shows the camera he used. - Technique: A single exposure lasting the length of a feature film (about two hours) on large-format film. - Try it: Write a Lens Studio or WebAR effect that averages the camera feed over the whole session and shows the average as a live overlay; point it at a TV, a street or a busy desk for 10 minutes. Twist: blend the average with the live view based on how long the phone has stayed still. #### Seascapes — Hiroshi Sugimoto (杉本博司) (1980) - Video: https://www.youtube.com/watch?v=JWh4t67e5GM - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Desktop, large-format camera, long exposure - Idea: One fixed rule repeated across many places makes a whole series feel like one view: AR installations can use a constant framing to connect distant sites. - What it is: Horizons of seas around the world, each split exactly in half between water and sky, photographed with long exposures in the same format for decades; discussed here in a Louisiana Channel interview. - Technique: Large-format camera with long exposures and a fixed composition that places the horizon at the exact center of the frame. - Try it: Make a WebXR portal that always shows a sea horizon at eye level, aligned with the viewer's real horizon using the device's gravity vector, and place it in a windowless room. Twist: the sea's time of day follows the local time of a coastline on the other side of the world. ### Holotronica (Stuart Warren-Hill) *Holographic effects studio and inventor of the Hologauze screen* London studio founded by audiovisual artist Stuart Warren-Hill (formerly of Hexstatic), known for Hologauze, a fine transparent mesh that catches projected light so images appear to float in mid-air in front of performers. Hologauze has been used in shows by Eric Prydz, Beyoncé and illusionists, and in Holotronica's own stereoscopic live sets. #### Hologauze — Holotronica (Stuart Warren-Hill) (2013) - Video: https://www.youtube.com/watch?v=LctpKrzGvQE - Interaction: Performance, Perception & Effects - Platform & tech: Projection, Hologauze, transparent mesh, projection - Idea: A semi-transparent layer between audience and stage creates floating content without glasses; the same layering logic (content in front of, around and behind real people) is the grammar of AR. - What it is: A showreel of Hologauze, a fine transparent mesh stretched in front of the stage that catches projected light so graphics seem to float in the air around performers, as in Eric Prydz's EPIC shows, Beyoncé's Tidal X performance and illusion acts. Uploaded by Stuart Warren-Hill in 2021; the product dates from about 2013 (year approximate). - Technique: A projector throws bright images onto a nearly invisible mesh that reflects part of the light while staying see-through, often combined with a second layer behind the performer for depth. - Try it: Stage a two-layer AR scene with a performer: one virtual layer in front of them and one behind, using people occlusion in ARKit or Lens Studio. Twist: the performer 'grabs' a graphic from the front layer and throws it to the back. #### Holotronica Live (British Science Festival) — Holotronica (Stuart Warren-Hill) (2019) - Video: https://www.youtube.com/watch?v=0SyxkCGWHpw - Interaction: Performance, Voice & Sound, Perception & Effects - Platform & tech: Projection, Hologauze, stereoscopic projection, live audiovisual - Idea: Combining stereo 3D with a floating screen pushes images out into the room; AR music sets can place visuals between performer and audience rather than on a back wall. - What it is: Stuart Warren-Hill performs Holotronica's live audiovisual set at the British Science Festival: music and stereoscopic 3D visuals projected onto Hologauze make shapes seem to hang in the space between the stage and the audience. - Technique: Stereoscopic video, viewed with 3D glasses, is projected onto a Hologauze screen in front of the performer and driven live with the music. - Try it: Perform a three-minute AR music set where audio-reactive shapes float between you and the audience, built in Unity AR Foundation or Lens Studio with microphone input. Twist: the audience's phones show the shapes from their own seats. ### HsienYu Cheng (鄭先喻) *Artist and software developer; interactive and bio-electronic installations* Taiwanese artist and software developer (born 1984, Kaohsiung) whose humorous electronic installations, software and bio-electronic devices question how technology conditions human behaviour. His works include a cube that locks up and drains visitors' phones, a 7-metre drawing machine whose operator cannot see the result, and an AI follow-spot that tracks passers-by; in 2024 he led A+E Lab's Hi-Tech Garden workshops connecting children to plant biodata. #### Around 7 Meters is more fun — HsienYu Cheng (鄭先喻) (2023) - Video: https://vimeo.com/865813598 - Interaction: Shared & Social, Drawing & Making, Perception & Effects - Platform & tech: Desktop, joystick, machine learning, computer vision, custom software - Idea: Split drawing and seeing across 7 metres so two strangers have to talk to make one image. - What it is: A participant steers a joystick at one end of a 7-metre structure while the image they draw appears on a screen at the far end that only observers can see; the drawer must rely on the observers' shouts and gestures, and the result cannot be photographed. - Technique: Custom software draws the joystick trajectory in real time, while a camera with machine learning detects phones pointed at the screen and blocks recording. - Try it: Pair students: one draws on a phone web canvas facing away from a projector, the other describes what appears on the wall 7 metres away. Twist: the describer may only use sounds, not words. #### Hi-Tech Garden — HsienYu Cheng (鄭先喻), AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) (2024) - Video: https://vimeo.com/970162531 - Interaction: Tangible Objects, Voice & Sound - Platform & tech: Desktop, plant biodata, sensors, microcontroller - Idea: Let children 'hear' a plant respond when they touch it. - What it is: In workshops at A+E Lab and Medway schools, children connect sensors to plants, read their biodata and turn it into sound and interactions, imagining a future garden where humans communicate with plants. - Technique: Likely capacitive or bio-electric sensors on leaves feed a microcontroller that maps changes in the plant's signal to sound and simple visuals. - Try it: Wire a houseplant to a Makey Makey or capacitive-touch board so touching different leaves plays different notes, then compose a short 'plant duet'. Twist: map the plant's soil moisture to the tempo so a thirsty plant plays slower. ### ILMxLAB (Lucasfilm) *Lucasfilm's immersive entertainment studio* ILM's immersive lab made Star Wars mixed-reality experiments with Magic Leap, from the 2016 'Lost Droids' test to Project Porg. #### Lost Droids (MR test) — ILMxLAB (Lucasfilm), Magic Leap Studios (2016) - Video: https://www.youtube.com/watch?v=lP5ZZI05A3g - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Headset, Magic Leap prototype, ILM character animation - Idea: Star Wars droids show up in a real office and project a hologram message for you. - What it is: Shot through Magic Leap optics: R2-D2 and C-3PO appear in a real office, R2 projects a hologram message and the droids react to the viewer. - Technique: Real-time rendered characters anchored to the physical room and filmed directly through the prototype's optics. - Try it: Use phone AR to have a character in the classroom "project" a recorded video message by playing a video plane above it. Twist: the message plays only when two people watch at the same time. #### Star Wars: Project Porg — ILMxLAB (Lucasfilm), Magic Leap Studios (2019) - Video: https://www.youtube.com/watch?v=qvXAhbM6y6M - Interaction: Spatial Mapping, Play, Hands & Body - Platform & tech: Headset, Magic Leap One, world meshing, character AI - Idea: Keep a Star Wars porg as a pet at home. - What it is: Adopt porgs that waddle around your real home, recognise surfaces and play fetch with you on Magic Leap. - Technique: Character behaviour trees query the room mesh for walkable surfaces and toys so the pets navigate and play in the real space. - Try it: Make a small phone AR pet that follows the phone across a tabletop and fetches a ball (using a navmesh or a simple follow script). Twist: the pet remembers where you last put a toy. ### Irene Alvarado *Creative technologist (Google Creative Lab, later GitHub Next)* Designer-engineer who built playful ML and AR experiments at Google Creative Lab, including NormanAR, Move Mirror and Teachable Machine, before working on GitHub Copilot. #### NormanAR — Irene Alvarado, James Paterson (Presstube), Jonas Jongejan (2017) - Video: https://www.youtube.com/watch?v=6xTGrRaL66U - Interaction: Drawing & Making, Spatial Mapping - Platform & tech: Phone, ARCore, openFrameworks - Idea: Hand-drawn 3D animation, made in VR, released into everyday places through a phone. - What it is: An open-source app that takes frame-by-frame animations drawn in VR with the Norman tool and lets you tap to place them in the real world, then walk around them. - Technique: Built with openFrameworks and ARCore; exported Norman stroke animations are played back as 3D line geometry anchored to detected planes. - Try it: Draw a 12-frame 3D loop in any VR or AR drawing app and place it somewhere unexpected on campus. Twist: the animation plays only when the viewer circles it. #### Move Mirror — Irene Alvarado, Google Creative Lab (2018) - Video: https://www.youtube.com/watch?v=JvzkFJW6LIU - Interaction: Hands & Body, Perception & Effects, Play - Platform & tech: Web, PoseNet, TensorFlow.js - Idea: Your body becomes a search query that finds strangers striking the same pose. - What it is: A web experiment where you move in front of your webcam and, frame by frame, it finds a photo of someone else in the same pose, turning your dance into a flipbook of strangers. - Technique: PoseNet in TensorFlow.js estimates body keypoints in the browser and matches them against a database of pose vectors from 80,000 images. - Try it: Collect 50 pose photos from classmates, then build a webcam page that shows the closest-matching photo to your current pose. Twist: match poses between two live webcams instead of a database. ### Irini Kalaitzidi *Choreographer, dancer and creative coder* Greek choreographer and developer who builds her own motion-capture and Kinect tools to create sound instruments inside the moving body and dances between humans and stick-figure agents. She took part in the 2022 Choreographic Coding Lab at A+E Lab with filmmaker Stathis Doganis. #### Deep Dancing — Irini Kalaitzidi (2020) - Video: https://vimeo.com/783283941 - Interaction: Hands & Body, Voice & Sound, Performance - Platform & tech: Desktop, OptiTrack/Motive motion capture, Max/MSP, spatial sound - Idea: Hear how someone moves, not only see it. - What it is: A dancer's whole body becomes a musical instrument: in an immersive room, her tracked coordinates move and shape a spatial soundscape, so viewers can hear the quality of her movement as well as see it. - Technique: Motion-capture data from a Motive system is sent to Max/MSP, where it drives spatial sound synthesis in the Immersive Arts Space at ZHdK. - Try it: Map a performer's position from a webcam to left-right panning and hand height to pitch in a Web Audio page, played on four speakers around the room. Twist: sound only moves when the body moves smoothly. #### yaGrid — Irini Kalaitzidi (2021) - Video: https://www.youtube.com/watch?v=3X6OONskFJo - Interaction: Shared & Social, Hands & Body, Perception & Effects - Platform & tech: Web, Kinect, stick-figure skeletons, video-call grid - Idea: Invite stick figures to your video call as dance partners. - What it is: A dance meeting in a video-call grid, where human dancers share windows with Kinect-captured stick figures that act as neutral, non-human dance partners. - Technique: Kinect skeleton data is rendered as minimal stick figures and composited into a Zoom-like grid alongside webcam feeds of human participants. - Try it: Use MediaPipe in the browser to turn each participant's webcam into a stick figure and show both views side by side in a shared grid. Twist: swap figures between windows so everyone dances with someone else's skeleton. ### Isaac Cordal *Street artist and sculptor* Spanish artist whose tiny cement figures of office workers and politicians (Cement Eclipses) are placed in gutters, on ledges and in puddles of cities worldwide. #### Cement Eclipses (Málaga) — Isaac Cordal (2012) - Video: https://www.youtube.com/watch?v=j_CDLuqlFgw - Interaction: Location & City, Tangible Objects - Platform & tech: Desktop, cement miniatures, city streets - Idea: Small interventions that reward attention are a subtle alternative to loud AR content. - What it is: In the streets of Málaga, tiny cement figures appear in gutters, on cornices and bus shelters, acting out scenes of city life that most passers-by miss. - Technique: Figures are glued or placed on urban surfaces at eye level or below, composed into small scenes. - Try it: Create five tiny AR scenes anchored to street furniture (bins, drains, railings) with image markers or plane detection. Twist: each scene is only visible when the phone is held below knee height. #### Cement Eclipses (Utsira) — Isaac Cordal (2016) - Video: https://www.youtube.com/watch?v=G3N3_-Hzbc4 - Interaction: Location & City, Tangible Objects, Perception & Effects - Platform & tech: Desktop, cement miniatures, island landscape - Idea: Tiny figures in a huge landscape reverse the usual scale, making viewers crouch and look closely, as small AR figures do. - What it is: On the Norwegian island of Utsira, Cordal places tiny cement figures of office workers on rocks, puddles and ledges, lost in a vast natural landscape. - Technique: Figures are cast in cement from small sculpted models and placed on site, then photographed at ground level. - Try it: Place 10 tiny AR office workers on real rocks, puddles and kerbs, with occlusion so they hide behind real objects, and photograph them at ground level. Twist: they form a queue that moves toward the nearest puddle. ### Jelmer Verhoog *Mixed-reality designer-developer; early ARKit prototyper* Dutch designer-developer who, weeks after ARKit's 2017 announcement, built Model3AR and the hit app AR Airplanes, and later led mixed-reality projects at Sopra Steria in Oslo. #### AR Airplanes — Jelmer Verhoog (2017) - Video: https://www.youtube.com/watch?v=5pI7ejzbQ1k - Interaction: Play, Spatial Mapping - Platform & tech: Phone, ARKit - Idea: A pocket RC plane that needs no field, no batteries and never crashes for real. - What it is: An iPhone app that lets you fly an RC-style airplane anywhere in augmented reality, taking off from the floor and banking around the room or a park. - Technique: ARKit world tracking provides a stable origin and ground plane while on-screen joysticks drive a simple flight model in world space. - Try it: Build an AR flying toy with two on-screen sticks and place three rings to fly through in the classroom. Twist: the rings are placed by a second player on another phone. #### Model3AR (Tesla Model 3 in AR) — Jelmer Verhoog (2017) - Video: https://www.youtube.com/watch?v=5KOTJT6A_V0 - Interaction: Spatial Mapping, Information & UI - Platform & tech: Phone, ARKit - Idea: See a product that does not exist in your city yet, at true scale, in your own street. - What it is: A life-size Tesla Model 3 parked in the real world through an iPhone 7: you walk around it, swipe to drive it, switch its lights on and change its paint colour. - Technique: An ARKit world-tracking app anchors a detailed car model to the detected ground plane, with touch controls for motion, lights and material swaps. - Try it: Place a true-scale model of something large (a car, a whale, a sofa) outside and let users change one property by touch. Twist: add a tape-measure mode so people can check it fits a real space. ### Jim Campbell *Artist and engineer; low-resolution LED works* American artist and electrical engineer whose sparse grids of LEDs show blurry moving figures that only resolve from a distance or in motion, as in Exploded Views and Scattered Light. #### Scattered Light — Jim Campbell (2010) - Video: https://www.youtube.com/watch?v=9xnI8p5xGPg - Interaction: Perception & Effects, Location & City - Platform & tech: Projection, LED light bulbs, grids, low-resolution video - Idea: Placing a ghostly crowd above a real park mirrors the people in it; AR can show a place's past or typical visitors as low-resolution presences. - What it is: About 2,000 LED light bulbs hung in three grids over Madison Square Park show the blurred silhouettes of commuters walking, visible best from a distance at night. - Technique: Each bulb is one pixel; footage of pedestrians (likely filmed at Grand Central) is mapped across the layered grids so the image gains depth. - Try it: Film classmates walking past a wall, reduce the clip to a 20 x 12 grid, and replay it as floating AR bulbs in the same place at night. Twist: show the ghosts only when the real place is empty. #### Exploded Views — Jim Campbell (2011) - Video: https://www.youtube.com/watch?v=W4T5kECTYaE - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Projection, LEDs, low-resolution video, 3D grid - Idea: The eye completes very low-resolution images; AR does not need high fidelity to be convincing, and distance-based detail can be a design choice. - What it is: A large cloud of thousands of LEDs hangs in the SFMOMA atrium; from a distance, flickering points turn into shadowy figures walking, running and falling, but up close they break into dots. - Technique: Video footage is downsampled and mapped onto thousands of LEDs arranged in a 3D grid, so the image reads only from certain distances and angles. - Try it: Render a video of people walking as a sparse 3D grid of glowing points in AR and fade the points to random noise as the viewer walks closer. Twist: let students record their own silhouettes as the source video. ### Joseph DeLappe *Media artist known for protest performances inside online games and virtual worlds* Joseph DeLappe, long a professor at the University of Nevada, Reno and now at Abertay University, stages interventions in shared virtual spaces: typing the names of US soldiers killed in Iraq into the recruiting game America's Army, and walking Gandhi's 240-mile Salt March on a treadmill that drove his avatar through Second Life. His work also includes drawing machines and large-scale landscape performances. #### dead-in-iraq — Joseph DeLappe (2006) - Video: https://www.youtube.com/watch?v=ejcZ3TR5YTs - Interaction: Shared & Social, Information & UI - Platform & tech: Desktop, Projection, America's Army, machinima - Idea: Use a shared virtual space as a temporary memorial that its own players cannot ignore. - What it is: Logged in as 'dead-in-iraq', DeLappe entered the US Army's recruiting game America's Army and typed, one by one, the name, age, branch and date of death of every US service member killed in Iraq into the game's text chat until he was killed or kicked out. - Technique: A manual performance through the game's standard text-message channel, recorded as machinima and shown as a large projection. - Try it: Choose a shared digital space your classmates use (a group chat, a Minecraft server, a shared AR scene) and design a five-minute respectful intervention that places a list of names or facts where people will meet them. Twist: make the intervention disappear when someone reads it all. #### The Salt Satyagraha Online: Gandhi's March to Dandi — Joseph DeLappe (2008) - Video: https://www.youtube.com/watch?v=34SxiWwOvHw - Interaction: Hands & Body, Shared & Social, Location & City - Platform & tech: Desktop, Projection, Second Life, treadmill, custom interface - Idea: A real body walking in a room carries its avatar through a shared virtual world. - What it is: Over 26 days at Eyebeam in New York, Joseph DeLappe walked the full 240 miles of Gandhi's 1930 Salt March on a modified treadmill, and every step moved his avatar MGandhi Chakrabarti across Second Life, where others joined the march. - Technique: The treadmill's motion was converted into keyboard input for the Second Life client, so walking speed on the belt drove the avatar's forward movement in real time. - Try it: Turn walking in place into movement in a virtual map: use a phone's step counter or accelerometer to move a marker along a real historical route on a projected map, and walk a symbolic distance as a class. Twist: show the route in AR on the floor so the walker sees how far is left. ### Julian Oliver *Critical engineer and artist* New Zealand-born artist and co-author of the Critical Engineering Manifesto, working with networks, surveillance and 'improved reality'. His early AR works used hacked binoculars and marker-tracked cubes to question who owns the view. #### levelHead — Julian Oliver (2007) - Video: https://www.youtube.com/watch?v=UJYKSFANuaQ - Interaction: Tangible Objects, Play - Platform & tech: Desktop, ARToolKit, Blender Game Engine - Idea: Several rooms hide inside a cube in your hand, and turning the real cube guides a little figure through them. - What it is: A spatial memory game: tilting a real cube with printed markers moves a tiny character through rooms that appear inside the cube on screen. - Technique: ARToolKit fiducial markers on each cube face give the cube's 6DoF pose, and the tilt angle drives a physics-based character moving between rooms rendered inside the cube. - Try it: Print a paper box with a different image marker on each face, use MindAR or AR Foundation to render small rooms inside it, and tilt the box to roll a ball into the next room. Twist: make the rooms spaces from your childhood memories, so a story comes together only once you have gone through them all. #### The Artvertiser — Julian Oliver (2008) - Video: https://www.youtube.com/watch?v=z4a8n8hotI4 - Interaction: Portals & Worlds, Location & City, Perception & Effects - Platform & tech: Wearable, OpenCV, Ferns tracking - Idea: Use modified binoculars to swap street ads for artworks in real time. - What it is: Custom binoculars recognise street billboards and replace each advert live with an artwork, turning advertising space into an exhibition surface. - Technique: Natural-feature (Ferns keypoint) tracking in OpenCV recognises known billboard images and computes a homography so replacement artwork is warped onto the billboard in the video feed. - Try it: Use 8th Wall or MindAR image tracking to recognize a few posters on campus and replace them live with classmates' work. Twist: make the replacement a response to or parody of the original ad rather than a simple cover-up. ### Julius Popp *Artist; kinetic information sculptures* German artist best known for bit.fall, a waterfall whose drops are released by computer-controlled valves to write words taken live from news feeds. #### bit.fall — Julius Popp (2006) - Video: https://www.youtube.com/watch?v=gg9LWsfqqrk - Interaction: Information & UI, Perception & Effects - Platform & tech: Desktop, solenoid valves, water, microcontroller, news feed - Idea: Information that exists only while it falls teaches AR designers that text in space can be temporary, physical-feeling and tied to gravity. - What it is: A row of nozzles releases water drops in precise patterns so that, for a moment, falling water spells out words picked live from online news, lit from the side. - Technique: Around 128 magnetic valves open for milliseconds in sequence, like a one-line printer, so each word is 'printed' as a curtain of drops whose shape holds for about a second. - Try it: Make words from a live feed (headlines, chat messages) fall as particle drops from a line anchored to a real doorway or ceiling in AR, dissolving when they hit the floor. Twist: let the viewer's hand catch and hold a word. #### bit.flow — Julius Popp (2008) - Video: https://www.youtube.com/watch?v=OD19qyL9a3Q - Interaction: Information & UI, Perception & Effects - Platform & tech: Desktop, pumps, valves, coloured liquid, tubing - Idea: Pixels do not need a screen: any flowing material can carry an image through space, which suggests AR content that travels along real pipes, rails or paths. - What it is: Coloured liquid and air bubbles are pumped through a long transparent tube wound across a wall, forming pixel patterns and words that travel along the tube. - Technique: Pumps and valves insert liquid and air segments at a computed rhythm so that, as the column moves through the coiled tube, the sequence lines up as a readable image; exact control is likely timing-based. - Try it: Trace a real handrail, cable or pipe with AR and send a stream of coloured pixel beads along it that spell a short message as they pass. Twist: let viewers tap the rail to inject their own colour into the stream. ### Kaito Tsutsumi (にー兄さん / drumath2237) *Software engineer working on WebXR, VFX Graph and Gaussian splatting* Japanese engineer who publishes Azure Kinect + VFX Graph samples, a generative music-video pipeline, and Babylon.js / Gaussian splatting tools. #### KinFuSDFVFX — Kaito Tsutsumi (にー兄さん / drumath2237) (2021) - Video: https://www.youtube.com/watch?v=ApPVp6Z3cgE - Source code: https://github.com/drumath2237/KinFuSDFVFX - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Azure Kinect, Houdini - Idea: Scan a real space once and give particles its shape to hold on to. - What it is: A real room scanned with Azure Kinect becomes a distance field that VFX Graph particles crawl over and cling to. - Technique: Kinect Fusion scans are converted in Houdini into a signed distance field volume, which VFX Graph samples to conform particles to the scanned surfaces in HDRP. - Try it: Scan a classroom object with a LiDAR phone, bake it into an SDF, and let AR particles crawl over the real object in place. Twist: the particles flee from the phone's flashlight. #### Generative VFX Music Video (Bitwig x Unity) — Kaito Tsutsumi (にー兄さん / drumath2237) (2022) - Video: https://www.youtube.com/watch?v=gARJwIS5VGc - Source code: https://github.com/drumath2237/Generative-VFX-Bitwig-Sandbox - Interaction: Voice & Sound, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Bitwig Studio - Idea: Let probability, not a timeline, direct both the music and the visuals. - What it is: A generative music video where particles, lines and colours change by chance and react to the spectrum of generative music made in Bitwig Studio. - Technique: Bitwig's Note Grid generates notes by probability, while Unity VFX Graph particle behaviour switches randomly and is driven by the audio spectrum of the track. - Try it: Drive an AR particle sculpture on a phone from live microphone spectrum and random state changes, placed in the classroom. Twist: each student's clap chooses the next random scene. ### Kasimir Lehto (Reality Crisis) *Co-founder and game director, Reality Crisis (Skatrix)* Former pro snowboarder turned filmmaker and game designer; in 2018 he co-founded Reality Crisis, which makes the AR skateboarding game Skatrix with skate legend Rodney Mullen and is co-published by Niantic. #### Skatrix — Kasimir Lehto (Reality Crisis) (2020) - Video: https://www.youtube.com/watch?v=BJjWhWfSc84 - Interaction: Spatial Mapping, Play, Location & City - Platform & tech: Phone, ARKit, ARCore, Niantic Lightship, Unity - Idea: Anything in the world can become a skate spot once the phone understands its edges. - What it is: Players scan a real curb, ledge, stair or table with their phone, and a small virtual skater grinds, flips and jumps across it under finger control; replays can be filmed from any angle. - Technique: Scene meshing and plane detection (later Niantic Lightship) convert real surfaces into colliders and grind rails for a physics-driven skater avatar. - Try it: Build a small character that detects edges of real surfaces from the AR mesh and slides along them when it reaches one. Twist: record the run and replay it from a camera that orbits the real object. #### Rodney Mullen's Skatrix Pro (Apple Vision Pro) — Kasimir Lehto (Reality Crisis), Niantic (John Hanke) (2024) - Video: https://x.com/Aidan_Wolf/status/1753158831558025589 - Interaction: Spatial Mapping, Play - Platform & tech: Headset, Apple Vision Pro, visionOS, Niantic Lightship - Idea: Your living room furniture becomes a miniature skate park you watch from above like a god. - What it is: A launch title for Apple Vision Pro, co-published by Niantic: a Rodney Mullen-inspired skater rides over the player's sofa, tables and floor, which the headset scans and turns into a skate park. - Technique: visionOS scene reconstruction provides a room mesh used as physics geometry, and gaze-and-pinch or a game controller steers the skater. - Try it: Use a headset or LiDAR phone mesh to find the three highest flat surfaces in a room and link them with a virtual ramp route. Twist: award extra points for routes that cross between two different pieces of furniture. ### Katie Dale-Everett Dance (KDE Dance) *Dance company working with motion capture, VR and participatory technology* Company led by choreographer Katie Dale-Everett that combines dance, verbatim theatre and emerging technologies such as motion capture, VR and live camera feeds, often for wellbeing and young audiences. It presented Playscape: How to Build a Galaxy at the Digital Bodies + Virtual Spaces symposium with AΦE. #### Empa — Katie Dale-Everett Dance (KDE Dance) (2019) - Video: https://www.youtube.com/watch?v=TKyzy7ttthU - Interaction: Hands & Body, Shared & Social, Perception & Effects - Platform & tech: Headset, VR headset, live camera feed - Idea: See yourself through your partner's eyes while you move together. - What it is: A participatory installation where pairs swap live camera feeds through VR headsets, dancing and holding hands while each sees through the other's eyes, to test whether technology can create empathy. - Technique: Head-mounted cameras stream each participant's point of view to the other person's VR headset, creating a video see-through body swap (a setup similar to The Machine to Be Another). - Try it: Pair students: one films at eye level with a phone while the other watches the live stream in a cardboard viewer, then they try to walk and mirror each other. Twist: swap roles mid-task without stopping. #### Playscape: How to Build a Galaxy — Katie Dale-Everett Dance (KDE Dance) (2022) - Video: https://www.youtube.com/watch?v=bhecnzfh3D4 - Interaction: Hands & Body, Performance, Play - Platform & tech: Projection, motion capture, real-time rendering - Idea: Dancing bodies literally build a galaxy that children can watch grow. - What it is: A family show in which two dancers, Gabby and Ed, explore space while their motion-captured avatars build planets and stars on a large screen, inviting children to play along. - Technique: Dancers in motion-capture suits drive avatars and particle effects in a real-time engine that is projected behind them; it premiered in a mocap streaming residency (likely Rokoko suits with Unreal Engine). - Try it: Let children 'place' planets by freezing in a pose that a webcam detects; each pose spawns a different projected planet. Twist: planets only orbit while someone keeps dancing. ### Keita Higuchi *HCI researcher; Lead Research Scientist at SB Intuitions (PhD from Rekimoto Lab, University of Tokyo)* Did his PhD in Jun Rekimoto's lab at the University of Tokyo, where he built drone telepresence such as Flying Head, interned twice at Microsoft Research on the ImmerseBoard telepresence whiteboard, and later worked on egocentric video, accessibility and HCI for machine learning. #### Dhalsim: Real-Time Body Transformation — Keita Higuchi (2014) - Video: https://www.youtube.com/watch?v=g2bg_vBVW1w - Interaction: Hands & Body, Perception & Effects, Play - Platform & tech: Desktop, Kinect, point cloud - Idea: Your own live body, but with arms that stretch across the room. - What it is: A Kinect captures a person as a live 3D point cloud and places them in a virtual checkerboard world, where their arm stretches several metres, like the Street Fighter character, to reach a distant ball or punch a second player across the space. - Technique: The depth camera's point cloud of the user is split by skeleton joints and the forearm segment is likely scaled along the arm direction in real time, so the stretched arm keeps the user's real appearance. - Try it: Use webcam body tracking (MediaPipe Pose in p5.js) to draw the user's arm as a line that grows longer the faster they punch, and let it knock virtual objects off the screen. Twist: arms only stretch when two players point at each other. #### ImmerseBoard — Keita Higuchi (2015) - Video: https://www.youtube.com/watch?v=TCU1Ifr5VUc - Interaction: Shared & Social, Hands & Body, Drawing & Making - Platform & tech: Desktop, Kinect, RGB-D - Idea: Let a remote collaborator share the board as if standing beside you, so gaze and pointing survive the video call. - What it is: Two people collaborate through large touch whiteboards; a depth camera beside each board renders the remote partner life-sized and in 3D, so they seem to stand next to you or behind the glass, and their arm can stretch out to point at what they mean. - Technique: An RGB-D camera (Kinect) mounted at the side of the display captures the writer; the depth image is reprojected into a 3D scene rendered for the viewer, with modes such as a tilted board, a mirror and a stretched pointing arm. - Try it: Build a shared sketch canvas between two laptops in which each person's webcam silhouette is overlaid at the side where they stand, so partners see where the other is pointing. Twist: mirror the partner so it feels like writing on two sides of a glass window. ### Ksawery Komputery (Ksawery Kirklewski) *New media artist and studio; interactive light installations* Studio led by new media artist Ksawery Kirklewski that builds large interactive LED installations on its own software and hardware, such as ENTER and FLUX, which mirror visitors as streams of light. #### ENTER — Ksawery Komputery (Ksawery Kirklewski) (2021) - Video: https://vimeo.com/606803803 - Interaction: Hands & Body, Portals & Worlds, Voice & Sound - Platform & tech: Projection, high-framerate camera, LED strips, custom software - Idea: A doorway that shows you as signal, as if about to step through a network. - What it is: A door-sized frame of thirty-two 5-metre LED strips that reacts to sound and to the person standing in front of it, like a digital portal made of light. - Technique: A high-framerate camera and audio input drive per-strip brightness patterns in custom software, turning the viewer into a stretched, time-smeared light portrait. - Try it: Make a 'portal' on a vertical monitor that shows a webcam feed reduced to 32 glowing columns with slit-scan delay. Twist: a clap freezes the portal for five seconds. #### FLUX — Ksawery Komputery (Ksawery Kirklewski) (2022) - Video: https://www.youtube.com/watch?v=MKt0uAd9a1w - Interaction: Hands & Body, Voice & Sound, Perception & Effects - Platform & tech: Projection, high-speed cameras, LED strips, custom software - Idea: Visualise the invisible signal that carries us across networks by turning bodies into flowing light data. - What it is: A 6-by-13-metre wall of LED strips that captures passers-by with high-speed cameras and replays their silhouettes as rippling streams of light and sound. - Technique: Four high-speed cameras feed custom software that maps silhouettes onto 144,000 LED pixels with time-delayed trails and an 8-speaker sound system. - Try it: Use a webcam silhouette mask to drive a vertical strip visualisation in p5.js where each column shows a delayed slice of your body. Twist: make the delay depend on how fast you move. ### Laterna Magika (Alfréd Radok & Josef Svoboda) *Multimedia theatre (film projection combined with live performers)* Created by director Alfréd Radok and scenographer Josef Svoboda for the Czechoslovak pavilion at Expo 58 in Brussels, Laterna Magika mixed live dancers and actors with film projected on multiple moving screens. It became a permanent company of the Prague National Theatre and is often called the first multimedia theatre. #### Laterna Magika (Expo 58 programme and repertoire) — Laterna Magika (Alfréd Radok & Josef Svoboda) (1958) - Video: https://www.youtube.com/watch?v=pmXrna52etU - Interaction: Performance, Projection - Platform & tech: Projection, 35mm film projection, multiple moving screens, choreography - Idea: A live body and a filmed body can pass one action back and forth; the illusion lives in the handover, the same seam an AR designer must hide when a virtual character meets a real person. - What it is: Dancers and actors perform in front of and between several film screens; filmed doubles of the performers continue their movements on screen, walk out of the image or answer the live person. This compilation from the company's repertoire shows the principle first seen at Expo 58 in Brussels. - Technique: Pre-shot 35mm film is projected onto several movable screens and timed precisely with choreography, so performers can step 'into' or 'out of' the image (Josef Svoboda's polyscreen scenography). - Try it: Record a 20-second clip of a classmate throwing a ball toward the camera, place it as a video plane in phone AR, and rehearse a live partner who 'catches' the ball as it leaves the frame. Twist: make the virtual clip react to the live person by triggering the next clip when they step into a marked spot. #### Kouzelný cirkus (Wonderful Circus) — Laterna Magika (Alfréd Radok & Josef Svoboda) (1977) - Video: https://www.youtube.com/watch?v=LrUMYz5_Syc - Interaction: Performance, Projection, Play - Platform & tech: Projection, film projection, scenography, choreography - Idea: Comedy is a good test for mixed reality: if the gag between the real clown and the projected one lands, the audience has accepted both as the same world. - What it is: A poetic circus without words: clowns, acrobats and dancers play with projected film on screens and scenery, climbing into pictures and being chased by their own images. It was co-created by Evald Schorm, Jiří Srnec, Jan Švankmajer and Josef Svoboda and ran for decades in Prague. - Technique: Film is projected onto screens that are also physical props (doors, curtains, frames), and performers match their blocking to the filmed action frame by frame. - Try it: Build a one-minute silent gag in phone AR where a virtual clown hides behind a real door or box in the room and pops out when you open it, using plane detection and occlusion. Twist: the virtual clown copies the last gesture the real performer made. ### Layar (Raimo van der Klein, Claire Boonstra, Maarten Lens-FitzGerald) *Mobile AR browser company* Layar launched in 2009 as the first mobile AR browser, overlaying geolocated 'layers' of information on the camera view, and later moved to interactive print. #### Layar Reality Browser — Layar (Raimo van der Klein, Claire Boonstra, Maarten Lens-FitzGerald) (2009) - Video: https://www.youtube.com/watch?v=liY898N2rPU - Interaction: Location & City, Information & UI - Platform & tech: Phone, Android, GPS, compass - Idea: The world's first phone AR browser, stacking layers of location data on the camera view. - What it is: The first mobile AR browser: on an Android phone, the camera view shows geolocated layers of nearby homes for sale, restaurants, ATMs and more, published by anyone through an open API. - Technique: Points of interest from third-party layers are fetched by API and rendered as camera-overlay markers using GPS, compass and accelerometer for bearing and elevation. - Try it: Fetch nearby cafes from an open data API (such as OpenStreetMap) and show them by direction in the camera view using GPS and compass. Twist: show only places you have never been. #### Layar for Google Glass — Layar (Raimo van der Klein, Claire Boonstra, Maarten Lens-FitzGerald) (2013) - Video: https://www.youtube.com/watch?v=rBPmG5mqWfI - Interaction: Information & UI, Tangible Objects - Platform & tech: Wearable, Google Glass, image recognition - Idea: Bring print-recognition AR to Google Glass. - What it is: Layar's image-recognition AR ported to Google Glass, so printed pages and posters trigger interactive content in the wearer's view. - Technique: Image recognition matches printed pages against a database, triggering linked content in a head-worn display without needing a touch screen. - Try it: Make a course poster into an image-recognition trigger that plays a 15-second introduction video when scanned. Twist: scanning the same poster at different times shows different content. ### Leon Denise *Shader artist, designer and developer* Paris shader artist from the demoscene who live-codes at Cookie Demoparty, teaches shader workshops and publishes Unity, Godot and Shadertoy experiments. #### CurveModifier — Leon Denise (2018) - Video: https://x.com/leondenise/status/965584020989005824 - Source code: https://github.com/leon196/CurveModifier - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, Unity, HLSL - Idea: Bend any mesh along a path, as Blender's curve modifier does, but live in the engine. - What it is: A mesh slides along a 3D curve and bends to follow it, like a train or a snake following a path drawn in space. - Technique: The curve is baked into a texture of positions and tangents, and a vertex shader looks up each vertex's position along the curve to rebuild it in the curve's local frame. - Try it: Let users draw a path through the room in AR by moving the phone, then send a long mesh creature crawling along it. Twist: have the creature's body stretch when the path passes through a doorway. #### godot-particles — Leon Denise (2022) - Video: https://x.com/leondenise/status/1509126396203126784 - Source code: https://github.com/leon196/godot-particles - Interaction: Perception & Effects - Platform & tech: Desktop, Godot, GLSL - Idea: A minimal GPU-particle template for an open-source engine. - What it is: Thousands of GPU particles swirl in curl-noise currents and gather toward targets, rendered as small shaded spheres; shared as a template for the Godot engine. - Technique: A Godot particles shader (shader_type particles) updates each particle with curl-noise, collision and 'go to' force-field rules, and a billboard shader draws every particle as a sphere impostor. - Try it: Port the template to a WebXR or Godot OpenXR passthrough scene and emit particles from the controllers or hands. Twist: freeze all particles in place when the user closes a fist. ### Lily & Honglei (Lily Xiying Yang & Honglei Li) *Artist duo working with painting, animation, virtual worlds and AR* Chinese-American artist collaborative based in Flushing, New York, whose paintings, Second Life worlds and AR interventions reflect on censorship, urbanization in China and immigrant life. They took part in Manifest.AR's Venice, Bushwick and ZERO1 interventions and received a Creative Capital award. #### The Great Firewall and Sunken City — Lily & Honglei (Lily Xiying Yang & Honglei Li) (2010) - Video: https://www.youtube.com/watch?v=BD3BvwFIPuU - Interaction: Location & City, Portals & Worlds - Platform & tech: Phone, Layar, GPS - Idea: Make online censorship physical by walling off a real street. - What it is: At the Bushwick AR Intervention, a virtual Great Firewall rises across Flushing Avenue, blocking the street for anyone who looks through a phone. - Technique: A wall model is geolocated across the avenue in a mobile AR browser, extending the duo's Second Life project Land of Illusion into the street. - Try it: Choose a website your class cannot access and build an AR wall with its name across a corridor. Twist: the wall lets you pass only if you type a password. #### Crystal Coffin — Lily & Honglei (Lily Xiying Yang & Honglei Li) (2011) - Video: https://www.youtube.com/watch?v=6TdEWBdHwj0 - Interaction: Location & City, Portals & Worlds - Platform & tech: Phone, Layar, GPS - Idea: Transplant a charged political relic into a tourist square. - What it is: During the Venice Biennale, a crystal coffin topped with a red star appears in Piazza San Marco and the Giardini, evoking Mao's embalmed body in Beijing. - Technique: A 3D coffin model is geolocated at Venice landmarks in a mobile AR browser as part of Manifest.AR's uninvited intervention. - Try it: Pick a politically loaded object from another country and place it in AR at the busiest spot on campus. Twist: its label changes language depending on who opens it. ### Liz West *Artist; colour and light installations* British artist who makes large, saturated colour-light installations, such as Our Colour Reflection, a floor of hundreds of coloured mirrors that throws colour onto historic interiors. #### Our Colour Reflection — Liz West (2016) - Video: https://www.youtube.com/watch?v=k74_GTdTiYg - Interaction: Perception & Effects, Spatial Mapping, Location & City - Platform & tech: Desktop, coloured mirrors, acrylic, daylight - Idea: Looking down to see up: a floor of reflective tiles lets people explore a ceiling; AR floor layers can reveal what is above or behind the viewer. - What it is: Hundreds of coloured mirrored discs are laid across the floor of a cathedral nave; they reflect the architecture and throw patches of coloured light onto the vaults. - Technique: Coloured acrylic mirrors of different sizes are placed in a modular layout; each reflects the space above and tints the light it bounces. - Try it: Scatter virtual coloured mirror discs on the real floor with AR Foundation plane detection, each showing a tinted reflection of the ceiling from the camera feed. Twist: let students drag the discs to compose a new pattern of reflected light on the ceiling. #### Our Spectral Vision — Liz West (2016) - Video: https://www.youtube.com/watch?v=DbpB9oOzj3w - Interaction: Perception & Effects, Information & UI - Platform & tech: Projection, dichroic glass, LEDs, acrylic - Idea: Colour can explain science: pairing a spectrum of light with how different animals see it is a model for AR that shows the world through another species' eyes. - What it is: A seven-metre wall of glowing colour made for the Natural History Museum's Colour and Vision exhibition surrounds visitors with saturated light from across the spectrum. - Technique: Dichroic glass and acrylic panels in front of LEDs split and reflect light into shifting spectral colours depending on the viewing angle. - Try it: Build a phone AR filter set that remaps the camera view to how a bee, a dog and a mantis shrimp might see colour, and let students switch between them in the room. Twist: add a hidden 'ultraviolet' message only visible in the bee mode. ### Lorenzo Quinn *Sculptor* Italian sculptor known for giant hands that rise out of Venice's canals to hold up buildings or join across water. #### Support — Lorenzo Quinn (2017) - Video: https://www.youtube.com/watch?v=OJjvUjczmgE - Interaction: Location & City, Hands & Body, Perception & Effects - Platform & tech: Desktop, resin hands, canal, Venice - Idea: Giant hands that seem to hold up a real building make a warning feel physical, a strong idea for AR interaction with architecture. - What it is: Two giant white hands rise out of the Grand Canal in Venice to hold up the side of the Ca' Sagredo Hotel, a warning about rising seas and climate change. - Technique: Scanned hands of the artist's son were enlarged, built in resin over a steel core and anchored in the canal bed. - Try it: Use hand tracking to capture a pair of hands, enlarge them in AR and place them rising out of the ground to 'hold up' a real building. Twist: the hands tremble more as a live sea-level or temperature value rises. #### Building Bridges — Lorenzo Quinn (2019) - Video: https://www.youtube.com/watch?v=-s2QOSx4yZM - Interaction: Location & City, Hands & Body, Shared & Social - Platform & tech: Desktop, six pairs of hands, arsenale basin - Idea: Hands reaching across water to meet make a bridge out of a gesture, a model for shared AR interactions between two sides of a space. - What it is: For the 2019 Venice Biennale, six pairs of giant hands arch over the water of the Arsenale basin to meet, each pair standing for a universal value such as friendship, wisdom or hope. - Technique: Six pairs of hands, 15 metres tall, were cast in sections and installed on foundations on both sides of the basin. - Try it: Let two users on opposite sides of a courtyard each reach out with tracked hands; their giant AR hands meet in the middle only when both reach at the same time. Twist: each pair of hands is labelled with a value the users choose together. ### Madeline Gannon *Artist-roboticist; principal of ATONATON* Designs ways for people to communicate with machines - from projection-mapped on-skin 3D modelling (Tactum) to industrial robots that watch and follow visitors (Mimus). #### Tactum — Madeline Gannon, Golan Levin (2015) - Video: https://www.youtube.com/watch?v=tOVommpNzPA - Interaction: Projection, Hands & Body, Drawing & Making - Platform & tech: Projection, depth sensing, projection mapping, 3D printing - Idea: Pinch a 3D-printable piece of jewelry into shape right on your own arm. - What it is: Touch and pinch gestures on your own forearm are tracked and a projector shows a live 3D-printable wearable form growing on your skin as you design it. - Technique: Tracks the arm with a depth camera, interprets skin touches as modelling gestures and projection-maps the resulting geometry back onto the arm. - Try it: Use phone AR hand tracking to place a stretchable virtual bracelet on your arm, adjust its shape with gestures, then export a screenshot and make a paper model from it. Twist: have a projector above cast the design onto a real arm. #### ExoSkin — Madeline Gannon, Autodesk Research (Fraser Anderson, Tovi Grossman, George Fitzmaurice et al.) (2016) - Video: https://www.youtube.com/watch?v=slaDdfWaruk - Interaction: Hands & Body, Drawing & Making, Tangible Objects - Platform & tech: Desktop, on-body modelling, 3D printing - Idea: Design and print wearables directly on the body. - What it is: A hybrid fabrication system for designing and 3D-printing wearables directly on the body, combining on-body modelling with printing onto a body-shaped surface. - Technique: Scans the body part, lets the designer model around it in 3D, and prints the wearable in place on a body-shaped form. - Try it: Scan a classmate's wrist with a phone (Polycam or similar), draw a band around the model in 3D software, and recreate it to scale with paper strips. Twist: try it on in AR before making it. ### Marpi (Mateusz Marcinowski) *Digital artist; founder of Marpi Studio* Polish-born digital artist who builds generative, multiplayer ecosystems of creatures, plants and landscapes that audiences grow and play with from screens, projections and their own phones. #### Binary Garden — Marpi (Mateusz Marcinowski) (2017) - Video: https://vimeo.com/203962961 - Interaction: Shared & Social, Voice & Sound, Perception & Effects - Platform & tech: Desktop, Phone, WebGL, WebSockets - Idea: Your phone becomes a seed pouch for a public, ever-growing garden. - What it is: Nine large touch screens grow sound-reactive 3D noise gardens, and visitors can also join from their own phones without installing an app to plant and remix patterns. - Technique: A browser-based web app on visitors' phones sends input over WebSockets to real-time noise-driven WebGL scenes that also react to live audio (likely). - Try it: Make a shared screen garden where each phone that opens a URL can plant one procedurally generated flower whose shape comes from the user's name. Twist: flowers wilt when the room goes quiet and bloom with applause. #### Extinct Systems — Marpi (Mateusz Marcinowski) (2017) - Video: https://vimeo.com/211763949 - Interaction: Shared & Social, Projection, Perception & Effects - Platform & tech: Projection, Phone, WebGL, Projection - Idea: One shared living world that a whole crowd grows together instead of one person at a time. - What it is: A large-scale immersive projection fills a hall with generative creatures and landscapes, and more than a hundred visitors shape the same canvas at once. - Technique: Real-time WebGL generative graphics are projected at room scale, with many simultaneous inputs merged into one simulation (likely phones connecting through a browser). - Try it: Build a projected pond where every student joins from their phone browser and spawns one creature that lives in the shared scene. Twist: creatures only reproduce when two phones tap at the same moment. ### Matt Moss *iOS developer & founder (Hawkeye Access, Locket Camera)* Young iOS developer whose 2018 ARKit 2 eye-tracking prototype went viral and became Hawkeye Access; later founded the widget photo app Locket. #### Control Your iPhone with Your Eyes — Matt Moss (2018) - Video: https://x.com/thefuturematt/status/1004821303486906369 - Interaction: Gaze & Attention, Face - Platform & tech: Phone, ARKit 2, TrueDepth - Idea: Select whatever you look at and blink to click: control an iPhone with your eyes. - What it is: Look at a button to select it and blink to press it: an ARKit 2 prototype that turns eye gaze into a pointer on iPhone X. - Technique: ARKit 2 face tracking provides a lookAtPoint and per-eye transforms, which are intersected with the screen plane to get a gaze cursor, and eyeBlink blend shapes act as clicks. - Try it: Show a gaze cursor on screen using ARKit face tracking's lookAtPoint (or WebGazer.js) and trigger buttons by blinking. Twist: make a simple game you can play only with your eyes. #### Hawkeye Access — Matt Moss (2018) - Video: https://www.youtube.com/watch?v=fLktElD6QfU - Interaction: Gaze & Attention, Face, Information & UI - Platform & tech: Phone, ARKit 2, TrueDepth - Idea: Turn an eye-control prototype into a hands-free browser on the App Store for accessibility. - What it is: The shipped version of his prototype: browse any website hands-free on iOS by looking at links and blinking to open them. - Technique: Calibrated gaze estimation from ARKit face tracking is smoothed and mapped to web page link targets, with dwell or blink detection triggering navigation. - Try it: Build a web page with WebGazer.js that opens a link automatically after you stare at it for 1.5 seconds. Twist: design a gaze interface for one specific user (such as someone with an injured hand). ### Merge Labs (Merge Cube) *AR/VR toy and education company* Merge makes the Merge Cube, a patterned foam cube that phones track so a hologram can be held and turned in the hand, used widely in schools. #### Merge Cube — Merge Labs (Merge Cube) (2017) - Video: https://www.youtube.com/watch?v=_sL1HfdVxg4 - Interaction: Tangible Objects, Information & UI - Platform & tech: Phone, image tracking, iOS, Android - Idea: A foam cube that lets kids really hold a hologram in their hands. - What it is: A patterned foam cube that a phone camera tracks, so a hologram such as a planet, skull or artefact appears in the user's hand and can be turned and examined. - Technique: The cube's six unique patterns form a single tracked object target, giving full 6DoF pose so a hologram can be held and turned in the hand. - Try it: Print and fold a patterned paper cube, track it with Vuforia or 8th Wall, and show a rotatable planet in your hand. Twist: bring two cubes close together to trigger a chemical reaction. ### Michael Flückiger *Interaction designer and creative coder; lecturer at Bern University of the Arts* Swiss interaction designer who builds playful hacks and installations with openFrameworks, from a bicycle that casts an animal shadow to a game where one player remote-controls another. #### Elephant Walk — Michael Flückiger (2015) - Video: https://vimeo.com/142179343 - Interaction: Projection, Location & City, Perception & Effects - Platform & tech: Projection, Wearable, openFrameworks, iPad - Idea: Your bike casts the shadow of an elephant. - What it is: At night a bicycle rides through the city and casts the shadow of a walking elephant on the walls beside it; the faster the bike goes, the faster the elephant walks. - Technique: A projector and car battery on the bike show a looped elephant animation driven by speedometer data, played by an openFrameworks app on an iPad mini. - Try it: Mount a phone projector or phone screen on a moving object (bike, trolley, skateboard) and play a walk-cycle animation whose speed follows the phone's accelerometer. Twist: when the object stops, the creature sits down. #### Real Life Arcade Game — Michael Flückiger (2017) - Video: https://vimeo.com/199587998 - Interaction: Play, Hands & Body, Shared & Social - Platform & tech: Desktop, openFrameworks, Pozyx, Arduino - Idea: A video game whose character is a real person following your button presses. - What it is: One player holds a game controller in front of a screen; every button press is sent as a spoken order to a second player in another room, whose tracked movements move the on-screen figure. - Technique: An openFrameworks game converts controller input into audio instructions and moves the avatar from Pozyx ultra-wideband position tracking of the second player. - Try it: Pair students: one sends commands from a phone, the other hears them in headphones and walks, while a webcam turns the walker's position into a game character. Twist: add a delay so the controller must plan ahead. ### Michael Murphy (Perceptual Art) *Artist; perceptual sculptures* American artist whose studio Perceptual Art hangs hundreds of objects on strings in a volume so that they snap into a single image, such as a portrait or a gun, only from one spot. #### Perceptual Shift — Michael Murphy (Perceptual Art) (2011) - Video: https://www.youtube.com/watch?v=fh9Hx-Uujno - Interaction: Perception & Effects, Gaze & Attention, Spatial Mapping - Platform & tech: Desktop, suspended spheres, fishing line, perspective - Idea: Depth hides the image and one viewpoint reveals it: a cloud of AR particles can carry a message that people have to physically find, which is far more memorable than showing it. - What it is: Hundreds of black spheres hang on fishing line in a seemingly random cloud; as the camera moves to one position, they lock into a portrait of Barack Obama. - Technique: Each sphere is placed along the line of sight from the chosen eye point to a pixel of the portrait at a varied depth, so the 2D image survives only from that point. - Try it: Write a WebXR or Unity script that takes a 32x32 image, casts rays from the current camera pose and places spheres at random depths along each ray. Twist: let the spheres slowly drift back to random depths unless someone stands on the spot. #### Gun Country — Michael Murphy (Perceptual Art) (2014) - Video: https://www.youtube.com/watch?v=802kLR9_cSk - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Desktop, suspended objects, fishing line - Idea: One set of points can hold two readings from two viewpoints: AR artworks can hide a second, opposing message that appears only when the visitor walks to the other side. - What it is: Suspended pieces float in a loose cloud; from one angle they form the silhouette of a gun, and walking around it the image falls apart into the shape of the United States and back. - Technique: Objects are likely positioned by solving for two target silhouettes from two eye points, so each hanging element sits where both lines of sight are satisfied. - Try it: Place AR cubes so that from the front they spell one word and from the side another; compute positions from two orthogonal pixel grids. Twist: pick two words that argue with each other. ### Mojang Studios (Minecraft Earth) *Game studio; Minecraft Earth* Mojang and Microsoft built Minecraft Earth (2019–2021), a location-based AR game where players built life-size Minecraft structures in the real world. #### Minecraft Earth — Mojang Studios (Minecraft Earth) (2019) - Video: https://www.youtube.com/watch?v=GNo38kNy_EU - Interaction: Location & City, Drawing & Making, Shared & Social - Platform & tech: Phone, ARKit, ARCore, Azure Spatial Anchors - Idea: Place Minecraft block builds life-size on the street and walk in together. - What it is: Players collect blocks around their neighbourhood, build on tabletop 'build plates', then place builds at life size in the real world and play in them together. - Technique: Build plates are authored at tabletop scale and rescaled to life size with the same anchor, while Azure Spatial Anchors let multiple players share the placement. - Try it: Build a small model from AR blocks on a table, then scale it to life size with one tap and walk inside. Twist: after scaling, another classmate can see it in the same spot. ### Moonbot Studios (William Joyce, Brandon Oldenburg) *Storytelling studio; makers of Wonderbook: Diggs Nightcrawler and the IMAG-N-O-TRON AR book apps* Oscar-winning studio behind The Fantastic Flying Books of Mr. Morris Lessmore; it built interactive picture-book apps and a film-noir Wonderbook game that treat printed pages as windows into animated worlds. #### Wonderbook: Diggs Nightcrawler — Moonbot Studios (William Joyce, Brandon Oldenburg) (2013) - Video: https://www.youtube.com/watch?v=dVnkwYGQ6OI - Interaction: Tangible Objects, Play, Portals & Worlds - Platform & tech: Desktop, Wonderbook, PlayStation Eye, PlayStation Move - Idea: Turning a real page works like a camera cut in a detective movie. - What it is: A film-noir detective story for Wonderbook: turning the pages of the physical book opens pop-up scenes of Library City, where players help bookworm detective Diggs investigate Humpty Dumpty's fall. - Technique: Page patterns printed in the Wonderbook are tracked by the PlayStation Eye so pop-up dioramas and characters stay glued to the open spread as it moves. - Try it: Write a three-page mystery and make each page an AR image target that shows one pop-up diorama with a hidden clue; readers must tilt the phone around the book to spot it. Twist: make the final page show a different culprit depending on which clues were tapped. #### The Numberlys IMAG-N-O-TRON — Moonbot Studios (William Joyce, Brandon Oldenburg) (2014) - Video: https://www.youtube.com/watch?v=hqplgV3_EsQ - Interaction: Tangible Objects, Information & UI, Voice & Sound - Platform & tech: Phone, iPad, image tracking - Idea: The printed book stays the main object; the tablet is a lens that animates what is already on the page. - What it is: A companion app for the printed picture book The Numberlys: pointing an iPad at the pages makes the art-deco number characters and the letters they invent spring off the paper as animated 3D scenes with sound. - Technique: Likely natural-feature image tracking of each illustrated spread, with pre-authored 3D animation anchored to the page. - Try it: Pick a favorite children's book page, scan it as an image target, and animate one character stepping out of the illustration with a sound effect. Twist: the character should only appear when a second reader covers a specific word with a finger. ### Myron Krueger *Computer artist and 'artificial reality' pioneer* Computer artist who from 1969 built responsive environments such as GLOWFLOW, METAPLAY and Psychic Space, culminating in VIDEOPLACE, where live video silhouettes of visitors interact with graphic creatures and each other. #### VIDEOPLACE — Myron Krueger (1975) - Video: https://www.youtube.com/watch?v=d4DUIeXSEpk - Interaction: Hands & Body, Shared & Social, Projection - Platform & tech: Projection, Desktop, silhouette tracking, custom video hardware, artificial reality - Idea: Your camera silhouette is your body in the computer world, no gear required. - What it is: A visitor's live silhouette appears on a large screen and can draw with a finger, juggle graphic objects, or touch the silhouette of another person in a different room. - Technique: A camera against a backlit wall extracts the body outline in real time with custom hardware, and programs test the outline for contact with graphic objects (the video shows the 1985 version). - Try it: Use a webcam and body segmentation in p5.js to make five mini VIDEOPLACE scenes: draw with a fingertip, bounce a ball off your arm, hold a creature in your palm. Twist: connect two laptops so two remote silhouettes can 'shake hands'. #### CRITTER (VIDEOPLACE) — Myron Krueger (1984) - Video: https://www.youtube.com/watch?v=VdrujesfIBQ - Interaction: Hands & Body, Play - Platform & tech: Projection, Desktop, silhouette tracking, behavior rules - Idea: A virtual pet that understands the shape of your body. - What it is: A small graphic creature chases a visitor's silhouette, climbs up their arm, dangles from a fingertip and does a little dance when it reaches the top of the head. - Technique: The creature's behavior rules react to the tracked silhouette's outline, finding hands and head from extremities of the contour. - Try it: Build a creature in a webcam sketch that walks along the edge of your body mask, jumps to your highest point and falls if you move too fast. Twist: give it moods so it avoids people who wave aggressively. ### Nathalie Djurberg & Hans Berg *Artist duo combining clay animation, sculpture and music* Swedish duo: Djurberg makes unsettling claymation worlds and Berg composes their hypnotic scores. They made the VR piece It Will End in Stars with Acute Art (2018) and the AR fairy-tale walk This Is It for Apple's [AR]T program (2019). #### It Will End in Stars — Nathalie Djurberg & Hans Berg, Acute Art (2018) - Video: https://www.youtube.com/watch?v=25laN3xx2GM - Interaction: Gaze & Attention, Perception & Effects - Platform & tech: Headset, VR, photogrammetry, Unity - Idea: The artwork looks back at you and reacts to your presence. - What it is: A VR fairy-tale landscape of charcoal drawings, text and scanned clay sculptures; a creature at its centre meets the viewer's gaze and responds to their movements, making them feel like a watched intruder. - Technique: Room-scale VR with photogrammetry-scanned clay figures and a creature whose head and eyes follow the headset position. - Try it: Place a character in AR or WebXR whose eyes always follow the camera, and add a rule that it steps back when stared at for three seconds. Twist: make it approach only while you look away. #### This Is It — Nathalie Djurberg & Hans Berg, Apple (ARKit / visionOS teams) (2019) - Video: https://www.youtube.com/watch?v=req-KX02PvY - Interaction: Location & City, Portals & Worlds, Voice & Sound - Platform & tech: Phone, ARKit, clay animation, spatial audio - Idea: An ordinary-looking tree hides a fairy-tale world inside. - What it is: On Apple's [AR]T Walk, a seemingly hollow tree appears in the street and reveals the strange, magical digital creatures hiding inside, like an outlandish fairy tale. - Technique: Location-anchored ARKit scene with animated, scanned clay-style characters and Hans Berg's score as spatial audio. - Try it: Pick a real tree or cupboard on campus and build an AR scene where creatures peek out of it and hide again when you come closer. Twist: the creatures tell a story only when the viewer stays completely still. ### Nick Cave *Visual and performance artist, known for his Soundsuits* American artist, dancer and educator (not the musician) whose wearable Soundsuits hide the body and turn it into sound and colour. For Apple's [AR]T program he created Amass, an AR installation available inside every Apple Store, and an AR walk with his Soundsuits. #### Accumul-Istic Quest ([AR]T Walk Soundsuits) — Nick Cave, Apple (ARKit / visionOS teams) (2019) - Video: https://www.youtube.com/watch?v=ih-9Ww6sB8I - Interaction: Location & City, Performance - Platform & tech: Phone, ARKit, iPhone - Idea: A wearable sculpture becomes a crowd of AR figures you can walk among. - What it is: A stop on Apple's [AR]T Walk where Nick Cave's wearable Soundsuits are reimagined as AR figures and colourful elements that accumulate around the walking group in the street. - Technique: Location-anchored ARKit scene on the guided [AR]T Walk, experienced on iPhones in small groups. - Try it: Scan or model a costume and place several copies of it along a campus path as an AR procession that starts dancing when the group arrives. Twist: each copy responds to a different sound in the environment. #### Amass — Nick Cave, Apple (ARKit / visionOS teams) (2019) - Video: https://www.youtube.com/watch?v=ebkkHAb6C80 - Interaction: Location & City, Play, Perception & Effects - Platform & tech: Phone, ARKit, Apple Store app - Idea: A chain of retail stores becomes one shared, global AR artwork. - What it is: Inside any Apple Store, opening the Apple Store app turns the shop into a swirling AR universe of Nick Cave's colourful 'Ikon Elements' that visitors collect and gather around them. - Technique: An ARKit experience inside the Apple Store app, likely triggered by store location and anchored to the store's floor and ceiling space. - Try it: Create an AR scene in Lens Studio or Reality Composer where tapping collects floating colourful objects that then orbit around the user. Twist: the objects only appear in one specific room of the school. ### Nova Sky Stories *Drone light-show studio* Drone-show company that frames its shows as 'sky stories': thousands of lit drones recreate artworks and historical scenes, such as Michelangelo's Sistine Chapel over St. Peter's Square in 2025. #### Grace for the World: the Sistine Chapel in drones — Nova Sky Stories (2025) - Video: https://www.youtube.com/watch?v=PozwuT-67pc - Interaction: Location & City, Perception & Effects, Performance - Platform & tech: Projection, drones, LEDs, choreography software - Idea: Moving a ceiling painting into the open sky turns a private indoor view into a public one; AR can release images from inside buildings and show them outside. - What it is: Above St. Peter's Square, 3,000 drones recreate images from Michelangelo's Sistine Chapel ceiling, including the hands of God and Adam, during the Grace for the World concert. - Technique: Artwork is converted into 3D point formations and flown by thousands of synchronised light drones with precise positioning and timed colour changes. - Try it: Take a famous painting from inside a campus building and 'release' it in AR as a drone-point image hovering above the building's roof, with the two hands meeting last. Twist: the image assembles only when viewers stand where the original hangs inside. #### America's founding story in the sky (Boston Pops) — Nova Sky Stories (2026) - Video: https://www.youtube.com/watch?v=xiQuQFh13c0 - Interaction: Location & City, Performance, Information & UI - Platform & tech: Projection, drones, LEDs, storyboarding - Idea: A drone swarm can tell a story in scenes, like a storyboard in the sky; AR narratives outdoors also need few, bold frames that read from far away. - What it is: Over the Charles River during the Boston Pops Fourth of July concert, more than 1,000 drones tell scenes from America's founding in a sequence of animated pictures. - Technique: Scenes are storyboarded, converted to drone waypoints and flown to the orchestra's timing, with transitions animated between each picture. - Try it: Storyboard a local history in five sky frames and play them in AR above the actual site, each frame held for ten seconds and linked by a morph. Twist: tell the same story with only 30 points and compare which frames survive. ### Novum Analytics *Indie studio; Night Terrors AR horror* Small studio behind Night Terrors, a crowdfunded phone AR survival horror game played at home in the dark with headphones. #### Night Terrors — Novum Analytics (2016) - Video: https://www.youtube.com/watch?v=QLcN2NclOTM - Interaction: Spatial Mapping, Play, Voice & Sound - Platform & tech: Phone, iOS, Android, computer vision - Idea: Lights off, headphones on, and your own home becomes a horror game full of monsters. - What it is: A survival horror game played at home in the dark with headphones: the phone becomes the only way to see the monsters and apparitions that move through your own rooms. - Technique: The phone camera feed is darkened and lit by a virtual flashlight, and apparitions are overlaid with jump-scare timing and spatial audio through headphones. - Try it: Build a scene where AR content appears only inside the phone's "flashlight" circle, paired with directional sound in headphones. Twist: the smaller the circle, the closer the horror gets. #### Night Terrors: Bloody Mary — Novum Analytics (2019) - Video: https://www.youtube.com/watch?v=ayL9mU1dnQw - Interaction: Spatial Mapping, Play - Platform & tech: Phone, ARKit, ARCore - Idea: Bring the Bloody Mary legend into your home, with a ghost stalking you in AR. - What it is: A follow-up that brings the Bloody Mary legend into the player's home, with the ghost stalking them through the phone's AR view. - Technique: ARKit or ARCore world tracking lets a ghost character move through room space relative to the player, with audio cues giving direction. - Try it: Have an AR character move behind you so you can only locate it by sound, and make it hide out of view when you turn around. Twist: use the front camera to detect whether you are standing in front of a 'mirror'. ### Otavio Good (Quest Visual) *Creator of Word Lens* Otavio Good founded Quest Visual and built Word Lens (2010), which translated printed signs live in the camera view; Google acquired it for Google Translate. #### Word Lens — Otavio Good (Quest Visual) (2010) - Video: https://www.youtube.com/watch?v=h2OfQdYrHRs - Interaction: Information & UI, Perception & Effects - Platform & tech: Phone, iPhone, OCR, computer vision - Idea: Point at a foreign sign and the words turn into your own language on the spot, like magic. - What it is: Point an iPhone at a sign or menu and the printed Spanish or English text is replaced in place, in the same style, with its translation, running offline in real time. - Technique: On-device OCR finds text regions, a dictionary translates them, and the original text is painted over with sampled background color and redrawn in a matching font. - Try it: Use Google ML Kit or Tesseract to recognize text in the camera view, replace it with another language or another sentence, and paste it back in place. Twist: replace every sign with its antonym. #### Word Lens for Glass — Otavio Good (Quest Visual) (2013) - Video: https://www.youtube.com/watch?v=pZKWW3rzT2Q - Interaction: Information & UI, Gaze & Attention - Platform & tech: Wearable, Google Glass, OCR - Idea: Look at a sign through the glasses and the translation appears right in your view. - What it is: Word Lens on Google Glass: the wearer looks at a sign and sees it translated in their view hands-free. - Technique: The OCR and in-place replacement pipeline runs on head-worn camera frames, so translation appears in view without holding a device. - Try it: Simulate a glasses view with a phone's cameras to build hands-free text translation, and discuss how the interface would need to be simplified for glasses. Twist: only read the translation aloud by voice, with no text shown. ### Otto Piene *Artist; co-founder of ZERO, pioneer of light and sky art* German artist (1928–2014), co-founder of the ZERO group and later director of MIT's Center for Advanced Visual Studies, who made rotating perforated lamps flood rooms with moving light ('Lichtballett') and later inflatable sky sculptures. #### Light Ballet (Lichtballett) — Otto Piene (1961) - Video: https://www.youtube.com/watch?v=MitGag5UUVE - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Projection, perforated metal, rotating lamps, motors - Idea: One rotating lamp with holes turns an entire room into a moving display; AR can get huge spatial effects from one small source projected onto real surfaces. - What it is: Perforated metal spheres and cylinders with lamps inside rotate in a dark room, sending thousands of moving points of light across walls, ceiling and visitors. - Technique: Light shines through hand-punched stencil patterns in rotating shells, so the projected dots sweep over all surfaces of the room in rhythmic choreography. - Try it: Place a virtual perforated lamp on a real table and cast its moving light dots onto the detected walls, floor and ceiling using a spotlight cookie in Unity AR Foundation. Twist: let the user punch new holes into the lamp by tapping it. #### Electronic Light Ballet — Otto Piene (1969) - Video: https://www.youtube.com/watch?v=M13FEIZgn0g - Interaction: Perception & Effects - Platform & tech: Projection, light, television, video - Idea: The same light choreography can live in a room or on a screen; AR designers can prototype spatial light pieces as simple films first. - What it is: A televised light ballet in which moving lamps and perforated forms are filmed and transformed into a flowing composition of light for the screen. - Technique: Likely filmed live light-ballet objects combined with broadcast-TV image processing to create a light composition made for television. - Try it: Film a 1-minute light ballet with a phone flashlight shining through a punched paper cup, then rebuild the best moment as a looping AR light effect anchored in the same room. Twist: sync the rotation to a song's tempo. ### Peter Mohr *Researcher, Institute of Visual Computing, Graz University of Technology* Works with Denis Kalkofen and Dieter Schmalstieg on turning existing manuals and how-to videos into 3D AR instructions, and on using a phone as a tracked controller for headsets. #### TrackCap — Peter Mohr, Tobias Langlotz, Denis Kalkofen (2019) - Video: https://www.youtube.com/watch?v=2fspIrY3TXg - Interaction: Hands & Body, Tangible Objects - Platform & tech: Headset, Phone, marker tracking, smartphone controller, 6DoF - Idea: Flip the tracking: the phone watches the headset, not the other way round. - What it is: A small cap with a marker on the headset lets an ordinary smartphone track itself relative to the head, turning the phone into a precise 6DoF controller and second screen for mobile headsets. - Technique: The phone's camera tracks a marker cap mounted on the HMD, and the inverse pose gives the phone's 6DoF position in the headset's frame, even outside the headset's field of view. - Try it: Tape a printed marker to a cardboard viewer and use a second phone's image tracking to follow it, then use that phone as a pointer in a shared scene. Twist: turn the second phone's screen into a paint palette. #### Video-Annotated AR Assembly Tutorials — Peter Mohr, Denis Kalkofen (2020) - Video: https://www.youtube.com/watch?v=m8uUDR8DERY - Interaction: Information & UI, Tangible Objects - Platform & tech: Phone, Headset, CAD tracking, assembly graph, video analysis - Idea: Convert any existing assembly video into step-by-step AR, keeping the video as backup. - What it is: A normal 2D how-to video of assembling a toy is analysed automatically and turned into a 3D AR tutorial on the real object, with the original video clips pinned to each step. - Technique: Tracks parts in the input video using a CAD model and an assembly graph of blocking relationships, then renders the detected steps in AR with the matching video snippets attached. - Try it: Pick a five-step LEGO model, film it, and build a phone AR guide that shows a 3D arrow and the matching video clip over each step on the real bricks. Twist: let the guide skip steps that the camera sees are already done. ### Phoenix Perry *Game designer, artist and educator; founder of Code Liberation Foundation* Designer of physical and embodied games who founded the Code Liberation Foundation to teach women and non-binary people to code games, and who later led physical computing at Goldsmiths and UAL's Creative Computing Institute. Her work ranges from Kinect-driven music performance to Bot Party, a game played through skin contact. #### 000000swan — Phoenix Perry (2011) - Video: https://www.youtube.com/watch?v=dpW0wRkijQY - Interaction: Hands & Body, Performance, Voice & Sound - Platform & tech: Projection, Desktop, Kinect, Wekinator, Unity - Idea: Teach the computer your own gestures instead of using its built-in ones. - What it is: A live music-and-game performance in which the performer's gestures, read by a hacked Kinect, move graphics through a real-time 3D game world. - Technique: Kinect depth data is fed to Wekinator machine learning, trained on the performer's gestures rather than skeleton tracking, whose outputs control objects in Unity. - Try it: Train Wekinator or Teachable Machine on three of your own gestures and map each to an effect in a projected scene or AR view. Twist: the gestures must come from a dance you already know. #### Bot Party — Phoenix Perry (2017) - Video: https://vimeo.com/252776298 - Interaction: Shared & Social, Tangible Objects, Play - Platform & tech: Wearable, custom electronics, body conductivity - Idea: Human touch is the network cable. - What it is: Made with Freida Abtan, a party game where players must hold hands and touch skin to skin to pass secret messages between their little bots, turning technology into a reason for physical contact. - Technique: Handheld bots send and receive signals through the players' bodies, likely by passing a low-level electrical or audio signal skin to skin, so a message only flows while people are touching. - Try it: Prototype a two-player game with touch-sensing (Makey Makey or capacitive pads) where a sound or AR effect plays only while players hold hands. Twist: the message changes each time a third person joins the chain. ### Pierre Huyghe *Artist* French artist who builds living, self-running environments with animals, bacteria, plants and algorithms, such as Untilled and After ALife Ahead. #### Untilled — Pierre Huyghe (2012) - Video: https://www.youtube.com/watch?v=CyXy7Aok_WM - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, compost site, living dog, beehive sculpture, psychoactive plants - Idea: The work is a system of living parts left to run by itself, a model for AR ecosystems that evolve whether or not anyone is watching. - What it is: In a compost heap of Kassel's Karlsaue park for documenta 13, a reclining nude sculpture with a living beehive for a head, a white dog with a pink leg and toxic plants coexisted and changed without the artist. - Technique: Huyghe set conditions (plants, animals, a sculpture colonised by bees) and let biological processes shape the site over 100 days. - Try it: Build a small AR ecosystem on a table with simple agents (plants, insects, a creature) that keep evolving in a persistent cloud anchor between visits. Twist: users can only observe, never touch. #### After ALife Ahead — Pierre Huyghe (2017) - Video: https://www.youtube.com/watch?v=eWre6dlUAbo - Interaction: Spatial Mapping, Information & UI, Perception & Effects - Platform & tech: Desktop, former ice rink, cancer cells incubator, sensors, augmented reality app - Idea: A living process drives both physical architecture and digital overlays, showing how AR can be steered by a real-world signal rather than the user. - What it is: For Skulptur Projekte Münster, Huyghe cut open the concrete floor of a former ice rink, revealing layers of earth, water, algae, bees and a peacock; cancer cells growing in an incubator controlled the ceiling panels and an AR layer of pyramids. - Technique: Sensors track the growth of HeLa cells in an incubator and use the data to open ceiling panels and spawn augmented reality forms in an app. - Try it: Connect a real-world signal (a plant's soil moisture sensor or a webcam of a fish tank) to the spawn rate of AR shapes in a room. Twist: when the signal stops changing, the AR layer slowly dies. ### Rajinder Sodhi *HCI researcher (UIUC PhD); worked at Microsoft Research, Disney Research and Qualcomm on projected and haptic AR* Co-created LightGuide (projected hand-movement guidance), BeThere (3D mobile telepresence with spatial input) and Disney's AIREAL air-vortex haptics. #### LightGuide — Rajinder Sodhi, Microsoft Research — Hrvoje Benko & Andy Wilson (2012) - Video: https://www.youtube.com/watch?v=vNaw9GpuVLQ - Interaction: Projection, Hands & Body, Information & UI - Platform & tech: Projection, depth camera, projector, on-body projection - Idea: Project movement cues straight onto your hand and let the hand follow the light. - What it is: A ceiling projector draws arrows, color gradients and 3D cues directly on the user's hand to guide it along a precise movement path. - Technique: Tracks the hand with a depth camera and renders movement-guidance visualizations warped onto the hand surface in real time. - Try it: Project an animated path onto the table or a hand and have a classmate follow it with their hand to draw a letter. Twist: switch to phone AR that pins the arrows to the hand on screen, and compare the accuracy of the two kinds of guidance. #### BeThere — Rajinder Sodhi, Lightform (Brett Jones & Kevin Karsch) (2013) - Video: https://www.youtube.com/watch?v=3wVFcrowS74 - Interaction: Shared & Social, Hands & Body, Spatial Mapping - Platform & tech: Phone, depth sensor on mobile, 3D reconstruction, hand tracking - Idea: A remote person reaches their hands into your space to point things out. - What it is: A mobile 3D collaboration system where a remote user sees a reconstructed 3D copy of the local space and reaches in with tracked hands to point and manipulate. - Technique: Attaches a depth sensor to a mobile device to capture the scene and the user's hand, sharing both as 3D data so remote hands appear in the local AR view. - Try it: Set up a two-person video call: the remote person draws circles and arrows on a screenshot, and the local person uses phone AR to place those marks on real objects to complete an assembly task. Twist: only gestures are allowed, no talking. ### Rick Companje *Creative technologist; creator of Globe4D and co-founder of Doodle3D* Dutch creative technologist who co-created Globe4D, a physical globe you turn by hand while projected data and time layers move with it, and later co-founded Doodle3D. #### Globe4D — Rick Companje (2015) - Video: https://www.youtube.com/watch?v=WUL-u_Cx6uM - Interaction: Tangible Objects, Projection, Information & UI - Platform & tech: Projection, openFrameworks, rear projection, multi-touch - Idea: Touch and spin a glowing globe to travel through space and time. - What it is: A large touch-sensitive sphere shows a projected Earth from the inside; visitors spin it with their hands and move through time to watch continental drift, seasons, disasters or flight routes change on the globe. - Technique: An acrylic hemisphere coated for rear projection is lit from inside by a projector with a wide lens, maps are rendered in real time with a shader (the early versions in openFrameworks), and infrared multi-touch tracking reads the hands. - Try it: Project a map onto a white ball (or use AR image tracking on a ball) and let rotating a second object, like a jar lid, scrub through ten years of data. Twist: the ball shows another planet when turned upside down. #### Multi-player Pong on Globe4D — Rick Companje (2016) - Video: https://vimeo.com/170910332 - Interaction: Play, Shared & Social, Projection - Platform & tech: Projection, openFrameworks, Globe4D - Idea: A classic flat game wrapped onto a sphere so players stand around it. - What it is: Several players stand around the projected sphere and play Pong on its curved surface, each defending a segment of the globe. - Technique: The game is rendered in spherical coordinates and rear-projected inside the Globe4D sphere, with each player moving a paddle by touching the surface. - Try it: Take a simple two-player game and redesign it for a round table or ball so that players face each other around it. Twist: the ball's speed depends on how many people are playing. ### Rob Cupisz *Tech lead, Unity Demo Team* Tech lead of Unity's Demo Team (The Heretic, Enemies) and author of the classic LightShafts volumetric-shadow effect. He publishes small example projects that show how the team's real-time SDF and particle techniques work. #### mesh-to-sdf VFX Examples — Rob Cupisz (2022) - Video: https://x.com/robcupisz/status/1591063441225437187 - Source code: https://github.com/robcupisz/mesh-to-sdf-examples - Interaction: Hands & Body, Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, mesh-to-sdf, HDRP - Idea: Rebuild the character's shape as a field every frame, and any number of effects can hug, avoid or fill it. - What it is: An animated character is turned into a signed distance field every frame; a pink particle figure conforms to it, electric 'Sparky' particles collide with it, and green bubbles are raymarched around it. - Technique: Unity's mesh-to-sdf package voxelises a low-poly proxy of the animated character into a 3D SDF texture each frame; VFX Graph samples it with Position (SDF), Conform to SDF and Collide with SDF nodes, and a shader raymarches it. - Try it: Bake an SDF of a scanned real object (or an ARKit body-tracked avatar) and make VFX Graph particles swarm onto its surface in AR. Twist: the swarm leaves the body when the person stops moving. #### Flying Embers — Rob Cupisz (2025) - Video: https://x.com/robcupisz/status/1883842759825715217 - Source code: https://github.com/robcupisz/flying-embers - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, Shader Graph - Idea: Model how a real camera sees a moving hot spark instead of stretching a sprite. - What it is: Glowing embers drift and tumble through the air around a character and still look right in close-up and at grazing angles. - Technique: Instead of stretching quads along velocity, the VFX Graph and Shader Graph effect draws each ember as a camera-facing streak whose length and falloff approximate motion blur and depth-of-field, so it holds up from any angle. - Try it: Place a campfire in AR and compare velocity-stretched embers with camera-aware streaks while walking around it. Twist: the embers drift towards the phone's flashlight. ### Robert Irwin *Artist; Light and Space* American artist (1928–2023) who moved from painting to 'conditional' works: translucent scrims, tape lines and daylight that make visitors notice how they perceive a specific room, culminating in untitled (dawn to dusk) in Marfa. #### Scrim veil—Black rectangle—Natural light — Robert Irwin (1977) - Video: https://www.youtube.com/watch?v=J_oHzl8qG-0 - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Desktop, scrim, black band, natural light - Idea: A barely visible surface changes how a whole room is read; semi-transparent AR planes can re-divide real spaces without blocking them. - What it is: A sheet of white scrim divides the Whitney's fourth-floor gallery lengthwise, with a black band at its top edge; natural light from the Breuer window makes the fabric appear and disappear as you move. - Technique: A translucent scrim is stretched from wall to wall with a painted black line at ceiling height, so perception of the surface depends on viewing angle and daylight. - Try it: Place a nearly transparent vertical plane across the classroom in AR with a thin black line on top, and vary its opacity with viewing angle so it vanishes head-on. Twist: ask classmates to walk around it and mark where it disappears. #### untitled (dawn to dusk) — Robert Irwin (2016) - Video: https://www.youtube.com/watch?v=hgXeWyOd6LU - Interaction: Perception & Effects, Location & City, Gaze & Attention - Platform & tech: Desktop, scrim, windows, daylight - Idea: The work is the changing light, not the building: an AR piece can be designed around time of day, looking different every hour at the same place. - What it is: A former army hospital in Marfa, Texas, is rebuilt with long corridors of translucent scrim and rows of windows; walking from one wing to the other, the building changes with the daylight from dawn to dusk. - Technique: A U-shaped building is split into a dark and a light wing; scrims hung in the corridors catch and layer daylight from carefully placed windows. - Try it: Design a site-specific AR installation for one corridor at school with virtual translucent scrims whose opacity and tint follow the real clock time. Twist: let students visit at 9 am and 3 pm and compare notes. ### Robin Seibold *Game developer and devlog creator* Swedish game developer who documents his stylised Unity rendering in carefully edited devlogs and open-sources the pieces, including a URP cel shader and a screen-space outline renderer feature. #### Screen-Space Outlines — Robin Seibold (2021) - Video: https://www.youtube.com/watch?v=LMqio9NsqmM - Source code: https://github.com/Robinseibold/Unity-URP-Outlines - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, HLSL - Idea: Edges found in depth and normals turn any render into an illustration. - What it is: A stylised 3D scene gains crisp, even ink outlines around objects and creases, and the devlog walks through how they are detected. - Technique: A URP renderer feature renders view-space normals and depth, then a full-screen pass runs Roberts-cross style edge detection on both and composites dark lines where the differences exceed thresholds. - Try it: Apply the outline renderer feature to virtual objects in an AR Foundation scene and also feed LiDAR depth so real furniture gets inked too, making the room look like a comic panel. Twist: vary line thickness with distance to the user. #### Cel Shading (Devlog 3) — Robin Seibold (2022) - Video: https://www.youtube.com/watch?v=gw31oF9qITw - Source code: https://github.com/Robinseibold/Unity-URP-CelShadeLit - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, HLSL - Idea: Quantise light into a few bands and a 3D model reads like hand-painted animation. - What it is: Characters and props switch from smooth lighting to flat bands of light and shadow with sharp specular highlights, supporting several lights and cast shadows. - Technique: A custom URP lit shader computes diffuse, specular and rim terms per light and passes them through smoothstep thresholds, while sampling URP's shadow maps for the main and additional lights. - Try it: Use the cel shader on AR characters and drive the main light from AR Foundation light estimation so the toon shadows match the real room. Twist: add a slider that lets viewers change the number of light bands live. ### Sallia Goldstein *Technical artist and Snap Lens Creator* LA-based 3D technical artist whose career began with a viral Snapchat lens; she has built lenses for Coachella, UFC and LACMA and teaches in Snap's Lens Academy. #### Body Confetti lens — Sallia Goldstein (2022) - Video: https://www.youtube.com/watch?v=-X32ScGcVyA - Interaction: Hands & Body, Perception & Effects, Performance - Platform & tech: Phone, Lens Studio, Body Mesh - Idea: Confetti bursts out of your limbs as you dance. - What it is: A Snapchat lens tracks the full body so bursts of confetti shoot out from the right limbs as you move and dance. - Technique: Lens Studio body-mesh tracking provides limb positions and velocities that trigger particle emitters attached to the body. - Try it: Use Lens Studio body tracking (or MediaPipe Pose) to make confetti spray from your hand when you wave. Twist: the faster you move, the warmer the confetti colors. #### Character Mashup (Lensathon 2022) — Sallia Goldstein (2022) - Video: https://www.youtube.com/watch?v=xwteu9chtsY - Interaction: Hands & Body, Shared & Social, Drawing & Making - Platform & tech: Phone, Lens Studio, Connected Lenses, body tracking - Idea: Assemble your own monster avatar, drive it with your body, then play with friends on the same screen. - What it is: Users custom-build a mashup avatar, bring it to life with body tracking, and share it with friends through connected lenses. - Technique: Selected body parts are assembled onto a rig driven by full-body tracking, and connected lenses sync avatars between users. - Try it: Have students draw a head, a body and legs on paper, combine them into a character and drive it with MediaPipe Pose. Twist: swap one part with your deskmate. ### Shengdong Zhao — Synteraction Lab *Professor at City University of Hong Kong; formerly Associate Professor at the National University of Singapore* Shengdong Zhao leads the Synteraction Lab, which studies heads-up computing: smart-glasses interfaces that support people during everyday activities instead of pulling them into a phone. With Runze Cai and Nuwan Janaka he built PANDALens and AiGet, proactive LLM companions on optical see-through glasses. #### PANDALens — Shengdong Zhao — Synteraction Lab (2024) - Video: https://www.youtube.com/watch?v=WceDSG4I4cw - Source code: https://github.com/Synteraction-Lab/PANDALens - Interaction: Location & City, Voice & Sound, Information & UI - Platform & tech: Headset, optical see-through HMD, GPT, multimodal sensing - Idea: Glasses that interview you during the trip and write the diary with you. - What it is: A travel companion on optical see-through glasses that notices interesting moments from what you see, hear and do, asks you short reflective questions, and then drafts a travel blog with an LLM from those moments and your answers. - Technique: Multimodal context from the camera, audio and user behavior is used to detect notable moments that appear as prompts on a head-mounted display; an LLM turns the collected moments and replies into narrative text. - Try it: Walk around campus for 30 minutes with a phone app that takes a photo whenever you pause for more than ten seconds, asks you one spoken question about it and records your answer; at the end, send photos and answers to an LLM to write a one-page story. Twist: have the AI write in the voice of the place instead of yours. #### AiGet — Shengdong Zhao — Synteraction Lab (2025) - Video: https://www.youtube.com/watch?v=Z1WACpnZodg - Source code: https://github.com/Synteraction-Lab/AiGet - Interaction: Gaze & Attention, Location & City, Information & UI - Platform & tech: Headset, AR smart glasses, eye tracking, LLM - Idea: Turn idle moments into curiosity: the AI notices what you glance at and tells you something you did not know. - What it is: A proactive assistant on AR smart glasses that notices where you look while you walk or shop and slips in surprising, personal facts about things you usually overlook, such as the plant by the path or the origin of a snack. - Technique: Gaze patterns, the egocentric camera view and a user profile are sent to an LLM that picks low-disruption, personalized knowledge and shows it briefly on the glasses. - Try it: Build a phone or glasses prototype that snaps a frame every 20 seconds during a walk and asks a multimodal LLM for 'one surprising fact about something in this image that a person who loves [your hobby] would enjoy', shown as a single line. Twist: only speak the fact when the user's walking pace slows down. ### Squidsoup *Art collective; volumetric light* International art collective led by Anthony Rowe that hangs thousands of individually controlled LED points in space to make walk-in volumetric displays such as Submergence. #### Ocean of Light — Squidsoup (2010) - Video: https://www.youtube.com/watch?v=QzXaM_KXze4 - Interaction: Perception & Effects, Hands & Body - Platform & tech: Projection, LED grid, volumetric pixels, sensors - Idea: Think of light as a spatial medium with resolution: designing for a low-resolution 3D grid trains students to make forms readable with very few points. - What it is: An early Squidsoup volumetric light grid in which a cube of LEDs displays flocking and wave-like forms that respond to people nearby. - Technique: A three-dimensional array of LEDs acts as voxels; motion sensing (likely ultrasonic or camera-based) steers the animated behaviours. - Try it: Build an 8 x 8 x 8 AR voxel cube on a desk and write a simple flocking system that lights the voxels; let hand tracking scare the flock away. Twist: limit yourself to 50 lit voxels at a time and see if the flock is still readable. #### Submergence — Squidsoup (2013) - Video: https://www.youtube.com/watch?v=aJHRHa1D6OE - Interaction: Perception & Effects, Hands & Body, Spatial Mapping - Platform & tech: Projection, LED points, volumetric display, sound - Idea: A sparse grid of points is enough to make 3D forms you can stand inside; AR point clouds can create presence without heavy meshes. - What it is: Thousands of suspended LED points fill a room as a walk-in 3D display; waves, swirls and figures of light move through the grid while visitors walk inside it. - Technique: Hanging strings of addressable LEDs form a volumetric pixel grid, and software renders 3D animations into it, sometimes reacting to visitors' movement. - Try it: Fill the classroom with a 10 x 10 x 10 grid of AR points in AR Foundation and render a moving sphere of light through it by brightening the nearest points. Twist: let the sphere follow the student who is moving the fastest. ### Stanley Donwood (with Radiohead) *Artist; Radiohead's long-time visual collaborator* British artist who has made every Radiohead album cover since 1994 with Thom Yorke, and whose drawings became explorable worlds in the PolyFauna app and the KID A MNESIA EXHIBITION. #### PolyFauna — Stanley Donwood (with Radiohead), Universal Everything (Matt Pyke) (2014) - Video: https://www.youtube.com/watch?v=bXeoF4xq9x8 - Interaction: Voice & Sound, Portals & Worlds, Perception & Effects - Platform & tech: Phone, gyroscope camera, generative 3D, Unity - Idea: Your phone is a window into a living world built from an album's sketchbook. - What it is: A free app in which you look around a dark, evolving landscape of Stanley Donwood's drawings and creatures by moving your phone, follow a red dot, and hear Radiohead's 'Bloom' fragmented into the space. - Technique: The phone's gyroscope drives a 360-degree camera in a generative 3D environment, and touch spawns creatures whose shapes evolve over time (built by Universal Everything). - Try it: Scan ten of your own sketches, turn them into textured 3D shapes and place them in a gyroscope-controlled web scene that people explore by turning their phones. Twist: let each visitor's touch leave a creature that later visitors meet. #### KID A MNESIA EXHIBITION — Stanley Donwood (with Radiohead) (2021) - Video: https://www.youtube.com/watch?v=AOinMjQ9jo8 - Interaction: Voice & Sound, Portals & Worlds - Platform & tech: Desktop, Unreal Engine, spatial audio - Idea: An album becomes architecture: each song is a room you walk through. - What it is: A free explorable virtual exhibition where visitors wander an upside-down analogue/digital building made from Thom Yorke's and Stanley Donwood's artwork, with rooms that react to Radiohead's music. - Technique: Built in Unreal Engine with Namethemachine and Epic Games, the spaces use original drawings as textures and spatialized stems that respond to the visitor's position. - Try it: Pick three songs and design a small walkable 3D room for each in a game engine, using your own drawings as textures and placing sounds so they get louder as you approach. Twist: make one room only reachable by walking backwards. ### Stephen Wilkes *Photographer* American photographer whose 'Day to Night' series stitches up to 1,500 frames shot from one fixed position over 15 to 30 hours into a single image that flows from daylight to night. #### Day to Night — Stephen Wilkes (2009) - Video: https://www.youtube.com/watch?v=afev0ZjAhUA - Interaction: Perception & Effects, Location & City - Platform & tech: Desktop, fixed camera, timelapse, digital compositing - Idea: Time can be laid out across space so one glance shows a whole day: AR can map time onto a direction, letting viewers read history by looking left to right. - What it is: From a fixed position high above a city or landscape, Wilkes shoots for up to 30 hours and blends the best moments into one photograph whose left side is day and right side is night. - Technique: Hundreds to over a thousand frames from a locked camera are masked and blended so each region of the image comes from a different time. - Try it: Record a 1-hour timelapse of a window view on a phone, then build an AR filter that shows each vertical slice of the live view from a different moment of the timelapse. Twist: tie the time gradient to the phone's compass heading instead of screen position. #### This Fragile Earth: Day to Night — Stephen Wilkes (2023) - Video: https://www.youtube.com/watch?v=IlTY4KNGzAY - Interaction: Perception & Effects, Information & UI, Location & City - Platform & tech: Desktop, fixed camera, digital compositing - Idea: Compressing a day of animal visits into one image reveals patterns of use no single moment shows; AR can do the same for how people use a space. - What it is: Day to Night images of endangered species and habitats — waterholes, migrations, ice — where animals arriving and leaving over a day share one frame. - Technique: A fixed long-lens camera records a location for a full day and selected animals from different hours are composited into one scene. - Try it: Log where people stand in a hallway over a day with a phone and a person-detection model, then show all positions as colored ghosts by hour in AR at the real spot. Twist: let viewers filter to one hour by turning a virtual dial. ### Studio Swine / A.A. Murakami (Azusa Murakami & Alexander Groves) *Art and design duo making ephemeral sculptures from mist, bubbles and light* Japanese-British duo Azusa Murakami and Alexander Groves, who worked as Studio Swine and now as A.A. Murakami, build machines that make fleeting forms, such as mist-filled bubbles that bloom from a tree and burst on your hand. #### New Spring (for COS) — Studio Swine / A.A. Murakami (Azusa Murakami & Alexander Groves) (2017) - Video: https://www.youtube.com/watch?v=lOrkjk6FqIs - Interaction: Hands & Body, Tangible Objects, Perception & Effects - Platform & tech: Desktop, mist-filled bubbles, bubble machine, aluminium tree - Idea: A form that lives a few seconds and responds to your hand is irresistible — ephemeral, touchable AR objects hold attention better than permanent ones. - What it is: A tall aluminium tree releases bubbles filled with white mist from its branches; visitors wearing special gloves can hold and bounce them before they burst into a puff of vapour. - Technique: A machine in the tree forms soap bubbles and fills them with scented mist; special fabric gloves let the bubbles bounce without popping. - Try it: Make virtual mist bubbles bloom from a real plant with AR and let hand tracking bounce them until they burst into a particle puff. Twist: bubbles only survive on the palm of a user who moves very gently. #### Silent Fall — Studio Swine / A.A. Murakami (Azusa Murakami & Alexander Groves) (2022) - Video: https://www.youtube.com/watch?v=c-nBDKNiOdo - Interaction: Perception & Effects, Hands & Body - Platform & tech: Desktop, mist-filled bubbles, falling bubble machine, light - Idea: Constant gentle falling, like snow, is calming and endlessly watchable — AR ambient effects work best at slow, steady rates. - What it is: In a dark gallery, a continuous fall of mist-filled bubbles drifts down onto a reflective floor and bursts into small clouds of vapour among the visitors. - Technique: Machines above the room blow soap bubbles filled with mist that fall under gravity and burst on impact, releasing the vapour. - Try it: Rain slow, glowing bubbles from the ceiling in AR that burst into mist when they hit the real floor or the user's body mesh. Twist: let the fall rate follow the room's quietness. ### Supermassive Games *Game studio; developer of Wonderbook: Walking with Dinosaurs and Start the Party!* British studio best known for Until Dawn; for Sony it built camera-based PlayStation Move and Wonderbook titles that put players and paper books inside the game image. #### Start the Party! — Supermassive Games (2010) - Video: https://www.youtube.com/watch?v=TEveLTmqDng - Interaction: Hands & Body, Tangible Objects, Play - Platform & tech: Desktop, PlayStation Move, PlayStation Eye, PS3 - Idea: Your own mirror image is the game board, and one tracked stick turns into whatever tool each minigame needs. - What it is: A PlayStation Move party game in which players see themselves on the TV and the glowing controller in their hand becomes a hair dryer, a fly swatter, a paintbrush or a sword inside the camera image. - Technique: The PlayStation Eye tracks the glowing Move sphere in 3D and the game replaces it with a virtual prop composited over the live mirrored video. - Try it: Build a webcam mirror game in p5.js or Unity: track a bright-colored object by color and replace it with a virtual tool that pops balloons drifting across the video. Twist: switch the tool every 15 seconds so players must change how they move. #### Wonderbook: Walking with Dinosaurs — Supermassive Games (2013) - Video: https://www.youtube.com/watch?v=-XQ4EJjyVXI - Interaction: Tangible Objects, Information & UI, Play - Platform & tech: Desktop, Wonderbook, PlayStation Eye, PlayStation Move - Idea: A paper book becomes an excavation site, and each page turn advances a natural-history lesson. - What it is: A BBC-licensed Wonderbook title: players dig fossils out of the pages of a physical marker book, then rebuild and feed dinosaurs that stomp out of the book into the living-room camera image. - Technique: The PlayStation Eye tracks printed patterns on each Wonderbook page to anchor 3D scenes to the book, while the Move controller acts as brush, hand or food. - Try it: Print four image-target pages and staple them into a booklet; in Unity with Vuforia or AR Foundation image tracking, make each page reveal one stage of a fossil dig (buried, brushed, assembled, alive). Twist: require the reader to tilt or shake the book to uncover the next layer. ### THÉORIZ (David-Alexandre Chanel, Jonathan Richer & team) *Art and technology studio; developers of the Augmenta tracking system* Lyon studio that builds immersive installations and live shows and developed Augmenta, a people-tracking system; in 2017 it combined VR trackers with projection mapping to create a perspective-correct mixed-reality room. #### Noisy Skeleton — THÉORIZ (David-Alexandre Chanel, Jonathan Richer & team) (2014) - Video: https://vimeo.com/103438556 - Interaction: Hands & Body, Voice & Sound, Projection - Platform & tech: Projection, Kinect, projection, generative sound - Idea: A dialogue between a body and a machine that listens with sound and space. - What it is: An immersive room of abstract projected lines and digital sound that reacts to a visitor's movements, from precise control to accidental feedback. - Technique: Depth-camera skeleton tracking likely drives generative visuals projected around the room and a spatial soundscape that echoes the movement back. - Try it: Map a webcam pose tracker's wrist heights to pitch and line density in a projected sketch (p5.js + ml5). Twist: add a delay so the room answers your gesture two seconds later. #### Mixed Reality room (RnD tests) — THÉORIZ (David-Alexandre Chanel, Jonathan Richer & team) (2017) - Video: https://www.youtube.com/watch?v=NrO5WYG5QIw - Interaction: Projection, Hands & Body, Portals & Worlds - Platform & tech: Projection, Augmenta, Vive Tracker, projection mapping, real-time engine - Idea: Head-tracked projection turns bare walls into windows onto virtual space without headsets. - What it is: An empty white room fills with projected 3D objects and spaces that look correct from a tracked camera's or visitor's point of view as they walk around. - Technique: Vive trackers and THÉORIZ's Augmenta tracking feed the viewer position to a renderer that computes off-axis projections for each projector (fish-tank VR style). - Try it: Project a 3D box into a room corner and use a phone's AR pose (or a webcam face tracker) to update an off-axis camera so the box looks solid from one moving viewpoint. Twist: add a second viewer and see how the illusion breaks. ### Tatzu Nishi *Artist* Japanese artist who builds living rooms and hotel rooms around public monuments, so visitors meet a statue at eye level inside a domestic interior. #### The Merlion Hotel — Tatzu Nishi (2011) - Video: https://www.youtube.com/watch?v=yDYlr47crIk - Interaction: Location & City, Portals & Worlds - Platform & tech: Desktop, temporary hotel room, landmark statue - Idea: Turning a public icon into a private room flips the scale of intimacy, a prompt for AR rooms built around landmarks. - What it is: For the Singapore Biennale, Nishi built a hotel room around the Merlion statue, so guests could sleep next to the city's symbol and visitors could see it indoors. - Technique: A temporary structure enclosed the statue with walls, a bed and a bathroom, open to visitors by day and guests by night. - Try it: Place a virtual bedroom around a campus landmark in AR so that the landmark sits by the bed. Twist: at night, the room's lights turn on and the landmark 'sleeps'. #### Discovering Columbus — Tatzu Nishi (2012) - Video: https://www.youtube.com/watch?v=rF76AmzRGxs - Interaction: Location & City, Portals & Worlds, Perception & Effects - Platform & tech: Desktop, scaffolding, living room interior, monument - Idea: Changing the room around a monument changes what it means; AR portals can wrap a statue in an unexpected interior the same way. - What it is: Around the statue of Christopher Columbus in New York's Columbus Circle, Nishi built a fully furnished living room six storeys up, so visitors met the statue at eye level beside a sofa and TV. - Technique: A scaffold tower supports a room built around the top of the column, decorated with custom wallpaper and furniture. - Try it: Choose a statue and build an AR room around its head (walls, sofa, lamp) so a viewer standing on a bench is 'inside' with it. Twist: the wallpaper pattern tells the statue's history. ### The Heavy Projects (B.C. Biermann) *AR public art studio; Re+Public* B.C. 'Heavy' Biermann's studio behind Re+Public, which animated large street murals and billboards with AR from the early 2010s. #### Re+Public AR mural — The Heavy Projects (B.C. Biermann) (2014) - Video: https://www.youtube.com/watch?v=S20tCj5veFw - Interaction: Location & City, Tangible Objects - Platform & tech: Phone, image tracking - Idea: Point at a street mural and the figures in it start to move. - What it is: Re+Public's AR app animates large street murals: pointing a phone at the wall makes the painted characters and shapes come alive. - Technique: Image tracking recognises each mural as a target and overlays animated layers registered to the painting in the phone's camera view. - Try it: Pick a mural on campus or paint your own poster, and use 8th Wall or MindAR image tracking to animate elements in it. Twist: the animation only appears at certain times, for example it only "wakes up" at dusk. ### The LEGO Group (Hidden Side) *Toy company; LEGO Hidden Side AR sets* LEGO's Hidden Side line (2019) combined physical building sets with a phone app that revealed ghosts haunting the model. #### LEGO Hidden Side — The LEGO Group (Hidden Side) (2019) - Video: https://www.youtube.com/watch?v=OnHa4Cd03B0 - Interaction: Tangible Objects, Play - Platform & tech: Phone, iOS, Android, object tracking - Idea: Build a physical LEGO set, then use a phone to see the ghosts hiding inside it. - What it is: Physical LEGO sets such as a haunted school or graveyard are scanned by a phone app that reveals ghosts inside the model; players hunt them by moving around the build. - Technique: The app recognizes the physical LEGO set (model or image-based tracking), and ghosts are rendered inside it with occlusion from a known digital model of the set. - Try it: Use a real cardboard box model and a simplified 3D stand-in for occlusion so an AR ghost hides behind a window in the box. Twist: opening a door on the real box lets a ghost out. ### Tin Drum (Todd Eckert) *Mixed-reality production company* Studio founded by Todd Eckert that produces large-scale mixed-reality performances for headsets, including Marina Abramović's The Life, Ryuichi Sakamoto's KAGAMI and the mixed-reality play An Ark. #### KAGAMI — Tin Drum (Todd Eckert) (2023) - Video: https://www.youtube.com/watch?v=0ns8M_J7GdM - Interaction: Performance, Shared & Social, Voice & Sound - Platform & tech: Headset, volumetric capture, Magic Leap, synchronized playback - Idea: A concert that can keep happening after the performer is gone, in the same room as you. - What it is: Audiences sit in a circle wearing mixed-reality headsets and watch a volumetric recording of Ryuichi Sakamoto playing a grand piano in the middle of the room, surrounded by visual worlds that grow from his music. - Technique: Sakamoto's performance was captured volumetrically and rendered in synchronized mixed-reality headsets (Magic Leap), so every seat sees him from its own angle. - Try it: Record a classmate playing an instrument with a phone depth or Gaussian-splat capture app and place the result on a real chair in AR for a small audience. Twist: let the room's lights dim as each piece builds. #### An Ark — Tin Drum (Todd Eckert) (2025) - Video: https://www.youtube.com/watch?v=G9gR6JxZafo - Interaction: Performance, Shared & Social - Platform & tech: Headset, volumetric capture, mixed-reality headsets, spatial audio - Idea: Theater where actors sit among you in your own space instead of on a stage. - What it is: Billed as the first play written for mixed reality, An Ark seats the audience in a circle while four volumetric actors, including Ian McKellen, appear among them in headsets and speak directly to them. - Technique: Actors were captured volumetrically and placed in the shared physical room through synchronized mixed-reality headsets with spatial audio. - Try it: Write a 2-minute monologue addressed to someone sitting in a specific chair, record it volumetrically or as a cut-out video, and place it in AR facing that chair. Twist: the monologue changes if the listener stands up. ### Tom Ward *Digital artist and creative technologist* Kent-based digital artist working across 2D and 3D animation, modelling, games and installations, and a freelance creative technologist for Electric Medway since 2024. His AR pieces for Electric Medway fill night-time streets and rooms with glowing plants and deep-sea creatures, and he is adding AR windows to the Intra Stories murals. #### Techno-Botanica — Tom Ward (2024) - Video: https://www.youtube.com/watch?v=7UNSpaXIdvg - Interaction: Location & City, Spatial Mapping, Perception & Effects - Platform & tech: Phone, mobile AR, 3D modelling - Idea: A dark courtyard becomes a glowing alien garden on your phone. - What it is: An AR artwork for Nocturnal at Electric Medway Festival 2024: through a phone, a night-time courtyard fills with glowing, neon alien plants that people can walk among. - Technique: Likely a mobile AR scene anchored to the ground plane, with emissive 3D plant models animated in a game engine. - Try it: Model three glowing plants in Blender, place them in a Lens Studio or 8th Wall ground scene, and plant a garden outside at dusk. Twist: the plants only bloom when the phone's microphone hears singing. #### Tales of the Deep — Tom Ward (2025) - Video: https://www.youtube.com/watch?v=bf4EfRMeI_E - Interaction: Portals & Worlds, Spatial Mapping, Play - Platform & tech: Phone, mobile AR, 3D animation - Idea: The sea floods an everyday room and brings its ghosts with it. - What it is: An AR piece for Luton Lights 2025 in which a spectral pirate and deep-sea creatures swim into an ordinary community room through the visitor's phone. - Technique: Likely mobile AR with animated 3D characters and particle effects placed in the room using plane detection. - Try it: Place a swimming AR fish that follows the phone around the classroom, then add a larger creature that appears only in a dark corner. Twist: the water level rises the longer the room stays quiet. ### USC World Building Media Lab (Alex McDowell, Bradley Newman et al.) *Research lab for world building and immersive storytelling* Lab at the USC School of Cinematic Arts led by production designer Alex McDowell (Minority Report). With Intel Labs, including Ronald Azuma's team, it built the Leviathan Project: AR and VR experiments set in Scott Westerfeld's steampunk world, shown at CES 2014 and Sundance New Frontier 2016. #### Leviathan Mixed Reality Props — USC World Building Media Lab (Alex McDowell, Bradley Newman et al.) (2014) - Video: https://www.youtube.com/watch?v=BLH5VtcN9iE - Interaction: Tangible Objects, Spatial Mapping, Portals & Worlds - Platform & tech: Headset, Unity, photogrammetry, motion capture - Idea: Touch the virtual world through real furniture. - What it is: In a VR story set in the Leviathan world, a real chair recreated with photogrammetry and a tracked book let participants physically touch the virtual scene while embodied as a floating jellyfish. - Technique: Photogrammetry scans of physical props are placed in Unity at the exact positions reported by a motion-capture system, so what users see and touch line up. - Try it: Scan a classroom chair with a phone photogrammetry app, place the model at the real chair's position in a Quest or WebXR scene, and have classmates sit on it blind. Twist: turn the chair into a different creature each time someone sits. #### Leviathan at Intel CES 2014 Keynote — USC World Building Media Lab (Alex McDowell, Bradley Newman et al.) (2014) - Video: https://www.youtube.com/watch?v=vN3Ijgsd1RE - Interaction: Performance, Shared & Social, Spatial Mapping - Platform & tech: Phone, Projection, tablet AR, camera tracking, live compositing - Idea: Fly a virtual whale over a live audience. - What it is: During Intel's CES 2014 keynote, a giant steampunk flying whale from Scott Westerfeld's Leviathan leaves the screen and swims above an audience of 2,500 people, visible on the big screens and a handheld tablet. - Technique: Cameras and a tracked tablet in the hall are registered to the auditorium, and the animated whale is composited live from several points of view onto the stage screens. - Try it: Anchor a large animated creature above the classroom in AR (8th Wall, Lens Studio or AR Foundation) and mirror one phone's view to the projector so everyone watches it pass overhead. Twist: the creature reacts when the class claps. ### Velvet Spectrum (Luke & Morgan Choice) *3D design and animation studio* Studio of Australian visual artist Luke Choice and Morgan Choice, known for colorful 3D typography, who extended their work into Snapchat lenses and Adobe Aero AR pieces. #### Lenses for Emotional Oranges 'All That' — Velvet Spectrum (Luke & Morgan Choice) (2020) - Video: https://www.youtube.com/watch?v=iybLfuSFsL4 - Interaction: Face, Perception & Effects, Performance - Platform & tech: Phone, Lens Studio - Idea: Turn the studio's signature 3D visual language into AR filters for a music video. - What it is: Colorful 3D lenses in Velvet Spectrum's signature style transform the performers and scenes of the Emotional Oranges music video. - Technique: Face and world lenses with stylized 3D assets are layered over the performers and shot directly through the camera for the video. - Try it: Make an AR filter music video for a song, with three filters for different shots that share one palette and set of materials. Twist: the filters switch automatically on the chorus. #### AR birding guide (Adobe Aero) — Velvet Spectrum (Luke & Morgan Choice) (2021) - Video: https://www.youtube.com/watch?v=aeqw-hVz2W8 - Interaction: Tangible Objects, Information & UI, Perception & Effects - Platform & tech: Phone, Adobe Aero - Idea: Scan a postcard and its bird flies out, becoming an interactive birding guide. - What it is: A postcard series for bird-lovers becomes an interactive AR birding guide: illustrated birds fly off the cards and can be explored in 3D. - Technique: Adobe Aero image anchors recognize each postcard and trigger animated 3D birds with tap behaviors. - Try it: Use Adobe Aero to turn postcards drawn by students into an AR field guide where scanning a card brings up a 3D animal and science facts. Twist: when two cards are placed together, their animals interact. ### Violeta Ayala *Quechua filmmaker and XR / AI artist* Bolivian Quechua documentary filmmaker (Cocaine Prison, The Fight) who moved into VR, AR and AI with Prison X and Huk, the Jaguaress, building Indigenous-futurist characters that respond to viewers. She spoke about Las Awichas at the University of Nottingham VIP Studio event alongside AΦE. #### Las Awichas — Violeta Ayala (2024) - Video: https://www.youtube.com/watch?v=maAXs0HzNA8 - Interaction: Voice & Sound, Hands & Body, Information & UI - Platform & tech: Phone, Headset, AR, hand tracking, speech recognition, AI - Idea: Grandmothers' wisdom is carried by AR animals that listen back. - What it is: Andean grandmothers' stories live inside AR robotic animals, from geckos to jaguars, that appear around the viewer and respond to hand movements and voice commands while they narrate. - Technique: Phone or headset AR anchors animated robotic animals in the room, while hand tracking and speech recognition feed an AI dialogue layer that makes each animal reply (built with King's College London). - Try it: Record an elder in your family telling a short story, then attach it to an AR animal that plays the next sentence each time you wave. Twist: the animal asks the viewer a question back at the end. #### Huk, the Jaguaress — Violeta Ayala (2025) - Video: https://www.youtube.com/watch?v=4cDaYOr3QBM - Interaction: Face, Voice & Sound, Gaze & Attention - Platform & tech: Desktop, Projection, computer vision, LLM, real-time voice, robotic camera - Idea: A film that looks at you, judges you and answers. - What it is: An interactive installation where Huk, a digital jaguar deity, watches visitors through a robotic camera, senses them and speaks back, generating short films in real time about the burning of the Amazon. - Technique: Five AI systems are chained together: computer vision on a robotic camera, sensors and emotion analysis feed a language model and real-time voice and image generation that cut the film live. - Try it: Build a webcam 'creature' that detects when a face appears and responds with one of five recorded lines based on how long the person stays. Twist: it forgets everyone after ten seconds, so returning visitors are greeted as strangers. ### Vitukhin Serhii *Unity/AR developer; early HoloKit community developer* Built some of the first HoloKit + ARKit stereo demos in 2017, including a co-located multiplayer drone shooter shown at SIGGRAPH 2017. #### HoloKit 4DViews Rocker Hologram — Vitukhin Serhii (2017) - Video: https://www.youtube.com/watch?v=fGr2l-YpB5s - Interaction: Performance, Perception & Effects - Platform & tech: Headset, Phone, HoloKit, ARKit, 4DViews volumetric video - Idea: A volumetric video of a real rock singer performs in your room. - What it is: A volumetric-captured rock guitarist performs life-size in the room through a HoloKit stereo headset, viewable from any angle. - Technique: Plays back a sequence of textured meshes from volumetric capture anchored on the floor and rendered stereoscopically. - Try it: Film a classmate's short performance on a phone, key it into a transparent-background video, and place it in an AR scene as a "hologram show." Twist: combine several classmates' performances into an AR band. #### HoloKit Drone Attack — Vitukhin Serhii, Botao 'Amber' Hu (2017) - Video: https://www.youtube.com/watch?v=fht47xfyUco - Interaction: Shared & Social, Play - Platform & tech: Headset, Phone, HoloKit, ARKit, Unity UNet - Idea: Two people wearing cardboard AR headsets team up to shoot drones in the same room. - What it is: Two players wearing HoloKit cardboard headsets share one living room and shoot down virtual drones together in stereo AR, shown at SIGGRAPH 2017. - Technique: Both phones start from the same physical position and orientation to share a coordinate frame, then sync game state with UNet. - Try it: Have two classmates start AR on their own phones (optionally inside HoloKit or a cardboard viewer) from the same starting point and shoot the same set of virtual targets together. Twist: one person can only see the targets, and the other can only shoot. ### Walt Disney Imagineering (Haunted Mansion team: Yale Gracey, Rolly Crump, Marc Davis, X Atencio) *Theme park design and engineering* Disney's design and engineering arm. For the Haunted Mansion (Disneyland, 1969), effects designer Yale Gracey and colleagues staged a room-sized Pepper's ghost ballroom and projected filmed faces onto sculpted heads, such as the fortune teller Madame Leota in her crystal ball. #### The Haunted Mansion: Grand Hall ghosts (Pepper's ghost) — Walt Disney Imagineering (Haunted Mansion team: Yale Gracey, Rolly Crump, Marc Davis, X Atencio) (1969) - Video: https://www.youtube.com/watch?v=Dx-_SYsjwJk - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Projection, Pepper's ghost, animatronics, large glass - Idea: Room-scale Pepper's ghost is still the simplest 'hologram': it teaches that a virtual layer must be lit, scaled and staged against the real set to be believed. - What it is: Riders pass above a dining hall where transparent ghosts dance, feast and fly around a real table and chandelier. The ghosts are animated figures hidden below and above the ride, reflected in a huge sheet of glass. - Technique: A giant angled glass pane between the ride and the ballroom reflects brightly lit animatronic figures hidden out of sight, superimposing them on the dimly lit real ballroom (Yale Gracey's Pepper's-ghost staging). - Try it: Build a tabletop ballroom with a real cup and candle, and use a tablet laid flat under a 45° acrylic sheet to make a small ghost dance around them. Twist: rebuild it in phone AR and add real-time occlusion so the ghost can pass behind the cup, something the glass trick cannot do. #### The Haunted Mansion: Madame Leota and Little Leota (projected faces) — Walt Disney Imagineering (Haunted Mansion team: Yale Gracey, Rolly Crump, Marc Davis, X Atencio) (1969) - Video: https://www.youtube.com/watch?v=kaMSvaN_EM4 - Interaction: Face, Projection, Tangible Objects - Platform & tech: Projection, film projection, sculpted head, hidden projector - Idea: Projecting a face onto a head-shaped form gives far more presence than a flat screen; it is the root of face projection and of AR avatars anchored on physical props. - What it is: A fortune teller's head floats inside a crystal ball, and a tiny bride at the exit speaks to riders ('Hurry back...'). Both are blank sculpted heads with a filmed face projected onto them, so they talk and blink in three dimensions. - Technique: Film of actress Leota Toombs' face is projected from a hidden projector onto a white face-shaped sculpture, aligned so the features land on the sculpted nose, eyes and mouth. - Try it: Print or model a small white mask, film a classmate's face straight-on, and use a pico projector or phone AR with image tracking to lay the video onto the mask so it speaks a line. Twist: make the mask answer with a different face each time someone asks it a question. ### Xulipa (Allan Yde) *AR/XR game developer and Lens Studio educator* Danish developer who builds multiplayer AR games for Snap Spectacles and headsets, most notably the wizard-duelling game Wand Duel, and publishes Lens Studio tutorials on YouTube. #### Wand Duel — Xulipa (Allan Yde) (2021) - Video: https://www.youtube.com/watch?v=tR1acxdSXmE - Interaction: Shared & Social, Play, Hands & Body - Platform & tech: Headset, Snap Spectacles, Lens Studio - Idea: Turn AR glasses into a face-to-face magic duel where you can still see your opponent's real face. - What it is: Players wearing Snap Spectacles face each other in a real room and cast spells by flicking a wand, duelling like wizards in a shared AR arena. - Technique: Co-located multiplayer on Spectacles syncs player positions and spell projectiles, with hand or controller gestures mapped to spells (likely). - Try it: Prototype a two-player phone AR duel where each player's phone position is their shield and a swipe fires a spell. Twist: spells bounce off real walls detected by the phone. #### AR Lens for Blind People — Xulipa (Allan Yde) (2023) - Video: https://www.youtube.com/watch?v=pRbeE_P1tdM - Interaction: Spatial Mapping, Voice & Sound, Information & UI - Platform & tech: Phone, Lens Studio, World Mesh, Object Detection, Text to Speech - Idea: Use AR's understanding of space for someone who cannot see the screen at all. - What it is: A Snapchat Lens that works as a digital cane: it senses nearby walls and obstacles through the phone's world mesh and speaks the names of detected objects aloud. - Technique: An invisible physics 'cane' collides with the LiDAR/world mesh to trigger alerts, while object detection results are passed to text-to-speech. - Try it: Build an audio-only AR prototype that warns a blindfolded classmate before they walk into a wall and names one object in front of them. Twist: replace speech with spatial sound cues only. ### Yegor Ryabtsov *AR developer and Lens creator for Snapchat and Spectacles* London-based developer behind the Spectacles painting app Artel and the physics puzzle game Trajectory: The Object Liberation Front, which won Best Innovative Lens at Lens Fest 2025. #### Fireside Tales — Yegor Ryabtsov, Stijn Spanhove, Pavlo Tkachenko (2025) - Video: https://www.youtube.com/watch?v=XDcy9xj_BII - Interaction: Shared & Social, Voice & Sound, Play - Platform & tech: Headset, Snap Spectacles, Snap Cloud, Supabase, Generative AI - Idea: A campfire where the spoken story instantly becomes pictures floating in the shared space. - What it is: Players on Snap Spectacles sit as marshmallow avatars around a virtual campfire, take turns telling a story, and watch AI generate artwork from what they say in real time. - Technique: Speech is transcribed and sent through Snap Cloud/Supabase to an image-generation model, with results synced to all co-located Spectacles players. - Try it: Build a phone AR circle where each student speaks one sentence and an AI-generated image appears above their head. Twist: each new image must be generated from the previous two sentences combined. #### Cross-device product viewer (8th Wall open source) — Yegor Ryabtsov (2026) - Video: https://www.youtube.com/watch?v=n5buY23vU5Q - Interaction: Shared & Social, Information & UI - Platform & tech: Web, Phone, Headset, 8th Wall, Three.js, WebXR - Idea: One link, every screen: the same AR object shared live across phones, laptops and glasses. - What it is: After 8th Wall went open source, a single web codebase shows the same product visualisation on a phone, a desktop and Snap Spectacles, synced in real time and colocated between devices. - Technique: Three.js with the open-source 8th Wall engine and the WebXR Device API renders the scene per device, while a realtime sync layer shares state and a common anchor. - Try it: Build a WebAR page where rotating a 3D object on a laptop rotates it on every classmate's phone. Twist: each device shows a different material, as if seeing the object through a different lens. ### kaiware007 *Unity engineer and VJ* Japanese Unity engineer and VJ who shares GPU particle, plexus, hex-tiling and image-effect samples, plus slides from Unity shader study meetups. #### 3D Reaction Diffusion (Marching Cubes) — kaiware007 (2017) - Video: https://x.com/kaiware007/status/867068705741586432 - Source code: https://github.com/kaiware007/UnityReactionDiffusion3D - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, Compute Shader, Marching Cubes - Idea: Run reaction-diffusion in 3D and turn the chemical field into a living surface. - What it is: A glossy, coral-like blob keeps growing and reshaping itself as two simulated chemicals react inside a 3D volume. - Technique: A compute shader iterates the Gray-Scott reaction-diffusion equations on a 3D grid, and a GPU Marching Cubes pass extracts an isosurface of one chemical's concentration each frame. - Try it: Grow the reaction-diffusion volume on a real table with AR Foundation, seeded where the user taps. Twist: add chemical wherever the phone is held close, so the form grows toward the viewer. #### Sea of Clouds — kaiware007 (2017) - Video: https://www.youtube.com/watch?v=i8HpYQ0hmio - Source code: https://github.com/kaiware007/UnitySeaOfClouds - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Desktop, Unity, HLSL - Idea: Fake a volumetric cloud sea cheaply with a displaced, soft-shaded surface. - What it is: A soft, rolling sea of clouds seen from above, inspired by the cloud kingdom in Super Mario Odyssey, drifting slowly under a bright sky. - Technique: A subdivided plane is displaced by scrolling noise in the vertex shader and shaded with soft rim and subsurface-like lighting so it reads as fluffy cloud rather than a hard mesh (likely no raymarching). - Try it: Place the cloud sea at knee height across a real room with AR Foundation so visitors wade through it. Twist: push the clouds away wherever the user's feet (tracked floor position of the phone) move. ### realities.io *Small XR studio (Puzzling Places)* Small studio that builds 3D jigsaw puzzles from photogrammetry scans of real places, with mixed-reality, hand-tracking and Vision Pro versions. #### Puzzling Places on Apple Vision Pro — realities.io (2024) - Video: https://www.youtube.com/watch?v=86c1ckjA5Pw - Interaction: Gaze & Attention, Hands & Body, Play - Platform & tech: Headset, Apple Vision Pro, visionOS - Idea: On Vision Pro, use gaze and pinch to assemble 3D puzzles of real places in your room. - What it is: The 3D jigsaw game redesigned for Vision Pro, with scanned places assembled in your room using eye-and-pinch input. - Technique: Photogrammetry models are pre-split into pieces, and eye-gaze targeting plus a pinch gesture selects and moves pieces, with snap detection when a piece is near its correct pose. - Try it: Scan a real object, cut it into a few pieces, and drag them back together in AR, snapping each piece into place when it gets close to the right position. Twist: once assembled, the object comes alive and tells its own story. #### Puzzling Places: Local Mixed Reality Multiplayer — realities.io (2024) - Video: https://www.youtube.com/watch?v=7WE24U6QNpA - Interaction: Shared & Social, Hands & Body, Play - Platform & tech: Headset, Meta Quest 3, photogrammetry, shared spatial anchors - Idea: Two people assemble a miniature 3D puzzle of a real landmark together on the same real table. - What it is: Two people in the same room assemble a photogrammetry-scanned miniature landmark together over a real table in passthrough, passing pieces by hand. - Technique: Shared spatial anchors give both headsets the same table frame, and piece ownership is transferred over the network when one player grabs a piece from the other. - Try it: Align coordinates on two phones with a shared image marker and work together to put a 3D model split into four pieces back together. Twist: each person can see only half of the pieces. ### rngtm (Kamosoba) *Unity technical artist; Shader Graph and VFX tool author* A Japanese Unity technical artist who writes widely read Shader Graph and VFX articles on Qiita and Hatena and publishes the matching projects, such as Unity-VFXToolBox with aura effects and procedural ring and cylinder meshes for particle effects. #### Aura VFX Samples — rngtm (Kamosoba) (2021) - Video: https://x.com/rn49rn49/status/1346612720234205185 - Source code: https://github.com/rngtm/Unity-VFXToolBox - Interaction: Perception & Effects, Play - Platform & tech: Desktop, Unity, ParticleSystem, URP, Shader Graph - Idea: A convincing aura is a few simple meshes and particles with textures that flow over them. - What it is: Swirling aura effects rise around a point, built from particle systems and scrolling textures in a game-VFX style. - Technique: Unity ParticleSystem emitters use custom vertex streams and custom data to pass per-particle parameters to URP shaders that scroll and mask noise textures, and the toolbox generates the needed empty systems and shader presets. - Try it: Wrap an aura effect around a person's feet in AR using ARKit body tracking to place it. Twist: the aura's colour follows the loudness of the person's voice. #### Procedural Ring Mesh VFX — rngtm (Kamosoba) (2021) - Video: https://x.com/rn49rn49/status/1347823897211666432 - Source code: https://github.com/rngtm/Unity-VFXToolBox - Interaction: Perception & Effects, Portals & Worlds - Platform & tech: Desktop, Unity, C#, procedural mesh, ParticleSystem - Idea: Generate the effect's geometry in code, so its shape can be tuned like a parameter. - What it is: Dark, spinning ring and disc effects are built from procedurally generated meshes with animated textures sliding across them. - Technique: A C# script builds ring, disc and cylinder meshes with tunable radius, segments and UV layout, and particle shaders scroll gradients and noise along the generated UVs. - Try it: Generate a ring mesh in code and use it as a portal rim on a detected AR wall. Twist: the ring's segment count drops as you walk away, turning it into a low-poly shape. ### AMAGI (Takayosi Amagi) *Web graphics developer and VJ* Japanese web-graphics developer and VJ, author of the VEDA live-coding environment and the VFX-JS / REACT-VFX WebGL effect libraries, who also builds Unity VJ systems. #### ZIGSIM-Parallax (PC as a portal) — AMAGI (Takayosi Amagi) (2019) - Video: https://x.com/amagitakayosi/status/1197225844676026368 - Source code: https://github.com/fand/ZIGSIM-Parallax - Interaction: Portals & Worlds, Face - Platform & tech: Desktop, Unity, ZIG SIM, OSC, ARKit - Idea: Track the viewer's head with a phone and render off-axis perspective so a flat screen feels like a hole in the wall. - What it is: A desktop monitor turns into a window onto a 3D space: as the viewer moves their head, the scene behind the screen shifts in correct perspective. - Technique: The ZIG SIM Pro app streams face-tracking data from an iPhone over OSC (received with OscJack); a CameraController moves the Unity camera to the head position and a Mapper shader projects the rendered RenderTexture onto the screen plane (off-axis projection, likely). - Try it: Use ARKit face tracking on the front camera of a phone to drive the same off-axis camera, so the phone screen becomes a portal into a room behind it. Twist: place the portal on a real wall with the back camera and let two viewers see it from their own positions. ### Aaron Sherwood *Artist, composer and creative technologist* American media artist and composer (Purring Tiger, with Kiori Kawai) who builds interactive audiovisual installations and performances, including Firewall, a stretched spandex membrane that reacts to touch. #### Firewall — Aaron Sherwood (2012) - Video: https://vimeo.com/54882144 - Interaction: Hands & Body, Tangible Objects, Voice & Sound - Platform & tech: Projection, Kinect, spandex, projector, generative music - Idea: A soft surface that you can physically push makes digital content feel touchable — a reminder that AR benefits from real material feedback. - What it is: A stretched sheet of spandex acts as a membrane: when people press into it, fire-like visuals bloom from the point of contact and music plays, louder and faster the deeper they push. - Technique: A Kinect behind the fabric measures how deep each point is pushed, and the depth map drives a particle system projected onto the fabric and a generative music engine (software stack likely Max/MSP or openFrameworks). - Try it: Stretch a sheet of fabric in a frame, track hands pushing into it with a depth camera or phone LiDAR, and render AR flames on the fabric at the press points. Twist: let two people pressing from opposite sides make the flames collide. ### Ackroyd & Harvey (Heather Ackroyd and Dan Harvey) *Artist duo* British duo who project photographic negatives onto growing grass so that chlorophyll prints the image, and cover buildings and ships with living grass. #### Grass photographs — Ackroyd & Harvey (Heather Ackroyd and Dan Harvey) (1990) - Video: https://www.youtube.com/watch?v=RgUp8mZ_W7Q - Interaction: Projection, Perception & Effects - Platform & tech: Projection, seedling grass, photographic negative, chlorophyll - Idea: Light as a slow printer on a living surface shows projection as a biological process, a poetic ancestor of projection AR. - What it is: Ackroyd & Harvey grow grass on vertical walls and project a photographic negative onto it; where more light falls, the grass makes more chlorophyll, so a portrait appears in shades of green and yellow. - Technique: Seedling grass grown on a vertical clay surface is exposed for days to a projected negative image, and chlorophyll develops according to light intensity. - Try it: Project a black-and-white portrait onto a tray of cress for a week using a cheap projector, then photograph the result; build an AR version that 'grows' a portrait on a real lawn. Twist: the AR portrait fades when the real weather is cloudy. ### Adrián Ciborro Montes *Software engineer working on graphics and AI-driven streaming* Spanish engineer who published a Unity project that turns webcam-based 3D pose estimation into VFX Graph particles. #### MoCap VFX — Adrián Ciborro Montes (2022) - Video: https://www.youtube.com/watch?v=z2Kst0t0PBA - Source code: https://github.com/adcimon/mocap-vfx - Interaction: Hands & Body, Performance - Platform & tech: Desktop, Unity, VFX Graph, Barracuda - Idea: No mocap suit: a neural network turns a webcam into a particle body. - What it is: A person filmed by a normal webcam is tracked as a 3D skeleton, and glowing VFX Graph particles stream from their moving limbs. - Technique: A 3D pose-estimation network (ThreeDPoseUnityBarracuda) runs in Unity Barracuda to drive an avatar, whose skinned mesh emits VFX Graph particles. - Try it: Run a pose model (Sentis or MediaPipe) on the phone camera in AR and emit particles from the tracked joints of a dancing classmate. Twist: the particle colour follows the music's tempo. ### Akinori Goto *Media artist* Japanese media artist who creates 3D-printed zoetropes: a spinning lattice of frozen figures lit by a thin projected line of light, so that a walking or dancing body appears to move inside the object. #### toki- — Akinori Goto (2016) - Video: https://www.youtube.com/watch?v=4R9Enhw2Qd0 - Interaction: Perception & Effects, Projection - Platform & tech: Desktop, 3D printing, projector, zoetrope - Idea: Light chooses the frame: an AR designer learns that a sliced reveal can turn a dense, unreadable object into a clear animated moment. - What it is: A 3D-printed lattice of intertwined figures spins while a thin line of projected light cuts through it, revealing a single walking or dancing figure that moves in space. - Technique: Every frame of an animation is arranged radially in one printed object; as it rotates, a static projected light plane illuminates only one frame's slice at a time. - Try it: Build a ring of 24 frozen poses of a figure in AR, rotate it, and render only the pose crossing a virtual light plane. Twist: let the viewer move the light plane with their phone to scrub time. ### Alexandre Rivaux (Bonjour Lab) *Creative technologist; co-founder of Bonjour Lab* Creative technologist at the Paris studio Bonjour Lab, which builds interactive installations and open-sources Unity tools such as VATUtils for driving Houdini vertex-animation textures from VFX Graph. #### VAT Utils for VFX Graph — Alexandre Rivaux (Bonjour Lab) (2022) - Video: https://www.youtube.com/watch?v=Ms41cq63dQw - Source code: https://github.com/Bonjour-Interactive-Lab/Unity3D-VATUtils - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, VFX Graph, HDRP, Houdini - Idea: Simulate offline, replay anywhere: baked physics becomes a real-time particle output. - What it is: Houdini rigid-body shatters, soft bodies and remeshing fluid simulations play back as many instanced meshes inside Unity VFX Graph. - Technique: SideFX Labs VAT 3.0 exports encode per-vertex positions and rotations into textures, and the package provides VFX Graph examples that use those VAT meshes as particle output with the matching shader, one per VAT type. - Try it: Bake a small Houdini shatter as a VAT, spawn it with VFX Graph on an AR Foundation plane, and replay it when the user taps a real object. Twist: spawn many copies with random time offsets so the floor keeps shattering like rain. ### Alicja Kwade *Sculptor and installation artist* Polish-born, Berlin-based artist whose sculptures question time, space and how we perceive reality. With Acute Art she made an AR version of LinienLand (2019) and the AR-BEIT series of interactive AR sculptures. #### AR-BEIT — Alicja Kwade, Acute Art (2020) - Video: https://www.youtube.com/watch?v=Zd0nL6kiUKs - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Phone, Acute Art app, ARKit, ARCore - Idea: Everyday objects in AR behave in ways that question what is real. - What it is: Four interactive AR sculptures: a lamp that buries its head in the ground when you walk toward it, a silver spinning top that never stops, a watermelon striped like Mars floating in the air, and puzzle pieces of parallel universes you drag together. - Technique: Phone AR placement via the Acute Art app with proximity triggers (the lamp reacts to the user's distance) and tap or drag gestures on the other pieces. - Try it: Make an AR object that reacts to how close the viewer comes: it hides, shrinks or turns away when approached. Twist: it only behaves normally when nobody is looking at it through the phone. ### Aman Tiwari *Artist and programmer* Artist and programmer who came through Golan Levin's courses at Carnegie Mellon and works across real-time graphics, openFrameworks and machine learning; his MeshToSDF converter became a standard companion to Unity's VFX Graph. #### MeshToSDF — Aman Tiwari (2019) - Video: https://x.com/aman_gif/status/1100187796302635008 - Source code: https://github.com/aman-tiwari/MeshToSDF - Interaction: Perception & Effects, Hands & Body - Platform & tech: Desktop, Unity, VFX Graph, Compute Shader - Idea: If a moving body becomes a distance field, particles can feel its surface. - What it is: Particles swarm over and cling to an animated character as its mesh is converted into a signed distance field every frame and fed to Unity's VFX Graph. - Technique: A compute shader rasterises the mesh triangles into a 3D texture and runs a jump-flooding pass to fill it with distances, which VFX Graph's Conform to SDF and Collide with SDF blocks read in real time. - Try it: Convert the ARKit body-tracking skeleton's skinned mesh (or the face mesh) to an SDF every frame and let VFX Graph particles cling to the user in AR. Twist: switch the particles from clinging to fleeing when the user claps. ### Amanda Parer *Artist* Australian artist whose giant glowing inflatable rabbits (Intrude) sit in parks and squares, a comment on the rabbit plague in Australia. #### Intrude — Amanda Parer (2014) - Video: https://www.youtube.com/watch?v=0QpZ-02W-2g - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, inflatable, internal lighting - Idea: Cute, glowing giants in a city at night are the simplest crowd-pleasing AR scene, and here they carry an ecological message. - What it is: Giant white inflatable rabbits, up to seven metres tall and lit from within, sit in public squares and parks around the world; the video shows them at Townsville's Ephemera festival. - Technique: Nylon inflatables with internal lights are anchored in public spaces and inflated with fans. - Try it: Place a group of glowing AR animals in a dark courtyard and give them a short message about a local invasive species. Twist: the animals multiply every hour, like the species itself. ### Ana Mendieta *Artist* Cuban-American artist (1948–1985) whose Silueta series pressed, burned and planted the outline of her body into earth, sand, mud and water. #### Siluetas — Ana Mendieta (1973) - Video: https://www.youtube.com/watch?v=xyS5gsw-A5Q - Interaction: Hands & Body, Location & City, Perception & Effects - Platform & tech: Desktop, earth, flowers, fire, film - Idea: A body-shaped mark left in nature holds presence after the person has gone, a direct brief for AR silhouettes left at places. - What it is: In Iowa and Mexico, Mendieta pressed, burned and planted the outline of her body into earth, sand, mud and rivers, recording the traces in photographs and films; the MoMA video introduces the series. - Technique: She lay in the landscape or shaped her outline with flowers, gunpowder, mud or fire, then filmed the trace changing. - Try it: Capture your body outline with segmentation and leave it on the ground in AR, filled with a natural material (petals, fire, water). Twist: the silhouette is only visible to people who stand exactly where you lay. ### Andrés Monroy-Hernández & Lei Zhang — Princeton HCI Lab *Associate Professor of Computer Science at Princeton and director of the Princeton HCI Lab; Lei Zhang leads the Capybara project* Andrés Monroy-Hernández, formerly at Microsoft Research and Snap Research, studies social and creative computing. With Lei Zhang, known for AI-assisted XR authoring tools such as VRCopilot, he built Capybara, an AR and generative-AI programming environment where children create and program their own 3D characters. #### Capybara — Andrés Monroy-Hernández & Lei Zhang — Princeton HCI Lab (2025) - Video: https://www.youtube.com/watch?v=paJP8B8btVk - Interaction: Drawing & Making, Tangible Objects, Hands & Body - Platform & tech: Phone, text-to-3D, auto-rigging, body tracking, object recognition, visual programming - Idea: Make children the creators of AI-powered AR, not just its audience. - What it is: An AR visual-programming environment for children: they create 3D characters and accessories with text-to-3D AI, animate them with auto-rigging and their own body movements, and program how the characters react to real objects the camera recognizes. - Technique: Text-to-3D models generate the characters, auto-rigging and body tracking animate them, vision models recognize physical objects, and block-based code links recognition events to character behaviors. - Try it: Run a kids' workshop: each child types a character description into a text-to-3D tool, imports the model into a phone AR app, and uses a simple rule card ('when you see a banana, dance') implemented with an object detector to make it react. Twist: let children program two characters that react to each other, not only to objects. ### AnhPhu Nguyen & Caine Ardayfio *Harvard students behind I-XRAY; later founders of Halo AI glasses* AnhPhu Nguyen and Caine Ardayfio are student hardware hackers who connected Ray-Ban Meta glasses to face search and LLMs to show how quickly strangers can be identified. #### I-XRAY — AnhPhu Nguyen & Caine Ardayfio (2024) - Video: https://x.com/AnhPhuNguyen1/status/1840786336992682409 - Interaction: Face, Information & UI, Shared & Social - Platform & tech: Wearable, Phone, Ray-Ban Meta, face search, LLM, people-search databases - Idea: A privacy warning built as a working demo: glasses that dox at a glance. - What it is: Two students stream video from Ray-Ban Meta glasses, match strangers' faces against the public web and use an LLM to assemble names, addresses and relatives within seconds, to show how exposed people already are. - Technique: Livestreamed frames are sent to a reverse face-search engine, the resulting URLs are read by an LLM to infer identity, and people-search sites fill in personal details; the authors did not release the code. - Try it: Do not build face identification. Instead, design an AR 'privacy shield' concept: sketch and prototype in Figma or WebAR how glasses should signal recording and how bystanders could opt out. Twist: make the bystander's opt-out visible to the wearer. ### Antonin Fourneau *Artist and game designer* French artist who makes playful public installations and arcade machines; his Water Light Graffiti wall lights up wherever it is touched with water. #### Water Light Graffiti — Antonin Fourneau (2012) - Video: https://vimeo.com/47080920 - Interaction: Drawing & Making, Tangible Objects, Shared & Social - Platform & tech: Desktop, LEDs, conductive contacts, water - Idea: Painting with an everyday material that leaves no mark teaches AR designers to link virtual ink to a real gesture and to let drawings fade on their own. - What it is: A wall of thousands of LEDs lights up wherever it is touched with water, so people paint glowing messages with brushes, sprayers or wet fingers that fade as the water dries. - Technique: Each LED sits between two contacts on the surface; water bridges the contacts, closes the circuit and powers the light until it evaporates. - Try it: Build a WebXR or Lens Studio wall where a tracked fingertip or a sprayed 'virtual water' particle leaves glowing strokes that dry and fade over 30 seconds. Twist: make the strokes last longer the more people add to the same spot. ### Appletea *VRChat and Meta Quest developer* A Japanese VRChat and Quest developer who builds shader-heavy tools, including Depth Scanner, which reads the Meta Quest 3 depth sensor and turns the surroundings into a live point cloud that can be exported as PLY. #### Depth Scanner for Meta Quest 3 — Appletea (2024) - Video: https://x.com/Appletea_VRC/status/1861910477389086953 - Source code: https://github.com/Appletea0673/Depth-Scanner-Project - Interaction: Spatial Mapping, Location & City - Platform & tech: Headset, Unity, Meta XR SDK, Meta Quest 3, shader - Idea: The headset already sees depth; just show the user what it sees. - What it is: Walking outdoors with a Meta Quest 3, the headset's depth sensor paints the surroundings as a live cloud of points over passthrough; moving people and hands are tracked too, and the scan can be saved as a PLY file. - Technique: The Meta XR SDK depth API provides the Quest 3 ToF/stereo depth texture; a shader and compute pass unproject each depth pixel with the camera intrinsics and pose into world-space points that are rendered in real time and accumulated for PLY export. - Try it: Turn the depth map of a Quest 3 or LiDAR iPhone into VFX Graph points in AR, colour them by distance, and export one snapshot. Twist: only points that changed since the last snapshot glow, so moving people light up. ### Aras Pranckevičius *Graphics engineer; ex-Unity (2006–2021); open-source developer* One of Unity's earliest graphics engineers, who worked on the engine from 2006 to 2021 and now contributes to Blender. In 2023 he released UnityGaussianSplatting, an open-source real-time viewer for 3D Gaussian Splatting captures that became a standard starting point. #### Unity Gaussian Splatting — Aras Pranckevičius (2023) - Video: https://www.youtube.com/watch?v=iccfV0YlWVI - Source code: https://github.com/aras-p/UnityGaussianSplatting - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Desktop, Unity, 3D Gaussian Splatting, compute shader - Idea: A photo capture can be rendered directly as millions of fuzzy points, with no meshes or textures. - What it is: A photographed garden scene with tools is flown through in real time, rendered from millions of soft coloured splats and compressed to under 8 MB. - Technique: Trained 3D Gaussian Splatting data is compressed into GPU buffers, a compute shader sorts the splats by depth every frame, and each Gaussian is drawn as a screen-space ellipse blended back to front. - Try it: Capture a classroom object as a Gaussian splat (Luma, Polycam or Scaniverse) and place it back in the same room in AR. Twist: let the splats scatter like dust when you walk through the object. ### Arghya Sur (Genesis Interactive) *XR developer; founder of Genesis Interactive* Indian XR developer behind Genesis Interactive, a VR/AI studio that open-sourced QuestRoomScan, a real-time room reconstruction and Gaussian-splat pipeline for Meta Quest 3. #### QuestRoomScan — Arghya Sur (Genesis Interactive) (2026) - Video: https://www.youtube.com/watch?v=lEn3GkH7Yao - Source code: https://github.com/genesisinteractive/QuestRoomScan - Interaction: Spatial Mapping - Platform & tech: Headset, Unity, Meta Quest 3, Depth API, Gaussian Splatting, Sentis - Idea: The headset can scan the room while you use it, so the real space becomes editable material. - What it is: Wearing a Meta Quest 3, the user looks around and the room is rebuilt as a colored 3D mesh in real time, then refined and optionally turned into Gaussian splats. - Technique: Depth API frames are fused into a GPU TSDF volume, Surface Nets extracts a mesh, and passthrough camera images texture it; scans can be sent to a server for Gaussian-splat training and rendered back on device. - Try it: Scan a corner of the classroom with QuestRoomScan and apply a shader that makes the scanned mesh melt or ripple while passthrough shows the real room. Twist: freeze a scan, then compare it with the live room to reveal what moved. ### BERG (Jack Schulze, Matt Webb, Timo Arnall) *Design consultancy (2005–2014)* London design studio known for Little Printer, the 'Media Surfaces' films and experiments that made invisible technology visible, often with Timo Arnall's film work. #### Making Future Magic: iPad light painting — BERG (Jack Schulze, Matt Webb, Timo Arnall) (2010) - Video: https://www.youtube.com/watch?v=C-dVnG8drxs - Interaction: Drawing & Making, Perception & Effects, Tangible Objects - Platform & tech: Desktop, iPad, long exposure, stop motion - Idea: A moving screen can sweep out a volume: slicing a 3D model over time is a physical preview of how AR renders form in space. - What it is: Moving iPads through the air during long exposures, BERG and Dentsu London draw 3D letters and objects in light: the screen shows successive slices of a 3D model, so the tablet becomes a brush that paints solid forms. - Technique: An iPad app displays cross-sections of a 3D model in sequence as the tablet is pulled through space at a steady speed during a long exposure, with stop-motion animation of the results. - Try it: Write a web page that plays the slices of a simple 3D model on a phone screen, pull the phone through the air during a 10-second long exposure on a second phone, and compare with the same model placed in WebXR. Twist: use the phone's gyroscope to advance slices with real motion instead of time. ### Bangbang Yang *Researcher and developer in ByteDance's PICO mixed-reality group* Bangbang Yang received his PhD from Zhejiang University's ZJU3DV group and works on 3D vision for mixed reality at PICO (ByteDance). DreamSpace (IEEE VR 2024) restyles the textures of a real, scanned room from a text prompt. #### DreamSpace — Bangbang Yang (2024) - Video: https://www.youtube.com/watch?v=Ni5kXdj7cfw - Source code: https://github.com/ybbbbt/DreamSpace - Interaction: Portals & Worlds, Spatial Mapping, Perception & Effects - Platform & tech: Headset, diffusion model, panoramic texturing, inpainting, mesh texturing - Idea: Repaint your real room with a sentence while keeping its real geometry. - What it is: Type a style such as 'Van Gogh' or 'cyberpunk' and your own scanned room is re-textured in that style end to end, with coherent textures across walls, floor and furniture that you can walk through in a headset. - Technique: A diffusion model first imagines a stylized 360-degree panoramic texture from the room's center, which is then propagated to occluded areas with inpainting and imitation and baked onto the reconstructed mesh. - Try it: Scan a room corner with a LiDAR phone (Polycam or Reality Composer), take a 360 photo from the middle, restyle it with an image-to-image diffusion model, and project the stylized panorama back onto the mesh in Unity or Blender to view in AR or VR. Twist: restyle only the surfaces you have touched, so the room changes as you move through it. ### Berndnaut Smilde *Artist; indoor clouds* Dutch artist whose Nimbus series creates a single real cloud inside a gallery or church with a fog machine and humidity, then photographs it in the seconds before it dissolves. #### Nimbus — Berndnaut Smilde (2010) - Video: https://www.youtube.com/watch?v=I_Uu8epvcw0 - Interaction: Spatial Mapping, Perception & Effects, Portals & Worlds - Platform & tech: Desktop, fog machine, photography - Idea: An outdoor thing in an indoor place instantly feels uncanny; AR's strongest effects often move one familiar element into the wrong context. - What it is: A small white cloud hangs for a few seconds in the middle of an empty room before dissolving; the artist makes it with a fog machine and documents it as a photograph. - Technique: Fog from a machine is mixed with moisture and released into still air under controlled temperature and lighting, and the cloud is photographed at its peak before it disperses. - Try it: Place one volumetric AR cloud in the middle of a real corridor or classroom, lit to match the room's light direction, that dissolves if anyone walks through it. Twist: make it rain only on the person who walks under it. ### Bjarne Lundgren *iOS developer and early ARKit game prototyper* Developer who shared some of the first ARKit game prototypes in 2017, including tic-tac-toe on a table and a down-ball tennis game played across the living-room floor. #### Down-ball tennis in the living room — Bjarne Lundgren (2017) - Video: https://www.youtube.com/watch?v=iGJlroUukaM - Interaction: Play, Spatial Mapping, Hands & Body - Platform & tech: Phone, ARKit - Idea: The phone is both the window and the racket. - What it is: An early ARKit game in which colourful virtual balls bounce across the real living-room floor and the player hits them with a racket held in front of the phone view. Video: NEXTREALITY. - Technique: ARKit floor-plane detection gives the balls a physical bounce surface, and the racket is likely attached to the phone's tracked pose as a collider. - Try it: Make an AR game where the phone itself is the bat and a virtual ball bounces on the real floor. Twist: two phones rally the same ball. ### Bot & Dolly *Robotics and film studio (acquired by Google, 2013)* Studio that repurposed industrial robot arms for film motion control. With director Bradley 'GMUNK' Munkowitz they made Box, a live performance of projection mapping onto robot-moved surfaces shot in-camera. #### Box — Bot & Dolly (2013) - Video: https://www.youtube.com/watch?v=lX6JcybgDFo - Interaction: Projection, Perception & Effects, Performance - Platform & tech: Projection, industrial robots, projection mapping, motion control - Idea: Robot arms move the canvases and the camera, a live magic trick where projection and motion stay perfectly in sync. - What it is: Two industrial robots move large panels while projection-mapped imagery on them stays locked to a robot-moved camera, creating illusions shot entirely in-camera. - Technique: Motion-control industrial robots move both the screens and the camera on programmed paths, and projection is rendered from the known poses so the anamorphic imagery stays correct from the camera's viewpoint. - Try it: Using a hand-moved cardboard screen and a fixed phone camera, adjust the projection in real time in TouchDesigner based on ArUco markers on the cardboard so the image always "sticks" to it. Twist: design an illusion that only works from the camera's viewpoint, while the live audience sees gibberish. ### Boyu Li *HCI researcher working on mixed reality and generative video, with Zeyu Wang at HKUST (Guangzhou)* Boyu Li, with Lin-Ping Yuan and Zeyu Wang, created VideoCraft (UIST 2025), a workflow that uses mixed reality to give video-generation models precise spatial control when making concept videos of real spaces. #### VideoCraft — Boyu Li (2025) - Video: https://www.youtube.com/watch?v=uvozXgniRok - Interaction: Drawing & Making, Spatial Mapping, Perception & Effects - Platform & tech: Headset, mixed reality, video diffusion, video-to-video generation - Idea: Use MR to direct a generative video model with precise spatial layers. - What it is: A mixed-reality workflow for concept videos of real spaces: film a room, place and adjust new virtual objects in MR as spatial layers, and a video-to-video generation model renders the final footage with the new elements blended in. - Technique: Placing objects in MR yields depth-aware object layers and camera paths that condition a video-to-video diffusion model, so inserted objects stay in place across frames. - Try it: Record a short phone AR video in which a grey cube stands in for a new object, then feed the clip and a prompt ('a vintage armchair') to a video-to-video or image-to-video model and check whether the placement survives. Twist: film the same shot with and without the placeholder and discuss what the AI needs from AR. ### Bruce Shapiro *Kinetic artist and motion-control engineer* American artist and engineer who has built computer-controlled kinetic sculptures since the 1990s; his Sisyphus tables use a hidden magnet to roll a steel ball through sand, drawing and erasing patterns forever. #### Sisyphus — Bruce Shapiro (2016) - Video: https://www.youtube.com/watch?v=_Y3IA9fuXG0 - Interaction: Drawing & Making, Perception & Effects, Tangible Objects - Platform & tech: Desktop, magnet, steel ball, sand, motion control - Idea: A single moving point leaves a readable trace in a real material; AR trails that interact with real surfaces can feel as calm and physical as this. - What it is: A steel ball rolls silently through a bed of sand, pulled by a hidden magnet, drawing spirals and flower patterns that it then erases with the next drawing. - Technique: A two-axis polar robot under the table moves a magnet along programmed paths; the ball follows above and displaces sand, so the path becomes a drawing. - Try it: Detect a real tabletop in phone AR and roll a virtual ball that carves a persistent groove into a sand shader along a parametric path. Twist: let the viewer draw the path with a finger and watch the ball follow it. ### Bruno Catalano *Sculptor* French sculptor whose bronze travellers (Les Voyageurs) are missing large parts of their bodies, so the harbour or street behind them shows through. #### Les Voyageurs — Bruno Catalano (2013) - Video: https://www.youtube.com/watch?v=bZJ2sE-xyik - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, bronze, fragmented figures, harbour - Idea: The missing body is filled by the real landscape: the most direct physical example of diminished reality and see-through AR. - What it is: Life-size bronze travellers carrying suitcases are missing large parts of their torsos, so the harbour behind them shows through; the video shows them at the port of Étaples. - Technique: Figures are modelled in clay, cast in bronze and cut away, with the remaining parts supported through the suitcase and legs. - Try it: Place an AR human figure on a street and cut a large hole through its body with a mask, so the real street shows through. Twist: the hole grows as the figure 'travels' further from home (GPS distance). ### Camille Dunlop *Digital, spatial and speculative designer* Architecture-trained designer who has worked with the speculative design studio Superflux and teaches Cinematic and Videogame Architecture at the Bartlett (UCL) and at Central Saint Martins. As a 2025 A+E Lab artist in residence she built Anatomy of Embodied Ecologies, a live Unreal Engine landscape of the Thames Estuary mudflats that reacts to the bodies in front of it. #### Anatomy of Embodied Ecologies — Camille Dunlop (2025) - Video: https://vimeo.com/1085797818 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Projection, Desktop, Unreal Engine, body tracking, real-time simulation - Idea: Your movement becomes a force of nature inside a living digital mudflat. - What it is: A speculative real-time landscape of the Thames Estuary's intertidal mudflats, with microalgae and diatoms, reacts to the people moving in front of it, so bodies disturb and reshape the ecosystem on screen. - Technique: Likely camera or motion-capture body tracking feeds positions into Unreal Engine, where particle and vegetation systems respond to presence and movement. - Try it: Model a local habitat (pond, riverbank, lawn) as a particle system in TouchDesigner or p5.js, drive it with webcam body tracking, and project it at floor level. Twist: stillness should make the ecosystem recover, movement should damage it. ### Channel TWo (Adam Trowbridge & Jessica Westbrook) *Artist duo making networked, critical media and locative AR* Adam Trowbridge and Jess Parris Westbrook work as Channel TWo (CH2), making critical media art about networks, power and identity. Their polyCopRiotNode_ placed geolocated 3D riot police in AR at sites of protest in Chicago and Washington DC, and they took part in the 2013 Bushwick AR Intervention. #### polyCopRiotNode_ — Channel TWo (Adam Trowbridge & Jessica Westbrook) (2012) - Video: https://www.youtube.com/watch?v=rhhmRBsfMP4 - Interaction: Location & City, Information & UI - Platform & tech: Phone, Layar, 3D models, GPS - Idea: Make the police response to protest visible as permanent virtual sentries at the sites where it happened. - What it is: A network of low-poly 3D riot police in gas masks is placed in augmented reality at geolocations tied to local protest and policing, first in Chicago and then in Washington DC, visible to anyone who looks with a phone. - Technique: Low-poly riot-police models are published as geolocated augments in a location-based AR browser (likely Layar) at researched coordinates. - Try it: Research three places near school where an invisible rule or power operates, and place a simple 3D sentinel at each with a WebAR geolocation tool, adding a one-line caption. Twist: let the sentinels face whoever is looking at them. ### Charles Pétillon *Photographer and installation artist* French artist whose Invasions series fills houses, courtyards and landscapes with thousands of white balloons, including a 100,000-balloon heart in Covent Garden. #### Heartbeat (Invasions) — Charles Pétillon (2015) - Video: https://www.youtube.com/watch?v=2OCAqJkjPHs - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, 100,000 white balloons, pulsing light - Idea: A mass of one simple element plus a slow pulse makes a building feel alive, a cheap and effective AR composition. - What it is: A 54-metre cloud of 100,000 white balloons fills the Covent Garden market hall in London, lit by a slow pulsing light like a heartbeat. - Technique: Balloons are tied into nets and suspended in the hall; a lighting system inside pulses on a heartbeat rhythm. - Try it: Fill a real room or atrium with thousands of instanced AR balloons and pulse their brightness to the user's heart rate from a watch or camera. Twist: the balloons drift out of the building one by one when the room is empty. ### Charles Ross *Artist working with light and astronomy* American artist known for prisms and solar burns and for Star Axis, a naked-eye observatory carved into a New Mexico mesa and aligned with Earth's axis. #### Star Axis — Charles Ross (1971) - Video: https://www.youtube.com/watch?v=RJ1UVKOv03o - Interaction: Location & City, Perception & Effects, Information & UI - Platform & tech: Desktop, granite, earthwork, astronomical alignment - Idea: Each step up the stair corresponds to a moment in the precession cycle, so walking becomes time travel through astronomy, a strong model for data-bound AR paths. - What it is: Under construction since 1971 on a New Mexico mesa, Star Axis is a naked-eye observatory: a stairway aligned with Earth's axis lets you see Polaris framed by a circle whose size changes with the 26,000-year precession of the axis. - Technique: An 11-storey Star Tunnel is cut into the mesa along the axis of the Earth, with its aperture sized to track Polaris's apparent circle over millennia. - Try it: Build an AR staircase aligned with your latitude's polar axis that shows, at each step, where Polaris will appear in a different century. Twist: let the user walk down to go back in time. ### Chen Chen & Shaoze Zhou *HCI researchers at Florida International University working on mixed reality and AI assistance* Chen Chen's group at Florida International University studies mixed-reality and AI support for everyday activities. ChatMuse (UIST 2026), led by Shaoze Zhou with Christine Lisetti and others, is a proactive MR agent for in-person small-group conversation. #### ChatMuse — Chen Chen & Shaoze Zhou (2026) - Video: https://www.youtube.com/watch?v=IMoYRhvsHsI - Interaction: Shared & Social, Voice & Sound, Information & UI - Platform & tech: Headset, mixed reality headset, LLM, speech transcription, non-verbal cue sensing - Idea: An AI prompter whispering from the wings of a real conversation. - What it is: A proactive mixed-reality agent for in-person small-group conversations: it reads everyone's verbal and non-verbal cues and quietly suggests to the wearer what to say, when to jump in or how to include a quiet friend, learning from how the wearer responds. - Technique: Speech transcription and non-verbal cue sensing from all participants feed an LLM that generates real-time verbal and behavioral guidance shown as MR overlays; the wearer's reactions refine later suggestions. - Try it: Run a three-person conversation in which one person sees a live transcript on a phone or headset; every 30 seconds an LLM reads the transcript and suggests one short action ('ask Mia about her trip'). Debrief on which hints felt helpful and which felt creepy. Twist: show the same hint to everyone at once and watch how the group dynamic changes. ### Choi Jeong-hwa (최정화) *Artist and designer* South Korean artist known for colourful installations of cheap plastic objects and giant inflatable flowers and fruit that breathe in public squares. #### Breathing Flower — Choi Jeong-hwa (최정화) (2012) - Video: https://www.youtube.com/watch?v=K0FS7Z_EMt4 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, inflatable lotus, motorised fans - Idea: One slow looping motion gives an object life, a minimal animation principle for AR sculptures. - What it is: A giant red lotus inflatable, about nine metres across, slowly opens and closes its petals as if breathing, in plazas from Perth to San Francisco's Civic Center. - Technique: Fans inflate and deflate the petals in a timed cycle. - Try it: Place a giant AR flower in a plaza that breathes in sync with the real breathing of the nearest user (microphone). Twist: when many people breathe slowly together, the flower blooms fully. ### Chris Milk *Director and immersive artist; co-founder of Within* Music-video director who moved into interactive and immersive work; The Treachery of Sanctuary turned visitors' shadows into flocks of birds before he co-founded the VR company Within. #### The Treachery of Sanctuary — Chris Milk (2012) - Video: https://www.youtube.com/watch?v=I5__9hq-yas - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, Kinect, openFrameworks, projection - Idea: Across three screens your shadow falls apart, is devoured, then grows wings and flies. - What it is: A triptych of white screens: in the first a visitor's shadow disintegrates into flying birds, in the second birds peck the shadow apart, in the third the shadow grows giant wings and takes flight. - Technique: Kinect depth cameras capture each visitor's silhouette and skeleton, and openFrameworks/Unity-style real-time graphics replace parts of the shadow with animated birds and wings projected on the screens. - Try it: Get the arm skeleton from MediaPipe Pose and grow a pair of wings on the silhouette that flap with the arms. Twist: design your own "three acts", giving the shadow a different fate in each. ### Christian Brinkmann *Audiovisual artist working with plant electrophysiology* Hamburg-born audiovisual artist (b. 1989) trained in audio engineering and electronic music, who now captures the electrical signals of living plants and turns them into responsive sound and light with TouchDesigner, Ableton Live and Unreal Engine. #### Floral Resonance — Christian Brinkmann (2024) - Video: https://x.com/publicartad/status/2005927172448698436 - Interaction: Tangible Objects, Voice & Sound, Projection - Platform & tech: Projection, plant electrophysiology, TouchDesigner, Ableton Live - Idea: Make the silent relationship between a person and a plant audible and visible. - What it is: An interactive audiovisual installation in which a living plant responds to people: touching or approaching it shifts the surrounding sound and light, shown at Manar Abu Dhabi with the artist explaining the piece. - Technique: Electrodes pick up the plant's bioelectrical signals, which change with touch, and TouchDesigner and Ableton Live map them to generative sound and light. - Try it: Wire a houseplant to a Makey Makey or capacitive touch sensor (Arduino) and let each touch change a projected colour and a synth note. Twist: the plant 'sulks' and stays dark if it is touched too often. ### Christian Marclay *Visual artist and composer* Swiss-American artist known for sound and collage works such as The Clock, who created an audio AR lens for the Centre Pompidou with Snap. #### Playing Pompidou — Christian Marclay (2022) - Video: https://www.youtube.com/watch?v=jDTCoxuWRHk - Interaction: Voice & Sound, Location & City, Tangible Objects - Platform & tech: Phone, Lens Studio, Custom Landmarker - Idea: Turn the entire Centre Pompidou into an instrument you can play. - What it is: An audio AR lens turns the Centre Pompidou into a musical instrument: visitors 'play' the building's surfaces and compose music from sounds recorded inside the museum. - Technique: The lens localizes against the building and maps sound samples to architectural elements, so pointing and tapping at parts of the facade triggers and sequences them. - Try it: Record sounds around the teaching building (doors, stairs, water pipes), pin them to their locations in AR, and make each spot sound when tapped. Twist: record your tapping order as a melody. ### Christopher Manzione *Sculptor; founder of the Virtual Public Art Project; Manifest.AR member* Sculptor who in 2010 founded the Virtual Public Art Project, one of the first platforms for site-specific AR sculpture on smartphones, and joined the Manifest.AR collective of AR artists. #### Growth (Virtual Public Art Project) — Christopher Manzione (2010) - Video: https://www.youtube.com/watch?v=d7Gg_NiWnZk - Interaction: Location & City, Tangible Objects - Platform & tech: Phone, Layar, 3D scanning, GPS - Idea: Install a giant public sculpture that costs nothing to build and exists only through phones. - What it is: Visitors to Prospect Park in Brooklyn point an iPhone or Android phone at a spot in the park and see a 30-foot fractal polygon sculpture standing among the trees. - Technique: A hand-made physical form was 3D-scanned and placed at GPS coordinates using an early location-based AR browser (likely Layar), viewable from all sides. - Try it: Sculpt a small object from clay, scan it with a phone, and place it in a campus square at large scale using a geolocated AR tool. Twist: design it so it only makes sense from one path through the square. ### Claes Oldenburg and Coosje van Bruggen *Sculptors* Swedish-American Claes Oldenburg (1929–2022) and Dutch-American Coosje van Bruggen (1942–2009) made giant public sculptures of everyday objects such as a spoon, a clothespin and a shuttlecock. #### Spoonbridge and Cherry — Claes Oldenburg and Coosje van Bruggen (1988) - Video: https://www.youtube.com/watch?v=nb37ZIP4NV4 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, stainless steel, aluminium, fountain - Idea: An everyday object at giant scale makes a whole garden feel like a tabletop, the most common and effective trick in consumer AR. - What it is: A 15-metre spoon with a giant cherry in its bowl spans a pond in the Minneapolis Sculpture Garden; water sprays from the cherry's stem. - Technique: The spoon is built of stainless steel on a hidden frame and the cherry of painted aluminium, with a fountain pipe inside. - Try it: Scan a kitchen utensil and a fruit and place them at 15 metres across a real pond or lawn in AR. Twist: add a physics cherry that rolls off if someone 'bumps' the spoon. ### Clara Bacou *3D artist and Snapchat Lens creator* Street artist turned 3D and AR artist who makes Snapchat and Spectacles Lenses, digital sculptures and branded AR, from the 2019 Spectacles 3 launch lenses to the dragon-flock Spectacles Lens Blazer. #### Butterflies Lens for Spectacles 3 — Clara Bacou (2019) - Video: https://x.com/specsfordevs/status/1192495374478393344 - Interaction: Perception & Effects, Spatial Mapping - Platform & tech: Wearable, Snap Spectacles 3, Lens Studio, depth capture - Idea: Fill your own world with oversized, hand-drawn creatures to show how you feel inside it. - What it is: Filmed through Snap Spectacles 3 with director Karen X Cheng, huge hand-drawn butterflies flap around a person in a park and over a bridge, placed in depth in the captured scene. - Technique: Spectacles 3's two cameras capture depth, and a Lens Studio lens likely uses that depth to place and occlude animated 2D butterfly sprites in 3D around the subject. - Try it: Draw one animal on paper, scan it, and animate it as a flock in an AR scene (Lens Studio or Reality Composer) around a classmate. Twist: make the flock scatter when the person moves quickly. ### Colin Leung (NiloCat) *Technical artist; creator of NiloToon* A technical artist who writes rendering code for anime-style games and created NiloToon, a toon shader for Unity URP. His free UnityURPToonLitShaderExample is one of the most-starred learning repositories for custom URP shaders. #### Unity URP Toon Lit Shader Example — Colin Leung (NiloCat) (2020) - Video: https://www.youtube.com/watch?v=gcUCTLF5hwE - Source code: https://github.com/ColinLeung-NiloCat/UnityURPToonLitShaderExample - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, HLSL - Idea: Stylised lighting is a few deliberate rules on top of real lights, not a filter. - What it is: An anime character is lit with clean cel shading, rim outlines and a special face-lighting option that keeps facial shadows tidy as the light moves. - Technique: A hand-written URP shader quantises the Lambert term into a ramp, adds rim light and an inverted-hull outline pass, and for the face option replaces normal-based shading with a lighting rule that keeps shadows smooth. - Try it: Apply a toon lit shader to a character in AR and use AR Foundation light estimation as its main light direction. Twist: the outline thickens when the real room gets darker. ### Cornelia Konrads *Land artist* German artist whose site-specific installations make stones, sticks and fences appear to float, fly apart or dissolve in forests and gardens. #### Floating and scattering installations — Cornelia Konrads (2012) - Video: https://www.youtube.com/watch?v=V8trnhNs4t0 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, stones, sticks, hidden supports - Idea: Natural objects frozen mid-explosion look like a paused physics simulation, an easy bridge to AR physics effects. - What it is: Konrads builds site-specific works in forests and gardens where piles of stones seem to hover, fences break apart into flying sticks and gates dissolve into the air. - Technique: Stones and sticks are held on thin hidden rods or wires so they appear suspended in mid-air. - Try it: Scan a real pile of stones or a fence and, in AR, freeze it in mid-explosion using a physics simulation paused at one frame. Twist: let the user scrub time forward and backward to reassemble it. ### Crypton Future Media (Hatsune Miku live concerts) *Music software company; producer of Hatsune Miku concerts* Crypton released the Vocaloid voice Hatsune Miku in 2007. Since 2009–2010 her concerts have put the animated singer on stage with a live band, projected onto a transparent screen, and tours such as MIKU EXPO and Magical Mirai travel worldwide. #### Hatsune Miku live (MIKU EXPO 2024, Vancouver) — Crypton Future Media (Hatsune Miku live concerts) (2024) - Video: https://www.youtube.com/watch?v=b6VhAvtekeU - Interaction: Performance, Shared & Social, Voice & Sound - Platform & tech: Projection, transparent screen, 3D animation, live band - Idea: A virtual performer becomes 'real' through shared rhythm, lighting and crowd ritual; social AR can borrow the same cues (timing, call-and-response, shared lights). - What it is: The animated singer Hatsune Miku performs life-size on stage beside a live band, projected onto a transparent screen, while the audience waves glow sticks in sync. This clip is from the 10th-anniversary MIKU EXPO North America tour. - Technique: Pre-animated 3D choreography is rear- or front-projected onto a transparent screen at the front of the stage, locked to the band's click track and to stage lighting cues. - Try it: Place an animated character in shared phone AR on a real stage or desk, sync its dance to a song, and let everyone's phone flash the same colour on the beat. Twist: the character points at whichever phone cheers loudest. ### Cyborn *Belgian VR/MR game studio* Creators of Hubris and the Quest 3 mixed-reality town builder Wall Town Wonders. #### Wall Town Wonders — Cyborn, Creature (2024) - Video: https://www.youtube.com/watch?v=5icyAyKb9Zg - Interaction: Spatial Mapping, Play, Hands & Body - Platform & tech: Headset, Meta Quest 3, Scene API, hand tracking - Idea: Build a living little town on the walls of your home. - What it is: A Quest 3 mixed-reality town builder: tiny houses, streets and townsfolk grow out of your real walls, and you tend to their wishes by hand. - Technique: Uses scene-understanding wall planes as the build grid so buildings attach to vertical surfaces and characters walk along them. - Try it: Use vertical plane detection in phone AR to place little houses and people on a classroom wall and have the people 'walk' along it. Twist: make real windows or posters on the wall become landmarks in the town. ### Doug Wheeler *Artist; Light and Space* American artist of the Light and Space movement who builds rooms with curved, seamless walls and carefully tuned light so that visitors lose all sense of depth and distance. #### 49 Nord 6 EST 68 VEN 12 FL — Doug Wheeler (2012) - Video: https://www.youtube.com/watch?v=aBJv_SdIgUQ - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Projection, curved plaster walls, LED lighting, white room - Idea: Remove corners and shadows and a room becomes infinite; in AR, hiding the edges of virtual content is often more convincing than adding detail. - What it is: Visitors enter a seamless white room with curved corners at David Zwirner; slow changes of light make the far wall dissolve, so the room seems to have no depth and no end. - Technique: Coved walls, a floor that curves up into the walls and evenly diffused, slowly programmed light remove every visual cue for distance. - Try it: In WebXR, cover one real wall with a virtual surface whose brightness gradient slowly erases its edges until it reads as open space. Twist: add a single floating dot and test whether it helps or hurts the sense of depth. ### Elly Oldman *Illustrator and author of The Infinite Drawing* French illustrator who has drawn a single endless black-and-white picture, The Infinite Drawing, a little more each day on Instagram. She turned it into a giant wall fresco and floor puzzle whose characters come alive in augmented reality, animated by the studio &Friends with AR by Artefacto, and toured it to Rennes and A+E Lab's ALT*ASH in Ashford. #### The Great Story of the Infinite Drawing — Elly Oldman (2020) - Video: https://www.youtube.com/watch?v=2oksun7jx1c - Interaction: Drawing & Making, Tangible Objects - Platform & tech: Phone, image tracking, tablet AR, hand-drawn animation - Idea: A giant hand-drawn wall where every corner hides an animated scene. - What it is: A monumental black-and-white fresco of Elly Oldman's endless drawing covers a wall and a floor puzzle; visitors point tablets at it and the characters of her ecological fable start moving. - Technique: Sections of the printed drawing serve as image targets for a tablet AR app (AR by Artefacto) that overlays frame-by-frame animations by &Friends in the same line style. - Try it: Draw one long collaborative black-and-white strip on paper, then pick five details and make each one move in AR with Artivive or Lens Studio image markers. Twist: hide one animation that only appears when two tablets look at it together. ### Eric Freeman *Game developer* A game developer who shares small Unity utilities, including FakeStopMotion, a script that makes any animated character play back at a low, choppy frame rate like stop-motion puppetry. #### Fake Stop Motion — Eric Freeman (2021) - Video: https://x.com/_ericfreeman/status/1369299130678382593 - Source code: https://github.com/EricFreeman/FakeStopMotion - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, C#, Animator - Idea: Change the time signature of motion, not the look of pixels, to get a hand-made feel. - What it is: A smoothly animated 3D character is made to move in choppy, held poses like a stop-motion puppet, while the rest of the game runs at full frame rate. - Technique: A script drives the Animator manually at a chosen low frame rate, advancing it in discrete steps instead of every frame, so it can be combined with any retro or PSX-style shader. - Try it: Put a stop-motion AR character on a real desk that updates at 8 fps while the camera image stays smooth. Twist: the frame rate rises the longer someone watches it. ### Eric Staller *Light artist and designer* American artist known for 1970s 'Lightscapes' light-drawing photographs in New York streets and for public light works such as the Lightmobile, a Volkswagen Beetle covered in computer-controlled bulbs. #### Lightmobile — Eric Staller (1985) - Video: https://www.youtube.com/watch?v=6TxVpY_3iPc - Interaction: Location & City, Perception & Effects, Tangible Objects - Platform & tech: Desktop, light bulbs, microcontroller, car - Idea: Light content does not have to stay on buildings: attaching animated light to a moving object makes the street itself the stage, a cue for AR content that rides on vehicles. - What it is: A Volkswagen Beetle covered with about 1,600 computer-controlled light bulbs drives through city streets, its patterns chasing across the body as it moves. - Technique: Hundreds of bulbs on the car body are sequenced by an onboard computer to run chase and pattern animations. - Try it: Track a toy car or a bike with image or object tracking and wrap it in an animated pattern of virtual bulbs that chase in the direction of travel. Twist: make the chase speed follow the real speed of the object. ### Even Realities *Maker of the Even G1 / G2 display smart glasses* Even Realities makes lightweight prescription-style glasses with a green micro-display for notes, translation, teleprompting and an AI assistant. #### Even AI on Even G1 glasses — Even Realities (2024) - Video: https://www.youtube.com/watch?v=Quam8jLa8pY - Interaction: Voice & Sound, Information & UI - Platform & tech: Wearable, Even Realities G1, LLM, micro-display - Idea: An assistant whose answers are private text in your line of sight. - What it is: Even G1 glasses include a built-in AI that listens when asked and answers with short text in a green micro-display that only the wearer can see. - Technique: Speech is captured by the glasses' microphones, processed by a cloud LLM via the phone, and rendered as monochrome text on the waveguide display. - Try it: Prototype a 'one-line assistant' in a phone headset viewer: every answer must fit in 40 characters, shown in green text at the bottom of the view. Twist: the assistant speaks only when you are silent for five seconds. ### Fabrice Matulic (Preferred Networks) *Senior researcher at Preferred Networks* Fabrice Matulic is an HCI researcher at Preferred Networks in Tokyo who works on pen, touch and AR interaction. With Nozomu Iwai he turned household objects into LLM-driven talking AR characters (CHI 2025). #### Bringing Everyday Objects to Life with AI-Powered Talking Characters — Fabrice Matulic (Preferred Networks) (2025) - Video: https://www.youtube.com/watch?v=RkRc7TRDPDQ - Interaction: Tangible Objects, Voice & Sound, Play - Platform & tech: Headset, Phone, HoloLens 2, Android, LLM, vision models, speech synthesis - Idea: Give every object a voice and a personality that come from what the object is. - What it is: Household objects become talking AR characters: a kettle, a plant or a chair gets an animated face and a personality drawn from what it is, chats with you about the room and reacts when things around it change. - Technique: Vision models recognize the object and its surroundings, an LLM generates a personality and dialogue grounded in the object's function, and speech synthesis with an animated overlay runs on HoloLens 2 or an Android phone. - Try it: Stick AR googly eyes on three objects with Lens Studio or web image tracking, write a one-line persona for each ('I am a stapler who hates loose paper'), and connect them to an LLM and text-to-speech so students can interview them. Twist: let two objects gossip with each other about the user. ### Fernanda De La Torre (LLMR, with Microsoft) *Researcher; lead author of LLMR with Microsoft collaborators including Jaron Lanier* Fernanda De La Torre led LLMR (CHI 2024) with Cathy Mengying Fang, Han Huang, Andrzej Banburski-Fahey, Judith Amores and Jaron Lanier at Microsoft. LLMR lets people create and modify interactive mixed-reality worlds in real time by talking to a team of LLM agents. #### LLMR — Fernanda De La Torre (LLMR, with Microsoft) (2024) - Video: https://www.youtube.com/watch?v=9YdYkgNNpE8 - Source code: https://github.com/microsoft/LLMR - Interaction: Voice & Sound, Drawing & Making, Information & UI - Platform & tech: Headset, Desktop, GPT-4, Unity, DALL-E, CLIP, Sketchfab - Idea: Language becomes the runtime: objects, behaviors and scene edits are generated as code while you are inside the world. - What it is: A framework for speaking interactive mixed-reality worlds into existence: type or say 'make a timer out of a rotating cube' or 'recolor the scene for a color-blind user' and an LLM writes and runs the Unity code live. - Technique: Several GPT-4 roles (scene analyzer, planner, builder, inspector) collaborate using a scene summary and memory, generate C# for Unity and debug it before execution. - Try it: In Unity, expose a tiny runtime command layer (a JSON command set or Roslyn scripting) and let an LLM receive a scene summary plus a user request and return commands; try five requests ranging from spawning objects to changing physics. Twist: add an 'inspector' prompt that reviews generated code before it runs and count the errors it catches. ### Filipe Peregrino *Graphic designer; multilingual AR typography* Brazilian graphic designer based in London whose London College of Communication project The Languages Around Us places animated, multi-script type installations into real spaces with AR. #### The Languages Around Us — Filipe Peregrino (2020) - Video: https://www.youtube.com/watch?v=WSziZQP_TOw - Interaction: Information & UI, Location & City, Drawing & Making - Platform & tech: Phone, AR, Noto fonts, motion design - Idea: Make the many languages spoken around you visible in the places you share. - What it is: Animated words in more than 40 languages appear as AR typographic installations in real London spaces, morphing from one writing system into another. - Technique: Words with similar stroke counts in Google Noto fonts are paired, stroke-animated and morphed on a timeline, then anchored in place as AR scenes. - Try it: Collect the word 'hello' in the first languages of your classmates and animate a morph between two scripts, then place it in AR where that classmate usually sits. Twist: the word only appears when someone standing there speaks. ### FireDragonGameStudio *Independent XR developer and tutorial maker* FireDragonGameStudio publishes open-source Quest and AR experiments, from passthrough-camera object detection to Gemini-powered indoor navigation. #### Gemini Vision Navigator — FireDragonGameStudio (2026) - Video: https://www.youtube.com/watch?v=m61hl5wy9cE - Interaction: Location & City, Voice & Sound, Spatial Mapping - Platform & tech: Headset, Phone, Gemini Live API, passthrough camera, indoor navigation - Idea: Indoor navigation without beacons: an agent that reads the room and tells you where to go. - What it is: A Gemini Live agent looks through the device camera, understands the indoor scene and guides the user by voice and AR cues toward a destination, submitted to the Gemini Live Agent Challenge. - Technique: Camera frames and speech are streamed to the Gemini Live API, whose function calls place AR arrows and markers in the passthrough view. - Try it: Use a multimodal live API on a phone to guide a blindfolded partner (with a spotter) from a classroom door to a chosen desk by voice only. Twist: the guide describes landmarks as characters in a story. ### Frank & Lillian Gilbreth *Industrial engineers; inventors of the cyclegraph and chronocyclegraph* American engineers and time-and-motion researchers who fixed small lights to workers' hands and photographed them with long exposures, producing light traces of movement and wire sculptures of the ideal motion path. #### Cyclegraphs and Chronocyclegraphs — Frank & Lillian Gilbreth (1914) - Video: https://www.youtube.com/watch?v=FgYC1qQ9lmA - Interaction: Hands & Body, Information & UI, Perception & Effects - Platform & tech: Desktop, long exposure, pulsed lamps, stereo camera - Idea: A light trail is also a measurement: AR hand-tracking trails can double as feedback that teaches a smoother, shorter motion. - What it is: Small bulbs on a worker's hands leave bright paths on long-exposure photographs of drilling and assembly tasks; interrupted lights add dots that show speed and direction. - Technique: Long exposures record small lamps on the hands; in the chronocyclegraph the lamps are pulsed so dot spacing shows velocity (dates approximate). - Try it: Use hand tracking on a headset or phone to record the path of a fingertip while a student ties a knot, then draw the path as dots spaced by time. Twist: show an expert's path in a second color and score the difference. ### Fujiko Nakaya *Fog artist* Japanese artist who since 1970 has made fog sculptures: timed clouds of pure water mist that roll over gardens, ponds and buildings and react to wind. #### Fog Sculpture (Neue Nationalgalerie) — Fujiko Nakaya (2021) - Video: https://www.youtube.com/watch?v=k7VeOZbUZyk - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, high-pressure water mist, timed nozzles, wind - Idea: Fog is a real volume that people walk into and that the weather shapes, the benchmark for any volumetric AR effect. - What it is: In the sculpture garden of the Neue Nationalgalerie in Berlin, clouds of pure water fog pour out at timed intervals, wrapping visitors, trees and Mies van der Rohe's building and drifting with the wind. - Technique: High-pressure pumps push water through hundreds of tiny nozzles that create droplets small enough to hang in the air as fog. - Try it: Build a volumetric fog cloud in AR (raymarched or particle-based) that pours from a real bench at set times and drifts with live wind direction. Twist: the fog parts around people using body segmentation. ### Gjon Mili *Photographer and filmmaker, LIFE magazine (1904–1984)* Albanian-American photographer who worked with Harold Edgerton on stroboscopic and multiple-flash photography and, in 1949, photographed Pablo Picasso drawing in the air with a small light. #### Picasso Light Drawings — Gjon Mili (1949) - Video: https://www.youtube.com/watch?v=X-i9eqlRzks - Interaction: Drawing & Making, Hands & Body, Perception & Effects - Platform & tech: Desktop, long exposure, flashlight, strobe flash - Idea: A drawing in the air lasts only as long as someone records it: AR finally makes these gestures persistent and walkable. - What it is: In a darkened room in Vallauris, Pablo Picasso draws a centaur, a bull and his signature in the air with a small flashlight while Mili's open-shutter camera records the glowing lines. - Technique: Long exposure in the dark with a moving point light, combined with a short flash to reveal the artist's figure. - Try it: Build a WebXR or Lens Studio air-drawing tool that traces the phone or fingertip as a glowing tube, and ask each student to draw one animal in a single continuous line. Twist: the line fades after 10 seconds unless someone else walks through it. ### Hamish Fulton *Walking artist* British artist who defines his art as walking; he records long walks in text and photographs and leads slow group walks in cities. #### Group Walk, Birmingham — Hamish Fulton (2012) - Video: https://www.youtube.com/watch?v=2XaX4ktvQ_0 - Interaction: Location & City, Shared & Social, Hands & Body - Platform & tech: Desktop, slow walking, group score - Idea: A simple shared rule for walking becomes the artwork; an AR score can guide groups through a site in the same way. - What it is: At Curzon Park in Birmingham, Fulton led a group walk in which participants moved slowly along set lines, turning a derelict site into a collective, silent performance. - Technique: Participants follow spoken instructions (walk slowly, in silence, along parallel lines), with no objects left behind. - Try it: Write an AR walking score where each participant sees a personal line on the ground and a speed indicator, and all lines converge at one point. Twist: the lines only appear while the walker keeps below a set speed. ### Harvard Visual Computing Group — Hanspeter Pfister *Research group at Harvard SEAS led by Hanspeter Pfister* Hanspeter Pfister's Visual Computing Group works on visualization, computer vision and sports analytics. ViSTAR (CHI 2026), led by Chunggi Lee and Hayato Saiki, is an AR basketball trainer with an LLM coaching agent. #### ViSTAR — Harvard Visual Computing Group — Hanspeter Pfister (2026) - Video: https://www.youtube.com/watch?v=aaCfLeenGdE - Interaction: Hands & Body, Play, Information & UI - Platform & tech: Headset, 3D motion reconstruction, LLM, AR headset - Idea: Let an LLM read your 3D motion data and talk like a coach. - What it is: An AR basketball trainer: a 3D avatar demonstrates the shot, overlays and rhythm cues show your balance, posture and timing, and an AI coach turns your reconstructed 3D motion into short spoken tips. - Technique: 3D motion reconstruction extracts spatio-temporal joint features that an LLM maps to natural-language coaching cues within a behavioral-skills-training loop of instruction, modeling, rehearsal and feedback. - Try it: Record a squat or a free throw with MediaPipe Pose on a phone, compute three numbers (knee angle, back angle, timing), send them to an LLM with the prompt 'give one coaching cue in under eight words', and show the cue in AR next to your body. Twist: ask the LLM to coach in the style of a famous coach or a cartoon character. ### Henry Daubrez & Samuel Honigstein (Dogstudio / KIKK in Town) *Curators and producers of the KIKK Festival 2018 AR art exhibit* Dogstudio founders Henry Daubrez and Samuel Honigstein conceived and produced the first AR art exhibit of KIKK in Town (2018): seven code-generated pieces by creative coders such as Bruno Simon, Matt DesLauriers and Edan Kwan, unlocked around Namur. #### KIKK Festival 2018 – Augmented Reality Exhibit — Henry Daubrez & Samuel Honigstein (Dogstudio / KIKK in Town) (2018) - Video: https://vimeo.com/299883293 - Interaction: Location & City, Perception & Effects, Information & UI - Platform & tech: Phone, WebGL, mobile AR, generative art - Idea: Curate a street exhibition where the frames are real but the art only exists in AR. - What it is: Seven code-generated AR artworks by creative coders are hung around the city of Namur on physical frames; festival visitors unlock each one with a phone to watch it grow out of the wall. - Technique: Each physical frame is an image target for a festival app that renders a real-time generative WebGL-style piece anchored to it (artists included Bruno Simon, Matt DesLauriers and Edan Kwan). - Try it: Hang five empty paper frames around the school, each an image target that loads a different student's generative sketch in WebAR. Twist: the pieces change with the time of day. ### Ho Tzu Nyen *Artist and filmmaker working with history and theory* Singaporean artist who makes films, installations and performances that start from historical and philosophical texts; he represented Singapore at the 2011 Venice Biennale. His AR work Language (2022, Acute Art, National Gallery Singapore) stages texts of the Kyoto School depending on where you point your phone. #### Language — Ho Tzu Nyen, Acute Art (2022) - Video: https://www.youtube.com/watch?v=jKLx6V6FFec - Interaction: Gaze & Attention, Voice & Sound, Information & UI - Platform & tech: Phone, Acute Art app, gyroscope, spatial audio - Idea: Where you point the phone (sky, ground, horizon) decides which story you hear. - What it is: Point the phone at the sky and giant anime-style mecha appear while one Kyoto School text is read; point it at the ground and a disintegrating scholar appears with another text; at the horizon there is nothing to see, only a voice speaking about absolute nothingness. - Technique: Phone AR that reads the device's pitch from the gyroscope to switch between three layered scenes and audio tracks. - Try it: Build a WebXR or phone web page that plays three different audio stories depending on whether the phone points up, down or level. Twist: the level direction shows nothing on screen at all. ### Iregular (Daniel Iregui) *Digital art studio making interactive audiovisual installations* Montreal studio founded in 2010 by Colombian-Canadian artist Daniel Iregui; its minimal, generative installations react to the bodies and sounds of the visitors who enter them. #### Control No Control — Iregular (Daniel Iregui) (2016) - Video: https://vimeo.com/1171876250 - Interaction: Hands & Body, Projection, Perception & Effects - Platform & tech: Projection, depth cameras, projection, generative graphics - Idea: Let the audience feel how much — and how little — control they have over a living surface. - What it is: A minimal geometric structure covered in projections that react when visitors touch and move near it, shifting between order and chaos. - Technique: Depth sensors detect hands and bodies around the structure and feed a generative system projection-mapped onto its faces (likely custom real-time software). - Try it: Build a cardboard pyramid, projection-map its faces, and use a webcam to detect how many people stand near it; more people means more chaotic patterns. Twist: one face ignores the crowd and only reacts to a single touch. ### Isamu Noguchi *Sculptor and landscape architect* Japanese-American artist (1904–1988) who treated whole landscapes as sculpture; his last work, Moerenuma Park in Sapporo, turned a landfill into a park of geometric hills. #### Moerenuma Park — Isamu Noguchi (2005) - Video: https://www.youtube.com/watch?v=ajgj-f-sDVM - Interaction: Location & City, Play, Perception & Effects - Platform & tech: Desktop, landfill conversion, earth pyramid, playground sculptures - Idea: A whole park conceived as one sculpture shows how land shape itself can be the interface for play, a guide for AR in parks. - What it is: Designed by Noguchi before his death in 1988 and completed in 2005, Moerenuma Park in Sapporo turns a garbage landfill into a park of geometric hills, a glass pyramid, fountains and sculptural playgrounds. - Technique: The landfill was capped and reshaped with earth into cones, a mound and a pyramid, following Noguchi's model. - Try it: Model a small park of geometric AR hills and play objects on a flat lawn, then test with children how they move through it. Twist: the shapes subtly reshape toward where people walk most. ### Jaagup Kuhi *Graphics programmer* Estonian programmer who open-sourced a real-time raymarched volumetric cloud renderer for Unity, with weather maps, noise shapes and lighting controls. #### Realtime Volumetric Clouds in Unity — Jaagup Kuhi (2018) - Video: https://www.youtube.com/watch?v=zOyHy969Y6I - Source code: https://github.com/jaagupku/volumetric-clouds - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL - Idea: Clouds are noise sampled along a ray, lit by how much sun reaches each step. - What it is: Fluffy, sunlit clouds drift across the sky and change density and coverage in real time. - Technique: A raymarching shader samples 3D noise shaped by a weather map and height gradient (likely the Perlin-Worley approach from Horizon Zero Dawn) and estimates lighting at each step with a short march toward the sun; the code is the author's bachelor's thesis. - Try it: Shrink the cloud volume to a small box and place it above a real table with AR Foundation, lit by the estimated main light direction. Twist: make it rain particles onto the table when the user blows into the microphone. ### Jakob Kudsk Steensen *Artist working with ecology and game engines* Danish artist who builds ecological worlds from field recordings, scans and game engines; his Serpentine commission The Deep Listener brought them into Kensington Gardens as location-based AR. #### The Deep Listener — Jakob Kudsk Steensen (2019) - Video: https://www.youtube.com/watch?v=xAlbHU6GgnQ - Interaction: Location & City, Voice & Sound, Information & UI - Platform & tech: Phone, location-based AR, photogrammetry, spatial audio, Unity - Idea: Use AR in a park to 'hear' the lives of five urban species. - What it is: A location-based AR and sound walk through Kensington Gardens and Hyde Park in which five of London's species — plane trees, bats, parakeets, damselflies and reedbeds — appear as scanned, sonified creatures at their habitats. - Technique: GPS-triggered AR places photogrammetry-scanned, game-engine-rendered species at specific park locations, paired with spatialised field recordings. - Try it: Pick three plants or animals on campus, record the sounds around them, and use AR to place enlarged models on the spot. Twist: the sound appears only after you stand still and quiet for 10 seconds. ### James Landay — Stanford HCI *Professor of Computer Science at Stanford University; co-founder of Stanford HAI* James Landay studies user interfaces for education, health and sustainability. His group's Moon Story (CHI 2024, led by Alan Y. Cheng) combines mobile AR, narrative and LLMs to teach children astronomy and environmental science. #### Moon Story (Scientific and Fantastical) — James Landay — Stanford HCI (2024) - Video: https://www.youtube.com/watch?v=ZpKhhMcS8Jw - Interaction: Information & UI, Voice & Sound, Play - Platform & tech: Phone, mobile AR, LLM, interactive narrative - Idea: Blend a fantasy story with real science and let the characters answer the child's own questions. - What it is: A mobile AR story app for elementary schoolers that blends fantasy with astronomy and environmental science: story scenes appear in the child's own surroundings and LLM-driven characters let children talk with the story as they learn. - Technique: AR scenes are anchored in the child's environment on a phone, while an LLM drives the characters' dialogue within the narrative and the lesson content. - Try it: Write a five-scene AR story about the Moon for a nine-year-old, place each scene on a table with AR Foundation or Adobe Aero, and give the guide character an LLM system prompt that keeps answers scientifically accurate and under two sentences. Twist: ask the child where their family comes from and let the LLM retell one scene with a Moon myth from that culture. ### Jeff Koons *Artist* American artist known for balloon-animal sculptures and Puppy, a 12-metre West Highland terrier covered in living flowers outside the Guggenheim Bilbao. #### Puppy — Jeff Koons (1992) - Video: https://www.youtube.com/watch?v=DdUyxFRwz_g - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, stainless steel frame, living flowering plants, irrigation - Idea: A giant cute figure with a living, changing skin shows how an AR object's surface can follow seasons. - What it is: A 12-metre West Highland terrier covered in tens of thousands of living flowering plants guards the entrance of the Guggenheim Museum Bilbao; the flowers are replanted twice a year. - Technique: A steel armature holds soil and an internal irrigation system; flowers are planted in pockets over its surface. - Try it: Place a giant AR animal on a plaza whose surface is covered with flowers that change by month using a seasonal flower list. Twist: users can plant a flower on it and see it grow over the week. ### Jeremy Hight *Writer and locative-media pioneer* Jeremy Hight co-created 34 North 118 West (2002), one of the first GPS-triggered narratives, and wrote the essay on 'narrative archaeology' that framed locative storytelling. He later joined the Manifest.AR circle and staged AR interventions, including a mock battle with Mark Skwarek across Washington DC. #### Skwarek vs Hight: AR battle over Washington DC — Jeremy Hight, Mark Skwarek (2013) - Video: https://www.youtube.com/watch?v=j2NadLDHsss - Interaction: Location & City, Play, Shared & Social - Platform & tech: Phone, Layar, creatAR, GPS - Idea: Turn a capital city into the battlefield of a poetry war fought in AR. - What it is: For the Corcoran's Gallery 31 show, Jeremy Hight and Mark Skwarek staged an 'epic battle' in augmented reality: monsters, ships and poems attack the Capitol and other DC landmarks on visitors' phones, and the public can add its own augmentations. - Technique: Geolocated 3D models and texts are placed with the creatAR tool on a Layar layer, so they appear at fixed GPS positions and heights over the landmarks. - Try it: In pairs, place geolocated AR objects around campus with a WebAR location tool or Adobe Aero, one student defending a landmark and the other attacking it, then walk the 'battlefield' together. Twist: every object must be a line of poetry whose height above the ground carries meaning. ### Johanna Jaskowska *AR designer; creator of Beauty3000* Polish designer whose Instagram effect Beauty3000 (2018), a glossy chrome face mask, went viral and helped launch the AR filter-artist scene. #### Beauty3000 — Johanna Jaskowska (2018) - Video: https://www.youtube.com/watch?v=d789vqCo_-A - Interaction: Face, Perception & Effects - Platform & tech: Phone, Spark AR, Instagram - Idea: Turn your face into shiny, plastic-like future skin and ask what beauty means. - What it is: An Instagram face effect that coats the face in a shiny, iridescent plastic-like skin that catches light as you move; it went viral in 2018 and helped start the AR filter-artist movement. - Technique: The face mesh is re-textured with a glossy material that uses an iridescent or matcap environment reflection, so highlights shift as the head moves. - Try it: Give the face mesh a rainbow matcap material in Effect House or Lens Studio and watch the highlights shift as you turn your head. Twist: swap the material for the texture of a real object around you. ### John Edmark *Artist, inventor and design lecturer* American artist and Stanford design lecturer who 3D-prints sculptures built on the golden angle; spun under a strobe light, his Blooms appear to unfold endlessly like living flowers. #### Blooms — John Edmark (2015) - Video: https://www.youtube.com/watch?v=1cSR3FTQTyc - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, 3D printing, strobe light, golden angle - Idea: When the flash is synchronised to the geometry, a static object becomes animation; AR depends on the same sync between rendering and real motion. - What it is: 3D-printed sculptures spin under a strobe light and appear to bloom endlessly, petals flowing outward from the centre like a living flower. - Technique: Petals are placed at successive golden-angle (137.5°) steps; spinning so that one rotation of 137.5° happens between flashes makes each petal appear to move to its neighbour's position. - Try it: Generate a golden-angle bloom mesh in three.js WebXR and rotate it by exactly 137.5° per frame on a real table. Twist: let the viewer change the angle slightly and watch the bloom spiral in reverse. ### John Giorno *Poet and performance artist (1936-2019)* American poet who pioneered Dial-A-Poem and turned short lines of text into colourful paintings and performances. His [AR]T Walk piece Now at the Dawn of My Life (2019) placed a rainbow trail of his words over city streets. #### Now at the Dawn of My Life — John Giorno, Apple (ARKit / visionOS teams) (2019) - Video: https://www.youtube.com/watch?v=eZKMHjBrSQY - Interaction: Location & City, Information & UI - Platform & tech: Phone, ARKit, 3D typography - Idea: Poetry becomes a rainbow sculpture of words in the street. - What it is: On Apple's [AR]T Walk, bold rainbow-coloured words from Giorno's poems rise and float in the street, forming a journey of homespun wisdom against the city backdrop. - Technique: Location-anchored ARKit scene of extruded 3D typography animated in rainbow gradients, following the style of Giorno's text paintings. - Try it: Write a four-line poem and place each line as 3D text at a different spot on a walking route using Lens Studio or WebXR. Twist: each line only appears when the previous one has been read aloud. ### Jon Barron *Research scientist at Google DeepMind (NeRF, mip-NeRF)* Jon Barron is a Google DeepMind researcher known for neural radiance fields and 3D reconstruction. #### Vibe-coding an AR game inside Quest 3 with Gemini 3 — Jon Barron (2025) - Video: https://x.com/jon_barron/status/1990909319701758157 - Interaction: Voice & Sound, Play - Platform & tech: Headset, Web, Gemini 3, Meta Quest 3, WebXR, Gemini Canvas - Idea: The holodeck is real: describe a game and play it in your room a minute later. - What it is: Without leaving the headset, the author asks Gemini 3 by voice in the Quest browser to build a WebXR game, and plays the generated AR game in his room moments later. - Technique: Gemini's Canvas generates a single-page WebXR app from a spoken prompt, which runs immediately in the headset browser with passthrough. - Try it: In a headset or phone browser, ask an AI assistant for a one-file WebXR game in one sentence, play it, then improve it with three more spoken requests. Twist: the game must use the real floor as the main mechanic. ### Jon Foreman *Land artist* Welsh artist (Sculpt the World) who arranges beach stones by size and colour into spirals and circles before the tide returns. #### Sculpt the World — Jon Foreman (2024) - Video: https://www.youtube.com/watch?v=yGKes_A5nIM - Interaction: Location & City, Tangible Objects - Platform & tech: Desktop, beach stones, tide, aerial photography - Idea: Imposing a simple order (size, colour) on natural material reads immediately as intention, useful for AR layout and generative placement. - What it is: A short film of Foreman's beach works over several years: stones sorted by size and colour into spirals, circles and gradients on Welsh beaches, before the tide returns. - Technique: Stones are gathered and placed by hand following a sketched geometry, working within one low tide. - Try it: Scan 30 real stones and write a script that places them in AR in a spiral sorted from smallest to largest on any detected floor. Twist: the user can switch the sorting rule (size, colour, roughness) live. ### Jonathan Forder *AR/game developer (Discover Studios)* Game and AR developer who, at Discover Studios in 2017, built an ARKit prototype that drops a flat, cartoon-shaded character and set into a real room. #### AR Cartoon Prototype — Jonathan Forder (2017) - Video: https://www.youtube.com/watch?v=LYFk25OQJ_8 - Interaction: Perception & Effects, Play - Platform & tech: Phone, ARKit, Unity - Idea: AR does not have to look real: a cartoon style can sit convincingly in a real room. - What it is: An ARKit prototype that stands Gumball and Darwin from The Amazing World of Gumball in a real living room as flat, hand-drawn characters that keep their 2D cartoon look. Video: Made With ARKit. - Technique: Animated 2D character sprites are likely rendered as camera-facing billboards with drop shadows, anchored to the floor via ARKit plane detection. - Try it: Animate a 2D character you have drawn as a sprite sheet and stand it on a real desk in AR as a camera-facing card with a shadow. Twist: when viewers walk behind it, reveal a hand-drawn back side. ### Jonpasang (전파상) *Seoul media art collective* Seoul collective of media artists and engineers that builds large kinetic works, best known for Hyper-Matrix, the moving cube wall of the Hyundai Motor Group Pavilion at Expo 2012 Yeosu. #### Hyper-Matrix (Hyundai Motor Group Pavilion, Expo 2012 Yeosu) — Jonpasang (전파상) (2012) - Video: https://www.youtube.com/watch?v=nGz1vyCNe4I - Interaction: Perception & Effects, Projection - Platform & tech: Projection, motorized cubes, stepper motors, projection - Idea: A wall that physically moves is shocking because walls never move — AR can make solid architecture appear to breathe. - What it is: A giant wall made of thousands of white cubes pushed in and out by motors forms waves, faces and 3D patterns, with projected light enhancing the relief. - Technique: Each cube is likely mounted on a stepper-driven rod; a height-map animation is sent to the actuator grid and synchronized with projection. - Try it: Replace a real wall in AR with a grid of cubes that push out to form the silhouette of whoever stands in front of it. Twist: use depth from the LiDAR to make the relief follow the user's body. ### Juampi *Developer and maker* Juampi is a developer who hacks consumer AI glasses into small, useful apps. #### Ray-Ban Meta product scanner — Juampi (2026) - Video: https://x.com/juampitech/status/2068387708276527265 - Interaction: Tangible Objects, Information & UI, Voice & Sound - Platform & tech: Wearable, Ray-Ban Meta, Meta Wearables SDK, vision model - Idea: Look at any product and hear what it is and where it comes from. - What it is: A hacked Ray-Ban Meta app explains unfamiliar products the wearer looks at in a shop and says where to buy them. - Technique: The Meta Wearables SDK streams frames to a phone app that queries a vision model and a shopping search, then speaks the answer through the glasses. - Try it: Build a phone 'shelf explainer' that photographs a product and reads out a two-sentence explanation plus a cheaper alternative. Twist: explain the product as if to someone from 100 years ago. ### Julie C. Stamm *Dance and media artist working with sensing, feedback and neuroaesthetics* Germany-based dance and media artist with a background in contemporary dance and neuroaesthetics research (Goldsmiths) who designs live-camera performances, pressure sensors and participatory games that translate movement into data and back into experience. At the 2026 CCL at the Royal Ballet and Opera she made Horrorscope, a walking game based on sea-level-rise maps. #### The Placeholders — Julie C. Stamm (2025) - Video: https://vimeo.com/1118720620 - Interaction: Projection, Performance, Perception & Effects - Platform & tech: Projection, live camera, image dataset, projection - Idea: Make a stage where a camera, a dataset and dancers invent meaning together live. - What it is: Three performers: one dances, one films, one extracts still frames from the live feed; each frame is projected immediately next to an image from a curated dataset, so the audience watches random meaning form and collapse. - Technique: A camera feed is sampled frame by frame on a laptop and paired with images from a curated dataset on a dual-image projection. - Try it: Let one student film a dancer with a phone while another grabs a frame every ten seconds and projects it beside a random image from a class photo folder. Twist: the dancer must react to the pairing on screen. ### Julie Curtiss *Painter and sculptor* French-born, New York-based painter whose surreal images of hair, food and female archetypes read almost like sculpture. Lune (2021) was her first AR work, made with Acute Art. #### Lune — Julie Curtiss, Acute Art (2021) - Video: https://vimeo.com/812460459 - Interaction: Gaze & Attention, Perception & Effects - Platform & tech: Phone, Acute Art app, ARKit, ARCore - Idea: A figure that always turns away makes the viewer aware of their own gaze. - What it is: A nude woman from one of Curtiss's paintings stands in AR with her back turned; however you walk around her, she keeps turning away and avoids your gaze. - Technique: Phone AR where the figure's orientation is continuously driven by the camera position so that it always faces away from the viewer. - Try it: Place a 3D figure in AR that rotates to keep its back to the camera, and ask classmates to try to see its face. Twist: let it turn around only when two phones look at it from opposite sides. ### Jure Triglav *WebXR and open-science software developer* Slovenian developer who builds open-science tools and WebXR experiments, including Shaderworlds, which puts raymarched Shadertoy worlds into a VR headset through the browser. #### Shaderworlds — Jure Triglav (2021) - Video: https://www.youtube.com/watch?v=XoRwpPPUAbc - Source code: https://github.com/jure/shaderworlds - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Headset, Web, WebXR, three.js, GLSL, Shadertoy - Idea: A fragment shader is already a world; WebXR just lets you stand inside it. - What it is: Raymarched Shadertoy landscapes such as clouds and entangled vines surround the viewer in a VR headset, opened simply from a web page; the controllers move the sun and vines. - Technique: three.js renders a raymarching shader per eye, likely rebuilding each ray from the WebXR view and projection matrices so the SDF scene has correct stereo depth; controller poses are passed in as shader uniforms. - Try it: Port one Shadertoy SDF scene into the Shaderworlds template and switch the session to immersive-ar so it renders only inside a sphere in the room. Twist: map a hand controller to a parameter of the SDF. ### Karen X Cheng *Creative director and filmmaker; viral AR and AI video maker* Karen X Cheng is a filmmaker and creative director known for short viral videos that mix practical effects, AR filters and new AI tools. #### AR + AI effect: a generated scene that reacts to your hands — Karen X Cheng (2023) - Video: https://x.com/karenxcheng/status/1661474527673749504 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Phone, Adobe Photoshop Generative Fill, AR filter, hand tracking - Idea: Use generative fill to paint the world, then let your hands play inside it. - What it is: An AR filter whose imagery was extended with Photoshop's generative AI reacts to the viewer's hands, so the AI-painted scene moves and opens around them. - Technique: The background art was inpainted with Photoshop's generative fill and layered into an interactive AR filter driven by hand tracking (built with collaborator Sergey Galkin, likely in Adobe's AR tools). - Try it: Extend a photo of your classroom with any generative-fill tool into a fantasy scene, cut it into three depth layers and make them parallax with hand position in a web AR filter. Twist: each hand reveals a different season. ### Karthik Ramani — Convergence Design Lab (Purdue) *Professor of Mechanical Engineering at Purdue University* Karthik Ramani's Convergence Design Lab has produced many AR authoring, tangible and human-robot systems. Recent work such as agentAR, led by Chenfei Zhu, lets an LLM agent assemble complete AR applications from a natural-language description. #### agentAR — Karthik Ramani — Convergence Design Lab (Purdue) (2025) - Video: https://www.youtube.com/watch?v=vVUakvZij0Q - Interaction: Voice & Sound, Spatial Mapping, Information & UI - Platform & tech: Headset, LLM agent, function calling, Unity, mixed reality headset - Idea: Let an AI agent assemble AR apps from research-proven building blocks while you only describe what you want. - What it is: An in-situ AR authoring system: describe an AR app in plain language and an LLM agent builds it on the spot, choosing tracking, interface and logic tools from a library distilled from AR research. - Technique: A tool-augmented LLM agent plans against a structured AR application template and calls functions from a tool library (object detection, anchoring, UI, triggers) to generate and run the app in the headset. - Try it: Define five Unity or Lens Studio functions (spawnObject, attachLabel, onObjectSeen, playSound, showTimer), expose them to an LLM through function calling, type 'remind me to drink when I look at my cup' and watch the agent wire them together. Twist: make the agent ask one clarifying question before it builds anything. ### Keiken (Hana Omori, Isabel Ramos, Tanya Cruz) *Collaborative art practice working with AR filters, game worlds and performance* Cross-dimensional collective that builds online worlds, Instagram AR face filters, game-engine environments and performances about future bodies and empathy, including the ongoing Morphogenic Angels world shown at Serpentine and at Hellerau's Blackbox Whitecube XR symposium. #### Augmented Empathy — Keiken (Hana Omori, Isabel Ramos, Tanya Cruz) (2020) - Video: https://www.youtube.com/watch?v=LYNKEGqHS3o - Interaction: Face, Perception & Effects - Platform & tech: Phone, Spark AR, Instagram filters - Idea: A face filter as a way to rehearse empathy with other beings. - What it is: An exhibition at FACT Liverpool built around Instagram AR face filters and installations that let visitors wear speculative future bodies and try on the feelings of other species. - Technique: Face-tracked effects built in Spark AR are published as Instagram filters and paired with gallery installations, so the work continues on visitors' own phones. - Try it: Design a face filter that shows how a non-human (a tree, a bat, a river) might 'feel' today, using Lens Studio or Effect House. Twist: the filter changes only when two faces are in frame together. ### Kianí del Valle *Choreographer and director of KDV Performance Group* Puerto Rican choreographer Kianí del Valle leads KDV Performance Group and works as a movement director for musicians and visual artists. With the Barcelona studio Hamill Industries and AI researchers from UPC she created Engendered Otherness, a dance ensemble in which computer vision turns her movement into a chorus of generated creatures, followed by the stage work CORTEX. #### Engendered Otherness (Ai Transmutations) — Kianí del Valle (2021) - Video: https://www.youtube.com/watch?v=hXiDYN1IFfY - Interaction: Hands & Body, Performance, Perception & Effects - Platform & tech: Projection, computer vision, generative AI - Idea: A live dancer leads a chorus of AI-generated creatures that only exist while she moves. - What it is: Kianí del Valle dances with Hamill Industries while computer vision reads her body and an AI generates a visual ensemble of not-yet-existing hybrid organisms that move with her choreography, to a score by Floating Points. - Technique: Pose estimation on live camera footage feeds generative image models trained by researchers from UPC and ISTHMO, which synthesise creature forms following the detected body in real time. - Try it: Feed MediaPipe Pose keypoints from a webcam into a pose-conditioned image generator (or a hand-made set of creature sprites mapped to limbs) and project the result behind a dancer. Twist: add a second creature that copies the dancer with a two-second delay. ### Kluge Interactive (Synth Riders) *Immersive game studio; developer of Synth Riders* Studio founded in 2008 that makes the VR rhythm game Synth Riders, which added a free passthrough mixed-reality mode for the Meta Quest 3 launch. #### Synth Riders Mixed Reality mode — Kluge Interactive (Synth Riders) (2023) - Video: https://www.youtube.com/watch?v=YzjiSRi28wU - Interaction: Voice & Sound, Hands & Body, Portals & Worlds - Platform & tech: Headset, Meta Quest 3, passthrough, Unity - Idea: Turn your living room into the dance floor by letting the level pour in through a portal. - What it is: The rhythm game's notes, rails and obstacles fly out of a portal into the player's passthrough living room, so dancing along the rails happens among their own furniture. - Technique: Quest passthrough replaces the virtual stage, with a stencil-masked portal where notes spawn and colour-tinted effects blended over the camera feed. - Try it: Make a passthrough rhythm prototype where beat-synced orbs emerge from a portal on a real wall and must be touched with the hands on the beat. Twist: let the song's intensity grow the portal until the virtual world covers the whole wall. ### Koo Jeong A *Installation and drawing artist* Korean artist known for quiet, site-specific works about everyday objects, scents and near-invisible gestures. Her AR ice cube density (2019) was Frieze's first augmented reality artwork, followed by the Prerequisites 7 drawings and the AR architecture OLO. #### density — Koo Jeong A, Acute Art (2019) - Video: https://www.youtube.com/watch?v=PUobAocKdXE - Interaction: Location & City, Perception & Effects, Spatial Mapping - Platform & tech: Phone, Acute Art app, environment mapping, ARKit - Idea: An invisible melting ice cube stands among real sculptures and mirrors the park. - What it is: A translucent, slowly dripping ice cube floats in mid-air among the bronze sculptures of Frieze Sculpture in Regent's Park, visible only on a phone screen and reflecting whatever surrounds it. - Technique: Geolocated phone AR with environment mapping: the camera feed is used as a reflection and refraction probe so the virtual ice picks up the real surroundings. - Try it: Place a glass-like object in AR using environment texturing (AR Foundation or Reality Composer) and photograph it in three very different locations. Twist: make it melt faster the warmer the colours around it. ### Kumi Yamashita *Artist; light and shadow* Japanese artist who places carved wood, folded paper or cut shapes under a single light so that their shadows become faces and figures, and who also 'draws' portraits with one thread wound around nails. #### Shadow works — Kumi Yamashita (2010) - Video: https://www.youtube.com/watch?v=65BRB3kuvv8 - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, single light source, carved wood, paper - Idea: The shadow, not the object, is the content: in AR a virtual light can make a real object cast an unexpected virtual shadow, a quiet way to add story to everyday things. - What it is: A single lamp hits a few carved wooden pieces, folded paper or cut numbers on a wall, and their shadows complete detailed faces and figures that the objects themselves do not show. - Technique: Relief shapes are carved or folded so that a light at a fixed angle casts a calculated silhouette; the lamp position is part of the work. - Try it: Detect a real object on a desk with image or object tracking and render a virtual shadow behind it that is a portrait, not its true outline. Twist: move a virtual lamp with your hand and let the portrait age as the shadow lengthens. ### Kunabi Brother *Indie studio; Euclidean Lands* Small Viennese studio (Miro Straka) behind the rotating-cube puzzle game Euclidean Lands and one of the first ARKit ports. #### Euclidean Lands AR — Kunabi Brother (2017) - Video: https://www.youtube.com/watch?v=f6vA1s97_-0 - Interaction: Play, Spatial Mapping - Platform & tech: Phone, ARKit, Unity - Idea: A puzzle world that rotates like a Rubik's cube sits on your table, and you walk around it to think. - What it is: A turn-based puzzle game on a rotating cube world, adapted for ARKit so the Rubik's-cube-like diorama sits on the player's table and can be walked around. - Technique: A rotating cube-world puzzle is placed on a detected table, and its grid-based rotation logic is unchanged while the camera view comes from the player walking around it. - Try it: Build a small AR cube world that can rotate as a whole, so that after each rotation the character can walk onto a new path. Twist: the only way to rotate it is by walking around the table. ### Kurt Perschke *Artist* American artist behind the RedBall Project, a 4.6-metre red inflatable ball squeezed into alleys, arches and doorways of cities around the world since 2001. #### RedBall Project — Kurt Perschke (2001) - Video: https://www.youtube.com/watch?v=kNSpgFoUR6E - Interaction: Location & City, Spatial Mapping, Play - Platform & tech: Desktop, 4.6 m inflatable ball, urban sites - Idea: One simple object that deforms to each site's geometry is the ideal spatial-mapping demo, and people come back daily to see where it goes next. - What it is: A 4.6-metre red inflatable ball is squeezed into alleys, archways, bridges and doorways, a new site each day, in cities from Barcelona to Sydney; Danny Cooke's film follows its UK tour. - Technique: The ball is inflated in place so its soft skin presses against the surrounding architecture. - Try it: Use ARKit or AR Foundation mesh collision to squeeze a soft red AR ball into real doorways, gaps and corners so it deforms against real surfaces. Twist: publish a new location every day as a clue. ### Lachlan Turczan *Artist working with light, water and sound* American artist who has spent a decade exploring the optical and sonic properties of water and light; his Lucida light sculptures premiered at Milan Design Week 2025. #### Lucida — Lachlan Turczan (2025) - Video: https://www.youtube.com/watch?v=Rne2JgYM0s8 - Interaction: Hands & Body, Perception & Effects - Platform & tech: Projection, light sculpture, water optics, sound - Idea: Make light feel like a material you can touch and shape. - What it is: Rooms sculpted from curtains of light that ripple, bend and seem to solidify into planes as visitors reach into them. - Technique: Likely projected light passed through water optics and haze, with sound and movement modulating the ripples, so the beams read as physical sheets. - Try it: In a hazy dark room, shine a projector through a shallow tray of water and film how hand movements and a speaker under the tray change the light sheet. Twist: map a hand-tracking signal to the speaker so gestures 'play' the light. ### Laura Schütz *Researcher in sonification and multisensory AR at the Technical University of Munich* Laura Schütz works with Nassir Navab's group at TU Munich on sonification and multisensory augmented reality. Sonify Anything recognizes real materials with computer vision so that virtual objects colliding with real ones sound believable. #### Sonify Anything — Laura Schütz (2025) - Video: https://www.youtube.com/watch?v=tN6JE3kNUbk - Interaction: Voice & Sound, Spatial Mapping, Tangible Objects - Platform & tech: Headset, material segmentation, modal sound synthesis, AR headset - Idea: Let the materials of the real world decide how virtual collisions sound. - What it is: When a virtual object hits a real one in AR, the system recognizes the real material (glass, wood, metal) and synthesizes a physically plausible impact sound on the fly instead of a generic click. - Technique: Computer-vision models segment and classify the materials of real objects, and their physical properties plus the impact dynamics drive real-time modal sound synthesis. - Try it: Throw virtual balls at real furniture in a phone AR app; on each hit, classify the surface at the impact point with a material-recognition model or a vision-LLM prompt and play a synthesized sound (Tone.js or a physical-model patch) tuned to that material. Twist: exaggerate one material so the whole room sounds as if it were made of glass. ### Lita Albuquerque *Artist* American artist who places pigment, stones and spheres in deserts and ice fields to map the stars onto the ground, including Stellar Axis in Antarctica. #### Stellar Axis: Antarctica — Lita Albuquerque (2006) - Video: https://www.youtube.com/watch?v=gvxU7GSXiqI - Interaction: Location & City, Information & UI, Perception & Effects - Platform & tech: Desktop, 99 blue spheres, star map, ice - Idea: Mapping the sky onto the ground turns astronomy into a walkable place, a clear template for AR star maps anchored to terrain. - What it is: Near the South Pole, Albuquerque led an expedition that placed 99 blue spheres on the ice, each at the position of a star in the sky above, making the first large ephemeral artwork on the continent. - Technique: Fibreglass spheres of different sizes (by star brightness) were placed on the ice by GPS according to a projected star chart. - Try it: Project tonight's star chart onto a field in AR, placing a sphere on the ground under each bright star, sized by magnitude. Twist: when the user looks up, draw a line from each sphere to its real star. ### Local Projects *Experience design studio for museums and public spaces (founded by Jake Barton)* New York studio behind media installations for the 9/11 Memorial Museum, Cooper Hewitt and the Cleveland Museum of Art's Gallery One, many built with openFrameworks. #### Strike A Pose (Gallery One) — Local Projects (2013) - Video: https://vimeo.com/60866008 - Interaction: Hands & Body, Play, Information & UI - Platform & tech: Desktop, Kinect, openFrameworks - Idea: Understand a sculpture by imitating it with your own body. - What it is: In the Cleveland Museum of Art's Gallery One, visitors copy the pose of a sculpture from the collection and a screen scores how closely their body matches it. - Technique: A Kinect skeleton is compared joint by joint with a stored pose of each sculpture, and the similarity percentage is shown next to the artwork. - Try it: Use webcam pose estimation to score how well students match the pose of three artworks or photos. Twist: two people must together match a sculpture of two figures. ### Louise Bourgeois *Sculptor* French-American artist (1911–2010) whose giant steel spider Maman, over nine metres tall, stands in plazas in Ottawa, Bilbao, Tokyo and elsewhere. #### Maman — Louise Bourgeois (1999) - Video: https://www.youtube.com/watch?v=UTsFuZ80OyY - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, bronze, stainless steel, marble eggs - Idea: A giant creature you can walk under creates awe and a feeling of being protected, a scale relationship AR creatures can reproduce. - What it is: A spider more than nine metres tall, carrying a sac of marble eggs, stands on long thin legs over plazas in Bilbao, Ottawa, Tokyo and Seoul; visitors walk underneath it. - Technique: Cast bronze and steel legs support a body with a mesh sac holding marble eggs. - Try it: Place a giant AR creature with long legs over a courtyard so people can walk beneath it, and add soft footsteps that respond to their movement. Twist: the creature slowly steps to stay over the largest group of people. ### Loóna (Sergey Gonchar, Andrew Yanchurevich, Dmitry Doryn, Eugene Nevgen) *App studio from the MSQRD founders; makers of Loóna* Founded by the team behind the MSQRD face-filter app, Loóna began as a calming bedtime app and was rebuilt for Apple Vision Pro as a spatial puzzle game with an AI creature. #### Loóna: Cozy Puzzle Games (Apple Vision Pro) — Loóna (Sergey Gonchar, Andrew Yanchurevich, Dmitry Doryn, Eugene Nevgen) (2024) - Video: https://www.youtube.com/watch?v=SqEGR3XDSFE - Interaction: Hands & Body, Play, Voice & Sound - Platform & tech: Headset, Apple Vision Pro, visionOS, RealityKit - Idea: Puzzle solving as a calm ritual that rewards you by animating the finished world. - What it is: Floating 3D dioramas hover in the player's room; pinching and placing their pieces with the hands completes each little world, which then comes alive with sound and a friendly AI creature called Lumo. - Technique: visionOS gaze-and-pinch plus direct hand manipulation snap pieces into RealityKit scenes, with spatial audio layered as each part locks in. - Try it: Build a five-piece 3D jigsaw that floats at chest height; each correctly placed piece adds one audio loop and one animation. Twist: hide one piece somewhere in the real room so the player must look around to find it. ### Ludwig Sidenmark & Haoyu Wang *HCI researcher known for gaze interaction in XR (University of Toronto); Haoyu Wang leads the generative haptic-prop work* Ludwig Sidenmark, who did his PhD on eye-head interaction at Lancaster University, works on gaze and mixed-reality interaction at the University of Toronto. With Haoyu Wang and colleagues he built Prop-Chromeleon, which uses generative AI to turn everyday objects into adaptive haptic props. #### Prop-Chromeleon — Ludwig Sidenmark & Haoyu Wang (2026) - Video: https://www.youtube.com/watch?v=IKRM-_YxGpM - Interaction: Tangible Objects, Hands & Body, Portals & Worlds - Platform & tech: Headset, generative AI, text-to-3D, object tracking, mixed reality headset - Idea: Reskin the object in your hand with a prompt, using its real shape as the constraint. - What it is: A mixed-reality system that turns any everyday object into a haptic prop: type 'a magic lantern' while holding a water bottle and a virtual lantern shaped to fit the bottle is generated and overlaid on it, so what you see matches what you feel. - Technique: An AI pipeline captures the prop's geometry, generates a 3D asset from the text prompt under shape constraints, and anchors it to the tracked physical object in the headset. - Try it: Pick three props with different shapes (a cylinder, a box, a ball), generate three virtual skins for each with a text-to-3D or text-to-texture tool, and attach them to the tracked objects with image or object tracking in Lens Studio or AR Foundation. Twist: let a squeeze or a shake trigger a new prompt, so the object transforms in your hand. ### Luxloop *New York creative technology studio* New York studio that builds interactive installations and location-based audio experiences; its projection piece If The Walls Had Eyes was shown at Refest 2014. #### If The Walls Had Eyes — Luxloop (2014) - Video: https://vimeo.com/113986453 - Interaction: Gaze & Attention, Projection - Platform & tech: Projection, computer vision, projection - Idea: The digital eyes that watch us online become real eyes on the wall. - What it is: A wall of projected video eyes turns to watch each visitor who walks past, following them along the gallery. - Technique: A camera tracks passers-by and the software likely picks, for each projected eye, the video frame whose gaze direction points at the tracked person. - Try it: Film an eye looking in nine directions and build a webcam piece that shows the frame looking at wherever the viewer stands. Twist: the eyes blink when the viewer takes out their phone. ### MAOTIK (Mathieu Le Sourd) *New media artist; immersive and interactive installations* French digital artist based in Montreal who designs immersive environments and interactive floors where generative visuals and sound respond to how people move through the space. #### INNER LIFE — MAOTIK (Mathieu Le Sourd) (2022) - Video: https://vimeo.com/1064295716 - Interaction: Hands & Body, Projection, Voice & Sound - Platform & tech: Projection, TouchDesigner, laser tracking, projection - Idea: Walking becomes painting and composing on a floor that behaves like a living organism. - What it is: An immersive interactive floor where visitors' footsteps generate watercolour-like, nature-inspired visuals and a changing sound composition; every two minutes the system switches to a new perspective of scale. - Technique: Laser (LiDAR) touch tracking of feet feeds a TouchDesigner parametric audiovisual system projected on the floor. - Try it: Point a projector at the floor and use a webcam from above to find people's positions; spawn ink blobs that spread where they stand and play a note per blob. Twist: switch the 'zoom level' every two minutes so the same steps create cells, then galaxies. ### MSQRD (Masquerade Technologies) *Face-filter app studio (2015–2016), acquired by Facebook* Masquerade Technologies, founded by Eugene Nevgen, Sergey Gonchar and Andrew Yanchurevich, launched the MSQRD selfie app in late 2015; its live face masks and face swaps went viral, and Facebook bought the company in March 2016 and folded the technology into its camera effects. The founders later started Loóna. #### MSQRD — MSQRD (Masquerade Technologies) (2016) - Video: https://www.youtube.com/watch?v=MCZotkdo3wI - Interaction: Face - Platform & tech: Phone, face tracking, iOS, Android - Idea: Wear a living mask on your face through the phone camera. - What it is: A selfie video app that tracks the face live and turns it into a crying clown, an old man, a chimpanzee or a white dog, with masks that follow expressions and open mouths, recorded as short clips to share. - Technique: Real-time facial landmark tracking on the phone fits a face mesh every frame, onto which animated masks and face-swap textures are warped. - Try it: In Lens Studio or Effect House, make a face mask that switches character when the user opens their mouth or raises their eyebrows. Twist: the mask ages a little every time the user blinks. ### MalyaWka (Pavel) *Technical artist; shader developer* A technical artist who shares Unity URP rendering features, most notably URP-ScreenSpaceCavity, a port of Blender's viewport cavity effect that darkens creases and brightens edges in screen space. #### URP Screen Space Cavity — MalyaWka (Pavel) (2021) - Video: https://x.com/_malyawka_/status/1471816770604441603 - Source code: https://github.com/malyawka/URP-ScreenSpaceCavity - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, HLSL - Idea: Emphasise geometry by drawing its curvature, the way an illustrator inks edges. - What it is: Low-poly scenes gain crisp white ridges and dark creases, the Blender viewport 'cavity' look, applied to the whole camera image in Unity. - Technique: A URP renderer feature reads screen-space normals and depth, estimates curvature (ridges and valleys) and a cavity term by sampling neighbours, and composites light and dark accents into the lit image. - Try it: Add a cavity pass to virtual objects in an AR scene so their edges stay readable against the busy camera background. Twist: apply it to the LiDAR room mesh too and draw the real room like a sketch. ### Marc Wakefield *AR effects artist; co-founder of Augmented Reality Design Studio* Illustrator turned AR artist whose Spark AR world effects and character filters have been used by millions, including work for U2; he runs Augmented Reality Design Studio with Susie Wakefield. #### Spark AR world effects — Marc Wakefield (2019) - Video: https://www.youtube.com/watch?v=KpEGpggfPXw - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, Spark AR, Instagram - Idea: Turn hand-drawn characters and fantasy worlds into Instagram AR effects you can drop into reality. - What it is: Marc Wakefield turns his drawn and painted characters and worlds into Instagram world effects that place them into the user's surroundings, including effects for U2. - Technique: Hand-drawn 2D artwork is placed as billboards or layered planes in world space via plane tracking, preserving the illustration style in 3D. - Try it: Split your own hand-drawn character into front, middle and back layers and place them in AR as a paper theater with parallax. Twist: as you walk around it, the character always turns to face you. ### Marco Brambilla *Video installation artist and director* Italian-Canadian artist known for dense video collages built from found footage and early use of 3D in video art. With Acute Art he made the VR work Creation (Megaplex) and The Four Temperaments AR (2020) starring Cate Blanchett. #### The Four Temperaments AR — Marco Brambilla, Acute Art (2020) - Video: https://www.youtube.com/watch?v=_xG8mEQ8Mk4 - Interaction: Performance, Voice & Sound, Gaze & Attention - Platform & tech: Phone, Acute Art app, video texture, spatial audio - Idea: The same line spoken by four versions of one actor changes meaning as you move among them. - What it is: Four floating crystal balls each hold Cate Blanchett playing one of Galen's temperaments; each looks at the viewer and says 'I love you', and the meaning shifts from tender to threatening as you walk between them. - Technique: Video of the actor mapped inside transparent spheres placed in phone AR, each billboarded to face the viewer, with spatial audio for each sphere. - Try it: Film a classmate saying one sentence in four different moods and place the four clips in AR as floating video spheres around the room. Twist: only the sphere you are closest to is audible. ### Marek Šimoník (Record3D) *Developer of the Record3D app* Independent developer of Record3D, an iPhone/iPad app that records and streams TrueDepth and LiDAR depth video, with open Unity demos that feed the stream into VFX Graph. #### Record3D → Unity VFX Graph Point Cloud Streaming — Marek Šimoník (Record3D) (2019) - Video: https://x.com/record3d/status/1175234894529798145 - Source code: https://github.com/marek-simonik/record3d_unity_demo - Interaction: Hands & Body, Spatial Mapping - Platform & tech: Phone, Desktop, Unity, VFX Graph, Record3D, TrueDepth - Idea: The depth camera already in your pocket can replace a Kinect for live volumetric effects. - What it is: A person filmed by an iPhone X's front depth camera appears in Unity as a live, Kinect-like point cloud, rendered and animated by a VFX Graph. - Technique: The Record3D app streams RGB and TrueDepth frames over USB; a native plugin in the Unity demo writes them into position and color textures that a VFX Graph samples as a point cache (the graph is one of Keijiro Takahashi's). - Try it: Stream yourself from an iPhone into the demo and restyle the point cloud with your own VFX Graph (sparks, dust, ribbons). Twist: bring it back into AR by recording with Record3D, then playing the capture as a floating hologram on a table with AR Foundation. ### Marina Abramović *Performance artist* The 'grandmother of performance art', who worked with Acute Art (Rising, VR) and Tin Drum (The Life, mixed reality) to extend presence into virtual bodies. #### The Life — Marina Abramović (2019) - Video: https://www.youtube.com/watch?v=VeajXYdTEiE - Interaction: Performance, Spatial Mapping - Platform & tech: Headset, volumetric capture, Magic Leap - Idea: Through an MR headset, the artist's holographic double performs in front of you. - What it is: The first mixed-reality performance artwork: a 19-minute volumetric capture of Abramović appears in the room through an MR headset, walking within a marked circle. - Technique: Volumetric capture of the performer (multi-camera studio) is played back as a 3D video mesh in a Magic Leap headset, anchored inside a physical circle so viewers walk around her. - Try it: Capture a classmate's one-minute silent performance with a phone LiDAR scan or multi-angle video, and place it in AR inside a circle drawn on the floor for people to walk around. Twist: when a viewer steps into the circle, the performer stops and 'looks' at them. ### Marinus Boezem *Conceptual artist* Dutch conceptual artist whose Green Cathedral in Flevoland (1987–1996) is 178 poplars planted in the exact floor plan of Reims Cathedral. #### De Groene Kathedraal (The Green Cathedral) — Marinus Boezem (1996) - Video: https://www.youtube.com/watch?v=yye2qZ-o0lk - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, 178 Lombardy poplars, floor plan of Reims Cathedral - Idea: An invisible building drawn with trees is exactly an architectural overlay on land, and its 'negative' twin shows the same plan by subtraction. - What it is: In Almere, Flevoland, 178 Lombardy poplars are planted along the ground plan of Reims Cathedral, so the trees form the columns and nave of a living church; beside it, a clearing in the forest shows the same plan as a void. - Technique: Trees were planted at the positions of the cathedral's columns; a nearby grove was cut to leave the same plan open. - Try it: Take the floor plan of a famous building and place its columns as AR trees at 1:1 scale on a field. Twist: toggle between 'trees as columns' and 'empty plan cut from a forest'. ### Mark Jenkins *Street artist and sculptor* American artist who casts life-size figures in packing tape, dresses them and places them in streets in odd poses, provoking passers-by. #### Street installations — Mark Jenkins (2014) - Video: https://www.youtube.com/watch?v=XeT11IfxIyM - Interaction: Location & City, Hands & Body, Perception & Effects - Platform & tech: Desktop, packing tape casts, clothes, urban placement - Idea: Realistic bodies in impossible poses test people's reactions, which is the social experiment behind every public AR character. - What it is: Life-size figures cast in packing tape and dressed in real clothes are placed in odd situations in the street (head in a wall, lying in a fountain); the video shows his piece for Nuit Blanche Paris 2014. - Technique: Tape is wrapped around a model's body (sticky side out, then over), cut off and reassembled, then dressed and placed. - Try it: Place a photoreal AR person in an absurd pose (half inside a wall, face down in a planter) and record passers-by through the phone. Twist: the figure changes pose when nobody is looking. ### Masahiko Sato + EUPHRATES *Creative group making educational media and experimental films* Group founded by Masahiko Sato (creator of Pythagora Switch) that turns ways of thinking into films, exhibitions and TV; Ballet Rotoscope (2011) traces a ballerina's movement into geometric curves. #### Ballet Rotoscope — Masahiko Sato + EUPHRATES (2011) - Video: https://www.youtube.com/watch?v=yzJk6ww3LD0 - Interaction: Performance, Information & UI, Perception & Effects - Platform & tech: Desktop, rotoscoping, motion analysis, computational geometry - Idea: Reveal the hidden geometry of a dance by drawing its trajectories on top of the body. - What it is: A short film in which a ballerina's movements are precisely traced and overlaid with glowing curves, polygons and mathematical figures that reveal the geometry of each pose. - Technique: Frame-by-frame rotoscoping extracts joint positions from filmed ballet, from which trajectory curves and computed geometric animations are generated and composited. - Try it: Film a classmate doing a slow movement, run a pose estimator on the video, and draw the trails of wrists and ankles plus the triangle between hands and head. Twist: do it live in AR so the dancer sees the geometry while moving. ### MatrixRex *Unity developer and 3D artist* Frontend and Unity developer from Dhaka who builds editor tools and stylised shaders, and released the highly customisable Uber Stylized Water shader for Unity 6 URP. #### Uber Stylized Water — MatrixRex (2024) - Video: https://www.youtube.com/watch?v=NtMFK1EBNDU - Source code: https://github.com/MatrixRex/Uber-Stylized-Water - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, URP, Shader Graph - Idea: Depth-based colour and foam make any shoreline read as water. - What it is: A stylised cartoon water surface with depth colour, foam along shorelines and animated waves, shown in a short sneak peek before release. - Technique: The URP water shader compares scene depth with the water surface depth to blend shallow and deep colours and draw intersection foam, likely combined with vertex waves and scrolling normal maps. - Try it: Place the water plane slightly above a real floor with AR Foundation and use LiDAR depth so foam appears where the water meets real feet and furniture. Twist: raise the water level slowly to flood the room. ### Michael Wesely *Photographer* German photographer who builds pinhole-like cameras with extremely long exposures, recording construction sites such as Potsdamer Platz in Berlin and the Museum of Modern Art in New York over two to three years in a single image. #### Open Shutter: Long-Exposure Construction Series — Michael Wesely (1997) - Video: https://www.youtube.com/watch?v=uGuSf_FuYXM - Interaction: Perception & Effects, Location & City, Portals & Worlds - Platform & tech: Desktop, pinhole camera, long exposure, ND filters - Idea: A site's history can be shown as translucent layers in one view; AR 'time layers' over buildings can borrow this ghosted look. - What it is: Cameras with tiny apertures stay open for up to three years over building sites such as Potsdamer Platz in Berlin, MoMA in New York and IMS Paulista in São Paulo, so cranes and new towers appear as translucent layers. - Technique: Custom cameras with very small apertures and heavy filtering expose one sheet of film for months or years (series from 1997 onward). - Try it: Collect 10 photos of one building from different years (archives, Street View history), align them in AR over the real facade, and blend them with a slider. Twist: let each layer appear only when the viewer stands where that photo was taken. ### Michal Skalsky *Graphics programmer* Graphics programmer who released an open-source volumetric lighting system for Unity in 2016, raymarching light shafts for point, spot and directional lights. #### Volumetric Lights for Unity — Michal Skalsky (2016) - Video: https://www.youtube.com/watch?v=JPxLCYXB-8A - Source code: https://github.com/SlightlyMad/VolumetricLights - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, HLSL - Idea: Make the air visible by marching rays through the light. - What it is: Point lights and spotlights fill a hazy scene with visible light shafts and shadowed beams that react to occluders. - Technique: For each pixel the shader raymarches through the light volume at half resolution, samples the shadow map and a 3D noise at each step to accumulate in-scattering with a phase function, then upsamples bilaterally. - Try it: Add a volumetric spotlight to an AR scene and anchor it to a real lamp detected with AR Foundation, so virtual dust glows in the beam. Twist: let the phone's flashlight direction steer the virtual beam. ### Mighty Coconut *Game studio; 57° North (Merge Cube), Walkabout Mini Golf* Indie studio that made 57° North, a choose-your-own-adventure story played on the handheld Merge Cube, before its VR hit Walkabout Mini Golf. #### 57° North — Mighty Coconut, Merge Labs (Merge Cube) (2017) - Video: https://www.youtube.com/watch?v=EvedPF7S3BU - Interaction: Tangible Objects, Play - Platform & tech: Phone, Merge Cube, Unity - Idea: A branching story packed into a tiny cube theater that sits in your palm. - What it is: A choose-your-own-adventure story played on the Merge Cube: each chapter appears as a small diorama in the palm that the player turns and chooses within. - Technique: The Merge Cube's patterned faces are tracked as a multi-face image target, so a diorama follows the cube's rotation and choices are made by turning it. - Try it: Make your own Merge Cube by putting image markers on all six faces of a paper box, show a small story scene on it, and let each face you turn to trigger a different choice. Twist: shaking the box makes it rain in the scene. ### Naho Matsuda *Visual artist working with language, data and public space* German-Japanese visual artist whose site-specific works examine language, communication and everyday power structures. EVERY THING EVERY TIME, produced with FutureEverything, turns live data from a city's sensors and systems into anonymous poetry on a street-side split-flap display; it has run in Newcastle, at SXSW and in Ashford for ALT*ASH. #### EVERY THING EVERY TIME — Naho Matsuda, AΦE / A+E Lab (Aoi Nakamura & Esteban Lecoq) (2018) - Video: https://www.youtube.com/watch?v=6bGqpGS-H88 - Interaction: Information & UI, Location & City - Platform & tech: Desktop, open data, split-flap display, generative text - Idea: Turn a city's live sensor data into poetry, shown on the street where it happens. - What it is: A mechanical split-flap display on a city street clatters through short, anonymous poems generated from live city data, such as traffic lights changing, car parks filling or trains arriving. - Technique: Software pulls real-time open-data feeds from the city, strips identifying details and rewrites events as poetic lines, which are sent to a custom split-flap display (software by Paul Angus, development by Dan Hett). - Try it: Pick three live data feeds for your campus or city (weather, buses, Wi-Fi users), write rules that turn each change into a one-line poem, and show them on a phone WebAR sign at the spot. Twist: the poem may never mention a number. ### Naoto Hieda *Live-coding artist and creative technologist* Japanese artist known for glitchy live-coded visuals in Hydra and long-running improvised sessions with dancer Jorge Guevara, where the webcam image of the dancer is fed through code in real time. He was a participant at the 2nd Choreographic Coding Lab at A+E Lab in 2025, where he also filmed interviews with fellow participants. #### Best Practices in Contemporary Dance — Naoto Hieda (2021) - Video: https://www.youtube.com/watch?v=PYbN2FJ3BpU - Interaction: Hands & Body, Perception & Effects, Performance - Platform & tech: Web, Projection, Hydra, live coding, webcam - Idea: A dancer and a live coder improvise a duet through one webcam image. - What it is: An ongoing improvised series with dancer Jorge Guevara: as he dances, Naoto live-codes glitch visuals in Hydra that sample and distort the webcam image of the dancer, and both react to each other. - Technique: Hydra, a browser-based video synth, takes the webcam as a source and chains feedback, displacement and colour functions typed live by the coder. - Try it: Open Hydra in a browser, use the webcam as source and take turns: one student dances for thirty seconds, the other changes one line of code. Twist: the dancer calls out words the coder must translate into effects. ### Nick Hardeman *Creative technologist; interactive installation developer* Creative coder and openFrameworks contributor who has worked with Design I/O on Living Library, Studio Play and Elements, and makes body- and voice-driven interactive installations of his own. #### GLADES — Nick Hardeman (2019) - Video: https://vimeo.com/323247011 - Interaction: Hands & Body, Projection, Information & UI - Platform & tech: Projection, depth sensors, openFrameworks, body tracking - Idea: Your own silhouette becomes the source of ecological harm, so the message is felt, not read. - What it is: Visitors move in front of two displays: on one their body grows toxic blue algae on a virtual water surface, on the other an invasive python wraps around their silhouette, while a large projection shows an anhinga flying between them. - Technique: Depth sensors track visitors' bodies without wearables, and openFrameworks (likely) simulates algae growth and a python rig that follows the silhouette. - Try it: Use webcam body segmentation to make your silhouette leave a spreading 'pollution' trail on a projected pond, and a clean-up creature that follows others. Twist: the pond only recovers when everyone stands still. ### Nicola Buttari (PROFORMA Videodesign) *Video designer; projection mapping on large surfaces and garments* Italian video designer behind the PROFORMA studio, specialising in projection mapping on buildings, stages and fashion, often driven by Kinect depth cameras. #### t-shirt mapping — Nicola Buttari (PROFORMA Videodesign) (2016) - Video: https://vimeo.com/179772378 - Interaction: Hands & Body, Projection - Platform & tech: Projection, Kinect, projection mapping, generative graphics - Idea: Clothing becomes a screen when projection follows the body. - What it is: Generative graphics are projected onto a white t-shirt and follow the wearer's body, turning plain clothing into a moving display. - Technique: A Kinect depth camera segments the torso and drives a generative mask that is projection-mapped onto the shirt in real time. - Try it: Use a webcam body-segmentation model to mask a projector's output so a pattern lands only on a white shirt. Twist: the pattern changes with the wearer's heartbeat or breathing sound. ### Nicole He *Programmer and artist; former Google Creative Lab creative technologist* Programmer and artist making playful work with voice interfaces, games and AI; SFPC and ITP community member. #### Garden Friends — Nicole He (2018) - Video: https://www.youtube.com/watch?v=d90F-61T6KQ - Interaction: Voice & Sound, Spatial Mapping, Play - Platform & tech: Phone, ARCore, Dialogflow, Unity - Idea: Say nice things to little AR plants and they grow into unique flowers shaped by your conversation. - What it is: An AR + voice experiment with animator Eran Hilleli: you grow personalized trees and flowers on real surfaces by chatting with a quirky character, speech driving the scene. - Technique: ARCore plane detection places plants on real surfaces, while speech recognition and an intent-matching dialog system (Dialogflow) map what you say to parameters that grow and style the plants. - Try it: Place a small character with AR Foundation plane detection and use speech recognition plus keywords or an LLM to interpret what you say and decide which plants grow. Twist: the character responds only to praise, and scolding makes the garden wither, so you feel what politeness toward machines means. ### Niki de Saint Phalle *Artist* French-American artist (1930–2002) who built the Tarot Garden in Tuscany, a hillside of giant mosaic-covered figures based on tarot cards. #### The Tarot Garden — Niki de Saint Phalle (1998) - Video: https://www.youtube.com/watch?v=fLcmyExFqdM - Interaction: Location & City, Play, Perception & Effects - Platform & tech: Desktop, mosaic, concrete, sculpture garden - Idea: A walkable set of characters from one symbolic system turns a landscape into a story you move through, a model for AR theme trails. - What it is: On a Tuscan hillside at Garavicchio, Niki de Saint Phalle spent almost 20 years building 22 monumental figures of the tarot's Major Arcana covered in mirror, glass and ceramic mosaic, some of them inhabitable. - Technique: Steel armatures were sprayed with concrete and covered by hand in mirror and ceramic mosaic. - Try it: Choose a card set (tarot, zodiac, a local deck) and place each card's character as an AR figure along a park trail. Twist: each visitor draws a random card and the path reorders around it. ### Nina Chanel Abney *Painter and public artist* American painter whose bold, flat, graphic compositions mix abstraction and figuration to capture the pace of contemporary life. Her first AR work, Imaginary Friend (2020, Acute Art), launched in parks and squares from London to New York. #### Imaginary Friend — Nina Chanel Abney, Acute Art (2020) - Video: https://www.youtube.com/watch?v=hugy9hJlrqg - Interaction: Location & City, Shared & Social - Platform & tech: Phone, Acute Art app, ARKit, ARCore - Idea: A painted character becomes an always-available companion in public space. - What it is: A large, flat-coloured figure in Abney's painting style appears in parks and squares; the sage-like character offers a blessing that its friend refuses, with the line 'sometimes we believe nothing good can ever happen to us'. - Technique: Phone AR via the Acute Art app, with the figure available at geolocated launch sites (London, New York) and placeable anywhere. - Try it: Turn a 2D character drawing into a flat, layered AR cut-out that says one short line when tapped. Twist: the line changes depending on how many people are standing in front of it. ### Nino Filiu *Generative artist, VJ and web developer* Paris-based artist, VJ and developer who built Supermosh, the first browser-based datamosh editor, so anyone can glitch video without codecs or command lines. #### Supermosh — Nino Filiu (2025) - Video: https://www.youtube.com/watch?v=M1OCjF-aJyo - Source code: https://github.com/supermosh/supermosh.github.io - Interaction: Perception & Effects - Platform & tech: Web, TypeScript, WebCodecs - Idea: Datamosh without codecs or command lines, right in the browser. - What it is: A browser editor where clips are glitched with datamosh: motion from one shot drags the pixels of another, as shown in the author's tutorial. - Technique: Supermosh likely decodes video in the browser, extracts per-block motion between frames and re-applies that motion to pixels of a held or different frame instead of refreshing them, reproducing the missing-keyframe glitch. - Try it: Recreate a real-time datamosh on a WebXR or phone camera feed by estimating motion with optical flow in a shader and advecting the previous frame instead of drawing the new one. Twist: only mosh inside a virtual portal placed in the room. ### Ninsky *AR developer (Lens Studio, Spectacles)* Lens Studio developer who posts AR effect experiments and co-developed the Spectacles racing game Specs Racer with Max van Leeuwen. #### Summer Snow — Ninsky (2022) - Video: https://x.com/Ninsky_AR/status/1556994932363214852 - Interaction: Portals & Worlds, Perception & Effects, Location & City - Platform & tech: Phone, Lens Studio, particles, ground tracking - Idea: Overwrite the season: bring winter into a summer street. - What it is: In the middle of a hot summer day, a phone lens covers a sunny Dutch street in deep snow, with flakes still falling and a snow bank lying on the pavement. - Technique: A Lens Studio world lens likely uses ground tracking and a particle system for falling snow plus a deformable snow mesh laid on the detected ground plane. - Try it: Make a 'wrong season' AR filter for the schoolyard in Lens Studio or Effect House: falling leaves in spring or snow in summer. Twist: let footprints appear where people walk. ### Noboru Seto (ruccho) *Game and tool developer* A Japanese developer who publishes Unity tools under the name ruccho, best known for WaterRW, a 2D interactive water system whose waves are simulated on the GPU and react to rigidbodies. #### WaterRW — Noboru Seto (ruccho) (2021) - Video: https://x.com/ruccho_vector/status/1461970803428405248 - Source code: https://github.com/ruccho/WaterRW - Interaction: Play, Perception & Effects - Platform & tech: Desktop, Unity, compute shader, 2D physics - Idea: Simulate only the surface of the water, and it can be both cheap and interactive. - What it is: A 2D water surface ripples and splashes when objects fall in; floating rigidbodies bob naturally on the waves. - Technique: The wave equation is solved on the GPU along the water surface and the result deforms the surface mesh; rigidbodies push the height field and receive buoyancy back, and a shader renders reflection and refraction. - Try it: Put a GPU height-field pond on a real table in AR and let the phone's position drop virtual pebbles into it. Twist: real objects detected by the LiDAR mesh block the ripples. ### NoiseCrime *Unity developer and graphics experimenter* Long-time Unity developer who publishes rendering experiments such as stencil portal cubes and instanced-indirect examples. #### Portal Room Cubes via Stencil Buffer — NoiseCrime (2013) - Video: https://www.youtube.com/watch?v=5DKIP9N-OB4 - Source code: https://github.com/noisecrime/Unity-StencilPortalRoomCube - Interaction: Portals & Worlds, Perception & Effects - Platform & tech: Desktop, Unity, Stencil buffer - Idea: Several rooms can share the same space if each is only visible through its own face. - What it is: A small cube holds a different room behind each face; walking around it reveals several full-size spaces packed into one box. - Technique: Each cube face writes its own stencil value, and each room's shaders test that value so the rooms render only through their faces. - Try it: Place a stencil room cube on a real desk in phone AR and let students design one room per face. Twist: the rooms connect so a ball rolling out of one face enters the next. ### Not Vital *Sculptor and architect* Swiss artist who builds sculptural houses, towers and 'houses to watch the sunset' in remote places from Niger to Patagonia, and runs a sculpture park in the Engadin. #### Sculpture park and houses to watch the sunset — Not Vital (1999) - Video: https://www.youtube.com/watch?v=-KAxEmX3a2Y - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, sculpture park, castle, architecture - Idea: A building that exists only to frame one moment of the day is a pure example of designing an experience around time and place, as AR can. - What it is: In a Louisiana Channel film, Not Vital shows his sculpture park in Sent, his foundation in Tarasp Castle and the buildings he makes around the world, including houses built only to watch the sunset. - Technique: Simple sculptural structures with stairs and openings are oriented to the sunset and built with local materials and labour. - Try it: Place an AR platform with a single framed opening on a hill so it aligns with today's sunset azimuth, and invite people to climb it at sunset. Twist: the opening closes after the sun has gone down. ### Onat Hekimoglu & Tobias Kreter *Game designers (Cologne Game Lab); creators of Klanglichter* Onat Hekimoglu and Tobias Kreter built Klanglichter at Cologne Game Lab, a laser harp reimagined as the controller of an audiovisual game. #### Klanglichter — Onat Hekimoglu & Tobias Kreter (2015) - Video: https://vimeo.com/128600307 - Interaction: Hands & Body, Voice & Sound, Play - Platform & tech: Projection, Unity, Arduino, laser harp - Idea: Touch photons: make light beams both the instrument and the joystick. - What it is: A laser harp becomes a game controller: players interrupt visible laser beams with their hands to hit geometric objects in a projected world and trigger chain reactions of light and sound. - Technique: Light sensors on an Arduino detect which beams are blocked and send events to a Unity game that renders the projected visuals and sound. - Try it: Make three 'beams' with a flashlight and light sensors (or virtual beams in AR) and map each blocked beam to a note and a projected burst. Twist: a chord of all three beams unlocks a hidden level. ### Ouchhh (Ferdi Alıcı & Eylül Duranağaç) *New media studio; data sculptures and projection mapping* Studio founded by Ferdi Alıcı and Eylül Duranağaç, known for AI and data-driven projections; their 2016 Dare to Dream combined real-time face tracking with face projection mapping. #### Dare to Dream — Ouchhh (Ferdi Alıcı & Eylül Duranağaç) (2016) - Video: https://vimeo.com/160396410 - Interaction: Face, Projection, Performance - Platform & tech: Projection, real-time face tracking, projection mapping - Idea: Let dreams appear on a face as a living, tracked mask of light. - What it is: Animated patterns and 'dreams' are projected onto a performer's face in real time, staying locked to her features as she moves. - Technique: Real-time face tracking (with Fikirbazzenger) estimates the face mesh each frame and re-renders the content from the projector's calibrated viewpoint. - Try it: Use a webcam face-mesh tracker and a projector aligned next to it to project a simple pattern onto a volunteer's face; calibrate with four points. Twist: the pattern reacts when the person closes their eyes. ### Owlchemy Labs *VR/MR studio (Job Simulator), owned by Google* Makers of Job Simulator and Vacation Simulator, now leading hand-tracking design research and mixed-reality play on Android XR. #### Inside[JOB] — Owlchemy Labs (2025) - Video: https://www.youtube.com/watch?v=ioCCn_qrbuo - Interaction: Hands & Body, Play, Spatial Mapping - Platform & tech: Headset, Android XR, hand tracking, passthrough - Idea: The JobBot robot bursts into your real room and has you finish silly little jobs with your bare hands. - What it is: JobBot leaps into your real room on Android XR; you fix quirky system settings, 'phish' for data and stream updates in playful hand-tracked mini-jobs across MR and VR. - Technique: Uses high-fidelity hand tracking with physics-based grabbing so every virtual prop can be poked, pulled and thrown in passthrough space. - Try it: Use phone AR gesture recognition to make three 'absurd little jobs' (pinching a virtual plug and plugging it in next to a real socket, slapping a glitching screen, and so on). Twist: each task can be done with one hand only. ### PARRATORO (Rafael Parra Toro) *Kinetic and optical artist; AR op-art books* Venezuelan-born artist-engineer based in Buenos Aires who makes op-art and kinetic prints and books, from the lenticular POP ON OP to the phone-activated POP AR. #### POP AR — PARRATORO (Rafael Parra Toro) (2025) - Video: https://www.youtube.com/watch?v=s8CRSKZrNJU - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Phone, image tracking, WebAR, 3D animation - Idea: Extend op art's optical illusion into a real third dimension with AR. - What it is: A book of op-art pages; scanning a page with a phone makes the flat geometric artwork rise off the paper as animated 3D sculptures. - Technique: Each printed page is an image target; the app anchors animated 3D versions of the same geometric patterns to the page (likely image tracking in a mobile AR SDK). - Try it: Draw a black-and-white stripe pattern, use it as an image target in Lens Studio or 8th Wall, and extrude its stripes into moving 3D fins. Twist: the fins flatten back into the print when the phone moves too close. ### PRELOADED *Games studio (London) making games with purpose, including AR and VR for museums and broadcasters* Independent studio known for educational and cultural games. With BBC Studios it made BBC Earth - Micro Kingdoms: Senses for Magic Leap, narrated by Stephen Fry. #### BBC Earth - Micro Kingdoms: Senses — PRELOADED (2020) - Video: https://www.youtube.com/watch?v=8HHL9S_vhZM - Interaction: Perception & Effects, Spatial Mapping, Voice & Sound - Platform & tech: Headset, Magic Leap, Unity, spatial mapping - Idea: Put tiny creatures in your room at your size, then let you borrow their senses. - What it is: A leafcutter ant colony and a wandering spider appear on your floor at giant scale; narrated by Stephen Fry, you switch into their senses to perceive vibrations and smells as they do. - Technique: Places high-detail animated creatures on the mapped room surfaces of Magic Leap 1 and switches rendering modes to visualise non-human senses. - Try it: Make a phone AR scene where a scaled-up insect walks on the table, then add a 'sense mode' button that turns vibrations from a tapped table into visible ripples. Twist: design a mode for a bat's echolocation using the microphone. ### Patrick Dougherty *Sculptor* American artist who weaves saplings and sticks into huge swirling huts and towers on museum lawns and campuses, built with local volunteers in about three weeks. #### Stickwork — Patrick Dougherty (2013) - Video: https://www.youtube.com/watch?v=2WpVAq0qb-c - Interaction: Location & City, Drawing & Making, Shared & Social - Platform & tech: Desktop, saplings, weaving, volunteers - Idea: Line drawn in space with natural material shows how volumetric sketching can make architecture; AR 3D drawing apps can reach the same feeling. - What it is: At the Peabody Essex Museum, Dougherty and volunteers weave thousands of saplings into swirling hut-like forms that look like windblown drawings in 3D. - Technique: Flexible saplings are bent and interwoven over a rough frame, with the outer layer of sticks laid in one direction to suggest motion. - Try it: Use a 3D drawing tool (Open Brush in passthrough or a WebXR line tool) to weave a hut-sized structure over a real bench or tree with a team. Twist: every stroke must follow the direction of the real wind. ### Patrick Hughes *Artist; reverspective paintings* British artist who invented 'reverspective': paintings on protruding pyramid-shaped panels where perspective is painted in reverse, so the scene seems to move and turn as the viewer walks past. #### Reverspective paintings — Patrick Hughes (1964) - Video: https://www.youtube.com/watch?v=I-OuPNZmX-0 - Interaction: Perception & Effects, Gaze & Attention - Platform & tech: Desktop, painted relief, reverse perspective - Idea: Reversing depth cues makes a static object feel alive: AR designers can exploit the same conflict between shading and parallax to make flat labels or signs seem to watch the viewer. - What it is: Painted streets and bookshelves seem to turn and follow you as you walk past, because the parts that look far away actually stick out toward you. - Technique: Perspective is painted in reverse onto protruding pyramids and wedges, so the brain trusts the painting over the parallax and the scene appears to rotate as the viewer moves. - Try it: Model a reverspective (three pyramids pointing at the viewer) in AR, texture it with a hallway image, and compare it with a hollow version. Twist: print the texture and fold a physical copy to hold next to the AR one. ### Patrick Lichty *Media artist, curator and writer; member of Second Front* Media artist and theorist who co-founded Second Front, the first performance-art group in Second Life, animated projects for The Yes Men, and joined Manifest.AR's early AR interventions. He writes widely on virtual worlds, AR and 'Techspressionism'. #### Survival Suit Pod — Patrick Lichty (2010) - Video: https://www.youtube.com/watch?v=XQhLqwVLKQ0 - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, Layar, GPS - Idea: A satirical climate-survival gadget parked on an ordinary street. - What it is: At a Bushwick street corner, an AR survival pod designed for The Yes Men (a parody of their SurvivaBall climate-disaster suit) appears for passers-by with phones. - Technique: A 3D model of the pod is geolocated at the corner of Irving and Flushing in a mobile AR browser for the Bushwick Augmented Reality Intervention. - Try it: Design an absurd AR 'survival device' for a near-future disaster and park it where students wait every day. Twist: tapping it plays an infomercial. ### Peter Uithoven *Interaction designer; co-founder of Doodle3D* Dutch interaction designer who, as a student at HKU, made the shadow-boxing game ShadowFighter with openFrameworks, and later co-founded Doodle3D with Rick Companje. #### ShadowFighter — Peter Uithoven (2010) - Video: https://vimeo.com/12486954 - Interaction: Hands & Body, Play, Shared & Social - Platform & tech: Projection, openFrameworks, OpenCV - Idea: Your shadow fights your opponent's shadow. - What it is: Two players stand side by side in front of a white wall lit by two lamps and box in the air; their real shadows hit each other on the wall, and projected comic-book 'BANG!' effects and sounds score the fight. - Technique: A camera detects when the two shadows overlap on the wall, and an openFrameworks app projects hit effects and plays sounds; the game starts automatically when two players step in. - Try it: Light a wall with two desk lamps, detect shadow overlap with a webcam, and project a score and hit effects. Twist: make the shadows grow when a player gets hit. ### Phil Walton *Official Snap Lens Creator and author* Lens creator behind the viral 'Potato Boss' lens and author of The Designer's Guide to Snapchat's Lens Studio. #### Potato Boss lens — Phil Walton (2020) - Video: https://www.youtube.com/watch?v=PR5n2ogiu28 - Interaction: Face, Voice & Sound - Platform & tech: Phone, Desktop, Lens Studio, Snap Camera - Idea: Turn your face into a talking potato. - What it is: A Snapchat/Snap Camera lens replaces you with a talking potato whose mouth and eyes are cut from your own face; it went viral when a manager got stuck with it in a video meeting. - Technique: Face tracking crops the user's eyes and mouth textures and composites them onto a 3D potato model that follows head pose. - Try it: Use Lens Studio to paste your eyes and mouth onto a fruit or a piece of stationery to make a talking object. Twist: the object changes color when you say a keyword. ### Physio 3D *Mixed-reality physiotherapy app studio* Physio 3D makes a Vision Pro physiotherapy app that combines 3D exercise guidance with a real-time AI voice coach. #### Physio3D: mixed-reality physiotherapy with a real-time AI voice coach — Physio 3D (2026) - Video: https://www.youtube.com/watch?v=TlQLjDA5XSY - Interaction: Hands & Body, Voice & Sound, Information & UI - Platform & tech: Headset, Apple Vision Pro, real-time AI voice, 3D guidance - Idea: A therapist who talks back is placed where you actually exercise. - What it is: A Vision Pro app places a 3D physiotherapy guide in the patient's room and adds a real-time AI voice coach that explains exercises, answers questions and tracks activity. - Technique: 3D exercise demonstrations are anchored in mixed reality while a speech-to-speech AI model with the training plan as context answers the user live. - Try it: Place an animated 3D figure doing a stretch next to the user in WebAR and connect a voice assistant that knows the three steps of the stretch. Twist: the coach adapts the exercise if the user says it hurts. ### Pixelcase *Immersive media studio (VR, 360 video, AR)* Australian immersive studio founded in 2007 that produces VR, 360 video and AR experiences; in 2018 it shared AR portals captured at the Hobbiton movie set in New Zealand. #### AR portal at Hobbiton — Pixelcase (2018) - Video: https://x.com/pixelcase/status/986445860442587136 - Interaction: Portals & Worlds, Location & City - Platform & tech: Phone, ARKit, 360 capture, portal rendering - Idea: A doorway you can physically walk through into another place. - What it is: Standing on a path at the Hobbiton movie set, a phone shows a doorway hanging in the air; walking through it puts the viewer inside a captured 360° world. - Technique: A mobile AR app (likely ARKit) places a portal whose interior uses stencil masking to reveal a 360° captured environment that surrounds the user after crossing the threshold. - Try it: Build an AR Foundation portal with a stencil shader that shows a 360° photo of another campus spot inside the door. Twist: the portal only opens after you knock twice on the real floor. ### Raymond Salvatore Harmon *Artist, filmmaker and muralist* American artist working across graffiti, experimental film and painting; MIRAGE combined an architecture-scale painted wall with a mobile AR layer. #### MIRAGE — Raymond Salvatore Harmon, The Heavy Projects (B.C. Biermann) (2014) - Video: https://www.youtube.com/watch?v=rhtFEY1Bk54 - Interaction: Location & City, Drawing & Making, Perception & Effects - Platform & tech: Phone, image-recognition AR, mural - Idea: A giant mural only comes alive when you look through your phone. - What it is: An architecture-scale painted mural in LA's Fashion District that becomes an interactive animated artwork when viewed through a companion AR app. - Technique: Image-recognition AR (developed with The Heavy Projects) tracks the painted façade and overlays animated layers aligned to the mural. - Try it: Paint a mural on a large sheet of white paper and use image-recognition AR to animate several of its elements. Twist: show different AR layers on the same mural by day and by night. ### Richard Serra *Sculptor* American sculptor (1938–2024) of massive weathering-steel plates and curves, including East-West/West-East, four 15-metre plates across a Qatari desert. #### East-West/West-East — Richard Serra (2014) - Video: https://www.youtube.com/watch?v=0VbVxY98zG0 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, weathering steel plates, desert, Brouq Nature Reserve - Idea: Level tops over uneven ground turn a desert into a measuring device, a strong idea for AR objects that reveal terrain by staying level. - What it is: Four steel plates, each over 14 metres tall, stand in a line more than a kilometre long across a valley in the Brouq Nature Reserve in Qatar; their tops are level while the ground dips beneath them. - Technique: Plates of different heights were set in a surveyed line so that their tops share one elevation across the dip in the terrain. - Try it: Place four tall AR slabs in a line across a sloping field with their tops locked to the same world height, and walk the line from end to end. Twist: add a fifth slab that the user places by eye, then check how far off it is. ### Richard Shilling *Land artist* British artist who makes ephemeral works from leaves, stones, ice and sticks outdoors, photographs and films them, and teaches land art workshops. #### Transience — Richard Shilling (2018) - Video: https://www.youtube.com/watch?v=0O3d4n3_FuU - Interaction: Location & City, Tangible Objects, Drawing & Making - Platform & tech: Desktop, leaves, stones, ice, stop-motion - Idea: Sorting found things by colour instantly makes them look designed, a fast exercise that translates directly to AR collecting and arranging. - What it is: A film of Shilling's nature art from 2008 to 2018: leaves sorted into colour gradients, balanced stones, ice circles and spirals made in woods and streams, some with stone balancer Michael Grab. - Technique: Materials are collected on site and arranged by hand, often photographed at each stage for stop-motion. - Try it: Photograph 20 leaves or objects outdoors, cut them out with segmentation, and arrange them in AR into a colour gradient spiral on the ground. Twist: the spiral re-sorts itself when a new object is added. ### Robert Henke *Artist and composer; laser and audiovisual works* German composer (Monolake) and co-developer of Ableton Live who makes precise laser and kinetic works, including the Lumière laser performances and kinetic pieces with Christopher Bauder. #### Lumière — Robert Henke (2013) - Video: https://www.youtube.com/watch?v=fJzGd_KfLeY - Interaction: Performance, Voice & Sound, Perception & Effects - Platform & tech: Projection, lasers, custom vector software, live electronics - Idea: When every visual event is also a sound event, the audience believes the image; AR feedback that pairs light and sound one-to-one feels physical. - What it is: In a dark hall, high-power lasers draw fast abstract vector shapes on a screen while each flash is fused with a sound, performed live by Robert Henke. - Technique: Custom software generates vector shapes for lasers and the matching synthesized sound from the same events, so image and audio stay locked together. - Try it: Build an AR instrument where tapping the air spawns a laser-like line shape and a matching synthesized note at that point in space. Twist: make the shape's size follow the pitch and ask others to play a duet. ### Rune Skovbo Johansen *Indie game developer and procedural-generation researcher* Indie developer behind The Cluster and LayerProcGen who invented Surface-Stable Fractal Dithering, a dither pattern that sticks to 3D surfaces and keeps constant dot size as you move. #### Surface-Stable Fractal Dithering — Rune Skovbo Johansen (2025) - Video: https://www.youtube.com/watch?v=EzjWBmhO_1E - Source code: https://github.com/runevision/Dither3D - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, ShaderLab, HLSL - Idea: Dither that belongs to the surface, not the screen. - What it is: A 3D scene is rendered only with black-and-white dots that stick to the surfaces as the camera moves, while the dots keep the same size on screen through fractal subdivision. - Technique: A 3D dither texture of Bayer-like dot layers is sampled in surface UV space, and the shader picks the fractal level from UV derivatives so dots subdivide smoothly as surfaces get closer or farther. - Try it: Apply the dither shader to virtual objects in an AR Foundation scene and to the LiDAR room mesh, so the whole space turns into a stable one-bit engraving as you walk. Twist: map dot density to sound volume. ### Ryan Hickman *Robotics and AR product lead; co-founder of TickTock AI* Co-founded Google's Cloud Robotics group and worked on Project Tango before co-founding TickTock AI in 2017, where phone SLAM and AR became the brain and remote control of a home robot. #### TickTock AR-controlled home robot — Ryan Hickman (2018) - Video: https://www.youtube.com/watch?v=qAIdzAF_-Hc - Interaction: Tangible Objects, Spatial Mapping, Voice & Sound - Platform & tech: Phone, Project Tango, Asus Zenfone AR, Kobuki, Google Assistant - Idea: Tap a point in AR and a real robot goes there. - What it is: A phone running TickTock's software rides on a small mobile robot as its brain; in the AR app, users tap a spot on the floor and the real robot plans a path and drives there, also responding to Google Assistant voice commands. - Technique: Project Tango SLAM on an Asus Zenfone AR localizes the robot and maps the room, and taps on the AR view are raycast to floor coordinates for the path planner. - Try it: Mount a phone on a Bluetooth toy car, raycast taps in AR to floor points, and drive the car toward the tapped point. Twist: let a second phone see the car's planned path as a glowing line in AR. ### Ryan McLeod *Indie designer-developer of Blackbox (Apple Design Award winner)* Solo designer of Blackbox, a puzzle game with no buttons that uses the device's own sensors, twice an Apple Design Award winner, later rebuilt for Apple Vision Pro. #### Blackbox for Vision — Ryan McLeod (2024) - Video: https://www.youtube.com/watch?v=5LShA47LhIc - Interaction: Gaze & Attention, Hands & Body, Play - Platform & tech: Headset, Apple Vision Pro, visionOS, RealityKit - Idea: Puzzles with no buttons: solve the riddles floating in your room with your gaze, gestures, voice and body movement. - What it is: Blackbox rebuilt for Apple Vision Pro: bubble-like puzzles float in your room and are solved by looking, reaching, speaking and moving through space instead of pressing buttons. - Technique: Puzzles hide their input rules in non-obvious channels such as gaze dwell, reach distance, voice and body movement, detected with visionOS gaze, hand tracking and audio input. - Try it: Design an AR puzzle with no buttons that players must solve in unusual ways such as walking closer, speaking or staying still. Twist: the hints can only be heard, never seen. ### SCE Japan Studio (The Eye of Judgment) *Sony first-party studio* Sony's Japan Studio co-created The Eye of Judgment (2007) with Wizards of the Coast, a PS3 card battle game read by the PlayStation Eye camera. #### The Eye of Judgment — SCE Japan Studio (The Eye of Judgment) (2007) - Video: https://www.youtube.com/watch?v=rO9PN638q_k - Interaction: Tangible Objects, Play, Shared & Social - Platform & tech: Desktop, PlayStation 3, PlayStation Eye, CyberCode markers - Idea: Level up physical card battles: the monsters on the cards appear in 3D and fight on the table shown on TV. - What it is: Physical trading cards laid on a 3x3 mat are read by the PlayStation Eye, and the creatures on them rise up as animated 3D monsters that fight on the TV image of the table. - Technique: CyberCode markers on each card are decoded by an overhead camera to identify the card and its grid cell, and animated creatures are rendered on the video of the mat. - Try it: Make four cards with different image markers that summon matching characters when placed on a 2x2 grid, and decide the winner from which cards sit next to each other. Twist: a card's rotation decides which way its character faces. ### Sam Van Aken *Artist* American artist who grafts up to 40 varieties of stone fruit onto one tree, so it blossoms in many colours in spring. #### Tree of 40 Fruit — Sam Van Aken (2008) - Video: https://www.youtube.com/watch?v=uU2L5nTSHtc - Interaction: Location & City, Information & UI, Perception & Effects - Platform & tech: Desktop, grafting, heirloom fruit varieties - Idea: An artwork that reveals its design only in one season and keeps an archive of lost varieties alive shows how living data can be an artwork. - What it is: By grafting, Van Aken makes single trees that bear up to 40 kinds of stone fruit; in spring they blossom in patches of pink, white and crimson, and the trees are planted in public places across the US. - Technique: Buds of different varieties are grafted onto the branches of a host tree over several years following a planned map. - Try it: Place an AR tree in a courtyard whose branches blossom in different colours, each linked to a local heirloom variety with its story. Twist: the blossoms appear only during the real flowering week in your region. ### Sarah Ticho *XR director and founder of Hatsumi; co-director of Soul Paint* Artist and producer working where immersive technology meets health and wellbeing. With Niki Smit she directed Soul Paint, a VR body-mapping experience in which people paint where they feel emotions on and around a virtual body; it won the SXSW 2024 XR Special Jury Prize, and she co-founded the XR Health Alliance. #### Soul Paint — Sarah Ticho (2024) - Video: https://www.youtube.com/watch?v=BmYAtoJA9Wk - Interaction: Drawing & Making, Hands & Body - Platform & tech: Headset, VR, hand tracking, 3D painting - Idea: Paint your feelings where they live in your body, then see everyone else's. - What it is: In VR you stand before a translucent body and paint where you feel emotions and sensations, with colours, textures and movement, then walk through a gallery of other people's painted bodies. - Technique: Hand-tracked 3D painting tools in a headset let participants body-map sensations onto a shared avatar template; finished maps are saved and displayed as a collective archive. - Try it: Print an outline of a body, let each student colour where they feel 'nervous' and 'calm', then place all outlines in a WebXR room as floating panels. Twist: let viewers walk into a body and hear the owner's one-word description. ### Seeing the Invisible (Jerusalem Botanical Garden & Outset Contemporary Art Fund) *Touring AR contemporary art exhibition in botanical gardens* Exhibition initiated by the Jerusalem Botanical Garden and Outset Contemporary Art Fund, curated by Hadas Maor and Tal Michael Haring, which opened in 2021 in about a dozen botanical gardens around the world at the same time. Thirteen artists, including Ai Weiwei, El Anatsui, Refik Anadol, Daito Manabe, Isaac Julien and Jakob Kudsk Steensen, made site-specific AR works viewed through one app. #### Seeing the Invisible — Seeing the Invisible (Jerusalem Botanical Garden & Outset Contemporary Art Fund) (2021) - Video: https://www.youtube.com/watch?v=u6uJhr9BPsI - Interaction: Location & City, Perception & Effects - Platform & tech: Phone, geolocated AR, mobile app - Idea: One exhibition that exists in many gardens at once, visible only through a phone. - What it is: An exhibition of thirteen AR artworks by artists such as Ai Weiwei, El Anatsui, Refik Anadol and Sigalit Landau, installed at the same time in botanical gardens around the world and viewed through one phone app while walking the grounds. - Technique: Each artwork is geolocated to a spot in every participating garden and anchored with markerless phone AR, so visitors find the pieces by walking a route. - Try it: Have each student make one small AR piece in Lens Studio or WebXR about something invisible on campus (wind, noise, history) and place all of them along a walking route. Twist: install the same show in a second location and compare how the site changes each piece. ### Shigeo Fukuda (福田繁雄) *Graphic designer and sculptor; visual tricks* Japanese designer (1932–2009) famous for posters and sculptures built on visual tricks, such as Lunch with a Helmet On, a heap of welded cutlery whose shadow is a motorcycle. #### Lunch with a Helmet On — Shigeo Fukuda (福田繁雄) (1987) - Video: https://www.youtube.com/watch?v=aFUGx-DaKz4 - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Desktop, welded cutlery, single light source - Idea: A playful title plus one light turns junk into a joke: AR pieces that pair a mundane object with a surprising projected identity are easy to understand in five seconds. - What it is: A spiky ball of 848 welded forks, knives and spoons looks like scrap metal, until a lamp casts its shadow as a detailed motorcycle. - Technique: Cutlery is welded into a 3D armature whose outline, from one light direction, matches a motorcycle drawing; other angles show only chaos. - Try it: Import a random-looking spiky mesh into an AR scene with a virtual light and render a shadow that shows a bicycle; let users walk the light around it. Twist: use real cutlery on a table, tracked as an object, as the caster. ### Sigur Rós *Icelandic post-rock band; co-creators of the Magic Leap soundscape Tónandi* Icelandic band that spent five years with Magic Leap Studios building Tónandi, an interactive audio-visual soundscape. #### Tónandi — Sigur Rós (2018) - Video: https://www.youtube.com/watch?v=5_A-qdfx1iw - Interaction: Voice & Sound, Hands & Body, Perception & Effects - Platform & tech: Headset, Magic Leap One, hand tracking - Idea: Reach out and touch glowing creatures in your room to play and shape the music. - What it is: An interactive Magic Leap soundscape in which glowing sound creatures spread across your real room; touching, gathering and waving at them with your hands layers and shapes Sigur Rós's music. - Technique: Room-anchored creature agents each carry a music stem or sample, and hand-tracking contact or proximity triggers, layers and modulates those audio sources. - Try it: Scatter five glowing AR creatures around a room, each tied to a music stem that fades in as you approach. Twist: when they are gathered together, the stems combine into a complete piece. ### Simular *AI company building the computer-use agent Sai* Simular builds Sai, an autonomous agent that operates a computer, and released an open-source bridge to Meta Ray-Ban glasses. #### Sai-Fi: talk to a computer-use agent through Ray-Ban glasses — Simular (2026) - Video: https://x.com/SimularAI/status/2090926852974297577 - Source code: https://github.com/simular-ai/sai-fi - Interaction: Voice & Sound, Information & UI - Platform & tech: Wearable, Desktop, Ray-Ban Meta, computer-use agent, Android - Idea: Your glasses delegate; an agent does the clicking at home. - What it is: An open-source Android companion app lets the wearer speak to Sai, an autonomous computer-use agent, through Meta Ray-Ban glasses, sending it tasks it then carries out on a computer. - Technique: The phone app relays voice (and optionally camera) input from the glasses to a cloud agent that controls a desktop through screenshots, mouse and keyboard actions. - Try it: Connect a voice note on your phone to a script that performs one safe desktop action (rename a file, open a playlist) and reports back by voice. Twist: the agent must describe what it did as a sports commentator. ### Singtaa *Developer of OneJS; Unity compute-shader tinkerer* The developer behind OneJS, which brings JavaScript UI to Unity, and Spark2D, an open library of compute shaders for noise, 2D signed distance fields, jump-flood SDF generation and gradients. #### Spark2D — Singtaa (2024) - Video: https://x.com/Singtaa/status/1837274411667444140 - Source code: https://github.com/Singtaa/Spark2D - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, Unity, compute shader, OneJS, SDF - Idea: Treat every shape as a distance function, and outlines, glows and blends come for free. - What it is: Inigo Quilez's 2D distance-field shapes (hearts, stars, rounded boxes and more) morph, glow and blend live in a Unity UI, scripted from JavaScript. - Technique: Compute shaders evaluate 2D signed distance functions, fBm noise and jump-flood SDF generation into textures, which gradient and blur kernels then colour; OneJS bindings expose them to JavaScript. - Try it: Draw floating AR stickers from 2D SDF shapes whose glow radius follows the viewer's distance. Twist: two stickers melt together with a smooth-min blend when you push them close. ### Slinkachu *Street artist* British artist behind Little People, a long-running project that leaves hand-painted model-railway figures in London streets and photographs them in tiny scenes. #### Little People — Slinkachu (2006) - Video: https://www.youtube.com/watch?v=VqTSqUOHTtg - Interaction: Location & City, Tangible Objects, Perception & Effects - Platform & tech: Desktop, model railway figures, street photography - Idea: Street details become full stories at 1:87 scale; AR can re-scale everyday surfaces into worlds in the same way. - What it is: Hand-painted model-railway figures are left in London's streets in tiny scenes (a man fishing in a drain, a couple on a crisp packet) and photographed close-up, then left to be found or lost. - Technique: Commercial model figures are repainted and modified, placed in real street settings and photographed with a macro lens. - Try it: Scan or model tiny AR people and stage a story on a real kerb, drain or leaf, then take a macro-style AR photo. Twist: the scene reacts to real rain by opening tiny umbrellas. ### Spencer Byles *Forest sculptor* British artist who spends a year at a time in a forest, building large sculptures only from what he finds there and leaving them to decay. #### A Year in a Catalan Forest — Spencer Byles (2023) - Video: https://www.youtube.com/watch?v=GcJbxM9iIEo - Interaction: Location & City, Tangible Objects, Perception & Effects - Platform & tech: Desktop, found wood, forest, year-long residency - Idea: Works hidden off the path reward exploration, like geo-anchored AR pieces placed away from main routes. - What it is: On the edge of La Garriga in Spain, Byles spends a year building large sculptures in the forest from fallen branches and found materials, leaving them for walkers to discover and for nature to reclaim. - Technique: Branches, vines and wood are woven and balanced into large forms using only what is found on site. - Try it: Hide three AR sculptures made from scanned forest materials at least 20 metres off a trail, with only a vague hint to find them. Twist: each sculpture slowly decays after it has been found. ### Stephen Orlando *Photographer* Canadian engineer-photographer whose Motion Exposure series attaches colour-changing LED sticks to paddles, swimmers and athletes to draw their motion in long exposures. #### Motion Exposure: whitewater kayak — Stephen Orlando (2016) - Video: https://www.youtube.com/watch?v=gO9IQDsL2-s - Interaction: Hands & Body, Location & City, Perception & Effects - Platform & tech: Desktop, LED light sticks, long exposure, colour-changing timing - Idea: Colour-coding time along a motion path turns one stroke into a readable diagram, a visual grammar for AR motion trails. - What it is: In the Elora Gorge in Ontario, LED sticks mounted on a kayak paddle change colour over time while a long exposure records the paddle's path over the whitewater. - Technique: Programmable LED strips attached to the paddle cycle through colours on a timer during a long camera exposure at night or dusk. - Try it: Track the phone or a hand with AR and draw its path in a colour that changes every second, then freeze the result as a 3D sculpture. Twist: record a sports movement (a throw, a swing) and compare two people's traces. ### Stuart Langfield *Director and designer* Film and motion director whose 2017 Super Mario AR test rethought a side-scroller as a ring-shaped level wrapped around the player. #### Super Mario AR — Stuart Langfield (2017) - Video: https://vimeo.com/246824152 - Interaction: Play, Hands & Body - Platform & tech: Phone, ARKit, Unity - Idea: A side-scrolling Mario level wraps around you; you turn to follow him and reach out to grab power-ups for him. - What it is: A Super Mario World level is bent into a ring around the player; you turn in place to follow Mario and reach out to grab power-ups floating in real space and drop them onto him. - Technique: The level geometry is wrapped onto a cylinder centered on the ARKit world origin so rotating the phone scrolls the game, and a raycast pick lets you move items. - Try it: Wrap the level of a small 2D game into a ring around the player (in Unity or Reality Composer) so the player advances by turning in place. Twist: players must crouch to see a hidden level. ### Sutu (Stuart Campbell) *Artist, comic creator and XR director* Australian comic artist and XR director whose Modern Polaxis was one of the first AR comic books, later known for Tilt Brush work and XR art at festivals and museums. #### Modern Polaxis — Sutu (Stuart Campbell) (2014) - Video: https://www.youtube.com/watch?v=5jhDF0CJToA - Interaction: Tangible Objects, Perception & Effects - Platform & tech: Phone, Unity, Vuforia - Idea: Scan a printed comic with your phone and every page comes to life. - What it is: A printed comic about a paranoid time-traveller; pointing a phone at each page brings panels to life with animation, sound and 3D depth. - Technique: Vuforia image targets recognise each printed comic page, and Unity renders animated layers, 3D depth and sound anchored to the page. - Try it: Draw and print a four-panel comic, then use MindAR or AR Foundation image tracking to animate one panel and add sound. Twist: have the AR layer tell a "truth" that contradicts the page. ### Tadashi Kawamata (川俣正) *Artist* Japanese artist who attaches hut-like structures of scrap wood to buildings, trees and bridges, as if a second, informal city were growing on the first. #### Tree Huts — Tadashi Kawamata (川俣正) (2008) - Video: https://www.youtube.com/watch?v=MacwNa-vP88 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, scrap wood, trees, building facades - Idea: Small structures attached to real surfaces at unexpected heights are a physical preview of facade-anchored AR content. - What it is: In Berlin and other cities, Kawamata attaches small huts of scrap wood high in trees and on building corners, as if informal shelters were growing on the city. - Technique: Huts are built from salvaged boards and fixed with brackets and ropes to trees and walls without damaging them. - Try it: Anchor five small AR huts to the corners and trees of real buildings using vertical plane detection, each containing a tiny scene. Twist: new huts appear wherever people linger longest. ### Takahiro "Poly" Horikawa *CEO and engineer at Curiosity Inc.; AR developer* Engineer and CEO of Curiosity Inc. in Tokyo (the company behind the Rememory volumetric capture studio) and a member of Placeholder. He rebuilds AR effects from commercial apps in a few lines of Unity code and shares the source. #### AR MeshWave — Takahiro "Poly" Horikawa (2021) - Video: https://x.com/thorikawa/status/1387068106757992449 - Source code: https://github.com/thorikawa/ar-meshwave - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Phone, Unity, AR Foundation, ARKit, LiDAR - Idea: A light wave that only exists on real surfaces makes the scanned room itself the effect. - What it is: A rainbow of 'gaming light' sweeps across the walls, floor and furniture of a real room, re-creating the LiDAR AR Spaces effect of Apple Clips in Unity. - Technique: AR Foundation meshing turns LiDAR scans into a room mesh; a roughly 50-line shader colours each fragment by its world-space distance from an origin minus time, so rainbow bands expand over every real surface. - Try it: Re-create the mesh wave on a LiDAR iPhone with AR Foundation meshing and a distance-based Shader Graph material. Twist: start a new wave wherever you tap, and let waves from different taps interfere. ### Tatsuro Ogata (llcheesell) *Director and producer of XR projects* A Tokyo-based director and producer of virtual live events and XR projects who builds his own production tools, including LEDScreenShader, a realistic LED-panel shader for Unity used for virtual stages. #### LEDScreenShader — Tatsuro Ogata (llcheesell) (2022) - Video: https://x.com/llcheesell/status/1492332637976936448 - Source code: https://github.com/llcheesell/LEDScreenShader - Interaction: Performance, Perception & Effects - Platform & tech: Desktop, Unity, HLSL, HDRP, URP - Idea: Imitate the physical structure of a display, not just its image, to make a virtual screen believable. - What it is: A virtual stage screen looks like a real LED wall: from far away it shows a clean image, and up close individual red, green and blue diodes and cabinet seams appear. - Technique: The HLSL shader multiplies the input image by an RGB subpixel mask (texture or procedural stripe, grid or honeycomb layout), draws cabinet seams, and uses screen-space derivatives (ddx/ddy) to fade LED detail and avoid moiré at a distance. - Try it: Hang a virtual LED billboard on a real wall in AR, feed it the phone camera or a video, and let viewers walk up to discover the diodes. Twist: the billboard shows a different message to each viewer depending on their distance. ### Tender Claws (Samantha Gorman & Danny Cannizzaro) *Immersive game and art studio* Studio led by Samantha Gorman and Danny Cannizzaro that makes experimental VR and AR works such as Virtual Virtual Reality, The Under Presents and the AR pet TendAR. #### TendAR — Tender Claws (Samantha Gorman & Danny Cannizzaro) (2018) - Video: https://www.youtube.com/watch?v=v4YMQRwsNr8 - Interaction: Face, Play, Tangible Objects - Platform & tech: Phone, ARCore, face emotion detection, object recognition - Idea: A small AR fish that feeds on your expressions and emotions. - What it is: An ARCore pet game where a talking guppy fish lives in the camera view and feeds on the player's emotions, read from their facial expressions, and asks them to show objects around them. - Technique: Front-camera facial expression classification feeds the pet's emotional state, while rear-camera object recognition checks whether the player shows the requested item. - Try it: Use expression recognition (MediaPipe or ARKit blend shapes) to tell whether you are smiling, and have an AR fish feed on your emotions. Twist: the fish asks you to photograph a specific object and checks it with object recognition. ### Thibaud Goiffon (HeyTibo) *Game developer and technical artist* Game developer publishing as HeyTibo who shares quick Godot 4 visual-effect sketches, shaders and characters, and open-sourced the whole VFX sketchbook under MIT. #### VFX Sketchbook (Godot 4) — Thibaud Goiffon (HeyTibo) (2023) - Video: https://www.youtube.com/watch?v=M8iTroIF1ag - Source code: https://github.com/gtibo/VFX-sketchbook-Godot-4.x - Interaction: Perception & Effects - Platform & tech: Desktop, Godot, GDShader - Idea: A sketchbook habit: one quick effect at a time, all shared openly. - What it is: A montage of stylised real-time effects made in Godot 4, from fireballs, fireworks and a force field to portals, a tornado portal, butterflies and a water hand, each a small sketch. - Technique: Each effect combines GPU particles with custom spatial shaders that scroll noise textures, erode alpha and push emissive colour, organised as reusable scenes in one project. - Try it: Port one sketch to an AR-capable engine (Godot OpenXR passthrough or Unity AR Foundation) and trigger it on a real surface with a tap. Twist: chain three sketches into a spell that the user casts with a gesture sequence. ### Thomas Ratliff *Indie XR game developer* Independent VR/MR game developer (Party Crashers) who posts mixed-reality experiments on Meta Quest 3, from passthrough camera shaders to LiDAR-style particle scans of his room. #### Passthrough Camera Shaders (QuestCameraKit) — Thomas Ratliff, Roberto Coviello (2025) - Video: https://x.com/devtom7/status/1902033672041091453 - Source code: https://github.com/xrdevrob/QuestCameraKit - Interaction: Perception & Effects, Hands & Body - Platform & tech: Headset, Unity, Meta Quest 3, Passthrough Camera API, Shader Graph - Idea: Once the headset camera is readable, the real world can be filtered like a photo, in place. - What it is: On Meta Quest 3 the user pinches a slider in the air and a region of the real room seen through passthrough becomes pixelated, one of several camera-image shaders. - Technique: The Passthrough Camera API delivers the headset camera image as a texture; the shaders likely reproject it onto world-space quads using the camera pose and intrinsics, then pixelate, blur or recolor it. - Try it: Open the camera-shader scene of QuestCameraKit and write a new filter (thermal vision, halftone, edge glow) for a hand-held window. Twist: let two hands define the window's corners. ### Tianyi Xiao (ETH Zurich) *Researcher in geoinformation, spatial cognition and AR at ETH Zurich* Tianyi Xiao works on spatial cognition and augmented reality at ETH Zurich. Sketch2Terrain (CHI 2025) combines freehand 3D sketching with a generative model so that non-experts can map terrain in AR. #### Sketch2Terrain — Tianyi Xiao (ETH Zurich) (2025) - Video: https://www.youtube.com/watch?v=DPvd_EmxzVg - Source code: https://github.com/TX-XR/Sketch2Terrain-CHI25 - Interaction: Drawing & Making, Spatial Mapping, Information & UI - Platform & tech: Headset, pix2pix, 3D sketching, AR headset, Unity - Idea: Sketch a few ridgelines in the air and let AI fill in the whole landscape. - What it is: Draw a few freehand 3D strokes in AR and an AI model turns them into a complete terrain with hills and valleys in real time, so non-experts can sketch the landscapes they remember. - Technique: Strokes are turned into a curve network and a height-map image, which a pix2pix image-to-image model completes into a terrain height map rendered as a 3D mesh in the headset. - Try it: Have students draw a few contour lines on paper or in a web canvas, turn the image into a height map with an image-to-image model (or a height-map ControlNet), and load it as a displaced plane on a table in phone AR. Twist: a second student walks a tiny virtual figure across the terrain and describes the route in words. ### Tim Noble & Sue Webster *Artist duo; shadow sculptures* British artist duo who pile rubbish, scrap metal or taxidermy into shapeless heaps that, lit from one lamp, cast precise shadow portraits of the two artists. #### Shadow sculptures (Dirty White Trash (With Gulls)) — Tim Noble & Sue Webster (1998) - Video: https://www.youtube.com/watch?v=u7DGkkxBS5w - Interaction: Perception & Effects, Tangible Objects, Projection - Platform & tech: Projection, rubbish, projector light, shadow - Idea: The gap between a messy object and its clean projection is the punchline: AR can reveal a hidden portrait inside clutter by choosing where the virtual light comes from. - What it is: A heap of the artists' own rubbish sits on a plinth; a projector beam throws its shadow on the wall, where it becomes a crisp silhouette of the two artists sitting back to back. - Technique: The pile is arranged piece by piece under a single projector light until its shadow matches a traced silhouette of the artists. - Try it: Scan a messy desk with LiDAR, then place a virtual spotlight whose shadow of the real mesh forms your classmate's profile (fake it by rendering the profile as the shadow). Twist: the portrait only appears when the desk is at its messiest. ### Tomoki Itamiya *Professor, Aichi University of Technology; AR disaster-education researcher* Tomoki Itamiya builds AR and VR apps that let people rehearse earthquakes, floods and fires in their own rooms for disaster education. #### Generative-AI earthquake rehearsal overlaid on a real room — Tomoki Itamiya (2025) - Video: https://x.com/t_itamiya/status/1914896389152133364 - Interaction: Spatial Mapping, Information & UI, Portals & Worlds - Platform & tech: Headset, Apple Vision Pro, generative video, spatial video - Idea: Rehearse a disaster in the exact room where it could happen. - What it is: From a single photo of a real room, a generative video model creates an earthquake scenario with falling ceiling panels, which is converted to spatial video and overlaid on the same room in Vision Pro, with rewind and slow motion. - Technique: An image-to-video model (such as Veo or Kling) animates the photo into a quake, a 2D-to-stereo step makes it a spatial video, and it is aligned to the real room in the headset. - Try it: Photograph a corner of the classroom, generate a 5-second 'what if the shelves fell' video with an image-to-video tool, and play it in AR aligned to the real corner. Twist: students redesign the corner and generate again to see if it is safer. ### Tooster (T3sT3ro) *Programmer* Programmer who built SdfManipulator, an interactive Unity editor for raymarched signed-distance-field scenes that generates HLSL shaders from gizmo-driven primitives, as his bachelor's thesis. #### SdfManipulator — Tooster (T3sT3ro) (2024) - Video: https://github.com/user-attachments/assets/f52581a2-1cf7-49b1-a8d6-941ec3a6ccc7 - Source code: https://github.com/T3sT3ro/SdfManipulator - Interaction: Drawing & Making, Perception & Effects - Platform & tech: Desktop, Unity, HLSL, Raymarching - Idea: Model with distance fields the way you would with clay, without writing shader code. - What it is: In the Unity editor, smooth SDF shapes are dragged, blended and carved with gizmos while the raymarched result updates live. - Technique: A tree of SDF primitives and operators with editor controllers is compiled into an HLSL raymarching shader through a generated ShaderLab/HLSL syntax-tree API, so each gizmo edit rewrites or re-parameterises the shader. - Try it: Export a scene built with SdfManipulator as a raymarched cube and place it on a table with AR Foundation, then let two fingers control the blend radius between shapes. Twist: drive one primitive with the user's hand position. ### Virendra Suryawanshi *Developer* Virendra Suryawanshi builds applied computer-vision prototypes for mixed reality headsets. #### AR pothole detection, grading and geotagging — Virendra Suryawanshi (2025) - Video: https://x.com/Virendra0698/status/2001634595494855158 - Interaction: Location & City, Information & UI - Platform & tech: Headset, Meta Quest 3, YOLO, passthrough camera, GPS - Idea: Inspect a street by looking at it; the headset keeps the report. - What it is: A Quest 3 app detects potholes in the road view with a custom YOLO model, grades their severity and tags their location, turning a headset into a road-inspection tool. - Technique: A custom-trained YOLO detector runs on passthrough frames, and detections are graded by size and logged with coordinates. - Try it: Train a small detector on 50 photos of cracks or litter on campus, run it on a phone walk, and drop an AR pin with a severity colour at each detection. Twist: the pins stay for the next person who walks by. ### Wenbo Tao *Independent visionOS developer (Magic Room, Mission Hub)* Wenbo Tao makes Apple Vision Pro apps such as Magic Room, which repaints the scanned room mesh with live effects. #### Room restyled as Studio Ghibli, live on Vision Pro — Wenbo Tao (2026) - Video: https://x.com/wenbq_me/status/2104174834276061569 - Interaction: Spatial Mapping, Perception & Effects - Platform & tech: Headset, Apple Vision Pro, GPT-6, scene reconstruction - Idea: Rebuild your room as editable objects first, then restyle it any way you like. - What it is: An AI model reconstructs virtual objects located and sized exactly like the real furniture of a room, which can then be restyled, here as a Studio Ghibli scene, in real time on Vision Pro. - Technique: Room-scanning data and camera views are given to a multimodal model that proposes matching 3D objects, which are then placed over the real furniture and re-textured in a chosen style. - Try it: Scan a corner of a room with a phone LiDAR app, replace each piece of furniture by a primitive of the same size, and texture them with AI images in one art style. Twist: every visitor sees a different style. ### Wingnut AR (Peter Jackson, Alasdair Coull) *AR studio founded by Peter Jackson and Fran Walsh* Wellington studio staffed largely by former Weta Digital artists; creative director Alasdair Coull premiered its Unreal Engine tabletop battle on stage when Apple launched ARKit at WWDC 2017. #### Wingnut AR WWDC 2017 ARKit demo — Wingnut AR (Peter Jackson, Alasdair Coull) (2017) - Video: https://www.youtube.com/watch?v=S14AVwaBF-Y - Interaction: Spatial Mapping, Play, Performance - Platform & tech: Phone, ARKit, Unreal Engine 4, iPad - Idea: A tabletop becomes a cinematic miniature battlefield you can walk around like a film set. - What it is: On the WWDC 2017 stage, an iPad placed a desolate outpost on a real table; as the presenter walked around it, airships arrived, bombed the base and exploded into debris, all in Unreal Engine. - Technique: ARKit world tracking and horizontal plane detection anchor an Unreal Engine 4 scene to the table, with real-time lighting estimation and particle effects. - Try it: Place a miniature diorama on a detected table plane and script a 30-second cinematic event (arrival, conflict, aftermath) that plays when the viewer walks to a marked viewpoint. Twist: let the viewer's distance to the table control the time scale. ### Wita (witalosk) *Creative technologist; Unity tool and shader developer* A Tokyo-based developer who writes Unity tools for live visuals, including a package that compiles fragment and compute shaders at runtime, and experiments with position-based and particle fluids. #### Unity Runtime Shader — Wita (witalosk) (2024) - Video: https://x.com/witalosk/status/1805984379543638156 - Source code: https://github.com/witalosk/UnityRuntimeShader - Interaction: Drawing & Making, Performance - Platform & tech: Desktop, Unity, HLSL, compute shader, DirectX 11 - Idea: Edit an effect while it is running, so tuning happens in front of the audience or inside the scene. - What it is: Fragment and compute shader code typed into a running Unity app recompiles instantly and repaints the output, like a live-coding session for visuals. - Technique: The package compiles HLSL at runtime through the DirectX 11 shader compiler and writes the results into a RenderTexture or GraphicsBuffer that the rest of the scene can use (Windows and DX11 only). - Try it: Build a tiny live-coding panel in a WebXR page where a GLSL snippet paints a floating AR canvas and edits apply on save. Twist: two students edit the same canvas from two phones. ### Xuan Luo *Computer-vision and graphics researcher* Researcher who, as a PhD student at the University of Washington, created Pepper's Cone with Jason Lawrence and Steven Seitz, a DIY 3D display made from a plastic cone and a tablet, with an open Unity implementation. #### Pepper's Cone — Xuan Luo (2017) - Video: https://www.youtube.com/watch?v=dppCXj11I_Q - Source code: https://github.com/roxanneluo/Pepper-s-Cone-Unity - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Phone, Unity, iPad - Idea: A cheap cone plus a pre-warped image makes a walk-around hologram. - What it is: A thin plastic cone on an iPad shows a 3D object suspended inside it, and the view stays perspective-correct as the display is rotated on its base (UIST 2017 video). - Technique: The Unity app reads the tablet's gyroscope to know the viewer's relative orientation, renders the object from that direction and pre-distorts the image so that its reflection in the cone appears as an undistorted, perspective-correct 3D object. - Try it: Build a Pepper's-ghost pyramid or cone for a phone and feed it a pre-warped render of an AR Foundation scene, using the phone's gyroscope or front-camera face tracking to update the view. Twist: display a live 3D scan of a classmate inside the cone. ### Yasuhito Nagatomo *visionOS and iOS developer* Tokyo visionOS and iOS developer who publishes small open samples for RealityKit, ARKit and Shader Graph materials, often with a short screen recording on X. #### Shader Graph Material Examples for visionOS — Yasuhito Nagatomo (2024) - Video: https://x.com/AtarayoSD/status/1791701412147253643 - Source code: https://github.com/ynagatomo/SGMExamples - Interaction: Perception & Effects - Platform & tech: Headset, visionOS, RealityKit, Reality Composer Pro, Shader Graph - Idea: A small library of readable node graphs is the fastest way into shaders on a new platform. - What it is: On Apple Vision Pro a model dissolves away along a noisy, glowing edge and reappears, one of a set of open Shader Graph materials that also includes fresnel, caustic waves, flowing rivers and gaseous motion. - Technique: Materials are built in Reality Composer Pro's Shader Graph: a noise texture compared against an animated threshold drives opacity and an emissive edge band, and the material is applied to entities in RealityKit. - Try it: Rebuild the dissolve material in Reality Composer Pro or Unity Shader Graph and trigger it when a virtual object is placed in the room. Twist: make the dissolve start from the point the user touched. ### Yinka Shonibare *Artist* British-Nigerian artist whose sculptures use Dutch wax-print fabric to question colonial history, including Nelson's Ship in a Bottle on Trafalgar Square's Fourth Plinth. #### Nelson's Ship in a Bottle — Yinka Shonibare (2010) - Video: https://www.youtube.com/watch?v=voEgrnPqKxo - Interaction: Location & City, Information & UI - Platform & tech: Desktop, glass bottle, Dutch wax-print sails, Fourth Plinth - Idea: An answer placed on a plinth beside an old monument rewrites the square's story, the way AR counter-monuments can. - What it is: A 1:30 scale model of Nelson's flagship HMS Victory with sails of Dutch wax-print fabric sits in a giant glass bottle on the Fourth Plinth of Trafalgar Square, facing Nelson's Column. - Technique: A detailed ship model was built with printed-fabric sails and sealed in a large acrylic bottle. - Try it: Find a monument near you and place an AR object on an empty plinth or ledge nearby that answers its story from another viewpoint. Twist: visitors vote each week on which answer stays. ### Youngki Lee (Seoul National University) *Professor of Computer Science and Engineering at Seoul National University* Youngki Lee's lab works on mobile, wearable and AR systems. Bubbleu (CHI 2023), led by Minji Kim with Kyungjin Lee and Rajesh Balan, studies how AR games should be designed around the errors of object-detection AI. #### Bubbleu — Youngki Lee (Seoul National University) (2023) - Video: https://www.youtube.com/watch?v=nrq8WfGFB8s - Interaction: Play, Tangible Objects - Platform & tech: Phone, object detection, mobile AR, game design - Idea: Treat the AI's mistakes as game material instead of hiding them. - What it is: An AR pet-breeding game in which you feed your creature by pointing the camera at real objects; because object detection often fails, the game is designed around that uncertainty through ambiguity, transparency and player control. - Technique: An object-detection model recognizes real items as food; game variants expose or hide the model's confidence, blur what counts as a match, or let players correct results, and were compared in a user study. - Try it: Build a phone game where a creature eats whatever the camera detects (COCO-SSD in the browser is enough), then make three versions: hide confidence, show confidence as the pet's mood, and let players argue with the pet when it is wrong. Twist: let the pet's personality grow out of its most frequent misrecognitions. ### Yu Wang & Yulu Lu (Beihang University) *Researchers at the School of New Media Art and Design, Beihang University* Yu Wang, Yulu Lu, Shuo Yan and Xukun Shen at Beihang University explore expressive AI agents in mixed reality. Their CHI 2025 late-breaking work turns everyday objects into personalized AI characters. #### If My Apple Can Talk — Yu Wang & Yulu Lu (Beihang University) (2025) - Video: https://www.youtube.com/watch?v=XyrT846glfE - Interaction: Tangible Objects, Voice & Sound - Platform & tech: Headset, mixed reality headset, LLM, object recognition - Idea: An object's physical traits become the design brief for its AI personality. - What it is: A mixed-reality study that turns everyday objects such as an apple, a cup or a book into personalized AI agents, exploring how an object's shape, material and use can shape the agent's look, voice and personality. - Technique: A formative study produced design dimensions for object agents; the mixed-reality prototype likely combines object recognition, an LLM persona and expressive virtual features anchored to the real object. - Try it: Choose one object and write its persona from four traits (shape, material, function and age), build a phone AR character on it with an LLM voice, and compare it with a classmate's persona for the same object. Twist: let the persona change as the object is used up, like an apple being eaten or a candle burning down. ### Yujie Tao *HCI researcher at Stanford SHAPE Lab (previously UChicago Human Computer Integration Lab)* A PhD researcher in Sean Follmer's SHAPE Lab at Stanford working on audio augmented reality such as Audio Personas, after research with Pedro Lopes at the University of Chicago on haptic illusions and mixed reality. #### Integrating Real-World Distractions into Virtual Reality — Yujie Tao, Pedro Lopes (2022) - Video: https://www.youtube.com/watch?v=PO8ZlQGYMY8 - Interaction: Voice & Sound, Portals & Worlds, Perception & Effects - Platform & tech: Headset, VR, sensors - Idea: Turn real-world interruptions into events inside the virtual story. - What it is: Instead of blocking out the real room, the VR game absorbs it: a fan's gust sways a curtain in the virtual world, a sudden bang becomes an explosion and the coffee machine's noise becomes falling debris. - Technique: A small sensing module detects simple signals such as loud sounds, wind and temperature shifts, and the game maps each one to a matching virtual event at the same moment. - Try it: Use the phone microphone in a WebXR or Unity scene to trigger thunder, a slammed door or rustling leaves whenever a real sound passes a threshold. Twist: map a classmate's footsteps to a giant walking nearby. ### Zappar (Caspar Thykier & team) *AR platform company; makers of ZapWorks and ZapBox* London company founded in 2011 that runs the ZapWorks/Mattercraft WebAR tools and made ZapBox, a cardboard mixed-reality headset that uses printed markers and a phone camera. #### ZapBox — Zappar (Caspar Thykier & team) (2017) - Video: https://www.youtube.com/watch?v=RnnZ3YQD4ig - Interaction: Tangible Objects, Hands & Body, Spatial Mapping - Platform & tech: Headset, Phone, ZapBox, pointcodes, cardboard headset - Idea: Room-scale MR with tracked controllers, built from paper, cardboard and a phone camera. - What it is: A $30 cardboard headset turns a phone into a room-scale mixed-reality device: printed pointcode markers scattered on the floor map the room, and two cardboard controllers with codes act as tracked hands. - Technique: The phone camera detects printed circular pointcodes to build a map of the room and track handheld cardboard controllers in 6DoF, with a fisheye lens widening the view. - Try it: Print six ArUco markers, tape two onto cardboard paddles, and build a phone AR scene where the paddles become tools that hit a floating ball. Twist: use a marker taped to the floor as a portal that swaps the ball into another world. ### Zinn Labs (Robert Konrad, Nitish Padmanaban, Kevin Boyle) *Eye-tracking company for smart eyewear, founded by alumni of Gordon Wetzstein's Stanford lab* Zinn Labs develops low-power eye tracking for smart glasses. Together with Gordon Wetzstein they published GazeGPT, which uses the wearer's gaze to tell a multimodal LLM which object in the camera view the question is about. #### GazeGPT — Zinn Labs (Robert Konrad, Nitish Padmanaban, Kevin Boyle) (2024) - Video: https://www.youtube.com/watch?v=AuDFHHTK_m8 - Interaction: Gaze & Attention, Voice & Sound, Information & UI - Platform & tech: Wearable, eye tracking, multimodal LLM, smart eyewear - Idea: Use your gaze as the pointer for an AI that looks through your glasses. - What it is: Smart eyewear that tells a multimodal LLM exactly what you are looking at: glance at a dog and ask 'what breed is this?', and the eye tracker tells the AI which of the many things in view you mean. - Technique: An eye tracker maps gaze into the world-facing camera image; the region around the gaze point is cropped and sent with the spoken question to a multimodal LLM. - Try it: Without an eye tracker, fake gaze with the phone's screen center or a fingertip tracked by MediaPipe: crop that region, send it with a spoken question to a multimodal LLM, and compare answers with and without the crop on a cluttered desk. Twist: record where the user looked for a minute and ask the AI what they seem to be interested in. ### dECOi / Mark Goulthorpe (HypoSurface) *Architecture practice; kinetic surface research* dECOi is the architecture practice of Mark Goulthorpe; with MIT they developed HypoSurface, a wall of hundreds of pneumatically driven metal plates that ripples into waves, patterns and letters in response to sound and movement. #### HypoSurface — dECOi / Mark Goulthorpe (HypoSurface) (2001) - Video: https://www.youtube.com/watch?v=ANXQRJ2zksI - Interaction: Voice & Sound, Hands & Body, Spatial Mapping - Platform & tech: Desktop, pneumatic pistons, metal tiles, sensors - Idea: A physical surface that moves like a screen shows what AR could add to real walls: content that seems to deform the wall itself rather than float in front of it. - What it is: A wall of hundreds of small metal plates driven by pneumatic pistons ripples into waves, patterns and even letters in response to sound and people moving in front of it. - Technique: Each tile sits on pneumatic actuators under computer control; sensors (microphones, motion detection) feed a real-time pattern generator that sets piston positions. - Try it: Detect a real wall and apply a vertex-displacement shader in AR so the wall seems to bulge into waves around the user's hand or voice. Twist: make the ripples spell a word when viewed from exactly one spot. ### dotsimulate (Lyell Hintz) *TouchDesigner artist; creator of StreamDiffusionTD* Lyell Hintz, known as dotsimulate, builds real-time AI tools for TouchDesigner, most notably StreamDiffusionTD. #### StreamDiffusionTD: real-time diffusion in TouchDesigner — dotsimulate (Lyell Hintz) (2024) - Video: https://www.youtube.com/watch?v=X4rlC6y1ahw - Interaction: Perception & Effects, Performance - Platform & tech: Desktop, Projection, TouchDesigner, StreamDiffusion, NVIDIA GPU - Idea: Any live video signal can be restyled by a prompt as it happens. - What it is: A TouchDesigner component that runs StreamDiffusion so any live camera, 3D render or audio-reactive visual can be repainted by a diffusion model in real time for installations and VJ sets. - Technique: StreamDiffusion's pipelined, few-step image-to-image generation runs on an NVIDIA GPU inside TouchDesigner, taking a texture input and a prompt and outputting frames at interactive rates. - Try it: Point a webcam at a table with objects, run a real-time diffusion node (TouchDesigner or a hosted API), and project the restyled image back onto the same table. Twist: let the audience change the prompt by moving objects. ### herman de vries *Artist* Dutch artist and former biologist who fences off small pieces of land as sanctuaries, letting nature grow undisturbed inside cities. #### sanctuarium — herman de vries (1997) - Video: https://www.youtube.com/watch?v=3vaA37kDS44 - Interaction: Location & City, Perception & Effects - Platform & tech: Desktop, fenced land, wild growth, city park - Idea: Protecting a patch of nature and letting time do the work makes an invisible process the artwork, a stance for slow AR. - What it is: For Skulptur Projekte Münster 1997, de Vries fenced a circle of land in a city park with a wall and left it untouched, so wild plants grew freely inside, visible only through small openings. - Technique: A round enclosure with a few viewing slits is built around a plot of land that is then left unmanaged for years. - Try it: Mark a small patch of real ground in AR with a virtual fence and log photos of it once a week to build a time-lapse. Twist: the virtual fence grows taller the longer the patch remains undisturbed. ### setchi *Unity engineer* Japanese Unity engineer known for the FancyScrollView UI library, who also writes compact animated ShaderLab sketches and GLSL tweets. #### Unity-ShaderSketches — setchi (2017) - Video: https://x.com/setchi/status/947253545111134209 - Source code: https://github.com/setchi/Unity-ShaderSketches - Interaction: Perception & Effects - Platform & tech: Desktop, Unity, ShaderLab - Idea: A daily habit of tiny fragment-shader sketches builds a library of moving patterns. - What it is: A grid of animated patterns — pulsing circles, rotating tiles, interference waves — each drawn by a few lines of ShaderLab code. - Technique: Each sketch is a single unlit ShaderLab fragment shader that computes colour from UV, time and math (polar coordinates, fract tiling, smoothstep) with no textures. - Try it: Pick three sketches and apply them as materials on AR planes detected on the floor and walls, so the room is covered in moving patterns. Twist: feed the phone's distance to each surface into the shader so patterns speed up as you approach. ### xr masiso *XR developer and tutorial maker* xr masiso publishes step-by-step Quest mixed-reality tutorials. #### Stable Diffusion AR with Quest passthrough camera access — xr masiso (2025) - Video: https://www.youtube.com/watch?v=FXFgkAmvpgo - Interaction: Perception & Effects, Tangible Objects - Platform & tech: Headset, Meta Quest 3, Passthrough Camera API, Stable Diffusion, Unity - Idea: The first day the headset can see, it can also dream about what it sees. - What it is: A step-by-step tutorial that reads the Quest 3 passthrough cameras in Unity, then uses them for object detection and for Stable Diffusion generations based on what the headset sees. - Technique: A WebCamTexture from the passthrough camera is sent to a Stable Diffusion image-to-image endpoint, and the output is displayed on a panel in the room. - Try it: Follow a camera-access tutorial for any AR device, send one frame to an image-to-image API, and hang the result as a framed picture on your real wall. Twist: the picture updates only when someone new enters the room.