MediaVerse, SERMAS, XR4DRAMA, XReco – DW ReCo XR Updates

We'll go through the projects in alphabetical order, so there's

• MediaVerse (multi-asset co-creation feat. XR tools, HLT and accessibility, AI-supported content analytics, decentralized content sharing, and IP rights management via blockchain tech)

• SERMAS (socially acceptable XR models and systems)

• XR4DRAMA (extended reality for disaster management and media production planning)

• XRECO (towards a cutting-edge, data-driven XR ecosystem for the entire content industry)

MediaVerse

This one was actually wrapped up in October 2023, after three years of research, development, and learning. A comprehensive review post ("All good things must not come to an end"), co-authored by our Oliva Stracke, is still available on the project's official website.

As for DW's focus, we mainly explored the potential of interactive 360° video for immersive journalism, and how that technology can be made easily accessible for people with little experience in the field. Among other things, we created templates for Fader, a WebXR-based storytelling platform we had already worked on in other projects. In a workshop with DW journalists, we generated new interest for the tool; if time and budget allow, we aim to follow up and create more journalistic 360° stories with them.

Another interesting, very different aspect of MediaVerse are the AI safety filters developed by CERTH and DW. They're meant to protect individuals who handle gruesome and potentially harrowing imagery. These filters lessen the severity of the impact and function as an early warning system. This is particularly advantageous for journalists or human rights abuse investigators who deal with images from conflicts, natural disasters, and other such events. For more information, check out this (freely accessible) paper: Mitigating Viewer Impact from Disturbing Imagery using AI Filters: A User-Study

SERMAS

SERMAS entered its second year in October 2023, and progress has been swift. A rather sophisticated (albeit still internal) demonstrator combines customized avatars with large language models (LLMs) and allows for voice interaction via standard browsers, and cameras/mics on standard laptops or tablets. At the moment, users can basically ask anything. In the next iterations, we'll try and make the model more domain-specific (i.e. focused on safety/security for DW journalists and assistance/background info for Poste Italiane customers) and also add immersive features (avatars as guides in physical space). Here's a screenshot of the prototype:

For more info on the DW pilot, check out this SERMAS page. On the project's official website, you'll also find news and background info on the other use cases and prototypes, some of which also deal with robotics (and physical agents!).

The results of a recently completed open call (cascade funding for third parties) are summarized in this LinkedIn post. That's right, we have big plans regarding the integration of avatar and 3D object generators, privacy-preserving facial recognition, multilingual speech recognition, and artificial empathy.

In other SERMAS news, parts of the crew met at Immersive Tech Week in Rotterdam (our Axel Primavesi has shared his impressions over here)–and the entire consortium will meet again for a three-day plenary and development workshop in Bonn in late February. Stay tuned for more news on the project's social media channels.

XR4DRAMA

"How to handle disasters and prepare media productions with AR, VR and lots of data" – that's the title of a pretty neat and (very well received) documentary film on our XR4DRAMA project, which was also completed in 2023. The film manages to sum up almost all aspects of the innovation action in roughly 12 minutes, so there's no need to repeat ourselves here.

However, we'd like to point out that the resulting XR4DRAMA software suite (desktop tool + award winning mobile app) is still "up for grabs" in terms of further development and exploitation. So if you're looking for an application that allows you to create a customizable, detail-rich, immersive, and completely private 3D map with multimedia notebook and task managing functions, don't hesitate to get in touch.

XReco

XReco, coordinated by DW, has also entered its sophomore year – and is still dedicated to supporting the creation of next-generation XR assets and experiences by providing easy-to-use and affordable tools for all kinds of media industry players. To ensure the relevance of its activities, XReco is banking on the concept of human-centered design, which was also promoted in two successful workshops in October and November 2023. Up next: The (second) XReco Joint Business Clinic, scheduled for January 24th. Don't forget to register and tell the consortium what the European XR ecosystem needs and why.

Regarding recent achievements, XReco partners FFP and RAI gained international recognition with the digital reconstruction of both Maria Callas and a Budapest opera house where Pablo Larrain shot a part of his upcoming movie about the famous Greek-Italian singer, (played in the film by American actress Angelina Jolie). Watch the "Maria Callas 100th Tribute" PanoCAST clip to learn more about stable diffusion, NeRFs, Gaussian splatting, virtual production, and other tools and technologies involved in the process.

Last, but not least the XReco consortium have started to connect with the Women in Immersive Tech (WIIT) movement, and are planning on co-organizing a very special hackathon.

What else?

Here's a piece of news that isn't directly related to a project, but still captured the attention of all XR people at DW Innovation: Apple's Vision Pro will soon be available to "regular" customers. On Mastodon, we've asked whether fellow media tech people think it's a really fantastic or totally overrated gadget – or maybe something in between.