of NA
JakubFlotyński
Knowledge-Based
Explorable
Extended
Reality
Environments
Knowledge-Based Explorable Extended Reality
Environments
Jakub Floty
´
nski
Knowledge-Based
Explorable Extended
Reality Environments
Jakub Floty
´
nski
Department of Information Technology
Pozna
´
n University of Economics and Business
Pozna
´
n, Poland
ISBN 978-3-030-59964-5 ISBN 978-3-030-59965-2 (eBook)
https://doi.org/10.1007/978-3-030-59965-2
© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland
AG 2020
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse
of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and
transmission or information storage and retrieval, electronic adaptation, computer software, or by similar
or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Switzerland AG.
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
To Dominika, for her understanding,
patience, and support.
Preface
Extended reality (XR) is a term coined based on virtual reality (VR) and augmented
reality (AR). While VR is a view of a world consisting only of virtual objects, which
may be interactive, AR is a view of the real world with superimposed virtual objects.
XR covers different forms of combined real and virtual environments, which range
between these two complementary technologies in the virtuality-reality continuum
(J. P. Gownder, C. Voce, M. Mai, D. Lynch 2016). Real and virtual objects in XR
environments may also have combined properties, which can be borrowed from real
objects, inspired yet different from real, or entirely virtual.
Elements that are inherent to virtually every XR environment are users and
interactive three-dimensional (3D) content. XR users and 3D content objects
typically have behavior, which encompasses autonomous actions and interactions
of users and objects. Users’ and objects’ behavior influences the XR environment,
including creation, modification, and destruction of objects in 3D scenes. Modeling
and tracking users’ and objects’ behavior in XR is fostered by the available systems
for motion and eye tracking, controllers, and haptic devices.
This book is devoted to explorable XR environments—their concept, architec-
tures as well as methods and tools for spatial–temporal composition based on
domain knowledge, including geometrical, presentational, structural, and behavioral
elements. Explorable environments’ features and behavior, covering past, current,
and potential states, can be subject to knowledge exploration with automated
reasoning and queries constructed by XR users and external services. It enables
monitoring, analyzing, comprehending, examining, and controlling XR environ-
ments as well as users’ skills, experience, interests, and preferences, and XR objects’
features.
The approach we propose in this book relies on two main pillars. The first are
knowledge representation technologies, such as logic programming, description
logics, and Semantic Web standards, which permit reasoning and queries. The
second are imperative programming languages, which are a prevalent solution
for building XR environments. We combine both within widely used integrated
development environments. XR exploration can be especially useful when the
acquired knowledge is represented using domain terminology, which is intelligible
vii
viii Preface
to specialists who are the primary users of the environments. Potential applications
of our approach can be indicated for a variety of domains, e.g., education,
training, medicine, design, tourism, marketing, merchandising, engineering, and
entertainment.
Explorable XR environments are an emerging field of research and applications
on the intersection of extended reality, artificial intelligence, software design and
development, modular and service-oriented architectures as well as programming,
data engineering, and data modeling. In recent years, we observe the intensive devel-
opment of new IT solutions in all of these disciplines as well as the growing interest
of researchers, practitioners, students, and users. Nonetheless, the connection of
these disciplines is still very new and hardly addressed by the available literature and
solutions. The book is a guide to novel approaches and tools for building explorable
XR environments.
The key benefit for the book’s readers is understanding the emerging domain
of knowledge-based explorable XR environments—its concept, motivations, appli-
cations, and system development. We pay attention to an in-depth discussion of
the field with taxonomy and classification of the available related solutions. We
analyze relationships between behavior-rich XR and knowledge representation,
which are supported by numerous examples of modeling, reasoning, and querying
temporal objects. We also provide examples and design patterns of knowledge-based
composition and exploration of XR behavior. Last but not least, we have extensively
evaluated and analyzed the proposed approaches.
The problems and solutions addressed by the book are relevant to the interna-
tional community. The book’s target audience encompasses industrial, training, and
educational institutions, including IT enterprises, universities, and colleges. The
book can be useful for IT researchers, practitioners, freelancers, undergraduate and
graduate students as well as a wide range of creators and users of IT systems in the
domains mentioned earlier.
The book leads readers step by step from the basic ideas to advanced concepts of
XR systems, knowledge representation, and programming. In the vast majority of
cases, we address the foundations and provide illustrative examples to make users
acquainted with the topic or at least to suggest knowledge sources that readers could
consult on their own. The most important prerequisite courses helpful to understand
the book are virtual and augmented reality, object-oriented programming, and logic
programming. In addition, the book could be adopted as a supplementary textbook
for knowledge representation and logic programming.
We address scientists’, lecturers’, teachers’, instructors’, and trainers’ needs by
starting the book with a general introduction to XR systems, 3D modeling tools,
and game engines. Then, we go into more advanced aspects of XR composition
and sophisticated cases of knowledge engineering, which we thoroughly discuss.
We intend to stimulate intuitive understanding of the presented concepts to enable
next its formal and precise understanding. Hence, the book can help scientists who
conduct research in these areas as well as lecturers and students who search for
clearly presented information supported by use cases.
Preface ix
For XR and game developers as well as graphic designers, the book can be a
valuable source of knowledge and examples in XR development and composition.
Moreover, it may show a completely new approach to the development and use of
XR, opening yet undiscovered opportunities related to XR-based knowledge explo-
ration. Due to the presentation of modeling behavior by composing independent
XR components, this work may also interest graphic designers who are keen to get
familiar with the augmentation of 3D models and scenes by activities designed in a
reusable service-oriented fashion.
Programmers, data engineers and modelers, and database and web developers
can find the book interesting as the proposed ideas are illustrated by various
examples demonstrating design patterns and development guidelines in object-
oriented, procedural, and declarative programming. Multiple discussed examples
are related to semantic knowledge-based representation of temporal data, including
objects, properties, and relations, which change over time. In this regard, the book
can be an exciting proposition for programmers, data engineers, and data modelers
who want to extend knowledge and skills about more complex solutions. In addition,
the book may gain web developers’ interest by connecting XR and web-based
systems, which mostly benefit from semantic technologies. On the one hand, these
target groups may realize what possible alternatives to widely used programming
languages and approaches to data modeling are. On the other hand, they can learn
how to apply the best practices of knowledge representation to describing dynamic,
temporal domains, such as virtual worlds.
Although we guide readers through details of creating and using explorable
XR environments, the more general introductory sections can also interest non-
professionals who would like to get acquainted with the related disciplines of
computer science.
Pozna
´
n, Poland Jakub Floty
´
nski
Acknowledgments
I would like to thank Professor Wojciech Cellary and Professor Krzysztof Walczak
for their support and valuable remarks on this book.
I would also like to thank the members of the Department of Information
Technology and the students at the Pozna
´
n University of Economics and Business:
Anna Englert, Marcin Krzyszkowski, Mikołaj Maik, Adrian Nowak, and Paweł
Soboci
´
nski for the collaboration on implementing XR environments presented in
this book.
xi
Contents
1 Introduction ................................................................. 1
2 Extended Reality Environments .......................................... 11
2.1 Relationships Between Users, Content, and Interfaces ............. 11
2.2 Creation of XR Environments ....................................... 12
2.2.1 Programming XR Environments........................... 12
2.2.2 3D Content Formats ........................................ 14
2.2.3 Visual Modeling of XR Environments .................... 15
2.3 Human–System Interfaces for Extended Reality ................... 16
2.3.1 Motion Tracking ............................................ 17
2.3.2 Eye Tracking ................................................ 19
2.3.3 Controllers .................................................. 21
2.3.4 Displays ..................................................... 22
3 Knowledge Representation with Logical Systems....................... 29
3.1 Foundations of Logical Systems..................................... 29
3.1.1 Statements and Knowledge Bases ......................... 29
3.1.2 Reasoning in Logical Systems ............................. 32
3.1.3 Computational Properties of Logical Systems ............ 34
3.2 First-Order Logic ..................................................... 37
3.2.1 Syntax and Semantics of First-Order Logic ............... 37
3.2.2 Queries and Reasoning on Knowledge Bases ............. 40
3.2.3 Knowledge Engineering.................................... 40
3.2.4 Representation of Spatial–Temporal Entities.............. 41
3.2.5 Logic Programming ........................................ 43
3.3 Description Logics ................................................... 45
3.3.1 Semantics of Description Logics .......................... 46
3.3.2 Types of Statements ........................................ 47
3.3.3 Constructors of Concepts and Roles ....................... 50
3.3.4 Reasoning Tasks in Description Logics ................... 55
3.3.5 Families of Description Logics ............................ 56
3.3.6 Spatial and Temporal Description Logics ................. 57
xiii