Large-scale Multi-agent Simulation and Crowd Sensing with Humans in the Loop

Overview

Introduction to Augmented Virtuality

Main topic of this talk is Fusion of Real and Virtual worlds creating Augmented Virtuality by using Mobile Agents!

Simulation of Socio-Technical Systems

• Socio-technical systems are characterized by interactions of:

  • Human-Human (initiated by a human)
  • Human-Machine (initiated by a human)
  • Machine-Human (initiated by a machine, e.g., a chat bot)
  • Machine-Machine (initiated by a machine)

• The simulation of social ensemble behaviour requires simplification of interactions and individual behaviour

• Commonly simulations are performed with less than 1000 entities (humans, machines, ..) in a sandbox world

• Agent-based Modelling (ABM) is a suitable behaviour model for simulation

Augmented Reality

• Augmented reality is well known for extending the real world by adding computer-generated perceptual information and overlaid sensory information

[1]

Field Studies

• Experimental field studies are commonly used in social science to test social models or to derive social models
• The ensemble size in field studies is often limited to less than 1000 individuals or entities

Data Mining and Machine Learning are important tools to derive meaningful information from experimental and aggregated data.

Taxonomy of Data Mining

[3]

Crowd Sensing

• Crowd data can be used in field studies to extend the information data base or replace classical (survey) field studies

• Mobile Crowd Sensing combines aggregation of user data and mobile computing, i.e., creating spatially annotated data traces

• Among data supplied by users explicitly, sensor data of mobile devices can be used, too. But: Weakly correlated data!

[2]

Augmented Virtuality

• Simulation worlds are commonly closed and rely on artificial sensory information generated by the simulator program or using data collected off-line (→ field studies).

A new Simulation paradigm providing augmented virtuality

Integration of Crowd Sensing and social Data Mining in simulation worlds

• The simulation world interacts with real world environments, humans, machines, and other virtual worlds in real-time using …

Agent-based Modelling and Computation

• Agents can represent artificial humans, bots, machines ..
• Agents can be used for distributed mobile computing → Crowd Sensing

Augmented Virtuality

Augmented Virtuality

Multi-Virtual Worlds

• Multiple virtual worlds can be connected by the augmented virtuality approach.
• Each virtual world can be considered as a world and model part
• Multiple virtual worlds can be simulated independently and in parallel → Speed-up!
• Virtual and physical world fusion by mobile agents …

Challenges

Time

• Different time scales in real- and virtual worlds

  • Simulation of distributed computing is much slower than Distributed Computing
  • Simulation is discrete (processed step-wise) with respect to the time scale
  • Interaction (reactivity) is different in virtual and real worlds

• Short-term versa long-term:

  • A simulation is performed commonly on a short time interval (window with defined start and end point)
  • Real world environments with interaction between humans and machines do not pose a well defined starting or end point → long-term execution

• Big Data and Information Strength

Challenges

Space

• Different spatial scales in real and virtual worlds

• Three different spatial models:

  1. Real and virtual worlds cover non-overlapping areas
  2. Real and virtual worlds cover overlapping areas
  3. Real world area is mapped on virtual world

Sensor Aggregation

Operational Layers

Sensor Aggregation

Sensor Aggregation

Virtual Sensor

• A virtual sensor consists of different components:

  • The environment of a sensor is a set of input streams of data generated from physical or virtual sensors.
  • The environment defines the context within the virtual sensor operates.
  • An aggregator processes the input streams and performs sensor data fusion
  • A filter produces a set of output streams.

• Agents can be used to implement virtual sensors

Sensor Aggregation

Sensor Aggregation

Virtual Sensor Network

• Networks of Virtual sensors compose processing chains (user defined or ad-hoc and self-organizing).

• Virtual sensors can operate in real-world environments (e.g., executed on mobile devices) or in virtual worlds (simulation).

• Agents implement the sensor data aggregator and filter function, performing the fusion, storage, and communication and represent mobile transport entities.

• Agent processing is virtualized by an unified Agent Processing Platform.

Sensor Aggregation

Agents and Simulation

Agent-based Modelling

Emergence

• Complex systems are characterized by emergent phenomena — patterns that appear to be quite complex can often be generated by simple rules.

• Emergence is a property classically exhibited by many agent-based models

• It occurs when an attribute that can be described at a system level is not specifically encoded at the individual level.

Modelling

• Agent-based modelling investigates the emergence behaviour of complex systems by defining a set of simple behaviour rules for individuals.

• Artificial Agents are characterized by:

  • Autonomy, loosely coupled, social
  • Reactivity, Pro-activity, goal-driven
  • Learning, Adaptation, Dynamic behaviour
  • Self-* (organization, adaptation, learning, ..)

Agent-based Computation

Distributed Computing

• Agents can be used for distributed and mobile computation in heterogeneous environments, too!

• Mobile agents can migrate between different host platforms:

  • Mobile Devices and Embedded Devices (Sensornetworks)
  • Stationary Computers
  • Servers and Clouds

• Computational agents are loosely coupled to their environment and agent processing platforms

Multi-Agent Systems (MAS)

• Large collection of loosely coupled computational processes interacting with each other. MAS can be used to implement:
  • autonomous,
  • reliable, and
  • adaptive data processing in distributed networks.

Sensor Clouds and the Internet

• Deployment of Agents can overcome interface and network barriers
  • Closing the gap arising between different platforms and environments
  • Integration of Sensor Networks and Crowd Sensing in Clouds

Agent Processing Platform

JavaScript Agent Machine (JAM)

• Agents are programmed in JavaScript; JAM is programmed in JavaScript ⇒

• One JAM can process up to 1000 agents/s and can be embedded in

  • WEB pages;
  • Mobile Apps (Android, iOS); or executed standalone on
  • Embedded system devices, desktop, notebooks, and server computer
  • Simulations

• Agents behaviour is modelled with Activity-Transition Graphs (ATG)

Agent Processing Platform

• JAM agents are mobile (carrying data and behaviour / code)

• JAM agents can interact (communicate) with eachother via:

  • Tuple spaces (public)
  • Signals (lightweight messages, private)

Simulation Environment for JAM

Simulation Environment for JAM

Crowd Sensing

Crowd Sensing with Mobile Agents

Crowd Sensing App

• Three versions:
  1. WEB Browser App that can be loaded from any WEB server or integrated in other WEB pages
  2. Mobile device App (Android, iOS)
  3. Desktop Program

• Layered Stack Architecture:

  1. JAM Agent Processing Platform
  2. Graphical user Interface (GUI)
  3. Visualization, Communication
  4. Multi-page Navigation with Chat Dialog and a Chat Mediator Agent
  5. Agents performing chat presentation and scheduling, mobile exploration, sensor aggregation

• Mediated Chat Dialog: External agents can engage in dialog with the user.

Crowd Sensing App

Crowd Sensing App

Security and Privacy

• The main goal of explorer agents is to collect crowd and social data!

• Therefore, an explorer agent is capable to interact with humans by performing textual dialogs (question-answer surveys).

• Due to the distributed and parallel processing of loosely coupled agents filtering and scheduling of agent dialog requests have to be performed by:

  • Chat Mediator Agent

• Capabilities (protected object-rights-key tuples) can be used to perform specific private chats and surveys

  • Explorer agents have to provide keys with specific rights that are evaluated by the chat agent allowing or denying the dialog request
  • Access of physical and virtual sensors can be protected by capabilities

All together

Human-in-the-loop simulation for Augmented Virtuality consists of:

• Virtual world Simulation Framework SEJAM2 based on the JAM platform and virtual JAM networks connected to the Internet

• Mobile devices hosting the JAM platform connected to the Internet

• Non-mobile devices and beacons connected to the virtual simulation world via the Internet

All together

Demonstrator

Smart City: Self-Organizing Light Control

Goal

Simulation and investigation of Crowd Interaction with Smart Cities

• Self-organized control of ambient light conditions (e.g., in streets or buildings) using Crowd Sensing

Virtual World

• World consists of streets and buildings

  • Beacons placed in buildings and beside streets
  • Smart Light Devices illuminating streets and buildings

• Agents (can) represent:

  • Technical devices → Light Control
  • Network Infrastructure and Communication
  • Computational units and Smart Controllers
  • Chat Bots performing Crowd Sensing
  • Robots
  • Artificial Humans (Artificial Crowd)

Smart City: Self-Organizing Light Control

Real World

• Mobile Devices using a WEB App with Chat Bot dialog

  • Users interact with remote agents via a chat dialog;
  • Remote agents investigate user location and an assessment of satisfaction of ambient light situation.

• Additionally, the device sensor data is collected (light, position, ..) if available

Self-organizing Light Control

• Beacons sent out mobile explorer agents performing question-answer dialogs on mobile devices (random walk)

• Based on collected crowd data:

  • The light conditions in buildings and streets should be adapted (darker or brighter);
  • Addressing 1. Crowd demands 2. Energy Saving.

Smart City: Self-Organizing Light Control

• If action is required, mobile notification agents are sent out to neighbouring nodes to change light intensity based on:
  • Directed diffusion
  • Random walk
  • Divide-and-Conquer

Smart City: Self-Organizing Light Control

Crowd Sensing App

Conclusions

1. Fusion of real and virtual worlds can contribute to investigate complex socio-technical systems with ensemble sizes beyond Millions of entities (users, devices, machines)

2. Using agent-based modelling and simulation with mobile agents create an unified model and representation for:

   • Artificial humans
   • Chat Bots
   • Devices and Machines
   • Distributed Computation

3. Augmented Virtuality means the integration and tight coupling of ABM simulation with Crowd Sensing and technical devices interaction

4. Issues:

   • Time scales in real and virtual worlds
   • Spatial scales and spaces in real and virtual worlds
   • Mapping of real on virtual worlds

References

1. Anna Chakravorty, User Experience Design Principles for Mobile Augmented Reality Applications, https://medium.com/@annadesign30/user-experience-design-principles-for-mobile-augmented-reality-applications-8ac0e5ebcdab (2.11.2018)
2. 3. ian Borcea, M. Talasila, and R. Curtmola, Mobile Crowdsensing. CRC Press, 2017.
3. L. Rokach and O. Maimon, DATA MINING WITH DECISION TREES Theory and Applications. World Scientific Publishing, 2015