Collection Mémoires et thèses électroniques
Accueil À propos Nous joindre

CHAPTER 9: Verify/Validate, Test and Document the Multiagent Geosimulation

Table des matières

In this chapter, we present two steps of our proposed method. The first step aims to verify and validate the multiagent geosimulation models, while the second aims to test and document the geosimulation. We also present how we can illustrate these steps using the shopping behavior as a case study.

The sixth step of the proposed method aims at developing an operational geosimulation prototype that can be used by end-users (see Chapter 7). This prototype may be employed to (1) merely visualize, in 2D and 3D modes the phenomenon to be simulated in a geographic environment or (2) to make decisions about the phenomenon to be simulated, or about the simulation environment. In order to increase the users’ confidence in the geosimulation models, it is important to include in our method a step designed at verifying and validating the geosimulation models. This step is the first one which is presented in this chapter.

The geosimulation prototype is intended at being used by end-users. Hence, it is relevant to test and document this prototype which represents a grood practice in computer science. The test and the documentation of the geosimulation prototype are the purpose of the second step to be presented in this chapter.

This chapter is organized as follows: Section 9.1 presents generic descriptions of the two steps aiming, respectively, at verifying and validating the geosimulation models and at testing and documenting the geosimulation. Section 9.2 illustrates these two steps using the shopping behavior simulation models. The last section of the chapter (Section 9.3) presents the conclusion.

Using a simulation is a surrogate for the experimentation with a real phenomenon (existing or proposed), where experimentation with that phenomenon could be disruptive, non-cost effective, or even not feasible. Therefore, the model or simulation must be able to provide valid representations of the actual phenomenon. The verification and validation steps of a simulation insure that the simulation models accurately represent the real phenomenon to be simulated. These two steps which are theoretically distinct are closely related in practice (Arthur and Nance, 1996) (Balci, 1988). They not only insure that the model assumptions are correct, complete, and consistent, but also enhance the users' confidence in the simulation models (Anu, 1997). When we talk about simulation verification and validation, it is pertinent to keep in mind the following elements:

■ The verification and validation step establish the reliability of the model and simulation of the thing being represented. Verification and validation provide a crucial piece of evidence to support the simulation model credibility for a particular simulation application.

■ The ease or difficulty of the verification and validation processes depends upon the complexity of the phenomenon being modeled.

■ The simulation of a complex phenomenon can only approximate the actual phenomenon, no matter how much time and money are spent on simulation construction. There is no such thing as absolute simulation validity, nor is it even desired. Indeed, a simulation model is supposed to be an abstraction and simplification of reality.

■ A simulation should always be developed for a particular set of objectives (goals). Indeed, a simulation that is verified and valid for one set of objectives may not be so for another set of objectives.

In our method, we propose a step that takes into account the verification and validation of the geosimulation models. Because we deal with geosimulations, it is relevant to focus on the spatial aspects of the models during the verification and validation. This step contains the following sub-steps (see Fig 9.1):

(1) Choose the verification and validation technique : Our literature review revealed that there exist several verification and validation techniques that can be used to verify and validate a simulation model. These techniques are separated into four categories: informal, static, dynamic, and formal.

Informal verification and validation techniques are among the most commonly used. They are called informal because their tools and approaches rely heavily on human reasoning and subjectivity, without stringent mathematical formalism (DMSO, 2005).

■ Static verification and validation techniques assess the accuracy of the static model design and source code. Static techniques do not require machine execution of the model, but mental execution can be used. These techniques are very popular and widely used, and many automated tools are available to assist in the verification and validation process. Static techniques can reveal a variety of information about the structure of the model, the modeling techniques used, data and control flow within the model, and syntactical accuracy (Whitner and Balci, 1989).

Dynamic verification and validation techniques require model execution; they evaluate the model, based upon its execution behavior. Most dynamic verification and validation techniques require model instrumentation, the insertion of additional code into the executable model to collect information about model behavior during execution. Dynamic verification and validation techniques are usually applied in three steps: executable models are instrumented; instrumented models are executed; and models’ outputs are analyzed and dynamic models’ behaviors are evaluated (DMSO, 2005).

Formal verification and validation techniques (or formal methods) are based upon formal mathematical proofs of correctness, and are the most thorough means of model verification and validation. The successful application of formal methods requires that the model development process must be well defined and structured. Formal methods should be applied early in the model development process in order to achieve maximum benefit. Because formal techniques require significant effort they are best applied to complex problems, which cannot be handled by simpler methods (DMSO, 2005). 

The taxonomy, presented in Table 9.1, lists the majority of the verification and validation techniques of simulation models that exist in the literature. They are grouped according to the categories described above. The detail of each technique is not presented in this dissertation.

In our method, the first part of this verification and validation step aims at choosing the suited technique that can be used to verify and validate the geosimulation models. It is relevant to note that the choice of a verification and validation technique depends upon the nature of the phenomenon to be simulated.

(2) Verify and validate the geosimulation models : Using the technique chosen in the previous sub-step, we verify and validate the geosimulation models. In multiagent geosimulation models, we distinguish two parts: The first is related to the phenomenon to be simulated, while the second concerns the simulated environment. Hence, it is relevant to verify and validate these two parts, as well as their spatial characteristics.

(3) Adjust the geosimulation models : Based upon the results of the verification and validation processes, we may notice some divergence between the geosimulation models and the real phenomenon to be simulated. If this is the case, we must adjust some elements of the geosimulation models. Hence, and depending upon the required adjustments, we may return back to some steps of the method in order to adjust them.

Once the geosimulation is verified and validated, it can be used by end-users. Before that, it is important to test and document the geosimulation. Hence, in our method, we propose a step that takes into account the test and documentation tasks of the geosimulation. This step contains the following sub-steps (see Fig 9.2):

(1) Test the multiagent geosimulation : Testing the geosimulation is a vital part of our method because testing can show the presence of problems in the geosimulation models and prototype. The step of geosimulation testing answers some questions like: Does the geosimulation prototype really work as expected? Does it meet the users’ needs? Does it produce what users expect? Do the users like it? What is its performance? Is it ready for release? If we answer this kind of questions, we can: save time and money by identifying defects early, avoid or reduce development downtime, know that we’ve satisfied our users’ requirements, build a list of desired modifications and enhancement, for later versions of the geosimulation prototype, etc.

(2) Document the multiagent geosimulation : When the geosimulation is tested, and before it is used by end-users, it must be documented. The geosimulation documentation is an important aspect. In geosimulation, we should write documents, describing the process of the geosimulation development from the users’ needs identification to the final step. We also need to provide documents describing the analysis results of the simulation, as well as the geosimulation prototype. The main documents that can be generated during the documentation step are:

Documenting the users’ needs (users’ requirements document URD) : This document describes the problems from the users’ point of view. After the problem description, we describe what the users would like the geosimulator to do. Briefly, in this document, we find: a user’s view of the problem, a brief description of the problem, a complete description of the problem, what is expected from the simulator (the solution), and what is not expected from the simulator.

Documenting the geosimulation models : In this document, we describe the geosimulation models. It contains the detailed design of the geosimulation, as well as any architectural information.

Documenting the simulation data : This document presents the detail of data collected to feed the geosimulation models, as well as the technique used to collect this data.

Documenting the program code, or the used platforms/tools/languages : In this document, we must document the program code or the platform/tool/language used to execute the geosimulation models (user guide). If we use our own code for the simulation, we are expected to fully document this code.

Documenting the simulation results : In this document, we need to write out the details of the simulation results (simulation outputs, analysis results of these outputs, etc.).

Documenting the verification, validation results : In this document, we present the techniques used for the verification and validation of the geosimulation models and prototype.

Testing documentation : The testing documentation describes how we tested the geosimulation prototype in order to prove that it works adequately. We should include testbeds for the prototype. We should provide test datasets where applicable.

User documentation : In this document, we present how users can use the geosimulation prototype. We have two styles of user documentation: ‘instruction’ or ‘tutorial’ style, useful for the new user, and ‘reference’ style, useful for the more experienced user who needs information on specific topics. These two styles are often known as the ‘User guide’ and the ‘Reference manual’.

A user guide (or introductory manual) : Is an instructional document, giving step-by-step procedures and explanations for accomplishing specific tasks when using the geosimulation prototype. It is not necessary to cover, in detail, all the geosimulation prototype features in this document, but it must provide enough information for all but the most experienced users. This document should be designed to ‘teach’ new users how to use the geosimulation application, so worked examples of useful operations should also be included. As with other kind of documents, the most important principle in designing user guides is to write with readers in mind. How will readers use the geosimulation application? Will they install the geosimulation prototype themselves? Will they need to transfer the prototype from one computer environment to another?

A reference manual : It contains the basic operations, facts and key definitions, ordered for easy reference (e.g., alphabetically). The aim of this document is to allow an experienced user to look up terms and procedures, as simply and rapidly as possible. This document should be more rigorous, formal, and exhaustive than the user guide.

In the remaining part of the chapter, we illustrate these steps using the customers’ shopping behavior in a mall as case study.

In Chapter 5, we presented the details of the shopping behavior multiagent geosimulation models. These models are composed of two main parts: the first is related to the simulation environment (the mall), while the second concerns the main actor of the geosimulation (the shopper). If we want to verify and validate the shopping behavior geosimulation models, it is relevant to take into account these two parts of the models. In the following points, we present the verification and the validation of some elements of the two parts of the shopping behavior geosimulation models.

In order to verify and validate the shopping mall model (mall) we propose to use the face validation technique. This technique is defined as follows: ‘ Face validation technique is an informal technique in which certain persons, such as the simulation projects team members, potential users of the simulation model, and subject matter experts, review simulation outputs (e.g., numerical, results, graphics, animation, etc.) for reasonableness. They use their estimates and intuition to compare model and the phenomenon to be simulated subjectively, in addition to judging whether the model and its results are reasonable’ (Hermann, 1967). We propose such technique, because it is based upon the visualization and animation modes to verify and validate the simulation models. These modes are efficient to represent the spatial characteristics of the simulated environment in a geosimulation. Hence, the persons carrying out the verification/validation processes, can use this technique to compare the geographic features of the real mall (on the plans) (see Fig 9.3) with those of the virtual mall, presented in 2D (in the GIS) (Fig 9.4) and 3D (Fig 9.5) models. The validation of the geosimulation environment model (mall) depends on the geographic data coming from the GIS of the mall and which fed the environment model.

In order to verify and validate the shopper model we propose to use the visualization/animation technique. This technique is defined as follows: ‘ Visualization/Animation technique is a dynamic technique. Visualization and animation of a simulation greatly assist in a model’s verification and validation. Seeing the model in action is very useful for uncovering errors (Sargent, 1992)’. We chose this technique for the following reasons:

- It allows us to verify and validate the model when it is in action during the simulation. Unlike the environment model, the shopper model is active during the simulation (it has a behavior) and it is relevant to verify and validate its actions;

- The visualization and animation aspects of the technique are efficient to verify and validate the spatial aspects of the shopper model which are relevant when dealing with geosimulations.

As mentioned in (DMSO, 2005), verification and validation of human behaviors are extremely complex: ‘ Verifying and validating human behavior (such as the shopping behavior) is very different from the other behaviors of non-human systems. What differentiates human behavior simulation from other kinds of behaviors comes primarily from their highly inherent complexity. Human behavior arises from numerous nonlinear relationships all interacting chaotically over many different orders of magnitude. Also, the verification and validation of human behavior is extremely difficult. Advances in information system technology make the construction of fine-grained simulations of human behavior, for a variety of situations, more feasible and practical. Developers have built many primarily cognitive simulations of human behavior, and the sophistication and fidelity of these systems continue to improve. However, the technology for the verification and validation of human behavior has not kept pace with these advances, despite facing particularly vexing problems since their very first applications. Human behavior manifests an intricate fabric of effects, coupled over many orders of magnitude, a property shared by complex chaotic systems. Small situational changes often create greatly different responses in the same system. Thus, verification and validation of human behaviors, even for simple tasks, can prove extremely difficult because of the large number of behavioral paths that must be explored for any given purpose. The lack of well-established techniques to support the verification and validation of human behaviors further exacerbates the difficulty of these problems. At the most abstract level, verification and validation of a human behavior involves comparing human behavior capabilities to the model, and simulation requirements for human behaviors in order to determine its fitness for the purpose represented by those requirements. Usually, defining human behavior capabilities involves testing the human behavior within the simulated environment, in which it will be used and determined if the behaviors manifested in test results are sufficient ’ (DMSO, 2005).

The complexity of the behavior to be verified and validated (the shopping behavior in a mall) leads us to make some choices and define some limits concerning the shopper model’s elements to be verified and validated. First, we are more interested to validate the shopper agent’s decision-making process. Furthermore, we focus on the spatial aspects of the decision-making process (movement decisions of the shopper agent inside the virtual environment). Second, we focus on some specific shopping situations in order to evaluate the movement decisions made by the shopper agent. Finally, since the shopping behavior is composed of several patterns (see Chapter 7), we focus on some shopping behavior patterns. To sum up, our verification and validation of the shopper model is based upon the evaluation of the shopper agent’s spatial decision-making process, of some shopping patterns, during specific shopping situations. In the following points, we present some results of this verification and validation step. At each point, we focus on a specific shopping behavior pattern, we put the shopper agent in a specific shopping situation (where some variables and factors (internal or external) are defined), and we evaluate the spatial decisions made by the shopper agent.

Go to a shop :

Deciding about which shop to visit is influenced by several factors: internal variables (shopping list, preferences, etc.), knowledge (via the memorization process), in addition to external variables (the geographic characteristics and the knowledge of the environment) via the shopper agent’s perception process, and the priority of the objectives. In order to verify and validate this shopping pattern, we try to put the shopper agent in specific shopping situations (based upon the factors mentioned above) and observe how it behaves and makes decisions. In the following points, we present some of these situations.

- Situation 1: Evaluate the influence of distance on the decision-making process : This situation aims at evaluating how the shopper agent makes decisions taking into account the distance factor. This situation is described by the following elements: the two shopper agents, A and B, enter the virtual mall by the doors A and B, respectively (see Fig 9.6). These agents’ starting points are presented in red in Figs 9.7 and 9.8. The shopping lists and memories’ contents of these agents are presented in Table 9.2.

During the simulation we can verify that agent A visits shop 64 (the second element in its shopping list) and agent B visits shop 134. This shows that the agents are rational about distance and visit the shop nearest to them.

- Situation 2: Evaluate the influence of perception on the decision making process : This situation aims at evaluating how the shopper agent makes decisions based upon its perception capability. This situation is described by the following elements: The locations of shopper agents A and B are presented in Fig 9.9. These positions are presented in red in Fig 9.10. The shopping lists and the agents’ memories contents are presented in Table 9.3.

During the simulation presented in Fig 9.10 we can verify that agent A does not visit shop 134, even though it is on its shopping list. This happens because the agent A does not perceive the shop 134 (this shop is not in the perception field of agent A). On the other hand, shop 99 is in agent B’ perception field, therefore, agent B visits shop 99 once it perceives it.

- Situation 3: Evaluation of the influence of the memorization process on agent’s decision-making process : This situation aims at evaluating how the shopper agent makes spatial decisions based upon its memorization process. In this situation, the positions of agents A and B are presented in Fig 9.11. The agents’ shopping lists and their memories contents are presented in Table 9.4.

During the simulation execution presented in Fig 9.12 we can verify that agent A visits shop 63 because it is in its memory (agent A knows this shop). This is not the case with agent B. Shop 63 is on agent B’ shopping list, but this agent does not know it, therefore, it moves closer to the known shop which is shop 1.

Go to a restroom :

In order to verify and validate this pattern, we illustrate it with a specific situation. This situation aims to evaluate how the shopper agent’s dynamic variables can influence their decisions in the virtual mall. In particular, we show how the agent can change its destination in order to satisfy a need having a greater priority (the need to use the restroom has more priority than the need to visit a shop). Fig 9.13 shows this situation which presents the positions of two agents A and B. It also shows the location of the restroom in the virtual mall. Moreover, the shopping lists, the agents’ memories, and the coefficient of variation of the agents’ dynamic states are presented in Table 9.5. In this table, we can see that the dynamic state of agent A in relation with its need to use the restroom varies more quickly than that for agent B. Therefore, agent A feels the need to use the restroom before agent B.

During the executed situation in the MAGS platform, presented in Fig 9.14, we can verify that agents A and B visit shop 1. When they move to agent 99 (next destination), agent A feels the need to use the restroom. Agent B continues it way to shop 99 and agent A changes its direction to go to the restroom. When agent A satisfies its need of using the restroom, it visits shop 99 as planned before it felt the need to visit the restroom (see Fig 9.15).

Leave the mall :

In order to verify and validate this pattern, we consider the situation presented in Fig 9.16. Based on this figure, we can verify that the agent visited store 1 and decided to visit store 99. When it moves to store 99, it is aware that it reaches the time limit of its shopping trip and decides to leave the mall. Thus, the agent leaves the mall by the closest door.

The verification and validation of the shopper agent’s decision-making process, which is presented above, is simple because:

- we verify and validate the spatial decision-making process of the shopper agents;

- some patterns of the shopping behavior; and

- in very limited shopping situations.

These limitations are due to a lack of the gathered data concerning the shopper in Square One mall. We recall that in our questionnaire we did not gather data concerning the decision-making process of the shoppers (their choices, their decisions, etc.). In order to improve the verification and validation processes, we can:

■ elaborate a survey whose the purpose is to gather data concerning the shopping decisions made by the shoppers during their shopping trip;

■ conduct this survey;

■ simulate the shopping behavior in the MAGS platform using the collected data; and

■ compare the shopping decisions made by the shopper agents (during the simulation) with those of the real shoppers in order to calibrate the geosimulation and to verify and validate the models.

In this chapter we presented two important steps of our generic method. The first step aims to verify and validate the geosimulation models. The verification and validation of the simulation models are important (1) to insure that the simulation models represent the real phenomenon to be simulated, and (2) to enhance the users’ confidence in the simulation models. The second step aims to test and document the geosimulation prototype and results.

After the verification, validation, test, and documentation of the shopping behavior geosimulation models, the shopping behavior geosimulation prototype can be used by end-users (mall managers) to make efficient decisions about the shoppers’ behavior, or about the configuration of their mall. In the next chapter, we present the last step of our proposed method which consists in the use of the geosimulation by the end-users. We also illustrate it using the shopping behavior as a case study.

© Walid Ali, 2006