Evaluation of J-Sim

October 17, 2003

• 1 Qualitative Comparison
• 2 Quantitative Comparison
• 2.1 Experiment 1

1 Qualitative Comparison

In what follows, we summarize notable research efforts on network simulation and compare J-Sim against them wherever appropriate.  Some of the simulation packages are out-dated (but the results of which have been leveraged in subsequent projects) and are listed here for historical purpose:

• NEtwork Simulation Testbed (NEST) is a general-purpose simulation package designed to simulate distributed networked systems and protocols. It provides a client-server based graphical environment for simulation construction and execution control.
  
• Maryland Routing Simulator (MaRS) is a network simulation package based on a general-purpose network simulator NetSim developed earlier. As the name suggests, it is mainly used to study different routing algorithms.
  
• The REalistic And Large (REAL) network simulator is a substantially improved, and faster, version of NEST and is designed specifically for studying different congestion and flow control mechanisms in TCP/IP networks.
  
• Ns-2 is part of the collaborative VINT project involving USC/ISI, Xerox PARC, LBNL, and UC Berkeley. Ns-2 began as a variant of the REAL network simulator in 1989, and has evolved substantially over the past few years. It provides substantial support for simulation of TCP, routing, and multicast protocols, but due to the special node structure in ns-2, it is non-trivial, and sometimes difficult, to include other protocols/algorithms or accommodate other network architectures in ns-2. In addition, the not-so-structured software architecture and the mixture of compiled and interpreted classes make it difficult to understand and validate ns-2 codes.
  
• The S3 project and SSF are collaboration efforts of Rutgers University, Dartmouth College, Georgia Tech, and Boston University, and come closest to our proposed work. The project focuses on distributed and parallel simulation for large network simulation and proceeds along two fronts: design and implementation of (i) a simulator-independent network modeling framework, called Scalable Simulation Framework (SSF); and (ii) parallel and distributed simulation kernels (such as Georgia Time Warp (GTW)). SSF consists of 5 base classes: Entity, inChannel, outChannel, Process, and Event, and defines APIs to separate simulation modeling from parallel simulation internals. Two independent implementations of the SSF API have been provided: JSSF (written in Java) and DaSSF (written in C++). They also include a Domain Modeling Language (DML) to synthesize a model and instantiate a simulation.

SSF and J-Sim are similar in terms of the (i) object-oriented, layered software architecture design and (ii) isolation of modeling from simulation internals. There are nevertheless several important differences between SSF and J-Sim, which are tabulated in Table 1.

Table 1. A comparison between SSF and J-Sim.

Aspect	SSF	J-Sim
Object-oriented paradigm	Five base classes: (i) Entity serves as a container mechanism for defining alignment of a group of components to common local time (e.g., ProtocolGraph extends Entity); (ii) inChannel and (iii) outChannel are communication endpoints for event exchange; (iv) Event is the base class for information exchange; and (v) Process is the base class for describing dynamic behavior. Each instance of process is owned by a specific entity and may wait for input to arrive on channels owned by the entity, wait for some amount of simulation time to elapse, or do both.	A simple and well-defined component-based software architecture: Component is the base class. Ports are the only interfaces of a component to send/receive data. When data arrives at a port, an execution context (a Java thread) is created for the component to process the data. Components are asynchronous in the sense that two components may process different data at the same time without synchronizing between each other. Components can be hierarchically structured. A component may be itself a container mechanism and consist of subcomponents. This facilitates hierarchical modeling of complex systems.
Simulation framework	Classes interact with the underlying simulation engine with defined APIs	Simulation engine is built in the runtime and is transparent to components.
Process-oriented modeling	A set of wait() methods is provided for modelers to write process-based models.	Provides, in addition to the set of wait() methods, synchronization methods to further extend programming flexibility
Simulation technique	Discrete event simulation	Can work with both discrete event simulation and real-time process-based simulation
Performance scalability	Highly scalable by use of parallel simulation kernels	Will implement a parallel simulation engine (that takes advantage of the conservative approach) in the autonomous component architecture

As INET/J-Sim is laid atop the autonomous component architecture, a component layer, called SSFNET, is positioned on top of SSF, contains classes for network hosts and routers, and implements major Internet protocols (IP, TCP, UDP, OSPFv2, BGP-4, and the pseudo-network interface drivers, sockets), global Internet topology construction and IP address assignment. The differences between SSFNET and INET are tabulated in Table 2.

Table 2. A comparison between SSFNET and INET/J-Sim.

Aspect	SSFNET	INET/J-Sim
Configuration	The configuration files are in the Domain Modeling Language (DML) format. They are used to synthesize a model and instantiate a simulation run with the help of the configuration database package SSF.DML. The language is, however, not suitable to be used in an interaction environment.	A dual-language environment is provided: Java is used to create components and a script language is used as a glue or control language that integrates components in a desired fashion at run time and provides high-level, dynamic control. This facilitates fast prototyping of customized simulation scenarios, and runtime configuration and diagnosis.
Modeling	A node is an Entity. The infrastructure inside a node does not follow the container semantics in SSF but follows the protocol graph/protocol session semantics in x-kernel. It is unclear on how to extend the structure.	A node is a composite component and all modules in a node are components. A general node structure is developed in INET to facilitate constructing specific node structures.
Network and Protocol Models	Support major Internet protocols (IP, TCP, UDP, OSPFv2, BGP-4, and the pseudo-network interface drivers NIC, sockets), global Internet topology construction and IP address assignment.	Support, in addition to those supported by SSFNET (except BGP), also multicast routing protocols (such as DVMRP, MOSPF, CBT), QoS-driven protocols (such as QoS extension to OSPFv2, QoS extension to CBT, and various rate-based message scheduling disciplines), protocols in the integrated services architecture (such as RSVP signaling), components in the differentiated services architecture (marker/tagger at edge routers and buffer management algorithms at core routers).

• Fluid simulation also aims to improve scalability, but from a different perspective. It consists of two major approaches: (i) hierarchical fluid modeling: it abstracts a potentially complex component of the simulation by a simpler model and substitution of this simpler model into the larger simulation; (ii) modeling at multiple time scales: it simulates system behavior at coarser time-scales, allowing for significant savings in simulation execution times. Conceptually, network traffic is modeled as a continuous fluid model and time-driven simulation is used to discretize the continuous fluid mode and simulate the fluid backlogs (e.g., the back-logged fluid at a server, the departure process at a server) at discrete times. They prove that the discretization error is bounded by a constant proportional to the discretization interval length. Although conceptually innovative, it is not clear whether or not most, if not all, traffic can be well characterized in fluid models without losing modeling accuracy.
  
• Although the Ptolemy project of UC Berkeley is not focused on network simulation, it has also come to our attention because of its similarity to the autonomous component architecture in building component-based architectures for general-purpose simulation.

Ptolemy mainly focuses on modeling and simulation of concurrent, real-time, embedded systems, but shares similar philosophy with us in terms of component-based models. They use well-defined models of computation to govern the interaction between components. The models of computation they have developed include communicating sequential processes, continuous time, discrete events, distributed discrete events and so on. These models, however, have their own communication semantics. Components derived from different models can not be mixed together to form truly heterogeneous systems. J-Sim, on the other hand, has a simple, unified and concrete component model. Components developed in different domains of application can easily be integrated as long as the contracts are matched.

2 Quantitative Comparison

2.1 Experiment 1

• The number of the packets being transmitted over the bottleneck link (and therefore the number of packet events) is irrelevant to n.
• Event queue is at least n (TCP timeout event).
• Routing table size is (n+1).

1. Scenario setup time, often neglected, takes a non-negligible, and even larger portion of the entire time of a simulation run. For example, it takes about 393 seconds for ns-2 to set up the 4000 TCP connections scenario and takes about 181 seconds to complete a 1000-second simulation.
2. J-Sim scales better in terms of memory usage. While Javasim can run through 10,000 TCP connections without using up memory, ns-2 causes system to thrash at about 4,500 TCP connections, and SSFNET throws out-of-memory exception at about 3,310 TCP connections.
3. J-Sim suffers from inter-component communication overhead and JVM. And the real-time simulation engine incurs more overhead than the pure event simulation engine. The offsets of the completion time at small number of TCP connections in all figures are the evidence.
4. As for the scalability in terms of the completion time v.s. number of connections, J-Sim performs better than SSFNET as J-Sim's curves grow more slowly than SSFNET's. J-Sim also performs better than ns-2 when simulation time is 100 seconds, but worse than ns-2 when simulation time is 500 and 1000 seconds, before ns-2 uses up memory.

• J-Sim: (v1.0)
  • Java: OneBottleneck.java.

    Usage:
    	java OneBottleneck <#connections> <simulationTime> ?interval<nn>? ?event? ?radix?
    Argument:
    	interval<nn>: interval (in simulation time) to report wall time elapsed
    	event: use event simulation engine instead of real-time simulation engine (default)
    	radix: use radix-tree routing table instead of multifield routing table (default)
    Example:
    	java -Xms200000000 -Xmx200000000 OneBottleneck 100 1000 interval200 event radix 

  • Tcl: onebottleneck.tcl.

    Usage:
    	java drcl.ruv.System -u onebottleneck.tcl <#connections> <simulationTime> ?interval<nn>? ?event? ?radix?
    Argument:
    	See "Java" above.
    Example:
    	java -Xms200000000 -Xmx200000000 drcl.ruv.System -u onebottleneck.tcl 100 1000 interval200 event radix 

  • Python: onebottleneck.py.

    Usage:
    	java drcl.ruv.System -u -s drcl.ruv.ShellPython onebottleneck.py <#connections> <simulationTime> ?interval<nn>? ?event? ?radix?
    Argument:
    	See "Java" above.
    Example:
    	java -Xms200000000 -Xmx200000000 drcl.ruv.System -u -s drcl.ruv.ShellPython onebottleneck.py 100 1000 interval200 event radix 

• ns-2: (v2.1b8a)
  • Script: ns2_test.tcl.

    Usage:
    	ns2 ns2_test.tcl <#connections> <simulationTime>
    	

  • Modified source files: (Look for "Tyan" in the files for modifications)
    • ns-2.1b8a/tcl/lib/ns-lib.tcl: Added the "startTime_" argument in proceduer "Simulator instproc run" for calculating the additional setup time. So for ns-2, the scenario setup time is the sum of the time spent in the script before invoking "$ns_ run" and the setup time collected above.
    • ns-2.1b8a/scheduler.h and ns-2.1b8a/scheduler.cc: added fields and codes to collect simulation engine information such as number of events, maximum queue length, current queue length and so on.
• SSFNET: (v1.3)
  • Script: gen.tcl for generating the DML file.

    Usage:
    	tclsh gen.tcl <#connections>
    	

  • Makefile: executing the DML file generated by gen.tcl. Simulation time (1000 second) is hard coded in the Makefile.
  • Modified source files: (Look for "Tyan" in the files for modifications)
    • src/SSF/Net/Net.java: added the "MyTimer" inner class and codes for printing out execution profiles.