A quick tutorial

SealTest allows a user to generate test sequences of a wide range of types, according to potentially unlimited coverage metrics, by defining very simple building blocks that interact with existing functions and algorithms.

Event Types

The basic classes in SealTest make no assumption on the type of event contained in a trace, or on the type of objects used as categories by the triaging function. To this end, the top-level definitions of triaging function and trace are all parameterized with generic types, generally called T for events and U for categories. This makes it possible for the user to define arbitrary objects as events (provided they inherit from the Event superclass), and similarly for triaging functions.

Currently, SealTest does provide one predefined type of event, called the AtomicEvent. An atomic event is represented by a single symbol, which can be any character string. Atomic events can be used in finite-state automata and in propositional Linear Temporal Logic.

Writing Specifications

Finite-State Machines

FSMs can be created programmatically, by manually adding states and transitions:

Automaton aut = new Automaton();
Vertex<AtomicEvent> v0 = new Vertex<>(0);
v0.add(new Edge<>(0, new AtomicEvent("a"), 1));
aut.add(v0);

Alternatively, SealTest also provides a parser that can build an automaton directly from a text file in GraphViz (DOT) format:

Scanner s = new Scanner(new File("fsm.dot"));
Automaton aut = Automaton.parseDot(s);

LTL Specifications

SealTest also provides ways of writing LTL formulæ:

Operator<AtomicEvent> op = new Globally<>(
  new Implies<>(
    new EventLeaf(new AtomicEvent("a")),
    new Next<>(
      new Or<>(new AtomicEvent("b"),
        new AtomicEvent("c"))
)));

Alternatively, SealTest also provides a parser that can build an LTL formula directly from a string:

String formula = "G (a -> (X (b | c)))";
AtomicParserBuilder parser = 
  new AtomicParserBuilder(formula);
Operator<AtomicEvent> op = parser.build();

Traces

SealTest provides AtomicEvent; in turn, a trace of atomic events is an AtomicTrace. A trace can be created programmatically, or parsed from a string; the following code snippet shows both methods. Since a trace is a list of events, its concents can be enumerated and accessed like any other Java list.

Trace<AtomicEvent> trace1, trace2;
trace1 = new AtomicTrace();
trace1.add(new AtomicEvent("a");
trace2 = AtomicTrace.readTrace("a,b,c");
for (AtomicEvent e : trace1) {
  System.out.println(e);
}

Generating Traces

Triaging Functions

A triaging function is…

Automata-Based Functions

Given a FSM specification, SealTest already provides a number of built-in triaging functions. For example, to classify a trace according to their state shallow history of length 2 with respect to some FSM, one writes:

Automaton aut = ...
Trace t = ...
StateShallowHistory f = 
  new StateShallowHistory(aut, 2);
MathSet<Integer> category = f.getStartClass();
for (AtomicEvent e : trace1) {
  category = f.processTransition(e);
}

At the exit of the loop, variable category contains the category in which f places the trace.

Hologram-Based Functions

The same applies to LTL-based specifications. The following code example evaluates an LTL formula on a trace, and then applies a hologram transformation on the resulting evaluation tree.

Operator<AtomicEvent> op = ...
Trace t = ...
for (AtomicEvent e : t)
  op.evaluate(ae);
FailFastDeletion<AtomicEvent> f = 
  new FailFastDeletion<>();
Operator<> new_op = f.transform(op);

In addition, SealTest allows the hologram to be exported as a picture, by producing a text file in Graphviz (DOT) format:

GraphvizHologramRenderer<AtomicEvent> renderer 
  = new GraphvizHologramRenderer<AtomicEvent>();
new_op.acceptPrefix(renderer, true);
System.out.println(renderer.toDot());

User-Defined Functions

One simply needs to create a new class that extends TriagingFunction; for example, here is a simple triaging function that categorizes each trace with respect to the number of events named a it contains:

public class MyFunction 
extends TriagingFunction<AtomicEvent,Integer> {
  // Number of a's
  int num_a = 0;
  
  public Set<Integer> getStartClass() {
    return new MathSet<Integer>(0);
  }
  public Set<U> read(AtomicEvent e) {
    if (e.getLabel().compareTo("a") == 0)
      num_a++;
    return new MathSet<Integer>(num_a);
  }
  public void reset() {
    num_a = 0;
  }
}

Coverage Metrics

TODO

Trace Generation

Cayley Graph Method

A first method to generate traces is by exploiting the Cayley graph of a triaging function. Here, T is the generic type of the events inside a sequence, and U is the type of the categories returned by the function. The new class needs to implement getStartCategory(), which returns the equivalence class of the empty trace, and processTransition(), which returns the equivalence class of the current sequence to which a new event is to be appended. Computing the Cayley graph associated to a function, as well as computing its prefix closure, can also be easily done using predefined objects:

TriagingFunction&ltT,U> f = ...
CayleyGraph&ltT,U> g = f.getCayleyGraph();
PrefixClosure&ltT,U> closure 
  = new PrefixClosure<>(graph);
CayleyGraph&ltT,U> closure_graph =  closure.getCayleyGraph();

Note that the function can define its Cayley graph directly by overriding the getCayleyGraph() method; otherwise it defaults to calling a generic graph exploration algorithm. Finally, generating a set of test sequences from a graph is also simple; for example, the following syntax shows how to use the hypergraph/Steiner tree algorithm to generate a set of sequences from an existing Cayley graph: %

TraceGenerator gen 
  = new HypergraphTraceGenerator<>(g);
Set> traces = gen.generateTraces();

Greedy Random Algorithm

SealTest also provides an alternate method of generating traces, using a random algorithm. This algorithm requires a coverage metric, an alphabet of possible events and a random number generator:

Automaton aut = ...
CoverageMetric metric = ...
Random rand = new Random();
GreedyAutomatonGenerator gen = 
  new GreedyAutomatonGenerator<>(aut, rand, metric);
Set> traces = gen.generateTraces();

Test Driver and Test Hooks

A test hook is an object that receives events and is responsible for executing them on some system under test. Suppose for example that we want to test a Microwave object that has methods like start, stop, open, setFood, etc. We can create a simple TestHook object that receives atomic events, and depending on their name, makes a specific method call on a Microwave object it is given. The resulting hook could look like this:

class MicrowaveHook
  implements TestHook {
  
  Microwave oven;
  
  public MicrowaveHook(Microwave o) {
    oven = o;
  }
  public Object execute(AtomicEvent event) {
    String event_name = event.getLabel();
    if (event_name.compareTo("start") == 0)
      oven.start();
    if (event_name.compareTo("stop") == 0)
      oven.stop();
    ...
    return null;
  }
}

Obviously, a test hook can be programmed to do other tasks, such as printing the event, logging it into a database, sending an HTTP request to some online system, etc. A test hook is used in conjunction with a test driver. The driver is given a set of traces, and executes each event they contain by calling the test hook. This set of traces can be built by hand, or, obviously, be constructed by a trace generator as described earlier.

TestSequenceGenerator gen = ...
TestSuite<AtomicEvent> suite = gen.generateTraces();
UnidirectionalTestDriver driver 
  = new UnidirectionalTestDriver<>();
driver.setTestSuite(suite);
MicrowaveHook hook = new MicrowaveHook(new Microwave());
driver.setHook(hook);
driver.run();

Since a driver implements Java's Runnable interface, its interaction with the SUT can be placed inside a separate thread.