We can evaluate empirically whether, in practice, getting slightly
more or fewer execution traces would have significantly changed which
dependencies were detected in execution traces gathered from
Phoenix. We do so by seeing how many of the dependencies would not
have been detected if the counts in row one in the contingency table
varied by a small amount. For example, if the contingency table in
Table 1 had [52,35,240,643] instead of
[52,33,240,643], then G=40.42, which is not much different than the
value for Table 1 of G=42.86. To determine whether
the dependencies detected in the execution traces are vulnerable to
noise, we can do the following test: 1) construct the contingency
table for dependencies detected in execution traces, 2) vary, one at a
time, the counts of row one, column one and row one, column two by
, and 3) run a G-test on the resulting contingency tables. By
tweaking the contingency table cell values in this manner, we check
the sensitivity of G to noise in the data. Both columns of the first
row were varied because some of the first column counts were 1, which
makes it impossible to test whether a lower ratio of first column to
second column might not have influenced the value of G more than a
higher ratio. We tweaked the counts by because many contingency
tables contained cell counts of less than 5, varying by spans
that range.
Table 2: Dependencies remaining after tweaking contingency tables. The
table includes the number of dependencies remaining after tweaking
over the total number of dependencies found in the execution traces
from the four experiments.
Table 2 shows how many of the dependencies found in each
of four sets of execution traces for Phoenix would remain if their
contingency tables are so tweaked. About 65% of the dependencies
remain after tweaking their contingency table values, meaning that the
counts in the first row of the contingency table can be changed by
without dropping the significance of G below the level of
. So, 35% of the dependencies detected would disappear if a
few patterns more or less were included in the execution traces. Based
on this testing of execution traces from Phoenix, dependency detection
is sensitive to small differences in the content of the execution
traces. Most of the dependencies that were vulnerable to the tweaking
were based on execution traces that included few instances of the
precursor/failure pattern, 23 out of 44 or 52% of the dependencies
that disappeared were based on contingency tables in which one of the
counts in the first row was less than five.
The implication of the sensitivity of dependency detection to noise in the execution traces is that rare patterns are especially sensitive to noise and so should be viewed skeptically. One must interpret the results of dependency detection with care: if ``sensitive" dependencies are discarded, then rare events may remain undetected; at the same time, one does not wish to chase chimeras. Interpreting dependencies requires weighing false positives against misses. If we are trying to identify dependencies between precursors that occur rarely or failures that occur rarely, then additional effort should be expended to get enough execution traces to ensure that the dependency is not due to noise.