Distributed systems contain multiple hardware and software components
that can interact across multiple nodes/subsystems in sometimes
unforeseen and complicated ways. As a result, determining the root
cause of failures in these systems can be a very frustrating
experience that might take several hours or even days.
Problem diagnosis (or fingerpointing) involves instrumenting systems
to yield meaningful data, detecting errors and/or failures within
these systems, and ascertaining their root-cause, i.e., the underlying
fault. Fingerpointing is difficult because the distributed
interactions, protocols and inter-component dependencies in computer
systems can cause a problem to change ``shape'' or manifestation,
leading to potential red herrings in problem determination. There can
be many root causes of an outward manifestation of a problem and there
might be insufficient information to distinguish between the various
root causes. On the other hand, too much monitoring and too many
error messages might overwhelm the system, obscure the root cause, and
lead to increased latencies and additional resource costs.
We are currently developing a variety of techniques for automated
fingerpointing in distributed systems -- the aim is to perform online
and offline root-cause analyses in order to identify a faulty
node/process, diagnose the source of the problem, and report it to the
user or administrator in a meaningful/useful manner.
We ultimately aim for a preemptive strategy (where we need not wait
for any instability or problem to manifest into system-wide outage
before taking remedial action) that might improve the system's overall
responsiveness and availability. The idea is to observe the trends of
various key metrics in the system to ascertain which of these can be
good indicators of the overall health of the system, and which metrics
(if monitored appropriately and at the right frequency) could herald a
potential outage. Thus, our techniques aim for two key elements:
diagnosis of the root cause of the problem, and (where possible) a
proactive indication of an imminent critical problem in the system
that averts a total system failure.