>Peter Dolce hit the nail on the head.  It's the protection factor (compliance)
>that is so difficult to assess......

The "protection factor" has always been a significant measurement
issue and challenge in health, safety, law enforcement, and many other fields.
You can't count stuff that doesn't happen, so you're stuck with counting stuff
that does happen, and that might serve as a proxy for measuring what's been
avoided. Accident rates, insurance claims, mortality rates, etc have long been
used as measures of effectiveness. Similarly, in research administration it may
be possible to measure effectiveness using these types of indicators (e.g. audit
disallowance trends might be an example of such a measure). The BIGGEST problem
with these (and all types of effectiveness) measures is not developng an
indicator, but proving causality. That is, are we getting better because of our
intervention? Or is it because there are other factors at work? These questions
require a much different type of evaluation. There's no magic bullet in this or
any other measurement area.

===========================================================================

Bill Kirby
NSF

xxxxxx@nsf.gov
703-306-1102