Ruling on Reliability and Relevance in a Daubert
The Methodology-Conclusion Debate and Other Issues
by Eric D. Brown, Blake Snider, and Victor Svilik - Harvard Law
The Gatekeeping Role
The Daubert opinion fundamentally altered
the structure of judicial decision-making in regard to the admissibility
of expert testimony. Under the Frye general acceptance test, which Daubert
superseded (or, more precisely, according the ruling of Federal Rules of
Evidence superseded), the judge did not, herself, assess the reliability
of an expert's opinion. Instead she judged whether the expert was qualified
and the opinion relevant an only touched upon the reliability through inquiring
whether others in the expert's field considered the methodology used by
the expert to be reliable.
Daubert changed this by placing the responsibility
to make a reliability determination squarely on the shoulders of the trial
judge. In a post-Daubert setting, the Frye inquiry is merely
one component to which a judge may look in making a decision whether or
not an expert's methodology is reliable and his opinion ultimately admissible.
This critical structural change is what places judges in a gatekeeping
The most commonly heard objection from judges
to this role is that it forces them to perform a function for which they
are not well suited. After all, many contend, judges are not amateur scientists.
While this is a valid objection to some extent, it may be overstating the
case. In fact,
Daubert does not ask judges to cull through scientific
opinions and pronounce a select few to be "good science." Rather, the gatekeeping
framework asks judges to filter out testimony that falls outside the bounds
of acceptable reliability. Certainly Daubert requires more of a
judge than the
Frye test did, but the judicial role becomes less
daunting when it is seen as requiring an exclusionary function, as filtering
out "bad science" rather than anointing "good science." In this sense,
the judge need only determine whether there are holes in the expert's reasoning,
a task that can be carried out successfully by an informed non-scientist.
Taking this as the task of a judge when confronted
with expert testimony, how should she approach the two-pronged Daubert
test of reliability and relevance? The first prong concerns the "scientific...knowledge"
component of Federal Rule of Evidence (FRE) 702 and asks judges to examine
the methodology employed by the expert witness.(1)The
second prong, often referred to as "fit," requires that the expert's testimony
"assist the trier of fact to understand the evidence or to determine a
fact in issue."(2)
Both reliability and relevance must be established for expert testimony
to be admissible.
At the heart of a Daubert hearing is
the question of the reliability of the expert's scientific testimony. To
aid judges in making this determination,
Daubert sets forth a non-exhaustive
list of factors to be used by judges to determine the reliability of an
expert's methodology. In addition to the familiar
analysis, the Court also notes that important inquiries include (1) whether
the methodology has been tested, (2) whether it has been subjected to peer
review and publication, and (3) the known or potential rate of error during
Courts in various jurisdictions have added to this list or focused on one
or more components of the analysis. In any case, noncompliance with one
or more of these (or other court-established) factors does not necessarily
invalidate the expert's methodology, so long as the court is satisfied
that the overall processes constitute reliable grounds upon which the results
The Methodology-Conclusion Debate
Ruling on the reliability of an expert's opinion
presents difficulties when the so-called methodology-conclusion debate
is taken into consideration. Justice Blackmun's opinion in Daubert
asserts that when determining the scientific validity of proffered testimony,
"the focus, of course, must be solely on principles and methodology, not
on the conclusions that they generate."(4)
On its face, this analysis seems to suggest that a trial judge should not
base her decision on an evaluation of the conclusions proposed by an expert.
This may seem clear cut, but in reality differences of opinion exist in
where to draw the line between the methodology used by an expert and the
conclusions reached. The methodology-conclusion debate takes on three different
(1) In one version, a judge struggles with
where her gatekeeping inquiry should fall along the line of generality/particularity.
At one extreme, the argument is that judges should inquire only at a general
level about the reliability of the expert's methodology. Such an inquiry
would be satisfied, for example, by evidence that the expert used methodology
generally accepted in the field of epidemiology. But does this collapse
into the defunct Frye test? Doesn't Daubert seem to require
a deeper inquiry? At the opposite pole, a judge's inquiry would entail
examination of every step in the expert's process of gathering data and
reasoning to a conclusion. This very particular examination of an expert's
methodology would look at every step leading up to, but not including,
the conclusion. Such an examination would mirror the evaluative inquiry
made by the most demanding peer review journals. Should judges and scientific
journals utilize the same standards in reviewing studies?
In Hall v. Baxter Healthcare Corporation,
Judge Jones of Oregon acknowledged that
Daubert limits the judge's
evaluation to the methodologies used by an expert, yet he later stated
that there appears to be no real demarcation between the scientific methodology
used and the conclusions that subsequently arise.(5)
Jones ultimately determined that all steps of an expert's reasoning process
must be evaluated, which includes the formulation of conclusions.(6)
Jones' decision appears concerned with situations in which the underlying
methodology does not support the expert's offered conclusions. An example
of a conclusion that does not follow from a given methodology arose in
re Paoli R.R. Yard PCB Litigation, in which the court excluded
an expert's testimony because the conclusions proffered were beyond the
scope of the methodology.(7)
In this case, where the expert's methodology required her to perform physical
examinations of the patients, she only examined two of them, yet tried
to apply her conclusions to all of the plaintiffs.(8)
(2) A second version of the controversy involves
looking at the conclusions reached not for their merit but as a means of
judging how particular the examination of an expert's methodology should
be. If an expert offers a conclusion that is at odds with the overwhelming
majority of similar studies in a field, can the judge consider this fact
to be a reason to take an especially careful look at the expert's methodology?
In this respect, a judge can only look at an expert's conclusion in order
to focus the inquiry into methodology, but not to throw the testimony out
solely because of the outlying conclusion.
(3) The third version of the debate involves
actually looking at the conclusion as a part of the methodology. The argument
is that conclusion is inseparable from methodology in that an expert's
choice of methodology and application of it determines the expert's conclusion.
According to this view, an expert who offers a conclusion that is at odds
with the weight of opinion in his field must explain why this is so as
part of the testimony offered. If this explanation is missing, the testimony
can be deemed unreliable. As such, an un-reconciled outlying conclusion
could automatically invalidate expert testimony. The Ninth Circuit in Lust
v. Merrell Dow Pharmaceuticals, Inc. indicated that, at the very least,
the proponent of expert testimony bears the burden of proving admissibility,
and, if few others share an expert's conclusions the court can use that
fact as a basis for excluding the testimony.(9)
Is this crossing Justice Blackmun's boundary between methodology and conclusion?
Reliability Determination by
the Judge or the Jury?
If a judge in a Daubert hearing makes reliability
determinations, a jury will never see the evidence offered. Should a judge
withhold testimony from the jury because an expert's conclusion is inexplicably
at odds with other studies in the same field or if the expert's methodology
is "questionable"? Or should the jury be allowed to consider this testimony
given complete information, for example with a caveat that an expert reached
a conclusion different from the thirty other studies in the expert's field?
One argument is that a conclusion that seems unpersuasive or novel does
not necessarily indicate that the expert's methodology was flawed in some
respect; such an evaluation, in any case, should be a question for the
jury, not the judge.(10)
The language in
Daubert notes that a methodology accepted by even
a minority of the scientific community may be admissible, as long as it
has some demonstrable basis of reliability. In the recent Oregon case Hall
v. Baxter Healthcare Corporation (involving claims against breast implant
manufacturers for injuries allegedly caused by silicone gel breast implants),
Judge Jones recognized that, in making an assessment of reliability, the
fact that an expert's conclusions differ from those of other experts is
not a sufficient basis alone for determining that the testimony is inadmissible.(11)
Those who wish to curtail the gatekeeping role
of judges often speak in terms of protecting the province of the jury.
If, they argue, a judge admits testimony over the objections of the other
side, opposing counsel still has adequate safeguards, such as cross-examination
and contrary evidence, to protect against conclusions perceived to be in
Those who construe Justice Blackmun's delineation between methodology and
conclusion as inviolable note that by limiting review to methodology, the
inquiry eliminates the temptation for judges to take an expert's credibility
into account, leaving this determination properly in the hands of the jury.(13)
According to this interpretation, the focus on methodology acts as a check
on the judge's gatekeeping power, ensuring the accepted division of authority
and protecting the plaintiff's right to his day in court.
Daubert's second prong, relevancy,
has received considerably less attention from commentators than the question
of reliability. Nonetheless, this dearth of scholarly treatment does not
indicate that the second prong is of any lesser importance, for the evaluation
of fit, like the reliability inquiry, requires a high degree of judicial
scrutiny. A determination of fit requires that a judge decide whether or
not proffered testimony will assist the trier of fact in drawing a conclusion.
This necessitates a determination as to whether or not testimony produced
is relevant to the facts at issue in a case. Justice Blackmun's opinion
in Daubert provides a useful example of how the question of relevance
comes into play. According to Justice Blackmun's hypothetical, while expert
testimony about the phases of the moon may be applicable in determining
how dark it was on a given night, the very same testimony should be inadmissible
if offered to demonstrate the psychological state of a defendant.(14)
Thus, an expert's opinion can be methodologically flawless yet it can still
be dismissed for failure to reach a reasonable conclusion. A judge, who
chooses to review how an expert reached his conclusion via his methodology,
as informed by the methodology-conclusion debate discussed above, could
well throw out on the grounds of reliability, testimony about the moon
aimed at proving psychological state. This ruling could be based on a belief
that the expert improperly reasoned to his conclusion, and underlying this
would be an assumption that this reasoning is part of the expert's methodology.
But what if the expert doesn't offer a conclusion but merely lays out scientific
evidence about the phases of the moon? While the expert himself draws no
conclusion about the effect of the phases of the moon on the psychological
state of the defendant, the lawyer for the defense will attempt to use
the testimony to this effect. A judge in her gatekeeping role can throw
out this testimony for a failure to "fit" the causation requirements of
the case at hand.
Beyond Rule 702
Unfortunately, questions of relevance rarely
present themselves in a form as simple as Justice Blackmun's example. Courts
must struggled to delineate how closely proffered scientific testimony
must be related to the facts at bar. Although Daubert focuses on
Rule 702 as the "primary locus" of a court's gatekeeping role, judges are
not precluded from looking to other rules for guidance.(15)
Some judges have relied on Rule 403 in determining what evidence will "assist
the trier of fact." Rule 403 establishes a balancing test weighing the
probative value of the evidence against "the danger of unfair prejudice,
confusion of the issues, or misleading the jury...." Going back to the
phases of the moon example, such testimony could be excluded on Rule 403
grounds that it has a high potential to mislead the jury even if the expert
did not offer a specific conclusion.
The Judge-Jury Question
Implicit in an admissibility decision is the
potential danger that by excluding the evidence, the judge may intrude
into the province of the jury. In exercising his gatekeeping role, the
judge has predetermined that this evidence cannot--under any circumstances--support
a verdict for the plaintiff. Is such a consideration proper, or should
the judge allow the evidence to reach the jury, trusting that the opposing
party's ability to impeach the testimony through cross-examination will
preclude an unjustified verdict? In deciding what testimony could "assist
the trier of fact," judges may choose, however, to look beyond the simple
question of whether or not the testimony fits the facts. If a scientific
study which indicates that the causal agent at issue keeps the relative
risk less than two (double the risk), a judge may choose to exclude the
study on grounds that such evidence would meet a "more likely than not"
standard of proof.(16)
A problem with exclusion on this basis is that while such a study alone
may not meet the requisite level of legal proof, it may be valuable in
conjunction with other evidence in the aggregate to meet that standard.
In excluding testimony about tests that fall below a certain level of probability,
a judge may risk denying a jury the opportunity to consider important parts
of a causation puzzle.
Judges acting as evidentiary gatekeepers are
bound by the two-pronged test set out in Daubert. In examining expert
testimony for reliability, judges first face the hurdle of distinguishing
between methodology and conclusion and choosing how far to extend their
reliability inquiry. Beyond this, even if testimony passes the reliability
test, a judge may exclude it for failing to meet the relevancy standard.
Analysis of fit gives judges a degree of leeway in assessing the validity
of applying an expert's conclusion to the facts at hand. The challenge,
then, of conducting a Daubert hearing lies in balancing an understanding
of the Daubert holding, a court's interest in judicial efficiency,
the parties' rights to due process, the jury's role as fact-finder, and
the judge's role as evidentiary gatekeeper.
v. Merrell Dow Pharmaceuticals, Inc. 113 S.Ct. 2786, 2795 (1993).
R. Evid. 702.
S.Ct. at 2797.
S.Ct. 2786, 2797.
F.Supp. 1387, 1387 (1996).
re Paoli R.R. Yard PCB Litigation, 35 F.3d. 717 (3rd Cir. 1994).
v. Merrell Dow Pharmaceuticals, Inc., 89 F.3d 594, 598 (9th Cir. 1996).
J. Chesebro, Taking Daubert's Focus Seriously: The Methodology/Conclusion
Distinction, 15 Cardozo L. Rev. 1745, 1749 (1994).
F.Supp. at 1387 (1996).
S.Ct. at 2798.
Black, Science and the Law in the Wake of Daubert: A New Search
for Scientific Knowledge, 72 Tex. L. Rev. 715, 749-750 (1994).
S.Ct. at 2796.
S.Ct. at 2795.
E. Bernstein, The Admissibility of Scientific Evidence After
v. Merrell Dow Pharmaceuticals, Inc., 15 Card. L. Rev. 2139, 2170