Virtual Test and Evaluation Facility
One may be faced with the problem of making a definite decision with respect
to an uncertain hypothesis which is known only through its observable
consequences. A statistical hypothesis test, or more briefly, hypothesis test,
is an algorithm to state the alternative (for or against the hypothesis) which
minimizes certain risks.
Evaluation is the systematic determination of merit, worth, and significance
of something or someone. Evaluation often is used to characterize and appraise
subjects of interest in a wide range of human enterprises, including the Arts,
business, computer science, criminal justice, engineering, foundations and
non-profit organizations, government, health care, and other human services.
A test method is a definitive procedure that produces a test result. (ASTM
definition)
The test result can be qualititive (yes/no), categorical, or quantititive (a
measured value). It can be a personal observation or the output of a precision
instrument.
Usually the test result is the dependent variable, the measured response based
on the particular conditions of the test or the level of the independent
variable. Some tests, however, involve changing the independent variable to
determine the level at which a certain response occurs: in this case, the test
result is the independent variable.
Importance of test methods
In engineering, science, manufacturing, and business, it is vital for all
interested people to understand and agree upon methods of obtaining data and
making measurements. This is to fully document experiments and measurements and
to provide needed definition to specifications and contracts.
A well written test method is important. However, even more important is
choosing a method of measuring the correct property or characteristic. Not all
tests and measurements are equally useful: usually a test result is used to
predict or imply sutiability for a certain purpose. For example if a
manufactured item has several components, test methods may have several levels
of connections:
* test results of a raw material should connect with tests of a component made
from that material
* test results of a component should connect with performance testing of a
complete item
* results of laboratory performance testing should connect with field
performance
* etc
These connections or correlations may be based on published literature,
engineering studies, or formal programs such as quality function deployment.
Validation of the suitability of the test method is often required.
Content of a test method
The test method might include:
* Descriptive title and scope of the test method
* Date of last effective revision and revision designation
* Person, office, or agency responsible for questions on the test method,
updates, and deviations.
* The significance or importance of the test method and its intended use.
* Terminology and definitions to clarify the meanings of the test method
* A listing of the types of apparatus (and the specific device) required to
conduct the test
* Safety precautions
* Environmental concerns and considerations
* Sampling procedures. How are samples to be obtained. Number of samples.
* Conditioning: temperature, humidity, etc, and Tolerance (engineering)
* Preparation of samples for the test
* Detailed procedure for conducting the test
* Calculations and analysis of data
* Interpretation of data and test method output
* Report: format, content, data, etc
* Accuracy, precision, bias, repeatability, reproducibility, and uncertainty of
test results
Design of experiments includes the design of all information-gathering exercises
where variation is present, whether under the full control of the experimenter
or not. (The latter situation is usually called an observational study.) Often
the experimenter is interested in the effect of some process or intervention
(the 'treatment') on some objects (the 'experimental units'), which may be
people. Design of experiments is thus a discipline that has very broad
application across all the natural and social sciences.
Early examples of experimental design
In 1747, while serving as surgeon on HM Bark Salisbury, James Lind, the ship's
surgeon, carried out a controlled experiment to discover a cure for scurvy.
Lind selected 12 men from the ship, all suffering from scurvy, and divided them
into six pairs, giving each group different additions to their basic diet for a
period of two weeks. The treatments were all remedies that had been proposed at
one time or another. They were
* A quart of cider per day
* Twenty five gutts of exilir vitriol three times a day upon an empty stomach,
* Half a pint of seawater every day
* A mixture of garlic, mustard and horseradish, in a lump the size of a nutmeg
* Two spoonfuls of vinegar three times a day
* Two oranges and one lemon every day.
The men who had been given citrus fruits recovered dramatically within a week.
One of them returned to duty after 6 days and the other became nurse to the
rest. The others experienced some improvement, but nothing was comparable to the
citrus fruits, which were proved to be substantially superior to the other
treatments.
In this study his subjects' cases "were as similar as I could have them", that
is he provided strict entry requirements to reduce extraneous variation. The men
were paired, which provided replication. From a modern perspective, the main
thing that is missing is randomized allocation of subjects to treatments.
A formal mathematical theory
The first statistician to consider a formal mathematical methodology for the
design of experiments was Sir Ronald A. Fisher. As an example, he described how
to test the hypothesis that a certain lady could distinguish by flavor alone
whether the milk or the tea was first placed in the cup. While this sounds like
a frivolous application, it allowed him to illustrate the most important means
of experimental design:
1. Comparison
In many fields of study it is hard to reproduce measured results exactly.
Comparisons between treatments are much more reproducible and are usually
preferable. Often one compares against a standard or traditional treatment that
acts as baseline.
2. Randomization
There is an extensive body of mathematical theory that explores the consequences
of making the allocation of units to treatments by means of some random
mechanism such as tables of random numbers, or the use of randomization devices
such as playing cards or dice. Provided the sample size is adequate, the risks
associated with random allocation (such as failing to obtain a representative
sample in a survey, or having a serious imbalance in a key characteristic
between a treatment group and a control group) are calculable and hence can be
managed down to an acceptable level. Random does not mean haphazard, and great
care must be taken that appropriate random methods are used.
3. Replication
Where measurement is made of a phenomenon that is subject to variation it is
important to carry out repeat measurements, so that the variability associated
with the phenomenon can be estimated.
4. Blocking
Blocking is the arrangement of experimental units into groups (blocks) that are
similar to one another. Blocking reduces known but irrelevant sources of
variation between units and thus allows greater precision in the estimation of
the source of variation under study.
5. Orthogonality
Orthogonality concerns the forms of comparison (contrasts) that can be
legitimately and efficiently carried out. Contrasts can be represented by
vectors and sets of orthogonal contrasts are uncorrelated and independently
distributed if the data are normal. Because of this independence, each
orthogonal treatment provides different information to the others. If there are
T treatments and T - 1 orthogonal contrasts, all the information that can be
captured from the experiment is obtainable from the set of contrasts.
6. Use of factorial experiments instead of the one-factor-at-a-time method.
These are efficient at evaluating the effects and possible interactions of
several factors (independent variables).
Analysis of the design of experiments was built on the foundation of the
analysis of variance, a collection of models in which the observed variance is
partitioned into components due to different factors which are estimated and/or
tested.
Some efficient designs for estimating several main effects simultaneously were
found by Raj Chandra Bose and K. Kishen in 1940 at the Indian Statistical
Institute, but remained little known until the Plackett-Burman designs were
published in Biometrika in 1946.
In 1950, Gertrude Mary Cox and William Cochran published the book Experimental
Designs which became the major reference work on the design of experiments for
statisticians for years afterwards.
Developments of the theory of linear models have encompassed and surpassed the
cases that concerned early writers. Today, the theory rests on advanced topics
in abstract algebra and combinatorics.
As with all other branches of statistics, there is both classical and Bayesian
experimental design.
Example
This example is attributed to Harold Hotelling in . Although very simple, it
conveys at least some of the flavor of the subject.
The weights of eight objects are to be measured using a pan balance that
measures the difference between the weight of the objects in the two pans. Each
measurement has a random error. The average error is zero; the standard
deviations of the probability distribution of the errors is the same number σ on
different weighings; and errors on different weighings are independent. Denote
the true weights by
We consider two different experiments:
1. Weigh each object in one pan, with the other pan empty. Call the measured
weight of the ith object Xi for i = 1, ..., 8.
2. Do the eight weighings according to the following schedule and let Yi be the
measured difference for i = 1, ..., 8:
Then the estimated value of the weight θ1 is
The question of design of experiments is: which experiment is better?
The variance of the estimate X1 of θ1 is σ2 if we use the first experiment. But
if we use the second experiment, the variance of the estimate given above is
σ2/8. Thus the second experiment gives us 8 times as much precision.
Many problems of the design of experiments involve combinatorial designs, as in
this example.
Observational error is the difference between a measured value of quantity and
its true value. In statistics, an error is not a "mistake". Variability is an
inherent part of things being measured and of the measurement process.
When either randomness or uncertainty modeled by probability theory is
attributed to such errors, they are "errors" in the sense in which that term is
used in statistics; see errors and residuals in statistics.
Every time we repeat a measurement with a sensitive instrument, we obtain
slightly different results. The common statistical model we use is that the
error has two additive parts:
1. systematic error which always occurs (with the same value) when we use the
instrument in the same way, and
2. random error which may vary from observation to observation.
The systematic error is sometimes called statistical bias. It is controlled by
very carefully standardized procedures. Part of the education in every science
is how to use the standard instruments of the discipline.
The random error (or random variation) is due to factors which we cannot (or do
not) control. It may be too expensive or we may be too ignorant of these factors
to control them each time we measure. It may even be that whatever we are trying
to measure is changing in time (see dynamic models), or is fundamentally
probablistic (as is the case in quantum mechanics -- see Measurement in quantum
mechanics). Random error often occurs when instruments are pushed to their
limits. For example, it is common for digital balances to exhibit random error
in their least significant digit. Three measurements of a single object might
read something like 0.9111g, 0.9110g, and 0.9112g.
In the scientific method, an experiment (Latin: ex- periri, "of (or from)
trying") is a set of observations performed in the context of solving a
particular problem or question, to support or falsify a hypothesis or research
concerning phenomena. The experiment is a cornerstone in the empirical approach
to acquiring deeper knowledge about the physical world.
Design of experiments
Design of experiments
1. That the independent variable is the only factor that varies systematically
in the experiment; in other words, that the experiment is appropriately
controlled - that confounding variables are eliminated; and
2. That the dependent variable truly reflects the phenomenon under study (a
question of validity) and that the variable can be measured accurately (i.e.,
that various types of experimental error, such as measurement error can be
eliminated).
In a pure application of the scientific method, hypotheses are tested by
critical experiments: ones that can falsify the hypothesis in the case of a
non-result (i.e., an experiment showing that the independent variable did not
affect the dependent variable as predicted). Such pure applications are rare,
however, in part because a result can sometimes be challenged on the basis that
an experiment was not sufficiently controlled, that the dependent variable was
not valid, or that various forms of error compromised the experiment. The
scientific method, as a result, builds in the need for reproducibility (usually
termed replication) and convergent evidence (see also: external validity).
The design of experiments attempts to balance the requirements and limitations
of the field of science in which one works so that the experiment can provide
the best conclusion about the hypothesis being tested. In some sciences, such as
physics and chemistry, it is relatively easy to meet the requirements that all
measurements be made objectively, and that all conditions can be kept controlled
across experimental trials. On the other hand, in other cases such as biology,
and medicine, it is often hard to ensure that the conditions of an experiment
are performed consistently; and in the social sciences, it may even be difficult
to determine a method for measuring the outcomes of an experiment in an
objective manner.
For this reason, sciences such as physics and several other fields of natural
science are sometimes informally referred to as "hard sciences", while social
sciences are sometimes informally referred to as "soft sciences"; in an attempt
to capture the idea that objective measurements are often far easier in the
former, and far more difficult in the latter.
In addition, in the social sciences, the requirement for a "controlled
situation" may actually work against the utility of the hypothesis in a more
general situation. When the desire is to test a hypothesis that works "in
general", an experiment may have a great deal of internal validity, in the sense
that it is valid in a highly controlled situation, while at the same time lack
external validity when the results of the experiment are applied to a real world
situation. One of the reasons why this may happen is the Hawthorne effect;
another is that partial equilibrium effects may not persist in general
equilibrium.
As a result of these considerations, experimental design in the "hard" sciences
tends to focus on the elimination of extraneous effects, while experimental
design in the "soft" sciences focuses more on the problems of external validity,
often through the use of statistical methods. Occasionally events occur
naturally from which scientific evidence can be drawn, which is the basis for
natural experiments. In such cases the problem of the scientist is to evaluate
the natural "design".
Controlled experiments
Experimental control
Many hypotheses in sciences such as physics can establish causality by noting
that, until some phenomenon occurs, nothing happens; then when the phenomenon
occurs, a second phenomenon is observed. But often in science, this situation is
difficult to obtain.
For example, in the old joke, someone claims that they are snapping their
fingers "to keep the tigers away"; and justifies this behavior by saying "see -
its working!" While this "experiment" does not falsify the hypothesis "snapping
fingers keeps the tigers away", it does not really support the hypothesis - not
snapping your fingers keeps the tigers away as well.
To demonstrate a cause and effect hypothesis, an experiment must often show
that, for example, a phenomenon occurs after a certain treatment is given to a
subject, and that the phenomenon does not occur in the absence of the treatment.
(See Baconian method.)
Enlarge picture
Standard curve
A controlled experiment generally compares the results obtained from an
experimental sample against a control sample, which is practically identical to
the experimental sample except for the one aspect whose effect is being tested.
A good example would be a drug trial. The sample or group receiving the drug
would be the experimental one; and the one receiving the placebo would be the
control one. In many laboratory experiments it is good practice to have several
replicate samples for the test being performed and have both a positive control
and a negative control. The results from replicate samples can often be
averaged, or if one of the replicates is obviously inconsistent with the results
from the other samples, it can be discarded as being the result of an
experimental error (some step of the test procedure may have been mistakenly
omitted for that sample). Most often, tests are done in duplicate or triplicate.
A positive control is a procedure that is very similar to the actual
experimental test but which is known from previous experience to give a positive
result. A negative control is known to give a negative result. The positive
control confirms that the basic conditions of the experiment were able to
produce a positive result, even if none of the actual experimental samples
produce a positive result. The negative control demonstrates the base-line
result obtained when a test does not produce a measurable positive result; often
the value of the negative control is treated as a "background" value to be
subtracted from the test sample results. Sometimes the positive control takes
the form of a standard curve.
An example that is often used in teaching laboratories is a controlled protein
assay. Students might be given a fluid sample containing an unknown (to the
student) amount of protein. It is their job to correctly perform a controlled
experiment in which they determine the concentration of protein in fluid sample
(usually called the "unknown sample"). The teaching lab would be equipped with a
protein standard solution with a known protein concentration. Students could
make several positive control samples containing various dilutions of the
protein standard. Negative control samples would contain all of the reagents for
the protein assay but no protein. In this example, all samples are performed in
duplicate. The assay is a colorimetric assay in which a spectrophotometer can
measure the amount of protein in samples by detecting a colored complex formed
by the interaction of protein molecules and molecules of an added dye. In the
illustration, the results for the diluted test samples can be compared to the
results of the standard curve (the blue line in the illustration) in order to
determine an estimate of the amount of protein in the unknown sample.
Controlled experiments can be performed when it is difficult to exactly control
all the conditions in an experiment. In this case, the experiment begins by
creating two or more sample groups that are probabilistically equivalent, which
means that measurements of traits should be similar among the groups and that
the groups should respond in the same manner if given the same treatment. This
equivalency is determined by statistical methods that take into account the
amount of variation between individuals and the number of individuals in each
group. In fields such as microbiology and chemistry, where there is very little
variation between individuals and the group size is easily in the millions,
these statistical methods are often bypassed and simply splitting a solution
into equal parts is assumed to produce identical sample groups.
Once equivalent groups have been formed, the experimenter tries to treat them
identically except for the one variable that he or she wishes to isolate. Human
experimentation requires special safeguards against outside variables such as
the placebo effect. Such experiments are generally double blind, meaning that
neither the volunteer nor the researcher knows which individuals are in the
control group or the experimental group until after all of the data has been
collected. This ensures that any effects on the volunteer are due to the
treatment itself and are not a response to the knowledge that he is being
treated.
In human experiments, a subject (person) may be given a stimulus to which he or
she should respond. The goal of the experiment is to measure the response to a
given stimulus.
Natural experiments
Natural experiment
The term "experiment" usually implies a controlled experiment, but sometimes
controlled experiments are prohibitively difficult or impossible. In this case
researchers resort to natural experiments, also called quasi-experiments.
Natural experiments rely solely on observations of the variables of the system
under study, rather than manipulation of just one or a few variables as occurs
in controlled experiments. To the degree possible, they attempt to collect data
for the system in such a way that contribution from all variables can be
determined, and where the effects of variation in certain variables remain
approximately constant so that the effects of other variables can be discerned.
The degree to which this is possible depends on the observed correlation between
explanatory variables in the observed data. When these variables are not well
correlated, natural experiments can approach the power of controlled
experiments. Usually, however, there is some correlation between these
variables, which reduces the reliability of natural experiments relative to what
could be concluded if a controlled experiment were performed. Also, because
natural experiments usually take place in uncontrolled environments, variables
from undetected sources are neither measured nor held constant, and these may
produce illusory correlations in variables under study.
Much research in several important science disciplines, including economics,
political science, geology, paleontology, ecology, meteorology, and astronomy,
relies on quasi-experiments. For example, in astronomy it is clearly impossible,
when testing the hypothesis "suns are collapsed clouds of hydrogen", to start
out with a giant cloud of hydrogen, and then perform the experiment of waiting a
few billion years for it to form a sun. However, by observing various clouds of
hydrogen in various states of collapse, and other implications of the hypothesis
(for example, the presence of various spectral emissions from the light of
stars), we can collect data we require to support the hypothesis. An early
example of this type of experiment was the first verification in the 1600s that
light does not travel from place to place instantaneously, but instead has a
measurable speed. Observation of the appearance of the moons of Jupiter were
slightly delayed when Jupiter was farther from Earth, as opposed to when Jupiter
was closer to Earth; and this phenomenon was used to demonstrate that the
difference in the time of appearance of the moons was consistent with a
measurable speed of light.
Observational studies
Observational study
Observational studies are very much like controlled experiments except that they
lack probabilistic equivalency between groups. These types of experiments often
arise in the area of medicine where, for ethical reasons, it is not possible to
create a truly controlled group. For example, one would not want to deny all
forms of treatment for a life-threatening disease from one group of patients to
evaluate the effectiveness of another treatment on a different group of
patients. The results of observational studies are considered much less
convincing than those of designed experiments, as they are much more prone to
selection bias. Researchers attempt to compensate for this with complicated
statistical methods such as propensity score matching methods (see hierarchy of
evidence). See also quasi-empirical methods
Field experiments
Field experiment
Field experiments are so named in order to draw a contrast with laboratory
experiments. Often used in the social sciences, and especially in economic
analyses of education and health interventions, field experiments have the
advantage that outcomes are observed in a natural setting rather than in a
contrived laboratory environment. However, like natural experiments, field
experiments suffer from the possibility of contamination: experimental
conditions can be controlled with more precision and certainty in the lab.
Acronym Definition
VTEF: Value TARP Echo Function
VTEF: Various Terrain Exposure Factor
VTEF: Virtual Test and Evaluation Facility
VTEF: Vary Testing Effort Function
VTEF: Virtual The Enterprise Foundation
VTEF: Value Thermal Electric Freezer
VTEF: Value Thermic Effect of Food
VTEF: Varies Thermo-Electric Fan
VTEF: Value Thermoelectric Freezer
VTEF: Virtual Threat Evaluation Function
VTEF: Various Time-Energy-Frequency (acoustics)
VTEF: Various Tonnes of Equivalent Fuel
VTEF: Virtual Total Equipment Failure
VTEF: Virtual Toxicity Equivalency Factor
VTEF: Virtual Tracheoesophageal Fistula
VTEF: Virtual Trailing Edge Flap
VTEF: Various Transient Energy Function
VTEF: Virtual Transverse Electric Field
VTEF: Virtual Tritium Extraction Facility
VTEF Vacuum Thermal Evaporation
VTEF Venous Thrombo Embolism
VTEF Video Transfer Engine
VTEF Vientiane, Laos - Wattay (Airport Code)
VTEF Virtual Terminal Environment
VTEF Vocational and Technical Education
Contact Information
Call our office today to set up an appointment. Learn more about how we can
help you, and learn more about the other services that we can offer you. All
messages we receive will be answered as soon as possible. We look forward to
hearing from you.
- Electronic mail
- General Information:
