- Check whether the explanatory variable was assigned by the researchers. - Distinguish observing a relationship from creating comparable treatment groups.

1. The study observes existing behavior rather than assigning exercise amounts. 2. It can show an association between exercise time and resting heart rate, but other variables may help explain the relationship. 3. Therefore, the study by itself does not establish causation.

No. It shows an association, not a cause-and-effect conclusion.

- Identify whether the random step chooses who enters the study or determines a treatment they receive. - Match that random step to the kind of conclusion it can support.

1. Random selection helps make the sample representative of the district's high school student population. 2. No treatment is assigned, so the study does not create the conditions needed for a causal comparison. 3. Therefore, random selection supports generalization, not causation.

It supports generalizing the survey result to high school students in the district, not making a causal claim.

- Compare the observed marker with the part of the histogram containing most results. - How often does the simulation produce values that far from the center?

1. Most simulated differences are near \(0\). 2. The observed difference lies far to the right, where the histogram has very few results. 3. The observed result is unusual under the no-effect model, providing evidence against it.

Unusual. The observed difference falls in a thin tail of the simulated distribution, which is evidence against the no-effect explanation.

- Who was randomly divided here: selected volunteers or the entire student population? - What happens when the volunteers are split into the two groups, and what happened before that split?

1. Random assignment helps balance student differences between the treatment and comparison groups by chance. 2. It does not make volunteers a random sample of all students.

It helps make the treatment groups more comparable. It does not make the volunteers representative of all students.

- Separate sampling error from measurement error. - The same device error affected every observation and therefore the mean. - Undo an additive error by subtracting it.

1. Measuring every cable makes the data a census of that shift's production, so there is no sampling error for that defined population. 2. The device introduces systematic measurement bias into every value. 3. The corrected mean is \(81.4-0.7=80.7\,\text{cm}\).

The corrected mean is \(80.7\,\text{cm}\). A census removes sampling variability for the shift, but it does not remove systematic measurement bias.

- Ask whether anyone was randomly assigned to the proposed cause. - Look for a third factor that changes both variables during the same months. - An observational association can have alternative explanations.

1. The evidence is observational: no one was randomly assigned to buy ice cream, so the association does not isolate a causal effect. 2. Temperature or season is a plausible confounder because warmer months can increase both ice-cream sales and swimming exposure, which can increase drowning incidents.

The study shows an observational association, not causation. Temperature or season is a plausible confounder affecting both ice-cream sales and swimming exposure.

- Use the sampling method to decide the scope of generalization. - Use the assignment method to decide whether causation is supported. - Random sampling and random assignment answer different inference questions.

1. Because the adults were selected with a simple random sample, the observed association can be generalized to the adult population represented by the sampling frame, subject to ordinary sampling and measurement limitations. 2. Exercise was observed rather than randomly assigned, so confounding or reverse causation can explain the association. The design therefore does not establish a causal effect.

The random sample supports a population-level association between exercise and sleep, but not a causal claim because exercise was not randomly assigned.

- Consider how treatment knowledge can influence both participants and evaluators. - Identify exactly which information blinding hides. - No single design feature controls every source of bias.

1. In Trial B, participants' expectations can change reported symptoms, and evaluators' expectations can influence measurement or interaction. 2. Double blinding in Trial A reduces both expectation effects tied to treatment knowledge and evaluator measurement bias. 3. Blinding does not eliminate other problems such as attrition, unrepresentative recruitment, poor adherence, or faulty outcome measures.

Trial A is more credible because double blinding reduces participant-expectation and evaluator-measurement effects. Blinding does not guarantee an unbiased result because attrition, recruitment, adherence, and measurement problems can remain.

- Compare the observed effect size with the decision threshold chosen before the analysis. - Keep evidence that an effect is nonzero separate from evidence that it is large enough to matter. - Sample size affects statistical precision, not the practical threshold itself.

1. The result is not practically significant because the observed reduction of \(0.2\) minute is far below the prespecified \(2\)-minute threshold. 2. With a very large sample, a small effect can be estimated precisely enough to produce strong statistical evidence against a no-effect model even when the effect is too small to matter operationally.

No. The \(0.2\)-minute reduction is below the \(2\)-minute practical threshold. A very large sample can make a tiny effect statistically detectable without making it practically important.

- Compare the requested input with the range of inputs used to fit the model. - A good fit inside one range does not determine behavior far outside it. - Ask what evidence the data actually provide near \(105^\circ\text{F}\).

1. The prediction at \(105^\circ\text{F}\) is a far extrapolation beyond the observed range of \(40^\circ\text{F}\) to \(80^\circ\text{F}\). 2. A relationship that is approximately linear within the fitted range can bend, level off, or change under more extreme conditions, so the data do not establish that the fitted line remains valid at \(105^\circ\text{F}\).

The prediction is an unsupported extrapolation far beyond the data range; a linear pattern from \(40^\circ\text{F}\) to \(80^\circ\text{F}\) need not continue to \(105^\circ\text{F}\).

- Compare the unit represented by each data point with the unit named in the conclusion. - Group averages can behave differently from relationships within groups. - Do not infer an individual relationship merely from an aggregate association.

1. The evidence compares county-level averages, while the conclusion concerns individual people. 2. Group averages do not show that the same relationship holds within counties or for individual residents. County characteristics such as income or school funding can also affect both averages. 3. Therefore, the aggregate association cannot by itself justify the individual-level claim.

The conclusion commits an aggregate-to-individual inference error: a relationship between county averages does not establish the same relationship for individual residents.

- Ask whether the proposed outcome could also influence the proposed cause. - Then look for a third factor connected to both measured variables. - The two explanations should be statistically different mechanisms.

1. Reverse causation is plausible: teenagers experiencing anxiety may turn to screens more often, so anxiety could partly influence screen time. 2. A confounder such as sleep disruption, social isolation, family stress, or school workload could affect both screen use and anxiety. 3. Because the survey is observational and cross-sectional, these alternatives prevent the association from establishing a causal direction.

For example, anxiety could lead to more screen use, and a third variable such as sleep disruption could increase both screen time and anxiety. The survey therefore does not establish that screen time causes anxiety.

- Separate what the model predicts from what management actually wants to measure. - Think of cases where the proxy and the desired outcome move in opposite directions. - Broad constructs usually require several valid indicators.

1. The model predicts email count, but email count is an unvalidated proxy for the broader construct of productivity. 2. High counts can reflect fragmented or unnecessary communication, while valuable focused work may generate few emails. 3. A better evaluation uses several role-appropriate outcomes, such as completed-work quality, timeliness, customer results, and peer or supervisor assessment.

Email count is not a validated measure of productivity: many emails can reflect inefficiency, while productive work may require few messages. Use multiple job-relevant quality and outcome measures instead.

- Judge predictive performance using data that were not used to fit the models. - Compare training performance with test performance for the highly flexible model. - Perfect fit to training observations can include accidental noise.

1. The quadratic model should be preferred because its test error of \(4\) units is much smaller than the degree-9 model's \(18\) units. 2. The degree-9 polynomial fits the training data perfectly but generalizes poorly to new data, which is overfitting.

Prefer the quadratic model. The degree-9 polynomial demonstrates overfitting: perfect training fit but much worse performance on new data.

- Randomization creates a probability mechanism, not guaranteed equality of every group summary. - Consider how much chance imbalance is possible with a small sample. - Separate an unusual realized imbalance from evidence that the assignment process itself was invalid.

1. Random assignment balances groups in expectation over repeated experiments; it does not guarantee identical baseline summaries in every realized sample. 2. With only \(20\) participants, chance imbalance can be substantial. 3. The difference should be reported and considered in analysis, but the imbalance alone does not prove manipulation or failure of the randomization procedure.

Valid random assignment can still produce unequal baseline means by chance, especially in a small sample. The imbalance alone does not prove that randomization failed.

- The test trips were not used in training, so what else could make the test misleading? - Compare the conditions in April's test set with the model's intended use. - Can a separate test set still leave out important kinds of trips?

1. April's test trips were not used to train the model, but they represent only clear weekdays. 2. Weekends and bad weather may have different traffic patterns and travel-time variability. 3. The test-set accuracy may therefore give an overly favorable picture of performance under all expected conditions.

The test set is separate from training but does not represent all conditions of use. Accuracy on clear weekdays may be better than accuracy on weekends or in bad weather.

- Ask what determined whether each completed study became visible. - Selection can operate on whole studies, not only on participants. - Search for eligible evidence before filtering by its result.

1. The review has publication bias because study visibility depends on whether results are favorable or statistically significant. 2. Omitting null studies makes the evidence appear more consistently positive than the full set of completed studies. 3. A stronger review should seek registrations, unpublished results, completed protocols, and all prespecified outcomes from every eligible study.

The review has publication bias. It should include registered, unpublished, and null studies, along with all prespecified outcomes, rather than selecting studies according to their results.

- Separate a total from a per-person average. - Use the relevant denominator for the word “generosity.” - Several components can change in opposite directions while the total rises.

1. The first-year mean is \(\frac{20000}{100}=\$200\) per donor. 2. The second-year mean is \(\frac{40000}{400}=\$100\) per donor. 3. Total donations doubled, but the average donation per donor fell by \(\$100\), or \(50\%\), so the headline's claim about individual generosity is unsupported. 4. A more accurate summary is that many more donors produced a higher total despite a lower average gift.

a) \(\$200\) and \(\$100\) per donor. b) The headline is misleading; average giving fell by \(50\%\). c) Total donations doubled because the donor count quadrupled, while the mean gift was cut in half.

- Identify whether the operational decision concerns an average or one future observation. - Individual outcomes vary around the mean even if the mean were known precisely. - Match the interval's target to the quantity the manager needs to plan for.

1. The confidence interval \((490,510)\) estimates uncertainty in the mean daily demand, not the variability of an individual future day. 2. The prediction interval \((420,580)\) includes both uncertainty about the mean and natural day-to-day variation. 3. Staffing one specific future day therefore calls for the prediction interval.

Use the prediction interval \((420,580)\), because the decision concerns one future day's demand rather than the population or period mean.

- Use the evaluation measure tied directly to the stated prediction goal. - Compare both methods on the same held-out period. - A goodness-of-fit summary and out-of-sample prediction error do not answer the same question.

1. The seasonal baseline is better on the stated prediction criterion because its mean absolute error is \(8\) units rather than \(12\). 2. Test-period prediction error directly measures performance on the goal being evaluated, while \(R^2\) is a different fit summary. 3. A high \(R^2\) therefore does not compensate for worse out-of-sample prediction error than a simple baseline.

The seasonal baseline predicts better because its test mean absolute error is \(8\) rather than \(12\) units. The model's high \(R^2\) is a different fit measure and does not override worse test error.

- Use the method of selecting participants to judge generalization. - Use the method of assigning conditions to judge causation. - Do not treat random sampling and random assignment as interchangeable.

1. Study A can support a population-wide association because it uses a simple random sample, but it cannot establish causation because sleep time was not assigned. 2. Study B can support a causal comparison for its participants because the schedules were randomly assigned, but volunteers may not represent a larger student population. 3. Random sampling supports generalization, while random assignment supports causal comparison. Each study includes only one of these design features.

a) Study A. b) Study B. c) Random sampling supports generalization; random assignment supports causation. Neither study has both features.

- Follow how the residuals change as \(x\) increases. - What kind of organized pattern would concern you when checking a straight-line model? - Could the ups and downs be random variation, or do they follow a repeated pattern?

1. The residuals are scattered above and below zero with no clear arch or other curved pattern as \(x\) increases. 2. The plot provides no clear evidence of missed curvature; the residuals are consistent with a linear model.

No. There is no clear curved pattern in the residuals, so the plot gives no strong reason to reject a linear model for missing curvature.

- How were the students who answered different from a randomly chosen schoolwide group? - What does the fraction of favorable replies describe with certainty? - What would have to change before reporting an estimate for the entire school?

1. Among respondents, \(\frac{45}{80}=0.5625\), or \(56.25\%\), favor a later start. 2. Students chose whether to answer, so those with strong opinions may be overrepresented. The result is not necessarily representative of the whole school. 3. A defensible statement is that about \(56\%\) of the students who responded favor a later start. A schoolwide estimate needs a representative way to select students.

No. About \(56\%\) of respondents favor a later start, but an optional poll cannot reliably estimate all students' views. A representative sample is missing.

- Look at how the residuals change from left to right. - Do the points look randomly scattered, or do they form a recognizable shape?

1. The residuals are positive near both ends and negative near the middle, forming a clear U-shaped pattern. 2. This organized curved pattern suggests that a straight-line model is missing curvature in the original relationship.

Yes. The U-shaped residual pattern suggests that a straight line does not capture the full relationship.

- Which population did the selection process actually cover? - What could make shopping habits differ from one store to another? - What does random selection within one store tell you—and what does it not tell you?

1. Random selection may fairly represent customers at the store that was sampled. 2. Shoppers at other stores may have different incomes, shopping habits, or coupon policies. They had no chance of being selected. 3. The result alone does not justify a statewide estimate.

No. The sample comes from one grocery store, not shoppers across the state. Coupon use could differ at other stores, so it may not represent all grocery shoppers statewide.

- Examine the scale on the vertical axis before judging the bars' sizes. - Read the numerical bar values rather than comparing only their visible lengths. - Find the difference in percentage points, not the ratio of bar heights.

1. The chart's vertical scale begins at \(94\) rather than zero, exaggerating the visible difference between the bars. 2. The percentages are \(96\%\) before and \(99\%\) after, so the actual increase is \(99-96=3\) percentage points.

The y-axis is cut off below \(94\%\), exaggerating the increase. The actual change is \(3\) percentage points.

- Use the total in each row as the denominator for its pass percentage. - What might have differed between the students before they decided whether to attend tutoring?

1. In the tutoring group, \(\frac{18}{20}=0.90=90\%\) passed. In the other group, \(\frac{12}{20}=0.60=60\%\) passed. 2. Students chose whether to attend. They may differ in motivation, past grades, or available time. The comparison alone does not prove the effect of tutoring.

a) Tutoring: \(90\%\); no tutoring: \(60\%\). b) No. Students who choose tutoring may differ from other students, for example in motivation or prior achievement.

- Treat random assignment and random sampling as serving different inferential purposes. - Use the tail proportion to judge compatibility with a no-effect model. - Match the scope of the conclusion to how participants entered the study.

1. Random assignment makes the treatment groups comparable on average, so the small tail proportion supports a causal conclusion about the app for the participating students in this experiment. 2. Because participants volunteered rather than being randomly sampled, the result cannot automatically be generalized to all students at the school. 3. Under a no-effect randomization model, only about \(1.0\%\) of simulated assignments produced a difference at least as favorable as \(3.2\) points.

a) The study supports a causal effect of the app for the participating students in this experiment. b) It does not justify generalizing the effect to every student at the school. c) A difference at least this large occurred in about \(1\%\) of no-effect simulations.

- It is easier to compute the complement event in which none of the tests produces a false flag. - Then compare the resulting family-level probability with the single-test rate. - The analysts had twenty opportunities to find an apparently unusual result.

1. The probability of no false flags is \((0.95)^{20}\approx0.3585\). 2. Therefore, the probability of at least one false flag is \(1-0.3585\approx0.6415\). 3. With about a \(64\%\) chance of at least one false flag under the no-effect model, selecting the one favorable result after twenty searches provides much weaker evidence than a single prespecified test would.

\(1-(0.95)^{20}\approx0.6415\). Because at least one false flag is common across twenty independent tests, the selected tail probability \(0.030\) is weak evidence unless the multiple search is accounted for.

- Look at the signs and sizes in sequence rather than averaging them. - A useful residual plot should not leave a visible structure unexplained. - Match the shape of the residual pattern to a missing feature in the model.

1. The residuals begin negative, become positive in the middle, and return to negative, forming a systematic curved pattern. 2. A linear model is not appropriate because residuals should fluctuate without a clear pattern around zero; the model underpredicts in the middle and overpredicts at both ends. 3. A curved relationship, such as a quadratic model, should be considered and checked against the data.

a) The residuals form an arch-shaped pattern. b) No. Their systematic pattern shows that the line misses curvature. c) Consider a curved model, such as a quadratic, and recheck residuals.

- Use the axis values, not the apparent bar-height ratio. - Calculate both an absolute percentage-point difference and a relative change. - Compare the visible bar heights with the actual percentages they represent.

1. Option A has \(48\%\) support and Option B has \(52\%\). 2. The difference is \(52-48=4\) percentage points. The relative increase is \(\frac{52-48}{48}\approx0.0833=8.33\%\). 3. The y-axis begins at \(47\%\) rather than \(0\%\), so the visible bar heights are \(48-47=1\) and \(52-47=5\). The second visible height is five times the first, but that comparison uses distances above an arbitrary truncated baseline, not the support percentages.

a) A: \(48\%\); B: \(52\%\). b) Difference: \(4\) percentage points; relative increase: about \(8.33\%\). c) The visible heights are \(1\) and \(5\) percentage points above a truncated \(47\%\) baseline. The fivefold visual comparison is not a comparison of the actual support percentages.

- Notice that selection used an extreme first measurement. - Ask what may happen on a second measurement without any treatment. - A comparison group should experience the same timing and selection process.

1. The players were selected for extreme low results that partly reflect temporary random variation. 2. On a later week, the unusually poor variation is unlikely to repeat as strongly, so percentages can move closer to typical levels even if the routine has no effect. 3. A stronger design randomly assigns similarly low-performing players to the new routine or a comparison routine and compares later changes.

The selected extreme lows can be followed by less extreme results simply because temporary poor variation does not repeat; this is regression toward the mean. Randomly assign comparable low-performing players to treatment and control routines to test the effect.

- Count correct predictions separately for each actual class. - Examine performance on the rare outcome rather than only the total. - A useful metric should reflect the kind of error that matters for the task.

1. The model correctly labels all \(950\) legitimate transactions, so accuracy is \(\frac{950}{1000}=95\%\). 2. It detects \(0\) of \(50\) fraudulent transactions, so its fraud-detection rate is \(0\%\). 3. The classes are highly imbalanced, so a model can obtain high overall accuracy by always predicting the common class while failing completely on the important rare class. 4. The report should include the fraud-detection rate, false-alarm rate, or a full confusion matrix.

a) \(95\%\). b) \(0\%\). c) The common legitimate class dominates the accuracy calculation and hides total failure on fraud. d) Report fraud-detection rate and false-alarm rate, or the full confusion matrix.

- Use actual positives as the denominator for detection and actual negatives for false alarms. - Compare both rates before judging a threshold. - A threshold decision reflects the costs of different mistakes, not only mathematical accuracy.

1. At the low threshold, the detection rate is \(\frac{90}{100}=90\%\), and the false-alarm rate is \(\frac{180}{900}=20\%\). 2. At the high threshold, the detection rate is \(\frac{65}{100}=65\%\), and the false-alarm rate is \(\frac{45}{900}=5\%\). 3. Raising the threshold reduces false alarms but misses more positive cases. 4. The best threshold depends on the relative costs of missed positives, false alarms, follow-up resources, and the intended use.

a) Low: \(90\%\) detection and \(20\%\) false alarms. High: \(65\%\) detection and \(5\%\) false alarms. b) The higher threshold lowers both detections and false alarms. c) The choice depends on the consequences and costs of the two error types.

- Combine group totals rather than averaging the two group means equally. - Weight each mean by its group size. - Ask whether response status is associated with the measured variable.

1. The combined salary total is \(800\cdot62{,}000+200\cdot120{,}000=73{,}600{,}000\) dollars. 2. The full-sample mean is \(\frac{73600000}{1000}=\$73{,}600\). 3. The respondent-only mean is \(73{,}600-62{,}000=\$11{,}600\) too low. 4. Nonresponse is related to salary, so treating the observed salaries as representative of all selected employees is inappropriate.

a) \(\$73{,}600\). b) The respondent-only mean underestimates by \(\$11{,}600\). c) The missingness is related to the outcome, producing nonresponse bias.

- Count how many subgroup opportunities were examined before one was highlighted. - Ask whether the subgroup was chosen before or after seeing the outcomes. - Post hoc findings require stronger confirmation than prespecified tests.

1. The subgroup was selected after the analysts inspected many possible subgroup results, creating multiple opportunities for a chance finding. 2. A tail probability of \(0.04\) is therefore less persuasive than it would be for one prespecified subgroup analysis. 3. The result is better treated as a hypothesis to be tested in a planned, independent replication.

Because the subgroup was found after searching \(12\) groups, the result may be a chance selection from multiple comparisons. It should be treated as exploratory and confirmed with a prespecified independent analysis.

- Compare both centers and spreads shown by the boxes. - Consistency is reflected by how tightly outcomes cluster. - One summary statistic cannot determine an entire distribution.

1. Service A has median \(30\) and interquartile range \(32-28=4\) minutes. Service B has median \(25\) and interquartile range \(45-15=30\) minutes. 2. Service A is more consistent because its central half and full whisker range are much narrower. 3. Equal means describe only one measure of center; Service B's broad, skewed distribution can have the same mean while producing far less predictable delivery times.

a) A: median \(30\), IQR \(4\). B: median \(25\), IQR \(30\). b) Service A. c) Reliability depends on variability and distribution shape, not only the mean.

- First recover correctly classified negatives from the false-positive counts. - Use actual positives as the denominator for missed-positive rate. - A shared overall percentage can be built from very different kinds of errors.

1. Group A has \(90\) correct positives and \(90\) correct negatives, so accuracy is \(\frac{180}{200}=90\%\). 2. Group B has \(10\) correct positives and \(170\) correct negatives, so accuracy is also \(\frac{180}{200}=90\%\). 3. Group A misses \(10\) of \(100\) positives, a \(10\%\) missed-positive rate. Group B misses \(10\) of \(20\), a \(50\%\) rate. 4. Different class proportions allow the same total accuracy to hide a much worse error rate on positive cases in Group B.

a) Both groups have \(90\%\) accuracy. b) Group A: \(10\%\); Group B: \(50\%\). c) Accuracy combines error types and is affected by group class proportions, so it can conceal unequal missed-positive rates.

- What does the split between the two schools tell you about how to combine their mean changes? - Can the overall change tell you what happened at each school? - What might affect reading scores between two tests besides the program?

1. Weight each school's mean change by its share of the students: \(0.80\cdot10+0.20\cdot(-5)=7\) points. 2. The positive overall change hides the decrease at West High. 3. Scores might change for other reasons. Without a suitable comparison group, these before-and-after results do not show that tutoring caused the increase.

a) \(+7\) points. b) It hides the decrease in mean scores at West High. c) No. Other factors might have changed scores; a suitable comparison is needed to judge the effect of tutoring.

- Ask when each predictor becomes available relative to the prediction decision. - A held-out set is useful only when it contains the same information that would be available in real use. - Random splitting does not remove a predictor that already reveals future information.

1. “Course completed” is recorded only after the final outcome is known, so it leaks future or outcome information into the predictor set. 2. A random train-test split does not solve the problem because the leaked variable appears in both sets. 3. In real use before the exam, that information would be unavailable, so the reported test setting does not reproduce the intended prediction task.

The model uses future-information leakage. Because the leaked predictor appears in both random splits, the \(99\%\) test accuracy measures an unrealistic task rather than true pre-exam prediction performance.

- Compare the population on which performance was validated with the population now being scored. - Look for changes in predictor distributions and operating conditions over time. - A performance estimate is tied to the distribution on which it was evaluated.

1. The applicant distribution and possibly the relationship between predictors and default outcomes have shifted over time. 2. The \(2018\) validation data therefore no longer represent the \(2026\) deployment population. 3. This distribution shift explains why the old accuracy estimate cannot be assumed to describe current performance.

The evidence suggests distribution shift. The \(2018\) validation population no longer matches \(2026\) applicants, so the old performance estimate is not reliable for current deployment.

- Calculate change once relative to the original risk and once directly on the percentage scale. - Translate both rates into counts for a common population size. - A relative statement can sound large when the baseline rate is small.

1. The relative reduction is \(\frac{2\%-1\%}{2\%}=\frac{1}{2}=50\%\). 2. The absolute reduction is \(2\%-1\%=1\) percentage point. 3. Without treatment, the expected count is \(10{,}000\cdot0.02=200\); with treatment, it is \(10{,}000\cdot0.01=100\), a difference of \(100\) events. 4. The relative claim is mathematically correct but incomplete without the starting risk and absolute change.

a) \(50\%\) relative reduction. b) \(1\) percentage point absolute reduction. c) About \(100\) fewer events per \(10{,}000\) people. d) The claim is correct but should also report the baseline risk and absolute reduction.

- Calculate each change relative to its own starting value. - Compare periods that differ in whether an unusual disruption occurred. - A claim can be numerically correct yet misleading because of endpoint selection.

1. The February-to-March increase is \(\frac{840-600}{600}=0.40=40\%\). 2. The January-to-March change is \(\frac{840-1000}{1000}=-0.16=-16\%\). 3. The year-over-year March change is \(\frac{840-830}{830}\approx0.0120=1.20\%\). 4. The advertised comparison starts from an unusually low outage month and therefore exaggerates ordinary growth; broader and comparable periods give a different picture.

a) \(40\%\). b) January to March: \(-16\%\); year-over-year March: about \(+1.20\%\). c) The selected outage month creates a misleadingly low baseline and overstates growth.

- Compare the center and spread of residuals separately. - Prediction uncertainty should reflect the local size of typical errors. - A single error margin assumes similar variability everywhere.

1. The residuals remain centered near zero, but their spread increases as \(x\) increases. Constant residual variability is therefore questionable. 2. Predictions should be most precise at low \(x\), less precise in the middle, and least precise at high \(x\). 3. A constant \(\pm3\) rule is wider than needed for many low-\(x\) cases but far too narrow for high-\(x\) cases, so it misrepresents changing uncertainty.

a) Residual variability increases with \(x\), so constant spread is questionable. b) Prediction intervals should widen as \(x\) increases. c) One fixed margin ignores the strong change in residual spread across the domain.

- Compare the effect across levels of the environmental condition. - A single effect is questionable when the change depends strongly on another condition. - Context-specific estimates can be more informative than an unweighted average.

1. The report treats a context-dependent effect as one constant effect and therefore misrepresents both dry and wet conditions. 2. The fertilizer effect changes with moisture conditions rather than remaining constant. 3. A more accurate conclusion is that the estimated increase is about \(5\) bushels per acre when dry and \(20\) when wet; any overall average must be weighted by the frequency of those conditions and should not replace the separate effects.

a) The average is incorrectly applied uniformly to two conditions with very different effects. b) The fertilizer effect depends on whether conditions are dry or wet. c) Report the separate \(5\)- and \(20\)-bushel effects and, if needed, a properly weighted overall average.

- Track whether information from the supposed test set changed any modeling decision. - Selecting the largest value among many noisy estimates favors positive random fluctuation. - An independent test set must remain untouched until model choices are finished.

1. Because test-set results influenced which model was selected, the test set became part of the model-development process rather than remaining an independent final evaluation. 2. Choosing the maximum among many noisy test accuracies favors models that benefited from positive random fluctuation on that particular test set. 3. The selected accuracy is therefore likely biased upward relative to performance on genuinely new data.

Repeated test-set reuse makes the test data part of model selection. Choosing the best of \(50\) noisy test results favors an unusually high value, so the reported accuracy is likely optimistic.

- Compare the dashed line with where most simulated values lie. - How does the count of results at least as large as the observed one compare with all simulations? - What are you assuming when using the histogram to assign a probability?

1. The dashed line is in the far right tail, away from the bulk of the simulated results. 2. The estimated tail probability is \(\frac{12}{1000}=0.012\), or \(1.2\%\), under the no-effect model. 3. This is the probability of such a result assuming the model, not the probability that the model is true.

a) Far into the right tail. b) About \(0.012\), or \(1.2\%\). c) No. The estimate describes results assuming the no-effect model; it does not give the chance that the model is true.

- Compute rates using applicants within the same department as denominators. - Then recompute after pooling each gender across departments. - Compare how each group is weighted toward the high-rate and low-rate departments.

1. In Department A, the rates are \(\frac{80}{100}=80\%\) for women and \(\frac{18}{20}=90\%\) for men. In Department B, they are \(\frac{2}{20}=10\%\) for women and \(\frac{60}{100}=60\%\) for men. 2. Overall, women have \(\frac{82}{120}\approx68.3\%\), while men have \(\frac{78}{120}=65\%\). 3. More women applied to high-admission Department A, while more men applied to low-admission Department B. Different group weights reverse the aggregated comparison.

a) Men have higher admission rates in both departments: \(90\%\) versus \(80\%\) in A and \(60\%\) versus \(10\%\) in B. b) Overall, women have about \(68.3\%\) admitted and men \(65\%\). c) The applicant groups are distributed very differently across departments with very different base admission rates.

- Random assignment balances groups before later events occur. - Ask who is missing from each final comparison and why. - A trustworthy report should show how different reasonable assumptions about missing outcomes affect the conclusion.

1. Differential dropout can destroy the comparability created by random assignment among the participants who remain. 2. If exercise participants with little improvement or difficulty following the program are more likely to leave, the completers can make the program look more effective than it was for everyone assigned. 3. The report should compare dropout reasons and baseline traits, include outcomes for all assigned participants when possible, and show how conclusions change under reasonable assumptions about the missing outcomes.

a) Unequal attrition can make the remaining groups systematically different. b) For example, exercise participants who do not improve may be more likely to drop out. c) Analyze all assigned participants when possible and report how the conclusion changes under reasonable assumptions about missing outcomes.

- What does a good regression fit tell you, and what does it not establish? - Which variable changes along with both series from year to year? - How could the analysis reduce the effect of a shared time trend?

1. A high \(R^2\) shows a strong fitted association, not proof that one variable caused the other. 2. Both series rise with time, so later years are higher in both measures even without a causal link. 3. Compare changes over time, account for inflation, and examine predictors with a plausible connection to textbook prices.

A shared rise over time can produce a high \(R^2\) without causation. The analysis should account for time and inflation and consider relevant pricing factors.

- An estimate from one sample is not expected to match another exactly. - Compare both the centers and the full plausible ranges. - Investigate methodological differences before making claims about conduct.

1. Different random samples and study conditions can produce different estimates; disagreement does not by itself establish misconduct. 2. Study 1 supports positive effects from \(2\) to \(8\), while Study 2 includes negative, zero, and small positive effects from \(-4\) to \(2\). They share the boundary value \(2\) but give substantially different centers. 3. Researchers should compare protocols, populations, measurements, and attrition, and combine or repeat well-designed studies when appropriate.

a) Sampling variation or design differences can explain disagreement; the intervals alone do not show fraud. b) The first supports a positive effect, while the replication supports values from moderately negative through small positive. c) Audit design differences and conduct or combine additional carefully designed replications.

- Compare the predicted percentage with the observed relative frequency. - State both the direction and size of the discrepancy. - Probability accuracy and ranking performance are different model properties.

1. The observed event rate is \(\frac{90}{200}=0.45=45\%\). 2. The observed rate is \(45\%-30\%=15\) percentage points higher than predicted, so the model underpredicts risk for this group and is poorly calibrated here. 3. Calibration compares predicted probabilities with observed frequencies; ranking ability requires comparing whether cases with higher assigned risks actually tend to experience more events than cases with lower assigned risks.

a) \(45\%\). b) The model underpredicts by \(15\) percentage points for this group. c) Calibration within one risk group does not measure how well the model orders cases across different risk levels.

- Use falsely stopped simulations divided by all no-effect simulations. - Compare both the absolute and multiplicative increase over the target rate. - A decision rule includes when the data are examined and when collection stops.

1. The simulated false-positive rate is \(\frac{142}{1000}=0.142=14.2\%\). 2. This is \(14.2\%-5.0\%=9.2\) percentage points above the intended rate and is about \(\frac{14.2}{5}=2.84\) times as large. 3. Each additional look creates another opportunity for random fluctuation to cross the threshold; the original tail-probability interpretation assumes a fixed analysis plan.

a) \(14.2\%\). b) It is \(9.2\) percentage points higher, about \(2.84\) times the intended rate. c) Repeated optional checking adds chances to stop on a random extreme result.

- Subtract before from after in both cities. - Compare those two changes using the same sign convention. - Does the untreated city suggest that something besides the law changed? - What must be true of the cities' trends for the comparison to isolate the law's effect?

1. The law city's change is \(70-100=-30\) crashes. The comparison city's change is \(88-110=-22\) crashes. 2. The adjusted change is \((-30)-(-22)=-8\) crashes. 3. A wider trend may have reduced crashes even without the law. For a causal interpretation, assume that without the law the first city would have had the same underlying trend as the comparison city and that no other major changes explain the difference.

a) Law city: \(-30\) crashes; comparison city: \(-22\) crashes. b) \(-8\) crashes. c) Both cities were improving. Only if their no-law trends would have been comparable, without other important differences, could the extra drop be attributed to the law.

- Identify which observations share the same source. - Repeated measurements do not add as much new information as measurements from new independent units. - Either change the unit of analysis or use a model that represents the grouping.

1. Observations from the same student are likely correlated, violating the assumption that all \(300\) observations are independent. 2. Treating repeated observations as independent overstates the amount of separate information and usually makes standard errors and intervals too small. 3. The analysis can summarize each student first or use a method that treats observations from the same student as related; the design can also sample more students rather than only more days per student.

a) Independence is violated within each student's repeated measurements. b) Uncertainty is understated because the effective sample size is less than \(300\). c) Analyze student-level summaries or use a method that accounts for repeated observations from each student, and increase the number of students when possible.

- Consider what happens when many new values are placed exactly at the center. - A filled-in value is not the same as an observed value. - Valid uncertainty should include uncertainty about what was missing.

1. Replacing missing values by the observed mean leaves the completed-data mean at \(75\) if the observed mean was \(75\). 2. Every imputed value lies exactly at the center, so the completed data have artificially reduced variability. 3. The method treats guessed values as known and ignores uncertainty about missing outcomes, causing model uncertainty and confidence-interval width to be understated.

a) It preserves the observed mean of \(75\). b) It artificially reduces variability by adding many central values. c) The analysis ignores uncertainty in the imputed values and therefore appears too precise.

- Keep the groups defined by the random mechanism. - A choice made after assignment can be related to the outcome. - Distinguish the effect of being offered a program from the effect among self-selected users.

1. The assignment-based effect estimate is \(74-70=4\) units. 2. People who choose to participate may differ in motivation, availability, or baseline condition from nonparticipants and controls, so the \(78-70=8\) comparison mixes treatment with self-selection. 3. Comparing all people according to original assignment preserves randomized group comparability and estimates the effect of offering or assigning the program.

a) \(4\) units. b) Actual participation is self-selected after assignment, so the \(8\)-unit comparison is confounded. c) Compare all \(200\) people by their original randomized assignment.

- Think about how many of the \(10{,}000\) people belong to each group before using the test rates. - Which group does each of the two positive-test rates describe? - For part b), among which group of people is the percentage being requested?

1. Of \(10{,}000\) people, \(200\) have the condition and \(9800\) do not. 2. Expected true positives: \(200\cdot0.90=180\). Expected false positives: \(9800\cdot0.05=490\). 3. Among all positive results, the fraction from people who actually have the condition is \(\frac{180}{180+490}\approx0.2687\), or about \(26.9\%\) to the nearest tenth of a percent. 4. The \(90\%\) figure begins with people who have the condition. Part b begins with all people who test positive. Because the condition is rare, false positives form a large part of that second group.

a) \(180\) true positives and \(490\) false positives. b) About \(26.9\%\). c) The detection rate is measured among people with the condition, not among everyone who tests positive. False positives and the low population rate matter.

- Ask whether one participant's assigned treatment can change another participant's experience. - Consider what happens when the control group receives part of the active program. - Choose assignment units that reduce contact across treatment conditions.

1. The study's assumption that one group's treatment does not affect outcomes in the other group is threatened by spillover or interference. 2. Control students may receive part of the tutoring benefit, making the groups more similar and often reducing the estimated difference; other social effects could also complicate direction. 3. Researchers could assign more widely separated schools or groups, measure how much sharing occurs, and report how the results change when cross-group exposure is considered.

a) Spillover makes the treatment and control conditions overlap. b) Shared materials can contaminate the control condition and usually shrink the observed contrast. c) Assign more separated groups and measure or account for cross-group exposure.

- Separate how treatment was assigned from how outcomes are later analyzed. - Matching is useful when partners share variation that can cancel in a difference. - Use the same observational unit in the analysis that the randomization created.

1. The original within-pair random assignment still occurred, so discarding labels does not erase the random treatment mechanism or necessarily change the treatment-group mean difference. 2. Matching made partners similar at baseline, and the within-pair differences vary by only \(4\) points compared with \(15\)-point variation among individual outcomes. Ignoring pairs fails to remove that shared baseline variation and therefore loses precision. 3. The analysis should use the \(50\) within-pair differences, centered at the observed mean difference of \(3.2\) points, with a method designed for paired data.

a) No. Treatment was still randomly assigned within pairs, although the analysis ignores useful design information. b) Ignoring matching leaves large between-participant variation in the error instead of canceling it within pairs. c) Analyze the \(50\) A-minus-B pair differences, whose mean is \(3.2\) points.

- Separate uncertainty inside one chosen model from uncertainty about choosing the model itself. - Good historical fit does not make two different future assumptions equivalent. - Show how the conclusion changes under other plausible model assumptions.

1. The report omits model uncertainty: different plausible model structures and assumptions produce very different forecasts. 2. A narrow interval conditional on Model A can describe random error within that model while ignoring the larger uncertainty about which model is appropriate. 3. A responsible report should show forecasts from both justified models, explain their assumptions, evaluate them on relevant validation data, and show how the forecast changes under the different plausible models.

a) Uncertainty about which model and assumptions are appropriate. b) It can look precise while ignoring another plausible forecast far from \(120\). c) Report alternative models, assumptions, validation performance, and sensitivity of conclusions to model choice.

- Evaluate random assignment, sampling, missing outcomes, and uncertainty as separate parts of the design. - Ask whether the set of completers can still be treated as if it arose solely from the original random assignment. - Distinguish a sample average from a population effect and from every individual's response. - A defensible conclusion should preserve both the observed completer difference and the study's design limitations.

1. Random assignment initially creates comparable treatment groups and supports causal inference if outcomes are retained. The much larger dropout in the drink group can destroy that comparability among completers, so the completer-only difference is not a clean randomized estimate of the causal effect. 2. Volunteers from one fitness center are not a random sample of all adults, so broad population generalization is unsupported. 3. The \(4.0\) value is a sample mean difference, not an exact population effect; the confidence interval shows uncertainty about the completer mean difference. Also, an average difference does not imply that every individual changes by the same amount. 4. Shuffling treatment labels only among the observed completers does not reproduce the original random-assignment mechanism if treatment assignment affected who remained in the study. Therefore, the \(0.020\) tail proportion cannot by itself be treated as a clean randomization-test probability for the original experiment. The confidence interval likewise describes uncertainty for the analyzed completers and does not repair attrition bias. 5. A supported conclusion is that drink-group completers had a higher mean heart rate than placebo completers, with an estimated difference of \(4.0\) beats per minute and completer interval \((0.5,7.5)\), but differential attrition weakens causal interpretation and the volunteer sample does not support generalization to all adults.

a) Random assignment initially supports causation, but differential attrition means the completer-only comparison is not a clean causal estimate. b) The volunteer sample from one fitness center does not support generalization to all adults. c) \(4.0\) is an estimate rather than an exact population effect, and a mean effect is not an identical effect for every individual. d) Because dropout differs after assignment, shuffling labels only among completers need not represent the original randomization; the \(0.020\) tail proportion and completer confidence interval do not remove attrition bias. e) Drink-group completers had a higher mean heart rate, but differential attrition weakens causal interpretation and generalization to all adults is unsupported.