How Many Trials Back a Force-Aware Manipulation Claim?

Summary

Papers on force-aware manipulation usually support their claims with real-robot success rates measured over a handful of trials. We audited how much statistical evidence those numbers carry, following a protocol frozen before any data were extracted.

Our summary. The archived abstract is the authority: doi:10.31224/8576.

What it found

1. F1

   We screened 89 arXiv papers, included 58, and extracted 377 binary comparisons of the proposed method against its strongest baseline or its own ablation.

2. F2

   The median condition was evaluated on 20 trials (interquartile range 15–30). Only 17% of papers reported any interval or error bar on success.

3. F3

   Only 6% of comparisons had 80% power to detect a true 20-point improvement. Two equally good methods compared on 10 trials each show an apparent gain of 10 points or more 41% of the time.

Limitations

• None of this says the methods do not work: most reported gaps are large and in the expected direction.
• Extraction was done by instances of an AI assistant. A second, blinded extraction checked the data, and the author re-coded a seeded random 20% of papers by hand (19 of 19 inclusion decisions and 100% of 88 binary rows matched).
• A revision is in progress.

Cite

@misc{aidoo2026trialaudit,
  title  = {How Many Trials Back a Force-Aware Manipulation Claim? A Statistical Audit of 2026 Real-Robot Evaluations},
  author = {Aidoo, Clement Nana},
  year   = {2026},
  note   = {engrXiv preprint, version 2},
  doi    = {10.31224/8576},
  url    = {https://doi.org/10.31224/8576}
}