Redundancy detection and editing in prose

Abstract

Redundancy in both written and spoken language is a well established fact. In the usual message there are words, phrases and sometimes even sentences which add no information to the essential ideas. Researchers in the areas of both computing science and psychology have separately investigated techniques for condensing prose, but they have failed to implement human techniques of verbal processing in their efforts. RDE (Redundancy Detection Editing) is a system which implements the results of verbal learning studies as heuristics for prose comprehension and reduction. Furthermore, RDE is a hybrid system which incorporates techniques of information organization obtained from information science studies, and the techniques of prose processing obtained from verbal learning experiments. Researchers in computing science who have investigated the area of automatic condensation have not attempted to investigate the effects of the condensed material on verbal comprehension. If the reduced material is to be used in lieu of the original article, this factor is an important criterion of success. In order to investigate this criterion, a verbal learning experiment was conducted. In this experiment, one group of subjects (control) read the original article, and a second group read a version that was reduced in length (44.64%) under the supervisor of a MONITOR. The third group read a version reduced in length (50.19%) without MONITOR supervision. The MONITOR was a collection of heuristics that related syntax errors to sentence recall. The experiment was concluded when all three groups were administered a multiple choice test on the contents of the article. While the results of the experiment did not indicate a significant difference in comprehension between the control group and the experimental groups, the scores of the subjects reading the reduced versions were sufficiently high to recommend further development of the algorithms contained in RDE.