An experimental study investigating the relation between vertical and horizontal graphic rating scales

Citation preview

AN EXPERIMENTAL STUDY INVESTIGATING- THE RELATION BETWEEN VERTICAL AND HORIZONTAL GRAPHIC RATING SCALES

A Thesis Presented to the Faculty of the. Department of Psychology University of Southern California

In Partial Fulfillment of the Requirements for the Degree Master of Arts

by James Henry Carpenter January 1950

UMI Number: EP63981

All rights reserved INFORMATION TO ALL USERS The quality of this reproduction is dependent upon the quality of the copy submitted. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed, a note will indicate the deletion.

Dissertation RuhnsntftQ

UMI EP63981 Published by ProQuest LLC (2014). Copyright in the Dissertation held by the Author. Microform Edition © ProQuest LLC. All rights reserved. This work is protected against unauthorized copying under Title 17, United States Code

ProQuest LLC. 789 East Eisenhower Parkway* P.O. Box 1346 Ann Arbor, Ml 4 8 1 0 6 - 1346

Th is thesis, w ritten hy

James Henry Carpenter under the guidance of h.%A.. F a c u lty C o m m ittee, and app ro ved hy a l l its members, has been presented to and accepted by the C o u n c il on G ra d u ate S tu d y and Research in p a r t ia l f u l f i l l ­ ment of the requirements f o r the degree of

MASTER OF ARTS

D ate

Jaiuaj. 1.9.1Q

Faculty Committee

TABLE OF CONTENTS CHAPTER I.

PAGE THE P R O B L E M ...................................... Introduction

..................................

Statement of the p r o b l e m .............. . . . . Hypotheses

II.

1 1 2

..................................

2

Importance of the s t u d y .....................

2

Organization of remainder of the thesis . . . .

3

.....................

**

REVIEW OF THE LITERATURE •Introduction *to review Uses and benefits

of the literature

...

...........................

k 5

Subjectivity, objectivity, validity and 8

reliability Sex difference, rater characteristics, halo, ............................. .

12

Criterion and construction of the scale . . . .

16

Construction of rating scales .................

16

weighting

III.

DESIGN OF THE EXPERIMENT

AND GROUP STUDIED

...

21

...............

21

..................

23

Design and scale construction Method of item selection

The group and methods ofsampling

............

Method of scoring IV.

STATISTICAL TREATMENT OF THE D A T A ............... Variability

..................................

2k

26 30 30

iii CHAPTER

PAGE Analysis of the distributions Tests of differences

V.

.................

.........................

3° 36

SUMMARY AND C O N C L U S I O N S........................

46

Summary Conclusions BIBLIOGRAPHY APPENDIX

...............................

.......................................... .............................

47 49

LIST OF TABLES TABLE

PAGE

I.

Tally Chart for Vertical R a t i n g s .................

28

II*

Tally Chart for Horizontal Ratings ...............

29

Means and Standard Deviation

31

III* IV* V.

Tests for Normality

.....................

32

Test for Significance of Difference between Conditions

.

..................................

35

LIST OF FIGURES FIGURE

E£GE

I,

Graphs of all four

conditions for Item

1

II.

Graphs of all four

conditions for Item

2 .............

39

III.

Graphs of all four

conditions for Item

3 .............

40

IV.

Graphs of all four

conditions for Item

4 ..............41

V.

Graphs of all four

conditions for Item

5 •••••••••••••

42

VI.

Graphs of all four

conditions for Item

6 *••••••••••••

43

..... 38

VII. Composite graphs for the four conditions for all items 44 *

it

V U I . Graphs for all vertical ratings, all horizontal ratings, and composite total of all ratings ....................45

CHAPTER I THE PROBLEM I.

INTRODUCTION

Industrial surveys and educational studies have employed rating scales in their various forms rather extensively since 1 1905* When properly constructed and appropriately used, the rating scale has proved to he a worthwhile measuring device. Indiscriminate use of the rating scale without regard to the basic principles of its construction may hint falsely that rating procedures are unsound. The most common types of rating scales are: the manto-man, and graphic, both absolute; the check list; and the 2 rank order, a relative scale. Though the man-to-man scale was the first type extensively employed in industry, it has been superseded by the check list and graphic scales.

The

check list is a valuable aid in supervisory ratings of em­ ployees, but it has not become as popular as the graphic scale. The graphic scale easily lends itself to statistical computations.

Depending on the shape of the distribution

■^J. P. G-uilford, Psychometric Methods. (New York: McGrawHill Book Co., Inc., 193 & ) 1 P* 2 6 5 . 2rbid., pp. 263-64.

2

✓

curve plotted from the ratings, the most meaningful statistical analysis can be selected, hypotheses accepted or re­ jected, and levels of confidence established,, II. Hypotheses.

STATEMENT OF THE PROBLEM It is the purpose of this study to inves­

tigate three hypotheses;. (1) it is likely that greater Dractice in making judgments on a horizontal- plane in life situations generally causes a difference betvseen rating scales employ — r

y

ing a horizontal scale and those employing a vertical scale, and since (2), psychologically and physically f,hightf is thought of as being at the !ltopH, a difference in ratings is likely to occur on the vertical scale when the labelings of the ends of the graphic scale are interchanged in position, and (3) it is possible that a simple change in the format of the horizontal scales, that of reversing the ends of the continuum, will result in a significant difference. Importance of thja study.

Horizontal scales predomin-

ate in practice, but validating studies may indicate that ver­ tical scales are superior or equal in reliability to the hori­ zontal.

Unless within a high level of 'confidence the two ✓

scales are demonstrated to provide the same measurements, the if

Interchanging of scales, or grouping data resulting from rat­ ings of both types could add unknown and undesirable error. Results of this experiment indicate that a true difference does exist.

III.

ORGANIZATION OF REMAINDER OF THE THESIS

The remainder of the thesis is organized as follows: Chapter II discusses the literature of theoretical and scientific Chapter

nature pertinent to the problem. III describes the group supplying theratings,

lists the controls, used, relates the procedure of obtaining the ratings, and the method of scoring Chapter

the ratings.

IV gives the results of the rating and statis­

tics employed. Chapter V summarizes the study and gives a section on the conclusions.

CHAPTER II REVIEW OF THE LITERATURE I.

INTRODUCTION TO REVIEW OF THE LITERATURE

Prior to World War I rating scales were recognized, but had not been developed thoroughly. The first seriously 1 considered scale was that of Scott, a man-to-man scale, which proved unsuccessful in industry because the executives failed to prepare them carefully. Ruml suggested the changes that were presented by Miner's dot-on-a-line scale.

Scott then developed the

graphic scale that is now used.

During World War I both the

man-to-man scale and the graphic scale were helpful In ob­ taining ratings of officers.

However,, some students of the

problem are still skeptical of the value of the rating pro­ cedure.

"Only one thing can safely be said of a rating: that

is, that it records what the rater thinks about the subject. A consideration in evaluating rating scales now in

*^D. D. Paterson, "The Scott Company Graphic Rating Scale," Journal Personnel Research. 1, Dec.-Jan. 1922 - 2 3 , p. 3^1. Loo. oit. 3 F. F. Bradshaw, "American Council on Education Rating Scale: Its Reliability, Validity and Use, " Archives of Psy­ chology. 119, October, 1930, p. 52.

use is the possibility that the scale may actually cover only one or two unique factors rather than the larger number usually seen on rating- forms. A more optimistic note is heard from Pockrass,

“Though

some favor the criterion of fitness there is a place for 5 merit rating selection. " In 1939 Star and G-reenly sent questionnaires to sixtyfour companies representative of the major branches of indus­ try in the United States.

There were forty-four returns

from which it was estimated that one-third of them used merit rating techniques.

6

So its use is enjoying a hopeful

popularity from which worthwhile investigation is certifying the value of rating procedures. II.

USES AND BENEFITS

A variety of uses have been made of rating scales: aesthetic judgments of art, music, literature, oratory; pros- .. pective employee rating in selection; merit ratings of employ­ ees on the job; executive personnel ratings;

student ratings

A

E. Ewart, S. E. Seashore and Joseph Tiffin, “A Factor Analysis of an Industrial Merit Bating Scale,“ Journal of Applied Psychology. 25, 19^1, P* 486. 5 J. H. Pockrass, “Rating Training and Experience in Merit System Selection, " Public Personnel Review, 2, 1941. p. 222

.

6J. E. Zerga, “Developing an Industrial Merit Rating Scale," Journal of Applied Psychology, 2?, 1 9 A 3 . p. 192.

in the classroom;

instructors 1 ratings by students; sample

merchandise rated in industrial production; and occupational similarities. To obtain ratings of factors for which objective data are available is meaningless unless to furnish a test of validity of the rating system or else a check on the staff performing the ratings. A few typical examples illustrate the most common uses of rating.

In rating trait characteristics, Hughes

7

obtained classroom ratings of the following: industry, ac­ curacy, initiative, reliability, cooperation, leadership, and physical vitality.

He lists the outcome in relation

to uses made of the results: 1.

Better understanding of student.

2.

Modification of school and classroom procedures.

3.

More scientific counseling of student.

4. 5.

Turning student*s attention to develop proper habits and attitudes. Greater justice to the backward student.

, 6 . Greater justice to the superior student. 7.

Understanding and approval on part of the community.

W. Hardin Hughes, “General Principles and Results of Rating Trait Character is tics , 11 Journal Educational Methods. 4, 1925. p. 426. 8 Ibld., p. 429.

The F&rrel-Birmingham Company, Inc., of Ansonia, Gonnecticutt, used a merit-rating system essentially the same as that formulated by the Metal Trades Association.

Of eleven

evaluating factors which were classified by degrees, the following benefits were summarized: in establishing wage rates, for his actual skill,

(1 ) eliminated favoritism

(2 ) paid the individual worker

(3 ) improved employee morale, and (4)

made possible comparison of the company's wage rate with others. Lender ^ reported that after a rating system was employed

by the Starr Electric Company of Bloomingfield, Mew Jersey, twelve new job masters were added to the original thirteen following installation of new equipment, but a one-third de­ crease in personnel resulted*

This was due to a better dif10 11 ferentiation of what each man was to do. Five studies * *

12 iq

'

*—

utiold, in unison, the advantages of rating:

Edward Lender, "Rating Maintenance Jobs," Factory Man­ agement and Maintenance, 103, June, 19^5* P» 137* *^J. E. Walters, "Rating the Job and the Man," Factory Management and Maintenance, 103, June, 19^5» P* 393• ^Anonymous, "U. S. Steel Rates Executives, " Business Week, March 4, 1939, p. 26. •

12

John G. Watkins, "The Use of Service Ratings for Em­ ployee Guidance," Public Personnel Review, 4, July, 1943, p. 168 - 1 7 2 . 13 ^D. J. Bolanovich, "Statistical Analysis of. an Indus­ trial Rating Chart," Journal of Applied Psychology, 3°> 1946, p. 154. 14 Lender, o£. oit. , p. 154.

(1 ) basis for promotion, transfer, etc. of employee, training needs or items necessary for job success,

(2 )

(3 )

heightened morale or prevention of grievances, and (4) in­ creased analytical ability. advantages,

In addition to these common

the following were also cited:

(1 ) increased

efficiency, reduced cost of p r o d u c t i o n , (2 ) provide a 16 gauge of executive personnel, and (3 ) employee selection, 17 18 (4) incentive to the employee, * (5 ) eliminates job over-

lapping and fixes responsibilities, ating tests of personality,

19 7 (6 ) criterion in evalu­

(7 ) rating clinical cases, (8 )

mass effects of drugs on efficiency, and (9 ) any psychologi20 cal experiment involving verbal reports. III.

SUBJECTIVITY, OBJECTIVITY, VALIDITY, AMD RELIABILITY The scientific method demands a high degree of validity

and reliability, which are accomplished by strict adherence to objective observation.

Because of individual differences

and variability, psychological data often require subjective

Zerga, op. clt., p. 19 ^. 16 17 18

Anonymous, op. cit. p. 26. Watkins, 6 p. cit., p. 168. Lender, op. cit., p. 15^.

^Walters, op. cit., p. 3 9 3 . 20 Max Freyd, “The Oraphic Re.ting Scale, “ Journal of Edu­ cational Psychology, 1^, February, 1923, p. 101.

9 treatment* Adams

21

submits that a scale is subjective when self**

consistency is appreciably higher than group-eonsistency* The ratio of consistency obtained by dividing group consist­ ency by self-consistency of ratings is invariably less than

..

1 00

An objective score depends on a minimum of personal 22 23 opinion. Wilke quotes Thorndike’s definition of objectivity and subjectivity: What science means by a perfectly objective scale is a scale in respect to whose meaning all competent thinkers agree. A perfectly subjective scale is one in respect to whose meaning all competent thinkers disagree (save by chance). These are the limits between which the actual scales known seem to live.... 24 Adams finally summarizes the issue: Our conclusion is that when objective scales are em­ ployed in such a way as to avoid the constant errors brought to light by psychological techniques, self con­ sistency and group consistency are equal inside the limits of error imposed by the experimental procedure. This is equivalent to saying that objectivity excludes the appearance of constant errors, while admitting random or chance errors.

21

H. F. Adams, “An Objeetivity-Subjectivity Ratio.for Scales of Measurement,*' Journal of Social Psychology, 1*

1930, p. 134. 22

H. F. Adams, “Validity, Reliability, and Objectivity," Psychological Monograph. 1936, 4y, p. 333. ^ W a l t e r H. Wilke, “The Reliability of Summaries of Rating Scale Evaluations of Student Personality Traits, " Journal Genetic Psychology. 53» 1938» P* 124. 24 Adams, "An Objectivity-Subjectivity Ratio for Scales of Measurement,“ op. cit., p. 129.

Validity is that property of a test by virtue of which it measures what it proposes to measure* 2*5 Methods of determining validity advocated by Driver ** *

/

are: (1 ) comparison with some absolute, direct measurement, (2 ) comparison with other tests purporting to measure the same ability,

(3 ) comparison with work samples,

of distribution of results,

(^) analysis

(5 ) analysis for halo effects,

and (6 ) follow-up procedures. One pitfall to avoid is reporting coefficients of correlation based on predictor information which is illicitly 26 27 used* Van Dusen 'states the following may be operating when the validity of criteria is low:

(1 ) lack of similarity

in ability measured and actual proficiency,

(2 ) measurement

of only a small percentage of the factors operating, and (3 ) changes in production requirements. Reliability is consistency.

Most used methods of com­

puting reliability are comparison of re-ratings with original ratings and the agreement existing among the ratings of a

2 ^R. S. Driver, “The Validity and Reliability of Ratings Personnel, February, 19 Al, 1 ?, p. 185*

26

R. M. Bellows, ‘‘Procedures for Evaluating Vocational Criteria,“ Journal of Applied Psychology. 25, 19^1, P* 185* 27 A. C. Van Dusen, ‘‘Importance of Criteria in Selection and Training, n Educational and Psychological Measurements, 11 7, 19^7, p. 504.

11 number of independent raters,

Bradshaw

28

found that raters

agree closely on traits of popularity, conceit, or leader­ ship, hut not ori emotionality, tact or honesty; als are easier to rate than others;

some individu­

objective traits are

easier to rate and are- more reliable. A multiple B of ,81 was obtained from ratings of an objective nature.

The items used were, personality, effi­

ciency, resourcefulness, cooperation, job attitude, sales 29 30 ability, organizing ability. Armstrong found correlation between efficiency and seniority is seldom as large as .60. Reliabilities for first and second ratings by foremen are in the order of .76 — .91 reported by Bradshaw.^ Some common correlations of reliability of traits: Leadership vs. force of personality Quickness of thought vs. retention Self-confidence vs. aggressiveness Regularity-persistence vs. trustworthiness Accuracy vs. quickness of thought Respect for authority vs. cooperation Self-confidence vs. capacity for leadership Self-confidence vs. cooperation

28

32

.8 3 .82 .80 .79 .79 .71 .6 8 .59

Bradshaw, op. cit., p. 15.

2^Bolanovich, op. cit., p. 22. 30 M. A. Bills, HA Method for Classifying the Jobs and Rating the Efficiency of Clerical Workers, 11 Journal Personnel Research. 1, Dec.- Jan., 1922-23, p. 386. 31 J Bradshaw, op. cit., p. 16. 32 Hughes, op> cit., p. 4-28.

From these correlations it is evident that the less objective or generally defined traits are those giving the lower co­ efficients. In a study by Sabador and Bennett-Spooled data of four judges gave a fairly constant order of merit.

An estimate

by one judge was found to be unreliable."' A coefficient of correlation of .9 0 for ratings requires ten or more judges. 36 Busmore reports a coefficient of .8^ for pooled judgments

35

of 86 raters and a coefficient of .7 7 for a jury rating of the same raters. IV.

SEX DIFFERENCE, RATER CHARACTERISTICS, HALO, WEIGHTING

Men raters tend to be more lenient than women raters. This is indicated by a study in which sixteen men teachers and twenty-five women teachers rated fifty boys and fifty girls.

The ratings were on sincerity, kindness, self-control

honesty, energy and intelligence.

No sex difference was

found for average ratings of appeals on judgments of

^ N a d i a Sabador and G. H. Bennett, “Employee Rating In­ sures Unbiased Opinions, “ Factory Management and Maintenance 1 0 3 , Dec., 19 1+5, p. 133jk Loc. cit. ^Bradshaw, op. cit., P* 58. J. T. Rusmore, “An Experimental Comparison of the Com­ posite and Jury Methods of Obtaining Group Judgments , 11 (un­ published Doctor's dissertation, The University of California Los Angeles, 194*0, p. 6 7 .

13 persuasiveness*

37 Another real difference reported is that

men resemble one another in their preferences, and women 38 are closer in respect^ to. their aversions. In a study; of self-ratings among students in the New York Public Schools, boys tended to make more of the socially undesirable traits 39 than did the girls* AO Husmore enumerates several cautions and observations of raters: (1) Persons tend to overrate friends on desirable traits* (2) Long-time observation is better than casual ob­ servation. (3 ) Raters should rate exclusively on past or present conduct. (^) Raters having some contact with item to be rated tend to agree more closely than less informed raters. (5) Raters should be trained* (6 ) Hate one trait at a time. (?) Some items are easier to rate than others. (8 ) Self-ratings are high on desirable traits and .low on undesirable ones.

H. L. Hollingsworth, *Judgments of Persuasiveness,* Psychological Review,, 18, 1911* p* 2 A 5 . 38 Ibid., p. 2A8. 39 J. P. Guilford, Psychometric Methods, (New York: McGraw-Hill Book Co., Inc., 193&), p. 501* Ao Rusmore, ojd . cit., pp. 2 1 - 2 2 .

(9) Eaters frequently are unable to Justify ratings, or give absurd rationalizations* (10) Median of several Judgments are more reliable than single Judgment. . (11) People differ markedly in their ability to make ratings* Raters often are reserved and self-conscious in mak4l ing self-ratings* Self-ratings are usually too high on desirable traits and too low on undesirable ones.

42

One

tends to rate his own sex higher on desirable traits and 44 vioe versa. J Hoffman says that the tendency to over­ estimation of one’s self is a function of the trait and not a fixed characteristic of the individual*

This is further

substantiated by ratings of associates who rate an individual as conceited* This individual is less apt to over-estimate 4^ 46 himself* -'Hoffman adds that possession of a given trait is negatively related to its over-estimation*

This characteris­

tic, however, may be explained in terms of the end error. 41 Hollingsworth,

op* cit., p. 243*

42

F. F* Bradshaw, ’’Revising Rating Techniques, * Per­ sonnel Journal, 1 0 , 1931, p. 234. ^ Loc* cit* Gr. J. Hoffman, ”An Experiment in Self-Estimation, " Journal of Abnormal and Social Psychology, 18, 1923, p. 49*

15 Graphic rating scales tend to eliminate the halo 4? effect. Another technique of reducing halo is suggested otl ^8 hy Bingham Since ratings on specific traits correlate closely with final estimates of personal fitness, why should interviewers be asked to record the trait? The answer is that an overall judgment is more likely to he correct if made after the raters attention has been focused suc~ eessively on several of the candidates specific traits. We eliminate halo as much as possible because its Aq effect produces unduly high correlation between traits. Weighting of specific items is logical since some items contribute more to the correlation with the criterion. However, if items do not possess an intercorrelation of .50 or greater, weighting has little effect on the final corre­ lation coefficient.

Also, the more disparate the weighting,

the higher the reliability coefficients.

50

The rater places

more confidence in the weighting if weights are assigned *1 prior to the ratings rather than subsequently.-'

4,7 Hoffman, og. cit. , p. 99* W. V. Bingham, BHalo, Invalid and Valid, * Journal of Applied Psychology. 2 3 , 1939* p. 2 2 3 . Bradshaw, ££. cit. . p. 1 5 . 50

Edwin E. CrhiBelli and Clarence W. Brown, Personnel and Industrial Psychology, (New York: McGraw-Hill Book Go., Inc.7 1 9 3 6 ) 7 p . T 13 :--51 L. H. Gozad, MThe Use of Performance Tests by the L. A. City Civil Service Commission,M Public Personnel Review, 2 , 19^1, p. 2 8 5 .

16 V.

CRITERION AND CONSTRUCTION OF THE SCALE

Ratings are subjective when used as a criterion. Criteria do not always exist ready-made, to be discovered and then used.

On the contrary, they must be developed and

constructed. ^ An example of constructing a criterion indicates the effort and Ingenuity involved.

For the prediction of success

at a Naval Training School, the first correlations with the criterion were low*

This pointed to a poorly selected test

battery, or else a poorly defined criterion. a change in the criterion was attempted*

Consequently,

Objective course

examinations and comprehensive final achievement tests were introduced. 53 higher.

As a result, the correlation was markedly

VI.

CONSTRUCTION OF RATING- SCALES

"A logical and even necessary starting point is that a scale measures something, and this unambiguously. 11

^ C . H. Patterson, “On the Problem of the Criterion in Prediction Studies, n Journal Consulting Psychology, 10, 1946, p. 285* ■^Lt.bComdr. D. B. Stuit, USNR, and Lt. Comdr. J. T. Wilson, USNR, “The Effect of an Increasingly Well Defined Criterion on the Prediction of Success at Naval Training School (Tactical Radar), Journal Applied Psychology, 3 0 ,

17 Walters-*-* wrote1 that compared with other rating scales, the graphic scale was considered best* because t h e .Judgments were more normal and reliable when re-rated after a short time interval;

the correlation between raters was higher; and

to a great extent, the graphic scale established a better defined scale of values which eliminated much Individual or subjective interpretation. The Scott Company-* ^reported a practical situation in which the same advantages were revealed: (1) The person who is making the judgment is freed from direct quantitative terms in making his decision of merit in any quality, (2) the person who is making the judgment can make as fine a discrimination of merit as he chooses. These two facts eliminate the restriction on natural judgments which other rating methods impose. Freyd^fsuggests: that the traits should be defined; that their extremities should be delimited;

that the item

should be in question form; that the scale should be long enough to be easily calibrated;

that no breaks or divisions

in the line should appear; that a line longer than five inches cannot be easily grasped as a unit; that the number of descriptive terms should be 3-5;

that end phrases and the

^Walters, op, cit. , p, 401, 56 D. Gr. Paterson, “The Scott Company Graphic Rating Scale," Journal Personnel Research, 1, Dec.-Jan., 1922-23, p. 3 6 3 * 57 Max Freyd, “The Graphic Rating Scale,“ Journal Edu­ cational Psychology, 14, February, 1923, pp. 99-100.

center phrase should not contain many words (in order to avoid end-effects and central tendency); phrase should be neutral; tribution, meaning;

that the center

that, in order to spread the dis­

the center phrases should be closer together in

that phrases should be short and to the point; that

no terms should be used such as, average, extremely, very, good, poor, fair; and that favorable extremes of scale should be alternated. Paterson*s^®study of industrial scales revealed that most firms used scales with 6 -1 0 qualities to be appraised. Some, however, used 12-15 and occasionally one encounters 59 as many as twenty qualities to be rated. Freyd enumerated advantages of the graphic sealet understood;

(1 ) it is simple and easily

(2 ) requires little motivation since it is in­

teresting to the rater; it is easy to score;

(3 ) it is quickly filled out; (4)

(5 ) it frees rater from direct quanti­

tative terms;, (6 ) it permits the rater to discriminate as finely as he cares;

(7 ) it is a universal scale;

(8 ) its

fineness of scoring can be altered at will; (9 ) it permits 60 comparable ratings. Bradshaw adds two significant considera-

Van Dusen, op. cit., p. 22. -^Freyd, op,, cit., p. 9 ^. 60 F. F. Bradshaw, “American Council on Education Bating Scale: Its Reliability, Validity and Use,“ Archives of Psy­ chology. 119, October, 1930, p. 2 3 .

19 tions in the scale construction:

seven seems the optimum num­

ber of steps; and in agreement with Symonds, each trait should 6l be rated on a separate sheet* Sabador and Bennett feel that a sum of ratings of personal qualities and performance on a three-step scale eliminates marked differences that are often found in performance ratings alone. Hughes

62

recommends a five-step group scale indicating

to the judges distribution percentages of 10, 20, 40, 20, 63 10, respectively. Champney and Marshall ^came to a somewhat different decision from that of Symonds or Bradshaw in respect to the optimum number of steps: ..•.limiting the scale to five or seven point scales may often give inexcusably inaccurate results. Perhaps it would not be too conservative to suggest the usual 18-24 step standard to be applied to a rating scale prac­ tice unless it is shown that for a particular job either accuracy is not desirable or discrimination beyond seven points is not to be attained. 64Wilke obtained ratings with a nine-step scale on 1,000 students scheduled for teaching.

Rated by the faculty

on personal characteristics shown during four years of college, only two percent were as much as two steps apart on a nine-step

61 62

✓ Sabador and Bennett, op. cit., p. 116

Hughes, op. cit., p. 425*

H. Ohampney and H. Marshall, “Optimal Refinement of the Rating Scale,“ Journal of Applied Psychology, 4l, 1932, p. 64 Wilke, pp. cit. , p. 314.

20 scale* A final check was submitted by Bradshaw.

(55

(1) Traits

differ in the success to which they can be rated depending on how objectively stated, unambiguous, and specific,

(2)

ratings are more reliable when a general trait is broken into specific factors,(3 ) “general all around value“ is frequently more reliably rated than are some of the more specific qualities involved,

(A) be sure the trait is not

a composite of several that vary independently,

(5 ) each

quality should refer to a single type of activity, and (6) ratings should not be used when more dependable data are ob­ tainable •

CHAPTER III DESIGN OF THE EXPERIMENT AND GROUP STUDIED I*

DESIGN AND SCALE CONSTRUCTION

Four conditions were employed.

In order to obtain

equally often appearance of each, item in respect to all others, two forms were necessary.

On Form A, the vertical

scale questionnaire, the labeling of the continuum of the scale for the six items was staggered in the following order: X _____ __

_

JL.

_

JL

On Form B, which also was vertical, the questions were reversed in

sequence _

X

X

-

tinuum were

reversed thus: *, -, -,

This

permitted each question

and the X

endsofthe con-

_

*,

-, to

* be

. rated anequal

number of

times, half with

tinuum at

the top of the scale and half at the bottom of the

scale.

the positive labeling of the con­

Every rater, who performed ratings on the vertical

scale, thus made three ratings under Condition I and three ratings under Condition II, regardless of whether he received Form A or Form B. In order to get a comparison with the vertical, Form C and Form D contained the same items on a horizontal scale. The order

of the continuum .

tion of Form B.

was staggered as in Form A,

,

And Form D follows the construc­

The items were reversed In order of sequence

22 from Form G and the labeling of the continuum was reversed in order, -

* -, *

- -i- •

This, as in the

vertical forms, permitted each question to be rated an equal number of times, half with the positive labeling of the con­ tinuum at the right end of the scale and half at the left end.

Also, as with the vertical scale ratings, each rater

made three ratings under Gondition III and three under Con­ dition IV, regardless of whether he received Form G or Form D.

These formats are in the Appendix. SCALE CONSTRUCTION ON ALL FOUR FORMS Vertical Scale Form A

1. (Ql)

A. »

Horizontal Scale

Form B 1. (0,6 >

•

Form (3 1

.

Form D

(ftD *-

1

.

(Q6)

— A.

JL,

2. (Q2)

JL

2. (Q5)

3. (03)

4.

3. (Of*)

w

JL

(0*0

,

4. (Q3)

5- (Q5)

X

5. (Q2)

6. (Q6)

JL •

6. (OX)

2. (Q2)

— JL •

2.

(Q5)

JL

3. (03)

— JL •

3.

(Q^)

JL — •

(Q3)

— A. •

JL «

JL

JL «

•S. ♦

(q M

A.—

,

mm

5. (0,5)

_

JL

5*

(Q2)

JL — •

6. (Q6)

JL —

6.

(ftl)

— A. •

Forms A and B contained identical items, but the items ap­ peared in reverse order and the labeling was reversed on the scale ends.

Forms G and D contained identical items (the

same items as those in Form A and Form B), but the items ap­ peared in reverse order and the labeling was reversed on the scale ends.

23 Form A was the reference for the order of the other three forms.

When Question 1 is indicated, that will al­

ways foe taken as Question 1 on Form A.

The same applies to

the other five items. Condition I was defined as all ratings on the vertical scales (Form A plus Form B) with the positive labeling at the top (*). Condition

II was

cal scales (FormA plus

defined as

all ratings on the verti­

Form B) with the positive labeling

at the bottom (?). Condition III was defined as all ratings on the hori­ zontal scales (Form C plus Form D) with the positive label­ ing at the right end of the scale (+-). Condition IV was

defined as

all ratings on the hori­

zontal scales (Form G plus Form D)with the positive label­ ing at the left end of the scale (--5-). II.

METHOD OF ITEM SELECTION

In order to make valid statistical computations for the items rated, each item should have a near-normal distri­ bution.

In selecting the items, the experimenter originally

devised thirty-nine questions of a controversial nature. The questions were submitted to twelve judges, graduate students, who understood the rank order method of rating. The judges were requested to rank the questions in terms of

Zk

the following instructions: In your opinion which question do you think will have the widest dispersion when submitted to a large population of college students for rating? Give that ques­ tion a rank order of 1. Then select the question which in your opinion will ha„ve the second widest dispersion when rated by the same college group. Give this question a rank order of 2. Continue in this manner until all questions have been ranked. Six questions having the smallest average rank order by this method were chosen as the items that appeared on the formats, Forms A, B, C and D.

In spite of the precau­

tions taken in selecting the six items, ratings by the 3^8 college subjects gave a markedly skewed distribution for each item.

Graphs for the distributions appear at the end

of Chapter IV. III.

THE GROUP AMD METHODS OF SAMPLING

All of the subjects were students at the University of Southern California enrolled in the introductory course in psychology.

This select population was a control under­

taken to limit the experience of the raters within as nearly similar background as seemed necessary.

Their ages

ranged from seventeen to thirty-seven years with threefourths of the number between the ages of twenty and twentyfive years. 8^ women.

The sampling provided ratings from 3^6 men and Eight men were dropped due to incomplete data.

The distributions deviated so greatly from the normal that computations of random sampling would not have been reliable

for so few women raters, so their ratings were not used. Six ratings toy each of the 318 men resulted in a total of 1908 ratings, A sufficient number of raters for random sampling statistics was obtained toy requesting each student in eight sections of the introductory course in psychology to make ratings on one of the four forms.

(See Appendix.)

An equal

number of each of the four forms was distributed among the students in each section sampled. Before distributing the forms a brief instruction in rating technique was given to each section of students, sample question,

A

MHow would you rate dancing compared with

other types of recreation?11 was used.

Rating for the four

forms was demonstrated on the blackboard.

As soon as the ex­

perimenter was confident that all students understood the procedure, they were instructed as follows: Please mark every question to the best of your ability. If any rating is omitted, it renders the other five worth­ less. You are not compelled to participate if for some personal reason you prefer not to. There is no way by which you can be identified; consequently, your ratings will be absolutely confidential. Results of the question­ naires will be given the professor after statistical anal­ ysis is completed. Then the forms were distributed, having been previously arranged so that every fourth student received the same form. There were only 8^ women students in the eight sections and since their ratings were distributed among four conditions it was not thought feasible to employ a test for significance

of difference among the formats for so few female subjects* Further consideration deemed it best to altogether reject the ratings done by the women on the basis of a possible sex dif­ ference existing for ratings* Eight men of the 326 male subjects failed to comply with the instructions;

their scales were Improperly marked,

or they failed to rate all six items, or some item was marked twice.

The 318 raters remaining were divided as follows;

Form A - 84; Form B - 71; Form 0 - 7 9 ;

Form D - 84.

The

uneven distribution was due to pulling out the ratings done by the women from the total. IV.

METHOD OF SCORING-

The four forms used (see Appendix) had six unbroken lines for rating the six questions. was ten centimeters long.

Each line or scale

Every question was scaled by the

student by placing a mark across the scale where he wished his answer to fall* When all of the ratings had been collected and exam­ ined to determine whether the raters had complied with in­ structions, the experimenter measured the linear distance with a millimeter rule from the positive extreme of the scale to the point at which the rater marked on the scale.

This

distance, always measured from the positive end, represents the rater’s score in millimeters.

Thus, the greater the

score, the greater the rejection of the item; or stated positively, the smaller the score, the greater the acceptance of the item. By measuring every scale rated, and tallying the scores in Table I and Table II, it was possible to construct dis­ tribution curves (at end of Chapter IV) for each item for each condition, for each item under all four conditions, for each condition including all six items, or for any com­ bination of composites of items or conditions.

28

TABLE I TALLY CHART FOR VERTICAL RATING-S

Frequencies

ftl

Condition 1 ^ 2 3 ^ 5

6

Q1

Condition 2 ? 2 3 ^ 5

6

2

0

1

l

0

0

6

0

2

0

1

0

8

1

0

2

8

0

11

0

6

0

1

1

21

2

3

2

7

0

18

2

9

1

3

b

8

5

6

6

2

0

6

1

6

3

2

2

A

3

8

10

A

0

12

17

10

19

9

1

5

13

10

25

7

1

7

8

2

15

10

3

11

7

11

12

8

1

9

9

11

16

1

2

5

27

13

lA

8

b

7

17

lb

10

13

7

7

11

10

10

18

lb

6

11

12

lA

11

0

15

9

2

22

51

2

6

12

17

53

71

8¥

71

71

8A

W

3

8¥

71

55J

^55---930

29

TABLE II TALLY CHART FOR HORIZONTAL RATINGS

Frequencies Condition 3 Score in mm.

Condition 4 --f-

0,1

2

3

^

5

6

Q1

2

3

^

5