A study of the role which several factors play in determining the attitudes of judges toward internal organs and reliability of these attitudes as expressed on a rating scale

Citation preview

A STUDY OF THE ROLE WHICH SEVERAL FACTORS PLAY IN DETERMINING- THE ATTITUDES OF JUDGES TOWARD INTERNAL ORGANS AND THE RELIABILITY OF THESE ATTITUDES AS EXPRESSED ON A RATING SCALE

A Thesis Presented to the Faculty of the Graduate School The University of Southern California

In Partial Fulfillment of the Requirements for the Degree Master of Arts in Psychology

by Robert Walton Troutman January 19^0

UMI Number: EP64002

All rights reserved INFO RM ATION TO ALL USERS The quality o f this reproduction is dependent upon the quality o f the copy subm itted. In the unlikely event that the author did not send a com plete m anuscript and there are missing pages, these will be noted. Also, if m aterial had to be removed, a note will indicate the deletion.

Dissertation Publishing

UMI EP64002 Published by ProQ uest LLC (2014). Copyright in the Dissertation held by the Author. M icroform Edition © ProQ uest LLC. All rights reserved. This w ork is protected against unauthorized copying under Title 17, United States Code

ProOuesC ProQuest LLC. 789 East Eisenhower Parkway P.O. Box 1346 Ann Arbor, Ml 4 8 1 0 6 - 1346

T h is thesis, w r i t t e n by

....... under the guidance of h.%3... F a c u lty C o m m ittee, and app ro ved by a l l its members, has been presented to and accepted by the C ouncil on G raduate Study and Research in p a r t ia l f u l f i l l ­ ment of the requirements f o r the degree of

...............

....... in. Pjsyclmlogy............. Date..

Faculty Committee

hairman

TABLE OP CONTENTS GHAPTER I.

PAGE

THE PROBLEM AND DEFINITIONS OF TERMSU S E D ....... The p r o b l e m........... Statement of the problem

1 1

............

1

Importance of the s t u d y .............

1

Definitions of terms u s e d ...............

2

Panel method of judgment .................

2

Panel method of judgment within present s t u d y ........

2

Internal organs .......................... II.

2

REVIEW OP THE LITERATURE........ ..............

3

Literature on the use of ratingscales ........

3

Literature on the effect of prestige value on 6

ratings...... Literature on the use of the panel method of judgments .......................... III.

MATERIALS, RATERS, AND EXPERIMENTALCONDITIONS .. Rating magazines

.................

Experimental conditions

12 12 12

The materials..... Instructions to r a t e r s

9

.............. ..............

Rating organizations for prestigevalue ....... The materials .............................

12 13 llj. llj.

ii

CHAPTER

PAGE Instructions to raters Experimental conditions

IV.

.............. .....

METHODS AND RESULTS OF THESTUDIES Over-all merit

15> ............

.........

Reliability

17

.....

19 22

Effect of prestige.......... V.

1? 17

..........

Effect of physical factors

l£

SUMMARY AND CONCLUSIONS..............

23

Summary ..................

23

Conclusions..............

25

BIBLIOGRAPHY................................. ..

26

APPENDIX A - RAY/ DATA

29

Contest results Prestige results

............. ....

29

......................

30

APPENDIX B - PEARSON’S PRODUCT-MOMENT COEFFICIENT WORK SHEETS.......

31

Individual ratings correlated against groupratings ..

31

Group reliability (odd-even correlation)

3$

......

Physical factors correlated against groupratings ....

39

Prestige values correlated against group ratings

lj.0

LIST OF TABLES TABLE I* II.

Correlation of Individual with. Entire G-roup Effect of Physical Factors

........

CHAPTER I THE PROBLEM AND DEFINITIONS OF TERMS USED For the past few years a contest known as the industrial Editors* Annual Contest11 has been conducted at the University of Southern California utilizing the judgments of advanced industrial psychology students.

It has been the

purpose of this contest to rate a series of internal organs (company magazines) on the basis of several qualities, with the eventual objective of selecting from among the group that publication deemed to be the most excellent. I.

THE PROBLEM

Statement of the problem.

It was the purpose of this

study (1) to discover what effect, positive or negative, the physical components of the magazines had on the ratings they received; (2) to discover* what effect, positive or negative, the prestige value of the company publishing each magazine had on its rating; and (3) to determine the reliability of the judgment s• Importance of the 3tudy. importance.

This study has a two-fold

In a broad sense it shows how much influence the

physical make-up of a magazine, and the prestige value of the company behind the magazine exert in determining the ratings

2

accorded it, as well as something of the reliability of the panel method of judgments.

-In a more restricted sense it

points up the limitations and weaknesses in the present method of selecting judges for the industrial Editors1 Annual Contest.n II.

DEFINITIONS OF TERMS USED

Panel method of judgment.

This is a method of obtain­

ing judgments from a selected group, which requires that its members render judgments over a period of weeks, months, or even years. Panel method of judgment within present study.

In

this study selected groups were used, but asked to render only one series of judgments. Internal organs.

Internal organs are company magazines

directed toward the personnel of the organization concerned.

CHAPTER II REVIEW OP THE LITERATURE Literature on the use of rating scales.

For many

years rating scales have been used as a means of appraising various aspects of people, objects, and things.

Shortly

before the turn of the century, Major,*** using a rating method which he called the method of isolated exposure, evaluated colors with respect to their effective value.

Leaving the

field of psychology for a moment, and glancing backwards, one finds that the British Navy used rating devices to obtain estimates of wind velocity in l8o5*^ Some forty years ago V. A. G. H enmon^ had a group of subjects judge the respective lengths of a series of lines, and following each judgment asked them to "assign the degree of confidence in the accuracy of the judgment, a for ’per­ fectly confident1* b for .’fairly confident’, c for ’with little confidence’, and d for 1doubtful1•"3

in this instance

1 D. R. Major, "On the Affective Tone of Simple Sense Impressions," The American Journal of Psychology, 7:S7-77>

1895.

2 J. P. Guilford, Psychometr1 c Methods. New Yorks McGraw-Hill Book Company, IncV, 1936, p. 2614..’ 3 V. A. C. Henmon, "The Relation of the Time of a Judgment to Its Accuracy," Psychological Review, 18:186-201, I9II#

if.

he used a modified graphic scale, although he did not refer to it as such# About the same time E. K. Strong,^- in testing recog­ nition memory, used a modified graphic rating scale#

So

modified was this scale that he got entirely away from the pencil and paper aspects ordinarily accompanying this type of rating#

Strong had his subjects rate a series of adver­

tisements, to which they had been previously exposed, by placing each advertisement in one of four piles#

The respec­

tive piles were designated in the following manner:

(1) for

those advertisements the subject was absolutely sure he had seen; (2) for those advertisements the subject was reasonably sure he had seen; (3) for those advertisements the subject had a faint idea he had seen; and (I4.) for those advertisements the subject couldn*t remember having seen. Some years later in 1921, F. A. G* Perrin,^ used this approach when he had his judges rate individuals for physical attractiveness on a seven point scale ranging from 1, very low possession of the characteristic under consideration, to 7, very high possession of the characteristic under

t E# K. Strong, lfThe Effect of Length of Series Upon Recognition Memory,n Psychological Review, 19:^1*7-62, 1912. £ F* A. C. Perrin, "Physical Attractiveness and Repulsiveness,11 Journal of Experimental Psychology, i|.:203-17, 1921#

5 consideration* Mor has the use of this modified graphic scale been limited to these three studies*

L* A. Weiss, the author

of a rather elaborate classification of existing rating scales, has the following to say concerning scales of this nature: Some of the most successful of recent rating scales seem to have retained all the principles of the graphic scale except for the actual use of the dot-on-a-line idea, and to have combined these graphic scale princi­ ples with those of the ranking method*° P. H. Furfey? reports a study in which such a scale was used in rating a group of seventy-five boys on such qualities as changeability of mood, reaction to authority, and type of games played*

The judges were two recreational

leaders who were well acquainted with the boys concerned* When the ratings of one of the judges were correlated with those of the other judge, a reliability coefficient of .888 resulted, and the reliability of their combined ratings, as computed by the Spearman-Brown formula, was *9l4-0>* These reliabilities are certainly gratifying, but some question arises concerning the number of judges involved*

6 L. A* Weiss, “Hating Scales,” Psychological Bulletin, 30:185-208, 1933* 7 Paul Hanly Furfey, “An Improved Rating Scale Technique,*1 Journal of Educational Psychology, 17:l4-5-li8, 1926.

6 Symonds® maintains, ”The number of independent ratings which, should be obtained may be determined by means of the Spearman-Brown formula11 and goes on to list eight as the average number of independent ratings necessary in order to obtain a reliability of *900.

The work of Remmers, 9 done

in common with Shock and Kelly, and later work done by Remmers,^*0 would tend to indicate that if the judges are comparable, an assumption frequently made without sufficient grounds, the reliability of their combined judgments will increase with the number of judges according to the SpearmanBrown formula. In the present study, the ratings of twenty independ­ ent judges were utilized, so no question should arise as to the adequacy of the number of judges involved. Literature on the effect of prestige value on ratings. In a study designed to determine the effect of a previously established ”group standard11 upon a new group, Asch, Block,

8 P. M. Symonds, Diagnosing Personality and Conduct. New York: The Century Company , 1932 , P* 96• 9 H. H. Remmers, N. W. Shock, and E. L. Kelly, ”An Emperical Study of the Validity of the Spearman-Brown Formula as Applied to the Purdue Rating Scale,” Journal of Educational Psychology, 18:187-95, 1927. 3*0 H. H. Remmers, ,fThe Equivalence of Judgments to Test Items in the Sense of the Spearman-Brown formula, ” Journal of Educational Psychology, 22:66-71 » 1931.

7 and Hert^man^ asked a mixed class of ninety-six students to place ten professions in rank order for each of the following characteristics:

Intelligence; social usefulness; conscien­

tiousness; stability of character; and idealism.

They then

averaged the obtained rankings and transformed them into rankings from 1 to 10.

Following this the authors calculated

the correlations between the various rankings and found intelligence to correlate .92 with social usefulness, .71 with conscientiousness, .73 with stability of character, and .67 with idealism.

The same experiment was then conducted with

a new group consisting of sixty students, except the members of this second group were informed of the rankings accorded intelligence by the former group.

The rankings obtained

from the second group were treated by the same statistical methods as were those obtained from the first group with subsequent correlations of *90 between intelligence and social usefulness,.75 between intelligence and conscientious­ ness, .81 between intelligence and stability of character, and '.67 between Intelligence and idealism.

As may be noted,

the correlations between intelligence and the various other variables increased for the second group, with the exception

^ Solomon E. Asch, Helen Black, and Max Hertzman, 11Studies in the Principles of Judgments, and Attitudes: I. Two Basic Principles of Judgment,w The Journal of Psychology, 5 • 219-51> 193$*

8 of the correlation between intelligence and social usefulness, which decreased a very small amount*

No mention was made as

to the significance of these differences*

In this study the

authors drew the following conclusion: It is concluded that presentation of the group stand­ ard is responsible for the increase of the correlations* We suggest that a standard, functioning as a frame of reference, may produce organization at a higher level, when it carried with it the sanction of public approval, than when the same or a similar standard is evolved by the individual himself. This conclusion is qualified in the following manner: Attention should be called to the fact that, in com­ paring the results of the two experiments, the group of Experiment I is considered a control. We feel justified in making this assumption in view of the fact that the two groups were drawn from the same general population. No adequate reason can be assigned for expecting these groups to differ systematically under similar experimental conditions. Hence we conclude that differences between the two groups are to be referred to differences in experimental conditions. Farnsworth and Misumi,-^ in an attempt to discover the influence of artists1 names on the preference value of pictures, selected from.a student1s check list of fifty pictures the four best known and the four least known.

They

then had two groups of students rate a series of paintings, which in some cases had the well-known artists1 names attached and in other cases the little known artists1 names

12 P. R. Farnsworth and I. Misumi, "Further Data on Suggestion in Pictures," The American Journal of Psychology,

143:632, 1931.

attached.

The authors concluded that paintings attributed

to well-known artists were more often preferred, although the differences were not significant. Sheriff determined preferences of subjects for six­ teen authors, and then assertained preferences for sixteen literary passages when one passage was attributed to each author.

Although all sixteen of the passages were from the

same author, preference for the respective passages varied with respect to the prestige value of the author to whom it had been attributed.

The differences again were not

significant, but did occur in the expected direction. Li ter atxire on the use of the panel method of judgments.

Little material of a scientific nature is to be

found concerning the panel method of judgments.

One becomes

impressed with this fact upon checking the available literature. Material of a non-scientific nature is present in some abundance, but consists mainly of lfarm chair discussions, containing little or no data. The consumer-panel technique, a form of the panel method of judgments, has been used in market research for

-**3 m . Sherif, ”An Experimental Study of Stereotypes,” The Journal of Abnormal and Social Psychology, 29s371-75*

1935-

10

the past ten or twelve year s. ^

4 panel, known as the

Junior Fashion Council, was formed by Eaton* s Toronto de­ partment store in 1939* ^

It was composed of twelve girls,

selected from families numbered among Eaton*s long-standing customers, and its purpose was that of supplying information concerning lfteen-age trends, likes, and dislikes.1* So successful was this panel, that it was enlarged in lpij-O to include representatives from every high school in Toronto, and in 19i]-l a similar panel was formed for boys.

These

panels have proven to be "an unequalled source of buying information.11 In 19^-7 the New York Sun set up a panel to measure consumer buying in department s t o r e s . 1 >35>6 families, comprising 85 per cent of the families living within a fiftymile metropolitan trading zone, constituted the panel.

Such

findings as resulted from the study were validated by comparing them with Federal Reserve figures.

Comparison was

made between the newspaper* s own estimate of department

R. H. Ganly and R. D. G-risp, 11Consumer Purchase Panels Serve Advertisers, Agencies, and Media," Printer* s Ink, Part I, 57s29-30, 19^7. Lawrence McCrocken, lfCollege Students Point Way to Multi-million Sales,11 Printer* s Ink, Part II, 225si-l*0-J+2, 19W. 3-6 H. G-anly, !fConsumer Panel Shows Apparel Home Furnishing Buying In New York Department Stores," Printer* s Ink, Part II, 1:3^-36, 19U8.

11 store sales as based upon panel results, and actual depart­ ment store sales as determined through, the government1s records.

The Federal Reserve figures showed the average

amount spent on department sales per family per week to be Ilf..53*

3?he Hew York Sun1s per-family-week department store

sales figure was fc>.ll.

Since it might be expected that

department store sales in Hew York would be a little higher than through-out the rest of the nation, the Sun’s estimate of $5*11 might be considered quite close. The Industrial Surveys Company uses a panel approach in the form of trThe Hational Consumer Panel,” and has the following to say concerning this method. Vi/hen used by Intelligent management, the panel becomes the most accurate and evidently the only device by which we may accurately study total retail commodity movement. . Aside from the many additional values which may be derived, we have reasoned that the panel technique, with an average 97 per cent accuracy in measuring the total market for consumers1 goods, is better than techniques theoretically technically perfect which omit any estimate of movement of from J>0 to $0 per cent of the total market for various commodities.

17 s. G. Barton, The Movement of Branded Goods to The Consumer. A. B. Blankenship, editor How to Conduct Consumer and Opinion Research. Hew York: Harper anT^Brothers Publishers,

1^6, ppT”So^r:—

CHAPTER III MATERIALS, RATERS, AND EXPERIMENTAL CONDITIONS I.

RATING MAGAZINES THE MATERIALS

Thirty-nine internal organs published by thirty-five* organizations were used in this study.

Each of these organ­

izations Is represented in Southern California and some, In addition, are represented nationally. INSTRUCTIONS TO RATERS Twenty students, members of a graduate seminar in applied psychology, were assembled and given the following Instructions: Tonight we are going to engage in an activity which will give you valuable experience as an industrial psychologist and will, at the same time, render a valued service to the ’’Industrial Editors’ Annual Contest.ff You are asked to rate the quality of each of a group of publications on the basis of the skill and effective­ ness with which their editors have practiced sound employee relations through their respective publications. Since quality of paper, size of magazine, colored pic­ tures, etc., are matters of budget rather than of skill, these elements should be disregarded in making your ratings.

* Four of the organizations published two magazines.

13 Rat© the content of the publications on the basis of bow effectively they interest the employee in and inform him on each of the following: a.

Company policies affecting the employees.

b. Products manufactured or services rendered. In the case of a retail organization also take into consid­ eration the merchandise sold. c. Management's plans which affect the'rank and file of the employees. d. Recognition of individual employees for outstand­ ing accomplishments, not only in the performance of their jobs but in the life of the community. e. Evidence that the line of communication works in both directions, and that management is sincerely inter­ ested in having the opinions and attitudes of its employees openly and honestly expressed. Write the name of each publication on one of the white cards before you. Study the publication carefully for each of the points mentioned above and then rate each point on the following scale: 5 - very good I4. - good 3 - average 2 - poor 1 - very poor Sign your name and give the date on the bottom line of each card. EXPERIMENTAL CONDITIONS To insure that the criteria by which the magazines were to be judged, and the method by which the judgments were to be recorded would be constantly available to the

Illgroup, they were written on the board in full view of the entire class* The answers to any questions arising during the rating session were directed to the group as a whole, in an attempt to keep the conditions as standard as possible* The raters were given five minutes time in which to examine any particular magazine and to record their judgments. At the expiration of this time period and on the word fpass», each rater passed the magazine then in his possession to the individual on his right, and accepted from the individual on his left the next magazine to be rated.

This .process con­

tinued until each rater had in his possession the magazine with which he had started. In view of the fact that rating thirty-nine magazines at one sitting would be quite an arduous task, so arduous possibly as to result in a drop of motivation sometime during the process, the rating was done in two periods.

During

the first period nineteen of the magazines were rated, and during the second period, a week later, the final twenty magazines were rated. II.

RATING ORGANIZATIONS FOR PRESTIGE VALUE THE MATERIALS

The materials consisted of randomized lists of the

15 thirty-five organizations involved in the main study* INSTRUCTIONS TO RATERS Seventeen students, members of an advanced course in psychology, were presented with a list of organizations and given the following instructions: The following is a list of various organizations. You are to place these organizations in rank order with respect to prestige value. The organization having the greatest prestige value for you will be assigned number 1, that organization having the next greatest prestige value for you will be assigned number 2, etc. Should one or more of the organizations be unknown to you, rank it or them at the end of the list and indicate your unfamiliarity by underlining the organization in question. Study the entire list before you start the ranking process. Webster defines prestige as: "Power to command admiration. Ascendancy derived from general admiration or esteem; commanding position in men’s minds.11 EXPERIMENTAL CONDITIONS The answers to such questions as arose during the ranking process were directed to the group as a whole in an attempt to keep conditions as standard as possible* The judges were given one hour during which to place the thirty-five organizations in rank order* The judges used in this study, with the exception of one judge, were not the same as those used in the original rating process, and ideally they should have been the same. It has been assumed, however, that the two groups were

16 comparable in so far as each was constituted entirely of advanced students in an applied psychology course*

CHAPTER IV METHODS AND RESULTS OP THE STUDIES OVER-ALL MERIT The total number of points assigned each magazine was computed in order that the winner of the "Industrial Editors* Annual Contest” might be selected*

The raw data, along with

these totals, is contained in Appendix A* RELIABILITY After determining the contest winner, the correlations existing between each individual rating and a composite of the group ratings were computed by means of the Pearson r. The correlations thus obtained are shown in the column entitled "Pearson r" of Table I*

These correlations, which are

actually reliability coefficients showing the degree of agree­ ment between each rater and all the raters, vary considerably and in many cases are quite low*

Actually these correlations

are too high, for when a measurement is correlated against a whole of which it is a part, some positive correlation occurs because the variance of the total is in part made up of the variance of the component*

In Table I the column entitled

"Adjusted Pearson r" shows the same correlations after having been corrected for this spurious correlation.

The trend thus

18 TABLE I CORRELATION OP INDIVIDUAL WITH ENTIRE CROUP INDIVIDUAL

PEARSON r

ADJUSTED PEARSON r

1

• 2736

• 1843

2

.2 0 7 0

• 0846

3

.3980

.3318

4

•1463

• 3804

5

.5398

•4 6 6 6

6

.6 3 8 3

• 5676

7

.1 8 2 1

• 0982

8

• 7246

• 6776

9

.0054

-•0 6 8 8

10

.6711

.6 1 4 7

11

•9971

.9961

12

.5609

•4942

13

.3339

.2 0 65

l4

• 1054

.0 1 9 3

15

.4680

• 3952

16

.5047

•4 2 6 6

17

• 8666

• 8315

18

• 6 ll5

• 5222

19

• 6434

•55oo

20

• 26 $ l

• 1737

19 revealed tends to substantiate the ubiquitous contention that individuals vary rather markedly, and forces one to look to combinations of raters for greater reliability* In accordance with this notion, the raters were divided into two groups by the odd-even method, and the averaged ratings of the one group were correlated against the averaged ratings of the other group*

A Pearson r of *71 was obtained.

However a whole series of judgments is more reliable than either half, and in general there is an increase in reliability 18 going with the increased length of the whole series* So a correction in the form of the Spearman-Brown formula was applied and the reliability coefficient became *83. EFFECT OF PHYSICAL FACTORS To determine what effect physical factors, grade of paper used in the magazines, number of colored pictures con­ tained in the magazines, number of pages in the magazines, etc*, had on the raters, each'magazine was evaluated on its physical merits and assigned a certain number of points. To facilitate this evaluation, the scale contained in Table II was constructed*

This scale was so set up as to

give a heavier weight to those physical factors which would

^ J* P. Guilford, Fundamental Statistics In Psychology and Education* First Edition; l^ew York: kcG-raw hTTi book Company, Inc., 19^2, p. 282*

20 be expected to exert a greater influence upon the individuals involved in the rating process, and also to allow for the degree to which any such factor was present in any particular magazine•

Thus, slick paper, which is commonly associated

with higher type publications as opposed to pulp publications, was given a weight of three points*

Sketches, black and

white pictures, and colored pictures were weighted on a slid­ ing scale whereby black and white pictures were considered more influential than were sketches, and colored pictures were considered more influential than were black and white pictures; the weight also depended upon the'number of pic­ tures and sketches present in the magazine under consideration* If an internal organ had no colored pictures, but did contain some color, perhaps in the form of borders around black and white pictures or in the form of colored printing, it was assessed one point* Those publications which were printed were considered to be much more attractive than were those which were mimeographed and were given a weight of four points.

If a

magazine had a formal cover, such being defined as any sort of cover which prevented the text of the magazine from appearing as part of the front page, it was given two points. Finally, it was felt that a thicker magazine would exert more influence than would a thinner magazine and accordingly each organ was weighted with respect to the number of pages

21 TABLE II EFFECT OF PHYSICAL FACTORS PHYSICAL FACTORS 1* Slick paper 2* Sketches A few (One sketch to one-half as many sketches as pages) Moderate (One-half as many sketches as pages to as many sketches as pages) Many (As many sketches as pages to more sketches than pages) 3* Black and white pictures A few (One picture to as many pictures aspages) Moderate (As many pictures as pages to twice as many pictures as pages) Many (Twice as many pictures as pages tothree times as many pictures as pages) Ij.. Colored pictures A few (One picture to one-fourth as many pictures as pages) Moderate (One-fourth as many pictures as pages to one-half as many pictures as pages) Many (One-half as many pictures as pages to threefourths as many pictures as pages)

POIHTS 3

1 2 3 2 3 k.

I}5 6

5* Other color (Color present In publication other than in pictures)

1

6 * Printed (As opposed to mimeographed)

1}.

7* Cover (As opposed to no formal cover)

2

8. Humber of pages I to 10 II to 20 21 to 30 31 and up

2

22

it contained. The number of points assessed the magazines were in turn correlated against the original points accorded the magazines by the twenty judges. •00i|., was not significant.

The resulting correlation,

It appears, therefore, that the

physical attributes of the various magazines had little effect on the raters. EFFECT OF PRESTIGE In order to establish a prestige value for each organ­ ization, the prestige ranks assigned it were summed.

In

those cases where an individual was unfamiliar with some of the organizations, and did not rank them, they were put at the end of that particular list and given equivalent ranks. The resulting prestige values were correlated with the original points accorded each magazine.

The resulting

correlation, .039* was not significant, and strongly suggests that the prestige of the various organizations had very little effect upon the judgments rendered.

CHAPTER V SUMMARY AND CONCLUSIONS I.

SUMMARY

Each of thirty-nine internal organs was rated on a five point scale with respect to how effectively it inter­ ested the employee in and Informed him on the following: a* company policies affecting the employee; b. products manufactured and services rendered by the organization; c. management’s plans which affected the rank and file of the employees; d. recognition of Individual employees for outstanding accomplishments; e. evidence that the line of communication worked in both directions. The raters were twenty advanced psychology students, who met on two different occasions to do the rating. The ratings so obtained were correlated each in turn against a composite of all the ratings and individual con­ sistency was found to be lacking, although group consistency, as determined by the split-half method was of the order .83. The ratings were also correlated with the physical attributes as measured by physical indices set up logically, and a positive correlation too small to be significant resulted. An effort was made to measure the effect of .prestige

upon the ratings accorded the various magazines*

A group

of seventeen subjects, comparable in education and age with the group utilized in the original experiment placed the thirty-five organizations involved in rank order for prestige value*

These rankings were correlated Y/ith the original

ratings, and the resulting positive correlation was too small to be significant* II.

CONCLUSIONS

The physical attributes of the various magazines had little or no effect upon the raters, and the same may be said of the prestige values of the concerns publishing the magazines considered* Individual raters tend to show low reliability, while groups of raters demonstrate increased reliability. The suggestion seems Y/arranted on the basis of this study, that the present method of selecting judges for the industrial Editors1 Annual Contest11 be modified in such a manner as to permit the selection of a more homogenous group.

Such action would result in higher individual re­

liability and concomitantly higher group reliability.

The

methods by which such a goal could be achieved are outside the scope of this thesis.

B I B L I O G R A P H Y

26 BIBLIOGRAPHY'

Asch, Solomon E., Helen Black, and Max Hertzman, "Studies in the Principles of Judgments and Attitudes: I* Two Basic Principles of Judgment,” The Journal of Psychology, 5: 219-251, 1936, : ------------ ----Barton, S. G. , The Movement of Branded Goods to the Consumer. A. B. Blankenship, editor, How to "Conduct’ TJonsumer and Opinion Research. Hew York: Harper and Brothers Publish­ ers, 195-^>* 31I4.PP* Farnsworth, P. R. and I. Misumi, "Further Data on Suggestion in Pictures,” The American Journal of Psychology, 5-3:632, 1931* Purfey, Paul Hanly, ”An Improved Rating Scale Technique,” Journal of Educational Psychology, 17:5-S“5-8, 192o. Ganly, R. H. and R. D. Grisp, "Consumer Purchase Panels Serve Advertisers, Agencies, and Media,” Printer’s Ink, Part I, 57:29-30, 195-7. r; Ganly, R. H., "Consumer Panel Shows Apparel Home Furnishing Buying in New York Department Stores,” Printer’s Ink, Part II, 1:3^-36, 195-6• : Guilford, J. P., Psychometric Methods. New York: McGraw-Hill Book Comp any, inc ., 19 36. 5e>6pp. ____________ , Fundamental Statistics in P sychology and Education. First Edition; New York":"“McGraw-Hill Book Company, inc., 195-2. 333PP* Henmon, V. A. C., ”The Relation of the Time of a Judgment to Its Accuracy,” Psychological Review, 18:186-201, 1911. Major, D. R., ”0n the Affective Tone of Simple Sense Impressions,” The American Journal of Psychology, 7:5777, 1895. : McCrocken, Lawrence, "College Students Point Way to Multi­ million Sales,” Printer’s Ink, Part II, 225*5-0-14-2, 195-8. Perrin, F. A. C., "Physical Attractiveness and Repulsiveness,” Journal of Experimental Psychology, 5**203-17, 1921. Remmers, H. H., N. W. Shock, and E. L. Kelly, "An Bmp eric al

27 Study of the Validity of the Spearman-Brown Formula as Applied to the Purdue Rating Scale,” Journal of Ed­ ucational Psychology, 18:187-95, 1927* Remmers, H* H., ”The Equivalence of Judgments to Test Items in the Sense of the Spearman-Brown Formula,” Journal of Educational Psychology, 22:66-71, 1931* Sherif, M*, ”An Experimental Study of Stereotypes,” The Journal of Abnormal and Social Psychology, 292 371-75, WW* Strong, E» K., ”The Effect of Length of Series Upon Recog­ nition Memory,” Psychological Review, 19:lfl|7-62, 1912. Symonds, P. M*, Diagnosing Personality and Conduct» The Century Company, 1932. 602pp.

New York:

Weiss, L. A., ”Rating Scales,” Psychological Bulletin, 30: 185-208, 1933. ----- --------

A P P E N D I X

29

APPENDIX A - RAW DATA CONTEST RESULTS

MAGAZINES 1 2 3 4 1 2 3 4 5 6 7 S 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

10 16 14 13 15 12 15 14 15 10 17 8 13 13 11 14 17 13 10 15 16 16 18 14 11 12 9 11 15 14 16 22 n 10 14 12 13 10 16

10 19 15 12 12 12 15 7 17 12 17 20 15 15 11 13 12 11 9 7 15 15 9 9 17 20 8 13 5 18 20 16 14 12 15 12 12 12 14

12 20 11 16 15 17 22 13 17 13 19 15 18 17 16 10 22 10 17 12 15 16 16 14 16 14 11 11 14 20 10 19 14 15 11 14 13 11 13

15 15 14 15 17 16 15 14 15 13 10 13 14 18 15 13 11 11 16 15 13 17 14 11 12 7 11 14 14 11 14 13 13 13 11 13 11 8 12

5

6 7

'- - RATERS 8 9 10 11 12 13 14 15 16 17 18 19 20 TOTAL

14 20 15 21 19 23 24 20 22 23 21 20 19 19 19 12 20 17 15 13 21 16 18 18 19 16 18 17 17 17 16 17 20 22 15 17 14 20 16

12 16 11 15 17 17 11 12 13 17 20 15 19 14 15 9 17 13 13 13 18 18 12 18 21 8 14 14 11 11 15 12 14 10 16 10 15 10 19

14 16 16 17 19 17 13 13 10 12 10 16 17 14 16 7 16 18 14 15 13 18 17 17 17 12 16 14 18 16 17 18 15 12 20 15 15 18 20

17 18 18 20 17 22 19 17 12 17 22 23 19 17 18 13 13 20 15 17 19 18 17 16 19 22 20 13 19 20 17 21 17 17 20 18 19 16 19

13 15 14 15 14 13 12 14 13 10 13 11 12 18 17 13 10 17 10 13 18 13 16 14 17 14 11 13 16 16 9 10 15 13 11 14 12 12 11

13 19 14 16 16 15 9 14 12 11 18 19 18 10 13 12 22 14 10 13 17 16 18 15 17 11 16 13 18 15 15 15 16 16 14 15 13 14 16

12 13 14 13 18 12 14 17 16 16 16 22 12 15 16 10 16 13 16 12 13 17 14 17 16 13 16 13 15 14 17 16 20 18 15 14 19 14 20

12 14 12 18 17 15 14 10 10 14 21 19 14 13 10 12 12 14 12 15 19 16 16 15 12 11 16 17 17 14 16 12 17 14 18 14 18 13 19

7 18 15 11 9 15 12 16 11 9 15 15 11 15 20 5 6 17 12 9 9 17 18 19 16 13 10 12 11 9 11 15 17 16 19 17 19 8 14

18 16 17 19 20 22 18 11 14 13 18 20 16 18 18 12 18 17 15 21 20 21 18 20 19 19 20 14 19 18 20 19 18 20 19 21 15 17 20

14 18 13 17 17 18 13 9 11 15 14 18 22 14 16 13 14 15 13 16 19 19 20 19 15 14 14 15 18 11 14 16 15 16 16 15 18 17 15

15 15 11 13 15 17 14 12 11 18 15 17 19 13 16 12 13 14 15 14 16 13 19 17 16 11 16 10 14 14 17 16 14 12 15 14 16 12 21

10 17 12 16 20 15 11 13 11 15 20 25 22 15 12 9 16 12 12 10 19 18 14 17 19 13 22 11 16 11 22 17 15 15 13 23 19 13 21

11 18 11 17 17 13 9 8 12 10 22 24 16 12 14 11 12 14 16 14 10 20 12 13 12 16 19 11 17 17 15 14 12 11 20 15 17 8 19

12 15 11 10 18 14 18 11 8 9 20 20 21 12 16 7 12 21 17 18 22 15 17 17 16 9 21 9 18 13 15 20 13 16 19 13 13 11 21

18 17 11 13 10 15 13 16 18 12 15 20 16 14 19 14 16 12 10 12 16 17 13 13 15 12 20 15 16 15 17 15 18 12 14 15 19 17 22

259 335 269 307 322 320 291 261 268 269 351 360 333 296 311 221 295 293 267 274 328 336 316 313 322 267 298 260 308 294 313 323 308 290 315

301 310 264 348

30 APPENDIX A - RAW DATA PRESTIGE RESULTS

ORGANIZATION 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 26 28 31 32 33 34 35 36 37 38 39

1

24 24 24 10 (2 5 )* 24 24 2 (2 9 )* 24 (3 0 )* 4 7 24 8 24 24 12 9 13 24 20 (2 7 )* 16 22 1 15 23 24 24 5 3 14 19 6 21 11 18 17

2

3

4

5 6 7

27 27 27 12 27 27 1 27 20 11 27 27 19 21 14 15 2 3 10 13 25 5 7 24 22 17 4 27 8 9 6 16 23 18 26

21 21 21 13 19 21 21 21 21 9 21 21 21 5 18 14 6 17 10 3 11 7 16 12 21 21 4 21 2 15 8 20 21 21 1

23 23 18 19 23 9 13 23 15 10 23 23 20 5 12 8 4 23 2 1 21 17 16 23 23 23 11 23 6 14 22 7 23 23 3

14 14 14 14 14 14 13 14 1 10 14 14 14 14 5 14 14 14 n 14 14 2 9 14 14 14 8 14 4 6 7 12 14 14 3

28 28 28 27 20 12 22 21 3 1 28 15 28 9 14 28 2 28 4 7 18 13 26 16 19 8 6 5 10 24 17 25 28 23 11

7 30 29 17 30 30 21 24 14 3 23 27 30 30 12 22 8 19 5 16 28 2 13 20 26 18 1 15 9 10 4 6 30 25 11

RATERS -8 9 10 11 12 13 14 15 16 17 30 11 28 18 29 21 3 19 24 5 27 20 32 14 16 24 1 25 6 10 33 9 12 34 31 23 2 22 8 7 4 17 26 15 13

30 20 34 12 11 21 4 23 26 5 32 28 31 25 19 16 9 24 34 3 7 13 18 2 29 22 1 15 10 17 14 27 33 8 6

24 5 26 14 15 20 18 21 10 12 27 27 27 16 22 11 17 27 19 6 27 9 3 13 27 25 2 27 1 9 4 23 27 27 7

19 19 19 19 19 14 16 19 12 13 1 19 19 17 19 19 2 19 19 5 9 7 19 18 19 19 6 15 3 11 8 19 10 19 4

* Organizations publishing two magazines.

33 35 32 5 6 28 17 9 1 8 31 3 7 24 14 11 23 30 2 25 16 13 27 26 34 29 19 4 18 22 21 15 12 10 20

32 16 18 8 17 28 1 9 4 21 31 13 33 25 7 3 24 23 20 26 34 15 19 22 27 35 11 29 6 5 12 14 30 10 2

32 33 10 27 35 13 3 34 20 12 31 4 24 15 22 28 21 14 2 6 16 8 30 23 17 11 9 29 18 25 5 1 19 26 7

32 34 31 6 20 25 21 9 1 15 30 12 35 29 24 28 11 10 27 19 17 16 7 13 33 5 22 14 23 3 18 4 2 8 26

30 29 31 14 10 11 6 34 33 1 32 25 17 21 18 13 2 24 5 12 27 9 3 28 26 23 16 35 7 4 15 8 20 22 9

29 12 26 21 20 10 3 29 1 4 29 5 15 29 11 23 18 29 28 7 8 13 25 2 29 17 19 22 9 6 16 27 29 24 14

31 APPENDIX B - PEARSON'S PRODUCT-MOMENT COEFFICIENT WORK SHEETS INDIVIDUAL RATINGS CORRELATED AGAINST GROUP RATINGS

ENTIRE GROUP X Y2 12.9$ “ 167770 280.56 16.75 180.90 13.14-5 235.62 15.35 16.10 259.21 256.00 16.00 211.70 14-55 170.30 13.05 179.56 13.14-0 180.90 13.14-5 308.00 17.55 324.00 18.00 16.65 277.22 14.80 219.04 241.80 15.55 111.30 10.55 217.56 14.75 214.62 14.65 178.22 13.35 152.52 12.35 16.14.0 268.96 16.80 282.24 15.80 249.64 244.92 1$. 65 16.10 259.21 178.22 13.35 222.01 14.90 13.00 169.00 237.16 15.14-0 i4.70 216.09 15.65 244.92 260.82 16.15 237.16 15.40 14.50 210.25 248.06 15.75 226.50 15.05 15.50 240.25 13.20 174.24 302.78 17.40 8639.14

X 10 16 llj 13 15 12 14 15 10 17 8 13

RATER 1 X2 100 256 196 169 225 144 225 196 225 100 2Qi

169 169 196 196 l4 289 17 169 13 100 10 225 15 16 256 2$6 16 18 324 196 14 11 121 12 144 81 9 11 121 15 225 196 l4 16 256 22 4§4 11 121 100 10 II4. 196 i fi iI 12 "" Tl 169 13 10 100 16, 256 52F 7I4J4.8

XX

129.50 268*00 188*30 199.55 2l|i.50 192.00 218.25 182*70 201*00 134-50 298*35 l44.oo 216.45 192.40 217.70 147.70 250.75 . 190.45 133.50 185*25 262.4 1171

435 381 416 256 339 328 164 360 210 ll*7 1*31 291 391 323 259

296

177 353 224 189 333 168 265 313 256 427 189

3#

67081 112225 72361 9h2k? 103684 102k00

84681 68121 7182k 72361 123201 129600 110889

87616

96721 Ij.88i|.l 87025 858k9 71289 75076 107584 112896 99856 97969

103684 71289 88804

67600

164

156 206

182 239 358 288 180 10B5H

145161 173056 114921 107584 26896 129600 44ioo 21609 185761 84681

152881 104329 67081 87616 31329 124609 50176 35721 110889 28224 70225 97969 65536 182329 35721

97969 104329 94864 84ioo 99225 90601 96100 69696 121104 3554o 64

112665 127635 111904 78592 109158 104960 47724 93960 56280 39543

151281 104760 130203 95608 80549 654i6 52215 103429 59808 51786 109224 56448 83740 .97969 82432 114009 563,22

111556

868ko

26896

50512 61740 42881 91409 48048 59740 57330 71939 110980 76032 62640 3197?11

44100

210

137 283

189225

18769 80089 24336 33124 57121

128164 8294S 32400 32l4(00

University of Southern California Uhvry