Expected Value of a Sample Estimate \f ~B7tf'- Statistical Reporting Service • U.S.Department of Agriculture. SRS No.19 ) r Z ' ) FOREWORD The Statistical Reporting Service (SRS) has been engaged for manyyears in the training of agricultural statisticians from around the world. Most of these participants comeunder the support of the . .Agency for International Development (AID) training programs; however, many also comeunder sponsorship of the Food and Agr1cul,tur~.~.rganization I .' " r' ,I ~ .. into the International Statistical Programs Center of the Bureau of the Census, with which SRSis c·ooperating. This treatise was developed by the SRSwith the .cooperation of . AIDand the Center, in an effort to provide improved materials for. "' " ,~ teaching and reference in the area of agricultural statistics, not only for foreign students but also for ~evelopment of staff working for these agencies. ..!~~ ., ..... ", ,,_I \ I, \,)t Washington, D. C. i HARRY C. TRELOGAN Administrator Statistical Reporting Service September 1974 ~ I~~: ..~~'. ,", ~,:;jt::~~.0~ . " " ~ • 1.~;"•• PREFACE The author has felt that applied courses in s~lin~ should give more attention to elementary theory of expected values of a random variable. The theory .pertaining to a 'random variable and to functions of random variables is the foundation for probability sampling. Interpretations of the accuracy of estimates from probability sample surveys are predicated on, amongother things, the theory of expected values. There are many students with career interests in surveys and the application of probability sampli~g who have very limited backgrounds in mathematics and statistics. Training in sampling should go beyond simply .. in mathematics and -probability should be included. It can (1) add 'mU:ch' learning about sample designs in a descriptive manner. •• I h •• ~ The foundations -, . • !..·I " to the breadth of understanding of bias, random sampling error, components' , ' of error, and other technical concepts; (2) enhance one's ability to make practical adaptations of sampling principals and correct us~ of formulas; and (3) make communication with mathematical statisticians easier and more meaningful. This monograph is intended as a reference for the convenience 6£ students in sampling. It attempts to express relevant, introductory mathematics and p,robability in the context of sample surveys. Although some proofs are presented, the emphasis is more on exposition of mathe- matical language and concepts than. on 'the mathematics per se and rigorous proofs. Manyproblems are given as exercises so a student may test his interpretation or understanding of the concepts. Most of the mathematics is elementary. If a formula looks involved, it is probably because it represents a long sequence of arithmetic operations. 11 1 Each chapter begins with very simple explanations and ends at a much more advanced level. Most students with only high school algebra should have no difficulty with the first parts of each chapter. Students with a few courses in college ma~hematics and statistics ~ght review ~he: first parts of each chapteT and spend considerable time studying the latter parts. In fact, some students might prefer to start with Chapter III and refer to Chapters I and II only as ~eeded. Discussion of expected values of random variables, as in Chapter III, was the original purpose of this monograph. Chapters I and II were added as background for Chapter III. Chapter IV focuses attention on the dis- tribution of an estimate which is the basis for comparing the accuracy of alternative sampling plans as well as a basis for statements about the accuracy of an estimate f.roma ,.amp~e•.. The content of Chapter IV is . inclUded in books on sampling, but it is important that students hear or read IIIOrethan one discussion of the distribution of an estimate, espe- dally with reference to estimates from actual sample surveys. The author's interest and experience in training has been primarily with persons who had begun careers in agricultural surveys. I appreciate the opportunity, which the Statistical Reporting Service has provided, to .. prepa~ this monograph• Earl E. Houseman Statistician iii , ", '.'~ ' CONTENTS Chapter I. Notation and Summation 1.1 lntroduction 1.2 Notation and the Symbol for Summation 1.3 Frequency Distributions 1. 4 Algebra 1.5 Double Indexes and Summation 1.5.1 Cross Classification 1.5.2 Hierarchal or Nested Classification 1.6 The Square of a Sum 1. 7 Sums of Squares 1.7.1 Nested Classification 1.7.2 Cross Classification , • , .I I , I ,,'ll" I Chapter II. RandomVariables and Probability 2.1 RandomVariables ", " ,; 2.2 Addition of Probabilities I 2.3 Multiplication of Probabilities 2.4 Sampling With Replacement 2.5 Sampling Without Replacement 2.6 Simple RandomSamples 2.7 Some Examples of Restricted RandomSampling 2.8 Two-Stage Sampling Chapter III. Expected Values of RandomVariables 3.1 Introduction 3.2 Expected Value of the Sumof TwoRandomVariables 3.3 Expected Value of an Estimate 3.4 Variance of a RandomVariable 3.4.1 Variance of the Sumof Two Independent Random Variables 3.4.2 Variance of the Sumof TwoDependent Random Variables iv Page 1 1 1 9 10 . 14 15 22 26 29 29 32 33 33 35 41 42 45 4.7 50 57 63 63 67 72 77 77 79 ~:,>lI;:.r.!l>:~ 'r "':~f;/~~:~': ':~ , , j \ \ r I ~, "I I CONTENTS(Continued) 3.5 Variance of an Estimate ·3.5.1 Equal Probability of Selection 3.5.2 Unequal Probability of Selection 3.6 Variance of a Linear Combination 3.7 Estimation of Variance 3.7.1 Simple RandomSampling 3.7.2 Unequal Probability of Selection 3.8 Ratio of TwoRandomVariables 3.9 Conditional Expectation 3.10 Conditional Variance Chapter IV. The Distribution of an Estimate 4.1 Properties of Simple RandomSamples 4.2 Shape of the Sampling Distribution 4.3 Sample Design 4.4 Response Error 4.5 Bias and Standard Error -. Page 80 80 92 84 89 89 92 94 97 103 113 113 117 119 125 135 III • 1 CHAPTER I. NOTATION AND SUMMATION 1.1 INTRODUCTION To wprk with large amounts of data, .an appropriate system of notation is needed. The notation must identify data by individual elements, and provide meaningful mathematical expressions for a wide variety of summaries from individual data. This chapter describes notation and introduces summation algebra, primarily with reference to data from census and sample surveys. The purpose is to acquai~t students with notation and summation t' rather than to present statistical concepts. Initially some of the expres- siona might seem complex or abstract, but nothiD:gmore than sequences of" II ..••• operations involving addition, subtraction, multiplication, and division. I •• ~- , is involved. Exercises are included so a student may test his interpreta- tion of different mathematical expressions. Algebraic manipulations are also discussed and some algebraic exercises are included. To a consider-'I .. able degree, this chapter could be regarded as a manual of exercises for students who are interested in sampling but are not fully familiar with the sUllll1ationsymbol, E. Familiarity with -the mathematical language will make the study of sampling mucheasier. 1. 2 NOTATION AND THE SYMBOL FOR SUMMATION "Element" will be used in this monograph as a general expression for a unit that a measurement pertains to. An element might be a farm, a per- son, a school, a stalk·of com. or an animal. Such units are sometimes called units of observation or reporting units. Generally, there are several characteristics or items of information about an element that one might be interested in. 2 Il1easurement" or "value" will be used as general tel'lll8 for the numerical value of a specified characteristic for an element. This includes assigned values. For example, the element might be a farm and the characteristic could be whether wheat is being grown or is not being grown on a farm. A value of "lfl could be assigned to a farm growing wheat and a value of flO" to a farm not growing wheat. Thus, the flmeasurement" or "value" for a farm growing wheat would be fllfl and for a farm not grow- ing wheat the value would be "0." Typically, a set of measurements of N elements will be expressed as follows: Xl' X2, ••• t~ where X refers to the characteristic that is measured and the index (subscript) 'to the various elements of the popula':" tion (or set)., For example, if there are N persons and the characteristic X is a person's" height, then Xl is the height of the first p~rsont etc. To refer to anyone of elements,· not a specific element, a subscript' '''i'' is used. Thus, Xi (read X sub i) means the' value of X for anyone of the N elements. A commonexpression would be "Xi is the value of X for the' i th element." The Greek letter r (capital sigma) is generally used to indicate a sum. Whenfound in an equation, it means "the sum of." For example, N L Xi represents the sum of all values of X from Xl to ~; that is, i-I N L Xi - Xl + X2 +..•+ XN. The lower and upper limits of the index of i-I sunmation are shown below and above the summation sign. For example, to 20 specify the sum of X for elements 11 thru 20 one would write LXi. i-II .~' 3 You might also see notation such as "E,Xiwhere i • 1, 2, ••• , Nil which indicates there are N elements (or values) in the set indexed by serial numbers l'thru N, or for part of a set you might see"EXi where i .,.11, 12, •••,20." Generally the index of summation starts with 1; so you will N often see a summation written as EXi• That is, only the upper limit of i the summation is shown and it is understood that the summation begins with I~ i-1. Alternatively, when the set of values being summed is clearly under- :rl :!~~.~, stood, the lower and upper limits might not be shown. Thus. it is under- : I <' ,. I'~ ; :1 '•• ' ! • stood that EXi or EXi is the sum,of X over all v,a1ues of the set under i consideration. Sometimes a writer will even drop the subscript and use EX for the sum of all values of X. Usually the simplest notation that is ~ 1 ~ adequate for the~purpose is adopted,. In this monograph, there will be ~ .. ' ~' ' some deliberate variation in notation to familiarize students with various representations .of data. An average is usually indicated by a "bar" over the symbol. For example, X (read "X bar," or sometimes "bar X") means the average value of mation makes it clear that the sum is being divided by the number of elements EX i However, T and X is the average of all elements. e' ".:; ~J,';;, f II oJ x. Thus, X •• In this case,showing the upper limit, N, of the sum- would also be inter- preted as the average of all values of X unless there is an indication to the contrary. Do not try to study mathematics without pencil and paper. Whenever the shorthand is not clear, try writing it out in long form. This will often reduce any ambiguity and save time. 4 Here are some examples of mathematical shorthand: ,I'" (1) Sum,of the reciprocals of X (2) Sum of the differences. between- Xi and ~ constant,C (3) Sum of the deviations of Xi from the average ~f X , NIl 1 1 1: X - X + X + ••• + 1L i-I i 1 2 -""N N • E (Xi-C)-(XI-C)+(X2-C)+•••+(~-C) i-I N E(Xi-X)-(Xl-i)+(X2-i)+•••+(~-X) i t., i I :.1 (4) (5) (6) Sum of the absolute values 6f • -' .'; I~' the ~ifferences between Xi' and X. (Absolute value, .,' I~I indicated by the vertical lines. means the positive value of the difference) Sum of the squares of Xi , I "[1'1'. I" , II' Sum of squares of the' " deviations of X from i 2 222 2 EXi - Xl + X2 + X3 +••• XN - 2 - 2 - 2 teXt-X) - (~-X) + •••+ (~-X) (7) Average of the squares of the deviations of X from i - - 2 - 2 (Xl-X) + ••• +(~-X) N (8) Sum of products of X and Y (9) Sum of quotients of X divided by Y (10) Sum of X divided by the sum of Y tXi Xl+X2+···+ ~ -- 1:Yi Yl+Y2+···+ YN (11) Sum of the first N digits (12) (13) N E i - 1+2+3+•••+ N i-I ,. I •• 5' Exercise 1.1. You are given a set of four elements having the following values of X: Xl - 2, X2 - 0, X3 - 5, X4 - 7. To test your understand'ing of the summation notation, compute the values of the follow- ing algebraic expressions: Expression Answer 4 (1) . t (Xi+4) 30 i-l (2) t2(Xi-1) I,:' 20 ., I' (3) 2t(X -1) ," 20 .1 f ( ." "",' , '1-, "'I"'X • 'III' '.' ~, " ~~,..2 I ';' II I. I< .' .; •••• ,~~ 91 I~!bl~ d .~ I l" • .11 ~ ;, ' (4) ~I" ! I. 27' tlK -1 1 ',' i _ tX 1 .- "(5) X-- , 3.5 N "- . (6) tx2 78 1 (7) t(-X1)2 78 (8) [tx1]2 19'6- (9) 2 64 t(Xi - X1) (10) t(X~) - tx 64 1 (11) ti(X1) 45 (12) t(-1)i(Xi) 0 4 (13) E (X~ - 3) 66 i-l 4 4 (14) E x2 - t (3) 66 i-l 1 i-l 4 Note: t (3) means find the sum of four 3's i-l " , 6 Definition 1.1. The variance of X where X - Xl' X2' ••• , ~, is defined in one of two ways: N E(Xi-X)2 " I~ ,;.., 2 i-I a = N , .. or N ," . S2 _ I (Xi-X) 2 i-I N-l The reason for the ~iO definitions will be explained in Chapter III. The variance formulas provide measures of how much the values of X vary (deviate) from the average. The square root of the variance of X is called the standard deviation of X. The central role that the above definitions of variance and standard deviation play in sampling theory will. become apparent as you study sampling. The variance of an estimate from a sample is one of the measures needed to judge the accuracy of the estimate and to evaluate alternative sampline designs. Much of the algebra and notation in this chapter is related to computation of variance. For ,'''"'' ~..:~.~ ..~ ,I ,I ').,.1 I/f,r~i·",~ ,?, .' ~~#1~:~ . '. -' :...: " ~" "i! I., 7 complex sampling plans, variance formulas are complex. This chapter should help make the mathematics used in samplin~ more readable and more meaningful when it is encountered. Definition 1.2. "Population" is a statistical term that refers to a set of elements from which a sample is selected ("Universe" is often used instead of "Population"). Some examples of populations are farms, retail stores, students, households, manufacturers, and hospitals. A complete definition of a population is a detailed specification of the elements that compose it. Data to be collected also need to be defined. Problems of defining popu- lations to be surveyed should receive much attention in courses on.sampling. From a defined population a sample of elements is selected, information , for each element in the sample is collected, and inferences from the sam- pIe are made about the population. Nearly all populations for sample surveys are finite so the mathematics and discussion in this monograph are limited to finite populations. In the theory of sampling, it is important to distinguish between data for elements in a sample and data for elements in the entire popula- tion. Many writers use uppercase letters when referring to the population and lowercase letters when referring to a sample. Thus Xl' ••• ' ~ would represent the values of some characteristic X for the N elements of the population; and xl' ••• ' xn would represent the values of X in a sample of n elements. The subscripts in xl' ••• ' xn simply' index the different elements in a sample and do not correspond to the subscripts in Xl' ••• ' ~ which index the elements of the population. In other words, xi could be anyone of the Xi'8. Thus, N ~ Xi i --. X N n r Xi i--. x n represents the population mean, and represents a sample mean 8 In this chapter we will be using only uppercase letters, except for constants and subscripts, because the major emphasis is on symbolic repre- sentation of data for a set of elements and on algebra. For this purpose, it is sufficient to start with data for a set of elements and not be concerned with whether the data are for a sample of elements or for all elements in a population. The letters X, Y, and Z are often used to represent different charac- I: teristics (variables) whereas the first letters of the alphabet are cOUIDOuly used as constants. There are no fixed rules regarding notation. ' For example, four different variables or characteristics might be called Xl' I. th ' X2, X3, and X4• In that case Xli might be used "to represent the i value of the variable Xl. Typically, writers adopt notation that is convenient for their problems. It is not practical to completely standardize notation. Exercise 1.2. In the list of expressions in Exercise 1.1 find the much is the variance of X changed? , l "" 2 variance of X, that is, find S • Suppose that X4 is 15 instead of 7. 2 1 Answer: From 9'3 to 44j . How Exercise 1.3.' You are given four elements having the following values of X and Y Y • 2 1 Y • 3 2 Y • 1 3 Y • 14 4 I I i .' I. f 9 Find the value of the following expressions: Expression Answer Expression Answer (1) tXiYi 107 (7) tXi-tYi -6 (2) (1:Xi)(1:Yi) 280 (8) 2 74 1:(Xi-Yi) (3) 1:(Xi-X)(Yi-Y) 37 (9) I:(X2_y2) -132 i i (4) I:XiYi-NXY 37 (10) I:X2_ty2 -132 i i (5) 1 Xi 1.625 (11) 2 36 - I:- [I:(Xi-Yi) ] N Yi "(6),1 'I:(X' Y l)1I"1 4 " i:1I ,:1 '-6I!' " "(12) [I:X12-[tY ]2 -204 ' 1111 "_ , 'I' , i i i i 1.3 FREQUENCY DISTRIB11l'IONS Several elements in a set of N might have the same value for some characteristic X. For example, many people have the same age. Let Xj L :' ~\ , •• be a particular age and let Nj be the number of people in a population K .(set) of N people who have the age Xj• Then I:Nj - N where K is the j-1 number of different ages found in the population. Also I:NjXj is the sum; I:NjXj of the ages of the N people in the population and ---- represents the I:N j average age of the N people. A listing of Xj and Nj is called the frequency distribution of ~since N j is the number of times (frequency) that the age X j is found in the population. On the other hand, one could let Xi represent the age of the ith individual in a population of N people. Notice that j was an index of age. We are now using i as an index of individuals, and the average age would I:X i be written as ~ The ·;·,,~~·~~t·: " '.•. 10 choice between these two symbolic representations of the age of people in the population .is a matter of convenience and purpose. Exercise 1.4. Suppose there are 20 elements in a set (that is, N - 20) and that the values of X for the 20.elements are: 4, 8, 3, 7, 8, 8, ~, 3, 7, 2, 8, 4, 8, 8, 3, 7,8,10, 3, 8. (1) List the values of X j and N j , where j is an index of the values 2, 3, 4, 7, 8, and 10. This is the frequency distribution of X. (2) What is K equal to? Interpret and verify the.following by making the calculations indicated: " . '. (3) (4) N EX- i-l i i) : i I,ll ::1 ',I . :.1 (5) 1. 4 ALGEBRA :' In arithmetic and elementary algebra, the order of the numbers when addition or multiplication is performed does not affect the results. The familiar arithmetic laws when extended to algebra involving the summation symbol lead to the following important rules or theorems: Rule 1.2 E8.'{i- aEXi where.!. is a constant Rule 1.3 1:(Xi+b) - 1:Xi+Nb where b is constant -tf" .••:' ,1 ,~ 'l'.s:.,,'. , r ,_ ;1,.'. ~C.~~~~·1!f i ), I 11 If it is not obvious that the above equations are correct, write both sides of each equation as series and note that the difference between the two sides' is a matter of the order in which the summation (arithmetic) is performed. Note that the use of parentheses in Rule 1. 3 means that b is contained in the series N times. That is, N ~ (Xi+b) - (Xl+b)+(X2+b)+"'+(XN+b) i-l " On the basis of Rule 1.1, we can write '!~ I ) The expression • I N E.b means"sum.the 'vaitie ~f b,which occurs N times. 11 Therefore, i-I N ~ b - Nb. i-l N Notice that if the expression had been E Xi+b,then b is an amoun~ to add i N to the sum, ~ Xi • i. N N In many equations X will appear; for example, E XXi or E (Xi-X). i i Since X is constant with regard to the summation, EXxi tX ~(Xi-X) - 1: X -tx - tx - _ i i NX. By definition,X - -;- • i i i i i i NX• tXi and t(Xi-X) - O. i i - itxi• Thus, Therefore, N To work with an expression like ~(Xi+b)2 we must square the quantity i in parentheses before summing. Thus, 12 E(Xi + b)2 - E(X~+ 2bXi + b 2) i • EX~7 E2bXi + Eb2 Rule 1 • EX~+ 2bEXi + Nb 2 Rules 2 and 3 Verify this result by using series notation. 2 2 Start with (Xl+b) +••• +(XN+b) • It is very important that the ordinary rules of algebra pertaining to the use of parentheses be observed. Students frequently make errors because inadequate attention is given to the placement of parentheses or to the interpretation of parentheses. Until you become familiar with the above rules t practice translating shorthand to series and series to sbort- hand. Study the followin~ examples' carefully: The left-hand side is the sum of the squares of Xi. The right~hand side is the square of the sum of Xi· On the right the parentheses are necessary. The left side could 2 have been written EXi • Rule 1.2 applies. 2 2 (2) {Xi} IXi N 2' N (3) 2 EX2+ Ey2 ,'" I , '.-:: ),~.:, E(Xi+Yi) ~ 1 i (4) E(X2 + y2) • EX2+ 1:y2 i i i i (5) EXiYi ~ (1:Xi)(EYi) A quantity in parentheses must be squared before taking a sum. Rule 1.1 applies The left side is the sum of products. The right side is the product of (6) (7) sums. N N Ea(Xi-b) ~ &LXi- ab i i (8) N N ta(Xi-b) - a1:X - Nab i i i 13 N N . (9)' a[tXi-b] - atX -ab i i i (10) Exercise 1.5. Prove the following: In all cases, assume i - 1,2,•••, N. (1) 't(X -X) - 0 i " (2) , , I~, J 'I " " O:X i )2 (3)NX2 _ -- I" ~ I, I" N I, ", •• '-'. ". " oj' ,~d' ".i ,N (4) ,t (aXi+bYi+C) -.aI:Xi+btYi+NC 3, j 'i-l ,,' I, , .. (5) (6) (7) ,Note: Equations (5) and (6) should be (or become) very familiar equations. t(X _X)2 • tX2 _ NX2 i i x t(...!+ Y )2 _.L t(X +aY )2 a i 2 i i a The total number of elements in the NO and N- Q. Prove that (8) Let Yi - a+bXi, show that Y - a+bX 2 - 2 2 and ty i - Na (a+2bX) + b tXi (9) Assume that Xi - 1 for Nl elements of a set and that Xi - 0 for NO of the elements. Nl set is N • Nl+NO' Let N - P t(Xt-X)2 N • PQ • (10) Hint: 14 , ,' •. ~~' '-I, , - - 2 as [(Xi-X)+(X-d)] • Recall from elementary algebra that 222 - - (a+b) - a +2ab+b and think of (Xi-X) as a and of (X-d) as b~ For what value of d is E(Xi-d)2 a minimum? 1.5 DOUBLEINDEXES AND SUMMATION When there is more tnan one characteristic for a set of elements, the different characteristics might be distinguished by using a different letter for each or by an index. For example, Xi and Yi might repres1ent the number of acres of wheat planted and the number of acres of wheat th harvested on the i farm. Or, Xij might be used where i is the index for the characteristics and j is the index for elements; that is, Xij th would be the value of characteristic Xi for the j element. However, , n : ~~'I :1 when data on each of several characteristics for a set of elements are to be processed in the same way, it mighe Jlotbe necessary to use notation that distinguishes the characteristics. Thus, one might say E(Xi-X)2 calculate N-1 for all characteristics. More than one index is needed when the elements are classified accord- ing to more than one criterion. For example, Xij might represent the value th th of characteristic X for the j farm in the i county; or Xijk might be th th th the value of X for the k household in the j block in the i city. As another example, suppose the processing of data-for farms involves classification of farms by size and type. We might let Xijk represent the value of characteristic X for the kth farm in the subset of farms classified as type j and size i. If Nij is the number of farms classified 15 as type j and size i, Nij E Xijk k then ---N----- - Xij• is the average value of X for ij • : :'; ,:~ •..:. .t""I!.;' .,':l"'7~j~1• .,,' the subset of farms classified as type j and size i. There are two general kinds of classification--cross classification and hierarchal or nested classification. Both kinds are often involved in the same problem. However, we will discuss each separately. An example of nested classification is farms within counties, counties within States, and States within regions. Cross classification means that the , "1 '''''11 • data can be arranged in two 'ormore dimensions as illustrated in the next section. 1.5.1 CROSS CLASSIFICATION ,,' ,: •. f I ~t ~I • I '. " As a specific illustration of cross classification and summation with I' , two indexes, suppose we are working wit~ the acreages of K crops on a set f N f Le X h of!the'ith crop on the jth farm o arms. t ij represent t e a~reage where i-I, 2,•••, K and j - 1,2, •••, N. In this case, the data could I I .' be arranged in a K by N matrix as follows: Column (j) Row Row (i) total ,,~;' ~~ ,~ 1 j N 1 XII Xlj XlN E Xlj j i XiI Xij XiN E Xij · j . · . · . K ~I ... ~j ~ E ~j j .. Column E Xu E Xij I EE Xij : E XiN: total i i i : ij 16 N The expression r Xij (or r Xij) means the sum of the values of Xij for a j j fixed value of i. Thus, with reference to the matrix, I Xij is the total j X could be written as farms and crop acreages, I th j whatever the i crop is. th of the values of X in the i row; or, with reference to the example about Xij would be the total acreage on all farms of . K Similarly, I Xij (or I Xij) is the column total i i ~ ~ for the j c.olumn, which in the example is the total for the j farm of the acreages of the K crops under consideration. The Bum of all values of KN I! Xij or !I Xij• , ij ij Double SUDIllationmeans the sum of sums. Breaking a double sum into parts can be an important aid to understanding it. Here are two examples: KN N 10{ l{ (1) II Xij - I Xlj + I X2j +.•• + r ~j ij j j j (1.1) With reference to the above matri~Equation (1.1) e~resses the grand total as the sum of row totals. (2) (1. 2) In Equations (1.1) and (1.2) the-double sum is written as the sum of K partial sums, that is, one partial sum for each value of i. Exercise 1.6. (a) Write an equation similar to Equation (1.1) that expresses the grand total as the sum of column totals. (b) Involved in Equation (1.2) are KN terms, Xij(Yij+a). Write these terms in the form of a matrix. , " 17 The rules given in Section 1.4 also apply to double summation. Thus , Study Equation (1.3) with reference to the matrix called for in Exercise 1.6(b). To fully und~rstand Equation (1.3), you might need to write out ... intermediate steps for getting from the left-hand side to the right-hand side of the equation. To simplify notation, a system of dot notation is commonly used, for example: (1. 3) The dot in Xi' indicates that an index in addition to i is involved and Xi' is interpreted as the sum of 'the values of X for a fixed value of i. Similarly, X.j is the sum of X for any fixed value of j, and X •• represents a sum over both indexes. As stated above, averages are indicated by use of a bar. Thus Xi. is the average of Xij for a fixed value of i, namely - X and X i· would represent the average of all values of Xij, 1:1: Xij namely _i_j_ NK Here is an example of how the dot notation can Simplify an algebraic expression. Suppose one wishes to refer to the sum of the squares of the row totals in the above matrix. 2 This would be written as r(xi.) • 1 The sum Without the dot notation the - 2 of squares of the row means would be E(Xi.) • i K N 2 K corresponding expressions would be !(!Xij) and! i j i important that the parentheses be used correctly. 18 2 It is very K N 2 For example, !(!Xij) is i j KN2 not the same as !!Xij• Incidentally, what is the difference between the ij last two ~xpressions?: Using the dot notation, the variance of the row means could be written as follows: (1.4) where V stands for variance and V(X1) is an expression for tbe variance of Xi•• Without the dot notation, or something equivalent to it, a formula for the variance of the row means would look muchmore complicated. 19 Find the value of the following algebraic expressions: Expression Answer Expression Answer N (1) EXlj j 42 54 (3) 6 21 18 78 (11) (10) (12) N . (;1 (13)' ~E(X -X )2, lj 1· .. j . 78 45 12 12 13.5 144 KN EEXij ij X.. (7) (6) (5) (4) (2) •.. .~~ , . ~\~:,~' '~~ ~'.'~;~:~~~~~~~ " ":' ;"AJ.~~ I K (8) NE(X i -X )2 i ..• 18 (14) 60 Illustration 1.1. To introduce another aspect of notation, refer to the matrix on Page 15 and suppose that the values of X in row one are to be multiplied by aI' the values of X in row two by a2, etc. The matrix would then be ~~l ••• ~~j ••• ~~ The general term can be written as aixij because the index of a and the 20 index i in Xij are the same. The total of all KN values of aiXij is KN I:taiXij• Since ai is constant with respect to summation involving j, ij N we can place a1 ahead of the summation symbol t • j Exercise 1.9. Refer to the matrix of values of Xij in Exercise 1.8. ' ••••••• \. kl I .-1. • ..,..,,'2 ,·:,:~'··;!:t:,:,,:;~~~~ ( Calculate: (1) ttaiXij ij (2) tt aixij ij N (3) 2 ttaiXij ij "I' ,/ ,1' Answer :-296 Show algebraically that: (4) ttaiXij - tX3j-t~j ij j j (5) tE aiXij - X -x ij N 3· 1· (6) 2 2 2 rtaiXij - tX3j-txlj j j Exercise 1.10. Study the following equation and if necessary write the summations as series to be satisfied that the equation is correct: Illustration 1.2. ,Suppose Yij • xij+ai+bj+c where i • 1, 2, ••• ,K and j - 1, 2, •••,N 21 The values of Yij can be arranged in matrix format as follows: Y11 - ~1 + a1+b1+c ••••••••• YlN - XlN + a1+bN+c 'Yij - Xij + ai+bj+c . . . . . Notice that ai is a quantity that varies from row to row but is constant within a row and that bj varies from column to column but is constant within a column. Applying the rules regarding the summation symbols we have -tXij + Nai + tb +Nc j j - tEXij + NEai + KIbj + KNc 'ij i j ".", r Illustration 1. 3. Wehave noted that t (XiYi) does !!2! equal (txi)(tYi). (See (1) and (2) in Exercise 1.3, and (5) on Page 12). But, tEXiYj - (EXi)(EYj) where i - 1, 2, ••• ,K and j - 1, 2, ••• ,N. This becomes ij i j clear if we write the terms of tEXiYj in matrix format as follows: ij RowTotals One could 22 The sum of the terms in each row is shown at the right. The sum of these row totals is Xl :EYj+... + ~:EYj - (X1+... + ~)rY j - rXiEY j. get the same final result by adding the columns first. Very often inter- mediate summations are of primary interest. Exercise 1.11. Verify that rrxiY j - (:EXi)(:EYj) using the values of ij X and Y in Exercise 1.3. In Exercise 1.3 the subscript of X and the sub- . script of Y were the same index. In the expression :ErXiYj that is no longer ij the case. Exercise 1.12. Prove the following: (1) (2) 1.5.2 HIERARCHAL OR NESTED CLASSIFICATION A double index does not necessarily imply that a meaningful cross classification of the data can be made. For example, Xij might represent th th the value of X for the j farm in the i county. In this case, j simply identifies a farm within a county. There is no correspondence, for example, between farm number 5 in one county and farm number 5 in another. In fact the total number of farms varies from county to county. Suppose there are th K counties and Ni farms in the i county. Ni county could be expressed as Xi. - r Xij j The total of X for the ith K In the present case rXij is i meaningless. KNi The total of all values of X is r1:Xij ij 23 When the classification is nested, the order of the subscripts (indexes) and the order of the sumnation symbols from left to right should be from the highest to lowest order of classification. Thus in the above example the index for farms was on the right and the summation symbol KN i involving this index is also on the right. In the expression 1:1: Xij, ij summation with respect to i cannot take place before summation with regard the summations can be performed in either order. to j. On the other hand, when the classification is cross classification I' It, ; '" . l~I' 1" .,"'" rt: ' ~ :.! !. I t \ :i"l .~n'· "£l~H~l~f, 'Xll, X12, th 'i"Inthe examp~e of '~~,'co~ties and Ni farms in the i county ,and in data as ~eingarranged in rows (or , , ' similar exampfes, you may ~hink C?f the 1I,.".l •••S..ll ~ II " •• ~ '~~.J' ~ I> ~ t.~~ I, •• ~, I' ' .•• ,. ,columns): , ' ~r ... !~J •.z., , X 2 ,,' N 2 ., ,!,' ~l' ~2' •••• Here are two double sums taken apart for inspection: , I ~ < (1) = ~l(X -x )2 j lj •• ( V ) Nl 2 - 2 2 1: (Xlj-X ••) - (Xll-X ••) +...+ (XlN -X ••) j 1 (1. 5) Equation (1.5) is the sum of squares of the deviations, (Xij-X ••), of all values of Xij from the overall mean. K There are 1:Ni values of Xij' and i a single index would be sufficient. x.. 24 If there was no interest in identifying the data by counties, N _ 2 Equation (1.5) would then be r(Xi-X) • i (2) (1. 6) Nl - 2 - 2 - 2 r (Xlj-X!.) - (XII-Xl.) +•••+ (XlN -Xl.) j 1 Nl - 2 With reference, to Equation :(1.6) do. you .re<:ognize t (~j-Xl'>? It involves . j only the subset of elements for' 'which i '. 1~ namely XII ~ X12, ••• that Xl. is the average value of X in XlN• Note 1 Nl - 2 this subset. Hence, r (Xlj-~'> is '" .j the sum of the squares of the deviations of the' X's in this subset from the subset mean. The double sum is the SUlll of K terms and each of the K terms is a sum of squares for a subset of X' s, the index for the subsets be.ing i. th Exercise 1.13. Let Xij represent the value of X for the j farm in th the i county. Also, ,let K be the number of counties and Ni be the number th of farms in the i county. Suppose the values of X are as follows: Xl3 - 5 X33 • 1 Find the value of the following expressions: Expression Answer 9 25 Expression (Continued) Answer (2) KNi 27 EE Xij ij (3) X - - and X 27 3 - - (4) N1 aX 9 E X1j 1- j (5) X2 _ and X3 _ 10 8 (6) X1_, X2_, and X3_ 3 5 2. (7) ENiXi_ 3 !Ni "", !,~1:] :1: I,. ,i. ') lili' i! ,.,,' ·1 I I :: ~,I : ' : I" ' .1 :;"'I~I I' ',I,' j 'l~l' • 1 1 24 I' • :. ~~ 11 ~It~1~:" I 'I!' •••••• I" 8, 2, and 14 for i - 1, 2, and 3 respectively 8 " .,,1 245 (8) K N~ 2 .K 2 1:(1:ij) or 1:Xi_ i j i (9) - 2 1:1: (Xij-X_.) ij , (10) N1 - 2 1:(X1j-X1) j (11) Ni - 2 E (Xij-Xi) j I • ~~:, ':" •• .., KNi - 2 , (12) E1:(Xij-Xi.) .. ij K (13) - - 2 12 1:Ni(Xi·-X••) i KUixijJ miXijr (14) L Ni ENj 12 i K (15) 1:NX2 _NX2 12 i i •.. i 26 Expressions (14) and (15) in Exercise 1.13 are symbolic representations of the same thing. By definition K ,and tNi - N i Substitution in (14) gives , . ~ , ,~I~~. '~:"\' t~~~ ~. ~ ';·f:~~r~1,. ~~~~??~~'~$: X and ~ • 3( N •• (1. 7) K Hence, by substitu:tion, Equation (1.7) becomes tN 3(2 - Ni2. i i· .• i " , • ' I~ ~, , ' Exercise 1.14. 'Prove the following: L': KNi K (1) • tx2 tt Xi.Xij i· ij i. KN (2) . i- - • 0 tt Xi. (Xij-Xi.> ij K K (3) , - - 2 -2-2 ·m (X -X ) - tN X -NX 1 i i··· i i i· •• Note that this equates (13) and (15) in Exp.rcise 1.13. The proof is similar to the proof called for in part (5) of Exercise 1.5. 1.6 THE SQUARE OF A SUM In statistics, it is often necessary to work algebraically with the square of a sum. For example, 22222 (I:Xi) - (Xl+X2+·· .+~) - X1+XlX2+··.+X2+X2Xl+··+XN+Yl+··· , .' ~r:l~.':I~:- ;;Il~~:~ ~~~;,~~' t>~:,'J'~" .:~tf~~~~: . :i;t ~ I i 27 The terms in the square of the sum can be written in matrix form as follows: The general term" i,~ this matrix is XiXj where Xi, and Xj come from the same set of XiS, namely, Xl' ••• '~. Hence, i and j are indexes of the same set. Note that the terms along the main diagonal are the squares of the value of X and 2 That is, on the main diagonal i = j could be w,rit ten ..,as,: EXi • and XiXj 2 The remaining all products of one value - XiXi - Xi terms are of X with some other value of X. For these terms the indexes are never equal. Therefore, the sum of all terms not on the main diagonal can be expressed as EXiX j where i ;. j is used to express the fact that the summa- i;j tion includes all terms where i is not equal to j, that is, all terms other than those on the main diagonal. Hence, we have shown that (EXi)2 - 2 EXi + EXiXj i;j Notice the symmetry of terms above and below the main diagonal: XlX2 - X2Xl,XlX3 - X3Xl ,etc. Whensymmetry like 'this occurs, instead of EXiXj you might see an equivalent expression 2E XiX j • The sum of all i~ i~ terms above the main diagonal is 1: XiX j • Owin~ to the symmetry, the sum i<j 28 of the terms below the main diagonal is the same. Therefore, ~ XiXj • i+j 2 1: Xi Xj • i<j Exercise 1.15. matrix format. Let Xl • 2, X2 • 0, X3 • 5, of ~X~ ' 2 1: XiXj t and [l:Xi]2. Show that i<j and X4 • 7. Com~ute the values 2 2 [l:Xi] • l:Xi+ 2 l:XiXj • 'f<j An important re~1U1t, which we w.ill use in Chapter 3, follows from the fact that , . : ~ (1. 8) , / • J II ~' ,1 : [l:(Yi-i»)2 • !(yi-y)2 + l: (Yi-Y)(Yj-Y) i+j , _ ' t2 .1 " . I. ~ ~.~ " We know that [l:(Yi-Y») • 0 because l:(Yi:"Y)- 0.' 'The'refore, '. I' I ':'. ,;, t"' l:(Yi-y~2 + l: (Yi-Y)(Yj-Y~ • 0 ,>, i+j 'I' It follows that ~ (y -Y)(Y -Y) • -1:(Y _i)2 i+ji j i Exercise 1.16. Consider (1. 9) Do you agree that l:y2 • N(N-l)y2? With reference to the matrix layout, i,cj -2 2 Y appears N times but the specification is i + j so we do not want to -2 count the N times that Y is on the main diagonal. Try finding the values of ~ Xi and ~ Xj and then show that i+j i+j Hint: Refer to a matrix l~yout. 29 In t Yi how many times does Yl appear? i"j Does Y2 appear the same number of times? • 1 I 1.7 SUMS OF sQuAREs For various reasons statisticians are interested in components of - . t, 1 ?'JI'I' t ••!, ...,_ ~. ,. . :.'. ~~ : ~ . To be somewhat speCific; reference.is ,made to the- ex-ample of K counties .. ; • .r- and Ni farms in the' i,th_ co~ty.;'- The':'sum~f_ the,. squareS' o~ the deviations variation, that i~?imeasuring the amoun~of variation attributable to each of more than one source. This involves computing sums of squares that correspond to the different sources of variation that are,of interest'. Wewill discuss a simple 'example of. ,it~.te.'d .classification and a simPle' " ,'.. ,~ '1 \ " I example of cross' classification. 1.7.1 NESTEDCLASSIFICATION ') : i.., . •• ~. I -. ~I .:' : I ~ , of Xij and X•• can be divided into two parts as shown by the following formula: (1.10) .' The quantity on the left-hand sid~ of Equation (1.l0)'is called the total sum of squares. In Exercise 1.13, Part (9), tne total sum of squares was 36. The first quantity on the right-hand side of the equation involves the, squares of (Xi. -X•• ), which are deviations of the class means from the over- , I all mean. It is called the between class sum of squares or with reference to the example the between county sum of squares. In Exercise 1.13, Part (13), the between county sum of squares was computed. The answer was 12. 30 The last term is called the within sum of squares because it involves deviations within the classes from the class means. It was .presented previously. See E,quation.(1.6) and the discussion pertaining'to it. In Exercise 1.13, the within class sum of squares was 24, which was calculated in Part (12). Thus, from Exercise 1.13, we have the total sum of squares, 36, which equals the 'between, 12, plus the within, 24. Equation (1.10). , This verifies t! r The proof of Equation 1.10 is easy if one gets started correctly. Write Xij-X •• ~ (Xij-Xi) +(Xi.-X ••). This simple technique of adding and subtracting Xi.' divides the deviation (Xij-X ••) into two parts. The. proof proceeds as follows: • L' ,i I' - Exercise 1.17. - LL[(Xij-Xi.) + (Xi.-X..)l2 ' .. ij .' '< - 2 - - - - - 2 - LL[(Xij-Xi.> :t.2(Xij-Xi.)(Xi.-X ••)?+ (Xi.-X ••) ). ij i and that Completion of Exercise 1.17 completes the proof. Equation (1.10) is written in a fo~·which displays its meaning rather than in a form that is most useful for computational purposes. For computa- tion purpo~es, the following relationships are commonly used: KN Total - LLi(X -X )2_ ij ij .. 31 K where N • I:Ni ' i leNi 2 Notice that the major part of arithmetic reduces to calculating I:I:Xij ij , ' I, ,':' There are variations of this that one might K X~. K -2 example, one could use I:N iristead of I:N'iXi. i i i Exercise 1.18. Show that use. For , ' '. ',,~, L ,. A special case that is useful occurs when Ni • 2. The within sum of squares becomes Xl1+Xi2 Since i ---- it is easy to show that i· 2 and Therefore the within sum of squares is which is a convenient form for computation. 32 1.7.2 C~SS 'CLASSIFICATION Reference is made to the matrix on Page 15 and to Exercise 1.8. The total sum of squares can be divided into three parts as shown by the following formula: <' .'1' I~, ~~'''' ':r~I:" I·,;'~t~~;~~ Turn to Exercise 1.·8 and find the total sum of squares and the three I. parts. They 'are: The three parts add to the total which"werifies Equation (l.ll). In Exercise 1.8, the sum of squares called remainder was computed directly (1.11) (1.12) , ' ~: ", " (see Part (10) of Exercise 1.8). In practice, the remainder sum of squares is usually obtained by subtracting the row and column sum of squares from the total. Again, the proof of Equation (1.11) is not difficult if one makes the right start. In this case the deviation, (Xij-X ••), is divided into three - - parts by adding and subtracting Xi. ~~ X.j as follows: (Xij-X ••) - (Xi.-X ••) + (X.j-X ••) + (Xij-Xi.-X.j+X ••) Exercise 1.19. Prove Equation (1.11) by squaring both sides of Equa- tion (1.12) and then doing the summation. The proof is mostly a matter of showing that the sums of the terms which are products (not squares) are zero. KN For example, showing that EE(Xi.-X ••)(Xij-Xi.-X.j+X ••) • 0 • ij element. 33 CHAPTER II. RANDO}1VARIABLES AND PROBABILITY 2.1 ~~DOM VARIABLES The ward IIrandamll has a wide variety af meanings. Its use in such terms as "randam events, II"random variable, II ar "random sample, IIhowever, implies a randam pracess such that the prababi~ity af an event accurring is knawn a priari. Ta select a randam sample af elements from a papulatian, tables af random numbers are used. There are variaus ways af using such tables ta make a random selectian sa any given element will have a specified prabability af being 8!electe,d•. : The theary af praba.b:il,;f.ty :8!ampl,in.g is :faunded an the cancept af a randam variable which is a ,variable that, by chance, might equal any ane , I af a defined set af values,.";::The value af.a randam :.variablean any partic- ular accasian is determined by a randam pracess'in .such a way that the chance (prabability) af its being equal ta anyspeclfied value in the set is known. This is in accard with the definitian af a prabability sample which states that every element af the populatian must have a known prab- ability (greater than zera) afbeing selected. A primary purpose af this chapter is ta present an elementary, minimum intraductian ar review af prabability as backgraund .far the next chapter an expected values af a randam variable. This leads ta a thearetical basis far sampling and far evaluating the accuracy af estimates fram a prabability-sample survey. In sampling theary, we usually start with an assumed papulatian af N elements and a measurement far each element af sOme characteristic X. A typical mathematical representatian af the N measurements ar values is Xl"",Xi""'~ where Xi is the value af the characteristic X far the ith th Associated with the i element is a prabability Pi' which is the prabability af abtaining it when ane element is selected at randam fram the 34 set of N. The Pits will be called sele~tion probabilities. If each element has an equal chance, of selection, Pi 1 - -N' The Pits need not be equal, but we will specify that each Pi>O. When referring to the probability of X being equal to Xi we will use P(Xi) instead of Pi' We need to be aware of a distinction between selection probability and inclusion probability, the latter being the probability of an element being included in a sample. In this chapter, much of the discussion is oriented toward selection probabilities because of its relevance to finding expected values of estimates from samples of various kinds •.. , Definition 2.1. A random variable is a variable that can''equal any value Xi' in a defined set, with a probability P(Xi). When an element is selected at random from a population and a measure- ment of a characteristic of it is made, the value obtained is a random variable. As we shall see later, if a sample,of elements is selected at random from a population, the s~le average and other quantities calculated from the sample are random variables. Illustration 2.1. One of the most familiar examples of a random variable is the number of dots that happen to be on the top side of a die when it comes to rest after a toss. This also illustrates the concept of probability that we are interested in; namely, the relative frequency with which a particular Qutcome will occur in reference to a defined set of possible outcones. With a die there are six possible outcomes and we expect each to occur with the same frequency, 1/6, assuming the die is tossed a very large or infinite number of times. Implicit in a statement that each side of a die has a probability of 1/6 of being the top side are SOMe assumptions about the physical structure of the die and the "rando1lllless" of the toss. 35 The additive and multiplicative laws of probability can be stated in several ways depending upon the context in which they are to be used. In sampling, 'our interest is primarily in the outcome of one random selection or of a series of random selections that yields a probability sample. Hence, the rules or theorems for the addition or multiplication of prob- abilities will be stated or discussed only in the context of probability .. 2.2 ADDITION OF PEQBABILITIES 1). " : , .~ I r Assume a population th " Xi for the i eleD:l~nt. That is, we h~ye a set of values of X, namely ,: 1, ~ l ., r. I 'd • !I' i ~ ~•~.r! j ~"I J:: Xl' •••,Xi':•.,~ ~- 'i.etPI'· ,••,Pi'••.,PN. be a set of selection probabilities, ~:~~.~'i.;~j~;~~~ ft(*~{~ ~~ ;1I!~'f~~, -f>, sampling • • t ~ ;;. ": ~ of N elements and a variable X which has a value where Pi is the p~obability of selecting, the ith element when a random " • ~ ! ~ selection 'is made. . J. ~ ,,~.:.;--:Q'~:~! We specify that each Pi must be greater than zero and N' that ~Pi - 1. When an element is selected at random, the probability that i th th it is either the i element ~ the j element ,isPi .+ Pj. This addition rule can be stated more generally. Let P be the sum of the selection s '\., : probabilities for the elements in a subset of the N elements. When a random selection is made from the whole set, P is the probability that the element s selected is from the subset and I-P is the probability that it is not from s the subset. With reference to the variable X, let P{Xi) represent the probability that X equals Xi. Then P{Xi)+P{Xj) represents the probability that X equals either Xi or Xj; and Ps{X) could b~ used to represent the probability that X is equal to one of the values in the subset. Before adding (or subtracting) probabilities one should determine whether the events are mutually exclusive and whether all possible events have been accounted for. Consider two subsets of elements, subset A and 36 subset B, of a population of N elements. Suppose one element is selected at random. What is the probability that the selected element is a member of either subset A or subset B? Let peA) be the probability that the selected element is from subset A; that is, peA) is the sum of the selec- tion-probabi1ities for elements in subset A. PCB) is defined similarly. If the two subsets are mutually exclusive, which means that no element is in both subsets, the probability that the element selected is from either subset A or subset B is peA) + PCB). If some elements are in both subsets, see Figure 2.1, then event A (whiCh is the selected element being a member .of subset A) and even.t B (which is the selected element being. a member of .~ I' II I X. I subset B) are not mutually exclusive events. Elements included in both I " subsets are counted once in peA) and once in P(B). Therefore, we must -, •••• - 'I' subtract P(A,B) from peA) + PCB) where P(A,B) is the sum of the probabilities ~I' (~ •• ~',. ~ \' ,•••.- •. for the elements that belong to both subset A and subset B. Thus , peA or B) - peA) + PCB) - P(A,B) I 'r' " ' • . . Figure 2.1 .. .. To summarize, the additive law of probability as used above could be stated as follows: If A and B are subsets of a set of all possible outcomes that could occur as a result of a random trial or selection, the probability .' 37 that the outcome is in subset A or in subset B is equal to the probability that the outcome is in A plus the probability that it is in B minus the probability that it is in both A and B. The additive law of probability extends without difficulty to three or more subsets. Draw a figure like Figure 2.1 with three subsets so that some points are common to all three subsets •• Observe that the additive : J law extends to three subsets as follows: ~i P(A or B'or C)-P(A)+P(B)+p(e)-p(Ao,B)-p(A,e)-p(B ,e)+p(A,B ,e) As a case for further discussion purposes, assume a population of N , " elements and "0,0 tdr{~~~f~ Jfor classification. A two-way classification of 'r " " the elements could be displayed in the format of Table 2.1 • . j;-' r Table '2~1--A 'two-way classification of N elements ,r' •• I • " ; X class ·,.------------------- Y class 1 j ... s Total ···· · . · . ----- ----------------------- ----- 1 ······ i ., ••• Nis'Pis : . · . · : t Nt1,Ptl ... Ntj ,Ptj ... N P N P · ts t ts t. t t. ·· : ··· . · . · Total N.l N .j N.s ~tP=l The columns represent a classification of the elements in terms of criterion Xj the rows represent a classification in terms of criterion Yj Nij is the number of elements in X class j and Y class i; and Pij is the sum of the 38 selection p~obabilities for the elements in X class j and Y class i. Any one of the N elements can be classified in one and only one of the t times s cells. Suppose one element from the population of N is selected. According to the additive law of probability we can state that ,,!.::;;1~~,ii~: , ." .~',".. EPij - P is the probability that the element selected is from i .j X class j ~ and !Pij • P is the probability that the element selected is from i· j Y class i ~ where .~ :;1 r, r ( • is the probability that the element selected is from I ; ,~ :, I'''' ~' : ; ~ t j • 1 .• 1' (belongs to both) X class j and Y class i. I I , 1 :::, ,r ~ The probabilities p'.j ,and"~i. ,arie,c"'~rl,~,d. ~lr~fi~al ,p.robab:ili~ies. The probability that one randomly selected element is from X class I.: ! :11' j ~ from Y-class i is p. j' + P{. - ~ij. (The answer is not p.j + Pi. because in p.j + Pi. there are Nij elements in X class ~ and Y class i that are counted twice.) _ Nij ~ If the probabilities of selection are equal~'~ij N ~ p.j - N ~ " and Pi. Also~ the probability that the selected student lives in dormitory A . 1600 a freshman is 5000 ' that he is a 1600 either a freshman or a sophomore is 5000 + Illustration 2.2. Suppose there are 5~000 students in a university. Assume there are 1~600 freshmen~ 1,400 sophomores, and 500 students living in dormitory A. From a list of the 5,000 students, one student is selected at random. Assuming each student had an equal chance of se1ection~ the probability that the selected student is 1400 sophomore is 5000 ' and that he is 1400 5000 • • .' •... 39 500 is 5000' But, what is the probability that, the selected student is either a freshman or lives in dormitory A? The question involves two classifica- tions: one pertaining to the student's class and the other to where the student lives. The inforlnation given about the 5000 students could be arranged as-follows: Class Dormitory Total Freshmen Sophomores Others · A 500 ·· Other · 4500 ··· Total 1600 1400 2000 5000 , , :!. From the above format, one can readily observe that the answer to the ques- .tion depends upon how many freshmen live in dormitory A. If the problem had stated that 200 freshmen live in dormitory A, the answer would have 1600 500 200 been 5000 + 5000 - 5000 • Statements about probability need to be made and interpreted with great care. For example, it is not correct to say that a student has a probability of 0.1 of living in dormitory A simply because 500 students out of 5000 live in A. Unless students are assigned to dormitories by a random process wi th known probabilities there is no basis for stating a studen t's probability of living in (being assigned to) dormitory A. We are consider- ing the outcome of a random selection. Exercise 2.1. Suppose one has the following information about a population of 1000 farms: ,40 600 produce corn 500 produce soybeans 300 produce wheat 100 produce wheat and corn 200 have one or more cows all farms that have cows also produce corn ll' 200 farms do not produce any ~rops One farm is selected at random with equal probability from the list of 1000. What is the probability that the'selected farm, (a) produces corn? Answer: 0.6 (b) does not:produ~e wheat? (c) produces corn but no wheat? Answer: 0.5 , , (d) ; , both? produces corn or wheat but not (e) has no cows? Answer: 0.8 (f) produces corn or soybeans? (g) produces corn and has no cows? Answer: 0.4 (h) produces either corn, cows, or both? (i) does not produce corn or wheat? . _ l One of the above questions cannot be answered. Exercise 2.2. Assume a population of 10 elements and selection probabilities as follows: Element Xi Pi Element X. Pi 1 ---- 1 2 .05 6 II .15 2 7 .10 7 2 .20 3 12 .08 8 8 .05 4 0 .02 9 6 .05 5 8 .20 10 3 .10 .' 41 One element is selected at random with probability Pi. Find: (a} P(X-2),. the probability that X - 2. (b) P(X>lO), the probability that X is greater than 10. (c) P(X;52), the probability that X is equal to or less than 2. (d) P(3<X>lO), the probability that X is greater than 3 and less than 10 (e) P(X;53 ,o~ X~lO), the probability that X is either equal to or less than 3 or is equal to or greater than 10. Note: The answer to (d) and the answer to (e) should add to 1. ~~ ~ • I :' ~ ~ ~!. i r] i~ .. ,I SO far, we have been discussing the probability of an event occurring as a result of a single randOm selection. When more than one random selection , : I " occurs simultaneously or in succession the multiplicat.1ve law of prob- A I·~.j~:::': ~ It gd:' 1 " ' : ~~:!~:' ab.11.1tyis useful. , '. ~i ,,:;' ~ , 'Ji'~ '1A '" l, r '. 2.3 MULTIPLICATION OF PROBABILITIES Assume a population of N elements and selection probabilities N Pl, ••• ,Pi,••• ,PN• Each Pi is greater than zero and EPi - 1. Suppose i two elements are selected but before the second selection .ismade the first element selected is returned to the population. In this case the outcome of the first selection does not change tpe selection probabilities for the second selection. The two selections (events) are independent. The probability of selecting the ith element first and the jth element second is, PiPj, the product of the selection ~robabilities Pi and Pj. If a selected element .isnot returned to the population before the next I selection is made, the selection probabilities for the next selection are __ changed. The selections are dependent. 42 The multiplicative law of probability, for two independent events A and B, states that the joint probability of A and B happening in the order A,B is equal to the probability that A happens times the prob- ability that B happens. In equation form, P(AB) - P(A)P(B). For the order B,A, P(BA) - P(B)P(A) and we note that P(AB) - P(BA). Remember, independence me~s that the probability of B happening is not "affected to any number of indepen'dent 'eve~ts; Thus, P(ABC) ••P(A)P(B)P(C). by the occurrence of A and vice versa. The multiplicative law extends For two dependent events A and B, the multiplicative law states that 1 , •• : ~ ~". i, the joint probability of A and B happening in the order A,B is equal to ~:; ;, .1;) I I ~.~ -l 1:..J l.t ~ ;:1 l!:,l',' ( 'I,~ 11: , ..•. :t i~~: )'1 the probability of A happening ti.wis the probability that B happens under '" ~ • II j "$If! r"" ·'J~1 ~;I ·.irf,~" ',·'1' :'" the condition that A has already happened. In equation form P(AB) •• ~ 4:' 'll ~ r!!( ~':~'I II ::~ ).::f...·~tjl."I:fr t!f(!~ 1',;' 1111!~ ': '" I " ~ ,,"'l")C P(A)P(BIA); or for the order i,A we have P(BA) ••P(B)P(AIB). The vertical bar can usually be translated as "given" or "given that." The notation on the left of the bar refers to the event under consideration and the nota- tion on the right to a condition under which the event can take place. p(BIA) is called conditional probability and could be read "the" prob- ability of B, given that A has already happened," or simply "the prob- ability of B given A." When the events are in~'!pendent,p(BIA) - P(B); that is, the conditional probability of B occurrin~ is the same as the unconditional probability of B. Extending the multiplication rule to a series of three events A,B,C occurring in that order, we have P(ABC) - P(A)P(BIA)P(cIAB) where p(CIAB) is the probability of C occurrinsgiven that A and B have already occurred. 2.4 SAMPLING WITH REPLACEMENT When a sample is drawn and each selected element is returned to the population before the next selection is made, the method of sampling is • 43 called "sampling with replacement." In this case, the outcome of one selection does not change the selection probabilities for another selection. Suppose a sample of n elements is selected with replacement. Let the values of X in the sample be xl,x2"",xn where Xl is the value of X obtained on the first selection, x2 the value obtained on the second selection, etc •. ~otice that Xl is a random variable that could be equal to any value in.the population set o~.values Xl'X2""'~' and the prob- ~ ability that Xl ,equals Xi is Pi' The ,same statement applies to x2' etc. Since the selections are;,independent,,:~tl1eprobability of getting a sample -.• I, of n in a particular order is the pr~4uct of the selection probabilities namely, P(xI)1~(~~);~:~.~~n~t1:~~~:"f~(xI~~~ ~~~,J~i,~~~~~~heelement, ~e~ected on ,the first draw, p(x2) is th~ Pi for the.el~ment ,selected on the second '" . ~ A, .I .} • ..:;: __Jt .. j, . '~. ~'l ,,' I ;~ draw, etc. Illustration 2.3. As an illustra~ion, consider a sample of two ,~ elements selected with equal probability and with replacement from a popu- lation of four elements. Suppose the values of some characteristic X for the four elements are Xl' X2, X3, and X4• There are 16 possibilities: Xl 'Xl X2,Xl X3 'Xl X4~Xr Xl ,X2 Xl ,X2 X3'X2 X4,X2 XI'X3 X2,X3 X3'X3 X4'X3 XI'X4 X2,X4 X3'X4 X4'X4 1 1 In this illustration p(~) is always equal to 4and p(x2) is always 4. Hence each of the 16 possibilities has a probability of (~) (t) - i6 • 44 Each of the 16 possibilities is a different permutation that could be regarded as a separate sample. However, in practice ~ we are not concerned about which element was selected first or second) it is more logical to disregard the order of selection. Hence, as possible samples and the probability of each occurring, we have: Sample Probability Sample Probability Xl,Xl 1/16 x2,X3 1/8 JS.,X2 1/8 x2,X4 1/8 :: . ~,~ I' ' i , I . I' ; ! •~ ,'I' Xl,X3 1/8 X3,X3 1/16 ' . I J' •• JS.,X4 1/8 x3,X4i 1/8 J ,=- : ! t· • :~I ldi,r .. J ~~,r ' X2,Xi 1/16 X4 ',Xl;" 1/16 l _ ~ :. • l"I. ., - "I' Note that the sum of the probabilities is 1. 'That must always be the I jo',l case if all possible samples have been listed with the correct prob- r~r ;; :J ,!" abilities • , , ' Also note ,that, since the probability (relative frequency • " ; I of occurrence) of each sample is known, the average for each 'sample is a randomvariable. In other words, there were 10 possible samples, and anyone of 10 possible sample averages could have occurred with the probability indicated. ,This-is a"simple illustration of the fact tbat the sample average satisfies the definition of a randomvariable. As the theory of sampling unfolds, we will be ex,amining the properties of a sample average that exist as a result of its being a randomvariable. Exercise 2.3. With reference to Illustration 2.3, suppose the 1 1 ,31 probabilities of selection were P1 • 4' P2 • 8' P3 • 8' and P4 • 4· Find the probability of each of the ten samples. Rememberthe sampling is with replacement. Check your results by adding the 10 probabilities. 45 For the sample composed of elements The sum should be 1. Partial answer: . I 1 1 I 1 2 and 4 the probability is (8) (4') + (1;) (8) I I a- 16. 2.5 SAMP~ING WITHOUT REPLACEMENT When a selected element is not returned to the population before the next selection is made, the sampling method is called sampling without replacement. In this case, the selection probabilities change from one ! As above, assume a population of N elements with values of some dr~ to the nextj that is, the selections (events) are dependent. characteristic X equal to Xl'X2' •••'~. , ' for the first Let the selection probabilities 1 s~,lection be PI ',•••,Pi'.,••PN where each P1>0 and I:Pi •••1.•.. , el,ements are,selected wi~~out replacement. Let Xl' x2' and. Suppose three t· x3 be the values'lof X obtained on the f~rst, second: and ;hird random draws, respecti~elY. What is the prob,a~.~litythat Xl •••XS' x2 •••X6' and" x3 • X7? . Let P~~~~X6,X7) represent t~~ :probability,whic~ is the prob- ability of selecting elements S, 6, and 7 in that order. According to the multiplicative probability law for dependent events, , ... P~X5,X6,X7) - P(XS)P(X6JXS)P(X7IXs,X6) . It is clear that P(~S) • PS. For the second draw the selection prob- abili~ies (after element S is eliminated) must be adjusted so they add i ' to 1. Hence, for the second draw the selection probabilities are PN , ..• , l-P • N P6 D--l-P • S Sindlarly, Therefore, (2.1) 46 Ps P7. Observe that P(X6,XS,X7) - (P6)(1-P )(l-P _p). Hence, P(XS,X6,X7) ~ 6 6 S P(X6,X5,X7) unless Ps - P6• In general, each permutation of n elements has a different probability of occurrence unless the Pi's are all equal. To obtain the exact probability of selecting a sample composed of e1e- ~nts 5, 6, and 7, one would need to compute the probability for each of the six possible permutations and get the sum of the six probabilities. Incidentally, in the actual process·of selection, it is not neces- sary to compute a new set of selection probabilities after each selection is made. Make each selection in the same way that the first selection l 'I was made. If an element is selected which has already been drawn, ignore the random number 'and continue the same process of random selection until a new element is drawn. As indicated by the very brief discussion in this section, the theory of sampling without replacement and with unequal probability of selection can be very complex. However, books on sampling present ways of circumventing the complex problems. In fact, it is practical and advantageous in many cases to use unequal probability of selection in sampling. The' probability theory for sampling with equal probability of selection and without replacement is relatively simple and will be discussed in more detail. Exercise 2.4. For a population of 4 elements there are six possible List the six pos.sib1e samples' and find the prob- samples of two when sampling without replacement. 1 - -4· 1 - -8' ability of getting each sample. Should the probabilities for the six samples add to 11 Check your results. ( (' I I 47 Exercise 2.5. Suppose two elements are selected with replacement and with equal probability from a population of 100 elements. Find the probability: (a),that element number 10 is not selected, (b) that ele- ment number 10 is selected only once, and (c) that element number 10 is . ~ selected twice? As a check, the three probabilities should add to 1. Why? Find the probability of selecting the combination of elements 10 and 20. t, Exercise 2.6. Refer to Exercise 2.5 and change the spec.1f1cation 0: "with replacement" to "without replacement." Answer the same questions. Why is the probability of getting the combination of elements 10 and 20 'Ii:: I ,1< , 1:! 1,,1 ' .. greater than it was in Exercise 2.5? . : ::' j 2.6 SIMPLE RANDOM SAMPLES I : j~: J :. t, ",: "II' , In practice, nearly all samples are selected without replacement. I, ; ~. "oil Selection of a random sample of n elements ,with equal probability and without replacement, from a population of N elements is called simple random sampling (srs). One element must be selected at a time, that is, n separate random selections are required. First, t.heprobability of getting a particular combination o(_n ... elements will be discussed. Refer t.oEquation (2.1) and the dtscussion 1 preceding it. The Pi's are all equal to N for simple random sampling. III Therefore, Equation (2.1) becomes P(X5,X6,X7) - (N)(N-l)(N-2)' All per- mutations of the three elements 5, 6, and 7 have the same probability of occurrence. There are 3! - 6 possible permutat ion.s• Therefore, the probability that the sample is composed of the elements 5, 6, and 7 is (1) (2) (3) N(N-l)(N-2) • Any other combination of three elements has the same probability of occurrence. 48 In general, all possible combinations of n elements have the same chance of selection and any particular combination of n has the following probability of being selected: (1)(2)(3) ••• (n) _ nl(N-n)I N(N-I)(N-2) ••• (N-n+I) NI (2.2) Nt According to a theorem on number of combinations, there are nl(N-n)I possible combinatioDs (samples) of n elements. If each combination of n elements has the same chance of being the sample selected, the probability. of selecting a specified combination must be the reciprocal of the number of combinations. This checks with Equation (2.2). An important feature of srs that will be needed in the chapter on th . expected values is the fact ·that the j element of the population is as I th likely to be selected at the i random draw as any other. A general th expression for the probability that the j element of the population is th selected at the i drawing is I - -N <.2.3) Let us check Equation 2.3 for i - 3. The equation becomes I - -N th The probability that the j element of the population is selected at the third draw is equal to the probability that it was not selected at either the first or second draw times the conditional probability of being selected at the third draw, given that it was not selected at the first or second draw. (Remembe~the sampling is without replacement). Notice ~l ~ that ~ ts the probability that the j element is not selected at the N-2 first draw and N-I is the conditional probability that it was not selected at the second draw. Therefore, eN;I) (~:i)is the probability that the j th 49 element has not been selected prior to the third draw. When the third This verifies Equation (2.3) for i • 3. draw is made, the conditional probability of 1 is N-2. ,Hence the probab'ility of selecting N-l N-2 1 1 draw is (T)(N-l)(N-2) • N • th selecting the j element th the j element at the third jth element in a population has a probability th at the i drawing. To summarize, the general result for any size of sample is that the 1 equal to N of being selected It means that xi (the value of X obtained at the ith 1 draw) is a random variable that has a proba~ility of N of being equal to any value of the set Xl' ••• '~' What probability does the jth element have of being included in 'a 1 sample of n? We have just shown that it has a probability of N of being selected at the ith drawing. Therefore, any given element of the popula- 1 tion has n chances, each equal to N t of being included in a sample. The· element can be selected at the first draw t 2!:. the second draw, ••• , .5!!.the nth draw and it cannot be selected twice because the sampling is without replacement. 1 Therefore the probabilities, N for each of the n draws, can n be added which gives N as the probability of any given element being included in the sample. Illustration 2.4. Suppose one has a list of 1,000 farms which includes some farms that are out-of-scope (not eligible) for a survey. There is no way of knowing in advance whether a farm on the list is out-of-scope. A simple random sample of 200. farms is selected from the list. All 200 farms are visited but only the ones found to be in scop~ are included in the sample. What probability does an in-scope farm have of being in the sam- ple? 1 Every farm on the list of 1000 farms has a probability equal to 5' 50 of being in the sample of 200. All in-scope farms in the sample of 200 are included in the final sample. 1 Therefore, the answer is 5. Exercise 2.7. From the following set of 12 values of X a srs of three elements is to be selected: 2, 10, 5,8, 1, 15, 7, 8, 13, 4, 6, and 2. Find P(x~l2) and P(J<x<12). Remember that the total possible number ~f samples of 3 can readily be obtained by formula. Since every possible sample of three is equally likely, you can determine which sam- ples will have an x<3 or an ~12 without listing all of the numerous possible samples. Answer: - 3 - 9 - 208 P(x~12) - 220 ; P(x~3) - 220 ; P(3<x<12) - 220· 2.7 SOME EXAMPLES OF RESTRICTED RANOOM SAMPLING There are many methods ,other,than srs that will g1ve every element an equal chance of being in the sample, but some combinations of n ele- ments do not have a chance of being the sample selected unless srs is used. th For example, one might take every k element begi~ning from a random starting point between I and k. This is called systematic sam- pIing. For a five percent sample k would be 20. The first element for the sample would be a random number between 1 and 20. If it is 12, then elements 12,32,52, etc., compose the sample. Every element has an 1 equal chance, 20 ' of being in the sample, but there are only 20 com- binations of elements that have a chance of being the sample selected. Simple random sampling could hav~ given the same sample but it is the method £f. sampling that characterizes a sample and determines how error due to sampling is to be estimated. One may think,of sample design as a matter of choosing a method of sampling; that is, choosing restrictions to place on the process of selecting a sample so the combinations which 51 have a chance of being the sample selected are generally ''better''than many of the combinations that could occur with simple random sampling. At the same time, important properties that exist for simple random sam- pIes need to be retained. The key properties of srs will be developed in the next two chapters. Another common method of samplin~ involves classification of all elements of a population into groups called strata. A sample is selected from each stratum. Suppose Ni elements of the population are in the i th stratum and a simple random sample of,ni elements is selected from it. This is called stratified random sampling. It is clear' that every ele- ment in the ith ~t:r'~tum~as a probability ,equal to :i of being in the "ni i sample. If the sampling fraction, N ' is the same f~r all I, ,', ; ," <, ':' ·i, " every element of the population has-an--equal chance, namely 'I 'I being in the sample. Again every element of strata, ni N ' of i the population has an equal " , t chance of selection and of being in the sample selected, but some combi- nations that could occur when the method is srs cannot occur when stratified random sampling is used. So far, our discussion has referred to the selection of individual elements, which are the units that data pertain to. For sampling purposes a population must be divided into parts which are called sampling units. A sample of sampling units is then selected. Sampling units and elements could be identical. But very often, it is either not possible or not practical to use individual elements as sampling units. For example, suppose a sample of.households is needed. A list of households does not exist but a list of blocks covering the area to be surveyed might be avail- able. In this case, a sample of blocks mi~ht be selected and all households ',l""."- 52 within the selected blocks included in the sample. The blocks are the sampling units and the elements are households. Every element of the population should belong to one and only one sampling unit so the list of sampling units will account for all elements of the population without duplication or omission. Then, the probability of selecting any given element is the same as the probability of selecting the sampling unit that it belongs to. I1lust~ati?n 2.5_. Suppos.e a population is composed of 1800 dwelling units located within 150 well-defined blocks. There are several possible , sampling plans. A srs of 25 blocks could be selected and every dwelling unit in the selected blocks could be included in the sample. In thi~ 1 case, the sampling fraction is '6 and every dwelling unit has a probability 1 of '6 of being in the sample. Is this a srs of dwellin~ units? No, but one could describe the sample as a random sample (or a probability sample) of dwelling -units and state that every dwelling unit had an equal chance of being in the sample. That is, the term "simple random sample" would apply to blocks, not dwelling units. As an alternative sampling plan, if there were twelve dwelling units in each of the 150 blocks, a srs of two dwelling units could be selected from each block. This scheme, which is an example of stratified random sampling, would also give every dwelling unit 1 a probability equal to '6 of being in the sample. Illustration 2.6. Suppose that a sample is desired of 100 adults living in a specified area. A list of adults does not exist, but a list of 4,000 dwellin~ units in the area is available. The proposed sampling plan is to select a srs of 100 dwelling units from the list. Then, the field staff is to visit the sample dwellings and list all adults living 53 in each. Suppose there are 220 adults living in the 100 dwelling units. A simple random sample of 100 adults is selected from the list of 220. Consider .the probability that an adult in the population has of being in the sample of 100 adults. Parenthetically, we should recognize that the discussion which follows overlooks important practical problems of definition such as the definition of a dwelling unit, the definition of an adult, and the.defini- ~ clear, that the list of dwelling units is complete, that no dwelling is tion of living in a dwelling unit. However, assume the definitions are on the list more than once, and that no ambiguity exists about whether I an adult lives or does not live in a particular dwelling unit. Incom- . plete definitions often lead to inexact probabilities or ambiguity, that gives difficulty in analyzing or interpreting results. The many practical problems should be discussed in an applied course on sampling. It is clear that the probability of a dwelling unit being in the 1 sample is 40. Therefore, every person on the list of 220 had a chance 1 of 40 of being on the list because, under the specifications, a person lives in one and only one dwelling unit, and an adult's chance of being on the list is the same as that of the dwelling unit he lives in. The second phase of sampling involves selecting a simple random sample of 100 adults from the list of 220. The conditional probability 100 5 of an adult being in the sample of 100 is 220 - 11. That is, given the 5 fact that an adult is on the list of 220, he now has a chance of 11 of being in the sample of 100. Keep in mind that the probability of an event happening is its rela- tive frequency in repeated trials. If another sample were selected 54 following the above specifications, each dwelling unit in the population 1 would again have a chance of 40 of being in sample; but, the number of adults listed is not likely to be 220 so the conditional probability at the second phase depends upon the number of dwellings units in the sample blocks. Does every adult have the same chance of being in the sample? Examine the case carefully. An initial impression could be misleading. Every adult in the population has an equal chance of being listed in the first phase and every adult listed has an equal chance of being selected at the second phase. But, in terms of repetition of the whole sampling .~ plan each person does not have exactly the same chance of being in the sample of 100. The following exercise will help clarify the situation and is a good exercise in probability. Exercise 2.8. Assume a population of 5 d.u.'s (dwelling units) with the following numbers of adults: Dwelling Uni t No. of Adults 1 2 2 4 3 1 4 2 5 3 A srs of two d.u.'s is selected. A srs of 2 adults is then selected from a list of all adults in the two d.u.'s. Find the probability that a speci- fied adult in d.u. No.1 has of being in the sample. Answer: 0.19. Find the probabIlity that an adult in d.u. No. 2 has of beinR in the sample. Does the probability of an adult being in the sample appear to be related to the number of adults in his d.u.? In what way? ~ to sample; in fact, the number of adults in the sample is a random variable. &'/ ' 55 An alternative is to take a constant fraction of the adults listed instead of a constant number. For example, the specification might have 1 been to select a random sample of 2of the adults listed in the first phase. In this case, under repeated application of the sampling speci- fications, the probability at the second phase does not depend on the outcome of the first phase and each adult in the population has an equal 1 1 . 1 chance, (40) (2) - 80 ' of being selected in the sample. Notice that under this plan the number of adults in a sample will vary from sample For some surveys, interviewing more than one adult in a dwelling unit is inadvisable. Again, suppose the first phase 0'£ sampling is to select a srs of 100 dwelling units. For the second phase, consider the following: .1! When an interviewer completes the listing of adults in a sample dwelling, he is to select one adult, from the list of those living in the dwelling, at random in accordance with a specified set of instructions. He then interviews ··theselected adult if available; otherwise, he returns at a time when the selected adult is available. What probability does an adult living in the area have of being in the sample? According to the multi- plication theorem, the answer is P'(D)P(AID) where P#(D) is the probability of the dwelling unit, in which the adult lives, being in the sample and p(AID) is the probability of the adult being selected given that his specifically, P#(D) = ~o and p(AID) th in the i dwe~ling. where ki is the number of adults 1 1 chance, (40)(k)' of being in a sample i number of adults in his dwelling unit. dwelling is in the sample. More 1 -k' i Thus, an adult's is inversely proportional to the Exercise 2.9. Suppose there are five dwellin~ units and 12 persons living in the five dwelling units as follows: 'r"" .: ' • ~~ " , .,~, :t:~!~··(~ , ., - •..• 56 Dwelling Uni t Individuals 1 1, 2 2 3, 4, 5, 6 3 7, 8 4 9 5 10, 11, 12 1. A sample of two dwelling units is selected with equal probability and without replacement. All individuals in the~elected dwelling units are in the sample. What probability does individual number 4 have of being in the sample? Individual number 9? , 1.111111 2. Suppose from a list .of the 'twelve individuals that one individual is selected with equal probability. From the selected individual two items of information are obtained: his age and the value of the dwelling in which he lives. Let Xl' X2, ••'~,X12 represent the ages of the 12 indi- viduals and let Yl ,••• 'Y5 represent the values of the five dwelling·:units. th 1 Clearly, the probability of selecting the i individual is 12 and there- 1 fore P(Xi) • 12. Find the five probabilities P(Yl), ••• ,P(Y5). Do you 2 agree that P(Y3) • l2? As a check, ~P(Yj) should equal one. 3. Suppose a sample of two individuals is selected with equal prob- ability and without replacement. Let Ylj be the value of Yj obtained at the first draw and Y2j be the value of Yj obtained at the second draw. Does P(Ylj) • P(Y2j)? That is, is the probability of gettin~ Yj on the second draw the same as it was on the first? If the answer is not evident, refer to Section 2.5. Exercis~ 2.10. A small sample of third-grade students enrolled in public schools in a State is desired. The following plan is presented only ., 57 as an exercise and without consideration of whether it is a good one: A sample of 10 third-grade classes i~ to be selected. All students in the 10 classes will be included in the sample. Step 1. Select a srs of 10 school districts. Step 2. Within each of the 10 school districts, prepare a list of public schools having a third grade. Then select one school at random from the list. Step 3. For each of the"10 schools resulting from Step 2, list the third-grade classes and select one class at random • (If there is only 'one'"f'thi'rd-igrad'e' 'class in the school, it is in the sample). This will give a sample of 10 classes. , , • "" 1 '", Describe third-grade classes i~',the 'population which have relatively small chances of being selected. D~fine needed, notation and write a " ) l" ,l ~l j mathematical expression representing the"proba~ility of a third-grade I' ' l I ': ,II .': I I ". I' \, 'l l.~. ,I:' :. I '~t r class beins; in the sample. 2.8 TWO-STAGE SA}~LING For various reasons sampling plans often employ two or more stages of sampling. For example, a sample of counties .might be selected, then within each sample county a sample of farms might be selected. Units used at the first stage of sampling are usually called primary sampling units or psu's. The sampling units at the second stage of sam- pIing could be called secondary sanplinr, units. However, since there has been frequent reference earlier in this chapter.to "elements of a popula- tion,lI the sar.mling units at the second stage will be called elements. In the simple case of two-stage samplin~, each element of the popu- lation is associated with one and only one primary sampling unit. Let i , , ' 'Ir./""'.;.4~ I,; I l r-~ ~ '~~~~ 58 be the index for psu's and let j be the index for elements within a psu. th Thus Xij represents the va~ue of some characteristic X for the j element th. in the i psu. Also, let M - the total number of psu's, m'· the number of psu's selected for a sample, th N - the total number of elements in the i psu, and i th ni - th~ number of elements in the sample from the i psu. Then, M LNi - N, the total number of elements in the population, and i m Ln1 - n, the total number of elements in the sample., 1 Now consider the probability of an element being selected by a two step process: (1) Select one psu, and (2) select one element!·within the selected psu. Let, th P - the probability of selecting the i psu, i th the conditional ,probability of selecting the j th th element in the i psu given that the i psu has been selected, and already Then, Pij - the the th overall probability of selecting the j element in th i psu. If the product of the two probabilities, Pi and Pjli' is constant for every element, then every element of the population has an equal chance of 59 being selected. In other words, given a set of selection probabilities 1 Pl,·,.,PM for the psu's, one could specify that Pij - N and compute Pjli ' 1 where 'Pjli - NP i ' so every element of the population will have an equal ,,' chance of selection. , Exercise 2.11. Refer to Table 2.1. An element is to be selected by a three-step process as follows: (1) Select one of the Y classes (a row) , N with probability Ni., (2) within the selected row select an X class (a I N column) with probability Nij , (3) within the selected cell select an £ i. element with equal probability. Does each element in the population of N , \ 'I: " : elements have an equal probabili,ty of being drawn? What is the probability? ':. 'I The probability of an element being included in a two-stage sample ! ~: r l.t •.~j \1 ,. I is given by (2.4) where th p" - the probability that the i psu is in the sample i of psu's, and Pili • the conditional probability which the j element has th of being in the sample, given that the i psu has been selected. The inclusion probability Pij will be discussed very briefly for three important cases: ability and without replacement. The probability, Pi ' being in the sample is f1 -: where fl is the sampling fraction for the (1) Suppose a random sample of m psu's is selected with equal prob- th of the i psu first-stage units. In the second stage of sampling assume that, within each of the m psu's, a constant proportion, f2, of the elements is selected. 60 th That is, in the i psu in the sample, a simple random sample of ni ele- ments out of Ni is selected, the condition being that ni - f2Ni• Hence, th th the j element in the i psu being in the the conditional probability of ni sample is P ~I - -. f2 • j i ~i - flf2 which shows that an SUbstitutin~ in Equation 2.4, we have element's probability of being in the sample is equal to the product of the sampling fractions at the two stages. In this case Pi j is constant and is the overall sampling fraction. UnleSs Ni is the same for all psu's, the size of the sample, ni - f2Ni ' varies from psu to psu. Also, since the psu's are selected m In at random the total'size of the sample, n - En - f2 ENi, is not constant . i i i I with regard to repetition of the sampling plan. In practice variation in , , the size, ni, of the sample from psu to psu might be very undesirable. If appropriate information is available, it is possible to select psu's with probabilities that will equalize the sample sizes ni and also keep Pij constant. (2) N i Suppose one psu is selected with probability Pi - ~ • This k - -N is commonly known as sarnplinp,with pps (probability proportional to size). Within the selected psu, assume that a simple random sample of k elements is selected. (If any Ni are less than k, consolidations could be made so all psu's have an Ni greater than k). Then, N P'" i p'" k d P~ i - ~, jli - N i ' an ij which means that every element of the population hgs an equal probability, kN 'of being included in a sample of k elements. Extension of this sampling scheme to a sample of m psu's could encounter the complications indicated in Section 2.5. However, it was \ 61 ... I sta~d that means exist for circumventing those complications. Sampling bOO~l/ discuss this mat~er quite fully so we will not include it in this monog~aph~ The point is that one can select m psu's without replacement \ Ni th in such a way that m ~ is the probability of including the i psu in Ni the sample. That is, Pi - 1U. N. -If a random sample of k elements is se1ect~d with equal probability from each of the selected psu's, and mk n - -a- N N ~us, if the ,Ni are known,l~xact1y for all M psu's'in the population, and if a list of .e1ements in each pS.u,is available, it is possible to select ,a two-stage sample of n e1eme t nts so that k elements for the sample come from each of m psu' a and every element of the population has an equal .. - j chance of being 'in the sample. In practice, however, one usually finds one of two situations: (a) there is no information on the number of e1e- ments in the psu's, or (b) the information that does exist is out-of-date. Nevertheless, out-of-date information on number of elements in the psu's can be very useful.' It is also possible that a measure of size might exist which will serve, more efficiently, the purposes of sampling. Yi be the value M where Y - l:Yi • Y i i th Pi - my-is the probability that the i pau has of being in the sample. A sample of m psu's is selected in such a way that is used as a measure of size. Y i Y c- (3) Suppose that characteristic Y th of Y for the i psu in the population and let P - i Let 1/ For example, Hansen, Hurwitz, and !1adow. Sample Survey Methods and Theory. Volume I, Chapter 8. John Wiley and Sons. 1953. With regard to the second stage of sampling. let f21 be the samplin~ th fraction for selecting a s~mple random sample within the i psu in the sample. That is. Pili - f2i• Then. Yi P1j .. (m y-> (f21> (2.5) In setting sampling specifications one would decide on a fixed valu' for Pij' In this context Pij is the overall sampling fraction or propor; tion of the population that is to be included in'the sample. For example. l if one wanted a 5 percent sample. p!j would be .05. Or. if one knew there ~. were approximately 50.000 elements in the population and wanted a sample;: of about 2.000. he would set 'P1j - .04. Hence. we will let f be the over- all sampling fraction and set p!j equal to f•. Decisions are also made on . the measure of size to be used and on the number'. m. of psu's to be selected. In Equation 2.5. this leaves f2i to be determined. Thus. ~2i is computeCt ': as follows for each psu in the sample: fY f --- 2i mYi Use of the samplin~ fractions f2i at the second stage of sampling will give every element of the population a probability equal to f of bein~ in the sample. A sample wherein every element of the population has an equal chance of inclusion is often called a self-weighted sample. . " The .theory of expec~ed values of random variables is used exten- \ CHAPTER III. INTRODUCTION EXPECTED VALUr.S OF RANDOM VARIABLES 63 I, ;if~\~t~:,:, ) '., SiVelj in the theory of sromplinr,;in fact, it is the foundation for sampling theory. Interpretations of the'accuracy of estimates from l I proba~ility samples depend heavily on the theory of expected values. I ~e definition of a random variable was discussed in the previous chapter. It is a variable that can take (be equal to) anyone of a defined set of values wi th known probability. Let Xi be the value of X for the ith element in a set of N elements and let Pi be the probability that th ' " of bein? selected by some chance operation the i element has so that Pi is known a priori. tfuat is the expected value of X? _D_e_f~n_i_t_i_~~~!. The expected value of a random v~riable X is N N r PiXi where r Pi-I. The mathematical notation for the expected value i=l i=l N of X is E(X). Hence, by definition, E(X) = E PiXi i=l Observe that EPiXi is a weighted avera~e of the values of X, the weights being the probabilities of selection. "Expected value" is a substitute expression for "avera~e value.1I In other words, E means "the average value of" or "find the average value of" whatever follows E. For 2 2 example, E(X ),read "the expected value of X ,"refers to the averar.e value of the squaresof the values that X can equal. That is, by definition, If all of the N elements have an equal chance of bein~ selected, all 1 values of Pi must equal N because of the requirement that rp i = 1. In Illustration 3.1. Assume 12 elements having values of X as follows: this case, E(X) - N 1 E N Xi i-I for all N elements. EX i -N - X , which is the simple average of X 64 Xl - 3 X - 5 X • 10 5 9 X2 • 9 X - 3 XIO - 3 6 X • 3 X7 - 4 XII - S 3 X4 -,5 Xs - 3 Xl2 - 4 F hi set,E(X) - 3+9+•••+4 5 umi h 1 h h or t s 12 -, ass ng eac e ement as t e same chance of selection. Or, by counting ,thenumber of times that each unique value of X occurs, a frequen~y distribution of X can be obtained , , as follows: ~ ~ 3 5 4 2 5 2 8 1 9 1 10 1 where X j is a unique value of X and N j is the number of times X j occurs. EX i - -- - X • N Suppose one of the X j values is selected at random with a probability equal • N", • N", to Pj whe re Pj --L- .....L EN j N 'What is the expected value of X j ? By N de~inition E(Xj) - EPjXj - Ej! Xj that in this illustration E(Xj) - 5. 65 The student may verify Note that the selection specifica- are very similar. The probability distribution with reference to Xj would be: l I" ~'!;'~~:~t ,1 '.: '~~2~' tions were equivalent to selecting one of the 12 elements at random with equal probability. Incidentally, a frequency distribution and a ~robability distribution ~ i 3 5/12 ,I 4 .. 2/12 1, ", ,~1 5 2/12 ~ 1/12 9 1/12 ,'..1 , 10 1/12 f'I't" ,.' , ~i i')'1 "!:'I :..•, . ':l 1 The 12 values, Pi - N ' for the 12 elements are also a probability distri- bution. This illustration shows two ways of treating the set of 12 elements. When finding expected values be sure that you understand the defini- tion of the set of values that the random variable might equal and the probabilities involved. Definition 3.2. \olhenX is a random variable, by definition the expected value of a function of X is N E[f(X)] - E Pi[f(Xi)] i-l Some examples of simple functions of X are: f(X) - aX, f(X) - X2, f(X) - a + bX + cX2, and f(X) - (X-X)2. For each value, Xi ' in a defined set there is a corresponding value of f(Xi). .,':',. •••If: •........-.':. It . 66 Illustr~~io~_3.2. Suppose f(X) = 2X+3. With reference to the set of 12 elements discussed above, there are 12 values of f(Xi) as follows: f(X1) - (2)(3) + 3 = 9 f(X2) = (2)(9) + 3 = 21 f(X12) = 2(4) + 3 = 11 1 Assuming Pi = - the expected value of f(X) = 2X+3 would be N ,. ; f i' ,. E(2X+3) (3.1) In algebraic terms, for f(X) - aX+b, we have ~ r • , N E(aX+b) - L Pi(aXi+b) = LPi(aXi~'+ ~Pib i-I By definition LPi(aXi) - E(aX), and LPib - E(b). Therefore, E(aX+b) - E(aX) + E(b) (3.2) Since b is constant and rpi,- 1, LPib - b, which leads to the first important theorem in expected values. Theorem 3.1. The expected value of a constant is equal to the constant: E(a) - a. another i1'!lPortanttheorem: Theorem 3.2. The expected value of a constant times a variable equals the constant times the expected value of the variable: E(aX) - aE(X). Applying these two theorems to Equation (3.2) we have E(aX+b) - aE(X) + b. Therefore, with reference to Illustration 3.2, E(2X+3) = 2E(X) + 3 - 2(5) + 3 = 13, which is the same as the result found in Equation (3.1). " 67 Exercise 3.1. Suppose a random variable X can take any of the following four values wi~h the.probabilities indicated: Xl = 2 X ••5 X = 4 X •••6 2 3 4 p = 2/6 P = 2/6 P •••1/6 P4 = 1/6 1 2 3· (a) Find E (X) Answer: 4 (b) Find E (X2) Answer: 1 Note that E(X2) ~ [E(X)]2 18). (c) Find E(X-X) Answer: 0 Note: By defini don 4 E(X-X)'" E P (X -X) i=l i i (d) .;., 2 1 Find E(X-X) Answer: 2). Note: : By definition 4 E(X-X)2 ••• EP (X _X)2 i-I i i , Exercise 3.2. From the following set of three values of Yi one value is to be selected with a probability Pi: Yl = -2 Y •••2 Y = 4 2 3 p" = 1/4 p" •••2/4 P 3' = 1/4 1 2 (a) Find E(Y) Answer: 11 2 (b) 1 3/16. 1 '"E~~) Find E(y) Answ'er : Note: E(Y) (c) Find E(y_y)2 Answer: 41 4 3.2 EXPECTED VALUE OF THE SUM OF TWO RAl'IDOMVARIABLES The sum of two or more random variables is also a random variable. If X and Yare two random variables, the expected value of X + Y is equal to the expected value of X plus the expected value of Y:E(X+Y) •••E(X)+E(Y). Two numerical illustrations will help clarify the situation. Illustration 3.3. Consider the two random variables X and Y in Exercises 3.1 and 3.2: 68 x a 2 PI 2 YI ••-2 P" I .. - =7; I 6 1 X ••5 P2 2 Y ••2 P" 2 .. - .. - 2 6. 2 2 4 X3 - 4 P3 1 Y ••4 P" 1 .. - .. - 6 3 3 4 X4 a 6 P4 1 .. - 6 Suppose one element of the first set and one element of the second the two selections are independent. lIenee by definition set are selected with probabilities as li5ted above. t\~at is the expected value of X + Y? The joint probability of getting.Xi and Yj is PiP; because , I~ I , ~I , 4 3 E(X + Y) a ~ r ~iPj" (Xi + Yj) i-I j-l The possible values of X + Y and the probability of each are as follows: X+Y PiP; X+Y PiL Xl + Y1 - 0 PIPi 2 X + Y - 2 P3Pi I a- .. - 24 .", 3 1 24 Xl + Y2 4 PIPi 4 X3 + Y2 = 6 P3Pi 2 - -- =- 24 24 Xl + Y3 ••6 PIP) 2 X 3 + Y 3 0: 8 P3P:3 I 0:- .. - 24 24 X2 + Yl - 3 P2Pi 2 X4 + Y1 ••4 Pli 1 .- =- 24 24 X2 + Y2 • 7 P2Pi 4 X4 + Y2 •• 8 P4Pi 2 .- -- 24 24 X2 + Y3 - 9 P2P:3••~4 X4 + Y 3 10 P4P:3 I = 10- 24 As a check the sum of the probabilities must be 1 if all possible sums have been listed and the probahility of each nas been correctly (3.3) determined. Substitutin~ the values of Xi + Yj and PiP; in Equation (3.3) we obtain 5.5 as follows for expected value of X + Y: 69 From Exercises 3.1 and 3.2 we have E(X) = 4 and E(Y) = 1.5. There- fore, E(X) + E(Y) = 4 + 1.5 = 5.5 which verifies the earlier statement that E(X + Y) - E(X) + E(Y). Illustration 3.4. Suppose a random sample of two is selected with replacement from the population of four elements used in Exercise 3.1. ., Let xl be the first value selected and let x2 be the second. Then xl and x2 are ;random variables and xl + x2 is a random variable. The possible values ~f xl +ix2 and the probability of each, P(xl ,x2).are listed below • Notice that each possible order of s~lection is treated separately. "," 'I f , !' :1 r! " Xl Xl 4/36 Xl X2 4/36 Xl X3 2/36 Xl X4 2/36 X2 Xl 4/36 4 7 6 8 7 Xl x2 P(xl,x2) +-- ! I X3 Xl 2/36 X3 X2 2/36 X3 X3 1/36 X3 X4 1/36 X4 Xl 2/36 6 9 8 10 8 By definition E(xl + x2) is 11 12 10 2/36 1/36 1/36 9 11 10 4/36 2/36 2/36 442 1 36(4) + 36(7) + 36(6) + ... +16(12) = 8 In Exercise 3.1 we found E(X) - 4. Since xl is ~he same random variable as X, E(x1) - 4. Also, x2 is the same random variable as X, and E(x2) - 4. Therefore, E(x1) + E(x2) - 8, which verifies that E(xl+x2) = E(X1) + E(x2). In general if X and Yare two random variables, where X might equal Xl' ••• '~ and Y might equal Yl, ••• ,YM, then E(X + Y) = E(X)+E(Y). The 70 NM proof is as follows: By definition E(X+Y) - LL Pij(Xi+Yj) where Pij is ij According to the probabi~ity of summation is over getting the sum Xi + Yj,and EEPij • 1. all possible values of Pij(Xi+Yj). The double '••..••• t • 1 ':t ••• ....•,' '~.,'..: ,"::~i~~;~i: 'l~~:':.;, . j', :.~ ••••! the rules for summation we may write (3.4) In the first term 'on the n:~ht, Xi is 'constant with regard to the summation " . I over j; and in the''second "term on the 'right, Y j is constant with regard I,' I j 11 t' I \. 'I II, ,I I ,1 t, I I' • I to the summation over 1. Therefore, 'the ri'ght-hand side of Equation (3.4) , ': can be written as "" M N And, since r Pij - Pi and E Pij - P , Equation (3.4) becomes j i j N M By definition r XiPi - E(X) and r YjPj - E(Y) • i j Therefore E(X+Y) - E(X) + E(Y) • If the proof is not clear write the values of Pij(Xi+Yj) in a matrix format. Then, follow the summation manipulations in the proof. The above result extends to any number of rand~m variables; that is, the expected value of a sum of random variables is the sum of the expected values of each. In fact, there is a very important theorem that applies to a linear combination of random variables. 71 Theorem 3.3. Let u - alul + •••+ ~~, where uI' ••• '~ are random variables and al'."'~ are constants. Then or in summation notation k k E(u) - E E aiui - E aiE(ui) i i The generality of Theorem 3.3 is impressive. For example, with refer- ence to sampling from a population Xl' ••• ' XN, ui might be the value of X obtained at the first draw, u2 the value obtained at the second draw, etc. -, ~. " ',$'. LJ J I :1 The constants could be weights. Thus, in this case~' 'uwould be a weighted average of the sample measurements. , - - Or, suppose xl,x2' ••• '~ are averages from a random sample for k different age groups. The averages are random variables and the theorem could be applied to1any linear combination of the random variable. "" , averages. In fact ui could be any fWlction of random variables. ~l\. }jlil;t- the only condition on which the theorem is based is that u~~~t ~' That is, be a Illustration 3.5. Suppose we want to find the expecte~ v~ue of 2·' (X + Y) where X and Yare random variables. Before Theorem 3.3 .can be applied we must square (X + Y). Thus E(X + y)2 _ E(X2 + 2XY + y2) • The application of Theorem 3.3 gives E(X + y)2 - E(X)2 + 2E(XY) + E(y)2. Illustration 3.6. We will now·show that E(X-X)(Y-Y) - E(XY) - XY where E(X) - X and E(Y) - Y Since (X-X) (Y-Y) - XY - XY - XY + XY we have E(X-X)(Y-Y) - E(XY-XY-xY+XY) and application of Theorem 3.3 gives E(X-X)(Y-Y) - E(XY) - E(XY) - E(YX) + E(XY) 72 Since X and Yare constant, E(XY) - X E(Y) - XY, E(YX) = YX, and E(XY) - XY. Therefore, E(X-X)(Y-V) = E(XY) - Xi Exerc~ 3.3. Suppose E(X) = 6 and E(Y) = 4. Find (a) E(2X+4Y) Answer: 28 (b) [E(2X)]2 Answe r : 144 (c) IEfi) Answer: 2 (d) E(5Y-X) Answer: 14 Exercise 3.4. Prove the following, assuming E(X) = X and E(Y) - Y: (a) E(X-X):= 0 (b) E(aX-bY) + cE(Y) = aX + (c-b)Y (c) E[a{X-X) + b(Y-Y)} :=0 (d) E(X+a)2:= E{X2) + 2aX + a2 (e) E(X-X)2. E{X2) _ X2 " ' I • (f) 'E{aX+bY) := 0 for any values of a and b if E{X) • ~ and E{Y) ,. O. 3.3 EXPECTED VALUE OF AN ESTIMATE Theorem 3.3 will now be used to find the expected value of the mean of a simple random sample of n elements selected without replacement from a population of N elements. The term "simple random sample" implies equal probability of selection without replacement. The sample aver~e is n th where Xi is the value of X for the i element in the sample. Without loss of generality, we can consider the subscript of x as corresponding th to the i draw; i.e., Xl is the value of X obtained on the first draw, x2 the value on the second, etc. As each Xi is a random variable, x is a linear combination of random variables. Therefore, Theorem 3.3 applies and