Download Measurement and Statistics - Computer Systems Performance Evaluation - Lecture Slides and more Slides Computer Science in PDF only on Docsity!

Performance

Engineering

1

MEASUREMENT AND STATISTICS

In order to get you in the mood for doing some measuring, statistics, and estimating, here are

some quotations with the right flavour:

"Figures don't lie, but liars figure." Mark Twain

"There are three kinds of untruths; lies, damn lies, and statistics." Mark Twain

The following are from "Policy Paradox and Political Reason" by Deborah Stone.

"Numerals hide all the difficult choices that go into a measurement."

"Certain kinds of numbers, big ones, numbers with decimal points, ones not multiples of ten, seemingly advertise the prowess of the measurer."

"How accurate a number is depends on the cost of acquiring it and on how important it is."

2

Purpose:

This section is about the methodology of measurement. What goes into designing an experiment, gathering some numbers, interpreting the results, and presenting those results to management in a way that allows them to make the necessary decisions.

Warm-up Experiment:

Divide into teams and measure the length of an object in the classroom. To do so you will need to make team decisions about tools, techniques, and reporting metrics.

Upon completion, discuss what can be learned from this experiment.

4

FUNDAMENTAL QUESTIONS ABOUT MEASUREMENT:

What kind of accuracy can you expect from a computer (or any other) measurement?

When you make a measurement, can you believe the result? How sure are you of the

result?

How should you state the result of an experiment? How do you reflect your belief in its

accuracy?

Can one number represent the performance of a product?

When have you measured enough?

Figures don't lie, but liars figure. How do you extrapolate from what you know to what

you'd like to know?

5

1. What kind of accuracy can you expect from a computer (or any other) measurement?

Associated Questions are:

• What are some sources of uncertainty when measuring a computer and its software.
• Is a computer deterministic? (What is the meaning of deterministic? Do a detour on predictable, deterministic, stochastic and chaotic.)
• What are the pros and cons of taking all the variation out of an environment. Repeatability vs. believability.

Here are some factors that lead to experimental variation:

• System/Component/Molecule/Atom – how granular is the measurement.
• Background Activity
• End effects and incomplete cycle effects. Measurement error.
• Randomness doesn't mean equality (stochastic process).

Example: Travelling around a monopoly board.

• Randomness from resource contention ( stochastic process ).

Example: Six processes do nothing but read randomly from a single disk. Do they each make approximately the same number of accesses after 1 second? 1 minute? 1 hour? 1 day?

7

1. What kind of accuracy can you expect from a computer (or any other) measurement?

Here are some factors that lead to experimental variation (continued):

• Changing hardware.

Example: Variations in fullness of a disk, CPU boards, interrupt traffic.

• Tool granularity

Example: Our experiment in class.

Example: You write a program that measures time in seconds. What percentage accuracy can you get from your experiment.

Example: You want to measure the time required to execute a routine and have available a system call named get_time_of_day. get_time_of_day returns time in units of 1/65535 seconds = 16 microseconds. The time required to execute the get_time_of_day routine itself is 100 microseconds. What is the shortest routine that can be measured with this tool? How would you do it?

Bottom Line: Never believe a real system number to better than 5 - 10%. Artificial numbers can sometimes be repeated to 1 - 2%, but are susceptible to spurious factors.

8

2. When you make a measurement, can you believe the result? How sure are

you of the result??

Then the mean and standard deviation are:

10

[ ] ( )∑

= =

n

i i

M E m n m 1

1 /

( ) { 1 }

( )

[ ] [ ]

2 2 0.^5

−

−

= − =

∑

N

M m

SD E m E m

i

i

s = σ^2 = variance = SD 2

The first form of the Standard Deviation is the form of the underlying data. The second form is that of the measured data. They are the same for an infinite amount of data and close enough for a large set of numbers.

NOTE: Use of these equations assumes that the measurements are independent of each other.

Confidence Intervals:

We'd like to say  “I'm p% sure that with n samples the actual value is within d of the mean of the measurements.” In this section, we develop simple ways to be able to make that statement.

Example of Standard Deviations using Normal Distributions:

By quoting the standard deviation of a measurement, we say we're 68% sure the true mean is within a standard deviation of the measured mean. Unfortunately, that 68% depends on having a large number of samples. For smaller numbers, the percentage will change.

11

2. When you make a measurement, can you believe the result? How

sure are you of the result??

Normal distribution showing mean and variance.

The Burns Co. is now making laptop computers in its Shelbyville plant. Mr. Burns is too cheap to wreck

too many computers in a test, so he's letting his QA guru, Homer, smash five of them. Homer is

to record from how high in the air he can drop each laptop on the floor before it won't work

anymore.

Mr. Burns' wants laptops that can survive a fall from his height of five feet, two The t-test will tell us if

we can accept that the average breaking point for a Burns laptop is greater than 5'2", given what

we know about the sample.

Let's say the five computers broke at drops of:

• 4 feet, 8 inches
• 5 feet, 1 inch
• 2 feet, 3 inches
• 6 feet, 10 inches
• 7 feet, 1 inch

13

2. When you make a measurement, can you believe the result? How

sure are you of the result??

Using the formula:

(avg. of sample) - (presumed avg. of larger pop.)

t = --------------------------------------------------

(st. dev. of sample) / (sq. root of sample size)

• we get an average breaking height of 62.2 inches, St Dev of 23.4, and a t-score of 0.0191.
• Let's go to the t-score table. There we find the t-value for four degrees of freedom and a 90-

percent confidence interval (that's p=.05, since taking .05 off each side of the bell curve leaves

us with .90 in the middle). That value is 2.13.

• Since the value we calculated is less than the table's t-value, that means we cannot accept the

assumption that all Burns laptops together have an average breaking drop of over 62 inches.

Even though our sample's average came in (just) over that.

14

2. When you make a measurement, can you believe the result? How

sure are you of the result??

2. When you make a measurement, can you believe the result? How sure are you of the

result??

Example – Use of Student-T:

Let's work through in detail the numbers in "f". We find the

mean = (14.40 + 14.29 + 14.43 + 14.36 + 14.44 )/5 = 14.

SD = SQRT( (.02 + .09 + .05 + .02 +.06 )/4 ) = SQRT( 0.00375 ) = 0.

s = variance = SD 2 = 0.

Suppose we want to find the confidence interval for 95% confidence. With 5 variables, we have n = 4 degrees of freedom. Read the table for t(0.975) ( there's 2.5% UNconfidence on each side of the curve ) giving 2.78.

d = t * SQRT( s / n ) = 2.78 * SQRT( 0.00375 / 5 ) = 2.78 * 0.027 = 0.

The number is 14.38 +- 0.075 with 95% confidence. (How should you round off this number to accurately reflect your confidence?)

16

Example of Normal Distribution:

Suppose we ’ ve been making measurements as shown in the first column in the Table below. By inserting those numbers in Excel, the spreadsheet will calculate all kinds of things for us automatically.

17

Measurements (Sorted) Mean^

Standard Deviation <-- From Excel'sFunctions

1.9 3.90 0.

2.8 1.9 <-- From Tools-> 2.8 Data_Analysis-> 2.8 Mean 3.961290323 Descriptive Statistics 2.9 Standard Error 0. 3.1 Median 3.9 (Note: Excel has 3.1 Mode 2.8 eliminated the 3.2 Standard Deviation 0.889448301 outlying value.) 3.2 Sample Variance 0. 3.3 Kurtosis -0. 3.4 Skewness 0. 3.6 Range 3. 3.7 Minimum 2. 3.8 Maximum 5. 3.9 95% Confidence 0.

Etc. etcetera

2. When you make a measurement, can you believe the result? How sure are you of the result?

COMPARING TWO SETS OF MEASUREMENTS:

In essence this is a way to combine the confidences for the two data sets so as to determine the confidence in the difference between the two sets. This is called a t-test.

Excel can do a t-test as shown in the data below:

19

Data Set Data Set 1 2 5.36 19. 16.57 3. 0.62 3. 1.41 2. 0.64 3. 7.26 1.

5.31 5.64 <-- Average =AVERAGE(A3:A8) 6.16 6.64 <-- Standard Deviation =STDEV(A3:A8) 0.465703 <-- Result of the t-test says there is a 46% chance these are from the same distribution =TTEST(A3:A8,B3:B8,1,1)

So for these sets of data, the answer is inconclusive. We can’t tell if there’s a significant difference between the data sets.Docsity.com

2. When you make a measurement, can you believe the result? How sure

are you of the result?

CHECKING A SERIES OF VALUES:

We'd like to know if a series of values matches a predicted distribution. In other words, we have a theory of what an experiment should give - do the results in fact match the theory? Chi-Squared tables are available for this purpose.

Calculate Chi - Squared

20

n

n n E

O E

2 2 (^ − ) χ = (^) ∑

where O = Observed and E = Expected.