






































Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Community
Ask the community for help and clear up your study doubts
Discover the best universities in your country according to Docsity users
Free resources
Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors
Some concept of Computer Systems Performance Evaluation are Measurement and Statistics, Performance Evaluation, Performance Metrics, Queueing Lingo, Software Performance Engineering. Main points of this lecture are: Measurement and Statistics, Mark Twain, Damn Lies, Deborah Stone, Choices, Decimal Points, Multiples, Seemingly Advertise, Prowess, Numerals Hide
Typology: Slides
1 / 46
This page cannot be seen from the preview
Don't miss anything!
1
"Figures don't lie, but liars figure." Mark Twain
"There are three kinds of untruths; lies, damn lies, and statistics." Mark Twain
The following are from "Policy Paradox and Political Reason" by Deborah Stone.
"Numerals hide all the difficult choices that go into a measurement."
"Certain kinds of numbers, big ones, numbers with decimal points, ones not multiples of ten, seemingly advertise the prowess of the measurer."
"How accurate a number is depends on the cost of acquiring it and on how important it is."
2
Purpose:
This section is about the methodology of measurement. What goes into designing an experiment, gathering some numbers, interpreting the results, and presenting those results to management in a way that allows them to make the necessary decisions.
Warm-up Experiment:
Divide into teams and measure the length of an object in the classroom. To do so you will need to make team decisions about tools, techniques, and reporting metrics.
Upon completion, discuss what can be learned from this experiment.
4
FUNDAMENTAL QUESTIONS ABOUT MEASUREMENT:
What kind of accuracy can you expect from a computer (or any other) measurement?
When you make a measurement, can you believe the result? How sure are you of the
result?
How should you state the result of an experiment? How do you reflect your belief in its
accuracy?
Can one number represent the performance of a product?
When have you measured enough?
Figures don't lie, but liars figure. How do you extrapolate from what you know to what
you'd like to know?
5
1. What kind of accuracy can you expect from a computer (or any other) measurement?
Associated Questions are:
Here are some factors that lead to experimental variation:
Example: Travelling around a monopoly board.
Example: Six processes do nothing but read randomly from a single disk. Do they each make approximately the same number of accesses after 1 second? 1 minute? 1 hour? 1 day?
7
1. What kind of accuracy can you expect from a computer (or any other) measurement?
Here are some factors that lead to experimental variation (continued):
Example: Variations in fullness of a disk, CPU boards, interrupt traffic.
Example: Our experiment in class.
Example: You write a program that measures time in seconds. What percentage accuracy can you get from your experiment.
Example: You want to measure the time required to execute a routine and have available a system call named get_time_of_day. get_time_of_day returns time in units of 1/65535 seconds = 16 microseconds. The time required to execute the get_time_of_day routine itself is 100 microseconds. What is the shortest routine that can be measured with this tool? How would you do it?
Bottom Line: Never believe a real system number to better than 5 - 10%. Artificial numbers can sometimes be repeated to 1 - 2%, but are susceptible to spurious factors.
8
2. When you make a measurement, can you believe the result? How sure are
you of the result??
10
= =
n
i i
M E m n m 1
1 /
( ) { 1 }
( )
[ ] [ ]
−
−
= − =
∑
N
M m
SD E m E m
i
i
The first form of the Standard Deviation is the form of the underlying data. The second form is that of the measured data. They are the same for an infinite amount of data and close enough for a large set of numbers.
NOTE: Use of these equations assumes that the measurements are independent of each other.
Confidence Intervals:
We'd like to say “I'm p% sure that with n samples the actual value is within d of the mean of the measurements.” In this section, we develop simple ways to be able to make that statement.
Example of Standard Deviations using Normal Distributions:
By quoting the standard deviation of a measurement, we say we're 68% sure the true mean is within a standard deviation of the measured mean. Unfortunately, that 68% depends on having a large number of samples. For smaller numbers, the percentage will change.
11
2. When you make a measurement, can you believe the result? How
sure are you of the result??
Normal distribution showing mean and variance.
13
2. When you make a measurement, can you believe the result? How
sure are you of the result??
14
2. When you make a measurement, can you believe the result? How
sure are you of the result??
2. When you make a measurement, can you believe the result? How sure are you of the
result??
Example – Use of Student-T:
Let's work through in detail the numbers in "f". We find the
mean = (14.40 + 14.29 + 14.43 + 14.36 + 14.44 )/5 = 14.
s = variance = SD 2 = 0.
Suppose we want to find the confidence interval for 95% confidence. With 5 variables, we have n = 4 degrees of freedom. Read the table for t(0.975) ( there's 2.5% UNconfidence on each side of the curve ) giving 2.78.
d = t * SQRT( s / n ) = 2.78 * SQRT( 0.00375 / 5 ) = 2.78 * 0.027 = 0.
The number is 14.38 +- 0.075 with 95% confidence. (How should you round off this number to accurately reflect your confidence?)
16
Example of Normal Distribution:
Suppose we ’ ve been making measurements as shown in the first column in the Table below. By inserting those numbers in Excel, the spreadsheet will calculate all kinds of things for us automatically.
17
Measurements (Sorted) Mean^
Standard Deviation <-- From Excel'sFunctions
1.9 3.90 0.
2.8 1.9 <-- From Tools-> 2.8 Data_Analysis-> 2.8 Mean 3.961290323 Descriptive Statistics 2.9 Standard Error 0. 3.1 Median 3.9 (Note: Excel has 3.1 Mode 2.8 eliminated the 3.2 Standard Deviation 0.889448301 outlying value.) 3.2 Sample Variance 0. 3.3 Kurtosis -0. 3.4 Skewness 0. 3.6 Range 3. 3.7 Minimum 2. 3.8 Maximum 5. 3.9 95% Confidence 0.
Etc. etcetera
2. When you make a measurement, can you believe the result? How sure are you of the result?
In essence this is a way to combine the confidences for the two data sets so as to determine the confidence in the difference between the two sets. This is called a t-test.
Excel can do a t-test as shown in the data below:
19
Data Set Data Set 1 2 5.36 19. 16.57 3. 0.62 3. 1.41 2. 0.64 3. 7.26 1.
5.31 5.64 <-- Average =AVERAGE(A3:A8) 6.16 6.64 <-- Standard Deviation =STDEV(A3:A8) 0.465703 <-- Result of the t-test says there is a 46% chance these are from the same distribution =TTEST(A3:A8,B3:B8,1,1)
So for these sets of data, the answer is inconclusive. We can’t tell if there’s a significant difference between the data sets.Docsity.com
2. When you make a measurement, can you believe the result? How sure
are you of the result?
We'd like to know if a series of values matches a predicted distribution. In other words, we have a theory of what an experiment should give - do the results in fact match the theory? Chi-Squared tables are available for this purpose.
Calculate Chi - Squared
20
n
n n E
O E
2 2 (^ − ) χ = (^) ∑
where O = Observed and E = Expected.