Performance Evaluation - Computer Systems Performance Evaluation - Lecture Slides, Slides of Computer Science

Download Performance Evaluation - Computer Systems Performance Evaluation - Lecture Slides and more Slides Computer Science in PDF only on Docsity!

Projects Overview

Performance Evaluation

Your Projects This Semester

Performance Measurement Warm-ups

• A series of small tasks to get you in the mode for measuring and

reporting

• Follow closely the material in the text – “Network Performance”

The Main Performance

• A large project following your own interests.
• Contains a series of steps to ensure success.
• These steps include:

Project Pre-Proposal

Project Proposal

Design Proposal

Project Report

Warm-Ups

Does Correlation

Imply Causation?

Examples:

There's a strong correlation between foot size and ability to do arithmetic.

(Of course, it's not causation; both measured variables are strongly correlated with

the age of the children. Older ones have bigger feet and better math ability.)

People who drink diet soft drinks don't lose weight. In fact, they gain weight, a

new study shows. (Best Theory: When we offer our bodies the sweet taste of

diet drinks, but give them no calories, it makes us crave more calories.)

The more firemen fighting a fire, the bigger the fire is observed to be.

(Therefore firemen cause an increase in the size of a fire? )

As ice cream sales increase, the rate of drowning deaths increases sharply.

(Therefore, ice cream consumption causes drowning.)

Warm-Ups

Use of Performance Nomenclature:

It is expected that by the time you finish each of these warm-ups, you will have

made use of Performance concepts.

It is SOOO easy to talk only about throughput, for instance, and ignore all the

other nice concepts.

But I’m expecting that you will talk about:

Response Time

Utilization

Throughput

Capacity/Bandwidth

Environments and Tools

In doing these warmups, we’re exploring three different “levels” of environments. Warmup #1 focusses on very LOW level environments – where we look at individual actions

• for instance, individual network packets, individual disk accesses, individual process scheduling. We can study only ONE action – and get as much possible information as we can. Or we can look at multiple instances of this simple action, with a goal of getting more information about its performance.

Warmup #2 looks at the next “higher” level of environment. An aggregate action – a stream of data made up of many packets, disk accesses over a span of time, and so on.

Warmup #3 in intended to look at a complex environment. For instance, the disk activities that occur to a “user” environment, the network traffic coming from multiple sources, the car and truck traffic associated with an intersection, etc. These do not lend themselves to simple analysis, and we get only aggregate results.

Once we’ve identified the environment, then we can select the tool. Low level environments require tools that can measure individual actions – Wireshark to watch an individual network packet, timing of individual left turns at an intersection. Medium level experiments probably won’t measure individual actions, but require tools that can measure throughput, response time, queueing time for a series of actions. High level, complex environments will perhaps need tools that understand the environment. For instance, a database in the cloud will support transactions made up of a number of smaller actions – it may involve network, disk, and other resources.

Warm-Up

Use a software snoop device to measure network traffic on a machine.

What you need to know:

1. Chapters 1 – 2 in “Network Performance”
2. Class notes from the first lecture on measurement. In this project, you will install a sniffer on a machine of your choice and use it to measure low level network performance. A sniffer is a program running on a machine that is able to capture packets and packet information on that machine. In considering what machine to use, be aware that you will need generally administrative access to that hardware. Chapter 2 of “Network Performance” talks about tcpdump, windump, and Wireshark (formerly ethereal). I’m a Wireshark fan myself, but it doesn’t really matter which you use. Here’s your assignment: Measure some aspect of network traffic where you measure the same thing but change some one or two variables. Organize your results so they are easily understandable (presentation is everything!) including a plot or graph. Here are several possible examples – but I encourage you to think up your own:
3. While Wireshark is running, connect to a website with your browser. How long does it take to complete the handshaking required to establish the connection. Now vary the distance to the site you’re connecting to. Does this change the connection time? Does the connection time depend on other things – like time of day, number of hops, etc. ( Remember that correlation does not imply correlation! What is the real cause of a dependency you find?)
4. While Wireshark is running, ping a number of sites and determine the exact time it takes for that ping to complete (rather than the very general number reported by ping.) The ping time depends on what

• distance, hops, etc???

Mid-Level Tools

In Warm-Up #1, there was a low-level tool you used; Wireshark is an excellent example – it gives lots of very detailed information.

Warm-Up #2 will use tools at the next higher level. On Windows these might be the Task Manager (fairly simple statistics) or the Performance Monitor that gives you detailed information. Neither of these give packet level details, but they provide information about aggregate network behavior. On LINUX, there are not such nice tools that come with the OS (at least, not that I’ve found.) There are numerous tools out there that you can download. Netstat and traceroute come with LINUX, but the command line interface isn’t much – you can get free GUI interfaces for them. Also take a look at: http://www.uperf.org/ http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html and many others.

Note that there are two flavors of measurement here – and either is appropriate for your experiment:

1. The measuring device is independent of the generator of traffic; it’s essentially standing “outside” of the generator and gathers what’s happening.
2. The tool that generates the traffic also incorporates a means of measuring that traffic

The network generation tool is usually designed to max out the connection between A and B. It’s a measurement of capacity, and will find a bottleneck somewhere along the path. This is more of a network measuring tool (fine for this Warmup) than a system or application tool.

Warm-Up

Measure Network Performance - throughput

What you need to know:

1. Chapters 4 – 12 in “Network Performance”
2. Class notes from the first lecture on measurement. In this project, you will measure the performance of a network. This network might be a local LAN consisting of machines on the same switch, it could be a metropolitan network of machines near us, or it might span the globe – your choice. Here’s your assignment: Use several of the tools described on the previous slide to measure network traffic. As in Warm-Up #1, vary the properties of what you are measuring in order to get a variety of measurements, and again display and explain your results.

Here is one possible example – but I encourage you to think up your own:

1. Determine the throughput between your machine and an off-site location as a function of the time of day. When is the best and worst time to use your machine as a source across the WPI network? Are your numbers consistent from day to day? What matters in getting good throughput?

Your report should include the data analysis as you did in Warm-Up #1, (see Ground Rules) but in addition you should report on the relative advantages and disadvantages of the several tools you used..

Warm-Up

Measure Application Performance What you need to know:

1. Chapters 14 – 18 in “Network Performance”
2. Class notes on simulation. In this project, you will determine the performance of an application. You can do that in one of two ways – with a traffic generator/benchmark/workload (Chpt. 16) or with a simulator (the other chapters) Here’s your assignment: Use a tool of your choosing to emulate the behavior a system would have. This “system” is more complex than the simple traffic patterns that you would get from the network measurements. This tool can be one you build, it can be one you’ve previously used (but “enhanced” to emulate a whole system) or could be a new tool.

You are free to come up with an environment of interest to you. But as before, measurement and reporting are essential.

Please try to DESIGN YOUR EXPERIMENT : There are a series of steps in Section 12 of the “Measurement and Statistics” Slides. Use the steps from there that are applicable. That section asks questions that will help you break down your project into do-able steps. Too many of us do an experiment and then say “that’s what I was planning on measuring all along.” This design process is extremely important.

Your report should include the data analysis as you did in Warm-Up #2, (see Ground Rules) but in addition you should report on the relative advantages and disadvantages of the tool you used. Docsity.com

Warm-Up # Measuring an Application Environment

1. Does the Design and Measurement accomplish the assignment?

A. Shows a use of the 12-step design process. B. Uses a tool to measure traffic. C. Measures at an application level – a fairly complex environment. Show that the benchmark you are using is representative of a real environment.

2. Measurement Criteria :

A. Is the experiment conducted such that there’s only one independent variable at a time? And that one independent variable leads to a change in the dependent variable – i.e., the dependent variable is really dependent? B. Are there several measurements under the same conditions – do we have the ability to assess the uncertainty in the results obtained? Is there discussion of significant figures, standard deviation and variation so we know how much we trust the numbers? C. Is there an analysis of why the variations seen in the numbers might occur? Is there a sanity check that the information presented makes sense? D. Is there an analysis that the dependence is due to causation and not just correlation? E. Is there a physical model presented that explains why the observed behavior occurs?

3. The presentation:

A. The presentation is mailed to me by noon on the date it is due. B. Includes a discussion of the environment being studied. C. Is at the right level; Not too much or too little detail. D. There’s a plot or graph presented. The visual information represents what was measured.

E. There was an in-class delivery and material given to me for further evaluation.

F. Presentation Completed within the time limit.

Name______________________

The Main Performance

Choosing a Project.

Simulations (Continued):

a) You build the simulation model. You need to collect some data for this, but I’m much less concerned about the extend of that data – you’ve doing a lot of hard work building the model, so that makes up for the lesser time spent measuring.

EXAMPLE: On the following pages, I give two examples of models built by former students; one build a monopoly game – it had a mechanism for moving the pieces around the board, included buying and selling of properties, for Chance and Community Chest, etc. Another student built a network model including several routers and nodes and determined what happens as traffic increases in this environment.

The Main Performance

Choosing a Project.

Measurements:

People almost always measure something that’s computer or network related; it’s so much easier to determine the performance characteristics of such things. You get to use all the things you’ve learned this semester. Talking about the “performance of weather” or the “performanceof responses to a questionnaire” while theoretically possible, I suppose, are pretty hard to do in practice.

Usually in this case your project would be a bigger example of the kind of thing you did for your warm-ups. It would follow the twelve-step process, and would be of a bigger scope and in more detail than you did for those warm-ups.

Here’s a list of measurement tools that people have used in the past:

Tracing and Tools Profiling Project in Eclipse, Iometer, Perfmon, Tcpdump, Wireshark, Iperf Simgrid  simgrid.gforge.inria.fr, NetworkTrafficGenerator (ntg), Network Simulator (ns)

The Main Performance

Choosing a Project.

Projects From Last Year

Web Server Performance: What factors affect the performance of Apache Tomcat.

Home WLAN Performance: Studies factors affecting the performance of home networks.

Mesh Networks: Measured and analyzed the performance of networks of computers.

Hadoop: A study of the measured performance of a “big data” database.

Data Base Behavior: Performance factors affecting the measured performance of data base updates.

Mobile Hard Drive: Measured the performance of a hard drive under a large number of conditions.

The Main Performance

Project Pre-Proposal

Goal: To get agreement on a topic in an informal way. Deliverable: Here you and I have a conversation/e-mail/brief write-up that contains a very short statement of what you want to do for a project. I Owe You: Very rapid feedback that the direction you have chosen is OK. What To Look For: Major issues I will check for include:

• Scope/size - does the project you are proposing contain the appropriate amount of work?
• Diversity – does the project contain several components such as measurement, simulation, etc. from the topics we are discussing in class.

It is assumed that you will choose a topic an aspect of performance that is of

interest to you or that you can use in your job.

The project contains several components as shown below. The due dates for

these components are given on the main syllabus page.