COM-503 Performance evaluation

Performance Evaluation is often the critical part in evaluating the results of a research project. Many of us are familiar with simulations, but it is often difficult to address questions like

  • I want to estimate a probability of error and I see no error in n experiments: what can I say about the error probability ?
  • I would like to characterize the fairness of my protocol. Should I use Jain's Fairness Index or the Lorenz Curve Gap ?
  • Should I eliminate the beginning of the simulation in order to wait until the system stabilizes ?
  • I would like to fit an explanatory model to my data, I was told to use least squares for that; is that the right thing to do ? Why ?
  • I simulate a random way point model but the average speed in my simulation is not as expected. What happened ?
  • The reviewers of my paper complained that I did not provide confidence intervals. What is that ? How do I get them ?
  • How do I analyze the power consumption of my system ?
  • I would like to fit a distribution to the flow sizes that I measured but all my measurements are truncated to a maximum value; how do I account for the truncation~?
  • What is Palm calculus and why should I care about it ?

These and other questions are the topic of this lecture.