This page describes the workload portion of Olio i.e. the load that is generated by the Olio driver against the Olio application.
Definitions
- Concurrent Users: The number of users emulated during a test run of the workload. The workload is scaled by increasing the number of Concurrent Users. We also use the term Active Users to represent the set of users actually performing work against the application.
- Users : The term Users is usually a short-hand for Concurrent Users or Active Users.
- Registered Users: This is the number of users known to the application i.e. the number of users in the database. The number of registered users is 100 times the number of Concurrent Users and reflects the fact that social networking applications have a large number of users but typically only a fraction of them are actively using the application.
- Operations: An operation is a request for a particular page made by the emulated user. Each operation may involve multiple HTTP request/response cycles depending on the embedded objects on a page. On average, there are 20 HTTP requests made per operation.
- Response Time: This is the time taken between the first byte of the first http request for an operation to the time the last byte of the last http request has been received. As such, this is a measure of the server response time and does not include any client-side processing.
- Cycle Time: This is the inter-arrival time between subsequent operations made by a concurrent user. To more realistically represent real workloads, each emulated user will wait for some time before submitting the next operation. In Olio, the mean cycle time is 5 seconds. This allows us to drive a larger emulated user load although to be truly realistic, the cycle time should be upto 30 seconds or larger. Using larger cycle times will require emulating 100's of thousands (or even millions) of users, so to provide a workload that is easier to run, we use 5 seconds.
- Operations Mix: This is the ratio of the various operations submitted by concurrent users. Olio uses a state-transition matrix mix to define the probability of transitioning from one operation to the next. See the Faban documentation for more details.
Olio Operations
There are 7 different Operations defined by Olio. A brief description of each is given below.
| Operation |
Description |
|---|---|
| HomePage |
Rretrieves the home page including all static content and thumbnails. |
| Login |
Log into the application as a particular registiered user, chosen at random between [1..REGISTERED_USERS]. Note that if this emulated user is already logged in as another user, an implicit logout is first performed. |
| TagSearch |
Search for events with a particular tag, limiting the search to an avg. of 125 events. The tag is chosen using a negative exponential distribution to use more commonly used tags. |
| EventDetail |
Rretrieve the EventDetail page for the specified event. The event is chosen at random from the list of events displayed on the home page. |
| PersonDetail |
Retrieves the PersonDetail page for the specified user. The user is chosen at random from the list of registered users. |
| AddPerson |
Register a new user. All of the user information is generated using random values. The user profile image is uploaded using the image in resources/person.jpg. Note that if this emulated user is logged in as another user, a logout is first performed. |
| AddEvent |
Add a new event. All of the event information is generated using random values. The event image and associated document are also uploaded, using files event.jpg and event.pdf in the resources directory. |
The AddPerson and AddEvent operations are relatively expensive in terms of cpu time consumed, since they involve image thumbnail generation.
Requirements and Metrics
Throughput
The primary performance metric is a throughput metric of operations/second representing the total number of operations performed during the steady-state interval divided by the number of seconds in the steady-state interval. The metric is valid only if all of the requirements for a run have passed as indicated by the summary report.
Response Times
There are requirements for 90% response time (the time in which 90% of all operations in staeady state should have completed) for all of the operations as specified below :
| Operation |
90% Response Time |
|---|---|
| HomePage |
1 second |
| Login |
1 second |
| TagSearch |
2 seconds |
| EventDetail |
2 seconds |
| PersonDetail |
2 seconds |
| AddPerson |
3 seconds |
| AddEvent |
4 seconds |
Operation Mix
A valid run needs to meet the following operation mix requirements. Note that the driver is designed to meet these requirements as long as there are a reasonable number of operations that were executed during the run.Note that since the AddPerson is a very small percentage of the mix, you will need to ensure that a sufficient number of operations are executed to meet this requirement. In practice, we have had no problem in achieving passing runs even with low concurrent users (50 or 100 users) as long as the steady state period is at 600-900 seconds.
| Operation |
%Total Mix |
|---|---|
| HomePage |
26.15% |
| Login |
10.22% |
| TagSearch |
33.45% |
| EventDetail |
24.68% |
| PersonDetail |
2.61% |
| AddPerson |
0.84% |
| AddEvent |
2.84% |
Cycle Times
Olio uses a negative exponential distribution of cycle times (see Cycle timings section in Faban documentation) to simulate a realistic web workload. The distribution uses a mean of 5 seconds, starting at 0 seconds and is truncated at 25 seconds for all of the operations except the AddPerson and AddEvent operations.
For the Add operations, a shifted negative exponential distribution is used - the mean is still 5 seconds, but the distribution uses a larger minimum ( 2 seconds for AddPerson and 3 seconds for AddEvent) and is truncated at 25 seconds.
The figure below shows the targeted cycle times, taken from a run with 300 concurrent users :
