Yeast-templates benchmarking.

Experimental environment

In assessing Yeast-Templates we compared four different implementations of two different web pages. These implementations are

  • JSP, implementing the pages in the traditional way using JSPs
  • Yeast, using Yeast-templates, together with Yeast-Server API, to develop the pages
  • Cached Yeast, similar to the previous, but in this case the templates are cacheable Yeast templates. In these, the original template can be replaced by a much more compact document containing the engine, the view’s model data and, as body, a single script tag that loads a cacheable file with the original template body. Therefore, a template like these has two parts: a skeleton and a cacheable body.
  • Accessible Yeast, Yeast templates processed in the server using the Yeast-Server accessibility facilities.

Note that initially, the cached case assumes a cache hit of 100%, equivalent to download only the skeleton of the template.

In our study we considered two different applications, with different load features:

  1. A dynamic web page simulating the timeline of a Twitter user. The page size varies depending on a parameter which represents the number of tweets in the page. We study the cases of 10, 40 and 80 tweets. This case models Web pages that repeat many times the same HTML structure, need a lot of processing time, and have a large resulting size.
  2. A page simulating the account-info page of a Twitter user. In this case the page has little dynamic data, is processed quickly and the resulting size is very similar in both the JSP and Yeast implementations.

The source code of the application used in the tests is available here.

The deployable application used in the tests is available here.

The running application is available here.

The server throughput evaluation was conducted using five identical Intel(R) Core(TM)2 Duo CPU E8400 3 GHz computers , with 2 Gb. RAM and OS Windows 7 Professional, connected through a 100 Mb. LAN. To measure the server throughput we use Apache JMeter 2.4, configuring three slaves, running in StrippedBatch mode, and one master to record the results. Each slave invoked 20 simultaneous requests (each request in a different Java thread), making a total of 60 simultaneous requests. Each thread is repeats 60 times, so the total of request was 3600. We used Apache Tomcat 6.0.26 as the server.  All Yeast based implementations use our Yeast-Server API.  Results are expected to be similar for other Browser Side Templating  (BST) systems, since the burden of the request process is on the transformation from the application model to the view’s model, and this is similar for every BST.

The test plans for JMeter are available here. To reproduce the experiments, first you have to disable the Yeast-Server accessible mode. Then run the test plans for the JSP, Yeast and Cached-Yeast cases. When finished, enable the Yeast-Server accessible mode and run the Accessible Yeast case. To enable/disable the Yeast-Server accessible mode, look for the yst.properties file in the WEB-INF/classes directory of the deployed application. Change the value of the property manager.accessibility.support (on - off).

Finally we looked at the overall latency for a browser rendering the pages under study, assuming that this time is the total time elapsed from the request to just after the browser displays the rendered page to the user. We compared the Yeast (YST) and JSP cases (cached Yeast performs very similar to Yeast in terms of server and client-side processing times). We run the experiment using the last versions of the more popular browsers (Ms. Internet Explorer 8.0.7600.16285, Firefox 4.0, Opera 11.01 (Compilation 1190)  and Google Chrome 10.0.648.204). The used computer was a Intel(R) Core(TM)2 Duo CPU T6570 2.1 GHz, 4Gb RAM, 32 bits, Windows 7 Home Premium. To determine the total latency of the client we considered the partial sum of three times:

  1. The “Server-response” time was measured with JMeter, which considers it as the “elapsed time from just before sending the request to just after the last response has been received” (see http://jakarta.apache.org/jmeter/usermanual/glossary.html).  
  2. The “DOM” variable represents the time to load the DOM of a page. This time was assessed storing the page in the local file system, loading it in the browser, and measuring the time until the onLoad event was fired. We repeated the experiment 20 times for each pages and browser, and computed the mean time.
  3. The “Template” variable records the time spent processing the Yeast code. Again the template was loaded from the file system 20 times, computing the mean time.

To measure the client-side times we followed the methodology described in PPK’s benchmarking methodology.

The time-line application

Transferred data average size

Table 1 shows that the transferred data are greatly reduced when Yeast is used, which is easily explained considering that the HTML block for each tweet is repeated in JSP, whereas it appears only once in Yeast. This is a particular feature of this application, which likewise influences the server throughput and the client latency, both evaluated later.

Table 1 Transferred data average size (bytes)
# tweets JSP Yeast Cached Yeast
10 37.371 10.383 3.476
40 128.308 17.352 10.435
80 250.100 26.731 19.790

The size of the Yeast-template engine is 45.507 bytes (but is downloaded only once for all possible templates), and the size of the cacheable template body is 6.160 bytes (that is downloaded once per each template).

Server throughput


Figure 1 Server throughput

As Figure 1 illustrates, the use of Yeast templates improves the throughput with respect to the JSP implementation at least 98%, even 138% if Yeast is combined with browser-side cache techniques. It is necessary to take into account the large size of the returned page to justify the poor performance of the JSP implementation when the number of tweets increases.

Looking at the accessible Yeast, the throughput decreases drastically when Yeast templates are processed in the server, which is explained simply due to the large amount of processing that requires the template used in the test, especially in the case of 80 tweets.

The response time, representing the elapsed time from just before sending the request to just after the last response byte has been received, is shown in the following chart (we exclude the Accessible Yeast case due to its great response time):

Figure 2 Response time in ms.

The server latency, representing the time elapsed from just before sending the request to just after the first response has been received is shown in the next figure. We see that the JSP case latency is lower than the others. This is due to the way Yeast-Server API processes the request, transforming in first place it the application' model into the view's model, and then writing the HTML to the client. JSP immediately start writing HTML code, thereby reducing latency.

Figure 3 Server latency (ms)

 

It is interesting to assess the effect that the cache has in the server throughput. In the previous tests we were assuming that the Yeast template engine was cached in the browser. This is a valid assumption for most of the requests, because the engine, which is shared by all templates, is cached the first time you get a template, and does not disappear unless the user cleans the browser cache. The situation is different if we consider the effect of caching the bodies of the cacheable templates. The maximum effect is obtained when the template is used a lot, but for little-used templates the browser caching technique can be counterproductive, because it's necessary to download separately the body and the skeleton of the template. To assess the effect of template-body cache hit ratios we have conducted an experiment to study the throughput in three situations: (1) assuming 100% cache hit ratio of the body of the template (equivalent to download only the skeleton of the template), (2) 50% hit ratio (modeling the situation in which half of the client requests download the template, skeleton and cacheable body, for the first time and the remaining half are subsequent downloads) and (3) 0% hit ratio (all of the client requests download the template for the first time). The results are shown in Figure 2, which includes as reference the previously obtained throughput of the JSP implementation.


Figure 2 Throughput for various cache hit ratios

As expected, the throughput decreases for lower cache hit ratios, but in any case it is greater than the obtained with the JSP implementation.

Overall client latency

Figure 3 summarizes the overall client latency in different browser, showing that the DOM and Yeast times dominate the latency, whereas the server response time is negligible 4-6 ms for Yeast, and 8-37 ms for JSP.
The chart illustrates that in certain browsers (Opera, Chrome) and for large pages, Yeast performs better than JSP. Only in one case (IE8, 80 tweets) Yeast is significantly slower than JSP, whereas in the rest of the cases, the difference is not perceptible to users.  Therefore, we can conclude that the use of Yeast does not affect the client’s experience.


Figure 3 Overall client latency (ms)

The account-info application

Transferred data average size

Table 2 shows that, in this case the size of the transferred of the Yeast template is slightly greater than the JSP generated page. This is due to the Yeast boilerplate. Again, data size is greatly reduced when Yeast Cached is used.

Table 2 Transferred data average size (bytes)
JSP Yeast Cached Yeast
30.363 31.068 3.204

The size of the Yeast-template engine is 45.507 bytes (but is downloaded only once for all possible templates), and the size of the cacheable template body is 26.867 bytes (that is downloaded once per each template).

Server throughput

In this case instead of varying the size of the page, we play with the number of concurrent user that requests it. Qe have assessed three configurations with 60, 90 and 120 concurrent user respectively.


Figure 4 Server throughput

As Figure 4 illustrates, in this case the use of Yeast does not improve the throughput with respect to the JSP implementation. It is even a little worse (2% - 8%). But, again, if we combine Yeast with browser-side caching techniques, the throughput improvement is over 135%.

Looking at the accessible Yeast, the throughput does not decrease so drastically as in the previous time-line application. This is due to the template requiring a very little processing.

The server response time and latency are the following, confirming that the JSP implementation is quicker than Yeast but slower than Cached Yeast:

Figure 7 Response time in ms.

Figure 8 Server latency (ms)

If we assess the effect that the cache has in the server throughput, we see the 0% body-cache hit ratio performs worse than the JSP implementation. This means that for seldom used templates with little BST code, the use of Yeast templates is counterproductive.


Figure 5 Throughput for various cache hit ratios

Overall client latency

Figure 3 summarizes the overall client latency in different browser, showing that the DOM time dominate the latency, whereas the server response time is a negligible 7-8 ms for all the cases. The results shows that in this case the little Yeast processing that the template requires does not affect the overall latency (Yeast takes a minimum of 24 ms in Firefox, and 69 ms in IE 8). Again, we can conclude that the use of Yeast does not affect the client’s experience.


Figure 6 Overall client latency (ms)