Performance test on Jetty/HSQLDB with a single wiki between 8.4.6 and 11.8

Version 243.3 by Thomas Mortagne on 2019/09/30 16:34
Warning: For security reasons, the document is displayed in restricted mode as it is not the current version. There may be differences and errors due to this.

Environment

Failed to execute the [velocity] macro. Cause: [The execution of the [velocity] script macro is not allowed in [test:Performances.Jetty HSQLDB single wiki 846 to 1108]. Check the rights of its last author or the parameters if it's rendered from another script.]. Click on this message for details.

  • Ubuntu 19.04 64b
  • Quad core i7-8750H
  • Oracle JDK 1.8.0_222 64b
  • Dumbbench 0.111
  • Yourkit 2018.04-b87
  • standard jetty+hsqldb
  • permdir in a SSD
  • jetty/hsqldb package
  • only main wiki
  • configuration
    • Admin user:
      • hidden documents enabled
  • import perf.xar

Most of response time statistics are made with Dumbbench, you can install it using cpan.
For example on Debian/Ubuntu:

$ sudo apt-get install cpanminus
$ sudo cpanm Module::Install
$ sudo cpanm Dumbbench

Unless specifically indicated otherwise all the tests are executed with a fully initialized instance (core extensions don't run anymore, SOLR index is fully up to date, etc.) and logged in with Admin user. Basically except for a few special tests we are using an instance that already been started and run until the proc is flat for some time.

Login with Admin user:

wget --save-cookies cookies.txt --post-data 'j_username=Admin&j_password=admin&j_rememberme=true' -O /dev/null -o /dev/null "http://127.0.0.1:8080/xwiki/bin/loginsubmit/XWiki/XWikiLogin?j_username=Admin&j_password=admin&j_rememberme=true"

Tests

Summary

All the performance boosts are coming from the asynchronous rendering and caching of various elements of the skin. We need to analyze in details what is stored in the increased memory use but various new caches have been added so it's not suprising (notification preference cache, groups cache, cached skin elements). The degraded indexing speed will need to be analyzed too. Even if there are more documents to index in 10.10 (+3%), it's probably not enough to explain the x2 difference.

"similar": difference is lower than 10%
"slightly": difference is lower than 20%

Note that most of the speed related values are an average of very moving results, a lot of things is happening during a HTTP request and it's far from stable duration (that's why 10% may sounds a lot for something called "similar" but the variable can go up and down around 5% sometimes so 10% average is really not that much of a clear win). Dumbbench based tests are executed several times and the lowest result is selected.

Speed

ActionsDifference
Jetty startup 
First accessnot existing page without UI 
not existing page with UI 
Reloadnot existing page without UI 
not existing page with UI 
empty page without UI 
empty page with UI 
Main.WebHome without UI 
Main.WebHome with UI 
SOLRFull SOLR reindex 
SOLR sync when index is empty 
SOLR sync when there is nothing to do 
Result of search finding lots of results 
Result of search finding one result 
RenderingPage with 1000 macros without UI 
Page with 1000 html macros without UI 
Wiki creationFrom flavor 
From template 

Memory

ActionsDifference
Heap Memory after jetty startup 
Heap Memory after full SOLR index 

Jetty startup

Unit: milliseconds

You get this from the log with the line looking like 2016-10-19 16:52:25.309:INFO:oejs.Server:main: Started @8434ms.

  • 8.4.6:
  • 11.8: 9670

First access

First HTTP request after restarting Jetty.

First access to not existing page without UI

(/xwiki/bin/get/NoSpace/NoPage)

Unit: milliseconds

Firefox network statistics to load the root resource (image/js/css/etc. resources are not taken into account).

  • 8.4.6:
  • 11.8: 5838

First access to not existing page with UI

(/xwiki/bin/view/NoSpace/NoPage)

Unit: seconds

Done by hand with a chronometer (image/js/css/etc. resources are not taken into account).

  • 8.4.6:
  • 11.8: 10

Reload

XWiki is doing nothing on background and everything is both in server and client cache already. Values are calculated using dumbbench (image/js/css/etc. resources are not taken into account).

Reload of not existing page without UI

dumbbench --float -p 0.1 -i 100 -- wget -q -o /dev/null -O /dev/null --load-cookies cookies.txt "http://127.0.0.1:8080/xwiki/bin/get/NoSpace/NoPage"

Unit: milliseconds

  • 8.4.6:
  • 11.8: 4.2

Reload of not existing page with UI

dumbbench --float -p 0.1 -i 100 -- wget -q -o /dev/null -O /dev/null --load-cookies cookies.txt "http://127.0.0.1:8080/xwiki/bin/view/NoSpace/NoPage"

Unit: milliseconds

  • 8.4.6:
  • 11.8: 64.99

Reload of empty page without UI

dumbbench --float -p 0.1 -i 100 -- wget -q -o /dev/null -O /dev/null --load-cookies cookies.txt "http://127.0.0.1:8080/xwiki/bin/get/EmptySpace/EmptyPage"

Unit: milliseconds

  • 8.4.6:
  • 11.8: 3.7

Reload of empty page with UI

dumbbench --float -p 0.1 -i 100 -- wget -q -o /dev/null -O /dev/null --load-cookies cookies.txt "http://127.0.0.1:8080/xwiki/bin/view/EmptySpace/EmptyPage"

Unit: milliseconds

  • 8.4.6:
  • 11.8: 70.4

Reload of Main.WebHome without UI

dumbbench --float -p 0.1 -i 100 -- wget -q -o /dev/null -O /dev/null --load-cookies cookies.txt "http://127.0.0.1:8080/xwiki/bin/get/Main/WebHome"

Unit: milliseconds

  • 8.4.6:
  • 11.8: 6.4

Reload of Main.WebHome with UI

dumbbench --float -p 0.1 -i 100 -- wget -q -o /dev/null -O /dev/null --load-cookies cookies.txt "http://127.0.0.1:8080/xwiki/bin/view/Main/WebHome"

Unit: milliseconds

  • 8.4.6:
  • 11.8: 72.2

SOLR

Full SOLR reindex

Unit: seconds

Delete the data/solr folder, restart, load home page and calculate the time of activity of the SOLR index thread using Yourkit (Yourkit is slowing down a bit all that but what is important is the difference anyway).

  • 8.4.6:
  • 11.8:

SOLR sync

Only the thread determining what need to be indexed and not the actual indexing.

<perm dir>/data/jobs/status/solr/indexer/status.xml

when index is empty

Unit: nanoseconds

  • 8.4.6:
  • 11.8:

when there is nothing to do

Unit: nanoseconds

  • 8.4.6:
  • 11.8:

Search

XWiki is doing nothing on background and everything is both in server and client cache already. Values are calculated using dumbbench (image/js/css/etc. resources are not taken into account).

Unit: milliseconds

Result of search finding lots of results

Login with Admin user and make sure hidden document display is enabled.

dumbbench --float -p 0.1 -i 100 -- wget -q -o /dev/null -O /dev/null --load-cookies cookies.txt "http://127.0.0.1:8080/xwiki/bin/view/Main/Search?sort=score&sortOrder=desc&r=1&f_type=DOCUMENT&text=*"
  • 8.4.6:
  • 11.8: 148

Result of search finding one result

dumbbench --float -p 0.1 -i 100 -- wget -q -o /dev/null -O /dev/null --load-cookies cookies.txt "http://127.0.0.1:8080/xwiki/bin/view/Main/Search?sort=score&sortOrder=desc&r=1&f_type=DOCUMENT&text=name%3A1000Macros"
  • 8.4.6:
  • 11.8: 102

Rendering

Page with 1000 macros without UI

dumbbench --float -p 0.1 -i 100 -- wget -q -o /dev/null -O /dev/null --load-cookies cookies.txt "http://127.0.0.1:8080/xwiki/bin/get/Test/1000Macros"

Unit: milliseconds

  • 8.4.6:
  • 11.8: 57

Page with 1000 html macros without UI

dumbbench --float -p 0.1 -i 100 -- wget -q -o /dev/null -O /dev/null --load-cookies cookies.txt "http://127.0.0.1:8080/xwiki/bin/get/Test/1000HTMLMacros"

Unit: milliseconds

  • 8.4.6:
  • 11.8: 56.4

Wiki creation

Unit: nanoseconds

From flavor

Create the wiki test1 from default flavor and get the time spend from the job (</rootStep> in <permdir>/jobs/status/wikicreation/createandinstall/test1/status.xml).

  • 8.4.6:
  • 11.8: 16015857760

From template

Make test1 a template wiki.

Create the wiki test2 from wiki test1 and get the time spend from the job (</rootStep> in <permdir>/jobs/status/wikicreation/createandinstall/test2/status.xml).

  • 8.4.6:
  • 11.8: 41731479072

Heap Memory

Attach Yourkit to get the information.

Unit: MB

Heap Memory after jetty startup

Start jetty and wait until Java is not doing anything on both (wait for potential initialization of various stuff like core extension cache, etc). Force garbage collector before looking at "used".

  • 8.4.6:  used,  allocated
  • 11.8:  used,  allocated

Heap Memory after full SOLR index

Delete the data/solr folder, restart, load home page then wait until nothing is happening in background. Force garbage collector before looking at "used".

Unit: MB

  • 8.4.6:  used,  allocated
  • 11.8:  used,  allocated
Tags:
   

Get Connected