andi/perfdb
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Activity
199 Commits 2 Branches 0 Tags
2cd32805d98a3336e010fb3c7e159655ff2070de
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
andi 2cd32805d9 show better error message when there are too many parallel requests 2018-03-25 18:03:20 +02:00
andi 540b6c79f4 restart waiting game 2018-03-25 17:59:51 +02:00
andi c581e352e4 add method that returns a string representation of the tags in Tags 2018-03-19 19:29:22 +01:00
andi 5343c0d427 reduce memory usage 
Reduce memory usage by storing the filename as string instead of
individual tags.
 2018-03-19 19:21:57 +01:00
andi b4a0514267 draw points and percentile lines with the same color 2018-01-21 14:09:34 +01:00
andi bb7701e7c4 use enum for line type instead of string 2018-01-21 11:01:30 +01:00
andi 8f15aba0d5 replace individual percentile aggregates with a single one for all 2018-01-21 10:54:13 +01:00
andi b439c9d79a update third-party libs 
antlr4: 4.7 -> 4.7.1
commons-lang3: 3.6 -> 3.7
 2018-01-21 08:44:30 +01:00
andi 469dc20411 add gradle plugin: gradle-versions-plugin 2018-01-21 08:43:45 +01:00
andi 9f45eb24ca add trace logging for creation of new writer 2018-01-21 08:36:40 +01:00
andi a6a2236d18 do not compute counts when proposing all keys 2017-11-18 13:03:45 +01:00
andi 14d1367c4a remove duplicate enums 2017-11-18 12:30:45 +01:00
andi f2868fcc1b reduce memory footprint: old generation by 100 MB 
This reduces the size of the old generation by 100MB (300MB down to
200MB). Unfortunately the total JVM size didn't change and is still
512MB.

Doc stores the path as byte array instead of Path.
 2017-11-18 10:39:01 +01:00
andi cc49a8cf2a open PdbReaders only when reading 
We used to open all PdbReaders in a search result and then interate over
them. This used a lot of heap space (> 8GB) for 400k files.
Now the PdbReaders are only opened while they are used. Heap usage was
less than 550 while reading more than 400k files.
 2017-11-18 10:12:22 +01:00
andi a636f2b9bd update primitive collections to 0.1.20171007100354 2017-11-18 10:09:47 +01:00
andi 0555691864 update gradle to 4.3.2 and spring boot to 1.5.8 2017-11-18 09:32:49 +01:00
andi 995558588a add median and 90% percentile 2017-11-18 09:28:41 +01:00
andi 11b3610971 make invaders better 
add kill count
do not move all invaders at once
 2017-10-01 19:08:59 +02:00
andi 08f1961f51 replace spinner with a little game 2017-10-01 17:23:59 +02:00
andi 386f211377 make it possible to draw the legend outside of the plot area 2017-09-30 17:51:33 +02:00
andi d4fd25dc4c replace LinkedHashMap with a more memory efficient implementation 
This saves approximately 50MB of heap space.
 2017-09-30 17:51:02 +02:00
andi 7e00594382 add helper class that returns the size of objects 2017-09-30 17:49:21 +02:00
andi e0655f66fa skip invalid entries 2017-09-24 17:21:20 +02:00
andi a7cd918fc6 skip empty files 2017-09-24 17:12:17 +02:00
andi dc8262c37e replace deprecated API 2017-09-24 17:00:08 +02:00
andi 4b53baacae add scrollbar for proposals, again 2017-09-24 13:32:51 +02:00
andi 35500e387a add vertical marker lines that show which area will be used after 
zooming
 2017-09-24 12:52:44 +02:00
andi 8b74992e66 reduce memory allocations by 25% with a cache of formatted longs 2017-09-24 09:53:25 +02:00
andi 37eb05aebf use unix timestamps in the CSVs used by gnuplot 
This is faster by factor 4, because we don't have to format the date.
 2017-09-24 09:38:08 +02:00
andi 955127dc4a use buffered writer 
despite my previous test, writing the csv is a little bit faster with
a buffered writer (~ 10-15%)
 2017-09-23 20:22:15 +02:00
andi 9d66c9e0da zoom in/out 2017-09-23 20:20:56 +02:00
andi 70e586b7e9 limit by max/min value 2017-09-23 18:56:02 +02:00
andi 347f1fdc74 update 3rd-party libraries 2017-09-23 18:24:51 +02:00
andi c9ff8b5586 only propose value if the existing prefix is a real prefix 2017-09-23 13:31:34 +02:00
andi 4360944683 remove keyhandler for enter key again 
It didn't work with autocomplete.js when selecting a proposal.
 2017-09-23 13:27:53 +02:00
andi d797a3c9ec send form when pressing enter 2017-09-23 12:56:32 +02:00
andi 79d860cee7 make computation of mean value optional 2017-09-23 12:42:22 +02:00
andi adbde57d95 add hours as valid time range 2017-09-23 10:55:40 +02:00
andi 38873300c8 print last inserted entry during ingestion 2017-09-23 10:55:03 +02:00
andi b7dc22275d trigger garbage collection periodically 
This reduces the memory usage, because the old generation can be made
smaller and we don't have to wait until max heap usage has been reached.
 2017-09-23 10:54:28 +02:00
andi 9e3dd27eb0 use a default date of seven days ago instead of last year 2017-09-23 10:52:33 +02:00
andi c2ee632f76 plot exactly the selected time frame 
Before gnuplot would only plot the range with values.
But that is not really helpful, when you navigate with the new
navigation buttons.
 2017-04-22 11:23:13 +02:00
andi c783fd2830 add aria-hidden="true" to some icons 2017-04-22 10:49:19 +02:00
andi c758cee428 add validation pattern for the date offset 2017-04-22 10:47:23 +02:00
andi 8ed64a4c0e add navigation buttons 
Added buttons to shift left/right, so that you do not have to
edit the date offset.
I decided to replace icons.css with font-awesome.min.css,
because I am too lazy to add the CSS for the icons every
time I use a new icon.
 2017-04-22 10:34:43 +02:00
andi 87858a79c1 compute proposals for blank strings 
Before we would only provide proposals for empty strings.
But blank and empty is not that different.
 2017-04-20 19:05:21 +02:00
andi fb40139d75 use available space for image 2017-04-20 19:03:17 +02:00
andi 0ebd657b99 add placeholder into the search box 
The placeholder should help a little bit with the syntax.
 2017-04-19 20:20:11 +02:00
andi 5ab6309fc5 reduce size of x-axis labels 2017-04-19 20:02:02 +02:00
andi dc716f8ac4 log more information in a more predictable manner when inserting entries 2017-04-19 19:32:23 +02:00
andi a99f6a276e fix missing/wrong logging 
1. Log the exception in PdbFileIterator with a logger instead
   of just printing it to stderr.
2. Increase log level for exceptions when inserting entries.
3. Log exception when creation of entry failed in TcpIngestor.
 2017-04-17 18:27:25 +02:00
andi c58e7baf69 make sure there is an exception if the file is corrupt 2017-04-17 17:52:11 +02:00
andi bcb2e6ca83 add query completion 
We are using ANTLR listeners to find out where in the
query the cursor is. Then we generate a list of keys/values
that might fit at that position. With that information we
can generate new queries and sort them by the number
of results they yield.
 2017-04-17 16:25:14 +02:00
andi f6a9fc2394 propose for an empty query 2017-04-16 10:39:17 +02:00
andi 44f30aafee add a new facade in front of DataStore 
This is done in preparation for the proposal API.
In order to compute proposals we need to consume the
API of the DataStore, but the code does not need to
be in the DataStore. 
Extracting the API allows us to separate these concerns.
 2017-04-16 10:11:46 +02:00
andi 43d6eba7b7 skip entries if we cannot search for the pdb file 
Happened when the project was 'http:'.
 2017-04-16 09:49:21 +02:00
andi ac1ee20046 replace ludb with data-store 
LuDB has a few disadvantages. 
  1. Most notably disk space. H2 wastes a lot of valuable disk space.
     For my test data set with 44 million entries it is 14 MB 
     (sometimes a lot more; depends on H2 internal cleanup). With 
     data-store it is 15 KB.
     Overall I could reduce the disk space from 231 MB to 200 MB (13.4 %
     in this example). That is an average of 4.6 bytes per entry.
  2. Speed:
     a) Liquibase is slow. The first time it takes approx. three seconds
     b) Query and insertion. with data-store we can insert entries 
        up to 1.6 times faster.

Data-store uses a few tricks to save disk space:
  1. We encode the tags into the file names.
  2. To keep them short we translate the key/value of the tag into 
     shorter numbers. For example "foo" -> 12 and "bar" to 47. So the
     tag "foo"/"bar" would be 12/47. 
     We then translate this number into a numeral system of base 62
     (a-zA-Z0-9), so it can be used for file names and it is shorter.
     That way we only have to store the mapping of string to int.
  3. We do that in a simple tab separated file.
 2017-04-16 09:07:28 +02:00
andi 85e45f74b7 update gradle to 3.5 2017-04-14 07:43:42 +02:00
andi f22be73b42 switch the byte prefix of DATE_INCREMENT and MEASUREMENT 
Date increments have usually higher values. 
I had hoped to reduce the file size by a lot. But in my example data
with 44 million entries (real life data) it only reduced the storage 
size by 1.5%.
Also fixed a bug in PdbReader that prevented other values for the 
CONTINUATION byte.
Also added a small testing tool that prints the content of a pdb file.
It is not (yet) made available as standalone tool, but during
debugging sessions it is very useful.
 2017-04-13 20:19:29 +02:00
andi 1163c1ca22 ignore 'test-output' directory 
This is a TestNG output directory.
 2017-04-13 20:12:55 +02:00
andi 58f8606cd3 use special logger for insertion metrics 
This allows us to enable/disable metric logging without having to log 
other stuff.
 2017-04-13 20:12:00 +02:00
andi ce44c3d8d6 add logarithmic scaling for the y-axis 
Often we have a few very high values and a lot
low values. With a linearly scaled y-axis the
plot is mostly useless.
 2017-04-12 19:59:35 +02:00
andi 8baf05962f group by multiple fields 
Before we could only group by a single field. But it is acutally
very useful to group by multiple fields. For example to see the
graph for a small set of methods grouped by host and project.
 2017-04-12 19:16:19 +02:00
andi 6cc6e679a4 sort fields in 'group by' 2017-04-11 18:39:12 +02:00
andi ee15594070 remove TODOs 
They don't make sense anymore.
E.g. the Tags class is used by classes outside of
org.lucares.performance.db.
 2017-04-11 18:09:29 +02:00
andi cc7e461ebf replace PdbRepository with @Bean annotated method 2017-04-11 18:06:33 +02:00
andi b8b4a6d760 remove deprecated constructor and getter 2017-04-10 20:15:22 +02:00
andi ac8ad8d30f close open files when no new entries are received 
If for 10 seconds no new entry is received, then all open 
files are flushed and closed.
We do this to make sure, that we do not loose data, when
we kill the process.
There is still a risk of data loss if we kill the process
while entries are received.
 2017-04-10 20:13:10 +02:00
andi 24259d7d72 replace log4j2.properties with log4j2.xml 
The properties support for log4j2 is not well documented and
possibly buggy. I couldn't figure out how to change the log
level of a logger.
 2017-04-10 19:50:49 +02:00
andi 72436e9c8c extract utility method 
a method to send json over tcp can be used by several tests
 2017-04-08 08:21:50 +02:00
andi 81b7cfa7bd add default size so that we don't have to specify it in tests 2017-04-08 08:20:59 +02:00
andi d72d6df0f4 update third-party libraries 2017-04-08 08:18:39 +02:00
andi ed17d84da4 remove obsolete and disabled test 2017-04-08 08:18:17 +02:00
andi 7b92a306f3 remove warning by using the logger 2017-04-02 11:15:41 +02:00
andi 2d78a70883 duration for inserts was wrong 
The bug was, that we computed the difference between millis and nanos.
Also log duration for flushes.
 2017-04-02 11:15:24 +02:00
andi cd6b71d35a use shorter folder names 
reduces the risk of too long file names
 2017-04-02 11:13:08 +02:00
andi 8ddae957f5 make location of log4j config file configurable 2017-04-02 11:12:22 +02:00
andi 4ec91a4d17 update console log layout pattern 
Use pattern similar to spring boot's default.
 2017-04-02 11:11:49 +02:00
andi 5fb285a4b4 gnuplot needs forward slashes even on windows 2017-04-01 19:15:28 +02:00
andi 46ce445010 return information about the plotted sequences 
this information can be used for tests
 2017-04-01 18:27:43 +02:00
andi 2875237272 replace the dateTo with an interval 
I hope an interval is easier to handle than having 
to change two date fields
 2017-03-27 20:07:32 +02:00
andi 726258020f make it possible to specify the time in the range 2017-03-27 19:38:02 +02:00
andi 364997e611 show better error message, when no data points are found 2017-03-26 17:30:50 +02:00
andi ee00ecb4b5 remove obsolete class 2017-03-20 19:02:01 +01:00
andi 7cf638400c ignore application properties for the 'dev' profile 2017-03-20 18:55:59 +01:00
andi c245a1f7d4 fix typo that broke the gnuplot integration 2017-03-20 18:52:48 +01:00
andi cc71a32d27 make location of gnuplot configurable 2017-03-19 20:04:07 +01:00
andi ea905c2315 replace sysout with logger 2017-03-19 09:20:00 +01:00
andi 9ab5d76d93 better exception logging 2017-03-19 09:08:41 +01:00
andi aadc9cbd21 move TcpIngestor to pdb-ui 
and start it in the web application.
Also use the spring way of handling property files.
 2017-03-19 08:00:18 +01:00
andi 16f9c92d13 add special logger for throughput metrics 
so that we can enable/disable it individually
 2017-03-18 10:20:14 +01:00
andi a01c8b3907 fix flaky test and improve error handling 
just ignore invalid entries
 2017-03-18 10:14:41 +01:00
andi a221259417 remove project pdb-keyword-db 
it was only created to test how fast a keyword db implementation would
be that works with integer arrays
 2017-03-17 16:26:00 +01:00
andi 513c256352 update third party libraries 2017-03-17 16:23:21 +01:00
andi ade8463e46 update to jquery 3.2.0 2017-03-17 16:12:18 +01:00
andi 182a907466 replace flex boxes with new grids 2017-03-17 13:13:12 +01:00
andi 3456177291 add date range filter 2017-03-17 11:17:57 +01:00
andi 8cc42916a4 scroll in autocomplete proposals 2017-02-12 19:43:35 +01:00
andi f178603bc1 add time to x-axis labels only if range is less than two weeks 2017-02-12 19:08:42 +01:00
andi 5aee6f5e4d use label '<none>' to for values that have not value for groupBy field 2017-02-12 18:56:37 +01:00