andi/perfdb
1
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Activity
94 Commits 2 Branches 0 Tags
44f30aafeeb8b655fef8e4c3b864f79b6dd138f2
Commit Graph
94 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
andi 44f30aafee add a new facade in front of DataStore 
This is done in preparation for the proposal API.
In order to compute proposals we need to consume the
API of the DataStore, but the code does not need to
be in the DataStore. 
Extracting the API allows us to separate these concerns.
 2017-04-16 10:11:46 +02:00
andi 43d6eba7b7 skip entries if we cannot search for the pdb file 
Happened when the project was 'http:'.
 2017-04-16 09:49:21 +02:00
andi ac1ee20046 replace ludb with data-store 
LuDB has a few disadvantages. 
  1. Most notably disk space. H2 wastes a lot of valuable disk space.
     For my test data set with 44 million entries it is 14 MB 
     (sometimes a lot more; depends on H2 internal cleanup). With 
     data-store it is 15 KB.
     Overall I could reduce the disk space from 231 MB to 200 MB (13.4 %
     in this example). That is an average of 4.6 bytes per entry.
  2. Speed:
     a) Liquibase is slow. The first time it takes approx. three seconds
     b) Query and insertion. with data-store we can insert entries 
        up to 1.6 times faster.

Data-store uses a few tricks to save disk space:
  1. We encode the tags into the file names.
  2. To keep them short we translate the key/value of the tag into 
     shorter numbers. For example "foo" -> 12 and "bar" to 47. So the
     tag "foo"/"bar" would be 12/47. 
     We then translate this number into a numeral system of base 62
     (a-zA-Z0-9), so it can be used for file names and it is shorter.
     That way we only have to store the mapping of string to int.
  3. We do that in a simple tab separated file.
 2017-04-16 09:07:28 +02:00
andi 85e45f74b7 update gradle to 3.5 2017-04-14 07:43:42 +02:00
andi f22be73b42 switch the byte prefix of DATE_INCREMENT and MEASUREMENT 
Date increments have usually higher values. 
I had hoped to reduce the file size by a lot. But in my example data
with 44 million entries (real life data) it only reduced the storage 
size by 1.5%.
Also fixed a bug in PdbReader that prevented other values for the 
CONTINUATION byte.
Also added a small testing tool that prints the content of a pdb file.
It is not (yet) made available as standalone tool, but during
debugging sessions it is very useful.
 2017-04-13 20:19:29 +02:00
andi 1163c1ca22 ignore 'test-output' directory 
This is a TestNG output directory.
 2017-04-13 20:12:55 +02:00
andi 58f8606cd3 use special logger for insertion metrics 
This allows us to enable/disable metric logging without having to log 
other stuff.
 2017-04-13 20:12:00 +02:00
andi ce44c3d8d6 add logarithmic scaling for the y-axis 
Often we have a few very high values and a lot
low values. With a linearly scaled y-axis the
plot is mostly useless.
 2017-04-12 19:59:35 +02:00
andi 8baf05962f group by multiple fields 
Before we could only group by a single field. But it is acutally
very useful to group by multiple fields. For example to see the
graph for a small set of methods grouped by host and project.
 2017-04-12 19:16:19 +02:00
andi 6cc6e679a4 sort fields in 'group by' 2017-04-11 18:39:12 +02:00
andi ee15594070 remove TODOs 
They don't make sense anymore.
E.g. the Tags class is used by classes outside of
org.lucares.performance.db.
 2017-04-11 18:09:29 +02:00
andi cc7e461ebf replace PdbRepository with @Bean annotated method 2017-04-11 18:06:33 +02:00
andi b8b4a6d760 remove deprecated constructor and getter 2017-04-10 20:15:22 +02:00
andi ac8ad8d30f close open files when no new entries are received 
If for 10 seconds no new entry is received, then all open 
files are flushed and closed.
We do this to make sure, that we do not loose data, when
we kill the process.
There is still a risk of data loss if we kill the process
while entries are received.
 2017-04-10 20:13:10 +02:00
andi 24259d7d72 replace log4j2.properties with log4j2.xml 
The properties support for log4j2 is not well documented and
possibly buggy. I couldn't figure out how to change the log
level of a logger.
 2017-04-10 19:50:49 +02:00
andi 72436e9c8c extract utility method 
a method to send json over tcp can be used by several tests
 2017-04-08 08:21:50 +02:00
andi 81b7cfa7bd add default size so that we don't have to specify it in tests 2017-04-08 08:20:59 +02:00
andi d72d6df0f4 update third-party libraries 2017-04-08 08:18:39 +02:00
andi ed17d84da4 remove obsolete and disabled test 2017-04-08 08:18:17 +02:00
andi 7b92a306f3 remove warning by using the logger 2017-04-02 11:15:41 +02:00
andi 2d78a70883 duration for inserts was wrong 
The bug was, that we computed the difference between millis and nanos.
Also log duration for flushes.
 2017-04-02 11:15:24 +02:00
andi cd6b71d35a use shorter folder names 
reduces the risk of too long file names
 2017-04-02 11:13:08 +02:00
andi 8ddae957f5 make location of log4j config file configurable 2017-04-02 11:12:22 +02:00
andi 4ec91a4d17 update console log layout pattern 
Use pattern similar to spring boot's default.
 2017-04-02 11:11:49 +02:00
andi 5fb285a4b4 gnuplot needs forward slashes even on windows 2017-04-01 19:15:28 +02:00
andi 46ce445010 return information about the plotted sequences 
this information can be used for tests
 2017-04-01 18:27:43 +02:00
andi 2875237272 replace the dateTo with an interval 
I hope an interval is easier to handle than having 
to change two date fields
 2017-03-27 20:07:32 +02:00
andi 726258020f make it possible to specify the time in the range 2017-03-27 19:38:02 +02:00
andi 364997e611 show better error message, when no data points are found 2017-03-26 17:30:50 +02:00
andi ee00ecb4b5 remove obsolete class 2017-03-20 19:02:01 +01:00
andi 7cf638400c ignore application properties for the 'dev' profile 2017-03-20 18:55:59 +01:00
andi c245a1f7d4 fix typo that broke the gnuplot integration 2017-03-20 18:52:48 +01:00
andi cc71a32d27 make location of gnuplot configurable 2017-03-19 20:04:07 +01:00
andi ea905c2315 replace sysout with logger 2017-03-19 09:20:00 +01:00
andi 9ab5d76d93 better exception logging 2017-03-19 09:08:41 +01:00
andi aadc9cbd21 move TcpIngestor to pdb-ui 
and start it in the web application.
Also use the spring way of handling property files.
 2017-03-19 08:00:18 +01:00
andi 16f9c92d13 add special logger for throughput metrics 
so that we can enable/disable it individually
 2017-03-18 10:20:14 +01:00
andi a01c8b3907 fix flaky test and improve error handling 
just ignore invalid entries
 2017-03-18 10:14:41 +01:00
andi a221259417 remove project pdb-keyword-db 
it was only created to test how fast a keyword db implementation would
be that works with integer arrays
 2017-03-17 16:26:00 +01:00
andi 513c256352 update third party libraries 2017-03-17 16:23:21 +01:00
andi ade8463e46 update to jquery 3.2.0 2017-03-17 16:12:18 +01:00
andi 182a907466 replace flex boxes with new grids 2017-03-17 13:13:12 +01:00
andi 3456177291 add date range filter 2017-03-17 11:17:57 +01:00
andi 8cc42916a4 scroll in autocomplete proposals 2017-02-12 19:43:35 +01:00
andi f178603bc1 add time to x-axis labels only if range is less than two weeks 2017-02-12 19:08:42 +01:00
andi 5aee6f5e4d use label '<none>' to for values that have not value for groupBy field 2017-02-12 18:56:37 +01:00
andi e0a2c90cfe make active row in autocomplete proposals better visible 2017-02-12 18:23:52 +01:00
andi 141e090129 ignore user properties 2017-02-12 18:20:50 +01:00
andi a0f67c383c remove user config 2017-02-12 18:20:38 +01:00
andi d137f7c9cb limit the number of plots 2017-02-12 18:17:36 +01:00
andi e47519e6d9 change button 'search' to 'plot' 
add label to 'groupBy' option
 2017-02-12 12:34:18 +01:00
andi becd04ed13 set terminal mode for gnuplot to noenhanced 
otherwise labels are parsed as LaTeX
 2017-02-12 11:15:37 +01:00
andi 562dadb692 group plots by field 2017-02-12 09:59:14 +01:00
andi b238849d65 use text input for filtering, again 2017-02-12 09:32:46 +01:00
andi c2e42ea5fb test didn't stop correctly 
because I used offer() on a blocking queue of length 1 (chances are
high, that the element won't be added).
 2017-02-05 11:21:09 +01:00
andi 0c9195011a use log4j in pdb-ui 2017-02-05 11:20:00 +01:00
andi 3722ba02b1 add slf4j via log4j 2 logging 2017-02-05 09:53:25 +01:00
andi 175a866c90 update third-party libraries 2017-02-05 08:54:49 +01:00
andi 30b92459a7 update gradle to 3.3 2017-02-05 08:32:15 +01:00
andi 9c86f8136b you can now make runnable distributions via './gradlew distZip' 2017-02-05 08:29:59 +01:00
andi ba2076cbb5 check performance with primitive ints instead of strings as doc id 
Queries can be done in less than a millisecond even for hundreds of
thousands of documents.
 2017-02-04 10:11:09 +01:00
andi 4f77515bbd test for keywords db performance 2017-01-07 09:10:42 +01:00
andi c283568757 group plots by a single field 2016-12-30 18:45:01 +01:00
andi 62437f384f minor unimportant changes 2016-12-30 13:16:30 +01:00
andi 58bb64c80a save 12ms in when checking if cached writer can be used 2016-12-29 19:33:45 +01:00
andi f520f18e13 leverage the cached pdbwriters 
this increased performance from 500 entries per second to 4000.
 2016-12-29 19:24:16 +01:00
andi de241ceb6d finalize refactoring 2016-12-29 18:27:15 +01:00
andi 68ac1dd631 reuse pdb writers 2016-12-28 08:39:20 +01:00
andi db0b3d6d24 new file format 
Store values in sequences of variable length. Instead of using 8 bytes
per entry we are now using between 2 and 20 bytes. But we are also able
to store every non-negative long value.
 2016-12-27 10:24:56 +01:00
andi c5f0e8514c remove debug output 2016-12-23 19:28:11 +01:00
andi 580733d267 only store the tag specific base folder in the database 
before that we added each file (one per day and tag combination) to the
db
 2016-12-23 19:12:30 +01:00
andi 6969c8ce46 all storage files for the same tags use the same storage folder 
- added an additional data folder as first level
 2016-12-23 16:35:00 +01:00
andi 85eaee940e change directory structure 
- the tags come first, then the date, 
  e.g. "mykey=myvalue_<uuid>/2016/01/01/<uuid>"
- We do this, so that we don't have to tag each file, 
  but only the root folder. This should speed up searches
 2016-12-23 15:07:08 +01:00
andi 5efab12063 test which verifies the dates in each file are monotonically increasing 2016-12-23 13:04:05 +01:00
andi 470f3c730d add UT for testing multiple files for different days 2016-12-23 12:48:26 +01:00
andi 61de6695f9 use better contrast colors for autocomplete 2016-12-23 10:44:43 +01:00
andi 24ee0fa60a scroll selected element into view 2016-12-23 10:42:39 +01:00
andi 95e34831d3 simple auto-completion for the search box 2016-12-23 10:32:51 +01:00
andi bc5d1b0b7b show rendered image 
added loading icon
image is scaled to available space
 2016-12-21 20:06:11 +01:00
andi d1e39513f3 create web application 2016-12-21 17:48:36 +01:00
andi 35054b00b8 check what starts faster json, ludb or mapdb 2016-12-17 10:54:54 +01:00
andi d4c694dea3 group results by a single field 2016-12-14 19:36:38 +01:00
andi b25060a5d2 add first most simple result object 2016-12-14 17:59:04 +01:00
andi 2fd7c39ef9 remove obsolete classes 2016-12-13 18:44:00 +01:00
andi a4d09ee4b3 remove svak and grok 2016-12-13 18:41:46 +01:00
andi fa4921fcc9 use custom csv writer for performance 2016-12-13 18:41:19 +01:00
andi 876520eb4c do not create a new ObjectMapper per entry 
also read value with MappingIterator.
This made reading 20-30 times faster. 
We can now read and index 100k-500k per second.
The varianz might be due to LuDB slowness.
 2016-12-12 18:45:02 +01:00
andi 89fbaf2d06 TcpIngestor that receives a stream of json objects and stores them 2016-12-11 18:40:44 +01:00
andi e936df6f7e render plot with a single dataseriew 2016-12-10 18:50:29 +01:00
andi 81b39c5675 small enhancements 2016-12-10 15:36:06 +01:00
andi 4376f8f783 log4j does not guarantee monotonically increasing date values 2016-12-10 15:35:29 +01:00
andi 34ee64fff1 insert entries for different tags in one stream 2016-12-10 14:10:41 +01:00
andi a409c4c5d0 use java.time for time 2016-12-10 08:16:55 +01:00
andi 256b278428 inital commit 2016-12-04 10:41:37 +01:00