- split the 'sortby' select field into two fields
- sort by average
- legend shows plotted and total values in the date range
- removed InlineDataSeries, because it was not used anymore
Sometimes it is useful to specify the certain y-axis range. For example
when you are only interested in the values that take longer than a
threshold. Or when you want to exclude some outliers. When you want to
compare plots in a gallery, it is very handy when all plots have the
same data-area.
Eventually we want to only support what is now called aggregate, but
not have to implement different plot types. So instead of supporting
percentile plots for dashboards I removed them. You can still get
percentile plots together with the scatter plot.
Scaling big plots to small thumbnails results in bad images that barely
show any details.
We solve this by calling gnuplot a second time to generate the
thumbnails. They don't have any labels and are rendered in the required
size, so that not scaling is necessary.
Thumbnails have to be requested explicitly, because it can be expensive
to compute them.
The last improvement of memory usage introduced a performance
regression. The ingestion performance dropped by 50%-80%, because
for every inserted entry the Tags were created inefficient.
For each entry we executed two calls to System.nanoTime().
The resulting numbers aren't very reliable and calling nanoTime
that often (160k - 300k per second) is quite expensive.
Plotting can take a long time and use a lot of resources.
Multiple plot requests can cause the machine to run OOM.
We are now allowing plots for 500k files again. This is mainly to
prevent unwanted plots of everything.
1. Log the exception in PdbFileIterator with a logger instead
of just printing it to stderr.
2. Increase log level for exceptions when inserting entries.
3. Log exception when creation of entry failed in TcpIngestor.
This is done in preparation for the proposal API.
In order to compute proposals we need to consume the
API of the DataStore, but the code does not need to
be in the DataStore.
Extracting the API allows us to separate these concerns.
LuDB has a few disadvantages.
1. Most notably disk space. H2 wastes a lot of valuable disk space.
For my test data set with 44 million entries it is 14 MB
(sometimes a lot more; depends on H2 internal cleanup). With
data-store it is 15 KB.
Overall I could reduce the disk space from 231 MB to 200 MB (13.4 %
in this example). That is an average of 4.6 bytes per entry.
2. Speed:
a) Liquibase is slow. The first time it takes approx. three seconds
b) Query and insertion. with data-store we can insert entries
up to 1.6 times faster.
Data-store uses a few tricks to save disk space:
1. We encode the tags into the file names.
2. To keep them short we translate the key/value of the tag into
shorter numbers. For example "foo" -> 12 and "bar" to 47. So the
tag "foo"/"bar" would be 12/47.
We then translate this number into a numeral system of base 62
(a-zA-Z0-9), so it can be used for file names and it is shorter.
That way we only have to store the mapping of string to int.
3. We do that in a simple tab separated file.
Before we could only group by a single field. But it is acutally
very useful to group by multiple fields. For example to see the
graph for a small set of methods grouped by host and project.
