replace ludb with data-store

LuDB has a few disadvantages. 
  1. Most notably disk space. H2 wastes a lot of valuable disk space.
     For my test data set with 44 million entries it is 14 MB 
     (sometimes a lot more; depends on H2 internal cleanup). With 
     data-store it is 15 KB.
     Overall I could reduce the disk space from 231 MB to 200 MB (13.4 %
     in this example). That is an average of 4.6 bytes per entry.
  2. Speed:
     a) Liquibase is slow. The first time it takes approx. three seconds
     b) Query and insertion. with data-store we can insert entries 
        up to 1.6 times faster.

Data-store uses a few tricks to save disk space:
  1. We encode the tags into the file names.
  2. To keep them short we translate the key/value of the tag into 
     shorter numbers. For example "foo" -> 12 and "bar" to 47. So the
     tag "foo"/"bar" would be 12/47. 
     We then translate this number into a numeral system of base 62
     (a-zA-Z0-9), so it can be used for file names and it is shorter.
     That way we only have to store the mapping of string to int.
  3. We do that in a simple tab separated file.
This commit is contained in:
2017-04-16 09:07:28 +02:00
parent 85e45f74b7
commit ac1ee20046
56 changed files with 2243 additions and 677 deletions

View File

@@ -1,5 +1,6 @@
package org.lucares.pdbui;
import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;
@@ -20,7 +21,7 @@ public class MySpringConfiguration {
private static final Logger LOGGER = LoggerFactory.getLogger(MySpringConfiguration.class);
@Bean
PerformanceDb performanceDb(@Value("${db.base}") final String dbBaseDir) {
PerformanceDb performanceDb(@Value("${db.base}") final String dbBaseDir) throws IOException {
final Path dataDirectory = Paths.get(dbBaseDir);
LOGGER.info("using database in {}", dataDirectory.toAbsolutePath());