replace ludb with data-store
LuDB has a few disadvantages.
1. Most notably disk space. H2 wastes a lot of valuable disk space.
For my test data set with 44 million entries it is 14 MB
(sometimes a lot more; depends on H2 internal cleanup). With
data-store it is 15 KB.
Overall I could reduce the disk space from 231 MB to 200 MB (13.4 %
in this example). That is an average of 4.6 bytes per entry.
2. Speed:
a) Liquibase is slow. The first time it takes approx. three seconds
b) Query and insertion. with data-store we can insert entries
up to 1.6 times faster.
Data-store uses a few tricks to save disk space:
1. We encode the tags into the file names.
2. To keep them short we translate the key/value of the tag into
shorter numbers. For example "foo" -> 12 and "bar" to 47. So the
tag "foo"/"bar" would be 12/47.
We then translate this number into a numeral system of base 62
(a-zA-Z0-9), so it can be used for file names and it is shorter.
That way we only have to store the mapping of string to int.
3. We do that in a simple tab separated file.
This commit is contained in:
@@ -14,8 +14,8 @@ import java.util.Map;
|
||||
|
||||
import org.lucares.pdb.api.Entry;
|
||||
import org.lucares.pdbui.TcpIngestor;
|
||||
import org.lucares.performance.db.FileUtils;
|
||||
import org.lucares.performance.db.PerformanceDb;
|
||||
import org.lucares.utils.file.FileUtils;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.testng.Assert;
|
||||
@@ -23,8 +23,6 @@ import org.testng.annotations.AfterMethod;
|
||||
import org.testng.annotations.BeforeMethod;
|
||||
import org.testng.annotations.Test;
|
||||
|
||||
import liquibase.exception.LiquibaseException;
|
||||
|
||||
@Test
|
||||
public class TcpIngestorTest {
|
||||
|
||||
@@ -42,7 +40,7 @@ public class TcpIngestorTest {
|
||||
FileUtils.delete(dataDirectory);
|
||||
}
|
||||
|
||||
public void testIngestDataViaTcpStream() throws LiquibaseException, Exception {
|
||||
public void testIngestDataViaTcpStream() throws Exception {
|
||||
|
||||
final OffsetDateTime dateA = OffsetDateTime.now();
|
||||
final OffsetDateTime dateB = OffsetDateTime.now();
|
||||
|
||||
Reference in New Issue
Block a user