| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| ... | |
| | |
|
| | |
|
| |
|
|
|
|
| |
Instead of busy waiting on the active job count, a condition variable is
now used to wait for all active jobs to finish before freeing an epoch's
resources.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
get_memory_usage, get_aux_memory_usage, get_record_count,
get_tombstone_count, and create_static_structure have been adjusted to
ensure that they pull from a consistent epoch, even if a change-over
occurs midway through the function.
These functions also now register with the epoch as a job, to ensure that
the epoch they are operating own isn't retired midway through the
function. Probably not a big issue for the accessors, but I could see it
being very important for create_static_structure.
|
| | |
|
| |
|
|
|
|
|
| |
I started moving over to an explicit Epoch based system, which has
necessitated a ton of changes throughout the code base. This will
ultimately allow for a much cleaner set of abstractions for managing
concurrency.
|
| | |
|
| |
|
|
| |
currently there's a race condition of some type to sort out.
|
| |
|
|
| |
I'll probably throw all this out, but I want to stash it just in case.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a big one--probably should have split it apart, but I'm feeling
lazy this morning.
* Organized the mess of header files in include/framework by splitting
them out into their own subdirectories, and renaming a few files to
remove redundancies introduced by the directory structure.
* Introduced a new framework/ShardRequirements.h header file for simpler
shard development. This header simply contains the necessary includes
from framework/* for creating shard files. This should help to remove
structural dependencies from the framework file structure and shards,
as well as centralizing the necessary framework files to make shard
development easier.
* Created a (currently dummy) SchedulerInterface, and make the scheduler
implementation a template parameter of the dynamic extension for easier
testing of various scheduling policies. There's still more work to be
done to fully integrate the scheduler (queries, multiple buffers), but
some more of the necessary framework code for this has been added as well.
* Adjusted the Task interface setup for the scheduler. The task structures
have been removed from ExtensionStructure and placed in their own header
file. Additionally, I started experimenting with using std::variant,
as opposed to inheritence, to implement subtype polymorphism on the
Merge and Query tasks. The scheduler now has a general task queue that
contains both, and std::variant, std::visit, and std::get are used to
manipulate them without virtual functions.
* Removed Alex.h, as it can't build anyway. There's a branch out there
containing the Alex implementation stripped of the C++20 stuff. So
there's no need to keep it here.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Fixed a few issues that manifested during the tiering tests,
1) When a version is copied, it now contains copies of the levels,
not just pointers (the levels themselves still hold pointers to
the shards, though).
2) Ensure that tasks are scheduled with the correct timestamp, they
were originally being scheduled backwards. The get_merge_tasks()
method already returns them in the correct order, so reversing
them again put it in the wrong order.
|
| |
|
|
|
|
|
|
|
|
|
| |
Merges are now executed from a seperate thread within the scheduler that
wakes up via condition variables when new merge tasks are scheduled. In
addition, tombstone limits are now enforced by the scheduler, with new
merges being scheduled as needed.
There are still a few tests failing, notably the zero tombstones in the
last run invarient is not holding under tiering with tombstones. Need
to look into that yet.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change is made in anticipation of scheduling each task using a
specific thread, and required some modification to the interface of
ExtensionStructure. Namely,
1. ExtensionStructure now supports a get_merge_tasks() interface,
which returns a list of the individual level merges that would
need to be performed to complete a buffer flush of specified
size.
2. merge_levels and merge_buffer have been promoted to the public
interface, to allow their use within the scheduler.
3. merge_buffer has been modified to assume that the structure
already can support a direct flush of the buffer into L0, it is
now the responsibility of the caller to ensure that the necessary
merges have already been completed prior to calling this method.
Currently, preemptive tombstone compactions are non-functional, so some
unit tests are failing. This will be fixed when the thread scheduling
system is set up.
|
| | |
|
| |
|
|
|
| |
The project is now in a state where it builds, but it probably has a lot
of bugs still.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
Query interface now enables skipping of delete processing and stopping
query processing when first match is found.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
Note: cosine similarity doesn't seem to work for VPTree--I don't think
that it is actually a metric, upon further research. At the very least I
can't find anyone claiming it is, and I've found several people claiming
it isn't. On testing with the Word2Vec data, Euclidean distance works
insofar as the M-Tree and VPTree return the same KNN results for test
queries, whereas Cosine Similarity does not work.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|