| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
Tweak the reconstruction trigger code to ensure that multiple
reconstructions won't be triggered at the same time.
|
| |
|
|
|
|
| |
Need to figure out the best way to do the detailed tracking in
a concurrent manner. I was thinking just an event log, with parsing
routines for extracting statistics. But that'll be pretty slow.
|
| |
|
|
|
|
|
|
| |
A poorly organized commit with fixes for a variety of bugs that were
causing missing records. The core problems all appear to be fixed,
though there is an outstanding problem with tombstones not being
completely canceled. A very small number are appearing in the wrong
order during the static structure test.
|
| | |
|
| | |
|
| |
|
|
|
| |
It isn't working right now (lotsa test failures), but we're to the
debugging phase now.
|
| |
|
|
|
| |
Clarified the reasoning for a few things in comments that just tripped
me up during debugging.
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
Currently, proactive buffer tombstone compaction is disabled by forcing
the buffer tombstone capacity to match its record capacity. It isn't
clear how to best handle proactive buffer compactions in an environment
where new buffers are spawned anyway.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
In InternalLevel::clone(), the m_shard_cnt variable was not being set
appropriately in the clone, resulting in the record counts reported for
a multi-shard level to be reported incorrectly.
In DynamicExtension::merge(), the merges were being performed in the
wrong order, resulting in multi-level merges deleting records. The
leveling tests all passed even with this bug for some reason, but it
caused tiering tests to fail. It isn't clear _why_ leveling appeared to
work, but the bug is now fixed, so that's largely irrelevant I suppose.
|
| |
|
|
|
|
|
|
|
|
| |
1. The system should now cleanly shutdown when the DynamicExtension
object is destroyed. Before now, this would lead to use-after-frees
and/or deadlocks.
2. Improved synchronization on mutable buffer structure management to
fix the issue of the framework losing track of buffers during Epoch
changeovers.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
This function wasn't ensuring that that the epoch pinned and the epoch
returned were the same epoch in the situation where the epoch was
advanced in the middle of the call. This is now resolved, and further the
function will return the newer epoch, rather than the older one, in
such a situation.
|
| |
|
|
|
|
| |
Fixed a few bugs with concurrent operation of internal_append, as well as
enabled the spawning of multiple empty buffers while merges are currently
active.
|
| |
|
|
| |
Fixed an incorrectly initialized lock guard
|
| |
|
|
|
|
|
|
| |
Add empty buffer now supports a CAS-like operation, where it will only
add a buffer if the currently active one is still the same as when the
decision to add a buffer was made. This is to support adding new buffers
on insert outside of the merge-lock, so that multiple concurrent threads
cannot add multiple new empty buffers.
|
| | |
|
| |
|
|
|
| |
Reordered some code in internal_append() to avoid use-after frees on the
mutable buffer reference used for insertion.
|
| | |
|
| |
|
|
|
|
| |
Added a new scheduler for ensuring single-threaded
operation. Additionally, added a static assert to (at least for now)
restrict the use of tagging to this single threaded scheduler.
|
| |
|
|
|
|
|
|
|
| |
The epochs must be released in the destructor prior to releasing the
buffers and structures, as otherwise there are references remaining to
these objects and their destructors will fail.
Additionally, fixed a bug in the constructor resulting in a memory leak
due to allocating an extra starting version and buffer.
|
| |
|
|
|
| |
This is mostly just for testing purposes at the moment, though I'd
imagine it may be useful for other reasons too.
|
| | |
|
| | |
|
| |
|
|
|
|
| |
Instead of busy waiting on the active job count, a condition variable is
now used to wait for all active jobs to finish before freeing an epoch's
resources.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
get_memory_usage, get_aux_memory_usage, get_record_count,
get_tombstone_count, and create_static_structure have been adjusted to
ensure that they pull from a consistent epoch, even if a change-over
occurs midway through the function.
These functions also now register with the epoch as a job, to ensure that
the epoch they are operating own isn't retired midway through the
function. Probably not a big issue for the accessors, but I could see it
being very important for create_static_structure.
|
| | |
|
| |
|
|
|
|
|
| |
I started moving over to an explicit Epoch based system, which has
necessitated a ton of changes throughout the code base. This will
ultimately allow for a much cleaner set of abstractions for managing
concurrency.
|
| | |
|
| |
|
|
| |
currently there's a race condition of some type to sort out.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a big one--probably should have split it apart, but I'm feeling
lazy this morning.
* Organized the mess of header files in include/framework by splitting
them out into their own subdirectories, and renaming a few files to
remove redundancies introduced by the directory structure.
* Introduced a new framework/ShardRequirements.h header file for simpler
shard development. This header simply contains the necessary includes
from framework/* for creating shard files. This should help to remove
structural dependencies from the framework file structure and shards,
as well as centralizing the necessary framework files to make shard
development easier.
* Created a (currently dummy) SchedulerInterface, and make the scheduler
implementation a template parameter of the dynamic extension for easier
testing of various scheduling policies. There's still more work to be
done to fully integrate the scheduler (queries, multiple buffers), but
some more of the necessary framework code for this has been added as well.
* Adjusted the Task interface setup for the scheduler. The task structures
have been removed from ExtensionStructure and placed in their own header
file. Additionally, I started experimenting with using std::variant,
as opposed to inheritence, to implement subtype polymorphism on the
Merge and Query tasks. The scheduler now has a general task queue that
contains both, and std::variant, std::visit, and std::get are used to
manipulate them without virtual functions.
* Removed Alex.h, as it can't build anyway. There's a branch out there
containing the Alex implementation stripped of the C++20 stuff. So
there's no need to keep it here.
|
| |
|
|
|
|
|
|
|
|
|
| |
Merges are now executed from a seperate thread within the scheduler that
wakes up via condition variables when new merge tasks are scheduled. In
addition, tombstone limits are now enforced by the scheduler, with new
merges being scheduled as needed.
There are still a few tests failing, notably the zero tombstones in the
last run invarient is not holding under tiering with tombstones. Need
to look into that yet.
|
| | |
|
| |
|
|
|
| |
The project is now in a state where it builds, but it probably has a lot
of bugs still.
|
| | |
|
| |
|
|
|
| |
Query interface now enables skipping of delete processing and stopping
query processing when first match is found.
|
| |
|
|
|
| |
This is necessary for KNN, but it unused for all currently implemented
query types.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Renamed the get_record_cnt() functions to get_record_count() for
consistency, dropped references to SSIs, and added openmp build flags
for PGM related targets.
Also adjusted dynamic_extension_tests.inc to fail when an erase fails
during the static structure testing as part of debugging a delete
cancellation problem under leveling.
|
| |
|
|
|
|
| |
Layout and Delete Policies are now specified via template parameters,
and rejection sampling has been removed as an argument to the mutable
buffer constructor.
|
| |
|
|
|
| |
This is used for setting up the query alias structure stuff for sampling
queries.
|
| | |
|
| | |
|