Shared metadata cache #8819
Open
Shared metadata cache #8819
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…s related fixes around
…witch there are no DbTriggers
TreeHunter9
reviewed
Dec 2, 2025
src/common/dsc.h
Outdated
| { | ||
| if (dsc_sub_type == isc_blob_text) | ||
| return dsc_scale | (dsc_flags & 0xFF00); | ||
| return TTypeId(CSetId(dsc_scale), CollId(dsc_flags > 8)); |
Contributor
There was a problem hiding this comment.
I guess bitshift was intended here?
Same at 305 line.
Noremos
reviewed
Dec 2, 2025
|
|
||
| } // namespace | ||
|
|
||
|
|
Contributor
There was a problem hiding this comment.
I think it's best to avoid unnecessary changes like these to simplify PR. I'll mark them in case you decide to undo them
|
|
||
| namespace Firebird { | ||
|
|
||
| namespace Firebird { |
| const UCHAR* pos; | ||
| }; | ||
|
|
||
|
|
| return 0; | ||
| } | ||
|
|
||
| SSHORT getBlobSubType() const noexcept |
| dsc_address = address; | ||
| } | ||
|
|
||
| bool operator==(const dsc& v) const noexcept |
Contributor
There was a problem hiding this comment.
The presence of DSC_EQUIV and this overload operator is a bit confusing. While this may not be relevant to this PR, it would be a nice to remove external functions related to dsc, such as DSC_EQUIV and DTYPE_IS_TEXT.
| { | ||
| DEB_BATCH(fprintf(stderr, "BLOBs %d were not used in messages\n", m_blobMap.count())); | ||
| ERR_post_warning(Arg::Warning(isc_random) << "m_blobMap.count() BLOBs were not used in messages"); // !!!!!!! new warning | ||
| ERR_post_warning(Arg::Warning(isc_random) << "m_blobMap.count() BLOBs were not used in messages"); // @@@@@@ new warning |
Contributor
There was a problem hiding this comment.
Unnecessary change. Or am I misunderstanding this
| @@ -2188,7 +2188,6 @@ void SortOwner::releaseBuffer(UCHAR* memory) | |||
| buffers.push(memory); | |||
| } | |||
|
|
|||
|
|
|||
| @@ -2442,3 +2441,4 @@ sort_record* PartitionedSort::getMerge() | |||
|
|
|||
| return eof ? NULL : record; | |||
| } | |||
|
|
|||
| return node; | ||
| } | ||
|
|
||
|
|
| ValueExprNode* UdfCallNode::pass2(thread_db* tdbb, CompilerScratch* csb) | ||
| { | ||
| if (function->fun_deterministic) | ||
| Function* f = function(tdbb); |
Contributor
There was a problem hiding this comment.
It's probably better to be consistent with the names and call it func, as in UdfCallNode::execute below
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tasks to be solved.
The old metadata cache is essentially single-threaded. For me, the problem I faced with attempts to introduce synchronization into this structure and get off with relatively little blood was taking into account the use of procedures and functions using a second reference counter, and in order to decide whether to delete an object, a global recalculation of these references is required throughout the cache. dsqlcache is also extremely unsympathetic, which puts objects created and still unmixed in a single transaction for all transactions of the same attach. An additional consideration is that objects stored in the cache are used in a huge number of places in the code, and they can be modified from other places (and threads) during MET_scan_xxx. This could still be dealt with using CopyOnWrite, but how to approach the first two points was unclear.
The new metadata cache is versioned. Each object is divided into 2 parts - a constant and a variable. For relation, for example, the type (table, view, external table, etc.), name, all locks (GCLock, partners lock, rescan lock) will be constant, and format, list of fields, triggers will be variable. The list of versions is linked to a permanent part of the object. As versions become obsolete, the list is cleared (although I won't say that it's very active - only as the cache is accessed, there is no special garbage collector).
In order to access the desired object for reading without locks, it was possible to allocate memory for the entire cache at once. In principle, there are not so many - 64K pointers to each type of cached objects. But I kept in mind the possible increase in the number of objects in the database and therefore made a 2-level array - each lower-level block of N objects is allocated once and for all (until the database is closed), and the upper-level block has the right to grow when the required number of lower-level blocks does not fit there. Therefore, the upper level is implemented using a read-safe array. For reading, we would take a slice of the current state of the array and this slice remains unchanged while we work with it. An external mutex is used to modify the array. This array (SharedReadVector) turned out to be very convenient, I also use it for requests in a statement and a list of formats in a relation. The 2-level arrays (CacheVector) for all objects are the same (template), and all this ultimately lives in DBB.
Ideally, I would like to remove dsqlcache altogether. So far, it has been removed for relations, procedures, and functions. At the same time, I did not abandon the dsql_rel format and the like - they are needed in order to be able to work with it when creating/changing an object (dsql_rel), although there is no such object in the metadata cache yet / or the format is not the same. Temporary objects of this format are created in dsqlScratch and die with it (by pool).
libcds is used to implement secure pointers (HazardPtr), they are needed for version headers in the list of object versions and for a slice in the SharedReadVector.
In all requests, instead of a pointer directly to objects, the cache stores a structure of 2 elements - a pointer to the permanent part of the object and a index in an array of pointers to versions of objects. These arrays may differ in different query clones. To reduce duplication, a RefCounted array is used. rpb_relation's are registered in rpb when creating a clone. The decision on whether to update a set of versions in a clone of the request is made in GetRequest, that is, when a clone of the request is received.
At the moment, TCS is fully working. QA makes mistakes - but I think it's time to pour metacache into the master so as not to prevent merge of other branches. And it's probably time to decide with the indexes of the system tables, which ones are needed and which ones are not very necessary. For example, I need to add indexes by ID for relations and other cached tables.