-
Notifications
You must be signed in to change notification settings - Fork 99
Commands
Commands are resources and are represented in tnvme as a C++ classes. Not all classes used to define a command are meant to be instantiated. Some classes are intermediate and they simply add characteristics to the hierarchical inheritance tree. The easiest way to tell if a command can be instantiated is to reference the Resource Manager. If RsrcMngr::AllocObj() allows to specify an enumeration representing the class of interest then it is meant to be instantiated.
The basic idea behind a command class is to allow a test to instantiate one and in doing so creates the logic to build upon. The initial creation of an object does not guarantee it is a completely operational resource. In most cases there will be missing components, and if it is used as is, it will most likely be rejected by any DUT. Specifically it won’t have any user data or meta data buffers associated. These concepts are, in most cases, highly configurable and so no assumptions are made as to their alignment, size, and initialization state. However, a commands DWORD's will be set to default values upon creation. The most likely scenario of using a command will be to 1st instantiate one using a shared_ptr for either test or group lifetime. Then referencing the doxygen generated output, or perusing the C++ header file for the class, call upon the public methods to setup a more detailed operation. And if the command has user data associated with a read/write operation, peruse the base class PrpData to allow this support. Additionally, the command could make use of meta data buffers. Meta data buffer which are not formed by extended the user data, but rather are meant to be placed as separate buffers, are supported using the base class MetaData.
As a general note, a test case should feel welcome to access any public method within any of the classes encompassing any command. This includes calling the public methods of intermediate classes. If you find yourself wanting to call a private or even a protected method, from a test case then you are most likely going to cause a framework failure. These protected methods are intended to support resource operation within the guidelines of the framework, the public methods are meant for safe test case development.
The user data is any buffer which is meant to hold the data associated with a command’s read or write action. This is read or writes in the general sense, not necessarily reading/writing to/from any media. This could mean to read the configuration space or to write features to the DUT. User data may take two forms which are either contiguous or discontiguous allocation. All contiguous allocated memory must occur within dnvme, and so tnvme must request dnvme to allocate and setup a buffer for its use beforehand. On the other hand, all discontiguous buffers are allocated in user space and are represented by the class MemBuffer. MemBuffer is also a resource and can be allocated with test or group lifetime.
Discontiguous memory is represented in user space by the class MemBuffer. All discontiguous memory must be allocated by tnvme; dnvme does not support allocated discontiguous memory, dnvme only allocates contiguous memory. Discontiguous memory can be allocated in 1 of 3 mutually exclusive ways. One can specify an offset into the first page of allocation where the memory starts, and hence this also implies the alignment of the buffer. One can specify the alignment, but not the offset of a buffer. One can allocate memory not specifying anything and accept whatever is handed by the OS from the heap. These 3 ways of memory allocation are achieved by calling the appropriate MemBuffer::Initxxxxx() method after the class has been instantiated. After the memory has been allocated and initialized, it may be associated with a command. This association is performed by supplying the shared_ptr to MemBuffer to method PrpData::SetPrpBuffer() of the command. Please understand that the interface setup of discontiguous buffers wildly differs from the setup of contiguous buffers. PrpData::SetPrpBuffer() is only used for discontiguous setup.
It may not be obvious that since discontiguous buffers are only virtually contiguous they must be presented by a PRP list when dnvme passes reference to a DUT. The creation of the PRP list is strictly guarded by dnvme. dnvme will create and track all PRP lists for all commands when the command is issued to any Submission Queue (SQ). The PRP list is deallocated by dnvme when the command ends up in the correct Completion Queue (CQ). However tnvme still controls the actual memory backing any command and thus has insight into the user buffer long after the command completes, only the PRP list is deallocated.
One obvious question may arise, what would happen if the Completion Element (CE) is placed in to the incorrect CQ by a DUT? This should be caught by a tnvme test case and return an error as the result of failing compliance. The tnvme framework will then exit that group and report an error to the logging system. If errors are chosen to be ignored while running tnvme, then the framework will disable and subsequently re-enable the DUT before the next group executes. This disabling is destructive at all levels and cleans up the system to prepare for the next test. This cleanup is thoroughly detailed in resource lifetime.
Contiguous memory can only be allocated by the kernel and so tnvme requests dnvme to do so via IOCTL’s which are buried in the framework. The details of the dnvme-tnvme interface are advanced and will not be discussed here. Contiguous memory which is associated with a command only comes into discussions when creating IOQ’s, because all other commands do not require contiguous memory. Some may require at most 1 discontinuity, but here again this is considered discontiguous and is handled by user space allocations. Furthermore, access to creating contiguous memory is privatized and protected from direct test case invocation due to its complexity. Contiguous memory is created on behalf of a test if a test creates an IOQ resource and specifies the desire to use contiguous memory to back that queue. Afterwards the 2nd half of IOQ Creation must occur by sending a Create IOQ command to the DUT. This command will be instantiated and subsequently initialized by calling its Init() method by passing the corresponding IOQ resource being created. When this association occurs is when the framework handles the custom IOCTL’s to dnvme to actually create the contiguous memory which is to back the IOQ. In other words, there would exist two resources, one resource to represent the physical IOQ and its memory, and another resource to represent the command which will be sent to the DUT to perform the 2nd half of creation. The moment these two resources are associated is when the contiguous memory is created within dnvme by the framework. Only after the command completes successfully will there be physically contiguous memory known by the DUT, by dnvme and by tnvme which can be used to issue commands. The amount of physically contiguous memory is calculated when the IOQ resources is initialized stating the size of each element and the number of elements it will contain.
Meta data buffers can either be embedded within the user data by extending the user’s data size, or it can be transferred as separate buffers. This discussion only applies to the latter case of sending as separate buffers. The specification states that meta data buffers must be physically contiguous when they are handed off to a DUT. In order to reduce complexity and address speed issues, this architecture allows for tnvme to request dnvme for contiguous meta data buffers of a constant set size. This size requirement must be maintained until such time the DUT is disabled. Afterwards a new meta data buffer size may be specified. This size is set by calling RsrcMngr::SetMetaAllocSize().
The complexity of handling meta data buffers is addressed by the interaction of a command resource and the gRsrcMngr. The gRsrcMngr manages the actual allocation requests to dnvme for the memory. And when any buffer is no longer used by a command the memory is not immediately released, rather the gRsrcMngr places previously used meta data buffers into a collection pool. Later any meta data buffers in the collection pool which can be doled out upon request. This saves times by eliminating multiple calls into dnvme to request deallocation and reallocation of memory. Since each buffer must be of the same size, gRsrcMngr will know that the next request will be for a known size.
Any particular command can request a meta data buffer regardless if it makes sense to do so. Each command does this by calling base class MetaData::AllocMetaBuffer(). This in turn requests memory from the gRsrcMngr. This memory is contiguous and dedicated as meta data memory. The gRsrcMngr will first check its pool of previously used buffers and return one to the command, otherwise a specific call into dnvme must occur to request a new allocation. After a successful call to MetaData::AllocMetaBuffer() the command will own that memory until its lifetime expires. The destructor of the command notifies the gRsrcMngr that the meta data buffer is no longer owned by the command and so it will become recycled.
A command which chooses to use a meta data buffer won’t have to worry about initializing the command DWORD 4 & 5 (MPTR) values, because dnvme will populate those fields within the command when it is issued to a SQ. A test application simply needs to make sure that the meta data memory contains something useful by gaining access to the memory. Access to the memory can be gained by calling the commands base class MetaData::GetMetaBuffer(). This method doesn’t return a shared_ptr to that memory, because that memories lifetime needs to be owned by the command. A test case can read/write using a local C++ character pointer.