IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008]

[Kais Belgaied]

- Case boundary question: since this marks a flag day for both TI and 
CI, can you list the components
  that are affected by this flag day?

- I am not clear on the consumer side of this new  interface: What 
prompts a ULP to start using this interface?
  Is it expected to attempt ibt_alloc_io_mem() until it  exhausts all 
resources?
  It would be easier to assess the completeness and the usefulness of 
the TI if you either extended the case's scope
  to include at least the changes on one transport consumer or gave a 
real example thereof.
 
- mi_ibt_version seems to be an enumeration of apparently mutually 
exclusive values  IBTI_V{1,2,3}
  yet the definition suggests a combination of independent (discrete) 
capabilities
  (FMR support, DMA wrapper support, etc.)
     . Is there any consumer of this interface that uses DMA wapper but 
not FMR?
   . For future evolution, is the mi_ibt_version always intended to 
express a monotonically increasing set
    of capabilities (capabs of V(n+1) includes all capabs of N(n)) ?
   Basically I'm trying to see if information of different nature  if 
being encoded in the same field. Without slipping
  in a design discussion you should consider if two fileds are more 
appropriate: 1 version (number or enum) and one capabs (bitmask).

- Under what condition can the caller of 
ibt_alloc_io_mem()/ibt_free_io_mem()  expect the following error
  to be returned?
 59     IBT_MR_ACCESS_REQ_INVALID   Invalid Access Control Specified.
 60                                 Remote Write or Remote Atomic access is
 61                                 requested without specifying Local 
Write.

 - in ibc_alloc_io_mem.9e
 these two sections are in conflict:
  11     ibt_status_t prefix_ibc_alloc_io_mem(ibc_hca_hdl_t hca_hdl,
  12         size_t size, ibt_mr_flags_t mr_flag, caddr_t *kaddrp,
  13         ibc_mem_alloc_hdl_t *mem_alloc_hdl);
and
  23     hca_hdl       IBTF channel Interface (TI) HCA Handle previously 
obtained
  24                   by calling ibt_open_hca(9F).
  25

    Kais.

On 10/10/08 11:04, Ted Kim wrote:
> Template Version: @(#)sac_nextcase %I% %G% SMI
> This information is Copyright 2008 Sun Microsystems
> 1. Introduction
>     1.1. Project/Component Working Name:
> 	 IBTF IO Memory
>     1.2. Name of Document Author/Supplier:
> 	 Author:  Lida HornI
>     1.3  Date of This Document:
> 	10 October, 2008
> 4. Technical Description
>
> A. Background
>
> The DDI distinguishes between different "types" of memory. Memory from
> ddi_dma_mem_alloc(9F) is usable for DMA and takes into account various
> factors such alignment and other device attributes. Memory from
> kmem_(z)alloc is not guaranteed to be usable for DMA, though most of
> the time it does work for that purpose, because of the capability of
> modern platforms. Nevertheless, there is value in maintaining these
> DDI distinctions, especially when considering certain platform issues
> such as memory DR.
>
> In the context of InfiniBand, registered memory is the target of DMA
> operations. This case introduces new InfiniBand related interfaces
> analogous to the ddi_dma_mem_alloc family of functions to IBTF
> (InfiniBand Transport Framework, PSARC/2002/132 and follow-on
> cases). This addition will help the InfiniBand stack maintain the
> proper DDI memory distinctions important for certain types of
> platforms.
>
>
> B. Proposal
>
> The proposal is to make additions to the IBTF Channel and Transport
> interfaces. The functionality added to the Transport Interface (TI) is
> used by the ULPs to allocate memory suitable for DMA and IB memory
> registration. In turn, the framework uses new entry points in the
> Channel Interface (CI) to request memory allocation from the
> underlying HCA driver. These interfaces are basically a wrapper for
> DDI functions which on the one hand abstract away HCA device specific
> details at the ULP level, but at the same time allow for the HCA
> driver to adjust the memory attributes (alignment, etc.) as necessary
> for efficiency.
>
> These additions include an IBTF ABI change, so this case also marks an
> internal flag day, incrementing our interface version numbers for both
> the TI and CI as noted below.
>
>
> All interface additions and changes in this proposal have a
> micro/patch binding.
>
> Transport Interface (ON Consolidation Private):
>
>   ibt_alloc_io_mem() - Allocates DMA memory (at the transport level)
>   ibt_free_io_mem() -  Deallocates DMA memory 
>   IBTI_V3 - TI version change
>  
> Channel Interface (ON Consolidation Private):
>
>   ibc_alloc_io_mem() - Allocates DMA memory (at the HCA driver level)
>   ibc_free_io_mem() - Deallocates DMA memory 
>   IBCI_V3 - CI version change
>
>
> C. Summary of Changes by man page 
>
> See materials directory for copies of man pages. Modified man pages
> have change bars in the left margin.
>
>   ibci.9 - modfied (new CI entry points added)
>
>   ibc_alloc_io_mem.9e - new (alloc & free CI entry points)
>
>   ibt_alloc_io_mem.9f - new (alloc & free TI functions)
>
>   ibt_clnt_modinfo_t.9s - modified (IBTI_V3 version)
>   ibc_hca_info_t.9s - modified (IBCI_V3 version)
>   ibc_operations_t.9s - modified (new CI entry points added)
>
>
> 6. Resources and Schedule
>     6.4. Steering Committee requested information
>    	6.4.1. Consolidation C-team Name:
> 		ON
>     6.5. ARC review type: FastTrack
>     6.6. ARC Exposure: open
>
>
>