AMD IOMMU [PSARC/2008/561]

[Garrett D'Amore]

Vikram Hegde wrote:
> Hi all,
>
> ok, what I am hearing is that we keep these properties Sun internal 
> and maybe use for debugging and testing. So I will keep this in the 
> code but will not document it. Is this acceptable to all ?

It is with me.

    - -Garrett
>
> Vikram
>
> Garrett D'Amore wrote:
>> Furthermore, drivers that don't want to re-spin for this should 
>> continue to function normally.  The only reason to avoid use of the 
>> IOMMU is either performance optimization or to workaround incorrect 
>> driver implementation.
>>
>> As indicated, I don't mind have have some properties "undocumented" 
>> usage, but we really should be encouraging folks that care to avoid 
>> the IOMMU to use the documented API.
>>
>> (And, IMO, there really should be no reason to avoid the IOMMU if 
>> you've written your code to perform well under regular DDI compliant 
>> DMA.  The IOMMU itself affords some significant benefits, making it 
>> much easier to get single contiguous DMA segments, reducing the need 
>> to deal with multiple cookies -- scatter gather -- or DMA windows.  
>> Never mind the huge benefits it affords for virtualization and the 
>> fault tolerance benefits that come for free as well.)
>>
>>    -- Garrett
>>
>> Wesley Shao wrote:
>>> FORCE_PHYSICAL allows finer granularity and the proposed properties
>>> make the drivers non-portable to SPARC platforms. They should really be
>>> used for internal debug.
>>>
>>> Besides, they make the drivers hard to run on virtualized platforms.
>>>
>>> Wes
>>>
>>> Vikram Hedgde wrote:
>>>> Hi,
>>>>
>>>>> Yes.  I'd prefer not to "document" the alternative mechanism, 
>>>>> since I view that as a hack/workaround rather than the correct 
>>>>> behavior of using the DMA attributes.
>>>> How about 3rd party drivers that driver vendors don't want to rev 
>>>> and re-release solely for this. Shouldn't there be a switch that 
>>>> they can suggest to customers to use with their existing driver 
>>>> binaries ?
>>>>
>>>> Vikram
>>>>
>>>>
>>>> Garrett D'Amore wrote:
>>>> Vikram Hegde wrote:
>>>>> Garrett D'Amore wrote:
>>>>>> It would be cooler if this could be dealt with using ordinary DMA 
>>>>>> attributes.  Won't DDI_DMA_FORCE_PHYSICAL (see the dma_attr_flags 
>>>>>> for ddi_dma_attr(9s)) work for this?
>>>>>>
>>>>> Yes, that would work as well and is the preferred and established 
>>>>> solution. This flag will be respected by the IOMMU. However, it 
>>>>> seems that an alternative mechanism that involves no touching of 
>>>>> the driver code is desirable as well. Do you agree ?
>>>>
>>>> Yes.  I'd prefer not to "document" the alternative mechanism, since 
>>>> I view that as a hack/workaround rather than the correct behavior 
>>>> of using the DMA attributes.
>>>>
>>>>    -- Garrett
>>>>>
>>>>> Thanks,
>>>>> Vikram
>>>>>>    - Garrett
>>>>>>
>>>>>>
>>>>>> Vikram Hegde wrote:
>>>>>>> Hi Kais,
>>>>>>>
>>>>>>> Below is the text I am planning to add to the PSARC case. And 
>>>>>>> yes, the disable properties will be a part of this PSARC case.
>>>>>>>
>>>>>>> Vikram
>>>>>>>
>>>>>>> ===================================================================== 
>>>>>>>
>>>>>>> Enabling/Disabling the IOMMU
>>>>>>> =============================
>>>>>>> It is recognized that driver or platform developers may not in 
>>>>>>> certain cases desire the
>>>>>>> IOMMU to be enabled. To accomodate such needs we propose the 
>>>>>>> following global properties
>>>>>>> in the iommu driver.conf file
>>>>>>>
>>>>>>> "global-disable" - can be set to 1 to disable IOMMU platform wide.
>>>>>>> "exclude-list"   - list of driver names that want the IOMMU 
>>>>>>> disabled.
>>>>>>> ====================================================================== 
>>>>>>>
>>>>>>>
>>>>>>> Vikram
>>>>>>>
>>>>>>> Kais Belgaied wrote:
>>>>>>>> Vikram, Jerry,
>>>>>>>>
>>>>>>>> So, is the ability to disable the IOMMU on a per driver basis 
>>>>>>>> in this case's scope?
>>>>>>>> what interface is being proposed for doing this?
>>>>>>>>
>>>>>>>> Jerry, assuming the answer  the to first question above is 
>>>>>>>> 'yes', the case cannot be in
>>>>>>>> a "closed approved automatic" state while waiting for some of 
>>>>>>>> its spec.
>>>>>>>>
>>>>>>>>    Kais.
>>>>>>>>
>>>>>>>> On 09/03/08 23:16, Matthew Jacob wrote:
>>>>>>>>> Garrett D'Amore wrote:
>>>>>>>>>> Vikram Hegde wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>>> Abilities to disable the IOMMU as a choice? Performance 
>>>>>>>>>>>> impact on various systems and interactions with other 
>>>>>>>>>>>> projects that have gone to some length to work within S/G 
>>>>>>>>>>>> and its performance characteristics. How will this affect 
>>>>>>>>>>>> them? Cache effects? Interactions with some VM work 
>>>>>>>>>>>> currently being undertaken?
>>>>>>>>>>>>
>>>>>>>>>>>> I can't imagine such a major change being 
>>>>>>>>>>>> closed/approved/automatic. 
>>>>>>>>>>> Performance is driver dependant. There is no way a single 
>>>>>>>>>>> project team can test the gamut of drivers out there and 
>>>>>>>>>>> certify that the IOMMU will  give acceptable performance for 
>>>>>>>>>>> all needs (which is itself a subjective opinion). We have 
>>>>>>>>>>> always planned to provide the ability to disable the IOMMU 
>>>>>>>>>>> on a per driver basis. Does that address your performance 
>>>>>>>>>>> concerns ?
>>>>>>>>>>
>>>>>>>>>> Actually, for many cases, IOMMU may *improve* performance.  
>>>>>>>>>> This is because it will make it easier to satisfy device 
>>>>>>>>>> constraints for DMA resources without requiring the use of a 
>>>>>>>>>> bounce buffer or dma windows.
>>>>>>>>>
>>>>>>>>> That is almost always the exception now. The other issue IIRC 
>>>>>>>>> is the interaction between the IOMMU and memory caches. In 
>>>>>>>>> many data movement appliance contexts you use scatter-gather 
>>>>>>>>> (scatter-gather in name only- contiguous memory is good) to 
>>>>>>>>> physical pages which are never mapped in.
>>>>>>>>>
>>>>>>>>> The main point here is to avoid a situation that we're 
>>>>>>>>> currently in now with respect MSI-X and Interrupt/CPU binding 
>>>>>>>>> and make sure that there are sufficient control surfaces to 
>>>>>>>>> allow for a broad range of possible implementation choices.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> That said, drivers which assume that setup and tear down of 
>>>>>>>>>> DMA is "cheap" may take a perf. hit when using an IOMMU.  
>>>>>>>>>> This is already the case for SPARC systems, btw.
>>>>>>>>>
>>>>>>>>> Except for those (ancient) ones that used 36-bit MBus physical 
>>>>>>>>> addresses and bypassed the IOMMU.
>>>
>>>
>>
>
>