AMD IOMMU [PSARC/2008/561]

[Garrett D'Amore]

Matthew Jacob wrote:
> Follow up to this as it actually turns out to be related to what I'm 
> working on. It turns out to be a not-serious problem, but it required 
> a small bit of thinking and planning ahead.
>
> I'm working on a driver for a board that needs to have DMA S/G lists 
> in coherent memory (as do many boards). These S/G lists are filled in 
> from the dma cookies you get back for mapping a command.
>
> If the underlying mapping mechanism is an IOMMU, then the number of 
> S/G list elements to allocate equals (roughly) the maximum command 
> load for each board.
>
> If the underlying mapping mechanism is DMA S/G, then the number of S/G 
> list elements to allocate equals (roughly) the maximum command load 
> for each board times the (roughly) the expected average page length 
> for each command. Yes, a DMA S/G mechanism /may/ have coalesced pages, 
> but more often than not doesn't.
>
> It turns out to be prudent to build an allocation of card S/G list 
> memory a page at a time anyway as load increases- this way you avoid 
> problems trying to allocate a bunch of contiguous consistent memory at 
> load time. (only to have it fail as this memory, at least on x86, 
> tends to fragment over time).
>
> I'm fortunate in that I can chain these chunks together for this card. 
> Other cards aren't so lucky.
>
> It might  helpful to know the properties of the underlying DMA mapper 
> so different strategies could be implemented at load time.

Knowing that the IOMMU is in play might allow you to assume that only 
one cookie will be needed.  But it would be poor form to rely on that 
behavior.  (This is one area where SPARC drivers sometimes have troubles 
being ported to x86.)

The driver can "force" this using restrictive DMA attributes, but 
experience is that such attributes simply fail to succeed to allocate 
resources on x86.

But in any case, the failure mode here is that going from IOMMU to 
physical only mapping often gives problems, but that introduction of an 
IOMMU layer should itself not cause any problems.  (Except to a driver 
that poorly assumed that the system always used a single cookie per 
page.  Which would very often likely *not* be the case, so  I think even 
those kinds of drivers would fail occasionally without an IOMMU.)

    -- Garrett
>
> Just a thought....
>