From lubaolu at gmail.com  Fri Nov  9 06:17:08 2007
From: lubaolu at gmail.com (Baolu Lu)
Date: Fri, 9 Nov 2007 22:17:08 +0800
Subject: [intel-platform-dev] how to bind my iommu nexus driver
Message-ID: <89da17a10711090617n4e04bfdat8c7d4562b4dab031@mail.gmail.com>

Hi all,

I am here to ask somebody's help on my kernel coding. First, let me
try to describe what I am doing.

I am developing lntel IOMMU driver for OpenSolaris. Intel IOMMU is a
part of Intel VTD (virtual technology
for direct IO), which spec can be downloaded from
http://www.intel.com/technology/virtualization/.

The BIOS exports the IOMMU information through the DMAR ACPI table.
So, first I get the IOMMU information
through parsing the ACPI table in the pci_enumverate( ). And create
the dev_info structure under the
root neuxs. Then want to let the kernel to load my IOMMU nexus driver
at the boot time.

My questions are:

1. What can I do to let the kernel load the IOMMU nexus driver module
automatically during the boot time?

2. The platform may have more than one IOMMU units. Each has it's own
register set and interrupt. But they share
the same bus ops. I want to bind only one driver for them. Does this
make sense? To do so, should I create
a dev_info structure for each IOMMU unit or just a single one?

Thanks for your reading and I am very appreciate for your help.


From lubaolu at gmail.com  Mon Nov 12 19:26:56 2007
From: lubaolu at gmail.com (Baolu Lu)
Date: Tue, 13 Nov 2007 11:26:56 +0800
Subject: [intel-platform-dev] how to bind my iommu nexus driver
In-Reply-To: <89da17a10711090617n4e04bfdat8c7d4562b4dab031@mail.gmail.com>
References: <89da17a10711090617n4e04bfdat8c7d4562b4dab031@mail.gmail.com>
Message-ID: <89da17a10711121926i5116df04n98186c9f17c9566@mail.gmail.com>

OK, sounds nobody is interested by my questions. :)

On Nov 9, 2007 10:17 PM, Baolu Lu <lubaolu at gmail.com> wrote:
> Hi all,
>
> I am here to ask somebody's help on my kernel coding. First, let me
> try to describe what I am doing.
>
> I am developing lntel IOMMU driver for OpenSolaris. Intel IOMMU is a
> part of Intel VTD (virtual technology
> for direct IO), which spec can be downloaded from
> http://www.intel.com/technology/virtualization/.
>
> The BIOS exports the IOMMU information through the DMAR ACPI table.
> So, first I get the IOMMU information
> through parsing the ACPI table in the pci_enumverate( ). And create
> the dev_info structure under the
> root neuxs. Then want to let the kernel to load my IOMMU nexus driver
> at the boot time.
>
> My questions are:
>
> 1. What can I do to let the kernel load the IOMMU nexus driver module
> automatically during the boot time?
>
> 2. The platform may have more than one IOMMU units. Each has it's own
> register set and interrupt. But they share
> the same bus ops. I want to bind only one driver for them. Does this
> make sense? To do so, should I create
> a dev_info structure for each IOMMU unit or just a single one?
>
> Thanks for your reading and I am very appreciate for your help.
>


From Mark.Johnson at Sun.COM  Tue Nov 13 07:54:31 2007
From: Mark.Johnson at Sun.COM (Mark Johnson)
Date: Tue, 13 Nov 2007 07:54:31 -0800
Subject: [intel-platform-dev] how to bind my iommu nexus driver
In-Reply-To: <89da17a10711121926i5116df04n98186c9f17c9566@mail.gmail.com>
References: <89da17a10711090617n4e04bfdat8c7d4562b4dab031@mail.gmail.com>
	<89da17a10711121926i5116df04n98186c9f17c9566@mail.gmail.com>
Message-ID: <4739C8B7.2040105@sun.com>



Baolu Lu wrote:
> OK, sounds nobody is interested by my questions. :)

Sorry Baolu, I was on vacation on Friday and in
a meeting all day yesterday.


> On Nov 9, 2007 10:17 PM, Baolu Lu <lubaolu at gmail.com> wrote:
>> Hi all,
>>
>> I am here to ask somebody's help on my kernel coding. First, let me
>> try to describe what I am doing.
>>
>> I am developing lntel IOMMU driver for OpenSolaris. Intel IOMMU is a
>> part of Intel VTD (virtual technology
>> for direct IO), which spec can be downloaded from
>> http://www.intel.com/technology/virtualization/.
>>
>> The BIOS exports the IOMMU information through the DMAR ACPI table.
>> So, first I get the IOMMU information
>> through parsing the ACPI table in the pci_enumverate( ). And create
>> the dev_info structure under the
>> root neuxs. Then want to let the kernel to load my IOMMU nexus driver
>> at the boot time.
>>
>> My questions are:
>>
>> 1. What can I do to let the kernel load the IOMMU nexus driver module
>> automatically during the boot time?

The Intel IOMMUs are unlike other IOMMUs we have come
across. In my experience, the IOMMUs have been integrated
into the IO bridge (and IO bridge driver).

Since there are multiple IOMMUs, each covering specific
devices, which is specified by the BIOS, I don't think
we will be able to have a traditional parent/child driver
relationship between the IOMMUs device drivers.

At this point, I believe we should probe for and manually
setup/load the IOMMU "drivers" from the rootnex driver.
But maybe we can fit multiple IOMMU support into the PCI
express nexus driver. I have to think about that some
more.


>> 2. The platform may have more than one IOMMU units. Each has it's own
>> register set and interrupt. But they share
>> the same bus ops. I want to bind only one driver for them. Does this
>> make sense? To do so, should I create
>> a dev_info structure for each IOMMU unit or just a single one?
>>
>> Thanks for your reading and I am very appreciate for your help.

I'm not sure if we should have one driver/module, or one for
each at this point. You won't be using busops since I don't think
the driver will be a parent of the device using it (unless
we try to fit it into the PCI Express nexus).

Lets start at the beginning... We need a probe routine which
will query if the system has an intel IOMMU and then return
any relevant info we need including info we need to setup
the infrastructure for interrupts/registers.

Just putting place holders for the names for now, if we
has a function which returned the number of intel IOMMUs
in the system (0 if not supported), and then an array
for each IOMMU with the info we need to set it up, what
would the intel_iommu_probe_t struct look like? e.g.
what will you get out of the ACPI tables.


int intel_iommu_probe(intel_iommu_probe_t *info);
void intel_iommu_probe_free(intel_iommu_probe_t *info);



MRJ



From ashok.raj at intel.com  Tue Nov 13 15:18:09 2007
From: ashok.raj at intel.com (Raj, Ashok)
Date: Tue, 13 Nov 2007 15:18:09 -0800
Subject: [intel-platform-dev] probe and setup questions...
Message-ID: <8012EB0B6BDD82478B53C8ED1EA17FD805E7D9D4@orsmsx411.amr.corp.intel.com>

Hi Mark

There are a couple of questions, but to keep the responses manageable I
thought I will shoot each question separately so we can address them
separately.

Details below: Basically would like get support for the following.

1. map reg for dmar
2. provide intr hookup to receive fault events.

In the following discussion, sometime you will see reference to drhd,
iommu, dmar essentially all are the same! 

In general we would use struct drhd * for the struct exported by ACPI,
and struct iommu * for the mapped iommu reg space definition.

If we could get some pointers on where to instrument to get this support
would be great help.

Discovery & Initialization
--------------------------

1. Probe routine

   int probe_intel_iommu (void)

   The above API will scan ACPI table and create a list of IOMMU's and
its 
   associated list of pci devices, A list of RMRR regions and its device
scope
   for each RMRR.

   Result: List of DMAR's, RMRR's and device list is built. 

   ACPI gives us a list of remapping units via DMAR tables.

   DRHD - DMA remapping hardware device
   RMRR - Reserved Memory Region Reporting Structure

   where do we put this function? 
    rootnex.c/dma_rootnex_init()?

   Could we do this someplace in pci_boot.c/pci_setup_tree() after the 
   enumeration is complete?

   if we do the scanning a bit later after the pci tree is setup, we
could
   do more of the initialization of iommu's as well and setup reserved
regions
   etc. It will be required the entire tree is setup to build all the
device
   tree relationship with dmar's.


   /*
    * 
    * Returns 0: 	Found Intel IOMMU
    *        -ENODEV: 	No Intel IOMMU found
    */
    probe_intel_iommu(void)
    {
    	scan and build list of dmar's, and list of devs for them

	scan and build list of rmrr's and list of devs for them

	for each rmrr
	   	for each dev in rmrr
	      		/*
		 	* find the iommu for this dev, and set
			* identity map for them
		 	*/
	      		prepare_rmmr_fordev(rmrr, pcidev);
	   	endfor /* dev */
	endfor /* rmrr */

	for each dmar
		allocate dev_info_t for dmar
		ddi_map_regs_setup()
		ddi_intr_alloc()
	endfor /* dmar */
    }

    Questions: 
    1. Do we make ddi_root_node() the parent for the dmar devices?
    2. when we call map_regs, will the rootnex be able to field that
request? 
       or do we need to add some support in rootnex.c?
    3. Interrupts for dmar devices are like MSI interrupts. But the
config
       is in memory space, unlike in pci config space.

       So we would need to handle enable_intr, add_handler,
get_supported_intrs
       calls to handle this new type of interrupt.
    4. If we need to support suspend/resume support it would require
some
       sort of driver module framework, so it would be able to intercept

       SUSPEND/RESUME calls. Even though we might not support them
initially
       getting the right framework would help in future
Cheers,
ashok raj
- Open Source Technology Center
   


From ashok.raj at intel.com  Tue Nov 13 16:44:13 2007
From: ashok.raj at intel.com (Raj, Ashok)
Date: Tue, 13 Nov 2007 16:44:13 -0800
Subject: [intel-platform-dev] IO Virtual Address management questions.
Message-ID: <8012EB0B6BDD82478B53C8ED1EA17FD805E7DB41@orsmsx411.amr.corp.intel.com>

Hi Mark

Here goes the second question about iova management. Basically iam
looking for 2 things

1. A way to carve out ranges that cannot be allocated. 
2. A way to clone the iova areana for each new domain I allocate.

Seems like I can use the same for other domain allocation code, looks
like I can use it generically even as a bitvec allocator. Say I have 256
domains, I could just use this with granularity of 1, just like how the
dtrace code is using it? 

More details below.

IO Virtual Address space management.
------------------------------------
1. We need a method to allocate io virtual address (IOVA). Solaris
already has
   support for this via vmem_create(), vmem_alloc() and vmem_free().

   We need to create holes in the address space for the following items.

   - Remove APIC address range 0xfee00000 - 0xfeefffff
   - Remove certain 1-1 ranges, such as RMRR's from an address map
   - Remove any mem ranges that exist under p2p side to avoid peer to
peer
   pci side effects. It would be simple to just remove them from the
address 
   range.
   
   What is the best way to punch holes in the vmem arena. I see two ways
   but not sure if there is a better way.

   If there is a new api like vmem_del(vmem_t *, void *addr, size_t
size)

   And is there a way to clone the areana created? we would need to do
this 
   each time we create a iova for each domain to manage. It might be
easy if
   i had an iova areana with holes, and just clone them for each new
domain
   i create.

   it would be great if vmem could do the splicing. Without something
like this

   - We could add and create the areana leaving holes with multiple
vmem_add()
   calls.


   - Just create a large address space, and use vmem_xalloc() to
allocate and
   remove these holes. Performance is not a problem since this is
startup time
   not speed path.

   say

	since we will only map at page granurality its enough if we 
	account just page numbers.

	iova_area = vmem_create(name, 1, BTOP(UINT64_MAX), 1, NULL,
NULL, NULL,
		    0, VM_SLEEP | VMC_IDENTIFIER);
   
   	but to cut out the un-allocatable holes say for e.g 

	vmem_xalloc(iova_area, 0xff, 1, 0, 0xfee00, 0xfeeff, VM_SLEEP);

Cheers,
ashok raj
- Open Source Technology Center
   


From mark.johnson at sun.com  Wed Nov 21 09:33:06 2007
From: mark.johnson at sun.com (mark.johnson at sun.com)
Date: Wed, 21 Nov 2007 09:33:06 -0800 (PST)
Subject: [intel-platform-dev] partial callback support
Message-ID: <20071121173306.4074511DEC5@mail.opensolaris.org>

Author: Mark Johnson <mark.johnson at sun.com>
Repository: /hg/intel-platform/onnv-intel-ioat
Latest revision: ee7e0709610ca9257a92e16240ce5603c9716039
Total changesets: 1
Log message:
partial callback support
add dom0 support
partial v2 support

Files:
	create: usr/src/uts/i86xpv/ioat/Makefile
	delete: usr/src/pkgdefs/SUNWdcopyh/Makefile
	delete: usr/src/pkgdefs/SUNWdcopyh/pkginfo.tmpl
	delete: usr/src/pkgdefs/SUNWdcopyh/prototype_com
	delete: usr/src/pkgdefs/SUNWdcopyh/prototype_i386
	update: usr/src/pkgdefs/Makefile
	update: usr/src/pkgdefs/SUNWdcopy/prototype_i386
	update: usr/src/uts/common/fs/sockfs/sockstr.c
	update: usr/src/uts/common/fs/sockfs/socksubr.c
	update: usr/src/uts/common/fs/sockfs/socktpi.c
	update: usr/src/uts/common/inet/ip.h
	update: usr/src/uts/common/inet/ip/ip_if.c
	update: usr/src/uts/common/inet/tcp/tcp.c
	update: usr/src/uts/common/inet/tcp/tcp_fusion.c
	update: usr/src/uts/common/io/dcopy.c
	update: usr/src/uts/common/io/dld/dld_proto.c
	update: usr/src/uts/common/os/move.c
	update: usr/src/uts/common/os/streamio.c
	update: usr/src/uts/common/sys/Makefile
	update: usr/src/uts/common/sys/dcopy.h
	update: usr/src/uts/common/sys/dlpi.h
	update: usr/src/uts/common/sys/mac.h
	update: usr/src/uts/common/sys/sodirect.h
	update: usr/src/uts/i86pc/io/ioat/ioat.c
	update: usr/src/uts/i86pc/io/ioat/ioat_chan.c
	update: usr/src/uts/i86pc/io/ioat/ioat_ioctl.c
	update: usr/src/uts/i86pc/io/ioat/ioat_ring.c
	update: usr/src/uts/i86pc/sys/ioat.h
	update: usr/src/uts/i86xpv/Makefile.files
	update: usr/src/uts/i86xpv/Makefile.i86xpv.shared
	update: usr/src/uts/i86xpv/Makefile.rules