From lubaolu at gmail.com Sat Dec 1 22:37:07 2007 From: lubaolu at gmail.com (Baolu Lu) Date: Sun, 2 Dec 2007 14:37:07 +0800 Subject: [intel-platform-dev] IO Virtual Address management questions. In-Reply-To: <8012EB0B6BDD82478B53C8ED1EA17FD805E7DB41@orsmsx411.amr.corp.intel.com> References: <8012EB0B6BDD82478B53C8ED1EA17FD805E7DB41@orsmsx411.amr.corp.intel.com> Message-ID: <89da17a10712012237r37c57710sc82c136b1b761475@mail.gmail.com> Hi Mark, In addition to these questions, can I add another one? We need function interfaces to allocate/free the iommu pages. The page size is 4k. Can the following steps work? If so, what are the functions in each step? 1. allocate a physical page 2. lock this page 3. map this page to virtual address Can you give me some reference code in the kernel source tree? Thanks On Nov 14, 2007 8:44 AM, Raj, Ashok wrote: > Hi Mark > > Here goes the second question about iova management. Basically iam > looking for 2 things > > 1. A way to carve out ranges that cannot be allocated. > 2. A way to clone the iova areana for each new domain I allocate. > > Seems like I can use the same for other domain allocation code, looks > like I can use it generically even as a bitvec allocator. Say I have 256 > domains, I could just use this with granularity of 1, just like how the > dtrace code is using it? > > More details below. > > IO Virtual Address space management. > ------------------------------------ > 1. We need a method to allocate io virtual address (IOVA). Solaris > already has > support for this via vmem_create(), vmem_alloc() and vmem_free(). > > We need to create holes in the address space for the following items. > > - Remove APIC address range 0xfee00000 - 0xfeefffff > - Remove certain 1-1 ranges, such as RMRR's from an address map > - Remove any mem ranges that exist under p2p side to avoid peer to > peer > pci side effects. It would be simple to just remove them from the > address > range. > > What is the best way to punch holes in the vmem arena. I see two ways > but not sure if there is a better way. > > If there is a new api like vmem_del(vmem_t *, void *addr, size_t > size) > > And is there a way to clone the areana created? we would need to do > this > each time we create a iova for each domain to manage. It might be > easy if > i had an iova areana with holes, and just clone them for each new > domain > i create. > > it would be great if vmem could do the splicing. Without something > like this > > - We could add and create the areana leaving holes with multiple > vmem_add() > calls. > > > - Just create a large address space, and use vmem_xalloc() to > allocate and > remove these holes. Performance is not a problem since this is > startup time > not speed path. > > say > > since we will only map at page granurality its enough if we > account just page numbers. > > iova_area = vmem_create(name, 1, BTOP(UINT64_MAX), 1, NULL, > NULL, NULL, > 0, VM_SLEEP | VMC_IDENTIFIER); > > but to cut out the un-allocatable holes say for e.g > > vmem_xalloc(iova_area, 0xff, 1, 0, 0xfee00, 0xfeeff, VM_SLEEP); > > Cheers, > ashok raj > - Open Source Technology Center > > _______________________________________________ > intel-platform-dev mailing list > intel-platform-dev at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/intel-platform-dev > From ashok.raj at intel.com Mon Dec 3 08:08:21 2007 From: ashok.raj at intel.com (Raj, Ashok) Date: Mon, 3 Dec 2007 08:08:21 -0800 Subject: [intel-platform-dev] IO Virtual Address management questions. In-Reply-To: <89da17a10712012237r37c57710sc82c136b1b761475@mail.gmail.com> References: <8012EB0B6BDD82478B53C8ED1EA17FD805E7DB41@orsmsx411.amr.corp.intel.com> <89da17a10712012237r37c57710sc82c136b1b761475@mail.gmail.com> Message-ID: <8012EB0B6BDD82478B53C8ED1EA17FD8060E98B4@orsmsx411.amr.corp.intel.com> Hi Allen I think mark has this request from us from a while back and is working on an interface not necessarily the same steps as you indicate. - allocate 4k aligned page that is locked - get a virtual and physical addr as output. We would also need to easily translate between virtual/physical and vice versa. Since when we walk the pagetables we would need a way to convert the physical addr to virtual instead of keeping a separate lookup table. Cheers, ashok raj - Open Source Technology Center >-----Original Message----- >From: Baolu Lu [mailto:lubaolu at gmail.com] >Sent: Saturday, December 01, 2007 10:37 PM >To: Raj, Ashok >Cc: Mark.Johnson at sun.com; Lu, Baolu; intel-platform-dev at opensolaris.org >Subject: Re: [intel-platform-dev] IO Virtual Address management questions. > >Hi Mark, > >In addition to these questions, can I add another one? > >We need function interfaces to allocate/free the iommu pages. The page >size is 4k. Can the following steps work? If so, what are the >functions in each step? > >1. allocate a physical page >2. lock this page >3. map this page to virtual address > >Can you give me some reference code in the kernel source tree? > >Thanks > >On Nov 14, 2007 8:44 AM, Raj, Ashok wrote: >> Hi Mark >> >> Here goes the second question about iova management. Basically iam >> looking for 2 things >> >> 1. A way to carve out ranges that cannot be allocated. >> 2. A way to clone the iova areana for each new domain I allocate. >> >> Seems like I can use the same for other domain allocation code, looks >> like I can use it generically even as a bitvec allocator. Say I have 256 >> domains, I could just use this with granularity of 1, just like how the >> dtrace code is using it? >> >> More details below. >> >> IO Virtual Address space management. >> ------------------------------------ >> 1. We need a method to allocate io virtual address (IOVA). Solaris >> already has >> support for this via vmem_create(), vmem_alloc() and vmem_free(). >> >> We need to create holes in the address space for the following items. >> >> - Remove APIC address range 0xfee00000 - 0xfeefffff >> - Remove certain 1-1 ranges, such as RMRR's from an address map >> - Remove any mem ranges that exist under p2p side to avoid peer to >> peer >> pci side effects. It would be simple to just remove them from the >> address >> range. >> >> What is the best way to punch holes in the vmem arena. I see two ways >> but not sure if there is a better way. >> >> If there is a new api like vmem_del(vmem_t *, void *addr, size_t >> size) >> >> And is there a way to clone the areana created? we would need to do >> this >> each time we create a iova for each domain to manage. It might be >> easy if >> i had an iova areana with holes, and just clone them for each new >> domain >> i create. >> >> it would be great if vmem could do the splicing. Without something >> like this >> >> - We could add and create the areana leaving holes with multiple >> vmem_add() >> calls. >> >> >> - Just create a large address space, and use vmem_xalloc() to >> allocate and >> remove these holes. Performance is not a problem since this is >> startup time >> not speed path. >> >> say >> >> since we will only map at page granurality its enough if we >> account just page numbers. >> >> iova_area = vmem_create(name, 1, BTOP(UINT64_MAX), 1, NULL, >> NULL, NULL, >> 0, VM_SLEEP | VMC_IDENTIFIER); >> >> but to cut out the un-allocatable holes say for e.g >> >> vmem_xalloc(iova_area, 0xff, 1, 0, 0xfee00, 0xfeeff, VM_SLEEP); >> >> Cheers, >> ashok raj >> - Open Source Technology Center >> >> _______________________________________________ >> intel-platform-dev mailing list >> intel-platform-dev at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/intel-platform-dev >> From mark.johnson at sun.com Tue Dec 11 05:19:37 2007 From: mark.johnson at sun.com (mark.johnson at sun.com) Date: Tue, 11 Dec 2007 05:19:37 -0800 (PST) Subject: [intel-platform-dev] Sync up Message-ID: <20071211131944.67F55319B8@mail.opensolaris.org> Author: Mark Johnson Repository: /hg/intel-platform/onnv-intel-ioat Latest revision: 69d43f4fcb1a66d123796eabc47039f28f8fe703 Total changesets: 1 Log message: Sync up Files: update: usr/src/uts/common/fs/sockfs/sockstr.c update: usr/src/uts/common/fs/sockfs/socktpi.c update: usr/src/uts/common/fs/sockfs/sockvnops.c update: usr/src/uts/common/inet/tcp/tcp.c update: usr/src/uts/common/inet/tcp/tcp_fusion.c update: usr/src/uts/common/os/move.c update: usr/src/uts/common/sys/uio.h update: usr/src/uts/i86pc/io/ioat/ioat.c update: usr/src/uts/i86pc/io/ioat/ioat_chan.c update: usr/src/uts/i86pc/io/ioat/ioat_ioctl.c update: usr/src/uts/i86pc/io/ioat/ioat_ring.c