[osol-discuss] Project proposal: full FMA support for generic x86 machine-check architecture

[Gavin Maltby]

The Project

	This project proposes improving the generic support of x86
	CPUs that implement the Machine Check Architecture (MCA).  It
	also proposes to create a model-specific plugin interface whereby
	model-specific MCA implementation may supplement and extend
	the generic support rather than stand alone as today.  Finally, the
	project proposes to redesign the cpu-module interface such that
	the MCA support it provides will be usable in both
	virtualized and bare-metal environments.

Background

	Prior to the "amd64 FMA" project (putback in February 2006)
	support for MCA was woeful, with no intelligent enabling of
	error detectors, no polling for errors that do not
	produce a machine check, and a clumsy machine-check handler
	which barfed an incomplete and raw dump of error telemetry to
	the console before panic.  There was no model-specific MCA
	support whatsoever.

	The "amd64 FMA" project, putback in February 2006, delivered
	detailed model-specific support for AMD family 0xf revisions B to E
	(also known as AMD Opteron/Athlon64/Turion64, K8 etc; revisions
	B to E all have DDR1 memory) and moderately improved generic
	MCA support.

	The AMD family family 0xf revs B to E support included:

		- performing MCA and model-specific configuration
		  beyond what the BIOS had setup
		- enabling all MCA banks and machine-check on all
		  uncorrectable error types
		- an error poller for errors that do not machine-check
		  (such as correctable memory errors)
		- an error handler that collects all telemetry and
		  bakes it into a model-specific error report which
		  is dispatched for logging and diagnosis via a robust
		  mechanism that will carry telemetry across a panic/reboot
		  cycle
		- detailed knowledge of model-specific error types, able
		  to produce an ereport error class from the raw telemetry
		  and to specify the appropriate ereport payload members
		  to include for that error class
		- a memory-controller driver able to discover all aspects
		  of memory configuration and to map system addresses to
		  memory resource
		- a topology enumerator to describe cpu and memory topology
		  and properties to the topology library, for use in
		  diagnosis software
		- eversholt diagnosis rules to diagnose all telemetry
		- corresponding fault classes and sun.com/msg knowledge
		  articles

	The generic support, applied to anything other than the AMD models
	above including all Intel processors and any AMD models outside the
	model range above, included:

		- generic MCA detector and error type enabling
		- a machine check exception handler which dumps all
		  MCA telemetry to the console, still in raw form
		- no error poller
		- no preparation of ereports (even for panic events), and
		  therefore no diagnosis of any kind

	The above support was backported to Solaris 10 Update 2.
	The model-specific AMD support was updated in build 51 to include
	AMD family 0xf revision F and G support.  Those revisions use
	DDR2 memory, and most work was required in the memory-controller
	driver.  This additonal support has been backported to Solaris 10
	Update 4 (pending release at this time).

The Problem

	1) The first problem is that the generic cpu MCA support is light
	on features:

		- it lacks an error poller;  we won't log memory
		  correctable ECC errors, for example
		- it does not raise ereports for logging and diagnosis,
		  even for errors that result in a panic
		- on an error that results in a panic at best you have
		  some raw, uninterpreted error register values and no
		  error classification or interpretation; if the system
		  lacks console logging no evidence at all may remain
		  when the system resets.

	This generic support is applied to all Intel processors, and to
	AMD processors not handled by already-delivered AMD model-specific
	support.  Where a platform with full model-specific support
	is upgradeable to new processors that drop in to existing sockets
	and where the new processor family/model/stepping is no longer
	covered by the existing model-specific support, the platform
	after upgrade will regress to the dumb generic cpu support.

	So for all Intel and for new AMD processors there is a window of
	OpenSolaris builds during which MCA support is very light - it
	takes some time to implement full model-specific support for a cpu.
	Improving generic cpu support will raise the base level of support
	for all processors that implement MCA/MCE features.  Purely from the
	Sun point of view, and not a concern for OpenSolaris itself, the
	delay in introducing new model-specific support is further
	stretched in having to backport to a suitable Solaris 10 Update
	release;  it is also possible that products based on a new
	cpu model can revenue-release before corresponding full FMA
	support for the cpu model is available in Solaris 10.
	
	2) The second problem is that the current cpu module
	implementations (generic and AMD family 0xf) and the
	cpu module interface itself are completely invalildated when
	not running natively.  A separate project will be proposed for
	FMA/Xen support, but generic cpu MCA support and the cpu module
	interface changes required for the current project proposal
	should be made with that requirement in mind.

	3) A third problem is that the current cpu module design
	leads to substantial code duplication across model-specific
	implementations (e.g., implementing an error poller in each).
	This should not be necessary given the common base MCA that they
	adhere to.

Key Objectives

	1) Enrich the generic cpu support to include an error poller,
	error type classification (as far as the architectural definition
	allows), raising of ereports capturing all the telemetry for
	logging and some degree of diagnosis (again as much as is possible
	generically).

	2) Design a new cpu model-specific MCA module interface which
	will supplement generic cpu MCA support to avoid code duplication
	and improve maintainability.  This new interface should be
	rich enough to allow the refactoring and redelivery of exising
	AMD model-specific support, and should be designed with upcoming
	Intel model-specific support in mind.

	3) Identify vendor-specific but nevertheless fairly widely-
	applicable (to many models from that vendor) features that
	can be captured in an intermediate level of model-specific
	support for new CPU models from that vendor.  For example in
	the absence of a fully-featured memory-controller driver we may
	be unable to determine the full memory topology and provide
	full error address to memory resource translation, but both
	AMD and Intel in recent models have hardware registers that
	can identify where errors have occured (albeit at a lower
	resolution than a full memory-controller driver may be able to
	provide).  Such support could provide a level of "generic AMD"
	and "generic Intel" support for new cpu releases that is
	better than the lowest common denominator generic support,
	and which is later not loaded in favour of full model-specific
	support once that is available.

	4) Redesign the cpu module interface to cope with running both
	natively and in a virtualized setting, Xen Solaris domain 0
	in particular.  The new interface should not assume any 1:1
	relationship between Solaris logical cpus (cpu_t structures)
	and real physical cores on the system, should not assume that
	being bound to a Solaris cpu means we can perform reads from
	hardware registers all from the same real core, etc.
	Instead the points at which we acquire telemetry and other
	hardware details should be well-defined, and the information
	gathered via calls into a newly-defined native or virtualized layer.
	Implementation of the virtualized layer itself is left to another
	project.

	5) The project deliverables must include some degree of hardware
	error-injection, or of error simulation if that is not possible.
	Existing test harnesses should be extended to include generic
	cpu scenarios.

	6) As with any FMA project, this should include an FMA portfolio
	detailing all events, diagnosis thereof, ASRU boundaries etc.
	To date there are no open FMA portfolios so those outside of Sun
	will have little idea of what this involves - that will become
	clear during the progres of the project.

	7) This project will not deliver full model-specific support
	for any Intel cpus nor for upcoming "AMD family 0x10" processors.
	It should, however, proceed with the needs of those projects
	in mind and should colloborate with those projects to understand
	those needs.

Community Participation

	Participation and contribution from the OpenSolaris community is
	encouraged.  This project will use public documentation from both
	AMD and Intel, available from the websites of those companies.
	As such, there is no vendor NDA or similar barrier to community
	participation.  It may be necessary for some design decisions to
	be made by or in conjunction with those with access to NDA
	documentation, but these are expected to be the exception.

	Error injector sources are currently in the closed tree.  Otherwise
	all applicable source is available in OpenSolaris.

	The event registry is now available externally to Sun, so there is no
	reason that the FMA portfolio cannot be prepared and published
	externally as part of this project.

Gavin

-- 
Gavin Maltby, Solaris Kernel Development.