zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008]

[Matthew Ahrens]

Jeff Bonwick wrote:
>> Yup, this was Eric's proposed solution: an L2ARC_IS_CACHABLE flag that
>> is added when the arc buffer is allocated.
> 
> That doesn't seem quite right.  Consider this sequence of events,
> where C and NC are two datasets with and without L2ARC caching:
> 
> 	NC: reads block B, puts it in the ARC with L2ARC_IS_CACHEABLE clear
> 	C: reads block B, but since it's not the one doing the allocation,
> 	   doesn't set L2ARC_IS_CACHEABLE
> 	feed thread: discards block B
> 	C: reads block B, going to disk instead of L2ARC as it should have
> 
> But if C had read the block before NC, everything would have been fine.
> 
> It seems to me that the order of access should not determine the policy.
> Whenever a block is looked up in the ARC, if the caller wants that block
> to be L2ARC cached, then the L2ARC_IS_CACHEABLE flag should be ORed in.
> In short, there are three possibly policies:
> 
> (1) If *anyone* wants the block cached, it is cached.
> (2) If *anyone* doesn't want the block cached, it is not cached.
> (3) It might or might not be cached depending who gets there first.
> 
> To me, (1) is the only policy that makes sense, or is even defensible.

Agreed.  Seems like this could be accomplished in the above scenario by 
having C set L2ARC_IS_CACHEABLE when it hits in the cache.

--matt