[glue-wg] Some thoughts on storage objects

[Paul Millar]

Hi all,

As an exercise, I've tried to jot down as precise and complete a description 
of each GLUE storage object as possible, also describing and how they relate 
to each other.  I've also tried to do this without any forward-references 
(so, in theory, the document is readable in a single pass).  In almost all 
cases, I've left out the attributes.

I don't know how useful this is.  It's just my point-of-view of things as 
stand now.  I'm sure there are bits that are "wrong" (either I've 
misunderstood and/or this description breaks a use-case), but if so, 
helpfully people can point which bits are wrong and (perhaps) it will 
stimulate some discussion.

BTW, I'm implicitly assuming that StorageEnvironment.RetentionPolicy can be 
multivalued.  If this isn't true and we have the use-case of the same 
physical disks being part of, for example, both Custodial and Output storage, 
then it starts to get complicated.

As always, comments appreciated.

Cheers,

Paul.

---

UserDomain:

 A collection of one or more end-users; a VO is an instance of a
 UserDomain.  All end-users that interact with the physical storage
 are a member of a UserDomain and, in general, derive their
 authorisation from that membership.


StorageCapacity:

 A StorageCapacity object describes the ability to store data within a
 homogeneous storage technology.  This storage technology provides a
 common access latency.

 All StorageCapacity objects are specified within a certain context.
 The context is determined by an association between the
 StorageCapacity object and precisely one other higher-level object.
 These associations are not listed here, but are described in later sections.

 In general, a StorageCapacity object will record some
 context-specific information.  Examples of such information include
 the total storage capacity of the underlying technology and how much
 of that total has been used.

 The underlying storage technology may affect which of the
 context-specific attributes are available.  For example, tape storage
 may be considered semi-infinite, so the total and free attributes have
 no meaning.  If this is so, then it affects all StorageCapacity objects with
 the same underlying technology, independent of their context.

 Different contexts may also affect what context-specific attributes
 are recorded.  This is a policy decision when implementing GLUE, as
 recording all possible information may be costly and provide no great
 benefit.

 [Aside: these two reasons are why many of the attributes within
 StorageCapacity are optional.  Rather than explicitly subclassing
 the objects and making the values required, it is left deliberately
 vague which attributes are published.]

 A StorageCapacity may represent a logical aggregation of multiple
 underlying storage technology instances; for example, a
 StorageCapacity might represent many disk storage nodes, or many
 tapes stored within a tape silo.  GLUE makes no effort to record
 information at this deeper level; but by not doing so, it requires
 that the underlying storage technology be homogeneous. Homogeneous
 means that the underlying storage technology is either identical or
 sufficiently similar that the differences don't matter.

 In most cases, the homogeneity is fairly obvious (e.g., tape storage
 vs disk-based storage), but there may be times where this distinction
 becomes contentious and judgement may be required; for example,
 the quality of disk-base storage might indicate that one subset is
 useful for a higher-quality service.  If this is so, then it may make
 sense to represent the different class of disk by different
 SpaceCapacities.


StorageEnvironment:

 A StorageEnvironment is a collection of one or more StorageCapacities
 with a set of associated (enforced) storage management policies.
 Examples of these policies are Type (Volatile, Durable, Permanent)
 and RetentionPolicy (Custodial, Output, Replica).

 StorageEnvironments act as a logical aggregation of
 StorageCapacities, so each StorageEnvironment must have at least one
 associated StorageCapacity.  It is the associated StorageCapacities
 that allow a StorageEnvironment to store data with its advertised
 policies; for example, to act as (Permanent, Custodial) storage of
 data.

 Since a StorageEnvironment may contain multiple StorageCapacities, it
 may describe a heterogeneous environment.  An example of this is "tape
 storage", which has both tape back-end and disk front-end into
 which users can pin files.  Such a StorageEnvironment would have two
 associated StorageCapacities: one describing the disk storage and
 another describing the tape.

 If a StorageCapacity is associated with a StorageEnvironment, it is
 associated with only one.  A StorageCapacity may not be shared
 between different StorageEnvironments.

 StorageCapacities associated with a StorageEnvironment must be
 non-overlapping with any other such StorageCapacity and the set of all
 such StorageCapacities must represent the complete storage available
 to end-users.  Each physical storage device (e.g., individual disk drive or
 tape) that an end-user can utilise must be represented by (some part
 of) precisely one StorageCapacity associated with a StorageEnvironment.

 Nevertheless, the StorageCapacities associated with
 StorageEnvironments may be incomplete as a site may deploy physical
 storage devices that are not directly under end-user control; for
 example, disk storage used to cache incoming transfers.  GLUE makes
 no effort to record information about such storage.


StorageResource:

 A StorageResource is an aggregation of one or more
 StorageEnvironments and describes the hardware that a particular
 software instance has under its control.

 A StorageResource must have at least one StorageEnvironment,
 otherwise there wouldn't be much point publishing information
 about it. [This isn't a strict requirement, but I think it makes sense
 to include it.]

 All StorageEnvironments must be part of precisely one
 StorageResource.  SoftwareEnvironments may not be shared between
 StorageResources.  This means that all physics hardware must
 be published under precisely one StorageResource.


StorageShare:

 A StorageShare is a logical partitioning of one or more
 StorageEnvironments.

 Perhaps the simplest example of a StorageShare is one
 associated with a single StorageEnvironment with a single
 associated StorageCapacity, and that represents all
 the available storage of that StorageCapacity.  An example of
 a storage that could be represented by this trivial
 StorageShare is the classic-SE.

 StorageSpaces must have one or more associated StorageCapacities.
 These StorageCapacities provide a complete description of the different
 homogeneous underlying technologies that are available under the space.

 In general, the number of StorageCapacities associated with a
 StorageShare is the sum of the number of StorageCapacities associated
 with each of the StorageShare's associated StorageEnvironments.

 Following from this, there is an implicit association between the
 StorageCapacity associated with a StorageShare and the corresponding
 StorageCapacity associated with a StorageEnvironment.  Intuitively, this
 association is from the fact that the two StorageCapacities share the
 same underlying physical storage.  This implicit association is not
 recorded in GLUE.

 StorageSpaces may overlap.  Specifically, given a StorageCapacity
 (SC_E) that is associated with some StorageEnvironment and which has
 totalSize TS_E, let the sum of the totalSize attributes for all
 StorageCapacities that are:
	1. associated with a StorageSpace, and
	2. that are implicitly associated with SC_E
 be TS_S.  If the StorageSpaces are covering then TS_S = TS_E.  If
 the StorageSpaces overlap, then TS_S > TS_E.
  [sorry, I couldn't easily describe this with just words without it sounding
  awful!]

 StorageSpaces may be incomplete.  Following the same definitions
 as above, this is when TS_S < TS_E.  Intuitively, this happens if
 the site-admin has not yet assigned all available storage.

 End-users within a UserDomain may wish to store or retrieve files.  The
 StorageShares provides a complete, abstract description of the
 underlying storage at their disposal.  No member of a UserDomain may
 interact with the physical hardware except through a StorageShare.

 The partitioning is persistent through file creation and deletion.  The
 totalSize attributes (of a StorageSpace's associated StorageCapacties)
 do not change as a result of file creation or deletion.  [Does GLUE need to
 stipulate this, or should we leave this vague?]

 A single StorageShare may allow multiple UserDomains to access
 storage; if so, the StorageShare is "shared" between the different
 UserDomains.  Such a shared StorageShare is typical if a site
 provides storage described by the trivial StorageShare (one that
 covers a complete StorageEnvironment) whilst supporting multiple
 UserDomains.


StorageMappingPolicy:

 The StorageMappingPolicy describes how a particular UserDomain is
 allowed to access a particular StorageShare.  No member of a
 UserDomain may interact with a StorageShare except as described by a
 StorageMappingPolicy.

 The StorageMappingPolicies may contain information that is specific
 to that UserDomain, such as one or more associated StorageCapacities.
 If provided, these provide a UserDomain-specific view of their usage
 of the underlying physical storage technology as a result of their
 usage within the StorageShare.

 If StorageCapacities are associated with a StorageMappingPolicy,
 there will be the same number as are associated with the
 corresponding StorageShare.


StorageEndpoint:

 A StorageEndpoint specifies that storage may be controlled through a
 particular interface.  The SRM protocol is an example of such an
 interface and a StorageEndpoint would be advertised for each instance
 of SRM.

 The access policies describing which users of a UserDomain may use
 the StorageEndpoint are not published.  On observing that a site
 publishes a StorageEndpoint, one may deduce only that it is valid for
 at least one user of one supported UserDomain.


StorageAccessProtocol:

 A StorageAccessProtocol describes how data may be sent or received.
 The presence of a StorageAccessProtocol indicates that data may be
 fetched or stored using this interface.

 Access to the interface may be localised; that is, only available
 from certain computers.  It may also be restricted to specified
 UserDomains.  However, neither policy restrictions are published in
 GLUE.  On observing a StorageAccessProcol, one may deduce only that
 it is valid for at least one user of one supported UserDomain.


StorageService:

 A StorageService is an aggregation of StorageEndpoints,
 StorageAccessProtocols and StorageResources.  It is the top-level
 description of the ability to transfer files to and from a site, and
 manipulate the files once stored.