[glue-wg] feedback on storage entities

[Maarten.Litmaath at cern.ch]

Hi Owen,

> > Whichever way, we would like to publish such back-end implementation
> > names and versions explicitly.  Flavia has a use case: the name and
> > version of the back-end storage system should be available, such that
> > it can be deduced from the information system which sites are likely
> > to suffer from which open issues.  This is important information for
> > debugging operational problems in WLCG (and other grids).
> 
> 
> Dear all,
> 
> Could some one please explain why back-end storage system is important
> for Glue?
> 
> The backend tape system which is only applicable to a handful of sites

If e.g. StoRM becomes popular, there may be more than a handful.
StoRM relies a lot on GPFS to provide the desired functionality
(e.g. space reservation and just-in-time ACLs).
Various implementations may come to rely on some NFSv4 release
that we might want to publish explicitly.
The upshot of this discussion is that it seems at least undesirable
to have the schema constrain us to a single implementation name and
version.

> (less than 14 sites I would guess) does not need to be discovered as we
> tend to hand code special provision for such sites in our data flow
> models. We have a white board in desy where we track the deployment of
> all tier 1's. The data is not so large that it demands discovery for
> bug fixing.
> 
> For me Glue is mostly about Tier 2/3 sites endpoint discovery which
> have low resources and even lower manpower and yet will be expected to
> fill in some (not all) of this relational information by hand. I feel
> that modeling a Tier 1 or a tier 0 is exactly what we should not be
> focusing upon. Since these sites are already known by name, have FTE's
> who can fill in Glue schema values by hand (and our Grid specific
> extensions). Tier 2/3 sites should ideally do less hand coding as
> manpower is an issue their.

Tier-2 and -3 sites can use YAIM to handle most of the configuration
issues.  These days the documentation is in fairly good shape and the
code explicitly requires the necessary parameters to be filled in,
so it should become easier for sites to publish sensible stuff.

> I'm in favour of a catalogue mapping service so catalogue
> synchronisation can occur which if such was added to the LCG frame work
> would make the tiny amount of "monitoring" data in the Glue useless as
> the catalogues would know what data was used at each site and far more
> usefully could delete files rather than just know they need to be
> deleted.

Each of the LHC VOs probably will implement its own catalogue mapping
or consistency service, but there will be no general service for other
VOs on EGEE for quite a while, if ever.  Other grids may have the same
problem.

It is true that having "monitoring" data in the information system
may lead to false expectations, and can only do a superficial job
w.r.t. accounting.  Still, we have experience with implementing
GLUE schema info providers and those we have some control over,
while a proper accounting service has no clear timeline, AFAIK.

Furthermore, just about everything is optional, so we may decide
later on that some attributes are not worth the effort after all:
cf. the storage library in GLUE 1.x, finally deprecated in 1.3.

> Patrick has told me dCache position is to accept all requests we can
> deliver and fight against all we cant do. Paul now represents dCache
> for Glue, I just drop and email in now and again.
> 
> My comments are not dCache related, I just fear we are missing a rare
> opportunity to remove attributes from an inter grid standard. I am much
> happier we extend the Glue to resolve missing services for
> LCG/OSG/Nordugrid within our grids and MOU's, but to add these as
> fields all Grids should use seems bloated and bad form, when their
> function is just to resolve architectural floors in our current grids
> ability to mange its cataloges. Its consequence is to prevent us
> focusing on missing data and syncronisation services, make inter
> operation harder by adding what I see as non use cases, such as data we
> should just know (backend tape system is not a job selection issue),
> and dynamic trivia such as poor quality monitoring (that does not
> include enough data to resolve running out of space anyway which seems
> the only use case), but that's a longer email. Remember all fields (and
> worse relations) we add to Glue wont go away for years and the costs
> are most apparent with tier 2/3 sites. I feel we are not being hard
> enough on people bringing use cases and looking at alternative
> solutions. I feel that we could resolve all the SE service discovery
> with just two (or three to be consistent with the rest of GLue)
> entities if we removed what I currently see as counter productive use
> cases.

I agree with your concerns, but here is a possible counter argument:
if GLUE 2.0 does not include all the 1.3 functionality that users
and developers appreciate, it may be a long time before we start
seeing 2.0 used for anything real.  We can take a very purist view
of what should be in the information system and find that grids
immediately have to hack it, so that they can get their work done.
It has happened with GLUE 1.x, it can happen again.
Cheers,
	Maarten