[glue-wg] New attribute for GLUE 2.1: StorageShare.ViewID

[Paul Millar]

Hi Maria,

On 25/09/14 09:04, Maria Alandes Pradillo wrote:
> If we need to write GLUE 2.1 info providers this is not in my opinion
> a backwards compatible change.

No, please re-read my explanation: nobody needs to write a GLUE 2.1 
info-provider.

Some groups (like dCache) may *choose* to write a GLUE 2.1 info-provider 
to better describe the storage; however, existing (GLUE 2.0) 
info-providers will continue to work fine.


> Could you please explain why suddenly this is an issue,

I *think* what has happened is this:

dCache published some Storage Shares.  Broadly speaking, there are two 
types: ones that show SRM space-reservations and ones that show physical 
hardware.

The Storage Shares representing SRM space-reservations are correct and 
easily selected as they have a non-empty Tag attribute.  This satisfies 
use-cases, like ATLAS, who do space accounting through SRM 
space-reservations.

The Storage Shares representing physical hardware were intended for 
"installed capacity" calculations.

I know that, for at least one Tier-1, the installed capacity numbers are 
provided as a spreadsheet and are calculated manually, NOT from 
information published by GLUE.  I suspect this is true for other sites, 
but I could be wrong here.

The problem is that, for dCache, the Storage Shares representing SRM 
space-reservations and those representing physical hardware are two 
views on the same storage which overlap.  The Storage Shares in the "SRM 
space-reservations" view do not overlap.  Similarly, the Storage Shares 
in the "physical hardware" view do not overlap.

In GLUE-2.0, there is no way to represent this.  In GLUE-2.0, pairs of 
Storage Shares either overlap or they are non-overlapping.

> who has requested this use case,

The ultimate use-case is for calculating the installed capacity, which 
is a long established use-case in WLCG.  I believe introducing an extra 
attribute is needed to provide the necessary information for a dCache 
instance.

 > how many VOs or users are affected otherwise

I believe all WLCG VOs that take part in the installed capacity 
calculations are affected.

> and why it can't be addressed with the existing GLUE 2.0 schema?

Because GLUE 2.0 provides only the SharingID attribute, which describes 
whether or not two Storage Shares overlap.  This is not a rich enough 
language to describe a dCache instance.

(SharingID may have some other problems, but those we can work-around.)

> Before proposing the change to the GLUE WG, I think we need further
 > discussion also among the rest of the Storage
> developers.

I thought the GLUE WG was the correct place to discuss a proposed change 
to GLUE.

> What happens with DPM or StoRM?

They need do nothing: the proposed change is backwards compatible for 
GLUE 2.0 info-providers.

 > If they don't publish this attribute, will those sites or users
 > who have requested that use case be happy?

If they (DPM and StoRM info-providers) don't publish this attribute, 
sites and users will continue to be as happy as they are now.

> Will they target only dCache installations?

I'm not sure what you mean by "they" and "target"; however, the proposal 
was carefully crafted to be backwards compatible for GLUE 2.0 
info-providers.

The overall effect is to unify how storage systems are represented by 
allowing a client to discover information about any kind of storage system.

 > I would like all Storage systems to be aligned.

Me too.  Currently they are not.  I would like to change this; hence the 
proposal.

> I don't like the idea that we introduce an optional attribute that
 > will be mandatory for dCache but not for the others.

Storage systems are different and behave differently.  Publishing 
different subset of attributes is a natural consequence of this.

The point here is that GLUE 2.0 is incapable of describing a dCache site 
in such a way that the installed capacity may be calculated from the 
published data.

GLUE 2.0 is "broken" in the sense that it cannot describe a dCache site 
such that the installed capacity may be calculated.  I would like to 
"fix" GLUE 2.0 so an info-provider can describe a dCache instance.

> And why do you say that storage accounting is broken in dCache only
> now?

Simple: because I didn't realise, until now, that is was broken.


> This is the first time I hear this, when I have spent the whole
> summer trying to improve storage info providers and the feedback I
> got from dCache is that everything was OK for GLUE 2 publishing.

Sure, dCache publishes some wonderful GLUE 2.0 information.  One can do 
many things with that information.  However, it turns out that 
calculating the installed-capacity isn't one of them.

Like many things, it's only when there are users that the problems are 
found.

However, please view this as a normal bug-fixing process: a problem has 
been discovered.  We are in the process of understanding it.  A solution 
has be proposed, which will be discussed based on its merits.

Cheers,

Paul.