[ur-wg] Usage vs Job accounting and the UR framework

[Matthew Ford]

Donal Wrote:

>This point has two distinct aspects.
> 1) The StartTime and EndTime values refer to The time period to which
>    the Usage Record itself applies. For a job (whether grid, batch or
>    flipping hamburgers for minimum wage) one natural interpretation is
>    that the start and end indicate an accounting period that matches
>    when the job is "consuming resources" in some appropriate sense. But
>    strictly the start and end indicate the accounting period, not the
>    job.

Whether this is true or not depends on which part of the spec you read!  Compare
10.10 to 3.13.  The former defines StartTime in terms of a "job" and the latter
defines in terms of "usage".

It is the lack of a clear definition of a "job" and the intermingling of "job"
and  "usage" in the definitions that are leading, I think, to a *lot* of
confusion with the schema.  I use job from now on to mean _batch system job_.
It goes to the heart of the framework.

Regarding the framework
-----------------------


Donal's email of the 27th noted the confusion of UsageRecord and JobUsageRecord
and talks about the dividing of terms(elements) into generic and computational
groups.

The division of elements is alluded to in the current spec: base and job level
accounting - see the first sentence of Secs. 3 and 4 respectively.  It just
doesn't seem to be consistently applied.

Further, Donal's composite suggestion, also sort-of exists in the current spec:
the UsageRecords element (note: it does not allow Sven's nested requirement) -
see section 7, Global Element Definitions.

Below is a summary of what we have now:

Record A:
<UsageRecord>
  <Base Properties..../>
</UsageRecord>

Record B:
<JobUsageRecord>
  <Base Properties..../>
</JobUsageRecord>

Record C:
<UsageRecords>
  <UsageRecord>
  </UsageRecord>
  ...
  <JobUsageRecord>
  </JobUsageRecord>
  ...
</UsageRecords>

Records A and B are at the moment equivalent.    Record C illustrates the
composite structure.  It is a *syntactical change of name only*, between
Records A and B.  The issue is that the Base properties in records A and B are
an identical set.  The JobUsageRecord whilst inheriting the UsageRecord type
does not extend it in anyway (but this may be intentional, I would like to
understand the original authors original intent at this point - see questions
at end).  Without the extension or the separation of the current base
properties there is little point to having the two elements UsageRecord and
JobUsageRecord.

Personally I would like to see the spec use a combination of types and
sub-elements to capture usage and to split the current base elements
accordingly.  I second Donal's proposal.  For example;

<UsageRecord>
  <Base Properties>
  <JobUsageRecord>
    <CPUDuration/>
    <Processors/>
  </JobUsageRecord>
</UsageRecord>

This allows consistency of data format through types and extensibility through
additional elements.  It will provide a sensible place for future work on
aggregation and allow grid-jobs for example to be much more easily defined.

For example:

<UsageRecord>
  <Base Properties>
    <GlobalJobId/>
    <Accounting Period/>
    <Aggregate Info/>
  </Base Properties>
  <Grid Job>
    <JobUsageRecords>
       <JobUsageRecord/>
       ...
    </JobUsageRecords>
    <DataRecord/>
  </Grid Job>
</UsageRecord>

I'm also painfully aware that such a change will mean changes/breaks for
existing accounting solutions in place.  For example the NGS and it's current
RUS service....it may be too big an ask given the current spec.  But I believe
it's necessary in order to make the spec something more that job accounting.

Further it is not feasible for a small group like this to define the elements
such as DataRecord and Grid Job (we have a head start on JobUsage).  I think
the schema should be made extensible to allow other groups to "plug-in" to this
structure and that the group should aim to act somehow in a arbitrators role.

Notes
-----
Historical Questions:

Is it that the UsageRecordType should be inherited by elements such as
JobUsage or, hypothetically DataUsage?  And where possible appropriate elements
are mapped onto the global UsageRecord type - this would ensure consistency of
data (Duration fields for example through-out the record) but require a certain
amount of shoe horning (and potentially a very large global type) to make things
work.  Why was this approach taken?  The extensibility talked about in section 7
- how was that envisaged working?

Thanks,

Matt.

--

Grid Support, Manchester Computing.