[Pgi-wg] Professional Software Engineering (was Gridiron, or standardization gone backwards)
Etienne URBAH
urbah at lal.in2p3.fr
Tue Jul 20 11:53:42 CDT 2010
Morris, Oxana, David and all
Lot of thanks to Morris for his general leadership and his positive mail
oriented toward future success.
I fully understand that many PGI-WG members are frustrated by the
non-achievements and internal struggles, and I try to present below some
explanations and proposals.
'Very advanced draft' produced by the PGI founders in September 2008
--------------------------------------------------------------------
Did this 'Very advanced draft' ever exist ?
Inside GridForge PGI documents, the 'Input_Documents/Pre-OGF_material'
contains various little files, some of which containing only some
fragments of requirements or specifications.
Only on 11 May 2009 did Balazs KONYA upload the first version of the
'Rendering of the GES strawman' in GridForge at
http://forge.gridforum.org/sf/go/doc15628?nav=1 with comment 'initial
rather incomplete draft'.
NONE of the previous files clearly describes :
- the context :
- Distributed resources managed inside separate administrative domains,
- Local and Grid-level credentials,
- Computing resources forbidden (or authorized) to have internet access,
- Input files available inside the Grid or only available on the
local machine of the Submitter of the Activity,
- ...
- the problem(s) to be solved (AUTHN, AUTHZ, Delegation, Automatic /
Manual Data Staging, ...),
- a summary of the solution(s) proposed to solve the problem(s),
- the interactions between the software components used in the solution(s),
- the semantics and syntax of the data that the software components
have to exchange,
- the sequence of the exchanged messages, and their relationships with
states of the software components.
Only on 31 July 2009 did Moreno MARZOLLA upload the 'Strawman
functionality' in GridForge at
http://forge.gridforum.org/sf/go/doc15736?nav=1
At that time, PGI should have worked to validate this foundational
document, which contains at least some hints about the above topics.
But instead of working on the validation of this foundational 'Strawman
functionality' document, PGI regrettably continued struggling on the
'Rendering of the GES strawman'.
This document regrettably dives directly into Port-Types, which are
details of communication protocol totally irrelevant at this stage.
They didn't have to waste time on formalizing use cases and
requirements, because all these were in their heads.
-----------------------------------------------------------
That is the main cause of FAILURE for most projects :
Founders ERRONEOUSLY think that the use cases and requirements are clear
in their heads, but in most projects, this proves FALSE, and
misunderstanding pops up everywhere.
In particular, I is clear that the founders do NOT agree on what an
Activity ID should be, and do NOT understand even now the relationships
of this issue with the stability of the Endpoint of the Execution
Service (Single Endpoint, Factory + Manager, Transient Endpoints).
This is the reason why I have published the 3 corresponding Sequence
Diagrams inside the 'Input Documents / Execution Service' folder.
Using the lessons of dozens of years of bad organization and thousands
of projects with bad software design, the most talented researchers
(Booch, Jacobson, Rumbaugh, ...) have gathered best practices and
propose methods permitting stakeholders to :
- understand the problems,
- write them down in a manner understandable for others,
- begin their work with foundations which will not collapse too quickly.
Just like Alchemy used the rules of science to evolve into Chemistry :
Computing is using methods like CMMI or ITIL, and languages like UML, to
evolve into professional Software Engineering and Operation. In
particular, we have to consider :
- Use Cases,
- Requirements,
- Use Case Diagrams,
- Class Diagrams,
- Collaboration Diagrams,
- Sequence Diagrams,
- State Diagrams,
- Deployment Diagrams,
- Flow Charts,
- ...
Professionals preferably begin with Use Cases. Therefore, Use Cases are
often the foundation of the whole Software Engineering work.
In order that these foundations are good, it is absolutely necessary
that the Use Cases correctly describe the System, the Actors, the
Preconditions, the Interactions between the Actors and the System, ...
This is the reason why I strongly reject a bad Use Case Template.
Analogies with 'Gridiron' and with 'Foreign people driving foreign cars'
------------------------------------------------------------------------
I am afraid that one of the main problem is that 'Gridiron' is an
appropriate analogy for simple computing with different programs on
different machines, which is far simpler that our real problem.
Though, 'Gridiron' can still propose some useful foundation rules, such as :
- Players must respect each other,
- In order to accommodate variations in 'Gridiron' rules, the whole
playground must be large enough, the detailed surfaces of play should be
delimited with washable or removable paint, and there must be an
available assortment of balls of different forms and sizes.
- ...
But 'providing middleware for a distributed infrastructure managed
locally by separate administrative domains and authorizing users to
submit activities on remote resources' is VERY FAR from simple computing.
This is a completely different story, of much higher complexity, having
analogy with 'Foreign people driving foreign cars' :
- In each country, traffic is managed by the local police, which has
the right to stop any car and/or any driver.
- In order to be allowed in a given country :
- A driver must have a valid driver license,
- A car must have a valid plate number and a valid insurance.
- Generally, countries trust each other, so that
- A driver license issued in one country is valid in all other
countries,
- An insurance issued in one country is valid in all other countries.
(But currently, my own car insurance is NOT valid for Irak and North
Korea)
- In order to ease driving and prevent accidents :
- Traffic is everywhere on the right side of the road (except in
the United Kingdom !)
- All indications, and especially warnings, should be given with
commonly agreed pictograms
(Do you all understand what 'Pont effondré' means ?)
- Street names should be indicated using street plates
(But street plates are parallel to the street in France, and as
far as I remember, perpendicular to the street in the USA)
- If necessary, street plates should display a commonly agreed Latin
transliteration
(Ever tried to find 'Avenue Croutchef', who is a former Soviet
Union leader ?)
*** use cases must be provided by users
---------------------------------------
Theoretically, yes.
But computing scientists can only provide 'computing' use case.
And our problem is NOT 'computing', but 'providing middleware for a
distributed infrastructure managed locally by separate administrative
domains and authorizing users to submit activities on remote resources'.
Who are able to understand this problem, and write down relevant Use Cases ?
- Surely the members of the teams having designed and now operating WLCG,
- Perhaps the members of OGF PGI-WG, if we accept professional Software
Engineering.
Therefore, I suggest that each member of PGI-WG :
- carefully studies the 2 Use Case Templates and the 9 Use Case
documents published,
- if necessary seeks advise of professionals working daily on Use Case
capture,
- makes his/her own mind,
- presents his/her position.
use case where these different systems must be brought together
---------------------------------------------------------------
1) http://forge.gridforum.org/sf/go/doc16010?nav=1
'Enforce Security of a Production Service Grid'
2) http://forge.gridforum.org/sf/go/doc16022?nav=1
'from Service Grid to Desktop Grid'
The EDGI project will implement this Use Case for :
- gLite, ARC and Unicore as Service Grid middleware,
- BOINC, XtremWeb-HEP and perhaps also OurGrid as Desktop Grid
middleware
3) If absolutely necessary, the EDGI project is also able to implement
following Use Case : 'Marshal Activities between gLite, ARC and Unicore'
But we hope that this will NOT be necessary.
Let's work with enthusiasm and professionalism !
Best regards.
-----------------------------------------------------
Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS
Bat 200 91898 ORSAY France
Tel: +33 1 64 46 84 87 Skype: etienne.urbah
Mob: +33 6 22 30 53 27 mailto:urbah at lal.in2p3.fr
-----------------------------------------------------
On Tue, 20/07/2010 16:10, Morris Riedel wrote:
> Dear Oxana,
>
> thanks for your summary.
>
>
> (1)
> In fact, I can deeply understand your sadness of what the path for PGI - initially was - and now finally became.
>
>
>
> (2)
> Nevertheless, PGI was created with participation of a small community (only EU) but since then it was a constant struggle in process because of numerous reasons that I don't want to list all.
>
> Examples include that we at least doubled the initial three involved technologies to six and beyond that are currently involved in PGI thus extending to Asia and US. This leads to a huge increase in time required for the agreement processes and discussions, but also significantly increase the impact of PGI.
>
> Another example, is the initial lack of funding leading to best-efforts-only and the proposal process in Europe, then US, that in turn was limiting our time to work with PGI with the necessary enthusiasm.
>
>
>
> (3)
> But instead of complaining - important is that we agreed at OGF to move back to the use cases process because of numerous reasons!
>
> Like a graph algorithm in a complex graph, it might be sometimes necessary to perform one or more 'back steps' in order to achieve a solution and to find 'the path'.
>
>
>
> (4)
> More importantly, the first time the very major technology providers of our distributed computing community actually 'sit on the same table' together to produce something useful t o g e t h e r.
>
> I therefore still believe that we can achieve something in PGI, although partly struggling, we managed to get the right players on board to hopefully have a much higher impact of PGI than initially aimed for.
>
>
>
> (5)
> So, let's don't give up so quickly.
>
> We already have a much better mutual understanding in PGI as before - let's build on this and keep on working little by little achieving something.
>
>
> Your co-chair,
> Morris
On Tue, 20/07/2010 14:43, Oxana Smirnova wrote:
> Hi David,
>
> you actually bring me back to my first post about use cases, which said:
>
> *** use cases must be provided by users
>
> I still stand by it. Indeed, if we have a community that plays gridiron, they'll never agree to twist the existing rules in order to make it looking a bit like soccer. Yet, we are trying to do exactly this.
>
> I never claim that one technological solution is better than another, just like I never will dare to claim that soccer is better than gridiron :-) What I say, is that they are *different*.
>
> If anybody has a use case where these different systems must be brought together, I'd like to have *this* use case, described in details, what exactly is needed, what exactly has to be unified and standardised.
>
> Cheers,
> Oxana
>
>
> On 20.07.2010 11:52, David Wallom wrote:
>> Hello Oxana,
>>
>> Though I agree in the most parts with your analogy this does seem to be in
>> one way or another the technologists again not getting what the user
>> community want. Take for example Structural Biologists, they are
>> collaborating across continental Europe, UK and US by at the moment having
>> themselves to devise the translators/metalayers necessary to bridge across
>> different infrastructures.
>>
>> We need to move beyond the 'my technical solution is better than yours' and
>> realise that if we are not careful the user communities will just see that
>> we are infighting and go their own way (or tell our funders that we are
>> inefficient and not worth continuing). We must move towards standards and
>> 'give' a little on all sides.
>>
>> The research community in the UK sees US collaboration as increasingly
>> important with small amounts of EU collaboration because of funding. If we
>> are not careful they will go their own way totally and we will not get them
>> back. To go back to your analogy, If the user community wants to play
>> gridiron you can shout until you are blue in the face about soccer but they
>> will ignore you, it isn't suitable for them.
>>
>> David
>>
>>
>> On 18/07/2010 01:39, "Oxana Smirnova"<oxana.smirnova at hep.lu.se> wrote:
>>
>>> Hi all,
>>>
>>> I intended to comment on use cases, but feel like commenting on the very fact
>>> of their appearance.
>>>
>>> When the PGI founders first met in September 2008 (yes, 2008), they produced a
>>> very advanced draft, almost a ready-made specification. The only reason they
>>> could not call it a specification was that they were nobody - that is, OGF did
>>> not know of them.
>>>
>>> They didn't have to waste time on formalizing use cases and requirements,
>>> because all these were in their heads.
>>>
>>> In football terms, they played by similar rules, and didn't have to explain to
>>> each other the gory details. They were driven by the desire to produce common
>>> rules of the game for themselves, such that they can play in the same league.
>>>
>>> Then OGF kindly adopted the team, but at the cost of putting forward formal
>>> requirements. No forward movement happened since. First step backwards was to
>>> trim the specs to a "strawman". Second step backwards was to drop the strawman
>>> and collect requirements. The third step backwards was to go back to use
>>> cases. I dread to think what will be the next PGI decision? To create itself?
>>>
>>> In September 2008 we thought that by December same year we'll have the core
>>> specs. Two years later we are discussing what is the best template for use
>>> cases and which teleconferencing tool to use. This is, well, unbelievable.
>>>
>>> And Ithink I know the reason. We try to compare incomparable things.
>>>
>>> Imagine an international football federation that brings together association
>>> football, American football, rugby, Australian rules football and all such
>>> things. And imagine this federation introducing common rules. What would this
>>> rule be? Right, "the game is played on a large field by two teams". Is there
>>> any practical use of this rule? No, every league will have to keep own
>>> "extensions".
>>>
>>> Despite often looking brain-damaged, footballers are clever enough not to
>>> invent common football rules. They realize that the term is overloaded, and
>>> they manage to disambiguate it.
>>>
>>> Grids brought together by PGI are as different as gridiron is different from
>>> soccer. Let's face it. They still can be played on the same pitch - meaning,
>>> they can use same hardware - but attempts to device common
>>> rules/specifications so far lead nowhere. It is as if gridiron guys would be
>>> keeping insisting that soccer has to be played with a ball that doesn't even
>>> look like a ball, and rugby folks would be agreeing, and soccer guys would be
>>> scratching their heads and meekly saying that their use case is actually to
>>> kick it with feet, not carry in armpits.
>>>
>>> The analogy is probably not exactly accurate, but I am quite frustrated, as I
>>> can see no progress whatsoever. Of the original PGI "creators" only three are
>>> still attending the meetings - Johannes, Aleksandr and myself.
>>>
>>> Any suggestions are welcomed.
>>>
>>> Cheers,
>>> Oxana
>
> Oxana Smirnova <oxana.smirnova at hep.lu.se>
> Lund University / NDGF
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5073 bytes
Desc: S/MIME Cryptographic Signature
Url : http://www.ogf.org/pipermail/pgi-wg/attachments/20100720/197cd961/attachment.bin
More information about the Pgi-wg
mailing list