[Pgi-wg] Professional Software Engineering (was Gridiron, or standardization gone backwards)

Tue Jul 20 14:25:36 CDT 2010

Hello Etienne,

thank you for the thorough analysis!

Of course, my gridiron analogy was half-joking, but only half ;-)

You are right pointing out problems of non-professional software 
engineering; only PGI was not meant to engineer any software.

PGI wanted to produce BES, JSDL and GLUE2 _profiles_ common for gLite, 
UNICORE and ARC. *That* simple. This is why pre-PGI documents are short 
and simple.

Well, some also wanted to do more dramatic changes to BES, but that was 
not likely to happen.

So, before getting bogged down in state diagrams and port-types again, 
I'd like to know, what are we trying to achieve now, on a high level? 
Are we still into profiling BES, JSDL and GLUE2 for the purpose of 
having a single infrastructure, or do we do something different? Like I 
wrote in other mails, I don't believe we can produce a single worldwide 
Grid infrastructure using all possible technologies; I don't even 
believe it is needed.

And allow me a very high-level comment on the last proposed use case: 
why (conceptually) would it be necessary to "marshal activities between 
ARC, gLite and UNICORE", when the infrastructure will be mixed, when all 
services from different providers will be used interchangeably? When it 
will be straightforward to deploy at one site an ARC computing service, 
a gLite storage and a UNICORE monitoring service? That's why we need 
standards, no? Beckham can move from MU to Real and even to LA Galaxy 
without having to learn new rules. He may decide to move to Miami 
Dolphins, but something tells me they neither will need his bending 
kick, nor will be willing to play by his rules ;-)

Cheers,
Oxana

On 20.07.2010 18:53, Etienne URBAH wrote:
> Morris, Oxana, David and all
>
> Lot of thanks to Morris for his general leadership and his positive mail
> oriented toward future success.
>
> I fully understand that many PGI-WG members are frustrated by the
> non-achievements and internal struggles, and I try to present below some
> explanations and proposals.
>
>
> 'Very advanced draft' produced by the PGI founders in September 2008
> --------------------------------------------------------------------
> Did this 'Very advanced draft' ever exist ?
>
> Inside GridForge PGI documents, the 'Input_Documents/Pre-OGF_material'
> contains various little files, some of which containing only some
> fragments of requirements or specifications.
>
> Only on 11 May 2009 did Balazs KONYA upload the first version of the
> 'Rendering of the GES strawman' in GridForge at
> http://forge.gridforum.org/sf/go/doc15628?nav=1 with comment 'initial
> rather incomplete draft'.
>
> NONE of the previous files clearly describes :
>
> - the context :
> - Distributed resources managed inside separate administrative domains,
> - Local and Grid-level credentials,
> - Computing resources forbidden (or authorized) to have internet access,
> - Input files available inside the Grid or only available on the local
> machine of the Submitter of the Activity,
> - ...
>
> - the problem(s) to be solved (AUTHN, AUTHZ, Delegation, Automatic /
> Manual Data Staging, ...),
>
> - a summary of the solution(s) proposed to solve the problem(s),
>
> - the interactions between the software components used in the solution(s),
>
> - the semantics and syntax of the data that the software components have
> to exchange,
>
> - the sequence of the exchanged messages, and their relationships with
> states of the software components.
>
>
> Only on 31 July 2009 did Moreno MARZOLLA upload the 'Strawman
> functionality' in GridForge at
> http://forge.gridforum.org/sf/go/doc15736?nav=1
>
> At that time, PGI should have worked to validate this foundational
> document, which contains at least some hints about the above topics.
>
> But instead of working on the validation of this foundational 'Strawman
> functionality' document, PGI regrettably continued struggling on the
> 'Rendering of the GES strawman'.
> This document regrettably dives directly into Port-Types, which are
> details of communication protocol totally irrelevant at this stage.
>
>
> They didn't have to waste time on formalizing use cases and
> requirements, because all these were in their heads.
> -----------------------------------------------------------
> That is the main cause of FAILURE for most projects :
>
> Founders ERRONEOUSLY think that the use cases and requirements are clear
> in their heads, but in most projects, this proves FALSE, and
> misunderstanding pops up everywhere.
>
> In particular, I is clear that the founders do NOT agree on what an
> Activity ID should be, and do NOT understand even now the relationships
> of this issue with the stability of the Endpoint of the Execution
> Service (Single Endpoint, Factory + Manager, Transient Endpoints).
> This is the reason why I have published the 3 corresponding Sequence
> Diagrams inside the 'Input Documents / Execution Service' folder.
>
> Using the lessons of dozens of years of bad organization and thousands
> of projects with bad software design, the most talented researchers
> (Booch, Jacobson, Rumbaugh, ...) have gathered best practices and
> propose methods permitting stakeholders to :
> - understand the problems,
> - write them down in a manner understandable for others,
> - begin their work with foundations which will not collapse too quickly.
>
> Just like Alchemy used the rules of science to evolve into Chemistry :
> Computing is using methods like CMMI or ITIL, and languages like UML, to
> evolve into professional Software Engineering and Operation. In
> particular, we have to consider :
> - Use Cases,
> - Requirements,
> - Use Case Diagrams,
> - Class Diagrams,
> - Collaboration Diagrams,
> - Sequence Diagrams,
> - State Diagrams,
> - Deployment Diagrams,
> - Flow Charts,
> - ...
>
> Professionals preferably begin with Use Cases. Therefore, Use Cases are
> often the foundation of the whole Software Engineering work.
> In order that these foundations are good, it is absolutely necessary
> that the Use Cases correctly describe the System, the Actors, the
> Preconditions, the Interactions between the Actors and the System, ...
> This is the reason why I strongly reject a bad Use Case Template.
>
>
> Analogies with 'Gridiron' and with 'Foreign people driving foreign cars'
> ------------------------------------------------------------------------
> I am afraid that one of the main problem is that 'Gridiron' is an
> appropriate analogy for simple computing with different programs on
> different machines, which is far simpler that our real problem.
>
> Though, 'Gridiron' can still propose some useful foundation rules, such
> as :
> - Players must respect each other,
> - In order to accommodate variations in 'Gridiron' rules, the whole
> playground must be large enough, the detailed surfaces of play should be
> delimited with washable or removable paint, and there must be an
> available assortment of balls of different forms and sizes.
> - ...
>
>
> But 'providing middleware for a distributed infrastructure managed
> locally by separate administrative domains and authorizing users to
> submit activities on remote resources' is VERY FAR from simple computing.
>
> This is a completely different story, of much higher complexity, having
> analogy with 'Foreign people driving foreign cars' :
>
> - In each country, traffic is managed by the local police, which has the
> right to stop any car and/or any driver.
>
> - In order to be allowed in a given country :
> - A driver must have a valid driver license,
> - A car must have a valid plate number and a valid insurance.
>
> - Generally, countries trust each other, so that
> - A driver license issued in one country is valid in all other countries,
> - An insurance issued in one country is valid in all other countries.
> (But currently, my own car insurance is NOT valid for Irak and North Korea)
>
> - In order to ease driving and prevent accidents :
> - Traffic is everywhere on the right side of the road (except in the
> United Kingdom !)
> - All indications, and especially warnings, should be given with
> commonly agreed pictograms
> (Do you all understand what 'Pont effondré' means ?)
> - Street names should be indicated using street plates
> (But street plates are parallel to the street in France, and as far as I
> remember, perpendicular to the street in the USA)
> - If necessary, street plates should display a commonly agreed Latin
> transliteration
> (Ever tried to find 'Avenue Croutchef', who is a former Soviet Union
> leader ?)
>
>
> *** use cases must be provided by users
> ---------------------------------------
> Theoretically, yes.
>
> But computing scientists can only provide 'computing' use case.
>
> And our problem is NOT 'computing', but 'providing middleware for a
> distributed infrastructure managed locally by separate administrative
> domains and authorizing users to submit activities on remote resources'.
>
> Who are able to understand this problem, and write down relevant Use
> Cases ?
> - Surely the members of the teams having designed and now operating WLCG,
> - Perhaps the members of OGF PGI-WG, if we accept professional Software
> Engineering.
>
> Therefore, I suggest that each member of PGI-WG :
> - carefully studies the 2 Use Case Templates and the 9 Use Case
> documents published,
> - if necessary seeks advise of professionals working daily on Use Case
> capture,
> - makes his/her own mind,
> - presents his/her position.
>
>
> use case where these different systems must be brought together
> ---------------------------------------------------------------
> 1) http://forge.gridforum.org/sf/go/doc16010?nav=1
> 'Enforce Security of a Production Service Grid'
>
> 2) http://forge.gridforum.org/sf/go/doc16022?nav=1
> 'from Service Grid to Desktop Grid'
> The EDGI project will implement this Use Case for :
> - gLite, ARC and Unicore as Service Grid middleware,
> - BOINC, XtremWeb-HEP and perhaps also OurGrid as Desktop Grid middleware
>
> 3) If absolutely necessary, the EDGI project is also able to implement
> following Use Case : 'Marshal Activities between gLite, ARC and Unicore'
> But we hope that this will NOT be necessary.
>
>
> Let's work with enthusiasm and professionalism !
>
> Best regards.
>
> -----------------------------------------------------
> Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS
> Bat 200 91898 ORSAY France
> Tel: +33 1 64 46 84 87 Skype: etienne.urbah
> Mob: +33 6 22 30 53 27 mailto:urbah at lal.in2p3.fr
> -----------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oxana_smirnova.vcf
Type: text/x-vcard
Size: 293 bytes
Desc: not available
Url : http://www.ogf.org/pipermail/pgi-wg/attachments/20100720/7d342f9a/attachment-0001.vcf 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2357 bytes
Desc: S/MIME Cryptographic Signature
Url : http://www.ogf.org/pipermail/pgi-wg/attachments/20100720/7d342f9a/attachment-0001.bin 