[saga-rg] Unedited notes from first two days

[Andre Merzky]

Many thanks, David!

And here the text version for the Windows agnostic part of the
world :)

Cheers, Andre.


Quoting [David E. Konerding] (Dec 15 2004):
> Date: Wed, 15 Dec 2004 10:38:43 -0800
> From: "David E. Konerding" <dekonerding at lbl.gov>
> To: Simple API for Grid Applications WG <saga-rg at ggf.org>
> Subject: [saga-rg] Unedited notes from first two days
> 
> 





-- 
+-----------------------------------------------------------------+
| Andre Merzky                      | phon: +31 - 20 - 444 - 7759 |
| Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 444 - 7653 |
| Dept. of Computer Science         | mail: merzky at cs.vu.nl       |
| De Boelelaan 1083a                | www:  http://www.merzky.net |
| 1081 HV Amsterdam, Netherlands    |                             |
+-----------------------------------------------------------------+
-------------- next part --------------
SAGA-RG design team meeting 12/13/2004

DK and SJ are taking minutes to post on the web

CL- We?re just making a straw man document, not final API

TG- minor timing details.  Start 9AM CST each day; go 3 hrs

Hartmut- sick, will be here weds on.

SJ- everybody email shantenu

TG- style details.  OO vs. C and fortran.  
Discussion- target audience, how to make API which is OO for C and fortran.
Scripting interfaces vs. programming API

Chris Smith: notification?

DK- low priority: asynch

CL- priority list.  

Hrabri- similar issues in drmma.  Some ops return right away.  For synch ops, use timeouts.  

Discussion around style/design of polling/async, callbacks, etc.

AM- Java CoG- simple API, synchronous.  On top ? task module.  

TG- start w/ nonblocking calls & build blocking calls around it

CL- how much oo?  No Multiple inheritance, 

JS- c-style OO.  Common ops across handles

Extensive discussion re: handle objects, which can point to different types of files

TG- How to support multiple backend file types?

SJ- how much of OO are we going to use- no simple inheritance?

DK- structured error handling.  Error handle which can be queried to get all exception
Information

TG- error tree (since a single call can have ?multiple errors? from different tplugins)

TG- style.  Security.

CL?security mech as base part of api?

TG- cmplex if multiple security tokens have to be available

CL- features sppted by api will determine security features exposed

CS- frequent mode- get_auth_token(), provide token to api calls

TG- session handling must support more than one auth type

CL- do we have a style/approach to security?

Consensus: ?session handle?

JS ? explicit management of security via API, or implicitly (user runs grid-proxy-init in shell, then runs program)

TG- next step of agenda- use cases

First: Cover the ?Related groups in GGF? document by Andre and Gab

TG- resource management vs. job startup

AM- document should have been more fine-grained?

Subcategorization of job reosources (job submission, job control, job monitoring, resource discovery, resource description)

What do we mean by data access?
	Files
	Databases

Remote file access

Data management?treat file as object to move around

Databases?information management/lookup.  Remote access to DBs?
Genome sequences.  Life science consortia working on this?

DAIS- unique interface to dbs. Exposed interface is not that complex

Extensive discussion of meaning of ?data X?

Gregor- want to narrow down & focus
JS- data streaming underlying upper level features.  Current apis to do authenticated sockets are crappy

SJ- logical files

Logical file (such as replica catalog)

CS- does steering mean checkpointing?  Sept category?

Identifying ?objects? we will be dealing with-
	Files
	Jobs
	Databases
	Applications
	Stream
	Logical files
	Event
	Advertisement(?)


Should visualization be *specifically* implemented in the API?  No.  App itself does all that; just uses API for things like streaming data

Information- 
publish & search for info

Communication-
	Messaging
	Information encoding/organization (?)

Data access/streaming/events?flavors of communication

Bulk operations?one per each distinct section (job, file, etc) 

Errors, Audits, transactions, workflows:

Error ?objects?

Audit ?objects?
	Log function in api?
	Extensive discussion on meaning of auditing and whether it belongs in the api
	Maybe it should be low priority?
	Still a standing issue

Enabling/disabling logging desirable

CS- Is there standard audit record format?and can we put hooks into api?

Transactions?  Probably not

Workflow?  

CS?workflow, bulk, task ? how related?

Task = interface to define task, asynch run, notification callbacks
Task-> interfaces unique to type of operation
One job at a time; synch and asynch task
Cog kit- task specification, arb. Task, bind to resource
Cog kit- Event model.  Being adopted by GT 4.
Task- when event changes in task, fire something back/throw exception

Workflow- built on top of task.  Doesn?t need its own API.  Just dependent tasks

QoS?lightpaths, LSPs, others (JS) and diffserve.  Application Interfaces- ?I want this much bandwidth?.  Do we want an API for this set of actions?  

CL?leave it till later, but make sure we don?t design ourselves into a corner so we can?t support it


Priorities/TODO:
Files/Data streaming
Jobs/Tasks
Simple hooks for resource discovery
Error handling

Set aside:
Events
Workflow
QoS
Auditing
Information advertising, search
Steering

Considerations
Bulk operations
Security

Eliminate:
Transaction
databases

Agenda for next session:
What objects in different areas?




After lunch

Job APIs

DRMAA
JSDL (language)
Globus (GRAM)
Ssh
Unicore (abstract job object) 
Condor SOAP
GAT
CoGKits (python/java)


Gregor walked us through the CoG Task/Job interface 
(globus-cog-api.pdf and javacog.pdf)

TG- Other APIs:

 

Extensive in-depth discussions about task/job interfaces, use cases.

CS- Platform?s ?lsrun?

CS walked us through the Platform LSF SDK

Discussion of what people are going to do overnight

Jobs

Files
Posix-like I/O







Dec 14 2004
Jobs:

CS read over CoG, GAT API.
Job service object; job description option
Wants more direct mapping between job objects and JSDL
Discussion of relationship between

AM- what about DRMAA?  

CS- very similar to our C api

What does job id mean across APIs?  In ssh?

Job id?go to portal and reference your running 


Data streams:

Need authentication, authorization; not so much call for explicit encryption
Opaque security token
IO handle: read, write, disconnect

Server side- must be able to connect/disconnect, without disturbing simulation

Don?t expose fine-grained stuff (socket options)


Iostream = open(sc, sd)
Send(iostream)
Receive(iostream)
Disconnect(iostream)

Service_open(sd, auth, enc, authorize), read, write, disconnect
Service description opaque in case impl has to deal with firewall, rather than just host and port

Associate ACLs with the service connection

Files:
AM- presented file and directory interfaces
Discussion around iterator vs. full list of files in directory calls

Discussion re: should ?directory object? have state like current directory; does it belong in the service object?

Directory- cp, mv, rm (within and across namespace)  etc

Files- read, write, seek, etc
Extended read/write calls for efficiency

Above is physical file/directory; extend this to logical files/directoryes
New directory methods: add_location, del_location, get_locations(), replicate()
(IE for a replica service)

Locking?should we have operations?

Jobs: where should the general property list for job descriptions come from

CS- JSDL
Extensive discussion of what JSDL is, what it isn?t, and how it maps to SAGA

Extensive discussion & hashing out of the job API

Extensive discussion &hashing out of the secure channel API


 

Weds December 15 2004

Discussing file interface.  Keep POSIX semantics AMAP.
Discussion of various readdir, getdents, other calls

Long discussion on task interface, eventually stopped and
Went back to files

Discussion of async vs. sync routines, what are appropriate things to expose in things like stat() return value?