[ogsa-wg] Re: [cddlm] Basic POSIX Component & template draft

[Steve Loughran]

Hiro Kishimoto wrote:
> Hi all,
> 
> Jun Tatemura and I took a home work assignment about "Basic POSIX
> Component" for BLAST application deployment.
> 
> Attached is my first-cut draft and hope to discuss at GGF17 OGSA EMS
> session.
> Please have a look and give your feedbacks before or
> at the GGF17. Your feedbacks are very important.
> 
> Thanks and see you in Tokyo!


This is really good, and I do agree, we do need a set of foundational 
things to deploy.

One thing I would argue, having learned the lessons of both Ant and 
SmartFrog, is that a consistent model of file system components comes 
first. That is, we need components to model

-files (with a liveness test that verifies the file is present)
-paths (an ordered list of files/directories)
-directories (with components that can create a dir on deployment, 
optionally to delete it and its contents on termination)
-wildcarded sets of files (**/*.csv)
-temporary files and directories
-cached downloads
-text files

I'm attaching the PDF file describing the current smartfrog set of 
components that do this.  The underlying way that this is done is that 
all components that consider themselves to be sources of filenames/paths 
set the attribute absolutePath at runtime. They usually also export an 
RMI interface for RPC-style operations, along side their RESTy state.

A standard set of filesystem components lets you include the operations 
to create and clean up directories, temporary files and other 
housekeeping operations into the workflows of a deployment.

Looking at the Posix proposal, I'd suggest starting with the reusable 
set of filesystem types, including a <fs:mount> component that could 
mount a local or remote filesystem, and which would then pass the local 
location to interested parties as the filesystem:absolutePath attribute.

You absolutely don't need to introduce a new shared datatype 
FileSystemName, because that implies some kind of alternate 
naming/locating scheme outside of what CDL and the runtime has built in. 
Dynamic CDL references are sufficient to crosslink information inside 
the graph of deployed things, and offer flexibility as well as ease of 
management (you see them when you walk the graph).

First, we'd need filesystem types:

<tmp cdl:extends="fs:TempFileSystem">
    <fs:Description> … </fs:Description>
    <fs:DiskSpace>
       <fs:LowerBoundedRange>10737418240.0</fs:LowerBoundedRange>
    </fs:DiskSpace>
</tmp>

<home cdl:extends="fs:Directory">
    <fs:Description>Chris's home directory</fs:Description>
    <fs:dir>/home/csmith</fs:dir>
</home>

When deployed, both of these would add absolutePath as an attribute,
so that tmp/absolutePath woudl resolve to, say, tmp/work01234/ and
home:absolutePath to /home/csmith.


You'd then mount a blast dir, say on an nfs filesys whose URL you provide

<blastfs cdl:extends="fs:RemoteFileSystem">
    <fs:url>nfs://filestore/csmith/blast</fs:url>
    <fs:mountUser>csmith</fs:mountUser>
</tmp>

The database file is under here, and you declare that it must exist. 
Deployment will fail
if it is absent:

<db cdl:extends="fs:File">
    <fs:dir cdl:ref="/tmp/absolutePath" cdl:lazy="true" />
    <fs:filename>db/ncbiblast/est</fs:filename>
    <fs:filemustexist>true</fs:filemustexist>
</db>


To use it, just refer to its path

<blast:database cdl:ref="/db/absolutePath" cdl:lazy="true"/>

Paths could be represented by some list-like element, whose entries are
evaluated at deploy time and then a platform-specific path created as a 
result

<path cdl:extends="fs:Path">
   <pathentry cdl:ref="/blastfs/absolutePath" cdl:lazy="true" />
   <pathentry cdl:ref="/home/absolutePath" cdl:lazy="true"/>
</path>

On Windows /path/absolutePath would become something like
   "\\filestore\csmith\blast;c:\documents and settings\csmith"
while on unix it could be
   "/nfs/filestore/csmith/blast:/home/csmith"

This information now goes down to the jsdl component

<jsdl-posix:Environment name="PATH"
   cdl:ref="path:absolutePath" cdl:lazy="true"/>
<jsdl-posix:Environment name="TMPDIR"
   cdl:ref="tmp/absolutePath" cdl:lazy="true"/>


-Steve







-------------- next part --------------
A non-text attachment was scrubbed...
Name: filesystem.pdf
Type: application/pdf
Size: 125547 bytes
Desc: not available
Url : http://www.ogf.org/pipermail/ogsa-wg/attachments/20060508/8782419a/attachment.pdf 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: java.pdf
Type: application/pdf
Size: 95346 bytes
Desc: not available
Url : http://www.ogf.org/pipermail/ogsa-wg/attachments/20060508/8782419a/attachment-0001.pdf