[gin-data] Re: gridftp between sites
Gregor von Laszewski
gregor at mcs.anl.gov
Fri May 5 12:07:36 CDT 2006
I will unfortunately not be there, but if there is a conference call at
GGF or outside of GGF I could phone in.
One thing I like to point out is that we are flexible in modifying what
we have to the needs of the group.
Also I like to point out that it is not sufficient in my view to present
a program that is installed just between the sites. You need in addition
a program that can be run by the users easily on their clients also.
So maybe in my absence Bill or you can put this point into the discussion.
Gregor
Erwin Laure wrote:
> Hi Gregor,
>
> Many thanks for this input. Will you or Bill be in Tokyo next week? I
> think it would be best if we could discuss this in the data mgmt part of
> the GIN meeting on Thursday.
>
> Cheers,
>
> -- Erwin
>
> Gregor von Laszewski wrote:
>> PS: some screenshots are now at
>> http://wiki.cogkit.org/index.php/Testing
>>
>> Naturally we can make them more pretty, but the focus here was on
>> functionality. We could naturally report the results in some xml
>> document that you than could than render yourself, but we thought the
>> simple html page we have is descriptive enough to demonstrate that we
>> meet GIN goals.
>>
>> Gregor
>>
>> Gregor von Laszewski wrote:
>>
>>> Bill asked me to look into some programs that we developed in the
>>> past as part to verify the functionality between gridFTP servers
>>> between different sites.
>>>
>>> One of the things we have is
>>>
>>> a) a program that can test elementary gridftp functionality between a
>>> client and a service. This helps the user to test if he has access to
>>> the services from his preferred client. This was one of the most
>>> requested features by users before each SC in the past.
>>>
>>> b) this program can be run on a server and while using delegation you
>>> can create a matrix between sites that report on the ability to
>>> conduct a compatibility tests.
>>>
>>> We ran a form of this program before SC04, SC05 to test
>>> interoperability between the various versions of gridftp servers as
>>> well as various job execution clients.
>>>
>>> Some of the test have a history, this way you could see if this has
>>> worked in the past. It is easy to publish the result to public_html.
>>> As the output is all in HTML it can be easily browsed with a browser.
>>>
>>> I suggest that we collect some relevant machine names and port
>>> numbers in order to see if we can run it and to apply for accounts
>>> and place certs at the relevant locations.
>>>
>>> We are in the process of making screenshots and putting up a
>>> documentation that is easier to find than the one that we have at
>>> http://www.cogkit.org/viewcvs/viewcvs.cgi/src/cog/modules/testing/karajan/README.txt?rev=1.1&content-type=text/vnd.viewcvs-markup
>>>
>>>
>>>
>>> Gregor
>>>
>>> PS: A subset of information from our documentation may show you what
>>> you can customize (please be aware that some text may :
>>>
>>>
>>> Constant example value description
>>>
>>> COLOR:FAILED "#ff4000" The background color of
>>> the table
>>> cell of a failed test
>>> COLOR:PASSED
>>> "#ffffff" The background color of the table
>>> cell of a passed test
>>> COLOR:TIMEOUT
>>> "#ffba00" The background color of the table
>>> cell of a timed-out test
>>>
>>> OUTPUT_DIR "output" The local directory in
>>> which the
>>> output files are created
>>>
>>> PUBLISH false Whether to publish the
>>> output files
>>> on a remote server after
>>> the tests
>>> are done. Copying
>>> happens using
>>> GridFTP.
>>>
>>> PUBLISH_HOST "wiggum.mcs.anl.gov" If publishing is
>>> enabled, the host
>>> to which files are copied
>>> PUBLISH_DIR
>>> "public_html/testing2" If publishing is enabled, the remote
>>> directory in which files
>>> are copied
>>>
>>> TEST_TIMEOUT 60*1000 The number of
>>> miliseconds after which
>>> a test is considered to
>>> have timed-out
>>> TEST_FILE "testfile" A file used in some
>>> of the file operation
>>> tests
>>> TEST_FILE_DIR
>>> user.home The directory containing the test file
>>> MAX_HISTORY_SIZE 365 The maximum number
>>> of samples stored in
>>> the history. Older
>>> samples may be
>>> discarded in order to
>>> enforce this
>>> setting.
>>> COG_DIR "{user.home}/cog-4_1_4" The path to a CoG
>>> binary distribution.
>>> This is needed for
>>> indirect tests
>>> (which use command line
>>> tools rather
>>> than library calls).
>>>
>>> 3. The hosts file
>>>
>>> The hosts.k file contains a list of machines and corresponding
>>> services on which the tests are run. The format of the file is a flat
>>> sequence of task:host elements (our manual that we write will have
>>> more detail on this). It is possible to specify multiple logical
>>> hosts for the same physical host for the purpose of separating
>>> different versions of the same services (like for example
>>> transferring files between the GT 4.0.1 and GT 4.0.2 GridFTP servers
>>> on the same machine).
>>>
>>> 4. Included tests and suites
>>>
>>> a) Execution (job submission)
>>>
>>> i) Direct - uses the task:execute call to submit jobs
>>> ii) Indirect - uses the cog/bin/cog-job-submit tool for the
>>> submission (effectively this would allow testing for GT2, GT3, GT4
>>> submission).
>>> b) File operations
>>> i) Direct
>>> A) Put - copies a file from the local host to the remote host
>>> B) Get - copies the above file back from the remote host to the
>>> local host
>>> C) List - lists files in a directory on the remote host
>>> D) Rename - renames a file on a remote host
>>> E) Remove - removes a file on a remote host
>>> F) Exists - tests the existence of a file
>>> G) Make Dir - creates a directory
>>> H) Is Dir - tests the isDirectory() implementation
>>> I) Remove Dir - removes a directory
>>> J) Bug - Runs a sequence of operations that used to cause a
>>> problem with
>>> certain combinations of client/GridFTP server versions (a
>>> list on a
>>> nonexistant directory/file followed by a simple operation -
>>> exists())
>>> c) Transfer
>>> Runs partial third-party transfers from /dev/urandom to /dev/null
>>> of 1MB of data between all pairs of hosts and displays the total time
>>> in a table. This test will only work properly with GridFTP servers.
>>> The transfer time will be noted within a table between all involved
>>> hosts.
>>>
>>
>
More information about the gin-data
mailing list