[gin-data] Re: gridftp between sites
Erwin Laure
Erwin.Laure at cern.ch
Fri May 5 09:12:28 CDT 2006
Hi Gregor,
Many thanks for this input. Will you or Bill be in Tokyo next week? I
think it would be best if we could discuss this in the data mgmt part of
the GIN meeting on Thursday.
Cheers,
-- Erwin
Gregor von Laszewski wrote:
> PS: some screenshots are now at
> http://wiki.cogkit.org/index.php/Testing
>
> Naturally we can make them more pretty, but the focus here was on
> functionality. We could naturally report the results in some xml
> document that you than could than render yourself, but we thought the
> simple html page we have is descriptive enough to demonstrate that we
> meet GIN goals.
>
> Gregor
>
> Gregor von Laszewski wrote:
>
>> Bill asked me to look into some programs that we developed in the past
>> as part to verify the functionality between gridFTP servers between
>> different sites.
>>
>> One of the things we have is
>>
>> a) a program that can test elementary gridftp functionality between a
>> client and a service. This helps the user to test if he has access to
>> the services from his preferred client. This was one of the most
>> requested features by users before each SC in the past.
>>
>> b) this program can be run on a server and while using delegation you
>> can create a matrix between sites that report on the ability to
>> conduct a compatibility tests.
>>
>> We ran a form of this program before SC04, SC05 to test
>> interoperability between the various versions of gridftp servers as
>> well as various job execution clients.
>>
>> Some of the test have a history, this way you could see if this has
>> worked in the past. It is easy to publish the result to public_html.
>> As the output is all in HTML it can be easily browsed with a browser.
>>
>> I suggest that we collect some relevant machine names and port numbers
>> in order to see if we can run it and to apply for accounts and place
>> certs at the relevant locations.
>>
>> We are in the process of making screenshots and putting up a
>> documentation that is easier to find than the one that we have at
>> http://www.cogkit.org/viewcvs/viewcvs.cgi/src/cog/modules/testing/karajan/README.txt?rev=1.1&content-type=text/vnd.viewcvs-markup
>>
>>
>>
>> Gregor
>>
>> PS: A subset of information from our documentation may show you what
>> you can customize (please be aware that some text may :
>>
>>
>> Constant example value description
>>
>> COLOR:FAILED "#ff4000" The background color of
>> the table
>> cell of a failed test
>> COLOR:PASSED
>> "#ffffff" The background color of the table
>> cell of a passed test
>> COLOR:TIMEOUT
>> "#ffba00" The background color of the table
>> cell of a timed-out test
>>
>> OUTPUT_DIR "output" The local directory in
>> which the
>> output files are created
>>
>> PUBLISH false Whether to publish the
>> output files
>> on a remote server after
>> the tests
>> are done. Copying happens
>> using
>> GridFTP.
>>
>> PUBLISH_HOST "wiggum.mcs.anl.gov" If publishing is enabled,
>> the host
>> to which files are copied
>> PUBLISH_DIR
>> "public_html/testing2" If publishing is enabled, the remote
>> directory in which files
>> are copied
>>
>> TEST_TIMEOUT 60*1000 The number of miliseconds
>> after which
>> a test is considered to
>> have timed-out
>> TEST_FILE "testfile" A file used in some
>> of the file operation
>> tests
>> TEST_FILE_DIR
>> user.home The directory containing the test file
>> MAX_HISTORY_SIZE 365 The maximum number of
>> samples stored in
>> the history. Older
>> samples may be
>> discarded in order to
>> enforce this
>> setting.
>> COG_DIR "{user.home}/cog-4_1_4" The path to a CoG
>> binary distribution.
>> This is needed for
>> indirect tests
>> (which use command line
>> tools rather
>> than library calls).
>>
>> 3. The hosts file
>>
>> The hosts.k file contains a list of machines and corresponding
>> services on which the tests are run. The format of the file is a flat
>> sequence of task:host elements (our manual that we write will have
>> more detail on this). It is possible to specify multiple logical hosts
>> for the same physical host for the purpose of separating different
>> versions of the same services (like for example transferring files
>> between the GT 4.0.1 and GT 4.0.2 GridFTP servers on the same machine).
>>
>> 4. Included tests and suites
>>
>> a) Execution (job submission)
>>
>> i) Direct - uses the task:execute call to submit jobs
>> ii) Indirect - uses the cog/bin/cog-job-submit tool for the
>> submission (effectively this would allow testing for GT2, GT3, GT4
>> submission).
>> b) File operations
>> i) Direct
>> A) Put - copies a file from the local host to the remote host
>> B) Get - copies the above file back from the remote host to the
>> local host
>> C) List - lists files in a directory on the remote host
>> D) Rename - renames a file on a remote host
>> E) Remove - removes a file on a remote host
>> F) Exists - tests the existence of a file
>> G) Make Dir - creates a directory
>> H) Is Dir - tests the isDirectory() implementation
>> I) Remove Dir - removes a directory
>> J) Bug - Runs a sequence of operations that used to cause a
>> problem with
>> certain combinations of client/GridFTP server versions (a
>> list on a
>> nonexistant directory/file followed by a simple operation -
>> exists())
>> c) Transfer
>> Runs partial third-party transfers from /dev/urandom to /dev/null
>> of 1MB of data between all pairs of hosts and displays the total time
>> in a table. This test will only work properly with GridFTP servers.
>> The transfer time will be noted within a table between all involved
>> hosts.
>>
>
More information about the gin-data
mailing list