[gin-data] Re: gridftp between sites

[Gregor von Laszewski]

PS: some screenshots are now at
http://wiki.cogkit.org/index.php/Testing

Naturally we can make them more pretty, but the focus here was on 
functionality. We could naturally report the results in some xml 
document that you than could than render yourself, but we thought the 
simple html page we have is descriptive enough to demonstrate that we 
meet GIN goals.

Gregor

Gregor von Laszewski wrote:
> Bill asked me to look into some programs that we developed in the past 
> as part to verify the functionality between gridFTP servers between 
> different sites.
> 
> One of the things we have is
> 
> a) a program that can test elementary gridftp functionality between a 
> client and a service. This helps the user to test if he has access to 
> the services from his preferred client. This was one of the most 
> requested features by users before each SC in the past.
> 
> b) this program can be run on a server and while using delegation you 
> can create a matrix between sites that report on the ability to conduct 
> a compatibility tests.
> 
> We ran a form of this program before SC04, SC05 to test interoperability 
> between the various versions of gridftp servers as well as various job 
> execution clients.
> 
> Some of the test have a history, this way you could see if this has 
> worked in the past. It is easy to publish the result to public_html. As 
> the output is all in HTML it can be easily browsed with a browser.
> 
> I suggest that we collect some relevant machine names and port numbers 
> in order to see if we can run it and to apply for accounts and place 
> certs at the relevant locations.
> 
> We are in the process of making screenshots and putting up a 
> documentation that is easier to find than the one that we have at
> http://www.cogkit.org/viewcvs/viewcvs.cgi/src/cog/modules/testing/karajan/README.txt?rev=1.1&content-type=text/vnd.viewcvs-markup 
> 
> 
> 
> Gregor
> 
> PS: A subset of information from our documentation may show you what you 
> can customize (please be aware that some text may :
> 
> 
> Constant           example value             description
> 
> COLOR:FAILED       "#ff4000"                 The background color of the 
> table
>                                              cell of a failed test
>                                            
> COLOR:PASSED       "#ffffff"                 The background color of the 
> table
>                                              cell of a passed test
>                                            
> COLOR:TIMEOUT      "#ffba00"                 The background color of the 
> table
>                                              cell of a timed-out test
> 
> OUTPUT_DIR         "output"                  The local directory in 
> which the
>                                              output files are created
> 
> PUBLISH            false                     Whether to publish the 
> output files
>                                              on a remote server after 
> the tests
>                                              are done. Copying happens 
> using
>                                              GridFTP.
> 
> PUBLISH_HOST       "wiggum.mcs.anl.gov"      If publishing is enabled, 
> the host
>                                              to which files are copied
>                                            
> PUBLISH_DIR        "public_html/testing2"    If publishing is enabled, 
> the remote
>                                              directory in which files 
> are copied
> 
> TEST_TIMEOUT       60*1000                   The number of miliseconds 
> after which
>                                              a test is considered to 
> have timed-out
>     
> TEST_FILE          "testfile"                A file used in some of the 
> file operation
>                                              tests
>                                            
> TEST_FILE_DIR      user.home                 The directory containing 
> the test file
>     
> MAX_HISTORY_SIZE   365                       The maximum number of 
> samples stored in
>                                              the history. Older samples 
> may be
>                                              discarded in order to 
> enforce this
>                                              setting.
>     
> COG_DIR            "{user.home}/cog-4_1_4"   The path to a CoG binary 
> distribution.
>                                              This is needed for indirect 
> tests
>                                              (which use command line 
> tools rather
>                                              than library calls).
> 
> 3. The hosts file
> 
> The hosts.k file contains a list of machines and corresponding services 
> on which the tests are run. The format of the file is a flat sequence of 
> task:host elements (our manual that we write will have more detail on 
> this). It is possible to specify multiple logical hosts for the same 
> physical host for the purpose of separating different versions of the 
> same services (like for example transferring files between the GT 4.0.1 
> and GT 4.0.2 GridFTP servers on the same machine).
> 
> 4. Included tests and suites
> 
>   a) Execution (job submission)
> 
>     i) Direct - uses the task:execute call to submit jobs
>     
>     ii) Indirect - uses the cog/bin/cog-job-submit tool for the 
> submission (effectively this would allow testing for GT2, GT3, GT4 
> submission).
>     
>   b) File operations
>     i) Direct
>       A) Put - copies a file from the local host to the remote host
>       B) Get - copies the above file back from the remote host to the 
> local host
>       C) List - lists files in a directory on the remote host
>       D) Rename - renames a file on a remote host
>       E) Remove - removes a file on a remote host
>       F) Exists - tests the existence of a file
>       G) Make Dir - creates a directory
>       H) Is Dir - tests the isDirectory() implementation
>       I) Remove Dir - removes a directory
>       J) Bug - Runs a sequence of operations that used to cause a 
> problem with
>          certain combinations of client/GridFTP server versions (a list 
> on a
>          nonexistant directory/file followed by a simple operation - 
> exists())
>        
>   c) Transfer
>     Runs partial third-party transfers from /dev/urandom to /dev/null of 
> 1MB of data between all pairs of hosts and displays the total time in a 
> table. This test will only work properly with GridFTP servers. The 
> transfer time will be noted within a table between all involved hosts.
>