[gin-data] gridftp between sites

[Gregor von Laszewski]

Bill asked me to look into some programs that we developed in the past 
as part to verify the functionality between gridFTP servers between 
different sites.

One of the things we have is

a) a program that can test elementary gridftp functionality between a 
client and a service. This helps the user to test if he has access to 
the services from his preferred client. This was one of the most 
requested features by users before each SC in the past.

b) this program can be run on a server and while using delegation you 
can create a matrix between sites that report on the ability to conduct 
a compatibility tests.

We ran a form of this program before SC04, SC05 to test interoperability 
between the various versions of gridftp servers as well as various job 
execution clients.

Some of the test have a history, this way you could see if this has 
worked in the past. It is easy to publish the result to public_html. As 
the output is all in HTML it can be easily browsed with a browser.

I suggest that we collect some relevant machine names and port numbers 
in order to see if we can run it and to apply for accounts and place 
certs at the relevant locations.

We are in the process of making screenshots and putting up a 
documentation that is easier to find than the one that we have at
http://www.cogkit.org/viewcvs/viewcvs.cgi/src/cog/modules/testing/karajan/README.txt?rev=1.1&content-type=text/vnd.viewcvs-markup


Gregor

PS: A subset of information from our documentation may show you what you 
can customize (please be aware that some text may :


Constant           example value             description

COLOR:FAILED       "#ff4000"                 The background color of the 
table
                                              cell of a failed test
											
COLOR:PASSED       "#ffffff"                 The background color of the 
table
                                              cell of a passed test
											
COLOR:TIMEOUT      "#ffba00"                 The background color of the 
table
                                              cell of a timed-out test

OUTPUT_DIR         "output"                  The local directory in 
which the
                                              output files are created

PUBLISH            false                     Whether to publish the 
output files
                                              on a remote server after 
the tests
                                              are done. Copying happens 
using
                                              GridFTP.

PUBLISH_HOST       "wiggum.mcs.anl.gov"      If publishing is enabled, 
the host
                                              to which files are copied
											
PUBLISH_DIR        "public_html/testing2"    If publishing is enabled, 
the remote
                                              directory in which files 
are copied

TEST_TIMEOUT       60*1000                   The number of miliseconds 
after which
                                              a test is considered to 
have timed-out
	
TEST_FILE          "testfile"                A file used in some of the 
file operation
                                              tests
											
TEST_FILE_DIR      user.home                 The directory containing 
the test file
	
MAX_HISTORY_SIZE   365                       The maximum number of 
samples stored in
                                              the history. Older samples 
may be
                                              discarded in order to 
enforce this
                                              setting.
	
COG_DIR            "{user.home}/cog-4_1_4"   The path to a CoG binary 
distribution.
                                              This is needed for 
indirect tests
                                              (which use command line 
tools rather
                                              than library calls).

3. The hosts file

The hosts.k file contains a list of machines and corresponding services 
on which the tests are run. The format of the file is a flat sequence of 
task:host elements (our manual that we write will have more detail on 
this). It is possible to specify multiple logical hosts for the same 
physical host for the purpose of separating different versions of the 
same services (like for example transferring files between the GT 4.0.1 
and GT 4.0.2 GridFTP servers on the same machine).

4. Included tests and suites

   a) Execution (job submission)

     i) Direct - uses the task:execute call to submit jobs
	
	ii) Indirect - uses the cog/bin/cog-job-submit tool for the submission 
(effectively this would allow testing for GT2, GT3, GT4 submission).
	
   b) File operations
     i) Direct
	  A) Put - copies a file from the local host to the remote host
	  B) Get - copies the above file back from the remote host to the local 
host
	  C) List - lists files in a directory on the remote host
	  D) Rename - renames a file on a remote host
	  E) Remove - removes a file on a remote host
	  F) Exists - tests the existence of a file
	  G) Make Dir - creates a directory
	  H) Is Dir - tests the isDirectory() implementation
	  I) Remove Dir - removes a directory
	  J) Bug - Runs a sequence of operations that used to cause a problem with
          certain combinations of client/GridFTP server versions (a list 
on a
          nonexistant directory/file followed by a simple operation - 
exists())
		
   c) Transfer
     Runs partial third-party transfers from /dev/urandom to /dev/null 
of 1MB of data between all pairs of hosts and displays the total time in 
a table. This test will only work properly with GridFTP servers. The 
transfer time will be noted within a table between all involved hosts.