[jsdl-wg] Process Topology

[Andreas Savva]

Chris,

Thanks for kick-starting this discussion. I'm commenting somewhat out of 
order so I've extracted some of the points here and left the entire text 
from your original email at the bottom of the email:

> ProcessCount (total number of _processes_ that the job will use to
> execute the job)

As I mention in section 8 of the spec ProcessCount is misnamed. (The 
usage of the term 'process' is unfortunate.) It isn't intended to 
control how many actual processes the job will use at execution time but 
to indicate the aggregate processing power that the application needs. 
So I see it as 1-1 mapping to a CPU and I have actually translated it as 
such in the examples of section 8.

> Note: I'm making an assumption here that CPUCount is the number of
cpus that
> I want from the resource, rather than an expression of how many cpus 
> the host needs to have configured. If it is the latter, then we do 
> need TileSize, and replace CPUCount in my examples below with 
> TileSize.

CPUcount is defined as the number of CPUs you want the resource to have. 
So it is the latter one.

> 4. A "homegrown" master/slave parallel job (say a ligand docking
> job).
Wants
> 32 processors. No tiling constraints at all.
	:
 > 4. ResourceCount == 32, CPUCount == 1
 >   -> oops ... it doesn't care about tiling
 >    ResourceCount == 1, CPUCount == 32
 >   -> hmm ... artificial constraint ... would suck on a blade cluster
 >    ResourceCount == 1-32, CPUCount == 1,32
 >   -> oops again ... I might get a total allocation of 32*32 cpus
 >
 >   * there seems to be a gap!
 >
 > If we had a term called "TotalCPUCount" for the entire job, I could do:
 >
 > 4. TotalCPUCount == 32
 >   -> LSF : "-n 32"
 >   -> PBS : "not sure how to express"

Using the current terminology (bear with me) I would translate it as

<Application>
    ...
    <jsdl-posix:POSIXApplication>
	...
	<ProcessCount>32</ProcessCount>
    </jsdl-posix:POSIXApplication>
</Application>
<Resource>
   ...
   <ResourceCount>
      <LowerBoundedRange>1.0</LowerBoundedRange>
   </ResourceCount>
</Resource>

I translate "No tiling constraints" as meaning TileSize=1  and since it 
is the default value I have omitted it.

 > So ... in summary ... I propose:
 >
 > CPUCount (as is if it's allocated cpus per resource)
 > TileSize (iff CPUCount is an expression of configured cpus in a host)
 > ResourceCount (as is ... hmmm ... maybe the default value needs to 
change)
 > TotalCPUCount (how many cpus this jobs needs to run in total)

We need TileSize. I agree that the default ResourceCount=1 definition 
should be changed to  'undefined' as you mention in a subsequent email.

So there are two issues:
1. Whether the topology requirements should be in the Application 
section or not. If they are in the Application section then the terms 
used should not be resource-flavored, i.e., not TotalCPUCount but 
something else.
2. How to rename 'ProcessCount' to eliminate the confusion with the term 
'process'

My answer to (1) is to keep these in the Application section. I am not 
sure how to rename ProcessCount though.

Could I also ask you to let me know if the examples in section 8 of the 
spec actually make sense or not?

Andreas

Christopher Smith wrote:
> Ok ... here are my thoughts on process topology and what's currently
> expressible in JSDL.
> 
> First, I'll list some use cases (they're all parallel jobs):
> 
> 1. Simple MPI job. Wants 32 processors with 1 processor per resource (in
> JSDL, a host is a "resource").
> 
> 2. OpenMPI job. Wants 32 processors with 8 processors per resource.
> 
> 3. An OpenMP job. Wants 32 processors. Shared mem of course, so one
> resource.
> 
> 4. A "homegrown" master/slave parallel job (say a ligand docking job). Wants
> 32 processors. No tiling constraints at all.
> 
> * Note that I'm specifically leaving out the Naregi "coupled simulation" use
> case (sorry guys), since we determined at the last GGF that it was a case
> which could be decomposed into multiple JSDL documents.
> 
> Second ... what is process topology? It provides the user a way to express
> how resources should be _allocated_ given the characteristics of the job
> (usually in terms of IO patterns ... e.g. network communication, disk IO
> channel contention, etc). Thus, it's used when the resource manager is
> _allocating_ the resources, not when the job is being started/launched.
> Therefore, none of the elements used to express process topology should be
> in the POSIXApplication section
> 
> What we have in JSDL now:
> 
> ResourceCount (how many "resources" i.e. hosts I want)
> CPUCount (how many processors _per resource_)
> TileSize (how many processors to allocate per resource as a unit)
> ProcessCount (total number of _processes_ that the job will use to execute
> the job)
> 
> I will argue that ProcessCount is useless for the purposes of process
> topology, since a) it isn't about allocation, and b) there isn't enough
> information to tell me how to start/launch a parallel job. It isn't about
> allocation since it is irrelevant to the scheduler whether I'll be computing
> using threads or processes. It isn't useful for launching because it doesn't
> tell me how to spread the ProcessCount processes given a particular
> allocated topology.
> 
> So that leaves the rest of them.
> 
> TileSize and CPUCount are pretty much the same thing. At least for 80% (or
> more) of the uses I've seen. The only thing that might cause them to differ
> is that I could possibly allocate more than one tile on a host. Given that
> CPUCount is a range and that we could express step values in the range (we
> can express step values in the range, right?), we don't need TileSize any
> more. 
> 
> Note: I'm making an assumption here that CPUCount is the number of cpus that
> I want from the resource, rather than an expression of how many cpus the
> host needs to have configured. If it is the latter, then we do need
> TileSize, and replace CPUCount in my examples below with TileSize.
> 
> So let's see how these map to the use cases.
> 
> 1. ResourceCount == 32, CPUCount == 1
>   -> LSF : "-n 32 -R span[ptile=1]"
>   -> PBS : "-l nodes=32:ppn=1"     (ppn=1 might be the default)
> 
> 2. ResourceCount == 4,  CPUCount == 8
>   -> LSF : "-n 32 -R span[ptile=8]"
>   -> PBS : "-l nodes=4:ppn=8"
> 
> 3. ResourceCount == 1, CPUCount == 32
>   -> LSF : "-n 32 -R span[hosts=1]"  (hosts=1 equivalent to ptile=<-n val>)
>   -> PBS : "-l nodes=1:ppn=32"
> 
> 4. ResourceCount == 32, CPUCount == 1
>   -> oops ... it doesn't care about tiling
>    ResourceCount == 1, CPUCount == 32
>   -> hmm ... artificial constraint ... would suck on a blade cluster
>    ResourceCount == 1-32, CPUCount == 1,32
>   -> oops again ... I might get a total allocation of 32*32 cpus
> 
>   * there seems to be a gap!
> 
> If we had a term called "TotalCPUCount" for the entire job, I could do:
> 
> 4. TotalCPUCount == 32
>   -> LSF : "-n 32"
>   -> PBS : "not sure how to express"
> 
> It basically means to grab 32 cpus, regardless of how they are spread.
> Basically I just need cpus. This is used a whole hell of a lot within our
> customer base. 
> 
> So ... in summary ... I propose:
> 
> CPUCount (as is if it's allocated cpus per resource)
> TileSize (iff CPUCount is an expression of configured cpus in a host)
> ResourceCount (as is ... hmmm ... maybe the default value needs to change)
> TotalCPUCount (how many cpus this jobs needs to run in total)
> 
> -- Chris
> 

-- 
Andreas Savva
Fujitsu Laboratories Ltd