[occi-wg] VM memory ranges and weight (was "XHTML5 Rendering Proposal...")

Wed Nov 11 15:17:50 CST 2009

Andy,

It's not that the hypervisor today allows OSes to grab all the memory, it's that modern OSes today grab all the memory given to them upon startup. AFAIK, when the kernel for all modern OSes start up, they automatically check the quantity of RAM available to them. This is a fixed value and remains a fixed value until the machine restarts. i.e. "Memory is not hot-swappable."

The only hypervisor that I know of that does trying to optimize and aggressively reuse active memory is VMware's hypervisor. That's why it is capable of memory overselling and why it has memory ballooning. However for it to work, it is necessary that VMware tools be installed on the VM OS. As soon as special tools such as this are installed, that VM ceases to use a standard "boundary specification" such as OCCI. It now has tools/code running on it that is tied to a specific hypervisor. That way of the hypervisor managing memory requires the hypervisor to mess with the memory management internals of the OS. 

Does it work? Yes. Is it the most efficient way to manage memory? No. 

That's why I was suggesting the idea below. Using something similar to the idea below, memory gets the same treatment as CPU does on Xen and VMware once you switch to the CPU scheduling mode. It effectively permits Memory Scheduling. The reason I suggested it for OCCI is because it's one solution that permits the development of a common memory management interface between the hypervisor and the VM OS. Without some sort of boundary specification for memory management, Operating Systems will continue to treat the virtual machine as if it is a physical machine to which it has full ownership of (which of course includes all the memory in that VM or physical machine).

The way I see it, we are creating a specification that simply seeks to further what we have today, instead of rethinking what we have today, and offering a better solution as well. By offering both a legacy solution and a better solution we permit evolution of the interface between the VM and the hypervisor. Memory management is one area where we desperately need an alternative to current memory management practices at the operating system level. I say this as the product manager for a cloud computing provider. We have several thousand virtual machines and from what I've observed, memory is the one resource that you define before boot and then all that memory belongs to the VM until shut down. The only exception to this is VMware's solution that I described above. 

Ideally, you shouldn't even have to define how much memory is available to the VM. The VM should simply use what it needs, actively returning what memory isn't in use to the hypervisor. This is how an application functions on top of an OS and is how the OS/VM should function on top of the Hypervisor. The reason we define a memory quantity at all is because the OS running on the VM requires a fixed memory amount. However, there's no reason that this amount couldn't simply be a range that can grow. 

For example, today I can guarantee a client a certain amount of processor and the client can use more than that amount if it is not in use. However with memory, the amount I define is both the minimum and maximum available to that machine. If I give it 4GB, it will have 4GB. But if it needs more than 4GB, it cannot get more than 4GB without restarting. 

In the future it would even be interesting to be able to sell clients memory MB-hours, just like the electric company sells KW-hours of energy. VMs today need a "memory dimming" ability, using only the memory necessary and not the full amount of memory given. 

I don't know if OCCI is the right place for this idea or if the DMTF would better be equipped to define this. Either way, without an interface for the OS to offload memory management responsibility to the hypervisor, we may never see operating systems modified to do so.

With regard to OCCI treating both IaaS and PaaS, I think it is better to divide these in two. There are very very different beasts. IaaS is concerned with providing standardized virtual hardware interfaces. PaaS ideally should be concerned with the creation of a platform optimized for parallelization that overcomes the Von Neumann bottleneck. PaaS solutions may run on top of a IaaS environment, but do not necessarily have to do so.

Andrew

On Nov 11, 2009, at 4:03 PM, Edmonds, AndrewX wrote:

> Hey Andrew
> The assumption in your mail that I see (correct me if I’m barking up the wrong tree) is the hypervisor allows the OS grab all memory and that it is the host OS that has control over what the hypervisor does. However, hypervisors aggressively optimise and reuse inactive memory in a means to do more with less. My view on what you are discussing are more to do with internal optimisations either within the service provider’s domain or within the internal technology domain of hypervisors. As OCCI is a “boundary” specification and protocol it shouldn’t be too concerned with such things. Also when talking of operating systems, with respect to OCCI we fall into a grey area between IaaS and PaaS. What might be worth consideration is the characteristic of “burstability” or in more general terms, “elasticity” in various attributes and that these considerations could pave the way to countering scenarios that you have outlined.
>  
> Andy
>  
> From: occi-wg-bounces at ogf.org [mailto:occi-wg-bounces at ogf.org] On Behalf Of Andrew de Andrade
> Sent: 11 November 2009 17:08
> To: occi-wg
> Subject: [occi-wg] VM memory ranges and weight (was "XHTML5 Rendering Proposal...")
>  
> (new topic: VM memory ranges and weight)
>  
> As I am not super technical, the following may be a bit confusing to understand. I'm going to try my best to explain. Keep in mind that this is not a critique of the XML currently being discussed, but in the way we are defining memory allocation for a VM in general. Here goes:
>  
> I noticed that the XML is defining memory in the following line:
> <li>Memory: <span property="occi:compute.memory">2048</span>
>  
>  
> I have one concern about defining a fix memory amount in our VM specification. The way I see it, allocating a fixed memory amount to a VM caters to a legacy issue associated with Operating Systems. The typical modern operating system for x86 chipsets (Windows, Linux variants, Unix variants etc.) all having one major memory management problem --  they're greedy. When the OS starts up, it first checks the machine it is running on, in the case of a IaaS cloud this is a virtual machine. Upon discovering how much memory the hypervisor has allocated to it, it automatically grabs all this memory whether it needs it or not. It assumes full ownership of the memory. 
>  
> Long-term this is very wasteful of memory, because there is no way (other than the black magic VMware is doing) to return this memory to the hypervisor when the applications running on top of the OS are not using it. 
>  
> i.e. if we have 48GB of RAM on the host and we start up 12 identical VMs with 4GB allocated to each via the OCCI specification, we then can end up with a lot of waste. If one of these VMs needs more than 4GB of RAM, it cannot take this RAM from another VM on the host that is not really using their memory. 
>  
> We should create a specification that allows this legacy memory allocation issue to degrade gracefully:
> -- The specification permits a memory management model for future operating systems, whereby the machine only takes the memory it needs and can get more memory when necessary and also return memory that it is not using. This would most likely best be defined not as one memory value, e.g. 2048MB like in the XML above, but instead would be expressed as a range and a weight.
> -- The typical modern OS today should still be able to get memory via this old legacy model (it gobbles up all the memory allocated to it). For these systems we can simply set the minimum and maximum to be the same value.
>  
>  
> -- The range would specify the minimum and maximum amount of RAM available to the Operating System. e.g. the machine is guaranteed a minimum of 2048 MB of RAM and can use up to  8192 MB of RAM.
> -- In non-multi-tenant environments, i.e. private clouds, the weight would be use to prioritize the machine relative to others as memory is freed up. This would only be necessary when all the host's memory is used and the host owner wants to prioritize some VMs over others. 
> -- In a multi-tenant environment, i.e. a public cloud, all the machines would ideally have the same weight (equal priority) or in some cases the cloud may offer service levels (e.g. a website would pay for a lower service level while a VoIP box would pay for a higher service level).
>  
> I realize that this is a part of the specification that clearly would not be used now or soon, but it permits OS developers to modify/optimize their memory management to best use the host memory in a IaaS cloud environment. This is a chicken and the egg problem. We need to create a specification that permits OS developers to make this change, so they have a reason to improve OS memory management procedures at the kernel level. If we don't do this, it's very likely that the modern OS will continue to function as a very resource-expensive runtime environment.
>  
> I wouldn't be surprised if in the future this permits the creation of memory management algorithms at the hypervisor level analogous to the way TCP/IP and FCP congestion algorithms manage competition for a finite amount of  bandwidth.
>  
> Feel free to break off the above discussion into a new discussion thread if necessary.
>  
> Andrew de Andrade
> Product Manager - Cloud Computing
> Locaweb (São Paulo, Brazil)
>  
>  
>  
> On Nov 11, 2009, at 1:24 PM, Gary Mazz wrote:
> 
> 
> Nicely done...
> 
> This proposal provides OCCI with significantly increased options while 
> maintaining support for more traditional representation methods. If 
> adopted, this proposal could be used to establish baseline for a 
> capabilities road map without requiring significant industry investment 
> in custom technologies.
> 
> cheers
> gary
> 
> Edmonds, AndrewX wrote:
> 
> Hi all,
>  
> From the previous discussions had and agreeing in principle to OCCI providing a XHTML5 rendering, I'm submitting the following proposal [1] for the group's review. XHTML5 is used to make the representation machine readable and as it is well-formed any XML toolset can digest and manipulate it. Complimenting XHTML5 is the use of RDFa which is used to demarcate information that has semantic significance and in our particular case, used to expose attributes defined by the OCCI specification. The combination of XHTML5 and RDFa is a W3C recommendation [2]. Your thoughts are welcome as ever.
>  
> Regards,
>  
> Andy
>  
> [1] Attached; source available here: http://code.google.com/p/occi/source/browse/docs/occi-xhtml5.xml
> [2] http://www.w3.org/TR/rdfa-syntax
>  
>  
> Andy Edmonds
> skype: andy.edmonds
> tweets: @dizz
>  
>  
> -------------------------------------------------------------
> Intel Ireland Limited (Branch)
> Collinstown Industrial Park, Leixlip, County Kildare, Ireland
> Registered Number: E902934
>  
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>  
> ------------------------------------------------------------------------
>  
> _______________________________________________
> occi-wg mailing list
> occi-wg at ogf.org
> http://www.ogf.org/mailman/listinfo/occi-wg
>  
> 
> _______________________________________________
> occi-wg mailing list
> occi-wg at ogf.org
> http://www.ogf.org/mailman/listinfo/occi-wg
>  
> -------------------------------------------------------------
> Intel Ireland Limited (Branch)
> Collinstown Industrial Park, Leixlip, County Kildare, Ireland
> Registered Number: E902934
> 
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
> _______________________________________________
> occi-wg mailing list
> occi-wg at ogf.org
> http://www.ogf.org/mailman/listinfo/occi-wg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.ogf.org/pipermail/occi-wg/attachments/20091111/41483e81/attachment.html 