The separate tools used to manage the LAN, SAN, blade server connectivity, NICs and HBAs also contribute to data center management complexities. Often administrators can see only what is in their direct line of responsibility and don’t have the overall view of the entire network environment. Now, imagine if they could:

• Logically eliminate the management of multiple switching layers
• Apply policies and manage traffic across many physical switches as if they were one switch
• Scale network bandwidth without manual reconfiguration of switch ports and network policies
• Provide a single, customized view of network status available to server, network, and storage administrators

Ethernet data center bridging and its new Ethernet fabric allows organizations to broaden the sphere of application mobility, provide VM awareness, and optimize server resources for applications just as it improves networking for storage.

Using this new data center bridging Ethernet fabric, organizations can simplify network architecture, more rapidly scale their networks, and significantly reduce data center management overhead. This is still a work in progress, but here at ASG at least, we see converged enterprise DCB Ethernet networks as the future, and we’re focused on helping companies get there as easily as possible.

Reliability

Despite trends in NAS virtualization and other appliance-based virtualization, nothing beats the rock solid reliability of Hitachi’s VSP—probably its top feature. Any growing enterprise with mission critical data should take a close look at the VSP for this reason. The VSP’s unique controller architecture can handle multiple cascading controller failures without going down, with very limited performance degradation. Software upgrades can be done without server reboots, which makes it the least disruptive to applications during SW upgrades.

One of our clients recently implemented the Hitachi VSP for this very feature. You can read about the full project in the CRN article, Denali Deploys Virtual Desktops To Give Doctors More Time With Patients. ASG handled the Hitachi VSP portion of this project. Here’s a quick summary from our own Pacific Northwest Regional Architect, Grant Loveridge:

“Seattle Children’s Hospital was having issues with their virtual desktop infrastructure (VDI) system going down intermittently, and VDI plays a very important role at Seattle Children’s Hospital.  Doctors and nurses rely on certain information to make critical decisions about patient care, etc.  In short, the system just can’t go down! In the process of rolling out their VDI deployment, the customer quickly realized their infrastructure was not robust enough. There were some concerns about the backend storage not being able to handle the demands and patient information not being available to the hospital staff.  Plus, the hospital operates 24 x 7, which means maintenance windows are very difficult to schedule.”

“The customer purchased the VSP to provide the reliability and scalability they needed—with just enough capacity to support the VDI pilot. The customer was able to get started with the VSP at an affordable level, using emergency funding.  All upgrades were accomplished with no downtime to users or applications. They have since scaled the system to support many more users and applications.  They also purchased a second VSP for disaster recovery and are using Hitachi Universal Replicator between sites—a project that is now underway. All in all, the customer is very happy with the VSP solution.”

With these benefits, I imagine the popularity of the Hitachi VSP can only continue to grow. I’ll be sure to keep you updated on its progress. If you have any questions about the VSP or our other Hitachi solutions, feel free to contact us.

About the Author
Anthony Sayre is an Advanced Product Specialist at ASG and our resident expert for Hitachi Data Systems solutions. You can contact him at asayre (at) virtual.com.

A quiet storm has been brewing at Advanced Systems Group. The Hitachi Data Systems Virtual Storage Platform (VSP), Hitachi’s flagship enterprise storage solution, has been quietly taking the data center world by storm. The Hitachi VSP can only be described as a smash hit since its introduction a year ago. With a full sales pipeline and 24 units sold or in process of implementation (that’s a lot—trust me), the VSP is fulfilling mid-range to enterprise needs for mission critical primary storage, disaster recovery, virtual desktop infrastructure, public and private cloud, archival and more.

We can’t sell VSP solutions fast enough, for two reasons: the scale/consolidation abilities of the VSP and its unmatched reliability. Today, I’ll just discuss the first of these features—scale/consolidation.

Scale/consolidation

The VSP can be a diskless storage virtualization engine, and/or it can scale up to over 2000 2.5” disk drives. Plus, it can virtualize up to 255 petabytes (!) of external storage, which is key to Hitachi’s ability to scale and consolidate older legacy systems under one management tool. Here at ASG, we’re witnessing first-hand just how powerful the scale/consolidation story can be. Here’s an example:

One of our clients, a Colorado based e-marketing company, provides many cloud services to fortune 500 companies around the world. As new projects for their clients come up, they often need to move data on the fly from lower to higher storage tiers, run the project, and then move the data back. Without Hitachi’s storage virtualization features like tiered storage management, this wouldn’t be possible without hiring many more IT storage administrators, taking systems down, migrating data, and booting them back up again—a nightmarish task for any data center manager, but especially in this company’s multiple petabyte data center.

ASG engineers architected a VSP solution that’s in the implementation process now, and will consolidate the customer’s 1.3PB of storage spread across 111 devices to 1.6PB across just 11(!) devices in a streamlined virtualized storage environment with automated tiering and DR. This wouldn’t have been possible without the VSP’s consolidation and massive scaling abilities.

In my next post, I’ll discuss the second of the VSP’s most desired features—its outstanding reliability. In the mean time, feel free to share your own Hitachi success story in the comments below. Feel free to contact me if you’d like to learn more about the VSP or our other Hitachi solutions.

About the Author
Anthony Sayre is an Advanced Product Specialist at ASG and our resident expert for Hitachi Data Systems solutions. You can contact him at asayre (at) virtual.com.

Network World Editor in Chief John Dix recently assembled a panel of experts to discuss What’s Next with Hypervisors? Personally, I don’t believe there’s a one size fits all hypervisor available. I think that we should be choosing the hypervisor that is most compatible today with the application you are going to run. The first question to ask is “What problem are you trying to solve?”

Here’s a table with a few ideas to help you get started:

We’d love to hear what you think. What hypervisors are you running today and what are you finding?

Join us, along with NetApp, at the 20^th annual CISD Technology Conference and Expo.  This Council of Information Services Directors (CISD) show will be held on October 10-11 at the Crowne Plaza in Baton Rouge, Louisiana.

NetApp will be highlighting their FlexPod™ for VMware Infrastructure Solution. With the innovative FlexPod solution, you can move your data center to the cloud with a prevalidated, fully-tested architecture solution that utilizes the leading technologies from NetApp, VMware, and Cisco. It simultaneously supports a variety of mixed application workloads and can be optimized for virtual desktop or server infrastructures, multi-tenancy, or private/public cloud computing environments.

You can read one of our previous blogs on FlexPod or check out NetApp’s FlexPod for VMware solution brief.

We hope to see you at the show next week!

Backup and disaster recovery solutions for virtualized technology environments are getting better, and we can expect them to continue to improve and evolve. The industry has recognized the challenges of implementing backup and disaster recovery in virtualized environments. As a result, virtualization technology vendors and their supporting supplier ecosystems are introducing better tools to handle various needed tasks. Moreover, some third parties have developed a comprehensive understanding of the challenges presented by different IT environments and have uncovered best practices to handle them.

Backup and disaster recovery is critical. With the progress the industry has made and the solutions now available, organizations can move forward with virtualization technology confident that they can effectively protect their critical data and quickly recover it when needed.

Here are 9 technical recommendations for virtualization backup and disaster recovery:

• Set hard limits—no more than 15 VMs per physical host, # vCPUs < = # pCPU Cores
• Beware of SCSI reservations. They’re used for specific operations when metadata changes are made, and they prevent multiple hosts from concurrently writing to the metadata. While SCSI Reservations are necessary to avoid data corruption, they also degrade the performance of virtual servers
• Limit the number of running snapshots. Snapshots grow in 16MB increments, and each time they grow, they cause SCSI reservations
• Only vMotion a single VM per LUN at any one time
• Only cold migrate a single VM per LUN at any one time
• Don’t power on/off too many VMs simultaneously
• Limit VM/template creations and deployments to a single VM per LUN at any one time
• Consider using smaller LUN sizes (<600GB) and don’t use extensions to extend a VMFS volume
• Verify backup operations are complete

Server virtualization technology delivers clear benefits. But it also presents a number of challenges, including backup and disaster recovery solutions. While the traditional methods for backup will work, there are much better options.

With that said, here are six best practices to help you successfully migrate your servers and applications from physical servers to virtual servers—while establishing the most efficient, long-term implementation of your VMware infrastructure possible.

1. Create operating system templates. The Physical to Virtual (P2V) migration capability from VMware automates and simplifies physical to virtual machine conversions, though it can ultimately waste your virtual resources. With P2V, physical servers’ devices (e.g. serial bus) and associated drivers are replicated to the virtual server even though many of them may be unnecessary or unavailable. Additionally, P2V creates virtual server images with the same physical attributes as the physical servers, often resulting in an over-allocation of resources. Instead, use the VMware capability to create image templates of different operating systems. Later, you can use these templates to deploy a new virtual server in a matter of minutes.

2. Create single processor images. As you create operating system templates, make sure they’re single processor images. Guest operating system images share physical processors, so in most cases, virtualized single processor guest operating systems perform much better than multi-processor guest operating systems. For example, a multi-processor guest operating system will be scheduled for time slices on multiple physical processors, increasing scheduling time and resources utilized—which adversely affects the resource pool available to the other guest operating systems.

3. Minimize the resources allocated to virtual servers. To achieve the maximum performance levels, minimize the resources allocated to virtual servers. Deliberately set the processor and memory resources low. Then closely monitor the guest operating system resource utilization over an initial evaluation period. Increase the processor and memory resources only when necessary, preventing allocation of unneeded memory and processor capacity to a virtual server. This also prevents unneeded consumption of resources that you could make available to other virtual servers.

4. Be cautious of SCSI reservations. SCSI reservations are used for specific operations when metadata changes are made, and they prevent multiple hosts from concurrently writing to the metadata. You might see this situation when you:

• Create or delete a VMFS datastore
• Expand a VMFS datastore onto additional extents
• Power on or off a Virtual Machine (VM)
• Create a new VM
• Migrate a VM with vMotion
• Grow a snapshot file or a thin-provisioned Virtual Disk

While SCSI Reservations are necessary to avoid data corruption, they also degrade the performance of virtual servers. To help minimize SCSI reservations, you can:

• Only use vMotion to migrate a single guest operating system per LUN at any one time
• Only cold migrate a single guest operating system per LUN at any one time
• Avoid powering on/off too many VMs simultaneously
• Limit template creations and deployments to a single creation per LUN at any one time
• Limit the number of running snapshots—snapshots grow in 16MB increments, and each time they grow, they cause SCSI reservations

5. Keep operating system and application data sets on the same volume. In a virtual server environment, you should keep both the operating system and application on the same volume. Since all of the disk space in the virtual server environment is in the storage array, RAID protection is provided and multiple physical disks are available to the storage volume for performance. A single volume also eliminates the need for consistency groups across multiple volumes, ensuring that the operating system and application snapshots are performed at the same instant. And keeping both the operating system and application on the same volume simplifies the snapshot and recovery process.

6. Keep at least one physical active directory server. Be sure to keep at least one physical active directory server. If your data center experiences a disruption that could disrupt the entire ESX infrastructure, you can bring this server up before the virtual servers. This way, you’ll avoid the manual process of managing the order in which you start the virtual servers, and you’ll ensure your virtual servers are able to complete their boot process.

Hopefully, with these best practices, you can successfully achieve a virtualization implementation that maximizes your existing resources while providing a scalable, available, and reliable infrastructure.

In traditional Layer 2 Ethernet networks, organizations create highly available networks by designating paths as active or standby using STP. While this provides an alternate path, only one path can be used at a time, which means that network bandwidth is not well utilized. Since one of the goals of server virtualization is to increase utilization of the physical server, increased utilization of network bandwidth should also be expected.

To increase network utilization, Multiple Spanning Tree Protocol (MSTP) and similar protocols allow for separate spanning trees per VLAN. While this improves bandwidth utilization, the STP limit of one active path between switches remains. And, because traffic paths are manually configured with MSTP, complexity increases.

Another challenge with STP is network behavior when links fail. Links do fail, and when that occurs, the spanning tree needs to be redefined. This can take anywhere from five seconds with Rapid Spanning Tree (RSTP) to several minutes with STP—and this situation can vary unpredictably even with small topology changes. Furthermore, the demands for non-stop traffic flow increases with server virtualization technology, and, consequently, network convergence times must shrink accordingly. STP does not provide an adequate solution for these issues. Finally, when a redefined spanning tree is reconverging, broadcast storms can occur and result in network slowdown. All of these limitations of STP are why Layer 2 networks typically are kept small in the data center.

What’s needed are Layer 2 networks that:

• • Are highly available
  • Guarantee high-bandwidth utilization over equal-cost paths
  • Don’t stall traffic when links are added or removed due to failure or network reconfiguration
  • Make latency deterministic and is lossless
  • Can transport IP and mission-critical storage traffic over the same wire

This becomes even more important when an application is running in a VM rather than on a physical server. Since the VM is not tied to a specific physical server, it can move between physical servers when the application demands change, when servers need to be maintained, and when a quick recovery from a site disaster is necessary.

VM mobility should occur within a cluster of physical servers that are in the same IP subnet or Ethernet VLAN for the migration to be non-disruptive to client traffic. Otherwise, changes in the IP subnet are necessarily disruptive. As noted in the discussion of STP limitations, the sphere of VM migration also can be constrained. The solution for flexible VM mobility is a more scalable and available Layer 2 network with higher network bandwidth utilization.

For a VM to migrate from one server to another, many server attributes must be the same on the origination and destination servers. This extends into the network as well, requiring VLAN, Access Control List (ACL), Quality of Service (QoS), and security profiles to be the same on both the source and destination access switch ports. If switch port configurations differ, either the migration pre-flight will fail or network access for the VM will break. Organizations could map all settings to all network ports, but that would violate most networking technology and security best practices. The distributed virtual switch in VMware vSphere 4 addresses some of these issues, but at the cost of consuming physical server resources for switching, added complexity in administering network policies at multiple switch tiers, and a lack of consistent security enforcement for VM-to-VM traffic.

Automation provides only part of the answer. With automated VM migration, network administrators will have limited visibility to the location of applications. This makes troubleshooting a challenge, and pinpointing issues to a specific VM will be like finding the proverbial needle in a haystack. Now, consider again a Layer 2 network that:

• • Places no physical barriers in the way of VM migration
  • Is aware of VM locations and consistently applied network policies
  • Does not require manual intervention when a VM moves
  • Removes the overhead of switching traffic from the hypervisor for maximum efficiency and functionality and supports heterogeneous server virtualization in the same network

The new Ethernet fabric brought about by Ethernet Data Center Bridging allows organizations to broaden the sphere of application mobility, provide VM awareness, and optimize server resources for applications just as it improves networking for storage.

Ethernet utilizes upper level layer protocols (TCP) to manage end‐to‐end data delivery and integrity. When the amount of data entering the network exceeds network capacity, Ethernet networks become over‐subscribed and will drop data packets in certain circumstances, which results in lost data.

Fibre Channel (FC), by comparison, provides a buffer‐to‐buffer credit that ensures packets will not be dropped due to congestion in the network, making it lossless. Ethernet can be made lossless only by adopting higher level protocols such as TCP/IP, which have adapted to the intent of IEEE 802-based networks by incorporating end‐to‐end congestion avoidance and flow control algorithms. Ethernet is an 802-based network.

To deliver traffic differentiation and QoS with Ethernet, the existing IEEE Ethernet 802.1p/Q standards provide classification of traffic flows with 3-bit tagging. Network equipment devices (like bridges and routers) use this classification to put different classes of traffic into different queues, while the standard specifies strict priority scheduling of these queues. This allows higher priority traffic to be serviced before the lower priority queues, thus achieving lower latency and also lower drop probability for priority traffic. However, this also creates unfairness issues for other queues because higher priority queues use more bandwidth, effectively starving the lower priority queues. Although the standard does permit the use of other scheduling algorithms, the behavior isn’t specified—which means no single implementation exists, producing interoperability issues.

Enter Data Center Bridging, an architectural extension designed to improve and expand the role of Ethernet in the data center. Data center bridging allows organizations to logically manage networks end-to-end with QoS through four capabilities:

1. Congestion Notification (CN)—provides end-to-end congestion management
2. Priority‐based Flow Control (PFC)—provides a link-level, flow-control mechanism
3. Enhanced Transmission Selection (ETS)—provides a common management framework
4. Discovery and capability exchange protocol—conveys the capabilities and configuration of the above features to ensure a consistent configuration across the network

Data Center Bridging networks can be characterized by limited bandwidth‐delay and limited hop‐count. When multiple traffic types are transmitted over a single link, there needs to be assurance that each traffic type obtains the bandwidth that has been allocated for it, while at the same time, conditionally restraining any traffic types from exceeding their allocated bandwidth.

When multiple traffic types—LAN, SAN, IPC—are consolidated onto a single converged link, there is no inherent prioritization of traffic between these types; however, each traffic type needs to maintain its current usage model of a single interface with multiple traffic classes supported. Each type also needs to maintain its bandwidth allocations for a given virtual interface (VI), independent of traffic on other VIs. Data Center Bridging physical links provide multiple virtual interfaces for different traffic types.

The new features and capabilities of Data Center Bridging will need to operate within and across multiple network domains with varying configurations. Achieving interoperability across these environments requires that link partners exchange information about their capabilities and configuration and then select and accept feature configurations with their link partners. This is accomplished through the four capabilities (CN, PFC, ETS, Discovery and Exchange) noted above.

These capabilities, in conjunction with other Data Center Bridging technologies, enable support for higher layer protocols that are loss-sensitive while not affecting the operation of traditional LAN protocols utilizing other priorities. Converged enterprise data center networks, however, must do more than provide QoS for mixed traffic. Data center networks must address the needs of networked storage and the scaling of virtual server networks.

That’s what we’ll discuss in our next blog, so be sure to come back!

Our last blog post covered the security concern. This blog post addresses the performance and scalability concerns with virtualization technology.

Performance

Another aspect of virtualization technology shrouded in misinformation is performance; converting your underutilized, over-built systems to virtual servers does not compromise their performance. In fact, performance on virtual servers can exceed those of physical servers though optimization. Plus, the overhead of virtualization technology is decreasing and Red Hat Enterprise Virtualization (RHEV) technology is getting better at handling I/O-intensive workloads, such as guest environments that run database and CPU-intensive applications.

KVM workloads take full advantage of virtualization-enhanced, multi-core CPU technology. Red Hat benchmarks have demonstrated that the highest computing workloads (SAP, Oracle, Exchange and Java) experience performance that is 90 percent, or greater, on KVM. Some workloads, Linux/Apache/MySQL/PHP (LAMP) workloads, for example, achieve up to 140 percent higher performance than physical machines. Tapping into virtualization technology’s performance-enhancing wellspring could give your applications the boost they need.

Scalability

KVM’s multi-core technology exploitation also makes it far more scalable than adding more underutilized physical machines to your data center. Remember, scalability is less a question of how many systems you can spin up, than it is how well your systems perform under increased loads.

RHEL 6 virtual servers handle workloads quite well in stressed environments. For example, a 100 physical server application farm will scale significantly and perform admirably with fewer virtual systems. That’s because those servers can likely seek a smaller footprint once converted to a virtual server format. With the more efficient and hardware-optimized virtual infrastructure, you could probably even reduce the number of supported servers by 30-50 percent.

In terms of energy efficiency, RHEL 6 has no difficulty powering new servers with 64 processor cores and 2TB of memory. Actually, RHEV supports a theoretical maximum of 16TB of physical memory (assuming anyone wanted—and could afford—to build a RHEL server with such specifications). The operating system can automatically put unused cores into a low-power state until needed, thus conserving valuable energy and reducing power costs.

Moving Forward

Overall, Red Hat is still a little newer to the virtualization technology field than most of the other major vendors, which makes it one of the smaller players in the virtualization technology market. Considering its short history, it’s not surprising that many industry experts questioned Red Hat’s decision to switch its hypervisor from Xen to KVM just a couple of years ago. (Red Hat originally used the open source Xen hypervisor as its virtualization technology engine, until 2008 when it acquired Quamranet, creator of the KVM open source hypervisor.)

Though still a relatively new development, Red Hat’s decision to integrate KVM into RHEL was a strategic move—particularly when considering how KVM has improved some of the issues of P2V. Of course, most service providers continue to run Xen—which lends itself well to virtual private server deployments and has a longer track record in the market. However, KVM is a veritable virtualization technology. A growing number of companies believe it has matured enough that they feel comfortable deploying it in production. Although it’s unlikely to replace Xen altogether, KVM will provide an attractive alternative to Xen for many companies as it continues to develop.