www.vuemuer.it
Virtualization, the italian way.
Virtualization, the italian way.
May 17th
It happens often to have to do maintenance on Virtual Machines bacause of an exhaustion of computing resources. A web server with many more visitors than those predicted when it was sized, a database server with newly added application, there are many examples.
In thse situations, a sysadmin schedules the activity on the virtual machine in non working hours(usually nights or weekends), stops the VM, changes settings by adding more cpu or ram, and restarts the VM. Even if this task takes only few minutes, it would be possible to avoid it completely by choosing a compatible guest OS and pre-emptively configure the VM to support HotAdd and HotPlug, that is the ability to add (note carefully, ONLY add) RAM or CPU to a running VM without the need to stop it.
Which VMware licenses you need?
These features have been introduced first in 4.0, and is supported only if the virtual machine uses at least virtual hardware 7 and has VMware Tools installed. Even if these two requirements are satisfied, not all vSphere editions has those features, but you will need at least Advanced (vSphere 4.0 e 4.1) or Enterprise (vSphere 5.0).
Which Guest OSes are supported?
Once you have one of the correct vSphere license, you will have to check if the desired Gust OS is among those supported; not all the generally supported OS support hotadding CPU or RAM. I’ve found no official list from VMware about the supported OS, best way of checking is to take a look in the appropriate section in the VM settings (note that CPU Hot Plug has an even smaller support list):
(In my example, a VM with Windows 2003 Enterprise 64 bit supports Memory HotAdd but not CPU HotPlug). Jason Boche has tested several Windows OSes in regards to Hot Add, and wrote a blog post that you can read here.
Enabling HotAdd / HotPlug
Has you can already imagine looking at the previous screenshot, that is the section where you can change HotAdd/HotPlug behaviour, while the VM is powered off.
Adding CPU or RAM
Once the features are enabled, you will be able to add CPU or RAM while the VM is running. To do this you will go in the hardware section of the VM settings, where you will find a modified view of the usual settings:
Increasing the RAM of this VM, you will see the sudden change in the Performance Monitor inside vCenter:
while inside the guest VM you will see the change in RAM size while the uptime counter is going on without any pause or reset:
Why you should use it?
During the initial configuration is easy to enable these features since the VM in not in production yet, it requires only a few more clicks. When the VM is then in production having these features enabled can be a further step in the uptime of the VM, since you can increase RAM or CPU in a suffering VM without the need to stop it.
The best solution is to enable these features directly in a template, in order to have newly created VMs already enabled even if you forget to manually enable HotAdd/HotPlug on every VM.
Final Note: there are some issues regarding some linux kernel/distribution, and also in some Windows version. Check VMware KB.
May 14th
In some previous articles I described Fusion-IO technology and how to install on of their card in a ESXi server. In this post, as many of you were waiting for, it’s time to run some performance tests and see what is able to do this product.
For my tests I used VMware I/O Analyzer, a ready-to-run appliance specifically designed to do benchmarks. It can be installed in few minutes in a ESXi server and it can run several I/O tests on the storage is deployed upon.
Some preliminary notes about my WhiteBox, that is not a real server…
Fusion-IO cards have incredibly high IOPS and latency values thanks to the complete bypass of the disk scsi/sata bus, using instead a direct pci-e connection, and also thanks to other essentials elements like their drivers and the way they intercept system calls to the guest file system they host. To take full advantage of these cards however also other components of a system need to be good enough; one for all, the mainboard chipset. This is why my home whitebox has not reached the nominal values stated by Fusion-IO.
Another key component to obtain the best performances is RAM. Following Fusion-IO tables in their user guide, their drivers need a fair amount of System RAM based on the size of the Fusion-IO card you are going to use. If you are going to write on average at 4k blocksize, the driver is going to use 425 MB for each 80 GB card space. With the IoDrive 640 I’m usin, this translates in 3.32 Gb RAM.
Keep these calculations clear in your mind while you are going to size a server that will host a Fusion-IO card: in this scenario, ESXi is no longer a small footprint OS…
I used pre-defined tests already available in the VMware appliance, and each of them has been executed for 60 seconds. No other Virtual Machine was running during the tests.
I/O Analyzer tests on my WhiteBox
This first test used 512byte blocks, 100% read and 0% random. It was aimed to reach the highest IOPS value, and the Fusion-IO card did its job: 58716 IOPS throughout the test, with an outstanding latency of 0.10 milliseconds, that is 103 microseconds!
This second test was exactly as the first, with only and increase in blocksize to 4k. You can see a drop in IOPS (almost halved) and a rise in latency up to 433 microsecondi; we are anyway talking about values way below a single millisecond.
In the third al last test, I used again 4k blocks, but this time with 50% balance between reads and writes and a completely random access. IOPS dropped a little bit more but anyway stayed on high values, but the latency was once again incredible even in a completely random access, with a final value of 513 microseconds.
Conclusions
Fusion-IO has always stated hyper-low latency is the real goal of their products, much more than IOPS. The numbers I saw in my tests were a complete confirmation of this statement.
May 10th
Nuovo appuntamento con i VMUGIT HandsOnDay, ospite di questa sessione sarà Fusion-IO.
Lo scopo di questo Hands on lab è poter conoscere e lavorare su una delle tecnologie più innovative degli ultimi anni applicata al mondo dello storage e della virtualizzazione: le NAND Flash. Finalmente potrete scoprire e toccare con mano gli ioDrive di Fusion-io: noti sul mercato per essere il punto di riferimento in termini prestazionali ed in particolare per quanto riguarda i tempi di accesso. Fusion-io ha introdotto per prima sul mercato il concetto di NAND in formato PCI-e. In questa sessione testeremo sul campo uno di questi device come datastore di un ambiente ESX e verificheremo tutti i vantaggi prestazionali che sono in grado di fornire.
Agenda
- Introduzione al mondo delle NAND Flash
- SSD, PCIe, File systems e Virtual Storage Layer (VSL)
- Presentazione generale prodotti Fusion-io
- Use Cases nello specifico in ambito virtuale
- Sessioni pratiche
Dove e quando
28 Maggio 2012
Ore 10.00 – 15.00 (pranzi e pause caffè offerti da Fusion-IO)
Hotel Royal Garden di Assago (MI)
Via G.Di Vittorio – 20090 Assago Milano
http://www.monrifhotels.it/RoyalGardenHotel
Come partecipare
La registrazione e’ aperta e gratuita per tutti gli utenti registrati al VMUG. La form di registrazione rimarra’ aperta e disponibile fino al 27 maggio compreso. I posti disponibili sono 30.
Potete registrarvi attraverso la form disponibile su myvmug.com a questo link:
http://www.vmug.com/e/in/eid=463
May 9th
Burlington, MA – May 7, 2012 - StarWind Software Inc., an innovative provider of iSCSI SAN software and VM backup technology, announced today that their solution, StarWind iSCSI SAN, has been nominated as a finalist in The Storage Awards 2012 in two categories – Storage Virtualization Product of the Year and Storage Magazine “One to Watch” Award.
The Storage Awards is an annual event presented by Storage Magazine, a leading data storage publication in the United Kingdom in order to honor the best storage products in the industry. The readers of Storage Magazine vote for, and reward, companies and services that they believe have made significant contributions to the IT industry during the year.
Last year, the 2011 Awards saw more than 6,500 readers vote (32,000+ votes cast in total) for their favorite products, people, and companies.
StarWind has been nominated for its StarWind iSCSI SAN – highly available storage software that delivers an ideal mix of performance, simplicity, and enterprise-class features, such as High Availability, Synchronous Mirroring with Automatic Failover, Remote Replication for disaster recovery, CDP/Snapshots, Thin Provisioning, Global Deduplication etc. StarWind’s software makes it easy to manage and optimize storage capacity, performance, and data protection across the entire IT infrastructure. It ensures complete data consolidation, maximum hardware utilization, and considerable reduction of operation and capital costs (OpEx and CapEx).
“The Storage Awards is a renowned industry award. The nomination as a finalist emphasizes StarWind’s innovative potential and significant results in the area of business continuity and data protection,” said Artem Berman, Chief Executive Officer of StarWind Software. “We are honored to be recognized by Storage Magazine for our virtualization achievements in the IT industry as we continue our efforts to provide high-performance storage solutions that are ideal for the virtualized environment.”
The winners of The Storage Awards will be announced on June 14, 2012 at the Storage Awards Ceremony held at the Grand Connaught Rooms in London. Take a second to check out all of the categories and nominations and take the time to VOTE:
May 7th
Lat week we hosted, as part of VMUGIT HandsOnDay series, Acronis with their VMProtect 7 software.
Acronis is globally known for their TrueImage software, a cloning solution I used in the past like thousands of others with great success. So I’ve been suprised at first to know TrueImage nowadays is only 10% of Acronis incomes, a sign they are moving towards different solutions and markets.
VMProtect is a software developed specifically for virtualized environments, in fact it supports only vSphere and only from 4.0 and above. One of the distinghiusing features of the software, derived from the TrueImage experience and its Universal Restore, is the ability to do convertion from physical machines.
The product can be installed as a Windows software (useful for physical servers) or by deploying a Virtual Appliance (Linux based). This is an interesting option, since it allows fast deployment times and help to save on Windows licenses. Is it possible to install any number of appliances (licensing is based on the number of managed ESXi sockets) and to distribute backup load on the appliances.
At the moment, these appliances cannot be coordinated, so you need to manually created different backup jobs in every appliace and design accordingly your backup jobs and their executions, to avoid jobs from different appliances saving data from the same datastore, thus creating an overload on the datastore itself.
Backup activities, like in any other solution based on VMware vStorage API (VADP), get advantage of CBT and use Hot Add as the preferred method.
Even if are available different backup schemes like classic GFS and others, the preferred methos is called by Acronis “Always Incremental”: inside a unique backup files the software saves every virtual machine of the same job, and all the delta data coming from CBT in the following runs. These blocks are indexed inside the single backup file, that has inside itself all the restore points of the same VM. During a restore you can choose which restore point to use.
This approach is for sure innovative and different from other cuncurrent solutions, one of its pros is the indipendence of the backup file from the software (it can be moved and read by another appliance) but one of it cons is the risk of loosing all the restore points of all the VMs inside the job after a corruption or loss of this file.
Another interesting feature is a vClient plugin, able to manage part of the available features in the native web console. This is useful for those who prefer to have all the administration tools directly inside the vClient, even if it is possible to connect only one appliance.
Another option is the backup of a ESXi (4.1 o 5.0) configuration: for those who do not have VMware licenses with host profiles it can be a solution to save and restore ESXi configuration. For sure an ESXi node can be installed and configured in few minutes, but some sections like networking can be time consuming and boring, especially without distributed switches. Keep in mind tough the saved configuration cannot be “generalized” to clone several ESXi, but only to restore the original server, since all the values like hostname and ip addresses are restored.
Last, for the backup features, Acronis has its own “Cloud” remote backup solution, thanks to some datacenters where you can send vmProtect backups. For Europe, datacenter is located in Strasburg, so it’s compliant with European Community laws about private data ant their phisical location.
About recovery, it can obviously restore the entire VM or single files (File Level Recovery in 2012 is not even a feature to be declared in my opinion, but something to be blamed about if not available) but in this case is not able to restore files in the original position inside a VM, but it exports the requested files in a zip file to be than manually managed.
vmProtect can create specific backups of Exchange Server elements (2003 and later) and to extract than from the backup specific mailbox or email. Those objects are not directly re-injected in Exchange, but Acronis gives you a PST file, and you will then need MAPI libraries to restore the objects yourself.
another feature we have been able to see are replicas, that offer failover and failback as in Veeam Backup. Unlike this competitor, vmProtect cannot save multiple restore points of the same VM, but only the last one and updating it at every run.
As its main competitor, even vmProtect can start a saved VM directly from the backup file without the need to restore it, using the same “trick”: it publishes a temporary NFS datastore showing the VM files and registering both the datastore and the VM in vCenter.
To summarize, vmProtect is a good solution, it has a fair list of functions, some of them different from the competition (scheduling options and backup encryption for example) and other that left me doubtful (the single backup file for all the retention points, or the lack of PowerShell support to automate the tasks). None the less, they are amoving in the right direction and they can push their competitors to offer even better products.
For sure they need to offer some sort of centralized management between the appliances, so to have a coordination of backup jobs and be more attractive to enterprise customers. Its price (399 euros per socket, only 1 version available) has been adjusted in a way to compete in an aggressive market where there are already other ruling competitors, by technology or by market share.
May 4th
Veeam Backup è un prodotto eccezionale, e in moltissimi casi un setup con tutti i parametri di default funziona perfettamente.
Con l’introduzione della versione 6 tuttavia, l’architettura distribuita che è stata introdotta permette ora di progettare strutture e soluzioni di backup molto differenti tra di loro, coinvolgendo tantissimi parametri e scelte tecnologiche da fare, che crescono e si complicano al crescere dell’infrastruttura da dover proteggere.
Una possibile idea, che ho in mente da qualche mese, sarebbe quella di tenere delle sessioni “live” di un giorno, dove più che un corso vero e proprio con un docente e tanti studenti, si passi il tempo disegnando degli scenari tipo (fittizi o derivati da casi reali) e si provi a definire una o più possibili soluzioni, analizzando pro e contro delle varie scelte che potrebbero essere fatte.
Chiamatelo DeepDive, Design Sessions, come volete. Potrebbe essere un’idea interessante?
Fatemelo sapere nei commenti di questo post.
May 2nd
This is a guest post by Massimiliano Moschini, recentyl awarded vExpert 2012!
For those who want to test vSphere FT (Fault Tolerance) but does not have supported processors (have a look at the supported processors in KB 1008027), or if you have a lab made with nested ESXi, yes is it possible to force FT activation.
Disclaimer: thismode is NOT supported and absolutely deprecated in production environments, but is only for test purposes, also there are some problems and unstability. For example FT may not reactivate after a host crash, or sometimes secondary VM is not recreated after the original secondary has become primary.
Anyway, this is what you need to do:
- shutdown the VM
- go to Edit Settings – Options Tab – Advanced/General
- select Configuration Parameters
- add these three new lines with the listed parameters:
Name=replay.allowFT Value=true
Name=replay.allowBTOnly Value=true
Name=replay.supported Value=true
- Save and exit
At this point you will be able to enable FT logging on a VMkernel port on the hosts of the HA cluster, and do your tests.
Enjoy yourselves!
PS: if you are thinking about trying this on a multi-processor VM, forget about it. I already tried it and is not going to work
Apr 26th
This WhitePaper is also available for offline reading, download it here in PDF format.
Veeam Backup & Replication v6 allows users to choose among several different backup methods to fit their need.
Each method has its own strenghs and weeknesses, and there is always a tradeoff between used space on the backup storage and the amount of activity (both from the cpu and the IOPS generated on the storage) required to complete the backup.
However, backup designers often evaluate only the used space, keeping low attention on the impact each chosen method has on destination backup I/O. This is fundamental in the choice of the backup appliance so that backup times do not go over the desired backup window.
We will analyze in this paper the available method and their pros and cons. We will assume standard disk storage is used; different analysis may be required if deduplication storage appliances or other different systems are used.
Forward Incremental Backup
This is the default method when a new backup job is created.
Also indicated simply as “Incremental Backup”, this is by far the most common method used in Veeam Backup projects.
During the first run, Veeam Backup & Replication creates a full backup file (.vbk). At subsequent backups, it only gets changes that have taken place since the last performed backup (whether full or incremental) and saves them as incremental backup files (.vib) next to the full backup.
It’s the best choice if the backup files needs to be moved or saved to tape, since only the new incremental backup file is created, and the previous files are not touched anymore, so are their archive bits.
On the other hand, restore are slower than in other methods (such as Reverse Incremental) because Veeam Backup server needs to aggregate the full backup and its increments to show to the backup administrator the desired restore point.
Also, based on the desired retention policies, an old full backup must be kept until the retention period is reached, resulting in a higher used space if compared to other backup methods.
This is the lowest impacting method on backup storage: for every bit we have to save, 1 IO is used to write it on the storage itself. This makes the forward incremental method the best choice for backups involving low-end NAS.
Reversed Incremental Backup
This is the best choice if the backup storage is space-constrained and if fast restore of the last backup version is the highest goal of the users.
During the first run of a backup job a full backup of a VM is created and stored in a resulting full backup file (.vbk). All subsequent backups are incremental – that is, Veeam Backup & Replication copies only those data blocks that have changed since the last job run. During incremental backup, Veeam Backup & Replication “injects” changes into the .vbk file to rebuild it to the most recent state of a VM. It also creates a reversed incremental backup file (.vrb) containing data blocks that are replaced when the full backup file is rebuilt. Therefore, the most recent restore point is always a full backup, and it gets updated after every backup cycle.
This backup method lets you perform forever-incremental backup and save disk space as you have to store only one full backup. With reversed incremental backup, you do not have to perform periodic full backups to guarantee safety of data and to keep up with the specified retention policy. If the number of restore points allowed by the retention policy is exceeded,
Veeam Backup & Replication will simply delete the earliest reversed increment.
However, this method uses 3 I/O for every written bit:
- 1° I/O to read the bit in the .vbk file that needs to be replaced
- 2° I/O to substitute that bit with the new one extracted from the VM disk
- 3° I/O to write the old bit into the newly created .vrb file
Also, since substitutions comes from different and unpredictable blocks of the production VM, we will have random access to the saved data, and this will even worse the performances of backups if compared to the sequential writes done during a forward incremental.
The real value can be 4x of the original I/O.
Forward Incremental Backup with Syntethic Full
Forward incremental is the best choice for offloading backups to tape, but one of its cons is the lack of a full backup if you need to do a fast restore of the last version of a VM.
To bypass this problem, you can use Syntethic Full: along with the usual forward incremental schedule, you can choose a day to perform a syntethic full. This combines the advantages of forward incremental (tape-friendly) and reverse incremental.
The full version is not created reading again from the production storage, but Veeam Backup & Replication uses the chain of full and incremental backups that are already kept on backup storage to create a new full backup.
Every next backup job run will create an incremental backup starting from the new full backup until next schedule of the new synthetic full.
This method uses 1 I/O for every written bit during the forward incremental, and 2 I/O during the synthetic full:
- 1° I/O to read the bit from the .vbk or .vib files
- 2° I/O to write the same bit into the resultant .vbk file
Forward Incremental Backup with Syntethic Full and Transform
If you select to create a synthetic full backup, you can additionally choose to transform all previous full backup chains to a reversed incremental backup sequence.
Both .vib files as well as the full backup will be transformed to reversed increments (.vrb files), and you will only have a full backup. This option enables you to keep only one full backup image on disk and so reduce the amount of space required to store backups.
However, such transformation takes more time than simply creating a periodic full synthetic backup and is a really heavy operation on both the Veeam Proxy doing the transformation and the backup storage.
From an I/O perspective, the combined use of syntethic full and transform on a forward incremental requires 4 I/O for every bit:
- 1° I/O to read the bit from the .vbk or .vib files
- 2° I/O to write the same bit into the resultant .vbk file
- 3° I/O to read again the bit from the .vbk or .vib files
- 4° I/O wo write the resultant .vrb file
Conclusions
As I have described in the previous scenarios, the method you choose to backup your VMs have a deep impact on the backup storage.
If you take for example a small company, they usually have a constrained budget and they will likely choose a cheap NAS for their backup storage. That appliance will be filled with off-the-shelf SATA disks, and they will setup a raid5 to save space. This design is completely reasonable; however, if their IT guy will choose for example the Reversed Incremental to further save on disk space, there are chances the cheap NAS will perform really bad because it’s not powerful enough to load the stress of 2 I/O per saved bit.
There is not a best choice from the 4 ways you can do a backup with Veeam Backup & Replication, it always depends on your environment: for example if you do not need to offload to tape or off-site and you have fast storages, you can probably go for the Reversed Incremental. But, if you are going to save to a deduplication appliance, reversed is not the best choice, so you will go back to the usual Forward Incremental.
Just like in any (well designed) IT project, you need to think about every piece of the whole design, and the goal of this paper has been to highlight the importance of I/O load on the backup storage: this is definitively one of the elements you need to care about.
Apr 23rd
As I described in a previous article, Fusion-IO cards are not natively supported by VMware ESXi.
After installing the card and restart the server, you can see that it claims there is no persistent storage available, not even a lun or local disk to be formatted:
So, we need to intall Fusion-IO drivers. They are available for ESX(i) 4.0, 4.1 e 5.0. Having driver in .vib format, you can use the usual method like any other third party driver inside an ESXi server, first copying the .vib file inside ESXi and then issuing this command from the command line (or via ssh):
esxcli software vib install -v /tmp/Fusion-io_bootbank_block-iomemory-vsl_2.3.8.24-1OEM.500.0.0.406165.vib
Finished the installation you need to reboot ESXi, and when it comes up again you will have two new elements in ESXi: a new “IOMemory VSL” among the storage adapters, and a 600 Gb local disk (identified by ESXi as SSD):
From here, you can format the disk with VMFS and use it as a “common” local datastore, but also as host cache.
CIM Providers
If you want to monitor the card health and status, the only way to do it directly from ESXi is to install CIM Providers. Fusion-IO, like many other hardware vendors, gives you their CIM providers for several ESXi versions.
To install them in ESXi 5.0, you first need to place your server in maintenance mode.
You will then upload the software into ESXi, and you will install it using this command:
esxcli software vib install --no-sig-check --depot=/path_to_ZIP/fusionio-cimprovider-esxi5-bundle-<version>.zip
(no-sig-check is needed since the software is not digitally signed); you will finally put ESXi out of the maintenance mode.
the Fusion-IO card is now ready to host virtual machines.
Apr 18th
In the new v6 architecture, Veeam Backup & Replication allows the creation of complex and highly performing environments by adding proxies and repositories, instructed and orchestrated by the central Veeam Backup Server. This server however still holds its own proxy and repository roles, created during the installation.
Is it possible to disable those roles, and transform the Veeam Backup in a command console of our backup infrastructure?
Proxy Role
If you ever used several proxies, you noticed how easy is to do maintenance on them simply disabling them:
But if you try to do the same on the Veeam Backup server, you will find out this operation is not allowed:
however, this operation is not possible only via the graphical interface. If you installed Veeam Powershell cmdlets, you can disable the proxy via script with a simple command:
Get-VBRViProxy-Name"VMware Backup Proxy"| Disable-VBRViProxy
Once you did this, our proxy list will show the Veeam Backup proxy in disable state (look at the grey icon):
Then, if you want to reactivate the proxy, you will only need to run:
Get-VBRViProxy -Name "VMware Backup Proxy" | Enable-VBRViProxy
Default Repository
During Veeam Backup installation, the installer creates a default repository in c:backup. Right as in the default proxy, even this element cannot be removed through the GUI:
Even here, you can remove it via Powershell:
Get-VBRBackupRepository -name "Default Backup Repository" | Remove-VBRBackupRepository
In this way you have deleted the default repository.
Once you finished these two activities, Veeam Backup Server has become a pure central console to manage your backup infrastructure.