Hyper-V Server 2012 Cluster with Powershell Deployment Toolkit

I recently came across a lovely show on Channel 9. It talks about setting up a simple Hyper-V Server 2012 cluster for use in a lab or test environment or whatever. I won’t go over the details, watch the show, it’s great! In addition to that I had come across an article on the Building Clouds Blog, about the PowerShell Deployment Toolkit. So over Memorial Day weekend I decided to stand up my cluster and spin up a test environment similar to what I use at work.

In my environment I have 6 servers, I have 3 set aside for Hyper-V, one is my firewall, one is a Domain Controller and the last is a management server. I’m using my DC as the file server as well. I didn’t need the iscsi target stuff, as I’m using Windows Server 2012 and used the new File and Storage Services to configure my iscsi drives.

I decided to let vmcreator.ps1 build the vm’s for me, originally I had spun up my own, but I was having difficulties getting the installer to work properly. Turns out that there is a requirement that the PDT tools be run from the C: drive of your computer. Also if you’re running them from the server OS, you will need to install the Hyper-V role in order vmcreator.ps1 to function properly. I don’t recall seeing either of those things mentioned in the TechNet article, but I may have overlooked that part.

So, linked from the vmcreator.ps1 article is a great utility, Convert-WindowsImage.ps1 that I used to create my base OS image. The utility is super handy and has a gui or cmdline version. I wimped out initially and used the gui version, pointed it an ISO of Windows Server 2012 and after a while I had a lovely vhdx ready for vmcreator.ps1.

After renaming the half dozen vm’s the script had created for me, in record time btw, I ran the installer.ps1. There’s not really a whole lot mentioned on the article about it’s use, it is rather self-explanatory and once you realize the limitation to the C: drive then it’s a no-brainer. That part took me a bit to figure out as I had an external drive with all the bits the downloader.ps1 had downloaded for me.

The end result is I now have the basic System Center infrastructure that I can play with locally to try out new features, or test the scripts and apps I create for work. It was really very slick, and I could totally see how I would use something like this in our QA environment at work.

 

RedHat 6 Enterprise + VMware VSphere 4.0

Odd thing happened today while I was setting up a server for someone. The RHEL 6 install went just fine, it found the network card, asked if I wanted to configure it via DHCP and all that. When the server rebooted, there was no eth0. When I ran ifconfig eth0 up, the adapter showed up, but then wouldn’t get an IP from DHCP. A quick look at the messages showed the following

rhel6-nonic

No broadcast interfaces found?

Unhandled state?

Now I admit that I’m not much of a Linux admin, I have setup and done some cool stuff, but it’s just not my forte. So I resorted to Google, when Nick and Carson didn’t the answer for me!

I found this thread. In it the user stated that if he ran dhclient eth0 the adapter would go, so I did that and sure enough I got an IP address. When I started poking around I noted that in /etc/sysconfig/network-scripts/ifcfg-eth0 that ONBOOT was set to NO, I changed this to YES. Then to verify that it was working properly, I stopped the network service, shutdown the computer, and moved it to a different VLAN.

When the server came back up, it had an IP address, I’ve never had RHEL do that before, and I’ve done several installs, so I guess it’s a fluke? But if not, at least I’ve documented how I got it to go if it happens to me again!

HA/FDM fails to restart a virtual machine with the error: Failed to open file /vmfs/volumes/UUID/.dvsData/ID/100 Status (bad0003)= Not found

This came across my newsfeed last night and this morning, and before I lose the links I thought I’d post them up here.

VMware KB article

Description of the Problem

Perl script to detect and resolve

PowerShell script to detect and resolve

Preparing to upgrade DC’s to Windows Server 2008 R2

We decided it was time to move to R2 on the domain controllers, while we’re at it we also moved up to Windows 2008 Functional. Sadly we can’t go to 2008r2 functional until our last 2008 DC goes out of warranty, in 2014! Oh well, maybe something will happen to it…

Fairly straightforward process

  1. Transfer FSMO roles
  2. Take a VM snapshot
  3. Perform the DCPromo
  4. Uninstall ADDS binaries
  5. Export Logs
  6. Reset computer account
  7. Install Windows 2008R2 Core

Well this morning Boe Prox posted a nice function to poshcode, Get-FSMORoleOwner. This function returns what server owns which roles and it was very helpful as I performed my transfers. Originally I had wanted to use Powershell to perform my transfers, but without ADWS the built-in ActiveDirectory module wouldn’t load. So I performed the transfers the old-fashioned way, using the MMC. After each transfer I would run Get-FSMORoleOwner to verify that the role had been moved. This all went without a hitch as expected, then came the Domain Functional Level.

No big surprises here, we did a little digging and found that moving to 2008 Functional level was equivalent to staying at 2003. There are a few new features in 2008, the most interesting one for me at least, is the Interactive Logging. Raising the functional level was a breeze, and all domain controllers reported the same level within minutes.

Since I’ve been moving away from VBscript and over to PowerShell I decided to take my VM snapshot using the PowerShellCLI from VMware.

New-Snapshot -Name Pre-Demotion -VM dc1 -Description “Snapshot prior to demoting.”

In order to backup all the logfiles I threw together a quick little function Backup-EventLogs. It takes the name of the computer and uses Get-WinEvent to get all the logs available. It then writes out each log where the RecordCount is greater than 0 to a csv file, using Export-CSV. Carson pointed out that it may have been easier to copy the log files over…ya, well…dammit I wouldn’t have gotten a nifty function out of it though!

Anyway, resetting and re-installing are fairly vanilla, and I’m pretty sure I covered standing up a core server somewhere before.

Thanks,

Managing your network interfaces for VMWare Player

If you need to reconfigure your virtual network, you will need the vmnetcfg.exe utility. This is part of the VMware Player download, but you have to run the installer such that it only extracts it’s files. Once you have extracted the files to a temporary folder, you will then need to extract the network.cab file. This cabinet file contains almost everything you need.

In the temporary folder that contains all the VMware player files, you will need to copy the following files to the same location you extracted the network.cab file to.

  • sigc-2.0.dll
  • vmwarebase.dll
  • libxml2.dll
  • iconv.dll
  • zlib1.dll
  • vmwarecui.dll
  • glib-2.0.dll
  • intl.dll
  • gobject-2.0.dll
  • glibmm-2.4.dll
  • gmodule-2.0.dll
  • vmwarestring.dll
  • libcurl.dll
  • ssleay32.dll
  • libeay32.dll
  • libldap_r.dll
  • liblber.dll
  • vmwarewui.dll
  • vixdiskmountapi.dll
  • sysimgbase.dll
  • vnetlib.dll

Once all the supporting files are in the same folder as the vmnetcfg.exe file, you will be able to manually adjust the virtual network adapters.

Fun with VMware ESXi

In the course of rolling out new hardware we encountered some problems with our up-and-running VMware ESXi servers. I won’t go into that now, but I’ll go over the fun things we found out. Most of these are most likely documented already, but I figured I’d put them here that way I have some place to look the next time I need something!

SSH

Firstly, managing through the client is nice, but sometimes we need more so below are the steps to enable SSH to your ESXi host:

  1. ALT+F1 at the ESXi host console
  2. Type, “unsupported” no text will display
  3. vi /etc/inetd.conf
  4. remove the # sign in front of #ssh stream tcp nowait root…
  5. cat /var/run/inetd.pid
  6. kill -HUP , this is the number returned from step 5

You should now be able to ssh into your ESXi host using puTTY or ssh, or whatever!

LOGS

One of the most annoying things for us is the lack of logs from ESXi, not to mention that they roll after each reboot, so you lose them. You can configure local and remote syslogging, I will give you the steps for both. My remote syslog server is a Windows 2008 R2 box that I’m playing with, and I’m using Kiwi SysLog Server to receive the logs.

Install Kiwi

  1. Agree to the license terms
  2. Select to run as a service
  3. Use the LocalSystem account
  4. Need a license for WebAccess so it’s pointless to check
  5. Choose a “Normal” install unless you need to set things up different
  6. Select your destination and click Install

Enable SysLog

  1. Connect to your ESXi host from the VMware Client
  2. Click the “Summary” tab
  3. Browse your local datastore, not it’s name
  4. Create a folder called “logs” inside
  5. Close the Datastore Browser
  6. Click the “Configuration” tab
  7. Click “Advanced Settings”
  8. Select Syslog

    1. Syslog.Local.DatastorePath = [Name of Datastore] logs/messages.log
    2. Syslog.Remote.Hostname = IP Address of syslog server
  9. Click Ok

Connect to your syslog server and you should now see that logs are available. The nice things is that since these logs are store on a different server, when the ESXi host reboots, you don’t lose them. The logs that are stored on the datastore also survive reboots, it’s nice to have both in case your syslog server fails.

iSCSI

Apparently ESXi is very touchy about iSCSI targets. If something get’s fouled up it seems the only way to clear it out is to reboot. This is not always fun, and while rebooting was the only thing that worked for us, here are some interesting tidbits we found while working with technical support:

/etc/vmkiscsi.conf

This file contains connection information for iSCSI targets on the host.

/var/lib/iscsi/vmkdiscovery

This file contains all the iSCSI targets that this host has discovered. We believe this file is auto-gen’d but what we do notice is that old targets persist even though they are not listed in this file. More digging may be required in order to find additional files that may contain outdated iSCSI targets.

/var/lib/iscsi/vmkbindings

This file contains a listing of all the iSCSI targets that the host is currently bound to. The recomendation from technical support is that this and the preceding file be renamed to try and resolve the iSCSI issue.

esxcfg-swiscsi -d

This command disables the software iSCSI initiator

esxcfg-swiscsi -k

This command kills the software iSCSI initiator

esxcfg-swiscsi -e

This command enables the software iSCSI initiator

esxcfg-swiscsi -s

This command triggers a rescan on the iSCSI initiator, this is similar to clicking rescan on the storage tab.

The value of virtualization

From time to time I think that we all sometimes overlook the value of virtualization. I know that I do, and it came home to me today. We have a problem with storage, it seems we can’t get enough! Over the summer we migrated from an older 7TB fiber channel SAN, to a new 13TB iSCSI SAN. Since then we have watched as more and more of our storage has been consumed. We hit a crisis mode this past weekend when the 3.5TB lun we had allocated for user data, quite literally filled up. Feel free to yell at me for not being more mindful or more proactive, it’s a crazy situation and I don’t know if I had to do it over, I would make the same choices.

Since we run a Storage Server cluster we run a report, and based on those we can tell you based on file ownership who our top consumers are. Needless to say the same handful of users are typically on top. In particular one user’s consumption ballooned by maybe 300%, their data quite literally occupied 1/6th of the entire 3.5TB! After going back and forth about possible solutions, we decided that the data should remain on the server, but be moved to a different lun. The only problem is none of our other luns had enough space to handle that amount of data, so we had to extend one of our luns. In case you’re curious about how long it took to extend, I started the process yesterday morning and it was still going when I left last night, but was done when I got in to my office this morning.

The fun part is what happened next. From Microsoft, presenting iSCSI luns to a server as dynamic volumes is not supported. From experience we can confirm that it is very painful, especially on reboots. Carson tracked that down one day while I was on vacation, I don’t think he’s documented that so I’ll bug him. All of the luns on our cluster are Basic disks, and as you know you cannot extend a basic disk. That is unless you know how to work DiskPart! We used this tool early on when we were troubleshooting computer imaging issues, and one of the nifty things it can do is extend the size of a disk.

We are just a trifle leery of fiddling with the drives on the cluster since it’s in production and at the core of most of the services we provide. This is where virtualization steps in, thankfully we implemented a VMware cluster a few years ago, and most of our server infrastructure is now virtualized. I have a machine, which I named “Schmoopy”,  that I tend to abuse quite regularly so Carson pointed out that you can extend the size of a virtual disk from the VMware interface.

So we fired up the VMware Infrastructure client, browsed to Schmoopy and added a disk. Keep in mind this was all done while the server was running. We verified that the disk showed up in Disk Management, we then initialized the basic disk and formatted it. Back in the VMware interface we then changed the size of the disk from 8GB to 20Gb, Schmoopy reported that it now had a chunk of free space at the end of the newly created disk. We fired up the command line and followed the instructions from an article we found on the Windows IT Pro website. We then went back to Disk Management and verified we had a newly available 20GB drive. The process went just as smoothly on the production server and the user’s data took the rest of the day to move!

It was most certainly the highlight of our day, and like I said earlier, it really made me aware of how valuable virtualization can be for everything we as IT Pro’s do!

Windows/Linux Server Base Image

I do a lot of software evaluations and as such I have come to rely quite heavily on virtualzation so I don’t have to worry about setting up hardware. As the software that we evaluate can be either Windows or Linux based I have created two base images, and from there build out my test servers for software deployment. The steps for each OS are very similar and I won’t go into any great detail on specific setup steps. I tend to just do a basic Windows Server installation taking all the defaults, and likewise for my Ubuntu LTS installation with only exception being selecting the OpenSSH at the end.

My personal choice for virtualization software is Virtual PC, mainly because I have used it ever since it became available for Windows and I don’t need anything more complicated to evaluate software.

Virtual PC Settings:

  • Networking: Shared NAT
  • Memory: 512MB
  • Hard Disk 1: 40GB
  • Floppy: Disabled
  • Sound: Disabled
  • Hardware Virtualization: Enabled

I have .iso images of my server install disks, so I just boot the blank base vm from iso and run setup from there.

Windows Server Installation:

  • Default WIndows Server installation
  • Apply the most recent service pack
  • Install .net 2 & 3
  • Connect to Windows Update (critical and software)
  • Repeat until all updates applied
  • Copy the i386 folder from the cd to drive C:
  • Extract supptools.cab to C:Support
  • Run the Setup Manager and choose a sysprep install
  • Sysprep the server and shut down

Ubuntu LTS Installation:

  • F6 and add noreplace-paravirt
  • Default installation
  • Choose OpenSSH at the end
  • SSH into server modify menu.lst add vga=771 noreplace-paravirt
  • sudo apt-get update
  • sudo apt-get upgrade
  • sudo shutdown -h now

I’m not sure if there is something similar for sysprep in Linux, but I’m pretty sure that it doesn’t really matter as you can just edit files in /etc and you’re done.

That’s pretty much all I do, the fun stuff is when I create a new machine based off this one.

Building Servers

Building a server, using the method I outlined in my previous post, becomes cake.

Building a server:

  1. Create a new virtual machine, use the default option
  2. Configure the settings for your new VM
  3. Run the New Virtual Disk wizard

    1. New virtual disk
    2. New Hard Disk
    3. Browse to the folder where your VM is stored
    4. Choose differencing disk
    5. Browse to the folder where the Server Base Image VHD is located

  4. Add the disk to your VM, make sure to browse to the proper VHD
  5. Power on the VM

Depending on how long ago the base image was created you may wish to check the OS for any security updates to make sure you’re as patched as possible.

Ubuntu 8.04 LTS and Request Tracker

Looking at some different possibilities regarding desktop support, so I’m outlining the steps I took to get a working copy of RT up and running on an Ubuntu 8.0 LTS server.

I’m setting this up in a VM on my desktop, since it’s Virtual PC I need to add the following at the startup and then later in the menu.lst file.

vga=771 noreplace-paravirt

 

Install LTS like you normally would, the only option I chose at the end was the OpenSSH that way if/when I forget to configure the noreplace nonsense I can still login and work.

From what I’ve seen the only version of RT available through the repos is 3.6, which I’m sure isn’t that big of a deal, currently Best Practices is on release 3.8. I have found a couple of articles that talk about setting things up which I’ll list at the end of this posting.

The following command will show you a list of packages available:

sudo apt-cache search rt3

I’m going to install the following packages:

  • rt3.6-apache2 –
  • rt3.6-clients –
  • request-tracker3.6 –
  • apahce2-doc –
  • postfix –
  • lynx –
  • postgresql –
  • libdbd-pg-perl –

sudo apt-get install rt3.6-apache2 rt3.6-clients request-tracker3.6 apache2-doc postfix lynx postgresql libdbd-pg-perl

For postfix choose an internet site, you can go back later and configure anything else you may need.

Once everything is installed you will want to configure your RT_SiteConfig.pm file, this appears to contain just general settings for the system.

nano /etc/request-tracker3.6/RT_SiteConfig.pm

Once that is done you will want to create a user account that has the ability to manipulate the database.

sudo su

su – postgres

createuser -s -P dba

exit

You should now have a user who can login and do dba-type things. I was following the guide up to this point, and this is where I deviated, I created the user first then ran the script and things seemed to go fine.

sudo su

su – postgres

/usr/sbin/rt-setup-database-3.6 –action init –dba dba –prompt-for-dba-password

exit

Now that you have a database you will need to configure apache so you can serve it up. I’m using the default sites-available file as we’re testing this out, so if you’re customizing things you may want to choose a different method, but either way I beleive you will need to edit the VirtualHost section and add the following line.

Include /etc/request-tracker3.6/apache2-modperl2.conf

Then you will need to enable mod-rewrite

a2enmod rewrite

Once that is taken care of you should be able to restart apache and browse to http://yourserver/rt and see things.

/etc/init.d/apache2 restart

You should now see a login prompt, user is ‘root’ and password is ‘password’. You may want to check the main wiki for BestPractical to see how to configure the system once installed.

 

Resources:

BestPractical wiki: http://wiki.bestpractical.com/view/UbuntuHardyInstallGuide