Windows Server 8 Beta Failover Clustering Part 2

There were so many images that I decided to split this up over several posts. In Part 1, I got the Failover Cluster feature installed on my first server, and now I’m going to work through the Failover Cluster Management tool.

win8-hv1-FC1

So, the interface doesn’t look any different. I was hoping for something that was part of the Dashboard.

win8-hv1-FC-wizard1

So here is the wizard, it’s really almost identical to Windows 2008 R2, but what the heck.

win8-hv1-FC-wizard2

I’ve selected both servers to be nodes in my new cluster.

win8-hv1-FC-wizard3

Oh wait! I forgot to install the Failover Cluster feature on win8-hv2!

win8-hv1-AddFeatureRemote

No worries, I got it covered, I can add that feature remotely from the dashboard. That option has to be one of the coolest one’s I’ve seen.

win8-hv1-FC-wizard4

Needless to say the feature installed successfully and I’m able to proceed now.

win8-hv1-FC-wizard5

None of the hardware I’m using is technically support by Microsoft but it’s Beta software so who cares right? Let’s see what the report says though.

win8-hv1-FC-validation-wizard1

Welcome to the Wizards, is it just me or are there more of them?

win8-hv1-FC-validation-wizard2

Lets run everything, I know it will have issues, as both machines are different and don’t have the same sets of software available.

win8-hv1-FC-validation-wizard3

Here we go…

win8-hv1-FC-validation-wizard4

That looks good!

win8-hv1-FC-validation-wizard5

OUCH! Well, win8-hv2 doesn’t actually have Hyper-V installed and since that was in the test, that’s where it failed.

win8-hv1-FC-validation-report1

The report confirms this. But again, thanks to being able to remotely install Roles and Features, I installed Hyper-V on the other server and re-ran the tests.

win8-hv1-FC-validation-report2

Much happier! For the record, there are several warnings.

  • Hyper-V : The processors are different between the two machines.
  • Network : I don’t have redundant network cards
  • Storage : I don’t have any storage available suitable for some types of clustering, which could be an issue for Hyper-V (Clustered Shared Volumes)
  • System Configuration : To be honest, I didn’t actually look at this, I was aware of the others so I assume there is something minor missing or different between the two nodes.

win8-hv1-FC-wizard6

Here we go, this is good enough to move on to create the end-point for the cluster. This is one of the ways you’ll be able to manage it.

win8-hv1-FC-wizard7

Here it is saying it’s going to steal some IP’s from my range

win8-hv1-FC-wizard8

My cluster is forming!

win8-hv1-FC-wizard9

I was successful! But there were warnings, basically it’s telling me I don’t have any sort of network storage to use for this cluster, of which we were already aware.

win8-hv1-FC2

Huzzah! The one warning was from the cluster I built earlier with just the one node. Perhaps should have nuked the cluster log, oh well.

Windows Server 8 Beta Failover Clustering Part 1

I read a very nice article over on TechNet about some of the new features of Failover Clustering in Windows 8, so I decided to give it a spin. It’s not too bad, I’m really enjoying the new Server Manager interface, it’s pretty awesome.

Here are the screenshots of the setup process.

win8-hv1-step1

Here is the dashboard, this particular computer was upgraded to Windows 8 Server from Windows 2008 R2, so there were already a few roles installed.

win8-hv1-wizard1

Similar interface for adding the Failover Clustering Role

win8-hv1-wizard2

I’ve not tried the Remote Desktop VDI stuff yet, I’m thinking I may grab a couple of more machines and start over with fresh installs.

win8-hv1-wizard3

Here you can install roles onto one of the servers in your pool. I added win8-hv2.soecs.ku.edu to the pool so I could manage both servers from one interface, cool huh?

There is also the option to install the role to VHD, I will have to try that later!

win8-hv1-wizard4

Here you can see the installed roles from Windows 2008 R2

win8-hv1-wizard5

There’s the feature I’m after, Failover Clustering. Note the additional tools, there are 81 PowerShell cmdlets available for managing Failover Clustering. I’m going to post those up after this.

win8-hv1-wizard6

Here is everything that will be installed, I checked the box to Restart the destination if necessary, but for this feature it’s not necessary. But as I was messing around with various other components earlier this week, that’s a nice option.

win8-hv1-wizard7

The installation is starting, you can see the notification flag now has a 1 inside it’s little box. You can close this window and the install will progress.

win8-hv1-TaskStatus

Clicking on the notification flag, you can see all tasks that are currently running.

win8-hv1-TaskDetails

Here is what you see when you click details. By the time I got to this screen the installation was done.

win8-hv1-Manage1

I can manage the Cluster from the Dashboard | Tools menu

win8-hv1-Manage2

Since it’s installed on win8-hv1 I can right click on that server in the Server Pool and select Failover Cluster Manager from there.

win8-hv1-Manage3

Perhaps this is silly to point out, but the Failover Clustering feature was not installed on win8-hv2 and so you don’t see the option to manage it from there.

RDP over SSH

Before I start, while this will allow you to access your servers over a secure tunnel, this does not mean you should forego patching your systems.

Don’t be that kind of admin, install the patches, install the critical updates, do us all a favor and make your gear as secure as you can.

I know this is not a new topic, but it’s rather new to me. The university has decided to block RDP at the border after the latest RDP exploit. For the record the university does provide a VPN which will work for most folks, but I don’t often have a machine that I can do that from. The nice thing about putty is it’s a simple download and you don’t have to install it, just download and go.

I’m not going to tell you how to setup an ssh server, mostly because it’s pretty straightforward.

Here we go

  1. Download and start putty
  2. Type in your connection information admin@server.company.com
  3. Open Connections, SSH, Tunnels
  4. Set the source port to be 3391
  5. Set the destination port to be rdp-server.company.com:3389
  6. Click add, and then open the connection
  7. Start the RDP client
  8. Make a connection to localhost:3391
  9. You may be prompted for all that new connection stuff and then finally credentials

You should now have a connection established to your remote desktop server that is being tunneled through your SSH connection.

Defrag is not defragging

I posted a question in the forums the other day after attempting to defrag one of the drives on my file server. The drive in question is where we have all the application installation files for the School stored.
This particular drive was quite low on free space, so I used the following switches in an attempt to force the defrag to happen anyway.

  • W – Performs full defragmentation. Attempts to consolidate all file fragments, regardless of their size.
  • F – Forces defragmentation of the volume when free space is low.
  • V – Specifies verbose mode. The defragmentation and analysis output is more detailed.

C:Usersjeffpatton.admin>defrag s: -w -f -v
Windows Disk Defragmenter
Copyright (c) 2006 Microsoft Corp.

Defragmentation report for volume S: Software Drive
    Volume size                         = 512 GB
    Cluster size                        = 4 KB
    Used space                          = 450 GB
    Free space                          = 62.40 GB
    Percent free space                  = 12 %

File fragmentation
    Percent file fragmentation          = 55 %
    Total movable files                 = 1,716,060
    Average file size                   = 605 KB
    Total fragmented files              = 1,536
    Total excess fragments              = 1,363,219
    Average fragments per file          = 1.85
    Total unmovable files               = 11

Free space fragmentation
    Free space                          = 62.40 GB
    Total free space extent             = 1,013,736
    Average free space per extent       = 65 KB
    Largest free space extent           = 85 MB

Folder fragmentation
    Total folders                       = 127,411
    Fragmented folders                  = 1
    Excess folder fragments             = 2,819

Master File Table (MFT) fragmentation
    Total MFT size                      = 1.70 GB
    MFT record count                    = 1,777,742
    Percent MFT in use                  = 99
    Total MFT fragments                 = 3

    Note: On NTFS volumes, file fragments larger than 64MB are not included in the fragmentation statistics
Defragmentation report for volume S: Software Drive
    Volume size                         = 512 GB
    Cluster size                        = 4 KB
    Used space                          = 450 GB
    Free space                          = 62.40 GB
    Percent free space                  = 12 %

File fragmentation
    Percent file fragmentation          = 70 %
    Total movable files                 = 1,716,060
    Average file size                   = 605 KB
    Total fragmented files              = 1,533
    Total excess fragments              = 1,363,206
    Average fragments per file          = 1.85
    Total unmovable files               = 11

Free space fragmentation
    Free space                          = 62.40 GB
    Total free space extent             = 1,012,697
    Average free space per extent       = 65 KB
    Largest free space extent           = 85 MB

Folder fragmentation
    Total folders                       = 127,411
    Fragmented folders                  = 1
    Excess folder fragments             = 2,819

Master File Table (MFT) fragmentation
    Total MFT size                      = 1.70 GB
    MFT record count                    = 1,777,742
    Percent MFT in use                  = 99
    Total MFT fragments                 = 3

    Note: On NTFS volumes, file fragments larger than 64MB are not included in the fragmentation statistics
Well, here it is the next day and defrag finished sometime during the night. You will notice now that file fragmentation is where it should be. My assumption from this is that while you can choose the –F option, there is most likely a threshold at which even that won’t work.
C:Usersjeffpatton.admin>defrag s: -w -f -v
Windows Disk Defragmenter
Copyright (c) 2006 Microsoft Corp.

Defragmentation report for volume S: Software Drive
    Volume size                         = 640 GB
    Cluster size                        = 4 KB
    Used space                          = 445 GB
    Free space                          = 195 GB
    Percent free space                  = 30 %

File fragmentation
    Percent file fragmentation          = 45 %
    Total movable files                 = 1,717,746
    Average file size                   = 607 KB
    Total fragmented files              = 446
    Total excess fragments              = 1,159,235
    Average fragments per file          = 1.72
    Total unmovable files               = 11

Free space fragmentation
    Free space                          = 195 GB
    Total free space extent             = 995,580
    Average free space per extent       = 206 KB
    Largest free space extent           = 84.77 GB

Folder fragmentation
    Total folders                       = 127,414
    Fragmented folders                  = 1
    Excess folder fragments             = 0

Master File Table (MFT) fragmentation
    Total MFT size                      = 1.70 GB
    MFT record count                    = 1,778,403
    Percent MFT in use                  = 99
    Total MFT fragments                 = 3

    Note: On NTFS volumes, file fragments larger than 64MB are not included in the fragmentation statistics
Defragmentation report for volume S: Software Drive
    Volume size                         = 640 GB
    Cluster size                        = 4 KB
    Used space                          = 445 GB
    Free space                          = 195 GB
    Percent free space                  = 30 %

File fragmentation
    Percent file fragmentation          = 0 %
    Total movable files                 = 1,717,746
    Average file size                   = 607 KB
    Total fragmented files              = 7
    Total excess fragments              = 305,897
    Average fragments per file          = 1.19
    Total unmovable files               = 11

Free space fragmentation
    Free space                          = 195 GB
    Total free space extent             = 659,294
    Average free space per extent       = 311 KB
    Largest free space extent           = 2.67 GB

Folder fragmentation
    Total folders                       = 127,414
    Fragmented folders                  = 1
    Excess folder fragments             = 0

Master File Table (MFT) fragmentation
    Total MFT size                      = 1.70 GB
    MFT record count                    = 1,778,403
    Percent MFT in use                  = 99
    Total MFT fragments                 = 3

    Note: On NTFS volumes, file fragments larger than 64MB are not included in the fragmentation statistics

Repository and Blog

I migrated over all my scripts late last night, it only took about an hour. I think part of the problem was on my end, I had to basically restart the sync several times to get everything over. The longest part was moving over the Trac content.

While I didn’t have a whole lot, it was enough to make it tedious. The other pain point was some of the syntax was the same, and some was just enough different to give you a migraine! But that’s all been migrated over as well. I have updated all the code in /production so that it points to the current url for online help.

https://mod-posh.googlecode.com

Once I commit what I have on my laptop up, I’ll create a new “download” for it. I struggled with whether or not I actually wanted a download, and finally decided that a simple zip of the includes folder for now is good. That’s where all the goodies are anyway.

With regards to the blog, I’ve still not been able to use WLW or even the blogger app on my GalaxyTab. Apparently when you import your previous blog that counts against your daily allowance of bandwidth and posting. Oh well, I’ll leave my existing gear up the rest of the this month with re-directs then I can turn all that off.

SCCM + Dell – ServiceTag = HEADACHE

Well, it’s been a long time since I’ve posted anything and this will most likely be the last post before I hop platforms, but since the resolution took nearly my whole morning I felt it was worthy a posting. As you know we’ve been rolling with System Center Configuration Manager for nearly two years now and while it doesn’t necessarily follow best practices, it’s been as stable as that product appears to be.

So today I was given a machine to image, and I promptly checked our servers to see if it was still lurking out there somewhere.

  • In AD? Nope.
  • In SCCM? Nope.
  • In DHCP? Yes, but that’s ok.

So I fired up the SCCM console and expanded leaf objects until I got down to Computer Association. Right click and choose Import Computer Information and follow through the dialogs. What we’ve always done is add the computer by Name and MAC address, mainly because the GUID is entirely too long for any normal person to remember or have the time to write down. After adding the computer to the proper collection and finishing out of the wizard, I fired up the computer and hit F12, and let the PXE do it’s magic.

Sadly, there was no magic, after about a minute I heard two beeps which means the computer is unable to boot off the network. No worries, I know my view on time is different from that of SCCM so I waited about 5 minutes and tried again, still two beeps. I decided that perhaps I had typoed the MAC address, so decided I’d try it again; two beeps.

That’s when I began my trouble-shooting, the first thing I did was double-check the MAC, as well as make sure there were no duplicates in DHCP, which based on how we roll DHCP is impossible, but still doesn’t hurt to check. There is a report in SCCM that let’s you know if there are duplicate MAC’s inside it’s database, “MAC – Computers for a specific MAC address.” This will let you know if the MAC you enter is associated with more than one computer, it wasn’t.

I fired up the log and saw the following message, I decided to leave the typo intact, since that’s the way Microsoft left it!

The SMS PXE Service Point intructed device to boot normally since it has no PXE advertisement assigned.

Device MAC Address:00:1A:A0:B9:EF:A8 SMBIOS GUID:4C4C4544-0000-2010-8020-80C04F202020.

That seemed odd to me, since I knew that everything was set properly, I decided to restart the Windows Deployment Services (PXE) service. Often times that will fix small issues with PXE booting workstations for imaging, two beeps. That wasn’t it, so then I had to go to the bad place, SMSPXE.LOG. I’m not sure why, but apparently the SMS dev’s decided to punish admins and write absolutely horrible log entries that look like XML and reference line numbers in the source code. Sadly, there wasn’t much of anything different in here either except this:

Device found in the database. MacCount=1 GuidCount=4]LOG]!>

See what I mean by horrible? Anyway, the interesting tidbit is GuidCount=4, wtf? So a while back Carson wrote a report in SCCM that would show GUIDs, I suppose I should post that at some point because it is SUPER handy! But sure enough there were 4 computer’s with the exact same GUID. All of them but one were current, so I decided to nuke them, restart PXE and attempt my boot again, two beeps. I was not a happy camper.

So it was off to Google, since I wasn’t having any luck with the logs. As you can imagine there were lots and lots and LOTS of threads, postings and technical documents.  Most of what I read was from the Microsoft Technet social site, but as I began reading I began noticing that several of these were referencing Dell Optiplex computers. While not the same vintage Optiplex as what I was reading about I was working with Dell hardware nonetheless.

I finally fell on the answer in a two year old thread on the Dell support site. Turns out that service tag is more than just helpful on their website! The GUID for the computer is based on that service tag, and if the motherboard gets replaced and the tech doesn’t add it back into the BIOS, the computer will create a generic one. That’s the GUID or SMSBIOS GUID listed above, and it was a painfully easy fix! You will need to download the ASSET.COM utility from Dell’s Utility FTP site.

Once downloaded run that command with the /S switch and the service tag for your computer. Please be aware that if you muck up that entry, there is no way to remove it! So double, triple, quadruple check before you confirm that it’s ok to update that information.

The only other complicated part for me was finding a floppy drive, and more importantly a floppy disk to make bootable so I could run this and a BIOS update on the intended computer!

Supporting ancient hardware

Today we’ll be working on some moldy oldies! I give you the SGI Indigo, and it’s successor the SGI Octane! Bow in front of their immense glory!

SGI Indigo2 Circa 1994SGI Octane Circa 1997

Here is the problem:

We have research analysis software designed solely for SGI, and our main computer for these analysese, the SGI Octane unix computer, OCTANE, will not boot due to apparent hard drive RAM failure. This 10+year old may not be recoverable.
We can regain our analysis functionality using the functional SGI Indigo 2 computer that has the necessary software, INDIGO. However, the network functionality is not currently available on INDIGO to allow file transfer to/from the computer. Thus, my primary request is that INDIGO be configured to allow secure ftp and remote shell access only.
I believe the computer needs to be added to the workgroup (or in setup in that “domain”). Please assign someone to complete this network access task in the next few days. Please coordinate with my lab assistant to assure that sftp and secure shell from our PC computers is working.
Secondarily, if it is staightforward to recover an image of the failed SGI Octane hard drive, we may consider replacing the drive (if possible), restoring the image on the “new” (likely used) drive and getting the SGI Octane, OCTANE working again. This is a lower priority, as there is no substantial data loss, and we have already been looking at ways to move the analyses done on the SGI over to Windows PC software (or possibly Linux). We have no desire to invest large amounts of time and/or money to restore the failed SGI. But, if the process is simple and cost is “cheap”, then we would like to restore the SGI octane until a long-term solution is found.
SGI_Error_Mesage

So how do you start? Well, we started with one assumption, the SCSI card is dead. If the disk itself is dead, then there is really nothing we can do. Working on our assumption we were able to scrounge up a full length PCI SCSI card.
WP_001576
We verified that it was recognized in the BIOS and could see an attached disk.
WP_001574WP_001575
After our regularly scheduled lunch (Buffalo Wild Wings…yum) and our staff meeting we headed over to the lab. We had been there once before with Nick, and it wasn’t much fun then either. It was during an IP inventory that we found these machines, and it was on that visit that we configured them to use our network. Sadly the “Supercomputers” were so old, they pre-dated DHCP…but that’s another story. We spoke with the advisor who was unaware that SGI ceased to exist as a company about 8yrs ago, further he was also unaware that SGI stopped using MIPS and switched to Intel based CPU’s about 4yrs before the company filed for bankruptcy.
In the lab we found many things, lovely wiring:
WP_001577
We also found the sad little Octane, with it’s drive removed. The first thing we noted was that the disk was six years newer than the computer. This was good news for us as we had servers that we could mount that disk into natively. We also “fixed” their network issue, turns out that the Indigo2 had the subnet incorrectly specified, sounds like a stupid mistake, aside from the fact you had to set the network id in hex, ya…you heard me. We gave them the IP of their box and they were able to access it via sftp.
WP_001578
We informed the advisor that we would attempt to access the disk back in our office, but first we had to make a stop over to our storage room. We picked up a lovely Dell PowerEdge 2650, and swapped one of it’s disks for the failed SGI disk.
WP_001580
Upon booting into the BIOS we ran disk utilities and it informed us that the disk was in fact dead. Sadly this means we were unable to fix that part of the problem. But the good news is that we had a nice trip down memory lane playing with hardware that used to cost thousands of dollars, and now is up for sale on E-bay for about $400. Sorry this has been a rather rambling post, but I felt upon receiving this ticket in the help desk that it really merited some form of posting.

HOWTO: Setup IIS 7.5 to use IPv6

I took the SANS 546 class today, and it got me thinking about setting up my server to respond to IPv6 hosts. Steps thus far are pretty straightforward:

  1. Get an account with a tunnel broker
  2. Configure your host
  3. Test connectivity
  4. Configure IIS
  5. Create AAAA record on your DNS provider
  6. Troubleshooting

Tunnel Broker Account

This is very easy and painless! There are several to choose from, but one that was mentioned by the lecturer was Hurricane Electric. Fill out the form and check your email for your password, the whole process takes about 1 minute. Once you login you will need to create your first tunnel:

  1. Login to http://www.tunnelbroker.net/ with your username and password
  2. Click “Create Regular Tunnel”
  3. IPv4 endpoint is your webserver
  4. They will find a tunnel closest to your IP
  5. Click “Create Tunnel”

Configure your host

See? Pretty painless, now that you have your tunnel up you will need to configure your host, since I’m working with Windows 2008 R2, there is actually a set of netsh commands you run. They are specific to your configuration and you can access them by clicking, “Example Configurations” tab on the Tunnel Details page. From the dropdown select your Operating System, and it will give you the commands you need to set it up.

Test connectivity

Once everything is configured the easiest way is attempt to access an IPv6 host:

Configure IIS

Configuring IIS is pretty simple as well, I found that I had some extra stuff that I didn’t think I should need to do though.

  • Open IIS Manager
  • Select the site you wish to enable IPv6 on
  • From the Action pane choose “Bindings…”
  • For basic web server
    • type = http
    • Ip address = ipv6 address
    • port = 80
    • host name = the name you want the server to respond to
  • Click Ok

Create AAAA record on your DNS provider

I use GoDaddy.com for my dns, so you will just need to go into total dns manager and add the AAAA entry. This entry will need to be the same as the host name you specified in your IIS bindings

Troubleshooting

  • The first thing you will want to do is make sure that you are able to ping your own ipv6 address
  • Then try pinging your ipv6 address remotely
  • Repeat these steps with the ipv6 hostname that you set in DNS.
  • It may also be a good idea to visit test-ipv6.com
    • If all those tests fail you may have other issues, that I can’t really help you with

What I found is I was able to ping my ipv6 address locally and remotely, my name ipv6.patton-tech.com resolved locally and remotely, but when pointing a browser at that URL nothing showed up. It was the end of the day when I got this setup, and I had done some of the above basic troubleshooting that all returned successful. This morning I began again I ran the following command:

netstat –s

The output from this command showed several failed attempts over IPv6, this seemed to increase each time I attempted to open the website (could have been coincidence). Since I saw there were failures on IPv6, the next thing I did was run this command:

netstat –an

This should show what addresses are listening on what ports, I saw my IPv4 addresses, but no IPv6. That’s when I started browsing the forums looking for something useful, and I didn’t find much. Most of what I found talked about making sure you didn’t choose the temporary IPv6 address, but since ours is assigned statically via netsh I don’t think that was the problem. Running the command:

netsh interface ipv6 show address

Shows that my interface was a tunnel interface, which makes sense, but that got me spun off into checking the firewall, which wasn’t the problem at all. Finally I found a forum post on iis.net that was close enough to my issue that I was able to resolve it. One of the posters suggested running this command:

netsh http show iplisten

Unlike the original poster I was able to see my IPv4 public address but not my IPv6 address. The suggestion was to remove all iplisten entries which would force iis to listen on all ip addresses. Since I have several services running and listening on port 80, I couldn’t do that. But the syntax of that command led me to TechNet for the proper syntax to add a listener:

netsh http add iplisten ipaddress=ipv6addy

I posted a question in the forums to ask if there is something I have done wrong, or if perhaps the default is to not add the listener but no answer yet. It seems to me that when I add a binding to iis, it should also allow the web server to listen on that address. I know there was nothing I had to do for IPv4, so it’s either a default (not likely) or the fact that this address is set statically (more likely).

Do you suffer from “Premature Installation”?

Or, “What’s in a name?”

Turns out a whole hell of a lot! First I need to thanks Nick for the awesome title, as he completely pinpointed my issue after I told him what happened! The last article I posted talked about our desire to move away from vanilla Windows 2008 and up to Windows 2008 R2. What should have been a pretty straightforward process got slightly mangled by two things. I forgot to rename the computer, and I moved to fast, hence the “Premature Installation!”

Naming is important, there are some names you can change and some you can’t. How computers get names has also changed with 2008, it used to be that during installation you were prompted for a name, now you do that after. One of the things we found out was that a Domain Controller can have multiple names, while I don’t know how recent that change is, or isn’t it was new to us. Back to the naming process, while there’s nothing inherently wrong with a Domain Controller named WIN-LLF3467Q0, you would undoubtedly agree it doesn’t really roll off the tongue.

So that was the first problem, I installed Windows 2008 R2 without mishap, and Directory Services installed, and when I hopped over to the Domain Controller’s OU I noticed my problem. So the first thing I did was go to the above article and renamed my new Domain Controller, and this is where the second problem occurred.

Replication while speedy, it does take time, and the more things you have in your AD the longer it could potentially take. The end result of my fubar is that we wound up with no less than three different entries in DNS for the same server, only one of which was correct, and due to replication latency the name of the server in AD was completely wrong.

So I did what I imagine most people would do, and went to uninstall DS from the server and attempt to start over. But because things had gotten so trashed I was unable to uninstall DS, because the server name that I was on didn’t exist in AD, I really should have screenshot stuff but take my word, I was on dc1 and the error was dc1 didn’t exist…which was technically true. It was a crazy weird edge situation, you could actually connect to DC1 but you had to type it in manually in order to get there. At any rate I was unable to remove DS, so I turned off the computer and attempted to remove the computer account that was listed from the Domain.

The problem with that was in order to do it, you MUST be on a Domain Controller to remove a non-functional Domain Controller from the Domain. I’ve not found an article on TechNet that mentions that, but I’ve not looked in any great detail. This information was found on the TechNet Social site, after connective over RDP to the off-site Domain Controller I was able to remove the offending account.

So, in the future, remember to be patient and make sure you have a checklist!

  1. Install Windows OS
  2. Change the default name before network connectivity
  3. Make any needed changes
    1. Disable IPv6
    2. Apply 3rd party DNS Hotfix
  4. Install Directory Services
  5. Wait
  6. Wait
  7. Wait
  8. Verify successful replication

These are the steps I followed on my server rebuild yesterday, as well as the same instructions I followed when I migrated the second Domain Controller this morning.