How we use the Active Directory

We rely on a lot of Microsoft tech to keep things clicking here at the School, and nothing is more important than our Active Directory Infrastructure. When I first started about the only thing you could get out of the Active Directory was a list of computers, users and groups.

Currently, we use the Active Directory as much as possible. One of the scripts that I posted a while back performs our inventory and actually stores pertinent information regarding each computer in it’s description property. One of the cool things that I’ve started doing, and I know I may behind the curve here, is to place objects in the Active Directory.

Two things that are incredibly important to users are accessing their data, and printing it out. So I have taken advantage of our OU structure, based on location, to make this much simpler. I have moved all the PrintQueue objects out of the print server and placed them in the office OU where the printer is located. In addition I have started publishing all of our shares into the Active Directory, more at the building level as most drives are fairly common for everyone, but there are some that are in individual offices.

Then I modified my scripts to take advantage of this, basically at logon the scripts figures out what OU the computer is in, and starts mapping PrintQueue or SharedFolder objects there, and then progressively works up the tree until it gets to the root of the Active Directory. This has made things much simpler to manage as I can literally just edit the object if a server changes and the script automatically maps to the new location or server!

The scripts will be posted as soon as they have been commented.

VLANs, VMWare ESXi, and RedHat EL

In environments were more than 4 VLAN’s exist you will have some difficulties getting all of them trunked into your VM’s. The best way seems to be trunking all the VLAN’s into single interface in your VM and then building multiple virtual interfaces within the guest OS.

There are only 4094 VLAN’s allowed so in order to create this properly the interface in the guest OS needs to be set to 4095. Then you can create a virtual interface for each VLAN on your network.

The following information was taken from RedHat support article 3681.

Resolution:

When connected to a properly configured network device, your Red Hat Enterprise Linux 3 system can communicate over a network using 802.1q Virtual Local Area Network (VLAN) tagged frames. The necessary kernel module, 8021q, is already available in the 2.4 kernel.
To use tagged traffic exclusively, create additional ifcfg-ethX.Y files, where X is the interface on which you will use the VLAN and Y is the VLAN ID. For example, on a system with one network card (eth0) that needs to talk to two different VLANs with VLAN IDs 10 and 11, you’ll need these files:

/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/sysconfig/network-scripts/ifcfg-eth0.10
/etc/sysconfig/network-scripts/ifcfg-eth0.11

These files will configure your system to have two virtual Ethernet interfaces called eth0.10 and eth0.11 that use tagged frames for communication to VLANs 10 and 11. To create the configuration files for the virtual tagged interfaces, copy the contents of your original ifcfg-eth0 file to ifcfg-eth0.10 and ifcfg-eth0.11. Then comment out or remove everything in your ifcfg-eth0 file except for

 
DEVICE=eth0
ONBOOT=yes

Next, edit the DEVICE= line in the ifcfg-eth0.10 and ifcfg-eth0.11 files so that they read eth0.10 and eth0.11 respectively. Add the line VLAN=yes to both files. Finish configuring these virtual adapters with the correct IP address and subnet mask for each VLAN, or with a BOOTPROTO=dhcp line if addresses are given out via DHCP. Don’t forget to include a default gateway in /etc/sysconfig/network. It’s important to remember that you can only have one default gateway.
Issue the command:

# service network restart

to complete the process. The VLAN=yes entries cause the network startup scripts to automatically run the vconfig command to add the necessary VLAN entries in /proc/net/vlan for each VLAN tag.
Here are the completed files for a network set up to only transmit tagged frames and with both virtual adapters set to use DHCP:
/etc/sysconfig/network-scripts/ifcfg-eth0

 
DEVICE=eth0
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-eth0.10

 
DEVICE=eth0.10
BOOTPROTO=dhcp
HWADDR=XX:XX:XX:XX:XX:XX
ONBOOT=yes
TYPE=Ethernet
VLAN=yes

/etc/sysconfig/network-scripts/ifcfg-eth0.11

 
DEVICE=eth0.11
BOOTPROTO=dhcp
HWADDR=XX:XX:XX:XX:XX:XX
ONBOOT=yes
TYPE=Ethernet
VLAN=yes

If you accidentally created a virtual adapter with the wrong VLAN ID, you may need to use the vconfig command to remove it from the /proc filesystem. Just restarting the network service won’t do that for you. For example, if you accidentally created a virtual adapter called eth0.12, the following command will remove it from /proc/net/vlan:

# vconfig rem eth0.12

DNS and DHCP Implementation

The School hosts its own Active Directory Infrastructure, this infrastructure depends on a DNS server that fully supports RFC 2136. Our administrative model relies on the ability of our clients to receive their network address from a DHCP server that fully supports RFC 2131. In a production environment we require that the services we rely upon do not implement “draft” or “beta” code.

This is not the state we find ourselves in while using Central IT’s implementation of DHCP and DNS. Their appliance has two serious flaws:

  1. TTL’s for all records do not decrement, resulting in stale forward and reverse lookups.
  2. The appliance relies on expired DHCP draft to handle failover between servers, which results in the lease files never being updated.

In order to provide a reliable and stable network environment we have decided to provide the requisite services ourselves. We will continue to troubleshoot the various issues with Central IT, but must move forward before the issues get worse. We have also provided a path to migrate back once a stable environment can be provided to the School.

Several steps need to be accomplished before we can roll these services out.

  1. Build two servers to host DNS and DHCP and provide redundancy

    1.  

      • Assign IP’s to authorized MAC addresses
      • Create scopes based on MAC addresses

    2. DNS and DHCP services will be provided by two RedHat EL 5.1 virtual machines. We settled on Linux as the built-in services offered by Microsoft did not offer the flexibility we needed.

  2. Provide services to each VLAN

      Our virtual servers have all the VLAN’s that comprise the School trunked into their switch. This trunk allows us to deliver an interface into each VLAN for the two virtual servers.

  3. Configure DHCP
  4. Perform zone transfer for DNS
  5. Disable scopes on Central IT appliances
  6. Delegate DNS from Central IT down to our DNS server

Microsoft Clustering

Some things I’ve recently learned while working in a non dynamic environment.

  1. Clusters by nature are dynamic
  2. Even when you don’t think it matters, it so does!
  3. If DNS is broken, nothing else matters.

In a non dynamic environment, set your Resource Name as a static entry in DNS. With a static DNS entry it should be safe to then uncheck the box for “DNS registration must succeed.” The idea being that if you have the static entry pointing at the correct IP, then DNS just works.

In this type of environment it appears that if that option is checked, which I believe is the default, and your cluster fails-over, those resources may not come back online. Given our current painful implementation of DHCP/DNS here, a simple configuration change took close to two hours to effectively work.

SOE Overview

I have been with the School of Engineering since February of 2007 and in that time we have moved from technology that was 1995 centric to an almost current level of technology. This overview should provide some insight into how we have grown, what caused us to move in the direction we did, and where we see ourselves in the future. This document will forego an explanation of the eccentricities of KU as the author himself still struggles with them.

The School provides services to the 2000+ engineering students and staff/faculty at the University of Kansas. We provide Helpdesk support for the over 100 applications we deliver to a given student, as well as technical support in the case of hardware issues.

When I first started computers were “imaged” using Symantec Ghost product. I use the term imaged generously as the computers while imaged were never properly imaged. The School is primarily a Microsoft shop, so the standard desktop OS is Windows. To support that environment we have a separate AD infrastructure from the University at large.

In order for a computer to be properly joined to any Active Directory Domain it needs a unique SID. The tool used for this purpose is sysprep.exe, which basically forces the computer into a pre-setup mode in which everything can be recreated.

In the case of the School’s SOP, a “master” computer would be installed with all requisite software and verified to function. This would also entail joining said computer to the proper domain and verified that logons would also work as intended. Without any other configuration this machine would then be ghosted to a server, applications, current Windows settings (NETBios name) and all.

The image would then pushed to the lab in a very manual way. A cart would be wheeled into the lab complete with server, one or more switches and enough cables to get to each client. Each client would be booted into the server and imaging would commence once all clients were connected. After that the computers would reboot, the computer would be renamed and that would be it.

Literally.

The end result of this process was an environment in which any kind of management, remote or otherwise, was virtually impossible.

If a new application was requested to be available in the labs, that application would be rolled out manually. It’s unclear as to wether or not a given application underwent any sort of compatibility testing, but in looking at what historical data there is, it appears that the bulk of applications were just installed. This process would be carried out manually on each and every computer where the software was needed. If it was something that should be available to all computers that would require installing the application on over 250 separate computers.

Due to the nature of how things were configured problems would crop up on a regular basis. The majority of the fixes almost always entailed using a batch file which would call some arcane NT command or would modify the registry with the reg.exe command. While the batch files were more or less standardized the nature of things here was such that really only a handful of computers could successfully run them against computers.

Historically engineering students were provided logon accounts to the engineering domain and this practice ended long before I arrived. All students are provided with the ability to create a domain account in the University’s domain. It is these accounts that the students use to logon to the computers anywhere on campus. Under the previous administration a one-way trust was established between the University domain and the School domain, to provide authentication.

If you are keeping track of things this more or less places us into an NT master/slave multi-domain arrangement.

As the students can logon to most any computer in the School the need to centralize their profile became mandatory. It was understood at that time that the ability to have a roaming profile between two domains in different forest’s was impossible. So a login batch file was created that would map drives for the students, map printers (using reg.exe) and manage their “profile” using robocopy.exe.

The process went this way:

  • Student logs in
  • All drives are unmounted
  • If the user registry file exists on the network it is reg.exe imported
  • Robocopy.exe copies everything from the server to the local drive
  • Based on an Environment variable the printers are mapped using reg.exe add
  • Drives are mapped using net.exe /use
  • Student logs out
  • Robocopy.exe copies data back to the server
  • The HKEY_CU is reg.exe exported to a file and copied up to the server

I can give no good reason as to why everything did NOT blow up at this point other than freak luck. This is the environment that was in place for a significant amount of time. This is NOT the way things should be in any network and when I took over administration that Summer I worked on changing that.

Main campus IT provides a large number of services and my goal has been to avail ourselves of those services whenever possible. To that end as new services became available I express our interest in testing them out in our environment. My goal then is at some point move us into a position in which the only services we provide are services that are wholly unique to the School. In addition, I want the administration of this environment to be as automated as possible, leveraging all the technology that we currently posses to achieve this end.

The first order of business was to replace Ghost with a process that was potentially free of manual intervention. I make no excuses, I am most definitely a Microsoft fan and with that in mind it made sense to me that Remote Installation Services was exactly what we needed to have.

In most environments there are some infrastructure services that just go, namely DNS and DHCP. The requisite DNS structure was in place to allow RIS to work, but the campus had a homegrown DHCP implementation. This service was based soley on LDAP and for the most part, it just worked. But in order for RIS to be truly automated the DHCP service needed to provide PXE support, this was either not possible or not available to me. So I resorted to a rogue Microsoft DHCP server that was limited by MAC addresses.

The first round of imaging went surprisingly well, unlike Ghost, which in our environment if you choose the wrong option the network goes away, RIS was surprisingly light on the network. In fact during testing we were asked if we were even doing anything while in the middle of imaging 10 computers! There were some planning and logistical issues that I was unprepared for, but they were well documented and did not show up in subsequent imaging sessions.

With that process more or less in hand it was time to tackle application management. I started working on a process that would allow the staff/faculty to request software be installed at their leisure. The idea being that with proper planning and testing, we will deliver any application you want, wherever you need it. In that respect we are actually making headway within the School.

There was no built-in application that could do this for us from Microsoft, so I turned to AdminStudio. This product allows us to take a given application and create a custom installation package that can be either manually installed or pushed out over the network. This product is now in place and creating and we have significantly reduced the amount of help requests caused by poorly implemented software.

Getting us to that point was only half of the application management plan. We also needed the ability to deliver an application at will to a given client with no manual intervention. The first solution was Group Policies. GPO’s allowed us to dictate based on OU what computers had which applications installed. While we had no reporting capabilities this did provide us the ability to “push” software to clients.

A better solution was found through Microsoft System Center. We have used one product, Data Protection Manager, for quite some time. This software backs up our 7TB storage array to disk with an option for long-term storage to tape. We have been very pleased with this product as it is incredibly intuitive and easy to use. My only complaint is that outside of scripting, there is no way to manage it unless you RDP into the server.

System Center Configuration Manager was the next product we looked at. This allows us to push applications with much more granular detail. We now have the ability to push an application to one or more computers with absolutely no interaction at all. In addition we have replaced RIS/WDS with SCCM Operating System deployment. We can literally push out the OS to a client as though it were an application! We also now have metering data, and can tell management exactly how often a given application is run! We plan on using this data to help eliminate that vast software duplication that we currently experience within the School.

We are now in a current state of operation within the School. Most of the day-to-day administration is done using either VBscripts or Powershell scripts. Sane group policies are now in place to manage all aspects of our users experience.  We are very close to the idea of “Single Sign on” within the School, and this is actually very exciting for us!

In addition to everything has happened at more or less the Application Layer of the network, we have had significant modification to the underlying network infrastructure. We are still waiting for an underlying hardware overhaul to happen, but we have seen some major enhancements to other areas of the network.

When I first arrived we had at most a 100MB outbound connection to the main campus, this is now a 1GB connection. All the servers in the rack shared a single port out to the rest of the network, now each server has a dedicated 1GB port to the rest of the network, this alone has greatly reduced the strain that we have seen on our storage array.

The outdated DHCP service was replaced, with a marginally functional DHCP/DNS appliance. We have had very few problems with the DNS appliance, there has been some general silliness for having a very specific configuration, but we have been able to work within those rules and DNS for the most part just works. Currently our issue is with the DHCP appliance, historically most organizations within the University used DHCP Reservations for all of their hosts. In an effort to eliminate the amount of work required to setup and configure computers we have decided that we need to use DHCP and DDNS for the purposes for which they were intended.

We are struggling with a vendor that seems like they are using us for beta testing their product. Things that on any other DHCP implementation would just work, are just broken with this implementation. I hold out hope that at some point in the not-to-distant future this problem will be resolved, but I do have a backup plan ready to go if needed.

When I first arrived the School’s network was one large VLAN that had been assigned close to 2,000 IP addresses. We have now divided the School’s network into separate and distinct VLAN’s, each with a specific purpose and an appropriate amount of private IP’s. This has greatly reduced the amount of transient traffic we had been seeing for quite some time. This now gives us a much leaner network than we have ever had, which means it performs better and we are better able to actually manage it, to the extent we are able.

We now also have a firewall in place. This was something that we had not had until quite recently. The School had been very fortunate considering the only thing protecting the bulk of our computers had been the built-in Windows Firewall. The main campus IT had recently upgraded to a new firewall product which was the primary reason for why we needed VLAN’s. Now we have a perimeter defense as well as a much improved client firewall policy that locks out everything else.

Configure Quickbooks for Windows Server

In a client/server environment multiple users connect to the Quickbooks server to access data files. There are two methods for this:

  1. Configure a file share on the server and allow the clients access to this share. In this scenario the client machines themselves will “host” the data files. Whichever client was “hosting” the files last will be saved into the data files. The problem with this arrangement is if the previous “hosting” client isn’t available you will receive several network connectivity errors.
  2. Configure a file share on the server and allow the clients access share. In this scenario the client machines access the data files through the Quickbooks Database Manager. The server will authenticate the users as opposed to a given client workstation. The problem with this arrangement is apparently Quickbooks is unable to make their database manager a service which means a user must be logged on so the application runs, so the following procedure should be followed:

Quickbooks Database Manager Service

  1. Create a service account
  2. Place it into the local administrators group
  3. Create a scheduled task that runs the Quickbooks Database Manager on computer startup
  4. Configure the task to run in the context of your server account

Intuit Quickbooks 9.0 Upgrade

Summary:

Upgrading the current version of Quickbooks 8.0 to the most recent version, Quickbooks 9.0.

Tasks:

  1. Backup the current data files
  2. Uninstall Quickbooks from the server
  3. Install Quickbooks 9.0 onto server
  4. Upgrade one workstation
  5. Copy data files to local computer and open the files
  6. Copy the data files back up to the server
  7. Verify single-user connectivity
  8. Upgrade remaining clients
  9. Verify multi-user connectivity

Start Date: 10/08/2008

End Date:

Summer 2007

In an effort to manage a portion of the network I began working on replacing the current methodology for computer imaging. Imaging computers with Ghost in our network environment is very dangerous; the current switch configuration is not compatible with multicasting and not performing sysprep leaves the computers in an unmanageable state.

Remote Installation Services provides a method by which computers can be imaged over the network. The process involves pre-staging a computer with the required software and running sysprep. This operation removes the SID from the computer and allows the computer to be unique. With unique computers you are able to manage the computers over the network.

RIS requires some infrastructure to be in place in order for it to function. The most important component is DHCP; under the current DHCP server (ANSR) the necessary technology was not available. In order to make RIS function a separate DHCP server was configured. This server was configured to only assign addresses to computers who have had their MAC addresses prestaged.

A clustered file server was installed with 6tb of SAN-based storage, as well as a dedicated backup server with 6tb of SATA storage. These were installed to replace the failed NAS-based storage array that had been in service for several years. It was evident almost immediately that the backup software, Legato, was not compatible with a cluster.

Winter 2008

The main thrust of this term was moving away from manual application installation and into a software lifecycle that provided automatic application installation via GPO. We acquired the AdminStudio product that would allow us to repackage a given application and deliver it to any lab within the domain.

Much of what I have planned is moving away from duplication of effort and rely more upon main campus IT. Main campus provides DNS, DHCP, firewall services and AD authentication through HOME. We were among the first to opt-in to the new DNS/DHCP/firewall services that were provided.

With that in mind we moved ahead with plans to migrate our DNS infrastructure onto the new Adonis appliances as soon as possible. This involved moving several web domains as well as our Active Directory domain onto this new platform. This was achieved by the start of the summer session.

In addition we started working on plans to migrate the labs out of the main vlan and into their own separate vlan and behind the firewall. The labs provide a glimpse to the end goal of all the work we have been doing:

· Isolated network

· Firewall protection

· Campus hosted infrastructure services

· HOME authentication

· Computer Management

· Application Management

Another large project that commenced was virtualization of servers. Our test environment was a Dell 1850 with 3gb RAM, and a dual core CPU. We opted for the VMware Server product as it was free and could be run on a very stripped down Linux installation. This hardware quite easily supported about 10 Windows based server’s simultaneously.

Spring 2007

At the start of this semester nearly 80% of the labs had been imaged using the new RIS server, there were some problems that required some manual intervention right before the semester started, but those were documented and wouldn’t be an issue for future imaging sessions.

Two major projects were in progress during this time. The first project involved researching, testing and implementing a new backup solution. The second project was maintaining a stable networking environment for the newly installed cluster. Both of these projects were entwined with each other as the addition of the cluster necessitated its backup.

The original backup solution was a product called Legato; it was neither cluster-aware nor intuitive to use. As a stop-gap we were backing up the cluster using the built-in Windows Backup utility, it provided a stable backup as well as reporting that we were unable to get using Legato. System Center Data Protection Manager was the product that we finally settle on.

DPM gave us all the things that Legato was unable to provide short term storage onto disk, long term storage onto tape and the ability to report on the status of backup and restore jobs. DPM provides an incredibly intuitive interface that allows us the ability to easily locate and restore lost files from weeks ago.

The other major project was resolving the stability issues for the cluster. The cluster had repeatedly failed over for unknown reasons on a regular basis. After months of working closely with the support team at Dell, the root cause of the problem turned out to be a leftover configuration to allow the Legato product to provide backups.