Database backed up to VIRTUAL_DEVICE GUID

Working with a customer who wanted to make sure their database backups were consistent before going live, we found the following entry in the log.


Database backed up. Database: ProductionDB, creation date(time): 2016/10/26(15:46:13), pages dumped: 70152, first LSN: 50966:4624:2, last LSN: 50966:13328:1, number of dump devices: 1, device information: (FILE=1, TYPE=VIRTUAL_DEVICE: {'{9B4A021C-7A36-4358-B95B-3E93999DC1BD}8'}). This is an informational message only. No user action is required.

There were several things that we knew, first the SQL Server were running managed backups to Azure. A quick check of the storage account revealed the regular managed backups we expected to see.

Get-CloudBlob.ps1

We didn’t see any backups for the database when that log entry was created. We also knew that the server was being backed up by Azure Recovery Services, and this backup happened daily at 2am. This correlated with the stamp of the log entry, but we were unable to find where this backup was. We assumed that this had something to do with VSS and SQL so with this information we started searching for some folks experiencing similar issues and potential resolutions.

We found several articles that I’ll list below, but the general consensus was that backups in Windows will trigger VSS. The providers will then trigger the VSS writers to quiesce the SQL files. Nothing is being backed up per-se, but SQL is registering this is a backup event. There were several threads where the reported solution was to just disable the SQL VSS service, but I don’t think that’s wise.

To our thinking, if we disable the VSS writer for SQL, then when Azure attempts to perform a backup, it will have at the last an inconsistent backup, and at worst a corrupt backup as the files were in use at the time.

Knowing that the server itself is getting backed up, we launched the File Recovery (Preview) which is very slick. This launches from the portal, you select what date you want and it compiles a script that you download and run on the VM in Azure. This script will connect over iSCSI to the VHD and mount it for 12hrs letting you access the files on the server at that point in time. This would allow you to access the SQL Database files in their normal directories, albeit different driver letters. But they are not actual backups, it’s a snapshot of the data/files at that time.

While very cool, we needed to have a more reliable way to validate that our backups were vaild. Our concern was that the VSS triggered backups would break the restore chain, but turns out that our regular managed backups work as expected and we could restore the last full, and transact get us back to where we needed to be.

So, the thing to remember here is that if you see an entry like the above, keep calm. As long as you are doing your regular backups as you should, then everything will be ok.

And Happy National Backup Day!

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual-machines-windows-sql-backup-recovery

https://www.sqlservercentral.com/Forums/Topic666998-357-1.aspx

http://dba.stackexchange.com/questions/91812/where-is-the-sqls-database-backup-virtual-device-string-stored-in-the-system-tab

https://social.msdn.microsoft.com/Forums/sqlserver/en-US/c96bc256-99e5-453e-aa78-7910fe925f65/ntbackup-causes-sql-server-backups?forum=sqldatabaseengine

https://social.msdn.microsoft.com/Forums/en-US/e1ec1bc8-ee2b-4397-9b98-314d63e3e9a5/trying-to-find-out-whats-triggering-backups-and-where-they-are-going-to?forum=sqldatabaseengine

https://support.microsoft.com/en-us/help/956893/support-policy-for-microsoft-sql-server-products-that-are-running-in-a-hardware-virtualization-environment

Create VSTS Service Principal

Working with one of our CSP customer’s and they needed to connect their Visual Studio Team Services account to their CSP Azure account. If you have a regular Pay-As-You-Go subscription, then you have access to the old portal (manage.windowsazure.com) but if you’re a CSP that doesn’t work. So after talking Brian Moore at Microsoft I created a series of steps that I thought I’d get down for the next time I need to do this.

Step 1

You will need to logon to your Visual Studio Team Services account. As you can see I have logged into mine and I have a couple of projects.

Step 2

You will need to select a project that you will deploy/integrate with Azure, I’ve selected my sample project.

Step 3

This step is where you configure the project to connect to VSTS by creating an endpoint.

Step 4

I’ve given my endpoint an incredibly creative name and associated it with a specific subscription.

Step 5

Here the endpoint is complete and you have the option to change it’s configuration, manage the endpoint’s role within Azure as well as manage the service principal itself, and finally to disconnect the service principal. The disconnect will in fact delete the service principal from azure, so in production this service principal should only ever be used with Visual Studio.

As a side note, the manage service principal link kicks you over to the old portal, so for CSP customer’s this may in fact fail. See images below.

Update Service Configuration

You have a couple of options here, change the connection name, and change the subscription.

Manage Endpoint Roles

This will take you to the Azure portal and let you adjust and generally fiddle with the roles associated with this Service Principal.

Manage Service Principal

By default this appears to connect you over to the old portal. But this gives you the ability to manipulate the properties of the Service Principal.

Here is where you can find the same information in the new portal. This is difficult to see, but Azure Active Directory > App Registrations and then choose the Service Principal named VisualStudioSPN.

Disconnect

Finally, to remove the Endpoint and Service Principal, simply choose disconnect, and this will go through and clean everything up.

 

Fix Windows Server 2012 R2 DFSR Event ID 4614

Recently had a ticket come in where a newly created Domain with two DC’s was not replicating properly. Upon logging into the DC’s I noted the following log entry in the DFS Replication Log


The DFS Replication service initialized SYSVOL at local path S:\SYSVOL\domain and is waiting to perform initial replication. The replicated folder will remain in the initial synchronization state until it has replicated with its partner . If the server was in the process of being promoted to a domain controller, the domain controller will not advertize and function as a domain controller until this issue is resolved. This can occur if the specified partner is also in the initial synchronization state, or if sharing violations are encountered on this server or the synchronization partner. If this event occurred during the migration of SYSVOL from File Replication service (FRS) to DFS Replication, changes will not replicate out until this issue is resolved. This can cause the SYSVOL folder on this server to become out of sync with other domain controllers.

Additional Information:
Replicated Folder Name: SYSVOL Share
Replicated Folder ID: C0037E37-EF20-4CDF-968A-932E669ED810
Replication Group Name: Domain System Volume
Replication Group ID: 35F23B97-543A-4310-A08E-5F28D6342C18
Member ID: 41E8F809-5114-48DC-8297-B7E866502101
Read-Only: 0

Working backwards through the log I found this entry


The DFS Replication service encountered an error communicating with partner AD-02 for replication group Domain System Volume.

Partner DNS address: AD-02.contoso.com

Optional data if available:
Partner WINS Address: AD-02
Partner IP Address: 192.168.1.4

The service will retry the connection periodically.

Additional Information:
Error: 1726 (The remote procedure call failed.)
Connection ID: 3E337B81-109F-4A97-880C-63E30F52E63F
Replication Group ID: 35F23B97-543A-4310-A08E-5F28D6342C18

I didn’t see that error on AD-02 but I did see several alerts like this one leading up to when the above alert fired on AD-01.


The DNS server is waiting for Active Directory Domain Services (AD DS) to signal that the initial synchronization of the directory has been completed. The DNS server service cannot start until the initial synchronization is complete because critical DNS data might not yet be replicated onto this domain controller. If events in the AD DS event log indicate that there is a problem with DNS name resolution, consider adding the IP address of another DNS server for this domain to the DNS server list in the Internet Protocol properties of this computer. This event will be logged every two minutes until AD DS has signaled that the initial synchronization has successfully completed.

As both DC’s are also DNS servers I configured the binding to be the statically set IP address. Which should once and for all rule out any sort of weird resolution issues. This had no affect either. There is not a lot else to go on from either server around the date where I think things went sideways. There is an odd 15 day gap in the DFSR Log, but maybe that’s normal? I don’t really see any indication of a problem in the Directory Services, System or Application logs around that time.

I then installed the Active Directory Replication Status Tool (https://www.microsoft.com/en-us/download/details.aspx?id=30005), while it was interesting to run there were no issues reported. The next thing I ran was the health report from the DFS Management console, which told me more or less what I already knew, DFS was waiting on initial synchronization.

So the next thing to try was to figure out how to force that synchronization to happen. I found the following article (https://support.microsoft.com/en-us/kb/2218556) on Microsoft’s Support site. I ran through the Authoritative Synchronization steps and after going through that the servers were happy. I was able to drop a test file in SYSVOL and see it replicate to the other server, delete that file on the server and see it drop off the other server.

While I don’t really know what the root cause for this issue was, hopefully the next time it crops up I’ll be able to figure it out a lot more quickly. As a side note to the steps I followed below I would tell you that I restarted the DFSR service after making changes in ADSI, each time.

Prior to running dfsrdiag, you may need to install that feature

Add-WindowsFeature -Name RSAT-DFS-Mgmt-Con

In order to force Active Director Replication throughout the domain you will run this command


repadmin /syncall /APed

You want to force the non-authoritative synchronization of SYSVOL on a domain controller. In the File Replication Service (FRS), this was controlled through the D2 and D4 data values for the Burflags registry values, but these values do not exist for the Distributed File System Replication (DFSR) service. You cannot use the DFS Management snap-in (Dfsmgmt.msc) or the Dfsradmin.exe command-line tool to achieve this. Unlike custom DFSR replicated folders, SYSVOL is intentionally protected from any editing through its management interfaces to prevent accidents.

How to perform a non-authoritative synchronization of DFSR-replicated SYSVOL (like “D2” for FRS)

  1. In the ADSIEDIT.MSC tool modify the following distinguished name (DN) value and attribute on each of the domain controllers that you want to make non-authoritative:
    1. CN=SYSVOL Subscription,CN=Domain System Volume,CN=DFSR-LocalSettings,CN=servername,OU=Domain Controllers,DC=domainname
    2. msDFSR-Enabled=FALSE
  2. Force Active Directory replication throughout the domain.
  3. Run the following command from an elevated command prompt on the same servers that you set as non-authoritative:
    1. DFSRDIAG POLLAD
  4. You will see Event ID 4114 in the DFSR event log indicating SYSVOL is no longer being replicated.
  5. On the same DN from Step 1, set:
    1. msDFSR-Enabled=TRUE
  6. Force Active Directory replication throughout the domain.
  7. Run the following command from an elevated command prompt on the same servers that you set as non-authoritative:
    1. DFSRDIAG POLLAD
  8. You will see Event ID 4614 and 4604 in the DFSR event log indicating SYSVOL has been initialized. That domain controller has now done a “D2” of SYSVOL.

How to perform an authoritative synchronization of DFSR-replicated SYSVOL (like “D4” for FRS)

  1. Stop the DFSR service on all domain controllers
  2. In the ADSIEDIT.MSC tool, modify the following DN and two attributes on the domain controller you want to make authoritative (preferrably the PDC Emulator, which is usually the most up to date for SYSVOL contents):
    1. CN=SYSVOL Subscription,CN=Domain System Volume,CN=DFSR-LocalSettings,CN=servername,OU=Domain Controllers,DC=domainname
    2. msDFSR-Enabled=FALSE
    3. msDFSR-options=1
  3. Modify the following DN and single attribute on all other domain controllers in that domain:
    1. CN=SYSVOL Subscription,CN=Domain System Volume,CN=DFSR-LocalSettings,CN=servername,OU=Domain Controllers,DC=domainname
    2. msDFSR-Enabled=FALSE
  4. Force Active Directory replication throughout the domain and validate its success on all DCs.
  5. Start the DFSR service set as authoritative:
  6. You will see Event ID 4114 in the DFSR event log indicating SYSVOL is no longer being replicated.
  7. On the same DN from Step 1, set:
    1. msDFSR-Enabled=TRUE
  8. Force Active Directory replication throughout the domain and validate its success on all DCs.
  9. Run the following command from an elevated command prompt on the same server that you set as authoritative:
    1. DFSRDIAG POLLAD
  10. You will see Event ID 4602 in the DFSR event log indicating SYSVOL has been initialized. That domain controller has now done a “D4” of SYSVOL.
  11. Start the DFSR service on the other non-authoritative DCs. You will see Event ID 4114 in the DFSR event log indicating SYSVOL is no longer being replicated on each of them.
  12. Modify the following DN and single attribute on all other domain controllers in that domain:
    1. CN=SYSVOL Subscription,CN=Domain System Volume,CN=DFSR-LocalSettings,CN=servername,OU=Domain Controllers,DC=domainname
    2. msDFSR-Enabled=TRUE
  13. Run the following command from an elevated command prompt on all non-authoritative DCs (i.e. all but the formerly authoritative one):
    1. DFSRDIAG POLLAD

Stuck in Maintenance Mode

It’s been ages, I do have plans to start writing again, and in fact this particular posting came about because I was going to write up something I just came across at work. I went to login to my site, and was presented with a 500 error. Since I’ve recently moved the entirety of both my sites to Azure Web Apps, I was mildly concerned as I had no real access to any of the switches, buttons or levers that make things go. I checked to see if I could login to my other site and received the same error message, which I guess is good. They are both running on the same App Service in Azure and while the actual pages would load and such I just was unable to login.

I thought about restarting the App Service itself, but chose to start small. I restarted the Web Application and was able to login, I then repeated that process on the other site and logged in to it as well.

That’s when I noticed that there were some updates available, specifically the plugin for Windows Azure Storage, and WordPress itself. So I updated WP first, it took longer than HTTP was willing to hang around so I got a nifty timeout. Reloaded the page and it came back, and I updated the rest of my plugins. Then I rolled over to update this site, same process, same plugins, same timeout. Only a refresh didn’t bring the site back after the plugins updated. I gave it a little more time and still I got the maintenance page. Keep in mind there is no server that I have access to, to really do anything that I would normally do. So i did a search and came across this article.

https://premium.wpmudev.org/blog/wordpress-stuck-in-maintenance-mode

Apparently the update process drops a hidden file .maintenance in the root of the site, neat. All I needed to do was delete that file, but how do you do that when you don’t have a thing to login into?

Turns out Web Apps have a console, this can be found under the Development Tools on the Web App. This drops you into a shell on the root of your site, even though I know this is Windows it felt like Linux, so

> uname -a
D:\home\site\wwwroot
MSYS_NT-6.2-WOW RD0003FF42E552 2.5.0(0.295/5/3) 2016-03-31 18:26 i686 Msys
>

That’s neat, all I needed to do then was run ls -a to make sure .maintenance was there, and then just rm .maintenance to get rid of it, refresh and the site is back up and available. I’m going to paste the contents of that page here, in case it goes away.

Has your WordPress site ever gotten stuck in maintenance mode? Many people seem to run into this problem. They update a WordPress theme or plugin or maybe even WordPress itself, and in the middle of the update, something goes wrong, and their site gets stuck in maintenance mode with nothing appearing but the message, “Briefly unavailable for scheduled maintenance. Check back in a minute.”
Not only are visitors locked out, but even the Admin is locked out of the backend.
If this happens to you, try one of the following solutions.
1. The solution that seems to work for most is to delete a file called “.maintenance” from the root of your site. This is a temporary file that gets created in the update process, and more than likely, this is your culprit. (Notice the *dot* at the beginning of the file name.)
Again, this file rests on your server in your main WordPress install section. You CANNOT access it through the Admin area of your site. You will need to access the server through your webhost’s system (like Cpanel) or via FTP.
If you do not have access to files on your server, then contact your host and let them know the problem (and the solution, of course).

2. Although removing the .maintenance file seems to work for most, it doesn’t work for all. If it doesn’t work for you, then try the following:
a Delete the .maintenance file as outlined in option #1 above.
b Delete the plugin or theme that you were attempting to update.
c. If your site does is not back at this point, then in your wp-content folder, you will find a folder called “upgrade.” Delete the files or folders you find there.
* Remember to clear the cache in your browser (or use a different browser) to make sure that you aren’t getting an old version of your site.
Once you have your site back, you may want to run your updates again to make sure they’ve taken.