I’ve observed this issue on Windows Server 2016, where one of the nodes were quarantined after many node failover attempts in an hour. For the following few hours, the WSFC prevents the node from rejoining the cluster. This could be due to a network issue that I’ve noticed in the current environment. Quarantined receives a ping response from one node. Intermittently, the ping response from one node to the quarantined issue node receives a Timeout response.
For more details go through below link it helped me a lot :
Issue : For a few minutes, there was a cluster failure that affected the availability group, and the replicas went into a resolving state. The replicas returned to their regular primary and secondary states once the cluster was brought back online, however several databases were still not synchronising. Furthermore, the databases on the primary were unavailable.
Resolution : The only remedy we found was to restart the primary replica’s SQL instance. Even a simple restart of the SQL service through the config manager, however, was stuck on “stopping service.” We had to use the TSQL command “SHUTDOWN WITH NOWAIT” to make SQL to stop. The databases were available again after SQL was brought back up, and AG was in sync and healthy.
Issue : The secondary replica does not correctly transition to the primary role if an automated failover event fails. As a result, the availability replica will indicate that this replica is in the process of resolving. Furthermore, the availability databases state that they are not synchronising, and apps are unable to access them.
Reasons to occur :
Case 1 : The value of “Maximum Failures in the Specified Period” has been reached. The clustered resource will fail three times in a six-hour period if the default behaviour is followed. RESOLVING STATE is applied to the AG replica.
Case 2 : Local SQL Server NT AUTHORITY\SYSTEM login account has insufficient NT Authority SYSTEM account rights. The following permissions are provided by default to this local login account:
Alter Any Availability Group Connect SQL View server state
Case 3 : If one of the availability databases in the availability group is in the synchronizing or not synchronized state, automatic failover will not be able to successfully transition the secondary replica into the primary role.
5 . Diagnose Unexpected Failover or Availability Group in RESOLVING State
Lease Timeout : A lease timeout can be triggered if SQL Server does not react within the normal 20-second lease timeout period.
Lease Timeout CAUSE – 100% CPU Utilization: A lease timeout might occur if CPU utilization is extremely high for an extended length of time. Using Performance Monitor, keep an eye on your CPU usage.
Sync Issues :
how to troubleshoot Always On synchronization issue? Multiple reasons for the database status changing to not in sync
I had a circumstance where I needed to restore 100’s of databases from a source server to a destination server, and we had to do it manually one by one, or manually build restore scripts for all databases using T-SQL scripts, or automate the process using PowerShell. I’m planned to use PowerShell to move the databases, however while searching, I came across DBATOOLS, which appears to be a quick and straightforward operation. To accomplish this, we must first install the DBATOOLS Module, and then update it if it is already installed. We used to get Database Refresh requests from Production to DEV/TEST on a regular basis as Database Administrators.
I installed and upgraded DBATOOLS on my machine, then I developed 3 databases to test migration from one instance to another. I already have database backup files in a folder to conduct our work, and now I’m testing to start restoring all 3 databases using DBATOOLS in PowerShell. To restore a database using DBATOOLS, we need to know the Restore command in PowerShell. We can check this in PowerShell using the Get-help *restore* command, which will display all of the available restore commands as well as a synopsis to help us figure out which restore command we can use to complete our task.
I use the command Restore-DbaDatabase from the DBATOOLS commands listed above. I’m only using this command since my backup files are already in a directory with Databases Full Backups, and I need to restore databases from these backup files.
In the above screenshot you can see 3 backup full files for test purpose, I use above backup path in our restore command to restore databases in my destination instance. Also I need to specify some parameters in Restore command like Destination Instance Name, backup Path, destination DATA Directory and Destination LOG Directory
Below is the Complete Restore command used in PowerShell :
You can see in the above Snippet that I ran the Restore command with the parameters passed in, and at the top you can see the restore database operation is in progress, as well as the percent completion of the restore is showing in a Progress bar. Once the Database Restore is complete, it generates an Output that contains the information details. In the Output, we can see the Script Field, which contains the T-sql script used by DBATOOLS to restore databases. We may copy that T-sql script and use it for future restorations.
All 3 databases were successfully restored to my destination instance and the databases are ONLINE.
We may generate Restore Script for all Databases in a Backup path without restoring them to the Destination Instance by using the same command. We just need to add one parameter to the Restore command: -OutputScriptonly. Below is the Full Restore command :
Using a combination of PowerShell and T-SQL scripting, create a simple solution for automating database migration across SQL Server versions. In my instance, I need to locate multiple databases that can be migrated using a standard procedure or at the very least a set of standard activities before proceeding to a set of customized actions. In these situations, you may want to create generic migration scripts and create a system to automate their execution. If you’re going to iterate through numerous test sessions and want to migrate a new copy before each one, a general technique like this can be extremely useful.
The activities that can be taken to make the migration process easier are as follows:
1) The backup database from the Source Instance will be transferred. 2) Restore the database that will be moved to the new instance. 3) Refresh the allocation usage. 4) Data purity can be used to check database consistency. 5) Set the level of page verification. 6) Set the database compatibility level to match the version of the destination instance (e.g. 120). 7) Statistics should be updated. 8) Remove any orphaned schemas (schemas automatically created for each database user to maintain compatibility but with no objects belonging to them). 9) Fix Orphaned Users if any exists. 10) Run Rebuild or re-organize all databases based on Migration Change window and the databases size.
Today, I’m automating the databases backup and restore of the first two Point. I still need to work on automating the other points, which are currently in process. I’m hoping that this backup and restore PowerShell script may be useful to some of the DBAs, and that I can share it now.
When you issue backup and restore commands in T-SQL, you’ll use handlers that are immediately called back by SMO, and you’ll provide a simple progress notification like the one provided by SSMS or sqlcmd. Please keep in mind that the completion handler just displays the error message. Get-SQLInstance, DatabaseFullBackup, and RestoreDatabaseFromFullBackup are only a few of the functions in the Powershell script.
Below is the PowerShell script that will implement backup/restore procedures for each database which we want to migrate to complete the migration :
You’ll need to update a few parameters in the script above, such as SourceInstanceName, Destination Instance Name, Backup Path, DATA File Path, LOG File Path and the Databases Names .
Once all of the parameters have been adjusted to your environment, the script will begin taking database backups from the source instance and restoring them to destination instance.
You can see below snippet where I don;t have any backup files , any Data and Log files:
Also I don’t have any Databases in my Destination Instance.
Now I’m backing up and restoring my “BLOB”,”panda”,”dbareports” databases with the PowerShell Script above. When I run the script, it will back up the first database, restore it to the Destination Instance after the backup is complete, and then take backups for the second database, It will begin restoring after the backup has been completed.
You can see in above Screenshot when Panda Database backup completed immediately it start s to Restore Panda Database in Destination , once the restore completed it will start backup for next Database (dbareports database) and will restore
The script completed migration of databases which I mentioned the DatabasesField
All three databases have been successfully migrated from the Source Instance to the Destination Instance. Now all you have to do is run the script in PowerShell ISE and it will take care of migrating your provided databases.