SQL Compare is a tool for comparing and synchronizing the structure of SQL Server databases. More than 150,000 database administrators, developers, and testers are using it. When testing local databases, staging or activating databases on remote servers, SQL Compare automates the process of allocating databases.
Click to download the trial version of SQL Compare [Huidu.com]
Our team develops practical applications for commercial organizations; mainly Georgian Bank. These applications are based on .Net-Windows-Forms applications with MS SQL Server as the database. They have a lot of business logic, which is contained in database routines, such as stored procedures, functions, views, and SQL CLR.
Not surprisingly, given the nature of the customer’s business, we do not have the right to access customer databases or data for development, testing or deployment. Under TFS source code management, we only have the development database and its manual test data. Developers work on their own copy of the database, each copy has its own sample data, and they use Redgate SQL Source Control to submit development changes. Then, we use the SQL comparison command line to automate the database deployment. In this article, I will explain how to achieve this goal and give an example of how to compare two revisions of the database in the same or different branches and generate a deployment script.
Command line permission
Automation programs that require the SQL comparison command line to be installed on multiple computers require Redgate Deploy or SQL Toolbelt licenses. For complete details, see the "Change Command Line Distribution" page of the documentation.
Manage databases in source control
Our database source code control and branching strategy are simple. We have the latest code base in Trunk; the entire application is there, including the database part of it. All new features and bug fixes are initially carried out in the Trunk. Each branch we create is just a copy of Trunk, so it represents the complete point-in-time state of the code base. After applying some changes and checking in the Trunk, we can merge them into any of these branches as needed. Usually, this is to fix reported bugs, but when it’s important to our customers, we also need to be able to incorporate small feature changes. For example, not every customer can afford to deploy each version, so the version they deploy is usually three or four behind. However, they still need us to deploy emergency fixes for the current version, and occasionally use some "exclusive" features.
So, how does all this work when we develop software? Let us call it "under-source-control-application" (USCAPP for short). We have the latest code base in USCAPP_Trunk, and some release versions under the TFS branch, called v241, v242, etc.
All changes made directly or through merging can be done through the normal TFS check-in of the trunk and its branches. At every check-in, TFS creates something called changeset, which has a unique reference number. A change set represents a snapshot of the entire code base in source control. Like any other source code control system, TFS can generate the point-in-time state of the code base for any revision (corresponding to any given change set number).
Of course, for all TFS projects in a collection (including its branches), the TFS change set number is global, and its number will increase every time the project collection is checked in. For us, this means that USCAPP_Trunk and all its branches v241, v242, etc. all share the same, global, growing change set number.
Developers make changes, everyone works on their own dedicated database and checks in the changes through SQL Source Control, these changes will update the code in USCAPP_Trunk. As needed, we merge the required change sets to other branches and create new change sets in these branches. Therefore, suppose the latest version is v245, and we know that customer A has deployed v242 into the production environment. The customer has not yet upgraded to the latest version, but has deployed other upgrade scripts to fix some bugs and make some small improvements. In other words, customer A is running a very specific version of v242 and we can convert it to a TFS change set number, which uniquely identifies the point-in-time state of the code base of the branch v242 they have deployed.
Use SQL Compare command line to automatically generate change scripts
Our goal is to automate the process of generating a synchronous SQL script that covers all the changes that have occurred since the script was last published.
Assume that customer A has deployed branch v242, and the release version of the database is marked with a human-readable version number 2.4.2.0, which corresponds to the change set number 87300, that is, it was released when the change set 87300 is the current latest version The global change set number of the code base.
One month later, we have made changes in the database, and now the current number of change sets in TFS is 88100. Now, we want to generate a script that contains every change made in the current month, so we upgrade the v2.4.2.0 of the database to the state represented by the change set number 88100, which we call v2.4.2.1.
To do this, we need to retrieve the state of the database at two points in time from TFS, one representing the source database (will not change) and the other representing the target database (we want to upgrade). Therefore, for customer A, the change set 88100 represents the source and 87300 represents the target. We need to compare the two states to find the difference, and then generate a script to synchronize the target so that its state is the same as the source. For any database object that exists in both databases but is different, the definition of the object in the target must be changed to match its definition in the source. Any objects that exist in the source but not in the target should be created, and any objects that exist in the target but not in the source should be deleted.
The good news is that we don’t have to do this manually. Both SQL Compare GUI and SQL Compare command line support this feature. We want to automate the process, so we use the command line and pass the appropriate parameters to the command line to generate the synchronization script. We also need to carefully record the script to upgrade the 2.4.2.0 version of the database to v2.4.2.1. Of course, we also need some protective measures here. One of them is a check, which will stop running this script on any v2.4.2.0 or higher database. I will not demonstrate here, but in the end, I will discuss these requirements in more detail.
Compare two revisions in the same branch
First, I will describe how we release a script called "fix", which is mainly used to deploy some bug fixes and minor improvements. The major version remains unchanged.
We use the SQL Compare command line to do this, passing an XML parameter file (argfile) that contains the values of all required command line switches that instruct SQL Compare how to perform the comparison. Alternatively, you can specify every switch to the command line, or the "splat" parameter in PowerShell.
In this case, the only parameter that needs to be passed to SQL Compare is the qualified file name of the XML Argfile, called "shared.xml"
"%Programfiles(x86)%\Red Gate\SQL Compare 13\sqlcompare" /Argfile:"shared.xml"
The content of argfile should be filled in exactly according to the instructions in the online documentation of the SQL Compare command line. This is a real example:
<commandline>
<SourceControl1 />
<Revision1>88100</Revision1>
<SourceControl2 />
<Revision2>87300</Revision2>
<Options>NoDeploymentLogging,IgnoretSQLt,IgnoreFillFactor,IgnoreWhiteSpace,IgnoreFileGroups,IgnoreUserProperties,IgnoreFileGroups,IgnoreUserProperties,Ignore IgnoreDatabaseAndServerName,CaseSensitiveObjectDefinition,ObjectExistenceChecks,DropAndCreateInsteadofAlter,ForceColumnOrder,DoNotOutputCommentHeader,IgnoreUsersPermissionsAndRoleMemberships</Options>
<ScriptsFolderXML>Command Line\SourceControlAddress v242.xml</ScriptsFolderXML>
<Filter>
<ReportType>Interactive</ReportType>
<Report>Command Line\Output\Shared.html</Report>
<ScriptFile>Command Line\Output\Shared.sql</ScriptFile>
<Force />
<Verbose />
</commandline>
Argfile contains five command line switches, which we use to define the desired behavior. The /Sourcecontrol1 and /Sourcecontrol2 switches specify that our source, and target, are a folder of source control scripts, in this case, modify 88100 and 87300, respectively.
<SourceControl1 />
<Revision1>88100</Revision1>
<SourceControl2 />
<Revision2>87300</Revision2> The
<ScriptsFolderXML> switch contains the complete file path as an XML file, SourceControlAddress v242.xml. This file (shown below) contains the source code control address of the branch v242 database:
<?xml version="1.0" encoding="utf-16" standalone="yes"?>
<ISOCCompareLocation version="
;
<SourceControlFolder>$/USCAPP/Branches/v242/Database/Schema</SourceControlFolder>
</ISOCCompareLocation>
This is the address from which SQL Compare should restore the 87300 and 88100 change sets. When the command-line version of SQL Compare is executed, it restores these change sets to the "script folder" (at the time of writing, to a folder in Windows Temp) and uses 88100 as the source and 87300 as the target for comparison To generate the final upgrade script.
Compare databases in two different branches
The process we use to release all the new features that have been completed in Trunk is slightly different from the bugfix version, but the main concept remains the same. Also in this case, we must compare two different states of the database architecture. Even if their "source of truth" exists as a version in TFS source control, they will still be exported to a folder as what Redgate calls a "script folder". Then they can be compared as two database schemas. In this case, the difference is that instead of comparing the two revisions (or point-in-time states) represented by the change set in one TFS branch, we compare between the two branches that now represent the versions.
To proceed step by step: the process first creates a new branch from the trunk branch and assigns it an appropriate name. For example, if v2.4.2 is the last released version of the USCAPP application, then when this version is released, we have created a branch named v242. Now that we have made more changes to Trunk, logically we will release version v2.4.3, so our new branch will be called v243, and from then on, in terms of what it contains, it Will serve as an exact copy of the Trunk branch.
Now, we should compare the two change sets of two separate branches. The change set we use for comparison must be the change set of the new branch v243 just created, and the change set corresponding to the latest deployment script of the previous branch v242 that customer A has applied. This comparison will reveal changes that only occurred on the Trunk's database, which were missing from the database of the previous branch v242.
For this, we need to specify one instead of two source control folder locations, one for the TFS branch containing source/ScriptsFolderXML1 and the other for the branch containing target/ScriptsFolderXML2. We use the SQL Compare reserved keyword "HEAD" to specify that we want the latest set of source control changes for the source branch. The generated Argfile is as follows:
<commandline>
<SourceControl1 />
<Revision1>HEAD</Revision1>
<SourceControl2 />
<Revision2>88100</Revision2>
<Options>NoDeploymentLogging,IgnoretSQLt,IgnoreFillFactor,IgnoreWhiteSpace,IgnoreFileGroups,IgnoreUserProperties,IgnoreFileGroups,IgnoreUserProperties, ,IgnoreDatabaseAndServerName,CaseSensitiveObjectDefinition,ObjectExistenceChecks,DropAndCreateInsteadofAlter,ForceColumnOrder,DoNotOutputCommentHeader,IgnoreUsersPermissionsAndRoleMemberships</Options>
<ScriptsFolderXML1>Command Line\SourceControlAddress v243.xml</ScriptsFolderXML1>
<ScriptsFolderXML2>Command Line\SourceControlAddress v242.xml</ScriptsFolderXML2>
<Filter>Command Line\Filters\Shared.scpf</Filter>
<ReportType>Interactive</ReportType>
<Report>Command Line\Output\Shared.html</Report>
<ScriptFile>Command Line\Output\Shared.sql</ScriptFile>
<Force />
<Verbose />
</commandline>
这是目标的源代码管理脚本位置XML文件（SourceControlAddress v242.xml）：
<?xml version="1.0" encoding="utf-16" standalone="yes"?>
<ISOCCompareLocation version="1" type="TfsLocation">
<ServerUrl>http://tfs:8080/tfs/projects</ServerUrl>;
<SourceControlFolder>$/USCAPP/Branches/v242/Database/Schema</SourceControlFolder>
</ISOCCompareLocation>
This is one of the source codes (SourceControlAddress v243.xml):
<?xml version="1.0" encoding="utf-16" standalone="yes"?>
<ISOCCompareLocation version="1" type="TfsLocation">
<ServerUrl > http://tfs:8080/tfs/projects</ServerUrl> ;
<SourceControlFolder>$/USCAPP/Branches/v243/Database/Schema</SourceControlFolder>
</ISOCCompareLocation>
Once again, we only use the address of Argfile as the only Parameters to call the SQL Compare command line:
"%programfiles(x86)%\Red Gate\SQL Compare 13\sqlcompare" /Argfile:"shared.xml"
After completing its work on the SQL Compare command line, in the file "Shared.sql" , We have an upgrade script that can be run on the target database to upgrade it to the latest major version.
Further request
In real life, we always need to carefully check the automatically generated scripts, adding checks and controls to ensure that, for example, we have applied all the necessary upgrade scripts to the expected database version in the correct order. We also need to add and customize the automatically generated deployment scripts of SQL Compare, such as processing data insertion or adding header information to each script (when creating the script, copyright information, contact information, etc.). ), or append some dynamically generated SQL scripts to the end of each automatically generated script to identify customers.
By adjusting the deployment of SQL Compare with custom migration scripts, many of these goals can be achieved, although in fact we have encountered some difficulties, such as they slow down the running speed of SQL Source Control or pre-deployment and post-deployment scripts.
Custom state-based deployment
Phil Factor has several good articles showing how to use pre-deployment and post-deployment scripts to adapt to state-based deployment to handle tricky data migration, or to add version numbers to the target database, specify its database settings, and so on.
Another issue we need to consider is that SQL Compare migration and pre-deployment or post-deployment scripts are static, while our requirements are dynamically generated scripts. Instead, we built a simple, lightweight tool in Visual Studio that allows developers to make small dynamic additions and customizations to SQL Compare scripts.
Another complication that I avoid delving here is that, for each of our customers, our source code management backbone combines all the shared logic of the customer database with small routines that contain custom code specific to that organization. In this article, I demonstrated how to use the SQL Compare command line to deploy the database structure and code common to all customers. Although the process is basically the same as the customer-specific routines, some small adjustments are required to ensure that the unique features are always deployed only to that customer’s production database, and no customer can see specifically for another customer The logic of writing. I will describe how we achieve this goal in the next article!
in conclusion
Our experience is that Redgate Source Control and SQL Compare can work together and play a great role in our automated script generation process. SQL Compare allows very fine control of the way it extracts scripts from Git or TFS source code control, which saves us a lot of manual scripting. We can see more possibilities to automatically generate corresponding rollback (downgrade) scripts and upgrade scripts. It just needs to reverse the change set we used for the source and target and launch the SQL Compare command line! It is a multifunctional tool.