Menu

How to Detect and Remove Duplicate Records in SQL Server

Duplicate records in SQL Server can lead to inaccurate reporting, data inconsistencies, and performance issues. In this article, we’ll go over how to identify and safely remove duplicate rows while keeping at least one unique record.

Detecting Duplicates

To find duplicate records in a table, use the GROUP BY and HAVING clauses to count occurrences of each unique combination of values:

SELECT column1, column2, COUNT(*)
FROM YourTable
GROUP BY column1, column2
HAVING COUNT(*) > 1;

Replace column1, column2 with the columns that define a duplicate in your dataset.

If you need to see the actual duplicate rows, use a ROW_NUMBER() approach:

SELECT *
FROM (
    SELECT *, ROW_NUMBER() OVER (PARTITION BY column1, column2 ORDER BY id) AS row_num
    FROM YourTable
) t
WHERE row_num > 1;

Here, id should be a unique column to order the duplicates.

Removing Duplicates

Method 1: Using ROW_NUMBER()

The safest way to remove duplicates while keeping one unique record is by using ROW_NUMBER().

WITH CTE AS (
    SELECT *, ROW_NUMBER() OVER (PARTITION BY column1, column2 ORDER BY id) AS row_num
    FROM YourTable
)
DELETE FROM CTE WHERE row_num > 1;

This deletes all duplicate records while keeping the first occurrence.

Method 2: Using DISTINCT INTO a New Table

If you want to be extra cautious, create a new table with only unique records:

SELECT DISTINCT * INTO NewTable FROM YourTable;

Then, drop the old table and rename NewTable back to YourTable.

Final Thoughts

Always backup your data before running delete operations to prevent accidental data loss. By regularly cleaning up duplicates, you can keep your SQL Server database efficient and reliable.

3
35

Related

Closing a SqlDataReader correctly prevents memory leaks, connection issues, and unclosed resources. Here’s the best way to do it.

Use 'using' to Auto-Close

Using using statements ensures SqlDataReader and SqlConnection are closed even if an exception occurs.

Example

using (SqlConnection conn = new SqlConnection(connectionString))
{
    conn.Open();
    using (SqlCommand cmd = new SqlCommand("SELECT * FROM Users", conn))
    using (SqlDataReader reader = cmd.ExecuteReader())
    {
        while (reader.Read())
        {
            Console.WriteLine(reader["Username"]);
        }
    } // ✅ Auto-closes reader here
} // ✅ Auto-closes connection here

This approach auto-closes resources when done and it is cleaner and less error-prone than manual closing.

⚡ Alternative: Manually Close in finally Block

If you need explicit control, you can manually close it inside a finally block.

SqlDataReader? reader = null;
try
{
    using SqlConnection conn = new SqlConnection(connectionString);
    conn.Open();
    using SqlCommand cmd = new SqlCommand("SELECT * FROM Users", conn);
    reader = cmd.ExecuteReader();

    while (reader.Read())
    {
        Console.WriteLine(reader["Username"]);
    }
}
finally
{
    reader?.Close();  // ✅ Closes reader if it was opened
}

This is slightly more error prone if you forget to add a finally block. But might make sense when you need to handle the reader separately from the command or connection.

1
195

In C#, you can format an integer with commas (thousands separator) using ToString with a format specifier.

int number = 1234567;
string formattedNumber = number.ToString("N0"); // "1,234,567"
Console.WriteLine(formattedNumber);

Explanation:

"N0": The "N" format specifier stands for Number, and "0" means no decimal places. The output depends on the culture settings, so in regions where , is the decimal separator, you might get 1.234.567.

Alternative:

You can also specify culture explicitly if you need a specific format:

using System.Globalization;

int number = 1234567;
string formattedNumber = number.ToString("N0", CultureInfo.InvariantCulture);
Console.WriteLine(formattedNumber); // "1,234,567"
4
482

When working with URLs in C#, encoding is essential to ensure that special characters (like spaces, ?, &, and =) don’t break the URL structure. The recommended way to encode a string for a URL is by using Uri.EscapeDataString(), which converts unsafe characters into their percent-encoded equivalents.

string rawText = "hello world!";
string encodedText = Uri.EscapeDataString(rawText);

Console.WriteLine(encodedText); // Output: hello%20world%21

This method encodes spaces as %20, making it ideal for query parameters.

For ASP.NET applications, you can also use HttpUtility.UrlEncode() (from System.Web), which encodes spaces as +:

using System.Web;

string encodedText = HttpUtility.UrlEncode("hello world!");
Console.WriteLine(encodedText); // Output: hello+world%21

For .NET Core and later, Uri.EscapeDataString() is the preferred choice.

28
1409