0

I recently built an app to test the speed difference between writing 100000 lines to a text file and 100000 inserts into a database

Here is the heart of the code

    private void RunTestBtn_Click(object sender, RoutedEventArgs e)
    {
        Stopwatch FFtimer = new Stopwatch();
        FFtimer.Start();
        RunFFTest();
        FFtimer.Stop();
        TimeSpan FFts = FFtimer.Elapsed;
        string FFelapsedTime = String.Format("{0:00}:{1:00}:{2:00}.{3:00}",
        FFts.Hours, FFts.Minutes, FFts.Seconds,
        FFts.Milliseconds / 10);
        FFLabel.Content = "Flat File: " + FFelapsedTime;

        Stopwatch Datatimer = new Stopwatch();
        Datatimer.Start();
        DataFFTest();
        Datatimer.Stop();
        TimeSpan Datats = Datatimer.Elapsed;
        string DataelapsedTime = String.Format("{0:00}:{1:00}:{2:00}.{3:00}",
        Datats.Hours, Datats.Minutes, Datats.Seconds,
        Datats.Milliseconds / 10);
        DBLabel.Content = "Database: " + DataelapsedTime;

    }
    void RunFFTest()
    {
        using(StreamWriter writer = new StreamWriter(@"F:\test\FFtest.txt"))
        {
            for(int i = 0; i< 100000; i++)
            {
            writer.WriteLine("test");
            }

        }
    }

    void DataFFTest()
    {
        using (SqlConnection conn = new SqlConnection("Data Source=EMMY;Initial Catalog=MyDB;User Id=SqlServiceUser;Password=MyPassword;"))
        {
            conn.Open();
            for (int i = 0; i < 100000; i++)
            {
                using (SqlCommand cmd = conn.CreateCommand())
                {
                    cmd.CommandText = "INSERT INTO TestTable VALUES ('Test')";
                    cmd.ExecuteNonQuery();
                }
            }
        }

    }

The end result is that the Flat file writing took 1 millisecond and the sql inserts took nine minutes and forty one seconds. I know the database will take longer, but is there any way to speed this up?

2 Answers 2

2

Possible options ...

  1. Use transactions (do multiple inserts per transaction)

  2. Use SqlBulkCopy API

Sign up to request clarification or add additional context in comments.

4 Comments

So in the future we are planning to do inserts on different things, so I just want a way to speed up inserts or the database in general (if thats possible)
If by "different things", you mean multiple tables, the same advice still applies. A single transaction may include DML statements for multiple tables. In the case of SqlBulkCopy, you'd have to execute a separate statement per table. So, for example, if you have 5 tables, you would execute 5 SqlBulkCopy operations. In both cases, the underlying idea is to insert in batches so that you minimize the overhead of the locking and the transaction log.
Oops sorry, should be more clear. I meant to say it would not be inserting "test" over and over, it would be inserting a variety of strings and integers, which seems like it would not work for the sqlbulkcopy
You can still use SqlBulkCopy. The easiest way is probably to create a DataTable containing the rows you want to insert, then call SqlBulkCopy with the DataTable as the source. Here's an example: myadventuresincoding.wordpress.com/2012/05/27/…
0

Yes, do it as one single insert statement, rather than many.

INSERT INTO TestTable VALUES ('Test'), ('Test'), ('Test') ....";

3 Comments

There is a maximum of 1,000 rows that can be inserted in this way and it might actually be slower.
That article takes no account of CreateCommand or ExecuteNonQuery as is being done within the loop here. I think it would be worth a try as it would not take at all long to test under the conditions in question.
The OP there says " I've got similar results running it from C# code using SqlClient, which is even more suprising considering all the DAL layers roundtrips" for short strings like test with lots of duplicates things may be better though!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.