Exporting Large Amounts of Data from SQL Server using ASP.Net

I architected something like this in a former life. Essentially a C# command-line app that ran queries against our reporting server.

Some key points:

• It didn't matter how long they took because it was in the background - when a file was ready, it would show up in the UI and the user could download it. They weren't waiting in real time.

• It didn't matter how inefficient the query was - both because of the point above, and because the reports were geared to the previous day. We ran them off of a reporting copy of production, not against production, which was kept on a delay via log shipping.

• We didn't set any expiration on the files because the user could request a report on a Friday and not expect to look at it until Monday. We'd rather have a file sitting around on the disk than run the report again (file server disk space is relatively cheap). We let them delete reports once they were done with them, and they would do so on their own to prevent clutter in the UI.

• We used C# and a DataReader rather than bulk export methods because of various requirements for the data. We needed to provide the flexibility to include column headers or not, to quote data or not, to deal with thousand separators and decimal points supporting various cultures, to apply different line endings (\r\n, \n, \r), different extensions, Unicode / non-Unicode file formats, and different delimiters (comma, tab, pipe, etc). There were also different requirements for different customers and even for different reports - some wanted an e-mail notification as soon as a report was finished, others wanted a copy of the report sent to some FTP server, etc.

There are probably a lot of requirements you haven't thought of yet but I hope that gives you a start.