15

I am building a web application that in short is taking fairly poorly structured data in SQL Server and via Node.JS porting it over to MongoDB. The reason I need this is that I need access to data from a fairly poorly written application that is central to an organization that I do not have ability to change code on from which the initial data is getting entered on. Once translated, I can have my application do what the business is looking for.

Right now my application is polling SQL Server every 30 minutes for changes and then updating my MongoDB via Node.JS, and due to the volume of data, it is undesirable to poll much more frequently.

What I need to happen is to have real time notifications from SQL Server pushed to my Node.JS application in some way whether active or passive no Node.JS's end so that it can update my Mongo database.

The node library I am using to get the data is: https://github.com/patriksimek/node-mssql

A few possible ideas I had were:

  • Have SQL Server send out a notification of some kind to my NodeJS HTTP service endpoint
  • Have NodeJS run a streaming query that will run on my end each time changes are made
  • Write an application in C# that watches these changes and pushes them to my NodeJS HTTP endpoint.

There are a few out there that seem to talk about this, but most seem to talk about changes on the data source origination point (which I cannot change), not from SQL Server itself.

3
  • Thanks for the edit @JamesZ, making me sound smarter than I really am XD Commented Feb 12, 2016 at 22:03
  • 1
    This is a really important question. I wish it received more attention. Commented Jul 16, 2017 at 10:03
  • 1
    Note for those watching this, I am going to be bringing in some big guns for this one as I really need this functionality lol. I will be sure to post my findings here if I can resolve it! Commented Dec 27, 2018 at 16:32

3 Answers 3

10

I have a similar situation, and after researching, the solution I am going to go with is to write a .Net Core windows service that does nothing but listen for query notifications from SQL Server. When a notification arrives, it will hit a nodejs REST endpoint notifying it of the change, and allowing the nodejs application to query and process the data as needed.

I searched and searched for a way to do the Query Notifications from nodejs, but it seems nothing exists for this. You need a .Net application.

However, that said, another possibility instead of writing a .Net Core windows service (or an ASP.Net application if you want) is to use EdgeJs to actually run a .Net assembly inline with your nodejs application (in the same process).

I have decided not to go that route, and will do the windows service instead. But it should also be viable.

(I also realize this question is 2 years old, so you've probably already completed this project, but I'll leave these answers here for posterity)

Sign up to request clarification or add additional context in comments.

Comments

2

I've got to preface this by saying I don't have a real-time solution. If there's a real-time solution, I don't know it. I'm just gonna talk about decreasing the poll's load.

I'm assuming you have control over the SQL Server itself. You can set up something in SQL Server to keep track of changes and get polling to only look at changes not yet pulled by Node.

Do you have control over the update/insert process? If you do, easy, put the change tracking code alongside the insert/update code. I'm assuming you don't though, in which case, I'd recommend* looking in to triggers. These are basically event listeners attached to tables that allow you to execute SQL before, during or after the insert/update of a table, and give you access to the data being changed/inserted using the deleted and inserted tables.

* Triggers are not liked by the SQL Server community for a wrath of reasons, some of which are discussed on this SQLServerCentral article. The jist of things: They are difficult to debug, they slow down performance as they become part of the write operation, and be careful to not create circular triggers (table1 is updated which triggers trig1, which updates table1, which triggers trig1, etc.). As such, if you are doing change tracking using triggers on a table, create a separate table in which those changes are tracked, and only let the trigger update/insert that table.

Now you can set up a change track table that the trigger populates with any changes made, and when they were made, and perhaps even a bit field to say whether Node has pulled those changes yet. Now all you have to do is get Node to look at that table and only pull in changes it hasn't pulled in yet. This table's probably going to become huge pretty quick, and therefore slow for the poll, so I'd recommend putting an index on the flag or the field that states the date the change was made. Now since poll is quick, you should be able to decrease the interval.

This is not an ideal solution and may be a lot of work, so it's probably best to wait a while and see if anyone comes up with a better solution.

Comments

0

You can resolve the problem with Triggers and CLR Stored Procedures. After update a table you can execute a CLR stored procedure what do something like send a notification to your Nodejs for example.

2 Comments

It would be more helpful if you provided some code by way of example and how the OP would implement it.
Thanks for the suggestion @OneShoot! An example would be great, but no need to -1 you for it IMHO! :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.