how to dump csv data into mongoDB using angular and nodejs

Question

I've a csv file and want to dump the data in it into mongodb using angular and nodejs.

Need help to read csv file's data using angular and parse it and store in mongodb

import { Injectable } from '@angular/core';
import { Http } from '@angular/http';
import 'rxjs/add/operator/map';

@Injectable()
export class TemplateService {

  domain = 'http://localhost:3000';
  constructor(private http: Http) { }


  postCsvToMongo(csvJson: any) {
    console.log('CSV!!!' + csvJson);
    return this.http.post(this.domain + '/fu/fileupload', csvJson).map(res => res.json());
  }

}

this is the code i'm using to send json object over post routes.

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <title>Error</title>
    </head>
    <body>
        <pre>TypeError: Converting circular structure to JSON
            <br> &nbsp; &nbsp;at JSON.stringify (&lt;anonymous&gt;)
            <br> &nbsp; &nbsp;at router.post (/home/agam/projects/AMEX/routes/fileUpload.js:10:23)
            <br> &nbsp; &nbsp;at Layer.handle [as handle_request] (/home/agam/projects/AMEX/node_modules/express/lib/router/layer.js:95:5)
            <br> &nbsp; &nbsp;at next (/home/agam/projects/AMEX/node_modules/express/lib/router/route.js:137:13)
            <br> &nbsp; &nbsp;at Route.dispatch (/home/agam/projects/AMEX/node_modules/express/lib/router/route.js:112:3)
            <br> &nbsp; &nbsp;at Layer.handle [as handle_request] (/home/agam/projects/AMEX/node_modules/express/lib/router/layer.js:95:5)
            <br> &nbsp; &nbsp;at /home/agam/projects/AMEX/node_modules/express/lib/router/index.js:281:22
            <br> &nbsp; &nbsp;at Function.process_params (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:335:12)
            <br> &nbsp; &nbsp;at next (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:275:10)
            <br> &nbsp; &nbsp;at Function.handle (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:174:3)
            <br> &nbsp; &nbsp;at router (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:47:12)
            <br> &nbsp; &nbsp;at Layer.handle [as handle_request] (/home/agam/projects/AMEX/node_modules/express/lib/router/layer.js:95:5)
            <br> &nbsp; &nbsp;at trim_prefix (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:317:13)
            <br> &nbsp; &nbsp;at /home/agam/projects/AMEX/node_modules/express/lib/router/index.js:284:7
            <br> &nbsp; &nbsp;at Function.process_params (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:335:12)
            <br> &nbsp; &nbsp;at next (/home/agam/projects/AMEX/node_modules/express/lib/router/index.js:275:10)
        </pre>
    </body>
</html>

this is the error i'm getting when I'm trying to debug the request received over the post request on node side. And how to receive the file over back end??

var csv = require('fast-csv');
var mongoose = require('mongoose');
var Bulletin = require('../models/bulletin');
var multer = require("multer");

module.exports = (router) => {

    router.post('/fileupload', (req, res) => {
        console.log('@@@ ' + req);
        res.send(req.files);
    });

    /*router.post('/fileupload', multer({dest: "./uploads/"}).array("uploads", 12), function(req, res) {
        res.send(req.files);
    });*/
    return router;
};

This is the nodejs code

1","Fed official says weak data caused by weather, should not slow taper","http://www.latimes.com/business/money/la-fi-mo-federal-reserve-plosser-stimulus-economy-20140310,0,1312750.story\?track=rss","Los Angeles Times","b","www.latimes.com","1394470370698
2","Fed's Charles Plosser sees high bar for change in pace of tapering","http://www.livemint.com/Politics/H2EvwJSK2VE6OF7iK1g3PP/Feds-Charles-Plosser-sees-high-bar-for-change-in-pace-of-ta.html","Livemint","b","www.livemint.com","1394470371207
3","US open: Stocks fall after Fed official hints at accelerated tapering","http://www.ifamagazine.com/news/us-open-stocks-fall-after-fed-official-hints-at-accelerated-tapering-294436","IFA Magazine","b","www.ifamagazine.com","1394470371550
4","Fed risks falling 'behind the curve', Charles Plosser says","http://www.ifamagazine.com/news/fed-risks-falling-behind-the-curve-charles-plosser-says-294430","IFA Magazine","b","www.ifamagazine.com","1394470371793
5","Fed's Plosser: Nasty Weather Has Curbed Job Growth","http://www.moneynews.com/Economy/federal-reserve-charles-plosser-weather-job-growth/2014/03/10/id/557011","Moneynews","b","www.moneynews.com","1394470372027
6","Plosser: Fed May Have to Accelerate Tapering Pace","http://www.nasdaq.com/article/plosser-fed-may-have-to-accelerate-tapering-pace-20140310-00371","NASDAQ","b","www.nasdaq.com","1394470372212

This is my sample csv file

> [email protected] start /home/agam/projects/AMEX
> node index.js

(node:6478) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
listening on port 3000
Conected to database: amex
@@@ [object Object]

this is the thing it is printing

// 1) Express for nodejs
const express = require('express');
const app = express();

const template = require('./routes/template');
const upload = require('./routes/upload');
const path = require('path');
// const fileUpload = require('express-fileupload');

const router = express.Router();
const fu = require('./routes/fileUpload')(router);

//const carts = require('./routes/carts')(router);

// 5) body parser
const bodyParser = require('body-parser');

const cors = require('cors'); // CORS is a node.js package for providing a Connect/Express middleware that can be used to enable CORS with various options.

// 2) Mongodb connection
var mongoose = require('mongoose');
const config = require('./config/database');
mongoose.connect(config.uri, (err) => {
    if (err) {
        console.log('error ' + err);
    } else {
        console.log('Conected to database: ' + config.db);
    }
});

app.use(cors({ origin: 'http://localhost:4200' }));
// 5) put bodyparser code before the routes (converts request to req.body)
app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json());


app.use(function(req, res, next) {
    res.header("Access-Control-Allow-Origin", "*");
    res.header('Access-Control-Allow-Methods', 'PUT, GET, POST, DELETE, OPTIONS');
    res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
    next();
  });

// 3) connect node.js to angular 
app.use(express.static(__dirname + '/client/dist/'));

// app.use(fileUpload());

app.use('/fu', fu); // Use Product routes in application
//app.use('/carts', carts);
// 



app.get('*', (req, res) => {
    res.sendFile(path.join(__dirname + '/client/dist/index.html'));
});



app.listen(3000, () => {
    console.log('listening on port 3000');
});

this is my index.js file which you are expecting

were you able to read the csv file in angular and convert it to json , because that seems the obvious approach. — Vaibhav
– Vaibhav, Commented Aug 16, 2018 at 4:57
Actually I don't know the approach and even don't know how to read csv file in angular and convert it to json. Please help me out as I'm new in this field and learning — agamjain14
– agamjain14, Commented Aug 16, 2018 at 5:32
stackblitz.com/edit/angular-mvq5ju Please use this link to access the code. I've created the project on stackblitz.com Please use the above link — agamjain14
– agamjain14, Commented Aug 16, 2018 at 9:10
@NicholasKyriakides Please find the project on stackblitz.com/edit/angular-mvq5ju — agamjain14
– agamjain14, Commented Aug 16, 2018 at 9:15

Vaibhav · Accepted Answer · 2018-08-17 17:10:48Z

Hi I have done a few changes in your StackBiltz example, so I'm able to achieve the payload to be posted via a post call to in the form of JSON array and created a service which accepts this payload and on subscription passes the payload to your node post api

Created a function named pass

test.component.ts

import { Component, OnInit } from '@angular/core';    
import { Router } from "@angular/router";
import { FileUtil } from './file.util';
import { Constants } from './test.constants';
import {TestService} from './test.service';

@Component({
  selector: 'app-test',
  templateUrl: './test.component.html',
  styleUrls: ['./test.component.css']
})
export class TestComponent implements OnInit {

  fileImportInput: any;
  data;
  csvRecords:any = [];

  constructor(private _router: Router,
    private _fileUtil: FileUtil,private testService:TestService
  ) { }

  ngOnInit() {
  }

  // METHOD CALLED WHEN CSV FILE IS IMPORTED
  fileChangeListener($event): void {

    var text = [];
    var target = $event.target || $event.srcElement;
    var files = target.files;

    if (Constants.validateHeaderAndRecordLengthFlag) {
      if (!this._fileUtil.isCSVFile(files[0])) {
        alert("Please import valid .csv file.");
        this.fileReset();
      }
    }

    var input = $event.target;
    var reader = new FileReader();
    reader.readAsText(input.files[0]);

    reader.onload = (data) => {
      let csvData = reader.result;
      let csvRecordsArray = csvData.split(/\r\n|\n/);

      var headerLength = -1;
      if (Constants.isHeaderPresentFlag) {
        let headersRow = this._fileUtil.getHeaderArray(csvRecordsArray, Constants.tokenDelimeter);
        headerLength = headersRow.length;
      }

      this.csvRecords = this._fileUtil.getDataRecordsArrayFromCSVFile(csvRecordsArray,
        headerLength, Constants.validateHeaderAndRecordLengthFlag, Constants.tokenDelimeter);
        this.passThisFunction(this.csvRecords);
        console.log('this.csvRecords',this.csvRecords);
      if (this.csvRecords == null) {
        //If control reached here it means csv file contains error, reset file.
        this.fileReset();
      }      
    }
    reader.onerror = function () {
      alert('Unable to read ' + input.files[0]);
    };
  };

  fileReset() {
    this.fileImportInput.nativeElement.value = "";
    this.csvRecords = [];
  }

  passThisFunction(records:any){
      console.log('inside function call');
      console.log('records',records);
      let json=JSON.stringify(records);
      console.log('jsonified',json);
      var str = json.replace(/},/g, "},\r\n");
        console.log('das' + str);
      this.testService.postCsvToMongo(str).subscribe((data) => {
                console.log(data);
      });
  }

}

test.component.html

<h4>Read .CSV File using Angular2, JQuery and Bootstrap</h4>

<table>
    <tr>
        <td>
            <input type="file" 
                #fileImportInput
                name="File Upload" 
                id="txtFileUpload" 
                class="btn btn-primary" 
                (change)="fileChangeListener($event)" 
                accept=".csv"
                />
        </td>

    </tr>
</table>

<div>
    <table class="table table-responsive table-hover" border="1" 
        style="width : 50%;">
        <tbody>
            <ng-container >
                <tr >                   
                        <td *ngFor="let csvRec of csvRecords;let i=index">{{csvRec}}</td>
                </tr>
            </ng-container>
        </tbody>
    </table>
</div>

file.util.ts

import { Injectable }       from '@angular/core';

@Injectable()
export class FileUtil {

    constructor() {}

    isCSVFile(file) {
        return file.name.endsWith(".csv");
    }

    getHeaderArray(csvRecordsArr, tokenDelimeter) {        
        let headers = csvRecordsArr[0].split(tokenDelimeter);
        let headerArray = [];
        for (let j = 0; j < headers.length; j++) {
            headerArray.push(headers[j]);
        }
        return headerArray;
    }

    validateHeaders(origHeaders, fileHeaaders) {
        if (origHeaders.length != fileHeaaders.length) {
            return false;
        }

        var fileHeaderMatchFlag = true;
        for (let j = 0; j < origHeaders.length; j++) {
            if (origHeaders[j] != fileHeaaders[j]) {
                fileHeaderMatchFlag = false;
                break;
            }
        }
        return fileHeaderMatchFlag;
    }

    getDataRecordsArrayFromCSVFile(csvRecordsArray, headerLength, 
        validateHeaderAndRecordLengthFlag, tokenDelimeter) {
          console.log(csvRecordsArray);
          console.log(headerLength);
          console.log(validateHeaderAndRecordLengthFlag);
          console.log(tokenDelimeter);
        var dataArr:any = [];

        for (let i = 0; i < csvRecordsArray.length; i++) {
            let data = csvRecordsArray[i].split(tokenDelimeter);

            if(validateHeaderAndRecordLengthFlag && data.length != headerLength){
                if(data==""){
                    alert("Extra blank line is present at line number "+i+", please remove it.");
                    return null;
                }else{
                    alert("Record at line number "+i+" contain "+data.length+" tokens, and is not matching with header length of :"+headerLength);
                    return null;
                }
            }
            dataArr=data;
            console.log('dataArr',dataArr);
        }   
        return dataArr;
    }

}

test.service.ts

import{Injectable} from '@angular/core';
import{Http} from '@angular/http';
import {map} from 'rxjs/operators';

@Injectable()
export class TestService{
  API_URL:string='your api url';
  constructor(private http:Http){}

  postCsvToMongo(csvJson:any){
    return this.http.post(this.API_URL,csvJson)
    .pipe(map((res)=>console.log(res)));
  }
}

Thanks for the help. I'm able to form json objects, but when i'm sending for post routes, it is giving bad request error
whats the error can you share your node api calls? and the node server files code?

Collectives™ on Stack Overflow

how to dump csv data into mongoDB using angular and nodejs

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related