Connect python script to node js using pythonshell package in node js

Question

I created a python program to scrape content from wikipedia page. I want to display this scraped data on localserver with node js server using express node js having pythonshell package. This is my code for Wiki.py(python program that i want to print on local host) and start.js(node js program to upload on local server) but it prints on terminal only but doesnot send any data to local host. Task is to do in this manner specifically.

Wiki.py Code:

from bs4 import BeautifulSoup
import requests

source = requests.get('https://en.wikipedia.org/wiki/Will_Smith').text
soup = BeautifulSoup(source,'html.parser')

heading = soup.find('h1',{'id':'firstHeading'}).text
print(heading)
print()

for item in soup.select("#mw-content-text"):
    required_data = [p_item.text for p_item in item.select("p")][3:6]
    print('\n'.join(required_data))

Start.js: Node js code:

const express = require('express')
const app = express()
const port = 3000
app.get('/', (req, res) => {
  var myPythonScriptPath = 'Wiki1.py';
  const {PythonShell} = require("python-shell");
  var pyshell = new PythonShell(myPythonScriptPath);

  pyshell.on('message', function (message) {
    console.log(message);
});
})
app.listen(port, () => console.log(`Example app listening on port 
${port}!`))

after your console.log write, res.send(message) to send data on hitting localhost:3000. — shiva2492
– shiva2492, Commented Jan 9, 2020 at 6:29
so I've added app.listen(port, () => console.log(Example app listening on port ${port}!)) but its not giving me any data..just data could not be received. Can you please elaborate your answer.Thank you. — Drac
– Drac, Commented Jan 9, 2020 at 8:10
I have elaborated the above point in my answer. Please check if that is what you were looking for. — shiva2492
– shiva2492, Commented Jan 9, 2020 at 9:49
so i added res.send(message);....but it only executes" heading = soup.find('h1',{'id':'firstHeading'}).text print(heading) print()" and gives heading Will smith..i.e. heading part but while executing second line of code it gives Error:: _http_outgoing.js:485 throw new ERR_HTTP_HEADERS_SENT('set'); ^ — Drac
– Drac, Commented Jan 9, 2020 at 10:29

pkoulianos · Accepted Answer · 2020-01-09 12:04:26Z

0

You can try to use node.js child_process module

index.js

const express = require('express')
const { spawn } = require('child_process');
const app = express()
const port = 3000
app.get('/', (req, res) => {
    const pyProg = spawn('python', ['script.py']);
    pyProg.stdout.on('data', function(data) {
        console.log(data.toString());
        res.send(data.toString())
    });

})
app.listen(port, () => console.log(`Example app listening on port 
${port}!`))

script.py

print('data return from python')
print('data return from python')
print('data return from python')
print('data return from python')
print('data return from python')

edited Jan 9, 2020 at 12:04

answered Jan 9, 2020 at 5:55

pkoulianos

1872 silver badges5 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

pkoulianos Over a year ago

I have just edit the index.js file with express . You have to add sys.stdout.flush() under your last print command in order to send data to node.js

pkoulianos Over a year ago

Try again and delete the last line of code ''sys.stdout.flush()"

pkoulianos Over a year ago

Download the sample app from github github.com/petranb2/node_python

shiva2492 · Accepted Answer · 2020-01-09 12:45:26Z

0

So, If all you are looking for is sending the "string" message back as response to the API then,


const express = require('express')
const app = express()
const port = 3000
app.get('/', (req, res) => {
  const myPythonScriptPath = 'wiki.py';
  const {PythonShell} = require("python-shell");
  const pyshell = new PythonShell(myPythonScriptPath);
  let string = ''

  pyshell.on('message', function (message) {
      console.log(message);
      string+=message
  });

  pyshell.end(function (err,code,signal) {
      if (err) throw err;
      console.log('The exit code was: ' + code);
      console.log('The exit signal was: ' + signal);
      console.log('finished', string);
      res.status(200).send(string)
  });  
})
app.listen(port, () => console.log(`Example app listening on port 
${port}!`))

edited Jan 9, 2020 at 12:45

answered Jan 9, 2020 at 9:48

shiva2492

4391 gold badge3 silver badges11 bronze badges

2 Comments

Drac Over a year ago

I executed this code but it only executes "heading = soup.find('h1',{'id':'firstHeading'}).text print(heading) print()" and gives heading Will smith..i.e. heading part but while executing second line of code it gives Error:: _http_outgoing.js:485 throw new ERR_HTTP_HEADERS_SENT('set');

shiva2492 Over a year ago

I have edited the answer, Please check if its what you were looking.

Collectives™ on Stack Overflow

Connect python script to node js using pythonshell package in node js

2 Answers 2

3 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related