Cannot POST data to URL using Pythons requests library from external file

Question

I'm trying to POST data to a URL using Pythons requests library.

If I try and do this by setting a multiline string variable which contains the post data in my script, everything works fine.

If I try to read in an external file with the same data in, the request fails on the application server I'm posting to, because it thinks there is invalid XML.

For example:

This works

starturl="http://myserver.example.com/location/where/I/post"
username=user
password=mypassword

# Set the XML data
xmldata="""<?xml version="1.0" encoding="utf-8"?>
(Lots more xml)
"""

# POST the job data

session = requests.Session()

request = session.post(starturl, auth=(username,password), data=xmldata, headers=post_headers)

Server side application processes the request just fine. However, if the only change I make is to read the xml data from an external file, this no longer works.

This does not work

xmlfile="/path/to/my/xmldata.xml"
xmldata = open(xmlfile,'r')


session = requests.Session()
request = session.post(start_url, auth=(username,password), data=xmldata.read(), headers=post_headers)

The server side application, then errors with:

"Data at the root level is invalid. Line 1, position 1"

When inspecting with wireshark I can see there is a difference in the request body of my POST. Three little dots are appearing from somewhere

When it works:

Content-Type: application/xml
Authorization: Basic c3BvdGFkbTpQQHNzdzByZA==

<?xml version="1.0" encoding="utf-8"?>

When it fails:

Content-Type: application/xml
Authorization: Basic c3BvdGFkbTpQQHNzdzByZA==

...<?xml version="1.0" encoding="utf-8"?>

I'm not sure what's causing the 3 leading dots to appear in the request body. I've inspected the source XML file, tried stripping newlines from it. Nothing seems to do the trick?

Well, you can simply post requests with data from file xmldata without reading it. See this part of documentation. Just use request = session.post(starturl, auth=(username,password), data=xmldata, headers=post_headers) where xmldata is opened file. However I am not sure if this will solve your problem. — running.t
– running.t, Commented Nov 22, 2017 at 12:16
Try with binary read: xmldata = open(xmlfile,'rb') As long as error can not be reproduced, there will be no exact answer. — Elis Byberi
– Elis Byberi, Commented Nov 22, 2017 at 12:23
It's impossible to debug without having your xml file, but you maight have a BOM at the beginning of your file. Microsoft is notably (in)famous for insisting on putting useless BOM on all utf-8 files. You can check the first three characters of your file for the codecs.BOM_UTF8 sequence ('\xef\xbb\xbf') and strip it out if it's there. — bruno desthuilliers
– bruno desthuilliers, Commented Nov 22, 2017 at 12:47
@brunodesthuilliers You were right! The XML file was copied from a Windows system and the Python script was running on a Linux system. After careful inspection of the source, a BOM was indeed present. Removing it solves the problem. Many thanks! — Aditya K
– Aditya K, Commented Nov 22, 2017 at 14:27
@AdityaK glad I could help. I allowed myself to repost the comment as an answer, feel free to accept it ;) — bruno desthuilliers
– bruno desthuilliers, Commented Nov 22, 2017 at 15:37

bruno desthuilliers · Accepted Answer · 2017-11-22 15:09:24Z

1

It's impossible to tell for sure without having your xml file, but you might have a BOM at the beginning of your file. Microsoft is notably (in)famous for insisting on putting useless BOM on all utf-8 files.

You can check the first three characters of your file for the codecs.BOM_UTF8 sequence ('\xef\xbb\xbf') and strip it out if it's there.

answered Nov 22, 2017 at 15:09

bruno desthuilliers

78.3k6 gold badges102 silver badges129 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Cannot POST data to URL using Pythons requests library from external file

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related