Extract form key value pairs form html page in python or parse html page

Question

I want to extract key value pairs of some form elements in a html page

for example

name="frmLogin" method="POST" onSubmit="javascript:return validateAndSubmit();" action="TG_cim_logon.asp?SID=^YcMunDFDQUoWV32WPUMqPxeSxD4L_slp_rhc_rNvW7Fagp7FgH3l0uJR/3_slp_rhc_dYyJ_slp_rhc_vsPW0kJl&RegType=Lite_Home"

while the original line is

<form name="frmLogin" method="POST" onSubmit="javascript:return validateAndSubmit();" action="TG_cim_logon.asp?SID=^YcMunDFDQUoWV32WPUMqPxeSxD4L_slp_rhc_rNvW7Fagp7FgH3l0uJR/3_slp_rhc_dYyJ_slp_rhc_vsPW0kJl&RegType=Lite_Home">

is there any method using which I can safely get the key and value pairs. I tried using splitting by spaces and then using '=' characters but string inside quotes can also have '='.

is there any different kind of split method which can also take care of quotes?

If you search for the values of the input elements of a form (as opposed to the attributes), then this answer fits: stackoverflow.com/a/65603777/633961 — guettli
– guettli, Commented Jan 7, 2021 at 7:19

stranac · Accepted Answer · 2012-11-17 17:09:44Z

7

Use a parsing library such as lxml.html for parsing html.

The library will have a simple way for you to get what you need, probably not taking more than a few steps:

load the page using the parser
choose the form element to operate on
ask for the data you want

Example code:

>>> import lxml.html
>>> doc = lxml.html.parse('http://stackoverflow.com/questions/13432626/split-a-s
tring-in-python-taking-care-of-quotes')
>>> form = doc.xpath('//form')[0]
>>> form
<Element form at 0xbb1870>
>>> form.attrib
{'action': '/search', 'autocomplete': 'off', 'id': 'search', 'method': 'get'}

answered Nov 17, 2012 at 17:09

stranac

28.4k5 gold badges28 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

rayfranco · Accepted Answer · 2012-11-17 17:07:41Z

2

You could use regular expressions like this one :

/([^=, ]+)="([^" ]+|[^," ]+)" ?"/

In python, you can do this :

#!/usr/bin/python

import re

text = 'name="frmLogin" method="POST" onSubmit="javascript:return validateAndSubmit();" action="TG_cim_logon.asp?SID=^YcMunDFDQUoWV32WPUMqPxeSxD4L_slp_rhc_rNvW7Fagp7FgH3l0uJR/3_slp_rhc_dYyJ_slp_rhc_vsPW0kJl&RegType=Lite_Home"';

ftext = re.split( r'([^=, ]+)="([^" ]+|[^," ]+)" ?', text )

print ftext;

edited Nov 17, 2012 at 17:07

answered Nov 17, 2012 at 16:58

rayfranco

3,6903 gold badges29 silver badges38 bronze badges

2 Comments

John Dvorak Over a year ago

Escaped double quotes could pose a problem here.

rayfranco Over a year ago

@JanDvorak I've never seen any quotes escape in HTML, but indeed, it could be a problem...

applicative_functor · Accepted Answer · 2012-11-17 17:01:06Z

1

s = r'name="frmLogin" method="POST" onSubmit="javascript:return validateAndSubmit();" action="TG_cim_logon.asp?SID=^YcMunDFDQUoWV
32WPUMqPxeSxD4L_slp_rhc_rNvW7Fagp7FgH3l0uJR/3_slp_rhc_dYyJ_slp_rhc_vsPW0kJl&RegType=Lite_Home"'
>>> lst = s.split('" ')
>>> for item in lst:
...     print item.split('="')
... 
['name', 'frmLogin']
['method', 'POST']
['onSubmit', 'javascript:return validateAndSubmit();']
['action', 'TG_cim_logon.asp?SID=^YcMunDFDQUoWV32WPUMqPxeSxD4L_slp_rhc_rNvW7Fagp7FgH3l0uJR/3_slp_rhc_dYyJ_slp_rhc_vsPW0kJl&RegType=Li
te_Home"']

answered Nov 17, 2012 at 17:01

applicative_functor

4,9862 gold badges25 silver badges34 bronze badges

Comments

alexvassel · Accepted Answer · 2012-11-17 17:17:06Z

0

{i.split('="')[0]: i.split('="')[1] for i in str.split("\" ")}

where str is your original string

edited Nov 17, 2012 at 17:17

answered Nov 17, 2012 at 17:11

alexvassel

10.8k3 gold badges33 silver badges32 bronze badges

Comments

kannanrbk · Accepted Answer · 2012-11-17 17:25:54Z

0

dict=eval('dict(%s)'%name.replace(' ',','))
print dict
{'action': 'TG_cim_logon.asp?SID=^YcMunDFDQUoWV32WPUMqPxeSxD4L_slp_rhc_rNvW7Fagp7FgH3l0uJR/3_slp_rhc_dYyJ_slp_rhc_vsPW0kJl&RegType=Lite_Home', 'onSubmit': 'javascript:return,validateAndSubmit();', 'method': 'POST', 'name': 'frmLogin'}

This will solve your problem .

answered Nov 17, 2012 at 17:25

kannanrbk

7,17413 gold badges58 silver badges97 bronze badges

Comments

guettli · Accepted Answer · 2021-01-06 20:29:12Z

0

You can use a library which has support for parsing HTML forms.

For example: https://mechanize.readthedocs.io/en/latest/

Stateful programmatic web browsing in Python. Browse pages programmatically with easy HTML form filling and clicking of links.

answered Jan 6, 2021 at 20:29

guettli

27.7k109 gold badges423 silver badges779 bronze badges

Collectives™ on Stack Overflow

Extract form key value pairs form html page in python or parse html page

6 Answers 6

Comments

2 Comments

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

Comments

2 Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related