Use urlparse library to iterate through url param

Question

I need some help with the following I'm trying to do: To modify individual dictionary values for each key (a param in this case) for only a single param at a time with a value from self.fooz for each iteration.

like this

So a url like: somesite.com?id=6&name=Bill would become somesite.com?id=<self.fooz>&name=Bill (iterating for the number of individual foozes) then, somesite.com?id=6&name=<self.fooz> (iterating for the number of individual foozes)

Finally, generating a full_param_vector and full_param values as discussed below

Can someone please help me?

I've done: 1) A set of raw paths are brought in via self.path_object) 2) parse the paths after the ? to grab all the raw parametrized key/values (via parse_after)

I wrote down some pseudocode of what I'm trying to accomplish:

if self.path_object is not None:
    dictpath = {}
    for path in self.path_object:
        #path.pathToScan - returns a full url e.g. somesite.com?id=6&name=Bill
        #parse_after returns a string with parameters only, like: {u'id': [u'2'], u'name': [u'Dog']}
        parse_after = urlparse.parse_qs(path.pathToScan[path.pathToScan.find('?') + 1:], keep_blank_values=0, strict_parsing=0)
        #for each params in 'parse_after':
            #replace a key's value from params with a value from self.foozs, 
            #loop over this single key inserting a single value from self.fooz for each param for all fooz_objects, then continue to the next param and do the same
            #create full_param_vector var with these new values
            #construct full_path made up of: path.pathToScan - <part before '?'> + "?" + full_param_vector
            #add all 'full_path' to a dictionary named dictpath
        #print dictpath

Any help is welcome. Thank you

use query = urlparse.urlparse(path.pathToScan).query; parse_after = urlparse.parse_qs(query, keep_blank_values=0, strict_parsing=0) — Antti Haapala
– Antti Haapala, Commented May 29, 2014 at 19:56
The issue isnt the urlparse() but rather the data. Path.pathsToScan returns a full url like somesite.com/path/here/?param=6&name=Bill whereas your query var just returns the parameters. The part that is missing, and is messing all this up is if a path has 1 or more paths after the domain — CodeTalk
– CodeTalk, Commented May 29, 2014 at 20:03

Antti Haapala · Accepted Answer · 2014-05-29 20:36:55Z

1

Something like this could do the trick, however, I still could not quite parse your question

from collections import defaultdict
import urllib
import urlparse

# parse the url into parts
parsed = urlparse.urlparse('http://somesite.com/blog/posting/?id=6&name=Bill')

# and parse the query string into a dictionary
qs = urlparse.parse_qs(parsed.query, keep_blank_values=0, strict_parsing=0)

# this makes a new dictionary, with same keys, but all values changed to "foobar"
foozsified = { i: 'foobar' for i in qs }

# make them back to a query string: id=foobar&name=foobar
quoted = urllib.urlencode(foozsified, doseq=True)

# the original parsed result is a named tuple and cannot be changed,
# make it into a list
parsed = list(parsed)

# replace the 4th element - the query string with our new
parsed[4] = quoted

# and unparse it into a full url    
print(urlparse.urlunparse(parsed))

prints

http://somesite.com/blog/posting/?id=foobar&name=foobar

thus you can do any modifications for the qs dictionary there, and then use the urlunparse back to a full url.

edited May 29, 2014 at 20:36

answered May 29, 2014 at 20:18

Antti Haapala

135k23 gold badges297 silver badges349 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

CodeTalk Over a year ago

So, thanks for the answer, but as included in the question path.pathToScan already outputs a full url e.g. somesite.com/blog/posting/?id=foobar&name=foobar . I want to take this path and sub-in values for each param. Does the "Like this" section make sense to you above?

MxLDevs Over a year ago

@CodeTalk it's parsing the URL and parametrizing each querystring parameter in a dictionary. You can then change individual parameters, and then generate a new URL. Isn't that what you want to do?

CodeTalk Over a year ago

@Antti - could you please add some comments to it so I can follow it? Thank you

CodeTalk Over a year ago

Yeah, what I'm trying to do is modify the parameters, but only 1 self.fozz object per url, per param. so like, somesite.com?id=<fozz object>&name=Bill (do that for all the fozz object) for 'id' param.. Then do next param: somesite.com?id=6&name=<fozz object> (do that for all fozz object) for 'name' param. does that make sense to you @MxyL ? If you have an answer, please feel free to list it too. Thank you both

Antti Haapala Over a year ago

done, it is just the matter of what you do for the foozsified dictionary.

|

Collectives™ on Stack Overflow

Use urlparse library to iterate through url param

1 Answer 1

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related