Erle Robotics Python Networking Gitbook Free

Downloading Pages Through Form Submission

The task of grabbing information from a web site usually starts by reading it carefully with a web browser and finding a route to the information you need.

Figure shows the site of the National Weather Service; for our first example, we will write a program that takes a city and state as arguments and prints out the current conditions, temperature, and humidity.

When using the urllib2 module from the Standard Library, you will have to read the web page HTML manually to find the form. You can use the View Source command in your browser, search for the words “Local forecast,” and find the following form in the middle of the sea of HTML:

<form method="post" action="" ...>
<input type="text" id="zipcity" name="inputstring" size="9"
» value="City, St" onfocus="this.value='';" />
<input type="submit" name="Go2" value="Go" />

The only important elements here are the <form> itself and the<input> fields inside; everything else is just decoration intended to help human readers. This form does a POST to a particular URL with, it appears, just one parameter: an inputstring giving the city name and shows a simple Python program that uses only the Standard Library to perform this interaction, and saves the result to phoenix.html.

import urllib, urllib2
data = urllib.urlencode({'inputstring': 'Phoenix, AZ'})
info = urllib2.urlopen('', data)
content =
open('phoenix.html', 'w').write(content)

On the one hand, urllib2 makes this interaction very convenient; we are able to download a forecast page using only a few lines of code. But, on the other hand, we had to read and understand the form ourselves instead of relying on an actual HTML parser to read it. The approach encouraged by mechanize is quite different: you need only the address of the opening page to get started, and the library itself will take responsibility for exploring the HTML and letting you know what forms are present. Here are the forms that it finds on this particular page:

>>> import mechanize
>>> br = mechanize.Browser()
>>> response ='')
>>> for form in br.forms():
... print '%r %r %s' % (, form.attrs.get('id'), form.action)
... for control in form.controls:
... print ' ', control.type,, repr(control.value)
None None
» hidden v:project 'firstgov'
» text query ''
» radio affiliate ['']
» submit None 'Go'
None None
» text inputstring 'City, St'
» submit Go2 'Go'
'jump' 'jump'
» select menu ['']
» button None None

Once we have determined that we need the zipcity.php form, we can write a program like that shown in You can see that at no point does it build a set of form fields manually itself, as was necessary in our previous listing. Instead, it simply loads the front page, sets the one field value that we care about, and then presses the form’s submit button. Note that since this HTML form did not specify a name, we had to create our own filter function—the lambda function in the listing—to choose which of the three forms we wanted.

import mechanize
br = mechanize.Browser()'')
br.select_form(predicate=lambda(form): 'zipcity' in form.action)
br['inputstring'] = 'Phoenix, AZ'
response = br.submit()
content =
open('phoenix.html', 'w').write(content)

Many mechanize users instead choose to select forms by the order in which they appear in the page—in which case we could have called select_form(nr=1). But I prefer not to rely on the order, since the real identity of a form is inherent in the action that it performs, not its location on a page.