У меня проблемы с загрузкой нескольких файлов в цикле с помощью Python Mechanize. Я также использую Beautiful Soup 4. Кажется, что в документации к любому пакету нет ответов.
Вот мой код - пожалуйста, переходите к самому циклу. Я включил все для справки:
import mechanize, cookielib, os, time
from bs4 import BeautifulSoup
fcList = ['abandoned mine land inventory points', 'abandoned mine land inventory polygons', \
'abandoned mine land inventory sites', 'coal mining operations', 'coal pillar location-mining', \
'industrial mineral mining operations', 'longwall mining panels', 'mine drainage treatment/land recycling project locations', \
'mined out areas', 'residual waste operations', 'underground mining permit']
dlLink = 'FTP Download'
dloadPath = 'C:\\Users\\SomeGuy\\Downloads'
# Browser
br = mechanize.Browser()
# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
# Select the first (index zero) form
br.select_form(nr=0)
# Input form data
br.form['Keyword']='mining'
br.submit()
html = br.response().read()
# Pass html to beautiful soup for parse
soup = BeautifulSoup(html)
htmlinks = soup.findAll("a")
# Find links with desired text
for htmlink in htmlinks:
string = str(htmlink.string)
if string.lower() in fcList:
print "Matched link!", string + ". attempting download...\n"
try:
req = br.click_link(text = string)
br.open(req)
print "URL: " + str(br.geturl)
html = br.response().read()
soup = BeautifulSoup(html)
the_tag = soup.find('a', text=dlLink)
fileURL = the_tag.get('href')
print fileURL
# attempt download
fnam = string.replace(" ", "_")
fnam = fnam.replace("/", "_")
f = br.retrieve(fileURL, os.path.join(dloadPath, fnam + ".zip"))
print f + "\n"
br.back()
except:
print "An unknown error occurred."
Выход:
>>>
Matched link! Abandoned Mine Land Inventory Points. attempting download...
URL: <bound method Browser.geturl of <mechanize._mechanize.Browser instance at 0x02D9D7B0>>
http://www.pasda.psu.edu/data/dep/AMLInventoryPoints2013_04.zip
An unknown error occurred.
Matched link! Abandoned Mine Land Inventory Polygons. attempting download...
An unknown error occurred.
Matched link! Abandoned Mine Land Inventory Sites. attempting download...
An unknown error occurred.
Matched link! Coal Mining Operations. attempting download...
An unknown error occurred.
Matched link! Coal Pillar Location-Mining. attempting download...
An unknown error occurred.
Matched link! Industrial Mineral Mining Operations. attempting download...
An unknown error occurred.
Matched link! Longwall Mining Panels. attempting download...
An unknown error occurred.
Matched link! Mine Drainage Treatment/Land Recycling Project Locations. attempting download...
An unknown error occurred.
Matched link! Mined Out Areas. attempting download...
An unknown error occurred.
Matched link! Residual Waste Operations. attempting download...
An unknown error occurred.
Matched link! Underground Mining Permit. attempting download...
An unknown error occurred.
>>>
Я считаю, что проблема может быть связана с отсутствием времени ожидания между загрузками. Этот код успешно загрузит самый первый файл в цикле, независимо от того, какой из них я выберу. Или, возможно, это какая-то другая ошибка, о которой я не знаю - я только вчера загрузил Mechanize и beautifulsoup!
