python - scrapy: post some forms following scrapy finishing processing urls -


i using scrapy scrape data member's site. perform login , scrape data successfully.

however, need submit forms on site when scraping of data finished. i.e: following reading of data, want write data site scraping (reading) data from.

my question is:

how informed of scrapy finished processing url scraping, can perform form submissions?

i noticed solution - see here (scrapy: call function when spider quits) reason cannot continue yielding more requests in self.spider_closed method called on over examples can write operations.

yes, cannot continue using spider after spider_closed signal has been fired - late, spider closed @ moment.

a better signal use spider_idle:

sent when spider has gone idle, means spider has no further:

  • requests waiting downloaded
  • requests scheduled
  • items being processed in item pipeline

Comments

Popular posts from this blog

css - SVG using textPath a symbol not rendering in Firefox -

Java 8 + Maven Javadoc plugin: Error fetching URL -

datatable - Matlab struct computations -