javascript - Is it possible to crawl directly through a site tree a site tree remotely or locally? -


i'm n00b web development , have n00b question.

suppose there's site is, example, like

index.php     page1.php     page2.php       page2-1.php       page2-2.php     page3.php

is there way can try go directly every subpage starting index, without knowledge of subpage names? in concrete terms, possible in, say, javascript, construct function works like

console.log(printsitetree("stackoverflow.com"); /* prints:      stackoverflow.com      stackoverflow.com/questions             .             .             .             stackoverflow.com/questions/29633992             .             .             .                 stackoverflow.com/questions/29633992/is-there-any-tool-to-calculate-the-distance-between-a-program-point-and-a-execut             .             .             .      stackoverflow.com/tags      .      .      . */

without relying on undue brute force?

theory

you can list of links on site, if site wants let have them. done via site map: http://en.wikipedia.org/wiki/site_map

usually, site provides location of sitemap in robots.txt file, crawlers can access it. xml file url's nested under sitemap/loc.

example

let's want links crawl http://www.msn.com/.
can go usual robots file location, is: http://www.msn.com/robots.txt , there can find line:
sitemap: http://sitemap.msn.com/xml
visit url , our url list:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <sitemap> <loc>http://sitemap.msn.com/xml/en-nz-autos-0</loc> </sitemap> <sitemap> <loc>http://sitemap.msn.com/xml/en-nz-entertainment-0</loc> </sitemap> <sitemap> <loc>http://sitemap.msn.com/xml/en-nz-entertainment-1</loc> </sitemap> <sitemap> <loc>http://sitemap.msn.com/xml/en-nz-finance-0</loc> </sitemap>

disclaimer

not sites give this, , there no guarantee links there or complete list. it's figure out if it's useful purpose.


-

Comments

Post a Comment

Popular posts from this blog

css - SVG using textPath a symbol not rendering in Firefox -

using textpath within symbol doesn't render in firefox. renders fine in latest chrome , ie, when try reference symbol, svg text doesn't render in firefox (37.0.1) - first box appears empty. code below (no external dependencies), there should 2 boxes word test flowing vertically in centre of each. edit: thought issue somehow involved flexbox layout, until paul pointed out issue exists without flexbox layout. the html is: <div> <svg id="not_working" viewbox="0 0 250 1200" preserveaspectratio="xmidymid meet"> <symbol id="test_symbol1" preserveaspectratio="xmidymid meet" viewbox="0 0 250 1200"> <path id="test_symbol_path" d="m 100 1200 l 100 0" /> <text font-size="100" fill="red"> <textpath text-anchor="middle" startoffset="50%" xlink:href="#test_symbol_path...
Read more

Java 8 + Maven Javadoc plugin: Error fetching URL -

i attempting generate javadoc links javadoc dependencies. have tried various means generate javadoc not produce qualified class names references classes dependencies. wanted links java doc simplified class names. however, java api classnames, no links , have qualified class names. working java 8. have following configuration: <plugin> <groupid>org.apache.maven.plugins</groupid> <artifactid>maven-javadoc-plugin</artifactid> <version>2.10.2</version> <configuration> <reportoutputdirectory>${project.basedir}/target</reportoutputdirectory> <destdir>javadoc</destdir> <windowtitle>epiphany</windowtitle> <doctitle>epiphany</doctitle> <show>private</show> <detectlinks>false</detectlinks> <detectoffline...
Read more

order - Notification for user in user account opencart -

thanks helps or points me in right direction. i want show pending notification user in account page after login. notification show when order status pending. after changing other status admin remove notification. this notification show like: your order id: order status: , [ button custom url] if multiple order show multiple order id. open cart version 1.5.6.3 custom theme. $this->load->model('account/order'); $results = $this->model_account_order->getorders(); foreach ($results $result) { $product_total = $this->model_account_order->gettotalorderproductsbyorderid($result['order_id']); $voucher_total = $this->model_account_order->gettotalordervouchersbyorderid($result['order_id']); if( $result['status'] =! 5){ // 5 order status means completed $this->data['orders'][] = array( 'order_id' => $result['order_id'], ...
Read more