hadoop - NameNode HA when using hdfs:// URI -


with hdfs or hftp uri scheme (e.g. hdfs://namenode/path/to/file) can access hdfs clusters without requiring xml configuration files. handy when running shell commands hdfs dfs -get, hadoop distcp or reading files spark sc.hadoopfile(), because don't have copy , manage xml files relevant hdfs clusters nodes codes might potentially run.

one drawback of approach have use active namenode's hostname, otherwise hadoop throw exception complaining nn standby.

a usual workaround try 1 , try if exception caught, or connect zookeeper directly , parse binary data using protobuf.

both of these methods cumbersome, when compared (for example) mysql's loadbalance uri or zookeeper's connection string can comma-separate hosts in uri , driver automatically finds node talk to.

say have active , standby namenode hosts nn1 , nn2. simplest way refer specific path of hdfs, which:

  • can used in command-line tools hdfs, hadoop
  • can used in hadoop java api (and tools depending on spark) minimum configuration
  • works regardless of namenode active.

in scenarion instead of checking active namenode host , port combination, should use nameservice as, nameservice automatically transfer client requests active namenode.

name service acts proxy among namenodes, divert hdfs request active namenode

example: hdfs://nameservice_id/file/path/in/hdfs

sample steps create nameservice

in hdfs-site.xml file

create nameservice adding id it(here nameservice_id mycluster)

<property>   <name>dfs.nameservices</name>   <value>mycluster</value>   <description>logical name new nameservice</description> </property>

now specify namenode ids determine namenodes in cluster

dfs.ha.namenodes.[$nameservice id]:

<property>   <name>dfs.ha.namenodes.mycluster</name>   <value>nn1,nn2</value>   <description>unique identifiers each namenode in nameservice</description> </property>

then link namenode ids namenode hosts

dfs.namenode.rpc-address.[$nameservice id].[$name node id]

<property>   <name>dfs.namenode.rpc-address.mycluster.nn1</name>   <value>machine1.example.com:8020</value> </property> <property>   <name>dfs.namenode.rpc-address.mycluster.nn2</name>   <value>machine2.example.com:8020</value> </property>

there many properties involved configure namenode ha nameservice

with setup hdfs url file looks this

hdfs://mycluster/file/location/in/hdfs/wo/namenode/host

edit:

applying properties java code

configuration conf = new configuration(false); conf.set("dfs.nameservices","mycluster"); conf.set("dfs.ha.namenodes.mycluster","nn1,nn2"); conf.set("dfs.namenode.rpc-address.mycluster.nn1","machine1.example.com:8020"); conf.set("dfs.namenode.rpc-address.mycluster.nn2","machine2.example.com:8020");  filesystem fsobj =  filesystem.get("relative/path/of/file/or/dir", conf);  // use fsobj perform hdfs shell operations fsobj ...

-

Comments

Post a Comment

Popular posts from this blog

css - SVG using textPath a symbol not rendering in Firefox -

using textpath within symbol doesn't render in firefox. renders fine in latest chrome , ie, when try reference symbol, svg text doesn't render in firefox (37.0.1) - first box appears empty. code below (no external dependencies), there should 2 boxes word test flowing vertically in centre of each. edit: thought issue somehow involved flexbox layout, until paul pointed out issue exists without flexbox layout. the html is: <div> <svg id="not_working" viewbox="0 0 250 1200" preserveaspectratio="xmidymid meet"> <symbol id="test_symbol1" preserveaspectratio="xmidymid meet" viewbox="0 0 250 1200"> <path id="test_symbol_path" d="m 100 1200 l 100 0" /> <text font-size="100" fill="red"> <textpath text-anchor="middle" startoffset="50%" xlink:href="#test_symbol_path...
Read more

Java 8 + Maven Javadoc plugin: Error fetching URL -

i attempting generate javadoc links javadoc dependencies. have tried various means generate javadoc not produce qualified class names references classes dependencies. wanted links java doc simplified class names. however, java api classnames, no links , have qualified class names. working java 8. have following configuration: <plugin> <groupid>org.apache.maven.plugins</groupid> <artifactid>maven-javadoc-plugin</artifactid> <version>2.10.2</version> <configuration> <reportoutputdirectory>${project.basedir}/target</reportoutputdirectory> <destdir>javadoc</destdir> <windowtitle>epiphany</windowtitle> <doctitle>epiphany</doctitle> <show>private</show> <detectlinks>false</detectlinks> <detectoffline...
Read more

order - Notification for user in user account opencart -

thanks helps or points me in right direction. i want show pending notification user in account page after login. notification show when order status pending. after changing other status admin remove notification. this notification show like: your order id: order status: , [ button custom url] if multiple order show multiple order id. open cart version 1.5.6.3 custom theme. $this->load->model('account/order'); $results = $this->model_account_order->getorders(); foreach ($results $result) { $product_total = $this->model_account_order->gettotalorderproductsbyorderid($result['order_id']); $voucher_total = $this->model_account_order->gettotalordervouchersbyorderid($result['order_id']); if( $result['status'] =! 5){ // 5 order status means completed $this->data['orders'][] = array( 'order_id' => $result['order_id'], ...
Read more