java - How to process two files in Hadoop Mapreduce? -


i have process 2 related files in hadoop environment using mapreduce. first file huge log files has been logging user's activity. , second file relatively small file contains details users. both .txt files. first file(log file) has format of:

userid | logintime | logouttime | roomnum | machineid

this file huge(couple of tb).

the second file(user file small file 20mb) is:

userid | userfname | userlname | dob | address

i have find out frequency of users's usage of lab machines, frequent user , list names.

i know how process 1 file if in there. since user details in other folder becoming hard me process it. new mapreduce seeking , advices here. problems similar joining 2 table in rdbms foreign key me.

you can use distributed cache save small file. distributed cache stored in memory , distributed across clusters running map reduce tasks.

add file distributed cache in following way.

configuration conf = new configuration(); distributedcache.addcachefile(new uri("/user/xxxx/cachefile/exmple.txt"), conf); job job = new job(conf, "wordcount");

and file setup method of mapper , play around data in map or reduce method.

public void setup(context context) throws ioexception, interruptedexception{     configuration conf = context.getconfiguration();     path[] localfiles = distributedcache.getlocalcachefiles(conf);     //etc }

alternatively can use different mappers process


-

Comments

Post a Comment

Popular posts from this blog

css - SVG using textPath a symbol not rendering in Firefox -

using textpath within symbol doesn't render in firefox. renders fine in latest chrome , ie, when try reference symbol, svg text doesn't render in firefox (37.0.1) - first box appears empty. code below (no external dependencies), there should 2 boxes word test flowing vertically in centre of each. edit: thought issue somehow involved flexbox layout, until paul pointed out issue exists without flexbox layout. the html is: <div> <svg id="not_working" viewbox="0 0 250 1200" preserveaspectratio="xmidymid meet"> <symbol id="test_symbol1" preserveaspectratio="xmidymid meet" viewbox="0 0 250 1200"> <path id="test_symbol_path" d="m 100 1200 l 100 0" /> <text font-size="100" fill="red"> <textpath text-anchor="middle" startoffset="50%" xlink:href="#test_symbol_path...
Read more

Java 8 + Maven Javadoc plugin: Error fetching URL -

i attempting generate javadoc links javadoc dependencies. have tried various means generate javadoc not produce qualified class names references classes dependencies. wanted links java doc simplified class names. however, java api classnames, no links , have qualified class names. working java 8. have following configuration: <plugin> <groupid>org.apache.maven.plugins</groupid> <artifactid>maven-javadoc-plugin</artifactid> <version>2.10.2</version> <configuration> <reportoutputdirectory>${project.basedir}/target</reportoutputdirectory> <destdir>javadoc</destdir> <windowtitle>epiphany</windowtitle> <doctitle>epiphany</doctitle> <show>private</show> <detectlinks>false</detectlinks> <detectoffline...
Read more

c - Expected expression before 'struct' - Error -

i'm using crossstudio ide make simple function prototype initializes uart baudrate , other parameters stm32f30x arm processor. goal of function prototype print baudrate initialization peripheral (stm32f30x.c), expected '9600'. instead error "expected expression before 'usart_init'" returned. there 3 files in total: 1) config_uart.c -> contains function prototype 2) stm32f30x_usart.c -> contains initializing function, call config_uart.c 3) stm32f30x_usart.h -> contains prototype struct definition config_uart.c body void comm_initial (void) { typedef struct usart_init usart_init; void usart_structinit(usart_initstruct); printf("%d\n", usart_init->usart_baudrate); } from stm32f30x_usart.c body void usart_structinit(usart_inittypedef* usart_initstruct) { /* usart_initstruct members default value */ usart_initstruct->usart_baudrate = 9600; usart_initstruct->usart_wordlength = usart_wordlength_8b;...
Read more