Read data into array slices in R -


how can read large csv files slices of array in r? have 500 csv files 480 rows , 640 columns contain data of thermal imaging camera. make array dimensions 640 x 480 x 500 in fact every slice of array contains 1 picture taken thermal camera (640 x 480 pixels (values in °c)). untill put csv files in list of matrices loop , unlist them make array. problem computers ram full might better store data in array @ once, don't know how that. need data in array format anyway further calculations.

matrices<-list()  (i in 0:endval){         filenumber<-sprintf("%03d",(i+1))         matrices[[i+1]]<-read.csv2(file=paste(nameoffile,filenumber,".csv",sep=""),header=f)         }  array1<-array(unlist(matrices), dim = c(nrow(matrices[[1]]), ncol(matrices[[1]]), length(matrices))) 

thanks in advance!

you should preallocate array once, , fill in each z-slice loop through csv files. testing, appears 640x480x500 array of doubles takes 1.2gb, , 500 csv reads each assigned directly z-slice of array took 3.5min. thus, tenable, in terms of both ram impact , computation time:

x <- 640; y <- 480; n <- 500; write.csv(matrix(1:(x*y),x,y),'pic1.csv',row.names=f); ## testing system.time({ x <- array(na_real_,c(x,y,n)); }); ## preallocate ##    user  system elapsed ##   1.640   0.109   1.743 object.size(x); ## 1228800208 bytes e <- 2; xv <- -((1+e):(x-e)); yv <- -((1+e):(y-e)); nv <- -((1+e):(n-e)); x[xv,yv,nv]; ## , , 1 ##  ##      [,1] [,2] [,3] [,4] ## [1,]   na   na   na   na ## [2,]   na   na   na   na ## [3,]   na   na   na   na ## [4,]   na   na   na   na ##  ## , , 2 ##  ##      [,1] [,2] [,3] [,4] ## [1,]   na   na   na   na ## [2,]   na   na   na   na ## [3,]   na   na   na   na ## [4,]   na   na   na   na ##  ## , , 3 ##  ##      [,1] [,2] [,3] [,4] ## [1,]   na   na   na   na ## [2,]   na   na   na   na ## [3,]   na   na   na   na ## [4,]   na   na   na   na ##  ## , , 4 ##  ##      [,1] [,2] [,3] [,4] ## [1,]   na   na   na   na ## [2,]   na   na   na   na ## [3,]   na   na   na   na ## [4,]   na   na   na   na ##  system.time({ (i in 1:n) { filename <- 'pic1.csv'; x[,,i] <- as.matrix(read.csv(filename)); }; }); ##    user  system elapsed ## 207.000   0.969 208.492 object.size(x); ## 1228800208 bytes x[xv,yv,nv]; ## , , 1 ##  ##      [,1] [,2]   [,3]   [,4] ## [1,]    1  641 305921 306561 ## [2,]    2  642 305922 306562 ## [3,]  639 1279 306559 307199 ## [4,]  640 1280 306560 307200 ##  ## , , 2 ##  ##      [,1] [,2]   [,3]   [,4] ## [1,]    1  641 305921 306561 ## [2,]    2  642 305922 306562 ## [3,]  639 1279 306559 307199 ## [4,]  640 1280 306560 307200 ##  ## , , 3 ##  ##      [,1] [,2]   [,3]   [,4] ## [1,]    1  641 305921 306561 ## [2,]    2  642 305922 306562 ## [3,]  639 1279 306559 307199 ## [4,]  640 1280 306560 307200 ##  ## , , 4 ##  ##      [,1] [,2]   [,3]   [,4] ## [1,]    1  641 305921 306561 ## [2,]    2  642 305922 306562 ## [3,]  639 1279 306559 307199 ## [4,]  640 1280 306560 307200 

Comments

Popular posts from this blog

css - SVG using textPath a symbol not rendering in Firefox -

Java 8 + Maven Javadoc plugin: Error fetching URL -

order - Notification for user in user account opencart -