Read data into array slices in R -
how can read large csv files slices of array in r? have 500 csv files 480 rows , 640 columns contain data of thermal imaging camera. make array dimensions 640 x 480 x 500 in fact every slice of array contains 1 picture taken thermal camera (640 x 480 pixels (values in °c)). untill put csv files in list of matrices loop , unlist them make array. problem computers ram full might better store data in array @ once, don't know how that. need data in array format anyway further calculations.
matrices<-list() (i in 0:endval){ filenumber<-sprintf("%03d",(i+1)) matrices[[i+1]]<-read.csv2(file=paste(nameoffile,filenumber,".csv",sep=""),header=f) } array1<-array(unlist(matrices), dim = c(nrow(matrices[[1]]), ncol(matrices[[1]]), length(matrices)))
thanks in advance!
you should preallocate array once, , fill in each z-slice loop through csv files. testing, appears 640x480x500 array of doubles takes 1.2gb, , 500 csv reads each assigned directly z-slice of array took 3.5min. thus, tenable, in terms of both ram impact , computation time:
x <- 640; y <- 480; n <- 500; write.csv(matrix(1:(x*y),x,y),'pic1.csv',row.names=f); ## testing system.time({ x <- array(na_real_,c(x,y,n)); }); ## preallocate ## user system elapsed ## 1.640 0.109 1.743 object.size(x); ## 1228800208 bytes e <- 2; xv <- -((1+e):(x-e)); yv <- -((1+e):(y-e)); nv <- -((1+e):(n-e)); x[xv,yv,nv]; ## , , 1 ## ## [,1] [,2] [,3] [,4] ## [1,] na na na na ## [2,] na na na na ## [3,] na na na na ## [4,] na na na na ## ## , , 2 ## ## [,1] [,2] [,3] [,4] ## [1,] na na na na ## [2,] na na na na ## [3,] na na na na ## [4,] na na na na ## ## , , 3 ## ## [,1] [,2] [,3] [,4] ## [1,] na na na na ## [2,] na na na na ## [3,] na na na na ## [4,] na na na na ## ## , , 4 ## ## [,1] [,2] [,3] [,4] ## [1,] na na na na ## [2,] na na na na ## [3,] na na na na ## [4,] na na na na ## system.time({ (i in 1:n) { filename <- 'pic1.csv'; x[,,i] <- as.matrix(read.csv(filename)); }; }); ## user system elapsed ## 207.000 0.969 208.492 object.size(x); ## 1228800208 bytes x[xv,yv,nv]; ## , , 1 ## ## [,1] [,2] [,3] [,4] ## [1,] 1 641 305921 306561 ## [2,] 2 642 305922 306562 ## [3,] 639 1279 306559 307199 ## [4,] 640 1280 306560 307200 ## ## , , 2 ## ## [,1] [,2] [,3] [,4] ## [1,] 1 641 305921 306561 ## [2,] 2 642 305922 306562 ## [3,] 639 1279 306559 307199 ## [4,] 640 1280 306560 307200 ## ## , , 3 ## ## [,1] [,2] [,3] [,4] ## [1,] 1 641 305921 306561 ## [2,] 2 642 305922 306562 ## [3,] 639 1279 306559 307199 ## [4,] 640 1280 306560 307200 ## ## , , 4 ## ## [,1] [,2] [,3] [,4] ## [1,] 1 641 305921 306561 ## [2,] 2 642 305922 306562 ## [3,] 639 1279 306559 307199 ## [4,] 640 1280 306560 307200
Comments
Post a Comment