Pfthb

I am making various ggplots on a very large dataset (much larger than the examples). I created a binning function on both x- and y-axes to enable plotting of such large dataset.

In the following example, the memory.size() is recorded at the start. Then the large dataset is simulated as dt. dt's x2 is plotted against x1 with binning. Plotting is repeated with different subsets of dt. The size of the ploted object is checked by object.size() and stored. After the plotting objects have been created, rm(dt) is executed, followed by a double gc(). At this point, memory.size() is recorded again. At the end, the memory.size() at the end is compared to that at the beginning and printed.

In view of the small size of the plotted object, it is expected that the memory.size() at the end should be similar to that at the beginning. But no. memory.size() does not go down anymore until I restart a new R session.

REPRODUCIBLE EXAMPLE

library(data.table)
library(ggplot2)
library(magrittr)

# The binning function
# x = column name for x-axis (character)
# y = column name for y-axis (character)
# xNItv = Number of bin for x-axis
# yNItv = Number of bin for y-axis
# Value: A binned data.table
tab_by_bin_idxy <- function(dt, x, y, xNItv, yNItv) 
 #Binning
 xBreaks = dt[, seq(min(get(x), na.rm = T), max(get(x), na.rm = T), length.out = xNItv + 1)]
 yBreaks = dt[, seq(min(get(y), na.rm = T), max(get(y), na.rm = T), length.out = yNItv + 1)]
 xbinCode = dt[, .bincode(get(x), breaks = xBreaks, include.lowest = T)]
 xbinMid = sapply(seq(xNItv), function(i) return(mean(xBreaks[c(i, i+1)])))[xbinCode]
 ybinCode = dt[, .bincode(get(y), breaks = yBreaks, include.lowest = T)]
 ybinMid = sapply(seq(yNItv), function(i) return(mean(yBreaks[c(i, i+1)])))[ybinCode]
 #Creating table
 tab_match = CJ(xbinCode = seq(xNItv), ybinCode = seq(yNItv))
 tab_plot = data.table(xbinCode, xbinMid, ybinCode, ybinMid)[
 tab_match, .(xbinMid = xbinMid[1], ybinMid = ybinMid[1], N = .N), keyby = .EACHI, on = c("xbinCode", "ybinCode")
 ]
 #Returning table
 return(tab_plot)


before.mem.size <- memory.size()

# Simulation of dataset
nrow <- 6e5
ncol <- 60
dt <- do.call(data.table, lapply(seq(ncol), function(i) return(runif(nrow))) %>% set_names(paste0("x", seq(ncol))))

# Graph plotting
dummyEnv <- new.env()
with(dummyEnv, 
 fcn <- function(tab) 
 binned.dt <- tab_by_bin_idxy(dt = tab, x = "x1", y = "x2", xNItv = 50, yNItv = 50)
 plot <- ggplot(binned.dt, aes(x = xbinMid, y = ybinMid)) + geom_point(aes(size = N))
 return(plot)
 
 lst_plots <- list(
 plot1 = fcn(dt),
 plot2 = fcn(dt[x1 <= 0.7]),
 plot3 = fcn(dt[x5 <= 0.3])
 )
 assign("size.of.plots", object.size(lst_plots), envir = .GlobalEnv)
)
rm(dummyEnv)

# After use, remove and clean up of dataset
rm(dt)
gc();gc()
after.mem.size <- memory.size()

# Memory reports
print(paste0("before.mem.size = ", before.mem.size))
print(paste0("after.mem.size = ", after.mem.size))
print(paste0("plot.objs.size = ", size.of.plots / 1000000))

I have tried the following modifications to the code:

• Inside fcn, removing ggplot and returning a NULL instead of a plot object: The memory leakage is totally gone. But this is not a solution. I need the plot.


• The less plots requested / less columns / less rows passed to fcn, the less is the memory leakage.


• Memory leakage also exists if I do not make any subset and make only one plot object (In the examples, I plotted 3).


• After the process, even after I call rm(list = ls()), the memory is still non-recoverable.

I wish to know why this happens and how to get rid of it without compromising my need to do binned plots and subset dt to make different plots.

Thanks for attention!