Generate random missing values in a dataframe using R

Question

I had a simple example of a dataframe as follows:

a    b   c

10  50  100

45  36  27

15  80  90

The output expected to be as shown below.

a   b   c

10  NA  100

NA  NA  NA

15  80  27

The programming that I tried as follows:

insert_nas <- function(x) {

  len <- length(x)

  n <- sample(1:floor((0.01*(dim(x)[1]))), 1)

  i <- sample(1:len, n)

  x[i] <- NA 

  x

}


> sapply(incomplete.data,insert_nas)

Error in 1:floor((0.01 * (dim(x)[1]))) : argument of length 0

However, there was an error showed up.
How can I generate random missing values with 1% of missing in the dataframe?

See also this, with the graphical represenation: stackoverflow.com/a/28368161/3871924 — agenis - code en bois
– agenis - code en bois, Commented Feb 19, 2016 at 13:50

tegancp · Accepted Answer · 2015-06-18 04:11:45Z

3

Where your error comes from:

The sapply call is trying to apply the function insert_nas to each element of incomplete.data (in this context, the elements of a dataframe are its columns). The function dim applied to an atomic vector yields NULL; multiplying by a constant gives a numeric vector of length 0; applying floor doesn't change this; and finally trying to generate a sequence bounded by an empty vector gives an error.

How to eliminate the error:

Presumably by dim(x)[1] you were intending to get the number of rows in the dataframe (which is what you get when x is the dataframe rather than one of its columns). Try replacing it with length(x).

For arbitrarily distributed selection of NAs:

To change some proportion p of values to NA, distributing without regard to column location, it seems most straightforward to just use a random sample of the appropriate size (p*df-size) over the whole dataframe to choose the elements to set to NA:

sel <- sample( nrow(df)*ncol(df), size = p*nrow(df)*ncol(df) )
for(t in 1:length(sel)){
    is.na(df[[sel[t]%/%nrow(df) +1]]) <- sel[t]%%nrow(df) + 1
}

edited Jun 18, 2015 at 4:11

answered Jun 18, 2015 at 2:06

tegancp

1,2026 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Nur Aifiah Ibrahim Over a year ago

Thanks for the help tegancp I really appreciate it. However, the random missing values were generated for 1% of missing from each of the columns in the dataframe but instead I still can not find ways to have a total of 1% missing from all of the columns in the data frame.

tegancp Over a year ago

So are you looking for a random 1% selection of NAs, distributed randomly without regard for the columns (so different columns would have different proportions of NAs)?

Nur Aifiah Ibrahim Over a year ago

Sorry for the late reply tegancp. Yes, tegancp how can I generate the random missing values by doing so?

Nur Aifiah Ibrahim Over a year ago

Dear tegancp, I had already tried with the updated programming but there is an error message Error in sample.int(length(x), size, replace, prob) : invalid 'size' as I include the first line of the programming as: insert_nas<-function(df){ sel<-sample(nrow(df)*ncol(df),size=p*nrow(df)*ncol(df)) for(t in 1:length(sel)){ is.na(df[[sel[t]%/%nrow(df) +1]])<-sel[t]%%nrow(df)+1} } How can I solve the problem tegancp?

tegancp Over a year ago

Sorry - playing with a small dataframe, so I used a larger value for the proportion of NAs. You'll have to set p= the value you want (0.01) before that line (or just replace p with your value).

|

Tiziano · Accepted Answer · 2015-10-10 09:46:13Z

I've used the prodNA from the missForest package.

My function is the follow

fn.df.add.NA <- function(df, var.name, prop.of.missing) {
  df.buf <- subset(df, select=c(var.name))                      # Select variable
  require(missForest, quietly = T)
  df.buf <- prodNA(x = df.buf, prop.of.missing)                 # chage original value to NA in casual sequence
  detach("package:missForest", unload=TRUE)

  df.col.order <- colnames(x = df)                              # save the column order       
  df <- subset(df, select=-c(which(colnames(df)==var.name)))    # drop the variable with no NAs    
  df <- cbind(df, df.buf)                                       # add the column with NA          
  df <- subset(df, select=df.col.order)                         # restore the original order sequence 

  return(df)  
}

It allow to change to NAs a random number of observations according to a given proportion.

Because prodNA function apply the NAs to all the data.frame columns I've used a "buffer" data structure in order to return the same data structure of the input data.frame. Perhaps some reader may suggest a more elegant way.

In every way you can do this test

set.seed(1)
df <- data.frame(a = as.numeric(runif(n = 100, min = 1, max = 100)),
                 b = as.numeric(runif(n = 100, min = 201, max = 300)),
                 c = as.numeric(runif(n = 100, min = 301, max = 400)))

 summary(df)
       a                b               c        
 Min.   : 2.326   Min.   :202.3   Min.   :303.8  
 1st Qu.:32.985   1st Qu.:229.2   1st Qu.:319.8  
 Median :49.293   Median :252.3   Median :338.4  
 Mean   :52.267   Mean   :252.2   Mean   :344.1  
 3rd Qu.:76.952   3rd Qu.:273.3   3rd Qu.:364.0  
 Max.   :99.199   Max.   :299.3   Max.   :398.2  

df <- fn.df.add.NA(df = df, var.name = "a", prop.of.missing = .1)
df <- fn.df.add.NA(df = df, var.name = "b", prop.of.missing = .2)
df <- fn.df.add.NA(df = df, var.name = "c", prop.of.missing = .3)

summary(df)
       a                b               c        
 Min.   : 2.326   Min.   :202.3   Min.   :303.8  
 1st Qu.:30.628   1st Qu.:229.2   1st Qu.:319.2  
 Median :48.202   Median :252.3   Median :342.2  
 Mean   :50.247   Mean   :252.5   Mean   :345.4  
 3rd Qu.:71.504   3rd Qu.:273.3   3rd Qu.:369.3  
 Max.   :99.199   Max.   :299.3   Max.   :396.2  
 NA's   :10       NA's   :20      NA's   :30

Samuel Saari · Accepted Answer · 2021-05-27 06:33:33Z

library(reprex)
set.seed(42)

mi_d <- mtcars

nr_of_NAs <- 30
for (i in 1:nr_of_NAs) {
  mi_d[sample(nrow(mi_d),1),sample(ncol(mi_d),1)] <- NA
}
mi_d

#>                      mpg cyl  disp  hp drat    wt  qsec vs am gear carb
#> Mazda RX4           21.0   6 160.0 110 3.90 2.620 16.46  0  1   NA    4
#> Mazda RX4 Wag       21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4
#> Datsun 710          22.8  NA 108.0  93 3.85 2.320 18.61 NA  1    4    1
#> Hornet 4 Drive      21.4   6 258.0 110   NA 3.215 19.44 NA  0   NA    1
#> Hornet Sportabout     NA   8 360.0  NA 3.15 3.440 17.02  0  0    3    2
#> Valiant             18.1   6 225.0 105 2.76    NA 20.22  1  0    3    1
#> Duster 360          14.3   8 360.0 245 3.21 3.570 15.84 NA  0    3    4
#> Merc 240D           24.4   4 146.7  62 3.69 3.190 20.00  1  0    4    2
#> Merc 230            22.8   4    NA  95 3.92 3.150 22.90  1  0    4    2
#> Merc 280            19.2   6 167.6 123 3.92 3.440 18.30  1  0    4    4
#> Merc 280C           17.8   6 167.6 123 3.92 3.440 18.90  1  0    4    4
#> Merc 450SE          16.4   8 275.8 180 3.07 4.070 17.40  0  0    3    3
#> Merc 450SL          17.3  NA 275.8 180 3.07 3.730 17.60  0  0    3    3
#> Merc 450SLC         15.2   8 275.8 180   NA 3.780 18.00  0  0    3    3
#> Cadillac Fleetwood  10.4  NA 472.0 205 2.93 5.250 17.98  0  0    3   NA
#> Lincoln Continental 10.4   8 460.0 215 3.00 5.424 17.82  0  0    3    4
#> Chrysler Imperial   14.7   8 440.0 230 3.23 5.345 17.42  0  0    3    4
#> Fiat 128            32.4   4  78.7  66 4.08 2.200 19.47  1  1   NA    1
#> Honda Civic         30.4   4  75.7  52 4.93 1.615 18.52  1  1    4    2
#> Toyota Corolla      33.9   4  71.1  65   NA 1.835 19.90  1  1    4    1
#> Toyota Corona         NA   4 120.1  97 3.70 2.465 20.01  1  0    3    1
#> Dodge Challenger    15.5   8 318.0  NA 2.76 3.520 16.87  0  0    3    2
#> AMC Javelin         15.2   8 304.0 150 3.15 3.435 17.30 NA  0    3    2
#> Camaro Z28          13.3  NA 350.0 245 3.73 3.840 15.41  0  0   NA    4
#> Pontiac Firebird      NA   8 400.0  NA 3.08 3.845 17.05  0  0    3    2
#> Fiat X1-9           27.3  NA  79.0  NA 4.08    NA 18.90  1  1    4    1
#> Porsche 914-2       26.0   4 120.3  91 4.43 2.140 16.70  0 NA    5    2
#> Lotus Europa        30.4  NA  95.1 113 3.77 1.513 16.90  1  1    5    2
#> Ford Pantera L      15.8   8 351.0 264 4.22 3.170 14.50  0  1    5    4
#> Ferrari Dino        19.7   6 145.0 175 3.62 2.770 15.50  0  1    5    6
#> Maserati Bora       15.0   8 301.0 335 3.54 3.570 14.60  0  1    5    8
#> Volvo 142E            NA   4 121.0 109 4.11 2.780 18.60  1  1    4    2
Created on 2021-05-27 by the reprex package (v2.0.0)

Hoffmayo · Accepted Answer · 2020-11-18 20:19:05Z

0

Hope it is still relevant. Created a function that receives a dataframe and desired NAs ratio, returns a dataframe that has randomly distributed NAs in it with desired specified ratio:

insertNA <- function(df,NAratio) {
  sel <- sample( nrow(df)*ncol(df), size = NAratio*nrow(df)*ncol(df) )
  for (i in c(1:length(sel))) {
    a <- as.integer((sel[i]-1)/ncol(df)+1)
    b <- sel[i] - (a-1)*ncol(df)
    df[a,b] <- NA
  }
  return(df)
}

answered Nov 18, 2020 at 20:19

Hoffmayo

1

Collectives™ on Stack Overflow

Generate random missing values in a dataframe using R

4 Answers 4

10 Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

10 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related