Subsetting a large vector uses unnecessarily large amounts of memory

I have a large raw vector, e.g.:

x <- rep(as.raw(1:10), 4e8) # this vector is about 4 GB

I just want to remove the first element, but no matter what I do it uses a huge amount of memory.

> x <- tail(x, length(x)-1)
Error: cannot allocate vector of size 29.8 Gb
> x <- x[-1L]
Error: cannot allocate vector of size 29.8 Gb
> x <- x[seq(2, length(x)-1)]
Error: cannot allocate vector of size 29.8 Gb

What's going on? Do I really have to rely on C to do such a simple operation? (I know it's simple to do with Rcpp but that's not the point).

SessionInfo:

R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.6 LTS

Matrix products: default
BLAS:   /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] dplyr_0.8.3

loaded via a namespace (and not attached):
 [1] tidyselect_0.2.5 compiler_3.6.1   magrittr_1.5     assertthat_0.2.1
 [5] R6_2.4.0         pillar_1.4.2     glue_1.3.1       tibble_2.1.3
 [9] crayon_1.3.4     Rcpp_1.0.2       pkgconfig_2.0.2  rlang_0.4.0
[13] purrr_0.3.2

Rcpp solution as @jangoreki asked for:

#include <Rcpp.h>
using namespace Rcpp;

// solution for the original question
// [[Rcpp::export]]
IntegerVector popBeginningOfVector(IntegerVector x, int npop) {
  return IntegerVector(x.begin() + npop, x.end());
}

// generic negative indexing
// [[Rcpp::export]]
IntegerVector efficientNegativeIndexing(IntegerVector x, IntegerVector neg_idx) {
  std::sort(neg_idx.begin(), neg_idx.end());
  size_t ni_size = neg_idx.size();
  size_t xsize = x.size();
  int * xptr = INTEGER(x);
  int * niptr = INTEGER(neg_idx);
  size_t xtposition = 0;
  IntegerVector xt(xsize - ni_size); // allocate new vector of the correct size
  int * xtptr = INTEGER(xt);
  int range_begin, range_end;
  for(size_t i=0; i < ni_size; ++i) {
    if(i == 0) {
      range_begin = 0;
    } else {
      range_begin = neg_idx[i-1];
    }
    range_end = neg_idx[i] - 1;
    // std::cout << range_begin << " " << range_end << std::endl;
    std::copy(xptr+range_begin, xptr+range_end, xtptr+xtposition);
    xtposition += range_end - range_begin;
  }
  std::copy(xptr+range_end+1, xptr + xsize, xtptr+xtposition);
  return xt;
}