Performance of list.extend slice vs islice

Question

It seems that even when islice would theoretically be better, in practice, it is slower than just using slice. So I am a bit puzzled by the difference in performance between the usage of slice and islice here:

from time import perf_counter
from itertools import islice
from random import choices
from string import ascii_letters
import sys

test_str = "".join(choices(ascii_letters, k=10 ** 7))

arr1 = []
t0 = perf_counter()
for _ in range(10):
    arr1.extend(test_str[slice(1, sys.maxsize)])
t1 = perf_counter()
print('%5.1f ms ' % ((t1 - t0) * 1e3))

arr2 = []
t0 = perf_counter()
for _ in range(10):
    arr2.extend(islice(test_str, 1, sys.maxsize))
t1 = perf_counter()
print('%5.1f ms ' % ((t1 - t0) * 1e3))

552.6 ms
786.0 ms

To justify why I believe the difference to be unexpected, here is my mental model for how the execution goes for both cases:

Strict Slicing:

Allocate tmp, where tmp is the immutable string from 1th index to the end of the original string.
Acquire an iterator over tmp.
Create string objects for each individual character.
Pass those string objects to be appended to the list.
Garbage collect tmp.

Lazy Slicing:

Acquire an iterator over test_str, discarding the first value.
Create string objects for each individual character.
Pass those string objects to be appended to the list.

I don't see how allocating the whole string object at once just to be garbage collected later leads to still better performance, especially considering the fact that both slicing methods are implemented in C.

I think I saw this question but with smaller results - first had 1.6 ms. And with k=10**6 instead of k=10**7 — furas
– furas, Commented May 3 at 1:25
I even found link to this question in my notes - but it seems it is deleted stackoverflow.com/questions/79604028/… — furas
– furas, Commented May 3 at 1:27
@furas Oh that was the question I posted earlier today. I deleted it because it was essentially asking why islice(some_list, big_number, bigger_number) was so much slower (1.6ms vs ~500ms) than some_list[big_number:bigger_number]. However, I realized the reason for it was quite trivial, as islice has to first "seek" to big_number by iterating & discarding all elements before big_number. After taking that into account, the difference became much less drastic (as we can see here), but islice was still worse, leading to a separate question. — Mathias Sven
– Mathias Sven, Commented May 3 at 2:09

Raymond Hettinger · Accepted Answer · 2025-05-02 23:51:35Z

2

I don't see how allocating the whole string object at once just to be garbage collected later leads to still better performance, specially considering the fact that both slicing methods are implemented in C.

The likely cause is that the islice object does not have a length hint while the size of the string slice is known. That lets list.extend preallocate to the correct size instead of having to execute periodic resize operations.

edited May 2 at 23:51

answered May 2 at 23:40

Raymond Hettinger

229k67 gold badges405 silver badges504 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Stefan Pochmann · Accepted Answer · 2025-05-03 21:41:25Z

The additional iteration layer of islice is far more costly than the string slices. Allocation optimization by length hint is insignificant, at least on the three systems where I tried this.

With the string slices, the extend method iterates directly over the string (slice).

With islice, it instead iterates over the islice iterator, which in turn iterates over the string. Every single character gets requested and passed through this additional iterator.

Benchmark results:

                       System 1     System 2     System 3
                      Py 3.11.10   Py 3.13.3    Py 3.13.0

extend_slice          552 ± 2 ms   413 ± 2 ms   823 ±  6 ms
extend_islice         849 ± 5 ms   648 ± 1 ms  1225 ± 28 ms
just_slice              4 ± 0 ms     7 ± 0 ms    20 ±  1 ms
iterate_slice         179 ± 1 ms   231 ± 1 ms   304 ±  8 ms
iterate_islice        523 ± 1 ms   366 ± 2 ms   680 ±  3 ms
extend_slice_len      553 ± 3 ms   414 ± 1 ms   840 ± 12 ms
extend_slice_nolen    558 ± 2 ms   430 ± 1 ms   842 ± 11 ms
extend_islice_nolen   852 ± 1 ms   649 ± 1 ms  1227 ± 22 ms
extend_islice_len     848 ± 2 ms   636 ± 1 ms  1195 ± 39 ms

extend_slice and extend_islice are your tests, we see time differences similar to what you saw.

just_slice just creates the string slices but does nothing with them. Wee see they're very cheap in comparison.

iterate_slice and iterate_islice don't extend a list, instead they just iterate the slices/islices (in a fast way). We see that the slices get iterated much faster than the islices.

The last four rows show times for your tests again, but with the slices/islices wrapped in little objects that provide or don't provide length hints. We only see little/unclear effects of having / not having length hints.

Benchmark code (system 3 has a low time limit, so instead of best 5 of 100 rounds I used best 3 of 7 there):

from time import perf_counter
from itertools import islice
from random import choices
from string import ascii_letters
from collections import deque
from statistics import mean, stdev
import random
import sys


def extend_slice():
    arr = []
    for _ in range(10):
        arr.extend(test_str[slice(1, sys.maxsize)])


def extend_islice():
    arr = []
    for _ in range(10):
        arr.extend(islice(test_str, 1, sys.maxsize))


def just_slice():
    for _ in range(10):
        test_str[slice(1, sys.maxsize)]


iterate = deque(maxlen=0).extend


def iterate_slice():
    for _ in range(10):
        iterate(test_str[slice(1, sys.maxsize)])


def iterate_islice():
    for _ in range(10):
        iterate(islice(test_str, 1, sys.maxsize))


class WithLen:
    def __init__(self, iterable):
        self.iterable = iterable
    def __iter__(self):
        return iter(self.iterable)
    def __len__(self):
        return len(test_str) - 1

class NoLen:
    def __init__(self, iterable):
        self.iterable = iterable
    def __iter__(self):
        return iter(self.iterable)


def extend_slice_len():
    arr = []
    for _ in range(10):
        arr.extend(WithLen(test_str[slice(1, sys.maxsize)]))


def extend_slice_nolen():
    arr = []
    for _ in range(10):
        arr.extend(NoLen(test_str[slice(1, sys.maxsize)]))


def extend_islice_len():
    arr = []
    for _ in range(10):
        arr.extend(WithLen(islice(test_str, 1, sys.maxsize)))


def extend_islice_nolen():
    arr = []
    for _ in range(10):
        arr.extend(NoLen(islice(test_str, 1, sys.maxsize)))


funcs = [
    extend_slice,
    extend_islice,
    just_slice,
    iterate_slice,
    iterate_islice,
    extend_slice_len,
    extend_slice_nolen,
    extend_islice_nolen,
    extend_islice_len,
]


test_str = "".join(choices(ascii_letters, k=10 ** 7))

def print(*a, p=print, f=open('out.txt', 'w')):
    p(*a)
    p(*a, file=f, flush=True)

times = {f: [] for f in funcs}
def stats(f):
    ts = [t * 1e3 for t in sorted(times[f])[:5]]
    return f'{mean(ts):4.0f} ± {stdev(ts):2.0f} ms '
    return f'{mean(ts):5.1f} ± {stdev(ts):3.1f} ms '
for _ in range(100):
    print(_)
    for f in random.sample(funcs, len(funcs)):
        t0 = perf_counter()
        f()
        t1 = perf_counter()
        times[f].append(t1 - t0)
for f in funcs:
    print(stats(f), f.__name__)

print('\nPython:', sys.version)

Collectives™ on Stack Overflow

Performance of list.extend slice vs islice

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related