Python disk caching with index

Question

My task is that of caching. If the folder exists then load the cache from disk, if it doesnt create new cache. This is trivial, what bothers me is if I want to create third mode of noop (don't do anything relating to cache).

Currently I have this:

class HeavyCalculationClass:
    def __init__(self, cache_dir=None)
        self.use_cache = cache_dir is not None
        if self.use_cache:
            if os.path.exists(cache_dir):
                if len(os.listdir(cache_dir) != 0:
                    self.cache_type = "write"
                else:
                    self.cache_type = "read"
            else:
                os.mkdir(cache_dir)
                self.cache_type = "write"

    def __getitem__(self, idx):
        if self.use_cache and self.cache_type=="read":
            data = numpy.load(os.path.join(cache_dir, str(idx)+".npy"))
        else:
            data = very_heavy_calculation()

        if self.use_cache and self.cache_type == "write":
            numpy.save(os.path.join(cache_dir, str(idx)+".npy"), data)
        return data

Is there any way to write this in a more elegant form?

Could you provide more code, currently the way I think would be an improvement could just be a hindrance. — Peilonrayz
– Peilonrayz ♦, Commented Mar 16, 2020 at 21:58
@Peilonrayz I've tried but adding any other code would just complicate too much, and won't focus on the core issue. Could you post your thoughts as an answer? Pretty much it needs to be iterable (hence getitem) , and if the file already exists on disk, just skip heavy calculation and read from disk. — Dusan J.
– Dusan J., Commented Mar 17, 2020 at 9:13
Welcome to Code Review! The current question title, which states your concerns about the code, is too general to be useful here. Please edit to the site standard, which is for the title to simply state the task accomplished by the code. Please see How to get the best value out of Code Review: Asking Questions for guidance on writing good question titles. — Toby Speight
– Toby Speight, Commented Mar 17, 2020 at 9:52
@TobySpeight I've edited it but I'm not sure how to name it precisely. Trying to optimize for easy recognition for someone who's done similar stuff but like this it is too specific and the same problem can be experienced with other stuff (not necesseraly caching). — Dusan J.
– Dusan J., Commented Mar 17, 2020 at 10:19

RootTwo · Accepted Answer · 2020-03-16 23:29:29Z

I don't know what you mean by "more elegant", but I would recommend using pathlib. This class provides for caching to be enabled/disabled for all files. If the class is not initialized with a cache_dir, then caching is disabled. If a cache_dir is provided, then __getitem__() loads data from cached files, if they exist. Otherwise the data is calculated and cached.

class HeavyCalculationClass:
    def __init__(self, cache_dir=None):
        # sets the path to the cache directory if it was specified
        # and creates the cache directory if needed
        if cache_dir:
            self.cache = pathlib.Path(cache_dir)

            if not self.cache.is_dir():
                self.cache.mkdir()

        else:
            # empty cache path means ignore the cache
            self.cache = None

    def __getitem__(self, idx):
        # if caching is 'on' and this file is cached,
        # then load it
        if self.cache:
            cache_file = self.cache / f"{idx}.npy"

            if cache_file.is_file():
                data = numpy.load(cache_file))
                return data

        # otherwise calculate the data
        data = very_heavy_calculation()

        # cache the data if a cache path is set
        if self.cache:
            numpy.save(self.cache / f"{idx}.npy"), data)

        return data

more elegant as in less lines of code less variables, simpler to read — Dusan J.
– Dusan J., Commented Mar 17, 2020 at 9:11

Stack Exchange Network

Python disk caching with index

1 Answer 1

You must log in to answer this question.

Hot Network Questions

Python disk caching with index

1 Answer 1

You must log in to answer this question.

Related

Hot Network Questions