Can python load data from a file "on demand" when a variable is accessed?

Question

I'm using python 3.6 and I have long lists of id numbers that I'd like to cache in files, but only load them to memory when needed. Ideally, I'd like them to appear as list-type variables that load from file when accessed, as follows.

""" contents of a library file, canned_lists.py """
list_of_ids = []
def event_driven_on_access_of_list_of_ids():
    # don't allow access to the empty list yet
    global list_of_ids
    with open("the_id_file.csv", "r") as f:
        list_of_ids = f.readlines()
    # now the list is ready, it can be accessed

""" contents of a calling file, script.py """
import canned_lists
for id in canned_lists.list_of_ids:  # at this point, list_of_ids should populate from a file
    print("do something with the ids")

Alternative option A would be to use a function rather than a variable. This works and is really not bad, but aesthetically, I'd like to import a list and use a list rather than a function.

""" contents of a library file, canned_lists.py """
def get_list_of_ids():
    with open("the_file.csv", "r") as f:
        return f.readlines()

Alternative option B would be to just store the data in code. The lists can be nearly 20,000 ids, so option B is clumsy and hard to manage. It also causes PyCharm to tell me my file sizes exceed configured limits of Code Insight features. I could probably increase the limits, but it seems more reasonable to just move data out of code.

""" contents of a library file, canned_lists.py """
list_of_ids = [123, 124, 125, 126, ]

Does python have a way to support changing a variable 'on access' to support the top option? Or does anyone have a better idea? I'll probably implement alternative A as it's perfectly functional, but I'm eager to learn from more advanced pythonistas.

nneonneo · Accepted Answer · 2020-01-06 02:57:45Z

2

You could use a property in the context of a class, which causes a function to be called upon variable access:

class X:
    @property
    def thing(self):
        return 42

print(X().thing) # prints 42, note no function call syntax

Two things to watch out for: (1) every property access will invoke the function, so you might want to cache the result in the class for performance; (2) property accesses are generally expected to be fast, that is, users of your code will not expect a property access to perform a slow file operation (that would violate the Principle of Least Surprise).

answered Jan 6, 2020 at 2:57

nneonneo

181k37 gold badges331 silver badges412 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

mightypile Over a year ago

Principle of Least Surprise is a good point that probably suggests the use of a function (alternative A) rather than 'pretending' it's just a list in memory. And a class is probably a better organizational structure than my globals. Very useful, thanks.

Collectives™ on Stack Overflow

Can python load data from a file "on demand" when a variable is accessed?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related