Writing multiple variables to a file using a function

Question

I am writing a function that exports variables as a dictionary to an external file. The problem comes when calling that function from another script. I think it has something to do with the globals() parameter.

import sys
import os

mydict = {}  #'initialising" the an empty dictionary to be used locally in the function below

def writeToValues(name):
    fileName = os.path.splitext(os.path.basename(sys.argv[0]))[0]
    valuePrint=open("values.py","a")
    def namestr(obj,namespace):
        return[name for name in namespace if namespace[name] is obj]
    b = namestr(name, globals())
    c = "".join(str(x) for x in b)
    mydict[(c)] = name
    valuePrint.write(fileName)
    valuePrint.write("=")
    valuePrint.write(str(mydict))
    valuePrint.write("\n")
    valuePrint.close()
    return mydict

a = 2
b = 3

writeToValues(a)
writeToValues(b)

I get the following result:

Main Junkfile={'a': 2, 'b': 3}

note the word Main Junkfile is the name of the script I ran as that is what the function first does, to get the name of the file and use that to name the dictionary.

Now help me as I cannot generate the same if I import the function from another script. Another problem is that running the script twice generates the values in steps.

Main Junkfile={'a': 2}
Main Junkfile={'b': 3, 'a': 2}

I cannot change the file open mode from append to write since I want to store values from other scripts, too.

if that another script is python too, you are missing typical: if name == "main": construct — Drako
– Drako, Commented May 24, 2017 at 11:14

Drako · Accepted Answer · 2017-05-24 11:27:58Z

1

this is not perfect but might help as an example:

import sys
import os

mydict = {}
def namestr(obj,namespace):
    return[name for name in namespace if namespace[name] is obj]

def writeto(name):
    fout = 'values.py'
    filename = os.path.splitext(os.path.basename(sys.argv[0]))[0]
    with open (fout, 'a') as f:
        b = namestr(name, globals())
        c = "".join(str(x) for x in b)
        mydict[(c)] = name
        data = filename + '=' + str(mydict) + '\n'
        f.write(data)
    return mydict

a = 2
b = 3

if __name__ == '__main__':
    writeto(a)
    writeto(b)

answered May 24, 2017 at 11:27

Drako

76812 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Drako Over a year ago

on imports filename creation may work not as you expect then use os.path to change current directory to what ever you want

zwer · Accepted Answer · 2017-05-24 13:28:41Z

First of all, to get the current executing script name, or rather the module that called your function you'll have to pick it up from the stack trace. Same goes for globals() - it will execute in the same context of writeToValues() function so it won't be picking up globals() from the 'caller'. To remedy that you can use the inspect module:

import inspect
import os

def writeToValues(name):
    caller = inspect.getmodule(inspect.stack()[1][0])
    caller_globals = caller.__dict__  # use this instead of globals()
    fileName = os.path.splitext(os.path.basename(caller.__file__))[0]
    # etc.

This will ensure that you get the name of the module that imported your script and is calling writeToValues() within it.

Keep in mind that this is a very bad idea if you intend to write usable Python files - if your script name has spaces (like in your example) it will write a variable name with spaces, which will further result in syntax error if you try to load the resulting file into a Python interpreter.

Second, why in the name of all things fluffy are you trying to do a reverse lookup to find a variable name? You are aware that:

a = 2
b = 2
ab = 5

writeToValues(b)

will write {"ab": 2}, and not {"b": 2} making it both incorrect in intent (saves the wrong var) as well as in state representation (saves a wrong value), right? You should pass a variable name you want to store/update instead to ensure you're picking up the right property.

The update part is more problematic - you need to update your file, not just merely append to it. That means that you need to find the line of your current script, remove it and then write a new dict with the same name in its place. If you don't expect your file to grow to huge proportions (i.e. you're comfortable having it partially in the working memory), you could do that with:

import os
import inspect

def writeToValues(name):
    caller = inspect.getmodule(inspect.stack()[1][0])
    caller_globals = caller.__dict__  # use this instead of globals()
    caller_name = os.path.splitext(os.path.basename(caller.__file__))[0]
    # keep 'mydict' list in the caller space so multiple callers can use this
    target_dict = caller_globals['mydict'] = caller_globals.get('mydict', {})
    if name not in caller_globals:  # the updated value no longer exists, remove it
        target_dict.pop(name, None)
    else:
        target_dict[name] = caller_globals[name]
    # update the 'values.py':
    # optionaly check if you should update - if values didn't change no need for slow I/O
    with open("values.py", "a+") as f:
        last_pos = 0  # keep the last non-update position
        while True:
            line = f.readline()  # we need to use readline() for tell() accuracy
            if not line or line.startswith(caller_name):  # break at the matching line or EOF
                break
            last_pos = f.tell()  # new non-update position
        append_data = f.readlines()  # store in memory the rest of the file content, if any
        f.seek(last_pos)  # rewind to the last non-update position
        f.truncate()  # truncate the rest of the file
        f.write("".join((caller_name, " = ", str(target_dict), "\n")))  # write updated dict
        if append_data:  # write back the rest of the file, if truncated
            f.writelines(append_data)
    return target_dict

Otherwise use a temp file to write everything as you read it, except for the line matching your current script, append the new value for the current script, delete the original and rename the temp file to values.py.

So now if you store the above in, say, value_writter.py and use it in your script my_script.py as:

import value_writter

a = 2
b = 3

value_writter.write_to_values("a")
value_writter.write_to_values("b")

a = 5

value_writter.write_to_values("a")

# values.py contains: my_script = {"a": 5, "b": 3}

Same should go for any script you import it to. Now, having multiple scripts edit the same file without a locking mechanism is an accident waiting to happen, but that's a whole other story.

Also, if your values are complex the system will break (or rather printout of your dict will not look properly). Do yourself a favor and use some proper serialization, even the horrible pickle is better than this.

zwer, the above answers all my questions. And you are right, the values file grows exponentially because it appends and not replace. Also the reverse lookup gets it wrong when the values are the same.I was trying to avoid pickle or sqlite3 databases. Is there a better way of exporting variables to be used in another script? Its mainly in a mathematics context and i'll be using the variable values for multiple calculations in diffrent files and updating them. I'm a complete newbie in programming
How will you serialize your data to share it between scripts depends on the data - in pure Python, if you only have number lists storing ",".join(your_numbers) in a file and then retrieving it by reading the file and doing your_numbers = [float(num) for num in file_content.split(",")] ought to be enough. For more complex structures you can use JSON - it's fast enough and will do most of the 'magic' itself. If you want to add numpy dependency to your project, it has some wickedly fast file storage and retrieval methods so be sure to check it out.

Collectives™ on Stack Overflow

Writing multiple variables to a file using a function

2 Answers 2

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related