Read a python variable in a shell script?

Consider something akin to the following:

#!/bin/bash
#      ^^^^ NOT /bin/sh, which doesn't have process substitution available.

python_script='
import sys
d = {}                                 # create a dict to store values in
exec(open(sys.argv[1], "r").read(), d) # exec code with that dict as namespace
for k in sys.argv[2:]:
  print("%s\0" % str(d[k]).split("\0")[0]) # extract values
'

read_python_vars() {
  local python_file=$1; shift
  local varname
  for varname; do
    IFS= read -r -d '' "${varname#*:}"
  done < <(python -c "$python_script" "$python_file" "${@%%:*}")
}

You might then use this as:

read_python_vars config.py week_date:dt cust_id:id
echo "Customer id is $id; date range is $dt"

...or, if you didn't want to rename the variables as they were read, simply:

read_python_vars config.py week_date cust_id
echo "Customer id is $cust_id; date range is $week_date"

Advantages:

• Unlike a naive regex-based solution (which would have trouble with some of the details of Python parsing -- try teaching sed to handle both raw and regular strings, and both single and triple quotes without making it into a hairball!) or a similar approach that used newline-delimited output from the Python subprocess, this will correctly handle any object for which str() gives a representation with no NUL characters that your shell script can use.

• Running content through the Python interpreter also means you can determine values programmatically -- for instance, you could have some Python code that asks your version control system for the last-change-date of relevant content.

  Think about scenarios such as this one:

  start_date = '01/03/16'
  end_date = '01/09/16'
  week_date = '%s-%s' % (start_date, end_date)
  

  ...using a Python interpreter to parse Python means you aren't restricting how people can update/modify your Python config file in the future.

Now, let's talk caveats:

• If your Python code has side effects, those side effects will obviously take effect (just as they would if you chose to import the file as a module in Python). Don't use this to extract configuration from a file whose contents you don't trust.
• Python strings are Pascal-style: They can contain literal NULs. Strings in shell languages are C-style: They're terminated by the first NUL character. Thus, some variables can exist in Python than cannot be represented in shell without nonliteral escaping. To prevent an object whose str() representation contains NULs from spilling forward into other assignments, this code terminates strings at their first NUL.

Now, let's talk about implementation details.

• ${@%%:*} is an expansion of $@ which trims all content after and including the first : in each argument, thus passing only the Python variable names to the interpreter. Similarly, ${varname#*:} is an expansion which trims everything up to and including the first : from the variable name passed to read. See the bash-hackers page on parameter expansion.

• Using <(python ...) is process substitution syntax: The <(...) expression evaluates to a filename which, when read, will provide output of that command. Using < <(...) redirects output from that file, and thus that command (the first < is a redirection, whereas the second is part of the <( token that starts a process substitution). Using this form to get output into a while read loop avoids the bug mentioned in BashFAQ #24 ("I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?").

• The IFS= read -r -d '' construct has a series of components, each of which makes the behavior of read more true to the original content:

  • Clearing IFS for the duration of the command prevents whitespace from being trimmed from the end of the variable's content.
  • Using -r prevents literal backslashes from being consumed by read itself rather than represented in the output.
  • Using -d '' sets the first character of the empty string '' to be the record delimiter. Since C strings are NUL-terminated and the shell uses C strings, that character is a NUL. This ensures that variables' content can contain any non-NUL value, including literal newlines.

  See BashFAQ #001 ("How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?") for more on the process of reading record-oriented data from a string in bash.