Making a preprocessor with Python

Question

I basically need to write a C preprocessor with Python, I've search around and since i need to fully custom my code and have a perfect understanding of what's going on, I better writting on my own.

Here's where I am : I first parse some header files (.h) to find #define keyword and build up a dictionnary with all the founded directives (with their values they have one). Then I need to parse source files (.c) depending of the directives I've found earlier. The mechanism I use at the moment to check if the code need to be processed is the following : I take all my define's name and their values and do a exec("define_name = define_value") (with the value '1' when not specified). Then to resolve a condition such as #if defined DEFINE_1 || defined DEFINE_2 && (DEFINE_3 == 10) .... I remove the C preprocessor keyword to make them Python style wich will produce DEFINE_1 or defined DEFINE_2 and (DEFINE_3 == 10).

And i finally use eval(...) on that string to find out the result.

THE QUESTION is I was wondering if the use of exec / eval is necessary and many people are somekind of reluctant to use them, is there a better solution?

In my experience with Python-based preprocessors, Python is much too slow to be practical for this. Also, the standard libraries for e.g. MSVC are bad to preprocess (though worse to compile). What problem are you trying to solve? — geometrian
– geometrian, Commented Apr 3, 2015 at 16:21

Wes Hardaker · Accepted Answer · 2013-06-05 13:09:31Z

4

Certainly the exec() isn't needed and shouldn't be used. I'm not even sure what you're expecting it to do anyway, as it would call a shell to set a variable that would only exist in the sub-shell.

And generally, you should avoid eval() statements as well as it's rarely the right thing to do.

So, what can you do?

1) first off, because programs can be written where one statement over-rides a previous statement, you can't pre-process the .h file (or even assume the #define's you're looking for are only in .h files in the first place) and have it work. Consider this:

#define foo 1
#if foo == 1
this line is true!
#endif
#define foo 0

If you pre-process everything you'll set "foo" to 1, then to 0 and then evaluate the #if later. You can't do that...

2) A more common thing to do would be to write a parser that goes line-by-line and deals with the contents of each line one at a time. This way you can even write a recursive function to deal with #include statements so you can start with just the .c file and let it pull in the right headers it's using, rather than requiring them to be specified in some other way.

In the end, you should end up with something like (in a function called "read_file"):

# ... file opening not shown ...

for line in file:
    includematch = re.match("#include\\s+\\"(.*)\\"", line)
    if match:
        # deal with an include statement by calling a function to process it
        read_file(includematch.group(1), definedict)

    definematch = re.match("#define\\s+(\\w+)\\s+(.*)")
    if definematch:
        # deal with define statements by saving it in a dict
        definedict[match.group(1)] = definedict[match.group(2)]

    #....

Obviously if I showed you the whole solution (and the above is hardly pretty code, but it's concise for showing purposes) I'd be solving your problem (homework?) for you. But the above is a better way to architect the whole thing than the path you were heading down.

answered Jun 5, 2013 at 13:09

Wes Hardaker

22.3k2 gold badges42 silver badges70 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Jaay Over a year ago

Thank you for your answer, it's not homework it is work ;). Just to precise some things, the project I work on is very strict and people are not allowed to put #define where they want (I just have 1 or 2 files just filled with #define so I just have to parse that), so there will not be redefines like you said. I have a working code at the moment but with loads of eval/exec, the main problem is how can I handle complex #if/#elif ... conditions with what have been parsed so far without using eval and exec?

Wes Hardaker Over a year ago

Well, you have to end up doing either a full-blown parser (which is harder but safer) or eval. Eval allows you to skip implementing paren matching, logic, etc, and you can concentrate on just replacement of values and then eval the rest. You can always start with eval and work your way from there, replacing it ideally in the future.

Wes Hardaker Over a year ago

note: just because you have a structured strict environment now with certain rules, doesn't mean those rules won't change in the future. You might consider doing it the right way first to mirror cpp as much as possible only to help future-proof yourself.

Jaay Over a year ago

Yes I've tried as much as possible to abstract from the current project, but things had to get done... like always, thank you for the answers and advices :)

This isn't my real name Over a year ago

Also note that it's entirely plausible that you might find defines and redefines as demonstrated in the answer, in the header files belonging to the implementation. So even if your coding standards don't allow it, you still need to account for it.

Collectives™ on Stack Overflow

Making a preprocessor with Python

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related