In Common Lisp, replacing "get-setf-method" with "get-setf-expansion" is not always working

The following function replaces the 3rd output from get-setf-expansion with newly generated gensym symbols:

(defun get-unique-setf-expansion (place &optional environment)
  "Calls GET-SETF-EXPANSION but replaces all store-vars with fresh gensyms."

  (multiple-value-bind (vars vals store-vars writer-form reader-form)
      (get-setf-expansion place environment)
    
    ;; 1. Generate new unique symbols for the store-vars list
    (let ((new-store-vars (loop for var in store-vars collect (gensym (concatenate 'string (symbol-name var) "-")))))
      
      ;; 2. Create an association list to map the old symbols to the new ones
      (let ((substitution-list (mapcar #'list* store-vars new-store-vars)))
        
        ;; 3. Substitute the old symbols with the new ones in the writer form
        ;;    (Reader-form and other lists usually don't contain store-vars)
        (let ((new-writer-form (sublis substitution-list writer-form)))
          
          ;; Return the new expansion
          (values vars vals new-store-vars new-writer-form reader-form))))))

When get-unique-setf-expansion substitutes for get-unique-method or get-unique-expansion, the expanded expression of the new sortf is similar with that of the original sortf:

;; Executed with "SBCL 2.1.11.debian"

(LET* ((#:NEW1-714 X)
       (#:AR715 AR)
       (#:G716 (INCF I))
       (#:NEW1-717 (AREF #:AR715 #:G716))
       (#:LST718 LST)
       (#:NEW1-719 (CAR #:LST718)))
  (UNLESS (> #:NEW1-714 #:NEW1-717) (ROTATEF #:NEW1-714 #:NEW1-717))
  (UNLESS (> #:NEW1-714 #:NEW1-719) (ROTATEF #:NEW1-714 #:NEW1-719))
  (UNLESS (> #:NEW1-717 #:NEW1-719) (ROTATEF #:NEW1-717 #:NEW1-719))
  (SETQ X #:NEW1-714)
  (FUNCALL #'(SETF AREF) #:NEW1-717 #:AR715 #:G716)
  (SB-KERNEL:%RPLACA #:LST718 #:NEW1-719))

It looks like get-setf-method was removed from the CL spec in 1991 (in Issue SETF-METHOD-VS-SETF-METHOD). So it's no wonder SBCL doesn't have it.

(I feel like this should have been a regular question, not the new advice thing)

I wrongly chose the type of question as "General Advice/Other", as @Shawn commented. I tried to change the type, but it cannot be changed.

Also, I bet the first one is working because the two instances of #:NEW1 are different uninterned symbols that just happen to print the same, but are treated distinctly when the code is executed. (eq '#:NEW1 '#:NEW1) is nil, after all.

@Shawn Thank you for explaining why the code works with uninterned symbols, which I was not familiar with.

Another reason that causes my misunderstanding was the use of a literal list. In the updated part of the question, the literal list '(25 30 35) is used.

Similarly, when I execute the following code in SBCL

(let ((lst '(0 1 2)))
  (setf (car lst) 10)
  (values
   (car lst)
   lst))

The output is

0
(10 1 2)

Modifying a literal list (or other object) is undefined, so weird stuff can happen if you try, yeah.

I guess that the result of modifying a literal depends on a compiler, as @Shawn commented that modifying a literal is undefined. In SBCL, if I executes (declare (optimize (debug 3))) in advance, the output is (values 10 (10 1 2)). So, high optimization made the inconsistent output (values 0 (10 1 2)). In addition, I should've carefully check the warning message about a literal by SBCL.

The literal list issue is a simple compiler optimization. It assumes that the list won't be modified, so (car lst) is replaced with the original car of the list.

It's a little surprising that the result wasn't 0 (0 1 2) since it could have optimized away the reference to lst itself. But since local variable references are cheap (maybe no more expensive than referencing the literal itself), it might not bother with this.

@Barmar, Thank you for the explanation about compiler optimization. As you said, calling the car function sounds more expensive than referencing a literal or a variable.

The representation #:NEW denotes an uninterned symbol.

Just because you see the same #:NEW in two different instances of a macro expansion doesn't meant that there is any issue.

First of all, we don't know whether it is actually the same symbol, because, look:

[1]> (make-symbol "ABC")
#:ABC
[2]> (make-symbol "ABC")
#:ABC
[3]> (eq '#:ABC '#:ABC)
NIL

The #: notation denotes a symbol with no home package, which is usually an uninterned symbol. (I have to say usually because Ron Garrett may be reading).

Secondly, even if a macro re-uses the same uninterned symbol for multiple expansions, that does not mean there is a hygiene problem. If only that macro knows about that symbol (it cannot clash with anything outside the macro) and if nested expansions of the macro do not clash with each other because of that symbol, then it is cool. For instance we can write a repeat macro like this:

(defmacro repeat (count-form &rest body-forms)
  `(let ((#1=#:count ,count-form))
     (do ((#2=#:dummy 0 (1+ #2#)))
         ((>= #2# #1#))
       ,@body-forms))) 

This works fine nested:

[2]> (let ((count 0)) (repeat 3 (repeat 4 (print (incf count)))))
1
2
3
4
5
6
7
8
9
10
11
12
NIL

The expansion appears to repeat symbols:

[3]> (macroexpand '(repeat x a b c))
(LET ((#:COUNT X)) (DO ((#:DUMMY 0 (1+ #:DUMMY))) ((>= #:DUMMY #:COUNT)) A B C)) ;
T
[4]> (macroexpand '(repeat y e f g))
(LET ((#:COUNT Y)) (DO ((#:DUMMY 0 (1+ #:DUMMY))) ((>= #:DUMMY #:COUNT)) E F G)) ;
T

And in fact they are repeated. The #:COUNT and #:DUMMY in both expansions are the same symbols. But, so what? The symbols are not known outside of the macro because they were created when the macro was read. The read notation #: produces a fresh symbol every time (as the eq expression above shows).

The macro uses these symbols in such a way that they are lexically scoped, so when the macro is nested with itself, the reuse makes no difference.

More Lisp macros should arguably be using symbols like this and not wasting resources allocating new gensyms on every call.

I didn't even bother using local variables in the macro to refer to these symbols; I used the circle notation to refer to them in multiple places.

#1=#:COUNT means the same thing as #:COUNT but #:COUNT is associated with the integer label 1. Later in the same expression, wherever we write #1#, it will reproduce the #:COUNT object.

Instead of the circle notation, we can bind those uninterned symbols to local variables and insert the local variables, exactly like we do with gensyms:

(defmacro repeat (count-form &rest body-forms)
   (let ((count '#:count) ;; usually we use (gensym)
         (dummy '#:dummy))
  `(let ((,count ,count-form))
     (do ((,dummy 0 (1+ ,dummy)))
         ((>= ,dummy ,count))
       ,@body-forms))))

Needless to say, if we write some other macro which also uses #:count, that #:count will be a different symbol. It's only the same symbol across multiple expansions of repeat, and we know that not to be a problem.

There is a risk to this techique and it occurs when a macro refers to itself in its own expansion. Suppose repeat had some reason to use (repeat ...) inside the expansion, and suppose that repeat inserted some of the variables into that syntax: (repeat ... ,dummy). Now it is in trouble because the dummy symbol is lexically intercepted inside the inner repeat. For that case, we need gensym. gensyms ensure correctness of even nestings that are instigated by the macro itself, rather than external invocation nestings.

@Kaz, Thank you for introducing the #: notation and the circle notation. As you said, gensym can be omitted in many macros, without creating many different uninterned symbols that share the same names.

In contrast, in the sortf macro, different #:NEW1 symbols should be used, and get-setf-expansion already makes a new uninterned #:NEW1 whenever it's used. In sortf, the number of #:NEW1 symbols is proportional to the number of arguments that are passed to sortf. Therefore, sortf cannot use the fixed number of symbols that can be generated by the #: notation.

use cl:*print-circle* set to t to see identity (-> which printed objects are the same) of objects in the list.

You are using an 4 year old version of SBCL. The current version of SBCL is 2.5.10.

In addition, modifying a literal, such as '(25 30 35), with an operation like setf can be problematic. In particular, the result of modifying a literal can vary depending on the level of compiler optimization.

There are possibly several reasons why one should not modify a literal constant. Effects of compiler optimization is only one.

(defun foo ()
  (let ((a '(1 2 3)))
    (setf (first a) 10)))

In above code one tries to modify a literal list, which is a part of the code. There is no compiler optimization involved -> one is modifying code.

This can have unwanted observable effects or may even lead to an error (runtime or compile-time). Imagine a Common Lisp compiler which places code into non-writeable memory -> then the runtime (or even the CPU) might detect that attempt to change code. This gets more relevant on architectures, where code is protected against runtime changes. In addition a compiler might detect at compile time the part which attempts to modify code and will report that.

@rainer-joswig, Thank you for the tip of cl:*print-circle* and the explanation about the problem of modifying a literal. With cl:*print-circle* that is set as t, macroexpand-1 explicitly shows differences between uninterned symbols of the same name. In addition, modifying a literal can be a problem even if compiler optimization is not applied, as a literal is defined as a fixed value which should not be updated while running a program.