Reinforcement Learning function approximation with Neural Networks

The w in the pseudocode does not strictly have to be just a single weight vector. The text in the beginning of the chapter does refer to w as a "weight vector" a couple of times, but the pseudocode itself only mentions that w are the parameters of a differentiable action-value function approximator. A Neural Network perfectly fits that description.

In the case of a Neural Network, you can think of w as the combination of all weight matrices (alternatively; you can view it as a really really long vector constructed by unrolling all of the weight matrices into a single vector). You can view the lines of pseudocode performing the update on w as regular backpropagation in Neural Networks, optimizing all the parameters w to make the prediction q(S, A, w) slightly closer to R + gamma*q(S', A', w).

That single line of pseudocode basically summarizes the entire backpropagation procedure in the case where w is a huge vector consisting of unrolled weight matrices of a Neural Network. In practice, it cannot be implemented in a single line of code, because partial derivatives of earlier layers of the network (components of that gradients-of-q vector) depend on partial derivatives in layers closer to the output layer, so those have to be computed sequentially (which is what backpropagation as you know it if you're familiar with Neural Networks does).