1

I'm following the book "Deep Learning with PyTorch Step By Step" and I have a question about the predict method in the StepByStep class (from this repository: GitHub).

The current implementation is:

def predict(self, x):
        self.model.eval()
        x_tensor = torch.as_tensor(x).float()
        y_hat_tensor = self.model(x_tensor.to(self.device))

        self.model.train()
        return y_hat_tensor.detach().cpu().numpy()

My question is: should I wrap the forward pass with torch.inference_mode() or torch.no_grad() even though I'm already calling self.model.eval() like :

def predict(self, x):
        self.model.eval()
        x_tensor = torch.as_tensor(x).float()

        with torch.inference_mode():
            y_hat_tensor = self.model(x_tensor.to(self.device))

        self.model.train()
        return y_hat_tensor.detach().cpu().numpy() # detach() is superfluous

I understand that model.eval() changes the behavior of layers like dropout and batch normalization, but I believe the context manager should still be added for better performance and memory efficiency. However, I'm not sure why the original implementation doesn't include it. What are the best practices here?

Improve this question
2
  • @cronoik Thanks :) I had already checked that thread before posting. That discussion is focused on validation inside a training loop, while my question is specifically about pure inference in a predict() method and the use of torch.inference_mode() in that context Different scenario, so that link doesn’t actually answer my question. Commented Nov 4 at 14:49
  • 2
    The linked question actually does answer yours, because it makes clear that model.eval() and no_grad() do different things and you should use both. inference_mode() has similar functionality to no_grad(), so you should still use it and mode.eval(). As for why the book doesn't use it, who knows. I don't know the book and make no assumptions about the author, but technically anyone can write a book, whether they fully grasp the material or not. Commented Nov 4 at 14:59
Related questions
702
Python exit commands - why so many and when should each be used?
949
What is __future__ in Python used for and how/when to use it, and how it works
0
Translation CNN from keras code to pytorch
Load 2 more related questions Show fewer related questions

0

Reset to default

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.