PyTorch: create non-fully-connected layer / concatenate output of hidden layers

Question

In PyTorch, I want to create a hidden layer whose neurons are not fully connected to the output layer. I try to concatenate the output of two linear layers but run into the following error:

RuntimeError: size mismatch, m1: [2 x 2], m2: [4 x 4]

my current code:

class NeuralNet2(nn.Module):
    def __init__(self):
        super(NeuralNet2, self).__init__()

        self.input = nn.Linear(2, 40)
        self.hiddenLeft = nn.Linear(40, 2)
        self.hiddenRight = nn.Linear(40, 2)
        self.out = nn.Linear(4, 4)

    def forward(self, x):
        x = self.input(x)
        xLeft, xRight = torch.sigmoid(self.hiddenLeft(x)), torch.sigmoid(self.hiddenRight(x))
        x = torch.cat((xLeft, xRight))
        x = self.out(x)

        return x

I don't get why there is a size mismatch? Is there an alternative way to implement non-fully-connected layers in pytorch?

I'm not sure if you posted the relevant parts, but that works flawlessly for me... I'm using the following code to instantiate the network: net = NeuralNet2(), as well as some input x = torch.Tensor(np.array([1,2])), and then simply call net(x). Which version are you using? — dennlinger
– dennlinger, Commented Oct 25, 2018 at 18:05
After trying out your suggestion x = torch.Tensor(np.array([1,2])) it worked for me too. Networks without concatenation do also accept input with shape [x, amountInputNeurons] while this network only accepts input with shape [amountInputNeurons]. Got to look deeper into concatenation to get back the old behavior. Thanks:) @dennlinger — Tengrath
– Tengrath, Commented Oct 25, 2018 at 20:28
Oh, I see. There might be a way to get this behavior by specifically telling the network which concatenation dimension to use. I believe for your case that would then be torch.cat((xLeft, xRight), axis=1). — dennlinger
– dennlinger, Commented Oct 26, 2018 at 6:24

Tengrath · Accepted Answer · 2018-10-26 12:50:49Z

2

It turned out to be a simple comprehension problem with the concatenation function. Changing x = torch.cat((xLeft, xRight)) to x = torch.cat((xLeft, xRight), dim=1) did the trick. Thanks @dennlinger

answered Oct 26, 2018 at 12:50

Tengrath

931 silver badge8 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

PyTorch: create non-fully-connected layer / concatenate output of hidden layers

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related