91 questions
0
votes
1
answer
37
views
why TensorDataset devide the data to minibatches?
Why TensorDataset devide the data to minibatches? For example, when putting in it 2D array, instead of yielding 2D tensors as batches, it sets the required batches to be minibatches, and its actual &...
- 305
1
vote
0
answers
481
views
What does "gradient_accumulation_steps" do in deepspeed?
I'm training my model with accelerate package which uses deepspeed internally. But I can't understand gradient_accumulation_steps param in its configuration.
In my knowledge, ...
- 216
0
votes
1
answer
45
views
How exactly does tensorflow perform mini-batch gradient descent?
I am unable to achieve good results unless I choose a batch size of 1. By good, I mean error decreases significantly through the epochs. When I do a full batch of 30 the results are poor, error ...
- 380
0
votes
1
answer
333
views
PyTorch LSTM: Should I use mini batch size or single batch but randomly remove n observation when calculating loss?
As I understood, the idea of mini batch size is equivalent with fitting the model to only a portion of all training data at each step (one epoch consists of many steps, depending on the batch size) to ...
- 63
0
votes
1
answer
871
views
Drop in scikit-learn KMeans replacement for GPU
I was wondering what is a good drop in replacement for from sklearn.cluster import KMeans. Others online have suggested using cuML from Nvidia's RAPIDS package, but this wasn't able to compile or ...
- 13
0
votes
0
answers
92
views
How to create mini-batches of predefined sizes from a sparse 2D matrix in Python?
I have a sparse 2D matrix saved on a disk (.npz extension) that I've created in preprocessing step with scipy.sparse.csr_matrix. It is a long sequence of piano-roll (a numerical form of MIDI ...
- 771
0
votes
0
answers
95
views
mini-batch gradient decent bad accuracy/loss
I’m trying mini-batch gradient descent on the popular iris dataset, but somehow I don’t manage to get the accuracy of the model above 75-80%. Also, I’m not certain if I’m calculating the loss as well ...
- 39
-2
votes
1
answer
2k
views
What is the right way of mini-batching the validation set while training?
I am training a neural network. For training I get 80% of my data and divide it to a number of mini-batches. I train on each mini batch, then update parameters, until all data is visited. I repeat the ...
- 2,098
0
votes
1
answer
402
views
The optimal way to split Eigen MatrixXd into fixed-size batches with randomly shuffled rows
I have input and target data represented as MatrixXd (N x M) and VectorXd (N). The goal is to create mini-batches of size K consisting of a subset of input and target data shuffled in the same way. ...
0
votes
0
answers
566
views
how to balance minibatches on keras when training a convolutional neural network?
I am training a convolutional neural network on images (with size 299, 299, 3). The images can have labels: 0, 1 or 2 (multiclass classification), and the 3 classes are very unbalanced. To improve the ...
- 11
1
vote
1
answer
1k
views
mini batch backpropagation clarification
Read through a lot of articles and now I am in my braindrain end, and need a fresh perspective on the concept of mini batch. I am new to machine learning and would appreciate any advice on whether my ...
- 27
0
votes
1
answer
64
views
How can I increase the number of mini-batch using the Standard Updater class in Chainer substantially?
How can I increase the number of mini-batch using the Standard Updater class in Chainer substantially?
In case of PyTorch,
I can increase the number of mini-batch substantially.
Execute loss.backward(...
3
votes
2
answers
2k
views
pytorch loss accumulated when using mini-batch
I am new to pytorch. May I ask what is the difference between adding 'loss.item()' or not? The following 2 parts of code:
for epoch in range(epochs):
trainingloss =0
for i in range(0,X.size()[...
- 449
-1
votes
1
answer
3k
views
PyTorch minibatch training very slow
When training my model on the adult income data set and using minibatches training is very slow regardless if I use PyTorch's DataLoader or a basic implementation for minibatch training.
Is there a ...
- 75
0
votes
1
answer
1k
views
In tensorflow, why is there only one validation loss, when there are many mini-batch of validation data?
In tensorflow, if we provide validation_data in .fit(), we get validation loss. But there is only one validation loss even if the validation dataset has many mini-batches. So I was wondering how ...
- 245
0
votes
1
answer
1k
views
How to handle samples with multiple images in a pytorch image processing model?
My model training involves encoding multiple variants of a same image then summing the produced representation over all variants for the image.
The data loader produces tensor batches of the shape: [...
- 405
1
vote
2
answers
1k
views
Mini-batch performs poorly than Batch gradient descent?
I am able to get pretty good results from batch gradient descent(batch size 37000) but when I try out mini-batch gradient descent, I get very poor results (even with adam and dropout).
In batch gd, I'...
1
vote
0
answers
24
views
Should sequence nature of input data within a mini-batch be maintained?
Suppose a multivariate time series prediction problem for the following data
rows = 2000
cols = 6
data = np.arange(int(rows*cols)).reshape(-1,rows).transpose()
print(data[0:20])
print('\n {} \n'....
- 416
1
vote
0
answers
1k
views
loss.backward() with minibatch in pytorch
I came across this code online and I was wondering if I interpreted it correctly. Below is a part of a gradient descent process. full code available through the link https://jovian.ml/aakashns/03-...
- 11
1
vote
1
answer
653
views
Do I calculate one loss per mini batch or one loss per entry in mini batch in deep reinforcement learning?
New to neural network and Pytorch.
I have 300 replay memories in each mini batch. I've seen people calculate one loss for the 300 replay memories, but it doesn't really make sense to me. The 300 ...
- 11
0
votes
1
answer
479
views
Trying to understand shuffle within mini-batch in tensorflow Dataset
From here I understand what shuffle, batch and repeat do. I'm working on Medical image data where each mini-batch has slices from one patient record. I'm looking for a way to shuffle within the ...
7
votes
1
answer
3k
views
Zero predictions despite masking support for zero-padded mini batch LSTM training in keras
Problem Statement
I’m training a many-to-many LSTM in keras with tensorflow backend (tf version 1.13.1) on tagged text sequences to predict the tag of each element in the sequence using pretrained ...
- 570
11
votes
2
answers
16k
views
What is 'mini-batch' in deep learning?
I'm taking the fast-ai course, and in "Lesson 2 - SGD" it says:
Mini-batch: a random bunch of points that you use to update your weights
And it also says that gradient descent uses mini-...
- 323
1
vote
0
answers
599
views
How to generate custom mini-batches using Tensorflow 2.0, such as those in the paper "In defense of the triplet loss"?
I want to implement a custom mini-batch generator in Tensorflow 2.0 using tf.data.Dataset API. Concretely, I have image data, 100 classes with ~200 examples each. For each mini-batch, I want to ...
-2
votes
1
answer
1k
views
Pytorch minibatching keeps model from training
I am trying to classify sequences by a binary feature. I have a dataset of sequence/label pairs and am using a simple one-layer LSTM to classify each sequence. Before I implemented minibatching, I was ...
- 1