Questions tagged [buffer]
This tag is for questions about kernel buffer caches, including pipe buffers. These are used to store the recently accessed files and/or frequently accessed files.
178 questions
3
votes
2
answers
167
views
What explains this very odd behavior of GNU grep interacting with buffering and pipes and how to stop it?
This is best illustrated with an example I feel:
{ printf 'foo\nbar\n' ; sleep 2 ; } | grep -m1 foo
{ printf 'foo\n' ; sleep 2 ; printf 'bar\n' ; sleep 2 ; } | grep -m1 foo
Both of these commands, ...
- 171
7
votes
3
answers
2k
views
Configure Linux to regularly sync cached data to disk
The sync command "writes any data buffered in memory out to disk". As far as I understand, data may be buffered in memory for very long time, even if the disks have no activity.
How can I ...
- 597
0
votes
0
answers
37
views
How can I get the _real_ progress of dd, ignoring fast/staging buffers? [duplicate]
I'm using the dd command to write to a USB stick. My command is pretty straightforward:
dd if=myimage.iso of=/dev/sdd bs=1M status=progress
and indeed, I seem to be getting the progress reported:
...
- 11.1k
0
votes
2
answers
165
views
Terminating the whole pipeline at once when its last process ends
Given the pipeline
printf '%s\n' 1 2 3 4 5 6 7 8 9 |
while read -r num
do
echo "$num" > /dev/stderr
echo "$num"
done |
while read -r num
do
echo $(( $num * 10 ))
[ &...
- 449
0
votes
1
answer
135
views
How do Kernel use pagecache?
I have a problem with pagecache that I don't understand.
As I understand, the pagecache will serve as disk cache for reading from and writing to disk. But I don't know how the kernel can map 10G ...
0
votes
1
answer
891
views
Linux: Getting the kernel buffer size for a socket
I have a C application which receives a lot of data over a TCP socket. Is it somehow possible to get the kernel buffer size for that file descriptor / socket? I would like to know how much data is ...
- 213
1
vote
1
answer
330
views
Create a file for writing with controlled flushing to disk in large chunks
On Linux I have a process (ffmpeg) that writes very slowly (even slower than 1kb / s sometimes) to disk. Ffmpeg can buffer this to 256kb chunks that get written infrequently but ffmpeg hangs ...
- 153
7
votes
1
answer
682
views
Why does tar's handling of stdout and - differ?
After a comment from OP, I discovered that /dev/stdout gives blocks of 10 KiB even after disabling buffering, but - does not. Why is this? I could not find anything regarding this in man tar nor man ...
- 460
0
votes
1
answer
2k
views
Is rsync really a culprit for causing low memory on our VM?
In a VM of 16GB RAM, we are running rsync as a cron job(for every 10 minutes) on our production to sync GBs of folder from AWS EFS to local storage. After few days of running we found that the VM is ...
- 103
0
votes
0
answers
121
views
Why piping cat into head -c 5 for a chardev results in many more calls to the driver's read than just calling head -c 5 on the chardev?
Taking inspiration from this blog post, I'm playing around with linux device drivers (which I'm studying from ).
The read field of the file_operations associated with the driver is initialized to the ...
- 2,362
3
votes
1
answer
325
views
Why is the stdout line buffering for cut -c differ from cut -b?
Our RHEL 7 machines have great long Log files and I asked about buffering of cut in this question. That question remains but a bit of experimentation showed a different issue.
I decided to try using ...
- 423
1
vote
1
answer
116
views
Piping line-by-line
I have a script that "listens" to amixer events (volume change and mute), piping the volume and mute status to xob. It works great, but the (ab)use of stdbuf -oL <command> looks wrong ...
- 13
1
vote
1
answer
3k
views
Is it safe to clear the buffers and cache on a linux server
After reading this link: How do you empty the buffers and cache on a Linux system?, I know that there are some commands that can help us to empty the buffers and cache of the OS.
But I'm not sure if ...
- 3,411
2
votes
1
answer
486
views
pv gives 100% progress with small buffer
I have to do a long task (convert image format) for every file of a folder. I achieved to use pv to write the estimation of duration with (used sleep here to simulated processing time):
pv -B 1 =(find ...
- 121
0
votes
0
answers
871
views
Why are packets being dropped (kfree_skb)?
I am currently sending ~9mpps, each packet is 72bytes large, and is a UDP DNS request. The packets are being sent to a Dell server I have using an intel i40e interface. The server is using all of its ...
- 732
1
vote
2
answers
919
views
why do we need to pass buffers to system calls in order to have information returned? Why can't system calls allocate the buffer internally?
I'm reading TLPI, and in section 4.4:
System calls don’t allocate memory for buffers that are used to return information to the caller. Instead, we must pass a pointer to a previously allocated ...
- 161
0
votes
1
answer
151
views
When is it beneficial to add -u to tr?
One of the arguments to the tr command is:
-u Guarantee that any output is unbuffered.
Under what circumstances is it a good idea to not buffer the output?
e.g. should I use -u if I expect each ...
- 867
6
votes
2
answers
787
views
Why is mawk's output (STDOUT) buffered even though it is the terminal?
I am aware that STDOUT is usually buffered by commands like mawk (but not gawk), grep, sed, and so on, unless used with the appropriate options (i.e. mawk --Winteractive, or grep --line-buffered, or ...
- 2,019
4
votes
1
answer
1k
views
configure vim to use both tmux and system clipboard
I'm fairly new to vim (I'm using nvim) so forgive my lack of knowledge, many questions are similar to mine, but not quite the same.
I would like this:
vim by default should cut/copy to a buffer, not ...
- 101
1
vote
0
answers
48
views
linux handling of files
I want to perform updates on a file via multiple processes in parallel. These processes all open this file for write in parallel.
Abbreviations used:
f : file,
p[i] : process i,
b[i] : buffer for FD ...
- 150
2
votes
0
answers
702
views
ping: sendmsg: No buffer space available
I am running a ping from one of the terminals:
ping www.google.com
This runs without any issues indefinitely.
Now on another terminal, I run:
for i in {1..5}; do speedtest; done
Now after sometime on ...
- 211
3
votes
2
answers
625
views
How do I feed data to a pipe until it's full, no more no less?
I want to use pipes on Linux as a synchronization primitive between a master process and a slave process. The classic way is to create two pipes, but I believe there's a way to use a single fd instead....
3
votes
1
answer
1k
views
broken pipe when tar extracted from stdin buffer
I am restoring files from a tar archive on an LTO-7 tape to a locally mounted network share. If I directly restore to the share it runs very slowly (90 MB/s). When I use an additional buffer I get the ...
- 205
0
votes
0
answers
993
views
Effect of Buffer Size in File I/O in Unix
I am trying to understand the inner workings of Unix based OSs. I was reading on buffered I/O, and how the buffer size affects the number of system calls made, which in turn affects the total time ...
- 101
2
votes
1
answer
394
views
Does it have 3 levels of buffering, or 2 levels of buffering, when using stdio library?
From The Linux Programming Interface:
(I've read the related sections. I pasted this sum-up diagram just for illustration.)
I know that for direct IO which uses read and write system calls. There is a ...
- 1,257