Questions tagged [text-processing]
Manipulation or examining of text by programs, scripts, etc.
8,526 questions
1
vote
2
answers
131
views
find awk grep - search and replace & passing modified contents to awk to overwrite the existing file
I have a folder with many subfolders full of various Quarto(reg) files & in those files there are links that are located in varying positions in the file lines.
UPDATE ON 3 November 2025 in ...
- 205
4
votes
4
answers
455
views
How can I find common prefixes in file names to group them?
I would like to be able to find all files in multiple directories whose file names start with the same string, but preferably not if that string is only one word or contains fewer than perhaps 5 ...
- 4,433
4
votes
4
answers
479
views
Remove new lines and everything after comment symbol with awk or sed
How to remove comments and newline symbols without using two pipes.
I have bookmarks.txt file with comments.
https://cookies.com # recipes cookbook
https://magicwands.com # shopping
I can copy link ...
- 133
2
votes
1
answer
112
views
Tmux pane with long-running session using wrong character set?
Today I connected to a long-running process in tmux over ssh for work, to find that the pane the process was running in seems to have started using the wrong character encoding for its output, leading ...
- 125
2
votes
1
answer
93
views
Redirect `rtf` output to file
System Info
alinuxchap@libertus-desktop:/usr/share/X11/xkb $ uname -a
Linux libertus-desktop 6.12.25+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.12.25-1+rpt1 (2025-04-30) aarch64 GNU/Linux
alinuxchap@...
- 53
3
votes
5
answers
716
views
Randomly pick single line from multiple lines while assigning value to environment variable
In a certain script that we run routinely we configure hostnames in environment variables. Since hostnames can change overtime, we try to dynamically pick the current set of hosts using linux's ...
- 359
2
votes
5
answers
146
views
formatting git log messages for later processing
I am trying to format and connect git log messages for later processing.
I am using git log --pretty=format:'%H %s' to get commit hash and the complete message at the moment.
I need commit messages to ...
- 359
1
vote
3
answers
128
views
edit all the values in a specific column based on row numbers range
I have a PDB file (coordinates of atoms in a protein) on a Linux machine:
ATOM 1 N GLY A 1 0.535 51.766 5.682 1.00 0.00
ATOM 2 CA GLY A 1 -0.712 50....
- 444
5
votes
6
answers
926
views
Remove the first field (and leading spaces) with a single AWK
Consider this input and output:
foo bar baz
bar baz
How do you achieve with a single AWK? Please explain your approach too.
These are a couple tries:
$ awk '{ $1 = ""; print(substr($0, 2)) ...
- 3,502
0
votes
2
answers
134
views
List and count ciphers used by cryptsetup in /dev/mapper devices
In my Linux Computer there are many files called file1, file2, file3 ... in /dev/mapper/.
Now I want to have an overview from the files what cipher is used how often.
I tried this
for i in /dev/...
- 539
1
vote
3
answers
105
views
Extracting "devname" from log message with re_extract
Can anyone help? I've exhausted my knowledge and troubleshooting skills trying to get this working.
Here is the example data from "msg":
date=2025-03-26 time=12:45:57 devname="this-is-...
- 11
0
votes
1
answer
171
views
Use sed to replace only part of a string
I'm trying to replace bobearl with jim in the following string
"billy" "bobearl" and "johnny"
I can do something like this:
sed 's/bob/jim/' /tmp/text.txt
"billy&...
- 1
0
votes
0
answers
109
views
Advanced CLI tool/code to determine text encoding (besides enca)
Looking for advanced CLI tool/code to determine text Codepage/Language (besides enca).
Goal: Automate as much as possible conversion of hundreds/thousands of 8-bit text files (including non-ASCII ...
- 113
9
votes
5
answers
2k
views
Run command on each line of CSV file, using fields in different places of the command
I have a CSV file and want to run a command for each line, using the fields of the file as separate arguments.
For example given the following file:
foo,42,red
bar,13,blue
baz,27,green
I want to run ...
- 312
-4
votes
5
answers
197
views
Command to display all the employees whose first name have more than 6 characters
From the script below I need to know the following:
EmpNo#Email#Name#JobLevel#Experience
641357#Amrit_Mohanty#Amrit Mohanty#3#2
678522#Puneet_Mishra#Puneet Mishra#3#1
670242#Vikas_Bharti#Vikas Bharti#...
- 183
3
votes
5
answers
706
views
removing braces statements containing nested braces inside
A typical latex problem:
\SomeStyle{\otherstyle{this is the \textit{nested part} some more text...}}
Now I want to remove all \SomeStyle{...} but not the content. Content contains nested braces. The ...
- 153
2
votes
2
answers
1k
views
Why is the file changing before being written to?
On Kubuntu Linux, The Google Chrome browser adds a checksum to the file, preventing simply editing the file by hand. So I'm writing a script to add the checksum.
$ cat .config/google-chrome/Default/...
- 16.5k
-2
votes
3
answers
189
views
How to replace two lines containing [tab] chars into one line with just [newline] char, using a bash script?
In a directory I have a bunch of text files. Some of the files contain double lines with a [tab] char only. I want to find and change these two "tabbed lines" into one line with a new line ...
- 21
6
votes
2
answers
390
views
Update object inside array inside another JSON object
I have a huge JSON object with an array of objects inside it. I have to add key:value pair to a specific object in the array. For example, let the input object is:
{
"a": {
"b&...
- 63
0
votes
1
answer
240
views
Find all files in directory and apply commands to each of them
I want to apply commands below to all files in a directory instead of one file.
cat file.txt | sed -E "s/\@([0-9]+)\W+~(.*?)/\1 \2/g" | tr -d '~'
cat file.txt | sed -E "s/\@([0-9]+).*\~...
- 103
0
votes
2
answers
121
views
BSD sed/awk moving portion of line to line above (switching attribute in HTML file)
My situation is simple : I have an HTML file with several lines containing only the indented <section> block tag, each line followed by an (also indented) <h3 id="YYYY">...</...
- 109
1
vote
1
answer
97
views
Delete lines containing partial string match
I have 2 files
file1
00:00:00:00:00:01
file2
00:00:00:00:00:02 foo bar
00:00:00:00:00:01 something else
What I want to do is compare the two files and remove 00:00:00:00:00:01 from file 2 so I end ...
- 125
1
vote
8
answers
229
views
linux shell script to remove 1 char in a particular field in file having lines of around 3000
My input file:
1oo+457864227yexaloo+6784536pkp8907654
2oo+499004227yexaloo+69008908pkp8907654
3oo+648968976yexaloo+53589094pkp8907654
4oo+490764578yexaloo+6784536pkp8907654
I want to find out the ...
- 11
3
votes
3
answers
486
views
duplicate columns with AWK and separate them by tab
I have a large file with the following format tab-separated:
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT recombination
chr1 586001 >63041388>63041391 G ...
- 283
0
votes
2
answers
139
views
Run program only on matching lines
Let's say I have a program blackbox, and a file with the following contents:
in this file
this line contains =TAG=
so does =TAG= this one
as =TAG= does this other line
this line does ...
- 1,191