0

I am trying to do ssh on a server and then do a grep to get the count of different errors in the log file. Once i have those details logs those into a CSV file. But when i am trying to run the grep command i am getting error.

#!/usr/bin/perl
my $addr = "user\@servername";
my $text = qq|Internal Server Error|;
my $remote_path = "/data/logs/error";
my $cmd = `ssh $remote_addr "grep -a $text $remote_path | awk -F " " '{print $4}' | sort -nr | uniq -c | sort -nr 2>/dev/null"`;
print $cmd;

But i am getting below error when i am running the script

grep: Internal: No such file or directory
grep: Server: No such file or directory
grep: Error: No such file or directory

Is there any suggestion how we can do this in shell script.

3
  • Why not do most (possibly all) of that directly in Perl? Commented Nov 5, 2018 at 22:43
  • @Shawn The only thing running in the local Perl program is the ssh command; everything else is executed by the shell on the remote host. Commented Nov 6, 2018 at 13:48
  • If you aren't willing to install the modules necessary to do this safely in perl (by using zdim's great answer), would you be interested in a shell-based answer that doesn't use perl at all? Commented Nov 6, 2018 at 20:13

1 Answer 1

5

First, in order to avoid a quoting nightmare and the many chances for shell injection I'd suggest to use a module, like String::ShellQuote

Then, I don't see that you need all those external tools, while such a long pipe-line is tricky and expensive. It invokes a number of programs, for jobs done really well in Perl, and requires very precise syntax.

Apart from ssh-ing, one other thing that an external tool may be good for here is to extract lines of interest with grep, in case that file is huge (otherwise you can read it into a scalar).

use warnings;
use strict;
use feature 'say';

use List::Util qw(uniq);  # in List::MoreUtils prior to module v1.45
use String::ShellQuote qw(shell_quote);

my $remote_addr = ...
my $remote_path = ...
my $text = 'Internal Server Error';

my $remote_cmd = shell_quote('grep', '-a', $text, $remote_path);
my $cmd = shell_quote('ssh', $remote_addr, $remote_cmd);

my @lines = qx($cmd);
chomp @lines;

# Process @lines as needed, perhaps
my @result = sort { $b <=> $a } uniq map { (split)[3] } @lines;
say for @result;

Once it comes to running external commands, there are many options. In the first place consider using a module. They all simplify things a lot, in particular with error checking, and are in general more reliable, while some also make harder jobs far more manageable.

An example with IPC::System::Simple

use IPC::System::Simple qw(capturex); 

my @lines = capturex('ssh', $remote_addr, $remote_cmd);

Since ssh executes the command when ran with one it doesn't need a shell (for that part) so capturex is used. See documentation for more options and for how to check for errors.

Some other options, from simple to more powerful, are Capture::Tiny, IPC::Run3, IPC::Run.

For more on all this see links assembled in this post (and search for more).


I can't see a need to run that pipeline as it stands but if there is one (stay on the remote host?) then form commands as above and then assemble the full pipeline

my $cgrep = shell_quote('grep', '-a', $text, $remote_path);
my $cawk  = shell_quote('awk', '-F', ' ', '{print $4}');
my $csort = shell_quote('sort', '-nr');
my $cuniq = shell_quote('uniq', '-c');

my $remote_cmd = "$cgrep | $cawk | $csort | $cuniq | $csort 2>/dev/null";

Note that the needed shell functionalities (| and redirection) shouldn't be quoted away.

The mere space in the awk piece may be awkward-looking but since it gets escaped it winds up right for -F. For me it's another sign of trouble with running external programs in shell pipelines; I couldn't figure out that bare space, thanks to Charles Duffy for the comment.

In this case the sort and uniq parts of the pipeline can be just typed as one string since it's only program names and options, but as soon as changes are made or any variables make their way in that becomes tricky. So I use shell_quote, for consistency and as a template.


Modules are reported missing and hard to obtain. Then escape what need be escaped (until you figure out how to get the modules, that is). In this case there happen to be little to fix, but that bit can serve as an example of common hoops to go through with complicated pipelines.

The string with $text needs to reach grep as such, one string. Since it passes through shell, which would break it by space into words, we need to protect (quote/escape) those spaces. Not to forget, we also need to get it to the shell in the first place, through Perl's parsing rules.

One way

my $text_quoted = q(') . quotemeta($text) . q(');

where quotemeta quotes all kinds of other things as well.

We also should protect the filename pattern, as it may rely on shell metacharacters (like *)

my $remote_path_quoted = quotemeta $remote_path;

but again, you have to inspect whether this is suitable in each case.

NOTE   If any dynamically generated variables (computed, come from user...) are interpolated in these commands they need be validated, with things carefully escaped and quoted.

Now your pipeline should work (it does in my simulated tests)

my $cmd = "ssh $remote_host grep -a $text_quoted $remote_path_quoted"
    . q( | awk F ' ' '{print $4}' | sort -nr | uniq | sort -nr 2>/dev/null);

This can be broken up into sensible components in their own variables, etc, but I really don't recommend such hand-patched solutions.

I suggest to only use the first part (ssh + grep) and do the rest in Perl though, as in the main part of the answer. And then to have those modules installed and switch to them.

No major computing tool works without (the many) libraries, and every production install contains a lot of "additional" stuff. As need for more libraries arises they are installed. Why would it have to be different with Perl? Yes, you can do it with builtins only but that may be much harder.


  One good reason would be so to use the system sort when files are huge, since it doesn't have to load the whole file at once, and for its speed. However, in this pipeline it is fed data over the pipe and invoked repeatedly so these advantages don't apply.

Sign up to request clarification or add additional context in comments.

10 Comments

@Developer Edited and added. Let me know if more/other is needed
With respect to the awk command, shell_quote('awk', '-F', ' ', '{print $4}') appears to Do The Right Thing.
I don;t have these modules installed which you are using and i am unable to install them because of permission
@CharlesDuffy Yes, thank you, edited. It makes sense (after the fact) -- the -F needs precisely a space, which does get escaped.
@Developer Welcome :) Let me know if questions come up.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.