2

I have hundreds of HTML files that are relatively simple in content and maintain a consistent format.

I need to convert them to tables, can I use a shell script to do this?

HTML source code

<html>
<head>
<title>Demo</title>
</head>
<body>
    <h1>Page Title</h1>
    <div class="row">
        <p class="text-1">Text 1</p>
        <p class="text-2">Text 2</p>
        <p class="text-3">Text 3</p>
        <p class="text-4">Text 4</p>
        <p class="text-5">Text 5</p>
        <p class="text-6">Text 6</p>
    </div>
    <div class="row">
        <p class="text-1">Text 1</p>
        <p class="text-2">Text 2</p>
        <p class="text-3">Text 3</p>
        <p class="text-4">Text 4</p>
        <p class="text-5">Text 5</p>
        <p class="text-6">Text 6</p>
    </div>
    <div class="row">
        <p class="text-1">Text 1</p>
        <p class="text-2">Text 2</p>
        <p class="text-3">Text 3</p>
        <p class="text-4">Text 4</p>
        <p class="text-5">Text 5</p>
        <p class="text-6">Text 6</p>
    </div>
</body>
</html>

Converted table source code

<table>
    <caption>Page Title</caption>
    <thead>
        <tr>
            <th>Hard Code</th>
            <th>Hard Code</th>
            <th>Hard Code</th>
            <th>Hard Code</th>
            <th>Hard Code</th>
            <th>Hard Code</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>Text 1</td>
            <td>Text 2</td>
            <td>Text 3</td>
            <td>Text 4</td>
            <td>Text 5</td>
            <td>Text 6</td>
        </tr>
        <tr>
            <td>Text 1</td>
            <td>Text 2</td>
            <td>Text 3</td>
            <td>Text 4</td>
            <td>Text 5</td>
            <td>Text 6</td>
        </tr>
        <tr>
            <td>Text 1</td>
            <td>Text 2</td>
            <td>Text 3</td>
            <td>Text 4</td>
            <td>Text 5</td>
            <td>Text 6</td>
        </tr>
    </tbody>
</table>

This is the mind map.

enter image description here

Before asking the question, I tried to look up information on the Internet and found that I could extract the HTML content using the pup tool, which is used as follows.

# Extracting page titles
cat demo.html | pup 'body > h1 text{}'

# Extracting paragraph text
cat demo.html | pup 'body > div.row > p.text-1 text{}'
cat demo.html | pup 'body > div.row > p.text-2 text{}'
cat demo.html | pup 'body > div.row > p.text-3 text{}'
cat demo.html | pup 'body > div.row > p.text-4 text{}'
cat demo.html | pup 'body > div.row > p.text-5 text{}'
cat demo.html | pup 'body > div.row > p.text-6 text{}'

In the next I ran into difficulty, I didn't know how to make it into a shell script. It involves shell loops and I spent a few days trying to figure it out, but without success.

Can you all help me? Thank you in advance!

Update

This is what I have tried to do. It has several problems.

  • It can only handle one <div class="row">...</div> data, which is the trickiest problem I've encountered (The problem is shown below). It involves the shell loop problem. enter image description here
  • It can only convert one HTML file at a time, and ideally works in batches of hundreds of HTML files (exported to another directory and saved, file names remain consistent).
#!/usr/bin/env bash

# Extracts HTML content
page_title=$(cat demo.html | pup 'body > h1 text{}')
paragraph_text_a=$(cat demo.html | pup 'body > div.row > p.text-1 text{}')
paragraph_text_b=$(cat demo.html | pup 'body > div.row > p.text-2 text{}')
paragraph_text_c=$(cat demo.html | pup 'body > div.row > p.text-3 text{}')
paragraph_text_d=$(cat demo.html | pup 'body > div.row > p.text-4 text{}')
paragraph_text_e=$(cat demo.html | pup 'body > div.row > p.text-5 text{}')
paragraph_text_f=$(cat demo.html | pup 'body > div.row > p.text-6 text{}')

# Print the contents in a predetermined format
cat << EOF
<table>
    <caption>$page_title</caption>
    <thead>
        <tr>
            <th>Hard Code</th>
            <th>Hard Code</th>
            <th>Hard Code</th>
            <th>Hard Code</th>
            <th>Hard Code</th>
            <th>Hard Code</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>$paragraph_text_a</td>
            <td>$paragraph_text_b</td>
            <td>$paragraph_text_c</td>
            <td>$paragraph_text_d</td>
            <td>$paragraph_text_e</td>
            <td>$paragraph_text_f</td>
        </tr>
    </tbody>
</table>
EOF
Improve this question
4
  • 3
    I wouldn't attempt to do this with a shell-script unless you can find a tool that specifically parses HTML in some useful way. I would use whichever scripting language or programming language I was comfortable with and make use of it's HTML-parsing libraries. (In my case, Perl and Go would be good candidates but your mileage will vary). If your data is absolutely 100% uniform you could maybe use a tool like sed even though it is dangerous to do so. Commented Aug 6, 2020 at 9:53
  • @RedGrittyBrick Indeed, I saw the same point on the web. But I have no programming experience and it would become difficult to make minor adjustments to this. Similarly, shell scripts run on the server and do not require a runtime environment to be installed. Commented Aug 6, 2020 at 10:01
  • Although my script below works in most cases, if you really want to avoid edge-cases you'll have to install the HTML::Element perl module and use a script based on this. The code will be safer and more readable. I didn't use it because now you only need perl (which is most likely already installed on your system) Commented Aug 6, 2020 at 11:54
  • @Garo Thank you! Can it be implemented using a shell script? I've updated the question to add what I've tried to do. Commented Aug 7, 2020 at 0:59

2 Answers 2

Reset to default
2

The following should more or less do it, keep in mind that I :

  • Only wrote it, I didn't test it. Edit: Now i did test it, fixed some bugs, and it seems to work.
  • Am ignoring edgecases (multiple <h1>'s, <tbody> inside a tablefield and so on, ...)

Put it in "scriptname.pl", change the filenames on line 2 and 3 and run it with perl scriptname.pl

#!/usr/bin/perl
open my $ifh, "inputfilename.html";
open my $ofh, ">outputfilename.html";
while(<$ifh>) {
  if(/<h1>(.*)<\/h1>/) {
    my $header = << "END";
  <table>
    <caption>$1</caption>
    <thead>
        <tr>
            <th>Hard Code</th>
            <th>Hard Code</th>
            <th>Hard Code</th>
            <th>Hard Code</th>
            <th>Hard Code</th>
            <th>Hard Code</th>
        </tr>
    </thead>
    <tbody>
END
    print $ofh $header;
  } elsif(/<div class="row">/) {
    print $ofh "<tr>\n";
  } elsif(/<\/div>/) {
    print $ofh "</tr>\n";
  } elsif(/<p class=".*?">(.*)<\/p>/) {
    print $ofh "<td>$1</td>\n";
  } elsif(/<\/body>/) {
    print $ofh "</tbody>\n</table>\n</body>\n";
  } else {
    print $ofh $_;
  }
}
close $ofh;
close $ifh;
Improve this answer
1
  • Thank you! It tests valid. Is it possible to implement it using a shell script? Use pup to select the HTML content. This allows me to make some minor adjustments and be more flexible. Commented Aug 6, 2020 at 12:31
1

You are trying to extract cell by cell, which will make it harder for you to rebuilt the table.

Using simple bash and pup only:

#!/bin/bash

count=$(grep '<div ' demo.html | wc -l)
page_title=$(cat demo.html | pup 'body h1 text{}')

tbody() {
    for ((i=1;i<count+1;++i)); do
        IFS=, row=$(cat demo.html | pup "body div.row:nth-of-type($i) text{}" | grep '\S' | paste -s -d, -)
        printf "\t\t<tr>\n"
        printf '\t\t\t<td>%s</td>\n' $row
        printf "\t\t</tr>\n"
    done
}

cat <<EOF
<table>
    <caption>$page_title</caption>
    <thead>
        <tr>
            <th>Hard Coded</th>
            <th>Hard Coded</th>
            <th>Hard Coded</th>
            <th>Hard Coded</th>
            <th>Hard Coded</th>
            <th>Hard Coded</th>
        </tr>
    </thead>
    <tbody>
`tbody`
    </tbody>
</table>
EOF

Output

<table>
    <caption>Page Title</caption>
    <thead>
        <tr>
            <th>Hard Coded</th>
            <th>Hard Coded</th>
            <th>Hard Coded</th>
            <th>Hard Coded</th>
            <th>Hard Coded</th>
            <th>Hard Coded</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>Text 1</td>
            <td>Text 2</td>
            <td>Text 3</td>
            <td>Text 4</td>
            <td>Text 5</td>
            <td>Text 6</td>
        </tr>
        <tr>
            <td>Text 1</td>
            <td>Text 2</td>
            <td>Text 3</td>
            <td>Text 4</td>
            <td>Text 5</td>
            <td>Text 6</td>
        </tr>
        <tr>
            <td>Text 1</td>
            <td>Text 2</td>
            <td>Text 3</td>
            <td>Text 4</td>
            <td>Text 5</td>
            <td>Text 6</td>
        </tr>
    </tbody>
</table>

Explanation

The idea is extract data row by row, by iterating until the last row. This snippet will give you the rows count:

grep '<div ' demo.html | wc -l

Then by using nth-of-type(n) as the selector, you can grab the whole row instead of column. You need to pass it to grep '\S' to get rid of blank lines. Then by passing to paste -s -d, -, will produce comma separated result.

IFS=, row=$(cat demo.html | pup "body div.row:nth-of-type($i) text{}" | grep '\S' | paste -s -d, -)

The printf '\t\t\t<td>%s</td>\n' $row will be expanded to printf '\t\t\t<td>%s</td>\n' 'Text 1' 'Text 2' ... and it will wrap each of the args with <td>...</td>

You can totally drop the \t part, it will just print indented result.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.