2

I have a an html source file which i need to extract the links within them, number of links varies from file to file and links are formatted as such and are within single quote:

../xxx/yyy/ccc/bbbb/nameoffile.extension

I need to get the text between the single quote, replace the .. by http:// and output the result to a file.

Im a newbie and looking for a solution to automate this process in terminal.

its html sources files and the links are everywhere in the file, I need to get them one link per lines outputted in a file to pass to my existing xargs curl for download.

sample file would is almost like that :

<head>
<body>
<html>

blabla
</>
blibli afg fgfdg sdfg <b> blo blo href= '../xxx/yyy/ccc/bbbb/nameoffile1.extension' target blibli bloblo href= '../xxx/yyy/ccc/bbbb/nameoffile2.extension'  blibli

bloblo href= '../xxx/yyy/ccc/bbbb/nameoffile3.extension'

…

result looking for is a file containing this:

http://z.z.com/xxx/yyy/ccc/bbbb/nameoffile1.extension
http://z.z.com/xxx/yyy/ccc/bbbb/nameoffile2.extension
http://z.z.com/xxx/yyy/ccc/bbbb/nameoffile3.extension

can someone be kind enough to help me find a solution please.

source file as close as possible:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"><HTML>
    <HEAD>
    <TITLE>Inter num num - nil</TITLE>
    <link rel="stylesheet" type="text/css" href="style.css" />
    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
    </HEAD>
    <BODY><table width=1200 align=center class=tabForm><tr><td align=left width=largeur_2 valign=top><img src=Img/logo.gif><br /></td><td align=center valign=center width=largeur_6><h1><font color='#CB150A'>Test d'épreuve</font></h1></td><td align=right valign=top width=largeur_2 class=dataLabel>Reçu le 11/03/2018 à 17:49<br /></td></tr>
    <tr><td width=1200 colspan=3 align=center><b><font color='#CB150A' size=+1>Client : zzz - Référence : 232323  - Désignation : Fiche d'accueil </font></b></color></td></tr>

    </table><BR/><table width=1200 align=center class=tabForm><tr><td class=dataLabelBig width=1200>M numnum ,<BR/><BR/>
    Job citée ci-dessus.<BR/>
    ci-joints toutes les informations nécessaires.
    <BR/><BR/>
    Sandy Jan<BR/>
    test@test.com</font></td></tr></table><br /><table width=1200 align=center class=tabForm><tr><td colspan=2  width=1200 class=dataLabel>Documents nécessaires à votre réponse</td></tr><tr><td colspan=2 width=1200 class=dataLabel><u><b>Job :</b></u> Suivi Travaux - <u><b>Article :</b></u> 232323  - Fiche d'accueil</td></tr><tr><td colspan=2 width=1200 class=dataLabel><a href='../path/path/path/path/path.html' target=_blank><img src=Img/pdf.png border=0> Fiche.html</a></td></tr><tr><td colspan=2 width=1200 class=dataLabel><a href='../path/path/path/path/pathd%27accueil%20traitant-20160621163240.pdf' target=_blank><img src=Img/pdf.png border=0> text.pdf</a></td></tr><tr><td colspan=2 width=1200 class=dataLabel><a href='../path/path/path/path/pathla%20S%E9curit%E9%20%281%29.doc' target=_blank><img src=Img/pdf.png border=0> Fiched'accueil.doc</a></td></tr></table><br /><table width=1200 align=center class=tabForm><tr><td colspan=2 class=dataLabelRed width=1200 >Notre commentaire</td></tr></tr><td colspan=2 class=dataLabel>mise a jour - Attention<br />
Impression <br /><br /></td></tr></table><br /><table width=1200 align=center class=tabForm><form method=post name=formvolume action=?&dossier=111734&coo=135&auth=b182f10b82ba&key=2e7c69213b28d7de6655&action=submit&type=volume enctype=multipart/form-data ><tr><td width=1200 align=left colspan=2 class=dataLabel><h3><img src=Img/h3Arrow.gif border=0>&nbsp;Remise de job  :</h3><br /></td></tr><tr><td align=left valign=top width=120 class=dataLabelRed>Votre commentaire</td><td width=1080 align=left class=dataLabel><textarea cols=200 rows=5 name=comment ></textarea></td></tr><tr><td align=left width=120 class=dataLabelRed>Votre fichier</td><td width=1080 align=left><input type=file name=fichier size=82></td></tr><tr><td align=center colspan=2 width=1200><br /><input type=button class=button value="&nbsp;&nbsp;Remettre votre réponse&nbsp;&nbsp;"  onClick="javascript: var ok=confirm('Etes vous certain de vouloir effectuer cette action ?');if(ok==true){ document.formvolume.submit();}else {return false}" ></form></td></tr><table></table></br><table width=1200 align=center class=tabForm><form method=post name=formvolume_complement action=?&dossier=111734&coo=135&auth=b182f10b82ba&key=2e7c69213b28d7de6655&action=submit_complement&type=volume enctype=multipart/form-data ><tr><td width=1200 align=left colspan=2 class=dataLabel><h3><img src=Img/h3Arrow.gif border=0>&nbsp;Demande de complément, votre réponse  :</h3><br /></td></tr><tr><tr><td align=left valign=top width=120 class=dataLabelRed>Votre commentaire</td><td width=1080 align=left class=dataLabel><textarea cols=200 rows=5 name=comment ></textarea></td></tr><td align=left width=120 class=dataLabelRed>Votre fichier</td><td width=1080 align=left><input type=file name=fichier size=82></td></tr><tr><td align=center colspan=2 width=1200><br /><input type=button class=button value="&nbsp;  Remettre votre réponse &nbsp;"  onClick="javascript: var ok =confirm('Etes v ?');if(ok==true){ document.formvolume_complement.submit();}else {return false}" ></form></td></tr><table></table></BODY></HTML></BODY>
</HTML>
grimdex
  • 41

4 Answers4

4

Utilities like sed, awk etc. are not made for parsing structured data such as html. Hence a much more viable solution would be to use python to do the same.

Firstly, make sure BeautifulSoup is installed by:

sudo apt-get install python3 python3-bs4

Now create a new file (for instance test.py) and paste the short script I've written for this purpose:

#!/usr/bin/env python3
import sys
from bs4 import BeautifulSoup

DOMAIN = 'z.z.com/'

if  len(sys.argv)<2 or not sys.argv[1].endswith('.html'):
    print("Argument not provided or not .html file", file=sys.stderr)
    exit()

with open(sys.argv[1], 'r', encoding='latin-1') as f:
    webpage = f.read()

soup = BeautifulSoup(webpage, "lxml")
for a in soup.findAll('a', href=True):
    print(a['href'].replace("../","http://"+DOMAIN))

Python 2 version on request:

#!/usr/bin/env python2
import sys
from bs4 import BeautifulSoup

DOMAIN = 'z.z.com/'

if  len(sys.argv)<2 or not sys.argv[1].endswith('.html'):
    print >> sys.stderr, "Argument not provided or not .html file"
    exit()

with open(sys.argv[1], 'rb') as f:
    webpage = f.read().decode("latin-1")

soup = BeautifulSoup(webpage, "html.parser")
for a in soup.findAll('a', href=True):
    print(a['href'].replace("../","http://"+DOMAIN))

Modify the DOMAIN variable to match your actual domain, save this script in the current directory and run it as follows:

./test.py yourfile.html > outputfile

For reference, this is the output produced by the script on running it with the provided example in the question:

http://z.z.com/path/path/path/path/path.html
http://z.z.com/path/path/path/path/pathd%27accueil%20traitant-20160621163240.pdf
http://z.z.com/path/path/path/path/pathla%20S%E9curit%E9%20%281%29.doc
David Foerster
  • 36,890
  • 56
  • 97
  • 151
Amith KK
  • 13,547
1

Another Perl solution that uses a proper HTML parser is the following (say get-links.pl):

#!/usr/bin/env perl

use strict;
use warnings;
use File::Spec;
use WWW::Mechanize;

my $filename = shift or die "Must supply a *.html file\n";
my $absolute_filename = File::Spec->rel2abs($filename);

my $mech = WWW::Mechanize->new();
$mech->get( "file://$absolute_filename" );
my @links = $mech->links();
foreach my $link ( @links ) {
    my $new_link = $link->url;

    if ( $new_link =~ s(^\.\./)(http://z.z.com/) ) {
        print "$new_link\n";
    }
}

You may need to install the WWW::Mechanize module first because it is not a core module (meaning it isn't installed by default together with Perl). To do so, run

sudo apt install libwww-mechanize-perl

The script reads the given file, converts the filename to an absolute path (because we want to build a proper URI like file:///path/to/source.html).

After extracting the links (my @links = $mech->links();) it examines each link's URL and if it starts with ../ then that part is replaced by http://z.z.com/ and printed.

Usage:

./get-links.pl source.html

Output:

http://z.z.com/path/path/path/path/path.html
http://z.z.com/path/path/path/path/pathd%27accueil%20traitant-20160621163240.pdf
http://z.z.com/path/path/path/path/pathla%20S%E9curit%E9%20%281%29.doc

As @Amith KK already said in his answer: Parsing HTML (or XML) is best done with a proper parser because tools like sed and their kind may fail when there are other elements in the source that look like a link but aren't.

David Foerster
  • 36,890
  • 56
  • 97
  • 151
PerlDuck
  • 13,885
0

To extract data between single quotes from file test.html with replacing of two dots .. in URLs with http:// and to save the extracted data to file newfile.txt do:

cat test.html | sed -ne 's/^.*'\''\([^'\'']*\)'\''.*$/\1/p' | sed -e 's/\.\./http:\//g' > newfile.txt

Or try without sed:

cat test.html | grep -Eo "'[^'() ]+'" | tr -d \'\" | perl -pe 's/../http:\//' > newfile.txt

This works for the file sample added to question by author:

cat test.html | grep -Eo "'[^|'() ]+'" | grep -wE "('..)" | tr -d \'\" | perl -pe 's/../http:\/\/mysite.mydomain.com/' > newfile.txt
Bob
  • 2,563
0

Convert HTML to Text

As mentioned in comments you need to convert html to text format. For this there is a one-liner which should cover all the bases:

sed 's/&nbsp;/ /g; s/&amp;/\&/g; s/&lt;/\</g; s/&gt;/\>/g; s/&quot;/\"/g; s/#&#39;/\'"'"'/g; s/&ldquo;/\"/g; s/&rdquo;/\"/g;'

If you are converting 100s of thousands of lines, bash builtin commands are many times faster:

#-------------------------------------------------------------------------------
LineOut=""      # Make global
HTMLtoText () {
    LineOut=$1  # Parm 1= Input line
    # Replace external command: Line=$(sed 's/&amp;/\&/g; s/&lt;/\</g; 
    # s/&gt;/\>/g; s/&quot;/\"/g; s/&#39;/\'"'"'/g; s/&ldquo;/\"/g; 
    # s/&rdquo;/\"/g;' <<< "$Line") -- With faster builtin commands.
    LineOut="${LineOut//&nbsp;/ }"
    LineOut="${LineOut//&amp;/&}"
    LineOut="${LineOut//&lt;/<}"
    LineOut="${LineOut//&gt;/>}"
    LineOut="${LineOut//&quot;/'"'}"
    LineOut="${LineOut//&#39;/"'"}"
    LineOut="${LineOut//&ldquo;/'"'}" # TODO: ASCII/ISO for opening quote
    LineOut="${LineOut//&rdquo;/'"'}" # TODO: ASCII/ISO for closing quote
} # HTMLtoText ()

Check if file exists

To test if file exists use a derivative of this function:

function validate_url(){
  if [[ `wget -S --spider $1  2>&1 | grep 'HTTP/1.1 200 OK'` ]]; then echo "true"; fi
}

Putting it all together

A final script still needs to be written based on sample data derived from a valid webpage with valid filenames.

WinEunuuchs2Unix
  • 105,762