how to remove duplicate nodes from xml file using perl

Question

I am creating one xml file from multiple, I need to remove duplicate nodes form output xml. I have script like this to generate new xml file

 #!/usr/bin/perl
 use warnings;
 use strict;
 use XML::LibXML;
 use Carp;
 use File::Find;
 use File::Spec::Functions qw( canonpath );
 use XML::LibXML::Reader;
 use Digest::MD5 'md5';

 if ( @ARGV == 0 ) {
     push @ARGV, "c:/main/sav ";
     warn "Using default path $ARGV[0]\n  Usage: $0  path ...\n";
 }

 open( my $allxml, '>', "combined.xml" )
     or die "can't open output xml file for writing: $!\n";
 print $allxml '<?xml version="1.0" encoding="UTF-8"?>',
  "\n<Datainfo xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\n";
 my %extract_md5;
 find(
      sub {
          return unless ( /(_str\.xml)$/ and -f );
          extract_information();
          return;
      },
      @ARGV
     );

 print $allxml "</Datainfo>\n";

 sub extract_information {
     my $path = $_;
     if ( my $reader = XML::LibXML::Reader->new( location => $path )) {
         while ( $reader->nextElement( 'Data' )) {
             my $elem = $reader->readOuterXml();
             my $md5 = md5( $elem );
             print $allxml $reader->readOuterXml() unless ( $extract_md5{$md5}++ );
         }

     }
     return;
 }

But from above script printing xml file like this

combined.xml:

<?xml version="1.0" encoding="UTF-8"?>
<Datainfo xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <data>
        <test>22</test>
        <info>sensor value</info>
        <sensor>
            <sensor value="23" temp="25"/>
        </sensor>
    </data>
    <data>
        <test>23</test>
        <info>sensor value</info>
        <sensor>
            <sensor value="24" temp="27"/>
        </sensor>
    </data>
    <data>
        <test>22</test>
        <info>sensor value</info>
        <sensor>
            <sensor value="22" temp="26"/>
        </sensor>
    </data>
</Datainfo>

In the above xml file I have data element test(22) is repeated in two times. I need to use test as the element to search in file if same test number is found what ever may be the information inside that node I need to delete that entire node information. I tried to do with md5 but it removing duplicate nodes from allxml files but now I need to search one specific element and delete entire node information if duplicate is occurred.please help me with this problem.
output like this

combined.xml:

<?xml version="1.0" encoding="UTF-8"?>
<Datainfo xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <data>
        <test>22</test>
        <info>sensor value</info>
        <sensor>
            <sensor value="23" temp="25"/>
        </sensor>
    </data>
    <data>
        <test>23</test>
        <info>sensor value</info>
        <sensor>
            <sensor value="24" temp="27"/>
        </sensor>
    </data>
</Datainfo>

Ross Attrill · Accepted Answer · 2013-08-20 22:29:05Z

1

I normally use XML::Simple for things like this.

XML::Simple stores your XML file in a hash/array structure. This would automatically eliminate the duplicate issue you're finding (depending how you configure it).

edited Aug 20, 2013 at 22:29

Ross Attrill

2,7321 gold badge28 silver badges32 bronze badges

answered Nov 22, 2011 at 13:45

David W.

107k40 gold badges224 silver badges349 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

biji Over a year ago

@David can you tel me how to configure to delete some information in xml file.please help me or provide some example.

DVK · Accepted Answer · 2011-11-22 13:38:40Z

0

You will have to do the duplicate checking by specifically checking <test> contents, instead of md5 of the entire node.

E.g. instead of my $md5 = md5( $elem ); and storing $md5 key in the hash, you need to extract the contents of <test> tag and store that.

I would prefer not to provide more details since you seem to be simply spamming SO as well as PerlMonks with requests to help you do your work and copying/pasting somewhat complicated code that you don't bother trying to understand how it works.

http://www.perlmonks.org/?node_id=939272

answered Nov 22, 2011 at 13:38

DVK

130k33 gold badges219 silver badges337 bronze badges

1 Comment

biji Over a year ago

Not like that I learned alot from so,actually I understand that md5 checks hole node and if any duplicates node and eliminates, Now I am extracting data node from all xml files , how to extract test tag in that and eliminate the entire node, please help me with this.

Collectives™ on Stack Overflow

how to remove duplicate nodes from xml file using perl

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related