| Author |
Message |
John Bokma
*nix forums Guru
Joined: 23 Feb 2005
Posts: 1136
|
 Posted: Sat Feb 26, 2005 1:59 am Post subject:
Re: Compare huge XML Files
|
|
|
junnuthala wrote:
| Quote: | The bottleneck is not in the XML::Parser when I use the "Stream" option
which returns all the tags in XML format itself.
But when I use the "Tree" options and it is processing each tag I am
getting much delay.
I guess I have to use "Stream" option and do my own callback functions
for startTag, endTag, startDocument and endDocument.
anyone have any suggestions on what would be the fastest way ?
|
How much memory does the process use?
--
John Small Perl scripts: http://johnbokma.com/perl/
Perl programmer available: http://castleamber.com/
Happy Customers: http://castleamber.com/testimonials.html |
|
| Back to top |
|
 |
junnuthala
*nix forums beginner
Joined: 23 Feb 2005
Posts: 10
|
| Posted: Fri Feb 25, 2005 9:01 pm Post subject:
Re: Compare huge XML Files
|
|
|
The bottleneck is not in the XML::Parser when I use the "Stream" option
which returns all the tags in XML format itself.
But when I use the "Tree" options and it is processing each tag I am
getting much delay.
I guess I have to use "Stream" option and do my own callback functions
for startTag, endTag, startDocument and endDocument.
anyone have any suggestions on what would be the fastest way ? |
|
| Back to top |
|
|
John Bokma
*nix forums Guru
Joined: 23 Feb 2005
Posts: 1136
|
| Posted: Fri Feb 25, 2005 4:42 pm Post subject:
Re: Compare huge XML Files
|
|
|
junnuthala wrote:
| Quote: | I have a 6MB XML file, but it has more than 300,000 elements.
XML::Parser is taking almost 35 minutes to get the result as a tree.
|
That sounds awfully slow. I parse over half a MB in seconds. I don't build
a tree but the overhead of doing so (if I do it) is insignificant.
--
John Small Perl scripts: http://johnbokma.com/perl/
Perl programmer available: http://castleamber.com/
Happy Customers: http://castleamber.com/testimonials.html |
|
| Back to top |
|
|
junnuthala
*nix forums beginner
Joined: 23 Feb 2005
Posts: 10
|
| Posted: Fri Feb 25, 2005 12:22 am Post subject:
Re: Compare huge XML Files
|
|
|
I have a 6MB XML file, but it has more than 300,000 elements.
XML::Parser is taking almost 35 minutes to get the result as a tree. |
|
| Back to top |
|
|
junnuthala
*nix forums beginner
Joined: 23 Feb 2005
Posts: 10
|
| Posted: Fri Feb 25, 2005 12:07 am Post subject:
Re: Compare huge XML Files
|
|
|
Thomas Malt wrote:
| Quote: | "junnuthala" <junnuthula@yahoo.com> writes:
John Bokma wrote:
junnuthala wrote:
Hello,
Can someone please suggest me a Perl module for comparing huge
XML
Files.
How do you want to compare them?
Parse and read the XML elements, attributes text into a tree or a
hash
and then compare.
I tried using Semanticdiff, but it is taking a lot of time to read
the
XML file into Hash.
That will take a lot of time no matter what you do. But define
"huge". And define "a lot of time"  .
|
I have a XML file of sixe 6MB, but it has more than 300,000 elements.
| Quote: | If you want efficiency more than anything else then XML::Parser is
still the fastest. Or at least it was the last time I checked.
|
XML::Parser is taking almost 35 minutes to get the result as a tree.
| Quote: | Implementing handlers to put attributes and CDATA into an HASH is
really straight forward, but if your files are in the 100MB area
that could still take several minutes depending on your hardware.
Thomas
--
: Thomas Malt.: tm@linpro.no ...: http://www.malt.no/ ...:
+4797748504 :
: Linpro AS...: info@linpro.no .: http://www.linpro.no/ .:
+4722871180 :
: :... >> Ledende på Linux i Norge >> Best på alt i
verden : |
|
|
| Back to top |
|
|
Thomas Malt
*nix forums beginner
Joined: 23 Feb 2005
Posts: 2
|
| Posted: Wed Feb 23, 2005 10:25 pm Post subject:
Re: Compare huge XML Files
|
|
|
"junnuthala" <junnuthula@yahoo.com> writes:
| Quote: | John Bokma wrote:
junnuthala wrote:
Hello,
Can someone please suggest me a Perl module for comparing huge XML
Files.
How do you want to compare them?
Parse and read the XML elements, attributes text into a tree or a hash
and then compare.
I tried using Semanticdiff, but it is taking a lot of time to read the
XML file into Hash.
|
That will take a lot of time no matter what you do. But define
"huge". And define "a lot of time" .
If you want efficiency more than anything else then XML::Parser is
still the fastest. Or at least it was the last time I checked.
Implementing handlers to put attributes and CDATA into an HASH is
really straight forward, but if your files are in the 100MB area
that could still take several minutes depending on your hardware.
Thomas
--
: Thomas Malt.: tm@linpro.no ...: http://www.malt.no/ ...: +4797748504 :
: Linpro AS...: info@linpro.no .: http://www.linpro.no/ .: +4722871180 :
: :... >> Ledende på Linux i Norge >> Best på alt i verden : |
|
| Back to top |
|
|
junnuthala
*nix forums beginner
Joined: 23 Feb 2005
Posts: 10
|
| Posted: Wed Feb 23, 2005 7:44 pm Post subject:
Re: Compare huge XML Files
|
|
|
John Bokma wrote:
| Quote: | junnuthala wrote:
Hello,
Can someone please suggest me a Perl module for comparing huge XML
Files.
How do you want to compare them?
|
Parse and read the XML elements, attributes text into a tree or a hash
and then compare.
I tried using Semanticdiff, but it is taking a lot of time to read the
XML file into Hash.
|
|
| Back to top |
|
|
John Bokma
*nix forums Guru
Joined: 23 Feb 2005
Posts: 1136
|
|
| Back to top |
|
|
junnuthala
*nix forums beginner
Joined: 23 Feb 2005
Posts: 10
|
| Posted: Wed Feb 23, 2005 8:01 am Post subject:
Compare huge XML Files
|
|
|
Hello,
Can someone please suggest me a Perl module for comparing huge XML
Files.
I tried XML::SemanticDiff, but it is taking a lots and lots of time to
load the XML File nodes, elements and attributes to the Hash.
Any suggestions would be really appreciated.
Thank you
-Venkat |
|
| Back to top |
|
|
Google
|
|
| Back to top |
|
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
Forum index
»
Programming
»
Perl
»
modules
