Comments on: processing large xml data files http://surgeworks.com/blog/processing-large-xml-data-files iPhone and iPad App Design and Development Tue, 17 Apr 2012 21:07:40 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Lolke Dijkstra http://surgeworks.com/blog/processing-large-xml-data-files#comment-43 Lolke Dijkstra Tue, 12 Oct 2010 17:29:57 +0000 http://surgeworks.com/blog/?p=681#comment-43 Hi Brad, I came across your site when searching for XML and large datasets. I agree with you that SAX makes a lot of sense, however also StAX may be considered. I have worked in many projects as a software engineer (mostly 90's) and architect (mostly last 10 years). One of the projects that I came across was processing extremely large XML datasets (containing 1,000,000 + financial transactions) and entered a team where development had been outsourced. The guys had come up with a 'solution' using JaXB. Of course that could not work. At least that is if you were to process all data into memory and work from there (which was exactly what they had been doing). I estimated the max dataset to be around 120,000 depending on available memory. In practice it was much worse of course (only about 30,000 max). So, I recommended a different solution based on SAX or StAX (actually I had done that already). As a spin off of that project I decided to further develop that solution into a MDE based approach. Based on the specification (XSD) the code generator would generate the XML parser, which could send events (logical) to the listening processor. This is what I did and it worked very well. You can find the information here: http://dijkstra-ict.nl/documents/XMLParserTechnologyForProcessingHugeXMLfiles.pdf. If this is of interest to anyone here, drop me a note. : lolke.dijkstra@dijkstra-ict.com Kind regards, Lolke Dijkstra Hi Brad,

I came across your site when searching for XML and large datasets.

I agree with you that SAX makes a lot of sense, however also StAX may be considered.

I have worked in many projects as a software engineer (mostly 90′s) and architect (mostly last 10 years). One of the projects that I came across was processing extremely large XML datasets (containing 1,000,000 + financial transactions) and entered a team where development had been outsourced. The guys had come up with a ‘solution’ using JaXB. Of course that could not work. At least that is if you were to process all data into memory and work from there (which was exactly what they had been doing). I estimated the max dataset to be around 120,000 depending on available memory. In practice it was much worse of course (only about 30,000 max). So, I recommended a different solution based on SAX or StAX (actually I had done that already). As a spin off of that project I decided to further develop that solution into a MDE based approach. Based on the specification (XSD) the code generator would generate the XML parser, which could send events (logical) to the listening processor. This is what I did and it worked very well. You can find the information here: http://dijkstra-ict.nl/documents/XMLParserTechnologyForProcessingHugeXMLfiles.pdf.

If this is of interest to anyone here, drop me a note.
: lolke.dijkstra@dijkstra-ict.com

Kind regards,
Lolke Dijkstra

]]> By: Ruby/Rails: Parse large XMLs (SAX parsers, Pull parsers) + example of Pull parser « Rails, Web 2.0, Data Modeling http://surgeworks.com/blog/processing-large-xml-data-files#comment-42 Ruby/Rails: Parse large XMLs (SAX parsers, Pull parsers) + example of Pull parser « Rails, Web 2.0, Data Modeling Fri, 21 May 2010 20:27:32 +0000 http://surgeworks.com/blog/?p=681#comment-42 [...] Processing large XML files (SAX example) [...] [...] Processing large XML files (SAX example) [...]

]]> By: Ruby/Rails: Parse large XMLs (SAX parsers, Pull parsers) – example of Pull parser « Rails, Web 2.0, Data Modeling http://surgeworks.com/blog/processing-large-xml-data-files#comment-41 Ruby/Rails: Parse large XMLs (SAX parsers, Pull parsers) – example of Pull parser « Rails, Web 2.0, Data Modeling Fri, 21 May 2010 20:24:39 +0000 http://surgeworks.com/blog/?p=681#comment-41 [...] Processing large XML files (SAX example) [...] [...] Processing large XML files (SAX example) [...]

]]> By: Bill Conniff http://surgeworks.com/blog/processing-large-xml-data-files#comment-40 Bill Conniff Sun, 14 Feb 2010 02:25:02 +0000 http://surgeworks.com/blog/?p=681#comment-40 The problem I have with such parsers is they read forward-only and do not cache. Xponent developed a caching parser, allowing one to do most anything, in one pass. Is is currently in public open beta test. The problem I have with such parsers is they read forward-only and do not cache. Xponent developed a caching parser, allowing one to do most anything, in one pass. Is is currently in public open beta test.

]]> By: anon_anon http://surgeworks.com/blog/processing-large-xml-data-files#comment-39 anon_anon Mon, 17 Aug 2009 19:32:51 +0000 http://surgeworks.com/blog/?p=681#comment-39 You might want to VTD-XML (http://vtd-xml.sf.net) for processing large or huge XML documents... it has two APis, the standard version processes XML up to 2GB in size, the extended version allows documents up to 256 GB in size... XPath 1.0 is built-in... it has tons other features You might want to VTD-XML (http://vtd-xml.sf.net) for processing large or huge XML documents… it has two APis, the standard version processes XML up to 2GB in size, the extended version allows documents up to 256 GB in size… XPath 1.0 is built-in… it has tons other features

]]>