Hello all,
Processing hundreds of thousands of zips, using db:add to to append small XML fragments from each into a single DB, I notice that the process becomes successively slower. Without having done any proper profiling, and aware that I might be looking in the wrong direction here, would it be reasonable to expect db:add to become steadily slower proportional to the size of the database being added to?
Cheers, C.
________________________________
Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.
Hi Hondros,
Processing hundreds of thousands of zips, using db:add to to append small XML fragments from each into a single DB, I notice that the process becomes successively slower. Without having done any proper profiling, and aware that I might be looking in the wrong direction here, would it be reasonable to expect db:add to become steadily slower proportional to the size of the database being added to?
Yes, this may happen – but spontaneously I cannot tell you what might the reason for that in your setup. Feel free to send me some profiling output (e.g. using -Xhrunprof:cpu=samples).
Christian
Cheers,
C.
Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.
Thanks Christian, but I've satisfied myself that the slowdown is due to the size of the DB being added to by timing a sample set against an empty database. No big deal, and I can easily work around it.
Cheers, Constantine
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: 06 January 2016 13:57 To: Hondros, Constantine (ELS-AMS) Cc: BaseX Subject: Re: [basex-talk] db:add speed proportional to DB size?
Hi Hondros,
Processing hundreds of thousands of zips, using db:add to to append small XML fragments from each into a single DB, I notice that the process becomes successively slower. Without having done any proper profiling, and aware that I might be looking in the wrong direction here, would it be reasonable to expect db:add to become steadily slower proportional to the size of the database being added to?
Yes, this may happen – but spontaneously I cannot tell you what might the reason for that in your setup. Feel free to send me some profiling output (e.g. using -Xhrunprof:cpu=samples).
Christian
Cheers,
C.
Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.
________________________________
Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.
basex-talk@mailman.uni-konstanz.de