Hi Hondros,
Processing hundreds of thousands of zips, using db:add to to append small XML fragments from each into a single DB, I notice that the process becomes successively slower. Without having done any proper profiling, and aware that I might be looking in the wrong direction here, would it be reasonable to expect db:add to become steadily slower proportional to the size of the database being added to?
Yes, this may happen – but spontaneously I cannot tell you what might the reason for that in your setup. Feel free to send me some profiling output (e.g. using -Xhrunprof:cpu=samples).
Christian
Cheers,
C.
Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.