Hi, baseX team. I have some problems of loading xml files into the database of BaseX. I am comparing the time to load xml files into database, there are 2 ways to load: Loading all the folder into the database and Loading each file into the database. The time to load all the folder and the time to load each file have a big difference. The time to load all the folder into the datebase is 84ms, whereas the time to load each file into the database is 912 ms. I have written the 2 ways to load in C#. The following code is to load all the folder into the datebase: private void CreateDBAndLoadDirectory(string directoryPath) { this.contextDirectory = new Context(); this.serverDirectory = new org.basex.BaseXServer(this.contextDirectory); Set.execute("createfilter", "*" + this.fileExtension, this.contextDirectory); Set.execute("CACHEQUERY", true, this.contextDirectory); Set.execute("SERIALIZE", false, this.contextDirectory); Set.execute("CHOP", true, this.contextDirectory); Set.execute("DBPATH", this.dbPath, this.contextDirectory); new CreateDB(this.dbDirectory, directoryPath).execute(this.contextDirectory); } And the following code is to load each file into the database: private void CreateDBAndLoadFiles(string directoryPath) { this.contextFile = new Context(); this.serverFile = new org.basex.BaseXServer(this.contextFile); Set.execute("CACHEQUERY", true, this.contextFile); Set.execute("SERIALIZE", false, this.contextFile); Set.execute("CHOP", true, this.contextFile); Set.execute("DBPATH", this.dbPath, this.contextFile); new CreateDB(this.dbFile).execute(this.contextFile); string[] fileList = Directory.GetFiles(directoryPath, "*" + this.fileExtension,SearchOption.AllDirectories); foreach (string fileName in fileList) { string path = fileName.Substring(this.otxPath.Length + 1); new Add(path, fileName).execute(this.contextFile); new Optimize().execute(this.contextFile); } } I don't know if I have made mistakes in these codes or not? I want to ask about the fastest way to load each file into the database.
Dear Anh,
bulk operations (your first solution) will always be faster than multiple single updates. However, you can speed up single updates by temporarily setting the AUTOFLUSH option to true [1].
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Options#AUTOFLUSH ___________________________
On Thu, Feb 23, 2012 at 10:11 AM, Anh Thuấn thuan.ly@emotive.de wrote:
Hi, baseX team. I have some problems of loading xml files into the database of BaseX. I am comparing the time to load xml files into database, there are 2 ways to load: Loading all the folder into the database and Loading each file into the database. The time to load all the folder and the time to load each file have a big difference. The time to load all the folder into the datebase is 84ms, whereas the time to load each file into the database is 912 ms. I have written the 2 ways to load in C#. The following code is to load all the folder into the datebase: private void CreateDBAndLoadDirectory(string directoryPath) { this.contextDirectory = new Context(); this.serverDirectory = new org.basex.BaseXServer(this.contextDirectory); Set.execute("createfilter", "*" + this.fileExtension, this.contextDirectory); Set.execute("CACHEQUERY", true, this.contextDirectory); Set.execute("SERIALIZE", false, this.contextDirectory); Set.execute("CHOP", true, this.contextDirectory); Set.execute("DBPATH", this.dbPath, this.contextDirectory); new CreateDB(this.dbDirectory, directoryPath).execute(this.contextDirectory); } And the following code is to load each file into the database: private void CreateDBAndLoadFiles(string directoryPath) { this.contextFile = new Context(); this.serverFile = new org.basex.BaseXServer(this.contextFile); Set.execute("CACHEQUERY", true, this.contextFile); Set.execute("SERIALIZE", false, this.contextFile); Set.execute("CHOP", true, this.contextFile); Set.execute("DBPATH", this.dbPath, this.contextFile); new CreateDB(this.dbFile).execute(this.contextFile); string[] fileList = Directory.GetFiles(directoryPath, "*" + this.fileExtension,SearchOption.AllDirectories); foreach (string fileName in fileList) { string path = fileName.Substring(this.otxPath.Length + 1); new Add(path, fileName).execute(this.contextFile); new Optimize().execute(this.contextFile); } } I don't know if I have made mistakes in these codes or not? I want to ask about the fastest way to load each file into the database.
basex-talk@mailman.uni-konstanz.de