BaseX-Talk

basex-talk@mailman.uni-konstanz.de

1 participants
5252 discussions

Re: [basex-talk] profiling function execution order
by Christian Grün 29 Aug '12

29 Aug '12

> plz find the sample code below. the particular URI will create a simple new > default xml under 'example' database. This was working fine in the previous > versions, but with the latest release(BaseX74-20120829.045851.zip), its just > hanging and never producing any response. Is there anywhere I need to do > configuration? I’m currently revising the RESTXQ locking mechanism. Do you work with a single BaseX instance, or does "10.18.48.180" refer to another machine? In the first case, your query might not be supported anymore in future versions, because it is conceptually dangerous to call an updating query from a reading query in the same instance. The reason is that your first query could reference database contents that are modified by the send-request() call, which would make them invalid in your query. – The preferred solution here is to use XQuery Update for all your updating operations. If you want to create new databases, we will soon offer a db:create() command. In the second case (if you work with two BaseX instances), I’m not sure what could be the reason. Christian

1 0

profiling function execution order
by Seenivasan Gunabalan 28 Aug '12

28 Aug '12

Dear all, We have an question regarding the prof:current-ns() / prof:current-ms() in basex7.4 beta. our need is to calculate the execution time of the calling function. consider the example below, let $o1 := prof:current-ns() let $resp := http:send-request(<http:request method="GET" href="http://starling.us/cgi-bin/gus_atom_xsl.pl?atom=starling_us_atom.xml!sort_o…" status-only = "false" username = "admin" password = "admin" send-authorization="true"> </http:request>) let $o2 := prof:current-ns() let $v := (validate:xsd($resp[2],"../http/hdatatest/xsd/atom.xsd")) let $o3 := prof:current-ns() return ($o2 - $o1, $o3 - $o2, $v) After spending some time in trying the understand why the timings were not coherent ( validation time is almost 0 when compared to the total time), we noticed that the query plan was modified in the following way, Query: let $o1 := prof:current-ns() let $resp := http:send-request(<http:request method="GET" href="http://starling.us/cgi-bin/gus_atom_xsl.pl?atom=starling_us_atom.xml!sort_o…" status-only = "false" username = "admin" password = "admin" send-authorization="true"> </http:request>) let $o2 := prof:current-ns() let $v := (validate:xsd($resp[2],"../http/hdatatest/xsd/atom.xsd")) let $o3 := prof:current-ns() return ($o2 - $o1, $o3 - $o2, $v) Compiling: - moving for/let clauses Result: let $o1 := prof:current-ns() let $resp := http:send-request(element { Q{http://expath.org/ns/http-client}request } { attribute { method } { "GET" }, attribute { href } { "http://starling.us/cgi-bin/gus_atom_xsl.pl?atom=starling_us_atom.xml!sort_o…" }, attribute { status-only } { "false" }, attribute { username } { "admin" }, attribute { password } { "admin" }, attribute { send-authorization } { "true" } }) let $v := validate:xsd(($resp)[2], "../http/hdatatest/xsd/atom.xsd") let $o2 := prof:current-ns() let $o3 := prof:current-ns() return ($o2 - $o1, $o3 - $o2, $v) Timing: - Parsing: 0.56 ms - Compiling: 0.28 ms - Evaluating: 3929.59 ms - Printing: 0.04 ms - Total Time: 3930.5 ms Result: - Hit(s): 2 Items - Updated: 0 Items - Printed: 15 Bytes Query plan: <QueryPlan> <FLWR> <Let var="$o1"> <FNProf name="current-ns()"/> </Let> <Let var="$resp"> <FNHttp name="send-request(request[,href,[bodies]])"> <CElem> <QNm value="http:request" type="xs:QName"/> <CAttr> <QNm value="method" type="xs:QName"/> <Str value="GET" type="xs:string"/> </CAttr> <CAttr> <QNm value="href" type="xs:QName"/> <Str value="http://starling.us/cgi-bin/gus_atom_xsl.pl?atom=starling_us_atom.xml!sort_o…" type="xs:string"/> </CAttr> <CAttr> <QNm value="status-only" type="xs:QName"/> <Str value="false" type="xs:string"/> </CAttr> <CAttr> <QNm value="username" type="xs:QName"/> <Str value="admin" type="xs:string"/> </CAttr> <CAttr> <QNm value="password" type="xs:QName"/> <Str value="admin" type="xs:string"/> </CAttr> <CAttr> <QNm value="send-authorization" type="xs:QName"/> <Str value="true" type="xs:string"/> </CAttr> </CElem> </FNHttp> </Let> <Let var="$v"> <FNValidate name="xsd(input[,schema])"> <IterPosFilter> <VarRef> <Var name="$resp" id="1"/> </VarRef> <Pos min="2" max="2"/> </IterPosFilter> <Str value="../http/hdatatest/xsd/atom.xsd" type="xs:string"/> </FNValidate> </Let> <Let var="$o2"> <FNProf name="current-ns()"/> </Let> <Let var="$o3"> <FNProf name="current-ns()"/> </Let> <Return> <List> <Arith op="-"> <VarRef> <Var name="$o2" id="2"/> </VarRef> <VarRef> <Var name="$o1" id="0"/> </VarRef> </Arith> <Arith op="-"> <VarRef> <Var name="$o3" id="4"/> </VarRef> <VarRef> <Var name="$o2" id="2"/> </VarRef> </Arith> <VarRef> <Var name="$v" id="3"/> </VarRef> </List> </Return> </FLWR> </QueryPlan> and then after modifying the validation line into the following, we noticed that the plan was not modified and the profile timings are as expected. let $v := (validate:xsd($resp[2],"../http/hdatatest/xsd/atom.xsd"),$o2) Is it possible to skip the profiling functions from query optimization process in order to avoid those kind of issues?. Thanks and kind regards, Seenivasan & Marco

2 2

goat whole milk powder
by Ms. Chocolate (Vitacore Co., Ltd.) 27 Aug '12

27 Aug '12

Ms. Chocolate (Vitacore Co., Ltd.) Dear friend, We are glad to be partner of ‘Meiling’ brand goat milk. Goat whole milk powder FOB $6100/MT, 20KG,25KG per bag, 16MT/20’FCL. Thanks for attention. Best wishes, Ms. Chocolate Vitacore Co., Ltd. Hebei Branch ADD: No.9,Unit 2,Building 17, Fengnanfu, Lubei Street, Tangshan, Hebei, China Tel: 0086 - 131 8018 4008 E-mail: chocolate(a)vitacore.cn Http://www.vitacore.cn

1 0

CPU bottleneck?
by Gilles CARRY 27 Aug '12

27 Aug '12

Hi, I'm currently setting up a prototype to store test log results in a XML DB and let users do reports with XQUERY. I'm experiencing performance issues and am trying to identify where the bottleneck is. The status: - I use a single database. - The computer is dedicated to the DB server. (all clients are on distant machines) - Each applications insert bursts of 100-400 xml nodes in each insert query. Between inserts there is a random delay. (seconds to minutes) - UPDINDEX=true - Autoflush is on - The delays start being unacceptable when CPU average reaches ~40% (on a dual core which means ~80% of a single CPU) - Disk i/o seems fine: iostat says write kB_wrtn/s is about 10% of max iorate supported by disk. (I also tried with an SSD with no better results) - After some trials, it seems the more nodes in the DB the more delays to insert. The delays seem proportional to the number of nodes in the DB. - The delays are also proportional to the number of inserter client application (it looks like the transaction monitor is responsible for this) So I suppose the bottleneck is the CPU. Is there a way to check where in Basex server the bottleneck is? Which functions crunch up all cpu? How can I be sure basex uses both CPU? Is there any chance to have multiple simultaneous write access in future versions? Thank-you, Best regards, Gilles. By the way, the output of 'info database': [BaseX]: info database Database Properties Name: logs2 Size: 202 GB Nodes: 81534781 Documents: 1 Binaries: 0 Timestamp: 24.08.2012 15:07:11 Resource Properties Timestamp: 24.08.2012 15:07:10 Encoding: UTF-8 Whitespace Chopping: ON Indexes Up-to-date: false Text Index: ON Attribute Index: ON Full-Text Index: OFF

2 1

I/O Read bytes information on Windows TaskManager
by Fabrice Etanchaud 24 Aug '12

24 Aug '12

Dear all at BaseX, I'm working with a 30 Go database, containing nearly 10 Millions records. When I iterate on all these record, and get only a few informations on each, I can see in the Windows TaskManager that I/O Read bytes is nearly the database's size... but I don't read all the informations, only a few elements. Do you think this is a 'bug' in the manager, or is BaseX really reading all the data ? Maybe seeking in a file is just considered like reading in that file ? best regards, Fabrice ETANCHAUD Senior Software Engineer edital Berkenlaan 1 B-1831 Brussels, Belgium <skype:fabrice.etanchaud?call> +32 2 716 32 32 general +32 2 716 32 20 fax fetanchaud(a)edital.com corsearch.com <http://www.corsearch.com/> | edital.com <http://www.edital.com/> <outbind://899-00000000E439C83B50BF17409D8C27B7AE0425AF07005AA868536216F 543BAB9422EAEA5338B00000011020900005AA868536216F543BAB9422EAEA5338B00000 13AA36B0000/cid:image001.gif@01CC0F02.2C91B6C0> Confidentiality Notice: This email and its attachments (if any) contain confidential information of the sender. The information is intended only for the use by the direct addressees of the original sender of this email. If you are not an intended recipient of the original sender (or responsible for delivering the message to such person), you are hereby notified that any review, disclosure, copying, distribution or the taking of any action in reliance of the contents of and attachments to this email is strictly prohibited. If you have received this email in error, please immediately notify the sender at the address shown herein and permanently delete any copies of this email (digital or paper) in your possession.

2 1

XQJ: "Error: [BXDB0007] Database 'factbook' is opened by another process" on server is not propagated to Java code?
by Arjan van Bentem 24 Aug '12

24 Aug '12

Hi all, Maybe this is related to http://xqj.net/basex/ instead of BaseX itself, but as I don't know allow me to ask here first. Whenever I have somehow locked a BaseX database, like by using the standalone command line utility or the GUI, then I get something like the following in the BaseX log files: QUERY(1) for $c in collection('factbook')//country[name='Albania']return( replace value of node $c/@population with xs:integer($c/@population + 1)) OK 39.92 ms QUERY(1) OK 0.62 ms FULL(1) Error: [BXDB0007] Database 'factbook' is opened by another process. CLOSE(1) OK 39.68 ms Good. However, I don't see that error in my Java code. The Java code just continues like all was fine. Can I somehow make the Java code fail when this happens? See example below, and the same thing in the attachment. Thanks, Arjan. private static final String DRIVER = "net.xqj.basex.BaseXXQDataSource"; private static final String SELECT = "xs:long(collection('factbook')//country[name='Albania']/(a)population[1]/data(.))"; private static final String UPDATE = "for $c in collection('factbook')//country[name='Albania']" + "return(" + " replace value of node $c/@population with xs:integer($c/@population + 1)" + ")"; @Test public void testUpdate() { try { XQDataSource xqd = (XQDataSource) Class.forName(DRIVER) .newInstance(); XQConnection xqc = xqd.getConnection("admin", "admin"); XQExpression xqe = xqc.createExpression(); XQSequence xqs = xqe.executeQuery(SELECT); xqs.next(); long before = xqs.getLong(); System.out.println("Initial value: " + before); System.out.println("Updating..."); xqs = xqe.executeQuery(UPDATE); xqs = xqe.executeQuery(SELECT); xqs.next(); long after = xqs.getLong(); System.out.println("Resulting value: " + after); xqc.close(); Assert.assertEquals("Update should have succeeded or failed", before + 1, after); } catch (Exception e) { e.printStackTrace(); } }

3 5

how to speed up //myTag requests ?
by Fabrice Etanchaud 23 Aug '12

23 Aug '12

dear all at Basex, In a /a/b/c example schema, Is there any index that could speed up //c request ? best regards, Fabrice ETANCHAUD Senior Software Engineer edital Berkenlaan 1 B-1831 Brussels, Belgium <skype:fabrice.etanchaud?call> +32 2 716 32 32 general +32 2 716 32 20 fax fetanchaud(a)edital.com corsearch.com <http://www.corsearch.com/> | edital.com <http://www.edital.com/> <outbind://899-00000000E439C83B50BF17409D8C27B7AE0425AF07005AA868536216F 543BAB9422EAEA5338B00000011020900005AA868536216F543BAB9422EAEA5338B00000 13AA36B0000/cid:image001.gif@01CC0F02.2C91B6C0> Confidentiality Notice: This email and its attachments (if any) contain confidential information of the sender. The information is intended only for the use by the direct addressees of the original sender of this email. If you are not an intended recipient of the original sender (or responsible for delivering the message to such person), you are hereby notified that any review, disclosure, copying, distribution or the taking of any action in reliance of the contents of and attachments to this email is strictly prohibited. If you have received this email in error, please immediately notify the sender at the address shown herein and permanently delete any copies of this email (digital or paper) in your possession.

2 1

Index auto update
by Gilles CARRY 22 Aug '12

22 Aug '12

Hello, I'm a little bit confused with indexes auto update mechanism. Having created a DB with UPDINDEX=true, and after inserting nodes, 'info database' shows : Indexes Up-to-date: false Text Index: ON Attribute Index: ON Full-Text Index: OFF whereas UPDINDEX=true. After issuing 'optimize', Up-to-date come back to true and FT index to ON. Does this mean index are not automatically updated if 'optimize' is not run? I use BaseX73-20120528.232116.jar Is there any comprehensive documentation about index update mechanism? Thank-you, Gilles.

2 1

Basex editing performance test
by Yoann Maingon 21 Aug '12

21 Aug '12

Hi, Continuing on my recent question about using multiple databases, we've been running some performance test on basex. I don't have the detail of the computer that hosted these test but they were all done on the same computer. Test was to edit a node's value from a php script in a database a million time, the database being choosen randomly in X amount of databases created. (this simulates multiple users accessing our app) *2 DB* Total time : 1517.95 seconds for 1000000 iterations with 2 DB Min time : 1.17 ms with database 1 Max time : 704.72 ms with database 0 Mean time : 1.52 ms *10 000 DB* Total time : 1515.75 seconds for 1000000 iterations with 10 000 DB Min time : 1.21 ms with database 8879 Max time : 645.16 ms with database 3822 Mean time : 1.52 ms *20 000 DB* Total time : 1680.29 seconds for 1000000 iterations with 20 000 DB Min time : 1.18 ms with database 3749 Max time : 285.49 ms with database 6518 Mean time : 1.68 ms *40 000 DB* Total time : 1813.53 seconds for 1000000 iterations with 40 000 DB Min time : 1.04 ms with database 786 Max time : 212.2 ms with database 6949 Mean time : 1.81 ms *80 000 db - test 1* Total time : 24728.94 seconds for 1000000 iterations with 80 000 DB Min time : 1.16 ms with database 25693 Max time : 2433.44 ms with database 22021 Mean time : 24.73 ms *80 000 db - test 2* Total time : 18661.74 seconds for 1000000 iterations with 80 000 DB Min time : 1.68 ms with database 5979 Max time : 1936.4 ms with database 30239 Mean time : 18.66 ms We can just see that there is an important difference from 40k to 80k databases. We haven't checked other mean method to see if this was due to a few edit actions. Does any one tried to have many databases and at some point reached a certain limit? In oder to do server sizing what is key for these actions? processor?ram? Thanks for your help! -- Yoann Maingon mydatalinx 0664324966

2 1

large number of xml files
by sateesh 21 Aug '12

21 Aug '12

Hi Basex, I have a requirement of querying on large number of xml files some where around 10,000 xml files , I have written the query and while executing the query it is taking huge amount of memory and time some where around 700mb memory and time around 4 -5 minutes .Is there a way to execute the query with less memory and with in short time. Thanks & Regards Sateesh.A

2 9

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk