BaseX-Talk

basex-talk@mailman.uni-konstanz.de

6 participants
5269 discussions

Re: [basex-talk] about performance
by Christian Grün 20 Jul '11

20 Jul '11

Serena, this is better than nothing, but I still need more info (at least an input document that allows me to run your query out of the box) to get a clue what might go on. Christian ___________________________ 2011/7/20 Lusha Wang <lwang40(a)syr.edu>: > hI Christian > I feel like this is not about my code. Because most of the time I can get results correctly. But I still pasted my code in the following： > > <html> > <body> > <?php > > include("BaseXClient.php"); > $start=$_GET['start']; > $end = $_GET['end']; > //$type=$_GET['type']; > > > if($start!=null && $end!=null) > { > echo "all entites that are within this time period:".$start."--".$end."<br>"; > > > $cmd='for $Document in doc("newdf")//Document[xs:date(./Date/text()) <=xs:date("'.$end.'") and xs:date(./Date/text()) >=xs:date("'.$start.'")] return <Document>{ for $d in $Document/HasDomain/@domain/data() where $Document/HasDomain return <Domain>{attribute{"id"}{$d}, attribute{"data"}{doc("newdf")//Domain[@id/data()=$d]/Name/text()}}</Domain>, for $u in $Document/HasURL/@url/data() where $Document/HasURL return <URL>{attribute{"id"}{$u}, attribute{"data"}{doc("newdf")//URL[@id/data()=$u]/Name/text()} }</URL>, for $i in $Document/HasIP/@ip/data() where $Document/HasIP return <IP>{attribute{"id"}{$i}, attribute{"data"}{doc("newdf")//Ip[@id/data()=$i]/Name/text()}}</IP>, $Document/Time,$Document/Author,$Document/IsReply}</Document>'; > } > > > > // commands to be performed > > $replace_string=array("<Document>"=>" ","</Document>"=>" ","<Domain"=>" ","/>"=>" ","<URL"=>" ","<IP"=>" ","<Author"=>" ","<Time>"=>" ","</Time>"=>" "); > try { > > set_time_limit(0); > ini_set('max_execution_time',1000); > // create session > $session = new Session("localhost", 1984, "admin", "admin"); > if($cmd!=null) > { > > $query = $session->query($cmd); > print $query->init(); > $count=0; > if(!$query->more()) echo "no results"; > while($query->more()) { > $result=$query->next(); > $result=strtr($result,$replace_string); > echo $result."<br>"; > > } > > print $query->close(); > } > print $session->close(); > } catch (Exception $e) { > // print exception > print $e->getMessage(); > } > ?> > </body> > </html> > ________________________________________ > From: Christian Grün [christian.gruen(a)gmail.com] > Sent: Wednesday, July 20, 2011 12:59 PM > To: Lusha Wang > Cc: basex-talk(a)mailman.uni-konstanz.de > Subject: Re: [basex-talk] about performance > > Serena, > > thanks for your e-mail, and.. Sorry, it's difficult to give you advice > without any reproducible code [1]. Please take some time and send us a > small code example that allows us to find out what's going wrong. > > Best, > Christian > > [1] http://sscce.org/ > ___________________________ > > On Wed, Jul 20, 2011 at 7:52 PM, Lusha Wang <lwang40(a)syr.edu> wrote: >> Hello Everyone >> When I tried the online basex demo, no mater how frequent I submit >> the query, i can always get results. However, in my own machine, i started >> the basex server, and then I use php script to query the server, sometimes, >> no results are returned. Is there any way to solve this problem? Thanks a >> lot. >> >> >> Sincerely >> Serena

1 0

about performance
by Lusha Wang 20 Jul '11

20 Jul '11

Hello Everyone When I tried the online basex demo, no mater how frequent I submit the query, i can always get results. However, in my own machine, i started the basex server, and then I use php script to query the server, sometimes, no results are returned. Is there any way to solve this problem? Thanks a lot. Sincerely Serena

2 1

a question about return values and data query
by Lusha Wang 20 Jul '11

20 Jul '11

Hi Everyone I have to questions: 1) Is it possible to return sth formatted string like this: return <Document>{$Document/HasDomain/@id /data()+"%%"+$Document/Time/text()}</Document> So far it shows errors.....I dont know if there are alternatives. 2) I have data like this <Documents> <Document id="doc1"> <Date>yyyy-mm-dd</Date> <HasIP ip="ip1"> <HasDomain domain="domain1"> <HasDomain domain="domain2"> </Document> <Document id="doc1"> <Date>yyyy-mm-dd</Date> <HasIP ip="ip2"> <HasDomain domain="domain1"> <HasDomain domain="domain3"> </Document> </Documents> <IPs> <IP id="ip1"> <data>111.111.111.111</data> </IP> <IP id="ip2"> <data>111.111.111.112</data> </IP> </IPs> <Domains> <Domain id="domain1"> <data>abc.com</data> </Domain> <Domain id="domain2"> <data>edf.com</data> </Domain> <Domain id="domain3"> <data>abc.com</data> </Domain> <Domains> I m trying to get all domains from documents that are within the following time period. for $Document in doc('test')//Document let $datestr := $Document/Date/text() let $date := xs:date($datestr) where $date<=xs:date("2010-05-02") and $date>xs:date("2010-05-01") and $Document/HasDomain return <Document>{$Document/HasDomain, $Document/Time/text()}</Document> However, only the domain ids are returned, I cant make the database return domain data rather than id. Can anyone offer me some suggestions? Thanks a lot. Sincerely Lusha

2 1

help! database query for specific time period
by Lusha Wang 18 Jul '11

18 Jul '11

Hi Guys Is it possible to query for specific time period of data? I really appreciate any help. <Documents> <Document id=" " time=" "> .............. </Document> <Document id=" " time=" "> .............. </Document> </Documents> Sincerely Serena

3 4

Re: [basex-talk] bulk-load
by Christian Grün 16 Jul '11

16 Jul '11

Dear Tomaso, dear Michael, today, I had a closer look into the BaseX routines that are responsible for adding new documents to the database, and I tweaked the update code of our document index to avoid linear costs for adding single documents. You are invited to check out the latest stable snapshot [1] and give us your valuable feedback. There are still some bottlenecks: - the default XML parser takes some time for initialization, which is particularly noticable for small documents. You'll get some performance boost by switching to the internal parser (Command: set intparse; see [2] for details). - as each BaseX database command is atomic, the data is flushed to disk after each update to avoid data loss. You may either specify a directory on disk to add multiple files at once, or choose to insert nodes instead of documents, which will give you better performance. Hope this helps, Christian [1] http://files.basex.org/releases/latest/basex-6.7.1-SNAPSHOT.jar [2] http://docs.basex.org/wiki/Parsers ___________________________ On Mon, Jul 4, 2011 at 8:22 PM, Christian Grün <christian.gruen(a)gmail.com> wrote: > Hi Tomaso, > >> From your answer I suppose there is something slower >> when Add() is called many times, and faster if we use CreateDb. >> Can you explain why the times increase? Is it because Add >> updates the index each time? > > Exactly; as the ADD operation is an atomic operation, there is > currently no way to define a batch operation (other than going more > low level, and looking at the Add command [1]). It might be that some > update operations could be delayed though; I've added this as an issue > (feel free to include more details) [2]. > > Christian > > [1] https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/core/c… > [2] https://github.com/BaseXdb/basex/issues/137 >

1 0

Re: [basex-talk] how to specify xml-stylesheet when generating xml ?
by Eric 15 Jul '11

15 Jul '11

Leo Wörteler <lw(a)basex.org> wrote: >Dear Fabrice, > >Am 15.07.2011 11:18, schrieb Fabrice Etanchaud: >> I would like to add something like <?xml-stylesheet type="text/xsl" >> href="linktomystylesheet.xsl"?>between the xml declaration and the >> generated xml. >> I did not find any xml-stylesheet option for serialization. > >as BaseX doesn't have to understand what xml-stylesheet does, it treats >it like any other processing instruction [1]. To add one to your output >document, just insert it as the first node: > > > declare option output:omit-xml-declaration "no"; > > > > let $generated-xml := <foo><bar/></foo> > > return document { > > <?xml-stylesheet type="text/xsl" href="linktomystylesheet.xsl"?>, > > text { '
' }, > > $generated-xml > > } > >This produces: > > > <?xml version="1.0" encoding="UTF-8"?> > > <?xml-stylesheet type="text/xsl" href="linktomystylesheet.xsl"?> > > <foo> > > <bar/> > > </foo> > >Note that 
 is the entity for the ASCII newline character, I >inserted it only for readability. > >> BaseX is just wonderful. > >Thank you! > >Leo >__________ >[1] http://en.wikipedia.org/wiki/Processing_instruction >_______________________________________________ >BaseX-Talk mailing list >BaseX-Talk(a)mailman.uni-konstanz.de >https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

1 0

how to specify xml-stylesheet when generating xml ?
by Fabrice Etanchaud 15 Jul '11

15 Jul '11

Dear all at BaseX, how can one specify xml-stylesheet (for example in XQuery 3 prolog) ? I would like to add something like <?xml-stylesheet type="text/xsl" href="linktomystylesheet.xsl"?> between the xml declaration and the generated xml. I did not find any xml-stylesheet option for serialization. BaseX is just wonderful. best regards, Fabrice ETANCHAUD Senior Software Engineer edital Berkenlaan 1 B-1831 Brussels, Belgium <skype:fabrice.etanchaud?call> +32 2 716 32 32 general +32 2 716 32 20 fax fetanchaud(a)edital.com corsearch.com <http://www.corsearch.com/> | edital.com <http://www.edital.com/> Confidentiality Notice: This email and its attachments (if any) contain confidential information of the sender. The information is intended only for the use by the direct addressees of the original sender of this email. If you are not an intended recipient of the original sender (or responsible for delivering the message to such person), you are hereby notified that any review, disclosure, copying, distribution or the taking of any action in reliance of the contents of and attachments to this email is strictly prohibited. If you have received this email in error, please immediately notify the sender at the address shown herein and permanently delete any copies of this email (digital or paper) in your possession.

2 1

Best approach for 10 million+ documents?
by Michael Paddon 14 Jul '11

14 Jul '11

Hi, I'm currently evaluating BaseX for a project. I've read all the online documentation but the underlying storage and indexing mechanisms are still a bit of a mystery to me, so I'm having trouble making optimal decisions in designing a large collection of documents. I have 10 million documents of moderate size. These are intended to be regularly replaced/updated. I have the choice of storing each document individually in a collection, or inserting/updating into a single document. Which approach will generally perform better? In an experiment, I found that after adding a few million documents, adding new documents got really slow. The JVM pegs at 100% CPU so it is doing a lot of work. What's going on here? Indexing? Would increasing the JVM memory help? Can indexing be disabled for bulk loads? Rather than try random things to see what worked, I was hoping to get some insight into how the system stores, indexes and uses resources. Many thanks, Michael

2 1

Re: [basex-talk] BaseX.org Webmailer
by BaseX Support 14 Jul '11

14 Jul '11

> I will use DELETE command in advance. > But supposing I have a DB with 5000 documents, which is the XQuery command > to find out if there are duplicated names? One popular way is to use the db:open() or collection() function, and include the path to the relevant documents: count( collection("db/path/to/document.xml") ) > Thank you You're welcome, Christian

1 0

(Attribute with) namespace problem.
by Huib Verweij 14 Jul '11

14 Jul '11

Hi guys, I'm having a weird thing with BaseX, it's probably something I do wrong, but I can't find what it is so I am asking your help. I am PUTting a XML document to the JAX-RX server: ================================================ PUT /basex/jax-rx/lexus/2e7a78c8-fdce-488b-993e-019ad812bc04 HTTP/1.1 Content-Type: text/xml; charset=UTF-8 Authorization: Basic YWRtaW46YWRtaW4= User-Agent: Jakarta Commons-HttpClient/3.1 Host: localhost:8489 Content-Length: 699099 <?xml version="1.0" encoding="UTF-8"?> <lexus id="uuid:2e7a78c8-fdce-4993e-019ad812bc04"> <meta version="1.0"> <name>Lexicon</name> <description>lexicon</description> <note>[This lexicon was imported by demo on 2011-06-24T12:48:02.194+02:00.] </note> <owner ref="uuid:2c9090a20ad6bf2e009f096563"/> <users> <user ref="uuid:2c9090a20ad6bf2e01f096563"> <permissions> <read>true</read> <write>true</write> </permissions> </user> </users> <schema> <container admin-info="" description="" id="MmM5MDkwYTIxNjdjMjFkNzAxMTY4MDQ2OTg5MTRhMDE=" mandatory="true" multiple="false" name="lexicon" note="" type="lexicon"> <container admin-info="" description="Represents a word, a multi-word expression, or an affix in a given language" id="uuid:5f5be48f-089b-4da3-9493-ea302b082fb7" note="" type="lexical-entry"> <container admin-info="" description="" id="uuid:09775cfd-acaf-46cd-98f8-ca878386d133" mandatory="false" multiple="true" name="lexemeGroup" note="" type="container"> <container xmlns:dcr="http://www.isocat.org/ns/dcr" admin-info="http://www.isocat.org/datcat/DC-1325" dcr:datcat="http://www.isocat.org/datcat/DC-1325" description="Minimal unit of language which : has a semantic interpretation and embodies a distinct cultural concept. " id="uuid:a8eb40c4-2068-462d-946b-031a74202a68" mandatory="false" multiple="true" name="lexeme" note="" type="data"/> ================================================ etc. Note the xmlns:dcr declaration. (I am using tcpmon to see what's sent and received. So tcpmon is sitting just in between my app and the JAX-RX server.) When I request the document I am getting something I do not expect, the xmlns:dcr seems to be replaced with a xmlns:xhtml and therefor I receive an error when processing the jax-rx result: The query: ================================================ POST /basex/jax-rx/ HTTP/1.1 Content-Type: application/query+xml; charset=UTF-8 Authorization: Basic YWRtaW46YWRtaW4= User-Agent: Jakarta Commons-HttpClient/3.1 Host: localhost:8489 Content-Length: 673 <?xml version="1.0" encoding="UTF-8"?> <query> <text> (: Returns lexus/meta element :) element lexus { attribute id { 'uuid:2e7a78c8-fdce-4993e-019ad812bc04' }, collection('lexus')/lexus[@id eq 'uuid:2e7a78c8-fdce-4993e-019ad812bc04']/meta } </text> <parameter name="output" value="media-type=text/xml; omit-xml-declaration=no"/> </query> ================================================ and the result: ================================================ HTTP/1.1 200 OK Content-Type: text/xml;omit-xml-declaration=no Content-Length: 15285 Server: Jetty(6.1.25) <jax-rx:results xmlns:jax-rx="http://jax-rx.sourceforge.net"><jax-rx:result><lexus id="uuid:2e7a78c8-fdce-4993e-019ad812bc04"> <meta version="1.0"> <name>Lexicon</name> <description>lexicon</description> <note>[This lexicon was imported by demo on 2011-06-24T12:48:02.194+02:00.]</note> <owner ref="uuid:2c9090a20ad6bf2e010b22509f096563"/> <users> <user ref="uuid:2c9090a20ad6bf2e010b22509f096563"> <permissions> <read>true</read> <write>true</write> </permissions> </user> </users> <schema> <container admin-info="" description="" id="MmM5MDkwYTIxNjdjMjFkNzAxMTY4MDQ2OTg5MTRhMDE=" mandatory="true" multiple="false" name="lexicon" note="" type="lexicon"> <container admin-info="" description="Represents a word, a multi-word expression, or an affix in a given language" id="uuid:5f5be48f-089b-4da3-9493-ea302b082fb7" note="" type="lexical-entry"> <container admin-info="" description="" id="uuid:09775cfd-acaf-46cd-98f8-ca878386d133" mandatory="false" multiple="true" name="lexemeGroup" note="" type="container"> <container xmlns:xhtml="http://www.w3.org/1999/xhtml" admin-info="http://www.isocat.org/datcat/DC-1325" dcr:datcat="http://www.isocat.org/datcat/DC-1325" description="Minimal unit of language which : has a semantic interpretation and embodies a distinct cultural concept. " id="uuid:a8eb40c4-2068-462d-946b-031a74202a68" mandatory="false" multiple="true" name="lexeme" note="" type="data"/> ================================================ I vaguely remember having seen this before, but I can't find it in the mailing list archive, so sorry if I missed an obvious answer, and please help me anyway. Hartelijke groet, Huib Verweij.

3 5

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk