BaseX-Talk

basex-talk@mailman.uni-konstanz.de

1 participants
5252 discussions

Re: [basex-talk] Out of memory
by Christian Grün 30 Mar '11

30 Mar '11

Hi Rupert, nice to hear that! Btw, I've added a little optimizations for mixed location paths (as the one containing the name() function after the location steps), which should perform the discussed optimizations on-the-fly: http://files.basex.org/releases/latest/ It's recommended to switch to that latest snapshot anyway, as it fixes some important index rewritings on collections, which have been temporarily removed in Version 6.6. More feedback is welcome, Christian ____________________________ > Hello Christian! > > You're my personal hero! The query > "dinstinct-values(/descendant::*[randnr]/name())" worked perfectly! 2 GB of > data analysed in 3 seconds, WOW! > > > Greetings and thanks again! > Rupert Jung > > -- > Rupert Jung > <pagina> GmbH > Gesamtherstellung wissenschaftlicher Werke > Herrenberger Str. 51 > D-72070 Tübingen > Handelsregister Stuttgart HRB 380249 > Geschäftsführer: Tobias Ott > > E-Mail: rupert.jung(a)pagina-tuebingen.de > Phone: (0 70 71) 98 76-37 > Fax: (0 70 71) 98 76-22 > > http://www.pagina-online.de > > > -----Ursprüngliche Nachricht----- > Von: Christian Grün [mailto:christian.gruen@gmail.com] > Gesendet: Dienstag, 29. März 2011 18:51 > An: rupert.jung(a)pagina-tuebingen.de > Cc: Andreas Weiler; bjoern.duenckel(a)pagina-tuebingen.de; > basex-talk(a)mailman.uni-konstanz.de > Betreff: Re: [basex-talk] Out of memory > > Rupert, > > thanks for your observation. My assumption is that the (hidden) > descendant-or-self step in your query causes a huge number of > intermediary nodes, which are then reduced to a small result set. In > other words, your query.. > > dinstinct-values(//*[randnr]/name()) > > ..equals the following query: > > > dinstinct-values(/descendant-or-self::node()/child::*[child::randnr]/name()) > > There are several choices how to possibly speed up your query; please > try e.g. to: > > 1. explicitly use the descendant step: > dinstinct-values(/descendant::*[randnr]/name()) > 2. wrap the name function around the location path: > dinstinct-values( name( //*[randnr] )) > 3. directly address the randnr nodes and use a parent step: > dinstinct-values( name( /descendant::randnr/.. )) > > We might add some optimizations to BaseX to automatize some of the > proposed steps. > > If this doesn't help, feel free to give us more feedback. > > Best, > Christian > ___________________________ > > On Tue, Mar 29, 2011 at 5:31 PM, Rupert jung > <rupert.jung(a)pagina-tuebingen.de> wrote: >> Hi Andreas and thanks for your answer, >> >> >> >> unfortunely that didn’t work (java.exe consumed 1.2 GB, then stopped). The >> expected result should not be longer then about 10 element names or so... >> >> >> >> Maybe this could be a bug in basex itself…? >> >> >> >> >> >> ________________________________ >> >> Rupert Jung >> >> <pagina> GmbH >> Gesamtherstellung wissenschaftlicher Werke >> Herrenberger Str. 51 >> D-72070 Tübingen >> >> Handelsregister Stuttgart HRB 380249 >> Geschäftsführer: Tobias Ott >> >> Phone: (0 70 71) 98 76-37 >> Fax: (0 70 71) 98 76-22 >> E-Mail: rupert.jung(a)pagina-tuebingen.de >> http://www.pagina-online.de >> >> >> >> Von: Andreas Weiler [mailto:andreas.weiler@uni-konstanz.de] >> Gesendet: Dienstag, 29. März 2011 16:25 >> An: rupert.jung(a)pagina-tuebingen.de >> Cc: basex-talk(a)mailman.uni-konstanz.de; > bjoern.duenckel(a)pagina-tuebingen.de >> Betreff: Re: [basex-talk] Out of memory >> >> >> >> Hi, >> >> >> >> as first hint you could start BaseX with the Xmx flag of Java: >> >> >> >> java -cp BaseX.jar -Xmx1G org.basex.BaseXGUI >> >> >> >> Probably that will solve this issue. >> >> >> >> Kind regards, >> >> Andreas >> >> >> >> Am 29.03.2011 um 16:14 schrieb Rupert jung: >> >> Hi there, >> >> >> >> I’m currently doing some tests with BaseX and a mid-sized database (around > 2 >> GB). >> >> >> >> I wonder myself why I’m not able to process this xquery-statement: >> >> >> >> dinstinct-values(//*[randnr]/name()) >> >> („Give me a list of all elements which have a child-element <randnr> and >> remove all double entries“) >> >> >> >> After about 10 seconds a got a „out of main memory“ error. What’s really >> strange about this: Processing the nodes itself with //*[randnr]/ >> >> works like a charm (but gives me a HUGE amount of text and is not really >> useful for me at all). >> >> >> >> My system: win7-x64, 4 GB RAM, Java 1.6.0_21 >> >> >> >> Thank you in advance, >> >> Rupert Jung >> >> >> >> ________________________________ >> >> Rupert Jung >> >> <pagina> GmbH >> Gesamtherstellung wissenschaftlicher Werke >> Herrenberger Str. 51 >> D-72070 Tübingen >> >> Handelsregister Stuttgart HRB 380249 >> Geschäftsführer: Tobias Ott >> >> Phone: (0 70 71) 98 76-37 >> Fax: (0 70 71) 98 76-22 >> E-Mail: rupert.jung(a)pagina-tuebingen.de >> http://www.pagina-online.de >> >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk(a)mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk(a)mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> > >

2 1

Re: [basex-talk] Out of memory
by Christian Grün 29 Mar '11

29 Mar '11

Rupert, thanks for your observation. My assumption is that the (hidden) descendant-or-self step in your query causes a huge number of intermediary nodes, which are then reduced to a small result set. In other words, your query.. dinstinct-values(//*[randnr]/name()) ..equals the following query: dinstinct-values(/descendant-or-self::node()/child::*[child::randnr]/name()) There are several choices how to possibly speed up your query; please try e.g. to: 1. explicitly use the descendant step: dinstinct-values(/descendant::*[randnr]/name()) 2. wrap the name function around the location path: dinstinct-values( name( //*[randnr] )) 3. directly address the randnr nodes and use a parent step: dinstinct-values( name( /descendant::randnr/.. )) We might add some optimizations to BaseX to automatize some of the proposed steps. If this doesn't help, feel free to give us more feedback. Best, Christian ___________________________ On Tue, Mar 29, 2011 at 5:31 PM, Rupert jung <rupert.jung(a)pagina-tuebingen.de> wrote: > Hi Andreas and thanks for your answer, > > > > unfortunely that didn’t work (java.exe consumed 1.2 GB, then stopped). The > expected result should not be longer then about 10 element names or so... > > > > Maybe this could be a bug in basex itself…? > > > > > > ________________________________ > > Rupert Jung > > <pagina> GmbH > Gesamtherstellung wissenschaftlicher Werke > Herrenberger Str. 51 > D-72070 Tübingen > > Handelsregister Stuttgart HRB 380249 > Geschäftsführer: Tobias Ott > > Phone: (0 70 71) 98 76-37 > Fax: (0 70 71) 98 76-22 > E-Mail: rupert.jung(a)pagina-tuebingen.de > http://www.pagina-online.de > > > > Von: Andreas Weiler [mailto:andreas.weiler@uni-konstanz.de] > Gesendet: Dienstag, 29. März 2011 16:25 > An: rupert.jung(a)pagina-tuebingen.de > Cc: basex-talk(a)mailman.uni-konstanz.de; bjoern.duenckel(a)pagina-tuebingen.de > Betreff: Re: [basex-talk] Out of memory > > > > Hi, > > > > as first hint you could start BaseX with the Xmx flag of Java: > > > > java -cp BaseX.jar -Xmx1G org.basex.BaseXGUI > > > > Probably that will solve this issue. > > > > Kind regards, > > Andreas > > > > Am 29.03.2011 um 16:14 schrieb Rupert jung: > > Hi there, > > > > I’m currently doing some tests with BaseX and a mid-sized database (around 2 > GB). > > > > I wonder myself why I’m not able to process this xquery-statement: > > > > dinstinct-values(//*[randnr]/name()) > > („Give me a list of all elements which have a child-element <randnr> and > remove all double entries“) > > > > After about 10 seconds a got a „out of main memory“ error. What’s really > strange about this: Processing the nodes itself with //*[randnr]/ > > works like a charm (but gives me a HUGE amount of text and is not really > useful for me at all). > > > > My system: win7-x64, 4 GB RAM, Java 1.6.0_21 > > > > Thank you in advance, > > Rupert Jung > > > > ________________________________ > > Rupert Jung > > <pagina> GmbH > Gesamtherstellung wissenschaftlicher Werke > Herrenberger Str. 51 > D-72070 Tübingen > > Handelsregister Stuttgart HRB 380249 > Geschäftsführer: Tobias Ott > > Phone: (0 70 71) 98 76-37 > Fax: (0 70 71) 98 76-22 > E-Mail: rupert.jung(a)pagina-tuebingen.de > http://www.pagina-online.de > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >

2 1

Re: [basex-talk] Out of memory
by Dimitar Popov 29 Mar '11

29 Mar '11

Hi Rupert, could you post the execution plans of the two queries? And also the version of BaseX that you use. Best regards, Dimitar Am Dienstag 29 März 2011, 17:31:26 schrieb Rupert jung: > Hi Andreas and thanks for your answer, > > > > unfortunely that didnt work (java.exe consumed 1.2 GB, then stopped). The > expected result should not be longer then about 10 element names or so... > > > > Maybe this could be a bug in basex itself? > > > > > > _____ > > Rupert Jung > > <pagina> GmbH > Gesamtherstellung wissenschaftlicher Werke > Herrenberger Str. 51 > D-72070 Tübingen > > Handelsregister Stuttgart HRB 380249 > Geschäftsführer: Tobias Ott > > Phone: (0 70 71) 98 76-37 > Fax: (0 70 71) 98 76-22 > E-Mail: rupert.jung(a)pagina-tuebingen.de > http://www.pagina-online.de <http://www.pagina-online.de/> > > > > Von: Andreas Weiler [mailto:andreas.weiler@uni-konstanz.de] > Gesendet: Dienstag, 29. März 2011 16:25 > An: rupert.jung(a)pagina-tuebingen.de > Cc: basex-talk(a)mailman.uni-konstanz.de; bjoern.duenckel(a)pagina-tuebingen.de > Betreff: Re: [basex-talk] Out of memory > > > > Hi, > > > > as first hint you could start BaseX with the Xmx flag of Java: > > > > java -cp BaseX.jar -Xmx1G org.basex.BaseXGUI > > > > Probably that will solve this issue. > > > > Kind regards, > > Andreas > > > > Am 29.03.2011 um 16:14 schrieb Rupert jung: > > > > > > Hi there, > > > > Im currently doing some tests with BaseX and a mid-sized database (around > 2 GB). > > > > I wonder myself why Im not able to process this xquery-statement: > > > > dinstinct-values(//*[randnr]/name()) > > (Give me a list of all elements which have a child-element <randnr> and > remove all double entries) > > > > After about 10 seconds a got a out of main memory error. Whats really > strange about this: Processing the nodes itself with //*[randnr]/ > > works like a charm (but gives me a HUGE amount of text and is not really > useful for me at all). > > > > My system: win7-x64, 4 GB RAM, Java 1.6.0_21 > > > > Thank you in advance, > > Rupert Jung > > > > _____ > > Rupert Jung > > <pagina> GmbH > Gesamtherstellung wissenschaftlicher Werke > Herrenberger Str. 51 > D-72070 Tübingen > > Handelsregister Stuttgart HRB 380249 > Geschäftsführer: Tobias Ott > > Phone: (0 70 71) 98 76-37 > Fax: (0 70 71) 98 76-22 > E-Mail: rupert.jung(a)pagina-tuebingen.de > http://www.pagina-online.de <http://www.pagina-online.de/> > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

2 1

Re: [basex-talk] Out of memory
by Andreas Weiler 29 Mar '11

29 Mar '11

Hi, as first hint you could start BaseX with the Xmx flag of Java: java -cp BaseX.jar -Xmx1G org.basex.BaseXGUI Probably that will solve this issue. Kind regards, Andreas Am 29.03.2011 um 16:14 schrieb Rupert jung: > Hi there, > > I’m currently doing some tests with BaseX and a mid-sized database (around 2 GB). > > I wonder myself why I’m not able to process this xquery-statement: > > dinstinct-values(//*[randnr]/name()) > („Give me a list of all elements which have a child-element <randnr> and remove all double entries“) > > After about 10 seconds a got a „out of main memory“ error. What’s really strange about this: Processing the nodes itself with //*[randnr]/ > works like a charm (but gives me a HUGE amount of text and is not really useful for me at all). > > My system: win7-x64, 4 GB RAM, Java 1.6.0_21 > > Thank you in advance, > Rupert Jung > > Rupert Jung > > <pagina> GmbH > Gesamtherstellung wissenschaftlicher Werke > Herrenberger Str. 51 > D-72070 Tübingen > Handelsregister Stuttgart HRB 380249 > Geschäftsführer: Tobias Ott > Phone: (0 70 71) 98 76-37 > Fax: (0 70 71) 98 76-22 > E-Mail: rupert.jung(a)pagina-tuebingen.de > http://www.pagina-online.de > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

2 1

Out of memory
by Rupert jung 29 Mar '11

29 Mar '11

Hi there, Im currently doing some tests with BaseX and a mid-sized database (around 2 GB). I wonder myself why Im not able to process this xquery-statement: dinstinct-values(//*[randnr]/name()) (Give me a list of all elements which have a child-element <randnr> and remove all double entries) After about 10 seconds a got a out of main memory error. Whats really strange about this: Processing the nodes itself with //*[randnr]/ works like a charm (but gives me a HUGE amount of text and is not really useful for me at all). My system: win7-x64, 4 GB RAM, Java 1.6.0_21 Thank you in advance, Rupert Jung _____ Rupert Jung <pagina> GmbH Gesamtherstellung wissenschaftlicher Werke Herrenberger Str. 51 D-72070 Tübingen Handelsregister Stuttgart HRB 380249 Geschäftsführer: Tobias Ott Phone: (0 70 71) 98 76-37 Fax: (0 70 71) 98 76-22 E-Mail: <mailto:rupert.jung@pagina-tuebingen.de> rupert.jung(a)pagina-tuebingen.de <http://www.pagina-online.de/> http://www.pagina-online.de

1 0

BaseX 6.6 Update - Soft-Files.com
by Soft-Files Team 28 Mar '11

28 Mar '11

Dear Developer, Your product BaseX 6.6 has been updated. Your product review page is located at: http://www.soft-files.com/basex_72311.html. Sincerely, The Soft-Files Team

1 0

xmldb api and child collections
by frans＠planet 27 Mar '11

27 Mar '11

Is it possible to create child Collections within a database through the XMLDB API? And is it possible to insert XMLResources in a child Collection through the XMLDB API? Regards, Frans

2 1

xmldb api and namespace handling
by frans＠planet 26 Mar '11

26 Mar '11

I'am accessing BaseX 6.5.1 through the XMLDB API. There are some issues with namespace handling when storing resources. Let me clarify that with some examples. First example I want to store the following as a resource: String example = "<?xml version=\"1.0\" encoding=\"UTF-8\" ?>" + "<tag xmlns=\"http://example.org\">" + "</tag>"; This yields <tag xmlns="http://example.org"/> in the GUI. So this is OK. Second example I want to store the following as a resource and INTPARSE = false: String example = "<?xml version=\"1.0\" encoding=\"UTF-8\" ?>" + "<tns:tag xmlns:tns=\"http://example.org\">" + "</tns:tag>"; This yields the following exception: org.xmldb.api.base.XMLDBException: "null" (Line 1): The prefix "tns" for element "tns:tag" is not bound Third example I want to store the following as a resource and INTPARSE = true: String example = "<?xml version=\"1.0\" encoding=\"UTF-8\" ?>" + "<tns:tag xmlns:tns=\"http://example.org\">" + "</tns:tag>"; This yields <tns:tag xmlnstns="http://example.org"/> in the GUI. The colon is missing in ' xmlnstns' . So this is not OK. Are the second and third examples bugs or should I change some setting to get a proper handling of namespaces? Regards, Frans

2 4

Fwd: BaseX REST Security
by Andreas Weiler 25 Mar '11

25 Mar '11

Forwarding to mailing-list. Anfang der weitergeleiteten E-Mail: > Von: Jan Vlčinský (CAD) <jan.vlcinsky(a)cad-programs.com> > Datum: 25. März 2011 14:46:47 MEZ > An: Andreas Weiler <andreas.weiler(a)uni-konstanz.de> > Betreff: Re: [basex-talk] BaseX REST Security > > Hi > Let me describe my vision of possible implementation. > Current REST communication would not declare anything about user and password in the xml documents being sent to server as request. > Web server (servlet container) would implement standard security protection to given url - either by means of basic or by means of digest authentication and possibly using https. > BaseX server would have to provide some method, how to let servlet check, that given credentials (username and password) are valid and servlet would use it to authenticate requests (thinking of using JAAS). > Servlet would also use username and password of the REST request to log into BaseX. > > Implementation for basic authentication would be relatively simple, as user provides full password and servlet can reuse it in logging into BaseX. > Using e.g. JAAS, BaseX would implement interface for authentication. > > With digest the situation is a bit more difficult as password from http request is probably unusable for logging into BaseX as it is already arriving somehow scrambled to the web server and reconstrucion is not possible (if I am correct). > Solutions could be > BaseX would have an option to reuse authenticated user and somehow reuse the available password data or simply trusting user, who logged into web server already. > At servlet there would be mapping from (authenticated) username to credentials of BaseX account (username and password). This would be used to log into BaseX. > Both options have some drawbacks and security risks, but we all know, security risk is general feature of almost any method. > The simplest solution could use basic authentication and rely on https encrypting open password over network. > > Just some ideas which came to my mind. > > With best regards > > Jan Vlčinský > > > 2011/3/25 Andreas Weiler <andreas.weiler(a)uni-konstanz.de> > Hi, > > in the BXJaxRX class, you can set String USER and String PASSWORD to use another user than the standard admin user. So each request will be executed/declined regarding these user permissions. > Currently it is not possible to send username/password with single requests to the server in the JAXRX mode. > > -- Andreas > > Am 25.03.2011 um 13:46 schrieb Евгений Хабаров: > > > When connection is made using Language Bindings, client need valid > > login/password to access database. > > When JAXRX is used - user authentication is NOT requested. > > Is it possible to protect JAXRX interface operations? > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk(a)mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > -- > Ing. Jan Vlčinský > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky(a)gmail.com > http://cz.linkedin.com/in/vlcinsky >

3 4

BaseX REST Security
by Евгений Хабаров 25 Mar '11

25 Mar '11

When connection is made using Language Bindings, client need valid login/password to access database. When JAXRX is used - user authentication is NOT requested. Is it possible to protect JAXRX interface operations?

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk