BaseX-Talk

basex-talk@mailman.uni-konstanz.de

1 participants
5252 discussions

Change basex home in the same execution enviroment
by Antonio Mele 28 Oct '15

28 Oct '15

Hello, I am using basex version 8.3-SNAPSHOT in a java stand alone application. After i set the property org.basex.path with the path of my DB (ex. /mnt/test/db_1) I call a new Context and then CreateDB. It creates the folder /mnt/test/db_1/data/db_1 After that (in the same execution enviroment) I change the property to another DB (ex. /mnt/test/db_2). It creates the folder /mnt/test/db_1/data/db_2 I would like to create the db_2 in the path /mnt/test/db_2/data/db_2. Follow the code I wrote, where inputFolder is a folder with inside some xml files Thanks in advance. System.setProperty("org.basex.path", "/mnt/test/db_1"); Context context = new Context(false); new CreateDB("db_1", inputFolder).execute(context); context.close(); System.setProperty("org.basex.path", "/mnt/test/db_2"); context = new Context(false); new CreateDB("db_2", inputFolder).execute(context); context.close();

2 3

Re: [basex-talk] Full-text index with lots of data
by Chuck Bearden 21 Oct '15

21 Oct '15

Sorry for not using "Reply All" earlier. Setting FTINDEXSPLITSIZE to 20000000 enabled the process to get a little further, if the meaning of each dot is the same. FTINDEXSPLITSIZE at default: ..............................|..................................................................|..........................................................................|...............................................................................|.. FTINDEXSPLITSIZE = 20000000 .......|.......|........|.......|......|........|.............|.............|.............|.............|.............|.............|.............|.............|..............|.............|.............|.............|.............|.............|.............|............ If it's a matter of making the indexing process take longer, that's not a problem. Thanks, Chuck On Tue, Oct 20, 2015 at 1:27 PM, Chuck Bearden <cfbearden(a)gmail.com> wrote: > Thanks Christian, I'll try the FTINDEXSPLITSIZE option. > > I'm also open to modifying the XML files it that would help. Because > of limitations of the service from which we harvest them RESTfully, I > have only 20 actual content elements in each file. If you think it > would make a difference, I could consolidate them to have, say, 200 or > 500 of the actual content elements per file, but I have no idea if > that would change how the indexing falls out. > > The files also have structures where some properties of each record > are each represented by a URL, and ID value, and a string. I could > XSLT the files to remove all but the string (human readable is better > for our purposes) to make them less verbose. > > BaseX is really super for doing data quality assessments of the XML, > and if we could get full-text indexing working, it would speed things > up considerably. Thanks to you & your team for all the work you've put > in to the application! > > Alles Gute > Chuck Bearden > > On Tue, Oct 20, 2015 at 12:55 PM, Christian Grün > <christian.gruen(a)gmail.com> wrote: > > I see; it seems that the index creation is failing at the very final > > step, in which partial index structures, which are temporarily written > > to disk, are merged. > > > > You could either to increase Xmx even more (to 6 or 7G?). If this > > doesn't work, you could try assign different values to the > > FTINDEXSPLITSIZE option [1] (start e.g. with 20000000). > > > > Sorry for the trouble. Feel free to keep me updated, maybe we find a > > way to fix this, > > Christian > > > > [1] http://docs.basex.org/wiki/Options#FTINDEXSPLITSIZE > > > > > > On Tue, Oct 20, 2015 at 7:48 PM, Chuck Bearden <cfbearden(a)gmail.com> > wrote: > >> Here's the stack trace: > >> > >> > =.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.= > >>> create db pure_20151019 pure_20151019 > >> Creating Database... > >> > ..;..;..;..;..;..;.;..;..;..;..;..;.;..;.;.;.;.....;.....;.....;......;.....;.....;.......;.;.;;.;.;;.;.;;.;.;;.;.;;.;.;;.;.................................................;..........................................................;..........................................................;..........................................................;..........................................................;..........................................................;................................................... > >> 677584.62 ms (1435 MB) > >> Indexing Text... > >> > ........................................................................................................................................................................................................................................................... > >> 98215794 operations, 178526.99 ms (1611 MB) > >> Indexing Attribute Values... > >> > ........................................................................................................................................................................................................................................................... > >> 178304119 operations, 135613.26 ms (2005 MB) > >> Indexing Full-Text... > >> > ..............................|..................................................................|..........................................................................|...............................................................................|..java.lang.OutOfMemoryError: > >> Java heap space > >> at org.basex.index.ft.FTList.next(FTList.java:93) > >> at org.basex.index.ft.FTBuilder.merge(FTBuilder.java:236) > >> at org.basex.index.ft.FTBuilder.write(FTBuilder.java:140) > >> at org.basex.index.ft.FTBuilder.build(FTBuilder.java:85) > >> at org.basex.index.ft.FTBuilder.build(FTBuilder.java:23) > >> at org.basex.data.DiskData.createIndex(DiskData.java:187) > >> at org.basex.core.cmd.CreateIndex.create(CreateIndex.java:103) > >> at org.basex.core.cmd.CreateIndex.create(CreateIndex.java:91) > >> at org.basex.core.cmd.CreateDB.run(CreateDB.java:104) > >> at org.basex.core.Command.run(Command.java:398) > >> at org.basex.core.Command.execute(Command.java:100) > >> at org.basex.api.client.LocalSession.execute(LocalSession.java:132) > >> at org.basex.api.client.Session.execute(Session.java:36) > >> at org.basex.core.CLI.execute(CLI.java:103) > >> at org.basex.core.CLI.execute(CLI.java:87) > >> at org.basex.BaseX.console(BaseX.java:191) > >> at org.basex.BaseX.<init>(BaseX.java:166) > >> at org.basex.BaseX.main(BaseX.java:42) > >> org.basex.core.BaseXException: Out of Main Memory. > >> You can try to: > >> - increase Java's heap size with the flag -Xmx<size> > >> - deactivate the text and attribute indexes. > >> at org.basex.core.Command.execute(Command.java:101) > >> at org.basex.api.client.LocalSession.execute(LocalSession.java:132) > >> at org.basex.api.client.Session.execute(Session.java:36) > >> at org.basex.core.CLI.execute(CLI.java:103) > >> at org.basex.core.CLI.execute(CLI.java:87) > >> at org.basex.BaseX.console(BaseX.java:191) > >> at org.basex.BaseX.<init>(BaseX.java:166) > >> at org.basex.BaseX.main(BaseX.java:42) > >> Out of Main Memory. > >> You can try to: > >> - increase Java's heap size with the flag -Xmx<size> > >> - deactivate the text and attribute indexes. > >>> d > >> > =.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.= > >> > >> Here's how the process looked in the output of 'ps -ef', in case > >> that's relevant: > >> > >> > =.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.= > >> cfbeard+ 88769 88757 46 12:15 pts/7 00:00:24 java -cp > >> > /home/cfbearden/opt/basex-8.3.0/BaseX.jar:/home/cfbearden/opt/basex-8.3.0/lib/basex-api-8.3.jar:/home/cfbearden/opt/basex-8.3.0/lib/basex-xqj-1.5.0.jar:/home/cfbearden/opt/basex-8.3.0/lib/commons-codec-1.4.jar:/home/cfbearden/opt/basex-8.3.0/lib/commons-fileupload-1.3.1.jar:/home/cfbearden/opt/basex-8.3.0/lib/commons-io-1.4.jar:/home/cfbearden/opt/basex-8.3.0/lib/igo-0.4.3.jar:/home/cfbearden/opt/basex-8.3.0/lib/jansi-1.11.jar:/home/cfbearden/opt/basex-8.3.0/lib/javax.servlet-3.0.0.v201112011016.jar:/home/cfbearden/opt/basex-8.3.0/lib/jdom-1.1.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-continuation-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-http-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-io-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-security-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-server-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-servlet-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-util-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-webapp-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-xml-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jing-20091111.jar:/home/cfbearden/opt/basex-8.3.0/lib/jline-2.13.jar:/home/cfbearden/opt/basex-8.3.0/lib/jts-1.13.jar:/home/cfbearden/opt/basex-8.3.0/lib/lucene-stemmers-3.4.0.jar:/home/cfbearden/opt/basex-8.3.0/lib/milton-api-1.8.1.4.jar:/home/cfbearden/opt/basex-8.3.0/lib/mime-util-2.1.3.jar:/home/cfbearden/opt/basex-8.3.0/lib/slf4j-api-1.7.12.jar:/home/cfbearden/opt/basex-8.3.0/lib/slf4j-simple-1.7.12.jar:/home/cfbearden/opt/basex-8.3.0/lib/tagsoup-1.2.1.jar:/home/cfbearden/opt/basex-8.3.0/lib/xmldb-api-1.0.jar:/home/cfbearden/opt/basex-8.3.0/lib/xml-resolver-1.2.jar:/home/cfbearden/opt/basex-8.3.0/lib/xqj2-0.2.0.jar:/home/cfbearden/opt/basex-8.3.0/lib/xqj-api-1.0.jar: > >> -Xmx4g org.basex.BaseX -d > >> > >> > >> On Tue, Oct 20, 2015 at 12:38 PM, Chuck Bearden <cfbearden(a)gmail.com> > wrote: > >>> It hasn't failed yet; I've gotten the progress indicators, along with > >>> the phases that have been completed: > >>> > >>> Creating Database... > >>> Indexing Text... > >>> Indexing Attribute Values... > >>> > >>> It's still working on "Indexing Full-Text...". I'll post whatever I > >>> get when it fails. Maybe it won't this time :) > >>> > >>> Chuck > >>> > >>> On Tue, Oct 20, 2015 at 12:33 PM, Christian Grün > >>> <christian.gruen(a)gmail.com> wrote: > >>>>> Creating Database... > >>>>> ..;..;..;..;..;..;.;..;.. > >>>> > >>>> Do you get any output after this line (I would expected to see a stack > >>>> trace, or at least an error message…)? > >>>> > >>>> > >>>> > >>>>> Where 'pure_20151019' is both the name of the database and the > >>>>> subdirectory where all my XML files are. > >>>>> > >>>>> It could well be that I'm missing a crucial option; I'm still > >>>>> relatively new to BaseX. It's great stuff, though. > >>>>> > >>>>> Because of my employer's IT environment, I have to run my Linux > >>>>> workstation in a VMWare VM, though I doubt that that makes a > >>>>> difference. > >>>>> > >>>>> Thanks, > >>>>> Chuck > >>>>> > >>>>> On Tue, Oct 20, 2015 at 11:15 AM, Christian Grün > >>>>> <christian.gruen(a)gmail.com> wrote: > >>>>>> Hi Chuck, > >>>>>> > >>>>>> Usually, 4G is more than enough to create a full-text index for 16G > of > >>>>>> XML. Obviously, however, that's not the case for your input data. > You > >>>>>> could try to distribute your documents in multiple database. As as > >>>>>> alternative, we could have a look at your data and try to find out > >>>>>> what's going wrong. You can also use the -d flag and send us the > stack > >>>>>> trace. > >>>>>> > >>>>>> Best, > >>>>>> Christian > >>>>>> > >>>>>> > >>>>>> On Tue, Oct 20, 2015 at 4:19 PM, Chuck Bearden <cfbearden(a)gmail.com> > wrote: > >>>>>>> Hi all, > >>>>>>> > >>>>>>> I have about 16G of XML data in about 52000 files, and I was > hoping to > >>>>>>> build a full-text index over it. I've tried two approaches: enable > >>>>>>> full-text indexing as I create the database and then loading the > data, > >>>>>>> and creating the full-text index after loading the data. If I > enable > >>>>>>> ADDCACHE and modify the basex shell script to use 4g of RAM > instead of > >>>>>>> 512M, I have no problem loading the data. If I try to load with > >>>>>>> FTINDEX or create the index afterward, the process runs out of > memory. > >>>>>>> > >>>>>>> I could believe that I'm overlooking some option that would make > this > >>>>>>> possible, but I suspect I just have too much data. I welcome your > >>>>>>> thoughts & suggestions. > >>>>>>> > >>>>>>> All the best, > >>>>>>> Chuck Bearden >

2 5

Circumvent authentication
by Marco Lettere 21 Oct '15

21 Oct '15

Dear all, is there a possibility to pass the Authorization header ahead to a restxq function without having it managed by BaseX itself? In a proof of concept project we have currently to face an authorization scheme which is not Basic or Digest. Unfortunately we always get an "Unsupported Authorization" exception. Workaround now is to place a proxy in front of Basex and transform the Authorization header into something like X-Authorization. Anyway I'd really like our code to be standard compliant. Thanks for any support as usual! Regards, Marco.

2 6

Full-text index with lots of data
by Chuck Bearden 20 Oct '15

20 Oct '15

Hi all, I have about 16G of XML data in about 52000 files, and I was hoping to build a full-text index over it. I've tried two approaches: enable full-text indexing as I create the database and then loading the data, and creating the full-text index after loading the data. If I enable ADDCACHE and modify the basex shell script to use 4g of RAM instead of 512M, I have no problem loading the data. If I try to load with FTINDEX or create the index afterward, the process runs out of memory. I could believe that I'm overlooking some option that would make this possible, but I suspect I just have too much data. I welcome your thoughts & suggestions. All the best, Chuck Bearden

2 3

Optimized query is empty when database is open and query uses a variable
by Amanda Galtman 20 Oct '15

20 Oct '15

Hi, When I run the following query with BaseX 8.2.3 or 8.3, and I have a certain database open, the optimized query is empty, so there are no results. let $x := 'x' let $doc := <root><el attr="x;y"/></root> return $doc//el[contains(@attr,$x)] This seems like a bug. If I close the database (which is not even being referenced in this query), open a different database, or replace $x in the last line with the literal string value, 'x', then I get the result I expect. The database that exhibits the problem and the database that doesn't exhibit the problem differ in the kinds of XML files in them and the indexes that are available for them. Does anyone know why the open database should affect the result of a query that doesn't reference the database? Thanks, Amanda

2 3

Which zip did BaseX choke on?
by Hondros, Constantine (ELS-AMS) 19 Oct '15

19 Oct '15

I love BaseX for the simplicity it brings to XML handling. But this is a problem I have not encountered before. I am creating a DB from about 17,000 small zipfiles, each containing a directory structure and somewhere within each, some XML. BaseX chokes on one of these files giving the error: "invalid entry size (expected 0 but got 11083 bytes)". So clearly one or more of the zips is invalid - but which one(s)? Is there any way that BaseX can echo to me the file that is causing the error? DEBUG is set to TRUE, but I don't get any morer verbose output. I am using the GUI, and running a BSX script too. Thanks in advance, C. ________________________________ Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.

4 6

STORE Command
by Maximilian Gärber 19 Oct '15

19 Oct '15

Hi, any way to add multiple binary files at once? <store path="source/_img/">../source/_img/*.*</store> would be nice. Regards, Max

2 3

restxq multipart/mixed application/octet-stream
by Dolejsi Jiri 19 Oct '15

19 Oct '15

When i send multipart/mixed document with 2.part application/octet-stream with value 0xFF0xFF Into the following function, it writes into file character ?. I am using BaseX 8.3 declare %rest:path("test") %rest:PUT("{$doc}") function nib:insert-doc($doc as item()*) as empty-sequence() { let $data := $doc[2] return file:write-text("d:\abc.bin", $data) }; ________________________________ Odes?latel tohoto e-mailu informuje adres?ta, ?e ?TK vy?aduje k platnosti pr?vn?ho jedn?n? p?semnou formu s v?jimkou jedn?n? realizovan?ho prost?ednictv?m elektronick?ch syst?m? ?TK (Fotobanka, Videobanka, Protext, NewsMarket, Infobanka). D?le informuje adres?ta, ?e smlouva s ?TK nem??e b?t uzav?ena, pokud nen? v?slovn? shoda o v?ech jej?ch n?le?itostech. ?TK si vyhrazuje pr?vo ukon?it jedn?n? o uzav?en? smlouvy kdykoli, a to z jak?hokoliv d?vodu nebo i bez uveden? d?vodu. Obsahuje-li tento e-mail nab?dku, ?TK vylu?uje p?ijet? nab?dky s dodatkem ?i odchylkou. Pokud je tento mail kvitanc? ve smyslu ? 1949, odst. 1 z?k. ?. 89/2012 Sb., ?TK v?slovn? sd?luje, ?e kvitance o zaplacen? jak?koliv spl?tky dluhu neznamen?, ?e byla sou?asn? zaplacena jak?koliv ??st t?ho? dluhu splatn? d??ve, a kvitance o spl?tce jistiny neznamen?, ?e bylo zaplaceno tak? p??slu?enstv? pohled?vky.

2 1

xml-to-json vs json:serialize
by Marco Lettere 16 Oct '15

16 Oct '15

Hi all, I have a bit of difficulty in understanding the rationale behind the XML to Json (and vice-versa) conversions. I must admit I'm a bit confused on when to use which format and how those map to serialization/parsing rules. In particular I'm wondering whether it's really necessary to use both the formalisms (and functions) in the following example. My feelings are that using only the standard (?) XQuery 3.1 representation would be much clearer and would enable a cleaner service chaining mechanism. Has anyone 2 minutes left to explain to me the model behind this sort of things? I really appreciate your support! Regards, Marco. module namespace jt = "urn:json:test"; declare %rest:path("jsonsrv") %output:method("text") %output:media-type("application/json") function jt:service() { xml-to-json( <map xmlns="http://www.w3.org/2005/xpath-functions"> <string key="a">12345678</string> </map>) }; declare %rest:path("testreq") %output:method("text") %output:media-type("application/json") function jt:test() { let $response := http:send-request( <http:request href="http://localhost:9984/jsonsrv" method="POST"> <http:body media-type="application/x-www-form-urlencoded" method="text"/> </http:request>) return json:serialize($response[2]) };

4 6

Location for import modules
by Paul Swennenhuis 15 Oct '15

15 Oct '15

Hi, I installed BaseX 8.3 as Tomcat webapp on a Ubuntu server. Everything works fine, including the DBA interface (very much appreciated, thanks for that!) Except one thing: I can't get the import of XQuery modules right. When I import a module in a .xq file, like this: import module namespace lib="My Library" at "library.xq"; BaseX gives me a "[XQST0059] Could not retrieve module '/library.xq' " error. Earlier I installed BaseX on a EC2 instance at Amazon, and there I found that import modules had to be placed in /usr/share/tomcat7 to be found. But that directory does not exist on this server, and even when I create it and place library.xq there it is still not found. I tried placing library.xq in the RESTPATH directory, in the REPOPATH directory, in the home directory of BaseX, but all in vain. I only get it to work when I place library.xq in the root of the server. And that is obviously not something I want to do. Or, alternatively, when I place it in [some directory] and then change the import line to import module namespace lib="My Library" at "[some directory]/library.xq"; In other words, when I change the relative path to an absolute one. But I'd have to do that for all of my XQuery files (a lot!) and, worse, I'd have to change it again if I deploy the .xq files on another server. So, to summarize: in which directory/directories does BaseX try to locate import modules? And what would I have to do to change the default location(s)? Thanks in advance. Paul

2 4

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk