- BaseX-Talk - mailman.uni-konstanz.de

sending variables to xquery module via client api (python)
by Cholfin, Bryan 17 Nov '15

17 Nov '15

Hi- Hoping someone can help me with this. I'm trying to write a python script that calls an xquery module in the database. It seems to be pretty easy to use a line such as: session.execute("run gettext.xqm") and get back the response I want. What I want to do is pass a bound variable to the module. The client api has a provision to bind a variable to a query object, but I'm not making a query object. I'm simply executing the stored query. I don't find anything in the documentation to suggest that a variable can be added to the run command this way. Bryan Cholfin Associate Editor for Digital Projects and Metadata Harvard University Press 617.496.9116 79 Garden Street | Cambridge, MA 02138-1400 www.hup.harvard.edu<http://www.hup.harvard.edu/>

3 8

Restxq
by meumapple 16 Nov '15

16 Nov '15

Hi All, I am using restxq. I wrote a function which takes one variable (this should be a chunk of text, not a single word). The output is xml. I can have the function work if I pass the content of the variable with curl -X POST --data "variable=my text is long" http://address How can I get the same result of the curl command with an xquery function? More in general, is this a good way to write a function in restxq when the variable is not a single word but the content of a file? (I am not even sure if a file can be passed as such in restxq) Thanks, Joseph

3 4

Treating SQL Results
by Ana Victória Ladeira 16 Nov '15

16 Nov '15

Good morning. I have an XML file of the following format: <?xml version="1.0" encoding="UTF-8"?> <resultsets> <resultset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <row> <column name="TITULO">A Adequação de Sistemas Baseados em Casos no Sequenciamento Inteligente de Ações</column> <column name="NOMECOMPLETO">José Pacheco de Almeida Prado</column> </row> <row> <column name="TITULO">Sistemas Distribuídos Inteligentes</column> <column name="NOMECOMPLETO">José Pacheco de Almeida Prado</column> </row> <row> <column name="TITULO">Uma Arquitetura para Planejamento Inteligente</column> <column name="NOMECOMPLETO">José Pacheco de Almeida Prado</column> </row> <row> <column name="TITULO">Navegação de robôs móveis com Lógica Paracpmsistente Anotada e método de Campos Potenciais</column> <column name="NOMECOMPLETO">José Pacheco de Almeida Prado</column> </row> <row> <column name="TITULO">Adaptation in a Traffic Control System</column> <column name="NOMECOMPLETO">José Pacheco de Almeida Prado</column> </row> ... The file contains titles for articles and their authors.. I was asked to develop a query that returns the author's name followed by the titles of all of its articles, so the result would be something like this: <Author1>:<Article1>, <Article2>, <Article3>... <Author2>:<Article1>, <Article2>... ... Any ideas on how can I do it? Thank you in advance.

2 1

database triggers
by Alexander Haus 16 Nov '15

16 Nov '15

Hello BaseX team, I've recently read, that you are currently working on a trigger mechanism for BaseX. Can you already tell, when you are planning to include it in a future release? Thanks a lot, Alex

2 1

Typo in docs?
by Marco Lettere 13 Nov '15

13 Nov '15

Hi all, Just one question. The documentation page related to the json module states: *Signatures* |*json:serialize*($input as node()) as xs:string| |*json:serialize*($input as node(), $options as map(xs:string, xs:string)) as xs:string| *Summary* Serializes the node specified by|$input|as JSON, and returns the result as|xs:string|instance. The node is expected to conform to the output created by thejson:parse() <http://docs.basex.org/wiki/JSON_Module#json:parse>function. All other items will be serialized as specified for thejson output method <http://docs.basex.org/wiki/XQuery_3.1#JSON_Serialization>of the official specification. Items can also be serialized as JSON if theSerialization Parameter <http://docs.basex.org/wiki/Serialization>|method|is set to|json|. The|$options|argument can be used to control the way the input is serialized. But I think the signature should be for item()? for the first parameter (since it allows also for maps) isn't it? In effect inspect:function(json:serialize#2) returns <function name="json:serialize" uri="http://basex.org/modules/json"> <argument type="item()" occurrence="?"/> <argument type="item()" occurrence="?"/> <return type="xs:string"/> </function> If it's confirmed I could proceed with fixing it. Regards, Marco.

2 1

Re: [basex-talk] Fwd: Bash command
by meumapple 10 Nov '15

10 Nov '15

I can add that if I use "cat file" on the bash I get the right Greek characters, while the same command via proc:execute returns gibberish characters (the optional argument "utf-8" in proc:execute has no effect). Thanks. Joseph Il giorno 09/nov/2015, alle ore 19:04, meumapple <meumapple(a)gmail.com> ha scritto: Hi, I do not have any LANG variable set. If I redirect into a file the output of the bash command, I get a message with "utf-8" when I analyze this file with the command "file". And indeed the characters are displayed correctly. So the problem arises only when I run everything through proc:execute. If I save the result of the xquery script and analyze them with "file" I get the message "output: ERROR : line 22 : regexec error 17, (illegal byte sequence). And of course the Greek characters are messed up (but not the ASCII ones for English) Hope this helps... Joseph Il giorno 09/nov/2015, alle ore 15:18, Jens Erat <jens.erat(a)uni-konstanz.de> ha scritto: Please report the encoding of your actual files. What is the output of `env | grep LANG` (when stored in your bash script)? If you pipe the output in another file, what will `file -i [output-file]` report (`file` applies some magic to discover file formats and encoding)? > Am 09.11.2015 um 14:06 schrieb meumapple: > Hi All, > > I have been able to run the command just putting it in a bash file and then running this file with proc:execute. It works. The problem now is with the encoding of the text returned by proc:execute. In my bash I can display Greek characters, but they are messed up if returned by the proc:execute function. I also added utf-8 as a parameter for the function but nothing changes. How can I solve this? Thanks. > > Joseph > > Il giorno 09/nov/2015, alle ore 10:41, Jens Erat <jens.erat(a)uni-konstanz.de> ha scritto: > > Hi Joseph, > > As far as I understand the documentation, BaseX `proc:system` does not > use a shell interpreter by default. `<` or also the pipe `|` are > features offered by your shell, not the operating system. > > Instead of passing a single parameter, pass the command as parameter to > `/usr/bin/env sh -c`, which will make BaseX run a shell that interpretes > I/O indirection: > > proc:system('/usr/bin/env', ('sh', '-c', 'opennlp postagger > model.bin < file.txt')) > > This might be a reasonable thing to provide as a built-in function, I'm > also not sure whether one can connect some kind of stream/strings to > STDIN, STDOUT and STDERR. > > Regards from Lake Constance, Germany, > Jens > > Am 09.11.2015 um 10:19 schrieb Christian Grün: >>> Any suggestions? >> >> Any example? ;) >> Christian >> >> >> >> Thanks. >>> >>> Joseph >>> >>> >>> Inizio messaggio inoltrato: >>> >>> Da: meumapple <meumapple(a)gmail.com> >>> Data: 06 novembre 2015 20:52:08 CET >>> A: BaseX <basex-talk(a)mailman.uni-konstanz.de> >>> Oggetto: Bash command >>> >>> Hi all, >>> >>> I am using proc:system for a bash command but I cannot run it. It looks like >>> opennlp postagger model.bin < file.txt. I suspect that the problem is the >>> less-than sign, which is converted into < Is there any way to keep the >>> less-than sign and pass it to the bash properly? Thanks. >>> >>> Joseph -- Jens Erat Universität Konstanz Kommunikations-, Infomations-, Medienzentrum (KIM) Abteilung Basisdienste D-78457 Konstanz Mail: jens.erat(a)uni-konstanz.de

1 0

Re :Collection DataBase Locking
by Adi Babu 10 Nov '15

10 Nov '15

While updating the Basex collection data base if it fails because of some unexpected reason the particular database cannot be opened anymore and the message shown in loggers is "*Database ... is being updated, or update was not completed*". I have found that this is because of *upd.basex*. My question is how can I found and remove this file through code to proceed further with a safe execution of the collection database creation process. Thanks & Regards, Adi

2 1

Re: [basex-talk] Sub : Adding Special Characters to the Collection Database
by Christian Grün 10 Nov '15

10 Nov '15

Hi Madhan, I see you are using BaseX 7.9. Does the problem persist when using our latest release? What is the output of db:info() (or INFO DB)? Best, Christian On Mon, Nov 9, 2015 at 3:33 PM, madhan gopal <madhangopal828(a)gmail.com> wrote: > Dear Christian, > > Here the problem is,the xml attribute values are replacing with Alphabet > "A" > > Example: > in my Xml file i have the tag <MyTag myAttribute="10" > ..</MyTag> > if i add this xml to collection database, the value of attribute > myAttribute is replacing 10 with 'A' > > Note:Please check once attachment. > > > > On Mon, Nov 9, 2015 at 4:52 PM, Christian Grün <christian.gruen(a)gmail.com> > wrote: >> >> Hi Madhan, >> >> > I am facing the problem of adding special characters to the collection >> > Database in below cases, >> >> What do you mean by "problem", and what are "special characters"? >> Christian >> >> >> > Case-1: if the xml file size is huge(in GBs),at the time the collection >> > Database creation tacking more time, >> > Case-2: if the more no.of xml files are adding to the particular >> > collection >> > Database continuously >> > >> > Example: >> > >> > in Xml file <MyTag myAttribute="10" ...</MyTag> >> > >> > The values are adding like the following >> > in collection <MyTag myAttribute="A" ...</MyTag> >> > >> > Sample Collection adding code : >> > -------------------------------------------- >> > /* >> > This method takes the xml files list as input and process the list >> > Create the xml collection DB, adds xml to the collection if already >> > exists >> > replace it >> > */ >> > >> > public static void createCollection(ArrayList<String> listFiles){ >> > ClientSession clientSession = null; >> > String strIndexedFile=""; >> > try { >> > clientSession = new ClientSession("<<IP_ADDRESS>>", 1984, >> > "<<USER>>", "<<PWD>>"); >> > String strDbName = "Sample_Collection"; >> > String strKeysFileLocation = >> > "D:\\TEMP\\7GB_Collection_TestingTeam\\Files"; >> > >> > String res = clientSession.execute("XQUERY db:exists('" + >> > strDbName + "')"); >> > //DB existence checking >> > if (!res.equalsIgnoreCase("true")) { >> > clientSession.execute("CREATE DB " + strDbName); >> > } else { >> > clientSession.execute("OPEN " + strDbName); >> > } >> > >> > for (int i = 0; i < listFiles.size(); i++) { >> > strIndexedFile = listFiles.get(i); >> > String tempFlag=clientSession.execute("XQUERY >> > db:exists('" + >> > strDbName + "','" + strIndexedFile + "')"); >> > clientSession.execute("ADD " + strKeysFileLocation + >> > File.separator + strIndexedFile); >> > } >> > System.out.println("InFormation:: >> > "+clientSession.execute("INFO")); >> > //optimizing the current opend DB using OPTIMIZE ALL command >> > clientSession.execute("OPTIMIZE ALL"); >> > System.out.println("InFormation:: >> > "+clientSession.execute("INFO")); >> > >> > } catch (Exception e) { >> > System.out.println("Exception files DB Updating: " + >> > e.getMessage()); >> > e.printStackTrace(); >> > } finally { >> > if (clientSession != null) { >> > try { >> > clientSession.close(); >> > } catch (IOException ex) { >> > ex.printStackTrace(); >> > } >> > } >> > } >> > } >> > >> > Hardware Specifications of Basex Sever : >> > ---------------------------------------------------------- >> > Operating System : Red Hat Enterprise Linux 64bit >> > OS Version : Linux 2.6.32 >> > Hard Disk : 500 GB >> > Memory Capacity : 16 GB Ram >> > Processor Family : Intel (R) Xeon(R) CPU E5 -2680 V2 >> > Processor Speed : 2.80 GHz >> > Physical Processors : 8 >> > Virtual Processors/cores : 4 >> > Server Version : basex 7.9 >> > -- >> > Thanks & Regards >> > Madhangopal.P > > > > > -- > Thanks & Regards > Madhangopal.P

2 2

Fwd: basic JSON problem
by Florent Gallaire 10 Nov '15

10 Nov '15

Hi Christian, that's what I was afraid, http://www.w3.org/TR/xslt-30/#func-xml-to-json is just too complex to understand for me. What about your simple-json try ? to have an easy: my_request&method=json&json=format=simple Best regards Florent 2015-11-09 10:17 GMT+01:00 Christian Grün <christian.gruen(a)gmail.com>: > Hi Florent, > > When using a JSON format other than jsonml, your XML document to be > output must comply with the rules specified in the particular format > [1,2]: > > xml-to-json(<string > xmlns="http://www.w3.org/2005/xpath-functions">JSON</string>) > > ...versus... > > xml-to-json(<string>JSON</string>) > > [1] http://docs.basex.org/wiki/Json_Module > [2] http://docs.basex.org/wiki/XQuery_3.1#fn:json-to-xml > > > On Mon, Nov 9, 2015 at 4:19 AM, Florent Gallaire <fgallaire(a)gmail.com> wrote: >> I would like to use the new "basic" JSON via REST. >> With basex 8.3.1 and 8.3.2beta, >> my_request&method=json&json=format=jsonml >> works fine, but >> my_request&method=json&json=format=basic >> returns : >> [FOJS0006] Element 'article' has invalid namespace: ''. >> >> What's the problem ? >> >> Best regards >> >> Florent >> >> -- >> FLOSS Engineer & Lawyer -- FLOSS Engineer & Lawyer -- FLOSS Engineer & Lawyer

2 3

Re: [basex-talk] Sub : Adding Special Characters to the Collection Database
by Christian Grün 09 Nov '15

09 Nov '15

Hi Madhan, > I am facing the problem of adding special characters to the collection > Database in below cases, What do you mean by "problem", and what are "special characters"? Christian > Case-1: if the xml file size is huge(in GBs),at the time the collection > Database creation tacking more time, > Case-2: if the more no.of xml files are adding to the particular collection > Database continuously > > Example: > > in Xml file <MyTag myAttribute="10" ...</MyTag> > > The values are adding like the following > in collection <MyTag myAttribute="A" ...</MyTag> > > Sample Collection adding code : > -------------------------------------------- > /* > This method takes the xml files list as input and process the list > Create the xml collection DB, adds xml to the collection if already exists > replace it > */ > > public static void createCollection(ArrayList<String> listFiles){ > ClientSession clientSession = null; > String strIndexedFile=""; > try { > clientSession = new ClientSession("<<IP_ADDRESS>>", 1984, > "<<USER>>", "<<PWD>>"); > String strDbName = "Sample_Collection"; > String strKeysFileLocation = > "D:\\TEMP\\7GB_Collection_TestingTeam\\Files"; > > String res = clientSession.execute("XQUERY db:exists('" + > strDbName + "')"); > //DB existence checking > if (!res.equalsIgnoreCase("true")) { > clientSession.execute("CREATE DB " + strDbName); > } else { > clientSession.execute("OPEN " + strDbName); > } > > for (int i = 0; i < listFiles.size(); i++) { > strIndexedFile = listFiles.get(i); > String tempFlag=clientSession.execute("XQUERY db:exists('" + > strDbName + "','" + strIndexedFile + "')"); > clientSession.execute("ADD " + strKeysFileLocation + > File.separator + strIndexedFile); > } > System.out.println("InFormation:: > "+clientSession.execute("INFO")); > //optimizing the current opend DB using OPTIMIZE ALL command > clientSession.execute("OPTIMIZE ALL"); > System.out.println("InFormation:: > "+clientSession.execute("INFO")); > > } catch (Exception e) { > System.out.println("Exception files DB Updating: " + > e.getMessage()); > e.printStackTrace(); > } finally { > if (clientSession != null) { > try { > clientSession.close(); > } catch (IOException ex) { > ex.printStackTrace(); > } > } > } > } > > Hardware Specifications of Basex Sever : > ---------------------------------------------------------- > Operating System : Red Hat Enterprise Linux 64bit > OS Version : Linux 2.6.32 > Hard Disk : 500 GB > Memory Capacity : 16 GB Ram > Processor Family : Intel (R) Xeon(R) CPU E5 -2680 V2 > Processor Speed : 2.80 GHz > Physical Processors : 8 > Virtual Processors/cores : 4 > Server Version : basex 7.9 > -- > Thanks & Regards > Madhangopal.P

1 0

basic JSON problem
by Florent Gallaire 09 Nov '15

09 Nov '15

I would like to use the new "basic" JSON via REST. With basex 8.3.1 and 8.3.2beta, my_request&method=json&json=format=jsonml works fine, but my_request&method=json&json=format=basic returns : [FOJS0006] Element 'article' has invalid namespace: ''. What's the problem ? Best regards Florent -- FLOSS Engineer & Lawyer

2 1

Keeping a database optimized
by Paul Swennenhuis 05 Nov '15

05 Nov '15

What is the best way to keep the indexes of a database up-to-date? I have created a database with the option UPDINDEX set to ON (and db-info tells me that it is indeed switched on) and I have also created text and attribute indexes. Even so, after several insertions (using XQUERY insert node) I see that the UPTODATE flag is no longer checked and I have to run OPTIMIZE ALL to get back my performance. a) Is there a way to keep UPTODATE checked under all circumstances? And if not b) Is is advisable to run OPTIMIZE ALL after every insertion? (I guess not)

2 4

PHP client update
by Paul Swennenhuis 04 Nov '15

04 Nov '15

Hi all, I wrote an update for the BaseX PHP client, to handle the Digest authentication method as introduced in BaseX 8.0 See attachment. Regards, Paul

1 0

request concerning unittest in the GUI
by Rob Stapper 02 Nov '15

02 Nov '15

Hi Christian, For debugging my xquery-code I using the test-facilities from the unit-module in combination with trace and prof:dump command in the code for code-tracing. The problem is that when a unit-test fails it overwrites the trace-output in the Query-info window. So the code-tracing-info is gone just when I need it. It would make debugging a lot easier if the trace-output is appended after the unit-test-output in the Info-window of the GUI. Best, Rob --- Dit e-mailbericht is gecontroleerd op virussen met Avast antivirussoftware. https://www.avast.com/antivirus

2 5

Best way to batch up a set of DB creates, updates and queries?
by Hondros, Constantine (ELS-AMS) 01 Nov '15

01 Nov '15

I'm looking for the best way to batch up a series of updating operations into a single pipeline that can be run at once. To give you an idea, we have to create multiple databases, perform a number of updating post-processes on each, before finally pulling certain fields out. I've got a lot of parameters that need to externalised somehow - database creation options, location of DTDs, XSDs, Catalogs, and unless I am trying this the wrong way I am finding it quite difficult. I can't write it all XQuery - I have a mixture of updating and non-updating functions. I can't get much traction out of the BXS scripting syntax - for example, I can't seem to create and use variables using this. What do other people out there use to glue these sorts of operations together? TIA, Constantine ________________________________ Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.

2 5

WebDav issues and auth clarification
by Régis Robineau 30 Oct '15

30 Oct '15

Hi all, I'm a beginner in BaseX and I'd need some clarification about the setup and configuration, especially about authentication process. The context is: - Debian 8 Jessie, with Basex 8.3 manually installed (i.e. not from APT). I'm using Apache in front of Tomcat to access web apps (through mod_proxy and mod_proxy_ajp): ProxyPass /BaseX83 ajp://localhost:8009/BaseX83 ProxyPassReverse /BaseX83 ajp://localhost:8009/BaseX83 - Webapp (WEBPATH): "/var/lib/tomcat7/webapps/BaseX83" - Data (DBPATH) : "/var/data/basex" (owned by tomcat7 user) - Client/server binaries (BaseX83): "/opt/basex/bin/" - conf file: "/opt/basex/.basex" Here are the main variables in my ".basex" file: DBPATH = /var/data/basex REPOPATH = /var/lib/tomcat7/webapps/BaseX83/repo HOST = localhost PORT = 1984 SERVERPORT = 1984 PROXYPORT = 80 WEBPATH = /var/lib/tomcat7/webapps/BaseX83 HTTPLOCAL = false STOPPORT = 8985 AUTHMETHOD = Basic And in my "WEB-INF/web.xml" I've set: - the value of "org.basex.dbpath" to reflect my local path (i.e. "/var/data/basex") - a new "org.basex.password" for the "org.basex.user" (admin) I have access to dba and REST service but I encounter various issues regarding authentication and WebDAV: 1/ I can't connect to WebDav service using the Finder (on Mac OS X 10.8.5) when "org.basex.authmethod" is set to "Digest". WebDav only works with the "Basic" auth method 2/ When the WebDav folder is mounted in the Finder, I can't copy files into it (permissions issues?). I've both tested as admin and as a test user with read/write permissions to the database... An empty file is created (zero byte, no content). But on the other hand, I'm able to access the REST interface with the same users (admin and test). I've created the test user like so: $ ./basexserver // in a second tab: $ ./basexclient -w > CREATE DB testdb > CREATE USER test password > GRANT READ ON testdb TO test > GRANT WRITE ON testdb TO test > SHOW USERS ON testdb 3/ The "org.basex.password" in WEB-INF/web.xml has no effect. The only way to define a password for admin is to use the console (ALTER PASSWORD admin password). Moreover the "service-specific credentials" for WebDav has no effect either. It just don't work. You can only connect to WebDav with the user credentials you defined in basex console (hence listed in users.xml). So what is the purpose of the "org.basex." settings? Does anybody ever faced the same issues with WebDav? Should I downgrade to 7.9 ? Thanks for your help. Regis

2 2

Creating Tim, ADate, Dtn, Map programatically
by James Wright 30 Oct '15

30 Oct '15

I have some Java methods which return time elements and or Maps and I was trying to convert these into XQuery atomic values with no luck. Is there a utility for these? Array has ``fromValues`` and all other types like ``Str`` have ``get`` but Map, Tim, Dtn, ADate do not appear to be construct-able. Any help would be appreciated. - James

2 1

BasexGui
by meumapple 28 Oct '15

28 Oct '15

Hi All, I love using the gui but I have a hard time dealing with autocomplete: is there a way to disable it in the txt editor? I think that it would also be very useful to allow view of whitespaces and tabs and have a function to prettify the code itself (do you know any for xquery?) Thank you very much for your work! Joseph

6 27

repo install issue
by Andy Bunce 28 Oct '15

28 Oct '15

Hi, The following installs in the repository, and can be used: " https://raw.githubusercontent.com/james-jw/xq-mustache/master/src/xq-mustac… " !repo:install(.) But it does not appear the repo:list() or the UI repository list.. I suspect the issue is the form of the namespace module namespace mustache = 'http://xq-mustache'; Regards /Andy

2 4

Change basex home in the same execution enviroment
by Antonio Mele 28 Oct '15

28 Oct '15

Hello, I am using basex version 8.3-SNAPSHOT in a java stand alone application. After i set the property org.basex.path with the path of my DB (ex. /mnt/test/db_1) I call a new Context and then CreateDB. It creates the folder /mnt/test/db_1/data/db_1 After that (in the same execution enviroment) I change the property to another DB (ex. /mnt/test/db_2). It creates the folder /mnt/test/db_1/data/db_2 I would like to create the db_2 in the path /mnt/test/db_2/data/db_2. Follow the code I wrote, where inputFolder is a folder with inside some xml files Thanks in advance. System.setProperty("org.basex.path", "/mnt/test/db_1"); Context context = new Context(false); new CreateDB("db_1", inputFolder).execute(context); context.close(); System.setProperty("org.basex.path", "/mnt/test/db_2"); context = new Context(false); new CreateDB("db_2", inputFolder).execute(context); context.close();

2 3

Re: [basex-talk] Full-text index with lots of data
by Chuck Bearden 21 Oct '15

21 Oct '15

Sorry for not using "Reply All" earlier. Setting FTINDEXSPLITSIZE to 20000000 enabled the process to get a little further, if the meaning of each dot is the same. FTINDEXSPLITSIZE at default: ..............................|..................................................................|..........................................................................|...............................................................................|.. FTINDEXSPLITSIZE = 20000000 .......|.......|........|.......|......|........|.............|.............|.............|.............|.............|.............|.............|.............|..............|.............|.............|.............|.............|.............|.............|............ If it's a matter of making the indexing process take longer, that's not a problem. Thanks, Chuck On Tue, Oct 20, 2015 at 1:27 PM, Chuck Bearden <cfbearden(a)gmail.com> wrote: > Thanks Christian, I'll try the FTINDEXSPLITSIZE option. > > I'm also open to modifying the XML files it that would help. Because > of limitations of the service from which we harvest them RESTfully, I > have only 20 actual content elements in each file. If you think it > would make a difference, I could consolidate them to have, say, 200 or > 500 of the actual content elements per file, but I have no idea if > that would change how the indexing falls out. > > The files also have structures where some properties of each record > are each represented by a URL, and ID value, and a string. I could > XSLT the files to remove all but the string (human readable is better > for our purposes) to make them less verbose. > > BaseX is really super for doing data quality assessments of the XML, > and if we could get full-text indexing working, it would speed things > up considerably. Thanks to you & your team for all the work you've put > in to the application! > > Alles Gute > Chuck Bearden > > On Tue, Oct 20, 2015 at 12:55 PM, Christian Grün > <christian.gruen(a)gmail.com> wrote: > > I see; it seems that the index creation is failing at the very final > > step, in which partial index structures, which are temporarily written > > to disk, are merged. > > > > You could either to increase Xmx even more (to 6 or 7G?). If this > > doesn't work, you could try assign different values to the > > FTINDEXSPLITSIZE option [1] (start e.g. with 20000000). > > > > Sorry for the trouble. Feel free to keep me updated, maybe we find a > > way to fix this, > > Christian > > > > [1] http://docs.basex.org/wiki/Options#FTINDEXSPLITSIZE > > > > > > On Tue, Oct 20, 2015 at 7:48 PM, Chuck Bearden <cfbearden(a)gmail.com> > wrote: > >> Here's the stack trace: > >> > >> > =.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.= > >>> create db pure_20151019 pure_20151019 > >> Creating Database... > >> > ..;..;..;..;..;..;.;..;..;..;..;..;.;..;.;.;.;.....;.....;.....;......;.....;.....;.......;.;.;;.;.;;.;.;;.;.;;.;.;;.;.;;.;.................................................;..........................................................;..........................................................;..........................................................;..........................................................;..........................................................;................................................... > >> 677584.62 ms (1435 MB) > >> Indexing Text... > >> > ........................................................................................................................................................................................................................................................... > >> 98215794 operations, 178526.99 ms (1611 MB) > >> Indexing Attribute Values... > >> > ........................................................................................................................................................................................................................................................... > >> 178304119 operations, 135613.26 ms (2005 MB) > >> Indexing Full-Text... > >> > ..............................|..................................................................|..........................................................................|...............................................................................|..java.lang.OutOfMemoryError: > >> Java heap space > >> at org.basex.index.ft.FTList.next(FTList.java:93) > >> at org.basex.index.ft.FTBuilder.merge(FTBuilder.java:236) > >> at org.basex.index.ft.FTBuilder.write(FTBuilder.java:140) > >> at org.basex.index.ft.FTBuilder.build(FTBuilder.java:85) > >> at org.basex.index.ft.FTBuilder.build(FTBuilder.java:23) > >> at org.basex.data.DiskData.createIndex(DiskData.java:187) > >> at org.basex.core.cmd.CreateIndex.create(CreateIndex.java:103) > >> at org.basex.core.cmd.CreateIndex.create(CreateIndex.java:91) > >> at org.basex.core.cmd.CreateDB.run(CreateDB.java:104) > >> at org.basex.core.Command.run(Command.java:398) > >> at org.basex.core.Command.execute(Command.java:100) > >> at org.basex.api.client.LocalSession.execute(LocalSession.java:132) > >> at org.basex.api.client.Session.execute(Session.java:36) > >> at org.basex.core.CLI.execute(CLI.java:103) > >> at org.basex.core.CLI.execute(CLI.java:87) > >> at org.basex.BaseX.console(BaseX.java:191) > >> at org.basex.BaseX.<init>(BaseX.java:166) > >> at org.basex.BaseX.main(BaseX.java:42) > >> org.basex.core.BaseXException: Out of Main Memory. > >> You can try to: > >> - increase Java's heap size with the flag -Xmx<size> > >> - deactivate the text and attribute indexes. > >> at org.basex.core.Command.execute(Command.java:101) > >> at org.basex.api.client.LocalSession.execute(LocalSession.java:132) > >> at org.basex.api.client.Session.execute(Session.java:36) > >> at org.basex.core.CLI.execute(CLI.java:103) > >> at org.basex.core.CLI.execute(CLI.java:87) > >> at org.basex.BaseX.console(BaseX.java:191) > >> at org.basex.BaseX.<init>(BaseX.java:166) > >> at org.basex.BaseX.main(BaseX.java:42) > >> Out of Main Memory. > >> You can try to: > >> - increase Java's heap size with the flag -Xmx<size> > >> - deactivate the text and attribute indexes. > >>> d > >> > =.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.= > >> > >> Here's how the process looked in the output of 'ps -ef', in case > >> that's relevant: > >> > >> > =.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.=.= > >> cfbeard+ 88769 88757 46 12:15 pts/7 00:00:24 java -cp > >> > /home/cfbearden/opt/basex-8.3.0/BaseX.jar:/home/cfbearden/opt/basex-8.3.0/lib/basex-api-8.3.jar:/home/cfbearden/opt/basex-8.3.0/lib/basex-xqj-1.5.0.jar:/home/cfbearden/opt/basex-8.3.0/lib/commons-codec-1.4.jar:/home/cfbearden/opt/basex-8.3.0/lib/commons-fileupload-1.3.1.jar:/home/cfbearden/opt/basex-8.3.0/lib/commons-io-1.4.jar:/home/cfbearden/opt/basex-8.3.0/lib/igo-0.4.3.jar:/home/cfbearden/opt/basex-8.3.0/lib/jansi-1.11.jar:/home/cfbearden/opt/basex-8.3.0/lib/javax.servlet-3.0.0.v201112011016.jar:/home/cfbearden/opt/basex-8.3.0/lib/jdom-1.1.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-continuation-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-http-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-io-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-security-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-server-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-servlet-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-util-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-webapp-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jetty-xml-8.1.17.v20150415.jar:/home/cfbearden/opt/basex-8.3.0/lib/jing-20091111.jar:/home/cfbearden/opt/basex-8.3.0/lib/jline-2.13.jar:/home/cfbearden/opt/basex-8.3.0/lib/jts-1.13.jar:/home/cfbearden/opt/basex-8.3.0/lib/lucene-stemmers-3.4.0.jar:/home/cfbearden/opt/basex-8.3.0/lib/milton-api-1.8.1.4.jar:/home/cfbearden/opt/basex-8.3.0/lib/mime-util-2.1.3.jar:/home/cfbearden/opt/basex-8.3.0/lib/slf4j-api-1.7.12.jar:/home/cfbearden/opt/basex-8.3.0/lib/slf4j-simple-1.7.12.jar:/home/cfbearden/opt/basex-8.3.0/lib/tagsoup-1.2.1.jar:/home/cfbearden/opt/basex-8.3.0/lib/xmldb-api-1.0.jar:/home/cfbearden/opt/basex-8.3.0/lib/xml-resolver-1.2.jar:/home/cfbearden/opt/basex-8.3.0/lib/xqj2-0.2.0.jar:/home/cfbearden/opt/basex-8.3.0/lib/xqj-api-1.0.jar: > >> -Xmx4g org.basex.BaseX -d > >> > >> > >> On Tue, Oct 20, 2015 at 12:38 PM, Chuck Bearden <cfbearden(a)gmail.com> > wrote: > >>> It hasn't failed yet; I've gotten the progress indicators, along with > >>> the phases that have been completed: > >>> > >>> Creating Database... > >>> Indexing Text... > >>> Indexing Attribute Values... > >>> > >>> It's still working on "Indexing Full-Text...". I'll post whatever I > >>> get when it fails. Maybe it won't this time :) > >>> > >>> Chuck > >>> > >>> On Tue, Oct 20, 2015 at 12:33 PM, Christian Grün > >>> <christian.gruen(a)gmail.com> wrote: > >>>>> Creating Database... > >>>>> ..;..;..;..;..;..;.;..;.. > >>>> > >>>> Do you get any output after this line (I would expected to see a stack > >>>> trace, or at least an error message…)? > >>>> > >>>> > >>>> > >>>>> Where 'pure_20151019' is both the name of the database and the > >>>>> subdirectory where all my XML files are. > >>>>> > >>>>> It could well be that I'm missing a crucial option; I'm still > >>>>> relatively new to BaseX. It's great stuff, though. > >>>>> > >>>>> Because of my employer's IT environment, I have to run my Linux > >>>>> workstation in a VMWare VM, though I doubt that that makes a > >>>>> difference. > >>>>> > >>>>> Thanks, > >>>>> Chuck > >>>>> > >>>>> On Tue, Oct 20, 2015 at 11:15 AM, Christian Grün > >>>>> <christian.gruen(a)gmail.com> wrote: > >>>>>> Hi Chuck, > >>>>>> > >>>>>> Usually, 4G is more than enough to create a full-text index for 16G > of > >>>>>> XML. Obviously, however, that's not the case for your input data. > You > >>>>>> could try to distribute your documents in multiple database. As as > >>>>>> alternative, we could have a look at your data and try to find out > >>>>>> what's going wrong. You can also use the -d flag and send us the > stack > >>>>>> trace. > >>>>>> > >>>>>> Best, > >>>>>> Christian > >>>>>> > >>>>>> > >>>>>> On Tue, Oct 20, 2015 at 4:19 PM, Chuck Bearden <cfbearden(a)gmail.com> > wrote: > >>>>>>> Hi all, > >>>>>>> > >>>>>>> I have about 16G of XML data in about 52000 files, and I was > hoping to > >>>>>>> build a full-text index over it. I've tried two approaches: enable > >>>>>>> full-text indexing as I create the database and then loading the > data, > >>>>>>> and creating the full-text index after loading the data. If I > enable > >>>>>>> ADDCACHE and modify the basex shell script to use 4g of RAM > instead of > >>>>>>> 512M, I have no problem loading the data. If I try to load with > >>>>>>> FTINDEX or create the index afterward, the process runs out of > memory. > >>>>>>> > >>>>>>> I could believe that I'm overlooking some option that would make > this > >>>>>>> possible, but I suspect I just have too much data. I welcome your > >>>>>>> thoughts & suggestions. > >>>>>>> > >>>>>>> All the best, > >>>>>>> Chuck Bearden >

2 5

Circumvent authentication
by Marco Lettere 21 Oct '15

21 Oct '15

Dear all, is there a possibility to pass the Authorization header ahead to a restxq function without having it managed by BaseX itself? In a proof of concept project we have currently to face an authorization scheme which is not Basic or Digest. Unfortunately we always get an "Unsupported Authorization" exception. Workaround now is to place a proxy in front of Basex and transform the Authorization header into something like X-Authorization. Anyway I'd really like our code to be standard compliant. Thanks for any support as usual! Regards, Marco.

2 6

Full-text index with lots of data
by Chuck Bearden 20 Oct '15

20 Oct '15

Hi all, I have about 16G of XML data in about 52000 files, and I was hoping to build a full-text index over it. I've tried two approaches: enable full-text indexing as I create the database and then loading the data, and creating the full-text index after loading the data. If I enable ADDCACHE and modify the basex shell script to use 4g of RAM instead of 512M, I have no problem loading the data. If I try to load with FTINDEX or create the index afterward, the process runs out of memory. I could believe that I'm overlooking some option that would make this possible, but I suspect I just have too much data. I welcome your thoughts & suggestions. All the best, Chuck Bearden

2 3

Optimized query is empty when database is open and query uses a variable
by Amanda Galtman 20 Oct '15

20 Oct '15

Hi, When I run the following query with BaseX 8.2.3 or 8.3, and I have a certain database open, the optimized query is empty, so there are no results. let $x := 'x' let $doc := <root><el attr="x;y"/></root> return $doc//el[contains(@attr,$x)] This seems like a bug. If I close the database (which is not even being referenced in this query), open a different database, or replace $x in the last line with the literal string value, 'x', then I get the result I expect. The database that exhibits the problem and the database that doesn't exhibit the problem differ in the kinds of XML files in them and the indexes that are available for them. Does anyone know why the open database should affect the result of a query that doesn't reference the database? Thanks, Amanda

2 3

Which zip did BaseX choke on?
by Hondros, Constantine (ELS-AMS) 19 Oct '15

19 Oct '15

I love BaseX for the simplicity it brings to XML handling. But this is a problem I have not encountered before. I am creating a DB from about 17,000 small zipfiles, each containing a directory structure and somewhere within each, some XML. BaseX chokes on one of these files giving the error: "invalid entry size (expected 0 but got 11083 bytes)". So clearly one or more of the zips is invalid - but which one(s)? Is there any way that BaseX can echo to me the file that is causing the error? DEBUG is set to TRUE, but I don't get any morer verbose output. I am using the GUI, and running a BSX script too. Thanks in advance, C. ________________________________ Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.

4 6

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk