- BaseX-Talk - mailman.uni-konstanz.de

GUI Textindex: false although true
by Benjamin Gathmann 16 Dec '15

16 Dec '15

I use the GUI of the current release 8.3.1. I have played around with the options in BaseX to test the speed of CREATE and ADD operations. Now the GUI "Open and manage" box shows me for any DB I create: Up-to-date: true TEXTINDEX: false ATTRINDEX: false However, I have never set TEXTINDEX or ATTRINDEX to false, and the files in the "data" directory suggest that the indexes are there, as does a "GET TEXTINDEX" (returns "true"). I only made the following changes: set updindex true; set autooptimize true; set autoflush false; -- joe-security-logo Joe Security LLC Benjamin Gathmann, Developer & Support business parc Reinach, Christoph Merian-Ring 11, 4153 Reinach, Switzerland benjamin.gathmann(a)joesecurity.org <mailto:benjamin.gathmann@joesecurity.org>, http://www.joesecurity.org ------------------------------------------------------------------------ The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.

2 2

Node Maximum 2^31
by Benjamin Gathmann 16 Dec '15

16 Dec '15

I read here that the current maximum number of nodes in a BaseX DB is 2^31 (2'147'483'648): http://docs.basex.org/wiki/Statistics Is this related to 32bit CPU architecture? And are there plans for a 64bit version of BaseX that would allow larger databases? I am asking because I am testing BaseX with large XML files (usually 5 to 15MB each) with a very deep nested structure and many small elements. Right now, I have created a DB with 3979 such documents. DB Size: 29,697 MB Nodes: 1,172,761,158 This means that I will reach the max. #nodes already with around 5500 reports and a DB size of 55 GB. This in turn will mean that I will have to split my documents on rather many database instances if I import e.g. 50,000 documents, which worries me a bit. Or do I worry with no reason?

2 1

Using external variable to perform node insert
by Mike Engledew 16 Dec '15

16 Dec '15

We are attempting to insert some new elements with attributes into an XML document from a string. The elements have a name space prefix such as asy: If I use an external variable containing XML to perform a node insert example :- declare namespace asy="http://xml.acme.com/asy"; declare variable $part external; insert nodes $part as first into db:open("PARTDB")/asy:assembly[@name="ZB09010"] The characters in the xs:string $path reserved for XML e.g. < > etc are converted to < > etc and the insert silently fails. if I change the insert statement to : insert nodes fn:parse-xml($part) as first into db:open("PARTDB")/asy:assembly[@name="ZB09010"] I get the message FODC0006 (Line 1): The prefix "asy" for element "asy:part" is not bound. If I add declare namespace asy="http://xml.acme.com/asy"; at the beginning of the $path variable contents before the XML I get FODC0006 (Line 1): Content is not allowed in prolog. If I then change the statement to : insert nodes xquery:eval($part) as first into db:open("PARTDB")/asy:assembly[@name="ZB09010"] The insert works as expected. Note that if the same XML is defined within the query using let $path := < new xml stuff > The insert works fine just using $path with no functions applied. Is this behavior correct and is using xquery:eval($part) the right approach or am I just lucky it works and I should have been doing something different? Thanks for any insight you can offer.

2 3

Re: [basex-talk] Extracting binary stored zip into database as binary stored elements not working
by Christian Grün 12 Dec '15

12 Dec '15

> In the meantime can you shed some light into the > [FODC0002] Resource > 'delta.zip' does not exist.' error? (cc to the list): It simply says that a database resource does not exist. Maybe you have more than one BaseX database directory in your environment? Could you try db:list('...db...') and check what results are returned in the GUI and via the client? > On Fri, Dec 11, 2015 at 9:53 AM, Christian Grün <christian.gruen(a)gmail.com> > wrote: >> >> > The raw stored file was slightly larger than the original zip. >> >> Do you have an example for that? >> >> > But when I submit the exact same query using the Java client I get the >> > following error: >> > >> > Stopped at ., 1/216: >> > [FODC0002] Resource 'delta.zip' does not exist. >> >> It shouldn’t make a difference. I’ll probably need more info to track this >> down. > >

2 6

Extracting binary stored zip into database as binary stored elements not working
by E. Wray Johnson 11 Dec '15

11 Dec '15

My zip file has files nested in subfolders. First I used db:store to store delta.zip into the database. My first problem was I got an error " *invalid entry CRC: expected 0x0 but got..."* The raw stored file was slightly larger *than the original zip.* So I replaced the zip file in the raw folder with the original one and the first error went away. Now I don't get any error bu the extracted file are not stored as binaries What is wrong with this code? I removed the updating annotation and returned $target instead of the try/store/catch inside the loop and got the expected result path name let $database := 'db' let $extractTo := 'target.zip' (: path to raw stored zip :) let $archivePath := 'alpha/bravo/charlie/delta.zip' let $archive := db:retrieve($database, $archivePath) let $basePath := replace($archivePath, '[^/]+$', '') || $extractTo let $entries := archive:entries($archive) let $contents := archive:extract-binary($archive) return for-each-pair($entries, $contents, %updating function($entry, $content) { let $target := $basePath || '/' || $entry/text() return try { db:store($database, $target, $content) } catch * { () } })

2 2

Concurrency model
by E. Wray Johnson 11 Dec '15

11 Dec '15

What is the best way to manage multiple users in a multi-threaded environment like servlets?

2 1

DB Contention
by buddyonweb-software＠yahoo.com 11 Dec '15

11 Dec '15

I have a Java process that continually scans for incoming .xml files deposited into a system folder on the OS. When it finds a file (say a.xml), it will create a DB for that file (call it DB-A) and load the XML file into that database (DB-A).Note: DB names will be guaranteed to be unique when created, so there will never be 2 DB-A databases created. As this file is processed by a seondar, other XML files will be added into DB-A - but all documents will relate only to the processing of the a.xml file (for example, various statistics, etc.). Now, there is a secondary/separate java process, that scans the database instances that have been created with the XML file loaded. This secondary process does some querying on that file and adds additional XML documents to that database. That is only this process will add new documents to the database. So my question is this: 1. With process 1 creating and inserting the original .xml file, is there a chance for database contention or is this architecture pretty safe from contention? Note: both Java process are simply using the BaseX.jar file. 2. If I added a third Java process in the future, that would a) only access existing documents in read only mode b) could add new documents to the database that no other process would read or update, is this safe from contention? Thanks in advance.

5 8

Question about baseX add "/path/to/filesystem/directory"
by ansel.jm＠free.fr 11 Dec '15

11 Dec '15

Hello I'm experiencing a trouble with the add "/path/to/filesystem/directory" on a large directory structure with thousands of files. The result is a strange error : > add "/data/PARTAGE/DOSSIERS EC&T/PC" "Configurations2/accelerator/current.xml" (Line 1): No text allowed before root element. And in the directory /data/PARTAGE/DOSSIERS EC&T/PC, no Configurations2 directory... I think a file is having a problem but with a thousand of files, how to know which of them ? On another directory, the add function works but finds xml files that aren't in the directory structure (find . -name "xxx" doesn't find them as basex finds !!!)... Is the "add function" fragile on a directory ? Many thanks in advance JMA

2 1

Broken Links on website
by E. Wray Johnson 08 Dec '15

08 Dec '15

On: http://docs.basex.org/wiki/Java_Examples Links in the following text seem to be broken: XQJ API > > The implementation of the BaseX XQJ API <http://xqj.net/basex/> > (closed-source) has been written by Charles Foster. It uses the > client/server architecture. The basex-examples repository contains various > examples > <https://github.com/BaseXdb/basex-examples/tree/master/src/main/java/org/bas…> > on how to use XQJ. > The broken links are: - http://xqj.net/basex/ - https://github.com/BaseXdb/basex-examples/tree/master/src/main/java/org/bas…

2 2

[bxerr:BXDB0002] Too many open files
by Mansi Sheth 08 Dec '15

08 Dec '15

Hello, I am importing BaseX, with tons of XML files. Currently I have roughly 1600 databases, I am starting basexhttp service, to access it over a web service endpoint, thru a xquery file. Using BaseX 8.2.3. I am receiving below error: [bxerr:BXDB0002] Resource "/veracode/msheth/BaseXDB/bi_output_713/inf.basex (Too many open files)" not found. basexhttp, is running with 10240M virtual memory. I can share the xquery file, if thats needed. Has anyone experienced this before ? Is there a limit on no of databases supported by BaseX ? Is there some configuration option, which I can use to close already queried database ? Thanks, - Mansi

4 5