BaseX-Talk

basex-talk@mailman.uni-konstanz.de

6 participants
5269 discussions

archive:extract-to out of memory
by Lizzi, Vincent 11 Oct '15

11 Oct '15

I've been using the new method archive:extract-to almost since it became available in BaseX, and it generally works very well. I've run into trouble with an out of memory error a few times trying to extract .zip files that are fairly large. For example, the .zip file I'm looking at now is 268 Mb. The out of memory error seems to actually be thrown on the file:read-binary method, which must be passed as input to archive:extract-to. Increasing the amount of memory available to BaseX by increasing -Xmx from 512 to 768 helps, but that doesn't seem like a good solution. For example, using the latest BaseX (BaseX831-20150929.165521.zip), a 268 Mb .zip file, a smaller 65 Mb .zip file, and the following code: declare namespace _ = "test"; declare function _:unzip1($zip as xs:anyURI) as xs:anyURI { let $tempDir := file:create-temp-dir('basex', 'unzip') return ( archive:extract-to($tempDir, file:read-binary($zip)), xs:anyURI($tempDir) ) }; declare function _:unzip2($zip as xs:anyURI) as xs:anyURI { let $tempDir := file:create-temp-dir('basex', 'unzip') return ( let $entries := zip:entries($zip) for $e in $entries//zip:entry let $path := string-join(($e/parent::zip:dir/@name, $e/@name), '/') return ( file:create-dir(file:parent($tempDir || $path)), file:write-binary($tempDir || $path, zip:binary-entry($zip, $path)) ), xs:anyURI($tempDir) ) }; declare function _:unzip3($zip as xs:anyURI) as xs:anyURI { let $tempDir := file:create-temp-dir('basex', 'unzip') return ( for $f in file:read-binary($zip) return archive:extract-to($tempDir, $f), xs:anyURI($tempDir) ) }; let $zip := xs:anyURI('C:\temp\bigfile.zip') let $unzipped := _:unzip2($zip) (: change this to test 1 2 3 :) return $unzipped When given a small .zip file _:unzip1, _:unzip2, and _:unzip3 extract all files from the .zip. _:unzip1 and _:unzip3 seem to complete in less time than _:unzip2. When given a large .zip file, _:unzip2 and _:unzip3 work but _:unzip1 produces an out of memory error. However, when I use the _:unzip1, _:unzip2 or _:unzip3 code in a larger program, which does some processing on the unzipped file, I still get an out of memory error and can see that not all files were extracted. Any help would be appreciated. Thanks, Vincent

2 1

Hitting index for multi-db query
by Marco Lettere 11 Oct '15

11 Oct '15

Hello all, I have to query all the databases on one server for docs matching a query. I do it with this query for example: db:list() ! db: open(.)/entry[@about='abc'] This query never hits the available attribute index. I could rewrite the query to something like db:list() ! attribute(., 'abc', 'about')/.. and in this case it takes advantage of the index of course. The problem is that I have to implement more queries which have much more generic predicate structures where I probably won't be able to use indexes explicitly. Is there any possibility to rewrite this query in order to use the indexes implicitly? Thanks, Marco.

2 3

Java from XQuery
by meumapple 09 Oct '15

09 Oct '15

Hi All, It is not clear to me if I can run from XQuery a .jar file (it is a simple linguistic parser). Any idea about that? Thanks. Joseph

3 7

ADD command
by Julien Charon 09 Oct '15

09 Oct '15

Hi, I have some trouble using the ADD command, but maybe I'm not understanding the command correctly. We have a java application that runs basex (7.7) embedded. I'm using the Add class/command to add an XML file to the database. After that, I'd like to delete that XML file from the local file system because I assume it has somehow be imported or copied to my database. However, from within my java application, I am not able to delete the file (File.delete() will always return false and the file won't be deleted). I tried the same thing with basex GUI: create a DB, add multiple files, delete those files (while DB is still open), do some queries (which works fine), close the DB, reopen it, do some queries again (which still works fine). Do I miss or misunderstand something? Do I need to release the XML file resource when using the java API? Any help would be appreciated. Mit freundlichen Grüßen / With kind regards, Julien Charon Avitech GmbH Engineering AxL Tel.: +49 (0)7541/282-177 Fax: +49 (0)7541/282-199 e-mail: julien.charon(a)avitech.aero<mailto:julien.charon@avitech.aero> ________________________________________________ Avitech GmbH Principal Office: Bahnhofplatz 1 | 88045 Friedrichshafen | Germany Court Registration: Amtsgericht Ulm | HRB 728293 Geschäftsführer/Managing Director: Antonio Maria Gonzalez Gorostiza http://avitech.aero<http://avitech.aero/> This message may contain confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

2 2

Origami presentation XML Amsterdam
by Marc van Grootel 04 Oct '15

04 Oct '15

Hi, Anyone on the list coming to Amsterdam the 6th of November? We may have some beers together. I will (finally) be presenting on the Origami templating library. I'm still working on it over on Github (in the unstable branch). After a pause in development it was almost completely rewritten. I intend to cut a new release just before the conference. Here's the anouncement I made for the conference website. XQuery 3.1 is still a candidate for recommendation by the W3C. Nevertheless there are already various implementations ready for use today. The introduction of maps and arrays, and the availability of functional weaponry such as fold and apply may change the way you write XQuery applications. You can still wield the trusty weapons of XPath and the FLWOR expression while, at the same time, adopting a more functional style of programming. Origami is an ongoing exploration of templating in XQuery and it makes heavy use of this functional style. It started with a quest for a more XSLT-like transformation capability and recently took a nice turn with the adoption of a flavor of micro-XML, with a heavy reliance on maps and arrays. Origami's unfolding design tries to honour principles of simplicity and composability. With Origami I aim to decomplect, or disentangle, several aspects common to templating problems and offer them in a powerful library using a couple of simple abstractions. With some practical code examples I will demonstrate how to use Origami to implement different templating styles. --Marc

2 1

BaseX 8.3 basexhttp - Class path contains multiple SLF4J bindings
by Neven Jovanović 03 Oct '15

03 Oct '15

Hello, in a fresh installation of BaseX 8.3, when running the basexhttp script (./bin/basexhttp -S from the basex directory) on Linux Mint Debian edition (Linux Mint LMDE 64-bit), I get the following error message: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/neven/Downloads/basex/lib/slf4j-simple-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/neven/Downloads/basex/lib/slf4j-simple-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory] Connection failed (port: 8984). Previously the same system ran BaseX 8.2, and everything was fine. Reinstalling 8.2 also brings everything back to normal. Is this my local problem, or something else? Best, Neven Neven Jovanovic Zagreb, Croatia

2 1

about xi:include and xml:base
by Marco Randazzo 01 Oct '15

01 Oct '15

In my application I have a main xml file (general.xml) which includes several xml files (file1.xml, file2.xml...) using the xi:include tag. I successfully imported general.xml in basex using the gui: the xi:include directives were successfully processed, merging all data in a single database. I noticed in the result window that the database keeps track of the origin of the data, adding an xml:base info for each included block. I then performed some find and replace operation on the database, modifying some values which were originally belonging to different files. Now I want to export data to file. I successfully obtained a single xml file using the export command of the gui. The generated file still preserves the xml:base infos. However, I'd like to recreate the original structure, i.e. obtain multiple files, the same which were merged by xi:include directives at the beginning of the import process. I cannot find such an option in the export tab. Is it possible? Thank you for the help, MR

2 1

JSON serializer - slight bug
by Hans-Juergen Rennau 30 Sep '15

30 Sep '15

Dear BaseX, I think there is a very slight bug in the JSON serializer: when an element with type="object" has a <_> child, I think the serializer should report an error (as underscores must be doubled, if not using the name "_", reserved for array items. Instead, the serializer creates a property with the empty string as name. Example: json:serialize( <json type="object"> <BaseX type="object"> <_>is phantastic</_> </BaseX> </json> )=>{ "BaseX":{ "":"is phantastic" } } Compare with: json:serialize( <json type="object"> <BaseX type="object"> <__>is phantastic</__> </BaseX> </json> )=>{ "BaseX":{ "_":"is phantastic" } } It may look like pedantry, but this misbehaviour may cause confusion, as it is of course an interesting practise to create JSON by creating the XML format and in the end serializing: if then inspecting the end result, it is not immediately clear that the "":... is caused by the serializer, rather than by own code. Kind regards,Hans-Jürgen

2 2

crypto module with binary files
by Michel Banguerski 29 Sep '15

29 Sep '15

Greetings all, I would like to perform cryptographic operations with arbitrary binary files but am unable to figure how to convert the output of file:read-binary() to something that could be processed by crypto:hmac() for example. Can You give me a hint please? TIA Best regards Michel

2 1

Java access to JSON-to-XML transformation
by Hans-Juergen Rennau 29 Sep '15

29 Sep '15

Dear BaseX team, I have a question concerning the availability of the JSON-to-XML transformation. Before asking, this remark: I am very glad and grateful that BaseX offers this functionality (json:parse), as it exposes JSON to XPath, which enables important evaluations that otherwise would not even be attempted. And now my question. Is it possible - does it appear reasonable - to access the functionality within a Java application by calling the BaseX code (org.basex.build.json.JsonParser, I suppose)) immediately, without launching XQuery? I expect that one would be treading dangerous ground, relying on internal APIs which may change any time, not being meant for external use. If this is so, a brief confirmation would suffice to make me drop the idea. Thank you, and kind regards -Hans-Jürgen PS: Background of my question - I evaluate alternative possibilities how to validate JSON within Java applications. One (and probably the best) option is validating against JSON schema (json-schema.org), an alternative would be a to-XML transformation followed by XSD validation. A first experiment lettig BaseX via XQJ execute the epic query "json:parse(.)" revealed a performance which would not be sufficient to be competitive. So I wondered if removing the overhead of launching BaseX might be an approach to consider.

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk