- BaseX-Talk - mailman.uni-konstanz.de

Re: [basex-talk] Memory requirements, release memory, heap shrink
by Dinu Marina 04 Nov '17

04 Nov '17

Indeed, I use the create function from the GUI, I just assumed it's the same 2 separate operations. Indeed, with CREATE DB it doesn't get out of memory at 1G. And it also gets GC'ed and returned to system afterwards with no additional intervention, after CREATE DB memory shrinks immediately back to ~30M. So confirmed, huge memory usage and memory "leak" (or whatever it is) is linked to ADD only. Thanks, Dinu On 04.11.2017 18:46, Christian Grün wrote: > Hi Dinu, > > yes, I have downloaded the file. > > Just one more question: > >> 2) using basexclient: >> >> CHECK somedb >> ADD /path/to/1_feed.zip > If you use the GUI, do you really add your zip file to an existing > database, or do you specify it as initial input when creating a new > database? The latter option is definitely more efficient, and the > command-line equivalent would be > > CREATE DB somedb /path/to/1_feed.zip > > For adding resources to existing databases, enabling ADDCACHE can help [1]. > > Cheers, > Christian > > [1] http://docs.basex.org/wiki/Options#ADDCACHE > > > >> Result: >> Out of Main Memory. >> >> To reproduce 2), start server with -Xmx2048m, repeat operations, then drop >> db, close client, check server memory usage. >> >> Thanks, >> Dinu >> >> >> >> On 04.11.2017 18:18, Christian Grün wrote: >>>> The fact is, the GUI runs with no problem with -Xmx512M to do the same >>>> thing, while basexclient fails without -Xmx2048M. >>> That’s surprising indeed – mostly because I would have expected the >>> BaseX client to always consume a small and constant amount of memory >>> (the BaseX server instance should be the process to consume all the >>> memory). I did some quick tests with large zipped input, but I failed >>> to reproduce the behavior you described. Feel free to provide me with >>> a step-by-step guide. >>> >>>> I will try that, thanks, but shouldn't this be the case automatically? >>>> Since >>>> I assume BaseX does free references to data structures, at least to a >>>> dropped DB? >>> Absolutely. Anything that’s reproducible is welcome. >>> >>> >>> >>>> On 04.11.2017 18:00, Christian Grün wrote: >>>>> Hi Dinu, >>>>> >>>>>> Question 1: >>>>> Memory consumption of the BaseX GUI is similar as on command-line, but >>>>> it may be due to garbage collection that some memory will be freed. >>>>> How do you add documents outside the GUI? >>>>> >>>>>> Question 2: >>>>> If a certain amount of memory is reserved by Java’s virtual machine, >>>>> it may still be used by other applications on your system (provided >>>>> that the memory can be freed by garbage collection). You can enforce >>>>> some GC calls by running the following XQuery expression (this should >>>>> only be done for testing purposes): >>>>> >>>>> (1 to 5) ! Q{java:java.lang.System}gc() >>>>> >>>>> Best, >>>>> Christian >>>>> >>>>> >>>>>> After the data is extracted, it's no longer needed and I DROP the DB; >>>>>> also >>>>>> connection is closed. But memory (the huge 2G mentioned above) is never >>>>>> returned to the system. >>>>>> >>>>>> The script I use to run BaseX is: >>>>>> >>>>>> export BASEX_JVM="-Xmx2048m -XX:MinHeapFreeRatio=10 >>>>>> -XX:MaxHeapFreeRatio=20 >>>>>> -XX:+UseSerialGC -Dorg.basex.LOG=false >>>>>> -Dorg.basex.DBPATH=/var/basex/data >>>>>> -Dorg.basex.REPOPATH=/var/basex/repo" >>>>>> BaseX/bin/basexserver -S >>>>>> >>>>>> So basically I tried specifying MaxHeapFreeRatio and SerialGC for java, >>>>>> but >>>>>> it's no improvement and it doesn't help so I assume the memory isn't >>>>>> hogged >>>>>> in java... is there a way to free up the memory once operations >>>>>> complete >>>>>> (like mentioned above, "complete" means created DB is dropped, >>>>>> connection >>>>>> closed, waiting for another batch to start over). >>>>>> >>>>>> Thanks, >>>>>> Dinu >>>>>>

2 11

Memory requirements, release memory, heap shrink
by Dinu Marina 04 Nov '17

04 Nov '17

Hi guys, I'm running BaseX as server; it has some processing to do every now and then, huge pieces of data. But most of the times it's idle. The "pieces of data" are huge XML files that are ADD'ed to a new DB, then read. Basically I'm using BaseX as an intermediate random-access parser/indexer for some huge XML files. Question 1: XML files are such that I need to specify -Xmx2048M, otherwise I get an out of memory error when ADDing the files. However, I notice that in the BaseX GUI adding the same XML files reports a memory usage of <300M. Is there some special option that the GUI is using that I could use too on the server so that memory usage is not so severe? Question 2: After the data is extracted, it's no longer needed and I DROP the DB; also connection is closed. But memory (the huge 2G mentioned above) is never returned to the system. The script I use to run BaseX is: export BASEX_JVM="-Xmx2048m -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:+UseSerialGC -Dorg.basex.LOG=false -Dorg.basex.DBPATH=/var/basex/data -Dorg.basex.REPOPATH=/var/basex/repo" BaseX/bin/basexserver -S So basically I tried specifying MaxHeapFreeRatio and SerialGC for java, but it's no improvement and it doesn't help so I assume the memory isn't hogged in java... is there a way to free up the memory once operations complete (like mentioned above, "complete" means created DB is dropped, connection closed, waiting for another batch to start over). Thanks, Dinu

2 3

Thanks for Enforce Index Rewriting!
by Omar Siam 03 Nov '17

03 Nov '17

This sounds like a great improvement for our code: http://docs.basex.org/wiki/Indexes#Enforce_Rewritings. Many thanks!

1 0

losses von tags?
by michael4you＠arcor.de 02 Nov '17

02 Nov '17

Hello, There is a problem in transforming data vom XML in HTML. In XML there are HTML-tags included. XML original: <dat>text is <b>important</b>.</dat> HTML goal: <div>text is <b>important</b>.</div> (1) result: <div>text is important.</div> let $data := <dat>text is <b>important</b>.</dat> return <div>{$data/data()}</div> (2) result: <div>text is important.</div> let $data := <dat>text is <b>important</b>.</dat> return <div>{$data/text()}</div> (3) result: <div><dat>text is <b>important</b>.</dat></div> let $data := <dat>text is <b>important</b>.</dat> return <div>{$data}</div> Is there another possibility to get the result desired? Thanking you, Michael

3 2

Re: [basex-talk] Rounding/parsing decimal vs float
by Marco Lettere 01 Nov '17

01 Nov '17

Thanks Kristian, unfortunately this introduces rounding errors which I'm trying to avoid because managing units, measures and monetary amounts. M. Il 01 nov 2017 12:57 PM, "Kristian Kankainen" <kristian(a)keeleleek.ee> ha scritto: You need to use xs:float or xs:double instead of xs:decimal to be able to use the 'e' or 'E' as the exponent separator. Br, Kristian K1. nov 2017 13:18 kirjutas kuupäeval Marco Lettere < m.lettere(a)gmail.com>: > > Hi all, > > I thought of asking this in parallel of hacking my own parsing procedure > ... > > Is there a native way to parse scientific notation string into > xs:decimal since a direct casting (xs:decimal("1e1")) is not allowed? > > Thanks, > > Marco. >

2 2

Re: [basex-talk] Rounding/parsing decimal vs float
by Kristian Kankainen 01 Nov '17

01 Nov '17

You need to use xs:float or xs:double instead of xs:decimal to be able to use the 'e' or 'E' as the exponent separator. Br, Kristian K1. nov 2017 13:18 kirjutas kuupäeval Marco Lettere <m.lettere(a)gmail.com>: > > Hi all, > > I thought of asking this in parallel of hacking my own parsing procedure > ... > > Is there a native way to parse scientific notation string into > xs:decimal since a direct casting (xs:decimal("1e1")) is not allowed? > > Thanks, > > Marco. >

1 0

Rounding/parsing decimal vs float
by Marco Lettere 01 Nov '17

01 Nov '17

Hi all, I thought of asking this in parallel of hacking my own parsing procedure ... Is there a native way to parse scientific notation string into xs:decimal since a direct casting (xs:decimal("1e1")) is not allowed? Thanks, Marco.

1 0

Flexible update via function
by michael4you＠arcor.de 01 Nov '17

01 Nov '17

Hello, How to add/replace a complete node in a node tree, with done by a function and by according function parameters. That means with flexible instructions. Such as: declare function local:NodeAdd($XML, $XMLPath, $XMLValue) { ?? }; let $XML := <root><xml>text</xml></root> let $XMLAdd := <B>BBB</B> let $XMLPath := $XML/xml return local:NodeAdd($XML, $XMLPath , $XMLAdd) Problem: Following examples didn't work, because an internal reference to copy-XML-structure is needed/forced. example 1: copy / modify declare function local:NodeAdd($XML, $XMLPath, $XMLValue) { copy $XML2 := $XML modify ( insert node $XMLValue into $XML2/xml ) return $XML2 }; But: Instead of into $XML2/xml, into $XML/xml is needed, respectively into $XMLPath Thanks a lot Michael

3 3

UnaryLookup expression failure
by Gunther Rademacher 01 Nov '17

01 Nov '17

While testing some XQuery lookup expressions on BaseX 8.6.7, I got a wrong result for this one: (map{'a':(1,3)},map{'a':(2,4)})!?a[2] It should have returned the second item of the 'a' entry for each of the maps, i.e. 3 4 but it returns just 3 Apparently the predicate is applied as if everything before it was in parentheses, i.e. as if it were ((map{'a':(1,3)},map{'a':(2,4)})!?a)[2] The (simplified) parse tree for the original expression and the XQuery 3.1 grammar however indicates that the predicate is part of the mapping target expression: <SimpleMapExpr> <ParenthesizedExpr> ... </ParenthesizedExpr> <TOKEN>!</TOKEN> <PostfixExpr> <UnaryLookup> <TOKEN>?</TOKEN> <NCName>a</NCName> </UnaryLookup> <Predicate> <TOKEN>[</TOKEN> <IntegerLiteral>2</IntegerLiteral> <TOKEN>]</TOKEN> </Predicate> </PostfixExpr> </SimpleMapExpr> Best regards Gunther

2 1

Re: [basex-talk] Peculiar External Binding and Casting Behaviour
by Shaun Flynn 31 Oct '17

31 Oct '17

Hello there, I found the problem! My code added a trailing space to the values of the variables when binding. After tokenizing in the XQuery, this creates a zero length string which when cast to xs:integer then becomes an xs:double according to the error message. Is this intended behaviour? Shaun On 31 Oct 2017 06:20, "Shaun Flynn" <scfy12(a)googlemail.com> wrote: > Hello there, > > I found the problem! My code added a trailing space to the values of the > variables when binding. > > After tokenizing in the XQuery, this creates a zero length string which > when cast to xs:integer then becomes an xs:double according to the error > message. > > Is this intended behaviour? > > Shaun > > On 30 Oct 2017 17:39, "Shaun Flynn" <scfy12(a)googlemail.com> wrote: > >> Hello there, >> >> I created a simple example using the BaseX Java API, and that worked >> fine, also using my code to create the query and submit it to a >> ClientSession worked ok too :( >> >> This was the query I ran: >> >> declare variable $items external := "123 NIL"; >> declare variable $tokenizedItems:= tokenize($items," "); >> >> for $item in $tokenizedItems >> return >> if($item = "NIL") then ("Nil") else (xs:integer($item)) >> >> Unable to reproduce the earlier result from today using my code as well :( >> >> So I can understand the error better: "[FORG0001] Cannot cast to >> xs:double: ." >> >> The full stop after xs:double , does that relate to a decimal point or >> the current item (xquery .) >> >> Shaun >> >> On 30 October 2017 at 10:31, Christian Grün <christian.gruen(a)gmail.com> >> wrote: >> >>> Thanks. Some few lines of Java Code might already be sufficient >>> (provided that the problem can be reproduced with the XQuery snippet >>> that you sent in your last mail). >>> >>> On Mon, Oct 30, 2017 at 11:24 AM, Shaun Flynn <scfy12(a)googlemail.com> >>> wrote: >>> > Hello there, >>> > >>> > I will put something together for you later on today in a .zip file. >>> > >>> > Shaun >>> > >>> > On 30 October 2017 at 10:22, Christian Grün <christian.gruen(a)gmail.com >>> > >>> > wrote: >>> >> >>> >> > However, when I bind variables externally in Java, still getting the >>> >> > same >>> >> > problem, and I definitely not putting quotes around them there. >>> >> >>> >> Do you have an example that might help me to reproduce it? >>> >> >>> >> >>> >> > $depotConstraint: PM >>> >> > $resourceConstraint: 143/6 >>> >> > $lowerDiagConstraint: 251 >>> >> > $upperDiagConstraint: 258 >>> >> > Oct 30, 2017 9:05:33 AM units.forms.UnitDiagramsController >>> >> > fetchSelectedDiagrams >>> >> > SEVERE: null >>> >> > org.basex.core.BaseXException: Stopped at ., 29/61: >>> >> > [FORG0001] Cannot cast to xs:double: . >>> >> > >>> >> > >>> >> > >>> >> > On 30 October 2017 at 09:57, Christian Grün < >>> christian.gruen(a)gmail.com> >>> >> > wrote: >>> >> >> >>> >> >> HiShaun, >>> >> >> >>> >> >> > I do not understand why that should be as there is clearly no >>> mention >>> >> >> > of >>> >> >> > xs:double in this query! >>> >> >> >>> >> >> I agree that the error message is misleading. You already get it >>> when >>> >> >> invoking the following expression: >>> >> >> >>> >> >> 'a' cast as xs:integer >>> >> >> >>> >> >> I need to check the code, but I assume that an additional double >>> >> >> conversion check takes place in the background in order to ensure >>> that >>> >> >> the value can actually be converted to a non-fractional number. >>> >> >> >>> >> >> For more information on your query, it would be great if you could >>> >> >> reduce it to a self-contained example that runs out of the box (I >>> >> >> tried to invoke it, but your code seems to rely on an opened >>> >> >> database). >>> >> >> >>> >> >> Best, >>> >> >> Christian >>> >> > >>> >> > >>> > >>> > >>> >> >>

1 0