BaseX-Talk

basex-talk@mailman.uni-konstanz.de

6 participants
5269 discussions

strange behavior of ft:mark() and ft:extract()
by Francis Avila 14 Dec '12

14 Dec '12

ft:mark() and ft:extract() cannot be used with any intermediate looping construct, at least in BaseX 7.3. For example: > create db ftex "<r><a>text example string</a></r>" > xquery ft:mark(db:open('ftex')/descendant::text()[. contains text 'example']) text example string > xquery db:open('ftex')/descendant::text()[. contains text 'example'] ! ft:mark(.) text example string Notice that the use of a loop means that ft:mark() no longer works. This doesn't work with ft:search() either: > create index fulltext Index 'FULLTEXT' created in 7.63 ms. > xquery for $n in ft:search('ftex','example') return ft:mark($n) text example string However this works if there's no intermediate variable binding (implicit copying) even if we do some xpath navigation. > xquery ft:mark(ft:search('ftex','example')/..) <a>text example string</a> Query executed in 0.96 ms. > xquery ft:mark(ft:search('ftex','example')/../..) <r> <a>text example string</a> </r> However (this is a separate bug), the document-node() parent/child ordering is messed up: > xquery ft:mark(ft:search('ftex','example')/../../..) <r> <a>text </a> </r> example string > xquery ft:mark(ft:search('ftex','example')/ancestor::document-node()) <r> <a>text </a> </r> example string It seems to me that probably there is some extra hidden metadata attached to the text node which is not preserved by any implicit copying, such as by loops. (Although I'm not sure what's going on with the document-node() example.) One example of where this is a big headache is when we want to know in which document was a the text match. Here is a natural implementation: QUERY: for $n in ft:search('ftex','example') return <r doc="{document-uri($n/ancestor::document-node())}">{ft:mark($n)}</r> RESULT: <r doc="ftex/ftex.xml">text example string</r> However in this case the text will not be marked. We can't even do this because the "marked" document does not have a reference to the document-node() (which is understandable, as there's probably a copy/modify/return transform under there): QUERY: xquery for $n in ft:mark(ft:search('ftex','example')/..) return <r doc="{document-uri($n/ancestor::document-node())}">{$n/(* | text())}</r> RESULT: <r doc="">text example string</r> The workaround I used is this: QUERY: let $matches := ft:search('ftex','example') for $n at $i in ft:mark(ft:search('ftex','example')/..) return <r doc="{document-uri($matches[$i]/ancestor::document-node())}">{$n/(* | text())}</r> This is extremely ugly, and shares with the last example the downside that I need to mark the *parent* node of the matched text node and then carefully unbox it in the return clause. Quite a "gotcha"! In addition to being ugly, I'm also worried that it may be incorrect because I don't know for sure that the order of search will always be exactly the same (although it *seems* to be). Even if the order is stable, what if the fulltext index is updated by another session between the first call to ft:search() and the second? I don't know what the right answer to all of this is, but the way things are seems very not-good. At the very least the problems with ft:mark() and ft:extract() should be documented with big red text! Perhaps a better method is to have a function with a data structure that contains the text matched text node (as a reference, so that node references are retained) *and* matching substrings explicitly and separately. E.g. a ft:search-with-info('ftex','string example') could return: (map { 'text' := text('test example string'), 'substrings' := (5,8, 14,6) }, ...) or maybe this: (map { 'text' := text('test example string'), 'substrings' := (<s start="5" length="8"/>, <s start="14" length="6"/> ) }, ...) or maybe just a sequence of alternating items: (text('test example string'), <matchinfo><s start="5" length="8"/><s start="14" length="6"/></matchinfo>, ...) Unfortunately I don't see how we can return a simple sequence of elements since the text node result would have to be a copy. With the results separated we could easily make a family of functions which accepts the same data structure and does mark() and extract()-like things with it. The pairs can be processed either with a tumbling window or a for loop with a "where $i mod 2" in it. -- Francis Avila

2 2

Perl BaseX API questions
by Liam R E Quin 13 Dec '12

13 Dec '12

(1) The sample perl files and the perl module implementing the API mention a readme file that explains the methods; does anyone know where one might find such a readme file? It might answer my other question... (2) if there's an error in my query and I catch the exception, the message I get is about how many milliseconds the query took, not the actual error. I can enclose my Perl script, but I'm wondering if others have seen this? Thanks, Liam -- Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/ Pictures from old books: http://fromoldbooks.org/ (fromoldbooks.org uses BaseX e.g. for Search and for the tag cloud)

2 6

Performance Question
by Thomas Kaltofen 13 Dec '12

13 Dec '12

Hi, I have a question regarding performance, because my database shows a somehow strange behavior. I have a database with around 3000 documents, the size of each document does not really matter, because I tried it with small and big documents and there is no real difference (total size of test database is around 40 MB). I am connecting to the database through the client/server interface to execute a query over all documents (all documents have the same structure). The first time the query is executed after the server has been started takes a little longer, subsequent executions of the same query are very fast (around 250 ms). Now I leave the server running for several hours without touching the database at all (e.g. over the night). If I now execute the same query again (the database is still running but was idle for several hours), the execution takes very long (several minutes), but only for the first execution after the idle time, subsequent calls to the query are fast again (around 250 ms). This behavior always happens whenever the database was idle for several hours. What could be the cause for this delay? What I tried already: - Run the database on another computer -> same behavior - Try out different JRE versions (6, 7) -> same behavior - Try out different BaseX versions (7.3, latest 7.5 snapshot) -> same behavior - Start the VM with different options regarding performance & garbage collection -> same behavior - Use smaller or larger documents in the database -> same behavior - Execute a different query -> same behavior I am really running out of ideas and this seems to be independent of the size of the individual documents. What I noticed is that the described lag after the idle phase increases with the amount of documents in the database. Any help and suggestions are appreciated. Thank you! Best regards, Thomas P.S. When (approx.) do you plan to release the next version of BaseX?

5 11

BaseX and Saxon8
by NORFARADILLA WAHID 13 Dec '12

13 Dec '12

Dear sir, I am currently using BaseX to support my research project. Unfortunately when I include Saxon8 into the class-path(for my additional feature) I keep receiving the following error: java.lang.NoClassDefFoundError: net/sf/saxon/om/Name Can you help me? p/s: BaseX 7.1.2 beta if that matter Thanks, Dilla

2 1

Fwd: BaseX Server
by Christian Grün 13 Dec '12

13 Dec '12

Hi all, attached is some interesting information from Raymond Meester on how to get BaseX running as Windows service. Everyone’s feedback is welcome, Christian ---------- Forwarded message ---------- From: Raymond Meester <raymondmeester(a)gmail.com> Hi Christian, A small update on implementing basex. Still working on the issue that BaseXServer suddenly isn't available more (started as BaseXServer -S on a Windows 2008 r2 server). I tried to monitor the process with procmon, unfortunaly due to a leak procmon flooded the page file of the server. So I had to go for another approach. As our administrator are familiar with windows services I tried to install basex as windows service. - First I tried the Java launcher Janel (http://www.redskaper.com/), but I could get it working. Don't know why, but it seemed to be a Janel issue. - Secondly I tried another Java launcher (http://winrun4j.sourceforge.net/). With this launcher I was able to got a working executable. I attached the ini file I used. However to use it as a windows service it needs a java method to work with services (see "Service example" on their webpage). - Last I used the working exe and this manual http://marckean.wordpress.com/2009/05/24/run-shoutcast-as-a-windows-service/ ((I attached the tools which the manual uses: instrv.exe and srvany.exe. (I changed the extension to .txt to get them through the email scanner. I believe they are part of the tool "resource hacker" http://www.angusj.com/resourcehacker/) And now I got BaseX running as a windows service. So of course I would like to see a native way to install Basex as Windows Service, but I thought I let you know how I manage this. Also when there is a better way (not changing the registry directly), please put it on the wiki. Regards, Raymond

1 0

Re: [basex-talk] Problems DST
by Randy reed 12 Dec '12

12 Dec '12

I think I have found my problem. I post this here in case it helps anyone else. The java date constructor is initialized with a long, my attempts use a literal are what caused it to fail. I cast my literal as a long xs:long("133554732") and it worked, though not perfectly because epoch is in seconds and java is milliseconds. Thus I had to multiply my literal by 1000 first and then cast it as a long let $fdate: =133554732*1000 let $edat := dt:new(xs:long($fdate)) I then can access the TimeZone.IsDaylightTime property Thanks to Christian for his help on this. Randy On Wed, Dec 12, 2012 at 1:13 AM, Randy reed <randywreed(a)gmail.com> wrote: > Hi Christian, > > Thanks for your response. I think that fn:implicit-timezone() would work for > current dates but I'm not sure if it would work for dates in the past since > it doesn't take any arguements. > > I need some way that it can take a date in utc and adjust to the current > timezone if its during that time period (i.e. we are currently in standard > time) or adjust it if it was in the past during daylight savings time. If > there was something that could read a utc date and then indicate whether it > was during the daylight savings time period or not I could make the time > adjustment. > > In java there is a inDaylightTime property but it requires a date and > timezone object. and it errors when I try to invoke it "invalid arguments" > when it tries to initialize the constructor.. > > declare namespace tz = "java:java.util.TimeZone"; > declare namespace dt = "java:java.util.Date"; > > let $fdate := dt:new(1347425558) > let $tmz := tz:getDefault() > let $day := tz:inDaylightTime($tmz, $fdate) > return $day > > Any help you could provide would be much appreciated. I am not a Java > programmer so I might not understand something here. > > Randy > > > On Dec 11, 2012 6:18 PM, "Christian Grün" <christian.gruen(a)gmail.com> wrote: >> >> Hi Randy, >> >> fn:implicit-timezone() may come in handy here. It returns your current >> local timezone: >> >> http://www.xqueryfunctions.com/xq/fn_implicit-timezone.html >> >> Best, >> Christian >> ___________________________ >> >> On Wed, Dec 12, 2012 at 12:13 AM, Randy reed <randywreed(a)gmail.com> wrote: >> > Hi All, >> > >> > I wrote a xquery in Basex that created an xml file that calculated >> > student grades. Points are dependent on students doing an activity in >> > a window which ends at midnight on sunday each week. The date/time of >> > the activities are stored in the source file in UTC. However, to know >> > if they met the deadline, I need it in local time. I do this by >> > creating a function that translates epoch to a regulate dateTime >> > variable. But I see that when we left daylight savings time, things >> > were thrown off by an hour. The command I use is: >> > >> > adjust-dateTime-to-timezone((xs:dateTime("1970-01-01T00:00:00-00:00") >> > + xs:dayTimeDuration(concat("PT", $v, >> > "S"))),xs:dayTimeDuration('-PT4H')) >> > >> > clearly the problem is the last part -PT4H, because dates that fall >> > after the switch back to standard time from daylight savings time >> > should be -PT5H. Is there a better way of doing this? Is there another >> > command that will give me a localized date that takes into account >> > daylight savings time issues? I tried to set a java calendar but I >> > could not make it work. >> > >> > Any help would be gratefully appreciated. >> > Thanks in advance >> > Randy >> > >> > -- >> > Dr. Randall Reed >> > Assistant Professor, Dept. of Philosophy and Religion >> > Appalachian State University >> > _______________________________________________ >> > BaseX-Talk mailing list >> > BaseX-Talk(a)mailman.uni-konstanz.de >> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk -- Dr. Randall Reed Assistant Professor, Dept. of Philosophy and Religion Appalachian State University

1 0

Re: [basex-talk] BaseX in Farsi?
by Masoumeh Seydi Gheranghiyeh 12 Dec '12

12 Dec '12

Hi Christian That would be a great idea. I am really interested in. Supporting Persian language makes support of the other right-to-left languages, like Arabic and Hebrew, also possible. As you've said, the first step is to find out how much effort is needed to add the right-to-left format to the editor. I will take a look at it to investigate how to do it and highly appreciate any suggestion. Regards, Masoumeh On Monday, December 10, 2012 22:30 CET, Christian Grün <christian.gruen(a)gmail.com> wrote: > Hi Masoumeh, > > yet another idea I got: as you may know, BaseX is available in various > languages. Is Farsi your native language? And would you be interested > in translating BaseX to your language? It may be, however, that BaseX > may get into troubles when writing text from right to left. > Particularly, it's the text editor that's very limited in this > regard.. > > Thanks in advance, > Christian > > PS: just in case: please forward your reply on basex-talk back to the list..

1 0

Problems DST
by Randy reed 11 Dec '12

11 Dec '12

Hi All, I wrote a xquery in Basex that created an xml file that calculated student grades. Points are dependent on students doing an activity in a window which ends at midnight on sunday each week. The date/time of the activities are stored in the source file in UTC. However, to know if they met the deadline, I need it in local time. I do this by creating a function that translates epoch to a regulate dateTime variable. But I see that when we left daylight savings time, things were thrown off by an hour. The command I use is: adjust-dateTime-to-timezone((xs:dateTime("1970-01-01T00:00:00-00:00") + xs:dayTimeDuration(concat("PT", $v, "S"))),xs:dayTimeDuration('-PT4H')) clearly the problem is the last part -PT4H, because dates that fall after the switch back to standard time from daylight savings time should be -PT5H. Is there a better way of doing this? Is there another command that will give me a localized date that takes into account daylight savings time issues? I tried to set a java calendar but I could not make it work. Any help would be gratefully appreciated. Thanks in advance Randy -- Dr. Randall Reed Assistant Professor, Dept. of Philosophy and Religion Appalachian State University

2 1

Re: [basex-talk] Unable to integrate with oxygen to run xquery
by Christian Grün 11 Dec '12

11 Dec '12

Dear Ionela, thanks a lot for your support and your feedback. I decided to also pass this on to our BaseX mailing list; hope it’s ok. All the best, Christian ____________________________________ On Tue, Dec 11, 2012 at 8:52 PM, oXygen XML Editor Support <support(a)oxygenxml.com> wrote: > Hello, > > I just wanted to let you know that this problem has been resolved in the > latest maintenance build of Oxygen 14.1, 2012121012 (released on December > 11th): >> >> Database/XQJ: BaseX via XQJ no longer worked because of conflicting jar >> libraries. > > > You can download it from our web site: > http://www.oxygenxml.com/download.html > > The list of bug-fixes can be found here: > http://www.oxygenxml.com/build_history.html#2012121012 > > You can follow the release/build RSS feed here: > http://www.oxygenxml.com/rssBuildID.xml > > To upgrade: > - For Windows and Linux you can install the new build in the same folder as > the previous installation, it will automatically upgrade it. > Before you upgrade, if you have added files or made changes to any of the > files from the Oxygen installation folder(especially the frameworks folder), > you may want to create a backup of them because they will be overwritten > during the upgrade procedure. Custom frameworks will be preserved but we > recommend backing them up anyway, just to be safe. > - For Mac OS X you will have to install in a different folder and copy any > files you may have changed from the old folder (if any) to the new folder. > The Oxygen preferences will be preserved since they are located elsewhere > (user home folder). > > Let us know if you encounter further problems with this new build. > > Regards, > Ionela > > -- > Ionela Istodor > > oXygen XML Editor and Author Support > > Tel: +1-650-352-1250 ext.301 > > Fax: +40-251-461482 > support(a)oxygenxml.com > http://www.oxygenxml.com > > > --------------------------------------------------------------------------- > This message (including any attachments) contains confidential information > and is intended only for the individual(s) named. If you are not the named > addressee you should not disseminate, distribute or copy this e-mail. Please > notify the sender immediately by e-mail if you have received this e-mail by > mistake and delete this e-mail from your system. Any quotation contained > herein has a validity of 30 days, unless otherwise specified. > > > On 11/26/2012 2:04 PM, Alex Muir wrote: >> >> That works on my side... >> >> Thanks a tonne! >> Cheers >> >> On Mon, Nov 26, 2012 at 5:13 AM, oXygen XML Editor Support >> <support(a)oxygenxml.com <mailto:support@oxygenxml.com>> wrote: >> >> Hi, >> >> We've identified the issue as being class loader related. Oxygen >> comes bundled with an xml-apis.jar which contains javax.xml >> interfaces slightly different from the ones present in the JRE. The >> main problem regards class javax.xml.stream.XMLInputFactory from >> xml-apis.jar. First of all it requires >> com.ctc.wstx.stax.WstxInputFactory (for which you must add >> org.apache.servicemix.bundles.woodstox-3.2.7_1.jar) and it's missing >> method newFactory(). The API basex-xqj-1.2.3.jar is compatible with >> it because it uses newInstance() instead of newFactory(). So if you >> use basex-xqj-1.2.3.jar in the connection and also add >> org.apache.servicemix.bundles.woodstox-3.2.7_1.jar in Oxygen lib, >> the connection will begin to work. >> >> But there should be no need for you to add >> org.apache.servicemix.bundles.woodstox-3.2.7_1.jar so I've attached >> an xml-apis.jar with these interfaces removed. Just replace the >> xml-apis.jar from {oXygenInstallDir}/lib with this new version and >> the BaseX connection will function with just basex-xqj-1.2.3.jar, >> xqj-api-1.0.jar and xqj2-0.1.0.jar added in the connection. No need >> to add org.apache.servicemix.bundles.woodstox-3.2.7_1.jar either. >> >> I've also added an issue to fix this issue on our side in the next >> Oxygen build. >> >> Thank you for following this through! >> Alex >> -- >> Alex Jitianu >> <oXygen/> XML Editor, Schema Editor and XSLT Editor/Debugger >> http://www.oxygenxml.com >> >> >> On 26-Nov-12 5:21 AM, Alex Muir wrote: >>> >>> Hi, >>> >>> I decided to try using basex.exe on windows 7 on a virtual machine >>> with java 7. I get the same results given a fresh download and >>> install of java, oxygen and xqj. I pasted some screen shots so >>> perhaps you'll see the error I'm making or get a better idea of >>> that is not working. When I started the server windows 7 asked >>> permission for the firewall which I gave. I started the server >>> from gui and as well closing the gui and starting from the start >>> menu basex server (start) and both gave the same result. >>> >>> There must be some difference between yours and mine system or >>> some setup thing I'm doing wrong. >>> >>> On linux, I also tried adjusting firewall settings on fedora >>> although this had no affect. >>> >>> If you want to do some remote desktop interaction to check out my >>> system, your highly welcome. >>> >>> Regards >>> Alex >>> >>> >>> On Fri, Nov 23, 2012 at 10:06 PM, Alex Muir <alex.g.muir(a)gmail.com >>> <mailto:alex.g.muir@gmail.com>> wrote: >>> >>> >>> >>> On Fri, Nov 23, 2012 at 9:12 PM, Christian Grün <cg(a)basex.org >>> <mailto:cg@basex.org>> wrote: >>> >>> > Also I wonder why I was getting this error in oxygen? >>> >>> At which stage does this error occur? >>> >>> >>> When running the xquery I would get that error message >>> initially, though it seemed to go away randomly.. the >>> following error is the more common problem. *"E [BaseX7.3] >>> Could not initialize class net.xqj.basex.bin.m"* >>> >>> >>> >>> > Is the woodstox-core-asl-4.1.4.jar required in anyway >>> and if so how? >>> >>> I can’t find this jar in my environment; where did you get >>> it from? >>> >>> >>> >>> Well based on the error message I thought this jar was needed >>> given a search online although I really don't know why I was >>> getting that >>> error message or what to do about it nor if that jar is the >>> required although seemed likely when searching for that error >>> online. >>> >>> From here http://wiki.fasterxml.com/WoodstoxDownload >>> >>> >>> -- >>> - >>> >>> Alex G. Muir >>> Software Engineering Consultant >>> Linkedin Profile : >>> http://ca.linkedin.com/pub/alex-muir/36/ab7/125 >>> Love African Kora Music? Take a moment to listen to Gambia's - >>> Amadu Diabarte & Jali Bakary Konteh www.bafila.bandcamp.com >>> <http://www.bafila.bandcamp.com> Your support keeps Africa's >>> >>> griot tradition alive... Cheers! >>> >>> >>> >>> >>> -- >>> - >>> >>> Alex G. Muir >>> Software Engineering Consultant >>> Linkedin Profile : http://ca.linkedin.com/pub/alex-muir/36/ab7/125 >>> Love African Kora Music? Take a moment to listen to Gambia's - >>> Amadu Diabarte & Jali Bakary Konteh www.bafila.bandcamp.com >>> <http://www.bafila.bandcamp.com> Your support keeps Africa's griot >>> tradition alive... Cheers! >>> >> >> >> >> >> -- >> - >> >> Alex G. Muir >> Software Engineering Consultant >> Linkedin Profile : http://ca.linkedin.com/pub/alex-muir/36/ab7/125 >> Love African Kora Music? Take a moment to listen to Gambia's - Amadu >> Diabarte & Jali Bakary Konteh www.bafila.bandcamp.com >> <http://www.bafila.bandcamp.com> Your support keeps Africa's griot >> tradition alive... Cheers! >> >

1 0

Are command scripts optimized before execution? / Optimization of my query
by Ben Companjen 10 Dec '12

10 Dec '12

Hi all, I was wondering whether command scripts are optimized before they are executed. I would think so, because when I had made an error in the, say, sixth command in my script, it was caught before the first command was executed. But more importantly for my application: it seems that combinations of "CLOSE; OPEN <same DB as just closed>" are skipped. I created an XQuery to copy 680000 records from my XML DB to MySQL. These records are MODS, found inside record elements at XPath /file/record. The database is about 2.5 GB and eats even more of my main memory when I try to copy all records in one run. I found out that doing subsequences of 50000 to 10000 records speeds up the process significantly and the server uses a lot less memory. For instance, when I start processing records from 400001, BaseX apparently doesn't keep the first 400000 in memory. But just doing 50000 records in a row, shifting the begin parameter of the subsequence and going on, eventually slows the process from tens to more than 100 records per second to less than one per second, probably because memory consumption goes up to all available memory (I assigned 3 GB to the JVM at startup of the server). Therefore I tried to free some memory by closing the database, re-opening it and then resume processing. But I don't see the memory usage reduction that I hoped for. >From my XQuery mods2sql5.xq: <namespace declarations> declare variable $begin as xs:integer external := 1; declare variable $length as xs:integer external := 50000; <function declarations: prepare statement return sql:execute-prepared(...)> (: main process :) for $record at $rodb in subsequence(/file/record[mods:mods][@id != ""], $begin, $length) <call functions to process records> return <functions' output as sequence> >From my command script mods2sql.bxs: SET QUERYINFO true OPEN monly2 SET BINDINGS $begin=1,$length=50000 RUN /Users/ben/Documents/xquery/metadata_only/mods2sql5.xq SET BINDINGS $begin=50001 RUN /Users/ben/Documents/xquery/metadata_only/mods2sql5.xq CLOSE OPEN monly2 SET BINDINGS $begin=100001 RUN /Users/ben/Documents/xquery/metadata_only/mods2sql5.xq SET BINDINGS $begin=150001 RUN /Users/ben/Documents/xquery/metadata_only/mods2sql5.xq SET BINDINGS $begin=200001 RUN /Users/ben/Documents/xquery/metadata_only/mods2sql5.xq CLOSE OPEN monly2 ... Are my observations about optimizations of command scripts correct? Is there any way to speed up my process? Thanks for any advice! Regards, Ben

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk