BaseX-Talk

basex-talk@mailman.uni-konstanz.de

1 participants
5252 discussions

serializing an xquery flwor in csv
by Emmanuelle Morlock 23 Oct '14

23 Oct '14

Hi all, I'm trying to do some xquery with the GUI in a standalone install. As I'm beginning with xquery, I rely a lot on the examples given in the doc. But I don't manage to be successful with http://docs.basex.org/wiki/CSV_Module for what I need. I'd like to get a result formatted in csv (separator : ';' or whatever) so tha I can directly import in a spreadsheet the results exported file. Here's the begining of my query : for $lettre in db:open('myDB')//TEI where $lettre//opener//addrLine[not(contains(.,'Adresse ?'))] let $title := $lettre//teiHeader//title let $destinataire := $lettre//profileDesc/creation/persName[@type='destinataire'] let $expediteur := $lettre//profileDesc/creation/persName[@type='expediteur'] let $date := $lettre//profileDesc//date let $locName := $lettre//profileDesc/creation/location/placeName[@type='PlaceExp'] let $locGeo := $lettre//profileDesc/creation/location/geo then I guess I should have something like : - a variable for the option... is let $options := {'separator':';'} correct ? something to put all the variable as a result let $fields := ... ? then return csv:serialize ($fields, $options) ??? thanks for helping even if this sounds too basic ! best emmanuelle -- Emmanuelle Morlock IGE CNRS - soutien aux projets d'édition TEI - humanités numériques UMR 5189 HISoMA http://www.hisoma.mom.fr/ 04 72 71 58 40 06 85 84 69 16 @emma_morlock

2 1

Re: [basex-talk] Architecture Question
by Christian Grün 23 Oct '14

23 Oct '14

Hi Mansi, Good to know! Keep us updated. Christian On Wed, Oct 22, 2014 at 8:13 PM, Mansi Sheth <mansi.sheth(a)gmail.com> wrote: > Christian, > > Actually, I am all set. I will just query using python or Angular or > whatever and do any data manipulation there and use XQUERY just to query and > no further processing. > > Btw, some initial very informal statistics: > > Took, 22 min to import ~2.1k documents with indexing as well. > ~2 min for query to return back. > > Impressive !!! > > Machine specs: 16 GB RAM, 2.7 GHz, i7 processor MacBook Pro. > > I am waiting on my colleague to get me some more production data, which > gives me access to some 10k XML files. Will keep you posted. > > - Mansi > > On Wed, Oct 22, 2014 at 12:04 PM, Mansi Sheth <mansi.sheth(a)gmail.com> wrote: >> >> Christian, >> >> Thanks for all your responses. It truly helps a lot. >> >> re: Importing data into databases: I realized, for the extent of this POC, >> I will just count no of docs in each database (currently programmed to be >> 50) and keep creating new databases. Structure of data is same, but its >> nested in nature. Like a folder can have folder, which can have file etc. >> Usually, it won't be more than 4 levels deep. Thats a good tip, to guess no >> of nodes based on byte size. I guess, for time being I will move on, with >> just storing 50 docs per DB. >> >> re: terabytes of data. Well, I am planning on using ~6 months worth of >> data for any analysis and discarding data prior to that (leaving it around >> in backups). Obviously, would be going some cloud route for such resources, >> will see how much budget I can manage to get :) Am very positive about this. >> So, no its not only a theoretical assumption as far as I can see. >> >> re: Currently, I am looking into querying these databases. I am exploring >> REST for it. From documentation, it seems our only option is supporting >> these queries (on server side) using XQUERY or RestXQ, no Java/Python ? I am >> well versed with XPATH and XSLT, gearing up towards XQUERY now. But, would >> be a little easier (just my personal preference :)) to manipulate data in >> Java/Python before serving it back to client. Is there any such facility ? >> Something like: >> >> "http://localhost:8984/rest?run=getData.java" >> >> similarly for python ? >> >> - Mansi >> >> Some preliminary statistics: Imported 2050 XML documents in 22 min >> (including indexing on attributes). >> >> On Sun, Oct 19, 2014 at 6:14 PM, Christian Grün >> <christian.gruen(a)gmail.com> wrote: >>> >>> Hi Mansi, >>> >>> > Is there some book/resource you can point me to, which helps better >>> > visualize NXD ? >>> >>> sorry for letting you wait. If you want to know more about native XML >>> databases, I recommend you to have a closer look at various articles >>> in our Wiki (e. g. [1,2]). It will also be helpful if you get into the >>> basics of XQuery [3]. >>> >>> Have you tried to realize some of the hints I gave in my previous mails? >>> >>> > I am trying to distribute data across multiple databases. I can't >>> > distribute >>> > based on day, as there could very well be situation, where single day's >>> > data >>> > could more than capacity of BaseX DB. >>> >>> If 2 billion XML nodes per day are not enough, you will probably need >>> to create more than one database per day. Via the "info db" command, >>> you see how many nodes are currently stored in a database, but there >>> is no cheap solution to find out the number of nodes of an incoming >>> document, because XML documents can be very heterogeneous. Some >>> questions back: >>> >>> * Do you have some more information on the data you want to store? >>> * Are all documents similar or do they vary greatly? If the documents >>> are somewhat similar, you can usually estimate the number of nodes by >>> looking at the byte size. >>> * Do you know that you will really need to store lots of terabytes of >>> XML data, or it is more like a theoretical assumption? >>> >>> Christian >>> >>> [1] http://docs.basex.org/wiki/Database >>> [2] http://docs.basex.org/wiki/Table_of_Contents >>> [3] http://docs.basex.org/wiki/Xquery >> >> >> >> >> -- >> - Mansi > > > > > -- > - Mansi

1 0

locking problem
by Lizzi, Vincent 22 Oct '14

22 Oct '14

This is an update, and further problem, on a problem I posted here a few months ago. I've written a program to import XML documents into BaseX in one database, and populate a second database with a small amount of metadata about each document. This works but crashes every so often, and I have to restore the databases from backup and restart the import. To get through a large set of documents I now have the program create a backup (using the CREATE BACKUP command) after every 1000 documents imported. Since my earlier post, I found that the connection between the client and server was occasionally being dropped. I've switched to opening and closing a connection for each document instead of using one long-running connection. The client and server are running on the same machine. I'm using the latest version of BaseX (BaseX80-20141013.213520.zip). Here is the error message reported by my program when it crashes. The error occurs on the Query.execute() method of org.basex.examples.api.BaseXClient.java. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 8.0 beta a91f5c2 Java: Oracle Corporation, 1.7.0_67 OS: Windows 7, amd64 Stack Trace: java.nio.channels.OverlappingFileLockException at sun.nio.ch.SharedFileLockTable.checkList(Unknown Source) at sun.nio.ch.SharedFileLockTable.add(Unknown Source) at sun.nio.ch.FileChannelImpl.tryLock(Unknown Source) at org.basex.io.random.TableDiskAccess.lock(TableDiskAccess.java:139) at org.basex.io.random.TableDiskAccess.<init>(TableDiskAccess.java:87) at org.basex.data.DiskData.init(DiskData.java:119) at org.basex.data.DiskData.<init>(DiskData.java:80) at org.basex.core.cmd.Open.open(Open.java:70) at org.basex.core.cmd.Open.run(Open.java:36) at org.basex.core.Command.run(Command.java:360) at org.basex.core.Command.execute(Command.java:94) at org.basex.server.ClientListener.run(ClientListener.java:145) The server instance reports the error with this stack trace: C:\Programs\basex\basex\bin>basexhttp BaseX 8.0 beta a91f5c2 [Server] Server was started (port: 1984) HTTP Server was started (port: 8984) java.io.FileNotFoundException: C:\Programs\basex\basex\data\article_xml_meta\doc.basex (The requested operation cannot be performed on a file with a user-mapped section open ) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(Unknown Source) at java.io.FileOutputStream.<init>(Unknown Source) at org.basex.io.out.DataOutput.<init>(DataOutput.java:47) at org.basex.io.out.DataOutput.<init>(DataOutput.java:36) at org.basex.index.resource.Docs.write(Docs.java:73) at org.basex.index.resource.Resources.write(Resources.java:51) at org.basex.data.DiskData.write(DiskData.java:141) at org.basex.data.DiskData.flush(DiskData.java:249) at org.basex.data.DiskData.finishUpdate(DiskData.java:239) at org.basex.query.up.ContextModifier.apply(ContextModifier.java:124) at org.basex.query.up.Updates.apply(Updates.java:129) at org.basex.query.QueryContext.iter(QueryContext.java:351) at org.basex.query.QueryProcessor.iter(QueryProcessor.java:80) at org.basex.server.ServerQuery.execute(ServerQuery.java:128) at org.basex.server.ClientListener.query(ClientListener.java:492) at org.basex.server.ClientListener.run(ClientListener.java:109) Here is the update statement where the error occurs, along with a bit of the surrounding Java. BaseXClient.Query queryIndex = client.query("declare variable $indexDb external; declare variable $hash external; declare variable $database external; declare variable $path external; let $p := <properties><database>{$database}</database><path>{$path}</path><sha1>{$hash}</sha1><loadedDate>{current-dateTime()}</loadedDate></properties> return db:replace('" + indexDatabaseName + "', $path, $p)"); try { queryIndex.bind("$indexDb", indexDatabaseName); queryIndex.bind("$database", managedDatabaseName); queryIndex.bind("$path", path); queryIndex.bind("$hash", hash); queryIndex.execute(); } finally { queryIndex.close(); } Previously this code used a variable for the first parameter of db:replace() but I changed it to "hard code" the database name into the query after reading about transactions and locking in the wiki. This change doesn't appear to have helped. To put this in context, the above update is run after the XML document is imported using BaseXClient.replace(). The next thing that happens in the program is an XQuery db:replace to add more properties for the document to the metadata database. I've added a Thread.sleep(300), to create a small delay between loading each document and this has helped the program to import more documents before crashing. Adjusting the delay doesn't seem to completely stop the crashes though. I'm not sure if this is a bug or improper use. Any help would be appreciated. Thanks, Vincent

2 2

Architecture Question
by Mansi Sheth 22 Oct '14

22 Oct '14

Hello, I have been going thru and comparing different Native XML Databases and so far I am liking BaseX. However, there are still a few questions unanswered, before I make a final choice: 1. I have 1000s of XML files (each between 50MB-400MB) and this is going to grow exponentially (~200 / per day). So, my question is how scalable is BaseX ? Can I configure it to use data from my external HDD, in my initial prototype ? 2. I plan to heavily use XPATH, for data retrieval. Does BaseX, use any multi-processing, multi-threading to speed up search ? Any concurrent processing ? 3. Can I do some post-processing on searched and retrieved data ? Like sorting, unique elements etc ? - Mansi

2 11

Using TcpCatcher as a proxy with Basex Client
by Arto Viitanen 20 Oct '14

20 Oct '14

I am debugging a Basex client built in C#. The client is based on the Git Hub example https://github.com/BaseXdb/basex/tree/master/basex-api/src/main/c%23. The problem is, every now and then the client drops the session. It might be, that the client drops the connection if the server takes too much time to answer (or something). So I need some way to emulate slow connection or connection dropping packages. I downloaded TcpCatcher (http://www.tcpcatcher.org/) and set it as a proxy (client talks to 21984; TcpCatcher catches packages and send them to BaseX listening to 11984). But It seems that BaseX does not respond. I took BasexClient.cs and Example.cs from the github and tried with them. I get three packages; one coming from the server and two going to client; but after that nothing. The Example prints "Access Denied". I guess this comes from: socket = new TcpClient(host, port); stream = socket.GetStream(); ehost = host; string ts = Receive(); Send(username); Send(MD5(MD5(pw) + ts)); if (stream.ReadByte() != 0) { throw new IOException("Access denied."); } that is: client gets ts (timestamp?), sends username and password with timestamp; but receives nothing. So how can I use TcpCatcher (or other proxy program) with Basex client? (Basex 7.7; windows 7) Arto Viitanen Microteam Oy

3 3

Templating
by Emmanuel Chateau 18 Oct '14

18 Oct '14

Hi, I’ve been trying to use the templating proposed by Andy Bunce here http://cubeb.blogspot.fr/2012/11/xquery-templating-engines-and-txq.html https://mailman.uni-konstanz.de/pipermail/basex-talk/2012-November/004209.h… I’ve tried with txq module in different ways. But, I can’t figure out why I couldn’t reproduce it. Here is the simpler exemple I’ve tried : In the webapp folder an xml file, named `view1.xml` wich contains : ```xml <div>{map:get($map, 'greeting')}</div> ``` In webapp a restxq file with : ```xquery module namespace test = 'test.test'; import module namespace xquery = "http://basex.org/modules/xquery"; declare function test:render($template as xs:string, $map as map(*)){ xquery:invoke($template,$map) }; declare %restxq:path('/test') %restxq:GET function test:test(){ let $map := map{ "greeting" : "hello", "who": "world"} return test:render('view1.xml', $map ) }; ``` Arrêté à /Users/emmanuelchateau/Documents/basex/webapp/view1.xq, 1/15: [XPST0008] Undefined variable $map. I’m running BaseX version 7.9 Is there still anybody with this kind of templating running ? Thanks a lot for your help. Emmanuel

2 4

Web hosting
by spaelti 16 Oct '14

16 Oct '14

I have now tried BaseX web server on my own machine, and it works great. But what I would really like to do is to use it on my website. This website is on a commercial web provider. Is it possible to use BaseX in such a situation? How does one go about this? Would I need to ask the commercial provider to install BaseX? Or is there some other web site hosting method, that one can use? Thanks in advance for any help or suggestions. Philip Spaelti spaelti(a)mac.com

2 1

Fw: Suggestion for a requirement
by Sateesh 16 Oct '14

16 Oct '14

? Hi Basex Team, We are starting a new project in that we have selected to use basex,i would tell you my requirement,request you to please suggest the best way to use basex. We have started a web application in that we have a requirement to create xml db's per user based means each user who logs in we need to create DB's on his xml files in the backend,for this we dont want to use the client-server architecture of the basex as we dont want that extra time to hit a http server request to basex server and get the response,what we would want is to create the DB's execute queries and drop the DB's with in the same web application and also please note that we dont have a requirement to edit the DB's data we would only read the data from those DB's by executing the queries and also note that volumes of the xml files are very huge they might go up to gb's also and query output will also be in huge volume and also we must make sure that this huge volumes of query output will not eatup all the heap memory as the number of parallel users using this site are very high and we cannot afford to go out of memory.Also we would need to configure the DB paths as this are huge volumes we cannot store them in the default locations to. Considering all the above points request you to please suggest the best way to use basex. Our web application is developed in java.Though we have used basex in our earlier projects some where i feel that we did not used it properly and gone through problems and the kind of support we got from your team in solving them earlier was immense and this time we want to go in a more systematic way so request you to please help us by suggesting the best way. Thanks & Regards SAteesh.A The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited.

4 8

Crashed BaseX import
by Arjan Loeffen 16 Oct '14

16 Oct '14

This is the report, bold is my logging steps in the process: *DELETE core-cbnl-20140901.xml* Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.7.2 Java: Oracle Corporation, 1.7.0_15 OS: Windows Server 2012, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 372 at org.basex.io.random.TableDiskAccess.flush(TableDiskAccess.java:124) at org.basex.data.DiskData.finishUpdate(DiskData.java:212) at org.basex.core.cmd.Add.run(Add.java:109) at org.basex.core.Command.run(Command.java:345) at org.basex.core.Command.exec(Command.java:321) at org.basex.core.Command.execute(Command.java:78) at org.basex.server.LocalSession.execute(LocalSession.java:122) at org.basex.server.LocalSession.execute(LocalSession.java:114) at org.basex.server.Session.execute(Session.java:50) at org.basex.http.rest.RESTCommand.run(RESTCommand.java:40) at org.basex.http.rest.RESTPost.run(RESTPost.java:108) at org.basex.http.rest.RESTServlet.run(RESTServlet.java:14) at org.basex.http.BaseXServlet.service(BaseXServlet.java:58) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2430) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2419) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) * ADD "C:\Servers\xslweb\home\webapps\CBNL-browser-v2\work\contexts\core-cbnl-20140901.xml" CREATE INDEX ATTRIBUTE (2d) Database 'CBNL' is currently opened by another process. CREATE INDEX TEXT (2e) Database 'CBNL' is currently opened by another process. CREATE INDEX FULLTEXT (2f) * Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.7.2 Java: Oracle Corporation, 1.7.0_15 OS: Windows Server 2012, amd64 Stack Trace: java.lang.RuntimeException: Data Access out of bounds: - pre value: 0 - #used blocks: 0 - #total locks: 0 - access: -1 (0 > -1] at org.basex.util.Util.notexpected(Util.java:53) at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:488) at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:189) at org.basex.data.Data.kind(Data.java:282) at org.basex.query.value.node.DBNode.<init>(DBNode.java:53) at org.basex.core.cmd.OptimizeAll$DBParser.parse(OptimizeAll.java:201) at org.basex.build.Builder.parse(Builder.java:73) at org.basex.build.DiskBuilder.build(DiskBuilder.java:90) at org.basex.core.cmd.OptimizeAll.optimizeAll(OptimizeAll.java:126) at org.basex.core.cmd.OptimizeAll.run(OptimizeAll.java:44) at org.basex.core.Command.run(Command.java:345) at org.basex.core.Command.exec(Command.java:321) at org.basex.core.Command.execute(Command.java:78) at org.basex.server.LocalSession.execute(LocalSession.java:122) at org.basex.server.LocalSession.execute(LocalSession.java:114) at org.basex.server.Session.execute(Session.java:50) at org.basex.http.rest.RESTCommand.run(RESTCommand.java:40) at org.basex.http.rest.RESTPost.run(RESTPost.java:108) at org.basex.http.rest.RESTServlet.run(RESTServlet.java:14) at org.basex.http.BaseXServlet.service(BaseXServlet.java:58) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2430) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2419) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) *OPTIMIZE ALL * Any idea what's going on here? Hope this is enough info. -- Arjan Loeffen, Armatiek BV 06-12918997

2 2

Xpath gives multiple results, finding the one originating from the latest imported document?
by Sam Eklund 15 Oct '14

15 Oct '14

A quick question, I have this simple xquery to retrieve a node that sometimes has multiple hits, and for this reason I am using the function: head. This gives me the first node that matches the xpath, but is there a way to retrieve the node that originates from the latest imported document? My current query: xquery head (//event[eventID="12345ABC"]) // Sam

2 3

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk