- BaseX-Talk - mailman.uni-konstanz.de

Re: [basex-talk] Large CSV File (~1M rows)
by Shaun Flynn 16 Oct '16

16 Oct '16

Hello there, The parsing runs out memory. I also tried doubling the memory to 1024MB, and still the same issue. Kind regards On 16 Oct 2016 09:15, "Christian Grün" <christian.gruen(a)gmail.com> wrote: Hi Shaun, > I do this through the MS DOS prompt, but -Xmx does not appear to be a > parameter any more, If you work with the ZIP distributions of BaseX, you can adjust the memory setting in the start scripts (in the bin directory). Otherwise, you’ll need to pass it to to java, not BaseX itself. > Is there a better method for parsing large CSV files? I then want to add the > resulting file tlk.xml to a new database. Did you check if it’s the CSV parsing or the fn:put call that causes the error? Christian [1] http://docs.basex.org/wiki/Database_Module#db:add

2 3

Making connections to a database
by Thomas Daly 16 Oct '16

16 Oct '16

Hello, I'm developing an application server with node.js. When the server starts, it creates a single connection to a BaseX database, to allow subsequent user queries to the database. I notice that when I connect separately to the BaseX database with the BaseXClient command line tool, the application server connection dies with a 'broken pipe' error. Am I doing this correctly, or should the application server connect then immediately disconnect from BaseX to serve each individual user query (in the same way that PHP works with MySQL)? Thanks Thomas

2 1

(no subject)
by Shaun Flynn 16 Oct '16

16 Oct '16

Hello there, I am trying to parse a large CSV file into XML by the following xquery: let $file := fetch:text("D:\BPLAN\tlk.txt") let $convert := csv:parse($file, map { 'header' : true(), 'separator' : 'tab'}) return fn:put( <tlks> {for $row in $convert/csv/record return <tlk>{$row/*}</tlk> }</tlks>, "D:\BPLAN\tlk.xml" ) Using the GUI, it runs out of memory -- when I click on the bottom right hand corner (where the memory usage is shown), it says to increase memory, restart using -Xmx <size>. I do this through the MS DOS prompt, but -Xmx does not appear to be a parameter any more, Is there a better method for parsing large CSV files? I then want to add the resulting file tlk.xml to a new database. Kindest Regards Shaun Connelly-Flynn

2 1

retrieve a sequence of all values within an attribute index
by Alex Muir 10 Oct '16

10 Oct '16

Greetings, I need to generate a list of all the distinct values within an index. Is there a way to retrieve a sequence of all values within an attribute index? Regards Alex tech.jahtoe.com bafila.jahtoe.com

3 9

http:send-request POST xml body is inheriting the http-client namespace
by chrisis 10 Oct '16

10 Oct '16

I'm using BaseX 8.4 but have recreated the issue with 8.5.4 beta 8d90b66 I'm POSTing xml files to a RESTXQ function using the following script let $fpath := 'C:\incoming\SAQD\mo' let $targetBasex := 'http://localhost:8984/saqd/mo/' let $list := sort(file:list($fpath), function($fname) {fn:analyze-string($fname,'.*_(.*?)\.xml$')/fn:match/fn:group[@nr="1"]/string()}) for $file in $list let $abspath := $fpath||'\'||$file let $body := try {fetch:xml($abspath, map { 'chop': true() }) } catch * {()} return if ( not(empty($body)) ) then (http:send-request(<http:request href="{$targetBasex||$file}" method='POST'> <http:body media-type="text/xml">{$body}</http:body> </http:request>), file:move($abspath,$fpath||'\processed\'||$file) ) else () The XML file has no namespace declarations. The RESTXQ function does a db:update using the body received and the http-client namespace has been added to the root node. If I POST to the same function using curl the stored document is unchanged. The function is as follows. declare %rest:path("/saqd/mo/{$fname}") (: %rest:consumes("application/xml", "text/xml") :) %rest:POST("{$body}") %updating function saqd:mo($fname, $body as document-node()) { try {(db:replace('SAQD', '/MO/'||$fname, $body ), db:output(http:send-request(<http:request href="{' http://example.org:8082/home/VALIDSCHEMA/xml/current/Metoffice_Data/'||$fname }" method='PUT' username='validuser' password='validpassword' send-authorization='true'> <http:body media-type='text/xml'>{$body}</http:body> </http:request>)) ) } catch * {db:output( <rest:response> <http:response status="500" message="{$err:code||' : '||$err:description}"> </http:response> </rest:response>)} }; The file PUT inside the db:output statement also gets the namespace added i.e. it happens even when the file was POSTed with curl so the recieved document still had no namespaces. Is this expected behaviour, a bug, or am I using http:send-request incorrectly? Regards, chris sweeney

3 6

Make BXElem return also namespace declarations
by Erdal Karaca 09 Oct '16

09 Oct '16

Hi, I am accessing the DOM tree of the collections in the embedded mode. Now, I need to process the DOM node returned by a query. For this, Item.toJava() returns a DOM Element (for my specific case/query). When calling Element#getAttributes(), I only get "direct" attributes, i.e. no namespace declarations are included in the list of attributes. Are the namespace declarations omitted intentionally? How can I make the BXElem also report them in the resulting attributes list? Thanks.

2 2

embedded mode thread-safe?
by Erdal Karaca 09 Oct '16

09 Oct '16

Hi all, Is the embedded mode thread-safe? I mean, once I have the Context object and opened a database, is it safe to access (read/write) it from multiple threads? Will the same transaction mechanism apply as if the server mode would be used? Thanks.

2 7

copy attributes of one node to another
by Erdal Karaca 09 Oct '16

09 Oct '16

Hi, I am trying to copy attributes of one node to another: copy $x := <test a="b" c="d"></test>, $y := parse-xml('<o a="x" c="y"></o>') modify ( delete nodes $x/@*, insert nodes $y/@* into $x ) return $x Unfortunately, the resulting $x is an empty <test/> node. I expected it to have the attributes of the parsed node. Am I using parse-xml() properly? Thanks!

2 2

problem getting -somit-xml-declaration=no to work
by Regine I. Heberlein 07 Oct '16

07 Oct '16

Hello all, I wonder whether someone could help me with what I'm sure is just some newbie error on my part. I'm running an XQuery Update using BaseX from the command line over an XML document, and I want the XML declaration to be preserved. My input is: basex -uVwsomit-xml-declaration=no <input xQuery Update file> That looks correct to me per the documentation, but when I run it, I lose the XML declaration nevertheless. (The xQuery Update executes correctly.) I've tried the following, all to no avail: --split out the -s flag, putting it as the first and last flag, respectively --setting the value to "yes" (well, I knew that wasn't going to work, but I had to check, right?) --upper case NO (ditto...) --adding the following two lines to my xQuery file: declare namespace output = "http://www.w3.org/2010/xslt-xquery-serialization"; declare option output:omit-xml-declaration "no"; I know this question has come up before, but I couldn't find an approach that worked for me in the list archives. Obviously, at this point I'm just throwing darts at the problem. I'd be grateful for any pointers! Sincerely, Regine Heberlein Principal Cataloger and Metadata Analyst, Rare Book Collections Princeton University Library One Washington Road Princeton, NJ 08540 609-258-6156

2 4

Cannot insert node into empty db/collection
by Erdal Karaca 06 Oct '16

06 Oct '16

Hi, When creating an empty database, then trying to insert a node into its root, I get the following error message: Command: create db test XQuery: insert node <test></test> into / Error: [XUDY0027] Insert target is an empty sequence. How can I populate an empty database? - I am using basex 8.5.3. Thanks!

3 5