- BaseX-Talk - mailman.uni-konstanz.de

EXECUTE syntax
by Ben Engbers 08 Jun '18

08 Jun '18

Hi, I want to use my R-client to insert csv in a database. These lines works: csv_add_run <- 'RUN "./DataScience/RBaseX/CSVexample.xq"' Session$command(csv_add_run) When I take the content from CSVexample.xq and incorporate that into a EXECUTE command, I get this: csv_add_exe <- 'EXECUTE "let $root := '/home/bengbers/DataScience/RBaseX/Examples/Parse/;)'; for $path in file:children($root)[ends-with(., '.csv')] return db:add('CSV_test', $path, 'CSV_API', map { 'parser': 'csv', 'csvparser': map { 'header': 'yes', 'separator': ';' }) "' Session$command(csv_add_exe) Stopped at , 1/9: Unknown command: 'EXECUTE. Did you mean 'EXECUTE'? My question is how to define the input for the EXECUTE-command? Cheers, Ben

2 1

Re: [basex-talk] Long queries on big data timing out.
by Christian Grün 08 Jun '18

08 Jun '18

Dear Peter, 1. and 2. If you are working with the client/server architecture of BaseX, you can adjust the default timeout of 30 seconds by assigning another number of seconds to the TIMEOUT option [1]. This can e.g. be done by editing the .basex configuration file (please find more information in the top of the referenced Wiki document). 3. There is a CACHETIMEOUT option for caching the results of jobs:eval; see [2] for more details. 4. The jobs:list-details function will also list finalized jobs with enabled 'cache' option: <job id="job1" duration="PT1.32S" state="cached" type="QueryJob" ... By checking the @state attribute, you will see if the query is still "running", or if a "cached" result is available. Hope this helps, Christian PS: Your mail did not survive my spam check; maybe something is wrong with the mail configuration? [1] http://docs.basex.org/wiki/Options#TIMEOUT [2] http://docs.basex.org/wiki/Jobs_Module#jobs:eval > Hello, > > We have some complicated queries that are submitted to a database that is really, really big. The system will time out, just bringing back a message saying "Interrupted". > > 1. I know that you can provide a timeout parameter when using jobs:eval etc for batch jobs, but is there a setting that we can tweak to set the default for interactive (i.e. non jobs:eval) queries? > 2. What is the default value for the timeout value? > 3. How long will the server hang on to results from jobs:eval()? I think I remember reading that this is configurable, but how? Is it per job or globally for the server? > 4. Can you consider adding an API that tells how long the job has been running?

1 0

Long queries on big data timing out.
by Peter Villadsen 07 Jun '18

07 Jun '18

Hello, We have some complicated queries that are submitted to a database that is really, really big. The system will time out, just bringing back a message saying "Interrupted". 1. I know that you can provide a timeout parameter when using jobs:eval etc for batch jobs, but is there a setting that we can tweak to set the default for interactive (i.e. non jobs:eval) queries? 2. What is the default value for the timeout value? 3. How long will the server hang on to results from jobs:eval()? I think I remember reading that this is configurable, but how? Is it per job or globally for the server? 4. Can you consider adding an API that tells how long the job has been running? Best Regards Peter Villadsen Principal Architect Microsoft Business Applications Group

1 0

XML well-formed error
by DK Singh 06 Jun '18

06 Jun '18

Hi All, I have a XML file when i apply xslt using xslt:transform function it is giving well formed error, even i tried on BaseX UI using xslt:transform function [FODC0002] "" (Line 1): The markup in the document following the root element must be well-formed. but when the same file i transform on the oxygeng editor it getting transformed, can anyone suggest what going wrong. Regards Dharmendra Kumar Singh

4 8

Script tags
by Philip Spaelti 02 Jun '18

02 Jun '18

Hi, I'm using the GUI to generate html documents. The xquery includes a <script src="my_script.js"></script> tag which Basex simplifies to <script src="my_script.js"/> which causes problems in the browser. Am I doing something wrong, or is this a bug? best Philip --------------------------------------------------- Philip Spaelti Kobe Shoin Women's University, Graduate school ---------------------------------------------------

2 1

Insert CSV into database
by Ben Engbers 01 Jun '18

01 Jun '18

Hi, My goals is to use my R clientdriver to insert csv-files into a new databases. But before that, I'm experimenting with the GUI. >From the documentation for the CSV-parser, I have taken this code: for $file in file:list("/home/bengbers/DataScience/RBaseX/Examples/Parse", false(), "*.csv") return db:add("CSV_test", $file, "", map { 'parser': 'csv', 'csvparser': map { 'header': 'yes', 'separator': ';' } }) BaseX returns: Error: Stopped at /home/bengbers/DataScience/Eindopdracht/Data/file, 2/14: [FODC0002] Resource 'Test_Parse.csv' does not exist. I didn't enter this path. It was used yesterday when browsing to the datafiles that were inserted into another test-database. let $file := file:list("/home/bengbers/DataScience/RBaseX/Examples/Parse", false(), "*.csv") return $file BaseX returns: Test_Parse.csv Test_Parse (exemplaar).csv If I create a new database, it neatly adds the two csv-files. My questions are: - which query I have to use to insert csv-files? - obviously, BasexGUI uses the wrong path. How should I adjust this path? Cheers, Ben

2 3

BaseX 9.0.2: New Maintenance Release
by Christian Grün 31 May '18

31 May '18

Hi all, A new BaseX maintenance release is available! Please visit http://basex.org to download the latest version. These are the changes: Critical bug fixes: - XQuery: Node ordering of mixed database/fragment instances - XQuery: Index rewritings of paths with nested positional predicates Improvements: - WebDAV: Much faster locking of database resources - XQuery: Enable ENFORCEINDEX for variable predicate arguments - GUI: Larger interaction components, unified font sizes Minor Bug fixes: - DBA: Suppress menu on login page - Commands: Nested evaluation with variable bindings - XQuery: Static typing of lookup expressions Enjoy, Christian

1 0

Serialize csv xquery map
by Andy Bunce 29 May '18

29 May '18

The xqueryformat of CSV records as a sequence of arrays [1] is very useful when parsing but there seems to be no easy way to serialize from this format. Is this correct? map { "records": ["Huber", "Sepp", "Hauptstraße 13", "93547 Hintertupfing"], "names": ["Name", "First_Name", "Address", "City"] } =>csv:serialize() [SENR0001] Items of type map(xs:string, array(xs:string)) cannot be serialized. ... would be nice if it could. /Andy [1] http://docs.basex.org/wiki/CSV_Module#XQuery

2 2

IETF Date in If-Modified-Since header
by Marco Lettere 29 May '18

29 May '18

Hello all, need to use the If-Modified-Since header in one of my APIs. The format requested is the IETF date format: If-Modified-Since: Wed, 21 Oct 2015 07:28:00 GMT as explained in [1]. If I pass the header as prescribed to a restxq made like: declare %rest:path("config") %rest:header-param("If-Modified-Since", "{$modifiedsince-ietf}") %rest:HEAD function config:check($modifiedsince-ietf as xs:string?){ I get the following exception: Cannot convert xs:string+ to xs:string?: ("Wed", "21 Oct 2015 07:28:00 GMT"). Which somehow demonstrates that , is treated as a separator of values for the header. I patched by replacing ? with * and using a string-join on the $modifiedsince-ietf before passing it to the follwoing parse-ietf-date function. But this is hard to explain and probably subject to other issues in the future. Is there a way to pass HTTP headers verbatim to the RestXQ function? thanks! M. [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Modified-Since

2 2

no value assigned to external variables in xq when running bxs->bxs->xq
by Freihalter Thomas 29 May '18

29 May '18

Hello, I am using basex 9.0.1 and have this 3 files: test.xq: declare variable $test external; $test test.bxs: <commands> <set option="BINDINGS">test=foo</set> <run file="test.xq"/> </commands> run.bxs: <commands> <run file="test.bxs"/> </commands> Running test.bxs is okay basex -v test.bxs BINDINGS: test=foo foo Query "test.xq" executed in 5.4 ms. basex -v run.bxs BINDINGS: test=foo Stopped at test.xq, 1/18: [XPDY0002] No value assigned to $test. It look like the bindings in test.bxs get lost, when executing the superior run.bxs. Is there any solution solving this problem? 8.6.7 does not work as well. regard Thomas echo "declare variable \$test external; \$test" > test.xq; echo "<commands><set option=\"BINDINGS\">test=foo</set><run file=\"test.xq\"/></commands>" > test.bxs; echo "<commands><run file=\"test.bxs\"/></commands>" > run.bxs; basex -v test.bxs; basex -v run.bxs

2 2

progress of query execution
by Ветошкин Владимир 29 May '18

29 May '18

Hi, everybody! My core has written in php and php-script execute xquery-scripts to get needed data. To get the progress of one large query I have to divide it into some small ones. Isn't it? Are there any more options? Maybe it is possible to get number of seconds, which needed to complete execution of query? -- С уважением, Ветошкин Владимир Владимирович

2 1

Controlling json serialization of JSON-XML
by Hondros, Constantine (ELS-AMS) 29 May '18

29 May '18

Hello all, I'm expecting the answer to be no, but is it possible to influence the JSON serialization of <number> elements at all? xml-to-json( <map xmlns="http://www.w3.org/2005/xpath-functions"> <number key="number">10111234</number> </map> ) Results in {"number":1.0111234E7} I read this in the spec, so it looks like the number is always being cast as a double: xs:string(xs:double(fn:string($E))) TIA, Constantine ________________________________ Elsevier B.V. Registered Office: Radarweg 29, 1043 NX Amsterdam, The Netherlands, Registration No. 33156677, Registered in The Netherlands.

2 1

Fwd: BaseX performance improvements
by Christian Grün 28 May '18

28 May '18

Hi Martin (cc to the list), > 1. each XML node => 500 bytes > 2. total size of XML DB => 10million nodes(which will grow continuously as > new files are added). So one node is one document, right? I’m just asking because, in the XML terminology, each XML document has nodes itself: element nodes, attributes nodes, text nodes, etc. If you are working with millions of documents, it can be helpful to work with a "daily" database, which contains auxiliary references on documents that have been deleted or updated. If there is a time interval in which no people access your database (e.g. each night), you can merge these databases and recreate your index structures. If you query your database, you can a few more lines of XQuery: first check your daily database, and (if it doesn’t contain the required document) look up the document in your fully indexed database. > I have to do a update of the nodes as I need to replace them. I chose delete > + insert instead of replace Please note that a replace operation can be much faster: If the structure of the updated document is similar to the old document, the document can be replace in-place. > Also, I am thinking of indexing on the node id, as that is like the primary > key on which I base my operations, any suggestions might also be helpful, Here it may also be interesting to know if you will only address documents or arbitrary nodes of your document. In the first case, addressing the document path via db:open should be sufficient [1]. In the latter case, you could possibly use the existing function db:node-id [2]. Hope this helps, Christian [1] http://docs.basex.org/wiki/Databases [2] http://docs.basex.org/wiki/Database_Module#db:node-id > Thanks again, > Regards > Martin Lourduswamy > > On Sun, May 20, 2018 at 9:28 AM, Christian Grün <christian.gruen(a)gmail.com> > wrote: >> >> Hi Martin, >> >> > I am new to BaseX, I would like ot speed up XQuery of insert and delete >> > and >> > replace through options. >> >> Welcome to the list. As there are numerous ways to do updates in >> BaseX, feel free to give us more information on your insert and delete >> operations. Do you work with large single documents or many small >> documents? How large is your database? >> >> > While I query basex through perl, and try to >> > connect through GUI, the perl connection aborts. Is there a parameters >> > for >> > parallel connections, please let me know. >> >> I can’t tell why your perl connection is interrupted by opening the >> GUI (because they should be completely independent from each other). A >> step-by-step description on how you proceeded might be helpful. >> >> Because there is no coupling between GUI and the client/server >> architecture, however, you should avoid running updates outside the >> GUI. Please check out [1] for more information. >> >> Best, >> Christian >> >> [1] http://docs.basex.org/wiki/Startup#Concurrent_Operations >> >> >> >> > >> > Thanks, >> > Regards >> > Martin Lourduswamy > >

2 2

-math:log10(1)
by Giuseppe Celano 28 May '18

28 May '18

- math:log10(1) returns -0 but -0 returns 0: is there a reason for that? Thanks! Giuseppe Universität Leipzig Institute of Computer Science, NLP Augustusplatz 10 04109 Leipzig Deutschland E-mail: celano(a)informatik.uni-leipzig.de E-mail: giuseppegacelano(a)gmail.com Web site 1: http://www.dh.uni-leipzig.de/wo/team/ Web site 2: https://sites.google.com/site/giuseppegacelano/

3 2

Data analysis
by Tim Clement 26 May '18

26 May '18

Hello, I’m new to basex and really like the functionality. I have very complex XML files with deep nesting. Each file is like a document with test output values and parameters. Basex ingested the files easily and I’m able to write queries to pull out data. My end goal is to plot, graph trend data. I can do basic plotting within Basex. Do you have any suggestions for tools to visualize the data trending ? I have Power BI , but its’ not great with XML input. Thanks in advance Tim

2 1

How to pass a document as context to QueryProcessor
by wd＠creativosdigitales.co 25 May '18

25 May '18

Hi, Im writing a Java program to execute an XQuery using the BaseX QueryProcessor class, and need to pass an XML document as context node. As far Ive found in the documentation and java examples, this is done with the context() method of QueryProcessor, which receives an instance of the Value class as parameter. But the examples available are about passing simple types (strings o numbers) but nothing about passing an XML node. My question is: Where can I find examples or documentation on how to pass XML nodes as parameters to QueryProcessor.context() ? Thanks for all your help, William David Velásquez Creativo de Software Creativos Digitales S.A.S. Calle 30A # 83 - 53 Local 1033 Tel: 322 1730 - 311 709 8421 Medellín, Colombia ¿Necesita Integrar Sistemas? Conozca nuestra solución de Integración no Invasiva: <http://creativosdigitales.co> http://creativosdigitales.co ¿Preguntas sobre Facturación Electrónica? Visite: <http://facturasyrespuestas.com/> http://facturasyrespuestas.com

2 2

Deploying BaseX + data + client
by Jonathan Robie 25 May '18

25 May '18

I have written a Python package for querying Greek syntax from Jupyter Notebook. It relies on BaseX and one or more open data resources found in GitHub repositories. I want to make it available to the public in a way that is easy to install. One possibility is to write a Python script that uses Docker to install BaseX, retrieves the data and installs it in the desired directory structure - perhaps in the container, then loads it into the database. I assume that would work cleanly cross-platform? Are there better approaches that I should consider? Are there security considerations that I should consider? Jonathan

1 0

Duplicate files when using webDAV and batch processes
by France Baril 25 May '18

25 May '18

Hi, We've been having this issue for a while and we think resolving it may be the key to resolving an intermittent server 500 error that we've been having. When a user tries to save a file while a batch process is running, BaseX saves duplicates of the file. How to reproduce: 1) Take a fresh BaseX 9.0.1 installation 2) Copy the attached .xqm in webapp 3) Create an empty DB called mydb 4) Access localhost:port-num/test/create-update-a-lot-of-files to populate your db. 5) In OxygenXML, set a webdav connection to the db and open a file, add a character in one of the elements, but don't save the file. 6) From the browser, access 'localhost:port-num/test/update-something' 7) While the process in the browser is still running, save the file in Oxygen. You'll get a message saying that read timed out. Click ok and do not try saving the file again. 8) When the update-something process is done running, don't resave the file in Oxygen, instead go to localhost:port-num/test/oups-duplicates. You'll get a message saying that some files are duplicated. If you don't try again from step #4 a few times. You'll only get duplicates if you get the time out message before the update-something process is still running. If you try to save the file many times, you'll get more duplicates, 4 or 6. We're not sure if it's a BaseX bug or if we are setting our user management and/or locking rules incorrectly. Do you have any suggestions? -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 10

Re: [basex-talk] basex trouble
by Christian Grün 24 May '18

24 May '18

Hi Ветошкин (cc to the list), Maybe we find a way to speed up your queries; you can attach them to your next mail. I guess that the parallel query execution leads to "random access patterns" on disk. You can enforce one query at a time by setting the PARALLEL option to 1 (see [1]). Hope this helps, Christian [1] http://docs.basex.org/wiki/Options#PARALLEL On Wed, May 23, 2018 at 9:40 PM, Ветошкин Владимир <en-trance(a)yandex.ru> wrote: > Hi, Christian! > > Thank you. > > I have two scripts, which make queries to one database. > If I run these queries at different times, each of them completes in about 5 > sec. > But if I run both scripts at one time, it takes about 40 sec. > > If the scripts make queries to different databases - it's ok, it takes about > 5 sec. > > Is it possible to improve this situation? > > I tried to use this code to load database in main-memory: > db:open('db1') update {} > But it didn't help. > > I hope, you understand me :) > > > 23.05.2018, 20:15, "Christian Grün" <christian.gruen(a)gmail.com>: > > Привет Ветошкин, > > Welcome to the list. Just send your question to this address. > > Best, > Christian > > > > Ветошкин Владимир <en-trance(a)yandex.ru> schrieb am Mi., 23. Mai 2018, 19:10: > > Hello! > > Sorry for my English. > May I ask you some questions here or I have to write them on github? > > -- > С уважением, > Ветошкин Владимир Владимирович > > > > -- > С уважением, > Ветошкин Владимир Владимирович >

2 5

Atomization
by Giuseppe Celano 24 May '18

24 May '18

I think I have identified a problem with atomization of attribute content (no database involved). I have a simple query: for $s in doc("doc1")//s//t for $d in doc("doc2")//case where $d/verb_lemma = $s/@l and $d//verb_form/@value = $s/@f and $d/aspect-values/@sign = "yes" return $s In order to get a result, I (necessarily) need to use the data() function in data($d/aspect-values/@sign) = "yes", otherwise the query never returns a result. Is this a bug? I would expect that the value of @sign is automatically atomized and compared to "yes", but this does not seem the case. Thanks. Ciao, Giuseppe Universität Leipzig Institute of Computer Science, NLP Augustusplatz 10 04109 Leipzig Deutschland E-mail: celano(a)informatik.uni-leipzig.de E-mail: giuseppegacelano(a)gmail.com Web site 1: http://www.dh.uni-leipzig.de/wo/team/ Web site 2: https://sites.google.com/site/giuseppegacelano/

2 3

basex trouble
by Ветошкин Владимир 23 May '18

23 May '18

Hello! Sorry for my English. May I ask you some questions here or I have to write them on github? -- С уважением, Ветошкин Владимир Владимирович

2 1

New BaseX setup
by France Baril 23 May '18

23 May '18

Hi, In one of my projects, we are trying to refactor our code so it uses a node.js client. We have extensive functions that do more than just get data in our .xqm. They were implemented around the http server and our web application was calling the REST functions we implemented using jquery AJAX request. All the examples we looked at in the documentation that don't use the http server pass entire queries as a string to the server. We were wondering if we can call functions from our .xqm modules directly (ex. namespace:do-this(param0, param1)) or if any request that uses functions from our .xqm files need to pass through the rest API. If we can call our .xqm functions more directly, could you point us to an example to help us understand how it would work? Thanks! -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

4 4

(no subject)
by Kirsten, Dirk 23 May '18

23 May '18

Dear list, I tried to insert a huge XML document (english wikipedia). When doing so and using the default parser I got the following warning: "/my/path/enwiki-latest-pages-articles.xml" (Line 61696838): JAXP00010004: The accumulated size of entities is "50,000,001" that exceeded the "50,000,000" limit set by "FEATURE_SECURE_PROCESSING". I solved this by setting the environment variables: -DentityExpansionLimit=2147480000 -DtotalEntitySizeLimit=2147480000 -Djdk.xml.totalEntitySizeLimit=2147480000 So it seems this is a security measure from JXB. Would it be possible to set these values by default or at least be able to change the limit with the GUI or BaseX options? Cheers Dirk Dirk Kirsten ______________________________ Senacor Technologies AG D?sseldorfer Stra?e 13 65760 Eschborn M +49 (151) 4435 1967 Dirk.Kirsten(a)senacor.com www.senacor.com Senacor Technologies Aktiengesellschaft - Sitz: Eschborn - Amtsgericht Frankfurt am Main - Reg.-Nr.: HRB 105546 Vorstand: Matthias Tomann, Marcus Purzer - Aufsichtsratsvorsitzender: Daniel Gr?zinger Diese E-Mail inklusive Anlagen enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten, informieren Sie bitte den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet. This e-mail including any attachments may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the materials in this e-mail is strictly forbidden.

2 1

Re: [basex-talk] BaseX-Talk Digest, Vol 101, Issue 37
by Kristian Kankainen 22 May '18

22 May '18

I have a setup using rsync to keep database directory copies synchronized with a master over the network. It works very well for me.22. mai 2018 10:16 kirjutas kuupäeval Alexander Holupirek <alex(a)holupirek.de>: > > Hi Martin, > > export/import is one option. backup/restore another [1]. > And a third could be to copy the database directory (provided the servers have a more or less similar setup). > > Cheers, > Alex > > [1] http://docs.basex.org/wiki/Backups (can also be done via DBA [2]) > [2] http://docs.basex.org/wiki/DBA > > > On 21. May 2018, at 18:32, Martin Lourduswamy <martin.louis(a)gmail.com> wrote: > > > > Hi, > > I would like to know how to copy a database from one server to another server, please let me know. > > Is it export and get and load xml or is there another way to do it, thanks for the clarification, > > Regards > > Martin Lourduswamy > > > > On Mon, May 21, 2018 at 6:00 AM, <basex-talk-request(a)mailman.uni-konstanz.de> wrote: > > Send BaseX-Talk mailing list submissions to > > basex-talk(a)mailman.uni-konstanz.de > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > or, via email, send a message with subject or body 'help' to > > basex-talk-request(a)mailman.uni-konstanz.de > > > > You can reach the person managing the list at > > basex-talk-owner(a)mailman.uni-konstanz.de > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of BaseX-Talk digest..." > > > > > > Today's Topics: > > > > 1. Fwd: BaseX performance improvements (Christian Gr?n) > > 2. Re: BaseX performance improvements (Christian Gr?n) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Sun, 20 May 2018 18:51:03 +0200 > > From: Christian Gr?n <christian.gruen(a)gmail.com> > > To: Martin Lourduswamy <martin.louis(a)gmail.com>, BaseX > > <basex-talk(a)mailman.uni-konstanz.de> > > Subject: [basex-talk] Fwd: BaseX performance improvements > > Message-ID: > > <CAP94bnPHW=NqzXxY-cfubGSLzWLoiiPzj_WfNj6tWYZFg0DtJA(a)mail.gmail.com> > > Content-Type: text/plain; charset="UTF-8" > > > > Hi Martin (cc to the list), > > > > > 1. each XML node => 500 bytes > > > 2. total size of XML DB => 10million nodes(which will grow continuously as > > > new files are added). > > > > So one node is one document, right? I?m just asking because, in the > > XML terminology, each XML document has nodes itself: element nodes, > > attributes nodes, text nodes, etc. > > > > If you are working with millions of documents, it can be helpful to > > work with a "daily" database, which contains auxiliary references on > > documents that have been deleted or updated. If there is a time > > interval in which no people access your database (e.g. each night), > > you can merge these databases and recreate your index structures. If > > you query your database, you can a few more lines of XQuery: first > > check your daily database, and (if it doesn?t contain the required > > document) look up the document in your fully indexed database. > > > > > I have to do a update of the nodes as I need to replace them. I chose delete > > > + insert instead of replace > > > > Please note that a replace operation can be much faster: If the > > structure of the updated document is similar to the old document, the > > document can be replace in-place. > > > > > Also, I am thinking of indexing on the node id, as that is like the primary > > > key on which I base my operations, any suggestions might also be helpful, > > > > Here it may also be interesting to know if you will only address > > documents or arbitrary nodes of your document. In the first case, > > addressing the document path via db:open should be sufficient [1]. In > > the latter case, you could possibly use the existing function > > db:node-id [2]. > > > > Hope this helps, > > Christian > > > > [1] http://docs.basex.org/wiki/Databases > > [2] http://docs.basex.org/wiki/Database_Module#db:node-id > > > > > > > > > Thanks again, > > > Regards > > > Martin Lourduswamy > > > > > > On Sun, May 20, 2018 at 9:28 AM, Christian Gr?n <christian.gruen(a)gmail.com> > > > wrote: > > >> > > >> Hi Martin, > > >> > > >> > I am new to BaseX, I would like ot speed up XQuery of insert and delete > > >> > and > > >> > replace through options. > > >> > > >> Welcome to the list. As there are numerous ways to do updates in > > >> BaseX, feel free to give us more information on your insert and delete > > >> operations. Do you work with large single documents or many small > > >> documents? How large is your database? > > >> > > >> > While I query basex through perl, and try to > > >> > connect through GUI, the perl connection aborts. Is there a parameters > > >> > for > > >> > parallel connections, please let me know. > > >> > > >> I can?t tell why your perl connection is interrupted by opening the > > >> GUI (because they should be completely independent from each other). A > > >> step-by-step description on how you proceeded might be helpful. > > >> > > >> Because there is no coupling between GUI and the client/server > > >> architecture, however, you should avoid running updates outside the > > >> GUI. Please check out [1] for more information. > > >> > > >> Best, > > >> Christian > > >> > > >> [1] http://docs.basex.org/wiki/Startup#Concurrent_Operations > > >> > > >> > > >> > > >> > > > >> > Thanks, > > >> > Regards > > >> > Martin Lourduswamy > > > > > > > > > > > > ------------------------------ > > > > Message: 2 > > Date: Sun, 20 May 2018 18:56:45 +0200 > > From: Christian Gr?n <christian.gruen(a)gmail.com> > > To: Martin Lourduswamy <martin.louis(a)gmail.com>, BaseX > > <basex-talk(a)mailman.uni-konstanz.de> > > Subject: Re: [basex-talk] BaseX performance improvements > > Message-ID: > > <CAP94bnN6w7ZaaNu43AqdzrKYsaqaxKGSamRNP8zy3fmEEJZ6Hg(a)mail.gmail.com> > > Content-Type: text/plain; charset="UTF-8" > > > > > Java heap space at C:\Users\lourduswamym\oracle_scripts/BaseXClient.pm line > > > 213. > > > > How much memory have you assigned to BaseX [1]? > > > > > Actually the error on the server log. The insert operation was taking 1+minutes due to slow network and then the error happened. I have been running this server instance continuously for 1 day with continuous insert from a per script. > > > > If you create a new database, you can specify a directory with the > > files that are to be initially added to your database. This might save > > you a lot of time. > > > > > So I think I should be careful to run only one single process of client at a > > > time or make sure memory is enough to handle multiple client operations, > > > please let me know your thoughts, > > > > If your documents are small, you shouldn?t usually encounter any > > errors. But it might have to do with the large number of documents you > > are dealing with. One more option is to distribute your docs across > > multiple databases (you can access all of them via a single XQuery > > expression). > > > > Best, > > Christian > > > > [1] http://docs.basex.org/wiki/Start_Scripts > > > > > > > > > > > Thanks again, > > > Regards > > > Martin Lourduswamy > > > > > > On Sun, May 20, 2018 at 9:48 AM, Martin Lourduswamy <martin.louis(a)gmail.com> > > > wrote: > > >> > > >> Hi, > > >> > > >> Thanks for the clarifications. My database is > > >> > > >> DB Size: > > >> ====== > > >> 1. each XML node => 500 bytes > > >> 2. total size of XML DB => 10million nodes(which will grow continuously as > > >> new files are added). > > >> 3. I will be having multiple DB's of same size and more size as the DB > > >> continuously grows > > >> > > >> I have to do a update of the nodes as I need to replace them. I chose > > >> delete + insert instead of replace for each of the node through perl, as I > > >> will make sure there are no duplicate nodes left in the database.(just a > > >> precaution that even if some nodes get duplicated, through some other > > >> processes I will be able to remove them by delete) > > >> > > >> DB Speed: > > >> ======== > > >> I will disable logging as it might speed up the database operations. > > >> > > >> DB Architecture > > >> ============ > > >> 1 server(windows or Linux) running baseX > > >> multiple client computers trying to query the same machine for select > > >> through fn:doc(...) API to get the data through perl > > >> then do a delete + insert from perl > > >> > > >> I would like to know if you think this might work for production and any > > >> suggestion for data retrieval and update > > >> Any changes to the way I architecture it might help > > >> > > >> Also, I am thinking of indexing on the node id, as that is like the > > >> primary key on which I base my operations, any suggestions might also be > > >> helpful, > > >> Thanks again, > > >> Regards > > >> Martin Lourduswamy > > >> > > >> On Sun, May 20, 2018 at 9:28 AM, Christian Gr?n > > >> <christian.gruen(a)gmail.com> wrote: > > >>> > > >>> Hi Martin, > > >>> > > >>> > I am new to BaseX, I would like ot speed up XQuery of insert and delete > > >>> > and > > >>> > replace through options. > > >>> > > >>> Welcome to the list. As there are numerous ways to do updates in > > >>> BaseX, feel free to give us more information on your insert and delete > > >>> operations. Do you work with large single documents or many small > > >>> documents? How large is your database? > > >>> > > >>> > While I query basex through perl, and try to > > >>> > connect through GUI, the perl connection aborts. Is there a parameters > > >>> > for > > >>> > parallel connections, please let me know. > > >>> > > >>> I can?t tell why your perl connection is interrupted by opening the > > >>> GUI (because they should be completely independent from each other). A > > >>> step-by-step description on how you proceeded might be helpful. > > >>> > > >>> Because there is no coupling between GUI and the client/server > > >>> architecture, however, you should avoid running updates outside the > > >>> GUI. Please check out [1] for more information. > > >>> > > >>> Best, > > >>> Christian > > >>> > > >>> [1] http://docs.basex.org/wiki/Startup#Concurrent_Operations > > >>> > > >>> > > >>> > > >>> > > > >>> > Thanks, > > >>> > Regards > > >>> > Martin Lourduswamy > > >> > > >> > > > > > > > > > End of BaseX-Talk Digest, Vol 101, Issue 37 > > ******************************************* > > >

1 0

Re: [basex-talk] BaseX-Talk Digest, Vol 101, Issue 37
by Martin Lourduswamy 22 May '18

22 May '18

Hi, I would like to know how to copy a database from one server to another server, please let me know. Is it export and get and load xml or is there another way to do it, thanks for the clarification, Regards Martin Lourduswamy On Mon, May 21, 2018 at 6:00 AM, <basex-talk-request(a)mailman.uni-konstanz.de > wrote: > Send BaseX-Talk mailing list submissions to > basex-talk(a)mailman.uni-konstanz.de > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > or, via email, send a message with subject or body 'help' to > basex-talk-request(a)mailman.uni-konstanz.de > > You can reach the person managing the list at > basex-talk-owner(a)mailman.uni-konstanz.de > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of BaseX-Talk digest..." > > > Today's Topics: > > 1. Fwd: BaseX performance improvements (Christian Gr?n) > 2. Re: BaseX performance improvements (Christian Gr?n) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 20 May 2018 18:51:03 +0200 > From: Christian Gr?n <christian.gruen(a)gmail.com> > To: Martin Lourduswamy <martin.louis(a)gmail.com>, BaseX > <basex-talk(a)mailman.uni-konstanz.de> > Subject: [basex-talk] Fwd: BaseX performance improvements > Message-ID: > <CAP94bnPHW=NqzXxY-cfubGSLzWLoiiPzj_WfNj6tWYZFg0DtJA(a)mail.gmail. > com> > Content-Type: text/plain; charset="UTF-8" > > Hi Martin (cc to the list), > > > 1. each XML node => 500 bytes > > 2. total size of XML DB => 10million nodes(which will grow continuously > as > > new files are added). > > So one node is one document, right? I?m just asking because, in the > XML terminology, each XML document has nodes itself: element nodes, > attributes nodes, text nodes, etc. > > If you are working with millions of documents, it can be helpful to > work with a "daily" database, which contains auxiliary references on > documents that have been deleted or updated. If there is a time > interval in which no people access your database (e.g. each night), > you can merge these databases and recreate your index structures. If > you query your database, you can a few more lines of XQuery: first > check your daily database, and (if it doesn?t contain the required > document) look up the document in your fully indexed database. > > > I have to do a update of the nodes as I need to replace them. I chose > delete > > + insert instead of replace > > Please note that a replace operation can be much faster: If the > structure of the updated document is similar to the old document, the > document can be replace in-place. > > > Also, I am thinking of indexing on the node id, as that is like the > primary > > key on which I base my operations, any suggestions might also be helpful, > > Here it may also be interesting to know if you will only address > documents or arbitrary nodes of your document. In the first case, > addressing the document path via db:open should be sufficient [1]. In > the latter case, you could possibly use the existing function > db:node-id [2]. > > Hope this helps, > Christian > > [1] http://docs.basex.org/wiki/Databases > [2] http://docs.basex.org/wiki/Database_Module#db:node-id > > > > > Thanks again, > > Regards > > Martin Lourduswamy > > > > On Sun, May 20, 2018 at 9:28 AM, Christian Gr?n < > christian.gruen(a)gmail.com> > > wrote: > >> > >> Hi Martin, > >> > >> > I am new to BaseX, I would like ot speed up XQuery of insert and > delete > >> > and > >> > replace through options. > >> > >> Welcome to the list. As there are numerous ways to do updates in > >> BaseX, feel free to give us more information on your insert and delete > >> operations. Do you work with large single documents or many small > >> documents? How large is your database? > >> > >> > While I query basex through perl, and try to > >> > connect through GUI, the perl connection aborts. Is there a parameters > >> > for > >> > parallel connections, please let me know. > >> > >> I can?t tell why your perl connection is interrupted by opening the > >> GUI (because they should be completely independent from each other). A > >> step-by-step description on how you proceeded might be helpful. > >> > >> Because there is no coupling between GUI and the client/server > >> architecture, however, you should avoid running updates outside the > >> GUI. Please check out [1] for more information. > >> > >> Best, > >> Christian > >> > >> [1] http://docs.basex.org/wiki/Startup#Concurrent_Operations > >> > >> > >> > >> > > >> > Thanks, > >> > Regards > >> > Martin Lourduswamy > > > > > > > ------------------------------ > > Message: 2 > Date: Sun, 20 May 2018 18:56:45 +0200 > From: Christian Gr?n <christian.gruen(a)gmail.com> > To: Martin Lourduswamy <martin.louis(a)gmail.com>, BaseX > <basex-talk(a)mailman.uni-konstanz.de> > Subject: Re: [basex-talk] BaseX performance improvements > Message-ID: > <CAP94bnN6w7ZaaNu43AqdzrKYsaqaxKGSamRNP8zy3fmEEJZ6Hg@mail. > gmail.com> > Content-Type: text/plain; charset="UTF-8" > > > Java heap space at C:\Users\lourduswamym\oracle_scripts/BaseXClient.pm > line > > 213. > > How much memory have you assigned to BaseX [1]? > > > Actually the error on the server log. The insert operation was taking > 1+minutes due to slow network and then the error happened. I have been > running this server instance continuously for 1 day with continuous insert > from a per script. > > If you create a new database, you can specify a directory with the > files that are to be initially added to your database. This might save > you a lot of time. > > > So I think I should be careful to run only one single process of client > at a > > time or make sure memory is enough to handle multiple client operations, > > please let me know your thoughts, > > If your documents are small, you shouldn?t usually encounter any > errors. But it might have to do with the large number of documents you > are dealing with. One more option is to distribute your docs across > multiple databases (you can access all of them via a single XQuery > expression). > > Best, > Christian > > [1] http://docs.basex.org/wiki/Start_Scripts > > > > > > Thanks again, > > Regards > > Martin Lourduswamy > > > > On Sun, May 20, 2018 at 9:48 AM, Martin Lourduswamy < > martin.louis(a)gmail.com> > > wrote: > >> > >> Hi, > >> > >> Thanks for the clarifications. My database is > >> > >> DB Size: > >> ====== > >> 1. each XML node => 500 bytes > >> 2. total size of XML DB => 10million nodes(which will grow continuously > as > >> new files are added). > >> 3. I will be having multiple DB's of same size and more size as the DB > >> continuously grows > >> > >> I have to do a update of the nodes as I need to replace them. I chose > >> delete + insert instead of replace for each of the node through perl, > as I > >> will make sure there are no duplicate nodes left in the database.(just a > >> precaution that even if some nodes get duplicated, through some other > >> processes I will be able to remove them by delete) > >> > >> DB Speed: > >> ======== > >> I will disable logging as it might speed up the database operations. > >> > >> DB Architecture > >> ============ > >> 1 server(windows or Linux) running baseX > >> multiple client computers trying to query the same machine for select > >> through fn:doc(...) API to get the data through perl > >> then do a delete + insert from perl > >> > >> I would like to know if you think this might work for production and any > >> suggestion for data retrieval and update > >> Any changes to the way I architecture it might help > >> > >> Also, I am thinking of indexing on the node id, as that is like the > >> primary key on which I base my operations, any suggestions might also be > >> helpful, > >> Thanks again, > >> Regards > >> Martin Lourduswamy > >> > >> On Sun, May 20, 2018 at 9:28 AM, Christian Gr?n > >> <christian.gruen(a)gmail.com> wrote: > >>> > >>> Hi Martin, > >>> > >>> > I am new to BaseX, I would like ot speed up XQuery of insert and > delete > >>> > and > >>> > replace through options. > >>> > >>> Welcome to the list. As there are numerous ways to do updates in > >>> BaseX, feel free to give us more information on your insert and delete > >>> operations. Do you work with large single documents or many small > >>> documents? How large is your database? > >>> > >>> > While I query basex through perl, and try to > >>> > connect through GUI, the perl connection aborts. Is there a > parameters > >>> > for > >>> > parallel connections, please let me know. > >>> > >>> I can?t tell why your perl connection is interrupted by opening the > >>> GUI (because they should be completely independent from each other). A > >>> step-by-step description on how you proceeded might be helpful. > >>> > >>> Because there is no coupling between GUI and the client/server > >>> architecture, however, you should avoid running updates outside the > >>> GUI. Please check out [1] for more information. > >>> > >>> Best, > >>> Christian > >>> > >>> [1] http://docs.basex.org/wiki/Startup#Concurrent_Operations > >>> > >>> > >>> > >>> > > >>> > Thanks, > >>> > Regards > >>> > Martin Lourduswamy > >> > >> > > > > > End of BaseX-Talk Digest, Vol 101, Issue 37 > ******************************************* >

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk