BaseX-Talk

basex-talk@mailman.uni-konstanz.de

5251 discussions

How to make an email database
by Finney, Tim 16 Jun '16

16 Jun '16

Hi Everyone, I'm thinking about what might be involved in making a BaseX email database that would allow large numbers of emails to be searched on things like sender, recipient, subject, date, contents, as well as an existing hierarchical structure for organizing emails. (E.g. all emails relating to a particular project are placed in a corresponding project folder.) The emails would have to be periodically exported from an existing email archive, presumable going by way of a (very big) PST file. Has anyone here already tried to do this kind of thing using BaseX? Any tips would be much appreciated. Does anyone know of a good open source program for converting PST files to an XML structure? (I don't care about attachments at the moment, though it would be good to eventually include non-binary attachments in the search facility.) Best, Tim Finney

2 1

climbing around in maps
by Graydon Saunders 15 Jun '16

15 Jun '16

Hi -- I'm using 8.4.4. So I'm got a 15 MiB XML file . It consists of a bunch of definitions, which reference other definitions in the same file. It's not guaranteed to not be circular in the general case but I expect it isn't in my particular use case. (And I've stuck a "don't get too deep" check in anyway, which is currently sitting at 1 in the hopes that wouldn't blow up.) The idea is to find the paths to some particular values by started at those values and climbing up the references until a magic "this is the top of a chain of references" point is reached. (Which can be detected via a pattern in the name of the definition.) I stuffed the things I expect to care about into maps, since that way I process it once, and can hopefully reference it thereafter. The downside is that the stack dies instantly when I attempt to recurse up the (virtual, rather than XML-document-actual) tree. (I've never seen BaseX run the query plan along the bottom bar in lieu of an error message before.) I've attached the full query info. I can't attach the data because it's somebody else's. (entry has relationship children; an entry describes an XSD schema, and a relationship element describes (possibly through reference to another entry element than its ancestor) either a complex type or a simple type that goes in that schema. There's only one entry with a given @name but many relationship elements may reference it. As a result, the scope of what's being climbed expands rapidly; this first test "do one name" turns into 20 names on the first pass. ) The query looks like: declare function local:climb($relByType as map(*), $built as xs:string*, $name as xs:string) { for $path in $relByType($name) return if (matches($path,'^\p{Lu}{4}_IN') or count($path) gt 1) then ($name,$built) else local:climb($relByType,($name,$built),$path) }; let $relationships as element(relationship)+ := //relationship[@type] let $relByType as map(*) := map:merge(for $x in $relationships let $key := $x/@type group by $key return map:entry($key,$x/ancestor::entry[1]/@name)) let $fixedRelationships as element(relationship)+ := //relationship[@type][@fixedValue] let $fixed := map:merge(for $x in $fixedRelationships let $key := $x/string-join((@name,@type,@fixedValue),'|') group by $key return map:entry($key,$x/ancestor::entry[1])) (:trying a single fixed value as the start of the climb for testing:) let $eventNames := map:get($fixed,'moodCode|CS|EVN.CRT')/@name (: so we can get the relationship element, but we want its event ancestor's name :) return for $x in $eventNames return local:climb($relByType,'',$x) My suspicion is that passing a big map to the function is causing the problem. But I don't know, and I don't think there's a way to declare a function with an external variable. ("take this value from the function declaration's context and put it in scope for the function"). Any suggestions? -- Graydon

1 1

group explicitly for performance improvements
by Graydon Saunders 15 Jun '16

15 Jun '16

I'm wading about in several tens of megabytes of health care data, trying to figure out what connects to which, as it were. There are a few thousand relationship elements that define some of the connections, and often more than one such element per @type, so it's not enough to just group them by _this_ relationship elements type, I need all the relationship elements which have a particular type value. let $relationships as element(relationship)+ := //relationship[@type] let $relTypes as xs:string+ := distinct-values($relationships/@type/string()) let $relByType := map:merge(for $x in $relTypes return map:entry($x,$relationships[@type eq $x])) return $relByType 82895.5 ms let $relationships as element(relationship)+ := //relationship[@type] let $relByType := map:merge(for $x in $relationships let $key := $x/@type group by $key return map:entry($key,$x)) return $relByType 170.79ms Holy Wow. Offered just in case anyone else is wondering why their query is so slow. -- Graydon

1 0

How to use REST API to add xml document to database
by Adil Hasan 14 Jun '16

14 Jun '16

Hello, Thank you for your work in creating this database management system! I have a question. I have an XML document that is created in an script. I would like to store the xml document in the xml database. The xml file is created in memory and I'm using python and the REST API to store the data. I can't figure out how to add an xml document to an existing database. I tried a POST request with the data being: <command xmlns="http://basex.org/rest"> <text>open test2; add to test21.xml <![CDATA[<greeting>hello3</greeting>]]></text> </command> That fails telling me I cannot do multiple commands at one time. So, I thought I could add 2 command statements with the POST request: <command xmlns="http://basex.org/rest"> <text>open test2</text> </command> <command xmlns="http://basex.org/rest"> <text>add to test21.xml <![CDATA[<greeting>hello3</greeting>]]></text> </command> But, that also doesn't work. Does anyone have a suggestion as to how to add an xml document to an existing database (ideally the xml document is not in a file)? many thanks for your help, adil

2 2

Re: [basex-talk] Nonstandard CSV in RESTXQ
by Christian Grün 13 Jun '16

13 Jun '16

Did you specify "text/comma-separated-values" as Content-Type? Am 12.06.2016 8:05 nachm. schrieb "Alexander Shpack" <shadowkin(a)gmail.com>: > Thank you, Christian! > > But it doesn't work as expected > > declare > %rest:path("/upload") > %rest:POST("{$csv}") > %input:csv("header=false,encoding=CP1251,separator=semicolon") > %output:method("html") > function page:uploadFile($csv as document-node()) > { > $csv > }; > > > [XPTY0004] Cannot cast xs:string to document-node(): "?????????? ?.... > > > On Sun, Jun 12, 2016 at 6:48 PM, Christian Grün <christian.gruen(a)gmail.com > > wrote: > >> Hi Alex, >> >> The following annotation should do what you want: >> >> %input:csv("header=true,encoding=CP1252") >> >> You were completely right, the existing documentation was a bit >> scrappy. I have slightly rewritten it (see [1] and outgoing links). >> >> Hope thie helps, >> Christian >> >> [1] http://docs.basex.org/wiki/RESTXQ#Input_options >> >> >> >> >> On Sun, Jun 12, 2016 at 4:03 PM, Alexander Shpack <shadowkin(a)gmail.com> >> wrote: >> > Hi, >> > >> > I have nonstandard CSV file with semicolon instead of comma as a >> separator >> > and cp1251 encoded. Documentation is not clear in this case, and the >> sample >> > doesn't work properly >> > >> > declare >> > %rest:path("/store.csv") >> > %rest:POST("{$csv}") >> > %input:csv("header=true") >> > function page:store-csv($csv as document-node()) >> > { >> > "Number of rows: " || count($csv/csv/record) >> > }; >> > >> > Even if I changed the encoding manually the next code does nothing >> > %input:csv("separator=semicolon") >> > and returns text instead of document-node() >> > >> > Could you check it? >> > >> > It would be nice if you add additional encoding parameter to csv module. >> > >> > Thanks! >> > > > > -- > s0rr0w >

3 16

Nonstandard CSV in RESTXQ
by Alexander Shpack 12 Jun '16

12 Jun '16

Hi, I have nonstandard CSV file with semicolon instead of comma as a separator and cp1251 encoded. Documentation is not clear in this case, and the sample doesn't work properly declare %rest:path("/store.csv") %rest:POST("{$csv}") %input:csv("header=true") function page:store-csv($csv as document-node()) { "Number of rows: " || count($csv/csv/record) }; Even if I changed the encoding manually the next code does nothing %input:csv("separator=semicolon") and returns text instead of document-node() Could you check it? It would be nice if you add additional encoding parameter to csv module. Thanks!

2 1

Re: [basex-talk] remotely access
by Henning Phan 11 Jun '16

11 Jun '16

hi,I failed to include the mail list the ports you gave me are different 9884 != 8984, could that be the problem? Date: Sat, 11 Jun 2016 20:46:02 +0000 From: med_khr(a)yahoo.fr To: henningphan(a)hotmail.com Subject: Re: [basex-talk] remotely access yes i have added a "NAT" in my router with 9884 portand the ip of server B is accessible outside from anywhere In the php file in servers Ai have wrote: $session = new Session("The_public_IP", 8984, "admin", "admin"); I'm wondering why it shows me this error. De : Henning Phan <henningphan(a)hotmail.com> À : Mohamed kharrat <med_khr(a)yahoo.fr> Envoyé le : Samedi 11 juin 2016 21h42 Objet : RE: [basex-talk] remotely access Hi,A common thing people miss is to ensure that the server B is remotely accessible. Is server B behind a router then you might have to do port forwarding. Another issue is that the client uses the wrong port.Hope that helpsDate: Sat, 11 Jun 2016 20:29:06 +0000From: med_khr(a)yahoo.frTo: christian.gruen(a)gmail.comCC: basex-talk(a)mailman.uni-konstanz.deSubject: Re: [basex-talk] remotely accessI have made a BaseX php client in a server Aand running BaseX server in server B in the php file containing the xquery query and the connection stringwhen i execute it, the browser shows me: "cannot communicate with server"Any idea? De : Christian Grün <christian.gruen(a)gmail.com> À : Mohamed kharrat <med_khr(a)yahoo.fr> Cc : BaseX Talk <basex-talk(a)mailman.uni-konstanz.de> Envoyé le : Samedi 11 juin 2016 20h29 Objet : Re: [basex-talk] remotely access > to execute remotely xquery queries, we have to run BaseX server and> BaseXclient in the same time?The BaseX server needs to be run on your remote machine, and you’llaccess it via your local BaseX client instance. But I’m pretty sure Igot you wrong? If yes, please try a bit harder ;)

2 1

remotely access
by Mohamed kharrat 11 Jun '16

11 Jun '16

Hi,to execute remotely xquery queries, we have to run BaseX server and BaseXclient in the same time? Best regards

2 2

Self-knowledge in xquery
by France Baril 11 Jun '16

11 Jun '16

When I'm in xquery and I need to reference something through the rest interface. Can I know from xquery my host/port or domain, or do I have to leave that information for myself somewhere in a config file. Case: - I keep an indexable version my .xsd in a DB so I can query it to provide guidance to users in the GUI. - Validation won't work with my .xsd because it won't resolve the includes if I just open the main .xsd file. - I keep a binary copy of my .xsd in a different DB so I can reference it though rest and therefore validate content, but I need to know my base url. So far I've been leaving that information in a config file that I query, but as I have multiple BaseX instances in implies that I enter my host:port in .basex and then again in my indexable config file. I'm trying to reduce duplication here. Thanks, France -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

3 3

Optimal size for large databases
by Lizzi, Vincent 09 Jun '16

09 Jun '16

I am importing a large (~3 million) set of XML documents to BaseX and am running into some problems as the databases grow beyond a few gigabytes. I am using BaseX 8.4.4, and have increased the memory available to BaseX to 12288m (12 Gb) using -Xmx (the machine has 20 Gb total). The documents are being stored in several databases that range in size from 300 Mb up to 25 Gb. My question is: what is an optimal maximum database size (in gigabytes)? I am hoping for some general advice. I appreciate that the answer can vary depending various factors. The problems that I've encountered with the larger databases are: 1. Running OPTIMIZE or OPTIMIZE ALL on the larger databases results in an out of memory error. I have switched to running CREATE INDEX to separately create text, attribute, token, and fulltext indexes, and found that creating these indexes separately produces fewer out of memory errors. I would like to be able to use OPTIMIZE ALL because over time some documents will be removed from the databases and the documentation indicates that optimize all will remove stale information from the indexes. 2. The command scripts that run CREATE INDEX or OPTIMIZE (ALL) seem to tie up the machine for a long time, maybe due to heavy disk access. 3. As the database grows in size the rate at which documents are added slows down. I have been measuring the number of documents imported, and observed rates over 100 documents per minute, and typical rates are around 60 - 30 documents per minute. As the database grows over a few gigabytes the speed slows to around 20 documents per minute. This is not much of a problem because when I see the rate slow down I can start a new database. Unfortunately I have been recording the number of documents, not the database size. In case this information is useful, my project is structured as follows: * There is 1 central index database which records for each document the BaseX database name and path where a document is stored, and some metadata that we use to identify or locate documents. * There are multiple content databases to store the actual documents. These content databases are organized by DTD and time period. * Each insert is done using the BaseX REST API. A BaseX HTTP server instance is running to receive documents, and a basex instance is running from the command line to locate and provide documents. Each insert is done by a POST that includes data and an updating query which adds (using db:replace) to the central index database and a document database in one "transaction". This helps to make the import resilient to duplicate documents, and to any problem that can prevent a single from document being added, and allows the process to continue if interrupted. I will probably need to re-organize the content databases so that each database is only a few gigabytes in size. Does anyone have advice on what would be a good maximum size for each database? Thanks, Vincent Vincent M. Lizzi - Electronic Production Manager Taylor & Francis Group 530 Walnut St., Suite 850, Philadelphia, PA 19106 E-Mail: vincent.lizzi(a)taylorandfrancis.com<mailto:vincent.lizzi@taylorandfrancis.com> Phone: 215-606-4221 Fax: 215-207-0047 Web: http://www.tandfonline.com/ Taylor & Francis is a trading name of Informa UK Limited, registered in England under no. 1072954 "Everything should be made as simple as possible, but not simpler."

2 3

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk