BaseX-Talk

basex-talk@mailman.uni-konstanz.de

6 participants
5270 discussions

Re: [basex-talk] Query Across/Many XML Files With BaseX
by Lukas Lewandowski 02 Jan '12

02 Jan '12

Hi Jon, Currently I work on querying distributed XML collections. Therefore I tested also the local BaseX instance (without distribution) for some query performance. My used collection consists of about 1,600,000 XML news articles, each 10 KB size. One of my queries searches for news articles which contain a special keyword (using the full text index). It takes about 100ms for the evaluation. Happy new year. Lukas On 01.01.2012, at 22:53, Jon Morehouse wrote: > Hello Lukas, > Thanks for your response. I am simply looking at storing text in my xml articles. I would be recreating articles with xml and all pictures would be hyperlinked on an external server. I would say each file would be no larger than say 20 kB max. > I am also looking at simply recreating a localized search engine. In terms of scalability, in your opinion, do you think that Basex would be more sufficient for scaling and searching through content. I am hoping my local database grows to hold hundreds of thousands, maybe millions of articles and would like to access any single article by searching with keywords for my users. > > Anyways, I appreciate your response. Have a good new year. > > > > > Jon Morehouse > Moeller High School Class of 2009 > Pepperdine University 2009-2010 > University of Southern California Class of 2013 > > > <Signature Final (2).jpg> > > On Jan 1, 2012, at 8:59 AM, Lukas Lewandowski wrote: > >> Hi Jon Morehouse, >> >> You can use BaseX to create a database (a collection) of all your XML files. You can find some information here [1]. >> Additionally you can create a full-text index to support fast access of keywords within text nodes, see also [2]. >> >> Afterwards you can access/query your database e.g., using the REST API [3] or a client API [4,5]. >> >> Which size does one XML file have? >> >> regards >> Lukas >> >> [1] http://docs.basex.org/wiki/Commands#CREATE_DATABASE >> [2] http://docs.basex.org/wiki/Full-Text >> [3] http://docs.basex.org/wiki/REST >> [4] http://docs.basex.org/wiki/Clients >> [5] https://github.com/BaseXdb/basex-api/tree/master/src/main/php >> >> >> On Jan 1, 2012, at 9:12 AM, Jon Morehouse wrote: >> >>> I am new to BaseX and am excited to be moving forward with this product. Right now I am setting up a website where I want to be able to query across millions of xml files. For instance, if each file contains different keywords, I would like to query across each file to match them with a list of say, my top 50 keywords, to find which files have the most keywords present, the most amount of times. Would something like this be possible with basex? It seems like it would be a simple xquery piece using php (the list of keywords is coming from mysql) but with each xml being its own xml file, would it be possible to search across each and every database/file. >>> >>> >>> >>> >>> >>> Jon Morehouse >>> Moeller High School Class of 2009 >>> Pepperdine University 2009-2010 >>> University of Southern California Class of 2013 >>> >>> >>> <Signature Final (2).jpg> >>> >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk(a)mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >

1 0

Query Across/Many XML Files With BaseX
by Jon Morehouse 01 Jan '12

01 Jan '12

I am new to BaseX and am excited to be moving forward with this product. Right now I am setting up a website where I want to be able to query across millions of xml files. For instance, if each file contains different keywords, I would like to query across each file to match them with a list of say, my top 50 keywords, to find which files have the most keywords present, the most amount of times. Would something like this be possible with basex? It seems like it would be a simple xquery piece using php (the list of keywords is coming from mysql) but with each xml being its own xml file, would it be possible to search across each and every database/file. Jon Morehouse Moeller High School Class of 2009 Pepperdine University 2009-2010 University of Southern California Class of 2013

2 1

Re: [basex-talk] HTTP Request returns 413 FULL. Where is configuration file?
by Lukas Lewandowski 31 Dec '11

31 Dec '11

Good news. Lukas On Dec 31, 2011, at 2:08 AM, France Baril wrote: > I had not try, and though I still haven't, I think that has just answered my question… since it lead me to read this: "Maximum URL length is 2083 characters". Src: http://www.cs.tut.fi/~jkorpela/forms/methods.html#exc. > > > > On Wed, Dec 28, 2011 at 2:32 PM, Lukas Lewandowski <lukas.lewandowski(a)uni-konstanz.de> wrote: > Did you try to send your query via POST? > > > On Dec 28, 2011, at 5:24 PM, France Baril wrote: > >> I use Ruby >> The request is: >> searchquery = searchquery + s_paramlist >> uri = URI.parse("http://localhost:8984/rest/KnowledgeBase/Content?query=" + searchquery ) >> http = Net::HTTP.new(uri.host, uri.port) >> request = Net::HTTP::Get.new(uri.request_uri) >> request.basic_auth("user1", "user1") >> response = http.request(request) >> # decode allows query to return %26 and ruby to understand '&' as not '&' >> result=URI.decode(response.body) >> if (response.code != "200") >> @search = "ERROR: " + response.code + ' ' + response.message >> else >> @search = result.html_safe >> $currentsearch = @search >> end >> The query works when the list of topics under 'limit to' is short, not when it's long: >> xquery version "1.0"; >> declare variable $kwoperator as xs:string := "all"; >> (:For each topic type, create box:) >> <div style="font-size: 10pt;"> >> <form action="start" method="get"> >> <div> >> <input type="text" name="keywords" value="Search for..." size="42"/> >> {if ($kwoperator ='any') then >> <span><br/> >> %26%23160%3B%0D%0A<input type="radio" name="kwoperator" value="all" /> Contains <b>all</b> these words <br/> >> %26%23160%3B%0D%0A<input type="radio" name="kwoperator" value="any" checked="yes"/> Contains <b>any</b> of these words <br/> >> </span> >> else >> <span><br/> >> %26%23160%3B%0D%0A<input type="radio" name="kwoperator" value="all" checked="yes" /> Contains <b>all</b> these words <br/> >> %26%23160%3B%0D%0A<input type="radio" name="kwoperator" value="any" /> Contains <b>any</b> of these words <br/> >> </span> >> } >> <input class="submit" type="submit" value="Search" /> >> <br/> >> </div> >> <div> >> <p><i>Limit topics to:</i></p> >> { for $x in distinct-values(//*[contains(@class, ' topic/topic ')]/name()) >> order by $x >> return <span>{"%26%23160%3B%0D%0A"}<input type="radio" name="topic" value="{$x}" />{concat(' ', $x)}<br/></span> >> } >> <input class="submit" type="submit" value="Refine" /> >> <br/> >> </div> >> </form> >> </div> >> >> Thanks, >> >> France >> >> On Wed, Dec 28, 2011 at 6:22 AM, Lukas Lewandowski <lukas.lewandowski(a)uni-konstanz.de> wrote: >> Hi France, >> >> I have several questions: >> >> What HTTP client do you use? >> Which HTTP method do you use? How does the request look like? >> How big is your data you want to store/request? >> >> regards >> Lukas >> >> On Dec 28, 2011, at 12:41 AM, France Baril wrote: >> >> > Hi, >> > >> > After about 2620 characters on a response for an HTTP:Request, I get a 413 FULL error. >> > >> > I looked at the .basex configuration file and at the httpserver startup options, but couldn't find a way to change configuration for response size. >> > >> > Should I be looking for the Jetty config? I searched for jetty.xml, but only found it under olderapp/jetty.xml. Nothing that resemble 'baseX/jetty.xml'. >> > >> > Do you know how I can resolve this issue? >> > >> > I'm on Mac OsX - Snow Leopard. >> > >> > Thanks, >> > >> > France >> > >> > _______________________________________________ >> > BaseX-Talk mailing list >> > BaseX-Talk(a)mailman.uni-konstanz.de >> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> >> >> >> -- >> France Baril >> Architecte documentaire / Documentation architect >> france.baril(a)architextus.com >> (514) 572-0341 > > > > > -- > France Baril > Architecte documentaire / Documentation architect > france.baril(a)architextus.com > (514) 572-0341

1 0

HTTP Request returns 413 FULL. Where is configuration file?
by France Baril 28 Dec '11

28 Dec '11

Hi, After about 2620 characters on a response for an HTTP:Request, I get a 413 FULL error. I looked at the .basex configuration file and at the httpserver startup options, but couldn't find a way to change configuration for response size. Should I be looking for the Jetty config? I searched for jetty.xml, but only found it under olderapp/jetty.xml. Nothing that resemble 'baseX/jetty.xml'. Do you know how I can resolve this issue? I'm on Mac OsX - Snow Leopard. Thanks, France

2 1

one text item per line output
by jidanni＠jidanni.org 26 Dec '11

26 Dec '11

Do I really need to do all this just to get one item per line $ basex -s format=no -q 'doc("...")//.../*:name/concat(text(),"
")' AM 594 復興1 AM 612 中央 AM 630 台灣 AM 657 正聲 I use -s format=no to turn off spaces. -s format=text not much help. There really ought to be a -s option to say I want newlines instead of spaces. In perl all we need is perl -l. OK, maybe Windows users also need CRLFs (\r\n).

2 4

jdbc driver
by Francesco Beretta 26 Dec '11

26 Dec '11

Hi, I'm trying to initialize a connection to a postgresql database : Query: sql:init("org.postgresql.Driver") Error: [FOSQ0007] Could not initialize specified driver: 'org.postgresql.Driver' What is the right syntax to initialize the driver? Could you provide me with an exemple ? Where should I put the jdbc driver jar ? Many thanks Francesco Beretta

3 2

Единая Дисконтная Система
by DISCOUNTKA 22 Dec '11

22 Dec '11

ООО Кандидат Здравствуйте, Вы подписаны на обновления сайта www.DISCOUNTKA.ru Наш информационно-торговый портал «DISCOUNTKA», поможет Вам своевременно узнавать об акциях, скидках и распродажах нашего города. Посещайте сайт www.DISCOUNTKA.ru – будьте в курсе всех новых предложений от наших Партнеров! Оставайтесь с Компанией «DISCOUNTKA» и получайте дополнительные бонусы, переходя на Страницу нашего Партнера ( http://discountka.ru/cat?orgTask=orgDetails&catid=739&orgId=2074 ). Выгодное предложение для Компаний нашего города ( http://discountka.ru/kabinet/71 ) P.S. не спешите отправлять это письмо в СПАМ, создайте отдельную папку «DISCOUNTKA» и настройте наши входящие письма в неё. Со временем Вы убедитесь, предложения от наших Партнеров Вам пригодятся, помогут сделать правильный экономический выбор! С Уважением и уверенностью на долгосрочное сотрудничество, Компания Discountka. здесь ( http://discountka.ru/component/acymailing/lists ) можно написать сообщение и отписаться от рассылок

1 0

Limit of namespaces per database
by Stefaniuk Marcin 22 Dec '11

22 Dec '11

I have fairly big set of webservice (WSDL+XSD). I'm considering switch from eXist to BaseX but when I'm trying to create database with initial import of the set but it stops after few elements. Limit of 256 namespaces is real shortcoming. Regards --- Marcin Stefaniuk PKO Bank Polski Departament Rozwoju Aplikacji Sprzedażowych i Wsparcia Wydział Rozwoju Platform Usług i Procesów Biznesowych Warszawa, ul. Puławska 15 tel. +48 22 580 30 39

2 3

Precompiling Queries
by Dave Glick 21 Dec '11

21 Dec '11

Hello again, I notice in the current sources for QueryProcessor.java that executing a query appears to be a three-stage process: parse, compile, and execute. Is it now possible to preparse and/or precompile a query before using it? If so, this raises a couple important questions: - Does executing the query "destroy" it (will subsequent executions return the same results)? - What can be done to the query after parsing or compiling? Can I rerun a query after changing the context? What about after adjusting the variable bindings? I notice that doing either of these in QueryProcessor does not reset the parse or compile flag... The last time I looked at this, neither of the operations above were possible making preparsing and precompiling kind of pointless for embedded use. If this works now, it's very exciting and could save a lot of cycles in cases where a complex query needs to be repeatedly executed against different contexts or with different variables. Thanks, Dave

2 4

Possible Optimization Bug (Or Maybe Document Insertion Bug)
by Dave Glick 20 Dec '11

20 Dec '11

Hello, I seem to have stumbled on a bug (or at least I think I have). Do the following: - Create a simple database with a single document - Insert a text node before the start of the document root ("insert node "test" before /root") - Attempt to optimize the database You will get a NullPointerException at org.basex.index.Names.index(Names.java:84). It appears as though the index is attempting to set statistics information for key id 0 (because the parent is a document), but the stats[] array only gets initialized with objects from index 1 and up. The other possibility is that text nodes should not be allowed before or after the document root element. I'm a little hazy on what the standards say in this regard. My understanding is that the document node can have prolog content before the root and other content after. I've also seen many documents which parse correctly that have text content before the opening tag (though again, this may not be strictly standard-compliant behavior). If this is the case, then the bug is probably that the insertion update operation is allowed to succeed. Dave

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk