Different issues than, my global lock has been set to false all along and I have the problems even on read-only operations.

I'm saving my PDF on the file system, redirecting and then loading it in the PdfReview DB in an update operation just to make sure that I don't have a write lock on the DB when the images are grabbed.

Thanks for sharing!

On Wed, Sep 25, 2013 at 2:14 PM, Joe Templeman joe@inkling.com wrote:

I found that it is locking up because of the database global lockhttp://docs.basex.org/wiki/Options#GLOBALLOCK which locks during any updating function. On the long updating call that was happening first, it was grabbing the lock and so every subsequent request was being blocked until the update had finished. I think the solution for us is to turn it off since we're never updating any content in place, only adding content, and that content is static so it doesn't matter if two requests clobber the data.

On Wed, Sep 25, 2013 at 9:30 AM, France Baril < france.baril@architextus.com> wrote:

...

I'm having similar issues where everything freezes on an http request done from inside a restxq function, but only if certain conditions are present.

The conditions are not easy to pinpoint. However, they affect our ability to validate content against modular dtds/xsds, to apply modular .xsl and .xsl that were grabbing pieces of content with doc(restxq...) accesses, and to create PDF with a FOP module because the fo references images stored as raw files and only accessible by http://..rest/ paths.

I'm in the 3rd week of refactoring our application so we can migrate to 7.7. I believe I am half way there. The main task is to augment content through query before the document is sent to .xsl to remove all needs for the use of the doc(restxq) functions. We also find that we can no longer have a self-contained application and that we have to create new steps to export pieces of content and applicative files to the file system.

Not having a self-contained application is a huge step back for us where we have to start considering that different OS handle different paths differently. For example, Windows is killing our capitalization on content re-import which multiplies the folder structures (looks like we'll need to naming conventions).

I'm hoping someone can provide a case simple enough to be submitted and that will help pinpoint the exact source of the problem and resolve the issue. Our refactoring for doc(restxq) will be done by there, but not having to export applicative files and images for processing would bring us back to a neat/clean self-contained application.

Regards,

France

On Fri, Sep 20, 2013 at 6:13 PM, Joe Templeman joe@inkling.com wrote:

...

Hi all,

I'm having trouble with the running a restxq API, set up as either a stand alone app (using basexhttp) or running the servlet in jetty.

When I make a request, the server will lazily load in the requested content using a small java library that I've built which can take quite a while, and unfortunately it completely blocks waiting for this request to complete. This means that if I make a request for content which is already loaded, which should return in a couple of milliseconds, it instead waits for the downloading of the first request to complete before returning.

I'm sure it's something simple that I'm missing in the config somewhere, but I have the threadpool parameters set up in my jetty.xml config file but it doesn't seem to make any difference.

Is this expected behavior? I can't imagine any HTTP server would ever be designed to be single threaded...

Any help is welcome, I've been trying to figure this out for days now.

Thanks, Joe

---

BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

-- France Baril Architecte documentaire / Documentation architect france.baril@architextus.com (514) 572-0341

Hi, I have divided this in sections, hoping it helps.

*Temp-dir* I have noticed, but did not tried the new temporary directory function, because I can't control when it's deleted. My impression is that it will be deleted when I get out of the function that created it, but since I redirect to an updating function that loads the resulting PDF into a DB, I am afraid that the result will be deleted before I can upload it. So I manage the directories myself, creating and deleting it when I know all processes are done with them. Let me know if I assumed wrong about deletion of the temp-dir.

*Case #1* I have attached an .fo that contains http://... links to images. When I use this with the fop module BaseX freezes (*let* *$pdf* := *fop:transform*(*$** fo*)).

It works when I change all the <fo:external-graphic src="*url(' http://localhost:8984/rest/ImagesAndGlobalContent/global/apps/smartactions/i... *')" inline-progression-dimension.maximum="100%" content-height= "scale-down-to-fit" content-width="scale-down-to-fit"/> to access images from the file-system.

The restxq from which it is called in non-updating. The update is performed after a redirect, where the pdf saved on the file-system is uploaded.

This worked on 7.6.

*Case #2* Other examples of what we were doing was process a document through .xsl. When the .xsl encountered a reference to a GUI item we called a restxq that would grab the string for the GUI item based on multiple parameters: phone model, os version, carrier, locale...

We do the same for images:* doc( http://localhost:8984/get-image-for?id=image-1&model=3083&carrier=sm.... ..*). This way we can create a link to the right image in the correct dpi for the user's model (get-image accesses our config file saved in a DB and returns the link to the image that matches the parameters). Now I resolve links to images in the xquery before applying the .xsl. In that case it's not such big issue because as long as I am processing indexed content I can pre-process it in xquery. It's more of an annoyance where I have to rewrite most of my code to match the new paradigm (I WAS RELYING HEAVILY on REST and RESTXQ). The rewriting is almost completed and I have not hit anything that cannot be done in a different way for indexable content. As a bonus, I seem to have gain in performance. The real issue we still have is when accessing raw files, see case #1, #3 and #4.

*Case #3* validate:xsd($xml-file, $xsd) where $xsd is at * http://localhost:8984/rest/AppResources/xsdname.xsd* gives me a headache too. I think it can't load the included .mod. So I have created a function called bypassvalidateissue ( ;-) ) that saves the dtds or xsds on file before applying them, and then deletes them.* I need the .xsd in the database to be able to run an query process that creates documentation of the structures and elements when the XSDs get updated*, so I want to keep them in the DB that holds our applicative resources. Our .xsd are saved as indexable, but we have the same issue with .dtds that are saved as raw content. A case that failed for me was trying a validate:dtd($xhtml-file, $html-transitional.dtd) when the dtd is in my AppResources DB. I think it's because the DTD is modular.

The validations using rest access worked for all cases in 7.6. Now I have to bypass all accesses by saving stuff on the file-system, processing, and deleting.

*Case #4* I have 3 .xsl saved in my Application Resouces DB. process-menu.xsl, process-topic.xsl. where both include common-inline-elements.xsl. xslt:transform($xml-file, http://.../rest-process.xsl) which used to work now fails me, and I can't just read the .xsl as an xml node, because it doesn't know to resolve the includes. My includes used to work when referenced as <include http://localhost:8984/rest/common-inline-elements.xsl/%3E. In 7.4, I think they also worked when a relative path was used... but I can't remember for sure.

*Summary* In summary, I don't feel as RESTED as I used to.

We use BaseX to create content from a single source of topics that get customized for different phone models that have different interfaces and dpi that run on different os, on different carriers that have their own custom needs, for different country that have different network set ups, etc. We also have adapted interfaces and navigation patterns for mobile/browser/pdf media targets. We make extensive use of queries ability to grab, aggregate and customize all pieces of content, all of that in 30 something languages.

We can't replicate some of these concurrent accesses in simple cases, but we are very careful with updating function, redirecting whenever necessary.

Having everything centralized in DBs is useful for faster deployment on different systems that use different OS (duplication of collection with high caps on Windows - we still have not changed our naming conventions), including a .WAR deployment that received users feedback and that doesn't allow for writing on the file system. Short term, we don't need the functions that write to the file system on the .WAR deployment, but if even 1 department asks for a PDF report, we could suffer from lack of read access to images (case #1).

Let me know if some of these cases are not clear, or if you need extra code samples.

On Mon, Sep 30, 2013 at 5:56 AM, Christian Grün christian.gruen@gmail.comwrote:

Hi France,

sorry for the inconvenience. Regarding REST(XQ) calls to the same BaseX instance, it’s indeed sth. that should be avoided, as it can cause all kinds of deadlocks, so it’s generally wise to address resources outside the database when using XSLT from an updating XQuery script. As you wrote, you’re encountering problems even with read-only operations.. Could you give me some more details on your query?

Talking about self-containedness, there should be ways to perform all steps from within an XQuery script. Have you thought about writing the relevant files to a temporary directory before running your XSLT script (using e.g. file:temp-dir)? However, I agree that this step may be troublesome if it’s not immediately clear which resources will be addressed by the XSLT script.

Christian ___________________________

2013/9/26 France Baril france.baril@architextus.com:

...

Different issues than, my global lock has been set to false all along

and I

...

have the problems even on read-only operations.

I'm saving my PDF on the file system, redirecting and then loading it in

the

...

PdfReview DB in an update operation just to make sure that I don't have a write lock on the DB when the images are grabbed.

Thanks for sharing!

On Wed, Sep 25, 2013 at 2:14 PM, Joe Templeman joe@inkling.com wrote:

...

I found that it is locking up because of the database global lock which locks during any updating function. On the long updating call that was happening first, it was grabbing the lock and so every subsequent

request

...

...

was being blocked until the update had finished. I think the solution

for us

...

...

is to turn it off since we're never updating any content in place, only adding content, and that content is static so it doesn't matter if two requests clobber the data.

On Wed, Sep 25, 2013 at 9:30 AM, France Baril france.baril@architextus.com wrote:

...

I'm having similar issues where everything freezes on an http request done from inside a restxq function, but only if certain conditions are present.

The conditions are not easy to pinpoint. However, they affect our

ability

...

...

...

to validate content against modular dtds/xsds, to apply modular .xsl

and

...

...

...

.xsl that were grabbing pieces of content with doc(restxq...)

accesses, and

...

...

...

to create PDF with a FOP module because the fo references images

stored as

...

...

...

raw files and only accessible by http://..rest/ paths.

I'm in the 3rd week of refactoring our application so we can migrate to 7.7. I believe I am half way there. The main task is to augment content through query before the document is sent to .xsl to remove all needs

for

...

...

...

the use of the doc(restxq) functions. We also find that we can no

longer

...

...

...

have a self-contained application and that we have to create new steps

to

...

...

...

export pieces of content and applicative files to the file system.

Not having a self-contained application is a huge step back for us

where

...

...

...

we have to start considering that different OS handle different paths differently. For example, Windows is killing our capitalization on

content

...

...

...

re-import which multiplies the folder structures (looks like we'll

need to

...

...

...

naming conventions).

I'm hoping someone can provide a case simple enough to be submitted and that will help pinpoint the exact source of the problem and resolve the issue. Our refactoring for doc(restxq) will be done by there, but not

having

...

...

...

to export applicative files and images for processing would bring us

back to

...

...

...

a neat/clean self-contained application.

Regards,

France

On Fri, Sep 20, 2013 at 6:13 PM, Joe Templeman joe@inkling.com

wrote:

...

...

...

...

Hi all,

I'm having trouble with the running a restxq API, set up as either a stand alone app (using basexhttp) or running the servlet in jetty.

When I make a request, the server will lazily load in the requested content using a small java library that I've built which can take

quite a

...

...

...

...

while, and unfortunately it completely blocks waiting for this

request to

...

...

...

...

complete. This means that if I make a request for content which is

already

...

...

...

...

loaded, which should return in a couple of milliseconds, it instead

waits

...

...

...

...

for the downloading of the first request to complete before returning.

I'm sure it's something simple that I'm missing in the config

somewhere,

...

...

...

...

but I have the threadpool parameters set up in my jetty.xml config

file but

...

...

...

...

it doesn't seem to make any difference.

Is this expected behavior? I can't imagine any HTTP server would ever

be

...

...

...

...

designed to be single threaded...

Any help is welcome, I've been trying to figure this out for days now.

Thanks, Joe

---

BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

-- France Baril Architecte documentaire / Documentation architect france.baril@architextus.com (514) 572-0341

-- France Baril Architecte documentaire / Documentation architect france.baril@architextus.com (514) 572-0341

---

BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk