We run out of memory.
Who/what is responsible for the OOM)? Could you give us some more information on the exact step in the process that causes the bottleneck?
Web controller is jquery, but we redirect from xquery. jquery just says: translate these files in these languages. The restxq function handles the steps, and calls itself back with an incremented step number when a group of transformations that could be handle without committing content (without a need to query the saved files or without running out of memory) are completed.
On Wed, Feb 11, 2015 at 1:47 AM, Christian Grün christian.gruen@gmail.com wrote:
Hi France,
I guess there is no simple answer to your question; it mostly depends on the architecture of your approach what would be the best solution and further steps. And I'm not quite sure what's the major challenge? Is it performance, is it technical restrictions, is it the overall concept?
As you were mentioning that you are working with a web interface, our approach would be to provide a RESTXQ function that triggers all the transformations whenever a user requests it. Have you thought about that? What language is your web controller built on?
Best, Christian
On Mon, Feb 9, 2015 at 8:12 PM, France Baril france.baril@architextus.com wrote:
Hi,
I have an item that I would like to bring to attention. We have developed a web controller to let users manages translation processes for BaseX content. Our process is something like this:
Users select content to translation (1 to 500 small files) + languages to translation to (1 to 32 selections). For each language, for each files:
The system transforms the content to xliff and sets the segments to translate="yes" if they have changed since the last translation for this language. Content is saved because we'll need to query it, we redirect to the next task. File without new segment to translate, content is processed and saved to the target languages (because attributes might have changed and/or segments might have been deleted). The xliff-file is deleted, we redirect to the next task because we'll need the new information for the next query. We query the server to offer the users stats on items to translate.
This is a simplification of the process, but it shows the logics. Redirects occur after each step for each language. We have grouped operations to limit commits/redirects to a minimum. We apply them:
After each language to avoid running out of memory. Before each operation that needs to query files based on the changes from the previous steps.
We have also split groups of tasks into smaller groups where too many tasks have led us to run out of memory in the past.
Our request would be for a way to force changes to commit without having to redirect. Refreshing the browser has a big impact on performance. Or maybe you have suggestions to improve batch processing when using a web interface for process management.
Thank you in advance for you input!
-- France Baril Architecte documentaire / Documentation architect france.baril@architextus.com
-- France Baril Architecte documentaire / Documentation architect france.baril@architextus.com