Hi,

Our issue is with performance.
Performing all the transformations lead to 2 issues:

We run out of memory.
We sometimes need to query the content that has been transformed after it has gone through some of the transformations.

Web controller is jquery, but we redirect from xquery. jquery just says: translate these files in these languages. The restxq function handles the steps, and calls itself back with an incremented step number when a group of transformations that could be handle without committing content (without a need to query the saved files or without running out of memory) are completed.

On Wed, Feb 11, 2015 at 1:47 AM, Christian Grün <christian.gruen@gmail.com> wrote:

Hi France,

I guess there is no simple answer to your question; it mostly depends
on the architecture of your approach what would be the best solution
and further steps. And I'm not quite sure what's the major challenge?
Is it performance, is it technical restrictions, is it the overall
concept?

As you were mentioning that you are working with a web interface, our
approach would be to provide a RESTXQ function that triggers all the
transformations whenever a user requests it. Have you thought about
that? What language is your web controller built on?

Best,
Christian

On Mon, Feb 9, 2015 at 8:12 PM, France Baril
<france.baril@architextus.com> wrote:
> Hi,
>
> I have an item that I would like to bring to attention. We have developed a
> web controller to let users manages translation processes for BaseX content.
> Our process is something like this:
>
> Users select content to translation (1 to 500 small files) + languages to
> translation to (1 to 32 selections).
> For each language, for each files:
>
> The system transforms the content to xliff and sets the segments to
> translate="yes" if they have changed since the last translation for this
> language. Content is saved because we'll need to query it, we redirect to
> the next task.
> File without new segment to translate, content is processed and saved to the
> target languages (because attributes might have changed and/or segments
> might have been deleted). The xliff-file is deleted, we redirect to the next
> task because we'll need the new information for the next query.
> We query the server to offer the users stats on items to translate.
>
> This is a simplification of the process, but it shows the logics. Redirects
> occur after each step for each language. We have grouped operations to limit
> commits/redirects to a minimum. We apply them:
>
> After each language to avoid running out of memory.
> Before each operation that needs to query files based on the changes from
> the previous steps.
>
> We have also split groups of tasks into smaller groups where too many tasks
> have led us to run out of memory in the past.
>
> Our request would be for a way to force changes to commit without having to
> redirect. Refreshing the browser has a big impact on performance. Or maybe
> you have suggestions to improve batch processing when using a web interface
> for process management.
>
> Thank you in advance for you input!
>
>
>
> --
> France Baril
> Architecte documentaire / Documentation architect
> france.baril@architextus.com

France Baril
Architecte documentaire / Documentation architect
france.baril@architextus.com