Re: [basex-talk] Multiple db:replace() = slower updates with 7.8 - BaseX-Talk - mailman.uni-konstanz.de

4 Apr 2014


      Hi France,
our latest release includes the fix for the slow-downs you encountered.
Best,
Christian
On Wed, Apr 2, 2014 at 10:29 PM, France Baril
france.baril@architextus.com wrote:
...
Performance is definitely back. This is great. The big drawback is that Java
7 will only work with 64 bits applications on Mac OS and Antenna House that
we use for PDF in xquery through the java API only run on 32 bits on Mac OS
X.
Anyhow, the end-client runs a Windows shop, so we'll manage.
Big thanks!
On Wed, Apr 2, 2014 at 2:56 PM, Christian Grün christian.gruen@gmail.com
wrote:
...
...
I can't think of any issue. I was looking in the documentation. Might be
interesting to add a note somewhere about Java 7 :-).
Thanks!
Yes, this will definitely be documented soon ;)
...
On Wed, Apr 2, 2014 at 2:47 PM, Christian Grün
christian.gruen@gmail.com
wrote:
...
...
Does Basex 8 use Java 7?
Exactly. As Java 6 is not supported by Oracle anymore, and as we
didn't get much votes against this consideration one year ago [1], we
felt that it's finally time to switch. Beside that, Java 7 gives us
much better native OS support for handling files on disk. This is why
the BaseX 7.8.x versions will be the last that will be based on Java
6.
Do you think you'll encounter problems migrating to Java 7?
Christian
[1]
https://mailman.uni-konstanz.de/pipermail/basex-talk/2013-February/004724.ht...
...
On Wed, Apr 2, 2014 at 2:31 PM, Christian Grün
christian.gruen@gmail.com
wrote:
...
> basexhttp returns an error and logs out:
Are you using Java 7?
>
> NoName:~ archie$ /Applications/basex/bin/basexhttp ; exit;
>
> Exception in thread "main" java.lang.UnsupportedClassVersionError:
> org/basex/BaseXHTTP : Unsupported major.minor version 51.0
>
> at java.lang.ClassLoader.defineClass1(Native Method)
>
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
>
> at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
>
> at
>
>
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>
> logout
>
>
> [Process completed]
>
>
>
> On Wed, Apr 2, 2014 at 1:49 PM, France Baril
> france.baril@architextus.com
> wrote:
>>
>> Timing is good, I should be able to try the new snapshot today.
>>
>>
>> On Wed, Apr 2, 2014 at 12:55 PM, Christian Grün
>> christian.gruen@gmail.com wrote:
>>>
>>> Hi France,
>>>
>>> we are also working towards some important commercial deadlines,
>>> which
>>> is why our mailing list feedback takes more time than expected
>>> (open
>>> source life would be so comfortable without financial pressure..
>>> ;).
>>> However, I have now uploaded a new 8.0 snapshot [1], and I would
>>> be
>>> glad to get your feedback if it fixes the problems you
>>> encountered.
>>> You may as well think about exporting and reimporting your
>>> database
>>> (or applying OPTIMIZE ALL on it), as we have also minimized the
>>> storage of redundant namespaces in the latest version.
>>>
>>> Looking forward to your response,
>>> Christian
>>>
>>> [1] http://files.basex.org/releases/latest/
>>>
>>>
>>>
>>>
>>> On Wed, Apr 2, 2014 at 6:25 PM, France Baril
>>> france.baril@architextus.com wrote:
>>> > Ok, thanks for the update.
>>> >
>>> > We are dependent on this to make the next release to our
>>> > production
>>> > server,
>>> > so if you have an idea on timing, it would really help us
>>> > adapt.
>>> >
>>> > France
>>> >
>>> >
>>> > On Tue, Apr 1, 2014 at 11:43 AM, Christian Grün
>>> > christian.gruen@gmail.com
>>> > wrote:
>>> >>
>>> >> Hi France,
>>> >>
>>> >> sorry for lettting you wait. We've already solved part of the
>>> >> problem,
>>> >> but we would first like to shed some more light on another
>>> >> related
>>> >> issue before relasing a new patch version.
>>> >>
>>> >> Christian
>>> >> ____________________________________
>>> >>
>>> >> On Tue, Apr 1, 2014 at 5:26 PM, France Baril
>>> >> france.baril@architextus.com wrote:
>>> >> > Any news on this issue?
>>> >> >
>>> >> >
>>> >> > On Tue, Mar 25, 2014 at 2:30 PM, France Baril
>>> >> > france.baril@architextus.com
>>> >> > wrote:
>>> >> >>
>>> >> >> Yes, we do use namespaces.
>>> >> >>
>>> >> >>
>>> >> >> On Tue, Mar 25, 2014 at 2:29 PM, Christian Grün
>>> >> >> christian.gruen@gmail.com wrote:
>>> >> >>>
>>> >> >>> We may have found a first evidence for the slow down. Do
>>> >> >>> your
>>> >> >>> documents use namespaces?
>>> >> >>>
>>> >> >>>
>>> >> >>> On Tue, Mar 25, 2014 at 2:41 PM, Christian Grün
>>> >> >>> christian.gruen@gmail.com wrote:
>>> >> >>> > Hi France,
>>> >> >>> >
>>> >> >>> > sorry, I know that an answer to your e-mail is still
>>> >> >>> > pending.
>>> >> >>> > If
>>> >> >>> > you
>>> >> >>> > should manage to build a self-contained example, it
>>> >> >>> > could
>>> >> >>> > help
>>> >> >>> > us to
>>> >> >>> > find the culprit.
>>> >> >>> >
>>> >> >>> > Best,
>>> >> >>> > Christian
>>> >> >>> >
>>> >> >>> >
>>> >> >>> > On Wed, Mar 19, 2014 at 6:59 PM, France Baril
>>> >> >>> > france.baril@architextus.com wrote:
>>> >> >>> >> Hi,
>>> >> >>> >>
>>> >> >>> >> We run batch processes on our files in order to send
>>> >> >>> >> them
>>> >> >>> >> out
>>> >> >>> >> to
>>> >> >>> >> translation
>>> >> >>> >> in xliff format (xml format for translations).
>>> >> >>> >>
>>> >> >>> >> I run a piece of code that:
>>> >> >>> >>   1. applies transformations
>>> >> >>> >>   2. file:writes the resulting files (for debug
>>> >> >>> >> purposes)
>>> >> >>> >>   3. and then db:replaces the resulting files.
>>> >> >>> >>
>>> >> >>> >> If I send 5 files into the process, it takes a few
>>> >> >>> >> seconds
>>> >> >>> >> before
>>> >> >>> >> the
>>> >> >>> >> files
>>> >> >>> >> appear on the file system. Yet full process with the
>>> >> >>> >> db:replace
>>> >> >>> >> takes
>>> >> >>> >> 2
>>> >> >>> >> minutes to complete.
>>> >> >>> >>
>>> >> >>> >> When I send 60 files... the process never ends. All
>>> >> >>> >> files
>>> >> >>> >> are
>>> >> >>> >> available on
>>> >> >>> >> the file system and are valid, which means I am fairly
>>> >> >>> >> certain
>>> >> >>> >> that
>>> >> >>> >> the
>>> >> >>> >> freeze doesn't occur because of an error in one of the
>>> >> >>> >> files.
>>> >> >>> >>
>>> >> >>> >> There is no error message in the terminal window. Is
>>> >> >>> >> possible
>>> >> >>> >> that
>>> >> >>> >> something
>>> >> >>> >> was changed after 7.7 that would affect performance?
>>> >> >>> >> On
>>> >> >>> >> our
>>> >> >>> >> production
>>> >> >>> >> server, which is on 7.7 we can run 100+ files in just
>>> >> >>> >> a
>>> >> >>> >> few
>>> >> >>> >> minutes.
>>> >> >>> >> Maybe
>>> >> >>> >> there are new server settings that we missed?
>>> >> >>> >>
>>> >> >>> >> For reference, here is the code. I understand that
>>> >> >>> >> updates
>>> >> >>> >> are
>>> >> >>> >> all
>>> >> >>> >> performed
>>> >> >>> >> at the end of the process, which is why we get the
>>> >> >>> >> debug
>>> >> >>> >> files
>>> >> >>> >> first,
>>> >> >>> >> but
>>> >> >>> >> the lag time between the time where we get all files
>>> >> >>> >> on
>>> >> >>> >> the
>>> >> >>> >> file
>>> >> >>> >> system and
>>> >> >>> >> the db:replace seems huge.
>>> >> >>> >>
>>> >> >>> >> declare updating function
>>> >> >>> >> translation:transform-to-xliff($menu-id
>>> >> >>> >> as
>>> >> >>> >> xs:string, $lang as xs:string*, $package-number as
>>> >> >>> >> xs:string){
>>> >> >>> >>
>>> >> >>> >>     let $menu := menu:xml-get-menu($menu-id, 'en-us')
>>> >> >>> >>
>>> >> >>> >>     (: Need to translate menu and its topics without
>>> >> >>> >> getting
>>> >> >>> >> duplicated
>>> >> >>> >> topics twice:)
>>> >> >>> >>     (: Always send the reusable strings from writers
>>> >> >>> >> and
>>> >> >>> >> from
>>> >> >>> >> dev...
>>> >> >>> >> doesn't
>>> >> >>> >> cost anything if no new string anyway :)
>>> >> >>> >>     let $file-ids := ($menu-id, for $id in
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >> distinct-values($menu/descendant::*[name()='topicref']/data(@id-ref))[1]
>>> >> >>> >> return $id,
>>> >> >>> >>                       'stringreferences',
>>> >> >>> >> 'stringreferencestransform')
>>> >> >>> >>
>>> >> >>> >>     return  (for $file-id in
>>> >> >>> >> distinct-values($file-ids)
>>> >> >>> >>              let $file :=
>>> >> >>> >> db:open('en-us')/*[@id=$file-id]
>>> >> >>> >>              let $skl-href := 'skeletons/' ||
>>> >> >>> >> $package-number
>>> >> >>> >> ||
>>> >> >>> >> '/'
>>> >> >>> >> ||
>>> >> >>> >> substring-after($file/base-uri(), 'en-us/')
>>> >> >>> >>              let $file-with-preserve :=
>>> >> >>> >>                      copy $copy := $file
>>> >> >>> >>                      modify(if
>>> >> >>> >> (exists($copy/@xml:space))
>>> >> >>> >>                             then ()
>>> >> >>> >>                             else insert node attribute
>>> >> >>> >> {'xml:space'}{'preserve'} into $copy)
>>> >> >>> >>                      return $copy
>>> >> >>> >>              let $all-trans-files :=
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >> xliff-transformer:create-translation-files($file-with-preserve,
>>> >> >>> >> $package-number, $menu-id, $lang, $skl-href)
>>> >> >>> >>
>>> >> >>> >>              let $skeleton := $all-trans-files[1]
>>> >> >>> >>
>>> >> >>> >>              let $xliffs := for $xliff in
>>> >> >>> >> $all-trans-files[position()
>>> >> >>> >> != 1]
>>> >> >>> >>                             return $xliff
>>> >> >>> >>
>>> >> >>> >>              let $debug :=
>>> >> >>> >> file:write('Debug-xliff/skls/'
>>> >> >>> >> ||
>>> >> >>> >> tokenize($skl-href, '/')[last()], $skeleton)
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >> return (db:replace('Translation-management',
>>> >> >>> >> $skl-href,
>>> >> >>> >> $skeleton),
>>> >> >>> >>                      for $xliff in $xliffs
>>> >> >>> >>                       let $target-lang :=
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >> $xliff/descendant::*[name()='file']/data(@target-language)
>>> >> >>> >>                       let $lang-pack-name :=
>>> >> >>> >> concat($target-lang,'_',
>>> >> >>> >> $package-number)
>>> >> >>> >>                       let $file-loc-end := $menu-id ||
>>> >> >>> >> '/'
>>> >> >>> >> ||
>>> >> >>> >> $lang-pack-name ||'/' ||
>>> >> >>> >> substring-after($file/base-uri(),
>>> >> >>> >> 'en-us/')
>>> >> >>> >>                       (:let $debug :=
>>> >> >>> >> file:append('debug-ids.xml',
>>> >> >>> >> '&#10;'
>>> >> >>> >> || $file-loc-end):)
>>> >> >>> >>                       let $file-loc := if
>>> >> >>> >> (exists($xliff/descendant::*[@translate='yes']))
>>> >> >>> >>                                        then
>>> >> >>> >> 'xliff-to-translate/'
>>> >> >>> >> ||
>>> >> >>> >> $file-loc-end
>>> >> >>> >>                                        else
>>> >> >>> >> 'xliff-no-new-segment/'
>>> >> >>> >> ||
>>> >> >>> >> $file-loc-end
>>> >> >>> >>                       let $debug :=
>>> >> >>> >> file:write('Debug-xliff/'
>>> >> >>> >> ||
>>> >> >>> >> tokenize($file-loc, '/')[last()], $xliff)
>>> >> >>> >>                       return
>>> >> >>> >> db:replace('Translation-management',
>>> >> >>> >> $file-loc,
>>> >> >>> >> $xliff)
>>> >> >>> >>                      )
>>> >> >>> >>              )
>>> >> >>> >> };
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >> --
>>> >> >>> >> France Baril
>>> >> >>> >> Architecte documentaire / Documentation architect
>>> >> >>> >> france.baril@architextus.com
>>> >> >>> >> (514) 572-0341
>>> >> >>> >>
>>> >> >>> >> _______________________________________________
>>> >> >>> >> BaseX-Talk mailing list
>>> >> >>> >> BaseX-Talk@mailman.uni-konstanz.de
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
>>> >> >>> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> France Baril
>>> >> >> Architecte documentaire / Documentation architect
>>> >> >> france.baril@architextus.com
>>> >> >> (514) 572-0341
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > France Baril
>>> >> > Architecte documentaire / Documentation architect
>>> >> > france.baril@architextus.com
>>> >> > (514) 572-0341
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > France Baril
>>> > Architecte documentaire / Documentation architect
>>> > france.baril@architextus.com
>>> > (514) 572-0341
>>
>>
>>
>>
>> --
>> France Baril
>> Architecte documentaire / Documentation architect
>> france.baril@architextus.com
>> (514) 572-0341
>
>
>
>
> --
> France Baril
> Architecte documentaire / Documentation architect
> france.baril@architextus.com
> (514) 572-0341
--
France Baril
Architecte documentaire / Documentation architect
france.baril@architextus.com
(514) 572-0341
--
France Baril
Architecte documentaire / Documentation architect
france.baril@architextus.com
(514) 572-0341
--
France Baril
Architecte documentaire / Documentation architect
france.baril@architextus.com
(514) 572-0341