- BaseX-Talk - mailman.uni-konstanz.de

fn:apply with an updating function
by Hans-Juergen Rennau 28 Jul '22

28 Jul '22

Dear BaseX people, I encounter a problem with standard function fn:apply(), applied to an updating function. declare %updating function f:apply-dbPut($db as xs:string, $input as item(), $path as xs:string) { let $fn := function-lookup(QName('http://basex.org/modules/db', 'put'), 3) let $args := [$db, $input, $path] return apply($fn, $args)}; Message:[XUST0002] Function body must be updating. Therer seems to be a checking looking for an updating function, and apply() cannot, of course, be classified as such. Do I make a mistake, or is it a limitation, or a bug? Thank you for checking, kind regards -Hans-Jürgen

2 2

BaseX YAML Parser?
by Eliot Kimber 26 Jul '22

26 Jul '22

I found myself needing to operate on some YAML documents and tried to find an XQuery YAML parser and did not find one, at least not one of sufficient completeness. It seems like kind of an obvious missing thing in the XQuery world so just curious if I missed something? I don’t have time myself to write one—it can’t be that hard but it’s not that critical of a need for me. I ended up using the Python PyYAML library, which let me quickly convert YAML to JSON, which then I could of course process just fine with XQuery. Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

6 6

Fwd: Available Integration with Apache POI for Excel Generation?
by Tim Thompson 26 Jul '22

26 Jul '22

Resending to the list (forgot to reply all). ---------- Forwarded message --------- From: Tim Thompson <timathom(a)gmail.com> Date: Mon, Jul 25, 2022 at 5:04 PM Subject: Re: [basex-talk] Available Integration with Apache POI for Excel Generation? To: Eliot Kimber <eliot.kimber(a)servicenow.com> Hi, Eliot, I'm not aware of any integration with Apache POI (which would be great), but here's a project--it seems unfinished, but may be enough to get started with? https://github.com/eliudmeza/OOXML-Library-XQuery-BaseXdb All best, Tim -- Tim A. Thompson (he, him) Librarian for Applied Metadata Research Yale University Library On Mon, Jul 25, 2022 at 2:29 PM Eliot Kimber <eliot.kimber(a)servicenow.com> wrote: > I tried searching for “BaseX Excel generation” but it didn’t get me > farther than a reference to the CSV module, which of course I am familiar > with. > > > > I need to add the ability to generate non-trivial Excel spreadsheets, > which I know how to do using the Apache POI library, but being lazy I was > hoping somebody had already done it in a shareable form. > > > > So my question: Does anyone have or know of a shareable integration of > Apache POI with BaseX for generating Excel spreadsheets or some other > Excel-generation solution that is not just downloading CSV files? > > > > Thanks, > > > > E. > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter > <https://twitter.com/servicenow> | YouTube > <https://www.youtube.com/user/servicenowinc> | Facebook > <https://www.facebook.com/servicenow> >

2 1

Available Integration with Apache POI for Excel Generation?
by Eliot Kimber 25 Jul '22

25 Jul '22

I tried searching for “BaseX Excel generation” but it didn’t get me farther than a reference to the CSV module, which of course I am familiar with. I need to add the ability to generate non-trivial Excel spreadsheets, which I know how to do using the Apache POI library, but being lazy I was hoping somebody had already done it in a shareable form. So my question: Does anyone have or know of a shareable integration of Apache POI with BaseX for generating Excel spreadsheets or some other Excel-generation solution that is not just downloading CSV files? Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 1

What Would "Server is running or permission was denied." Indicate?
by Eliot Kimber 18 Jul '22

18 Jul '22

One my RESTXQ pages constructs an HTML report that takes a few seconds to construct. The page works fine in my local dev environment but when I deploy the code on my production server the process fails after constructing the report markup with the log messages: Server is running or permission was denied. Server is running or permission was denied. Server is running or permission was denied. That is, the message is emitted three times in a row with about 50ms between and the page generation fails (page never appears in the browser). Any idea what condition this might indicate and how I might track it down? It must be something I’m doing in my code but I can’t find any obvious logic bugs so wondering what kind of thing I might look for. Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 1

First steps with client/server on Linux - bulk import; error message in dba
by Florian Schmitt 18 Jul '22

18 Jul '22

Hi all, i'm currently evaluating BaseX running on a headless Linux Server. It took me quite a while to get it running (the docs seem to be focused on Windows environments)... Is there a way to contribute to the documentation? First question: After importing some of documents nicely organized in collections, i've found that the REST client will display all documents on the same "level" instead of showing just the first level of collections. This isn't nice if the db contains a great number of documents.... When accessing the same db using WebDAV (Oxygen), one can nicelly navigate using the collection structure. With REST, adding the collection names to the URL will restrict the output to the db content of those collections. But without such a restriction, it would be great if the REST web interface would just return the first level of collections without any XML resource entries. Second question: What's the "best practice" for bulk imports while retaining the directory structure of the imported files using collections? I've found that Oxygen "translates" subdirectories into collections when using the "import directories" feature. Is there somehing similar using the command line client? And another issue: The "Logs" page of the DBA webapp shows the following error warning: "Unexpected error: Improper use? Potential bug? Your feedback is welcome: Contact..." Is this a critical error? How to debug this? So far, i'found the wrapper-basex log, but it doesn't contain anything useful. Since i'm using the Yajsw Wrapper, there's no debug output or debug option (?). Yours sincerely. Florian

4 8

Adding Attributes After the Fact: How To?
by Eliot Kimber 16 Jul '22

16 Jul '22

In my Mirabel project I’m loading large quantities of DITA content (40K topics per content database). I’m turning off DTD parsing because otherwise it takes hours to load the content (I haven’t had bandwidth to figure out how to use Xerces grammar caching with BaseX, which would solve the problem). DITA uses an attribute-based mechanism (@class) to associate elements with their base types. For example, a <concept> element is a kind of <topic>, as indicated by the @class value “- topic/topic concept/concept “. To find DITA elements you always search on @class values rather than tag names. In XSLT 3 this is best expressed as match=”*[contains-token(@class, ‘topic/topic’)]” to handle any topic element. These attributes are normally defaulted in grammars so they are not explicit in the source as imported and thus do not exist in my no-DTD content databases. To work around this I’ve implemented a function that hard codes the element-to-class-value mapping to enable selecting elements by @class value. (For any set of content you can know what the @class value is for any element type as its statically declared in the governing grammar, so it is possible to have this kind of hard-coded element-to-class mapping.) However, I now realize that if the attributes are in the data as stored in BaseX that lookup will be much faster using an attribute index. That is, rather than using a predicate like [dutils:class(., ‘topic/topic’)] I could do db:token($database, ‘topic/topic’, ‘class’). Because I have the element-to-class mapping I can add the attributes to the content after the initial database load. My question: How best to do that? I can have an updating function that simply adds the @class attribute to all the elements but I suspect there are some performance implications for doing that kind of bulk update to millions of nodes. So any guidance on this kind of bulk update would be helpful. Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

1 0

RESTXQ: How to Reflect Progress of Long-Running Querries
by Eliot Kimber 16 Jul '22

16 Jul '22

In my Mirabel system, I have multiple BaseX instances to handle long-running processes while a primary instance serves the web pages. For these background processes I’d like to be able to show progress in the main web app. My question: what are the options for doing this? I can imagine having a database to write progress data to and then read from that in my RESTXQ page (maybe using a simple JavaScript to poll the app). Are there better techniques? I’m trying to limit the amount of JavaScript I have to write but I can do it if I have to. Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 1

Possible to Speed Up This Lookup?
by Eliot Kimber 16 Jul '22

16 Jul '22

I generate a “where used” index as an XML doc in a separate database. I’m then doing lookups against this index like this where $node is the element I want to find the index entry for: collection($rkDatabase)/doc-where-used-index/where-used-entry[string(@key) eq string(db:node-id($node))] Where the markup is shown in [1] below. Using prof:track() I’m measuring a consistent 0.036 seconds per lookup, which seems very slow. Db:node-id() is instantaneous so the cost must be in the XPath itself. I’ve tried different variations on the predicate but nothing makes a measurable difference. There must be a more efficient way to do this kind of lookup or a better way to structure my index XML so that the lookup is optimized. Or is this something that could be affected by my index settings at the database level? Thanks, Eliot [1] Where-used index markup: <doc-where-used-index> <where-used-entry key="9151416" tagname="map" class="map/map" id=""> <title> <ph keyref="workplace-safety-management"/> </title> <conrefs/> <topicrefs> <noderef node-id="672471" database="tokyo" tagname="mapref" baseuri="/tokyo/qa-sushma-sundareswaran.ditamap" href="workplace-service-delivery-core.ditamap"/> <noderef node-id="898684" database="tokyo" tagname="mapref" baseuri="/tokyo/qa-ankit-jaiswal.ditamap" href="workplace-service-delivery-core.ditamap"/> <noderef node-id="9524996" database="tokyo" tagname="mapref" baseuri="/tokyo/workplace-service-delivery-suite.ditamap" href="workplace-service-delivery-core.ditamap"/> </topicrefs> <doc> <noderef node-id="9151416" database="tokyo" tagname="map" baseuri="/tokyo/workplace-service-delivery-core.ditamap"/> </doc> <xrefs/> <referencing-maps> <noderef node-id="672452" database="tokyo" tagname="map" baseuri="/tokyo/qa-sushma-sundareswaran.ditamap"/> <noderef node-id="898667" database="tokyo" tagname="map" baseuri="/tokyo/qa-ankit-jaiswal.ditamap"/> <noderef node-id="9524980" database="tokyo" tagname="map" baseuri="/tokyo/workplace-service-delivery-suite.ditamap"/> </referencing-maps> </where-used-entry> … </doc-where-used-index> _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

5 11

file:resolve-path, ".." steps
by Hans-Juergen Rennau 11 Jul '22

11 Jul '22

Dear BaseX people, extension function file:resolve-path() shows a behaviour which appears to me inconsistent and probably not intended: - if called with a single parameter, a path containing ".." is resolved as expected- if called with two parameters, the ".." steps are not resolved. I would expect the resolution of ".." steps to be always a part of path resolution. Example: cd C:\a\b\c file:resolve-path('../x') =>C:\a\b\x file:resolve-path('../x', 'file:///C:/a/b/c/') =>C:\a\b\c\..\x file:resolve-path('../x ', 'file:///C:/a/b/c/' ! file:resolve-path(.) =>C:\a\b\x Greetings, Hans-Jürgen

2 2

Explicit cacheing with store: vs. implicit cacheing in RESTXQ ?
by Majewski, Steven Dennis (sdm7g) 11 Jul '22

11 Jul '22

Thinking about use cases for the new store: cacheing module, and wondering if I understand the cacheing already provided in RESTXQ. https://docs.basex.org/wiki/RESTXQ#Preliminaries <https://docs.basex.org/wiki/RESTXQ#Preliminaries> So in a production environment where I have disabled monitoring for file changes by setting PARSERESTXQ to -1, variables declared in prolog of library modules have a long lifetime and are only recalculated when the modules are reloaded and parsed — which would, I assume, imply that if modifications are made to the database, I should call rest:init() to force a reload if I have declared values that are calculated from a collection that may have been changed. And that would also imply that I could, in many cases, use that implicit caching in place of explicit store: functions. Is that correct ? — Steve M.

2 1

Email integration
by Ben Pracht 09 Jul '22

09 Jul '22

Hi, This may seem a bit unusual, but I like the parsing capabilities of XQuery and I'd like to use it for my email. I'd like to have BaseX log into my IMAP server, fetch emails and put each email as a separate XML document. I'm willing to have it ignore attachments if necessary. Then, I'd like to query my email using normal BaseX XQuery. This isn't for normal emails, but I wanted to be able to query it for automated emails and for one-off type queries. Oh, to make things more difficult, I'm using GMail, so the login just got even more difficult. Any suggestions? Thanks, Ben Best Regards, Ben Pracht 919.809.2439 ben.pracht(a)gmail.com

3 3

Same prefix, same functions, different namespace?
by Graydon Saunders 07 Jul '22

07 Jul '22

Hello -- As part of a long and involved conversion process, the content starts in a lax version of the target vocabulary and proceeds to a strict version of the target vocabulary. By local convention, the two vocabularies use the same namespace prefix but are not the same namespace and use different namespace URIs. Ideally, I would be using the same test functions in a lot of cases, only changing the association for the prefix depending on the data. So far as I know, there isn't any way to declare a namespace dynamically; I can't create a "test this content" module and tell it which namespace to use in some parameterized way, so I'm going to have to create two versions of the module despite the only difference being a namespace declaration. Is that right, or is there a way to do this? Thanks! Graydon

3 2

Java 8 and XML Catalog
by Lizzi, Vincent 07 Jul '22

07 Jul '22

Hi Christian, I'm having trouble getting an XML Catalog to work in BaseX 9.7.2 running on Java 8. The XML Catalog is working properly in BaseX and Saxon using Java 17. However, if I keep everything the same and switch to Java 8 the XML Catalog doesn't work. The error messages when running on Java 8 indicate that the XML Catalog is not being used at all. The obvious solution of updating the environment to a more recent version of Java is not so easy in this case. I have the XML Catalog location configured as an absolute file system URI in the system properties org.basex.catfile, xml.catalog.files, and javax.xml.catalog.files. Option db:intparse is set to false and option db:dtd is set to true. I have set the class path to include both the Apache XML Resolver (xml-resolver:xml-resolver:1.2) and the xmlresolver.org XML Resolver (org.xmlresolver:xmlresolver:4.4.0). It is my understanding that BaseX will use Norm Tovey-Walsh's XML Resolver from xmlresolver.org if it is found on the class path, or fall back to Java's built in XML resolver. I see the logic for this check on which catalog resolver to use in org.basex.util.Resolver. Do you have any suggestions on what to try, or is this pointing to a possible bug somewhere? Thank you, Vincent ______________________________________________ Vincent M. Lizzi Head of Information Standards | Taylor & Francis Group 530 Walnut St., Suite 850, Philadelphia, PA 19106 E-Mail: vincent.lizzi(a)taylorandfrancis.com Web: www.tandfonline.com Taylor & Francis is a trading name of Informa UK Limited, registered in England under no. 1072954 "Everything should be made as simple as possible, but not simpler." Information Classification: General

4 5

RESTXQ: Serve images that are not within the web app dir
by Eliot Kimber 07 Jul '22

07 Jul '22

In my RESTXQ application, I have the XML content in a database but the images referenced from that content is only in a local clone of the git repository from which the XML content is loaded into BaseX. I would like to serve the images from RESTXQ-generated HTML Pages but I’m not sure how to do that other than to use URLs that include the base64 image data directly. What I think I need to do is implement a RESTXQ end point that handles image URLs that I know how to convert into image files on the file system but I don’t see how to then return the raw bytes as the return result from the RESTXQ handler function. I’m trying to avoid copying the images because there’s no other reason to have them in a database or otherwise copy them somewhere else on the file system. I suppose I could symlink the git directories under the webapp directory but that seems kind of icky too. Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

4 5

Re: [basex-talk] Sym links in BaseX 9.7.2
by Maud Ingarao (IHRIM - ENS de Lyon) 07 Jul '22

07 Jul '22

Hi Christian :-) The prior version we were using was 9.3.2 and everything worked fine. Now with 9.7.2 I get this error : Stopped at /path/to/basex/webapp/synopsx/workspace/bvm/models/mixed.xqm, 38/55: [XQST0059] Could not retrieve module: /path/to/basex/webapp/synopsx/models/synopsx.xqm. although nothing changed in our webapp. Thanks a lot, and all the best :-) Maud

2 1

How To: Sort nodes in document order?
by Eliot Kimber 04 Jul '22

04 Jul '22

I feel like I should know how to do this (and possibly I’ve previously been told how and have forgotten), but I’m at a loss. I am implementing a process that gathers nodes from a single document (a DITA map) and constructs a system of maps where I’m using the maps primarily to represent a tree and enable lookup of things by key. However, the order of the elements that make up these maps is also important (DITA’s rules for how you build DITA key spaces from maps depends entirely on document order). For example, I have a map that maps “key names” to the elements that declare those key names, i.e.: map { “key-01” : (<topicref keys=”key-01” href=”foo.dita”/>, <topicref keys=”key-01” href=”bar.dita”/>) } This is the initial state of the map (reflecting pass 1 over the DITA map document) but then I need to add additional elements to the sequence that is the value of entry “key-01” and I need the resulting sequence to be in document order. So my question: given two sequences of nodes from the same document, how does one construct a new sequence where the nodes are in document order? Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

3 4

Exporting XML with a relaxNG declaration
by France Baril 01 Jul '22

01 Jul '22

Hi, I am trying to output XML with a link to a RelaxNG schema without success. - It works with DTDs using doctype-public and doctype-systems in the serialization params of file:write - It works with .xsd because the declaration is in the attribute - For relaxNg, the intro line looks like a PI, but I can't seem to be able to keep PIs when outputting xml with file:write Specifically: let $map-params := map { 'method': 'xml', 'indent': 'no', 'omit-xml-declaration': 'no' } let $map := ( <?xml-model href="urn:oasis:names:tc:dita:rng:map.rng" schematypens="http://relaxng.org/ns/structure/2.x"?>, <map> <?test name="abc"?> <title>{lower-case($spez/child::*[name()='name']/text())}</title>{ <title>{lower-case($spez/child::*[name()='name']/text())}</title>{ ... }</map> ) return file:write($dir-path || '/spec-' || lower-case($spez/child::*[name()='name']/text()) || '.ditamap', $map) gets me: <map> <title>Tests for elements - version 2.0</title> ... </map> Both the declaration and the in-content test pi disappear. What am I missing here? -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 2

the implicit context of position
by Graydon Saunders 30 Jun '22

30 Jun '22

Hello -- I've got an element node that's part of a document. I need its position among the element children of its parent. $step/position() will (correctly!) always return 1, there's one thing in $step, of course it does. replace($step/path(),'^.*\[(\p{Nd}+)\]$','$1') => xs:integer() gets me the implicit position but is something of a horror. for $e at $in in $step/../* where $e is $step return $in works, too, but any notion of elegance and clarity have fled screaming into the night. I keep thinking there must be a better way; anyone got any suggestions? Thanks! Graydon

4 4

Sym links in BaseX 9.7.2
by Maud Ingarao 30 Jun '22

30 Jun '22

Hi everybody ! As we had problems getting our formatDateTime() outputs in french, we followed Christian's advice and updated to BaseX 9.7.2. Now it works and we get our outputs in French, thanks a lot ! But we face a new problem : symbolic links inside the webapp are no longer followed. We tried to had the following to jetty.xml : | <Call name="addAliasCheck"> <Arg><New class="org.eclipse.jetty.server.handler.AllowSymLinkAliasChecker"/></Arg> </Call> | But this doesn't seem to be the answer. I can't remember configuring anything in particular in the previous versions of BaseX and our webapp was just running fine. Thanks for your help. Maud || -- Maud Ingarao IHRIM - UMR 5317 Institut d’histoire des représentations et des idées dans les modernités École Normale Supérieure de Lyon 15 Parvis René Descartes - BP7000 - 69342 Lyon CEDEX 07 +33 4 37 37 65 79 - maud.ingarao(a)ens-lyon.fr */Présente les lundi - mardi - mercredi/* */At the office on mondays - tuesdays - wednesdays/* http://ihrim.ens-lyon.fr/ http://ahn.ens-lyon.fr/ https://cahier.hypotheses.org/

2 1

How To Determine if You're In Daylight Saving Time?
by Eliot Kimber 28 Jun '22

28 Jun '22

I’m showing times on RESTXQ generated pages that need to be adjusted for a particular time zone. Doing the time zone adjustment is easy enough but I realized that knowing whether or not a given time zone should be standard or daylight time is a challenge not directly addressed by the XQuery time functions. So my question: does anyone have a technique for determining if a given time zone or region is using DST? Of course I can hard-code some sort of DST calendar for the time zones I know I need (U.S. time zones, European Central Time, India Standard Time) but it seemed like there ought to be a more general solution somewhere. My server is running on Linux but is developed and tested on macOS. Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 1

Fonts problem with database list and system dialogs
by Jonathan Robie 28 Jun '22

28 Jun '22

I can't read the names of databases in my UI now - the font is tiny. And the systems dialogs I might otherwise be able to use to find and fix the problem also use tiny fonts. Other windows are fine. [image: image.png] Jonathan

2 7

XPATH injection / escaping
by trichel 26 Jun '22

26 Jun '22

Hello, How to write secure queries when the queried text nodes contain ampersands? For instance: declare variable $publisher external; (: $pub == 'Faber & Faber' :) declare variable $db := db:open('db'); let $records := $db/record/publisher[. = $publisher] (: publisher == 'Faber & Faber' :) The external variable is unsafe input, escaped by the sending application. Escaping the ampersand in the external variable with & (& a m p ;) doesn't work, Basex stops finding hits. Just letting the ampersand pass might expose the code to injection attacks? I could switch to a full-text query and remove the ampersand from the external variable, but that's a bit hackish. The expression is exact. How to proceed in a secure way?

2 4

Link Graph Construction: Anything I Can Crib or Learn From?
by Eliot Kimber 26 Jun '22

26 Jun '22

In the context of our Project Mirabel system that manages DITA content, I need to be able answer the question “for topic X, what other topics link to it directly or indirectly?” That is, say Topic A links to Topic B that Links to Topic C. Asking the question “What topics ultimately link to topic C?” I would like to get the answer “Topic A, Topic B”. Getting the answer for direct references is easy—I already build a where-used index that captures, for each DITA map or topic, what other maps and topics link directly to it. But to get the Topic A part of the answer I need some kind of link graph index and I’m not sure how best to go about calculating this or capturing it in some index or set of indexes. In our content the fan out from a single Topic C to the set of topics that ultimately reference it could be 10s of 1000s of topics. We have about 45K topics in the content for each version of the ServiceNow Platform and a number of topics that are used by a large number of other topics, so the explosion can be quite large. That suggests that a simple topic-to-ultimately-referenced-topics index would be very inefficient in that the entry for any given topic could potentially have 45K – 1 entries (we don’t care that a topic references itself). On the other hand, working backwards through chains of direct references can also be expensive and is probably too slow, so maybe the brute-force index is the best option? At the same time, I would like to be able to quickly visualize the link graph extending from or ending in any given topic or simply the link graph for the entire information set, which requires capturing the nodes and edges. My question: does anyone either have experience or insight into this kind of link graph challenge or know of relevant papers or general discussion of graph processing I might look at? Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

4 3

Construct URL-Friendly Base64 Strings
by Eliot Kimber 22 Jun '22

22 Jun '22

I’m trying to replicate a call to a remote REST API that uses hmac encoding to construct an authentication string. I’m working from existing Python code that does the same thing. What I’m seeing is that the crypto:hmac() function can return Base64 strings that include characters that must be escaped in URIs, i.e.: mTA76dRZkiCf84WUKM/PWySydamj3dMZ6Tm26s9QGNI (Note the “/”) Python’s base64 support includes the method urlsafe_b64encode(), which avoids the characters “+” and “/”. Unfortunately the handler of the API call fails if I escape the “/” or “+”—I’m not sure why. So I need to construct the same URI-friendly base64 string that the Python method is generating. I don’t see an obvious way to do this with the BaseX conversion package. Is there a way short of implementing my own Base64 encoding algorithm to get a URI-friendly base64 string? I’m using 9.6.4 at the moment but could upgrade if necessary. Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 2

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk