Installing Fiedler Data Tools

This guide installs Fiedler Data Import and Fiedler Data Export on the OpenShift cluster that runs your IBM Maximo Application Suite (MAS). At the end, people open the apps in the browser, sign in through MAS, and import and export Excel workbooks with their own rights in Manage.

Everything comes from two places. The container images are public on quay.io. Every script and every file of the installation is in one install kit, which you download from this page. The steps are the ones we run on our own test system, on the MAS 9.2 series.

Two people usually share the work. The OpenShift administrator deploys the apps, sections 1 to 9. The Maximo administrator installs the Manage-side components and gives people access, sections 10 to 12. You can install one app alone, and the other one later. They share the licence, and nothing else depends on the order.

What gets installed

#

For each app, in the MAS core namespace:

• One pod per Manage workspace, with a data volume of 5 GiB for the jobs and their files. It runs beside IBM’s own MAS components and signs people in through MAS.
• A Route under the suite’s domain. Under subdomain routing it is the host fiedler-data-import.<domain> or fiedler-data-export.<domain>. Under path routing it is the path /fiedler-data-import or /fiedler-data-export.
• The app’s own service account, with a Role that may read and write one Secret, fiedler-data-tools-license, and read MAS’s Suite for its domain.
• Optionally, monitoring: a ServiceMonitor and a PrometheusRule with the app’s alerts.

Shared by both apps:

• The licence Secret fiedler-data-tools-license, created empty by the first deploy. It holds the one licence of Fiedler Data Tools. Without a licence, the free edition applies.

In Manage, installed by each app’s administration screen with the administrator’s own credential:

• Four automation scripts per app: FDIMP_PING, FDIMP_META, FDIMP_ROWS and FDIMP_INSTALL for the import, FDEXP_PING, FDEXP_META, FDEXP_READ and FDEXP_INSTALL for the export. The export’s scripts only read.
• One application per app, FDIMPORT and FDEXPORT, with their signature options, and two security groups per app.

Optionally, for Fiedler Data Export: a nightly export, a CronJob that runs the command line client on a schedule. It needs a licence.

1. What you need

#

2. The images

#

Each app is one container image. It holds the server, the web interface with its fonts, and the app’s Manage-side scripts. The apps need no internet access at run time: they talk to MAS and Manage inside your cluster, and the browser loads nothing from another site. The nightly export uses a third image, which holds the command line client.

Each version has its own tag, and latest follows the newest version. Pull the images without a login. Refer to each image by its digest: a digest names exactly one build, and it stays the same in a mirror. This prints the digest of a version:

skopeo inspect --format '{{.Digest}}' docker://quay.io/fiedler-it/fiedler-data-import:1.0.1

The digests of the current version are also in the kit’s README.md.

Mirror the images when the cluster has no route to quay.io. skopeo copy --preserve-digests keeps the digest, and the second command shows the same digest from your mirror:

skopeo copy --preserve-digests docker://quay.io/fiedler-it/fiedler-data-import:1.0.1 docker://<your-mirror>/fiedler-data-import:1.0.1
skopeo inspect --format '{{.Digest}}' docker://<your-mirror>/fiedler-data-import:1.0.1

Do the same for fiedler-data-export, and for fde-cli when you use the nightly export. For a network that is fully disconnected, we send the images as OCI archives on request.

3. The install kit

#

The kit holds every file of the installation. Download it, check it and extract it:

• fiedler-data-tools-install-1.0.1.tar.gz, the kit of version 1.0.1
• fiedler-data-tools-install-1.0.1.tar.gz.sha256, its checksum

curl -LO https://fiedler.site/products/fiedler-data-tools/installation/fiedler-data-tools-install-1.0.1.tar.gz
curl -LO https://fiedler.site/products/fiedler-data-tools/installation/fiedler-data-tools-install-1.0.1.tar.gz.sha256
shasum -a 256 -c fiedler-data-tools-install-1.0.1.tar.gz.sha256
tar -xzf fiedler-data-tools-install-1.0.1.tar.gz
cd fiedler-data-tools-install-1.0.1
shasum -a 256 -c SHA256SUMS --quiet

On Linux, sha256sum -c does the same as shasum -a 256 -c. All further commands run in the kit’s folder.

Keep deploy/ and export/deploy/ side by side. The export’s deploy script also reads deploy/license.yaml.

4. Find your cluster’s values

#

The overlays need a few facts about your MAS installation. Each row gives the command that finds the value.

An empty routing mode means subdomain routing. A Manage split into several server bundles has one service per bundle, and the apps use the one marked as the default. The issuer must show Ready, and the right to create the Role must answer yes.

5. Write the overlays

#

The manifests are kustomize files. Each app’s base/ holds its objects for any cluster, and an overlay adds the values of one cluster. The kit brings two example overlays per app: example-subdomain and example-path. Copy the one for your routing mode for both apps, and name the copies after your cluster. This uses prod:

cp -r deploy/overlays/example-subdomain deploy/overlays/prod
cp -r export/deploy/overlays/example-subdomain export/deploy/overlays/prod

Every cluster value in them is a placeholder from the table of section 4. One sed fills both apps. Put your own values in place of the examples here:

sed -i.bak \
  -e 's/<core-ns>/mas-prod-core/g' \
  -e 's/<instance>/prod/g' \
  -e 's/<domain>/mas.example.com/g' \
  -e 's/<workspace>/main/g' \
  -e 's/<bundle>/all/g' \
  deploy/overlays/prod/*.yaml export/deploy/overlays/prod/*.yaml
rm deploy/overlays/prod/*.bak export/deploy/overlays/prod/*.bak

What the files of an overlay hold:

• kustomization.yaml: the namespace, the image names, and the optional components. Remove the # before components: and ../../components/monitoring when user workload monitoring is on. When you use a mirror, set each image’s newName to it.
• certificate.yaml: the app’s internal certificate, with your namespace in its DNS names and your issuer.
• deployment.yaml: a JSON patch with the app’s addresses: the routing mode, the domain, the MAS sign-in endpoints and Manage’s addresses. Each value comes after a test of the variable’s name, so a changed base stops the build before a value lands in the wrong variable.
• config.yaml, Fiedler Data Import only: its settings. A production system needs the one line Instance.Kind: production. The app checks the file when it starts, and the log line names any setting with a wrong value.
• cron-profile.yaml, Fiedler Data Export only: the nightly export’s profile, used only with that component. Section 14 explains it.

Two values may need a second look:

• MANAGE_INTERNAL_URL points the app at Manage’s service inside the cluster, which is faster than Manage’s public route. Empty, the app calls Manage through its public route.
• TRUST_PROXY_HOPS, Fiedler Data Import only, counts the proxies in front of the pod that write X-Forwarded-For. The OpenShift router is one. Behind a load balancer that ends TLS and sets the header, it is 2. The import’s audit record names the caller’s address only when this count is right.

The export’s settings are environment values too, and the defaults suit a first install. To set one, add a line to its deployment.yaml, for example three jobs at once:

- { op: add, path: /spec/template/spec/containers/0/env/-, value: { name: JOBS_CONCURRENT, value: "3" } }

Check the overlays before you apply them. Each must render, and no placeholder and no set-by-overlay may be left:

oc kustomize deploy/overlays/prod | grep -c -e set-by-overlay -e '<[a-z-]*>'          # prints 0
oc kustomize export/deploy/overlays/prod | grep -c -e set-by-overlay -e '<[a-z-]*>'   # prints 0

6. Deploy Fiedler Data Import

#

deploy/deploy.sh does the first deploy in one run. It reads the namespace, the host and the path from your overlay, so the Route and the pod always agree. Pin the image by its digest:

DIGEST=$(skopeo inspect --format '{{.Digest}}' docker://quay.io/fiedler-it/fiedler-data-import:1.0.1)
OVERLAY=prod IMAGE=quay.io/fiedler-it/fiedler-data-import:1.0.1 IMAGE_DIGEST=$DIGEST ./deploy/deploy.sh

The script runs these steps and prints each one:

1. It checks your login and the issuer.
2. It creates the OIDC client secret with a generated value. The value lands in the cluster Secret only.
3. It creates the pull secret. An image from quay.io needs no login, so the pull secret holds no credentials. Then it creates the app’s service account with its Role and RoleBinding, and the empty licence Secret when that is missing. A licence installed later is never reset by a deploy.
4. It applies the overlay: the Certificate, the PVC, the ConfigMap, the NetworkPolicy, the Deployment, the Service, and the monitoring objects when the overlay has them.
5. It waits for the certificate, then creates the Route with the issued CA.
6. It registers the OIDC client with MAS through a short Job in the cluster.
7. It waits for the rollout and checks the result.

A good run ends with internalapi HTTP 200 and DONE: open https://<host><path>/. The last step prints the app’s version through the Route, the sign-in redirect to MAS, the readiness answer, and a call from the pod to MAS’s internal API over mutual TLS.

From a mirror with a login, give the script the credentials once. It reads them only when the pull secret does not exist yet, and the password goes in without an echo:

export REGISTRY_USER=<user>
read -rs REGISTRY_PASSWORD; export REGISTRY_PASSWORD
OVERLAY=prod IMAGE=<your-mirror>/fiedler-data-import:1.0.1 IMAGE_DIGEST=$DIGEST ./deploy/deploy.sh

7. Deploy Fiedler Data Export

#

export/deploy/deploy.sh works the same way:

DIGEST=$(skopeo inspect --format '{{.Digest}}' docker://quay.io/fiedler-it/fiedler-data-export:1.0.1)
OVERLAY=prod IMAGE=quay.io/fiedler-it/fiedler-data-export:1.0.1 IMAGE_DIGEST=$DIGEST ./export/deploy/deploy.sh

It finds the licence Secret of Fiedler Data Import and keeps it. A good run ends with internalapi HTTP 200 and DONE: open https://<host><path>/ and sign in.

With the nightly export in the overlay, pin its image too, with CLI_IMAGE=quay.io/fiedler-it/fde-cli:1.0.1 and CLI_IMAGE_DIGEST=<its digest>. RENDER_ONLY=1 prints the manifests the script would apply, with the images pinned, and changes nothing.

From a mirror with a login, the export can copy the import’s pull secret, so no password passes through your shell:

REGISTRY_PULL_FROM=fiedler-data-import-registry-pull OVERLAY=prod \
  IMAGE=<your-mirror>/fiedler-data-export:1.0.1 IMAGE_DIGEST=$DIGEST ./export/deploy/deploy.sh

8. The service account, the Role and the licence Secret by hand

#

The deploy scripts create these objects. An OpenShift administrator who prefers to create them alone, or a pipeline without the right to create Roles, uses the same template. It holds the ServiceAccount, the Role and the RoleBinding of one app:

for app in fiedler-data-import fiedler-data-export; do
  sed -e "s|__APP__|$app|g" -e 's|__NS__|<core-ns>|g' deploy/license.yaml | oc apply -f -
done
oc get secret fiedler-data-tools-license -n <core-ns> \
  || oc create secret generic fiedler-data-tools-license -n <core-ns>

The Role allows get, update and patch on that one Secret, by name, and get and list on MAS’s Suite. Kubernetes cannot limit create to one name, so the Secret must exist before the first start. Check the result:

oc auth can-i update secret/fiedler-data-tools-license -n <core-ns> --as=system:serviceaccount:<core-ns>:fiedler-data-import   # yes
oc auth can-i list secrets -n <core-ns> --as=system:serviceaccount:<core-ns>:fiedler-data-import                            # no

9. Monitoring, optional

#

With user workload monitoring on and ../../components/monitoring in an overlay, the deploy creates a ServiceMonitor and a PrometheusRule for that app: eight alerts for the import and six for the export. Each alert carries a severity label, so your Alertmanager routes it. Check that Prometheus scrapes the pods: in the OpenShift console, under Observe and Metrics, the queries up{job="fiedler-data-import"} and up{job="fiedler-data-export"} give 1.

Each app also brings a Grafana dashboard, components/monitoring/dashboard.json, with the datasource as a variable. There are two ways to bring it in:

• A Grafana that the MAS CLI’s Grafana operator runs: add ../../components/grafana to the overlay’s components. It creates a GrafanaDashboard that this operator selects.
• A Grafana of your own: it needs read access to the cluster’s metrics first. export/deploy/grafana-datasource.sh creates a service account that may read them, and a Prometheus datasource in your Grafana with its token. The token goes from the cluster into Grafana through a pipe and is never printed. Then import dashboard.json in Grafana.

export GRAFANA_URL=https://<your-grafana>
read -rs GRAFANA_TOKEN; export GRAFANA_TOKEN
./export/deploy/grafana-datasource.sh mas-thanos

10. Accept the licence agreement and install the Manage-side components

#

Open each app’s administration screen as the Maximo administrator:

Before the first install, the app’s application does not exist in Manage, so nobody holds its options. The app therefore lets in a user who may save in the Automation Scripts and the Application Designer applications, and shows that user the components only.

The screen reads and writes the scripts through the object structure MXAPIAUTOSCRIPT, so that user needs its options too. On a Manage installed without demo data, even MAXADMIN lacks all three, and the screen says “No access”. Grant them to the group MAXADMIN in Security Groups: Automation Scripts and Application Designer on the Applications tab, MXAPIAUTOSCRIPT on the Object Structures tab. A grant on an object structure alone starts no profile sync in Manage. In our test on such a Manage, saving a small change to the user MAXADMIN in Users, such as the city, started it. Sign in again a minute or two later.

First the licence agreement of Fiedler Data Tools. The screen shows it with Read and accept: read it, tick the box, and press Accept. The app records the acceptance with your user, the time and the version of the agreement. Install or upgrade stays off until then. The same text is LICENCE-AGREEMENT.md in the kit.

Then press Install or upgrade. The app does three things with your own credential:

1. Scripts. It creates or updates its four scripts through Manage’s standard object structure MXAPIAUTOSCRIPT.
2. Application, options and groups. Its install script creates the application, the signature options and two security groups. MAXADMIN and you join both groups.
3. Profiles. It clears Manage’s cached profiles of the groups’ members. MAS takes the new options at Manage’s next profile sync, often within two minutes. Then open the screen again, and it shows the rest of the administration.

The report lists every script with its result and the options you now hold. Pressing the button again is safe: an install that finds everything in place changes nothing. Do the same on the other app’s screen.

Manage keeps an application only when its ORIGINALAPP holds a value. When a screen reports “Repair needed”, the install fills it, and Manage takes the repaired application at its next restart.

11. When the install cannot write the scripts

#

Some sites switch on object structure security for MXAPIAUTOSCRIPT, or allow no app to write a script. The install then reports the scripts as failed. You create the four scripts of that app by hand, and the app does the rest.

1. Download the scripts. On the administration screen, follow Download the scripts. The zip holds each script as <NAME>.js and the file scripts.json with every other field.
2. Create each script in Manage’s Automation Scripts application, in this order: PING, META, then ROWS or READ, then INSTALL. Use More Actions > Create > Script, the variant without a launch point, and take every value from scripts.json:

3. Finish on the administration screen. Press Install again. The four scripts now show as up to date, because their sources match, and the install script creates the application, the options and the groups as in section 10.

When the install script cannot run either, create its rows by hand. In Application Designer, create the application FDIMPORT with the description “Fiedler Data Import”, or FDEXPORT with “Fiedler Data Export”. Add the signature options of section 10, each with “Electronic signature” off and “Visible” on. Then create the two groups in Security Groups, grant their options, and add the users.

12. Give people access

#

In Manage’s Security Groups application, add each person who imports to FDIMPORT and each person who exports to FDEXPORT. Add the administrators of each app to FDIMPADM or FDEXPADM. Then open the app’s administration screen, enter the group under Refresh profiles, and press the button. The new options work at once, and nobody has to sign out.

• A user who should import through object structures only gets IMPORT and leaves out MBOPATH.
• Two export options stay out of the users’ group on purpose. WHERE lets a user add a where clause of their own, and OBJECTS lets a user export a main object without an application.
• Each app works with the user’s own rights in Manage. A user without READ in ASSET cannot export assets, whatever the export’s options say.

13. A licence

#

Both apps run without a licence in the free edition. An import takes up to 200 rows of each sheet, an export writes up to 200 records, and up to three users may use each app. A licence lifts the limits, and it unlocks the command line clients and scheduled exports.

An administrator installs the licence on either app’s administration screen. It takes effect at once in that app, and the other app reads it within a minute. An OpenShift administrator can also put the licence file into the Secret. The apps read it every minute:

oc create secret generic fiedler-data-tools-license -n <core-ns> --from-file=license=<licence file> \
  --dry-run=client -o yaml | oc apply -f -

The command line clients fdi and fde come with a licence, and we send them with it. Get in touch for a licence, a trial licence or a price.

14. The nightly export, optional

#

The component export/deploy/components/cron runs one export every night with the command line client, as an integration user, and keeps each night’s files on a volume of its own. The job shows in the app’s History under that user. It needs a licence.

The user. Pick an integration user who belongs to FDEXPORT, and create a Maximo API key for that user. Put the key into a Secret once. It is read without an echo and goes into the cluster through a pipe:

read -rs CRON_KEY
printf '%s' "$CRON_KEY" | oc create secret generic fiedler-data-export-cron-apikey -n <core-ns> --from-file=apikey=/dev/stdin
unset CRON_KEY

The overlay. In export/deploy/overlays/prod/kustomization.yaml, remove the # before components:, before ../../components/cron and before - path: cron-profile.yaml. cron-profile.yaml tells the client where the app is: through its Service, with the CA of the app’s internal certificate. The sed of section 5 has already filled it.

The selection and the time. As it comes, the component exports the assets of the demo site BEDFORD at 02:30, Vienna time. Replace both with a patch in the overlay. The selection can be any fde export command. This patch exports the open work orders of a saved query on weekdays at 03:00, in the time zone of New York:

- op: replace
  path: /spec/schedule
  value: "0 3 * * 1-5"
- op: replace
  path: /spec/timeZone
  value: America/New_York
- op: replace
  path: /spec/jobTemplate/spec/template/spec/containers/0/args/0
  value: |
    set -eu
    day=$(date +%Y-%m-%d)
    fde export --target WORKORDER --app WOTRACK --query "Open work orders" --preset records -o "/out/$day"
    find /out -mindepth 1 -maxdepth 1 -type d -mtime +7 -exec rm -rf {} +

Name the file in the overlay’s patches, with target: { kind: CronJob, name: fiedler-data-export-nightly }. Each night’s files go into a folder named by the date, and folders older than seven days are removed after a good run. Then deploy the export again with CLI_IMAGE and CLI_IMAGE_DIGEST, as in section 7.

15. Check the installation

#

16. Updates

#

A new version needs only the new images. With APPLY_ONLY=1, a deploy script applies the overlay and waits for the rollout. It leaves the secrets, the Route and the OIDC client as they are:

DIGEST=$(skopeo inspect --format '{{.Digest}}' docker://quay.io/fiedler-it/fiedler-data-import:<version>)
APPLY_ONLY=1 OVERLAY=prod IMAGE=quay.io/fiedler-it/fiedler-data-import:<version> IMAGE_DIGEST=$DIGEST ./deploy/deploy.sh

The same command with export/deploy/deploy.sh and the export’s image updates the export. Each version comes with its own kit, so take the scripts from the kit of that version.

The Deployment replaces its pod in one step, and the app is away for the few seconds the new pod needs to start. Plan an update outside the hours of long jobs. Then open each administration screen. When a version brings new scripts, the screen offers them, and Install or upgrade installs them.

17. Remove the apps

#

Remove the objects in Manage first, while the app still runs. Each administration screen’s Uninstall shows what it removes. Then it deletes the app’s application, its options and grants, the two groups with their members, and the four scripts. Records that people imported stay in Manage.

Then remove each app from the cluster. The undeploy script removes the OIDC client registration first, then every object of the app in the namespace. Name your namespace; KEEP_DATA=1 keeps the data volume with its jobs:

NS=<core-ns> KEEP_DATA=1 ./deploy/undeploy.sh
NS=<core-ns> KEEP_DATA=1 ./export/deploy/undeploy.sh

The licence Secret stays, so a licence survives a reinstall. oc delete secret fiedler-data-tools-license -n <core-ns> removes it as well. When an app is gone already, remove its objects in Manage by hand: the four scripts in Automation Scripts, the application in Application Designer, and the two groups in Security Groups.

Help

#

Write to support@fiedler-it.at with the app, its version and build from the administration screen, and what you saw. For a licence, a trial or a price, get in touch.

IBM and Maximo are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide.