Multi-ID Enrichment — Release Notes

Worker Injection — where enrichment enters the normal process

The enrichment module wraps three points in mpi_and_emit_worker.py without changing the core MPI call. Every step degrades gracefully if its flag is off or a dependency is unavailable.

GitLab: mpi_and_emit_v2 (enhancement_id)

① Pre-match enrichment — supplemental.py

Runs when ENABLE_MSG_METADATA_ENRICHMENT = True, before issue_mpi_call().

1. Opens a warehouse connection (metadata_conn, separate from the main connection).
2. Queries reliance_patient_ids by msg_id → a list of (assigning_authority, type, patient_id) rows.
3. Filters by MSG_METADATA_ALLOWED_ID_TYPES — currently mrn / mr / fmrn → fmrn. SSN excluded by decision.
4. Merges FMRNs into data_tags[FMRN_FIELD], deduping on (authority.upper(), id). A hard cap of 50 identifiers is enforced before the payload is passed to the MPI call — in practice most pipelines present 2–3 IDs; the cap prevents payload growth for edge-case contributors with large cross-reference lists.
5. Queries reliance_msg_facility_set_org by msg_id → sets sending_org_id in the payload.

If either query fails, the error is logged and the process continues without the enriched identifiers — the MPI call still fires.

② Post-match recording — outcome.py

Runs after issue_mpi_call() returns, when ENABLE_MATCH_QUALITY_EVAL = True.

1. Calls _fetch_pi_rule_with_retry(pi): polls store 1100 every 1 s for up to 10 s until rule_fired and match_type appear. Store 1100 indexes a PI asynchronously, so a single immediate read races the index.
2. Calls evaluate_fmrn_utilization() to set fmrn_utilized and fmrn_available_unused.
3. Upserts one row into warehouse.reliance_match_reconciliation with 38 fields covering identifiers present, rule fired, match type, timing and enrichment flags.

③ Rollback capture — rollback.py

Runs after a successful match, when ENABLE_ROLLBACK = True.

1. Writes one row to utilities.rollback_pipeline: msg_id, pi (e.g. 52309187.PI), mpid, imat_dest, psuri.
2. Reverse is via rollback_script.py --msg-id <id>, which strips the .PI suffix and calls extensions/mpi/transactions/delete.

What enrichment does NOT change

Settings & Flags

All three feature flags ship off. A flag change takes effect on the next batch, about 1 minute, with no worker restart.

Tuning settings

Confirm flags after deploy

cd /opt/tools/mpi_and_emit_v2
python -c "
from enrichment.message_enrichment_pipeline import MessageEnricher
e = MessageEnricher()
print('metadata_enrichment_available:', e.metadata_enrichment_available)
print('match_quality_eval_available: ', e.match_quality_eval_available)
e.close()
"
# Phase 0 expected output:
# metadata_enrichment_available: False
# match_quality_eval_available:  False

Database Tables

The enrichment module reads Mirth-owned tables and writes to tables it owns. Mirth-owned tables are never modified; the service user holds SELECT only on them.

Mirth-owned (read-only to enrichment)

Enrichment-owned (read/write)

Stage-only (not deployed to production)

Migration status

Entity Relationship Diagram

Full schema including service user access lines. Orange dashed = SELECT only (stage also UPDATE/DELETE); teal = SELECT·INSERT·UPDATE·DELETE; blue = SELECT·INSERT·UPDATE·DELETE.

Users that interact with the code

Three actors touch the system: the Mirth service that populates the source tables, the reliance DB user that the worker and rollback run as, and the operator who runs rollback_script.py manually.

Mirth (populates source tables)

Enrichment reads these tables but never writes to them. Every prod channel must populate them before enrichment is enabled — verify with the SQL in the Production Prerequisites section of the main doc.

reliance@% (worker, backfill, rollback)

Operator (rollback)

Runs rollback_script.py manually, always with --msg-id <id> or an explicit list. Never run bare in production — it would process every pending row in rollback_pipeline.

Also needs a RelianceApiCall session with access to:

• extensions/mpi/transactions/delete — to remove PI records from store 1100
• /cache/deleterecord — to remove source records from all search stores
• Read access to store 1100 and store 1101 (to verify cleanup)

Credentials file: /opt/search/appliance5/conf/.mariadb_connection.json (prod) or .mariadb_connection_local.json (stage), containing the reliance user and password. Must be chmod 600.

Release Plan

Four phases, one flag per phase, each started only when the previous gate passes. A flag change takes effect within about one minute with no restart.

PI recording — stays on until results are confirmed

When ENABLE_ROLLBACK = True, every matched message gets a row in utilities.rollback_pipeline recording its PI, MPID and source URI. This is the audit trail that enables a Level 4 data rollback if a wrong match is confirmed.

The PI value stored is the raw form from the MPI response (e.g. 52309187.PI). rollback_script.py strips the .PI suffix before calling the delete endpoint — confirmed working on stage after the fix in this release.

Split / merge reporting

The backfill job runs every 2 hours and captures persona-level stats including persona_shrank (a possible split) and multi_ssn_persona (a possible erroneous merge). These feed the weekly report generated by generate_match_quality_report.py.

A split signal: distinct MPIDs per day rising while messages per persona falls. A merge signal: multi_ssn_persona rate above the Phase 1 baseline, or mpid_reassigned appearing for a message that previously matched correctly.

Rollback plan summary

Level 1 — one-liner

sed -i.bak 's/^ENABLE_MSG_METADATA_ENRICHMENT *= *True/ENABLE_MSG_METADATA_ENRICHMENT = False/' \
  /opt/tools/mpi_and_emit_v2/settings.py && grep ENABLE_MSG settings.py

Gate query (run daily from Phase 1)

SELECT DATE(processed_at)                                                    AS day,
       COUNT(*)                                                               AS messages,
       ROUND(100*SUM(pi IS NOT NULL)/COUNT(*), 2)                             AS matched_pct,
       ROUND(100*SUM(pi IS NOT NULL AND rule_fired IS NULL)/COUNT(*), 2)      AS missing_rule_pct,
       ROUND(100*SUM(rule_fired IN (2,7))/COUNT(*), 2)                        AS identifier_rule_pct,
       ROUND(100*SUM(fmrn_utilized = 1)/COUNT(*), 2)                          AS fmrn_utilized_pct,
       ROUND(100*SUM(mpid_reassigned = 1)/COUNT(*), 3)                        AS reassigned_pct
FROM warehouse.reliance_match_reconciliation
WHERE processed_at >= CURDATE() - INTERVAL 14 DAY
GROUP BY DATE(processed_at) ORDER BY day;

Cron Updates

One cron addition (backfill) and two stage-only crons to remove before production deploy.

Add: persona backfill (every 2 hours)

0 */2 * * * root flock -n /var/lock/persona_backfill.lock \
  /opt/tools/miniconda3/envs/python3_12/bin/python \
  /opt/tools/mpi_and_emit_v2/enrichment/run_persona_backfill.py \
  --lookback-hours 3 \
  >> /opt/tools/mpi_and_emit_v2/enrichment/match_quality_eval/persona_backfill.log 2>&1

The flock -n wrapper is required. Stage runs show one persona-builder pass can take 17–67 minutes for 1,000–2,700 MPIDs, so overlap is possible without it. If the lock is already held, the new invocation exits silently and the next scheduled run takes over.

Remove: stage-only pressure test crons

These must not appear in production crontabs:

# Remove these lines from crontab on stage before promoting to prod:
* * * * * /usr/bin/python3 /opt/tools/sql_query/sql_query.py stage_ccd_preasure_prod_tests 1000 >> ...
* * * * * /usr/bin/python3 /opt/tools/sql_query/sql_query.py stage_hl7_preasure_prod_tests 1000 >> ...

Backfill health check

grep -E "can't open|failed|Traceback|Backfill complete" \
  /opt/tools/mpi_and_emit_v2/enrichment/match_quality_eval/persona_backfill.log | tail -20

Every successful run ends with Backfill complete: N processed, 0 failed. One run on 2026-09-16 failed 339 of 1,802 (19%) — cause unknown, all other runs showed 0 failures. Monitor this on Phase 1 day 1.

Expected throughput

Production will likely see larger windows. Measure during Phase 1 and confirm the 2-hour interval is sufficient.

Blocking and Non-Blocking Items

Blocking — must complete before deploy

Open Questions

Items marked with a status reflect answers received so far.

Rules Reference

Full rule set from production rules.xml, provided 2026-09-24. Rule 0 (exact re-match) is engine-level and does not appear in the file.

Key rules for Multi-ID analysis

Rule transition map (pressure test)

Full rule listing

Note: rule 35 is absent from the provided rules.xml. Rule 0 is engine-level and does not appear in the file.

Maintenance

Ongoing responsibilities after release: who reviews the reports, how often, and what happens when identifier data quality issues — including blanks — are found.

Weekly match quality report

ID quality issues — blank or missing identifiers

When reliance_patient_ids has no rows for a message, enrichment adds nothing and the match falls back to whatever IMAT extracted natively. This is silent — no error is raised. The feedback loop to surface this:

Feedback loop — from report finding to fix

1. Reviewer identifies an issue in the weekly report or gate query — e.g. a facility showing ids_available = 0 for all its messages, or a spike in mpid_reassigned.
2. Reviewer raises it with the Mirth channel owner (if the issue is blank/missing IDs in reliance_patient_ids) or with the rules owner (if the issue is a mis-merge pattern in reliance_match_reconciliation).
3. Channel fix: Mirth channel owner corrects the ID extraction mapping and reprocesses affected messages if needed. New messages from that channel should start populating reliance_patient_ids correctly on the next run.
Allowlist fix: If a valid ID type is present in the DB but excluded by MSG_METADATA_ALLOWED_ID_TYPES, Reliance data ops updates settings.py after confirming the type is safe to pass to the matcher. SSN requires a separate compliance decision before it can be added.
4. Rollback if needed: If a confirmed wrong match is traced to an enriched identifier, trigger a Level 4 rollback for the affected messages (see the Release Plan tab) and re-feed after the fix is in place.
5. Verify: Check the next weekly report to confirm the issue is resolved and metrics have returned to baseline.

ID quality flagging — rolling 3-month list

The enrichment module validates every row from reliance_patient_ids before injecting it into the FHIR payload. Rows that fail validation are skipped (not passed to the matcher) and written to warehouse.reliance_id_quality_flags with a reason code. This table retains 3 months of history; older rows are purged at the start of each backfill run.

Validation rules — what gets flagged and skipped

Reviewing flagged IDs

Run this weekly alongside the match quality report to surface new patterns. Results grouped by flag reason and facility — new combinations that haven't appeared before are the ones to investigate first:

# All flags in the last 7 days, grouped by reason and facility
grep "$(date -d '7 days ago' +%Y-%m-%d)" \
  /opt/tools/mpi_and_emit_v2/enrichment/logs/id_quality_flags.log \
  | python3 -c "
import sys, json
from collections import Counter
counts = Counter()
for line in sys.stdin:
    r = json.loads(line)
    counts[(r['flag_reason'], r['sending_facility'])] += 1
for (reason, fac), n in counts.most_common():
    print(f'{n:>6}  {reason:<40}  {fac}')
"

To pull all entries for a specific message:

grep '"msg_id": 672018724' /opt/tools/mpi_and_emit_v2/enrichment/logs/id_quality_flags.log

Feedback loop — from flagged ID to interface analyst

1. Reviewer runs the flag query weekly. Any flag_reason + sending_facility combination with unreviewed > 0 is a candidate for investigation.
2. Classify the flag:
   • blank_patient_id or blank_assigning_authority → Mirth channel is not populating the field. Raise with the interface analyst for that sending facility — they need to fix the CCD extraction mapping.
   • invalid_chars_in_authority or invalid_chars_in_id → The facility is sending malformed identifiers. Raise with the interface analyst — may need a transformation rule in the Mirth channel.
   • unlikely_id_pattern → Could be a placeholder or test value. Raise with the interface analyst to confirm whether the value is intentional or a channel bug.
   • blocked_type:ssn → Expected; no action unless the SSN allowlist decision changes.
3. Interface analyst investigates the sending facility's CCD structure and Mirth channel mapping, corrects the extraction, and notifies the reviewer.
4. Mark as understood — once an issue is investigated and either fixed or accepted as expected behavior, add a note in the team's tracking system. The log file itself is append-only; no DB update needed.
5. If a new edge case is found that the current rules don't cover — a new class of malformed ID — add it to the UNLIKELY_ID_PATTERNS list in extraction.py and document it in this table.

Log file — no new DB table needed

Flagged IDs are written as JSON lines to a log file. One line per skipped row, containing everything needed to investigate and route to the right interface analyst.

# Log location
/opt/tools/mpi_and_emit_v2/enrichment/logs/id_quality_flags.log

# Example line
{"ts":"2026-09-25T14:32:01","msg_id":672018724,"sending_facility":"ORG_ALPHA",
 "assigning_authority":"ORG_BETA","id_type":"mr","patient_id":"???001",
 "flag_reason":"invalid_chars_in_id"}

3-month rolling retention via logrotate — add to /etc/logrotate.d/id_quality_flags:

/opt/tools/mpi_and_emit_v2/enrichment/logs/id_quality_flags.log {
    daily
    rotate 90
    compress
    missingok
    notifempty
    copytruncate
}

Maintenance ownership — to be assigned

Feedback & Alerting

Two validation points catch data quality issues before they reach the MPI call. Both write to a single structured log file and send a daily email summary if issues were detected. Nothing is blocked — messages continue processing; only the alert fires.

Architecture

① patient_id validation rules

Runs in extract_patient_ids() in extraction.py for every row from reliance_patient_ids. Rows that fail are skipped — not passed to the MPI payload — and written to the alert log.

② facility_set_org validation rules

Runs in supplemental.py after fetch_facility_set_org(). An invalid value is set to None so it never reaches extract_sending_org() or the MPI call.

Alert log format

One JSON line per flagged row, written to enrichment/logs/id_quality_flags<YYYYMMDD>.log. Channel info is looked up from reliance_msg_changelog by msg_id.

# patient_id issue
{"ts":"2026-10-10T03:36:59+00:00","msg_id":672031651,
 "channel_id":"49f72119-4f3e-42a8-8d6b-904fa6505006",
 "channel_name":"OCHIN - HL7v2 - 30009","mirth_instance":"zen_gemini",
 "assigning_authority":"Mosaic Medical","id_type":"mr",
 "patient_id":"???001","flag_reason":"invalid_chars_in_id"}

# facility_set_org issue
{"ts":"2026-10-10T03:36:59+00:00","msg_id":672031652,
 "channel_id":"49f72119-4f3e-42a8-8d6b-904fa6505006",
 "channel_name":"OCHIN - HL7v2 - 30009","mirth_instance":"zen_gemini",
 "failure_point":"facility_set_org","value":"123456",
 "flag_reason":"digits_only_facility_set_org"}

Log rotation

Handled entirely in extraction.py via _DailyFlagHandler — no logrotate config needed.

Daily email alert

Run by cron at 5pm via enrichment/filter/run_notify.py. Only sends if issues were flagged that day — no email on clean days.

# Cron entry
0 17 * * * root /opt/tools/miniconda3/envs/python3_12/bin/python \
    /opt/tools/mpi_and_emit_v2/enrichment/filter/run_notify.py \
    >> /opt/tools/mpi_and_emit_v2/enrichment/logs/notify.log 2>&1

# Manual dry run
cd /opt/tools/mpi_and_emit_v2
python -m enrichment.filter.run_notify --dry-run

# Check a specific date
python -m enrichment.filter.run_notify --date 2026-10-09

Recipients are configured in server_config/server_config.py (stage only — not promoted to production):

ENHANCEMENT_ALERT_RECIPIENTS = [
    "interface-analyst@reliancehie.org",
    "data-ops@reliancehie.org",
]

Feedback loop — flag to fix

1. Email arrives at 5pm with a per-channel summary table and the full log attached. The channel_name in each row identifies exactly which Mirth channel to investigate.
2. Interface analyst opens the log and finds the flag_reason for their channel:
   • blank_patient_id / blank_assigning_authority → Mirth channel not populating the field — fix the CCD extraction mapping
   • invalid_chars_in_* → Channel sending malformed value — add a transformation rule in Mirth
   • digits_only_facility_set_org / patient_ids_present_but_no_facility_set_org → reliance_msg_facility_set_org mapping broken — fix the channel's org assignment
   • ssn_shaped_fmrn → Channel sending SSN values under an MRN type code — fix the type mapping
3. Interface analyst fixes the Mirth channel and notifies the reviewer.
4. Reviewer confirms the next day's log no longer contains that channel/reason combination.
5. New edge case found — add a new pattern to _UNLIKELY_ID_PATTERNS in extraction.py and document it in this table.

Files added by this feature

Match Quality Report — Weekly Review

Generated weekly. Example below uses prototype data — real output queries reliance_match_reconciliation and reliance_persona_quality_flags. Each chart answers one decision question.

Both reports are generated by generate_match_quality_report.py and written as self-contained HTML files to enrichment/logs/reports/. The longitudinal report link above is a prototype — the generated file uses live DB data.

How to generate

cd /opt/tools/mpi_and_emit_v2
python -m enrichment.generate_match_quality_report \
    --start 2026-10-03 --end 2026-10-10 \
    --output enrichment/logs/report_20261010.txt

Example report printout

Multi-ID Match Quality Report — 2026-10-03 to 2026-10-10
Generated: 2026-10-10 17:00:02
═══════════════════════════════════════════════════════

MATCH RATE
  Messages processed       :  14,310
  Matched (PI assigned)    :   99.8%  ✓  baseline 99.8%
  Limbo rate               :    0.2%  ✓  baseline  0.2%
  Missing rule_fired       :    0.0%  ✓

ENRICHMENT SIGNAL
  IDs pushed / available   :   98.2%  ✓  target ≥ 95%
  FMRN utilized            :   22.4%  ✓  target ≥ 15%
  Identifier rules (2+7)   :   25.5%  ✓  baseline 2.9%

SPLIT SIGNALS
  persona_shrank rate      :    0.3%  ✓  baseline 0.3%
  mpid_reassigned rate     :    0.01% ✓  baseline 0.01%
  Distinct MPIDs this week :  48,210  ↓  from 48,440 (consolidating ✓)

MERGE SIGNALS
  multi_ssn_persona rate   :    0.05% ✓  baseline 0.05%
  Rule 37 volume           :    3.1%  ↓  from 4.2% ✓

PIPELINE SPLIT
  CCD — identifier rule %  :   28.3%
  HL7 — identifier rule %  :    6.1%  ⚠  lower — HL7 FMRN gap known

OVERALL STATUS: ✓ ALL TARGETS MET — no action required

Gate query — run daily

SELECT DATE(processed_at) AS day, COUNT(*) AS messages,
       ROUND(100*SUM(pi IS NOT NULL)/COUNT(*),2)                          AS matched_pct,
       ROUND(100*SUM(limbo_reason IS NOT NULL)/COUNT(*),2)                AS limbo_pct,
       ROUND(100*SUM(rule_fired IN (2,7))/COUNT(*),2)                     AS identifier_rule_pct,
       ROUND(100*SUM(fmrn_utilized=1)/COUNT(*),2)                         AS fmrn_utilized_pct,
       ROUND(100*SUM(ids_pushed)/NULLIF(SUM(ids_available),0),1)          AS ids_pushed_pct,
       ROUND(100*SUM(mpid_reassigned=1)/COUNT(*),3)                       AS reassigned_pct,
       ROUND(100*SUM(rule_fired=37)/COUNT(*),2)                           AS rule37_pct
FROM warehouse.reliance_match_reconciliation
WHERE processed_at >= CURDATE() - INTERVAL 14 DAY
GROUP BY DATE(processed_at) ORDER BY day;