FHIR Bulk Export Workflow

Overview

The FHIR Bulk Export Service is intended to fulfill the 21st Century Cures bulk data export requirements. This feature leverages FHIR R4 APIs certified under the 2015 Edition CEHRT (g)(10) criterion. The FHIR Bulk Export service enables API consumers to make asynchronous requests to export USCDI (United States Core Data for Interoperability) clinical data for all patients in a particular athena context.

With FHIR Bulk Export, you can retrieve data for multiple patients or resources at a time. Potential use cases to consider for the FHIR Bulk Export service are:

  • One-time initial load of data for seeding system such as data warehouse 

  • Monthly data capture synchronizations 

  • Monthly export of a few specific resources (using _type parameter) 

  • Population health data transfer 

We do not recommend using FHIR Bulk Export for real-time access needs, daily data synchronizations, or scenarios when a single patient’s health information needs to be obtained. 

Access and Permission

Our FHIR APIs are gated using OAuth scopes, which can be requested in Developer Console. Your application will also need to be granted access to target specific athena practice(s) for data exports, either requested by the athenahealth customer you are working with or follow this process to request context access.

The following authorization scopes and associated resources are supported in our implementation of FHIR Bulk Export:

  • system/AllergyIntolerance.read
  • system/CarePlan.read
  • system/CareTeam.read
  • system/Condition.read
  • system/Device.read
  • system/DiagnosticReport.read
  • system/DocumentReference.read
  • system/Encounter.read
  • system/Goal.read
  • system/Immunization.read
  • system/Location.read
  • system/Medication.read
  • system/MedicationRequest.read
  • system/Observation.read
  • system/Organization.read
  • system/Patient.read
  • system/Practitioner.read
  • system/Procedure.read
  • System/Provenance.read

Glossary

  • Client – FHIR client application making API requests  
  • Practice – athenaOne context 
  • Targeted Practice – athenaOne context being targeted for data export   
  • Job – An export request made by a FHIR client app 
  • Application - number of jobs a FHIR client application can run at once 
  • Resource – type of JSON web resource defined in FHIR specification   
  • Organization – a resource that we use to represent a practice or department   
  • Group – a resource representing a group of people. Currently this has a 1-1 relation to all patients within a given practice 

How to run Bulk Export

The high-level experience after authentication is, as defined by the FHIR Bulk Export specification and based on the FHIR Asynchronous Request Pattern:

1. Export Invocation - GET/fhir/r4/Group/{logicalId}/$export

  • FHIR client makes an http GET request to initiate an export job (refer here for more details on Group-level export documentation).

  • Specify the following HTTP header values in your export request:

    • Prefer: respond-async

    • Accept: application/fhir+json

  • A polling URL containing the jobId is returned in the response Location header.

2. Export Polling - GET/fhir/r4/$export-poll-status/{jobId}

  • FHIR client makes an http GET request for polling the job status.

  • Polling URL returns progress indicator in response header until the job is completed.

    • 'X-Progress' which shows the percentage completed

    • 'Retry-After' which indicates how long to wait before polling again 

  • FHIR client continues polling the job status URL until the export job is 100% completed.

  • FHIR client downloads exported files using the Pre-Signed URLs returned in the response body of the polling URL (see example response body here).

3. (optional) Export Cancellation - DELETE/fhir/r4/$export-poll-status/{jobId}

  • FHIR client can cancel the export job at any time by making a DELETE request to the same polling location URL.

A more detailed sequence diagram of the workflow is also provided by HL7 here.

Additional Notes

  • Exported files are hosted in a protected AWS S3 bucket in .ndjson format.

  • Exported data will be available for download for the lesser of: (1) 30 days from the completion time of the bulk export job; or (2) 60 days from the request that initiated the bulk export.

  • Each file will be deleted after 60 days post creation.

  • The URLs returned in the job completion response manifest are AWS S3 Pre-Signed URLs. These URLs are valid for a period of 1 hour (60 minutes) after manifest retrieval.

  • If the pre-signed URL for a resource expires before all data is retrieved or the client needs to re-download a file, FHIR client can request a new pre-signed URL by re-requesting the job completion response. The filename of the resource file will be consistent across requests, so FHIR client can rely upon this to determine which files they have already downloaded.

  • When polling for job status, or canceling a job, FHIR client must have a valid auth token with the same client ID as the one used to initiate the export job.

  • athena's Group-level export targets Patient Compartment for resources required by USCDIv1. This means we export resources that are referenced by a resource within the patient compartment and excludes resources with no data available on the patient record.

Partial Failures

  • In the case of partial failure, the bulk export job completion results will contain an error file of OperationOutcome resources which include diagnostic strings that enumerate the list of patients which have failed in each task or the resources that failed for a patient.

Bulk Export Concurrency Limits & Throttling

Limit

Environment

Job Limit

Throttle (Retry-After)

Application

Preview

10

1 hour

Production

10

1 hour

Targeted Practice per Client App

Preview

2

1 hour

Production

2

1 hour

Service Denials / Throttling Behavior

For service denials or throttling behavior, the bulk export service will return a response with an error status code and a Retry-After header. Capacity throttling resulting in a 429 response will happen when the FHIR client application is hitting concurrency the limitations mentioned above. Availability throttling resulting in a 503 response may happen if there is a widespread system failure or if the bulk export service is experiencing failures itself, in which case the resulting OperationOutcome issue diagnostics will contain the message "Service is unavailable. Please try again later." instead of the message "Too many active exports. Please try again later" like you see for Capacity throttling.

Potential Exception Cases

The following are potential exceptions you may encounter during the different bulk export operations.

Export Invocation - GET/fhir/r4/Group/{logicalId}/$export

  • Accept Header not application/fhir+json => Status 406

  • Prefer header not respond-async => Status 406

  • Service is currently throttled/unavailable => Status 503

  • Client is not authenticated with correct OAuth scopes => Status 401

  • Client is not authorized for target group => Status 401

  • Client passes in an invalid group id => Status 404

  • Client is running too many concurrent jobs => Status 429

  • Server-side exceptions => Status 500

Export Polling - GET/fhir/r4/$export-poll-status/{jobId}

  • Auth header doesn’t exist => Status 401

  • Invalid auth token => Status 401

  • Client not authorized for job poll or job not found => Status 401 

  • Server-side exceptions => Status 500

  • Job has failed => Status 500

Export Cancellation - DELETE/fhir/r4/$export-poll-status/{jobId}

  • Auth header doesn’t exist => Status 401

  • Invalid auth token => Status 401

  • Client not authorized for job poll or job not found => Status 401 

  • Server-side exceptions => Status 500

  • Requests for unauthorized polling of jobs (i.e., polling jobs initiated by other client applications) => Status

Was this information helpful? Yes | No Thank you for your feedback! What went wrong? Incomplete or incorrect information | Irrelevant Content | Others
Submit

On this Page