Using serverless for cross-organizations information exchange in genomic analysis
Authored by Yudho Ahmad Diponegoro (Sr. Solutions Architect at AWS) and Ankur Sharma, PhD (MiRXES)
In today’s world, genomic sequencing and analysis can play an important role in discovering novel ways of improving health and life sciences. The computational intensity of these workloads necessitates scalable, cost-efficient cloud infrastructure. MiRXES , a Singapore-headquartered RNA technology company, leveraged Amazon Web Services (AWS) to deliver scalable, cost-optimized compute and storage solutions for genomic analysis. This architecture enables MiRXES to dynamically scale resources based on project-driven, variable workloads, ideal for genomic analysis.
Genomics workload often involve multiple parties, including the institution who holds the DNA samples and the company/institution who performs the computation for genomic sequencing and analysis. This blog post focuses on the serverless technologies such as Amazon Simple Notification Service , Amazon Simple Queue Service , AWS Lambda to facilitate notification exchanges between these parties. This post also briefly addresses how other services such as AWS Batch and AWS DataSync are used to perform the analysis and the data sharing respectively.
This blog post addresses a scenario where the client has these requirements:
- Genomic samples are to be sequenced and analyzed. Analysis result data is to be shared to the client through an automated data sharing mechanism.
- Once a sample analysis result is ready, the client should be programmatically notified with certain payload. Client is to send back a payload to acknowledge the data retrieval.
- Data can be purged after certain number of days. MiRXES must send a warning notification to the client few days before the scheduled deletion.
- Client can request to hold certain data where MiRXES should then extend the data availability period
The core architecture
Here is the core architecture depicting MiRXES infrastructure that consists of a sequencing hardware and a server on-premises connected with Amazon S3 in AWS cloud through AWS Direct Connect .
When the analysis result is ready, they are uploaded back by the Nextflow script to the S3 bucket. AWS DataSync is used to share the resulting data bundle with the client. Data is copied into the client’s S3 bucket.
The notification and raw data purging architecture
It is important to notify the client once the data bundle is ready. This is handled by multiple serverless components as shown in the diagram below. This section will also address the data purging architecture to save cost by deleting the data after certain period.
The Lambda function prepares the manifest of the sample files and stores it in S3 bucket. The manifest file contains data bundle details along with their location in S3, which the Lambda function constructs by listing the objects in S3 for the particular sample. This manifest file is needed to inform the processor on the client side about the file structure of the data bundle.
Upon constructing and uploading the manifest file, the Lambda function sends a notification to an SNS topic with payload which specifies the sample detail and location of the manifest file in S3.
An SQS queue on the client’s AWS account subscribes to this SNS topic using cross-account subscription . The queue will receive the notification payload. It will fetch the manifest file using the location information in the payload and use the manifest file to locate and fetch the data bundle for that sample. Since the actual data bundle is already copied from MiRXES’ S3 bucket into the client’s S3 bucket with AWS DataSync, the client’s notification processor can use the location defined in the manifest file to locate the actual copy of the intended data in client’s S3 bucket. The process will trigger further downstream processes.
The client will also send a notification back to MiRXES that the files for the sample are successfully received. The notification will be sent to the SQS queue in MiRXES’ AWS account as shown in the diagram below.
A Lambda function will poll the SQS queue and put an item into table in Amazon DynamoDB , a serverless, NoSQL, and fully managed database. The item will have information on the sample ID, the action to take (delete or archive), the scheduled data action date, the scheduled data action warning date, and other relevant information.
Initially, the item will have the “action” attribute to “delete” which means that the sample data is set to be deleted at the scheduled time. The “action_done” flag is set to false. So is the “warning_done” flag. These indicates that the deletion and the warning message sending have not yet happened.
A daily schedule is set with EventBridge to invoke another Lambda function. This function will scan the items in DynamoDB to find items which are getting closer to the deletion date and send the warning message payload to SNS topic accordingly. This message will be received by SQS queue on the client’s side. The function will also update the “warning_done” attribute of the item in the database to true so that the warning won’t be sent again. This is shown in the diagram below.
The Lambda function will also scan for items which require the deletion. It will delete the corresponding files for that genomic sample in the S3 bucket accordingly. For items whose “action” attribute are set to “archive” it will archive them instead. The archival can be implemented differently. For example, one could set the Lambda function to move the files from the S3 Standard storage class into the Amazon S3 Glacier storage classes with lower storage cost. For MiRXES, the objects are moved into the S3 Standard IA class (Infrequent Access). Read more about the S3 storage classes .
Since the “action” attribute for each item is always “delete” by default, there is a process which can cause it to change to “archive” when requested. This process is described in the diagram below.
The full architecture is shown in the diagram below. This, however, may not include some detailed components which are omitted on purpose for clarity and simplicity.
In addition to these components, the notifications are also logged in S3 bucket on MiRXES side to keep records on these notification exchanges. When the need arises, such as during audit or troubleshooting, MiRXES can refer to the notification records.
Conclusion
In this blog post we learnt how serverless architecture can be implemented to facilitate the notification exchanges between a service provider and the customer/client. More specifically, we learnt on how MiRXES implemented the information exchanges with the client in the genomics sequencing and analysis domain. We also learnt the core architecture where AWS Batch can be used along with EC2 Spot Instances and Nextflow.
With this architecture, both the provider and client could automate the data and information exchanges during a genomic project to reduce human error, increase operational efficiency, and improve observability and auditing.
Learn more about serverless on AWS and AWS for Healthcare & Life Sciences . Thank you for reading this blog post.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article