THE CHALLENGE
IAS had decided to gradually migrate their full infrastructure to AWS, including a solution which helps customers track and optimize ad campaigns across publishers and partners such as Facebook. Core to this technology is the ability to collect, process, aggregate, and analyze ad impressions, clicks, views, and other events related to individual ad sessions.
Over the years, their on-premise solution for ingesting ad events had become unwieldy to support and difficult to scale. The solution was not set up to work in the cloud. It needed to be refactored to simplify code management, take advantage of AWS native services, and perform optimally in the cloud. They needed coding and migration support from an AWS data migration veteran like EveryIT.
THE SOLUTION
In less than six months, EveryIT updated and migrated the IAS solution to AWS. Prior to the migration, EveryIT refactored the solution’s entire codebase from top to bottom and implemented a cloud-based ad event ingestion solution to support multiple partners, minimize operational costs and coding, and scale dynamically to accommodate sudden volume spikes.
To streamline the code and enable scalability, EveryIT analyzed the logic for each partner, identifying shared code and pinpointing standard areas for partner-specific logic. This effort laid the foundation for a simpler, more transparent codebase that could be quickly scaled to new partners while being easier and more cost-effective to support in the long term. In addition, the refactoring also optimized the solution for the cloud and leveraging AWS capabilities.
HOW IT WORKS
The comprehensive solution makes the most of AWS functionality while reacting dynamically with speed. To achieve optimal performance, it separates the ad events pipeline into two pieces: a receiver at the front end and a deliverer at the back end.
For the front-end receiver service, a lightweight Node.js web-service behind Elastic Load Balancing (ELB) provides the HTTP API endpoints to which partners submit ad events. The application has a very short start-up time and scales automatically based on the incoming traffic volume. The application validates and converts partner-specific API calls to standard ad event records. It then passes these records to a background Kinesis Producer Library (KPL) process that buffers and pushes the records to a Kinesis Data Stream which smooths the shape of the incoming traffic, enabling downstream processes to work more efficiently. Amazon ELB and auto-scaling Amazon Elastic Container Service (ECS)/AWS Fargate for the pipeline front-end enable low operational costs, while still being a serverless, easy-to-maintain solution.
On the back end of the Kinesis Data Stream—the deliverer service—a Kinesis Client Library (KCL)-based application enriches ad event records with additional properties derived from external services, and stores the records in the Amazon S3/AWS Glue Catalog-based data lake for further processing. Amazon Cloud Watch collects and surfaces pipeline metrics for dashboard monitoring, and alerts administrators when pipeline errors exceed specific thresholds.
RESULTS
- Reduced operational costs: The streamlined solution in AWS requires fewer resources.
- Increased scalability: IAS can quickly and cost effectively scale to new partners.
- Improved flexibility & performance: IAS leverages the performance and advanced capabilities of AWS.
- Improved integration: The new solution integrates with AWS-based client infrastructure for greater collaboration.
TECHNOLOGIES USED
Elastic Load Balancing (ELB), Amazon Elastic Container Service (ECS), AWS Fargate, Amazon Kinesis, Kinesis Data Streams (KDS), Kinesis Producer Library (KPL), Kinesis Client Library (KCL), Amazon S3, AWS Glue, AWS Auto Scaling, Amazon CloudWatch, Amazon DynamoDB, AWS Cloud Development Kit (CDK), AWS CloudFormation
OUR EXPERIENCE
EveryIT has performed hundreds of data migrations for large enterprise customers. Our certified practitioners have hands-on, best-practice knowledge of all the major platforms. As an AWS Data Migration Competency Partner, EveryIT has invested in building deep proficiency on the AWS platform.