Extending Manticore-Search DB for auto-scaling and replication on AWS

Soon after adding Manticore-search to enable search suggestion (read here for the full story) in less than 50 milli seconds, the architecture was ready for adulthood - handling scale. Since this was an educational portal, the nature of the usage was such that there would be a sudden burst of requests and then sudden contraction. This required us to enable auto-scaling to the architecture. The horizontal scaling automatically brought in the need for replication. In all, what we needed was a reliable, scalable and maintainable architecture.

The over arching guidance for the architecture were the following:

• Use AWS and its components to the fullest extent - for 'ease' of maintenance (availability of less-than expert operations team was the easier)

• Master-Master - to read and write on any node providing distributed workload processing

• Virtually Synchronous Replication ensuring minimal lag and zero data loss in the event of a node crash; no data inconsistencies or divergences

• Automatic node provisioning - both scaling in and scaling out

• Ease of deployment

Replication

Manticore has this Manticore Replication cluster based on the Galera Cluser popular in replicating MySQL. The salient points of this replication are:

• Each node runs the Manticore DB replicated using Manticore Replication Cluster running on each node (unlike the representation in the diagram above)

• Runs in a master-master mode

Auto Scaling

The above architecture runs on AWS and extensively uses the AWS components for the purpose except for the Manticore Replication Cluster itself. The EC2 instances are provisioned with the auto-scaling functionality of AWS Elastic Container Service with the minimum node configuration being 1. The AWS ECS takes care of provisioning new instances or removing unwanted instances based on the resource utilisation. AWS ECR calls it the scaling in and scaling out operation of the auto scaling group, which is essentially a capacity provider.

When a scale-out operation occurs, a new ECS node is provisioned and the (Docker) container image is pulled from ECR into the new node to run.

The AWS Network Load Balancer distributes the load among the nodes provisioned - which is called the target group in AWS parlance. The load balancer is the single point of contact for the front end application which is oblivious to the presence of multiple EC2 nodes.

The paraphernalia

When ECS decides that a EC2 node needs to be added to the cluster, an event is raised via the Event Bridge which in turn triggers a Lambda function which informs the replication cluster by downloading the credentials from the parameter store. The notification service from AWS is also used to keep the operations informed when scale in and out occurs.

Conclusion

Manticore Replication Cluster with AWS' ECS auto-scaling can be combined to form a sturdy architecture that auto-scales as well as replicates without data loss. What in the process might look easy but not very easy is the configuration of the various components which can be mind boggling. Never ever try to do it without infrastructure provisioning service like AWS CloudFormation which we used in here; although it was a pain to build the CloudFormation template, once done the operational folks were able to make multiple deployments with relative ease.

Let us know your experience in scaling Manticore-search with reliability. Is this architecture of ours a overkill or this is what it is?