Install AWS CLI Client

Please refer to the official documentation here to install the AWS CLI Client for your operating system.

AWS database high availability

This post will take you through the important concepts regarding AWS database high availability. It explains many concepts such as Amazon RDS, DB Instance, Availability zone, Region concepts, high availability, and Failover Process.

Amazon RDS

Amazon RDS stands for Amazon Relational Database Service. It is a service provided by AWS to create and operate on databases on the cloud very easily. Amazon RDS lets you collaborate with different database systems like MariaDB, SQL Server, MySQL, PostgreSQL, and Oracle. It provides backup of all databases. Managing databases and managing its administrative task is very difficult, but AWS Database provides you with a very user-friendly and easy-to-use service for managing and doing all the administrative work of database on the cloud.

DB Instance

It is the database instance of AWS DB which is created on the cloud. You can create, delete, redefine and modify database instance in AWS. It provides handling for database security on the cloud.

Availability Zone and Region Concepts

Amazon Web Service creates different Regions which are completely independent. Inside the Region, different Availability zones will be connected with each other.

So all the regions will be completely isolated from the other regions. For Region, Amazon EC2 Regions will be used. Two regions will also be able to communicate in AWS but that will use the public Internet. So for the communication part, necessary encryption methods are used to pass messages.

When you create any instance, you can choose its Availability zone. If you don’t choose an Availability zone, one Availability zone will be selected automatically. You can mask any failure of a database instance in your Availability zone by using the Elastic IP address. Availability zone is denoted by different region code.

High Availability

It is also known as Multi-AZ. Amazon used Multi-AZ deployment for MySQL, Oracle, MariaDB, and PostgreSQL. Multi-AZ deployment can be established using AWS CLI. In Multi-AZ deployment, the primary DB instance will be replicated with other availability zones to avoid I/O freezes.

Failover Process in Amazon RDS

If Amazon-AZ is available during the failover process, Amazon RDS automatically creates a replica of the Availability Zone. The failover time is between 60 to 120 seconds, but if the database transaction or recovery process is very lengthy then failover process time can be increased.

Amazon RDS automatically handles the failover process so you can resume back your database operation as soon as possible without any more administrative tasks.

There are different ways to check if your Multi-AZ DB has failed or not. You can create a configuration so if any failover occurs, its notifications will be sent by email or SMS. You can check your database even by the Amazon RDS APIs or console. You can also take a look at your current state of Multi-AZ deployment using Amazon RDS APIs and console.

The primary database instance will be switched automatically to replica if any of the below conditions are met:

  • If database instance server type is modified.
  • If manual failover of database instance is initiated by reboot with failover.
  • If the operating system of the database instance is undergoing software patching.
  • If your primary database instance is going to fail.

This post explained various important concepts regarding AWS database high availability. If you want to know more about it, have a look at our video tutorials.

Apache Mesosphere and DC/OS – Introduction to containerization at hyper scale

A concept that sits at the heart of Apache Mesos, Containerization is a virtualization method for the deployment & running of distributed applications without bringing VMs into the equation. Now, where does Apache Mesos fit into all this?

Consider that you’ve deployed some containers in your data center, including Analytics, Web Application, Software Networking, etc. Now, if you wish to deploy your web apps by integrating these containers, the first thing you’ll require will be selecting a subset of nodes for your application’s runtime environment. Also, there will be other details to take care of like virtual or physical locations for deployment.

However, you can automate these steps by scripting them out. You’ll require the details of all the resources you’re employing including the computers, their ports, DNS addresses and so on. The end product of all these operations would be a Statically-Partitioned Sub-Cluster. Now, suppose the need for another deployment arises. The only way to do so, while following the legacy topology would be to repeat all the steps mentioned above, which as it may seem brings redundancy & inefficiency into the equation.

It will be worse if your Web App becomes really popular. To fulfill the increased demand, you’ll have to shut down the existing system, bring disruption to the users and need to put resources into jobs that could’ve been rescaled.

Development times could be cut short, wasteful spending could be avoided, disruption time can be brought down and most importantly more efficient distribution of resources could take place if a new solution can be put to use. The solution comes in the form of Apache Mesos.

What is Apache Mesos?

Running Docker containers in a data center isn’t as easy as it seems when it comes to huge scale deployment where proper distribution of resources is a priority. An excellent way to do so would be to make the clusters treat the containers like CPU cores in a personal computer. Enter Apache Mesos!

Apache Mesos is a fault-tolerant cluster manager that used a centralized approach to the allocation of resources and their subsequent management. Mesos joins up the physical resources and presents them as one big unit that can then be scheduled across various clusters, similar to how the Linux Kernel works.

Developed by the Apache Software Foundation at the University of California, Berkley, the software is Cross Platform and has a stable release dated November 3rd, 2016.

Apache Mesos is primarily built for hyper-scalability. Its ability to scale to thousands to tens of thousands of nodes has made it a top-level open source project and is the driver for its popularity in companies like Microsoft, Twitter and Ebay when it comes to management of their data centers.

Also, Mesos is language independent and supports several development languages like C++, Java & Python.

DC/OS Mesosphere

Based on the Apache Mesos distribution systems kernel comes an operating system, i.e. DC/OS. The OS enables the visualization & management of several machines as one unit, automating several tasks such as process placement, resource management, inter-process communications, etc. The OS has a web interface as well as a CLI for remote administration tasks.

Notable to say, DC/OS is the only open source project that brings all these features under one roof.

Docker & Mesos go hand in hand because of their synergetic approach to pushing a container into production, making the entire process super-easy for developers.

DC/OS provides a level of abstraction between the scheduler and the machines where these tasks are to be executed. This essentially means that it is up to the OS to distribute resources accordingly, eliminating the need for a scheduler to deal with the tasks. Thus, static partitioning has been shut down.

How a distributed system is designed?

Two different sets of machines are implemented:

  • Coordinator machine: assigns tasks to workers
  • Workers machine: executes assigned tasks.

Mesos provides a level of abstraction b/w the scheduler & the machines, so in effect Mesos sits in between them. This provides the immediate benefit of running multiple distributed systems on the same cluster of machines without hogging down any resources or stealing any system’s share of resources.

DC/OS Features

Apache Mesos is made up of a set of masters and a set of workers, working in conjunction with a framework that runs in accordance with Mesos API, e.g. Hadoop. Whenever the framework wants to run a task on the Mesos cluster, a connection is made to the masters which trigger a distribution of resources.

To sum it up, DC/OS packs the following features:

  • High resource utilization
  • Mixed workload colocation
  • Container orchestration
  • Extensible resource isolation
  • Stateful storage support
  • Zero downtime upgrades
  • High availability & elastic stability
  • Web & Command Line Interface
  • Real-time interaction
  • Integration-tested components
  • Service discovery and distributed load balancing
  • And much more…

It all starts off with the Mesos master, which has a list of all the slaves as well as their specifications in one place. For instance, there may be 10 slaves with 4x CPUs & 4GB RAM each. These resources are presented to a framework scheduler, and whenever the need for task execution arises, the task is launched and handed over to the Mesos Master.

The Master handles the task to the slave according to resources from where the executor will take over. Meanwhile, the status of operations is sent back up, from the Master to the Scheduler. On the basis, of this information, a new task may be started or the current one may be killed or halted.

DC/OS Architecture

As mentioned before, DC/OS is a distributed operating system that sits between the resources of machines & provides services to apps. The services include service discovery, package management and running processes across several nodes.

The architecture can essentially be split into 3 parts:

  • User Space
  • Kernel Space
  • Hardware

The User Space consists of components like Distributed DNS proxy, Mesos DNS as well as services like Spark & Marathon. In addition, it spans systemd services like Chronos or Kafka.

Consider the DC/OS kernel a magnified & glorified version of the Linux kernel. The kernel comprises of:

  • Mesos Master: the process orchestrates tasks that are later-on run by Mesos agents. The process receives reports from various agents and allocates resources to each DC/OS service in need of it.
  • Mesos Agents: runs discrete Mesos tasks on behalf of the entire framework. Private agent nodes are employed to run the apps & services while public agent nodes compute the DC/OS apps in a publicly accessible network. The Mesos-Slave process also packs the ability to invoke an Executor for launching tasks via containerizers.

The Kernel Space in DC/OS is responsible for managing resource allocation & for performing two level scheduling across the clusters

Finally, the Hardware may be Amazon web services, Open stack, or any physical or virtual hardware.

A general way to look at all these processes is as follows:

  1. Client/scheduled initializes itself.
  2. Mesos master sends resources offers to scheduler.
  3. The scheduler declines resource offers as long as processes haven’t been initiated from the client side.
  4. Client proceeds with the launch.
  5. Mesos master sends a resource offer, which if matched is accepted and sent along with a “launchTask” request to the master.
  6. The Mesos agents are directed by the master.
  7. The executor reports the status of the tasks to the agents which report it to the master, from where they are sent to the scheduler.
  8. The scheduler informs the client.

If you like to learn more about this popular platform – check out our course on Docker, Apache Mesos and DC/OS here:

Introduction to OpenStack – Open Source IaaS Cloud Platform 101

Started as a collaboration project between Rackspace Hosting and the US Space Agency NASA, OpenStack has swiftly grown into one of the world’s biggest open source technologies delivering a flexible, efficient, scalable and cost-effective OS based on the cutting edge concept, i.e. “The Cloud”.

Work began in 2010 when the concept was still in its early stages, but the team was able to launch its first stable build in just four months, naming it Austin. At that time the main purpose of the initiative was to bring mainstream hardware within the folds of cloud computing. With the passage of time, OpenStack grew as an Infrastructure as a Service (IaaS) and the platform soon incorporated modules that let it control a variety of hardware components including those able to process, store or communicate with other entities.

Basically, OpenStack is a set of tools that support large-scale virtualization and allow for the creation & management of virtual machines through secure, easily accessible GUI.

Owing to the “Open-Source” label, the software was welcomed with open arms by the Linux community, and today OpenStack has been accepted by several companies thanks to its robust features-list:

  • 2011 – 2012: Ubuntu, Debian, SUSE, Red Hat, etc. come in
  • 2013: Oracle joins as a Sponsor, planned for Oracle Solaris
  • 2014: HP Helion Cloud Computing Solutions to be based on OpenStack

The latest & most stable version of OpenStack is the Newton released on 6th October, 2016 while Ocata is still in the pipeline.

Its 2017, and today the adoption of Cloud Computing is in full swing. Not-for-profit organizations, corporations, enterprises and even small startups, all are busy shifting over from private data-centers to the public, private or hybrid cloud infrastructure. The cloud landscape is rapidly changing and it is also true that the cloud technologies are just in their infancy. So there is tremendous amount of room for improvement.

This room for improvement can only be filled by skilled professionals, who are well experienced with these cloud technologies. The job market has seen a steady rise in the demand for these professionals, and this has resulted in a pay rise for this particular field. In the US for instance, the median income for professionals fluent in OpenStack is $120,000 – $140,000 a year!

The OpenStack project isn’t composed of a single, large program that offers all the features one is looking for. Instead, think of it as a platform consisting of several projects or services, all designed in parallel, aimed for specific purposes. Each “project” offers core features, unique to its own applications. As this is an Open Source project, experts from around the world can collaboratively contribute to its development.

Each individual service can then be accessed using its own API and modules be called to accomplish tasks on hand. Take a look at the individual services/projects and their purpose below:

1. The Identity service, code named Keystone:

The main purpose of OpenStack Identity Management is to create & provide management tools for users & their respective services. It acts as a central authentication mechanism for all OpenStack components and integrates itself with several directory services like the Lightweight Directory Access Protocol to facilitate multiple login possibilities.

Just like “Computer Management” and “Group Policy” in Windows’ OS, Keystone allows administrators to configure policies for user groups across systems and implement them with a single click. The entire system is controlled by a well-designed and easy-to-use GUI which makes managing the OpenStack system very straight forward.

2. The Compute service, code named Nova:

The Compute service is a very vital one, controlling the very core fabric of the platform, cloud computing. The service has been written in Python and provides an abstraction layer. This layer then virtualizes resources such as processing power, RAM, storage and network services, along with functions that greatly improve automation & utilization.

Examples of management functions include ones that can launch, suspend, resize, stop and reboot resources using hypervisors. An API, Application Programming Interface can be used to store & manage files programmatically while the image is running.

3. The Image service, code named Glance:

This service provides support for Virtual Machine images, especially system disks that can then be used to launch instances of Virtual Machines. In addition to the services, discovery, registration & activation, the project has functionalities to provide backups & snaps. The images are highly robust and can roll out new servers on the fly.

4. The Dashboard service, code named Horizon:

Horizon is basically a GUI project/service that allows users to interact with other OpenStack services like Nova, Cinder, etc. The entire interface is web-based and is well-versed in the art of effectively controlling & monitoring each service.

5. The Object Storage service, code named Swift:

This storage service is actually redundant, but an excellent choice for scale-out storage. Based on Rackspace Cloud Files, Swift ensures collection & distribution of data across all devices currently in the pool so that the users can make the best use of their hard drive resources. In the event of failure of a component, OpenStack automatically restores content to newer cluster members.

6. The Block Storage service, code named Cinder:

The service has been created to manage block-level storage systems that compute use of instances. The storage system is very much necessary & effective in places where performance constraints have to be strictly maintained, e.g. databases. The Linux server storage is the most common storage that employs Cinder, however other plugins exist as well like NetApp, Ceph and SolidFire. The system has an excellent interface to attach, create and detach devices to & from servers.

7. The Network service, code named Neutron:

Previously called Quantum, Neutron is a Networking Service that has a powerful set of tools, allowing it to control a range of network resources like LANs, dynamic host configuration protocol and IPv6. Users can easily define subnets, networks and routers, allocate IPs and other specifications, after which the entire system comes onboard. Users can assign floating IP addresses that allow users to assign fixed IPs to virtual machines.

8. The Orchestration service, code named Heat:

The entire mission of the OpenStack orchestration program is to create a human & machine accessible project that can efficiently manage the entire lifecycle of all the applications that lie within OpenStack clouds. Heat is the practical implementation for this and implements an engine across several composite cloud apps. The orchestration is based on templates that come in the form of text files, which may be treated as code.

9. The Metering service, code named Ceilometer:

The Ceilometer project is one of the most promising and actively developed project, very well-suited for controlling & monitoring the OpenStack infrastructure. The salient features of this service include:

  • Efficient collection of metering data,
  • Configuring the type of data collected so that operating requirements are met,
  • Collecting data by monitoring notifications from other services,
  • Using the REST API for accessing & inserting data,
  • Producing metering messages,

These were the nine blocks of OpenStack that form the current architecture. In time, more promising elements will also get added to the platform, making it even more resourceful for cloud computing.

If you like to learn more about this popular platform – check out our course on OpenStack here: