AWS Global Infrastructure

Region

Region is a physical location around the world where data centers are clustered together. Each AWS Region consists of multiple, isolated and physically separated Availability Zones within a geographic area.

Availability Zone

A group of logical data centers is called an availability zone. They are interconnected with high-bandwidth, low-latency networking, to provide low-latency networking between zones that is sufficient to accomplish synchronous replication (same time replication). Typically, 3 availability zone in one region.

Edge Location

Edge locations are connected to the AWS Regions through the AWS network across the globe. It cache copies of customer contents for faster delivery to users at any location.

Planning for Failure

• Storage - File stored in AWS S3, is redundantly copied into every Availability Zone in that Region.
• Compute - It is a best practice to spread out computing resources across multiple Availability Zones to guarantee high availability.
• Database - Database can be configured for Multi-AZ deployment.

Shared Responsibility

• Customer are responsible for the security of everything that they create and put IN the AWS cloud. (Security IN the cloud)
• AWS manages the security OF the cloud, specifically the physical infrastructure that hosts customer resources. (Security OF the cloud)

AWS Well-Architected Framework

• Operational Excellence - run and monitor systems to deliver business value and continually improve supporting processes and procedures.
• Security - ability to protect information, systems, and assets while delivering business value through risk assessments and mitigation strategies.
• Reliability - ability to recover from disruptions, dynamically acquire on demand computing resources and mitigate disruptions.
• Performance Efficiency - ability to use computing resources efficiently to meet system requirements and to maintain that efficiency as demand changes and technologies evolve.
• Cost Optimization - ability to run systems to deliver business value at the lowest price point.

Total Cost of Ownership (TCO)

It is a financial metric that is used to estimate and compare direct and indirect costs of a product or a service. It typically includes the actual costs of the procurement, management, maintenance and decommissioning of hardware resources.

Instances

• On-Demand instance are ideally used for computing needs that are less than a year.
• Spot instances are used for computing needs that need to run for low costs and can be interrupted.
• Lambda instances do not exist. You can use AWS Lambda to process snippets of code, when a Lambda function is triggered.

Networking

Classless Inter-Domain Routing

CIDR, method to define a network’s IP range contained in it. In below example, the netmask, 24, first 24 (left to right) cannot change. Remaining 8 bits are flexible and can go from 0 to 255 providing 256 IP addresses in the network.

192.0.2.0/24

• Fixed IP address: All 32 bits are fixed. Use case: Set up a firewall rule and give access to a specific host.
• Internet CIDR block: All 32 bits are flexible. Use case. Setup a firewall rules to allow internet traffic.

192.0.2.0/32 # Fixed IP address

0.0.0.0/0 # Internet CIDR block

Subnet

Subnet or sub-network, is a network within a network. With this access control over subnets, we can decide which subnets in the networks can communicate with each other. With a subnet, it reduces the bandwidth on the routers and traffic can reach its destination in a shorter amount of time. IP address needs to travel through the router to reach IP in another subnet. Strategic placement of subnets and IPs within the subnet help to reduce network complexities and reduces network loads for more efficient traffic flow.

Planning a subnet

Note: IP addresses are required by load balancers, switches, routers, servers, workstations, printers, fax machines, mobile. Router and switches require multiple IP addresses.

• Public Subnet: Allows internet traffic that is routed through an internet gateway to reach the subnet.
• Private Subnet: Denies traffic to subnet that is routed from the public internet. Access to public internet from private subnet requires a NAT device.

Amazon VPC

Amazon VPC is a virtual networking environment that gives you full control over your virtual networking environment, which includes resource placement, connectivity, and security. When you launch VPC, choose region, then launch subnets in availability zones.

VPC IDs, string of random numbers and letters to identify VPC and tags, more identifiable name, are used to associate VPC components to the correct VPC. e.g. vpc-012kkadf232321234 (Development VPC)

Creating VPC, main route table, network ACL, and security group will be created automatically.

• AWS Direct Connect establish a dedicated, private network connection between your network and one of the Direct Connect locations to reduce network costs, increase bandwidth throughput, and provide a more consistent network experience than internet-based connections.
• VPC Endpoints provide connections between VPC and supported services.
  • Interface Endpoints AWS PrivateLink provides private connectivity between VPCs, AWS Services, and on-premises network without exposing traffic to the public internet.
  • Gateway Endpoints provide reliable connectivity to Amazon S3 and DynamoDB without requiring an internet gateway or a NAT device for VPC.
• VPC Peering is network connection between two VPCs that you can use to route traffic between them privately. Restriction: IP address ranges cannot overlap and transitive peering is not supported, must explicitly establish peering connection to be able to communicate.
• AWS Transit Gateway connects VPCs and on-premises networks through a central hub.

Route Table

Main Route Table directs traffic in VPC, each route specifies a destination, CIDR block or range of address where traffic is supposed to go and a target, the gateway, network interface, or connection through which to send the destination traffic. Each subnet in VPC must be associated with route table and follows many to one rules, many subnets can be associated with same route table.

Security Groups

Act as firewalls to allow only the traffic that you authorize and route the traffic to only the parts of your network that you permit. Instance level firewall, filters inbound and outbound traffic that is allowed to instances. Security groups are stateful.

By default, there is no inbound traffic rules but there’s outbound rule that allows all outbound traffic.

Network ACLs

Provides an additional level of stateless granular-level security only if required. Network ACLs act at the subnet level and control traffic. Each subnet in VPC must be associated with network ACL. Network ACLs can be associated with multiple subnets. The network administrator would need to update the network ACLs before that return traffic could pass out of the subnet.

VPC Architecture

Setting up AWS account, will automatically provide a default VPC in each AWS Region. Default VPCs have a public subnet with internet access for each Availability Zone in the Region such that EC2 instances can be immediately launched.

Service quota: 5 VPCs per region per account. Can be increased by raising a ticket.

Public access to Presentation tier

Restricted access to Presentation tier

Reserved subnet IP addresses

Note: Best practice to create larger subnets (256 IP addresses) instead of small subnets (65 IP addresses)

Network Gateway

A gateway that determines the traffic that will be given access to network is network gateway.

• Internet Gateway enables communication between VPC and the internet.
• Virtual Private Gateway enables to connect on-premise site to VPC also referred as site-to-site VPN connection.

Elastic IP

Elastic IP address replaces the default public IP address, you can mask the failure of an instance or software by rapidly remapping the address to another instance in VPC. It operates in Region level, can be used in any VPC in the region where it was created.

NAT Gateway

You can use a NAT gateway so that instances in a private subnet can connect to services outside your VPC but external services cannot initiate a connection with those instances.

Security

Layer of Security

• Perimeter Layer - security guards, fencing, security feeds, intrusion detection technology, and other security measures.
• Environmental Layer - environmental considerations from site selection and construction to operations and sustainability.
• Infrastructure Layer - backup power equipment, HVAC system and fire suppression equipment.
• Data Layer

IAM

AWS Identity and Access Management is an AWS service that lets you control user access to services in the AWS Cloud.

Note: Creating AWS account, you access your account as the AWS account root user, has complete access to all AWS services and resources in the account.

• IAM user - entities, represents a specific person or service that uses IAM users to interact with AWS, no permission by default
• IAM policies - an JSON documents that define specific actions that the entity cna perform on a specific AWS service or resource.
• IAM group - a group of IAM users that have the same permissions, IAM users in a group inherit the permissions that are granted in IAM policy. It cannot be nested, is unique, aren’t distinguished by case.
• IAM role - similar to user, a role doesn’t have any credentials associated with it, intended to be used by anyone who needs short term access.
  • Using roles has benefit of specified security credentials not being tied directly to an entity.
  • IAM role trust policy JSON document where principals that you trust to assume the role are defined.
• IAM Access Analyzer - identify the resources in your organization and accounts that are shared with the external entity, review its findings to determine whether the access is unintended or safe.
  • set or grant, detailed permissions
  • verify who can access what
  • refine by removing overly broad access

IAM Credential Types

• AWS Management Console provided with a username and password.
• Programmatic access provided with a set of access keys, programmatic calls to AWS using AWS CLI, SDKs, HTTPS.
  • comprised of an access key ID and a secret key.
  • user have two active access keys useful to rotate the user’s access keys or revoke permissions.

IAM Identity Center

YOu need IAM user’s credentials having necessary permissions to perform create, modify or delete EC2 instance. However, you need instance’s key pair to perform patch or install software and add or remove file from the instance. The key pairs doesn’t provide accountability for who is using the keys.

AWS IAM Identity Center (successor to AWS Single Sign-On) help you securely create and connect workforce identities and manage their access centrally across AWS accounts and applications.

IAM Policies

IAM Policies are formal statement of permissions.

• Effect required, specifies statement results is allow or an explicit deny. Explicit Deny takes the precedence.
• Action describes the specific action or actions that will be allowed or denied.
• Resource - specifies the object or objects that the statement covers. Specify a resource using Amazon Resource Name (ARN).
• NotResource - advanced policy element that explicitly matches every resource except those specified.

{
  "Version": "2012-01-01",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["DynamoDB:*", "s3:*"],
      "Resources": [
        "arn:aws:dynamodb:region:account-number:table/table-name",
        "arn:aws:s3:::bucket-name",
        "arn:aws:s3:::bucket-name/*"
      ]
    },
    {
      "Effect": "Deny",
      "Action": ["DynamoDB:*", "s3:*"],
      "NotResources": [
        "arn:aws:dynamodb:region:account-number:table/table-name",
        "arn:aws:s3:::bucket-name",
        "arn:aws:s3:::bucket-name/*"
      ]
    }
  ]
}

Identity and resource based policies

Identity based policies are attached to a principal (or identity), such as an IAM user, role, or group.

• AWS managed policies created and managed by AWS. ~1,000 AWS managed policies.
• Customer managed policies created and managed by customer, precise control.
• Inline policies are policies that are embedded directly into a single user, group or role. [Not recommended].

Resource based policies are attached to a resource, such as S3 bucket or an AWS Key Management Service. You define the policy on the resource itself and is inline only.

Conflicting policies

• default implicit deny
• explicit allow overrides implicit deny
• explicit deny overrides explicit allow

Amazon Cognito

Amazon Cognito is fully-managed service that provides authentication and authorization for web and mobile apps. Two main components:

• User pools user directory that provides sign-up and sign-in options for your app users, federated through a third-party identity provider (IdP).
• Identity pools enable to create unique identities for your users and federate them with identity providers. temporary and limited privilege AWS credentials to access other AWS services.

AWS KMS and Secrets Manager

• AWS KMS - is data protection services, enables you to create and manage encryption keys, integrates CloudTrail.
  • Customer-managed keys main components: Key administrators, key policies and key users.
• AWS Secret Manager - centrally manage secrets used to access resources on AWS, on third-party services and on premises, automatically rotate secrets, control access to secrets, and audit and monitor secret usuage

Source

• Introduction to Cloud 101 - AWS Educate
• Getting Started with Networking (Lab) - AWS Educate

• Every action that you make in AWS is an API call that is authenticated and authorized.
• Security and compliance are a shared responsibility between AWS and you.
• When you first access AWS, you begin with a single sign-in identity known as the root user.
• AWS Identity and Access Management (IAM) is an AWS service that helps you manage access to your AWS account and resources.
• Maintaining roles is more efficient than maintaining users, users have long-term credentials, whereas IAM dynamically provides temporary credentials that expire after a defined period of time. (15 minutes to 36 hours).
• Policies attached to Role, Role can be assigned and unassigned efficiently to User or AWS Services to perform specific task without modifying any other policies.
• AWS IAM Identity Center allows to manage federated users, then can be used to assign a role to a federated user when access is requested through an identity provider.
• Three types of compute options are available: virtual machines (VMs), container services, and serverless.
• When architecting any application for high availability, consider using at least two EC2 instances in two separate Availability Zones.

Choosing the right AWS Region

AWS regions are independent from one another.

• Latency - application sensitive to latency (delay between a request for data and the response), choose a REgion that is close to your user base.
• Pricing - Due to the local economy and the physical nature of operating data centers, prices vary from one Region to another. AWS charges based on the financial factors specific to each Region.
• Service Availability - Some services might not be available in some regions.
• Data Compliance - Enterprise companies often must comply with regulations that require customer data to be stored in a specific geographic territory. (GDPR)

Availability Zones

Availability Zones consist of one or more data centers with redundant power, networking and connectivity, connected using redundant high-speed and low-latency links. e.g. us-east-1a, sa-east-1b.

To keep application available, we have to maintain high availability resiliency. At minimum, should use two Availability Zones.

Edge locations

Edge locations are global locations where content is cached. For example, if your media content is in London and you want to share video files with your customers in Sydney, you could have the videos cached in an edge location closest to Sydney. AWS has more than 400+ edge locations globally.

Amazon CloudFront is a specific Content Delivery Network (CDN) service provided by Amazon Web Services (AWS). It uses a network of Edge Locations around the world to cache and serve content to users with low latency and high data transfer speeds.

AWS Compute

The EC2 instance is the entity you interact with, where you can install your web server and serve your content to users. An AMI includes the operating system, storage mapping, architecture type, launch permissions, and any additional preinstalled software applications. In this case, the AMI is how you model and define your instance.

When you launch a new instance, AWS allocates a virtual machine that runs on a hypervisor. Then the AMI that you selected is copied to the root device volume, which contains the image that is used to boot the volume.

Amazon EC2 instance types

For example, c5n.xlarge can be broken down as follows:

• First position c, indicates the instance family. This indicates that this instance belongs to the compute optimized family.
• Second position 5, indicates the generation of the instance. This instance belongs to the fifth generation of instances.
• Remaining letters before the period – In this case, n indicates additional attributes, such as local NVMe storage.
• After the period, xlarge indicates the instance size. In this example, it’s xlarge.

Difference between stop and stop-hibernate

When you stop an instance, it enters the stopping state until it reaches the stopped state. AWS does not charge usage or data transfer fees for your instance after you stop it. But storage for any Amazon EBS volumes is still charged. While your instance is in the stopped state, you can modify some attributes, like the instance type. When you stop your instance, the data from the instance memory (RAM) is lost.

When you stop-hibernate an instance, Amazon EC2 signals the operating system to perform hibernation (suspend-to-disk), which saves the contents from the instance memory (RAM) to the EBS root volume. You can hibernate an instance only if hibernation is turned on and the instance meets the hibernation prerequisites.

Terminate

When you terminate an instance, the instance stores are erased, and you lose both the public IP address and private IP address of the machine

Pricing

On-Demand Instances are recommended for the following use cases:

• Users who prefer the low cost and flexibility of Amazon EC2 without upfront payment or long-term commitments
• Applications with short-term, spiky, or unpredictable workloads that cannot be interrupted
• Applications being developed or tested on Amazon EC2 for the first time

Spot Instances are recommended for the following use cases:

• Applications that have flexible start and end times
• Applications that are only feasible at very low compute prices
• Users with fault-tolerant or stateless workloads

Savings Plans are recommended for the following use cases:

• Workloads with a consistent and steady-state usage
• Customers who want to use different instance types and compute solutions across different locations
• Customers who can make monetary commitment to use Amazon EC2 over a 1-year or 3-year term

With Reserved Instances, you can choose the type that best fits your applications needs.

• Standard Reserved Instances: These provide the most significant discount (up to 72 percent off On-Demand pricing) and are best suited for steady-state usage.
• Convertible Reserved Instances: These provide a discount (up to 54 percent off On-Demand pricing) and the capability to change the attributes of the Reserved Instance if the exchange results in the creation of Reserved Instances of equal or greater value. Like Standard Reserved Instances, Convertible Reserved Instances are best suited for steady-state usage.
• Scheduled Reserved Instances: These are available to launch within the time windows that you reserve. With this option, you can match your capacity reservation to a predictable recurring schedule that only requires a fraction of a day, a week, or a month.

Dedicated Hosts can help you reduce costs because you can use your existing server-bound software licenses, such as Windows Server, SQL Server, and Oracle licenses.

• Dedicated Hosts can be purchased on demand (hourly).
• Dedicated Hosts can be purchased as a Reservation for up to 70 percent off the On-Demand price.

A container, powered by the containerization engine, is a standard unit of software that encapsulates the application code, runtime, system tools, system libraries, and settings necessary for programmers to build, ship, and run applications efficiently.

Characteristics

Challenges

• Security impacted if operating system affected
• Difficult to manage thousands of containers
• Complex to migrate legacy projects to container technology
• Difficult to right-size containers for specific scenarios

Vendors

• Docker - Robust and most popular container platform today
• Podman - Daemon-less architecture providing more security than Docker containers
• LXC - Preferred for data-intensive apps and ops
• Vagrant - Offers highest levels of isolation on the running physical machine

Docker

Docker is an open platform, or engine, written in Go programming language, uses Linux kernel’s features and namespaces technology to provide isolated workspace, where programmers can Develop -> Ship -> Run -> Containers. Docker became popular because:

• Simple architecture
• Scalability
• Easy portability

Docker Commands

Docker Objects

Dockerfile

is a text file that contains instructions needed to create an image.

• FROM - Defines base image, always begin from this instructions.
• RUN - Executes arbitrary commands
• CMD - Defines default command for container execution

CMD should always have one command instructions. If multiple CMD instructions then only the last command instruction will take effect.

Image

layers can be shared between images, which saves disk space and network bandwidth.

Naming

      hostname/repository:tag
        ^         ^        ^
      /           |          \
  Image       Container    Version
 Registry      Images   or Variant of Image


e.g. docker.io/ubuntu:18.04

The host name can be excluded using Docker CLI.

Container

• is runnable instance of an image
• can be created, stopped, started or deleted using the Docker API or CLI
• can connect to multiple networks, attach storage, or create a new image based on its current state
• is well isolated from other containers and its host machine

Network

• Networks are used for the isolated containers communication

Storage

• Docker uses volumes and bind mounts to persist data even after a container stops

Plugins

• Storage plugins provide the ability to connect to external storage platforms

Docker Architecture

consists of a Docker client, a Docker host, and a registry.

• based on client-server architecture
• provides a complete application environment
• includes the client, the host, and the registry components
• Docker host server include Docker daemon known as dockerd

Docker CLI or REST APIs -------> Docker host server
                  sends instructions

Daemon <----- Docker API requests or "docker run" commands
       listens

The daemon ----------------> to the registry
    builds, runs, and distributes containers

Registry ----------> images (either public or private)
          stores

Registry

• stores and distributed images, public (Docker Hub), private (implemented for security).
• registry locations are either hosted or self-hosted.

Commands

IBM Specific Commands

Under the Hood

Cgroups (control groups) are a kernel mechanism for limiting and measuring the total resources used by a group of processes running on a system. For example, you can restrict CPU, memory, network, or I/O quotas using cgroups.

Namespaces are a kernel mechanism for limiting the visibility that a group of processes has over the rest of a system. For example, you can limit visibility to certain process trees, network interfaces, user IDs, or filesystem mounts.

cgroups limit how much of a resource you can use whereas namespaces limit which resources you can use.

LXC Framework

LXC (Linux Containers) are an OS level virtualization method that allow running multiple isolated Linux containers on a host OS using a single Linux kernel. LXC (Linux Containers) uses the following kernel features to contain processes:

• LXC namespaces (IPC, Network, Mount, PID, User, UTS)

  Kernel version 4.10, there are six kinds of namespaces, IPC, Network, Mount, PID, User, UTS

• LXC cgroups (control groups)

  LXC cgroups framework provides Resource Limiting, Prioritization, Accounting, Control

• LXC security modules (AppArmor and SELinux profiles)

  Two most accepted modules in Linux Kernel are AppArmor and SELinux.

LXC Containers

• Privileged containers are defined as any container where the container UID 0 is mapped to the host’s UID 0.
• Unprivileged containers are safe by design. The container UID 0 is mapped to an unprivileged user outside of the container and only has extra rights on resources that it owns itself.

Container Security

• Securing the container development pipeline and corresponding applications.
• Securing the container deployment environment(s) and corresponding infrastructure.
• Integrating with enterprise security tools and enhancing existing security policies and procedures.

Challenges to DevOps

• Code Phase - use of third-party resources, GitHub repository is unavoidable
• Build Phase - Installing applications keeping default configuration, not implementing security safeguards or hardening controls.
• Test Phase - Deployed applications can be exploited through a variety of security testing methods.
• Deploy Phase - Exposing data in Docker files and embedded malware in container image.
• Operate Phase - Non-updated images, well-known vulnerability in dependencies, unrestricted admin access, unauthorized access.
• Monitor Phase - Hijacked repository, image registry and poisoned resources, exposed ports, Kubernetes/Docker API abuse

Defense in Depth in Container

• Cloud Provider is first layer involving supporting infrastructure, typically hosted by main cloud providers.
• Cluster allows to manage the workloads of nodes within a management layer are responsible for running application.
• Container third layer, static analysis of vulnerability in application container, image signing and enforcement to mitigate man-in-the-middle attacks.
• Code security awareness for engineers and developers.

Securing Container Layer

• Docker Content Trust - builtin and integrates TUF using Notary, strong cryptographic guarantees over code.
• Portieris - Kubernetes admission controller for enforcing content trust.
• Project Calico - Open source networking and network security solution for containers, VMs, and native host-based workloads.

Securing Code Layer

• Awareness of common vulnerabilities affecting this technology and way to test them.
• Grant access to application over TLS only.
• Limit port ranges of communication.
• Don’t hardcode secrets or passwords in source code and don’t reuse.
• Store secrets in secure repositories. Unsecure repo includes shared storage, unencrypted files, version control systems, S3 etc.
• Use third party dependency security, e.g. GitHub security alerts scan and alert developers.
• Perform static code analysis - scan codebases for common security errors.
• Perform dynamic testing - using tools to exploit well-known attacks, OWASP Top Ten Project.
• Perform secure API development.

Docker

Docker enables the organization to run, create, and manage containers on a single operating system.

Kubernetes

Kubernetes is a container orchestration system for Docker containers, address container issues, provide scaling as needed.

Architecture

Control Plane

• One or More API Servers: Entry point for REST / kubectl
• etcd: Distributed key/value store
• Controller-manager: Always evaluating current vs desired state
• Scheduler: Schedules pods to worker nodes

Data Plane

• Made up of worker nodes
• kubelet: Acts as a conduit between the API server and the node
• kube-proxy: Manages IP translation and routing

Components of Cluster

• Node - common term for VMs and/or bare-metal servers that Kubernetes manages.
• Pod - basic unit of deployment in Kubernetes, collection of related Docker containers that need to coexist.

Securing the Components of the Cluster

• Use transport layer security (TLS) for all API traffic.
• Implement API authentication and authorization can be infrastructure, like nodes, proxies, the scheduler, and volume plugins.
• Limit resource usage on a cluster, resource quota, max/min size of resources like CPU, memory, or persistent disk a namespace can allocate.
• Employ user permissions hardening as most application workloads need limited access to host resources (UID 0).
• Restrict network access.
• Restrict metadata access, the cloud platforms exposes metadata contain cloud credentials services locally to instances so running Kubernetes on cloud platform, limit permissions given to instance credentials, use network policies to restrict pod access metadata api.
• Control which nodes, pods may access, use policies to separate workloads.
• Restrict access to etcd write access equivalent to gaining root on entire cluster and read access can escalate quickly.
• Enable audit logging of the cluster.
• Don’t use alpha or beta features in production clusters.
• Rotate infrastructure credentials, shorter lifetime of secret, automate rotation.
• Review third-party integration libraries and requested permissions.
• Encrypt secrets at rest.

Container Orchestration

Container Orchestration means managing lifecycle of containers, especially in large, dynamic environments. It used used to deploy, scale, schedule and network applications. Some engineering teams container orchestration tasks are:

• Scaling up or down resources
• Provisioning and deployment
• Redundancy and availability
• Expose singular points of access
• Load balancing
• Resource sharing
• External exposure of services running in a container to the outside world
• Health monitoring
• Configuration of an application in relation to the containers running it

Reference

• The SANS Institute’s checklist for auditing Docker-based containers
• The article “Container Security: Examining Potential Threats to the Container Environment”
• Kubernetes Cloud Native Security
• Securing a Kubernetes cluster
• Orchestration with Kubernetes
• Notary
• Docker Content Trust
• IBM Portieris
• SELinux
• Project Calico
• What Is Container Orchestration?
• Security for Containers

• considering asymptotic run-times.
• how does runtime scale with input size.

Big O

• clarifies growth rate.
• cleans up notation.
• can ignore complicated details.
• Big-O is only asymptotic and loses important information about constant multiples.

$$ O(n^2) \approx 3n^2 + 5n + 2 $$

$$ O(n) \approx n + log_2(n) + sin(n) $$

$$ O(n \medspace log(n)) \approx 4n \medspace log_2(n) + 7 $$

Common Rules

Notation Graph

Logarithms Rules

$$ log_a(n^k) = k\space log_a\space n $$

$$ log_a(nm) = log_an + log_an $$

$$ n^{log_ab} = b^{log_an} $$

$$ log_a\space n\cdot log_b\space a = log_b\space n $$

Example:

$$ log_3 \medspace 81 == 3^x = 81, x = 4 $$

$$ log_k n == k^x = n, x\space and\space k\space are\space constants $$

Time Complexity

• time is a sum of the times required.
• is a mathematical notation to describe the algorithm’s performance or complexity of an algorithm.
• specifically how long an algorithm takes to run as the input size grows.
• it allows to analyze and compare the efficiency of different algorithms
• also make informed decisions about which one to use in a given situation.

Space Complexity

• memory is a measure of maximal heap and stack utilization.
• primitive variables (booleans, numbers, undefined, null) are constant space O(1)
• strings require O(n) space (where n is the string length)
• reference types are generally O(n), where n is the length (for arrays) or the number or keys (for objects)

Data Structures

• are collections of values
• the relationships among them
• the functions or operations that can be applied to the data
• commonly used data structures in JavaScript: array and object

Arrays

• contiguous and indexed in order
• insertion and deletion can be expensive
• lookup is great - can quickly be accessed at a specific index
• easy to sort
• small size-wise
• stuck with fixed size, no flexibility

Linked List

• insertion/deletion is easy.
• lookup is bad - have to rely on linear search
• relatively difficult to sort, instead sort as you construct
• relatively small size wise (not as small as arrays)

Note: Searching a value could be expensive while using linked list, however if you need to perform insertion and deletion in a large data set then it could be the better option than an array

Singly-Linked List

node1(value | next) --> node2(value | next) --> node3(value | next) --> NULL

#include <stdio.h>
#include <stdlib.h>

typedef struct node
{
    int number;
    struct node *next;
} node;

int main(int argc, char *argv[])
{
    node *list = NULL;

    for (int i = 1; i < argc; i++)
    {
        int number = atoi(argv[i]);
        node *new = malloc(sizeof(node));

        // Handling allocation error.
        if (new == NULL)
        {
            free(new);
            return 1;
        }

        // Assigning number.
        new->number = number;
        new->next = NULL;

        // If list is empty
        if (list == NULL)
        {
            list = new;
        }
        // If new number is small, insert at the beginning.
        else if (new->number < list->number)
        {
            new->next = list;
            list = new;
        }
        // if belongs later in the list
        else
        {
            for (node *ptr = list; ptr != NULL; ptr = ptr->next)
            {
                // if end of the list
                if (ptr->next == NULL)
                {
                    ptr->next = new;
                    break;
                }

                // if in the middle of list
                if (new->number < ptr->next->number)
                {
                    new->next = ptr->next;
                    ptr->next = new;
                    break;
                }
            }
        }
    }

    // new pointer to point all the list
    node *ls_ptr = list;

    while (ls_ptr != NULL)
    {
        printf("%i\n", ls_ptr->number);
        ls_ptr = ls_ptr->next;
    }

    return 0;
}

• do not have indexes
• connected via nodes with a next pointer
• random access is not allowed
• not applicable for binary searching.
• can only move in one direction.
• Time Complexity:
  • Insertion:
    • O(1) if unsorted (beginning)
    • O(n) if in sorted.
  • Deletion: Depends O(1) or O(n)
  • Search: O(n)
  • Access: O(n)

Doubly-Linked List

• has one more pointer previous than singly linked list.
• takes up more memory but has more flexibility than singly linked list.
• allows to move forward and backward through the list.
• Time Complexity:
  • Insertion: O(1)
  • Deletion: O(1)
  • Search: O(n)
  • Access: O(n)

Stack

• implementation using as an array or Linked List (LIFO).
• insert at the beginning and remove at the beginning.
• two operation: push and pop.
• Time Complexity:
  • Insertion: O(1)
  • Deletion: O(1)
  • Searching: O(n)
  • Access: O(n)

Queue

• implementation using as an array or a Linked List (FIFO).
• insert at the end and remove at the beginning.
• two operations: enqueue and dequeue.
• Time Complexity:
  • Insertion: O(1)
  • Deletion: O(1)
  • Searching: O(n)
  • Access: O(n)

Dictionary

• stores in a form of key (word) and value (definition).
• data structures like: arrays, hash tables.

Trees [DRAFT]

Binary Search Trees

• Time Complexity
  • Searching - O(log n)

Tries

• tree of an arrays.
• combines structures and pointers together to store data.
• data can be searched through roadmap.
• no collisions
• not widely used because huge NULL pointers (trade-off)
• insertion is complex - lot of dynamic memory allocations
• deletion is easy - just free a node
• lookup is fast - not fast as an array (but almost)
• already sorted
• rapidly becomes huge, even very little data present.
• not great if space is at a premium
• Time Complexity
  • Searching - O(k) -> constant value k -> O(1)

Hashing

• function in math or code that takes any number of inputs and maps them to a finite number of outputs (range).
• a good hash function
  • always give the same value for the same input (be deterministic)
  • use only and all (buckets) the data being hashed
  • an even (uniform) distribution of data across buckets
  • generate very different hash codes for very similar data
• analogy: separating different patterns in a cards. shuffled card to -> spade, heart, diamond, club

HASH_MAX = 100;

unsigned int hash(char & str)
{
    int sum = 0;

    for (int j = 0; star[j] != "\0"; j++)
    {
        sum += str[j];
    }

    return sum % HASH_MAX;
}

Hash Tables

• array of linked lists.
• allows random access ability of an array.
• widely used.
• insertion is two step process - hash and add
• deletion is easy - once found
• lookup is on average better than linked lists (we have constant factor)
• not an ideal data structure for sorting. (trade off)
• also called hash map.
• example: maps and sets
• Time Complexity
  • Searching - O(n) or ~O(1)
  • Insertion - ~O(1)
  • Deletion - ~O(1)

Collision

• occurs when two pieces of data, when run through the hash function, yield the same hash code.
• shouldn’t simply overwrite it.

Linear probing

• we try to place the data in the next consecutive element in the array.
• such that, if not find directly, can be find nearby.
• subject to a problem called clustering.

Chaining

• every element of the array hold multiple pieces of data.
• each element of the array is a pointer to the head of a linked list.
• multiple pieces of data can yield the same hash code.

Binary Heap

Priority Queue

Graph

Searching Algorithms

Linear Search

• algorithm iterate across the array from left to right, searching for a specified element.
• O(n) - look through entire array of n elements.
• Ω(1) - target is the first element, can stop immediately.

Binary Search

• should be sorted.
• algorithm divide and conquer, reducing the search area by half each time, trying to find a target number.
• O(log n) - split array of n element in half repeatedly to find target, either element exist or doesn’t exist.
• Ω(1) - target element is at the midpoint of the full array, and so we can stop looking immediately after we start.

Sorting Algorithms

Bubble sort

• algorithm to move higher valued elements generally towards the right and lower value elements generally towards the left.
• O(n²) - array in reverse order, bubble “n” elements n times.
• Ω(n) - array is already perfectly sorted, we make no swaps on the first pass.

Selection sort

• algorithm is to find the smallest unsorted element and add it to the end of the sorted list
• O(n²) - iterate over each of the n element of array, repeating this process n times. only one element sorted at each pash
• Ω(n²) - no guarantee the array is sorted until we go through this process.

Merge sort

• algorithm is to sort smaller arrays and then combine those arrays together (merge them) in sorted order
• O(n log n) - split n elements up and then recombine them, doubling the sorted sub-arrays as we build them up.
• Ω(n log n) - array is perfectly sorted, still have to split and recombine them back together with algorithm.

"""
Module: to demonstrate merge sorting
"""


def merge(left, right):
    """Merging two lists"""
    li, ri = 0, 0
    merged_lists = []

    # compare smallest to the list.
    while li < len(left) and ri < len(right):
        if left[li] <= right[ri]:
            merged_lists.append(left[li])
            li += 1
        else:
            merged_lists.append(right[ri])
            ri += 1

    # add all remaining
    merged_lists += left[li:]
    merged_lists += right[ri:]

    return merged_lists


def merge_sort(lists):
    """Merge sort function"""
    half = len(lists) // 2

    if len(lists) == 1:
        return lists

    return merge(merge_sort(lists[:half]), merge_sort(lists[half:]))


if __name__ == "__main__":
    output = merge_sort([6, 5, 4, 3, 2, 1])

    print(output)

Radix sort

Insertion sort

Algorithms and it’s Applications

Inheritance

Recursion

• the function that calls itself as part of execution, elegant because no other function, no loops.
• search recursion in Google.

Fibonacci Series

• developed to study rabbit populations.
• after n generation, it gives number of rabbits it has.

$$ F_n {\geq} 2^{n/2} for, n {\geq} 6 $$

"""
Module: Fibonacci series
"""

import os

os.system("clear")


def fibonacci_slow(position):
    """Recursive fibonacci function"""
    if position <= 1:
        return position

    response = fibonacci_slow(position - 1) + fibonacci_slow(position - 2)

    return response


memo = []


def fibonacci_memoized(position):
    """Memoized approach of fibonacci function"""
    if position <= 1:
        return position

    if position in memo:
        return memo[position]

    return fibonacci_memoized(position - 1) + fibonacci_memoized(position - 2)


def fibonacci_bottom_up(n):
    """Bottom up approach of Fibonacci"""
    prev = 0
    nxt = 1
    count = 0

    while count < n:
        print(nxt, end=" ")

        total = prev + nxt
        prev, nxt = nxt, total

        count += 1


def fibonacci_dynamic(n):
    """
    Dynamic programming approach of fibonacci function.
    => Memoization + Bottom-up
    """
    series = [0, 1]

    for position in range(2, n + 1):
        series.append(series[position - 1] + series[position - 2])

    return series


if __name__ == "__main__":
    TOTAL_SIZE = 10

    print("Fibonacci Slow - O(2^n)")
    for i in range(1, TOTAL_SIZE + 1):
        res = fibonacci_slow(i)
        print(res, end=" ")

    print("\nFibonacci Memoized - O(n)")
    for i in range(1, TOTAL_SIZE + 1):
        res = fibonacci_memoized(i)
        print(res, end=" ")

    print("\nFibonacci Bottom-up - O(n)")
    fibonacci_bottom_up(TOTAL_SIZE)

    print("\nFibonacci Dynamic - O(n)")
    output = fibonacci_dynamic(TOTAL_SIZE)
    print(output)

# Output
Fibonacci Slow - O(2^n)
1 1 2 3 5 8 13 21 34 55
Fibonacci Memoized - O(n)
1 1 2 3 5 8 13 21 34 55
Fibonacci Bottom-up - O(n)
1 1 2 3 5 8 13 21 34 55
Fibonacci Dynamic - O(n)
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

Collatz Conjecture

"""
Module: collatz conjecture applies to positive integers and
speculates that it is always possible to get "back to 1"
"""

import os

os.system("clear")


def collatz(position):
    """Recursive collatz conjecture function"""
    print(int(position), end="")

    if position <= 1:
        return 0

    print(end=" -> ")
    if position % 2 == 0:
        return 1 + collatz(position / 2)
    else:
        return 1 + collatz(3 * position + 1)


if __name__ == "__main__":
    for i in range(1, 11):
        collatz(i)
        print()

# Output
1
2 -> 1
3 -> 10 -> 5 -> 16 -> 8 -> 4 -> 2 -> 1
4 -> 2 -> 1
5 -> 16 -> 8 -> 4 -> 2 -> 1
6 -> 3 -> 10 -> 5 -> 16 -> 8 -> 4 -> 2 -> 1
...

DSA against Problem

1. If we are dealing with top/maximum/minimum/closest ‘K’ elements among ‘N’ elements, we will be using a Heap.

2. If the given input is a sorted array or a list, we will either be using Binary Search or the Two Pointers strategy.

3. If we need to try all combinations (or permutations) of the input, we can either use Backtracking or Breadth First Search.

4. Most of the questions related to Trees or Graphs can be solved either through Breadth First Search or Depth First Search.

5. Every recursive solution can be converted to an iterative solution using Stack.

6. For a problem involving arrays, if there exists a solution in O(n^2) time and O(1) space, there must exist two other solutions:

   • Using a HashMap or a Set for O(n) time and O(n) space,
   • Using sorting for O(n log n) time and O(1) space.

7. If a problem is asking for optimization (e.g., maximization or minimization), we will be using Dynamic Programming.

8. If we need to find some common substring among a set of strings, we will be using a HashMap or a Trie.

9. If we need to search/manipulate a bunch of strings, Trie will be the best data structures.

10. If the problem is related to a LinkedList and we can’t use extra space, then use the Fast & Slow Pointer approach.

Reference

• https://dev.to/arslan_ah/20-essential-coding-patterns-to-ace-your-next-coding-interview-32a3
• https://pll.harvard.edu/course/cs50-introduction-computer-science
• https://www.toptal.com/developers/sorting-algorithms
• https://www.linkedin.com/pulse/time-complexity-vs-space-sumaiya-rimu/

• the simplest algorithms that exist, and they often seem to be a very natural thing to try when you’re solving a problem.
• subproblem is a similar problem of smaller size.
• greedy choice is a safe choice if there is an optimal solution consistent with this first choice.

ingredients

reduction to subproblem

• make some first choice
• then solve a problem of the same kind
• smaller: few digits, fewer patients
• this is called a ‘subproblem’

safe choice

• a choice called safe if there is an optimal solution consistent with this first choice.
• not all first choices are safe.
• greedy choices are often unsafe.

general strategy

• make a greedy choice
• prove that it is a safe choice.
• reduce to a subproblem
• solve the subproblem

          greedy choice
Problem -------------------------> Safe Choice

When first starting a penetration test or any security evaluation on a target, a primary step is known as Enumeration which involves scanning of the open ports.

Tools

ping - Packet Internet or InterNet Groper

ping <target-ip-addr> -c 4

nmap - Network Mapper

sudo nmap -p- <target-ip-addr> # Scan all ports, takes longer.
sudo nmap -sV <target-ip-addr> # Name and description of identified services.

showmount - mount info for NFS server

showmount -e <target-ip-addr>
/SomeDir (everyone)

sudo mkdir nfs
sudo mount -t nfs <target-ip-addr>/SomeDir nfs

hash-id - Hash Identifier

hash-id -h <hash>
hash-id -f <file-name-path>

smbmap - SMB share drives

smbmap -H <target-ip-addr> -u 'user' -p 'pass'

enum4linux - Windows and Samba system

enum4linux <target-windows-ip-addr>

netdiscover - IP addresses on the network

netdiscover -r <network-addr>/<cidr>

netcat / whatweb / curl / dmitry - Banner Grabbing

netcat <target-ip-addr> <port>
220 (vsFTPd 2.3.4) # FTP
quit
221 Goodbye

whatweb http://<target-ip-addr>

dmitry -p <target-ip-addr> # -p port scan on 150 most used services
dmitry -pb <target-ip-addr> # -b switch version of the program running.

Port numbers

Challenges

telnet(port 23/tcp Linux telnetd)

Telnet is an old service used for remote management of other hosts on the network. Usually, connection requests through telnet are configured with username/password combinations for increased security.

Note: Due to configuration mistakes, some important accounts can be left with blank passwords for the sake of accessibility. Some typical important accounts have self-explanatory names, such as: admin, administrator and root leaving open to simple brute-forcing attacks.

$ telnet {target-ip-addr}

Meow login: root

cat flag.txt

FTP(port 21/tcp vsftpd 3.0.3)

File transfer services that may have high chances to be poorly configured, it can be easily misconfigured if not correctly understood. For secure transmission that protects the username and password and encrypts the content, FTP is often secured with SSL/TLS (FTPS) or replaced with SSH File Transfer Protocol (SFTP).

Note: FTP users may authenticate themselves with a clear-text sign-in protocol, generally in the form of a username and password. A typical misconfiguration for running FTP services allows the anonymous username, followed by any password whatsoever since the service will disregard the password for this specific account.

$ ftp {target-ip-addr}
Name: anonymous
331 Please specify the password.
Password: anon123
ftp> ls
200 : PORT command successful. Consider using PASV.
150 : Here comes the directory listing.

226 : Directory send OK.

> get flag.txt # Download the file to the same directory.
226 : Transfer complete.

> bye
421 Timeout

# Other options: Make sure from are in right path where the file exists.
> put exploit.php # Upload a file

> mdelete exploit.php # Delete a file

SMB(445/tcp microsoft-ds?)

SMB (Server Message Block) is communication protocol provides shared access to files, printers, and serial ports between endpoints on a network. We mostly see SMB services running on Windows machines. SMB runs at the Application or Presentation layers of the OSI model. Due to this, it relies on lower-level protocols for transport. The Transport layer protocol that Microsoft SMB Protocol is most often used with is NetBIOS over TCP/IP (NBT).

Note. An SMB-enabled storage on the network is called a share . SMB clients are required to provide a username/password combination to see or interact with the contents of the SMB share. A network administrator can sometimes make mistakes and accidentally allow logins without any valid credentials or using either guest_accounts or anonymous log-ons.

Four separate shares:

• ADMIN$ - Administrative shares are hidden network shares created by the Windows NT family of operating systems that allow system administrators to have remote access to every disk volume on a network-connected system. These shares may not be permanently deleted but may be disabled.
• C$ - Administrative share for the C:\ disk volume. This is where the operating system is hosted.
• IPC$ - The inter-process communication share. Used for inter-process communication via named pipes and is not part of the file system.
• WorkShares - Custom share.

$ smbclient -L {target-ip-addr}

$ smbclient \\\\{target-ip-addr}\\ADMIN$
    or
$ smbclient \\\\{target-ip-addr}\\C$
NT_STATUS_ACCESS_DENIED

# Seems to be human made, prone to misconfiguration.
$ smbclient \\\\{target-ip-addr}\\WorkShares
smb: \>

ls : listing contents of the directories within the share
cd : changing current directories within the share
get : downloading the contents of the directories within the share
exit : exiting the smb shell

# Passing username and password.
$ smbclient -U 'user' \\\\{target-ip-addr}\\Administrator

Redis(6379/tcp key-value store 5.0.7)

Redis (REmote DIctionary Server), which is an ‘in-memory’ database are the ones that rely essentially on the primary memory for data storage (meaning that the database is managed in the RAM of the system); in contrast to databases that store data on the disk or SSDs. Primary memory is significantly faster than the secondary memory, the data retrieval time in the case of ‘in-memory’ databases is very small, thus offering very efficient & minimal response times.

Note: In-memory databases like Redis are typically used to cache data that is frequently requested for quick retrieval. The Keyspace section provides statistics on the main dictionary of each database. The statistics include the number of keys, and the number of keys with an expiration.

$ redis-cli -h {target-ip-addr}
> info
# Keyspace
db0:keys=4

> select 0 # Select Redis logical database followed by index number.

> keys * # List all the keys present.

> get flag

$ redis-cli -h {target-ip-addr}
> AUTH <passkey>

Source

• Hack The Box

• Batch processing collection of data from multiple sources and combined to become a single source of information.
• a type of batch processing called Extract, Transform and Load (ETL)
• the process of extracting large amounts of data from multiple sources and formats and transforming it into one specific format before loading it into a database or target file.

Block Diagram of ETL

Extraction

Python Composite Functions

import glob
import pandas as pd

list_csv = glob.glob("*.csv")
list_json = glob.glob("*.json")

df_csv = pd.read_csv("source.csv")
df_json = pd.read_json("source.json", lines=True)

Extract Function

def extract():
    extracted_data = pd.DataFrame(columns=['name', 'height', 'weight'])

    for csvfile in glob.glob('*.csv'):
        extracted_data = extracted_data.append(
            pd.read_csv((csvfile),
            ignore_index=True
        )

    for jsonfile in glob.glb("*.json"):
        extracted_data = extracted_data.append(
            pd.read_json(jsonfile, lines=True),
            ignore_index=True
        )

    return extracted_data

If we did not set the parameter “ignore_index” to true, then the index of the data frame “extracted_data” would be the same as the row Number of the original file.

If we “ignore index” to true, then the order of each row would be the same as the order the row was appended to the data frame.

Transform

def transform(data):
    data['height'] = round(data.height * 0.0254, 2)
    data['weight'] = round(data.weight * 0.45359237, 2)

    return data

Load

def load(targetfile, data_to_laod):
    data_to_load.to_csv(targetfile)
    targetfile = 'transformed_data.csv'
    load(targetfile, transformed_data)

Logging Entries

from datatime from datetime

def log(message):
    timestamp_format = "%Y-%h-%d-%H:%M:%S"
    now = datetime.now()
    timestamp = now.strftime(timestamp_format)

    with open("logfile.txt", "a") as f:
        f.write (timestamp + ',' + message + '\n')

Hands-on Project

The complex Extract, Transform, and Loading operations involves following steps:

• Extract relevant information from websites using Webscraping and requests API.
• Transform the data to a required format.
• Load the processed data to a local file or as a database table.
• Query the database table using Python.
• Create detailed logs of all operations conducted.

Solutions

Top Countries GDP - ETL Process

"""Module: Code for ETL operations on Country-GDP data"""

import os
import sqlite3
import requests
import numpy as np
import pandas as pd

from bs4 import BeautifulSoup
from datetime import datetime

os.system("clear")

WIKIPEDIA_GDP_URL = (
    "https://web.archive.org/web/20230902185326/https://en.wikipedia.org/wiki/List_of_countries_by_GDP_%28nominal%29"
)
TABLE_ATTRIBUTES = ["Country", "GDP_USD_millions"]
DB_NAME = "world_economies.db"
TABLE_NAME = "countries_by_gdp"
CSV_PATH = "countries_by_gdp.csv"


def extract(url, table_attributes):
    """
    This function extracts the required
    information from the website and saves it to a dataframe. The
    function returns the dataframe for further processing.
    """
    page_output = requests.get(url, timeout=10)
    html_soup = BeautifulSoup(page_output.text, "html.parser")
    gdp_df = pd.DataFrame(columns=table_attributes)

    tables = html_soup.find_all("tbody")
    rows = tables[2].find_all("tr")

    for row in rows:
        data = row.find_all("td")

        if len(data) == 0:
            continue

        country = data[0].find("a")
        gdp = data[2]

        if country is None or "—" in gdp:
            continue

        row_dict = {"Country": country.text, "GDP_USD_millions": gdp.text}
        row_df = pd.DataFrame(row_dict, index=[0])
        gdp_df = pd.concat([gdp_df, row_df], ignore_index=[0])

    return gdp_df


def convert_currency_to_numeric(currency):
    """
    This function converts the currency text into numeric form.
    """
    return float("".join(currency.split(",")))


def convert_millions_to_billions(millions):
    """
    This function convert millions value to billions and round
    to 2 decimal places.
    """
    return np.round(millions / 1000, 2)


def transform(df):
    """
    This function converts the GDP information from Currency
    format to float value, transforms the information of GDP from
    USD (Millions) to USD (Billions) rounding to 2 decimal places.
    The function returns the transformed dataframe.
    """
    gdp_list = df["GDP_USD_millions"].tolist()
    gdp_list = [convert_millions_to_billions(convert_currency_to_numeric(gdp_value)) for gdp_value in gdp_list]
    df["GDP_USD_millions"] = gdp_list
    df = df.rename(columns={"GDP_USD_millions": "GDP_USD_billions"})

    return df


def load_to_csv(df, csv_path):
    """
    This function saves the final dataframe as a `CSV` file
    in the provided path. Function returns nothing.
    """

    df.to_csv(csv_path)


def load_to_db(df, sql_connection, table_name):
    """
    This function saves the final dataframe as a database table
    with the provided name. Function returns nothing.
    """
    df.to_sql(table_name, sql_connection, if_exists="replace", index=False)


def run_query(statement, sql_connection):
    """
    This function runs the stated query on the database table and
    prints the output on the terminal. Function returns nothing.
    """
    print("Query Statement:", statement)
    output = pd.read_sql(statement, sql_connection)
    print("Query Output:", output)


def log_progress(message):
    """
    This function logs the mentioned message at a given
    stage of the code execution to a log file.
    Function returns nothing
    """
    timestamp_format = "%Y-%h-%d-%H:%M:%S"
    now = datetime.now()
    timestamp = now.strftime(timestamp_format)

    with open("./code_log.txt", "a") as f:
        f.write(timestamp + " : " + message + "\n")


if __name__ == "__main__":
    log_progress("Preliminaries complete. Initiating ETL process.")

    extracted_data = extract(WIKIPEDIA_GDP_URL, TABLE_ATTRIBUTES)
    log_progress("Data Extraction complete. Initiating transformation process.")

    transform_data = transform(extracted_data)
    log_progress("Data Transformation complete. Initiating loading process.")

    load_to_csv(transform_data, "gdp.csv")
    log_progress("Data load to CSV file complete. Initiating SQL connection.")

    sql_connection = sqlite3.connect(DB_NAME)
    log_progress("SQL Connection Initiated. Initiating loading process.")

    load_to_db(transform_data, sql_connection, TABLE_NAME)
    log_progress("Data load to database complete. Running the query")

    query_statement = f"SELECT * from {TABLE_NAME} WHERE GDP_USD_billions >= 100"
    run_query(query_statement, sql_connection)
    log_progress("Run query complete. Closing SQL connection.")

    sql_connection.close()
    log_progress('Process Complete. Connection Closed.')

World top Banks - ETL Process

"""Module: Code for ETL operations on Bank data"""

import os
import sqlite3
import requests
import numpy as np
import pandas as pd

from bs4 import BeautifulSoup
from datetime import datetime

os.system("clear")

WIKIPEDIA_BANK_URL = "https://web.archive.org/web/20230908091635/https://en.wikipedia.org/wiki/List_of_largest_banks"
TABLE_ATTRIBUTES = ["Bank_Name", "USD_Billion"]
DB_NAME = "banks.db"
TABLE_NAME = "largest_banks"
CSV_PATH = "banks.csv"


def extract(url, table_attributes):
    """
    This function extracts the required
    information from the website and saves it to a dataframe. The
    function returns the dataframe for further processing.
    """
    page_output = requests.get(url, timeout=10)
    html_soup = BeautifulSoup(page_output.text, "html.parser")
    bank_df = pd.DataFrame(columns=table_attributes)

    tables = html_soup.find_all("tbody")
    rows = tables[0].find_all("tr")

    for row in rows:
        data = row.find_all("td")

        if not data:
            continue

        bank = data[1].find_all("a")
        billion = data[2]

        row_dict = {"Bank_Name": bank[1].text, "USD_Billion": billion.text}
        row_df = pd.DataFrame(row_dict, index=[0])
        bank_df = pd.concat([bank_df, row_df], ignore_index=[0])

    return bank_df


def convert_currency_to_numeric(currency):
    """
    This function converts the currency text into numeric form.
    """
    return float("".join(currency.split(",")))


def transform(df):
    """
    This function accesses the CSV file for exchange rate
    information, and adds three columns to the data frame, each
    containing the transformed version of Market Cap column to
    respective currencies
    """
    currency_df = pd.read_csv("exchange_rate.csv")
    exchange_rate = currency_df.set_index("Currency").to_dict()["Value"]
    df["USD_Billion"] = [convert_currency_to_numeric(x) for x in df["USD_Billion"]]
    df["GBP_Billion"] = [np.round(x * exchange_rate["GBP"], 2) for x in df["USD_Billion"]]
    df["EUR_Billion"] = [np.round(x * exchange_rate["EUR"], 2) for x in df["USD_Billion"]]
    df["NPR_Billion"] = [np.round(x * exchange_rate["NPR"], 2) for x in df["USD_Billion"]]

    return df


def load_to_csv(df, csv_path):
    """
    This function saves the final data frame as a `CSV` file
    in the provided path. Function returns nothing.
    """

    df.to_csv(csv_path)


def load_to_db(df, sql_connection, table_name):
    """
    This function saves the final data frame as a database table
    with the provided name. Function returns nothing.
    """
    df.to_sql(table_name, sql_connection, if_exists="replace", index=False)


def run_query(statement, sql_connection):
    """
    This function runs the stated query on the database table and
    prints the output on the terminal. Function returns nothing.
    """
    print("Query Statement:", statement)
    output = pd.read_sql(statement, sql_connection)
    print("Query Output:", output)


def log_progress(message):
    """
    This function logs the mentioned message at a given
    stage of the code execution to a log file.
    Function returns nothing
    """
    timestamp_format = "%Y-%h-%d-%H:%M:%S"
    now = datetime.now()
    timestamp = now.strftime(timestamp_format)

    with open("./code_log.txt", "a") as f:
        f.write(timestamp + " : " + message + "\n")


if __name__ == "__main__":
    log_progress("Preliminaries complete. Initiating ETL process.")

    extracted_data = extract(WIKIPEDIA_BANK_URL, TABLE_ATTRIBUTES)
    log_progress("Data Extraction complete. Initiating transformation process.")

    transform_data = transform(extracted_data)
    log_progress("Data Transformation complete. Initiating loading process.")

    load_to_csv(transform_data, "banks.csv")
    log_progress("Data load to CSV file complete. Initiating SQL connection.")

    sql_connection = sqlite3.connect(DB_NAME)
    log_progress("SQL Connection Initiated. Initiating loading process.")

    load_to_db(transform_data, sql_connection, TABLE_NAME)
    log_progress("Data load to database complete. Running the query")

    query_statement = f"SELECT * FROM {TABLE_NAME}"
    run_query(query_statement, sql_connection)
    log_progress("Run query 1 complete. Closing SQL connection.")

    query_statement = f"SELECT AVG(NPR_Billion) FROM {TABLE_NAME}"
    run_query(query_statement, sql_connection)
    log_progress("Run query 2 complete. Closing SQL connection.")

    query_statement = f"SELECT * FROM {TABLE_NAME} LIMIT 5"
    run_query(query_statement, sql_connection)
    log_progress("Run query 3 complete. Closing SQL connection.")

    sql_connection.close()
    log_progress("Process Complete. Connection Closed.")

• ease of use
  • intuitive web interface
  • easy to navigate
  • documentation and examples included
• open source and free
  • free to install and use
  • open source
• extensibility
  • plugins add functionality
  • new features can be developed

terminologies

project or job

• a user-configured description of the work that Jenkins will manage.

• is an outlines for what we want Jenkins to do.

• job and project used interchangeably.

• build - one run of a project, a verb or a noun.

• build step - a task inside a project.

• build trigger - criteria for starting a build, manual or automatic.

plugins

• a software package that extends jenkins’ core functionality.

pipelines

scripted pipeline

• Groovy-based DSL

node {}

declarative pipeline

• specifically designed for configuring Jenkins projects as code.

pipeline {
    agent any

    options {
        buildDiscarder(logRotator(daysToKeepStr: '10', numToKeepStr: '10'))
        timeout(time: 12, unit: 'HOURS')
        timestamps()
    }
    triggers {
          cron '@midnight'
    }
    stages {
        stage('Initialize') {
            steps {
                echo 'Initializing..'
            }
        }
        stage('Build') {
            steps {
                echo 'Building..'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying....'
            }
        }
    }
}

agent

• specifies where pipeline steps will be running
  • any - run on first available system.
  • label - { label 'linux' } - run on a system with this label.
  • docker - { docker { image 'maven' } } - run pipeline inside a docker container using the specified image.
  • none - defer agent selection to stages. allows to use different agent for each state in the pipeline.

stages and steps

• specific parts of the process being automated.
• use stages name: Build, Test, and Deploy
• steps:
  • echo - print message
  • git - check out code from a Git repo
  • sh - run a shell script or local command
  • archiveArtifacts - archive artifacts.

environment variables

• named with all capital letters.
• can be set globally (inside pipeline) or locally (inside stage).

echo env.MAX_SIZE
echo "$env.MAX_SIZE"
echo "${env.MAX_SIZE}"

currentBuild variables

• refers to the currently running build.
• properties of variable named currentBuild:
  • currentBuild.startTimeInMillis
  • currentBuild.duration
  • currentBuild.currentResult

parameter variables

• must include a name, default value, and description.
• use all capital letters in the name for easy identification.
• parameter types: string, text,boolean, choice and password

-> params.PARAMETER_NAME
-> "${params.PARAMETER_NAME}"

pipeline {
  agent any
  parameters {
    ...
  }
  ...
}

conditional expressions

• conditions for when:
  • branch
  • environment
  • expression

when {
  expression {
    params.ENVIRONMENT == 'PRODUCTION'
  }
}

manual approvals

pipeline {
  agent any
  stages {
    state('XYZ') {
      steps {
        input message: 'Confirm deployment to production...', ok 'Deploy'
      }
    }
  }
}

pipeline as code

• captures project configuration as code.
• Jenkinsfile (Pipeline script from SCM)
• supports GitOps approach by using the repo as the single source of truth.
• Changes are reviewed before being merged and applied using automation.

pipeline {
  tools {...}
  options {...}
  triggers {...}
  stages {...}
  post {...}
}

external scripts

sh('./scripts/build.sh')

bat('..\scripts\build.bat')

dir("${env.WORKSPACE}/environments/test") {
  sh('
    terraform init
    terraform plan
  ')
}

distribute builds with agents

• Jenkins controller provides a web interface to manage configuration.
• best practice is to limit jobs that are run on the controller to free up resources.
• Node: a server or system connected to a Jenkins controller.
• Agent: a process running on a node that manages jobs and reports status to the Jenkins controller.

ssh nodes

• connect to servers using SSH keys
• system must accept SSH connections
• a user and key must be in place for Jenkins to use
• Java must be installed.

docker nodes

• run jobs in new containers on every build

tips for using nodes and agents

• agent configuration, start using label instead of any:

agent {
  label 'linux'
}

agent {
  docker ...
}

• tool configuration, allowing more control over the version of specific tool:

tools {
  maven 'Maven-3.8.4'
}

• checking out code when a job is associated with a repository:

git branch: 'master',
  url: 'https://github.com/<org>/<repo>.git'

securing Jenkins

• out-of-the-box security
  • locked by default
  • initial admin password
• user accounts for individuals with usernames and passwords.

security realms

• controls how a person is authenticated to access resources.
• provides an interface to manage identities and their authorization to use a resource.
• Jenkins built-in user database is the default security realm.
• Jenkins can delegate authorization to other realms.
  • LDAP
  • Unix/Linux users and groups.

matrix-based security

• enable with matrix authorization strategy plugin.
• permissions are assigned per user.
• permissions are assigned per action.
• review “Disable Access Control” if you get locked out.

Note: Be sure to assign admin permissions to key accounts when implementing matrix-based security. Failing to assign permission correctly may lock out admin users.

project-based permissions

• permissions are applied globally with matrix-based authorization.
• project-based authorization can be used to limit permissions to specific jobs and folders.
• enabled the same as matrix-based permissions.
• additional permissions can be configured for individual projects.

secrets and credentials

• Jenkins can store and manage sensitive information.
• Sensitive information is referred to as a credential.
• Types of credentials:
  • Usernames and passwords
  • SSH keys
  • Files
  • Text strings (API keys or security tokens.)

accessing credentials

• credentials() - assigns values to one or more environment variables.

environment {
  STRING = credentials('secret-value') # string type
  LOGIN = credentials('login') # login type
}

env.STRING
env.LOGIN - username:password
env.LOGIN_USR - username
env.LOGIN_PSW - password

• withCredentials() {}

steps {
  withCredentials([string(credentialsId:'apiKey', variable: 'API_KEY')]) {
    sh "./build_script.sh ${env.API_KEY}"
  }
}

reference

• https://www.linkedin.com/learning/jenkins-essential-training-17420152/challenge-configure-users-and-permissions

Container orchestration automates the container lifecycle of containerized applications resulting in faster deployments, reduced errors, higher availability, and more robust security. The container lifecycle include:

• Deployment
• Management
• Scaling
• Networking
• Availability

Container orchestration is a critical part of an organization’s orchestration, automation, and response (SOAR) requirements. Some features:

• Defines container images and registry
• Improves provisioning and deployment
• Secure network connectivity
• Ensures availability and performance
• Manages scalability and load balancing
• Resource allocation and scheduling
• Rolling updates and roll backs
• Conducting health checks and automated error handling

Benefits of Container Orchestration:

• Increased Productivity
• Faster Deployment
• Reduced Costs
• Stronger Security
• Easier Scaling
• Faster Error Recovery

How it works?

• uses configuration files written in YAML or JSON uses to find resources, establish a network and store logs
• automatically schedules the deployment of new container to a cluster, finds the right host based on predefined settings or restrictions
• manages the container’s lifecycle based on specifications in the configuration file includes system parameters (like CPU and memory), and file parameters (like proximity and file metadata)
• supports scaling and enhances productivity, through automation

Tools

• Apache Mesos’s Marathon framework - open-source cluster manager, scales container infrastructure by automating the bulk of management and monitoring tasks
• HashiCorp’s Nomad - free, open-source cluster management and scheduling tools, supports various app types on all major operating systems
• Docker Swarm - open-source container orchestration platform, automates deployment of containerized apps, specifically to work with Docker Engine and other Docker tools
• zGoogle’s Kubernetes - standard for open-source container orchestration platforms, robust feature set, broadly supported, maintained by Cloud Native Computing Foundation (CNCF)

Kubernetes

• referred as (k8s), is system for automating deployment,scaling, and management of containerized applications.
• automates a host of container management tasks including
  • deployment,
  • storage provisioning,
  • load balancing and scaling,
  • service discovery, and
  • “self-healing”— the ability to restart, replace or remove a failed container.

Is Not

• traditional, all-inclusive as a service (PaaS)
• rigid or opinionated but a flexible model that supports a diverse variety of workloads and containerized applications.
• does not provide CI/CD pipelines to deploy source code or build applications
• does not prescribe logging, monitoring, or alerting solutions
• does not provide built-in middleware, databases, or other services

Capabilities

• Automated rollouts of changes to application or configuration, health monitoring, ensures instances are running, and rolling back changes
• Storage orchestration that mounts a chosen storage system including local storage, network storage, or public cloud
• Horizontal scaling of workloads based on metrics, or via commands
• Automated bin packing that increases utilization and cost savings using a mix of critical and best-effort workloads. Automated bin packing performs container auto-placement based on resource requirements and conditions without sacrificing high availability (HA)
• Secret and configuration management of sensitive information including passwords, OAuth tokens, and SSH keys, and handles deployments and updates to secrets and configuration without rebuilding images
• assigns both dual-stack IPv4 and IPv6 addresses to Pods and Services
• manages batch execution and continuous integration workloads and automatically replaces failed containers
• self-heals failing or unresponsive containers, exposes containers to clients only if healthy and running
• discovers Pods using IP addresses or a DNS name, and load balances traffic for better performance and high availability
• easily extensible by adding or providing additional features to Kubernetes cluster without modifying source code

Concepts

Ecosystem

Architecture

• A deployment of Kubernetes is called a Kubernetes cluster, is a cluster of nodes that runs containerized applications
• Each cluster has one master node (the Kubernetes Control Plane) and one or more worker nodes

Control Plane

• The control plane maintains the intended cluster state by making decisions about the cluster and detecting and responding to events in the cluster

An example of a decision made by the control plane is the scheduling of workloads. An example of responding to an event is creating new resources when an application is deployed.

kube-api-server

• the Kubernetes API server exposes the Kubernetes API. The API server serves as the front-end for the control plane
• all communication in the cluster utilizes this API, kube-apiserver which is designed to scale horizontally—by deploying more instances and balance traffic between them

An example the Kubernetes API server accepts commands to view or change the state of the cluster.

etcd

• highly available, distributed key value store that contains all the cluster data
• stores deployment configuration data, desired state, and meta data in a way that can be accessed in a common location
• defines the state in a Kubernetes cluster, and the system works to bring the actual state to match the desired state

kube-scheduler

• assigns newly created Pods to nodes, means it determines where workloads should run within the cluster
• selects the most optimal node according to Kubernetes scheduling principles, configuration options, and available resources

kube-controller manager

• runs all the controller processes that monitor the cluster state
• ensure the actual state of a cluster matches the desired state

cloud-controller manager

• runs controllers that interact with the underlying cloud providers
• effectively link clusters into a cloud provider’s API
• allows both Kubernetes and the cloud providers to evolve freely without introducing dependencies on the other

Worker Nodes

• are the worker machines in a Kubernetes cluster, user applications are run on nodes, can be virtual or physical machines
• are not created by Kubernetes itself, but rather by the cloud provider, allowing Kubernetes to run on a variety of infrastructures
• are then managed by the control plane and contain the services necessary to run applications
• include pods, which are the smallest deployment entity in Kubernetes; pods include one or more containers
• containers share all the resources of the node and can communicate among themselves

kubelet

• is the most important component of a worker node, this controller communicates with the kube-apiserver to receive new and modified pod specifications
• ensure that the pods and their associated containers are running as desired
• reports to the control plane on the pods’ health and status
• to start a pod, the kubelet uses the container runtime

Container runtime

• is responsible for downloading images and running containers
• rather than providing a single container runtime, Kubernetes implements a Container Runtime Interface that permits pluggability of the container runtime
• Docker is likely the best-known runtime, Podman and Cri-o are two other commonly used container runtimes

kube-proxy

• is a network proxy that runs on each node in a cluster
• maintains network rules that allow communication to Pods running on nodes — in other words, communication to workloads running on your cluster
• This communication can come from within or outside of the cluster

Objects Terms

• software object - a bundle of data that has an identity, a state, and a behavior.
  • Example include variables, data structures, and specific functions.
• entity - a person, place, or thing with an identity and associated data.
  • Example in banking, a customer account is an entity.
• persistent - that lasts even if there is a server failure or network attack.
  • Example is persistent storage.

Kubernetes Objects

• Kubernetes objects are persistent entities. Example: Pods, Namespaces, ReplicaSets, Deployments, and more.
• Kubernetes objects consist of two main fields:
  • object spec - provided by user and defines desired state
  • status - provided by Kubernetes and defines current state
• Kubernetes works towards matching the current state to the desired state.
• To work with these objects, use the Kubernetes API directly with the client libraries, and the kubectl command-line interface, or both

Labels and selectors

• Labels are key/value pairs attached to objects.
  • Intended for identification of objects.
  • Not unique. Many objects can have the same labels.
  • Helps to organize and group objects.
• Label selectors are the core grouping method in Kubernetes.
  • Identify and group a set of objects.

Namespaces and names

• Namespaces provide a mechanism for isolating groups of resources within a single cluster.
• This is useful when teams share a cluster for cost-saving purposes or for maintaining multiple projects in isolation.
• There are different patterns of working with namespaces.
  • There may be only one namespace for a user who works with one team which only has one project that is deployed into a cluster.
  • Alternatively, there may be many teams or projects, or users with different needs, where additional namespaces may be created.
• Namespaces provide a scope for the names of objects
  • Each object must have a unique name
  • Names are uniques for that resource type within that namespace

Pods

• is the simplest unit in Kubernetes
• represents a process or a single instance of an application running in the cluster.
• encapsulates one or more containers
• replicating a pod serves to scale applications horizontally
• YAML files are used to define the objects that you want to create.
• The YAML files shown defines a simple pod.
• A PodSpec must contain at least one container.

apiVersion: v1
kind: Pod # kind of object to be created
metadata:
  name: nginx
spec: # provides the appropriate fields for the object
  containers: # container and will run in this Pod
    - name: nginx # name of container
      image: nginx:1.7.9 # image that will run in Pod
      ports: # port that container exposes
        - containerPort: 80

ReplicaSet

• is a set of identical running replicas of a Pod that are horizontally scale
• the configuration files for a ReplicaSet and a Pod are different from each other
• the replicas field specifies the number of replicas that should be running at any given time.

apiVersion: v1
kind: ReplicaSet
metadata:
  name: nginx-replicaset
  labels:
    app: nginx
spec:
  replica: 3 # creates/deletes Pods to meet the desired number of replicas
  selector: # selector to identify which pods it can acquire
    matchLabels:
      app: nginx # same as template labels below
    template: # defines the Pods that should be created by the ReplicaSet
      metadata:
        labels:
          app: nginx # same as matchLabels above
      spec:
        containers:
          - name: nginx
            image: nginx:1.7.9
            ports:
              - containerPort: 80

Note: Creating ReplicaSets directly is not recommended. Instead, create a Deployment, which is a higher-level concept that manages ReplicaSets and offers more features and better control.

Deployment

• is a higher-level object that provides updates for both Pods and ReplicaSets.
  • run multiple replicas of an application using ReplicaSets
  • are suitable for stateless applications
• for stateful applications, Stateful Sets are used
• One key feature provided by Deployments but not by ReplicaSets is rolling updates
• A rolling update scales up a new version to the appropriate number of replicas and scales down the old version to zero replicas
• The ReplicaSet ensures that the appropriate number of Pods exist, while the Deployment orchestrates the roll out of a new version

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.7.9
          ports:
            - containerPort: 80

Service

• is a REST object, like Pods
• are a logical abstraction for a set of Pods in a cluster
• provide policies for accessing the Pods and cluster
• act as a load balancer across the Pods
• is assigned a unique IP address for accessing applications deployed on Pods
• eliminates the need for a separate service discovery process.

Service Properties:

• supports multiple protocols such as TCP, which is the default protocol, UDP, and others
• supports multiple port definitions
  • The port number with the same name can vary in each backend Pod
• can have an optional selector and can optionally map incoming ports to a targetPort

Why a Service is needed:

• is needed because Pods in a cluster are volatile, can be destroyed and new Pods can be created at any time
• this volatility leads to discoverability issues because of changing IP addresses
• it keeps track of Pod changes and exposes a single IP address or a DNS name
• utilizes selectors to target a set of Pods

Service types

ClusterIP

• is the default and most common service type
• assigns a cluster-internal IP address to the ClusterIP Service that makes the Service only reachable within the cluster.
• cannot make requests to Service from outside the cluster.
• You can set the ClusterIP address in the Service definition file
• provides Inter-service communication within the cluster

NodePort

• an extension of ClusterIP Service
• creates and routes the incoming requests automatically to the ClusterIP Service
• exposes the Service on each Node’s IP address at a static port
• exposes a single Service with no load-balancing requirements for multiple services.

Note that for security purposes, production use is not recommended.

External Load Balancer (ELB)

• an extension of the NodePort Service
• creates NodePort and ClusterIP Services automatically
• integrates and automatically directs traffic to the NodePort Service with a cloud provider’s ELB
• To expose a Service to the Internet, you need a new ELB with an IP address
• You can use a cloud provider’s ELB to host your cluster.

External Name

• maps to a DNS name and not to any selector
• requires a spec.externalName parameter
• maps the Service to the contents of the externalName field that returns a CNAME record and its value
• can use an External name to create a Service that represents external storage and enable Pods from different namespaces to talk to each other

Ingress

• is an API object that, when combined with a controller, provides routing rules to manage external users’ access to multiple services in a Kubernetes cluster
• in production, Ingress exposes applications to the Internet via port 80 (for HTTP) or port 443 (for HTTPS)
• while the cluster monitors Ingress, an external Load Balancer is expensive and is managed outside the cluster
• acts as a supervisor for external access, exposing routes from outside the cluster to internal services
• adheres to rules defined on the Ingress resource to regulate traffic routing.

DaemonSet

• is an object that makes sure that Nodes run a copy of a Pod
• As nodes are added to a cluster, Pods are added to the nodes
• Pods are garbage collected when removed from a cluster
• If you delete a DaemonSet, all Pods are removed
• are ideally used for storage, logs, and monitoring nodes

StatefulSet

• is an object that manages stateful applications
• manages deployment and scaling of Pods
• provides guarantees about the ordering and uniqueness of Pods
• maintains a sticky identity for each Pod request
• provides persistent storage volumes for your workloads

Job

• job creates Pods and tracks the Pod completion process
• are retried until completed
• Deleting a job will remove the created Pods
• Suspending a Job will delete its active Pods until the job resumes
• can run several Pods in parallel
• a CronJob is regularly used to create Jobs on an iterative schedule

Kubectl - The Kubernetes Command Line Tool

kubectl [command] [type] [name] [flags]

  [command] = operation performed (create, get, apply, delete)
  [type]    = resource type (pod, deployment, replicaset)
  [name]    = resource name (if applicable)
  [flags]   = special options or modifiers that override default values

kubectl get services
kubectl get pods --all-namespaces
kubectl get deployment my-dep
kubectl get pods -o wide

Key command types:

• Imperative commands

# No audit trails
kubectl run nginx --image nginx

• Imperative object configuration

# Inconsistency if changed configuration aren't merged
kubectl create -f nginx.yaml

• Declarative object configuration

# operation are identified by Kubectl, not the user
# ideal for production systems
kubectl apply -f nginx/

Kubernetes Anti-patterns

Avoid baking configuration in container images

Containers offer the advantage of using a consistent image throughout the production process. To achieve adaptability across different environments, building images without embedding configuration directly into containers is essential.

Issue: Problems arise when images contain environment-specific artifacts that deviate from the tested version, necessitating image rebuilds and risking inadequately tested versions in production. Identification of environment-dependent container images involves spotting features like hardcoded IP addresses, passwords, and environment-specific prefixes.

Best practice: Create generic images independent of specific runtime settings. Containers enable the consistent use of a single image throughout the software lifecycle, promoting simplicity and efficiency.

Separate application and infrastructure deployment

Infrastructure as Code (IaC) allows defining and deploying infrastructure like writing code. While deploying infrastructure through a pipeline is advantageous, separating infrastructure and application deployment is crucial.

Issue: Using a single pipeline for both infrastructure and application deployment leads to resource and time wastage, especially when changes in application code outpace infrastructure changes.

Best practice: Split infrastructure and application deployment into separate pipelines to optimize efficiency and resource utilization.

Eliminate specific order in deployment

Maintaining application stability despite delays in dependencies is crucial in container orchestration. Unlike traditional fixed startup orders, Kubernetes and containers initiate components simultaneously.

Issue: Challenges arise when poor network latency disrupts communication, potentially causing pod crashes or temporary service unavailability.

Best practice: Proactively anticipate failures, establish frameworks to minimize downtime, and adopt strategies for simultaneous component initiation to enhance application resilience.

Set memory and CPU limits for pods problem

The default Kubernetes setting without specified resource limits allows an application to potentially monopolize the entire cluster, causing disruptions.

Best practice: Establish resource limits for all applications, conduct a thorough examination of each application’s behavior under various conditions, and strike the right balance to optimize cluster performance.

Avoid pulling the latest tag in production problem

Using the “latest” tag in production leads to unintended pod crashes as images are pulled down sporadically, lacking specificity.

Best practice: Use specific and meaningful image tags, maintain the immutability of container images, store data outside containers in persistent storage, and avoid modifying containers post-deployment for safer and more repeatable deployments.

Segregate production and non-production workloads problem

Relying on a single cluster for all operational needs poses challenges. Security concerns arise from default permissions and complications with non-namespaced Kubernetes resources.

Best practice: Establish a second cluster exclusively for production purposes, avoiding complexities associated with multi-tenancy. Maintain at least two clusters—one for production and one for non-production.

Refrain from ad-hoc deployments with kubectl edit/patch problem

Configuration drift occurs when multiple environments deviate due to unplanned deployments or changes, leading to failed deployments.

Best practice: Conduct all deployments through Git commits for comprehensive history, precise knowledge of cluster contents, and easy recreation or rollback of environments.

Implement health checks with liveness and readiness probes problem

Neglecting health checks can lead to various issues. Overly complex health checks with unpredictable timings can cause internal denial-of-service attacks within the cluster.

Best practice: Configure health probes for each container, use liveness and readiness probes, and prioritize robust health checks for reliable application responsiveness.

Prioritize secret handling and use vault problem

Embedding secrets directly into containers is poor practice. Using multiple secret handling methods or complex injection mechanisms can complicate local development and testing.

Best practice: Use a consistent secret handling strategy, consider HashiCorp Vault, handle secrets uniformly across environments, and pass them to containers during runtime for enhanced resilience and security.

Use controllers and avoid running multiple processes per container problem

Directly using pods in production poses limitations. Pods lack durability, automatic rescheduling, and data retention guarantees. Running multiple processes in a single container without controllers can lead to issues.

Best practice: Utilize Deployment with a replication factor, define one process per container, use multiple containers per pod if necessary, and leverage workload resources like Deployment, Job, or StatefulSet for reliability and scalability.

• ensures the right number of pods are always up and running
• provide high availability through redundancy
• adds or deletes pods for scaling
• always tries to match the actual state of the replicas to the desired state
• replaces failing pods or deletes additional pods to maintain the desired state
• supersedes ReplicaControllers
• Deployments manage ReplicaSets, send pods declarative updates

Create ReplicaSet from scratch

# replicaset.yaml
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: hello-kubernetes
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-kubernetes
    template:
      metadata:
        labels:
          app: hello-kubernetes
      spec:
        containers:
          - name: hello-kubernetes
            image: paulbouwer/hello-kubernets:1.5
            ports:
              -containerPort: 8080

# Outputs ReplicaSet was created
kubectl create -f replicaset.yaml
# Confirm by using "get pods"
kubectl get pods
# Output shows details about ReplicaSet and pod created
kubectl get rs

Note: Creating a Deployment that includes a ReplicaSet is recommended over creating a standalone ReplicaSet.

# Create deployment
kubectl create -f deployment.yaml
kubectl get pods
kubectl get deploy

# Scale deployment
kubectl scale deploy hello-kubernetes --replicas=3

# Should shows 3 pods running
kubectl get pods

# Delete pod
kubectl delete pod hello-kubernetes-2324343-5mflw
# Here desired state (--replicas=3) does not
# match the actual state [running pods=2]
# and deleted pod replaced by new one

# Create pod extra pod
kubectl create pod hello-kubernetes-122323
# Here desired state (--replicas=3) does not
# match the actual state [running pods=4]
# and new pod is marked for deletion and removed automatically

Autoscaling

• ReplicaSets provide a good start for scaling, but you don’t always want 10 instances of your resource running.
• Kubernetes autoscaling helps optimize resource usage and costs by automatically scaling a cluster in line with demand as needed.
• Kubernetes enables autoscaling at two different layers:
  • the cluster or node level
  • the pod level.
• Three types of autoscalers are available in Kubernetes:
  • The Horizontal Pod Autoscaler (or HPA)
  • Vertical Pod Autoscaler (or VPA)
  • Cluster Autoscaler (or CA)

To create autoscaling:

kubectl get pods
kubectl get rs

kubectl autoscale deploy hello-kubernetes --min=2 --max=5 --cpu-percent=50

kubectl describe rs hello-kubernetes-<hash>

• List the current number or state of pods.
• A ReplicaSet is automatically created when you create a deployment. In order to autoscale, you simply use the autoscale command with the requisite attributes.
• Min is the number of minimum pods – notice that we have changed the value of “Min” to 2.
• Max is the number of maximum pods.
• CPU-percent acts as a trigger that tells the system to create a new pod when the CPU usage reaches 50% across the cluster.
• In the background, the deployment still uses the ReplicaSet to scale up and down.
• The describe command shows the number of replicas in the “autoscaled” ReplicaSet.

The Horizontal Pod Autoscaler (or HPA)

• adjusts the number of replicas of an application by increasing or decreasing the number of pods.
• automatically updates a workload resource (like a deployment) by horizontally scaling the workload to match the demand
• Horizontal scaling, or “scaling out,” automatically increases or decreases the number of running pods as application usage changes.
• uses a cluster operator that sets targets for metrics like CPU or memory utilization and the maximum and minimum desired number of replicas
• even though you can create an HPA autoscaler from scratch, you should use the autoscale command instead.

kubectl get hpa

For example,

• the system load is low early in the morning, so one pod is sufficient. The HPA autoscales the workload resource to meet usage demand.
• By 11am, peak load drives a need for three pods, so the HPA autoscales the workload resource to meet usage demand.
• Usage drops in the afternoon, so the third pod is marked for deletion and removed.
• And usage drops even lower by 5pm, so another pod is marked for deletion and removed.

# another way to enable autoscaling is to manually
# create the HPA object from a YAML file.
apiVersion: autoscaling/v1
kind: HorizontalPodAutoScaler
metadata:
  name: hello-kubernetes
  namespace: default
spec: # can set the minimum and maximum number of pods.
  maxReplicas: 5
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hello-kubernetes
  # The CPU-percent flag shows up as “targetCPUUtilizationPercentage”.
  targetCPUUtilizationPercentage: 10

Vertical Pod Autoscaler (or VPA)

• adjusts the resource requests and limits of a container by increasing or decreasing the resource size or speed of the pods.
• A best practice is to scale horizontally, but there are some services you may want to run in a cluster where horizontal scaling is impossible or not ideal.
• Vertical scaling, or “scaling up,” refers to adding more resources to an existing machine.
• lets you scale a service vertically within a cluster.
• uses a cluster operator sets targets for metrics like CPU or memory utilization, similar to an HPA.
• The cluster then reconciles the size of the service’s pod or pods based on their current usage and the desired target.

For example,

• the system load is low early in the morning, so system resources used by the pod are low.
• By 11am, peak load drives a need for more capacity.
• The VPA autoscales the pod by adding more system resources (CPU and memory) to meet the demand.
• Usage drops in the afternoon, so the pod is autoscaled to use fewer system resources.
• And usage drops even lower by 5pm, so the pod is autoscaled further to match the 7am levels.

Note: You should not use VPAs with HPAs on resource metrics like CPU or memory. However, you can use them together on custom or external metrics.

Cluster Autoscaler (or CA)

• adjusts the number of nodes in the cluster when pods fail to schedule, or demand increases or decreases in relation to the existing nodes’ capacity
• autoscales the cluster itself, increasing and decreasing the number of available nodes that pods can run on.
• Pods are autoscaled using HPA or VPA, but when the nodes themselves are overloaded with pods.
• can use a CA to autoscale the nodes so that the pods can rebalance themselves across the cluster.

For example,

• the system load is low early in the morning, so existing nodes can handle the load.

• When demand increases, new pod requests come in, and the CA autoscales the cluster by adding a new node and pods to meet the demand.

• By 11 am, peak load brings the new node to full capacity.

• When usage drops in the afternoon, unused pods are marked for deletion and removed.

• And when usage drops even lower by 5 pm, all pods in the new node are marked for deletion and removed.

• And then the node itself is marked and removed.

• cluster autoscaler ensures there is always enough compute power to run your tasks, and that you aren’t paying extra for unused nodes.

• clusters may have periods where all batch processing jobs are complete, and the new batch doesn’t start until later in the day.

• Each autoscaler type is suitable in specific scenarios, so you should analyze the pros and cons of each to find the best choice.

• Using a combination of all three types ensures that

  • services run stably at peak load times, and
  • costs are minimized in times of lower demand.

Deployment Strategies

• defines an application’s lifecycle that achieves and maintains the configured state for objects and applications in an automated manner. Effective deployment strategies minimize risk.

Kubernetes deployment strategies are used to:

• Deploy, update, or rollback ReplicaSets, Pods, Services, and Applications
• Pause/Resume Deployments
• Scale Deployments manually or automatically

There are are six types of deployment strategies:

Recreate

• is simplest deployment strategy
• has short downtime between the shutdown of existing and deployment and the new deployment
• Pods running the live version of the application are all shut down simultaneously, and a new version of the application is deployed on newly created Pods.
• The rollback process is completed in reverse order.

Rolling (ramped)

• each Pod is updated one at a time, single v1 Pod is replaced and is updated in this way until all Pods are v2.
• During rollback process, there is hardly any downtime since users are directed to either version.

Blue/green

• the blue environment is the live version of the application.
• the green environment is an exact copy that contains the deployment of the new version of the application.
• the green environment is created identical to the current production environment and is thoroughly tested.
• once all changes, bugs, and issues are addressed, user traffic is switched from the blue environment to the green environment.
• to perform a rollback, switch the environment back.

Canary

• the new version of the application is tested using a small set of random users alongside the current live version of the application.
• once the version of the application is successfully tested, it is then rolled out to all users.
• rollback has no downtime since few users are exposed to the new version.

A/B testing

• known as split testing, evaluates two versions of an application (version A and version B).
• each version has features that cater to different sets of users.
• can select which version is best for global deployment based on user interaction and feedback.

Shadow

• a “shadow version” of the application is deployed alongside the live version.
• user requests are sent to both versions, and both handle all requests, but the shadow version does not forward responses back to the users.
• lets developers see how the shadow version performs using real-world data without interrupting user experience.

Comparison

To create a good strategy:

• Consider the product type and the target audience
• Shadow and canary strategies use live user requests, as opposed to using a sample of users.
• The A/B testing strategy is useful if the version of the application requires minor tweaks or UI feature changes.
• The blue/green strategy is useful if your version of the application is complex or critical and needs proper monitoring with no downtime during deployment.
• The canary strategy is a good choice if you want zero downtime and are comfortable exposing your version of the application to the public.
• A rolling strategy gradually deploys the new version of the application. There is no downtime, and it is easy to roll back.
• The recreate strategy is a good choice if the application is not critical and users aren’t impacted by a short downtime.

Rolling updates

• are automated updates that occur on a scheduled basis.
  • They roll out automated and controlled app changes across pods,
  • Work with pod templates like deployments, and
  • allow for rollback as needed.

To prepare your application to enable rolling updates,

• Step 1: Add liveness probes and readiness probes to deployments. That way deployments are appropriately marked as ‘ready.’

livenessProbe:
  httpGet:
    path: /
    port: 9080
  initialDelaySeconds: 300
  periodSeconds: 15
readinessProbe:
  httpGet:
    path: /
    port: 9080
  initialDelaySeconds: 45
  periodSeconds: 5

• Step 2: add a rolling update strategy to the YAML file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-test
spec:
  replicas: 10 # creating a deployment with 10 pods.
  selector:
    matchLabels:
      service: http-server
  # to wait a few seconds before moving to the next pod in the rollout stage
  minReadySeconds: 5
  progressDeadlineSeconds: 600
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # strategy is to have at-least 50% of the pods always available
      # for a zero-downtime system, set the maxUnavailable to 0.
      # Setting the maxSurge to 100% would double the number of pods
      # and create a complete replica before taking the original set down
      # after the rollout is complete.
      maxUnavailable: 50%
      # there can only be 2 pods added to the 10 you defined earlier
      maxSurge: 2

Rolling out an application update

• Assume, You have a deployment with three pods in your ReplicaSet.
• Your application displays the message, “Hello world!”
• Your client has submitted a new request, and you have a new image for your application with a different message, “Hello world v2!’ to your users.

// Older
let port = process.env.PORT || 8080;
let message = process.env.MESSAGE || 'Hello world!';
// Newer
let port = process.env.PORT || 8080;
let message = process.env.MESSAGE || 'Hello world v2!';

• But you cannot have any downtime in your application.
• First, you need to build, tag, and upload this new image to Docker Hub.

docker build -t hello-kubernetes .
docker tag hello-kubernetes cham11ng/hello-kubernetes:2.0
docker push cham11ng/hello-kubernetes:2.0
# Your new software has been dockerized
# and then updated to Docker Hub with the name
# and tag cham11ng/hello-kubernetes:2.0”.

• Second, apply the new image to your deployment.

kubectl get deployments
# sets the image flag to the updated tag image on Docker Hub
kubectl set image deployments/hello-kubernetes \
  hello-kubernetes=cham11ng/hello-kubernetes:2.0

  deployment.extensions/hello-kubernetes image updated

# observe that version 2 deployment has been rollout
kubectl rollout status deployments/hello-kubernetes
  deployment "hello-kubernetes" successfully rolled out.

• Third, you can roll back your changes using the “rollout undo” command if there are errors in a deployment or the clients can change their minds.

kubectl rollout undo deployments/hello-kubernetes
  deployment.extensions/hello-kubernetes rolled back

kubectl get pods
# we see three new pods that were created as port of the rollback
# and can see some pods in terminating status

How rolling works

All-at-once

In an all-at-once rollout, all v1 objects must be removed before v2 objects can become active.

• Here you see version 1 of an app with three pods running that users can access.
• When version 2 is deployed, new pods are created.
• The version 1 pods are marked for deletion and removed and user access is blocked.
• Once the version 1 pods are removed, the version 2 pods become active and user access is restored.
• Notice the time lag between deployment and pod updates.

In an all-at-once rollback, all v2 objects must be removed before v1 objects can become active.

• Here you see version 2 of an app with three pods running that users can access.
• When version 1 of the app is deployed, new pods are created.
• The version 2 pods are marked for deletion and removed and user access is blocked.
• Once the version 2 pods are removed, the version 1 pods become active and user access is restored.

One-at-a-time

In a one-at-a-time rollout, the update is staggered so user access is not interrupted.

• Here you see version 1 of an app with three running pods that users can access.
• When version 2 is deployed, a new pod is created.
• The first version 1 pod is marked for deletion and removed, and the v2 pod becomes active.
• Now, a second v2 pod is created, and the second version 1 pod is marked for deletion and removed, and the second v2 pod becomes active.
• Then, a third v2 pod is created, and the third version 1 pod is marked for deletion and removed. And now the third v2 pod becomes active.

In a one-at-a-time rollback, the update rollback is staggered so user access is not interrupted.

• Here you see version 2 of an app with three running pods that users can access.
• When version 1 of the app is deployed, a new pod is created.
• The first version 2 pod is marked for deletion and removed, and the v1 pod becomes active.
• Now, a second v1 pod is created. The second version 2 pod is marked for deletion and removed, and the second v1 pod becomes active.
• Then, a third v1 pod is created, and the third version 2 pod is marked for deletion and removed. And the third v1 pod becomes active.

ConfigMaps

• helps developers avoid hard coding configuration variables in application code by keeping the configuration variables separate so that any changes in configuration settings do not require code changes.
• is an API object that stores non-confidential data in key-value pairs.
• provides configuration data to pods and deployments so that the configuration data is not hard coded inside the application code
• is meant for non-sensitive information as they do not provide secrecy or encryption.
• The data stored in a ConfigMap is limited and cannot exceed 1 megabyte
  • For larger amounts of data, consider mounting a volume or use a separate database or file service
• has optional data and binaryData fields and no “spec" field in the template
• the Config name must be a valid DNS subdomain name
• A ConfigMap is reusable for multiple deployments, thus decoupling the environment from the deployments themselves!
• Multiple ways to create
  • a ConfigMap by using string literals,
  • by using an existing “properties” or ”key” = “value” file,
  • or by providing a ConfigMap YAML descriptor file. You can use the first and second ways to help create such a YAML file.
• Multiple ways to reference from pod/deployment to consume a ConfigMap
  • reference by using environment variables with the configMapKeyRef attribute
  • by mounting a file using the volumes plugin. (mount as volume)
• Kubernetes applies the ConfigMap to the pod or the deployment just before running the pod or deployment.

Configuration: Environment Variable

• You’ll use the environment variable directly in the YAML file.
• Apply this deployment descriptor to our deployment.
• Here, the message is hard-coded in the descriptor file.

---
spec:
  containers:
    - name: hello-kubernetes
      image: cham11ng/myapp:latest
      ports:
        - containerPort: 8080
      env:
        # is used in the JavaScript file as process.env.MESSAGE
        - name: MESSAGE
          value: 'Hello from config file!'

Configuration: ConfigMap string literal

# provide a ConfigMap is to provide a key-value pair
# in the create ConfigMap command.
kubectl create ConfigMap my-config \
  --from-literal=MESSAGE="hello from first configmap"