Cloud siem icon white

Automate your SOC with Cloud SIEM Get started

Get started

Hemant Jain

Hemant Jain is the founder and owner of Rapidera Technologies, a full service software development shop. He and his team focus a lot on modern software delivery techniques and tools. Prior to Rapidera he managed large scale enterprise development projects at Autodesk and Deloitte.

Posts by Hemant Jain

Blog

Managing Container Data Using Docker Data Volumes

Docker data volumes are designed to solve one of the deep paradoxes of containers, which is this: For the very same reasons that containers make apps highly portable — and, by extension, create more nimble data centers — they also make it hard to store data persistently. That’s because, by design, containerized apps are ephemeral. Once you shut down a container, everything inside it disappears. That makes your data center more flexible and secure, since it lets you spin up apps rapidly based on clean images. But it also means that data stored inside your containers disappears by default. How do you resolve this paradox? There are actually several ways. You could jerry-rig a system for loading data into a container each time it is spun up ( via SSH, for example), then exporting it somehow, but that’s messy. You could also turn to traditional distributed storage systems, like NFS, which you can access directly over the network. But that won’t work well if you have a complicated (software-defined) networking situation (and you probably do in a large data center). You’d think someone would have solved the Docker container storage challenge in a more elegant way by now — and someone has! Docker data volumes provide a much cleaner, straightforward way to provide persistent data storage for containers. That’s what I’ll cover here. Keep reading for instructions on setting up and deploying Docker data volumes (followed by brief notes on storing data persistently directly on the host). Creating a Docker Data Volume To use a data volume in Docker, you first need to create a container to host the volume. This is pretty basic. Just use a command like: docker create -v /some/directory mydatacontainer debian This command tells Docker to create a new container named mydatacontainer based on the Debian Docker image. (You could use any of Docker’s other OS images here, too.) Meanwhile, the -v flag in the command above sets up a storage container in the directory /some/directory inside the container. To repeat: That means the data is stored at /some/directory inside the container called mydatacontainer — not at /some/directory on your host system. The beauty of this, of course, is that we can now write data to /some/directory inside this container, and it will stay there as long as the container remains up. Using a Data Volume in Docker So that’s all good and well. But how do you actually get apps to use the new data volume you created? Pretty easily. The next and final step is just to start another container, using the –volumes-from flag to tell Docker that this new container should store data in the data volume we created in the first container. Our command would look something like this: docker run --volumes-from mydatacontainer --volumes-from debian Now, any data changes made inside the container debian will be saved inside mydatacontainer at the directory /some/directory. And they’ll stay there if you stop debian — which means this is a persistent data storage solution. (Of course, if you stop mycontainervolume, then you’ll also lose the data inside.) You can have as many data volumes as you want, by the way. Just specify multiple ones when you run the container that will access the volumes. Data Storage on Host instead of a container? You may be thinking, “What if I want to store my data directly on the host instead of inside another container?” There’s good news. You can do that, too. We won’t use data storage volumes for this, though. Instead, we’ll run a command like: docker run -v /host/dir:/container/dir -i image This starts a new container based on the image image and maps the directory /host/dir on the host system to the directory /container/dir inside the container. That means that any data that is written by the container to /container/dir will also appear inside /host/dir on the host, and vice versa. There you have it. You can now have your container data and eat it, too. Or something like that. About the Author Hemant Jain is the founder and owner of Rapidera Technologies, a full service software development shop. He and his team focus a lot on modern software delivery techniques and tools. Prior to Rapidera he managed large scale enterprise development projects at Autodesk and Deloitte. Managing Container Data Using Docker Data Volumes is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

Blog

Setting Up a Docker Environment Using Docker Compose

Docker Compose is a handy tool for solving one of the biggest inherent challenges posed by container-based infrastructure. That challenge is this: While Docker containers provide a very easy and convenient way to make apps portable, they also abstract your apps from the host system — since that is the whole point of containers. As a result, connecting one container-based app to another — and to resources like data storage and networking — is tricky. If you’re running a simple container environment, this isn’t a problem. A containerized web server that doesn’t require multiple containers can exist happily enough on its own, for example. But if life were always simple, you wouldn’t need containers in the first place. To do anything serious in your cloud, you will probably want your containers to be able to interact with one another and to access system resources. That’s where Docker Compose comes in. Compose lets you define the containers and services that need to work together to power your application. Compose allows you to configure everything in plain text files, then use a simple command-line utility to control all of the moving parts that make your app run. Another way to think of Compose is as an orchestrator for a single app. Just as Swarm and Kubernetes automate management of all of the hundreds or thousands of containers that span your data center, Compose automates a single app that relies on multiple containers and services. Using Docker Compose Setting up a Docker environment using Compose entails multiple steps. But if you have any familiarity with basic cloud configuration — or just text-based interfaces on Unix-like operating systems — Compose is pretty simple to use. Deploying the tool involves three main steps. First, you create a Dockerfile to define your app. Second, you create a Compose configuration file that defines app services. Lastly, you fire up a command-line tool to start and control the app. I’ll walk through each of these steps below. Step 1. Make a Dockerfile This step is pretty straightforward if you are already familiar with creating Docker images. Using any text editor, open up a blank file and define the basic parameters for your app. The Dockerfile contents will vary depending on your situation, but the format should basically look like this: FROM [ name of the base Docker image you're using ] ADD . [ /path/to/workdir ] WORKDIR [ directory where your code lives ] RUN [ command(s) to run to set up app dependencies ] CMD [ command you'll use to call the app ] Save your Dockerfile. Then build the image by calling docker build -t [ image name ] Step 2. Define Services If you can build a Dockerfile, you can also define app services. Like the first step, this one is all about filling in fields in a text file. You’ll want to name the file docker-compose.yml and save it in the workdir that you defined in your Dockerfile. The contents of docker-compose.yml should look something like this: version: '2' services: [ name of a service ]: build: [ code directory ] ports: - "[ tcp and udp ports ] " volumes: - .: [ /path/to/code directory ] depends_on: - [ name of dependency image ] [ name of another service ]: image: [ image name ] You can define as many services, images and dependencies as you need. For a complete overview of the values you can include in your Compose config file, check out Docker’s documentation. Don’t forget that another cool thing you can do with Compose is configure log collection using Powerstrip and the Sumo Logic collector container. Step 3. Run the app Now comes the really easy part. With your container image built and the app services defined, you just need to turn the key and get things running. You do that with a command-line utility called (simply enough) docker-compose. The syntax is pretty simple, too. To start your app, call docker-compose up from within your project directory. You don’t need any arguments (although you can supply some if desired; see below for more on that). As long as your Dockerfile and Compose configuration file are in the working directory, Compose will find and parse them for you. Even sweeter, Compose is smart enough to build dependencies automatically, too. After being called, docker-compose will respond with some basic output telling you what it is doing. To get the full list of arguments for docker-compose, call it with the help flag: docker-compose —help When you’re all done, just run (you guessed it!) docker-compose down to turn off the app. Some Docker Compose Tips If you’re just getting started with Compose, knowing about a few of the tool’s quirks ahead of time can save you from confusion. One is that there are multiple ways to start an app with Compose. I covered docker-compose up above. Another option is docker-compose run. Both of these commands do the same general thing — start your app — but run is designed for starting a one-time instance, which can be handy if you’re just testing out your build. up is the command you want for production scenarios. There’s also a third option: docker-compose start. This call only restarts containers that already exist. Unlike up, it doesn’t build the containers for you. Another quirk: You may find that Compose seems to hang or freeze when you tell it to shut down an app using docker-compose stop. Panic not! Compose is not hanging. It’s just waiting patiently for the container to shut down in response to a SIGTERM system call. If the container doesn’t shut down within ten seconds, Compose will hit it with SIGKILL, which should definitely shut it down. (If your containers aren’t responding to standard SIGTERM requests, by the way, you may want to read more about how Docker processes signals to figure out why.) That’s Compose in a nutshell — or about a thousand words, at least. For all of the nitty-gritty details, you can refer to Docker’s Compose reference guide. Setting Up a Docker Environment Using Docker Compose is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.

Blog

How to Configure a Docker Cluster Using Swarm

Blog

Managing Containers with Docker Shipyard

Blog

LXC and LXD: Explaining Linux Containers

Blog

Container Orchestration with Mesos Marathon

Marathon, Apache Mesos’s framework for managing containers, can make your life much easier. (That may be surprising when you consider that the first guy to run a marathon died at the end of it, at least according to the ancient Greek story, but I digress.) Like other orchestration solutions such as Kubernetes and Docker Swarm, Marathon allows you to scale your container infrastructure by automating most of the management and monitoring tasks. Mesos Marathon has evolved into a sophisticated and feature-rich tool. You could literally write a book about it. (I’m betting someone is in the midst of that.) That means I can’t fit everything there is to know about Marathon into a single blog post. But I can go over the essentials, so I’ll explain when you should use Marathon, and introduce the basics of the tool. Is Marathon Right for You? The container orchestration field is now very large, with Swarm and Kubernetes representing only two of the most popular choices. (There are several others to boot, as this non-exhaustive list shows.) So you’re probably wondering why you would choose Marathon in the first place, given that you have so many other options. I’d be crazy if I said Marathon was the right tool for every cluster. It’s not. But it does have some features that set it apart and make it an ideal choice for certain container-orchestration situations. Marathon’s key distinguishing features include: Ultra-high availability. Marathon lets you run multiple schedulers at the same time so that if one goes down, the system keeps ticking. Swarm and Kubernetes are also designed to be highly available, but Marathon takes this to an extreme. Lots of ways to interact with and manage the service. It has a built-in Web interface. (Swarm and Kubernetes have Web UIs, too, but they’re not necessarily part of the core package.) It has multiple CLI clients (although these are add-ons, which you have to install separately). And it offers a rich API. In other words, it gives you lots of options for managing or scripting the tool in many different, complex ways. Application “health checks.” This feature gives you detailed information about the status of applications that Marathon is managing. If you like performance monitoring a lot, you’ll like Marathon. Easy to run locally for development purposes (in contrast to, for example, Kubernetes). Last but not least, I think it’s fair to say that Marathon is a more mature tool than most other container orchestration solutions. It has been around longer and has more features. Swarm, Kubernetes and the rest of the crowd will probably catch up (and may well end up with more features than Marathon offers). But if you want an orchestration tool that is completely mature today, Marathon is an excellent choice. Installing Marathon One of the things I like most about Marathon is how easy it is to install. Whereas some other orchestration tools require you to use scripts to download, build and install, you can easily grab the Marathon source with a straight-forward Git clone, then make an executable JAR using the distribution-specific scripts that ship with the source. In other words, run these commands to install Marathon on any GNU/Linux distribution: git clone https://github.com/mesosphere/marathon.git cd marathon sbt assembly ./bin/build-distribution The only potentially tricky part of these instructions (I say “potentially” because it’s not actually hard if you’re familiar with Scala applications) is the sbt tool. If you don’t already have sbt on your system, you can install it by following these instructions. Everything else is pretty straightforward. Running Marathon There are two different ways to use Marathon: locally for testing purposes, or in production. I’ll outline each method below. Running Marathon Locally As noted, a really cool thing about Marathon is that it’s easy to run locally, in case you want to play around with the UI or do other forms of testing. This feature sets it apart from a tool like Kubernetes, which you can’t run locally without resorting to some tricks (like setting up a single-node cluster through Docker running over localhost, or a resource-heavy testing environment like Vagrant). To run Marathon locally, just launch it from the CLI in local mode with a command like: ./bin/start --master local --zk zk://localhost:2181/marathon That’s all it takes to get the Web UI up and running at http://localhost:8080. (I told you the Web interface on Marathon was extremely easy.) Running Marathon in a Production Cluster Running Marathon in a production environment entails creating a Mesos cluster with Docker. Fortunately, it’s pretty simple if you use pre-built containers from Docker Hub. These are the basic steps: First, create a ZooKeeper container. You need this to take advantage of Marathon’s high-availability mode. Here’s the command: docker run -d -p 2181:2181 -p 2888:2888 -p 3888:3888 garland/zookeeper Second, start a Mesos master container, so you can build your cluster: docker run --net="host" \ -p 5050:5050 \ -e "MESOS_HOSTNAME=localhost" \ -e "MESOS_IP=localhost" \ -e "MESOS_ZK=zk://localhost:2181/mesos" \ -e "MESOS_PORT=5050" \ -e "MESOS_LOG_DIR=/var/log/mesos" \ -e "MESOS_QUORUM=1" \ -e "MESOS_REGISTRY=in_memory" \ -e "MESOS_WORK_DIR=/var/lib/mesos" \ -d \ garland/mesosphere-docker-mesos-master If your Docker server is on a different machine than the one you’re working from, replace “localhost” in the command above with the IP of the Docker server. Third, start Marathon: docker run \ -d \ -p 8080:8080 \ garland/mesosphere-docker-marathon --master zk://${HOST_IP}:2181/mesos --zk zk://${HOST_IP}:2181/marathon Again, replace “localhost” with the Docker server IP address as needed. Fourth, add a Mesos slave to your cluster: docker run -d \ --name mesos_slave_1 \ --entrypoint="mesos-slave" \ -e "MESOS_MASTER=zk://${HOST_IP}:2181/mesos" \ -e "MESOS_LOG_DIR=/var/log/mesos" \ -e "MESOS_LOGGING_LEVEL=INFO" \ garland/mesosphere-docker-mesos-master:latest And once again, replace “localhost” as appropriate. That’s all. Now, you can access the Mesos and Marathon Web UIs at http://localhost:5050 and http://localhost:8080, respectively. (If the Docker host is on a different machine, use its IP instead of localhost.) You can also use any of the various CLI clients for controlling Marathon. See this page (under section “Marathon Clients”) for a list. Additional Resources If you’re interested in learning more about topics like orchestration, composition and clustering in Docker, see Michael Floyd’s blog post explaining how containerization enables DevOps teams. If you’d like to learn more about docker logging, check out the Sumo Logic App for Docker. Read more blogs from the Sumo Logic DevOps community at devops.sumologic.com.