Learning Kubernetes with minikube and the kubernetes.io tuts

Introduction

In my previous post, I discussed my assertion that with new Cloud technologies, it made more sense to develop web backends that were targeted at individuals, i.e. Personal Backend Services, as opposed to Shared, and this is what I planned to use for taskUdo. However, I also admitted that this was not based on practical experience of either the technologies or costs associated with deploying applications to the Cloud.

Clearly a lack of practical knowledge was something that needed rectifying, so my plan was to:

  1. Bootstrap knowledge of Kubernetes (my chosen Cloud technology) by undertaking basic training with a focus on using it for application deployment.
  2. As a learning reinforcement exercise, attempt to move a non-trivial application (in this case ownCloud) onto Kubernetes, running on a:
    • Local, testing cluster.
    • Production cloud environment (for which I had chosen, no surprises, Google Cloud Platform).

Here’s what I did and found for the first part of this.

Bootstrapping Kubernetes knowledge

My normal approach when bootstrapping skills, even with bleeding edge stuff, is Amazon, find a couple of books and dig in. I think this continues to work because of the greater effort involved in publishing books (even if not in dead tree format). Which, in turn, tends to result in higher production standards and greater long term value to the information in books (usually anyway).

This time though I decided not to go with that approach. As yet, there aren’t that many books on Kubernetes, and, from what I can tell, the existing ones are more focused on building clusters and running complex backend services than I hope will be necessary. Instead, I decided to try the somewhat radical approach, for me anyway, of directly going with the OpenSource project’s training and documentation.

Relying on docs from projects where a lot of them make their money through books and support contracts can be a bit of a risk.
Relying on docs from projects where a lot of them make their money through books and support contracts can be a bit of a risk.

Plan

Work through the official Kubernetes web site’s list of official tutorials from here.

I’d start with their zero install Kubernetes Basics, interactive demo/tutorial 1, since not having to install anything, was one less thing for me to get wrong, or distract myself with.

Once through that, then I was going to set-up on my Mac a local Kubernetes test cluster by following the instructions in the Hello Minikube tutorial to get minikube installed.

With minikube and my test cluster in place, I’d then go through the rest of the tutorials (but omit the full on Online Training Course) and hopefully be competent enough at the end of them to know where to start with subsequently bringing up ownCloud.

Observations and thoughts

Getting a local Kubernetes test/training cluster setup working

minikube is the Kubernetes project current recommended front-end tool for creating and managing local Kubernetes’ test clusters.

The Hello Minikube tutorial itself steps you through installing minikube and getting everything else you need installed to set up a local Kubernetes test cluster and its tools.

On macOS, this list ends up being:

It’s not a huge list, but it’s coming from a variety of different providers, so the actual installation process can be a bit disjointed. However, I think if you follow the steps in the tutorial carefully you should reliably end up with a working local Kubernetes test cluster without too many problems, I did anyway.

Having dealt with both commercial and OpenSource installations of similar complexity, I have to admit that I was actually very relieved by how smooth the whole installation and setup process was (and how well it has largely worked since).

Perhaps not quite as spectacular, but we have Kubernetes lift off ... hopefully, nothing will blow up ...
Perhaps not quite as spectacular, but we have Kubernetes cluster lift off … hopefully, nothing will blow up …

Using local minikube and the local Kubernetes test

Whilst getting minikube installed and working was straightforward, and I haven’t encountered any major problems with it since, I have noticed a few wrinkles.

Recommend allocating more resources to minikube

I’m not sure precisely which tutorial it was, possibly the Connecting a Front End to a Back End Using a Service tutorial, but one of them wouldn’t run with the default minikube configuration because the resource configuration was too small (you can see the defaults with minikube start --help).

The giveaway that you might be bumping into a similar problem is when kubectl describe pods your_pod_that_is_not_starting mentions in its output that it can’t schedule it.

My trusty laptop could take it 2. So I fixed this by bumping up the defaults using minikube config set to allow it to use all four cores, double the amount of RAM (because I could), and at the same time, told it to use the xhyve driver by default (I was getting bored of typing it when starting minikube).

minikube config view now shows:

- memory: 4096
- vm-driver: xhyve
- WantReportError: true
- cpus: 4
Bug (maybe), minikube Persistent Volume Claims not picking up manually provisioned volumes

One frequent application requirement, is to be able to have a disk space volume that does not get thrown away between invocations. For instance, such a volume might hold a database of users and their passwords that have been added to the system via its web interface, or some other similar data.

If you are working your way through the Tutorials, the first time you bump into this is with Running a Single-Instance Stateful Application demo. In this example, the intent is to bring up a MySQL DB with the DB’s files stored on a Persistent Volume. Unusually for the tutorials, the example expects you to be running on Google’s Cloud Platform and, because that uses the gcloud tool to create volumes, you are left a bit in the dark if you are using anything else. However, with a bit of a search, the answer to this is actually contained in the docs Tasks->Configuring a Pod to Use a PersistentVolume for Storage article. In this article, they show how to write a file to a volume, and then have one of the Pods serve it up using minikube.

Except, in the last couple of weeks that example has stopped working because the system, with possibly the latest update, has started ignoring manually created Persistent Volumes and is instead always dynamically creating fresh, temporary volumes.

For those interested, the workaround, is to annotate the PersistentVolumeClaim with an empty storage type, which disables fallback automatic volume provisioning for this claim and causes it to correctly bind to the manually created one instead.

So for the task example, the workaround PersistentVolumeClaim looks like this:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: task-pv-claim
  "annotations": {
        # Prevent creating new volumes on the fly by not specifying.
        "volume.beta.kubernetes.io/storage-class": ""
  }

spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
Bug, leaks IP addresses when deleting cluster

One of the things I noticed after experimenting with using minikube delete as a way to completely reset the cluster, was that it would leak IP addresses 3. After each minikube delete operation, you would find minikube ip returns an IP address that is incremented from its previous one. As of the 14th of March 2017, I think this remains broken, there is certainly a ticket open for the issue on GitHub here.

In a meantime, the workarounds are:

  1. Live with it and use minikube delete sparingly.
  2. You can manually reset the numbers by removing a couple of system files, as a suggested in the cross-linked comment for a different ticket (708). I used this successfully, but be careful with this option as it’s all too easy to accidentally remove the wrong files and end up with a broken system.

Tutorials

I’m not an IT admin, full on Linux hacker or Jedi programmer in any way shape or form. However, I have been in the industry for a while and picked up the odd thing along the way. Specifically, I have been playing with Linux from mid 90’s, my team at ARM developed a large distributed Build System and I have quite a lot of background knowledge about services and cloud computing as a result of this.

So not a blank slate, I’ll usually know roughly what to search for and who to ask, but no previous practical experience and this is what I thought about Tutorials on the Kubernetes site.

Content and pace

I found the tutorials informative, they stepped me through at about the right pace and covered almost everything I needed to know to find my way around afterwards.

The only technical issues I had with the tutorials were the two with minikube that I have already mentioned.

Coverage

Ultimately I ended up skipping a few of the tutorials, not because I don’t think they would be useful for most folks, but because after I had glanced through them, they were more complicated than I’m hoping to need for taskUdo’s backend. Specifically, I did not really look at:

Other than that, and after tackling getting ownCloud going, I thought there was one obvious omission from the content, and few “nice to have” improvements.

The one major omission in the tutorials that I have so far noticed, was the lack of coverage of the use of Secrets. If you search the site there is good information present on how to create and use them in the Glossary here. I’d argue that the need to store and use passwords is likely to be a common requirement for most applications, so this should probably be represented in the tutorials directly.

Beyond that, and I know that some of it is not a core concern and there is a need for vendor neutrality, but it would have been nice to have seen:

  • Some examples of how to work with Google Cloud Platform, Amazon Web Services, Azure etc. Coverage of such things as Ingresses, static IP address provisioning, Persistent Volumes, all of which have an element of cloud vendor specificity.
  • A bit more guidance, or pointers to where to look when it came to figuring out why it wasn’t working, e.g. Tasks-> Monitoring, Logging, and Debugging
  • Working with containers, particularly what happens when using exec to permissions and entry points.
  • Getting encryption and SSL working, ideally something featuring the use of Let’s Encrypt

Tips

If you’re planning to use either Kubernetes or minikube here are a few things that you might find useful:

  1. If it is an option for you, then make sure you have extended command line completions installed. It’ll help you explore the command and sub-command options as well as saving you a lot of typing. You can find the official how to do this here and I mention how to do it in a more general macOS context over here.
  2. Familiarise yourself with the contents of Tasks->Debugging Init Containers and Support->Troubleshooting Applications (kubectl describe and kubectl log are your friends).
  3. Similarly kubectl exec -it bash pod_nameis very useful for logging into containers in your pod and looking at what the heck is happening inside of them.
  4. Most containers are very bare bones. If you log into them with kubectl exec, then you may find that many of the normal Linux tools are not available. Assuming the container is Debian based (which a lot of the are) then apt-get update followed by apt-get install tool_you_want (where tool_you_want is usually vim) is helpful to know (if you’ve not used to Linux based OS).
  5. The Kubernetes site search works really well. I’ve only really found the need to resort to more general searches/StackOverflow when I’ve been trying to figure out how to make something work with Google’s Cloud Platform, rather than with Kubernetes per se. So for the most relevant information on Kubernetes, I would recommend using the kubernetes.io’s site search by default.
  6. There is a lot of information in the kubernetes.io site and some of the best bits of it are not always where you might expect to find them. In addition to making sure to use the site search, I think it’s probably a good idea to at least know roughly what else is available under the other documentation areas. I think in particular you might want to have a browse around:

Conclusions

At the outset, the plan was to attempt to bootstrap a reasonable level of competency with Kubernetes by following the Tutorials and documents on the kubernetes.io website.

It appeared as a reasonable approach, as documentation and training are evidently something that is important to the project. For instance, if you just want to quickly try the system out and you go to the project’s Home page, then there, front and centre, is a link to a “Try Our Interactive Tutorials”. Whilst at the other end of the scale, if you follow the “Documentation” link from the Homepage, you’ll find the Tutorial’s page, which in turn links to a free, and I would guess comprehensive, Udacity online training course led by some of the leading lights in the Kubernetes’ community.

It was a few weeks ago now that I completed my training process using the tutorials. Since then I’ve gone on to get ownCloud running on minikube and on Google’s Cloud Platform (more on this in subsequent posts). Within the limitations I’ve already mentioned, I think there is more than enough good quality information on the website for most folks to get themselves up and running with Kubernetes.

Overall the docs, like the rest of the project, are in my opinion very good for such a large and complicated system. For me, the strongest testament to their quality is perhaps that I have felt no need to purchase additional training materials from Amazon or anywhere else.

 

  1. NB: As of 2017/03/12 this option works fine in Chrome and Firefox, but seems to be broken in Safari
  2. Late 2013 MacBook Pro, i7 with 16GB of RAM
  3. This was a bit irksome, especially when working with ownCloud because its security setup meant I needed to be able to pre-configure the IP address with ownCloud in order to be able to log into it.