etcd container not starting - oci runtime error

[brianrusso]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.2074+a092d8e0f95f52", GitCommit:"a092d8e0f95f5200f7ae2cba45c75ab42da36537", GitTreeState:"clean", BuildDate:"2016-12-13T17:03:18Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:57:05Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
  The connection to the server localhost:8080 was refused - did you specify the right host or port?

• Cloud provider or hardware configuration:
  packet.net

• OS (e.g. from /etc/os-release):
  NAME="Container Linux by CoreOS"
  ID=coreos
  VERSION=1298.5.0
  VERSION_ID=1298.5.0
  BUILD_ID=2017-02-28-0013
  PRETTY_NAME="Container Linux by CoreOS 1298.5.0 (Ladybug)"
  ANSI_COLOR="38;5;75"
  HOME_URL="https://coreos.com/"
  BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

• Kernel (e.g. uname -a):
  Linux kilauea-etcd-00 4.9.9-coreos-r1 kubeadm join on slave node fails preflight checks #1 SMP Tue Feb 28 00:06:10 UTC 2017 x86_64 Intel(R) Atom(TM) CPU C2550 @ 2.40GHz GenuineIntel GNU/Linux

• Others:
  Docker command I am using to load kubeadm onto CoreOS is:

docker run -it -v /etc/cni:/rootfs/etc/cni -v /etc/systemd:/rootfs/etc/systemd -v /opt:/rootfs/opt -v /usr/bin:/rootfs/usr/bin luxas/kubeadm-installer coreos

I do get this error when running that docker, not sure if it matters:
/entrypoint.sh: line 97: /etc/systemd/network/50-cni.network: No such file or directory

What happened?

I got the dreaded "waiting for control plane" error and I am very new to Kubernetes so I did a bunch of searching..
I figured out the API on 8080 was not running, so I looked at that container's logs, and it complained about the etcd container not running. Progress!

In fact.. it never ran.. just stuck at 'Created'
a48603e17aa4 gcr.io/google_containers/etcd-amd64:3.0.14-kubeadm "etcd --listen-client" About a minute ago Created k8s_etcd.c323986f_etcd-kilauea-etcd-00_kube-system_3a26566bb004c61cd05382212e3f978f_7bf5a5ad

Let's try starting it!
kilauea-etcd-00 core # docker start a48603e17aa4
Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"write /proc/self/task/1/attr/exec: invalid argument\""\n"
Error: failed to start containers: a48603e17aa4

Boo! Okay and that's where I'm stuck. I have no idea what this error means..

What you expected to happen?

It to work! Because kubeadm seems pretty good.

How to reproduce it (as minimally and precisely as possible)?

In my environment (packet.net coreos), it is very replicable using the above docker command.

Anything else we need to know?

I like puppies.

[brianrusso]

Doesn't happen with Ubuntu 16, so definitely something specific to either CoreOS or that particular kubeadm-installer by luxas

[brianrusso]

I am no longer pursuing Kubernetes on this platform; and have no reason to believe this is a significant bug; so am closing the ticket.

[ericchiang]

FYI I hit this too. Looking into it.

[luxas]

@ericchiang On CoreOS? Yeah, I could never figure out what the root cause what when I hit it back at that time.

One thing you must do in any case (and may fix the issue) is to remove spc_t from the etcd Static Pod.
We're looking into removing that anyway so (for CoreOS compat)