Alex Winter Alex Winter - 3 months ago 21x
SQL Question

kubernetes and debugging it in general

I have a kubernetes cluster setup in AWS.
We are using the EC2 container registry to store our docker images on.
We have the master/minions all setup and everything seems to be working with the cluster.

My spec file is as followed:

apiVersion: v1
kind: Service
name: apim-mysql
app: apim
# the port that this service should serve on
- port: 3306
app: apim
tier: mysql
clusterIP: None
apiVersion: extensions/v1beta1
kind: Deployment
name: apim-mysql
type: Recreate
app: apim
tier: mysql
- name: myregistrykey
- name: mysql
image: <This points to our EC2 Container Registry and retreives the image>
cpu: 100m
memory: 100Mi
- name: WSO2_ZIP_FILE
- name: WSO2_VERSION
value: 1.10.1
value: 'yes'
- name: MYSQL_USER
value: username
value: password
value: dns
# If your cluster config does not include a dns service, then to
# instead access environment variables to find service host
# info, comment out the 'value: dns' line above, and uncomment the
# line below.
#value: env
- containerPort: 3306
name: mysql

What this container does is just sets up a mysql.
We will need other nodes within the cluster to connect to this node.
Because they will need to use the mysql DB.

I guess my first question is does everything look okay with this spec file?
Or does anyone see something wrong?

I do the kubectl create command and it runs successfully:

kubectl create -f mysql.yaml
service "apim-mysql" created
deployment "apim-mysql" created

kubectl get pods shows the pod running:

apim-mysql-545962574-w2qz1 1/1 Running 1 8m

I sometimes when doing kubectl logs get an error showing this:

kubectl logs apim-mysql-545962574-w2qz1
Error from server: dial unix /var/run/docker.sock: no such file or directory

But eventually retrying enough it will go through... if anyone has information on why that occurs it would be great.

When it does work get something like this:

kubectl logs apim-mysql-545962574-w2qz1
Initializing database
2016-07-13T15:51:47.375052Z 0 [Warning] InnoDB: New log files created, LSN=45790
2016-07-13T15:51:52.029915Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2016-07-13T15:51:53.531183Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: b837bd45-4911-11e6-99ba-02420af40208.
2016-07-13T15:51:53.746173Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2016-07-13T15:51:53.746621Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2016-07-13T15:52:19.891437Z 1 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2016-07-13T15:52:19.891705Z 1 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2016-07-13T15:52:19.891733Z 1 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2016-07-13T15:52:19.891778Z 1 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2016-07-13T15:52:19.891831Z 1 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
Database initialized
MySQL init process in progress...
2016-07-13T15:52:34.632188Z 0 [Note] mysqld (mysqld 5.7.13) starting as process 49 ...
2016-07-13T15:52:49.814764Z 0 [Note] InnoDB: PUNCH HOLE support available
2016-07-13T15:52:49.814846Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-07-13T15:52:49.814859Z 0 [Note] InnoDB: Uses event mutexes
2016-07-13T15:52:49.814870Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2016-07-13T15:52:49.814928Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2016-07-13T15:52:49.814932Z 0 [Note] InnoDB: Using Linux native AIO
2016-07-13T15:52:50.243657Z 0 [Note] InnoDB: Number of pools: 1
2016-07-13T15:52:52.175079Z 0 [Note] InnoDB: Using CPU crc32 instructions
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...

After a little bit it the pod seems to restart and it will say Initializing database again..

A couple days ago when I ran kubectl logs it would return everything right away and be very fast now it is very slow and doesn't really show anything I have not really changed my spec file at all so I have no idea what is going on. To me it seems like the container isn't getting executed properly when it actually does show some logs... but I unsure.

If anyone has any clue on how to debug this further with some commands I can run it would be very appreciated. I am very stuck at this point and have google'd so much but with no luck.



My experience using kubernetes on aws while getting unhelpful errors as

Error from server: dial unix /var/run/docker.sock: no such file or directory

was resolved by choosing a more hefty aws cluster instance type ... here are relevant env vars

export MASTER_SIZE=t2.medium
export NODE_SIZE=t2.medium
export NUM_NODES=2   #  if not defined aws will auto guess

... also remove mention of resource limiting settings under tag resources until after it runs OK

Following commands are essential ... just leave off mention of namespace if you are not using

kubectl describe svc --namespace=xxx

kubectl get pods --namespace=xxx

kubectl describe pods --namespace=xxx

kubectl describe nodes

Also nice is ability to perform a live edit of a deployment ... first see your deployments .. issue

kubectl get deployments  --namespace=ruptureofthemundaneplane 

... output

loudspeed-deployment   1         1         1            1           1h
mongo-deployment       1         1         1            1           1h

so now we know name of deployment to do live edit issue

kubectl edit deployment/mongo-deployment

which will open an edit session in your terminal using default editor where you can change settings at will

I find when troubleshooting a database deployment its handy to also launch an image using below Dockerfile ... this allows you login using exec as per

kubectl exec -ti $(kubectl get pods --namespace=${PROJECT_ID}|grep ${GKE_NODEDEPLOYMENT}|cut -d' ' -f1) --namespace=${PROJECT_ID} -c ${GKE_NGINX} -- bash

where you are free to do interactive database login session (once you install needed client code or put same into below Dockerfile) ... here is matching Dockerfile for this troubleshooting deployment container

FROM ubuntu:16.04

ENV TERM linux
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update
RUN apt-get install -y wget  curl

COPY .bashrc /root/

# ENTRYPOINT ["/bin/bash"]
CMD ["/bin/bash"]

# docker build --tag stens_ubuntu .
# docker run -d  stens_ubuntu  sleep infinity
# docker ps
# ... find CONTAINER ID from above and put into something like this
# docker exec -ti 3cea1993ed28 bash