## Overview Hello all, We are running gitlab-runner on Kubernetes and our cloud provider recently upgraded the underlying containerd container runtime to v1.2.8. There is an underlying change, which causes the following errors on the runners `helper` image: ``` failed to create containerd task: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown ``` Steps to reproduce on kubernetes with containerd v1.2.8: ``` kubectl run -it --image gitlab/gitlab-runner-helper:x86_64-f100a208 msvechltest --command -- sh kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. Error from server (NotFound): pods "" not found # we dont get a shell... ``` Further debugging shows: ``` kubectl describe pod msvechltest-5c77c798b7-nn2dp Last State: Terminated Reason: StartError Message: failed to create containerd task: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown ``` [There already is an issue open on containerd about this](https://github.com/containerd/containerd/issues/3597), and a [MR is already scheduled for the next release](https://github.com/containerd/containerd/pull/3565) which fixes this behaviour in some cases. Apparently this happens on images where no default `$PATH` variable is set. While a fix for this is scheduled to be released soon, it might make sense to fix the environment of the helper image. ## Proposal The root-cause stems from https://gitlab.com/gitlab-org/gitlab-runner/blob/master/shells/bash.go#L227 where `sh` is executed without an absolute path. While it makes sense to have it without an absolute path here, adding a default `$PATH` to the helper image might make sense.