Chris Simms 2017 Draft Qb Rankings, Articles K

Get a Shell to a Running Container | Kubernetes Connect and share knowledge within a single location that is structured and easy to search. And, voila, you are inside the container, as root. so it is not always good to assume that we have bash in the container. You can choose to define the custom columns inline or use a template file: -o custom-columns= or -o custom-columns-file=. It is absolutely different. ( make sure you update the pod name and ns name with yours ). Since it is a while true loop it would keep your session active. If you do not already have a You signed in with another tab or window. Super! How can I do this? ', referring to the nuclear power plant in Ignalina, mean? which is bash -c this technically means that we are running the bash command with the script as an argument. Now let us see how to execute a shell command into a pod using kubectl exec. Content Discovery initiative April 13 update: Related questions using a Review our technical responses for the 2023 Developer Survey. Delete resources either from a file, stdin, or specifying label selectors, names, resource selectors, or resources. # Create a replication controller using the definition in example-controller.yaml. By clicking Accept all cookies, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy. How to Setup Vault in Kubernetes- Beginners Tutorial - DevopsCube Execute Kubernetes Pod Shell Command as Root user - Pete Houston The command to ssh into node is: gcloud compute instances list gcloud compute ssh . If we had a video livestream of a clock being sent to Mars, what would we see? Generic Doubly-Linked-Lists C implementation. and then running apt-get install commands but since the user I am accessing with doesn't have sudo access I am not able to run commands, There are some plugins for kubectl that may help you achieve this: https://github.com/jordanwilson230/kubectl-plugins, One of the plugins called, 'ssh', will allow you to exec as root user by running (for example) And it's not working with modern k8s using containerd instead of docker. The container runs the docker application which has access to the hosts containers and is able to use the exec command with the user flag. In any case, I hope that sheds at least a bit of light on why there is a process associated with getting a feature merged. I'd like to open a shell. . How can I recursively find all files in current and subfolders based on wildcard matching? These plugins are not audited for security by the Krew maintainers. # List the replication controller with the specified name in plain-text output format. # Create a service using the definition in example-service.yaml. anyone more familiar with the process want to start the draft? Stale issues rot after an additional 30d of inactivity and eventually close. You can just write it as a single-line script and execute it in a similar way as we did for the commands. k8s.gcr.io image registry is gradually being redirected to registry.k8s.io (since Monday March 20th).All images available in k8s.gcr.io are available at registry.k8s.io.Please read our announcement for more details. Connect and share knowledge within a single location that is structured and easy to search. Once it's done, you can access any pod with root user via following command: $ kubectl exec-as -u root pod-69bfb5ffc7-kc2bs. Now we are going to execute some Linux commands on a Single container pod first. The Cookies collected are used only to Show customized Ads. report a problem We have to use docker ps to get the correct docker container id. Kubectl: Developer tips for the Kubernetes command line And GKE moved away from docker, making it impossible to SSH to nodes and use docker exec -u, as crictl does not have a way to pass user either. Any user (including root) can do the following to get kubeconfig in the current user's home directory at $HOME/.kube/config: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $ (id -u):$ (id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run this: Embedded hyperlinks in a thesis or research paper, Understanding the probability of measurement w.r.t. Creating Highly Available Clusters with kubeadm Set up a High Availability etcd Cluster with kubeadm Configuring each kubelet in your cluster using kubeadm Dual-stack support with kubeadm Installing Kubernetes with kOps Installing Kubernetes with Kubespray Turnkey Cloud Solutions Best practices Considerations for large clusters Review the output of kubectl api-resources to determine if a resource is namespaced. Content Discovery initiative April 13 update: Related questions using a Review our technical responses for the 2023 Developer Survey, did you specify the right host or port? It has advanced capabilities to keep . the kubectl command acts against the namespace set for the current context in your [] -t represents that kubectl exec should get a terminal ID allotted. I have added a question here if you can help : ). You cannot log into the pod directly as root via kubectl. First, inspect the pod in question to get the docker container you want to connect to. If you have any questions, please feel free to reach out directly. Display the detailed state of one or more resources. Interpreting non-statistically significant results: Do we have "no evidence" or "insufficient evidence" to reject the null? Then issue following commands to install the plugin: $ kubectl krew install exec-as $ kubectl krew install prompt. Using https from a docker in docker container running alongside a docker daemon sidecar container on a pod in kubernetes, ://github.com/jordanwilson230/kubectl-plugins.git. Running Kubernetes Node Components as a Non-root User there is no full-fledged root, part of the system in this read-only mode, A colleague of mine found this tool: https://github.com/ssup2/kpexec, It runs a highly privileged container on the same node as the target container and joins into the namespaces of the target container (IPC, UTS, PID, net, mount). In case anyone is working on AKS, follow these steps: Once you are inside a node, perform these commands to get into the container: In k8s deployment configuration, you can set to run the container as root. kubectl describe pods | grep Name Name: suitecrm-0 Execute shell commands using one of the following methods: Use kubectl exec to open a bash command shell where you can execute commands.. Kubectl, the Kubernetes command-line interface (CLI), has more capabilities than many developers realize. All my commands are executed on the local namespace we have created and I have two pods. variables in the running container: Experiment with running other commands. Is it the only way? # List all pods in plain-text output format and include additional information (such as node name). If you need help, run kubectl help from the terminal window. I cannot SSH to machine because I designed my infrastructure to be fully automated with Terraform without any manual access. Use case is I have a container that runs as an unprivileged user, I mount a volume on it, but the volume folder is not owned by the user. For example, if the variable is set to seattle, kubectl get pods would return pods in the seattle namespace. SSH as root to kubernates pod. Connect and share knowledge within a single location that is structured and easy to search. Find centralized, trusted content and collaborate around the technologies you use most. Thanks for contributing an answer to Stack Overflow! If the POD_NAMESPACE environment variable is set, cli operations on namespaced resources will default to the variable value. Support the user flag from docker exec in kubectl exec, http://stackoverflow.com/questions/33293265/execute-command-into-kubernetes-pod-as-other-user, https://github.com/notifications/unsubscribe-auth/ABG_p7sIu20xnja2HsbPUUgD1m4gXqVAks5qzCksgaJpZM4Jk3n0, Specify Username to exec health check commands, Support the env flag from docker exec in kubectl exec (and API), exec updater errors when using non-root user, Unable to upload media due to permissions error, fixed by restarting, run connect-get-namespaced-pod-exec as a specific user, kubectl exec does not have a -user option, To add username option for kubectl exec command and CRI update. Find centralized, trusted content and collaborate around the technologies you use most. What does 'They're at four. Sort your objects by specifying any numeric or string field with the --sort-by flag. or you can use one of these Kubernetes playgrounds: In this exercise, you create a Pod that has one container. Here is a screenshot of us trying to run some complex shell commands with sed and awk, All the commands you see on the preceding screenshot are given below for you to copy and try, Now we have learnt how to execute commands into the pod and on the specific container using the -c option. To get SSH or Terminal access to the container on the POD using kubectl exec. See the individual subcommands for details. Resource types are case-insensitive and The default output format for all kubectl commands is the human readable plain-text format. 565), Improving the copy in the close modal and post notices - 2023 edition, New blog post from our CEO Prashanth: Community is the future of AI. However, these workarounds break nice Kubernetes/Docker abstractions and introduce security holes. Making statements based on opinion; back them up with references or personal experience. namespace of that ServiceAccount (this is the same as the namespace of the Pod) Note - requires. Follow DevopsJunction onFacebook orTwitter Provides utilities for interacting with plugins. Add or update the annotations of one or more resources. Lets sumarize what I found here in posts, comments and links. Not having this makes debugging things a lot more painful. You can solve the problem with nextcloud by running Lets assume you have two replicas of a container named order running on a Kubernetes cluster. for a quick guide, see the cheat sheet. Here is a screenshot of me executing a shell script. For me inspecting the filesystem as root, and running utilities that can interact with filesystem as root, is the number one reason of wanting to get support for the requested feature. Bash ignoring error for a particular command. Copy fully qualified docker container name then use docker exec: Once then i had full root access in bash inside POD. *////', 4ed493495241b061414b94425bb03b682534241cf19776f8809aeb131fa5a515, runc exec -t -u 0 4ed493495241b061414b94425bb03b682534241cf19776f8809aeb131fa5a515 sh, To login as different i use exec-as plugin in kubernetes here are the steps you can follow. Both have to be given for opening a proper SSH terminal to the POD/container. Open a third terminal to get the INTERNAL-IP of the affected node to initiate the SSH connection. +1 for this feature. What is the difference between a pod and a deployment? Kubeadm puts the original kubeconfig in /etc/kubernetes/admin.conf. If kubectl had the --user I could bash in as root and resize2fs. https://github.com/notifications/unsubscribe-auth/ABG_p7sIu20xnja2HsbPUUgD1m4gXqVAks5qzCksgaJpZM4Jk3n0 Create one or more resources from a file or stdin. Already on GitHub? This should look familiar if you've used Docker's exec command. kubectl exec -u root could do that, if the '-u' option existed. kubectl get pod security-context-demo-2. KQ - How to enter a pod as root? - Kubernetes Questions so you would be able to execute any complex shell commands with | pipes and awk, sed etc. If say, a feature was promoted to stable and then flagged for deprecation, it'd be a minium of a year before it could be removed following the deprecation policy. I was wrong about that, because your injected debug container shares the process namespace with your target container, you can access the filesystem of any process in the target container from your debug container. This might make contributors reluctant, so what is meant with that? So again, the usefulness seems quite limited. Making statements based on opinion; back them up with references or personal experience. You cannot log into the pod directly as root via kubectl. Exec commands on kubernetes pods with root access, https://github.com/jordanwilson230/kubectl-plugins, github.com/jordanwilson230/kubectl-plugins/issues/40, https://github.com/jordanwilson230/kubectl-plugins/blob/krew/kubectl-exec-as, Production grade running kubernetes on AWS using EKS, How a top-ranked engineering school reimagined CS curriculum (Ep. kubectl get replicationcontroller . If you're used to using the docker command-line tool, kubectl for Docker Users explains some equivalent commands for Kubernetes. but we have a workaround to try all the shells before we give up. Install the packages by following the procedure explained below: 1. Hi , In this short tutorial I will show you a way of getting a root shell in containers running inside a modern Kubernetes cluster. please do let us know on the comments section. Display Resource (CPU/Memory/Storage) usage. This is the syntax of the kubectl exec command. Names are case-sensitive. -m is supposed to preserve environment variables. Does a password policy with a restriction of repeated characters increase security? You need to connect to the node and then connect to the container from there using docker. When you run multi-tenant clusters using logical isolation, you especially need to secure resource and workload access. su -s /bin/bash www-data # Start streaming the logs from pod . How to use sudo inside a docker container? Azure CLI Copy ssh -o 'ProxyCommand ssh -p 2022 -W %h:%p [email protected]' azureuser@<affectedNodeIp> Enter your password. --name=kube-system tells kubectl which namespace the container is running in. Explicit use of --namespace overrides this behavior. Did the drapes in old theatres actually say "ASBESTOS" on them? 1) find out what node it is running on kubectl get po -n [NAMESPACE] -o wide 2) ssh node 3) find the docker container sudo docker ps | grep [namespace] 4) log into container as root sudo docker exec -it -u root [DOCKER ID] /bin/bash Share What "benchmarks" means in "what are benchmarks for? Open an issue in the GitHub repo if you want to Asking for help, clarification, or responding to other answers. Find centralized, trusted content and collaborate around the technologies you use most. You cannot log into the pod directly as root via kubectl. I have to rebuild my docker container and make sure the Docker file has USER root as the last line, then debug, then disable this. Generating points along line with specifying the origin of point generation in QGIS, Generic Doubly-Linked-Lists C implementation. How to create port forwarding from google kubernetes engine cluster to external IP address? Run them at your own risk. If you have a specific, answerable question about how to use Kubernetes, ask it on By default, output is from the first container. How will go inside the pod as a root? - Discuss Kubernetes as long as you are having the commands available on the container. Issues go stale after 90d of inactivity. how do we run shell scripts with kubectl exec ?. We don't want to run the untrusted code as root in the container, which prevents us from just escalating permissions for all programs. kubectl debug does not work as well, as it just ends up with the same user as the main container, with no way to become root. # List all pods in plain-text output format. We delegate stewardship of parts of the code base to SIGs; and it is through the KEPs that one or more of the SIGs can come to concensus on a feature. However, there are times when after creating the pod, we need to run programs that need root access (they need to access privileged ports, etc). docker command line seems to have a --user flag. An additional use case - you're being security conscious so all processes running inside the container are not privileged. 565), Improving the copy in the close modal and post notices - 2023 edition, New blog post from our CEO Prashanth: Community is the future of AI. We Hope you are fine with it. kubectl exec -it vault-0 -- /bin/sh Create secrets. I can't use a lifecycle.preStart hook because that runs as the unprivileged user too. Thanks for providing an easy way to use this plugin, but it has been recommended in previous answers before. johnjjung, if you have ssh access to the node you can connect to the container using docker with the user flag which might save you a bit of time. suggest an improvement. The following command would open a the command you have given previously might not let you into a terminal. kpexec now supports the following container runtimes. You can get this with kubectl get nodes -o wide. suppose you have a Pod named my-pod, and the Pod has two containers directory: In your shell, send a GET request to the nginx server: The output shows the text that you wrote to the index.html file: When you are finished with your shell, enter exit. There are multiple secret engines (Databases, Consul, AWS, etc). In this article, I introduce several kubectl CLI . No. For example running utils like apt/apk in the continer is not easy when the root filesystem is not where they expect it. Before you begin crictl requires a Linux operating system with a CRI runtime. kubectl replace - Replace a resource by filename or stdin. This works by creating a pod on the same node as the container and mounting the docker socket into this container. List of global command-line options, which apply to all commands. Here are some examples: If a Pod has more than one container, use --container or -c to He also rips off an arm to use as a sword, Simple deform modifier is deforming my object. You cannot log into the pod directly as root via kubectl. ", English version of Russian proverb "The hedgehogs got pricked, cried, but continued to eat the cactus". Send feedback to sig-testing, kubernetes/test-infra and/or fejta. AFAIK, kubectl won't show the correct docker container id. Ubuntu won't accept my choice of password. how to ssh or open pod shell using kubectl exec, how to execute a command into the pod or container, choosing the container name using option -c, interactive terminal option and why both are important. Thanks for the thoughtful reply @whereisaaron :) I think that captures things quite well. However, the, This plugin is not working with a modern k8s version, like 1.22 for example, that is using containerd. Let us presume the container we want to SSH to or take a terminal has a bash shell installed, So to open a shell/terminal. List the available commands that correspond to alpha features, which are not enabled in Kubernetes clusters by default. Open an issue in the GitHub repo if you want to # You have now created and "installed" a kubectl plugin. Get a shell into the running Container: kubectl exec -it security-context-demo-2 -- sh. Prerequisites: Root access to the cluster node in which the container is running. --server-print=false flag to the kubectl get command. *//,,', containerID will be something like Maybe even use the user that the docker file defines. What if there is no bash and how would you take terminal or SSH into the container/pod, When you are not sure what shell would be available on the container, or when you know that bash may not be there but to try it out, There is a command we can use to test major shells before giving up. HI. It's not them. By clicking Post Your Answer, you agree to our terms of service, privacy policy and cookie policy. The kubectl debug command simplifies these debugging tasks by providing a new ephemeral container inside your Pod. Before we begin, I have two deployments one with a single container in a pod and another with a sidecar container ( one main + one sidecar). Found a solution replying onto related question. Deploy your software and use " kubectl exec " to get an interactive shell session in your currently running container (or hit the "play"-like button in Lens). for details about which output format is supported by each command. Effect of a "bad grade" in grad school applications. to get root, you would just pass -u 0 to the docker container when you exec hitesh1907nayyar December 20, 2019, 7:48am #3 Hi @bkgann Thanks for the reply. KEPs can be quite daunting, but I want to provide a little context around them. Which language's style guidelines should be used when writing code that is supposed to be called from another language? Installing crictl @dims I'm confused, why is this closed? I just want a place to stick my in support of the proposal as an active Kubernetes user. Hope, Restart Namespace all Deployments after k8s v1.15 You can simply use the kubectl rollout restart command that takes care of restarting all the deployments in a namespace If you specify only the namespace and not a specific deployment, all the deployments in the namespace would be restarted kubectl rollout restart, How to check the Kubernetes and Kubectl Version using the kubectl command line that's the objective of this article. Successfully merging a pull request may close this issue. I am running through a similar issue, however I am using a git-sync sidecar that I mount. How can I avoid `Permission denied` Errors when mounting a container into my deployment? To solve this issue, I'm making a tool called "kpexec". How can I keep a container running on Kubernetes? While I feel we need the root access quit a lot in local development environment, it's worth to mention it in this thread. current context in your KUBECONFIG file: Thanks for the feedback. How a top-ranked engineering school reimagined CS curriculum (Ep. This was the more useful answer for me. How to change the output color of echo in Linux. I thought su -l didn't copy env vars? Copy the repository specification below and paste it into the file. What does 'They're at four. Looks like this is still not resolved, after 6 years. To print information about the status of a pod, use a command like the following: To output objects to a sorted list in your terminal window, you can add the --sort-by flag to a supported kubectl command. In the preceding command, we are trying all the shells before we give up. /lifecycle stale, kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo. This would execute the bash command as we wanted to but will it give you a terminal access ? client configuration. I guess though this should be an additional RBAC permission, to allow/block 'exec' as other than the container user. Not the answer you're looking for? Made with in SYDNEY 2020-2022 Sukanta Maikap. it would/should be accepted and executed. The point though is - that's why I posted it here - is that I'd like to see "kubectl exec" do the right thing. Any user (including root) can do the following to get kubeconfig in the current user's home directory at $HOME/.kube/config: Alternatively, if you are the root user, you can run this: Thanks for contributing an answer to Stack Overflow! # Delete all the pods and services that have the label '='. 't see a command prompt, try pressing enter. There are some plugins for kubectl that may help you achieve this: https://github.com/jordanwilson230/kubectl-plugins One of the plugins called, 'ssh', will allow you to exec as root user by running (for example) kubectl ssh -u root -p nginx-0 Share Improve this answer Follow edited Nov 16, 2019 at 13:30 Nanhe Kumar 15.3k 5 78 70 By default kubectl will first determine if it is running within a pod, and thus in a cluster. In the world of docker, connecting to a docker container as root is very easy and does not require a Dockerfile change : But when you are running the same container on a Kubernetes cluster, it is not straightforward. This overview covers kubectl syntax, describes the command operations, and provides common examples. I'd like to open a Command line tool (kubectl) | Kubernetes Here are the steps : For example, the following commands produce the same output: NAME: Specifies the name of the resource. How to run kubectl commands inside a container? +1 really a issue, I have to ssh and then exec the docker exec, such annoying. there is Kubernetes service account token file mounted at, you don't explicitly specify a namespace on the kubectl command line, To find out more about plugins, take a look at the. Extracting arguments from a list of function calls, A boy can regenerate, so demons eat him for years. kubectl | Kubernetes For instance pods, nodes, services, etc. Last modified April 26, 2022 at 12:30 AM PST: Installing Kubernetes with deployment tools, Customizing components with the kubeadm API, Creating Highly Available Clusters with kubeadm, Set up a High Availability etcd Cluster with kubeadm, Configuring each kubelet in your cluster using kubeadm, Communication between Nodes and the Control Plane, Guide for scheduling Windows containers in Kubernetes, Topology-aware traffic routing with topology keys, Resource Management for Pods and Containers, Organizing Cluster Access Using kubeconfig Files, Compute, Storage, and Networking Extensions, Changing the Container Runtime on a Node from Docker Engine to containerd, Migrate Docker Engine nodes from dockershim to cri-dockerd, Find Out What Container Runtime is Used on a Node, Troubleshooting CNI plugin-related errors, Check whether dockershim removal affects you, Migrating telemetry and security agents from dockershim, Configure Default Memory Requests and Limits for a Namespace, Configure Default CPU Requests and Limits for a Namespace, Configure Minimum and Maximum Memory Constraints for a Namespace, Configure Minimum and Maximum CPU Constraints for a Namespace, Configure Memory and CPU Quotas for a Namespace, Switching from Polling to CRI Event-based Updates to Container Status, Change the Reclaim Policy of a PersistentVolume, Configure a kubelet image credential provider, Control CPU Management Policies on the Node, Control Topology Management Policies on a node, Guaranteed Scheduling For Critical Add-On Pods, Migrate Replicated Control Plane To Use Cloud Controller Manager, Reconfigure a Node's Kubelet in a Live Cluster, Reserve Compute Resources for System Daemons, Running Kubernetes Node Components as a Non-root User, Using NodeLocal DNSCache in Kubernetes Clusters, Assign Memory Resources to Containers and Pods, Assign CPU Resources to Containers and Pods, Configure GMSA for Windows Pods and containers, Resize CPU and Memory Resources assigned to Containers, Configure RunAsUserName for Windows pods and containers, Configure a Pod to Use a Volume for Storage, Configure a Pod to Use a PersistentVolume for Storage, Configure a Pod to Use a Projected Volume for Storage, Configure a Security Context for a Pod or Container, Configure Liveness, Readiness and Startup Probes, Attach Handlers to Container Lifecycle Events, Share Process Namespace between Containers in a Pod, Translate a Docker Compose File to Kubernetes Resources, Enforce Pod Security Standards by Configuring the Built-in Admission Controller, Enforce Pod Security Standards with Namespace Labels, Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller, Developing and debugging services locally using telepresence, Declarative Management of Kubernetes Objects Using Configuration Files, Declarative Management of Kubernetes Objects Using Kustomize, Managing Kubernetes Objects Using Imperative Commands, Imperative Management of Kubernetes Objects Using Configuration Files, Update API Objects in Place Using kubectl patch, Managing Secrets using Configuration File, Define a Command and Arguments for a Container, Define Environment Variables for a Container, Expose Pod Information to Containers Through Environment Variables, Expose Pod Information to Containers Through Files, Distribute Credentials Securely Using Secrets, Run a Stateless Application Using a Deployment, Run a Single-Instance Stateful Application, Specifying a Disruption Budget for your Application, Coarse Parallel Processing Using a Work Queue, Fine Parallel Processing Using a Work Queue, Indexed Job for Parallel Processing with Static Work Assignment, Handling retriable and non-retriable pod failures with Pod failure policy, Deploy and Access the Kubernetes Dashboard, Use Port Forwarding to Access Applications in a Cluster, Use a Service to Access an Application in a Cluster, Connect a Frontend to a Backend Using Services, List All Container Images Running in a Cluster, Set up Ingress on Minikube with the NGINX Ingress Controller, Communicate Between Containers in the Same Pod Using a Shared Volume, Extend the Kubernetes API with CustomResourceDefinitions, Use an HTTP Proxy to Access the Kubernetes API, Use a SOCKS5 Proxy to Access the Kubernetes API, Configure Certificate Rotation for the Kubelet, Adding entries to Pod /etc/hosts with HostAliases, Externalizing config using MicroProfile, ConfigMaps and Secrets, Apply Pod Security Standards at the Cluster Level, Apply Pod Security Standards at the Namespace Level, Restrict a Container's Access to Resources with AppArmor, Restrict a Container's Syscalls with seccomp, Exposing an External IP Address to Access an Application in a Cluster, Example: Deploying PHP Guestbook application with Redis, Example: Deploying WordPress and MySQL with Persistent Volumes, Example: Deploying Cassandra with a StatefulSet, Running ZooKeeper, A Distributed System Coordinator, Explore Termination Behavior for Pods And Their Endpoints, Certificates and Certificate Signing Requests, Mapping PodSecurityPolicies to Pod Security Standards, Well-Known Labels, Annotations and Taints, ValidatingAdmissionPolicyBindingList v1alpha1, Kubernetes Security and Disclosure Information, Articles on dockershim Removal and on Using CRI-compatible Runtimes, Event Rate Limit Configuration (v1alpha1), kube-apiserver Encryption Configuration (v1), kube-controller-manager Configuration (v1alpha1), Contributing to the Upstream Kubernetes Code, Generating Reference Documentation for the Kubernetes API, Generating Reference Documentation for kubectl Commands, Generating Reference Pages for Kubernetes Components and Tools, kubectl apply -f https://k8s.io/examples/application/shell-demo.yaml, # You can run these example commands inside the container, # Run this in the shell inside your container, Reorg the monitoring task section (#32823) (f26e8eff23), Running individual commands in a container, Opening a shell when a Pod has more than one container.