Edge computing is a new paradigm in which data are locally collected, aggregated and preprocessed before being sent to a Cloud platform. Edge devices, typically IoT objects, are characterized by limited computational capabilities, alongside high energy efficiency operations. In smart building or smart city applications, the overall amount of available IoT devices may account for an important computational capacity. Aggregating the idle CPU cycles of several devices would allow performing on-site parallel and distributed computing, while reusing the available resources in the network. Timely and important computational tasks such as video surveillance processing, HPC jobs, or deep neural network training, can be performed with local available resources. This includes local servers or a private, on-site cloud, as opposed to the use of public clouds, thus avoiding security issues and fostering energy efficiency. An ongoing research topic in such a distributed and volatile context is to design novel algorithms for resource provisioning that allow tasks to be distributed over a set of available IoT devices in an efficient manner. This work presents two online, adaptive and robust scheduling techniques, based on the UCB and EXP3 Multi-Armed Bandit algorithms (MABs), for resource management in Cloud-Edge Computing based environments while improving performance and reducing energy consumption. The novelty of our adapted algorithms lies in the fact that we explicitly account for unavailable computing IoT devices resulting in a time-varying available set of arms at each stage. Our numerical results show the high relevance of our approach in reaching optimal provisioning policies in time-varying environments.