Sustainable Offloading in Mobile Cloud Computing: Algorithmic Design and Implementation Sustainable Offloading in MCC: Algorithmic Design and Implementation

University of Ottawa, Canada
Brock University, Canada

ACM Comput. Surv., Vol. 52, No. 1, Article 11, Publication date: February 2019.

Mobile Cloud Computing (MCC) has been extensively explored to be applied as a vital tool to enhance the capabilities of mobile devices, increasing computing power, expanding storage capacity, and prolonging battery life. Offloading works as the fundamental feature that enables MCC to relieve task load and extend data storage through an accessible cloud resource pool. Several initiatives have drawn attention to delivering MCC-supported energy-oriented offloading as a method to cope with a lately steep increase in the number of rich mobile applications and the enduring limitations of battery technologies. However, MCC offloading relieves only the burden of energy consumption of mobile devices; performance concerns about Cloud resources, in most cases, are not considered when dynamically allocating them for dealing with mobile tasks. The application context of MCC, encompassing urban computing, aggravates the situation with very large-scale scenarios, posing as a challenge for achieving greener solutions in the scope of Cloud resources. Thus, this article gathers and analyzes recent energy-aware offloading protocols and architectures, as well as scheduling and balancing algorithms employed toward Cloud green computing. This survey provides a comparison among system architectures by identifying their most notable advantages and disadvantages. The existing enabling frameworks are categorized and compared based on the stage of the task offloading process and resource management types, describing current open challenges and future research directions.

CCS Concepts: • Networks → Cloud computing; Mobile networks; • Computer systems organization → Cloud computing; • Software and its engineering → Scheduling; Cloud computing; • Information systems → Cloud based storage; Mobile information processing systems;

Additional Key Words and Phrases: Mobile Cloud Computing, offloading, energy consumption, resource

ACM Reference format:
Azzedine Boukerche, Shichao Guan, and Robson E. De Grande. 2019. Sustainable Offloading in Mobile Cloud Computing: Algorithmic Design and Implementation. ACM Comput. Surv. 52, 1, Article 11 (February 2019), 37 pages.


The constant demand for more sophisticated mobile devices has accentually propelled the industry to produce technology advancements and promote sharp improvements in the capabilities of mobile devices, as well as technology support devices. Several reports, such as the ones issued by Cisco Systems [49, 101], have demonstrated the rapid growth in the hardware capacities of these devices in recent years. As an example of this steep growth, a recently released mobile phone can easily outperform many static personal computers in the past decade in terms of execution and communication efficiency. On the other hand, these same pocket-size devices integrate more advanced sensors and resource-hungry functions, as well as richer applications associated with such functionalities and capabilities. Unfortunately, improvements in the battery lifespan have not progressed as other parts; the battery of mobile devices is fundamentally limited to the hardware layer. For instance, Lithium-based battery technologies [27] proposed 10 years ago are still used nowadays, while new graphene-based solutions [76] may launch on the market in the foreseeable future, though this is unlikely. As a result, several works have been putting significant efforts into the development of software-layer optimizations to support resource-hungry tasks so that they can execute on resource-constraint devices.

Mobile Cloud Computing (MCC) is rising in a convenient moment where it, from a simplistic viewpoint, intends to handle the conflict between mobile applications and mobile devices. MCC envisions the use of Cloud resources for the provisioning of external computing and storage utilities to mobile entities. When dealing with the provisioning and allocation of distributed resources, Grid Computing [51] has been a prominent solution and a mature standard for the dynamic control and management of processing and computing resources. The Grid middleware defined standardized internal and external APIs and provided functions such as security, resource management, information queue, and data management. Since the approach of the Grid has been proposed, many works [19, 23, 35, 59, 104] have endeavored to augment the mobile device capability and to enable MCC offloading by leveraging Grid resources.

Even though studies have questioned the distinction between the Grid and the Cloud due to their similar goals [11, 122], as well as the maturity [52] and performance [38] of Cloud Computing, we must point out that the Cloud is well suited to offloading because it enables customization, utilization, load balancing, scalability, and less energy consumption when compared with previous standards [111, 128]. Therefore, Cloud Computing can enable adequate service availability through its internal redundancy-based resource scheduling frameworks; Amazon [37] and Google [28] public Cloud infrastructures have already included this feature to support services. In addition to the high availability, Cloud Computing also grants elastic provisioning models [98]. These flexible service models can be particularly well suited to requirements on different granularities by varying resource capabilities and management levels. As a result, when compared to other large-scale systems, such as Cluster or Grid Computing systems, Cloud Computing stands out by genuinely empowering real-time resource provisioning in a self-manageable pay-per-use manner [6, 24, 52].

MCC then bridges Mobile and Cloud Computing through effective offloading. Certainly, connectivity and availability of communication channels play a fundamental role in enabling this offloading as a promising alternative that outsources mobile application tasks or parts of application components from mobile devices to external Cloud utilities. Inevitably, together with offloading, a substantial amount of energy can be preserved on mobile devices by alleviating computing loads from mobile elements. However, preserving energy can only be achieved if the communication channel management overhead is observed and properly controlled, but preserving energy can only be achieved if the communication channel management overhead is observed and properly controlled. Consequently, the lifespan of the mobile devices can be prolonged without hurting their mobility and portability.

Unfortunately, offloading-based energy reservation is not necessarily costless. The offloading process inevitably includes several issues that must be considered for achieving improvement of performance or reduction of energy consumption. For instance, the process of migrating application load might add computation and communication overhead, extra energy consumption of the external resources, usability concerns on users and applications, or QoS requirements of applications and services. All these aspects influence, or even impede, achieving energy efficiency improvement during the task offloading execution scenario. However, even with truly maximizing energy reservation in mobile devices, the offloading comes at the expense of Cloud resources, missing, to some extent, the purpose of Green Computing [85]. As result, the scale of the urban environment where mobile devices most likely reside incurs a high volume of offloading requests, exposing and deepening issues involved with unattainable scheduling, imbalance, and management costs, as well as inevitably increasing Cloud-side energy consumption.

The high interest in MCC has motivated the development of many works, so several recent survey studies have collected and classified existing works, focusing on architectural aspects, QoS constraints, augmentation of mobile device capabilities, and decision-making policies. As for defining a general layout for MCC, some surveys have regarded the overall challenges and general architectures of MCC [103, 108, 109, 127]. Other surveys have observed MCC solutions that have targeted constraints on QoS [4]. Some other survey works have observed the Cloud-based augmentation of mobile elements [2], focused on computation and storage enhancement (“augmentation”) through offloading [139], or have concentrated on producing an understanding of proposed policies and procedures on the decision-making process within the offloading [137]. Other survey studies have centralized in the synergy of novel technologies with MCC, such as Edge Computing [116, 117], IoT [1], and Big Data Stream Mobile Computing [7], or they have focused on scheduling and load balancing in data centers toward energy efficiency [66]. Contrary to these existing surveys, this study focuses on the particular scope of sustainable computing; it provides an overview of the state of the art on energy awareness in the MCC-based offloading and related Green Cloud Computing strategies. This survey thus classifies related works regarding the problematic models and strategies of preserving the energy of mobile devices while giving a Cloud-based perspective on energy-preserving techniques.

As a result, this survey introduces an ample view of the elements that concern energy aspects toward the profiling of applications and resources, partitioning of application tasks, and actual decision-making procedures from the perspective of both mobile devices and Cloud systems with discussions on simulation-based experimental evaluations and implementation-based principles. This work also develops further the actual open challenges in MCC, delineating the possible research directions in the area. As described in Figure 1, this study splits Sustainable MCC into two major groups: (1) one that concerns the works focused on promoting energy saving in mobile devices, represented by Energy-aware MCC (Figure 2), and (2) another that concerns the related approaches to contain energy consumption in Cloud resources, represented by Green CC.

Fig. 1.
Fig. 1. Sustainable Mobile Cloud Computing interest areas.

Fig. 2.
Fig. 2. Energy-aware Mobile Cloud Computing in a big picture.

The organization of the remaining sections is as follows. Section 2 introduces the concept of MC and MCC systems by detailing the components of their architectures and then illustrates the entire offloading process in the scope of energy-aware MCC systems. Section 3 describes the existing mobile-device-based works regarding the issue of energy reservation and tradeoffs. Section 4 presents the revolution and solutions toward Cloud-based energy-aware resource management. Section 5 describes actual open challenges and presents future research directions. Finally, Section 6 briefly summarizes this survey study.


Even though offloading of computation and data may be considered a complementary feature to mobile computing systems, it has become an essential tool for increasing the autonomy of smart mobile devices and enabling the delivery of rich applications. A thorough understanding of mobile computing and MCC Systems enables efficient design and implementation of offloading strategies, as well as awareness of involved limitations. Therefore, this section presents a brief background on MCC, including motivation, architecture, classification, and offloading basics.

2.1 Mobile Computing

Mobile Computing closely relates to ubiquitous computing, which initially originated from works presented in the early 1990s [129, 130]. These works were the first to delineate ubiquitous computing, describing scenarios that have become commonplace nowadays; they also forecast the aggressive development of mobile hardware and introduced the perception of everyday computing, which represents the unstructured technology-aided activities of the daily routine of common individuals. Stemming from this concept, we can easily conclude that Mobile Computing comprehends a combination of wireless communication, mobility, and portability [50, 114].

These factors have been presented as the ones responsible for defining Mobile Computing in its early years. Even in recent years, we can observe that this representation still stands true for shaping challenges, motivations, and research works behind Mobile Cloud Computing. When it comes to smart mobile devices (SMDs) and MCC, a large number of works, many listed in this survey study, showed the substantial importance of and interest in exploring energy consumption in mobile environments.

2.1.1 Wireless Communication. Wireless communication technology is fundamental for enabling mobile systems and permitting SMDs to have access to the Internet; on the other hand, it is the major concern in realizing such systems due to the already-existing physical challenges for maintaining connectivity and communication quality. In mobile systems, SMDs make use of wireless communication, typically via a wireless LAN or cellular network, to attach themselves to a stationary physical network.

2.1.2 Mobility. The free, independent movement of devices consists of the fundamental nature of Mobile Computing, which refers to support network availability independent of the location, as well as the physical displacement, of devices. As a vital requirement to enable mobility, SMDs usually need to follow protocols and conduct registration procedures whenever entering new environments and areas of interest. Many protocols, such as communication handoff, have been developed as a way to minimize communication interruptions and maintain connectivity as much as possible.

2.1.3 Portability. In broad terms, portability is one of the critical factors in enabling mobility of devices. The nonattachment of devices to a stationary power supply amplifies their independent movement capabilities. Consequently, the use of batteries consists of the feature that allows mobile devices to be carried around, directly enabling, and at the same restricting, mobility. The autonomy, lifespan, or charge of batteries immediately influences mobility, as well as capabilities of SMDs. Portability unconditionally suffers from the lack of a continuous power source.

2.2 MCC System Components

Besides the three major features described in the previous section, heterogeneity is also an inherent factor in Mobile Cloud Computing. Apart from the large, and growing, variety of services and applications, we can notice that the models, capacity, and behavior of SMDs, Cloud servers, and network infrastructure differ widely. However, the elements that compose an MCC system, in terms of supporting offloading and device extensibility, are classified into major categories: smart mobile devices, Cloudlets, a remote Cloud, and networking. These elements play an essential role in MCC and must be considered when offloading computation and data from devices.

2.2.1 Smart Mobile Devices. SMDs are the original motivators for designing and implementing Mobile Cloud Computing. In the whole architecture, they are the entities with enhanced capabilities through offloading techniques, and they are responsible for the original local task execution and data storage. As described later in this study, a local execution is preferred when conditions indicate that offloading to Cloudlets or remote Clouds cannot be performed efficiently, in terms of computation gain, data storage, and energy consumption. Such conditions are dynamic and dictated by evaluations obtained through profiling toward energy and computing efficiency, which includes task patterns, computing capabilities, and the networking environment.

2.2.2 Cloudlets. Cloudlets are the unique, distinct components in the Cloudlet-based architectural model. These elements in the model perform as local resource managers, responsible for connecting local devices, executing offloading requests, and forwarding offloading tasks to remote Cloud resources in case the specific local Cloudlet capabilities are insufficient. Based on the granularity of offloading requests, Cloudlets are designed and implemented differently to cater to device-level offloading or thread-level offloading.

2.2.3 Remote Cloud. The remote Cloud primarily handles the offloading of task executions and data storage, which might be forwarded from Cloudlets or directly forwarded by mobile elements. Based on the definition presented in [98], a remote Cloud performs as a resource pool that contains clones of all mobile tasks. This resource pool can also provide sufficient “unlimited” computing capability to process different types of tasks.

2.2.4 Networking. Networking plays a crucial role in defining the feasibility of offloading in the MCC model. Mobile devices, Cloudlets, and Cloud resources present a considerable variability of capacities, features, and configurations. This diversity, when it comes to the variety of communication interfaces and means, promotes a diversity of connections among these elements.

2.3 Taxonomy of Energy-Aware MCC

The great motivator behind Energy-aware MCC comprehends SMDs becoming increasingly more complex and powerful while recent batteries are still being made with low capacity. In order to achieve portability and availability for mobile devices, offloading has risen as the most effective approach, always seeking to transfer the tasks from mobile elements to externally accessible alternatives in the most efficient manner in the MCC architecture.

Based on the types of external offloading resources, the MCC offloading strategy can be briefly categorized into Local MCC, Remote Cloud MCC, and Cloudlet MCC, respectively. As summarized in Table 1, the local MCC, which consists of only local mobile elements, leads to small overhead, but it is limited to its available computing capability. The remote MCC ensures sufficient computing power, but it introduces the long-distance communication that will lead to larger communication overhead. Subsequently, the Cloudlet MCC enhances both computing and communication efficiency, yet it is restricted by the availability.

Table 1. MCC Deployment Model and Energy-Aware Issues
Deployment Type Pros Cons
Local MCC LAN-Level Latency Restricted Computing Resources
Low Communication Overhead Mobile Device Hotspot
Remote Cloud MCC Unlimited Computing Resource WAN-Level Latency
High Computing Efficiency High Communication Overhead
Cloudlet-based MCC Flexible Energy-Aware Scheduling High Scheduling Load
Low Latency Cloudlet Bottleneck
High Computing Efficiency

2.3.1 Local MCC. As shown in Figure 3, in this deployment model, mobile devices in the local area initiate and maintain a wireless ad hoc network to form a Mobile Ad Hoc Cloud. SMDs can collaboratively help each other in an ad hoc manner. Some designs have envisioned offloading and task delegation into Mobile Ad Hoc Clouds [109], following the most recent trends where Mobile Edge Computing (MEC) is implemented. This approach can drastically reduce communication latency but faces more constrained scenarios due to limited SMD capabilities. Within the network, idle mobile elements can provide the computing power for others. The devices can either trigger the offloading process as resource requesters or execute offloading tasks as resource vendors.

Fig. 3.
Fig. 3. Energy-aware local offloading MCC deployment model.

2.3.2 Remote Cloud MCC. The deployment model is based on remote public Cloud resources as described in Figure 4, typically provided and maintained by public cloud service providers, such as Amazon AWS, Google Cloud Platform, and Microsoft Azure. In order to enable task offloading for mobile devices, virtual instances are established and mapped to mobile users in an always-on mode. Each virtual instance hosts a virtual clone of the specific user device, capable of receiving offloading tasks, scheduling offloading executions, and returning results via a proxy. Due to the data-center-based resource integration and energy optimization technologies, the model based on remote resources can support “unlimited” computing power and storage for the end mobile users and trivially outperform the local-based counterpart in terms of computing efficiency.

Fig. 4.
Fig. 4. Energy-aware remote offloading MCC deployment model.

2.3.3 Cloudlet MCC. This mode, as depicted in Figure 5, involves both local and remote resources. Unlike Local MCC that is established and maintained by mobile devices, the Cloudlet-based model introduces the concept of the Cloudlet as the local resource manager. The Cloudlet typically refers to a static workstation or a series of workstations that are located on the edge of local wireless networks and connected with remote Cloud resources via wired networks. By pooling the local resources and coordinating with the remote Cloud, the Cloudlet can either schedule offloading tasks locally or forward them to the remote Cloud based on the computing capability requirements and energy cost.

Fig. 5.
Fig. 5. Energy-aware cloudlet offloading MCC deployment model.

2.4 MCC-Based Offloading

Understanding the major elements that compose the MCC environment and the essential targeted destinations for energy-aware offloading, we dissect the offloading process into three principal parts. First, in some general sense, offloading extends the Remote Procedure Call (RPC) approach [18], which allows a program, or parts of a program, to run across one or multiple communication networks. Differently from the traditional RPC model, the MCC offloading primarily concentrates on maximizing energy reservation on the mobile elements, which further introduces additional issues, such as coordinating heterogeneous resources on the unstable network.

An entire MCC-based offloading process is suggested to be decoupled into three relatively independent subtasks—partitioning, environment profiling, and decision making and execution, respectively.

2.4.1 Task Partitioning. Offloading is performed at either the application level or device level based on the diversity of applications and the size of the system. With respect to the application requirements, tasks may involve specific-purpose equipment, which may not be available in the Cloud resources, such as GPS, camera, and other sensors. In this case, offloading cannot consider these mobile application tasks due to the lack of corresponding hardware and capabilities in the Cloudlet or remote Cloud MCC solutions. Also, for security concerns, as shown in [118] and [141], applications with sensitive personal information may not be offloaded to a third-party Cloud as a matter of preventing data loss or inter-Cloud malicious attacks.

2.4.2 Profiling. Because of the nature of mobility, the networking and computing environments may change immensely during application execution, under which the offloading efficiency may also vary largely during runtime. In order to decrease the probability that offloading overhead exceeds its benefits, during a certain interval, the execution environment information is necessarily updated as references to make the real-time offloading scheduling decision. The profiled information involves three aspects: the current task load, the computing status of mobile devices and offloading resources, and the networking between them.

2.4.3 Decision Making and Execution. This step consists of the final part in the offloading procedure, which makes the distributed task scheduling decision from a global point of view. Although each task is already partitioned, tagged, and profiled, the entire task set needs a global evaluation to optimize the executions. This optimization consists of one of the most extensively explored aspects in recent works, listed in the following sections.

• Evaluation Model

As this work will describe in the subsequent sections, which summarily enlists recent studies in Tables 2 and 3, an MCC-based offloading approach cannot suit all mobile applications; performance profoundly relies on the execution environments. However, all these works share similar offloading fundamentals: reduction of application execution time and reservation of energy can be achieved if the computational gain and the communication overhead are carefully considered and evaluated. We can then define, in general terms, energy consumption involved with local processing and data transfers and energy reservation, or gain, related to offloading.

Table 2. Summary of Energy-Aware MCC Offloading-Oriented Works (Device Level)
Reference Granu Comp Comm Task M Issue Solution Avai WiFi Cell
BECMC [10] Device N/A Yes Clone Sync Logger Yes Yes Yes
DTMMC [60] Device Yes Yes VM Scale Migration Yes Yes Yes
VMBC [115] Device Yes Yes AbiWord, GIMP Preparation Overlay N/A Yes N/A
CSOSM [140] Device Yes Yes Math, Detection Channel TOPSIS Yes Yes Yes
CDroid [9] Device N/A Yes Clone Latency Proxy No Yes Yes
VD-ABC [40] Device N/A Yes Multi-task Deployment BCO Yes Yes Yes
EnaCloud [87] Device N/A Yes Multi-task Deployment Opt. heuristics Yes Yes Yes
CoopMCC [41] Device N/A Yes Multi-task Cooperation Power-splitting, contract Yes Yes Yes
EA-HRM2 [55] Device N/A Yes Stream Heterogeneity OHTA Yes Yes Yes
DECM [56] Device Yes Yes Multi-task Energy Dyn. Prog. N/A Yes Yes
TIM/EDA [74] Device Yes N/A Multi-task Fairness Auction N/A Yes N/A
SynMCC [82] Device N/A Yes Stream Consistency Sync Yes Yes Yes
MAYA [83] Device Yes N/A Multi-task Cost reduct ILP N/A N/A N/A
LODCO [95] Device Yes Yes Multi-task Exec Cost CPU cycle N/A N/A Yes
E2G [112] Device N/A Yes Stream Proxy Select SPNE N/A N/A Yes
EENA [113] Device N/A Yes App Brokering Decision Matrix Yes N/A Yes
EMOP [120] Device N/A Yes Stream Multichannel DTMC, VIA Yes Yes Yes
Off-MinED [126] Device N/A Yes Stream Tradeoff 0-1 ILP Yes Yes Yes
MCCWET [134] Device Yes Yes Multi-task Offloading/MPT Convex Opt. N/A N/A Yes
OF-MECO [135] Device N/A Yes Stream OFDMA Mix Int N/A Yes Yes
EECO [136] Device Yes Yes Multi-task Latency, 5G Priorities N/A N/A Yes

Granu = granularity; Comp = computation; Comm = communication; Avai = availability; Task M = mobile application/ task; Issue = major tackled issue; WiFi = access to WiFi networks; Cell = access to cellular networks.

Table 3. Summary of Energy-Aware MCC Offloading-Oriented Works (Task Level)
Reference Granu Comp Comm Task M Issue Solution Avai WiFi Cell
CloneCloud [34] Process Yes Yes Search, Scan Flexibility Translator N/A Yes Yes
ThinkAir [81] Process Yes Yes Scan, Puzzle Execution Parallelism Yes Yes N/A
UMCC [96] Process Yes Yes App MAC Shannon form Yes Yes Yes
PEDSA [132] Process Yes Yes Stream Throughput GA N/A Yes Yes
COMET [61] Thread Yes Yes Read-Write Multithread Memory Sharing Yes Yes Yes
CIMCA [47] Task N/A Yes IMCA Delay Routing Yes Yes Yes
MAUI [36] Task Yes Yes Video, Game Overhead Adaptation N/A Yes Yes
EACMA [3] Task Yes Yes Math function Delay Local Process N/A Yes Yes
JACCR [8] Task Yes Yes Multi-task Execution Joint Optimize N/A N/A Yes
MTCCO [94] Task Yes Yes DIA Latency Proxy, Encoding N/A N/A Yes
EaCMD [39] Task Yes Yes Stream Availability Collaborative Yes Yes Yes
ACOMC [48] Task Yes Yes GCM Updating Fuzzy Logic Yes Yes N/A
EeSPC [138] Task Yes Yes Linear Collaborate LARAC Yes N/A N/A
Odessa [107] Task Yes Yes Recognition Adaptation Greedy Profiling Yes Yes N/A
MCC-GNEP [26] Task Yes N/A Multitask Multiuser Game Theory N/A N/A N/A
MECC [32] Task Yes Yes Multiuser Multichannel Game Theory Yes Yes Yes
DNN-MCC [44] Task Yes Yes DNN apps Cooperation Convolution partitioning N/A N/A Yes
EA-HCM [54] Task Yes Yes Multitask Heterogeneity Assignment N/A Yes Yes
eDors [65] Task Yes Yes Multitask Energy Min Max N/A Yes Yes
EDTS [89] Task Yes N/A Multitask Dyn Sched CPA N/A N/A N/A
IDA [92] Task Yes Yes Multitask Decoupling N-LIP Yes Yes Yes
BCSA [93] Task Yes Yes Multitask Exec Cost Selection Yes Yes Yes
TaCO [64] Task Yes Yes Multitask Energy costs Dependency Yes Yes Yes

Granu = granularity; Comp = computation; Comm = communication; Avai = availability; Task M = mobile application/ task; Issue = major tackled issue; WiFi = access to WiFi networks; Cell = access to cellular networks.

Computational energy gain refers to the difference in computing capability between the mobile device and the offloading destination resource for the targeting task set. According to the Dynamic Voltage and Frequency Scaling (DVFS) model, as in Formula 1,

\begin{equation} Power_{dvfs} ()=Power_{static} + Power_{dynamic} = c * f * V * V, \end{equation}
where $Power_{static}$ stands for the leakage power consumption and $Power_{dynamic}$ represents the dynamic power consumption in the scope of minimum CPU frequency and maximum CPU frequency $[f (min), f (max)]$, the computational energy gain during the task execution interval ${\Delta t}$ can be defined as in Formula 2:
\begin{equation} E_{\Delta t} (gain) = \int _0^{\Delta t} (Power_{dvfs} (\textit{offload})-Power_{dvfs} (mobile))d (\Delta t), \end{equation}
where $Power_{dvfs} (\textit{offload})$ is the offloading resource power matrix and $ Power_{dvfs} (mobile)$ is the mobile device power matrix.

The calculation of communication overhead is even more complex due to the mix of different communication media and transmission protocols. Given a simple client/server-based offloading application, the overall communication energy overhead from the perspective of the mobile device can be defined as in Formula 3:

\begin{equation} E (cost) = E (upload) + E (download) + E (wait) + E (extra), \end{equation}
where $E (upload)$ and $E (download)$ calculate the energy utilized for uploading task inputs and downloading process results, respectively, that are highly related to the size of the task and the real-time available bandwidth; $E (wait)$ is the extra energy consumed by the mobile device while waiting for the calculation results of the remote resource; and E(extra) holds the extra energy cost of the mobile device when maintaining the communication channel.

• Implementation Methodology

Profiled information is typically abstracted as different types of parameters in mathematical models, and the decision-making process is to find the best value in these models. Even computation-only offloading scenarios involve several parameters and factors that substantially increase their modeling complexity. In more holistic approaches, energy, computation, and mobility factors are accounted for. New designs require more sophisticated evaluation tools to implement and analyze different experiment scenarios to compare to other works. Throughout the extensive study in this survey, we could observe that the experimental-based comparison is typically oriented in two directions: simulation based and implementation based, both of which concentrate on the calculation of the task execution efficiency and device energy consumption.

In simulation-based studies, the evaluation typically deploys simulators to support the targeting experiment scenarios. The mobile simulators, such as OMNet++ [123] and ONE [75], are utilized to investigate local mobile devices and communications, while the Cloud-based simulators [25, 79] handle Cloudlets and remote Cloud properties and their respective behaviors.

On the other hand, implementation-based evaluations lack standard MCC definition, adopting different implementation strategies and system architectures. In general, enabling mobile application execution externally involves the configuration of mobile devices, offloading resources, and channeling between them. SMDs are primarily smartphones and tablets that execute Android OS [10, 34] or Windows OS [36], which can utilize JVM (Java Virtual Machine) or CLR (.Net Common Language Runtime) to solve the cross-platform issue. The lack of iOS-based Apple devices is mainly due to the insufficient support of MacOS-based virtualization on the Cloud side. The offloading resources differ based on the granularity of the corresponding topics. For instance, the Android-based image manipulation application scenario [10] is built on the Amazon public EC2 Cloud for considering and testing access to remote resources. A VM task overlay [115] is implemented on local servers in order to manipulate local resources. VirtualBox-based Android x86 image processing [81] is deployed on private and public Clouds for testing offloading approaches with hybrid resources. The communication between mobile devices and offloading resources is usually via proxies or middleware. In terms of proxy-based communication, application servers in the middle of mobile devices [81] have been used to dispatch application tasks to corresponding computing units. For the middleware-based solution, MAUI runtime is deployed inside smartphones and servers, which maintain the client proxy and server proxy internally [36]. In the context of communication, Kimberley Control Manager [115] has been adopted to provide service discovery and message transmission.


As discussed in the previous section, the mobile-device-based components concern the offloading issues in the scope of the local region, which involve the handling of task partitioning, profiling, offloading scheduling, and interactions among local components. The feasibility of MCC-based offloading must be first discussed to identify the offloading benefits. Considering that the offloading decision on partitioned tasks is well-established, the suboffloading technologies and studies are then described in further detail.

Figure 6 describes the general process of energy-aware task offloading. The typically local offloading process starts with the target devices entering the coverage zone. The devices first register themselves with the Cloudlet manager so that the Cloudlet can download user images from their remote repository if no local copies are available. After the virtual instances are fully booted, the Cloudlet acknowledges the device about the virtual instance information and QoS information. Based on these profiling data, the devices trigger offloading tasks or only execute the tasks on the mobile devices, based on the energy consumption evaluation. Finally, after the device leaves, the corresponding Cloudlet resources are released for further usage.

Fig. 6.
Fig. 6. Energy-aware local offloading flow.

Figure 7 summarizes the existing, recent MCC works toward reducing the energy consumption of SMDs. The solutions are usually focused on a particular aspect of the MCC environment, such as Availability, Communication Costs, and Computation Efficiency, while at the same time minimizing energy consumption of mobile devices. Orthogonal to these objectives, different approaches have been used as an attempt to achieve the targeted minimization.

Fig. 7.
Fig. 7. Offloading objectives and approaches for reducing energy in SMDs.

3.1 Offloading Objectives

Enabling an MCC-based energy-aware offloading procedure may not always be beneficial regarding energy reservation. Several previous works have evaluated the potential overhead that the energy-aware task offloading might introduce, in terms of computational waste and communication overhead. On the one hand, regarding the communication overhead, as a form of RPC (remote procedure call), offloading introduces a new communication, I/O, and synchronization process inevitably when it is compared to the device-only executions. On the other hand, the computational waste is generated due to the nature of the mobile applications, within which only a certain level of parallelism can only be achieved. As a result, enabling remote offloading implies halting the execution process on the mobile device occasionally, which might significantly reduce resource utilization efficiency, especially for the high-end devices that are equipped with high-frequency computing units. As summarized in Tables 2 and 3, energy-aware MCC offloading approaches span over several goals. However, they in general aim at particular objectives when attempting to reduce or minimize energy consumption of SMDs, considering the availability of cloud resources, communication costs involved with offloading, and computation gain.

3.1.1 Availability. MCC heavily relies on the availability of Cloud services and resources due to its close dependency on mobile networks, such as WiFi and cellular communication. Besides being closely related to the existence of connectivity, availability might concern the selection of the most suitable communication channel and Cloud resources for minimizing offloading costs.

In the sense of identifying the most suitable Cloud resources, CloneCloud [34] has tackled the challenge of determining the most appropriate Cloud resource when migrating or offloading applications and tasks to the Cloud. It also considered the “distance” of the resources from the position of SMDs, defining acceptable offloading decisions based on communication latencies. Thus, CloneCloud conducted a study that emphasized the availability regarding the new MCC offloading process.

In CloneCloud, similar to MAUI [36], which utilizes the .Net framework [121], process-level virtualization is also deployed to establish an intermediate layer that can translate tasks to suit different instruction sets. Thus, mobile devices and static servers can share the same source code. Then, application tasks are suggested to be partitioned automatically based on rules, to keep application developers from partitioning details.

VD-ABC [40] deals with a similar problem as CloneCloud. It determines the optimal deployment of the cloned virtual machine in the Cloud to reduce the communication distance between SMDs and cloned VMS to minimize energy consumption. The approach employs the artificial bee colony optimization metaheuristic with Boltzmann selection polity to tackle the complexity of the problem when determining the best VM placement while acknowledging the offloading of several other SMDs. Consequently, VD-ABC must contemplate the feasibility of VM deployment in case the involved energy costs are substantially high and ultimately do not grant any benefit.

In a more restrictive perspective, EnaCloud [87] focused on the Cloud resources when performing offloading; it implemented a dynamic VM placement to enhance energy efficiency on the Cloud side. By abstracting the VM placement as a bin packing problem, EnaCloud made use of heuristics to achieve a suboptimal placement solution, where it reduced energy consumption and overhead by adjusting resource demands. Focused on the placement of Cloud VMs, EnaCloud disregarded communication costs and latencies when determining the SLA-based placement strategy.

Even though there might be suitable resources for fully implementing MCC, the high volume of offloading requests may lead to a fast lack of Cloud endpoints for receiving tasks and data. Within this context, DTMMC [60] has realized a cost analysis that concerned availability as the number of local users increases. This work first illustrates the lifespan of task offloading in the Cloud and then argues the inevitable task migration overhead brought by the unpredictable workload in such an offloading process. However, due to the lack of global-level information, hotspot servers may easily be overcommitted, which further downgrades the task offloading performance.

TIM/EDA [74] has designed an auction-based offloading within a Cloud-based MCC architecture. Resources are valued differently in this approach, where it holistically considers computation gain, communication cost, link performance, and energy consumption. The auction model considers both Cloudlets and SMDs when offloading tasks. The model aims at promoting collaboration of SMDs and Cloudlets with incentives and better fairness when hosting offloaded tasks. TIM/EDA implements a strategy for evenly distributing load and increasing the availability of resources for further offloading of other SMDs.

Finally, availability is closely related to the connectivity between devices and offloading Cloud resources, which involves coping with the instability or suitability of communication channels and interfaces. When the Cloudlet MCC model is contemplated, new communication opportunities might be available in the sense of additional communication channels, which are represented through different interfaces, and conditions, which are reflected in the proximity of resources. EaCMD [39] concerns the availability of public WiFi and proposes a collaborative offloading strategy that introduces cellular networks as offloading channels. The optimal offloading channel can be selected via channel evaluation between the WLAN and cellular network, which is performed by prefetching location-based caches. The maintenance and updates of the caches are guided by mobility prediction. Based on the comparison and prediction, the collaborative offloading strategy can achieve 80% more energy savings than the naive ones. In EeSPC [138], MCC collaboration is abstracted as a Markovian channel-based DAG, which is further handled by the LARAC protocol.

MECC [32] explores the opportunities in multichannel wireless communication. The approach considers the interference among multiple peers when attempting to access the communication medium for offloading. MECC makes use of game theory for implementing a distributed computation offloading algorithm, coping with the multichannel wireless contention. During the execution, the algorithm analyzes and identifies the benefit of offloading tasks in light of the communication conditions where transmission rates lower than a threshold do not benefit any SMDs.

EECO [136] is similar to MECC [32]; it also leads with the multichannel communication conditions and multiaccess characteristics of 5G when offloading to the remote Cloud in a Mobile Edge Computing scenario. EECO defines an optimization model to minimize the whole system energy consumption and to satisfy the latency constraints of mobile applications. Thus, the model accounts for the costs of local computation and data transmission in terms of energy consumption. The model follows a three-stage scheme for energy-efficient computation offloading where it classifies communication channels and prioritizes assignments to handle multiuser task offloading in 5G.

UMCC [96] places MCC in a much broader sense, Smart Cities, where it considers several Cloud topologies: Central Cloud, Cloudlet, and Distributed Mobile Cloud. The first objective of this approach consists of dealing with the heterogeneity of communication technologies. As a result, the model contains a utility function that considers application factors, such as latency, energy consumption, and bandwidth consumption. UMCC makes use of the Shannon Formula to solve this optimization problem where a weighted sum of offloading entities is implemented following different policies: greedy, cluster based, and biased randomization.

3.1.2 Communication Cost and Profiling. Due to the nature of the distributed MCC environment, energy-aware offloading cannot ignore communication characteristics involved with data transmissions. Availability closely relates to communication in terms of connectivity of SMDs over networks. Assuming that availability is already achieved, the cost of communication is a fundamental, delimiting factor for quantifying the actual benefit of MCC offloading.

The dynamic communication changes that might happen in mobile environments have served as a strong motivator to determine precise networking conditions as a decision-making parameter to promote MCC offloading. As a result, EACMA [3] analyzed communication overhead by evaluating the impacts of the communication distance and the number of intermediate hops. Similarly, this subject has been tackled in VMBC [115] and CIMCA [47], which suggested that WAN-level communication may compromise the usability of MCC offloading. The feasibility evaluation is also performed in terms of energy consumption and execution efficiency.

Bound to a narrower class of applications than in EACMA [3], BECMC [10] focused on determining communication overhead for dealing with data synchronization. It tested the task offloading performance of Android smartphones on the Amazon public EC2 Cloud platform [5]. In the proposed scenario, system-level information and the application code are tracked and monitored passively. Passive information is extracted through the Intents class, while Active information is driven by the instantiation of the Alarm class, which periodically triggers data collection.

Odessa [107] also analyzed the communication costs for MCC offloading, so it concerned dynamic profiling and its benefits toward adaptation during real-time offloading. This approach proposed a greedy profiler that can periodically extract piggyback environmental stage data, which is further utilized to estimate the execution bottleneck and scheduling adaptation.

Similar to BECMC [10], MAUI [36] has evaluated the energy consumption on both the data transmission and the profiling overhead. For the communication overhead, it tests a typical code uploading scenario under which a small piece of the task is sent from mobile devices to remote resources via 3G or WiFi. The results presented two positive correlations: RTT (Round-Trip Time) versus energy consumption and throughput versus energy consumption. Under a certain throughput value, 3G uploading may consume three to five times more energy than WiFi uploading. Specifically, the paper emphasizes the impacts of WiFi's Power-Save Mode (PSM). The PSM is intended to reserve energy by turning the devices into passive mode when no data is sent. However, the device state transmission cost from passive mode to active mode also introduces latency, which causes higher energy consumption for small transfers.

The restrictive communication conditions that constrained offloading in MCC models have been relaxed with Cloudlet MCC architectures. The shorter distance between SMDs and Cloudlets have allowed better offloading gains and energy reservation. CIMCA [47] has performed a transmission cost analysis study on the decision-making network, which has tested the offloading performance between mobile devices and Cloudlets in the scope of interactive offloading tasks. CIMCA achieves task offloading by one or multiple reachable Cloudlets that are peers to each other. The work also evaluates the task offloading performance based on three types of tasks with decentralized routing and centralized routing. CIMCA suggests that Cloudlet-based offloading can provide larger throughput and lower delay when compared to the remote Cloud.

As a way to counter the costly communication latencies, VMBC [115] has explored the reduction of the amount of data involved with MCC offloading. VMBC targeted the long-distance transmission overhead with the VM preparation process. The output of this work suggests that the WAN-level latency between mobile devices and Cloud resources may significantly restrict the usage of offloading applications even if sufficient bandwidth is provided. Concerning the reduction of VM state transmission overhead, the work proposed a VM overlay model to enable reduction of the size of the offloading image.

Correctly assessing the dynamic communication status of an MCC environment allows for more successful offloading decision making. Even though communication costs are not the primary objective in many offloading models, they have been recognized as critical for allowing overall execution and energy gains. EA-HRM2 [55] implemented a model that contemplates the communication costs for offloading tasks to the Cloud. In heterogeneous embedded environments, the access to the wireless communication interfaces may pose a high cost in terms of energy. Thus, the task assignment model of EA-HRM2, even though focused on computation gains, heavily relies on the energy costs involved with data transmissions.

Similar to EA-HRM2 [55], EMOP [120] developed a strategy to optimize offloading in terms of energy consumption considering the changing communication conditions within mobile environments. EMOP introduces an energy-efficient multisite offloading policy that uses Discrete Time Markov Chain (DTMC) to model fading wireless mobile channels and dynamicity of SMDs. The Markov Decision Process (MDP)-based framework formulates the multisite partitioning problem, which considers communication cost against the offloading gains concerning energy.

In the same context as both EA-HRM2 [55] and EMOP [120], eDors [65] dealt with conciliating two major dynamic offloading objectives: reducing execution time and energy consumption. The model focuses on the computation aspects of the tasks. However, to reduce energy consumption, it recognizes the communication costs for each offloaded task. Thus, eDors identifies the suboptimal tradeoff between the two objectives through Max-Min heuristics where transmission power is controlled to favor offloading depending on dynamic communication conditions.

Comparable to eDors [65], MinED [126] studied the tradeoff between energy reservation and communication delay. In this case, energy reservation is understood as primarily promoted through computation offloading. Thus, the model acknowledges that offloading promotes energy reservation by relieving local processing in SMDs. However, offloading incurs costly application delays due to inherent communication latencies. MinED then copes with this duality by formulating a joint optimization problem through 0-1 Integer Linear Programming (ILP).

Communication costs not only involve the surrounding networking environment of SMDs but also the characteristics of applications that are being offloaded. The frequency at which synchronization needs to be conducted for keeping offloaded tasks consistent dictates the communication overhead throughout the application lifespan. Consequently, SynMCC [82] introduced a model that acknowledges data synchronization for maintaining the consistency in the duality between SMDs and their respective cloned Cloud VMs. Synchronization frequency is dictated by the amount of offloaded data and changes in the data. Thus, SynMCC implements application and task profiling in SMDs for selective offloading decision making. The decision making attempts to reduce energy consumption through minimal data synchronization between SMDs and the Cloud.

Opportunism is another exploited factor in Smart City scenarios where there exist diverse network interfaces in SMDs that can extend connectivity and promote better conditions for MCC offloading. UMCC [96] considers heterogeneous urban environments for MCC. The framework focuses on the availability through the connectivity of SMDs over several Cloud topologies. However, besides maximizing connectivity, UMCC attempts to reduce the communication costs when there are many communication technologies. The framework achieves this cost minimization by using the Shannon Formula to solve the optimization problem.

Sharing the same objective of reducing the energy costs of communication as in UMCC [96], E2G [112] designed a model for efficiently transmitting data to the Cloud in an MCC system. The policy behind this approach assumes that an SMD bridging the costly communication with the remote Cloud is the most effective strategy to reduce data transmission costs of neighboring SMDs. In order to promote fairness among SMDs and avoid any selfishness, the model is based on Subgame Perfect Nash Equilibrium (SPNE). SPNE allows E2G to select an SMD responsible for bridging the communication and minimize the whole energy cost of transmissions.

The new trends in cellular networks have also enabled exploring the beneficial characteristics of 5G for enabling multichannel, multiaccess data transmissions with the purpose of boosting offloading. For instance, EECO [136] accounts for the costs of local computation and data transmission while acknowledging the multiaccess characteristics of 5G, observing Orthogonal FDMA (OFDMA). Besides concerning accessibility in 5G, the model minimizes the whole system's energy while satisfying latency constraints by considering the energy consumed in the macro and small base stations when offloading. The scheme behind the model presented a three-stage energy-efficient computation offloading through classification and priority assignment.

In the same 5G context of EECO [136], OF-MECO [135] introduced a model to maximize local communication resource allocation for Mobile Edge Computing scenarios. In a multiuser environment, the model explicitly deals with two Medium Access Control (MAC) protocols: TDMA and OFDMA. OF-MECO makes use of a threshold-based convex optimization through priorities to cope with infinite and finite Cloud capacities. On the other hand, the model formulates a mixed-integer problem to maximize channel gains against local computing energy consumption through Lagrange and Karush–Kuhn–Tucker (KKT) conditions.

Very close to EECO [136], EENA [113] implemented an energy-efficient and network-aware offloading algorithm where communication between SMDs and the Cloud is realized over 5G technology. The suggested model makes use of the 5G characteristics and contemplates the use of Cloudlets, or brokers, for connecting SMDS with the Cloud servers. EENA employs decision matrices with dynamic parameters for regulating offloading to minimize energy consumption and involved communication costs.

With technologies toward the practical implementation of Microwave Power Transfer (MPT), new possibilities can be explored to extend the battery lifespan of SMDs together with MCC strategies. MCCWET [134] focused on energy reservation through several strategies: controlling SMDs’ CPU cycles, harvesting energy through MPT, and selecting offloading. By assuming the full availability of MPT, the framework employs Convex Optimization Theory to minimize mobile energy consumption by selectively opting for offloading or controlling local power. MCCWET makes use of the threshold to interleave MPT with offloading and guarantees transmissions with optimal data allocation for dynamic channels.

CoopMCC [41] expands MCCWET [134] by collaboratively allowing SMDs to help each other in an ad hoc fashion. CoopMCC explored the cooperation among SMDs to extend the lifespan of their batteries. The approach implemented a cooperative contract where it attempts to achieve the mutual benefit of two types of mobile devices: energy-poor and data-poor. CoopMCC relies on a power-splitting (PS) policy to generate an optimization closed-form model where communication can be exchanged with energy. Energy-poor helps data-poor by relaying data, and data-poor helps and allows energy-poor to harvest its energy through power transfers. The model identifies the optimal PS factor and transmission power as a form to stimulate cooperation and exchange and achieve an optimal cooperation contract.

3.1.3 Computation Efficiency. Extending the computation capabilities of SMDs consists of a primary MCC functionality where communication costs cannot be denied due to their inherent presence in any mobile environments. With the objective of enhancing the processing of applications and preserving SMDs’ energy, offloading schemes have employed subpartitioning of tasks, scheduling based on task dependencies, and Cloud resource “limitations.”

Unlike the previous objectives that mainly concentrate on reducing overhead or coping with the availability of Cloud resources, some approaches sought to maximize the computing efficiency that MCC offloading can support. An offloading architecture has been introduced in ThinkAir [81], exploring the parallelism in the MCC-based offloading. In this proposed architecture, the application subtasks are partitioned with the ability to execute simultaneously. The offloading performance proves to be effective for computing-intensive tasks that involve little data communication during execution but large computational load.

Similar to ThinkAir [81], JACCR [8] evaluated offloading under which computation-oriented tasks are scheduled and mapped to many Cloud Computing units. This pairing enables multiple virtual instances to serve different application tasks simultaneously, increasing the level of parallelism of offloaded tasks in Cloud virtual servers. The results show that the proposed MIMO (Multi-Input Multioutput) offloading model can extensively reduce the task queueing compared to the device-only execution with the same task sets.

Following the task subpartitioning strategy and when it is possible, finer-grained tasks can lead to more flexible offloading. BCSA [93] introduced an optimal selection algorithm that makes use of adaptive task partitioning and dynamic selective offloading strategies. The primary target of BCSA consists of minimizing execution costs where local energy consumption, execution time, and execution costs are the major parameters of its search. These parameters consider the communication delay and costs, in terms of energy, involved with the offloading process.

In contrast to BCSA [93], DNN-MCC [44] explored the particular characteristics of Deep Neural Networks (DNNs) for optimizing the performance of applications when offloading them to the Cloud. Though a mobile-Cloud collaborative approach, rich DNN-based applications, such as speech recognition and intelligent personal assistant (Siri), can be deployed more effectively in SMDs. The model exploits the partitioning of heavy DNN convolutions for effectively boosting execution and reducing energy consumption, relying on data exchanges.

Due to the popularity of streaming and gaming applications in the mobile devices, some works try to augment the performance of these applications, such as MTCCO [94]. MTCCO evaluates the offloading performance in CDIA (Cloud-based Distributed Interactive Application) to traditional DIA (Distributed Interactive Application). Due to the limited CPU and GPU capabilities, the proposed scheme treats mobile devices as lightweight terminals only responsible for presenting streaming data. On the Cloud side, a mobile client is deployed to communicate with the application server on behalf of the corresponding mobile user, which forwards user commands to target proxies and routes streaming data back to the user's screen. In this case, mobile devices can leverage the gaming processing entirely to the Cloud and be in charge of only uploading instructions and showing results.

A relevant work is also shown in PEDSA [132] for data stream tasks. Concerning the maximization of the application makespan and throughput, a GA-based algorithm is proposed for the stream task partitioning, which begins with random selection and ends with maximal computing efficiency or largest throughput. The evaluation presents that using the proposed task partitioning model can always outperform the all-mobile or all-cloud naive approaches on the graph generation application. However, the computational cost of the mobile device is not fully discussed. Based on the DVFS mobile device computing model, a shorter task execution time may not necessarily lead to smaller energy consumption, due to the high computation load introduced by partitioning and channeling.

Even though task partitioning is necessary for allowing offloading to Cloud servers and achieving a certain level of parallelism of applications, scheduling decision making is fundamental for guaranteeing proper task execution to reach substantial computation gains. In that sense, CSOSM [140] introduced a client-server-based MCC system, tackling the entire task partitioning, profiling, and decision-making process. In CSOSM, the offloading scheduling process is abstracted as two aspects—offloading channel selection and resource mapping. The MCC system uses the TOPSIS offloading decision-making algorithm, which is deployed to the multicriteria wireless interface election, while the MIN-MIN heuristic is implemented on the Cloud components to process tasks. As the experiment shows, the proposed system can outperform ThinkAir [81], reducing execution time to two-thirds and preventing half of energy consumption.

In order to cope with the decision-making complexity, ACOMC [48] employed a fuzzy-logic-based code offloading engine on top of the COMET virtualization platform [90]. In ACOMC, environment profiling information is translated into crisp sets. Based on the fuzzy threshold rules, these crisp sets further generate offloading decisions. Experimental results showed offloading improvement that led to an average 10-second execution time for message delivery.

COMET [61] enabled distributed shared memory between mobile devices and the Cloud resources via Dalvik virtualization [20]. Instead of PRC, applying DSM can support flexible thread-level migration, which is presented to reach 15 times higher execution efficiency.

TaCO [64] introduced a DAG-based analysis over the offloading costs and benefits to select a task to run in either a Cloudlet or a remote Cloud server. It implements a task-centric strategy so that a finer granularity offloading model is enabled. Also, lighter application cloned VMs are used instead of the traditional SMD cloned VMs. This scheme imposes higher overhead toward low density of SMDs but compensates in saturated offloading requests.

In smart city or urban computing environments, the scale of the system adds multiuser scenarios where access introduces stress to resource sharing and application offloading. The challenge in such conditions consists of dealing with the selfishness of users and promoting fairness when allocating MCC resources for improving execution and reducing energy consumption. MCC-GNEP [26] explores “open offloading” in an unmanaged, multiuser scenario. The approach makes use of a game theory model using queuing theory to deal with the selfishness of users where the MCC system follows a Cloudlet architecture. MCC-GNEP employs a distributed algorithm for the model's architecture to achieve an equilibrium as a way to address the multiuser offloading scenario. The model assumes that applications have been already partitioned, statically or dynamically, and methodologies have determined which tasks to offload.

Similar to MCC-GNEP [26], MECC [32] considers a MEC scenario where it acknowledges multiuser computation offloading in multichannel wireless interference. Besides attempting to solve the availability of Cloud resources, the approach also uses game theory for the distributed computation offloading algorithm to cope with the selfishness of users when offloading. Also, it extends the offloading model into a multichannel wireless contention.

The distributed MCC environment inherently requires dealing with communication issues even when energy-area offloading approaches emphasize computation efficiency. Even disregarding the interdependencies of tasks, offloading depends on the networking infrastructure for sending tasks to run and retrieve their output. In this regard, EA-HCM [54] focused on task assignment into heterogeneous Cloud cores. This approach extends on a previous work, EA-HRM2 [55], where communication costs are heavily considered for the offloading decision making. EA-HCM implements a heterogeneous task assignment algorithm to reduce energy consumption involved with offloading but still considering wireless communication costs. Thus, this model solves an energy minimization problem observing heterogeneous computing and data preprocessing.

eDors [65] was listed in the previous section as an offloading strategy relying on controlling the transmission power of wireless channels to tackle high communication costs. eDors has the reduction of energy consumption and application execution time as its primary offloading objectives. Its model solves an energy-efficiency cost minimization problem in light of the task dependencies through a dynamic partition scheme over a DAG of task dependency. In the end, eDors implements a distributed algorithm based on computation offloading selection, clock frequency control, and transmission power allocation.

SynMCC [82] employs selective offloading decision making to minimize the frequency in which data is synchronized between SMDs and their respective cloned Cloud VM. Even though this approach has been listed as oriented to communication costs, its core offloading strategy relies on controlling the computation gain through different scheduling policies: periodic, programmatic, event-based, and resource-intensive update.

As described previously, for EMOP [120], although it closely considers communication costs in its model, its major focus formulates on multisite partitioning. The partitioning policy is built on a value iteration algorithm based on a state transition graph. A Markov Decision Process framework allows one to estimate mobile wireless channels for offloading tasks to multiple Cloud servers.

Most of the offloading techniques assume that applications allow partitioning in subtasks and certain level dependency on the local SMD resources. When we observe practical examples of mobile applications, we notice that they present time constraints for completing operations and returning outputs. Therefore, IDA [92] has dealt with applications’ time constraints in a multidevice scenario when offloading tasks. The model attempts to reduce the overall energy consumption of several devices in a Mobile Cloud where tasks are offloaded considering the involved cellular communication costs. The task scheduling of heterogeneous applications is mapped as a nonlinear integer programming problem where application time deadlines and transmission error rates are the major problem constraints. IDA, the interactive decoupling algorithm, is then used to combine two methods, linear relaxation and convex optimization, for reaching a scheduling solution.

In a similar context as IDA [92], EDTS [89] designed a dynamic task scheduling algorithm to minimize the energy consumption of mobile applications in light of constraints related to time and application probability. The algorithm makes use of schemes to reduce energy costs related to local processing and offloading by identifying critical paths in dependency task graphs. A suboptimal scheme solves the scheduling problem by using the Data Flow Graph Critical Path (DFGCP) and an optimal solution that addresses the Critical Path Assignment (CPA) with dynamic programming.

All described offloading techniques deal with a significant level of search complexity to achieve plausible suboptimal energy and execution gains. For the sake of simplicity, local offloading decision making is sought as an alternative to solve the energy and execution issues of SMDs in their local scope by assuming that Cloud resources are optimally available and suitable. LODCO [95] made use of a low-complexity online algorithm that implemented decision making depending only on the local mobile device's information and execution status, such as consumed CPU cycles and needed transmission power for offloading. The model minimizes the execution cost of mobile applications in the context of Mobile Edge Computing environments by using Lyapunov optimization-based dynamic computation offloading.

MAYA [83] dealt with a minimization of energy consumption and offloading costs in MCC environments. The model assumed that energy consumption was reduced through the execution of tasks in remote Cloud servers while the execution of the same offloaded tasks is posed as monetary service costs. The models had as a major parameter the classification of SMDs according to their capacity to afford homogeneous Cloud services. The energy and cost minimization is mapped as an Integer Linear Programming optimization problem to determine the best suitable task scheduling in a three-tier Cloud model.

3.2 Problem-Solving Approaches

From the enlisting and description of offloading techniques in previous sections, we can clearly notice that, orthogonally to them, many approaches have been used to solve the optimization issues involved with minimizing energy consumption, maximizing execution efficiency, and minimizing energy-related communication costs. These optimization constraints may be understood as disjoint and requiring tradeoff thresholds or policies to determine the precise “offloading level” of an application before it might start harming either computation efficiency or energy reservation. Heuristics, meta-heuristics, Stochastic Optimization, Game Theory, and Combinatorial Optimization have been used extensively in order for these offloading strategies to implement performance of such complex offloading scenarios.

3.2.1 Heuristics. As illustrated by the offloading approaches described in the previous sections, heuristics allows for solving scheduling and placement problems faster when compared to traditional, formal closed-form models. Usually, heuristics leads to approximate solutions that might be very close to the optimal solution of a problem, approximating the exact solution. For searching algorithms, the heuristic function allows one to rank alternatives while the search is conducted, performing as a fundamental element when defining the convergence of the search.

  • Convex Programming allows for minimizing convex functions, assuming that the local minimum is the global minimum in the general case of the problem to be solved. In the context of MCC offloading, MCCWET [134] has used convex optimization to minimize the mobile energy consumption.
  • Integer Linear Programming (ILP) “simplifies” the problem by assuming some or all variables to be integers when solving an optimization problem. Usually, the ILP objective function and constraints are linear. For solving an ILP problem, Linear Programming relaxation can be applied to relax an NP-hard problem into a polynomial solution; however, the relaxation may involve violating constraints, may not be possible, or may not be optimal. In the context of offloading, MAYA [83] uses integer linear programming to minimize energy and Cloud service cost when requesting Cloud servers to offload SMDs. MinED [126] also formulated a joint optimization problem by using 0-1 integer linear programming to minimize energy consumption and communication delay.
  • Nonlinear Programming allows one to optimize a nonlinear, or nonconvex, problem by defining an objective function that works over a set of constraints, system equalities and inequalities, and a set of unknown real variables. The method either maximizes or minimizes the objective function, the optimized targeted objectives. IDA [92] has employed Nonlinear Programming to optimize offloading in light of the cellular communication costs.
  • Dynamic Programming recursively breaks down a complex problem into simpler subproblems. Finding the optimal solution for the original problem consists of identifying and/or joining the optimal solution in the subproblems. EDTS [89] has used Dynamic Programming to identify the critical path in the task dependency graph and determine an optimal scheduling solution, considering time and application probability constraints.

3.2.2 Meta-Heuristics. Different from heuristics, which are designed to deal with the specific class of problems, meta-heuristics represent a set of “general purpose” heuristics that can be adapted, or applied, to any class of scheduling optimization problem. Evolutionary algorithms have been extensively used to search for optimization solutions. VD-ABC [40], for instance, has used the Artificial Bee Colony (ABC) algorithm with the Boltzmann selection policy to determine VM deployments as a way to minimize the communication cost with SMDs in MCC.

3.2.3 Stochastic Optimization. Stochastic optimization is broadly used to represent the random behavior of mobility and transmission uncertainty in wireless communication channels. With a random probability distribution, a stochastic model allows estimating communication conditions when they cannot be predicted accurately. EMOP [120] has employed Discrete Time Markov Chain to model wireless mobile channels.

3.2.4 Game Theory. Game Theory allows for solving conflict and cooperation problems. In the context of MCC, the cooperation of selfless users may lead to fairer split and access to Cloud resources, achieving overall higher energy preservation through offloading. MCC-GNEP [26] uses Queueing Theory to address a multiuser, unmanaged scenario and achieve an equilibrium where open offloading is implemented. MECC [32] also assumes a multiuser scenario and employs Game Theory for the distributed computation offloading algorithm that is then extended into a multichannel wireless contention environment. E2G [112] defines a model based on Subgame Perfect Nash Equilibrium (SPNE) to select an SMD to communicate with the Cloud.

3.3 Critical Analysis of the Offloading Techniques

The key explorations of recent MCC offloading works are summarized in Table 4. The open issues and potentials can be categorized based on three main offloading functionalities—task partitioning, profiling, and decision making—as follows:

Table 4. Open Issues
Process Issue Description
Task Partitioning Compatibility CPU instruction sets, computing capacity
Usability How to migrate traditional applications to MCC
Profiling Accuracy Runtime environment may change drastically
Overhead Frequent and accurate profiling may exhaust battery
Fast Response Needs to be simple and quick
Decision Making and Execution Complexity Quick optimization with estimated inputs
  1. Task Partitioning
    1. Compatibility. Hardware on mobile devices and Cloud resources differs in terms of architecture and instruction sets. A typical mobile ARM-based CPU [53] utilizes RISC instructions, while the newest server-side x86-based [22] Intel Skylake processor is based on MMX, AES-NI, CLMUL, and FMA3 instruction sets. This difference hinders servers to directly process applications that are designed for mobile devices.
    2. Usability. In the layered MCC architecture, application developers are supposed to concentrate on the issues only related to applications. Unfortunately, the standard MCC deployment model is not officially defined. As a result, application designers are typically required to estimate the task offloading performance and partition tasks for mobile devices and Cloud resources entirely via supportive APIs or partially through the automatic partitioning tools. A significant amount of effort on reworking might be inevitable to enable traditional applications to execute on the MCC execution flow.
  2. Profiling
    1. Accuracy. Unfortunately, the profiled data is not $100\%$ correct even if the most precise instruments are equipped. This situation occurs due to three main reasons. First of all, the profiled data, such as mobile device current battery volume, CPU frequency, and available network bandwidth, are all continuous values. During each profiling interval, these values are detected and assumed to be the average value of this particular interval. A small range can help these discrete numbers converge to the actual ones, but continual profiling potentially increases the computing overhead and similar energy cost, which in return reduces the entire offloading profits. Second, the profiled data needs to be transferred to the processing unit, so a delay exists between processing time and profiling time.
    2. Overhead. Unlike the task partitioning that can be processed prior to the real application execution, profiling is typically performed on the fly while the application is triggered to ensure the accuracy. In this case, the overhead of profiling exercises more influences the offloading efficiency than the task partitioning, which requires the extra occupied computing and storage to be minimized.
    3. Fast Response. Multiple objects are required to be profiled quickly because these results are the input values for the next stage, although the profiling involves different detecting models and parameters. For the profiling of task load, a commonly used energy evaluation model consists of DVFS in [33] and [106].
  3. Decision Making and Execution
    1. Complexity. Decision making is the core and the most complex part for offloading. As discussed in the profiling step of the process, all profiled data are “estimated” correctly, which cannot achieve full correctness. Based on only these inputs, the decision process is required to produce final global optimization to trigger the real execution. Furthermore, concerning the computing capability and execution overhead on Cloudlets or mobile devices, the optimization is supposed to consume negligible load and time overhead, as little as negligible as possible.

3.3.1 Task Partitioning. Task partitioning needs to handle the diversity of hardware and software. The Hardware Layer issue typically refers to the CPU architecture sets; this issue requires the source code to be translated into different machine-executable bits accordingly. The Software Layer problem consists of enabling developers to migrate traditional mobile applications to MCC applications with as little effort as possible.

To tackle the usability issues, MAUI [36] seeks to minimize the effort of developers when designing offloadable applications by introducing annotations. The proposed architecture encourages developers to simply tag all classes as remote executable if they are not related to mobile-specific components and interfaces. In terms of compatibility, MAUI employs a deployment strategy to coordinate an ARM-based mobile CPU and x86-based server CPU [110]. In this case, the intermediate language can be automatically translated into different machine languages respectively, and the same source code can be executed on different platforms.

CloneCloud [34] utilizes Dalvik virtual machines [70] integrated with a Java compiler front- end to handle the compatibility. In addition, the task partitioning components can smartly tag the tasks as remote executable or local executable on the class or method level, without the developer's efforts. In this case, traditional mobile-device-based Java applications can operate seamlessly in the Mobile Cloud environment. The self-managed task partitioning limits the scope of remote tasks due to mobile-specific features and the device-VM one-on-one mapping.

Instead of implementing virtualization at the application level, ThinkAir [81] has suggested deploying an Android-based x86 virtual machine [69] directly on the Cloud. This remote deployment may introduce larger overhead while it provides higher authority that can further enable parallel task execution. Instead of automatic partitioning, ThinkAir also requires manual annotation via APIs, which may further support user-defined parallel algorithms.

In other instances, VMBC [115] also argues to utilize hardware-level virtual machines to handle the compatibility. This approach advocates that virtual instances can support a broader range of applications. Also, they are not limited by the specific languages in which applications are coded. In addition, the establishment and recycling of virtual instances are already well managed on the Cloud, which can directly support MCC applications in the same way as traditional software.

On the one hand, hardware-layer solutions outperform process-level ones in terms of the flexibility and security that virtual isolation brings. On the other hand, the virtual instances generally involve larger computing power waste. Currently, standard sample base images are still not available from public Cloud service providers. It requires smartphone designers and mobile OS developers to coordinate, define, and tailor base offloading images for different phones and tasks.

3.3.2 Profiling. The data profiling refers to computing capacity profiling and networking profiling, respectively, the results of which are the main references of the final offloading decision.

  • Computing Profiling. The computing capacity profiling involves mobile CPU profiling and server computing unit profiling, both of which are performed on the DVFS energy management model. DVFS, as shown in the study described in [105], can reserve energy through delaying task execution. The model changes the voltage during runtime, and the CPU frequency can scale up and down based on the task requirements. In this case, achieving energy efficiency may not necessarily mean poor support for real-time applications. As proposed in [106], instead of calculating average execution time, the profiling model concerns the worst-case boundary and using look-ahead to reduce energy costs for embedded devices further.
  • Networking Profiling. Unlike the computing units, the network environment, especially the wireless LAN, is not stable. Although network parameters, such as bandwidth and RTT, can be predetected before offloading, they may change drastically during execution.

In CloneCloud [34], the profiling data are simplified as CPU availability, screen display, and network availability. Since it assumes a one-on-one task VM mapping and ignores the energy consumption of the Cloud resources, the proposed profiling model can detect these parameters easily and minimize the overhead that the profiling process brings.

MAUI [36] classifies profiling behavior into device profiling, program profiling, and network profiling, respectively. The device profiling concentrates on the CPU utilization and predefines a linear relationship between CPU usage and energy consumption for the specific mobile device based on multiple experiments. In terms of network profiling, MAUI requires basic network parameters including RTT, bandwidth, and package loss rate. Similar to device profiling and program profiling, MAUI also deploys the test-collect method to fetch these values.

ThinkAir [81] also organizes the profiling process into three subdomains—task, hard devices, and network. The hard device profiling is straightforward, which reads the CPU usage and screen and records WiFi and 3G sensors with two states. The task profiling in ThinkAir directly utilizes standard Android APIs to document the task-related information such as execution time, required resources, uploading task size, and others, specifically tailored for Android phones and pads. For the network characteristics, it requires data regarding RTT, bandwidth, and send/receive rates that are gathered by test packets.

As can be noticed in these profiling methods, the core concern consists of the overhead that profiling itself may bring. The profiling process not only needs to occupy at least one thread to read the APIs or prestored data but also consumes energy. The solution usually comprehends to prefetch test data in advance and simplify the execution time data collection.

3.3.3 Decision Making and Execution. Based on the profiled data, the decision-making process seeks to produce a global optimization for the entire task offloading. The problem is generally mapped to the network path forwarding issue. Given a task set that involves $N$ offloadable tasks and that each task is linked to some others based on the reliant relationship, the task execution scenario can be further presented as from the first one to the $N$ task, via many possible paths. In this case, the profiled coefficient can be added into the links between tasks and the decision goal is to find the path with the lowest weighted values.

Since both MAUI [36] and CloneCloud [34] assume a centralized execution, they suggest using Shortest Path algorithms that only run in $O (Nlog (N)),$ where $N$ is the number of subtasks in a typical mobile application in which the number of edges is highly limited. Although the decision-making overhead is controlled to a certain level, it depreciates repeated scheduling and suggests to use previous results unless the profiling results changed drastically.

The issue on the decision making drastically increases in complexity when distributed offloading is enabled. In ThinkAir [81], the program profiler tags the tasks that can be executed simultaneously and calls the execution controller to arrange extra VMs to parallelize the labeled tasks. Due to the complexity of profiling and decision making, only recursive algorithms and large-scale data analysis are supported in this mode.


In the scope of MCC offloading, the “Cloud” generally refers to all external utilities that are distributed outside the mobile devices. According to the mobility of the utility, the outside resources can be classified as static resources and mobile resources. In terms of the static utility, these raw resources are typically organized and clustered in the form of Grid Computing or Cloud Computing. The Grid and Cloud architecture encapsulate underlying resource management details and virtualize resource pooling services that can be easily accessed via the Internet. In terms of the mobile utility, the resources are organized based on the wireless ad hoc network, which dynamically maintains the offloading services according to the local device connectivity, the studies of which are discussed in the previous mobile-device-based section. In this section, the state of the art of Cloud service provisioning is investigated regarding energy-aware MCC offloading, based on the static utility.

The concept of Cloud Computing, as well as Grid Computing, is proposed for layering traditional computing capability and resources into the raw components, middleware, and applications. The advancement of applications and distributed systems has leveraged the demand for computation and communication resources, as well as simple, flexible, and transparent access to services, which have been made available through a web interface in most cases. This growing demand for resources has boosted the development of solutions that push toward paradigm shifts in the method in which resources are provisioned and managed.

As shown in Figure 8, the general “Cloud” system typically involves three layers—the soft layer SaaS, the intermediate layer PaaS, and the hard layer IaaS, which are responsible for task mapping, virtual machine management, and infrastructure deployment, respectively.

Fig. 8.
Fig. 8. Cloud offloading supportive model.

The offloading approaches described in the previous section, such as MAUI [36], CloneCloud [34], PEDSA [132], EaCMD [39], and Odessa [107], presented practical MCC use cases that involved data streaming, pattern recognition, and gaming applications in local-range and remote offloading. The elastic resource provisioning in cloud-based architectures allows for more flexibility in tailoring such relatively lightweight computing applications according to real-time load requirements.

According to the layered service provisioning model defined by NIST in [98], studies on energy-aware scheduling in Cloud Computing can be briefly divided into three areas: the handling of tasks and processes, the management of virtual resources, and the optimization of hardware resources. For each layered model, the energy-aware offloading studies are further categorized based on the Cloud resource functionality model—computing-centric model or communication-centric model—targeting the tradeoff between execution efficiency and energy reservation.

The type of raw resources consists of a conditioning factor in classifying the hardware resource optimization issues, which are divided into computing resource optimization, storage resource optimization, and networking optimization, respectively.

4.1 Hardware Supportive Infrastructure

The hardware-layer studies focus on the virtualization technologies, which are led by Xen, VMWare, KVM, and AWS.

In both Xen_Comp [11] and Xen_Net [99], the development of Xen-based virtualization technologies are demonstrated, and their performance is measured. In the context of performance, Xen_Comp [11] presents the concept of paravirtualization. It alleviates memory and CPU overheads by migrating partial tasks from virtual instances to hosts. Concerning the reduction of CPU overhead, Xen allows guest system calls accessed by processors precisely without bothering ring 0 VMM (Virtual Machine Monitor). In terms of networking, Xen_Net [99] introduces the high-level virtual network interface component that can migrate partial network processing from the guest domain to the physical NIC (Network Interface Card).

Similar to Xen, the KVM (Kernel-based Virtual Machine) presented by KVM_Sys [78] also caches virtual memory management units in the guests to improve computing efficiency. As for the storage devices and network I/O, programmed I/O and interrupts are utilized to enable and enhance the communication efficiency. As for VMware-based VMM, VM_I/O [119] and VM_Comp [124] explore the concept of paravirtualization further by attempting to minimize intermediate operations caused by the virtual monitor. In addition, MVP [12] explores the possibility of enabling virtualization on mobile phones based on the general desktop virtualization concept.

Besides the aforementioned efforts in the scope of virtualization technologies, a series of evaluations are designed and implemented in the public Cloud with several execution scenarios, and the real performances are observed. Most notably in EC2_Dis [13], the performance of public Cloud instance EC2 from Amazon is evaluated based on large-scale, tightly coupled, distributed tasks.

EC2_HPC [45] is another study in EC2 toward HPC applications, in which the public Cloud instance is tested with standard HPC benchmarks in terms of memory bandwidth, computation, and I/O with MPI implementations. The matter in this particular case consists of the Cloud virtual instances being managed by the Cloud platform but not the users, which means that it is still impossible for users to define the network topology for their instances. In addition, for every reboot, the location of the virtual instances may change due to the migration policy in the Cloud. As a result, the MPI implementations that require instances in the same subnet perform poorly.

In the same context as evaluating the performance of EC2, EC2_NASA [97] has realized a performance analysis of Amazon EC2 by comparing it to Pleiades, NASA's supercomputer. The results show that in the single-node mode where shared memories are used, the performance of EC2 is equivalent to Pleiades due to the utilization of the same node architecture. However, the EC2 instances increase the delay when scaling up due to the underlying network architecture. This performance behavior has been observed in a study presented in [72], which described similar results and pointed out that the interconnection issue is one key bottleneck in the Cloud to support HPC-based applications.

Empirical performance analysis has been conducted and presented in [71], which evaluated the efficiency of public Cloud EC2 instances from the perspective of Many-Task Computing (MTC). Based on the experiment scenarios, the work suggests that an order of magnitude improvement is required in the Cloud to achieve similar performance as Grid.

Table 5 briefly summarizes the works and studies above. From the table, we can conclude that the flexible Cloud solutions pose as a potential and promising feature to support lightweight multitenant MCC mobile applications. However, the virtualization technologies inevitably introduce nonnegligible performance loss regarding computing and communicating, and the Cloud nature is not suitable for the high-performance computing-intensive or communication-intensive environment. As a result, scheduling, load balance, and communication protocols must be tailored appropriately under the current hard-layer infrastructure for mitigating performance losses and leveraging offloading capabilities.

Table 5. Cloud-Based Hardware-Layer Studies
Work VMM Resource Target Evaluation
Xen_Comp [11] Xen Static Comp, I/O SPEC, lmbench
Xen_Net [99] Xen Static Net Zero-copy netperf
KVM_Sys [78] KVM Static Comp, I/O, Net N/A
VM_I/O [119] VMware Static I/O, Net nettest
VM_Comp [124] VMware Static Memory SPEC, database
MVP [12] VMware Mobile Comp kstats
EC2_Dis [13] Xen Static Comp, Net cavity3D
EC2_HPC [45] Xen Static Net HPC standard
EC2_NASA [97] Xen Static Comp HPC standard
EC2_MTC [97] Xen Static Comp, Net MTC standard

4.2 Virtual Instance Management

While the hard layer paves the foundations for the Cloud architecture, the intermediate virtual resource management layer directly tackles the resource provisioning issue in the MCC-based offloading. As the bridge that connects the physical resources and application tasks, this layer encapsulates underlying implementation details and hosts application tasks in the virtual machines. The core functioning of this layer is to instantiate and maintain the virtual instances in proper physical hosts to process offloading tasks in a way that the energy consumption of the Cloud is minimized and the requirements of incoming tasks are acceptably satisfied. To achieve this, typically the studies are oriented by VM live migration and resizing, which enable the VMs to dynamically modify the computing capability and location, as shown in Table 6.

Table 6. Cloud-Based VM-Layer Studies
Work Arch Serv VM Energy Scheduling Prediction Migration
PaPCR [77] Yes Yes Yes DVFS Min-Price RT-VM Lookahead N/A
PEGCA [43] Yes N/A N/A State Predict-based Neural Network N/A
EaACB [46] N/A N/A N/A State ACO History based N/A
MoACS [57] N/A N/A N/A DVFS ACO History based N/A
EaFVM [42] Yes N/A Yes VM-based Branching Heuristic N/A N/A
BbGbS [131] Yes Yes Yes N/A Predict based History based Yes
LBVMR [68] Yes Yes Yes N/A GA N/A Yes

The VM-level energy management originates from VM-based migration—to achieve high energy efficiency by gathering VMs and releasing low load physicals. In BbGbS [131], a migration-based framework is designed and evaluated, targeting the Xen hypervisor. The Sandpiper architecture can automatically trigger load-balancing behaviors in a self-organized manner without access to VMs. The data collection model is implemented to collect guests’ data outside the virtual guest regarding CPU, network, and RAM. Domain-0 VM and hypervisor in Xen are utilized to obtain the CPU task, network event, and RAM swap list of each virtual client in the physical host. The detection algorithm is based on historical data and the time-series prediction algorithm.

As mentioned in LBVMR [68], the overhead introduced by migration is inevitable in the migration-based load-balancing algorithms. In their research, a genetic-based algorithm is proposed. By simulating and evaluating resource scheduling solutions of a certain number of generations, an acceptable scheduling strategy can be found concerning load balancing and the number of migrations. The presentation of the system employs the concept of spanning trees, which involves management roots, physical machines, and virtual machines. An individual tree represents a scheduling solution, which performs mutation and crossover to produce offspring.

Due to the significant computational and communication overhead introduced by the live migration, several works sought to handle the scheduling issue without adjusting the VM position frequently. This approach has been explored by EaRAH [1416], which abstracted the virtual resource scheduling process as the bin packing problem where incoming VMs are presented as objects, while physical hosts are bins. In the scheduling process, VMs are first instantiated in a set of physical hosts. Then, VMs are migrated if there exist hosts that violate utilization boundaries.

Concerning the improvement of the scheduling efficiency, the scheduling components involve prediction functions to tailor the VM-host mapping, such as the work described in PaPCR [77]. It also proposes a DVFS-based scheduling solution, focusing on the processor speed adjustment. Instead of utilizing live migration to alleviate the unbalanced load, this work investigates the use of a lookahead prediction based on the current VM proportional share so that the processor can modify its frequency to achieve higher energy efficiency while meeting the task requirements.

Similarly, PEGCA [43] deploys a power-down prediction-based resource scheduling algorithm to solve the unknown load issue. In the described solution, neural network technology is utilized to learn the load variation behavior and predict further load change based on the current VM load. Based on the prediction, the Cloud management node can either reduce the number of VMs and suspend physical hosts if further system load decreases or invoke additional physical servers and increase live VMs for the increasing of loads.

Besides the predictions, heuristics have also been used to tackle the performance of scheduling in the Cloud. EaACB [46] and MoACS [57] introduced the ant colony optimization (ACO) protocol to achieve self-manageable energy-aware scheduling. Unlike EaRAH [14], which first generates random scheduling solutions and then reallocates the unbalanced load to achieve energy or execution efficiency optimization, EaACB [46] seeks to find the global optimization directly in the scheduling stage. In the proposed bin-packing ACO, the pheromone is defined as the minimum energy consumption scheduling solution generated by ants. In order to find the global optimization, a probabilistic rule is defined to enable ants to bypass local optimal value stochastically while the pheromone trail updates gradually converge all ants to the final solution.

Similar to this ACO-based scheduling technique, MoACS [57] also deploys the ACO algorithm to solve the multidimension bin-packing issue. Unlike the previous work that only concerns energy consumption according to the on/off states, this study further evaluates the pheromone on the CPU utilization, which leads to a more robust result and a faster convergence rate.

In addition to the above nature-inspired algorithms, the techniques described in both EaFVM [42] and EVRP [62] deploy Constraint Programming (CP) to search for the best scheduling solution. In the proposed design, task requirements and current system load are translated into QoS and availability constraints, while a VM-based power objective is established for a heuristic search. In the execution, the branching heuristic is implemented to improve scheduling efficiency.

4.3 Task and Process Scheduling

The task- and process-related energy-aware issues are inherited from traditional distributed systems, such as clusters and Grid [51]. The energy-aware handling of such conventional systems is typically conducted through balancing and remapping tasks so that computing resources or networking resources can be fully or partially powered down for energy reservation. This layer is required to balance the tradeoffs between the Cloud energy cost and the task execution quality as shown in Table 7.

Table 7. Cloud-Based Task-Layer Studies
Work Resource Scope Exec Eff Energy Eff Task Model Network Model Comp Model Scheduling Prediction Scalability
EMMC [84] Cluster N/A Yes Meta N/A State CS, AIS N/A N/A
EaGAT [30] Data Center Yes Yes Meta N/A DVFS GA N/A N/A
GATSC [58] Cluster Yes N/A Meta N/A MapReduce GA KCCA N/A
JSMCC [91] Inter-DC Yes N/A Meta N/A State GA N/A Yes
PBoHM [100] Data Center Yes Yes DAG N/A DVFS GA N/A N/A
MOAEA [133] Data Center Yes Yes DAG Delay, Size DVFS PSO N/A N/A
HBDSA [102] Data Center Yes N/A Web App N/A Service Honey Bee N/A N/A
DENS [80] Multi-DC Yes Yes Video App 3-Tier Net DPM Weighted Tier Buffer Yes
EeDRC [21] Data Center N/A Yes Database Traffic based DVFS Aggregate Net Traffic N/A
QGMMH [67] Grid Yes N/A Meta Bandwidth FLOP Min-Min GHS N/A
UPGMM [31] Data Center Yes N/A Meta N/A FLOPS Min-Min N/A N/A
LBTCC [125] Data Center Yes N/A Meta 3-Tier Net FLOPS OLB, Min-Min N/A N/A
EMmTS [17] Data Center Yes N/A Meta Bandwidth FLOPS Max-Min N/A N/A
IMMTS [88] Cluster Yes N/A Meta N/A FLOPS Max-Min Load N/A
DECM [56] Cloudlet Yes Yes Meta Route based Meta DP N/A Yes

EMMC [84] consists of an energy-aware cluster-based framework that describes two power-down-based task scheduling frameworks—Covering Set and All-In Strategy—presenting a performance comparison and discussion. By utilizing the power-down strategies, a promising amount of energy consumption can be reduced. In this same context, a task replication reconfiguration method [86] has been proposed to develop an energy-aware cluster. By limiting the size of covering subsets and the replicas, energy can be reserved with a certain level of availability.

Inspired by bio-inspired optimization heuristics, many works have employed approaches based on genetic algorithms to handle the energy-aware task scheduling issue. In that sense, GATSC [58], JSMCC [91], EaGAT [30], and PBoHM [100], as well as variations of these strategies described in [29, 73], have made efforts to handle the task scheduling issue in the Cloud based on the genetic algorithm (GA). In these works, the scheduling problem is formalized as a self-managed mapping process that applies DVFS based on resource models to application load models. The use of DVFS is guided by the energy-aware evaluation matrix, which typically targets the task execution time and the system energy cost as the multiobjective context.

MOAEA [133] used another bio-inspired optimization class of algorithms, which utilizes PSO (Particle Swarm Optimization)-based population heuristic optimization to pair the tasks, processors with different voltage levels. In the proposed solution, the task mapping issue is defined as finding the optimal task-process-voltage triplet sets that contain the best fitness value. In the evolution process, the fitness value is evaluated as a combination of the task makespan, cost, and energy, while the descendants are generated by comparing the distance to the current optimal. To achieve the global optimal, offspring can randomly select another particle position with a certain probability.

HBDSA [102] also introduced a load-balancing mechanism that is based on the honeybee optimization algorithm. Idle computing resources are divided into foragers and scouts. Scouts serve a random task in the task queue and record the task information after completing globally (on the advertising board) with a probability. The foragers randomly read the globally published task information based on adjusted probability and then choose and harvest the task. The probability is calculated based on the profit of the server and the whole system recorded by the global advertising board.

Besides the biological-based algorithms, several works attempted to shorten the makespan and increase the utilization ratio of the resources by using Min-Min optimization approaches. QGMMH [67] employed the concept of Min-Min to support QoS-aware Grid Computing. It calculates the Minimum Completion Time (MCT) of each task and prioritizes small tasks.

In the same trend, UPGMM [31] proposed an enhanced Min-Min algorithm with user-priority awareness and dynamic load balancing, which tackles the multitenant issue on the Cloud environment. In addition to the existing Min-Min functions, the proposed algorithm introduces an extra load comparison step, selecting small tasks from resources that are most overloaded. The selected tasks are then reassigned to low load resources to execute if the makespan allows. Although the proposed algorithm produces overhead due to the extra load comparison step, the experiment result shows an overall performance improvement in terms of task execution time, compared to Min-Min.

Compared to Min-Min, large tasks in Max-Min optimization strategies are considered to have higher priority and are scheduled first. In EMmTS [17], an enhanced Max-Min scheduling algorithm is proposed. In this approach, instead of selecting the largest tasks, the authors argue that choosing tasks with average or just above-average load can lead to better makespan and load balance. The experiment result shows an improvement of the proposed algorithm compared to the improved Max-Min algorithm when new large tasks are involved.

Similarly, IMMTS [88] employed a Max-Min optimization strategy to tackle the elasticity issue in the Cloud. In traditional Min-Min and Max-Min studies, the handling of new tasks is mainly under the assumption that the previously scheduled tasks are completed, which is not always true, especially in the multitenant Cloud environment. The proposed method designs and maintains a task table and a virtual resource table to trace the real-time task and resource change. The task status of the virtual machine is collected by the information agent inside the virtual machine. The collected data is then sent to a load balancer, based on which relocation decisions are made.

Unlike the previous studies that focused on computing-related issues, DECM [56], DENS [80], and EeDRC [21] have devised a bandwidth-based network model especially to tackle the communication-based energy-aware problems. DECM [56] deploys the concept of dynamic programming to determine Cloudlet routing and task scheduling. According to the sequence of tasks and the position of Cloudlets, energy-aware recursive functions are designed to pair tasks and resources.

DENS [80] and EeDRC [21] evaluated the energy consumption of ports, line cards, and switches from the view of data center level, rack level, and server level. As suggested, the energy consumption of the ports is highly related to the scale of traffic created by the application tasks. Traffic prediction models have also evaluated both works. DENS [80] evaluates the potential network traffic according to the current uploading queue, while EeDRC [21] defines traffic load prediction based on historical data. Based on the evaluation of task traffic, a tradeoff can be made between communication delay and bandwidth especially for delay-tolerant applications, which can reduce the system-level network energy cost.


As discussed in the sections above, many research efforts have been presented regarding the energy-aware issue in MCC-based offloading. However, due to the novelty of the MCC concept model, enabling energy-efficient offloading remains a challenge. The main obstacles and potential research opportunities are summarized as follows:

  1. Mobile oriented
    1. Availability. The availability issue in energy-aware MCC-based offloading is originated by the distinctness of computing architectures between mobile and static devices. However, the mobile devices cannot transparently translate the mobile-based task sets into runnable executions directly for the static offloading resources. This lack of interpreter standard in MCC necessitates the developers either to manually divide tasks into mobile execution tasks and offloading executions or to deploy an intermediate task partitioner that can translate the codes across the platforms. On the one hand, the manual approach requires a considerable amount of work when migrating traditional mobile tasks to the MCC platform. On the other hand, the auto-partitioning method inevitably introduces runtime computing and energy overhead.
    2. Mobility. Unlike the static computers that are connected through wired networks, the mobile elements are moving under the wireless channels. The unstable offloading environment may introduce unpredictable communication overhead, which further impedes the energy-efficient offloading as discussed in Section 3. Many profiling technologies are proposed to alleviate the influence of the fluctuated network that may lead to inappropriate offloading decisions. However, the profiling process itself introduces energy overhead, and the profiling accuracy can still be improved. Also, the decision-making algorithms should concern the coverage of the offloadees and the mobility of the offloaders. Penalty functions can be used to penalize the potential offloading interruptions in the channels.
  2. Cloud oriented
    1. Security. As a distributed paradigm, MCC requires data sharing fundamentally between personal mobile users and the public Cloud resources. However, considering the potential security threats, users should not always share the task-related data, especially when the users first enter a new local area and are surrounded by unknown devices. Instead of stamping and sharing user-based task information directly in the local area, protection mechanisms should be implemented on both the mobile part and the local Cloudlet to filter user-sensitive data and detect malicious behaviors in an energy-efficient manner.
    2. Execution Efficiency. Due to the deployment of RPC in MCC-based offloading, the task execution efficiency highly depends on the status of remote resources. However, from the perspective of Cloud operators, the Cloud system-level energy efficiency often conflicts with the task execution efficiency. According to the patterns of mobile tasks that are typically small and delay sensitive, traditional energy-aware migration-based scheduling protocols of heuristic-based solutions may easily lead to profiling failures due to intolerable communication latency.


In this article, which incorporates [63], the state of the art of works related to energy-aware offloading is extensively surveyed in the scope of MCC-based offloading. The origin of MCC is illustrated based on the nature of Mobile Computing and the comparison between Grid Computing and Cloud Computing. The offloading process, which consists of partitioning of tasks, profiling, decision-making, and offloading components, has been extensively explored in several works, considering the local devices, Cloudlets, and the remote Cloud. The networking aspects, as a determinant factor in performance, have also been investigated. All these elements have been broadly described and discussed in the survey study, guided through the functionality, issues, and solutions of the schemes, architectures, and techniques. This work gathered and examined the existing studies, systematically classified them according to the perspective of the mobile device and the Cloud, and observed performance aspects in terms of the energy-aware offloading processing and the tradeoff between energy reservation and execution efficiency.



This work was supported by PI: Prof. A. Boukerche through the Canada Research Chairs Program, NSERC Strategic Project Program, and NSERC CREATE TRANSIT Program and uOttawa.

Authors’ addresses: A. Boukerche and S. Guan, University of Ottawa, 800 King Edward Avenue, Ottawa, ON, K1N 6N5, Canada; emails:,; R. E. De Grande, Brock University, 1812 Sir Isaac Brock Way, St. Catharines, ON, L2S 3A1, Canada; email:

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from

©2019 Association for Computing Machinery.
0360-0300/2019/02-ART11 $15.00

Publication History: Received June 2017; revised August 2018; accepted October 2018