5g
Factories of the Future
Media & Entertainment
Smart Cities
Smart Energy
Smart Ports
SME Opportunities
Societal Impacts
Technology Development
Telecoms Providers
5G CAM
5G Automotive
5G CAM KPIs
5G CAM Standardisation
5G Corridors
5G Multimodal Connectivity
5G Transport Network
Artificial Intelligence & Machine Learning
Artificial Intelligence & Machine Learning in big data
Artificial Intelligence & Machine Learning technologies
Big data
Big data algorithms
Big data analytics
Collaborative Classification and Models
Business Models, Process Improvement, Contract Management, KPIs and Benchmarking Indexes
Collaboration Risk and Value Sharing
Collaborative Planning and Synchromodality
Customs & Regulatory Compliance
Environmental Performance Management
Logistics Optimisation
Stock Optimisation
Supply Chain Corrective and Preventive Actions (CAPA)
Supply Chain Financing
Supply Chain Visibility
Common Information Objects
Booking
Customs Declarations
Transport Service Description
Transport Status
Waybills
Computing and Processing
Big Data Management and Analytics
Cloud
Edge
Fog
Knowledge Graphs
Machine Learning
MIST
Stream Processing
Connectivity
Architecture
Blockchain
Connectivity Interfaces
Technologies (Bluetooth, Ethernet, Wifi)
Data Management, Simulation and Dashboards
Dashboards
Data Fusion
Data Governance, Integrity, Quality Management and Harmonization
Event Handling
Open Data
Simulation
Statistics and Key Performance Indicators (KPIs)
Data market
Data ecosystem
Data marketplace
Data Platform
Data Providers
Devices
IoT Controllers
IoT Gateways
IoT Sensors
Tracking Sensors
Digitisation Frameworks
Control Towers
Data Pipelines
e-Freight
e-Maritime
National Single Windows
Port Community Systems
Federation
Data Federation
Platform Federation
Industrial IoT Sectors
Rail Sector Active Predictive Maintenance
Interoperability
Data interoperability
Data interoperability mechanisms
Interoperability solutions
Platform interoperability
IoT Secuirty, Privacy and Safety Systems
PKI Technology
Privacy-preservation
Data privacy preserving technologies
Privacy preserving technologies
Project Results
5G-SOLUTIONS Deliverables
5G-SOLUTIONS Publications
CHARIOT Capacity Building and Trainings
CHARIOT Deliverables
CHARIOT Publications
SELIS Deliverables
SELIS Publications and Press Releases
Project Results - 5g Routes
5G-ROUTES Deliverables
5G-ROUTES Innovation
5G-ROUTES Publications
Project Results - TRUSTS
TRUSTS Deliverable
TRUSTS Publications
Safety, Security and Privacy Systems
Access Management
Coordinated Border Management
Information Security
International Organisations
Risk Assessment and Management
Risk Management
Safety and Security Assessment
Source Code Analysis
Sectors and Stakeholders
Airports and Air Transport
Banks, investors and other funding providers
Custom Authorities
Facilities, Warehouses
Freight Forwarders
Inland Waterways
Multimodal Operators
Ports and Terminals
Railway
Retailers
Road Transport
Shippers
Shipping
Smart Buildings
Trusties and other Intermediary Organizations
Urban and Countryside Logistics
Urban Logistics
Sectors and Stakeholders - TRUSTS
Audit & Law firms
Corporate offices
Enterprises
Financial Institutions
Telecommunications
Security
Secured Data
Secured Infrastructure
Secured Platform
Sovereignty
Data sovereignty
Standards
Good Distribution Practices
International data standards
International Organization for Standardization (ISO)
UN/CEFACT
World Customs Organization (WCO)
Supply Chain Management
Business Models, Process Improvement, Contract Management, KPIs and Benchmarking Indexes
Risk Management
Risk-Based Controls
Screening and tracking
Supervision Approach
Technologies
5g
Agile Deployment, Configuration Management
Business Applications
Business Integration Patterns, Publish-Subscribe
Cloud Technologies/Computing, Services Virtualisation
Cognitive
Community Node Platform and Application Monitoring
Connectivity Technologies (Interfaces and Block Chain)
Hybrid S/T Communication and Navigation Platforms
IoT (Sensors, platforms)
Mobile
Physical Internet (PI)
Public key infrastructure (PKI)
Radio-frequency identification (RFID)

SELIS Publications and Press Releases

BBQ: Elastic MapReduce over Cloud Platforms

 


Nikolaos Chalvantzis, Ioannis Konstantinou, Nectarios Koziris 19/09/2019 00:00:00

Cloud infrastructure services such as Amazon EMR allow users to have access to tailor-made Big Data processing clusters within a few clicks from their web browser, thanks to the elastic property of the cloud. In virtual cloud environments, resource management is desired to be performed in a way which optimizes utilization, thus maximizing the value of the resources acquired. As cloud infrastructures become increasingly popular for Big Data analysis, the execution of programs with respect to user selected performance goals, such as job completion deadlines, remains a challenge. In this work we present BARBECUE (joB AwaRe Big-data Elasticity CloUd managEment System), a system that allows a Hadoop MapReduce virtual cluster to automatically adjust its size to the workload it is required to execute in order to respect individual jobs’ completion deadlines without acquiring more resources than the least necessary. To that end, BBQ’s Decision Making module uses a Performance Model for MapReduce jobs which can express cluster resources (i.e., YARN Container capacity) and execution time as a function of the number of nodes in the cluster. BBQ leverages the abstraction of YARN, making it feasible for integration with other execution frameworks, such as Spark, with the necessary changes to its pluggable Decision Making module. We also add a new feature to Hadoop MapReduce which can now dynamically, on-the-fly update the number of selected ReduceTasks in cases where the cluster is expanded, so that our system makes full use of the resources it has acquired during the reduce phase of the execution. BBQ uses an adaptation of the hill climbing algorithm to estimate the optimal combination of number of nodes and reduce waves given a known job, its data input and an execution deadline. The attendees will be able to watch the system perform cluster resizes in real-time in order to execute its assigned jobs in time. 

 

 

Reference Link

Attached Documents

The “CHARIOT IoT Search Index” aims to provide a web location where publications, articles, and relevant documents can be centralized hosted in a well-structured and easily accessed way.

Tags

Contact Us
Enter Text
Contact our department
123movie