org.apache.spark.scheduler.cluster.mesos

Class CoarseMesosSchedulerBackend

Object

org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend

org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend

All Implemented Interfaces:

org.apache.mesos.Scheduler, ExecutorAllocationClient, Logging, SchedulerBackend

public class CoarseMesosSchedulerBackend
extends CoarseGrainedSchedulerBackend
implements org.apache.mesos.Scheduler, Logging

A SchedulerBackend that runs tasks on Mesos, but uses "coarse-grained" tasks, where it holds onto each Mesos node for the duration of the Spark job instead of relinquishing cores whenever a task is done. It launches Spark tasks within the coarse-grained Mesos tasks using the CoarseGrainedSchedulerBackend mechanism. This class is useful for lower and more predictable latency.

Unfortunately this has a bit of duplication from MesosSchedulerBackend, but it seems hard to remove this.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend

CoarseGrainedSchedulerBackend.DriverActor

Constructor Summary

Constructors

Constructor and Description
CoarseMesosSchedulerBackend(TaskSchedulerImpl scheduler, SparkContext sc, String master)

Method Summary

Methods

Modifier and Type	Method and Description
String	appId()
String	applicationId() Get an application ID associated with the job.
scala.collection.mutable.HashMap<Object,Object>	coresByTaskId()
org.apache.mesos.Protos.CommandInfo	createCommand(org.apache.mesos.Protos.Offer offer, int numCores)
void	disconnected(org.apache.mesos.SchedulerDriver d)
org.apache.mesos.SchedulerDriver	driver()
void	error(org.apache.mesos.SchedulerDriver d, String message)
void	executorLost(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.ExecutorID e, org.apache.mesos.Protos.SlaveID s, int status)
int	extraCoresPerSlave()
scala.collection.mutable.HashMap<String,Object>	failuresBySlaveId()
void	frameworkMessage(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.ExecutorID e, org.apache.mesos.Protos.SlaveID s, byte[] b)
boolean	isRegistered()
int	MAX_SLAVE_FAILURES()
int	maxCores()
int	newMesosTaskId()
int	nextMesosTaskId()
void	offerRescinded(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.OfferID o)
void	registered(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.FrameworkID frameworkId, org.apache.mesos.Protos.MasterInfo masterInfo)
Object	registeredLock()
void	reregistered(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.MasterInfo masterInfo)
void	resourceOffers(org.apache.mesos.SchedulerDriver d, java.util.List<org.apache.mesos.Protos.Offer> offers) Method called by Mesos to offer resources on slaves.
scala.collection.mutable.HashSet<String>	slaveIdsWithExecutors()
void	slaveLost(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.SlaveID slaveId)
void	start()
void	statusUpdate(org.apache.mesos.SchedulerDriver d, org.apache.mesos.Protos.TaskStatus status)
void	stop()
scala.collection.mutable.HashMap<Object,String>	taskIdToSlaveId()
int	totalCoresAcquired()
void	waitForRegister()

Methods inherited from class org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend

ACTOR_NAME, actorSystem, conf, createTime, defaultParallelism, driverActor, isReady, killExecutors, killTask, maxRegisteredWaitingTime, minRegisteredRatio, numExistingExecutors, removeExecutor, requestExecutors, requestTotalExecutors, reviveOffers, stopExecutors, sufficientResourcesRegistered, taskIdsOnSlave, totalCoreCount, totalRegisteredExecutors

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Methods inherited from interface org.apache.spark.ExecutorAllocationClient

killExecutor

Constructor Detail

CoarseMesosSchedulerBackend

public CoarseMesosSchedulerBackend(TaskSchedulerImpl scheduler,
                           SparkContext sc,
                           String master)

Method Detail

MAX_SLAVE_FAILURES

public int MAX_SLAVE_FAILURES()

isRegistered

public boolean isRegistered()

registeredLock

public Object registeredLock()

driver

public org.apache.mesos.SchedulerDriver driver()

maxCores

public int maxCores()

coresByTaskId

public scala.collection.mutable.HashMap<Object,Object> coresByTaskId()

totalCoresAcquired

public int totalCoresAcquired()

slaveIdsWithExecutors

public scala.collection.mutable.HashSet<String> slaveIdsWithExecutors()

taskIdToSlaveId

public scala.collection.mutable.HashMap<Object,String> taskIdToSlaveId()

failuresBySlaveId

public scala.collection.mutable.HashMap<String,Object> failuresBySlaveId()

extraCoresPerSlave

public int extraCoresPerSlave()

nextMesosTaskId

public int nextMesosTaskId()

appId

public String appId()

Specified by:

appId in interface SchedulerBackend

newMesosTaskId

public int newMesosTaskId()

start

public void start()

Specified by:

start in interface SchedulerBackend

Overrides:

start in class CoarseGrainedSchedulerBackend

createCommand

public org.apache.mesos.Protos.CommandInfo createCommand(org.apache.mesos.Protos.Offer offer,
                                                int numCores)

offerRescinded

public void offerRescinded(org.apache.mesos.SchedulerDriver d,
                  org.apache.mesos.Protos.OfferID o)

Specified by:

offerRescinded in interface org.apache.mesos.Scheduler

registered

public void registered(org.apache.mesos.SchedulerDriver d,
              org.apache.mesos.Protos.FrameworkID frameworkId,
              org.apache.mesos.Protos.MasterInfo masterInfo)

Specified by:

registered in interface org.apache.mesos.Scheduler

waitForRegister

public void waitForRegister()

disconnected

public void disconnected(org.apache.mesos.SchedulerDriver d)

Specified by:

disconnected in interface org.apache.mesos.Scheduler

reregistered

public void reregistered(org.apache.mesos.SchedulerDriver d,
                org.apache.mesos.Protos.MasterInfo masterInfo)

Specified by:

reregistered in interface org.apache.mesos.Scheduler

resourceOffers

public void resourceOffers(org.apache.mesos.SchedulerDriver d,
                  java.util.List<org.apache.mesos.Protos.Offer> offers)

Method called by Mesos to offer resources on slaves. We respond by launching an executor, unless we've already launched more than we wanted to.

Specified by:

resourceOffers in interface org.apache.mesos.Scheduler

statusUpdate

public void statusUpdate(org.apache.mesos.SchedulerDriver d,
                org.apache.mesos.Protos.TaskStatus status)

Specified by:

statusUpdate in interface org.apache.mesos.Scheduler

error

public void error(org.apache.mesos.SchedulerDriver d,
         String message)

Specified by:

error in interface org.apache.mesos.Scheduler

stop

public void stop()

Specified by:

stop in interface SchedulerBackend

Overrides:

stop in class CoarseGrainedSchedulerBackend

frameworkMessage

public void frameworkMessage(org.apache.mesos.SchedulerDriver d,
                    org.apache.mesos.Protos.ExecutorID e,
                    org.apache.mesos.Protos.SlaveID s,
                    byte[] b)

Specified by:

frameworkMessage in interface org.apache.mesos.Scheduler

slaveLost

public void slaveLost(org.apache.mesos.SchedulerDriver d,
             org.apache.mesos.Protos.SlaveID slaveId)

Specified by:

slaveLost in interface org.apache.mesos.Scheduler

executorLost

public void executorLost(org.apache.mesos.SchedulerDriver d,
                org.apache.mesos.Protos.ExecutorID e,
                org.apache.mesos.Protos.SlaveID s,
                int status)

Specified by:

executorLost in interface org.apache.mesos.Scheduler

applicationId

public String applicationId()

Description copied from interface: SchedulerBackend

Get an application ID associated with the job.

Specified by:

applicationId in interface SchedulerBackend

Returns:

An application ID