public class BlobServer extends Thread implements BlobService, BlobWriter, PermanentBlobService, TransientBlobService
Thread.State, Thread.UncaughtExceptionHandler
MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY
Constructor and Description |
---|
BlobServer(Configuration config,
BlobStore blobStore)
Instantiates a new BLOB server and binds it to a free network port.
|
Modifier and Type | Method and Description |
---|---|
boolean |
cleanupJob(JobID jobId,
boolean cleanupBlobStoreFiles)
Removes all BLOBs from local and HA store belonging to the given job ID.
|
void |
close()
Shuts down the BLOB server.
|
protected BlobClient |
createClient() |
boolean |
deleteFromCache(JobID jobId,
TransientBlobKey key)
Deletes the file associated with the blob key in the local storage of the blob server.
|
boolean |
deleteFromCache(TransientBlobKey key)
Deletes the (job-unrelated) file associated with the blob key in the local storage of the
blob server.
|
File |
getFile(JobID jobId,
PermanentBlobKey key)
Returns the path to a local copy of the file associated with the provided job ID and blob
key.
|
File |
getFile(JobID jobId,
TransientBlobKey key)
Retrieves the local path of a file associated with a job and a blob key.
|
File |
getFile(TransientBlobKey key)
Retrieves the local path of a (job-unrelated) file associated with a job and a blob key.
|
int |
getMinOffloadingSize()
Returns the configuration used by the BLOB server.
|
PermanentBlobService |
getPermanentBlobService()
Returns a BLOB service for accessing permanent BLOBs.
|
int |
getPort()
Returns the port on which the server is listening.
|
File |
getStorageDir() |
File |
getStorageLocation(JobID jobId,
BlobKey key)
Returns a file handle to the file associated with the given blob key on the blob server.
|
TransientBlobService |
getTransientBlobService()
Returns a BLOB service for accessing transient BLOBs.
|
boolean |
isShutdown()
Tests whether the BLOB server has been requested to shut down.
|
PermanentBlobKey |
putPermanent(JobID jobId,
byte[] value)
Uploads the data of the given byte array for the given job to the BLOB server and makes it a
permanent BLOB.
|
PermanentBlobKey |
putPermanent(JobID jobId,
InputStream inputStream)
Uploads the data from the given input stream for the given job to the BLOB server and makes
it a permanent BLOB.
|
TransientBlobKey |
putTransient(byte[] value)
Uploads the (job-unrelated) data of the given byte array to the BLOB server.
|
TransientBlobKey |
putTransient(InputStream inputStream)
Uploads the (job-unrelated) data from the given input stream to the BLOB server.
|
TransientBlobKey |
putTransient(JobID jobId,
byte[] value)
Uploads the data of the given byte array for the given job to the BLOB server.
|
TransientBlobKey |
putTransient(JobID jobId,
InputStream inputStream)
Uploads the data from the given input stream for the given job to the BLOB server.
|
void |
run() |
activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield
equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
serializeAndTryOffload
public BlobServer(Configuration config, BlobStore blobStore) throws IOException
config
- Configuration to be used to instantiate the BlobServerblobStore
- BlobStore to store blobs persistentlyIOException
- thrown if the BLOB server cannot bind to a free network port or if the
(local or distributed) file storage cannot be created or is not usablepublic File getStorageDir()
@VisibleForTesting public File getStorageLocation(@Nullable JobID jobId, BlobKey key) throws IOException
This is only called from BlobServerConnection
or unit tests.
jobId
- ID of the job this blob belongs to (or null if job-unrelated)key
- identifying the fileIOException
- if creating the directory failspublic void close() throws IOException
close
in interface Closeable
close
in interface AutoCloseable
IOException
protected BlobClient createClient() throws IOException
IOException
public File getFile(TransientBlobKey key) throws IOException
The blob server looks the blob key up in its local storage. If the file exists, it is
returned. If the file does not exist, it is retrieved from the HA blob store (if available)
or a FileNotFoundException
is thrown.
getFile
in interface TransientBlobService
key
- blob key associated with the requested fileIOException
- Thrown if the file retrieval failed.public File getFile(JobID jobId, TransientBlobKey key) throws IOException
The blob server looks the blob key up in its local storage. If the file exists, it is
returned. If the file does not exist, it is retrieved from the HA blob store (if available)
or a FileNotFoundException
is thrown.
getFile
in interface TransientBlobService
jobId
- ID of the job this blob belongs tokey
- blob key associated with the requested fileIOException
- Thrown if the file retrieval failed.public File getFile(JobID jobId, PermanentBlobKey key) throws IOException
We will first attempt to serve the BLOB from the local storage. If the BLOB is not in there, we will try to download it from the HA store.
getFile
in interface PermanentBlobService
jobId
- ID of the job this blob belongs tokey
- blob key associated with the requested fileFileNotFoundException
- if the BLOB does not exist;IOException
- if any other error occurs when retrieving the filepublic TransientBlobKey putTransient(byte[] value) throws IOException
TransientBlobService
putTransient
in interface TransientBlobService
value
- the buffer to uploadIOException
- thrown if an I/O error occurs while uploading the data to the BLOB serverpublic TransientBlobKey putTransient(JobID jobId, byte[] value) throws IOException
TransientBlobService
putTransient
in interface TransientBlobService
jobId
- the ID of the job the BLOB belongs tovalue
- the buffer to uploadIOException
- thrown if an I/O error occurs while uploading the data to the BLOB serverpublic TransientBlobKey putTransient(InputStream inputStream) throws IOException
TransientBlobService
putTransient
in interface TransientBlobService
inputStream
- the input stream to read the data fromIOException
- thrown if an I/O error occurs while reading the data from the input
stream or uploading the data to the BLOB serverpublic TransientBlobKey putTransient(JobID jobId, InputStream inputStream) throws IOException
TransientBlobService
putTransient
in interface TransientBlobService
jobId
- ID of the job this blob belongs toinputStream
- the input stream to read the data fromIOException
- thrown if an I/O error occurs while reading the data from the input
stream or uploading the data to the BLOB serverpublic PermanentBlobKey putPermanent(JobID jobId, byte[] value) throws IOException
BlobWriter
putPermanent
in interface BlobWriter
jobId
- the ID of the job the BLOB belongs tovalue
- the buffer to uploadIOException
- thrown if an I/O error occurs while writing it to a local file, or
uploading it to the HA storepublic PermanentBlobKey putPermanent(JobID jobId, InputStream inputStream) throws IOException
BlobWriter
putPermanent
in interface BlobWriter
jobId
- ID of the job this blob belongs toinputStream
- the input stream to read the data fromIOException
- thrown if an I/O error occurs while reading the data from the input
stream, writing it to a local file, or uploading it to the HA storepublic boolean deleteFromCache(TransientBlobKey key)
deleteFromCache
in interface TransientBlobService
key
- blob key associated with the file to be deletedpublic boolean deleteFromCache(JobID jobId, TransientBlobKey key)
deleteFromCache
in interface TransientBlobService
jobId
- ID of the job this blob belongs tokey
- blob key associated with the file to be deletedpublic boolean cleanupJob(JobID jobId, boolean cleanupBlobStoreFiles)
jobId
- ID of the job this blob belongs tocleanupBlobStoreFiles
- True if the corresponding blob store files shall be cleaned up
as well. Otherwise false.public PermanentBlobService getPermanentBlobService()
BlobService
getPermanentBlobService
in interface BlobService
public TransientBlobService getTransientBlobService()
BlobService
getTransientBlobService
in interface BlobService
public final int getMinOffloadingSize()
getMinOffloadingSize
in interface BlobWriter
public int getPort()
getPort
in interface BlobService
public boolean isShutdown()
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.