public class PermanentBlobCache extends AbstractBlobCache implements PermanentBlobService
When requesting BLOBs via getFile(JobID, PermanentBlobKey)
, the cache will first
attempt to serve the file from its local cache. Only if the local cache does not contain the
desired BLOB, it will try to download it from a distributed HA file system (if available) or the
BLOB server.
If files for a job are not needed any more, they will enter a staged, i.e. deferred, cleanup. Files may thus still be be accessible upon recovery and do not need to be re-downloaded.
blobClientConfig, blobView, log, numFetchRetries, readWriteLock, serverAddress, shutdownHook, shutdownRequested, storageDir, tempFileCounter
Constructor and Description |
---|
PermanentBlobCache(InetSocketAddress serverAddress,
Configuration blobClientConfig,
BlobView blobView)
Instantiates a new cache for permanent BLOBs which are also available in an HA store.
|
Modifier and Type | Method and Description |
---|---|
protected void |
cancelCleanupTask()
Cancels any cleanup task that subclasses may be executing.
|
File |
getFile(JobID jobId,
PermanentBlobKey key)
Returns the path to a local copy of the file associated with the provided job ID and blob
key.
|
int |
getNumberOfCachedJobs() |
int |
getNumberOfReferenceHolders(JobID jobId) |
File |
getStorageLocation(JobID jobId,
org.apache.flink.runtime.blob.BlobKey key)
Returns a file handle to the file associated with the given blob key on the blob
server.
|
void |
registerJob(JobID jobId)
Registers use of job-related BLOBs.
|
void |
releaseJob(JobID jobId)
Unregisters use of job-related BLOBs and allow them to be released.
|
close, getFileInternal, getPort
public PermanentBlobCache(InetSocketAddress serverAddress, Configuration blobClientConfig, BlobView blobView) throws IOException
serverAddress
- address of the BlobServer
to use for fetching files fromblobClientConfig
- global configurationblobView
- (distributed) HA blob store file system to retrieve files from firstIOException
- thrown if the (local or distributed) file storage cannot be created or is not usablepublic void registerJob(JobID jobId)
Using any other method to access BLOBs, e.g. getFile(org.apache.flink.api.common.JobID, org.apache.flink.runtime.blob.PermanentBlobKey)
, is only valid within
calls to registerJob(JobID) and releaseJob(JobID)
.
jobId
- ID of the job this blob belongs toreleaseJob(JobID)
public void releaseJob(JobID jobId)
jobId
- ID of the job this blob belongs toregisterJob(JobID)
public int getNumberOfReferenceHolders(JobID jobId)
public int getNumberOfCachedJobs()
public File getFile(JobID jobId, PermanentBlobKey key) throws IOException
We will first attempt to serve the BLOB from the local storage. If the BLOB is not in
there, we will try to download it from the HA store, or directly from the BlobServer
.
getFile
in interface PermanentBlobService
jobId
- ID of the job this blob belongs tokey
- blob key associated with the requested fileFileNotFoundException
- if the BLOB does not exist;IOException
- if any other error occurs when retrieving the file@VisibleForTesting public File getStorageLocation(JobID jobId, org.apache.flink.runtime.blob.BlobKey key) throws IOException
jobId
- ID of the job this blob belongs to (or null if job-unrelated)key
- identifying the fileIOException
- if creating the directory failsprotected void cancelCleanupTask()
AbstractBlobCache
This is called during AbstractBlobCache.close()
.
cancelCleanupTask
in class AbstractBlobCache
Copyright © 2014–2018 The Apache Software Foundation. All rights reserved.