class DataPool: public GPEnabled

Thread safe data storage.

Inheritance:


Public Methods

[more]void stop(bool only_blocked=false)
Tells the DataPool to stop serving readers.
[more]void load_file(void)
Loads data from the file into memory.
[more]static void load_file(const char * name)
This function will make every DataPool in the program, which is connected to a file, to load the file contents to the main memory and close the file.
[more]static void close_all(void)
This function will remove OpenFiles filelist.

Public

[more]class Incrementorclass Readerclass Triggerclass OpenFilesclass BlockListclass Initialization

o DataPool()
Default constructor.
o DataPool(ByteStream & str)
Creates and initialized the DataPool with data from stream str.
o DataPool(const GP<DataPool> & master_pool, int start=0, int length=-1)
Initializes the DataPool in slave mode and connects it to the specified offsets range of the specified master DataPool.
o DataPool(const char * file_name, int start=0, int length=-1)
Initializes the DataPool in slave mode and connects it to the specified offsets range of the specified file.
ovoid connect(const GP<DataPool> & master_pool, int start=0, int length=-1)
Switches the DataPool to slave mode and connects it to the specified offsets range of the master DataPool.
ovoid connect(const char * file_name, int start=0, int length=-1)
Connects the DataPool to the specified offsets range of the named file_name.

[more] Adding data.

[more]void add_data(const void * buffer, int size)
Appends the new block of data to the DataPool.
[more]void add_data(const void * buffer, int offset, int size)
Stores the specified block of data at the specified offset.
[more]void set_eof(void)
Tells the DataPool that all data has been added and nothing else is anticipated.

[more] Accessing data.

[more]int get_data(void * buffer, int offset, int size)
Attempts to return a block of data at the given offset of the given size.
[more]GP<ByteStream> get_stream(void)
Returns a ByteStream to access contents of the DataPool sequentially.

[more] State querying functions.

[more]bool is_connected(void) const
Returns TRUE if this DataPool is connected to another DataPool or to a file.
[more]bool has_data(int start, int length)
Returns TRUE if all data available for offsets from start till start+length-1.
[more]int get_length(void) const
Returns the length of data in the DataPool.
[more]int get_size(void) const
Returns the number of bytes of data available in this DataPool.

[more] Trigger callbacks.

[more]void add_trigger(int start, int length, void (* callback)(void *), void * cl_data)
Associates the specified trigger callback with the given data range.
[more]void add_trigger(int thresh, void (* callback)(void *), void * cl_data)
Associates the specified trigger callback with the specified threshold.
[more]void del_trigger(void (* callback)(void *), void * cl_data)
Use this function to unregister callbacks, which are no longer needed.


Inherited from GPEnabled:

Public Methods

oGPEnabled& operator=(const GPEnabled & obj)
oint get_count(void) const

Protected Fields

ovolatile int count


Documentation

Thread safe data storage. The purpose of DataPool is to provide a uniform interface for accessing data from decoding routines running in a multi-threaded environment. Depending on the mode of operation it may contain the actual data, may be connected to another DataPool or may be mapped to a file. Regardless of the mode, the class returns data in a thread-safe way, blocking reading threads if there is no data of interest available. This blocking is especially useful in the networking environment (plugin) when there is a running decoding thread, which wants to start decoding as soon as there is just one byte available blocking if necessary.

Access to data in a DataPool may be direct (Using get_data() function) or sequential (See get_stream() function).

If the DataPool is not connected to anything, that is it contains some real data, this data can be added to it by means of two add_data() functions. One of them adds data sequentially maintaining the offset of the last block of data added by it. The other can store data anywhere. Thus it's important to realize, that there may be "white spots" in the data storage.

There is also a way to test if data is available for some given data range (See has_data()). In addition to this mechanism, there are so-called trigger callbacks, which are called, when there is all data available for a given data range.

Let us consider all modes of operation in details:

  1. Not connected DataPool. In this mode the DataPool contains some real data. As mentioned above, it may be added by means of two functions add_data() operating independent of each other and allowing to add data sequentially and directly to any place of data storage. It's important to call function set_eof() after all data has been added.

    Functions like get_data() or get_stream() can be used to obtain direct or sequential access to the data. As long as is_eof() is FALSE, DataPool will block every reader, which is trying to read unavailable data until it really becomes available. But as soon as is_eof() is TRUE, any attempt to read non-existing data will read 0 bytes.

    Taking into account the fact, that DataPool was designed to store DjVu files, which are in IFF formats, it becomes possible to predict the size of the DataPool as soon as the first 32 bytes have been added. This is invaluable for estimating download progress. See function get_length() for details. If this estimate fails (which means, that stored data is not in IFF format), get_length() returns -1.

    Triggers may be added and removed by means of add_trigger() and del_trigger() functions. add_trigger() takes a data range. As soon as all data in that data range is available, the trigger callback will be called.

    All trigger callbacks will be called when EOF condition has been set.

  2. DataPool connected to another DataPool. In this slave mode you can map a given DataPool to any offsets range inside another DataPool. You can connect the slave DataPool even if there is no data in the master DataPool. Any get_data() request will be forwarded to the master DataPool, and it will be responsible for blocking readers trying to access unavailable data.

    The usage of add_data() functions is prohibited for connected DataPools.

    The offsets range used to map a slave DataPool can be fully specified (both start offset and length are positive numbers) or partially specified (the length is negative). In this mode the slave DataPool is assumed to extend up to the end of the master DataPool.

    Triggers may be used with slave DataPools as well as with the master ones.

    Calling stop() function of a slave will stop only the slave (and any other slave connected to it), but not the master.

    set_eof() function is meaningless for slaves. They obtain the EOF status from their master.

    Depending on the offsets range passed to the constructor, get_length() returns different values. If the length passed to the constructor was positive, then it is returned by get_length() all the time. Otherwise the value returned is either -1 if master's length is still unknown (it didn't manage to parse IFF data yet) or it is calculated as masters_length-slave_start.

  3. DataPool connected to a file. This mode is quite similar to the case, when the DataPool is connected to another DataPool. Similarly, the DataPool stores no data inside. It just forwards all get_data() requests to the underlying source (a file in this case). Thus these requests will never block the reader. But they may return 0 if there is no data available at the requested offset.

    The usage of add_data() functions is meaningless and is prohibited.

    is_eof() function always returns TRUE. Thus set_eof() us meaningless and does nothing.

    get_length() function always returns the file size.

    Calling stop() function will stop this DataPool and any other slave connected to it.

    Trigger callbacks passed through add_trigger() function are called immediately.

    This mode is useful to read and decode DjVu files without reading and storing them in full in memory.

oclass Incrementorclass Readerclass Triggerclass OpenFilesclass BlockListclass Initialization

ovoid stop(bool only_blocked=false)
Tells the DataPool to stop serving readers.

If only_blocked flag is TRUE then only those requests will be processed, which would not block. Any attempt to get non-existing data would result in a STOP exception (instead of blocking until data is available).

If only_blocked flag is FALSE then any further attempt to read from this DataPool (as well as from any DataPool connected to this one) will result in a STOP exception.

o Adding data.
Please note, that these functions are for not connected DataPools only. You can not add data to a DataPool, which is connected to another DataPool or to a file.

ovoid add_data(const void * buffer, int size)
Appends the new block of data to the DataPool. There are two add_data() functions available. One is for adding data sequentially. It keeps track of the last byte position, which has been stored by it and always appends the next block after this position. The other add_data() can store data anywhere.

The function will unblock readers waiting for data if this data arrives with this block. It may also trigger some trigger callbacks, which may have been added by means of add_trigger() function.

Note: After all the data has been added, it's necessary to call set_eof() to tell the DataPool that nothing else is expected.

Note: This function may not be called if the DataPool has been connected to something.

Parameters:
buffer - data to append
size - length of the buffer

ovoid add_data(const void * buffer, int offset, int size)
Stores the specified block of data at the specified offset. Like the function above this one can also unblock readers waiting for data and engage trigger callbacks. The difference is that this function can store data anywhere.

Note: After all the data has been added, it's necessary to call set_eof() to tell the DataPool that nothing else is expected.

Note: This function may not be called if the DataPool has been connected to something.

Parameters:
buffer - data to store
offset - where to store the data
size - length of the buffer

ovoid set_eof(void)
Tells the DataPool that all data has been added and nothing else is anticipated. When EOF is true, any reader attempting to read non existing data will not be blocked. It will either read ZERO bytes or will get an "EOF" exception (see get_data()). Calling this function will also activate all registered trigger callbacks.

Note: This function is meaningless and does nothing when the DataPool is connected to another DataPool or to a file.

o Accessing data.
These functions provide direct and sequential access to the data of the DataPool. If the DataPool is not connected (contains some real data) then it handles the requests itself. Otherwise they are forwarded to the master DataPool or the file.

oint get_data(void * buffer, int offset, int size)
Attempts to return a block of data at the given offset of the given size.

  1. If the DataPool is connected to another DataPool or to a file, the request will just be forwarded to them.
  2. If the DataPool is not connected to anything and some of the data requested is in the internal buffer, the function copies available data to buffer and returns immediately.

    If there is no data available, and is_eof() returns FALSE, the reader (and the thread) will be blocked until the data actually arrives. Please note, that since the reader is blocked, it should run in a separate thread so that other threads have a chance to call add_data(). If there is no data available, but is_eof() is TRUE the behavior is different and depends on the DataPool's estimate of the file size:

    • If DataPool learns from the IFF structure of the data, that its size should be greater than it really is, then any attempt to read non-existing data in the range of valid offsets will result in an "EOF" exception. This is done to indicate, that there was an error in adding data, and the data requested is supposed to be there, but has actually not been added.
    • If DataPool's expectations about the data size coincide with the reality then any attempt to read data beyond the legal range of offsets will result in ZERO bytes returned.
    .
.

Throws:
STOP The stream has been stopped
EOF The requested data is not there and will not be added, although it should have been.
Returns:
The number of bytes actually read
Parameters:
buffer - Buffer to be filled with data
offset - Offset in the DataPool to read data at
size - Size of the buffer

oGP<ByteStream> get_stream(void)
Returns a ByteStream to access contents of the DataPool sequentially. By reading from the returned stream you basically call get_data() function. Thus, everything said for it remains true for the stream too.

o State querying functions.

obool is_connected(void) const
Returns TRUE if this DataPool is connected to another DataPool or to a file.

obool has_data(int start, int length)
Returns TRUE if all data available for offsets from start till start+length-1. If length is negative, the range is assumed to extend up to the end of the DataPool. This function works both for connected and not connected DataPools. Once it returned TRUE for some offsets range, you can be sure that the subsequent get_data() request will not block.

oint get_length(void) const
Returns the length of data in the DataPool. The value returned depends on the mode of operation: .

oint get_size(void) const
Returns the number of bytes of data available in this DataPool. Contrary to the get_length() function, this one doesn't try to interpret the IFF structure and predict the file length. It just returns the number of bytes of data really available inside the DataPool, if it contains data, or inside its range, if it's connected to another DataPool or a file.

o Trigger callbacks.
Trigger callbacks are special callbacks called when all data for the given range of offsets has been made available. Since reading unavailable data may result in a thread block, which may be bad, the usage of trigger callbacks appears to be a convenient way to signal availability of data.

You can add a trigger callback in two ways:

  1. By specifying a range. This is the most general case
  2. By providing just one threshold. In this case the range is assumed to start from offset ZERO and last for threshold+1 bytes.

ovoid add_trigger(int start, int length, void (* callback)(void *), void * cl_data)
Associates the specified trigger callback with the given data range.

Note: The callback may be called immediately if all data for the given range is already available or EOF is TRUE.

Parameters:
start - The beginning of the range for which all data should be available
length - If the length is not negative then the callback will be called when there is data available for every offset from start to start+length-1. If thresh is negative, the callback is called after EOF condition has been set.
callback - Function to call
cl_data - Argument to pass to the callback when it's called.

ovoid add_trigger(int thresh, void (* callback)(void *), void * cl_data)
Associates the specified trigger callback with the specified threshold.

This function is a simplified version of the function above. The callback will be called when there is data available for every offset from 0 to thresh, if thresh is positive, or when EOF condition has been set otherwise.

ovoid del_trigger(void (* callback)(void *), void * cl_data)
Use this function to unregister callbacks, which are no longer needed. Note! It's important to do it when the client is about to be destroyed.

ovoid load_file(void)
Loads data from the file into memory. This function is only useful for DataPools getting data from a file. It descends the DataPools hierarchy until it either reaches a file-connected DataPool or DataPool containing the real data. In the latter case it does nothing, in the first case it makes the DataPool read all data from the file into memory and stop using the file.

This may be useful when you want to overwrite the file and leave existing DataPools with valid data.

ostatic void load_file(const char * name)
This function will make every DataPool in the program, which is connected to a file, to load the file contents to the main memory and close the file. This feature is important when you want to do something with the file like remove or overwrite it not affecting the rest of the program.

ostatic void close_all(void)
This function will remove OpenFiles filelist.


This class has no child classes.

Alphabetic index HTML hierarchy of classes or Java


DjVu is a trademark of LizardTech, Inc.
All other products mentioned are registered trademarks or trademarks of their respective companies.