Fastbufs

Generally speaking, a fastbuf consists of a buffer and a set of callbacks. All front-end functions operate on the buffer and if the buffer becomes empty or fills up, they ask the corresponding callback to handle the situation. Back-ends then differ just in the definition of the callbacks.

The state of the fastbuf is represented by a struct fastbuf, which is a simple structure describing the state of the buffer (the pointers buffer, bufend), the front-end cursor (bptr), the back-end cursor (bstop), position of the back-end cursor in the file (pos), some flags (flags) and pointers to the callback functions.

The buffer can be in one of the following states:

Rules for back-ends:

This structure contains the state of the fastbuf. See the discussion above for how it works.

Fastbuf flags

Tie a fastbuf to a resource in the current resource pool. Returns the pointer to the same fastbuf.

If you want to use fastbufs to access files, you can choose one of several back-ends and set their parameters.

Back-end types

When you open a file fastbuf, you can use this structure to select a back-end and set its parameters. If you want just an "ordinary" file stream, you can happily pass NULL instead and the defaults from the configuration file (or hard-wired defaults if no config file has been read) will be used.

Configuration section with which you can fill the fb_params

The default fb_params

Opens a file with file mode mode (see the man page of open()). Use params to select the fastbuf back-end and its parameters or pass NULL if you are fine with defaults.

Raises ucw.fb.open if the file does not exist.

Like bopen_file(), but returns NULL on failure.

Opens a temporary file. It is placed with other temp files and it is deleted when closed. Again, use NULL for params if you want the defaults.

Creates a fastbuf from a file descriptor fd and sets its filename to name (the name is used only in error messages). When the fastbuf is closed, the fd is closed as well. You can override this behavior by calling bconfig().

Same as above, but with an auto-generated filename.

Flushes all buffers and makes sure that they are written to the disk.

If you want to use the FB_STD back-end and not worry about setting up any parameters, there is a couple of shortcuts.

Equivalent to bopen_file() with FB_STD back-end.

Equivalent to bopen_file_try() with FB_STD back-end.

Equivalent to bopen_tmp_file() with FB_STD back-end.

Equivalent to bopen_fd() with FB_STD back-end.

Like bfdopen(), but it does not close the fd on bclose().

Usually, bopen_tmp_file() is the best way how to come to a temporary file. However, in some specific cases you can need more, so there is also a set of more general functions.

Generates a temporary filename and stores it to the name_buf (of size at least * TEMP_FILE_NAME_LEN). If open_flags are not NULL, flags that should be OR-ed with other flags to open() will be stored there.

The location and style of temporary files is controlled by the configuration. By default, the system temp directory ($TMPDIR or /tmp) is used.

If the location is a publicly writeable directory (like /tmp), the generated filename cannot be guaranteed to be unique, so open_flags will include O_EXCL and you have to check the result of open() and iterate if needed.

This function is not specific to fastbufs, it can be used separately.

Opens a temporary file and returns its file descriptor. You specify the file mode and open_flags passed to open().

If the name_buf (of at last TEMP_FILE_NAME_LEN chars) is not NULL, the filename is also stored in it.

This function is not specific to fastbufs, it can be used separately.

Sometimes, a file is created as temporary and then moved to a stable location. This function takes a fastbuf created by bopen_tmp_file() or bopen_tmp(), marks it as permanent, closes it and renames it to name.

Please note that it assumes that the temporary file and the name are on the same volume (otherwise, rename() fails), so you might want to configure a special location for the temporary files beforehand.

The fblim back-end reads from a file handle, but at most a given number of bytes. This is frequently used for reading from sockets.

Create a fastbuf which reads at most limit bytes from fd.

The fbmem back-end keeps the whole contents of the stream in memory (as a linked list of memory blocks, so address space fragmentation is avoided).

First, you use fbmem_create() to create the stream and the fastbuf used for writing to it. Then you can call fbmem_clone_read() to get an arbitrary number of fastbuf for reading from the stream.

Create stream and return its writing fastbuf.

Given a writing fastbuf, create a new reading fastbuf.

The fbbuf back-end stores the stream in a given block of memory. This is useful for parsing and generating of complex data structures.

Creates a read-only fastbuf that takes its data from a given buffer. The fastbuf structure is allocated by the caller and pointed to by f. The buffer and size specify the location and size of the buffer.

In some cases, the front-ends can take advantage of rewriting the contents of the buffer temporarily. In this case, set can_overwrite as described in Internals. If you do not care, keep can_overwrite zero.

It is not possible to close this fastbuf. This implies that no tying to resources takes place.

Creates a write-only fastbuf which writes into a provided memory buffer. The fastbuf structure is allocated by the caller and pointed to by f. An attempt to write behind the end of the buffer causes the ucw.fb.write exception.

Data are written directly into the buffer, so it is not necessary to call bflush() at any moment.

It is not possible to close this fastbuf. This implies that no tying to resources takes place.

Calculates, how many bytes were already written into the buffer.

The fbgrow back-end keeps the stream in a contiguous buffer stored in the main memory, but unlike fbmem, the buffer does not have a fixed size and it is expanded to accomodate all data.

At every moment, you can use fastbuf->buffer to gain access to the stream.

Create the growing buffer pre-allocated to basic_size bytes.

Create the growing buffer pre-allocated to basic_size bytes.

Reset stream and prepare for writing.

Prepare for reading (of already written data).

Can be used in any state of b (for example when writing or after fbgrow_rewind()) to return the pointer to internal buffer and its length in bytes. The returned buffer can be invalidated by further requests.

The write-only fbpool back-end also keeps the stream in a contiguous buffer, but this time the buffer is allocated from within a memory pool.

Structure for fastbufs & mempools.

Initialize a new fbpool. The structure is allocated by the caller, so bclose() should not be called and no resource tying takes place.

Start a new continuous block and prepare for writing (see mp_start()). Provide the memory pool you want to use for this block as mp.

Close the block and return the address of its start (see mp_end()). The length can be determined by calling mp_size(mp, ptr).

This fastbuf backend is designed for cases when several threads of a single program append records to a common file and while the record can mix in an arbitrary way, the bytes inside a single record must remain uninterrupted.

In case of files with fixed record size, we just allocate the buffer to hold a whole number of records and take advantage of the atomicity of the write() system call.

With variable-sized records, we need another solution: when writing a record, we keep the fastbuf in a locked state, which prevents buffer flushing (and if the buffer becomes full, we extend it), and we wait for an explicit commit operation which write()s the buffer if the free space in the buffer falls below the expected maximum record length.

Please note that initialization of the clones is not thread-safe, so you have to serialize it yourself.

Open an atomic fastbuf. If master is NULL, the file name is opened. If it is non-null, a new clone of an existing atomic fastbuf is created.

If the file has fixed record length, just set record_len to it. Otherwise set record_len to the expected maximum record length with a negative sign (you need not fit in this length, but as long as you do, the fastbuf is more efficient) and call fbatomic_commit() after each record.

You can specify record_len, if it is known (for optimisations).

The file is closed when all fastbufs using it are closed.

Declare that you have finished writing a record. This is required only if a fixed record size was not specified.

Creates a new "/dev/null"-like fastbuf. Any read attempt returns an EOF, any write attempt is silently ignored.

Can be used by any back-end to switch it to the null mode. You need to provide at least one byte long buffer for writing.

Checks whether a fastbuf has been switched to the null mode.

Imagine some code which does massive string processing. It takes an input buffer, writes a part of it into an output buffer, then some other string and then the remaining part of the input buffer. Or anything else where you copy all the data at each stage of the complicated process.

This backend takes multiple fastbufs and concatenates them formally into one. You may then read them consecutively as they were one fastbuf at all.

This backend is read-only.

This backend is seekable iff all of the supplied fastbufs are seekable.

You aren’t allowed to do anything with the underlying buffers while these are connected into fbmulti.

The fbmulti is inited by fbmulti_create(). It returns an empty fbmulti. Then you call fbmulti_append() for each fbmulti.

If bclose() is called on fbmulti, all the underlying buffers get closed recursively.

If you want to keep an underlying fastbuf open after bclose, just remove it by fbmulti_remove where the second parameter is a pointer to the removed fastbuf. If you pass NULL, all the underlying fastbufs are removed.

After fbmulti_remove, the state of the fbmulti is undefined. The only allowed operation is either another fbmulti_remove or bclose on the fbmulti.

Create an empty fbmulti

Append a fb to fbmulti

Remove a fb from fbmulti

Parameters that could be configured.

Configure a fastbuf. Returns previous value.

Close and free fastbuf. Can not be used for fastbufs not returned from function (initialized in a parameter, for example the one from fbbuf_init_read).

Throw exception on a given fastbuf

Write data (if it makes any sense, do not use for in-memory buffers).

Seek in the buffer. See man fseek for description of whence. Only for seekable fastbufs.

Set position to pos bytes from beginning. Only for seekable fastbufs.

Go to the beginning of the fastbuf. Only for seekable ones.

How large is the file? -1 if not seekable.

Where am I (from the beginning)?

Return next character from the buffer.

Return next character from the buffer, but keep the current position.

Have I reached EOF?

Return last read character back. Only one back is guaranteed to work.

Write a single character.

Return the length of the cached data to be read. Do not use directly.

Return the length of the buffer available for writing. Do not use directly.

Read at most l bytes of data into b. Returns number of bytes read. 0 means end of file.

Reads exactly l bytes of data into b. If at the end of file, it returns 0. If there are data, but less than l, it raises ucw.fb.eof.

Writes buffer b of length l into fastbuf.

Reads a line into b and strips trailing \n. Returns pointer to the terminating 0 or NULL on EOF. Raises ucw.fb.toolong if the line is longer than l.

The same as bgets(), but for 0-terminated strings.

Returns either length of read string (excluding the terminator) or -1 if it is too long. In such cases exactly l bytes are read.

Read a string, strip the trailing \n and store it into growing buffer b. Raises ucw.fb.toolong if the line is longer than limit.

Read a string, strip the trailing \n and store it into buffer allocated from a memory pool.

Read a string, strip the trailing \n and store it on the stack (allocated using alloca()).

Write a string, without 0 or \n at the end.

Write string, including terminating 0.

Write string and append a newline to the end.

Copy l bytes of data from fastbuf f to fastbuf t. UINT_MAX (~0U) means all data, even if more than UINT_MAX bytes remain.

Skip len bytes without reading them.

Begin direct reading from fastbuf’s internal buffer to avoid unnecessary copying. The function returns a buffer buf together with its length in bytes (zero means EOF) with cached data to be read.

Some back-ends allow the user to modify the data in the returned buffer to avoid unnecessary. If the back-end allows such modifications, it can set f->can_overwrite_buffer accordingly:

The reading must be ended by bdirect_read_commit() or bdirect_read_commit_modified(), unless the user did not read or modify anything.

End direct reading started by bdirect_read_prepare() and move the cursor at pos. Data in the returned buffer must be same as after bdirect_read_prepare() and pos must point somewhere inside the buffer.

Similar to bdirect_read_commit(), but accepts also modified data before pos. Note that such modifications are supported only if f->can_overwrite_buffer == 2.

Start direct writing to fastbuf’s internal buffer to avoid copy overhead. The function returns the length of the buffer in buf (at least one byte) where we can write to. The operation must be ended by bdirect_write_commit(), unless nothing is written.

Commit the data written to the buffer returned by bdirect_write_prepare(). The length is specified by pos which must point just after the written data. Also moves the cursor to pos.

printf into a fastbuf.

vprintf into a fastbuf.