Berkeley DB Programmer's Reference Guide

Legal Notice

Copyright © 2002 - 2019 Oracle and/or its affiliates. All rights reserved.

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. No third-party use is permitted without the express prior written consent of Oracle.

Other names may be trademarks of their respective owners.

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable:

U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government.

This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.

This software or hardware and documentation may provide access to or information on content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services.

1-Jun-2020


Table of Contents

Preface
Conventions Used in this Book
For More Information
Contact Us
1. Introduction
An introduction to data management
Mapping the terrain: theory and practice
Data access and data management
Relational databases
Object-oriented databases
Network databases
Clients and servers
What is Berkeley DB?
Data Access Services
Data management services
Design
What Berkeley DB is not
Berkeley DB is not a relational database
Berkeley DB is not an object-oriented database
Berkeley DB is not a network database
Berkeley DB is not a database server
Do you need Berkeley DB?
What other services does Berkeley DB provide?
What does the Berkeley DB distribution include?
Where does Berkeley DB run?
The Berkeley DB products
Berkeley DB Data Store
Berkeley DB Concurrent Data Store
Berkeley DB Transactional Data Store
Berkeley DB High Availability
2. Access Method Configuration
What are the available access methods?
Btree
Hash
Heap
Queue
Recno
Selecting an access method
Btree or Heap?
Hash or Btree?
Queue or Recno?
Logical record numbers
General access method configuration
Selecting a page size
Selecting a cache size
Selecting a byte order
Duplicate data items
Non-local memory allocation
Btree access method specific configuration
Btree comparison
Btree prefix comparison
Minimum keys per page
Retrieving Btree records by logical record number
Compression
Hash access method specific configuration
Page fill factor
Specifying a database hash
Hash table size
Heap access method specific configuration
Queue and Recno access method specific configuration
Managing record-based databases
Selecting a Queue extent size
Flat-text backing files
Logically renumbering records
3. Access Method Operations
Database open
Opening multiple databases in a single file
Configuring databases sharing a file
Caching databases sharing a file
Locking in databases based on sharing a file
Partitioning databases
Specifying partition keys
Partitioning callback
Placing partition files
Retrieving records
Storing records
Deleting records
Database statistics
Database truncation
Database upgrade
Database verification and salvage
Flushing the database cache
Database close
Secondary indexes
Error Handling With Secondary Indexes
Foreign key indexes
Cursor operations
Retrieving records with a cursor
Storing records with a cursor
Deleting records with a cursor
Duplicating a cursor
Equality Join
Data item count
Cursor close
4. Access Method Wrapup
Data alignment
Retrieving and updating records in bulk
Bulk retrieval
Bulk updates
Bulk deletes
Partial record storage and retrieval
Storing C/C++ structures/objects
Retrieved key/data permanence for C/C++
Error support
Cursor stability
Database limits
Disk space requirements
Btree
Hash
External File support
The external file threshold
Creating external files
External file access
External file storage
External Files and Replication
Access method tuning
Access method FAQ
5. Java API
Java configuration
Compatibility
Java programming notes
Java FAQ
6. C# API
Compatibility
7. Standard Template Library API
Dbstl introduction
Standards compatible
Performance overhead
Portability
Dbstl typical use cases
Dbstl examples
Berkeley DB configuration
Registering database and environment handles
Truncate requirements
Auto commit support
Database and environment identity checks
Products, constructors and configurations
Using advanced Berkeley DB features with dbstl
Using bulk retrieval iterators
Using the DB_RMW flag
Using secondary index database and secondary containers
Using transactions in dbstl
Using dbstl in multithreaded applications
Working with primitive types
Storing strings
Store and Retrieve data or objects of complex types
Storing varying length objects
Storing arbitrary sequences
Notes
Dbstl persistence
Direct database get
Change persistence
Object life time and persistence
Dbstl container specific notes
db_vector specific notes
Associative container specific notes
Using dbstl efficiently
Using iterators efficiently
Using containers efficiently
Dbstl memory management
Freeing memory
Type specific notes
Dbstl miscellaneous notes
Special notes about trivial methods
Using correct container and iterator public types
Dbstl known issues
8. Berkeley DB Architecture
The big picture
Programming model
Programmatic APIs
C
C++
STL
Java
Dbm/Ndbm, Hsearch
Scripting languages
Perl
PHP
Tcl
Supporting utilities
9. The Berkeley DB Environment
Database environment introduction
Creating a database environment
Sizing a database environment
Opening databases within the environment
Error support
DB_CONFIG configuration file
File naming
Specifying file naming to Berkeley DB
Filename resolution in Berkeley DB
Examples
Shared memory regions
Security
Encryption
Remote filesystems
Environment FAQ
10. Berkeley DB Data Store and Concurrent Data Store Applications
Concurrent Data Store introduction
Handling failure in Data Store and Concurrent Data Store applications
Architecting Data Store and Concurrent Data Store applications
Quick Start Guide
Data Store Environment
Concurrent Data Store Environment
Transactional Data Store Environment
11. Berkeley DB Transactional Data Store Applications
Transactional Data Store introduction
Why transactions?
Terminology
Handling failure in Transactional Data Store applications
Architecting Transactional Data Store applications
Getting Started with Transactional Data Store Applications
Opening the environment
Opening the databases
Recoverability and deadlock handling
Atomicity
Isolation
Degrees of isolation
Snapshot Isolation
Transactional cursors
Nested transactions
Environment infrastructure
Deadlock detection
Checkpoints
Database and log file archival
Log file removal
Recovery procedures
Hot failover
Using Recovery on Journaling Filesystems
Recovery and filesystem operations
Berkeley DB recoverability
Transaction tuning
Transaction throughput
Transaction FAQ
12. Berkeley DB Replication
Replication introduction
Replication environment IDs
Replication environment priorities
Building replicated applications
Replication Manager methods
Base API methods
Building the communications infrastructure
Connecting to a new site
Managing Replication Manager group membership
Adding sites to a replication group
Removing sites from a replication group
Primordial startups
Upgrading groups
Replication views
Managing replication directories and files
Replication database directory considerations
Managing replication internal files
Running Replication Manager in multiple processes
One replication process and multiple subordinate processes
Persistence of local site network address configuration
Programming considerations
Handling failure
Other miscellaneous rules
Running replication using the db_replicate utility
One replication process and multiple subordinate processes
Common use case
Avoiding rollback
When to consider an integrated replication application
Choosing a Replication Manager acknowledgement policy
Elections
Synchronizing with a master
Delaying client synchronization
Client-to-client synchronization
Blocked client operations
Clients too far out-of-date to synchronize
Initializing a new site
Bulk transfer
Transactional guarantees
Master leases
Changing group size
Read your writes consistency
Getting a token
Token handling
Using a token to check or wait for a transaction
Clock skew
Communicating between Replication Manager Sites
Configuring for Write Forwarding
Using Replication Manager message channels
Special considerations for two-site replication groups
Two-site strict configuration
Preferred master mode
Other alternatives
Network partitions
Replication FAQ
Ex_rep: a replication example
Ex_rep_base: a TCP/IP based communication infrastructure
Ex_rep_base: putting it all together
Ex_rep_chan: a Replication Manager channel example
13. Berkeley DB Slice Support
Enabling Slices
Slice APIs
Key Design
14. Distributed Transactions
Introduction
Berkeley DB XA Implementation
Building a Global Transaction Manager
Communicating with multiple Berkeley DB environments
Managing the Global Transaction ID (GID) name space
Maintaining state for each distributed transaction.
Recovering from the failure of a single environment
Recovering from GTM failure
XA Introduction
Configuring Berkeley DB with the Tuxedo System
Update the Resource Manager File in Tuxedo
Build the Transaction Manager Server
Update the UBBCONFIG File
Restrictions on XA Transactions
XA: Frequently Asked Questions
15. Application Specific Logging and Recovery
Introduction to application specific logging and recovery
Defining application-specific log records
Automatically generated functions
Application configuration
16. Programmer Notes
Signal handling
Error returns to applications
Globalization Support
Message Format
Enable Globalization Support
Localization Example
Environment variables
Multithreaded applications
Berkeley DB handles
Name spaces
C Language Name Space
Filesystem Name Space
Memory-only or Flash configurations
Disk drive caches
Copying or moving databases
Compatibility with historic UNIX interfaces
Run-time configuration
Performance Event Monitoring
Using the DTrace Provider
Using SystemTap
Example Scripts
Performance Events Reference
Programmer notes FAQ
17. The Locking Subsystem
Introduction to the locking subsystem
Configuring locking
Configuring locking: sizing the system
Standard lock modes
Deadlock detection
Deadlock detection using timers
Deadlock debugging
Locking granularity
Locking without transactions
Locking with transactions: two-phase locking
Berkeley DB Concurrent Data Store locking conventions
Berkeley DB Transactional Data Store locking conventions
Locking and non-Berkeley DB applications
18. The Logging Subsystem
Introduction to the logging subsystem
Configuring logging
Log file limits
19. The Memory Pool Subsystem
Introduction to the memory pool subsystem
Configuring the memory pool
Warming the memory pool
The warm_cache() function
20. The Transaction Subsystem
Introduction to the transaction subsystem
Configuring transactions
Transaction limits
Transaction IDs
Cursors
Multiple Threads of Control
21. Sequences
22. Berkeley DB Extensions: Tcl
Loading Berkeley DB with Tcl
Installing as a Tcl Package
Loading Berkeley DB with Tcl
Using Berkeley DB with Tcl
Tcl API programming notes
Tcl error handling
Tcl FAQ
23. Berkeley DB Extensions
Using Berkeley DB with Apache
Using Berkeley DB with Perl
Using Berkeley DB with PHP
24. Dumping and Reloading Databases
The db_dump and db_load utilities
Dump output formats
Loading text into databases
25. Additional References
Additional references
Technical Papers on Berkeley DB
Background on Berkeley DB Features
Database Systems Theory