The osquery "public API" or SDK is the set of osquery headers and a subset of the source "cpp" files implementing what we call osquery core. The core code can be thought of as the framework or platform, it is everything except for the SQLite code and most table implementations. The public headers can be found in osquery/include/osquery/.

osquery is organized into a core, additional, and testing during a default build from source. We call the set of public headers implementing core the 'osquery SDK'. This SDK can be used to build osquery outside of our CMake build system with a minimum set of dependencies. This organization better isolates OS API dependencies from development tools and libraries and provides a logical separation between code needed for extensions and module compiling.

The public API and SDK headers are documented via doxygen. To generate web-based documentation, you will need to install doxygen, run make docs from the repository root, then open ./build/docs/html/index.html.

Extensions

Extensions are separate processes that communicate over a Thrift IPC channel to osquery core in order to register one or more plugins or virtual tables. An extension allows you to develop independently: it may be compiled and linked using an external build system, against proprietary code. Your extension will be version-compatible with our publicly-built binary packages on https://osquery.io/downloads.

Extensions use osquery's Thrift API to communicate between osqueryi or osqueryd and the extension process. Extensions are commonly written in C++, but can also be developed in Python, in Go, or in really any language that supports Thrift.

Only the osquery SDK provides the simple startExtension symbol that manages the life of your process, including the Thrift service threads and a watchdog.

osquery extensions should statically link the core code and use the <osquery/sdk.h> helper include file. C++ extensions should link: boost, thrift, glog, gflags, and optionally rocksdb for eventing. Let's walk through a basic example extension in C++ (source for example_extension.cpp):

  1. // Note 1: Include the sdk.h helper.
  2. #include <osquery/sdk.h>
  4. using namespace osquery;
  6. // Note 2: Define at least one plugin or table.
  7. class ExampleTablePlugin : public TablePlugin {
  8.  private:
  9.   TableColumns columns() const override {
  10.     return {
  11.       std::make_tuple("example_text", TEXT_TYPE, ColumnOptions::DEFAULT),
  12.       std::make_tuple("example_integer", INTEGER_TYPE, ColumnOptions::DEFAULT),
  13.     };
  14.   }
  16.   QueryData generate(QueryContext& request) override {
  17.     QueryData results;
  18.     Row r;
  20.     r["example_text"] = "example";
  21.     r["example_integer"] = INTEGER(1);
  22.     results.push_back(r);
  23.     return results;
  24.   }
  25. };
  27. // Note 3: Use REGISTER_EXTERNAL to define your plugin or table.
  28. REGISTER_EXTERNAL(ExampleTablePlugin, "table", "example");
  30. int main(int argc, char* argv[]) {
  31.   // Note 4: Start logging, threads, etc.
  32.   osquery::Initializer runner(argc, argv, ToolType::EXTENSION);
  34.   // Note 5: Connect to osqueryi or osqueryd.
  35.   auto status = startExtension("example", "0.0.1");
  36.   if (!status.ok()) {
  37.     LOG(ERROR) << status.getMessage();
  38.     runner.requestShutdown(status.getCode());
  39.   }
  41.   // Finally, shutdown.
  42.   runner.waitForShutdown();
  43.   return 0;
  44. }

The osqueryi or osqueryd processes start an "extension manager" Thrift service thread that listens for extension register calls on a UNIX domain socket. Extensions may only communicate if the processes can read/write to this socket. An extension process running as a non-privileged user cannot register plugins to an osqueryd process running as root. Both the osquery core and C++ extensions using startExtension will deregister plugins if the communication becomes latent. Both of these settings are configurable using gflags or the osquery config options.

Using the example extension

Please see the deployment guide on extensions for a more-complete overview of how and why extensions are used.

If you build from source, you will build an example extension. The code can be found in the osquery/examples folder; it adds a config plugin called "example" and additional table called "example". There are two ways to run an extension: load the extension at an arbitrary time after shell or daemon execution, or request an "autoload" of extensions. The auto-loading method has several advantages, such as allowing dependencies on external config plugins and inheriting the same process monitoring as is applied to the osquery core worker processes.

The osqueryi shell also allows a quick and easy command-line autoload using —extension. Let's review both options.

First, to load the example extension in the osquery interactive shell, we start osqueryi:

  1. $ ./build/darwin/osquery/osqueryi
  2. osquery> SELECT path FROM osquery_extensions;
  3. +-------------------------------------+
  4. | path                                |
  5. +-------------------------------------+
  6. | /Users/USERNAME/.osquery/shell.em   |
  7. +-------------------------------------+
  8. osquery> ^Z
  9. [1]  + 98777 suspended  ./build/darwin/osquery/osqueryi

Here we have started the osquery shell process, inspected the UNIX domain socket path it uses to communicate with extension processes, and then suspended the process temporarily.

  1. $ ./build/darwin/osquery/example_extension.ext --help
  2. osquery 1.7.0, your OS as a high-performance relational database
  3. Usage: example_extension.ext [OPTION]...
  5. osquery extension command line flags:
  7.     --interval VALUE  Seconds delay between connectivity checks
  8.     --socket VALUE    Path to the extensions UNIX domain socket
  9.     --timeout VALUE   Seconds to wait for autoloaded extensions
  11. osquery project page <https://osquery.io>.
  12. $ ./build/darwin/osquery/example_extension.ext --socket /Users/USERNAME/.osquery/shell.em &

Before executing the extension we've inspected the potential CLI flags, which are a subset of the shell or daemon's CLI flags. We executed the example extension in the background, so now we can resume the osqeury shell and use the 'example' table provided by the extension.

  1. [2] 98795
  2. $ fg
  3. [1]  - 98777 continued  ./build/darwin/osquery/osqueryi
  4. osquery> SELECT * FROM example;
  5. +--------------+-----------------+
  6. | example_text | example_integer |
  7. +--------------+-----------------+
  8. | example      | 1               |
  9. +--------------+-----------------+
  10. osquery>

If the responsible osquery shell or daemon process ends, the extension will detect the loss of communication and also shutdown soon after. Read more about the lifecycle of extensions in the deployment guide.

The second, and simpler, way to manually load an extension is at the osqueryi command line with —extension:

  1. $ ./build/darwin/osquery/osqueryi --extension ./build/darwin/osquery/example_extension.ext

Building external extensions

Your "external" extension, in the sense that the code is developed and contained somewhere external from the osquery repository, can be built semi-automatically.

  • Symlink your external extension source code directory into ./external. (ln -s [extension_source_dir] [osquery_external_subdir] on Linux or macOS, mklink /D [osquery_external_subdir] [extension_source_dir] on Windows)
  • Make sure the symlink contains extension_ as a prefix.
  • Run make externals.This will find and compile all .*.{cpp,c,mm} files within your external directory. If you need something more complicated, add a CMakeLists.txt to your directory and add your targets to the externals target.

See CMake/CMakeLibs.cmake for more information about the ADD_OSQUERY_EXTENSION CMake macro.

Example:

  1. (osquery) $ ln -s ~/git/fun-osquery-extension ./external/extension_fun
  2. (osquery) $ ls ./external/extension_fun/
  3. fun.cpp
  4. (osquery) $ make externals
  5. [...]
  6. [100%] Built target libosquery
  7. Scanning dependencies of target external_extension_awesome
  8. [100%] Building CXX object external/CMakeFiles/external_extension_fun.dir/extension_fun/fun.cpp.o
  9. [100%] Linking CXX executable external_extension_fun.ext
  10. [100%] Built target external_extension_fun
  11. [100%] Built target externals

Bundling multiple extensions into a single-executable extension

All of the extensions declared with the add_osquery_extension_ex() CMake function will be automatically bundled into a single executable.

The executable name and version can be changed using the following two environment variables:

  • OSQUERY_EXTENSION_GROUP_NAME (default: osquery_extension_group)
  • OSQUERY_EXTENSION_GROUP_VERSION (default: 1.0)It is important to provide a header file that can be included by the generated main.cpp file; its purpose is to define the types used by the REGISTER_EXTERNAL directive.

An example is included in the osquery/examples/extension_group_example.

Please note that when using bundling, the source directory of each extension is added to the include list; developers should always use uniquely named include files. Additionally, if you are using RapidJSON documents in your extension, then you should instead leverage the osquery json.h header to avoid linking issues due to how we have configured RapidJSON #defines.

Thrift API

Thrift is a code-generation/cross-language service development framework. osquery uses Thrift to allow extensions to implement plugins for config retrieval, log export, table implementations, event subscribers, and event publishers. We also use Thrift to wrap our SQL implementation using SQLite.

Extension API

An extension process should implement the following API. During an extension's set up it will "broadcast" all of its registered plugins to the osquery shell or daemon process. Then the extension will be asked to start a UNIX domain socket and Thrift service thread implementing the ping and call methods.

  1. service Extension {
  2.   /// Ping to/from an extension and extension manager for metadata.
  3.   ExtensionStatus ping(),
  4.   /// Call an extension (or core) registry plugin.
  5.   ExtensionResponse call(
  6.     /// The registry name (e.g., config, logger, table, etc.).
  7.     1:string registry,
  8.     /// The registry item name (plugin name).
  9.     2:string item,
  10.     /// The Thrift-equivalent of an osquery::PluginRequest.
  11.     3:ExtensionPluginRequest request),
  12. }

When an extension becomes unavailable, the shell or daemon process will automatically deregister those plugins.

Extension Manager API (osqueryi/osqueryd)

  1. service ExtensionManager extends Extension {
  2.   /// Return the list of active registered extensions.
  3.   InternalExtensionList extensions(),
  4.   /// Return the list of bootstrap or configuration options.
  5.   InternalOptionList options(),
  6.   /// The API endpoint used by an extension to register its plugins.
  7.   ExtensionStatus registerExtension(
  8.     1:InternalExtensionInfo info,
  9.     2:ExtensionRegistry registry),
  10.   ExtensionStatus deregisterExtension(
  11.     1:ExtensionRouteUUID uuid,
  12.   ),
  13.   /// Allow an extension to query using an SQL string.
  14.   ExtensionResponse query(
  15.     1:string sql,
  16.   ),
  17.   /// Allow an extension to introspect into SQL used in a parsed query.
  18.   ExtensionResponse getQueryColumns(
  19.     1:string sql,
  20.   ),
  21. }