Protox is an Elixir library for working with Google's Protocol Buffers (proto2 and proto3): encode/decode to/from binary, generate code, or compile schemas at build time.

Protox emphasizes reliability: it uses property testing, mutation testing, maintains near 100% coverage, and passes Google’s conformance suite.

Given the following protobuf definition:

message Msg{
  int32 a = 1;
  map<int32, string> b = 2;
}

Protox will create a regular Elixir Msg struct:

iex> msg = %Msg{a: 42, b: %{1 => "a map entry"}}
iex> {:ok, iodata, iodata_size} = Msg.encode(msg)

iex> binary = # read binary from a socket, a file, etc.
iex> {:ok, msg} = Msg.decode(binary)

You can use Protox in two ways:

1. pass the protobuf schema (as an inlined schema or as a list of files) to the Protox macro;
2. generate Elixir source code files with the mix task protox.generate.

• Elixir >= 1.15 and OTP >= 26
• protoc >= 3.0 This dependency is only required at compile-time. It must be available in $PATH.

Add :protox to your list of dependencies in mix.exs:

def deps do
  [{:protox, "~> 2.0"}]
end

The following example generates two modules, Baz and Foo:

defmodule MyModule do
  use Protox, schema: """
  syntax = "proto3";

  message Baz {
  }

  message Foo {
    int32 a = 1;
    map<int32, Baz> b = 2;
  }
  """
end

Use the :files option to pass a list of files:

defmodule MyModule do
  use Protox, files: [
    "./defs/foo.proto",
    "./defs/bar.proto",
    "./defs/baz/fiz.proto"
  ]
end

Here's how to encode a message to binary protobuf:

msg = %Foo{a: 3, b: %{1 => %Baz{}}}
{:ok, iodata, iodata_size} = Protox.encode(msg)
# or using the bang version
{iodata, iodata_size} = Protox.encode!(msg)

You can also call encode/1 and encode!/1 directly on the generated structures:

{:ok, iodata, iodata_size} = Foo.encode(msg)
{iodata, iodata_size} = Foo.encode!(msg)

Here's how to decode a message from binary protobuf:

{:ok, msg} = Protox.decode(<<8, 3, 18, 4, 8, 1, 18, 0>>, Foo)
# or using the bang version
msg = Protox.decode!(<<8, 3, 18, 4, 8, 1, 18, 0>>, Foo)

You can also call decode/1 and decode!/1 directly on the generated structures:

{:ok, msg} = Foo.decode(<<8, 3, 18, 4, 8, 1, 18, 0>>)
msg = Foo.decode!(<<8, 3, 18, 4, 8, 1, 18, 0>>)

Protox honors the package directive:

package abc.def;
message Baz {}

The example above is translated to Abc.Def.Baz (package abc.def is camelized to Abc.Def).

You can prepend a namespace with a prefix using the :namespace option:

defmodule Bar do
  use Protox, schema: """
    syntax = "proto3";

    package abc;

    message Msg {
        int32 a = 1;
      }
    """,
    namespace: __MODULE__
end

In this example, the module Bar.Abc.Msg is generated:

msg = %Bar.Abc.Msg{a: 42}

One or more include paths (directories in which to search for imports) can be specified using the :paths option:

defmodule Baz do
  use Protox,
    files: [
      "./defs1/prefix/foo.proto",
      "./defs1/prefix/bar.proto",
      "./defs2/prefix/baz/baz.proto"
    ],
    paths: [
      "./defs1",
      "./defs2"
    ]
end

It's possible to generate Elixir source code files with the mix task protox.generate:

protox.generate --output-path=/path/to/messages.ex protos/foo.proto protos/bar.proto

The files will be usable in any project as long as Protox is declared in the dependencies as functions from its runtime are used.

• --output-path

  The path to the file to be generated or to the destination folder when generating multiple files.

• --include-path

  Specifies the include path. If multiple include paths are needed, add more --include-path options.

• --multiple-files

  Generates one file per Elixir module. It's useful for definitions with a lot of messages as the compilation will be parallelized. When generating multiple files, the --output-path option must point to a directory.

• --namespace

  Prepends a namespace to all generated modules.

Unknown fields are fields present on the wire that do not correspond to the protobuf definition. This enables forward-compatibility: older readers keep and re-emit fields added by newer writers.

When unknown fields are encountered at decoding time, they are kept in the decoded message. It's possible to access them with the unknown_fields/1 function defined with the message.

iex> msg = Msg.decode!(<<8, 42, 42, 4, 121, 97, 121, 101, 136, 241, 4, 83>>)
%Msg{a: 42, b: "", z: -42, __uf__: [{5, 2, <<121, 97, 121, 101>>}]}

iex> Msg.unknown_fields(msg)
[{5, 2, <<121, 97, 121, 101>>}]

Always use unknown_fields/1 since the field name (e.g. __uf__) is generated to avoid collisions with protobuf fields. It returns a list of {tag, wire_type, bytes}. See the protobuf encoding guide for details.

• The Any well-known type is partially supported: you can manually unpack the embedded message after decoding and conversely pack it before encoding;
• Groups (deprecated in protobuf);
• All options other than packed and default are ignored as they concern other languages implementation details.

• (Protobuf 2) Required fields encoding raises Protox.RequiredFieldsError when a required field is missing.

  defmodule Bar do
    use Protox, schema: """
      syntax = "proto2";
  
      message Required {
        required int32 a = 1;
      }
    """
  end
  
  iex> Protox.encode!(%Required{})
  ** (Protox.RequiredFieldsError) Some required fields are not set: [:a]

• (Protobuf 2) Nested extensions Fields names coming from a nested extension are prefixed with the name of the extender:

  message Extendee {
    extensions 100 to max;
  }
  
  message Extension1 {
    extend Extendee {
      optional Extension1 ext1 = 102;
    }
  }
  
  message Extension2 {
    extend Extendee {
      optional int32 ext2 = 103;
    }
  }
  
  message Extension3 {
    extend Extendee {
      optional int32 identical_name = 105;
    }
  }
  
  message Extension4 {
    extend Extendee {
      repeated int32 identical_name = 106;
    }
  }

  In the above example, the fields of Extendee will be:

    :extension1_ext1
    :extension2_ext2
    :extension3_identical_name
    :extension4_identical_name

  This is to disambiguate cases where fields in extensions have the same name.

• Enum aliases When decoding, the last encountered constant is used. For instance, in the following example, :BAR is always used if the value 1 is read on the wire:

  enum E {
    option allow_alias = true;
    FOO = 0;
    BAZ = 1;
    BAR = 1;
  }

• (Protobuf 2) Unset optional fields are assigned nil. You can use the generated default/1 function to get the default value of a field:

  defmodule Bar do
    use Protox,
    schema: """
      syntax = "proto2";
  
      message Foo {
        optional int32 a = 1 [default = 42];
      }
    """
  end
  
  iex> %Foo{}.a
  nil
  
  iex> Foo.default(:a)
  {:ok, 42}

• (Protobuf 3) Unset fields are assigned to their default values. However, if you use the optional keyword (available in protoc >= 3.15), then unset fields are assigned nil:

  defmodule Bar do
    use Protox,
    schema: """
      syntax = "proto3";
  
      message Foo {
        int32 a = 1;
        optional int32 b = 2;
      }
    """
  end
  
  iex> %Foo{}.a
  0
  
  iex> Foo.default(:a)
  {:ok, 0}
  
  iex> %Foo{}.b
  nil
  
  iex> Foo.default(:b)
  {:error, :no_default_value}

• Messages and enums names are converted using the Macro.camelize/1 function. Thus, in the following example, non_camel_message becomes NonCamelMessage, but the field non_camel_field is left unchanged:

  defmodule Bar do
    use Protox,
    schema: """
      syntax = "proto3";
  
      message non_camel_message {
      }
  
      message CamelMessage {
        int32 non_camel_field = 1;
      }
    """
  end
  
  iex> msg = %NonCamelMessage{}
  %NonCamelMessage{__uf__: []}
  
  iex> msg = %CamelMessage{}
  %CamelMessage{__uf__: [], non_camel_field: 0}

• The detailed reference of the generated code is available in documentation/reference.md.
• Please see documentation/types_mapping.md to see how protobuf types are mapped to Elixir types.

The Protox library has been thoroughly tested using the conformance checker provided by Google.

Run the suite with:

mix protox.conformance

See benchmark/launch_benchmark.md for running benchmarks.

Please see CONTRIBUTING.md for more information on how to contribute.