Skip to content

Idl parser#90

Merged
mikpe merged 21 commits into
klarna:masterfrom
seriyps:idl-parser
Mar 10, 2026
Merged

Idl parser#90
mikpe merged 21 commits into
klarna:masterfrom
seriyps:idl-parser

Conversation

@seriyps

@seriyps seriyps commented Jul 22, 2019

Copy link
Copy Markdown
Member

Implementation of idl parser. Fixes #68.

Usage Examples

// file: my_service.avdl

@namespace("com.example")
protocol MyService {
  enum Status { OK, ERROR }

  record Response {
    string id;
    Status status;
    union { null, string } message = null;
  }
}

Load a schema file and encode/decode

Store = avro_schema_store:new([], ["my_service.avdl"]),
LookupFun = avro_schema_store:to_lookup_fun(Store),
Encoder = avro:make_encoder(LookupFun, []),
Decoder = avro:make_decoder(LookupFun, []),

Record = [{"id", <<"req-1">>}, {"status", <<"OK">>}],
Bin = iolist_to_binary(Encoder("com.example.Response", Record)),
Record = Decoder("com.example.Response", Bin).

Convert .avdl to an AVPR map (JSON protocol representation)

{ok, Bin} = file:read_file("my_service.avdl"),
Avpr = avro_idl:str_to_avpr(
         binary_to_list(Bin),
         filename:dirname("my_service.avdl")),
%% Avpr is a map: #{<<"protocol">> => ..., <<"types">> => [...], <<"messages">> => ...}

In-memory schemas (useful for tests or embedded schemas)

Files = #{
    {"schemas", "common.avdl"} =>
        <<"protocol Common { record Address { string city; } }">>,
    {"schemas", "order.avdl"} =>
        <<"protocol Orders {  import idl \"common.avdl\"; record Order { string id; int amount; } }">>
},
ReadFun = fun(Cwd, Path) ->
    case maps:find({Cwd, Path}, Files) of
        {ok, Bin} -> {ok, Bin};
        error     -> {error, enoent}
    end
end,
{ok, Src} = maps:find({"schemas", "order.avdl"}, Files),
Schema = avro_idl:decode_schema(
           binary_to_list(Src), "schemas",
           [{read_fun, ReadFun}]).

@coveralls

coveralls commented Jul 22, 2019

Copy link
Copy Markdown

Pull Request Test Coverage Report for Build 599

  • 75 of 77 (97.4%) changed or added relevant lines in 2 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage decreased (-29.7%) to 70.257%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/avro_idl.erl 70 72 97.22%
Totals Coverage Status
Change from base Build 591: -29.7%
Covered Lines: 2870
Relevant Lines: 4085

💛 - Coveralls

Comment thread src/avro_idl_lexer.xrl Outdated
@seriyps
seriyps force-pushed the idl-parser branch 2 times, most recently from 88a4cf8 to 2b2406b Compare March 14, 2020 22:49
@CLAassistant

CLAassistant commented Jan 21, 2021

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

seriyps added 6 commits March 3, 2026 13:32
Lexer fixes:
- Support empty string literals ("") — previously caused illegal token error
- Support backslash escape sequences in strings (\", \\, \n, \t, \r)

Parser fixes:
- Allow annotations and multiple doc comments before function/message
  definitions, consistent with how enum/record/error/fixed declarations
  are handled (meta function rule instead of doc_v function)
- Allow empty record and error type bodies (record Foo {})
- The meta function fix also resolved the undeclared 3rd shift/reduce
  conflict, so Expect 2 is now correct
- process_imports/2 now loads all three import types instead of silently
  dropping them; imported types are returned as {avsc, Map} tagged tuples
  so they pass through protocol_to_avpr/2 without re-conversion
- load_idl_import/2 uses the imported file's dirname as cwd so nested
  relative imports resolve correctly
- Add tests for each import type and for nested directory imports
Detect ".avdl" extension and delegate to avro_idl:decode_schema/2,
then flatten the resulting union type and add each named type to the
store individually via do_add_type/2.
Add a {read_fun, fun((Cwd, Path) -> {ok, Bin} | {error, term()})}
option to str_to_avpr/3 and decode_schema/3. The default read_fun
uses file:read_file/1. The fun is threaded through recursive IDL
imports so all file I/O can be intercepted at every level.

Also add decode_schema/3 as a convenience wrapper accepting the
same options.
- Add encode_decode_test/0 in avro_idl_tests covering:
  - types from imported .avdl (idl import)
  - types from imported .avpr (protocol import)
  - types from imported .avsc (schema import)
  - cross-namespace type references (org.erlang.ftp.MyAnnotated)
- Fix protocol_with_typedefs.avdl: MyAnnotated referenced MyError
  (an error type filtered out by decode_schema), replaced with a
  valid reference to org.erlang.www.MyEnum2
@seriyps

seriyps commented Mar 4, 2026

Copy link
Copy Markdown
Member Author

@zmstone this is now ready. It took me some time to finalize it 😄

Comment thread src/idl.hrl Outdated

@mikpe mikpe left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the caveat wrt the record name.

The `record` would be a reserved word and would warn in OTP-29; the
rest of the records are updated for consistency
Comment thread rebar.config Outdated
@mikpe
mikpe merged commit 618a557 into klarna:master Mar 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Would be nice to have support for Avro IDL schema format (.avdl)

5 participants