Skip to content

Correct way to map unions in a large schema? #559

@atkinsj

Description

@atkinsj

I'm ingesting data from censys.io. Their schema is ~10,000 lines long and contains quite a few unions both in terms of arrays and in the form of "type" : [ "null", "long" ],.

What's the correct way to map these types? avrogen generates code that doesn't unmarshal which I don't think is unexpected but I'm not sure the best way to approach this.

Schema is attached. avrogen generated this:

type Root struct {
	IP               *string               `avro:"ip"`
	Location         *RootLocation         `avro:"location"`
	AutonomousSystem *RootAutonomousSystem `avro:"autonomous_system"`
	Whois            *RootWhois            `avro:"whois"`
	DNS              *RootDNS              `avro:"dns"`
	Services         []RootServices        `avro:"services"`
}

which fails to parse with IP: avro: decode union type: unknown union type. My understanding is that *string should be a nullable type and so it shouldn't need to be registered but it seems like it does?

example.schema.txt

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions