Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FlatGeobuf encoding #32

Draft
wants to merge 12 commits into
base: main
Choose a base branch
from
+68 −1
Draft

Add FlatGeobuf encoding #32

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
72cb6ca
Initial version of README.md for FlatGeobuf encoding
StefanBrand Jun 10, 2024
fb51d11
Second version of FlatGeobuf encoding
StefanBrand Jun 10, 2024
4118843
Create datatypes.md for FlatGeobuf encoding
StefanBrand Jun 10, 2024
b416728
Clarify columns encoding for FlatGeobuf
StefanBrand Jun 10, 2024
85e3632
Typo in FlatGeobuf name
StefanBrand Jun 10, 2024
a022ebe
Clarify bounding box data types for FlatGeobuf
StefanBrand Jun 10, 2024
26d8e77
Update CHANGELOG.md
StefanBrand Jun 10, 2024
3c1a039
Make side-car collection file mandatory for FlatGeobuf encoding
StefanBrand Jun 10, 2024
d3e3173
Clarify OGR driver capabilities for FlatGeobuf writing of metadata
StefanBrand Jun 10, 2024
f471e66
Remove obsolete notice about FlatGeobuf version
StefanBrand Jun 10, 2024
e0fb789
Update flatgeobuf/README.md
m-mohr Jun 11, 2024
40232ba
Remove OGR notice from FlatGeobuf spec
StefanBrand Jun 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.

### Added

- ...
- FlatGeobuf encoding

### Changed

Expand Down
41 changes: 41 additions & 0 deletions flatgeobuf/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# FlatGeobuf Encoding Specification

The FlatGeobuf encoding defines how field boundaries compliant to fiboa must be published.
The generic FlatGeobuf format is defined on the [FlatGeobuf website](https://flatgeobuf.org/).

> [!NOTE]
> The FlatGeobuf encoding is still work in progress. Feedback is welcome!

- **[Examples](examples/)**
- **[Data type mapping](datatypes.md)**

## Collection

The FlatGeobuf file must embed the [fiboa Collection](../core/README.md#collection) into the `Header` table's
[`metadata`](https://github.com/flatgeobuf/flatgeobuf/blob/ee7c8f5f45c67dd4a84a51fef518dfebc3e19d0a/src/fbs/header.fbs#L81)
field in a property named `fiboa` within a JSON-structured object.
It is recommended to also provide the fiboa Collection as a separate JSON file.

> [!NOTE]
> Starting in GDAL/OGR 3.9.0 the [FlatGeobuf driver](https://gdal.org/drivers/vector/flatgeobuf.html)
supports writing to the Header's `metadata` field with the `-mo` option, i.e. `-mo "fiboa_version=0.2.0"`.

## Features

Each [fiboa Feature](../core/README.md#features) corresponds to a Flatbuffer record in the
[`Feature`](https://github.com/flatgeobuf/flatgeobuf/blob/ee7c8f5f45c67dd4a84a51fef518dfebc3e19d0a/src/fbs/feature.fbs#L16-L20)
table of a FlatGeobuf file.

All properties defined for fiboa Features, except the Geometry, are made available as the
[`properties`](https://github.com/flatgeobuf/flatgeobuf/blob/ee7c8f5f45c67dd4a84a51fef518dfebc3e19d0a/src/fbs/feature.fbs#L18)
vector of a FlatGeobuf Feature. Key names and data types are encoded into the
[Header's `columns`](https://github.com/flatgeobuf/flatgeobuf/blob/ee7c8f5f45c67dd4a84a51fef518dfebc3e19d0a/src/fbs/header.fbs#L75) schema.

Properties that are optional can be omitted if all values are
[null values](https://github.com/flatgeobuf/flatgeobuf/blob/ee7c8f5f45c67dd4a84a51fef518dfebc3e19d0a/src/fbs/header.fbs#L52)
i.e. the column can be missing from the FlatGeobuf file.

The mapping between the
[FlatGeobuf data types](https://github.com/flatgeobuf/flatgeobuf/blob/ee7c8f5f45c67dd4a84a51fef518dfebc3e19d0a/src/fbs/header.fbs#L26-L42)
and the fiboa data types, can be found in the [data type mapping](datatypes.md).

30 changes: 30 additions & 0 deletions flatgeobuf/datatypes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Mapping of fiboa Data Types to FlatGeobuf

The following table shows the data types that are used by fiboa in the Property definitions.
It also shows the mapping to the [FlatGeobuf data types](https://github.com/flatgeobuf/flatgeobuf/blob/ee7c8f5f45c67dd4a84a51fef518dfebc3e19d0a/src/fbs/header.fbs#L26-L42).

| fiboa Schema data type | FlatGeobuf |
| --------------------------------------------------- | ------------------------------------------------------------ |
| boolean | Bool |
| int8 | Byte |
| uint8 | UByte |
| int16 | Short |
| uint16 | UShort |
| int32 | Int |
| uint32 | UInt |
| int64 | Long |
| uint64 | ULong |
| float<br />IEEE 32-bit | Float |
| double<br />IEEE 64-bit | Double |
| binary | Binary |
| string<br />charset: UTF-8 | String |
| array | Json Array |
| object<br />keys: string<br />values: any | Json Object |
Copy link
Contributor

@m-mohr m-mohr Jun 11, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about structs? I think both could work depending on the specificity of the schema.

I think fiboa, also foe GeoParquet, should do a better job at explaining which schema leads to a struct and which to a map. I'll open an issue: #34

| date | DateTime (ISO 8601 date) |
| date-time<br />with milliseconds<br />timezone: UTC | DateTime (ISO 8601 date time) |
Comment on lines +23 to +24
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't find datetime data types in the specification. Can you help me finding it? Otherwise it should probably string with a pre-defined format (e.g. ISO 8601 / RFC 3339)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I was refering to this line: https://github.com/flatgeobuf/flatgeobuf/blob/2828e07fd759aa8b44c6b1aaab41320b22dd0d29/src/fbs/header.fbs#L40 It is (probably) an ISO 8601 string, but it is it's own ColumnType in the FlatGeobuf schema.

Related comment: #32 (comment)

Copy link
Contributor

@m-mohr m-mohr Jun 11, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that clairifies it. We need to clarify how we handle date only more practically. What happens with the missing components (hour, minute, ...). Maybe add a link somewhere to the allowed types.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I agree DateTime is not well defined. I put a link to the allowed column types here: https://github.com/fiboa/specification/pull/32/files#diff-c33cc7b35504997d9be5d9db7ecaf8f5975eef4b5fe303b28c915033bfba02adR4 But we can also link to https://unpkg.com/[email protected]/dist/doc/enums/generic.ColumnType.html if you prefer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't see that link before, looks good. So here the Date issue is still open.

| geometry | [Geometry](https://github.com/flatgeobuf/flatgeobuf/blob/ee7c8f5f45c67dd4a84a51fef518dfebc3e19d0a/src/fbs/feature.fbs#L5-L14) |
| bounding-box<br />x and y only, no z | Json <br>Bounds Array [minX, minY, maxX, maxY]|

## Potential issues in conversion

TBD
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fiboa allowes enums for strings and integers, but flatbuffers don't.
It think we should mention this here.
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html