Adding new variable length field in a repeating group #26

v1-0-DRAFT_STANDARD/doc/01Introduction.md

@@ -149,7 +149,7 @@ References
 
 ### Related FIX Standards 
 
-*Simple Open Framing Header*, FIX Protocol, Limited. Release Candidate 1
+*Simple Open Framing Header*, FIX Protocol, Limited. Version 1.0 Draft Standard
 specification has been published at
 <http://www.fixtradingcommunity.org/>
 

v1-0-DRAFT_STANDARD/doc/03MessageStructure.md

@@ -4,10 +4,8 @@ Message Structure
 Message Framing
 ------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-SBE messages need framing when used with protocols that do not preserve
-message boundaries, such as when they are transmitted on a streaming
-session protocol or are persisted in storage. Be aware that framing
-features may or may not be encoded in SBE.
+SBE messages have no defined message delimiter. Version 2.0 of SBE makes it possible to walk the elements of a message to determine its limit, even when the message has been extended. Nevertheless, since internal framing depends on a correct starting point and not encountering malformed messages, it may be desirable to use an external framing protocol when used with transports that do not preserve message boundaries, such as when they are transmitted on a streaming
+session protocol or when persisting messages in storage. 
 
 ### Simple Open Framing Header
 
@@ -73,6 +71,10 @@ The fields of the SBE message header are:
 -   **Schema version** - the version of the message schema in which the
     message is defined
 
+-   **Group count** - the number of repeating groups in the root level of the message
+
+-   **Variable-length field count** - the number of variable-length fields in the root level of the message
+
 Block length is specified in a message schema, but it is also serialized
 on the wire. By default, block length is set to the sum of the sizes of
 body fields in the message. However, it may be increased to force
@@ -95,17 +97,21 @@ Recommended message header encoding
     <type name="templateId" primitiveType="uint16"/>
     <type name="schemaId" primitiveType="uint16"/>
     <type name="version" primitiveType="uint16"/>
+	<type name="numGroups" primitiveType="uint16" />
+    <type name="numVarDataFields" primitiveType="uint16" />
 </composite>
 ```
 
-The recommended header encoding is 8 octets.
+The recommended header encoding is 12 octets.
 
 | Element     | Description       | Primitive type | Length (octets) | Offset |
 |-------------|-------------------|----------------|----------------:|-------:|
 | blockLength | Root block length | uint16         | 2               | 0      |
 | templateId  | Template ID       | uint16         | 2               | 2      |
 | schemaId    | Schema ID         | uint16         | 2               | 4      |
 | version     | Schema Version    | uint16         | 2               | 6      |
+| numGroups   |Number of repeating groups | uint16 | 2               | 8      |
+| numVarDataFields | Number of variable-length fields | uint16 | 2   | 10     |
 
 Optionally, implementations may support any other unsigned integer types
 for blockLength.
@@ -141,6 +147,15 @@ attributes.
 The version number of the message schema that was used to encode a
 message. See section 4.3.1 below for schema attributes.
 
+### Number of repeating groups
+
+A count of repeating groups at the root level of the message. The count does not include nested repeating groups.
+
+### Number of variable-length fields
+
+A count of the variable-length fields at the root level of the message. The count does not include variable-length fields within repeating groups.
+
+
 Message Body
 ----------------------------------------------------------------------------------------------------------
 
@@ -425,12 +440,31 @@ Recommended encoding of repeating group dimensions
 <composite name="groupSizeEncoding">
     <type name="blockLength" primitiveType="uint16"/>
     <type name="numInGroup" primitiveType="uint16" semanticType="NumInGroup"/>
+	<type name="numGroups" primitiveType="uint16" />
+    <type name="numVarDataFields" primitiveType="uint16" />
 </composite>
 ```
 
-Wire format of NumInGroup with block length 55 octets by 3 entries
+#### Block length
+
+The total space reserved for the fixed-length fields of this repeating group, not counting
+any repeating groups or variable-length fields.
+
+#### Number of entries
 
-`37000300`
+The number of entries in this repeating group, called NumInGroup in FIX.
+
+##### Number of repeating groups
+
+A count nested repeating groups in this repeating group.
+
+#### Number of variable-length fields
+
+A count of the variable-length fields in this repeating group.
+
+Wire format of NumInGroup with block length 55 octets by 3 entries, containing one nested group and two variable-length fields.
+
+`3700030001000200`
 
 #### Restricting repeating group entries
 
@@ -439,10 +473,7 @@ The occurrences of a repeating group may be restricted to a specific range by mo
 Example of a restricted group encoding
 
 ```xml
-<composite name="restrictedGroupSizeEncoding">
-    <type name="blockLength" primitiveType="uint16"/>
-    <type name="numInGroup" primitiveType="uint16" semanticType="NumInGroup" minValue="1" maxValue="10" />
-</composite>
+<type name="numInGroup" primitiveType="uint16" semanticType="NumInGroup" minValue="1" maxValue="10" />
 ```
 
 Sequence of message body elements

v1-0-DRAFT_STANDARD/doc/04MessageSchema.md

@@ -3,16 +3,16 @@
 
 XML schema for SBE message schemas
 ---------
-See [SimpleBinary1-0.xsd](../resources/SimpleBinary1-0.xsd) for the normative XML Schema Definition (XSD) for SBE.
+See [sbe.xsd](../resources/sbe.xsd) for the normative XML Schema Definition (XSD) for SBE.
 
 
 XML namespace
 -----------------------------------------------------------------------------------------------------------
 
-The Simple Binary Encoding XML schema is identified by this URL:
+The Simple Binary Encoding XML schema is identified by this URL [*tentative*]:
 
 ```xml
-xmlns:sbe=http://fixprotocol.io/sbe/rc4
+xmlns:sbe=http://fixprotocol.io/2017/sbe 
 ```
 
 Conventionally, the URI of the XML schema is aliased by the prefix

v1-0-DRAFT_STANDARD/doc/05SchemaExtensionMechanism.md

@@ -22,12 +22,9 @@ Compatibility is only ensured under these conditions:
 
 -   Existing fields cannot change data type or move within a message.
 
--   A repeating group may be added, but only after existing groups and
-    if there are no subsequent variable data elements at the end of the
-    message.
+-   A repeating group may be added after existing groups at the root level or nested within another repeating group.
 
--   A variable data element may be added, but only after existing groups
-    and data.
+-   A variable-length data field may be added after existing variable-length data at the root level or within a repeating group.
 
 -   Message header encoding cannot change.
 
@@ -92,14 +89,6 @@ migration to replacement message layouts.
 Wire format features for extension
 ----------------------------------
 
-### Message size
-
-It is assumed that a either message boundaries are delimited by a
-transport or session protocol header conveys the size of the whole
-message. See section 3.1 above. This enables a consumer to properly
-frame messages even when the message has been lengthened in a later
-version of the schema.
-
 ### Block size
 
 The length of the root level of the message is sent on the wire in the
@@ -113,6 +102,10 @@ does not break parsing of earlier fields.
 Likewise, block size of a repeating group is conveyed in the NumInGroup
 encoding.
 
+### Number of repeating groups and variable data
+
+Message headers and repeating group dimensions carry a count of the number of repeating groups and a count of variable-length data fields on the wire. This supports a walk by a decoder of all the elements of a message, even when the decoder was built with an older version of a schema. As for added fixed-length fields, new repeating groups cannot be interpreted by the decoder, but it still can process the ones it knows, and it can correctly reach the end of a message.
+
 Comaptibility strategy
 -----------------------
 *This suggested strategy is non-normative.*

v1-0-DRAFT_STANDARD/doc/08ReleaseNotes.md

@@ -1,72 +1,16 @@
 Release Notes
 =============
-Release Candidate 4
--------------------
-
-These issues were resolved and accepted for Release Candidate 4. See issues and pull requests
-in GitHub for details and changes.
 
-| Issue | Description                                                    | Section |
-|------:|----------------------------------------------------------------|---------|
-| 2     | Schema extension is vague in terms of what compatibility means | 5
-| 3     | Extensibility of the Template ID                               | 5
-| 6     | Limiting maximum occurences of repeating group                 | 3
-| 8     | blockLength for repeating groups of variable length            | 3
-| 11    | Offsets within composite types                                 | 4
-| 12    | Composites reusing other types                                 | 4
+These are the changes made since SBE version 1.0. 
 
-Release Candidate 3
+SBE version 2.0 Release Candidate 1
 -------------------
+These issues were resolved and accepted for version 2.0 Release Candidate 1. See [issues](https://github.com/FIXTradingCommunity/fix-simple-binary-encoding/issues) and [pull requests](https://github.com/FIXTradingCommunity/fix-simple-binary-encoding/pulls) in GitHub for details and changes.
 
-This is a summary of document changes to Release Candidate 3 from RC2.
-Changes in this release were intended only as clarifications or to add
-capabilities. Message schemas that conformed to the RC2 specification
-should still conform to the RC3 wire format.
-
-**Section 1**
-
-References section expanded.
-
-**Section 2**
-
--   Statement added that non-FIX data types should not carry a
-    semanticType attribute in a message schema.
-
--   String encoding section split into two sections for strings (text
-    fields) and data (non-character data) to clarify the distinction.
-    Both text and non-text can be either fixed-length `<field>` or
-    variable-length `<data>`.
-
--   Timestamp encoding enhanced to allow time unit to either be
-    specified as a constant in a message schema or to be serialized on
-    the wire.
-
-**Section 3**
-
-Message structure is enhanced to allow variable-length `<data>` elements
-within a repeating group entry.
-
-**Section 4**
-
-Message schema XSD updated to support `<data>` in repeating groups and
-for various other refinements
-
-**Section 5**
-
--   Statements added to say whole repeating groups or variable data may
-    be added to a message without breaking compatibility so long as the
-    added elements are at the end of a message.
-
--   Added deprecated schema attribute to mark obsolete elements.
-
-**Section 6**
-
-No change
-
-**Section 7**
-
-Examples updated to use Simple Open Framing Header.
+| Issue | Description                                                    |
+|------:|----------------------------------------------------------------|
+| 26    | Adding new variable length field in a repeating group          |
 
-**Section 8**
+### Compatibility
 
-Release notes added.
+Version 2.0 is not back-compatible SBE version 1.0, either in wire format or XML schema.

v1-0-DRAFT_STANDARD/resources/sbe.xsd

@@ -1,10 +1,10 @@
 <?xml version="1.0" encoding="utf-8"?>
-<xs:schema xmlns:sbe="http://fixprotocol.io/2016/sbe" xmlns:xs="http://www.w3.org/2001/XMLSchema" targetNamespace="http://fixprotocol.io/2016/sbe" elementFormDefault="unqualified" version="1.0 Draft Standard">
+<xs:schema xmlns:sbe="http://fixprotocol.io/2017/sbe" xmlns:xs="http://www.w3.org/2001/XMLSchema" targetNamespace="http://fixprotocol.io/2017/sbe" elementFormDefault="unqualified" version="2.0-RC1">
 	<xs:annotation>
 		<xs:documentation>
 			Message schema for FIX Simple Binary Encoding
-			Version: 1.0 Draft Standard
-			© Copyright 2014-2016 FIX Protocol Limited
+			Version: 2.0 Release Candidate 1
+			© Copyright 2014-2017 FIX Protocol Limited
 			License: Creative Commons Attribution-NoDerivatives 4.0 International Public License
 		</xs:documentation>
 	</xs:annotation>