.. _encoding-overview:

Encoding Overview
=================


.. _object-format:

Object Format
-------------
The :term:`Relay` and :term:`Client` use the `CBOR <https://cbor.io/>`_ encoding scheme for objects and to communicate changes made to a :term:`Shop`. We are using this format for a number of reasons: First, it is a scheme standardized by the IETF (as `RFC 8949 <https://www.rfc-editor.org/rfc/rfc8949.html>`_) and thus comes with a wide range of implementations in many languages. Secondly, it comes with canonical encoding mode, which our initial protobuf implementation lacked. This last point allows us to encode all objects of a :term:`Shop` such that we can create a state, or merkle root. This hash allows :term:`Users<User>` to check if the data they have received is complete and consistent.

The object types are as follows:

.. TODO: link to generated code from either CDDL or the Go struct definitions

- :cbor:ref:`Manifest`: the shop's metadata
- :cbor:ref:`Listings`: the items a shop lists for sale
- :cbor:ref:`Tags`: the tags a shop uses to categorize its listings
- :cbor:ref:`Inventory`: how many items are currently available for sale
- :cbor:ref:`Orders`: the list of orders, what items were ordered and by whom
- :cbor:ref:`Accounts`: the list of user accounts

.. _object-visibility:

Object Visibility
-----------------

Not all objects are visible to every :term:`User` of a :term:`Shop`. Certain information is private to :term:`Clerks<clerk>`.

The following objects are publicly visible:

- The :cbor:ref:`Manifest`, :cbor:ref:`Listings` and :cbor:ref:`Tags`
- :cbor:ref:`Accounts` for :term:`Clerks<clerk>`, since :term:`Guests<guest>` need to verify the event signatures,

Orders and the patches to them are restricted. All of them are visible to :term:`Clerks<clerk>` and **Guest** users can only see those that they created.

Changes to the inventory are restricted to :term:`Clerks<clerk>` as well.

.. _object-patches:

Patching Objects
----------------
To not re-transmit objects all the time, we drew inspiration from `JSON Patch <https://jsonpatch.com/>`_ but applied it to using the CBOR scheme. In a nutshell, changes to an object are operationalized, and which value is changed is indicated by the path to the property.

Here is an example, changing the price of a :cbor:ref:`Listing` to 100:

.. code-block:: json

   { "op": "replace", "path": ["Listings", 1, "Price"], "value": 100 }


To build transactional blocks of patches, we are introducing the concept of a PatchSet. This is a list of patches that are applied in order to build up the next state of a :term:`Shop`. Either they all apply correctly or the :term:`Relay` refuses the write request.


More Patch Examples
~~~~~~~~~~~~~~~~~~~

Here are some examples of CBOR patches that can be applied to shop objects:

Adding Order Items and Increasing Quantity
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In this example we are patching the following Order:

.. code-block:: json

  {
    "ID": 42
    "State": 1
    "Items": []
  }

with the following PatchSet with four operations:

* Appending a new :cbor:ref:`OrderedItem` to an :cbor:ref:`Order`
* Appending another :cbor:ref:`OrderedItem` to the same :cbor:ref:`Order`
* Incrementing the quantity of the first :cbor:ref:`OrderedItem`
* and finally, updating the :cbor:ref:`OrderState` from 1 (`Open`) to 3 (`Committed`)

.. code-block:: json

   [
     { "op": "append", "path": ["Orders", 42, "Items"], "value": { "ListingID": 101, "Quantity": 1 } },
     { "op": "append", "path": ["Orders", 42, "Items"], "value": { "ListingID": 202, "VariationIDs": ["large", "blue"], "Quantity": 2 } },
     { "op": "increment", "path": ["Orders", 42, "Items", 0, "Quantity"], "value": 3 }
     { "op": "replace", "path": ["Orders", 42, "State"], "value": 3 }
   ]

The resulting Order is this:

.. code-block:: json

  {
    "ID": 42
    "State": 3
    "Items": [
      { "ListingID": 101, "Quantity": 4 },
      { "ListingID": 202, "Quantity": 2, "VariationIDs": ["large", "blue"] }
    ]
  }


Updating Shop Manifest
^^^^^^^^^^^^^^^^^^^^^^

This example shows a PatchSet with two operations that modify the :term:`Shop`'s :cbor:ref:`Manifest`:

* Adding a new (escrow) :cbor:ref:`Payee` to the :cbor:ref:`Manifest`
* Adding a new :cbor:ref:`ShippingRegion` with a :cbor:ref:`PriceModifier`

.. code-block:: js

   [
     {
       "op": "add",
       "path": ["Manifest", "Payees", 1, Uint8Array("0x1234567890abcdef1234567890abcdef12345678")],
       "value": { "CallAsContract": true }
     },

     { 
       "op": "add",
       "path": ["Manifest", "ShippingRegions", "Germany"],
       "value": {
         "Country": "Germany",
         /* empty fields work as wildcards */
         "PostalCode": "",
         "City": "",
         "PriceModifiers": {
           "standard": {
             "ModificationAbsolute": {
               "Amount": 100,
               "Plus": false
             }
           }
         }
       }
     }
   ]

Updating Listing Images
^^^^^^^^^^^^^^^^^^^^^^^

This example shows a PatchSet that updates the images of a :cbor:ref:`Listing`:

* Removing the first two images from the :cbor:ref:`ListingMetadata`
* Adding two new images to the end of the array

.. code-block:: json

   [
     { "op": "remove", "path": ["Listings", 123, "Metadata", "Images", 0] },
     { "op": "remove", "path": ["Listings", 123, "Metadata", "Images", 0] },
     { "op": "append", "path": ["Listings", 123, "Metadata", "Images"], "value": "https://example.com/images/product123-front.jpg" },
     { "op": "append", "path": ["Listings", 123, "Metadata", "Images"], "value": "https://example.com/images/product123-back.jpg" }
   ]

.. _object-signatures:

Object Signatures
-----------------

We use `EIP191 <https://eips.ethereum.org/EIPS/eip-191>`_ to sign the CBOR encoded header of a PatchSet.

The header contains the following information:

- The shop ID
- The KeyCard nonce (usually a counter)
- The MMR root hash of all objects in the PatchSet
- The timestamp of the PatchSet

A signed PatchSet thus consists of:

- The above header
- A signature of the header
- A list of patches

.. _patchset-mmr:

Merkle Mountain Ranges (MMRs)
-----------------------------

While our PatchSets don't grow, we still used the MMR scheme defined in `this IETF ID <https://datatracker.ietf.org/doc/html/draft-bryce-cose-merkle-mountain-range-proofs>`_. We picked it because it clearly defines how to create roots and accumulators, whereas we found a lot of ambiguity and diversions in other Merkle tree implementations.

In general, Merkle trees allow us to make inclusion proofs, which are a way to prove that a certain object is part of a Merkle tree by providing the necessary sibling nodes to reconstruct the path from a leaf to the root. This allows us to stream only parts of PatchSets to a client, which can then verify the integrity of the received data. Say, if they only require all the patches for a certain listing, they can download only those. See :ref:`ref_market.mass.SubscriptionPushRequest` for more.

.. _cbor-references:

CBOR References
---------------

.. cbor:file:: shop_logical.cddl
   :title: Shop datastructures