Re: Captain of the Ship


Sean Barnum
 

Agree. This is what I alluded to in a prior response.

Adding the type properties rather than a simple array makes serialization/deserialization more complicated and does not work with most standards-based ecosystem tooling.

 

sesan

 

From: Spdx-tech@... <Spdx-tech@...> on behalf of Gary O'Neall <gary@...>
Date: Thursday, July 21, 2022 at 3:53 PM
To: willbar@... <willbar@...>, Spdx-tech@... <Spdx-tech@...>, 'David Kemp' <dk190a@...>
Subject: [EXT] Re: [spdx-tech] Captain of the Ship

> Unfortunately, that one is a two-edged sword. If you don’t know the type (e.g. you’re trying to look something up by ID) then you need to search through all the types to find the ID. Conversely, if you want to find everything of a certain type then grouping by type is beneficial.

 

Good point if you’re using the serialization format to represent your internal storage of the graph.  In all my SPDX software, I use a different internal representation of the SPDX graph than what is represented in the serialization format so this particular situation never comes up.  This brings up another meta-issue – should we be optimizing the serialization format to be used as an internal storage format or optimizing it for deserialization and reserialization?  If the latter, than having arrays of types is much easier IMHO.  If you go the type property route, all the deserializers I’m familiar with would require writing custom deserialization code whereas using the arrays can use just the of the shelf libraries.  I’m happy to be proven wrong on this point if anyone knows of a deserializer for JSON (not JSON-LD) that can understand the type property.

 

To your second meta issue, Below are my thoughts based on past experience maintaining some of the SPDX tooling:

 

  • If we ONLY support JSON-LD, a number of issues go away and the tooling is vastly simplified.
  • Supporting JSON-LD and the RDF dialects are just slightly more complicated for the tooling since JSON-LD can be viewed as another dialect of RDF.
  • Supporting YAML and/or XML introduces some of the same issues as supporting a simplified JSON format.  If we support one of these, we might as well support all IMHO.
  • Tag/Value is it’s own set of (rather large) complexities.
  • Spreadsheets have a similar set of complexities as Tag/Value, but they are distinct enough that there isn’t much leverage between solving both at the same time.  I will be using spreadsheets myself, so I’ll probably continue to support some type of spreadsheet format in 3.0 if it is at all feasible.

 

Gary

 

 

From: Spdx-tech@... <Spdx-tech@...> On Behalf Of William Bartholomew (CELA) via lists.spdx.org
Sent: Thursday, July 21, 2022 12:20 PM
To: William Bartholomew (CELA) <willbar@...>; gary@...; Spdx-tech@...; David Kemp <dk190a@...>
Subject: Re: [spdx-tech] Captain of the Ship

 

There’s a meta-question here that we need to answer related to JSON serialization, would SPDX 3.0 support JSON and JSON-LD, just JSON, or just JSON-LD? I’d lean towards JSON-LD as long as we have a purely mechanical upgrade process from SPDX 2.x JSON to SPDX 3.x JSON-LD. If we adopt JSON-LD then a number of serialization design questions already have answers, and it is still parseable as JSON.

 

 

Regards,

 

William Bartholomew (he/him) – Let’s chat

Principal Security Strategist

Global Cybersecurity Policy – Microsoft

 

My working day may not be your working day. Please don’t feel obliged to reply to this e-mail outside of your normal working hours.

 

From: Spdx-tech@... <Spdx-tech@...> On Behalf Of William Bartholomew (CELA) via lists.spdx.org
Sent: Thursday, July 21, 2022 12:16 PM
To: gary@...; Spdx-tech@...; David Kemp <dk190a@...>
Subject: [EXTERNAL] Re: [spdx-tech] Captain of the Ship

 

Unfortunately, that one is a two-edged sword. If you don’t know the type (e.g. you’re trying to look something up by ID) then you need to search through all the types to find the ID. Conversely, if you want to find everything of a certain type then grouping by type is beneficial.

 

I’d lean towards not grouping by type because you can always create a type->id mapping when deserializing. Given that we’ll have more types with profiles, I think grouping by type will have more downsides than upsides.

 

William

 

From: Spdx-tech@... <Spdx-tech@...> On Behalf Of Gary O'Neall via lists.spdx.org
Sent: Wednesday, July 20, 2022 10:26 AM
To: Spdx-tech@...; David Kemp <dk190a@...>; SPDX-list <Spdx-tech@...>
Subject: [EXTERNAL] Re: [spdx-tech] Captain of the Ship

 

One additional consideration that came up in the 2.X discussion was how to handle the type for the elements.

In David's example, the type is one of the properties. For 2.X, we implemented separate arrays for each type. For some of the JSON serialization libraries, this affords a significant convenience when deserializing into objects of the same type.

Note that this isn't an issue for JSON-LD or RDF serialization formats which natively handle types.

Gary

On July 20, 2022 11:57:01 AM CDT, David Kemp <dk190a@...> wrote:

We discussed whether elements should be serialized as maps or arrays, and I provided an example map serialization for discussion.  The two serialization formats are equivalent, in that they deserialize to identical logical nodes.  But the discussion highlighted some practical distinctions:

1) Members of a map are pre-indexed by IRI, while an array must be searched member by member to find the element with a specified IRI.  Because looking up element references is a common operation, the first step after receiving an array of elements would be to build an index from IRI to element position in the array.

2) In order to find the captain of a ship with 1000 rooms, you'd need to search each room to look for someone wearing a captain's uniform.  Or in order to find an SBOM element in an array of 1000 elements, you'd need to examine all elements to determine which one(s) are the SBOM type.  That's true whether the 1000 elements are serialized as a map or an array.  BUT, if the 1000 elements were serialized as a map AND a rootElements property existed to list the SBOM IRI(s), no searching is required, the map points directly to the captain.

Conclusion: serialization as a map doesn't help finding the captain if the captain's ID isn't specified along with the map.  But if the captain's ID is specified, map serialization is hugely more efficient than having to search 1000 elements in an array to find that ID.

In any case, here is the JSON-serialized array equivalent of the previous map example, along with listing the 5 default properties at the top level instead of nested in a "defaults" property:

{
  "namespace": "urn:acme.dev:",
  "createdBy": ["identities:fred"],
  "created": "2022-04-05T22:00:00Z",
  "specVersion": "3.0",
  "profiles": ["Core", "Software"],
  "dataLicense": "CC0-1.0",
  "elementValues": [
    {
      "id": "artifacts:gnu-coreutils/v9.1/src/du.c",
      "type": {
        "file": {
          "filePurpose": ["APPLICATION", "SOURCE"]
        }
      }
    },
    {
      "id": "artifacts:gnu-coreutils/v9.1/src/echo.c",
      "type": {
        "file": {
          "filePurpose": ["APPLICATION", "SOURCE"]
        }
      }
    },
    {
      "id": "artifacts:gnu-coreutils/v9.1",
      "type": {
        "package": {
          "packagePurpose": ["APPLICATION", "SOURCE"],
          "downloadLocation": "http://mirror.rit.edu/gnu/coreutils/coreutils-9.1.tar.gz",
          "homePage": "https://www.gnu.org/software/coreutils/"
        }
      },
      "name": "GNU Coreutils"
    },
    {
      "id": "relationships:gnu-coreutils/v9.1",
      "type": {
        "relationship": {
          "relationshipType": "CONTAINS",
          "from": "urn:acme.dev:artifacts:gnu-coreutils/v9.1",
          "to": [
            "artifacts:gnu-coreutils/v9.1/src/du.c",
            "artifacts:gnu-coreutils/v9.1/src/echo.c"
          ]
        }
      }
    },
    {
      "id": "identities:fred",
      "type": {
        "actor": {}
      },
      "identifiedBy": [{"email": "fred@..."}]
    },
    {
      "id": "sboms:gnu-coreutils/v9.1",
      "type": {
        "sbom": {
          "elements": [
            "artifacts:gnu-coreutils/v9.1/src/du.c",
            "artifacts:gnu-coreutils/v9.1/src/echo.c",
            "artifacts:gnu-coreutils/v9.1",
            "relationships:gnu-coreutils/v9.1",
            "identities:fred"
          ]
        }
      }
    }
  ]
}

Regards,
David

--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Join Spdx-tech@lists.spdx.org to automatically receive all group messages.