SPDX Legal call this Thursday
https://web.archive.org/web/20060319014854/http://info.borland.com/devsupport/interbase/opensource/IPL.html
Firebird is still under this license, still used in Fedora.
http://www.sendmail.com/pdfs/open_source/sendmail_license.pdf
ANSWER: That is the correct sendmail license. We have updated our link.
recommendation at the end of the email.
ANSWER: Updated the Crystal Stacker entry in the Fedora license list to add the missing disclaimer text. License now matches license in download. I do not believe there is a different source license vs binary license here.
· should we accommodate this difference somehow? If so, due to this already being on the license list, this seems like it should be a priority to resolve for v2.1 release
opensource@...
3) License matching templates/markup:
We have a task to add markup to some of the standard headers and have also had input to add/edit markup on existing licenses. As a result of the latter, it has been raised that perhaps the markup could be improved. Before adding more markup (to standard headers, license text or both), it seemed prudent to start a discussion as to whether the existing markup is effective. Please ponder the following questions:
a) have you used the existing markup for matching purposes?
i) if no, why not?
ii) if yes, has it been helpful/effective? Could it be improved, and if so, how? (this will likely involve putting forward a proposal for review)
Please also add thoughts (preferably in a new section or with your initials if added to others) here: http://wiki.spdx.org/view/Legal_Team/Templatizing
I will share a few points from my experience in templatization. I currently use a different templatization syntax that predates SPDX, but the principle of using regular expressions embedded within the license text is similar.
The major barrier to me adopting the SPDX templates is insufficient templatization within the existing licenses. The SPDX templates currently encode what I perceive to be the ‘official’ variations, i.e. organization name, person name, product name etc. However, real-world licenses contain may minor variations that may be inconsequential from a legal perspective, but nonetheless do not warrant separating out as separate licenses. Here is an example from the GPL-3.0 notice where it is common to see two variations in one of the sentences:
distributed in the hope that it will be useful
distributed in the hope that they will be useful
The example above is fairly uncontroversial, I would hope. However, there are plenty of other examples that border on having a legal impact. For example, in these two BSD-2-Clause variations it is necessary to consider whether the additional word constitutes an acceptable minor variation or warrants a different classification altogether:
Redistributions of source code must retain the above copyright notice, this list of…
Redistributions of source code must retain the above copyright notice unmodified, this list of…
It is the grey cases like these that make expanding the use of templating difficult. Inevitably it leads to having to make some judgements about the impact of a particular word or phrase on the legal interpretation, something that I am aware SPDX tries to avoid.
Whether it is worth templating all the cases like these primarily depends on the goals of the SPDX templates. If they are for human use to see what official variations are permitted, then they are not necessary. On the other hand, if they are to be used by automated license scanning tools, then covering these cases is essential in order to have a tool that works effectively on real-world code. So I think an important point is to gain clarity on the purpose of the templates.
In terms of the current application of the templates, I have a technical concern over the use of unbounded regular expressions, for example:
<<var;name=copyrightHolderAsIs;original=THE COPYRIGHT HOLDERS AND CONTRIBUTORS;match=.+>>
This is unbounded because it will match any number of characters for the copyrightHolderAsIs field. The practical consequence of this is that regular expression matching can explode in terms of time. I don’t have a concrete example to hand, but my own experience with using the same unbounded regular expressions on real-world licenses is that I have seen it take minutes just to process one regular expression on a single file, and this does not scale well when there are millions of files to process. Clearly, in terms of English language there is no maximum size on the length of a copyright statement. Using an unbounded regular expression is therefore correct in theory but difficult to use in practice. I have had to use size bounded regular expressions in order to have a scanning tool that will complete in a reasonable time. The problem in switching to bounded regular expressions is in deciding on what is an acceptable upper bound on the size, and this can really only be judged by experimentation against real-world licenses, and does then require on-going tweaking as new license variations are discovered.
Neither of these are problems with templatization per-se, and they are more to do with the extent and way in which they are currently applied.
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782
Hi Sam,
Responses inline below:
Gary
Sent: Wednesday, September 16, 2015 4:00 AM
To: J Lovejoy; SPDX-legal
Cc: spdx-tech@...
Subject: RE: SPDX Legal call this Thursday
3) License matching templates/markup:
We have a task to add markup to some of the standard headers and have also had input to add/edit markup on existing licenses. As a result of the latter, it has been raised that perhaps the markup could be improved. Before adding more markup (to standard headers, license text or both), it seemed prudent to start a discussion as to whether the existing markup is effective. Please ponder the following questions:
a) have you used the existing markup for matching purposes?
i) if no, why not?
ii) if yes, has it been helpful/effective? Could it be improved, and if so, how? (this will likely involve putting forward a proposal for review)
[Gary] Yes - For the SourceAuditor commercial tools, the markup is used to validate that 2 licenses are equivalent per the matching guidelines. The open source SPDX tools uses the markup is used in a number of ways. The "compareSpdx" and "compareMultipleSpdx" commands use the markup to determine if the licenses are equivalent. There are library methods implemented to compare license text and report if the license text matches any of the SPDX LicenseList.
In all cases above, the markup is used to compare 2 existing known license text. It is NOT used to match license text against a library of possible license matches. In the commercial tool, a separate algorithm implements this functionality and the markup language turned out to be too inefficient for this purpose - at least for the performance requirements of our application.
Note: When we originally discussed the markup language, we debated whether to cover the use case of searching a library of possible license matches and the decision was taken not to support this.
In my opinion, the markup works fine for matching two license texts. If we wanted to support a searching use case, we would need to modify/extend the markup language to enable this to be efficient.
Please also add thoughts (preferably in a new section or with your initials if added to others) here: http://wiki.spdx.org/view/Legal_Team/Templatizing
I will share a few points from my experience in templatization. I currently use a different templatization syntax that predates SPDX, but the principle of using regular expressions embedded within the license text is similar.
The major barrier to me adopting the SPDX templates is insufficient templatization within the existing licenses. The SPDX templates currently encode what I perceive to be the ‘official’ variations, i.e. organization name, person name, product name etc. However, real-world licenses contain may minor variations that may be inconsequential from a legal perspective, but nonetheless do not warrant separating out as separate licenses. Here is an example from the GPL-3.0 notice where it is common to see two variations in one of the sentences:
distributed in the hope that it will be useful
distributed in the hope that they will be useful
The example above is fairly uncontroversial, I would hope. However, there are plenty of other examples that border on having a legal impact. For example, in these two BSD-2-Clause variations it is necessary to consider whether the additional word constitutes an acceptable minor variation or warrants a different classification altogether:
Redistributions of source code must retain the above copyright notice, this list of…
Redistributions of source code must retain the above copyright notice unmodified, this list of…
It is the grey cases like these that make expanding the use of templating difficult. Inevitably it leads to having to make some judgements about the impact of a particular word or phrase on the legal interpretation, something that I am aware SPDX tries to avoid.
Whether it is worth templating all the cases like these primarily depends on the goals of the SPDX templates. If they are for human use to see what official variations are permitted, then they are not necessary. On the other hand, if they are to be used by automated license scanning tools, then covering these cases is essential in order to have a tool that works effectively on real-world code. So I think an important point is to gain clarity on the purpose of the templates.
In terms of the current application of the templates, I have a technical concern over the use of unbounded regular expressions, for example:
<<var;name=copyrightHolderAsIs;original=THE COPYRIGHT HOLDERS AND CONTRIBUTORS;match=.+>>
This is unbounded because it will match any number of characters for the copyrightHolderAsIs field. The practical consequence of this is that regular expression matching can explode in terms of time. I don’t have a concrete example to hand, but my own experience with using the same unbounded regular expressions on real-world licenses is that I have seen it take minutes just to process one regular expression on a single file, and this does not scale well when there are millions of files to process. Clearly, in terms of English language there is no maximum size on the length of a copyright statement. Using an unbounded regular expression is therefore correct in theory but difficult to use in practice. I have had to use size bounded regular expressions in order to have a scanning tool that will complete in a reasonable time. The problem in switching to bounded regular expressions is in deciding on what is an acceptable upper bound on the size, and this can really only be judged by experimentation against real-world licenses, and does then require on-going tweaking as new license variations are discovered.
[Gary] Extremely good point - Performance would be greatly improved if we bounded the RE's. Something which is within the spec, but is not implemented in the templates. I would support implementing bounding all RE's in an upcoming release and changing the spec to encourage bounding. I think we could pick a pretty practical upper bound that would improve the performance.
Also note - this is more of an issue with the "matching against a library of licenses" issue than simply comparing two license texts since the former is more performance sensitive.
Neither of these are problems with templatization per-se, and they are more to do with the extent and way in which they are currently applied.
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782
3) License matching templates/markup:
We have a task to add markup to some of the standard headers and have also
had input to add/edit markup on existing licenses. As a result of the
latter, it has been raised that perhaps the markup could be improved. Before
adding more markup (to standard headers, license text or both), it seemed
prudent to start a discussion as to whether the existing markup is
effective. Please ponder the following questions:
a) have you used the existing markup for matching purposes?Yes and No: ScanCode uses an SPDX-inspired/derived markup, but
instead of reusing the markup directly from the main license texts,
markup is transformed in a simpler {{mustache-like}} syntax added to
copies of these texts used only for detection purpose.
i) if no, why not?Because:
- adding more markup to a reference license text makes this eventually
no longer usable as a reference text and harder to read by humans
- the many variations found in the wild make it hard to put all in a
single template.
- the markup syntax implies eventually an implementation using regular
expressions. ScanCode does not use regex, but inverted indexes and
string alignments.
ii) if yes, has it been helpful/effective? Could it be improved, and if so,I think a simple markup is a very effective way to detect licenses
how? (this will likely involve putting forward a proposal for review)
with minor text variations and still call this an exact match.
It is also a very effective way to indicate variations for humans.
I find it hard personally to mix the human readability and technical
detection concerns in the same file without compromises.
As food for thought, here are some examples of markup as used in ScanCode:
https://github.com/nexB/scancode-toolkit/blob/b37be4de78152fbd3ed54761627c960010ce26a3/src/licensedcode/data/rules/apache-1.1_38.RULE#L17
https://github.com/nexB/scancode-toolkit/blob/b37be4de78152fbd3ed54761627c960010ce26a3/src/licensedcode/data/rules/bzip2-libbzip-1.0.5_1.RULE#L1
The syntax is using double curly braces to enclose variable parts.
There is no regex involved.
Optionally a number can be used after the opening braces to indicate
the number of variable words, defaulting to 5 words.
For instance {{ Copyright (c) 2015 Myco }} would match up to 5 words
and {{ 10 Copyright (c) 2015 Myco inc.}} would match up to 10 words.
I hope this helps even though this is a slightly different take.
--
Cordially
Philippe Ombredanne
3) License matching templates/markup:
We have a task to add markup to some of the standard headers and have
also had input to add/edit markup on existing licenses. As a result
of the latter, it has been raised that perhaps the markup could be
improved. Before adding more markup (to standard headers, license
text or both), it seemed prudent to start a discussion as to whether
the existing markup is effective. Please ponder the following
questions:
a) have you used the existing markup for matching purposes?
i) if no, why not?
Didn't use the markup. At the time haven't found documented the pattern keywords to find inside each license term. For example, one can find [xxxx]-[xxxx], [Owner Organization], <AUTHOR>. However, didn't seemed to be used consistently throughout the list of licenses. That was my opinion at that time, might certainly be proven erroneous.
If help is welcome, I can volunteer for normalizing the data for the complete list (top to bottom) and document the used keywords. However, for our own tooling usage as template I'd replace the text on some specific license terms so that we could later generate new licence docs where only the copyright holder changes.
Such example would be http://spdx.org/licenses/BSD-3-Clause-Attribution.html where: "Universidad de Palermo, Argentina" (http://www.palermo.edu/).'" becomes "<AUTHOR>".
With kind regards,
Nuno
--
http://triplecheck.net
phone: +49 615 146 03187
a) have you used the existing markup for matching purposes?
No.
i) if no, why not?
In a nutshell, partly due to timing of our dev efforts ahead of SPDX templatization rollout, partly due to performance in context of our internal Use Case of scanning every single open source file (not just a handful of applications).
Details:
Black Duck has a corpus of license variants extracted from our Knowledge Base of around half a billion unique open source source/text and binary files.
Several years ago, prior to SPDX license templatization, we went through multiple iterations of grouping license text variants, as well as license name/nickname variants.
Our groupings were based on applying similarity algorithms, followed by human review. Our methodology has been in place for some time prior to the templates / regular expressions subsequently rolled out by SPDX.
Variations we've encountered, such as the street address of an organization, or typos / word substitutions that SPDX license templates might not have covered are some of the reasons we haven't yet gone through the exercise to see which license variants the SPDX templates might not 'match' to the license id's we've grouped them under.
We use our license scanner to scan (and rescan) every single open source file to populate our Knowledgebase of file-level license data. Our scanner uses multiple techniques to discover license references, but does not use regular expressions because of performance concerns.
Because Black Duck does not provide legal advice, consumers of our tools are able review the license text which our tools highlight in order to make a final determination. This fits well with the SPDX concept of separating discovered and concluded information.
Outlook:
SPDX license templates / regexes aren't as useful or efficient as other matching techniques we have, when automatically bulk scanning large codebases to determine ‘LicenseFoundInFile'.
But when a human is in the loop producing a final SPDX Document with Concluded License, SPDX license templates / regexes could be useful to focus legal review on deviations which may or may not be significant.
Date: Wednesday, September 16, 2015 at 7:00 AM
To: J Lovejoy, SPDX-legal
Cc: "spdx-tech@..."
Subject: RE: SPDX Legal call this Thursday
3) License matching templates/markup:
We have a task to add markup to some of the standard headers and have also had input to add/edit markup on existing licenses. As a result of the latter, it has been raised that perhaps the markup could be improved. Before adding more markup (to standard headers, license text or both), it seemed prudent to start a discussion as to whether the existing markup is effective. Please ponder the following questions:
a) have you used the existing markup for matching purposes?
i) if no, why not?
ii) if yes, has it been helpful/effective? Could it be improved, and if so, how? (this will likely involve putting forward a proposal for review)
Please also add thoughts (preferably in a new section or with your initials if added to others) here: http://wiki.spdx.org/view/Legal_Team/Templatizing
I will share a few points from my experience in templatization. I currently use a different templatization syntax that predates SPDX, but the principle of using regular expressions embedded within the license text is similar.
The major barrier to me adopting the SPDX templates is insufficient templatization within the existing licenses. The SPDX templates currently encode what I perceive to be the ‘official’ variations, i.e. organization name, person name, product name etc. However, real-world licenses contain may minor variations that may be inconsequential from a legal perspective, but nonetheless do not warrant separating out as separate licenses. Here is an example from the GPL-3.0 notice where it is common to see two variations in one of the sentences:
distributed in the hope that it will be useful
distributed in the hope that they will be useful
The example above is fairly uncontroversial, I would hope. However, there are plenty of other examples that border on having a legal impact. For example, in these two BSD-2-Clause variations it is necessary to consider whether the additional word constitutes an acceptable minor variation or warrants a different classification altogether:
Redistributions of source code must retain the above copyright notice, this list of…
Redistributions of source code must retain the above copyright notice unmodified, this list of…
It is the grey cases like these that make expanding the use of templating difficult. Inevitably it leads to having to make some judgements about the impact of a particular word or phrase on the legal interpretation, something that I am aware SPDX tries to avoid.
Whether it is worth templating all the cases like these primarily depends on the goals of the SPDX templates. If they are for human use to see what official variations are permitted, then they are not necessary. On the other hand, if they are to be used by automated license scanning tools, then covering these cases is essential in order to have a tool that works effectively on real-world code. So I think an important point is to gain clarity on the purpose of the templates.
In terms of the current application of the templates, I have a technical concern over the use of unbounded regular expressions, for example:
<<var;name=copyrightHolderAsIs;original=THE COPYRIGHT HOLDERS AND CONTRIBUTORS;match=.+>>
This is unbounded because it will match any number of characters for the copyrightHolderAsIs field. The practical consequence of this is that regular expression matching can explode in terms of time. I don’t have a concrete example to hand, but my own experience with using the same unbounded regular expressions on real-world licenses is that I have seen it take minutes just to process one regular expression on a single file, and this does not scale well when there are millions of files to process. Clearly, in terms of English language there is no maximum size on the length of a copyright statement. Using an unbounded regular expression is therefore correct in theory but difficult to use in practice. I have had to use size bounded regular expressions in order to have a scanning tool that will complete in a reasonable time. The problem in switching to bounded regular expressions is in deciding on what is an acceptable upper bound on the size, and this can really only be judged by experimentation against real-world licenses, and does then require on-going tweaking as new license variations are discovered.
Neither of these are problems with templatization per-se, and they are more to do with the extent and way in which they are currently applied.
-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782
This is really great information. We did not have time to discuss on the call today, as the focus was on items for the v2.2 release of the license list, but I’d like to capture this info in the wiki page, so it doesn’t get lost, etc.
Could you each add a section with your comments to this page? http://wiki.spdx.org/view/Legal_Team/Templatizing
I created a new section for just this purpose. If you are unable to edit the wiki page, let me know.
Thanks,
Jilayne
SPDX Legal Team co-lead
opensource@...
On Sep 17, 2015, at 3:22 AM, Nuno Brito <nuno.brito@...> wrote:3) License matching templates/markup:
We have a task to add markup to some of the standard headers and have
also had input to add/edit markup on existing licenses. As a result
of the latter, it has been raised that perhaps the markup could be
improved. Before adding more markup (to standard headers, license
text or both), it seemed prudent to start a discussion as to whether
the existing markup is effective. Please ponder the following
questions:
a) have you used the existing markup for matching purposes?
i) if no, why not?
Didn't use the markup. At the time haven't found documented the pattern keywords to find inside each license term. For example, one can find [xxxx]-[xxxx], [Owner Organization], <AUTHOR>. However, didn't seemed to be used consistently throughout the list of licenses. That was my opinion at that time, might certainly be proven erroneous.
If help is welcome, I can volunteer for normalizing the data for the complete list (top to bottom) and document the used keywords. However, for our own tooling usage as template I'd replace the text on some specific license terms so that we could later generate new licence docs where only the copyright holder changes.
Such example would be http://spdx.org/licenses/BSD-3-Clause-Attribution.html where: "Universidad de Palermo, Argentina" (http://www.palermo.edu/).'" becomes "<AUTHOR>".
With kind regards,
Nuno
--
http://triplecheck.net
phone: +49 615 146 03187
_______________________________________________
Spdx-legal mailing list
Spdx-legal@...
https://lists.spdx.org/mailman/listinfo/spdx-legal