Skip to content

model_card_toolkit.model_card

Model Card Data Class.

The Model Card (MC) is the document designed for transparent reporting of AI model provenance, usage, and ethics-informed evaluation. The model card can be presented by different formats (e.g. HTML, PDF, Markdown). The properties of the Model Card (MC) are defined by a json schema. The ModelCard class in the ModelCardsToolkit serves as an API to read and write MC properties by the users.

Owner Objects

@dataclasses.dataclass
class Owner(BaseModelCardField)

The information about owners of a model.

Attributes:

  • name - The name of the model owner.
  • contact - The contact information for the model owner or owners. These could be individual email addresses, a team mailing list expressly, or a monitored feedback form.
  • role - The role of the person e.g. owner, developer or auditor.

Version Objects

@dataclasses.dataclass
class Version(BaseModelCardField)

The information about verions of a model.

If there are multiple versions of the model, or there may be in the future, it’s useful for your audience to know which version of the model is discussed in the Model Card. If there are previous versions of this model, briefly describe how this version is different. If no more than one version of the model will be released, this field may be omitted.

Attributes:

  • name - The name of the version.
  • date - The date this version was released.
  • diff - The changes from the previous version.

License Objects

@dataclasses.dataclass
class License(BaseModelCardField)

The license information for a model.

Attributes:

  • identifier - A standard SPDX license identifier (https://spdx.org/licenses/), or "proprietary" for an unlicensed Module.
  • custom_text - The text of a custom license.

Reference Objects

@dataclasses.dataclass
class Reference(BaseModelCardField)

Reference for a model.

Attributes:

  • reference - A reference to a resource.

RegulatoryRequirement Objects

@dataclasses.dataclass
class RegulatoryRequirement(BaseModelCardField)

Regulatory requirements that the model should comply to

Attributes:

  • regulation - Name of the regulation.

Citation Objects

@dataclasses.dataclass
class Citation(BaseModelCardField)

A citation for a model.

Attributes:

  • style - The citation style, such as MLA, APA, Chicago, or IEEE.
  • citation - the citation.

ModelDetails Objects

@dataclasses.dataclass
class ModelDetails(BaseModelCardField)

This section provides a general, high-level description of the model.

Attributes:

  • name - The name of the model.
  • overview - A description of the model card.
  • documentation - A more thorough description of the model and its usage.
  • owners - The individuals or teams who own the model.
  • version - The version of the model.
  • licenses - The license information for the model. If the model is licensed for use by others, include the license type. If the model is not licensed for future use, you may state that here as well.
  • references - Provide any additional links the reader may need. You can link to foundational research, technical documentation, or other materials that may be useful to your audience.
  • citations - How should the model be cited? If the model is based on published academic research, cite the research.
  • path - The path where the model is stored.
  • regulatory_requirements - Provide any regulatory requirements that the model should comply to.

Graphic Objects

@dataclasses.dataclass
class Graphic(BaseModelCardField)

A named inline plot.

Attributes:

  • name - The name of the graphic.
  • image - The image string encoded as a base64 string.

GraphicsCollection Objects

@dataclasses.dataclass
class GraphicsCollection(BaseModelCardField)

A collection of graphics.

Each graphic in the collection field has both a name and an image. For instance, you might want to display a graph showing the number of examples belonging to each class in your training dataset:

Then, provide a description of the graph:


model_card.model_parameters.data.train.graphics.collection = [
  {'name': 'Training Set Size', 'image': training_set_size_barchart},
]


model_card.model_parameters.data.train.graphics.description = (
  'This graph displays the number of examples belonging to each class ',
  'in the training dataset. ')

Attributes:

  • description - The description of graphics.
  • collection - A collection of graphics.

SensitiveData Objects

@dataclasses.dataclass
class SensitiveData(BaseModelCardField)

Sensitive data, such as PII (personally-identifiable information).

Attributes:

  • sensitive_data - A description of any sensitive data that may be present in a dataset. Be sure to note PII information such as names, addresses, phone numbers, etc. Preferably, such info should be scrubbed from a dataset if possible. Note that even non-identifying information, such as zip code, age, race, and gender, can be used to identify individuals when aggregated. Please describe any such fields here.
  • sensitive_data_used - A list of sensitive data used in the deployed model.
  • justification - Justification of the need to use the fields in deployment.

Dataset Objects

@dataclasses.dataclass
class Dataset(BaseModelCardField)

Provide some information about a dataset used to generate a model.

Attributes:

  • name - The name of the dataset.
  • description - The description of dataset.
  • link - A link to the dataset.
  • sensitive - Does this dataset contain human or other sensitive data?
  • graphics - Visualizations of the dataset.

KeyVal Objects

@dataclasses.dataclass
class KeyVal(BaseModelCardField)

A generic key-value pair.

Attributes:

  • key - The key of the key-value pair.
  • value - The value of the key-value pair.

ModelParameters Objects

@dataclasses.dataclass
class ModelParameters(BaseModelCardField)

Parameters for construction of the model.

Attributes:

  • model_architecture - specifies the architecture of your model.
  • data - specifies the datasets used to train and evaluate your model.
  • input_format - describes the data format for inputs to your model.
  • input_format_map - describes the data format for inputs to your model, in key-value format.
  • output_format - describes the data format for outputs from your model.
  • output_format_map - describes the data format for outputs to your model, in key-value format.

Test Objects

@dataclasses.dataclass
class Test(BaseModelCardField)

Information about test that is run against the model.

Attributes:

  • name - The name of the test.
  • description - User-friendly description of the test.
  • threshold - Threshold required to pass the test.
  • result - Result returned by the test.
  • passed - Whether the model result satisfies the given threshold.
  • graphics - A collection of visualizations associated with the test.

PerformanceMetric Objects

@dataclasses.dataclass
class PerformanceMetric(BaseModelCardField)

The details of the performance metric.

Attributes:

  • type - What performance metric are you reporting on?
  • value - What is the value of this performance metric?
  • slice - What slice of your data was this metric computed on?
  • description - User-friendly description of the performance metric.
  • graphics - A collection of visualizations associated with the metric.
  • tests - A collection of tests associated with the metric.

QuantitativeAnalysis Objects

@dataclasses.dataclass
class QuantitativeAnalysis(BaseModelCardField)

The quantitative analysis of a model.

Identify relevant performance metrics and display values. Let’s say you’re interested in displaying the accuracy and false positive rate (FPR) of a cat vs. dog classification model. Assuming you have already computed both metrics, both overall and per-class, you can specify metrics like so:

model_card.quantitative_analysis.performance_metrics = [
  {'type': 'accuracy', 'value': computed_accuracy},
  {'type': 'accuracy', 'value': cat_accuracy, 'slice': 'cat'},
  {'type': 'accuracy', 'value': dog_accuracy, 'slice': 'dog'},
  {'type': 'fpr', 'value': computed_fpr},
  {'type': 'fpr', 'value': cat_fpr, 'slice': 'cat'},
  {'type': 'fpr', 'value': dog_fpr, 'slice': 'dog'},
]

Attributes:

  • performance_metrics - The performance metrics being reported.
  • graphics - A collection of visualizations of model performance.

ExplainabilityReport Objects

@dataclasses.dataclass
class ExplainabilityReport(BaseModelCardField)

Model explainability report.

Details of how the model works such as feature importance, decision trees or LIME or shapely analysis.

Attributes:

  • type - What explainability method are you conducting?
  • slice - What slice of your data was this analysis conducted on?
  • description - User-friendly description of the explainability metric.
  • graphics - A collection of visualizations related to the explainability method.
  • tests - A collection of tests associated with the explainability method.

ExplainabilityAnalysis Objects

@dataclasses.dataclass
class ExplainabilityAnalysis(BaseModelCardField)

Model explainability.

Analysis to explain how the model works and operates.

Attributes:

  • explainability_reports - The explainability studies undertaken.

FairnessReport Objects

@dataclasses.dataclass
class FairnessReport(BaseModelCardField)

Model fairness report.

Details on fairness checks and analysis.

Attributes:

  • type - What fairness assessment method are you conducting?
  • slice - What slice of your data was this analysis conducted on?
  • segment - What segment of the dataset which the fairness report is assessing?
  • description - User-friendly description of the fairness metric.
  • graphics - A collection of visualizations related to the fairness method.
  • tests - A collection of tests associated with the fairness method.

FairnessAnalysis Objects

@dataclasses.dataclass
class FairnessAnalysis(BaseModelCardField)

Model fairness.

Attributes:

  • fairness_reports - The fairness studies undertaken.

User Objects

@dataclasses.dataclass
class User(BaseModelCardField)

A type of user for a model.

Attributes:

  • description - A description of a user.

UseCase Objects

@dataclasses.dataclass
class UseCase(BaseModelCardField)

A type of use case for a model.

Attributes:

  • description - A description of a use case.

Limitation Objects

@dataclasses.dataclass
class Limitation(BaseModelCardField)

A limitation a model.

Attributes:

  • description - A description of the limitation.

Tradeoff Objects

@dataclasses.dataclass
class Tradeoff(BaseModelCardField)

A tradeoff for a model.

Attributes:

  • description - A description of the tradeoff.

Risk Objects

@dataclasses.dataclass
class Risk(BaseModelCardField)

Information about risks involved when using the model.

Attributes:

  • name - The name of the risk.
  • mitigation_strategy - A mitigation strategy that you've implemented, or one that you suggest to users.

FairnessAssessment Objects

@dataclasses.dataclass
class FairnessAssessment(BaseModelCardField)

Information about the benefits and harms of the model.

Attributes:

  • group_at_risk - The groups or individuals at risk of being systematically disadvantaged by the model.
  • benefits - Expected benefits to the identified groups.
  • harms - Expected harms to the identified groups.
  • mitigation_strategy - With respect to the benefits and harms outlined, please describe any mitigation strategy implemented.

Considerations Objects

@dataclasses.dataclass
class Considerations(BaseModelCardField)

Considerations related to model construction, training, and application.

The considerations section includes qualitative information about your model, including some analysis of its risks and limitations. As such, this section usually requires careful consideration, and conversations with many relevant stakeholders, including other model developers, dataset producers, and downstream users likely to interact with your model, or be affected by its outputs.

Attributes:

  • users - Who are the intended users of the model? This may include researchers, developers, and/or clients. You might also include information about the downstream users you expect to interact with your model.
  • use_cases - What are the intended use cases of the model? What use cases are out-of-scope?
  • limitations - What are the known limitations of the model? This may include technical limitations, or conditions that may degrade model performance.
  • tradeoffs - What are the known accuracy/performance tradeoffs for the model?
  • ethical_considerations - What are the ethical risks involved in application of this model? For each risk, you may also provide a mitigation strategy that you've implemented, or one that you suggest to users.
  • fairness_assessment - How does the model affect groups at risk of being systematically disadvantaged? What are the harms and benefits to the various affected groups?

ModelCard Objects

@dataclasses.dataclass
class ModelCard(BaseModelCardField)

Fields used to generate the Model Card.

Attributes:

  • model_details - Descriptive metadata for the model.
  • model_parameters - Technical metadata for the model.
  • quantitative_analysis - Quantitative analysis of model performance.
  • considerations - Any considerations related to model construction, training, and application.
  • explainability_analysis - Explainability analysis being reported.
  • fairness_analysis - Fairness analysis being reported.

to_json

def to_json() -> str

Write ModelCard to JSON.

from_json

def from_json(json_dict: Dict[str, Any]) -> None

Reads ModelCard from JSON.

If ModelCard fields have already been set, this function will overwrite any existing values.

Arguments:

  • json_dict - A JSON dict from which to populate fields in the model card schema.

Raises:

  • JSONDecodeError - If json_dict is not a valid JSON string.
  • ValidationError - If json_dict does not follow the model card JSON schema.
  • ValueError - If json_dict contains a value not in the class or schema definition.

get_test_results

def get_test_results() -> Dict[str, Counter]

Return overall number of tests passed and failed across performance metrics, explainability reports, fairness reports.

Returns:

Counter of test cases passed and failed for performance tests, explainability tests, and fairness tests in a dictionary.