Add comprehensive codec management functionality

[sumanjeet0012]

Summary

closes #23

This PR introduces codec management functionality to py-multicodec simillar to go-multicodec, bringing feature parity with the Go library.

File Mapping: Go → Python

Go File	Python File	Purpose
code.go	multicodec/code.py	Core Code type with Set(), String(), Tag() methods
code_table.go	multicodec/code_table.py	Named constants (SHA2_256, DAG_CBOR, etc.)
gen.go	tools/gen_code_table.py	Generator script for code_table.py

Changes

New Module: multicodec/code.py (mirrors code.go)

Code Type:

from multicodec import Code

# Create from integer
code = Code(0x12)
print(str(code))  # "sha2-256"
print(int(code))  # 18

# Create from string (mirrors Go's Set() method)
code = Code.from_string("sha2-256")
code = Code.from_string("0x12")  # hex also works

# Set method (mirrors Go exactly)
code = Code(0)
code.set("sha2-256")

# Tag method
print(code.tag())  # "multihash"

# Comparison operators
code1 = Code(0x12)
code2 = Code(0x13)
print(code1 < code2)  # True
print(code1 == 0x12)  # True

Reserved Range Constants:

from multicodec import RESERVED_START, RESERVED_END, is_reserved

print(RESERVED_START)  # 0x300000
print(RESERVED_END)    # 0x3FFFFF
print(is_reserved(0x300001))  # True

KnownCodes Function:

from multicodec import known_codes

codes = known_codes()  # List of all Code objects
print(len(codes))  # 460

New Module: multicodec/code_table.py (mirrors code_table.go)

460 Named Constants:

from multicodec.code_table import SHA2_256, IDENTITY, DAG_CBOR, IP4, TCP

# Use like Go's multicodec.Sha2_256
code = SHA2_256
print(str(code))  # "sha2-256"
print(int(code))  # 18

# Works with Code operations
from multicodec import Code
print(isinstance(SHA2_256, Code))  # True
print(SHA2_256 == Code(0x12))      # True

New Script: tools/gen_code_table.py (mirrors gen.go)

Generates code_table.py from constants.py:

python tools/gen_code_table.py
# Generated multicodec/code_table.py
# Total constants: 460

Usage Comparison: Go vs Python

Go:

import "github.com/multiformats/go-multicodec"

code := multicodec.Sha2_256
fmt.Println(code.String())  // "sha2-256"

var c multicodec.Code
c.Set("sha2-256")

codes := multicodec.KnownCodes()

Python (now):

from multicodec import Code, known_codes
from multicodec.code_table import SHA2_256

code = SHA2_256
print(str(code))  # "sha2-256"

c = Code.from_string("sha2-256")

codes = known_codes()

Backward Compatibility

This PR maintains full backward compatibility:

• All existing functions (add_prefix, remove_prefix, get_codec, etc.) work unchanged
• New functionality is additive only
• No breaking changes to the public API

[acul71]

Hi @sumanjeet0012 thanks, for this PR.
Please add a newsframent file related to the issue and better not to have eval in python code could become a security issue, why bother later ?, better fix it now.

1. Summary of Changes

This PR introduces comprehensive codec management functionality to py-multicodec, bringing feature parity with the Go implementation (go-multicodec). The changes are additive only and maintain full backward compatibility.

Issues Addressed

• Issue Missing codec management functionality #23: Missing codec management functionality
  • Problem: The current implementation lacked type-safe codec handling, named constants, reserved range support, codec enumeration, and string parsing capabilities that exist in go-multicodec.
  • Solution: This PR implements a Pythonic Code type with all the requested features.

Files Changed

New Files:

• multicodec/code.py (467 lines) - Core Code type implementation
• multicodec/code_table.py (955 lines) - Generated file with 460 named constants
• tools/gen_code_table.py (244 lines) - Generator script for code_table.py
• tests/test_code.py (305 lines) - Comprehensive test suite

Modified Files:

• multicodec/__init__.py - Exports new Code type and related functions

Breaking Changes

✅ None - All changes are additive. Existing functions (add_prefix, remove_prefix, get_codec, etc.) work unchanged.

---

2. Branch Sync Status and Merge Conflicts

Branch Sync Status

• Status: ✅ In sync with origin/master
• Details: Branch is 0 commits behind and 1 commit ahead of origin/master. The single commit ahead contains all the PR changes.

Merge Conflict Analysis

• Conflicts Detected: ✅ No conflicts - PR can be merged cleanly
• Details: Test merge completed successfully with no conflicts detected. The PR branch can be merged into origin/master without any manual resolution.

---

3. Strengths

1. Excellent Feature Parity: The implementation closely mirrors go-multicodec's architecture while following Python conventions, making it easy for developers familiar with the Go version to use.

2. Comprehensive Test Coverage: The test suite (test_code.py) includes 305 lines of tests covering:

   • Code creation from integers and strings
   • String parsing (name, hex, decimal)
   • Reserved range handling
   • Comparison operators
   • Hashability
   • Tag lookup
   • Named constants
   • Edge cases

3. Type Safety: The Code class provides type-safe codec handling with proper type hints throughout, improving IDE support and catching errors at development time.

4. Backward Compatibility: All existing functionality remains unchanged. The new Code type is additive and doesn't interfere with existing functions.

5. Well-Documented:

   • Clear docstrings for all public methods
   • Usage examples in docstrings
   • Comprehensive module-level documentation
   • Generator script includes clear instructions

6. Efficient Implementation:

   • Uses __slots__ for memory efficiency
   • Caches known_codes() result
   • Proper use of __index__ for integer contexts
   • Efficient comparison operators

7. Code Generation: The gen_code_table.py script properly generates the constants file from constants.py, ensuring consistency and maintainability.

8. Tag Categorization: The tag() method provides comprehensive categorization of codecs (multihash, multiaddr, IPLD, etc.), matching the Go implementation.

---

4. Issues Found

Critical

None - No critical issues found.

Major

Missing Newsfragment

• File: newsfragments/
• Issue: CRITICAL BLOCKER - Missing newsfragment file for issue Missing codec management functionality #23
• Impact: PR cannot be approved without a newsfragment file
• Suggestion:
  • Create newsfragments/23.feature.rst with content describing the new codec management functionality from a user perspective
  • The file must end with a newline character
  • Example content: "Added Code type and named constants for type-safe multicodec handling, matching go-multicodec functionality."

Generator Script Uses eval()

• File: tools/gen_code_table.py
• Line(s): 28
• Issue: The load_codecs_from_file() function uses eval() to parse the CODECS dictionary, which could be a security risk if the file is compromised
• Suggestion: Consider using ast.literal_eval() instead of eval() for safer parsing, or use a proper JSON/YAML parser if the format allows. However, since this is a code generation script run during development, the risk is low. This is acceptable but worth noting.

Minor

Inconsistent Format String in Error Message

• File: multicodec/code.py
• Line(s): 84
• Issue: Error message uses single quotes inside f-string: f'unknown multicodec: "{text}"'
• Suggestion: Use double quotes consistently: f'unknown multicodec: "{text}"' or f"unknown multicodec: '{text}'" for better consistency

Missing Type Annotation in Generator

• File: tools/gen_code_table.py
• Line(s): 15
• Issue: load_codecs_from_file() return type is not explicitly annotated
• Suggestion: Add return type annotation: def load_codecs_from_file() -> dict:

Potential Issue with __repr__ Format

• File: multicodec/code.py
• Line(s): 116-118
• Issue: The __repr__ method returns Code("sha2-256", 0x12) but the constructor only accepts an integer, not a string
• Suggestion: This is actually fine for display purposes, but consider documenting that repr() output is not meant to be eval()-able, or adjust the format to match the constructor signature more closely (e.g., Code(0x12) # sha2-256)

---

5. Security Review

Security Assessment

✅ No security vulnerabilities identified.

Analysis:

1. Input Validation: The Code class properly validates input:

   • Type checking for integer values
   • Non-negative value validation
   • String parsing with proper error handling

2. No External Input Risks: The code doesn't process untrusted external input in a way that could lead to injection attacks.

3. Safe String Operations: All string operations are safe and don't involve shell execution or file system access.

4. Generator Script: While eval() is used in the generator script, it's only executed during development and operates on trusted source files. The risk is minimal, but using ast.literal_eval() would be safer.

Recommendations:

• Consider replacing eval() with ast.literal_eval() in gen_code_table.py for defense in depth
• No other security concerns identified

---

6. Documentation and Examples

Documentation Quality

✅ Excellent - The PR includes comprehensive documentation:

1. Module Documentation:

   • Clear module-level docstrings in code.py
   • Explains purpose and usage

2. Method Documentation:

   • All public methods have docstrings
   • Usage examples in class docstring
   • Parameter and return type documentation

3. Code Generation Documentation:

   • Generator script includes usage instructions
   • Generated file includes regeneration instructions

Missing Documentation

⚠️ Minor gaps:

1. API Reference: The new Code type and functions should be added to the Sphinx documentation (docs/api_reference.rst or similar). However, since sphinx-apidoc is used, this may be automatically generated.

2. README Updates: Consider adding a section to README.rst showing usage examples of the new Code type and named constants.

3. Migration Guide: While not strictly necessary (no breaking changes), a brief note about the new features in the changelog would be helpful.

Examples

The PR includes good examples in:

• Class docstring showing basic usage
• PR description with Go vs Python comparison
• Test file demonstrating various use cases

Recommendation: Add a usage example to the README showing how to use the new Code type alongside existing functions.

---

7. Newsfragment Requirement

⚠️ CRITICAL BLOCKER

Issue Reference

• ✅ Issue exists: Issue Missing codec management functionality #23 is properly referenced in the PR description ("closes Missing codec management functionality #23")
• ❌ Newsfragment missing: No newsfragment file found for issue Missing codec management functionality #23

Required Action

MANDATORY: Create newsfragments/23.feature.rst with the following:

Filename: 23.feature.rst
Type: .feature (new functionality)
Content: A user-facing description of the change, for example:

Added Code type and named constants for type-safe multicodec handling, bringing feature parity with go-multicodec.

Important:

• The file must end with a newline character (\n)
• Content should focus on user impact, not implementation details
• Must use the issue number (23) in the filename

Status: ❌ BLOCKER - PR approval is blocked until this newsfragment is added.

---

8. Tests and Validation

Test Execution Results

✅ All tests passed

Summary:

• Total tests: 503
• Passed: 503
• Failed: 0
• Skipped: 0
• Errored: 0
• Execution time: 0.19s

Test Coverage:

• New tests in test_code.py cover:

  • Code creation (int, string, hex, decimal)
  • Reserved range handling
  • Comparison operators
  • Hashability
  • Tag lookup
  • Named constants
  • Edge cases and error conditions

• Existing tests in test_multicodec.py still pass, confirming backward compatibility

Linting Results

✅ All linting checks passed

Output Summary:

• ✅ check yaml - Passed
• ✅ check toml - Passed
• ✅ fix end of files - Passed
• ✅ trim trailing whitespace - Passed
• ✅ pyupgrade - Passed
• ✅ ruff (legacy alias) - Passed
• ✅ ruff format - Passed
• ✅ run mypy with all dev dependencies present - Passed

No errors or warnings reported.

Type Checking Results

✅ All type checks passed

Output Summary:

• ✅ run mypy with all dev dependencies present - Passed

No type errors or warnings reported.

Documentation Build Results

✅ Documentation build succeeded

Output Summary:

• ✅ Sphinx build completed successfully
• ✅ All 9 source files processed
• ✅ No errors or warnings
• ✅ HTML documentation generated successfully

Build details:

• Environment updated: 9 files added
• All pages generated successfully
• Search index created
• Object inventory dumped

---

9. Recommendations for Improvement

High Priority

1. Add Newsfragment (BLOCKER):

   • Create newsfragments/23.feature.rst as described in section 7
   • This is required for PR approval

2. Replace eval() in Generator:

   • Consider using ast.literal_eval() instead of eval() in tools/gen_code_table.py for improved security

Medium Priority

3. Add README Examples:

   • Add a section to README.rst demonstrating the new Code type usage
   • Show integration with existing functions

4. Improve __repr__ Format:

   • Consider making repr() output match the constructor signature more closely
   • Or document that repr() output is display-only

5. Add Type Annotations:

   • Add return type annotation to load_codecs_from_file() in generator script

Low Priority

6. Consistent Quote Usage:

   • Use consistent quote style in error messages

7. Documentation Updates:

   • Consider adding migration notes or feature highlights in changelog
   • Ensure API reference documentation includes new types

---

10. Questions for the Author

1. Integration with Existing Functions: Should the existing functions (add_prefix, get_codec, etc.) be updated to accept Code objects in addition to strings, or is the current string-only interface intentional for backward compatibility?

2. Code Table Generation: Should gen_code_table.py be run automatically as part of the build process, or is manual execution expected when constants are updated?

3. Reserved Range Usage: Are there any plans to add validation or special handling for reserved range codes in the existing prefix functions?

4. Performance Considerations: The known_codes() function caches results. Is there a use case where the cache should be invalidated (e.g., if constants are updated at runtime)?

5. Tag Implementation: The _get_tag() function uses hardcoded lists. Is this intended to match the Go implementation exactly, or would a more data-driven approach be preferred?

---

11. Overall Assessment

Quality Rating: Excellent

The PR is well-implemented, thoroughly tested, and maintains backward compatibility. The code quality is high, with proper type hints, documentation, and test coverage. The only blocker is the missing newsfragment file.

Security Impact: None

No security vulnerabilities identified. The use of eval() in the generator script is acceptable for a development tool but could be improved.

Merge Readiness: Needs fixes

Status: ⚠️ BLOCKER - Missing newsfragment file

Required before merge:

1. ✅ Code quality: Excellent
2. ✅ Tests: All passing
3. ✅ Linting: All checks passed
4. ✅ Type checking: All checks passed
5. ✅ Documentation build: Successful
6. ❌ Newsfragment: MISSING (BLOCKER)

After newsfragment is added:

• ✅ Ready to merge
• ✅ No merge conflicts
• ✅ All checks passing
• ✅ Backward compatible

Confidence: High

The implementation is solid, well-tested, and follows Python best practices. Once the newsfragment is added, this PR is ready for approval and merge.

---

Summary

This PR successfully implements comprehensive codec management functionality, bringing py-multicodec to feature parity with go-multicodec. The code is well-written, thoroughly tested, and maintains full backward compatibility.

The only blocker is the missing newsfragment file for issue #23. Once this is added, the PR is ready for merge.

Recommendation: ✅ Approve after adding newsfragment

[sumanjeet0012]

@acul71 I have replaced the eval() with ast.literal_eval(), and added the newsfragment.

[acul71]

@acul71 I have replaced the eval() with ast.literal_eval(), and added the newsfragment.

Well done, I've fixed some minor issues, I'm merging this.