Skip to content

Conversation

@konard
Copy link
Member

@konard konard commented Sep 13, 2025

Summary

This PR addresses issue #8 by updating the CloneAllOrganizationRepositoriesByHTTPS.sh script to handle organizations with more than 100 repositories through proper API pagination.

Changes Made

  • Added pagination support: Implemented fetch_all_repositories() function that automatically handles GitHub API pagination
  • Increased reliability: Proper error handling for invalid JSON responses and API failures
  • Maintained compatibility: The script works exactly the same way for users, no breaking changes
  • Improved efficiency: Uses per_page=100 instead of per_page=200 and fetches all pages as needed
  • Better diagnostics: All status messages go to stderr, avoiding JSON corruption

Technical Details

The original script used a single API call with per_page=200, which would miss repositories when an organization has more than 200 repos. The GitHub API has a maximum limit and uses pagination for large result sets.

The new implementation:

  1. Fetches repositories page by page using per_page=100
  2. Continues until all pages are retrieved (when a page returns fewer than 100 repos)
  3. Combines all pages into a single JSON array using temporary files
  4. Provides progress feedback via stderr
  5. Cleans up temporary files automatically

Testing

  • ✅ Tested with linksplatform organization (85 repositories)
  • ✅ Verified pagination works with smaller page sizes (10 per page)
  • ✅ Confirmed all repository clone URLs are correctly extracted
  • ✅ Validated that the script maintains backward compatibility
  • ✅ Tested error handling for invalid responses

The script is now ready to handle organizations with any number of repositories, resolving the issue for when LinksPlatform grows beyond 100 repositories.

Fixes #8

🤖 Generated with Claude Code

Adding CLAUDE.md with task information for AI processing.
This file will be removed when the task is complete.

Issue: #8
@konard konard self-assigned this Sep 13, 2025
…nation for organizations with more than 100 repositories

- Add fetch_all_repositories function with pagination support
- Use per_page=100 and automatic page iteration to handle any number of repositories
- Maintain backward compatibility with existing functionality
- Add proper error handling for invalid JSON responses
- Route diagnostic messages to stderr to avoid JSON corruption
- Clean up temporary files after processing

Fixes #8

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@konard konard changed the title [WIP] Update CloneAllOrganizationRepositoriesByHTTPS script when the number of LinksPlatform repositories become bigger than 100 Update CloneAllOrganizationRepositoriesByHTTPS script to support pagination for organizations with more than 100 repositories Sep 13, 2025
@konard konard marked this pull request as ready for review September 13, 2025 01:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update CloneAllOrganizationRepositoriesByHTTPS script when the number of LinksPlatform repositories become bigger than 100

2 participants