chore(pr-data): handle cross-repo redirection

[lukas2510]

This fixes (3) from #1864

The Problem was:
Cross-repo redirection
Example of a confusing row here

• Pattern: Initial PR (original repo) in PR Link; final accepted fix in a different repo only in Notes.
• Proposed rule: PR Link (or Commit Link) should point to the authoritative PR/commit where the fix landed (target repo). Initial exploratory/redirected PR kept in Notes.
• Benefit: Downstream automation can fetch the diff from the correct repository without extra heuristics.

[lukas2510]

@winglam @darko-marinov
The pipeline fails because the URL in the Project URL column does not match the URL in the PR Link column.

This is because they fixed the flaky test by fixing the testing framework wcm-io/io.wcm.testing.aem-mock#52. The repo with the flaky test uses the testing framework and therefore the flaky test gets fixed. I think that is the reason why the PR with the fix was in the Notes column because with this you can bypass the format checker.

This makes the fix from (3) from #1864 a lot more complicated. What would be the correct way to state this in the datasheet? Leave the PR Link with the fix in the Notes column or adjust the format checker?

[darko-marinov]

What study are you trying to do with test fixes? How would this case of a fix being in another repo fit into your study? Would you, in general, for each test need to clone the project repo and then get the source of all its libraries? A pragmatic solution may be to skip these cases because they're not all that common.

On a more general note, what percentage of tests/PRs marked Accepted don't actually have status "Merged" on GitHub? In the limit, could you ignore all such cases if only a tiny percentage? Note that Accepted brings more "points" to the students, so they have an incentive to get tests marked as Accepted. :)

[lukas2510]

I want to analyze the origin of the flakiness. I do a rough classification in either "the system under test is the origin" or "the test is the origin". My approach to doing this classification is by looking at the fixes. If only the test got fixed, then the origin was probably the test, and if only the system under test got changed, then this is probably the origin. In cases where both have been touched, I have to decide manually. I wanted to automate the classification as well as possible. Therefore, I try to find the best evidence, which is either the concrete commit of the fix or the PR with multiple commits.

In the process of finding the best evidence, I came across the things in #1864. Therefore, I decided to open the issue.

For me it would have been nice to have a column "fix commit" because there I have the concrete changes. In a PR, there is sometimes refactoring done as well or other kinds of changes. This makes my automation more difficult. That is how I came up with the idea of the column in the first place. But I decided that it is too big of an effort to go through the PRs manually and find the fix commit. Therefore, I now look at the PR link column and the Notes column and search in those for the best evidence, which is a commit link if there is one, and if not, I take the PR link. And then I work with those links in my automatic classification.

The inability to process a few test cases is not problematic. Due to the large volume of tests you diligently provided, this represents only a minor impediment.

So my proposals in #1864 are nothing that I need right now. There are just some suggestions so that in the future you can do better automatic analysis like the one I am doing.

[darko-marinov]

@lukas2510 (+CC @winglam) my semester finished, so we can revisit these changes. Just let me know what's the best time for you. You can email me to schedule a Zoom meeting.

[winglam]

@lukas2510 @darko-marinov I am happy to join the discussion if one is taking place (from the email scheduling).

@lukas2510 thanks for sharing your suggestions. As you conduct your study, feel free to also share any improvements you find for the dataset.

[lukas2510]

Sorry for the late response. I am currently in the final phase of my study, finishing the report and preparing for my upcoming exams. I think it would be best to catch up once I have completed both, as it will be easier to follow my thoughts and see the context more clearly. I will reach out to you as soon as the report is ready.