Reland Update the project dependencies for the lldb-remote-linux builders. #593
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add additional dependencies with the libc++/libc++abi/libunwind and compiler-rt libraries. Also removed unaffected LLDB_CAN_USE_LLDB_SERVER options.
|
Depended on #594 |
This patch disable soft deletion on the GCS Buckets used to cache object files. Soft deletion is a feature on GCS buckets where when files are deleted they are kept around for some duration of time (~7 days by default) in a "soft" deleted state before being fully deleted. Given we do not really lose anything by deleting object files and can easily regenerate them, it does not make sense to pay for storing them for an additional seven days. By the end of the time they will be long outdated. Reviewers: dschuff, cmtice, lnihlen, gburgessiv, Keenuts Reviewed By: cmtice Pull Request: llvm#518
This patch adds lifecycle rules to the GCS buckets to delete old objects. We do not want to keep object files around for very long as they quickly become out of date. GCS does not keep track of the last time a file was fetched, so we have to rely on the creation date. The granularity for when to delete an object can only be specified in days, so we set it to one day for now. Reviewers: dschuff, lnihlen, cmtice, gburgessiv, Keenuts Reviewed By: cmtice Pull Request: llvm#519
I was apparently following the example from the documentation too closely. We do not want to use days_since_noncurrent time since it only applies to versioned objects which we do not have. Use age like I originally intentioned.
Remove the arm32 buildbot as it has been replaced with the qemu arm32 buildbot.
This removes the single stage test suite bot, then adds stage 1 check and final test suite run to the 2 stage bot. Which is the equivalent but 1 fewer worker. It'll slow down the 2 stage builds a bit but there's no a lot of activity in the 32-bit Arm area anyway, so I don't think it's a problem.
And keep the latter. Since we (Linaro) don't commit to testing ld specifically, and recommend lld anyway. The ld bot had some extra testing that I've enabled for the lld version in this change.
This patch adds an explicit maintenance window to the premerge clusters. This is in response to some control plane upgrades that we recieved notice of that will prevent access to the k8s control plane for ~15 minutes which means we will not be able to start new jobs. We should have this anyways though as the current node upgrade strategy also breaks jobs.
This patch bumps the maintenance window end date so we can actually apply the changes. GKE requires that there be at least 48 hours of maintenance time available in >=4h blocks for every 32 day segment. The original rules did not meet this. These changes allow us to deploy and should not sacrifice too much in terms of ensuring we minimize commit traffic during the maintenance periods.
This patch adds some pod disruption budgets to runner pods that just sets the minimum number of available pods to the maximum. This ensure that the number of pods that k8s calculates can be disrupted is zero. This means that when GKE is updating the node pool, it must wait an hour before forcibly evicting the pod, giving it time to finish. Before this, when GKE wanted to upgrade a node, it would forcibly evict the pod very quickly (theoretically after the grace period which has a default of 30s) not realizing it is stateful.
…vm#524) The cronjob for scraping llvm-project commit data consistently fails due to running out of memory. `git clone https://github.com/llvm/llvm-project.git` consistently uses around 1.5 GiB of RAM, which is well above both the request and limit currently set for the cronjob. Thus, the job is repeatedly killed more memory than is allocated. Increasing the resource allocation should get this cronjob running as intended.
This patch updates the operationa metrics Cron job to use Mi for the memory requests rather than Gi as k8s will automatically convert them to Mi for decimal number of Gi which terraform then thinks differs from what it tried to apply. This leads to terraform trying to reapply the setting everytime one runs terraform apply despite the settings already being equivalent.
As GitHubArchive BigQuery is known to be lossy, it is likely that we currently overestimate the number of commits made to llvm/llvm-project without an associated pull request. To remedy this, we can make calls to the [GitHub Event API](https://docs.github.com/en/rest/activity/events). While we want to avoid using the API to get information regarding every single commit made to LLVM, we can narrow our calls down to only commits that don't have any pull request data available via BigQuery. From those calls, we can determine if a "push" commit actually does have a pull request and, if it does, whether or not it has been approved.
The builds for clang-sparc64-linux currently fail with a timeout during the linking stage, so lets increase the timeout to 1800 seconds which is the timeout also used on the builder clang-solaris11-sparcv9.
1. Do not restrict pull request branch to allow hacking on in stacked PRs. 2. Remove some extra new lines at the bottom. 3. Rename the push job to better reflect the current context rather than where the job was copied from.
This shouldn't be needed any more.
Add clang-stage1-RA-as job which runs on arm64 machines
As we are already making calls to the GitHub GraphQL API for data validation, we can just remove the added complexity of using GitHub Archive BigQuery as a data source and query the API directly. Using BigQuery has the advantage of not being rate-limited, but we often have to query for 50-70 commits via the API anyway due to missing records of events in GitHub Archive. With more than half of the BigQuery data points needing amending, it makes more sense to use the API as the original data source.
This patch adds the dispatch_job python script. This script is designed to be invoked from within a llvm-zorg AnnotatedBuilder and spawn jobs inside of kubernetes pods on the premerge cluster. This is not directly integrated into an AnnotatedBuilder given the additional dependencies that we have on the kubernetes client API. Reviewers: lnihlen, dschuff, Keenuts, gburgessiv, cmtice Reviewed By: cmtice Pull Request: llvm#526
This patch adds a container image and some necessary setup code so we can run this through buildbot. Reviewers: Keenuts, cmtice, dschuff, gburgessiv, lnihlen Reviewed By: cmtice, Keenuts Pull Request: llvm#527
This patch sets up a workflow in a similar vein to all the other container images for building the premerge-buildbot container and publishing it to GHCR. There is a lot of duplicate code and eventually this should be refactored into an action. This is on my TODO list. Reviewers: cmtice Reviewed By: cmtice Pull Request: llvm#528
…m#577) The builder builds host based BOLT tools only, no Aarch64 bolt runtime yet (not supported for the cross toolchain builds).
Move the buildbot that used to do the libc builds to the CMake cache file based approach. This is a bit of a WIP as we also sort out a few other things in the background.
This patch makes it so that the postcommit builders upload the lit timing files to the GCS buckets. Once the caches get populated we can add support to the premerge builders for pulling this in.
Otherwise the cache_lit_timing_files.py script will not work.
This PR adds support for running the LLVM test-suite on Windows buildbots using ClangBuilder. The changes ensure that the correct executable paths and file extensions are used when running Python and LNT scripts on Windows. This depends on llvm/llvm-project#155226 which adds llvm-lit.cmd wrapper to make llvm-lit directory runnable on windows.
This PR adds a new buildbot to run LLVM testsuite on windows. It adds a new builder and worker pair. The testsuite will run after the first stage build skipping the ninja check step as we cover that on other buildbots. This will primarily be the builder to test that ensures LNT and LLVM testsuite are running without regressions on windows. We will later build on it to add support for test more as our Windows hardware availability improves. This depends on llvm#575 and llvm/llvm-project#155226.
This patch ensures we test flang-rt postcommit so we are actually testing the configuration introduced by #156039.
llvm/llvm-project#155186 enabled this on the premerge side, and we want corresponding postcommit coverage.
As part of the DebugifyBuilder, we add cflag arguments to the test suite build step containing the path of a report file, where any bugs detected by debugify will be written. Debugify appends to the file rather than replacing it, as we wish to accumulate the bugs found in all invocations of Clang during the test suite build step to a single file. Currently however, this file is not cleaned up after builds, meaning that we are constantly appending bugs to the same file after each test run, rather than creating a fresh file - this contaminates the results for future builds, as bugs will never disappear from the list even if they no longer exist in LLVM. This patch moves the report file into the test suite build directory, which will always be cleaned before each test run, preventing the results of one run from seeping into subsequent runs.
…llvm#584) This is already the default, and I'm dropping this option in llvm/llvm-project#156336.
Fixes 60d2141. On Windows, it's Scripts/python, on Linux it's bin/python. Unfortunately there is no third common path to use. Even the virtualenv docs have different paths for Windows and everything else: https://virtualenv.pypa.io/en/latest/user_guide.html#quick-start
…er." (llvm#589) Reverts llvm#577 Noticed unstable builder behavior during the BOLT tests. Requires investigation and update https://lab.llvm.org/buildbot/#/builders/193/builds/10263
Having the argument as part of the script name is an error as it tries to find / exec that exact string. Pass theargument as extra_args instead. Not sure if we need to split argument name and value in separate array entries.
This patch adds the design document outlining how we plan on implementing post submit testing for the premerge configuration along with the motivation and alternatives considered.
This change adds new data points to track how many lines were modified per commit, captured under a new `diff` field. This field replaces `files_modified` in our BigQuery table, as it serves the same purpose but now contains additional data regarding each file.
When querying the GitHub API, we get a runtime error if the batch size is a multiple of the total number of commits. One extra batch is queried for 0 commits, which then raises an exception when trying to access the results. This change corrects the number of batches to query for in those cases.
…s for a few bots. (llvm#585) Update descriptions for several bots that were no longer accurate. Also remove jobs properties on two of the MacOS builders to not be hardcoded.
Co-authored-by: Joseph Huber <[email protected]>
we are going to enable building libcxx for the device on that builder and hence need to listen to changes in that project.
This patch disables the use of lit's internal shell for the profcheck builder. This was enabled recently in the monorepo but is causing some test failures on the profcheck buildbot that needs more investigation. Turning off the internal shell for now to give me time to investigate things.
llvm#594) Added missed 'enable_projects' argument for UnifiedTreeBuilder.getCmakeExBuildFactory to pass it to the base LLVMBuildFactory class.
`check-llvm` does not run tests for the `llvm-lit` utility, so run the `check-lit` target as well.
This changes adds additional fields for tracking whether a commit to llvm-project is a revert and what commit/pull request it is reverting. Such commits are detecting by pattern matching against the commit message.
Add additional dependencies with the libc++/libc++abi/libunwind and compiler-rt libraries. Also removed unaffected LLDB_CAN_USE_LLDB_SERVER options.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add additional dependencies with the libc++/libc++abi/libunwind and compiler-rt libraries.
Also removed unaffected LLDB_CAN_USE_LLDB_SERVER options.