Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add direct url repodata building #725

Merged
merged 54 commits into from
Jun 13, 2024
Merged

feat: add direct url repodata building #725

merged 54 commits into from
Jun 13, 2024

Conversation

ruben-arts
Copy link
Collaborator

@ruben-arts ruben-arts commented Jun 4, 2024 •
edited
Loading

Implements:

  • Creating "virtual" repodata from a single package file
  • Makes the matchspec parsing invalid if the url is not a conda archive style name. e.g. foo-1.2.3-bsle9f2_0.conda
  • Matches a matchspec on: Matchspec::url == PackageRecord::direct_url
  • Adds a "fake" entry in the PackageRecord for a direct_url to keep track in the record that it came from a direct url and which one.

Future improvements:

  • We now fetch the required package to the cache and then read the index.json from there. We could possible stream the package in and only get what we need before we do the solve.
  • It would be great to support "any" name package file. e.g. /home/user/pacakge/test.bla and read the info from it as part of the matchspec generation.
  • We don't have all information in the package record from the index.json, issue: Get sha256, size and md5 from direct url package. #742

@ruben-arts
Copy link
Collaborator Author

This doesn't build ontop of main yet. Let me fix that later

@ruben-arts ruben-arts force-pushed the feature/direct_url_query branch from cef239e to 5ee5d0f Compare June 5, 2024 13:36
baszalmstra
baszalmstra reviewed Jun 6, 2024
View reviewed changes
Copy link
Collaborator

@baszalmstra baszalmstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • We should make sure that when repodata is requested for a direct url that the name of its index.json file matches what we expect the name to be.
  • Does this also support file:// urls?

crates/rattler_conda_types/src/match_spec/mod.rs Outdated Show resolved Hide resolved
crates/rattler_conda_types/src/package_name.rs Outdated Show resolved Hide resolved
crates/rattler_conda_types/src/repo_data/mod.rs Outdated Show resolved Hide resolved
crates/rattler_conda_types/src/repo_data/mod.rs Outdated Show resolved Hide resolved
crates/rattler_conda_types/src/version/mod.rs Outdated Show resolved Hide resolved
crates/rattler_repodata_gateway/src/gateway/direct_url_gateway.rs Outdated Show resolved Hide resolved
crates/rattler_repodata_gateway/src/gateway/error.rs Outdated Show resolved Hide resolved
crates/rattler_repodata_gateway/src/gateway/mod.rs Outdated Show resolved Hide resolved
crates/rattler_repodata_gateway/src/gateway/mod.rs Outdated Show resolved Hide resolved
crates/rattler_repodata_gateway/src/gateway/query.rs Outdated Show resolved Hide resolved
@ruben-arts ruben-arts mentioned this pull request Jun 10, 2024
Open
@ruben-arts ruben-arts marked this pull request as ready for review June 10, 2024 19:19
baszalmstra
baszalmstra requested changes Jun 11, 2024
View reviewed changes
Copy link
Collaborator

@baszalmstra baszalmstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this also work for file://... urls?

crates/rattler_repodata_gateway/src/gateway/builder.rs Outdated Show resolved Hide resolved
crates/rattler_repodata_gateway/src/gateway/query.rs Outdated Show resolved Hide resolved
crates/rattler_repodata_gateway/src/gateway/query.rs Outdated Show resolved Hide resolved
crates/rattler_repodata_gateway/src/gateway/query.rs Show resolved Hide resolved
crates/rattler_solve/src/resolvo/mod.rs Outdated Show resolved Hide resolved
@ruben-arts
Copy link
Collaborator Author

Added a test to see if there is only one direct url and that it is located in the first subdir: 4cc3c50

@baszalmstra baszalmstra enabled auto-merge (squash) June 13, 2024 12:00
@baszalmstra baszalmstra merged commit c18aec9 into conda:main Jun 13, 2024
14 checks passed
This was referenced Jun 11, 2024
Closed
Closed
Closed
@baszalmstra baszalmstra mentioned this pull request Jul 8, 2024
Merged
baszalmstra added a commit that referenced this pull request Jul 8, 2024
@baszalmstra
chore: release (#769)
6fae9a4 
## 🤖 New release
* `rattler`: 0.26.4 -> 0.26.5
* `rattler_cache`: 0.1.0 -> 0.1.1
* `rattler_conda_types`: 0.25.2 -> 0.26.0
* `rattler_macros`: 0.19.3 -> 0.19.4
* `rattler_package_streaming`: 0.21.3 -> 0.21.4
* `rattler_networking`: 0.20.8 -> 0.20.9
* `rattler_shell`: 0.20.9 -> 0.21.0
* `rattler_lock`: 0.22.12 -> 0.22.13
* `rattler_repodata_gateway`: 0.20.5 -> 0.21.0
* `rattler_solve`: 0.24.2 -> 0.25.0
* `rattler_libsolv_c`: 0.19.3 -> 0.19.4
* `rattler_virtual_packages`: 0.19.15 -> 0.19.16
* `rattler_index`: 0.19.17 -> 0.19.18

<details><summary><i><b>Changelog</b></i></summary><p>

## `rattler`
<blockquote>

##
[0.26.5](rattler-v0.26.4...rattler-v0.26.5)
- 2024-07-08

### Added
- add direct url repodata building
([#725](#725))

### Fixed
- errors should not contain trailing punctuation
([#763](#763))
- run clippy on all targets
([#762](#762))
</blockquote>

## `rattler_cache`
<blockquote>

##
[0.1.1](rattler_cache-v0.1.0...rattler_cache-v0.1.1)
- 2024-07-08

### Added
- add direct url repodata building
([#725](#725))

### Fixed
- run clippy on all targets
([#762](#762))
</blockquote>

## `rattler_conda_types`
<blockquote>

##
[0.26.0](rattler_conda_types-v0.25.2...rattler_conda_types-v0.26.0)
- 2024-07-08

### Added
- add support for zos-z
([#753](#753))
- return pybytes for sha256 and md5 everywhere and use md5 hash for
legacy bz2 md5 ([#752](#752))
- add direct url repodata building
([#725](#725))
- add shards_base_url and write shards atomically
([#747](#747))

### Fixed
- allow version following package in strict mode
([#770](#770))
- Fix doctests and start testing them again
([#767](#767))
- skip over implicit `0` components when copying
([#760](#760))
- allow empty json repodata
([#745](#745))
- lenient and strict parsing of equality signs
([#738](#738))
- This fixes parsing of `ray[default,data] >=2.9.0,<3.0.0`
([#732](#732))
</blockquote>

## `rattler_macros`
<blockquote>

##
[0.19.4](rattler_macros-v0.19.3...rattler_macros-v0.19.4)
- 2024-07-08

### Other
- update README.md
</blockquote>

## `rattler_package_streaming`
<blockquote>

##
[0.21.4](rattler_package_streaming-v0.21.3...rattler_package_streaming-v0.21.4)
- 2024-07-08

### Fixed
- run clippy on all targets
([#762](#762))
</blockquote>

## `rattler_networking`
<blockquote>

##
[0.20.9](rattler_networking-v0.20.8...rattler_networking-v0.20.9)
- 2024-07-08

### Fixed
- errors should not contain trailing punctuation
([#763](#763))
</blockquote>

## `rattler_shell`
<blockquote>

##
[0.21.0](rattler_shell-v0.20.9...rattler_shell-v0.21.0)
- 2024-07-08

### Added
- add direct url repodata building
([#725](#725))
- allow passing custom environment to run_activation
([#743](#743))
</blockquote>

## `rattler_lock`
<blockquote>

##
[0.22.13](rattler_lock-v0.22.12...rattler_lock-v0.22.13)
- 2024-07-08

### Added
- Only save md5 in lock file if no sha256 is present
([#764](#764))
- return pybytes for sha256 and md5 everywhere and use md5 hash for
legacy bz2 md5 ([#752](#752))
- add direct url repodata building
([#725](#725))

### Fixed
- lock file stability issues with PyPI types
([#761](#761))
- errors should not contain trailing punctuation
([#763](#763))

### Other
- revert only save md5 in lock file if no sha256 is present
([#766](#766))
</blockquote>

## `rattler_repodata_gateway`
<blockquote>

##
[0.21.0](rattler_repodata_gateway-v0.20.5...rattler_repodata_gateway-v0.21.0)
- 2024-07-08

### Added
- improve error message when parsing file name
([#757](#757))
- add direct url repodata building
([#725](#725))
- add shards_base_url and write shards atomically
([#747](#747))

### Fixed
- direct_url query for windows
([#768](#768))
- Fix GatewayQuery.query to filter records based on provided specs
([#756](#756))
- run clippy on all targets
([#762](#762))
- allow empty json repodata
([#745](#745))

### Other
- document gateway features
([#737](#737))
</blockquote>

## `rattler_solve`
<blockquote>

##
[0.25.0](rattler_solve-v0.24.2...rattler_solve-v0.25.0)
- 2024-07-08

### Added
- add direct url repodata building
([#725](#725))
- add tool to generate resolvo snapshots
([#741](#741))

### Fixed
- run clippy on all targets
([#762](#762))
- This fixes parsing of `ray[default,data] >=2.9.0,<3.0.0`
([#732](#732))

### Other
- bump resolvo to 0.6.0
([#733](#733))
- Document all features on docs.rs
([#734](#734))
</blockquote>

## `rattler_libsolv_c`
<blockquote>

##
[0.19.4](rattler_libsolv_c-v0.19.3...rattler_libsolv_c-v0.19.4)
- 2024-07-08

### Other
- update README.md
</blockquote>

## `rattler_virtual_packages`
<blockquote>

##
[0.19.16](rattler_virtual_packages-v0.19.15...rattler_virtual_packages-v0.19.16)
- 2024-07-08

### Added
- add support for zos-z
([#753](#753))

### Fixed
- run clippy on all targets
([#762](#762))
</blockquote>

## `rattler_index`
<blockquote>

##
[0.19.18](rattler_index-v0.19.17...rattler_index-v0.19.18)
- 2024-07-08

### Other
- updated the following local packages: rattler_conda_types,
rattler_package_streaming
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/MarcoIeni/release-plz/).
@baszalmstra baszalmstra mentioned this pull request Jul 8, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@baszalmstra baszalmstra baszalmstra approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ruben-arts @baszalmstra