Skip to content

Add to_agen wrapper and aiofiles.os.walk and aiofiles.os.fwalk. - #193

Draft
stankudrow wants to merge 16 commits into
Tinche:mainfrom
stankudrow:extend-path-tools
Draft

stankudrow wants to merge 16 commits into
Tinche:mainfrom
stankudrow:extend-path-tools

Conversation

@stankudrow

@stankudrow stankudrow commented Nov 11, 2024 •

Copy link
Copy Markdown
Contributor

Closes #160 .
Also may close #167 .
Potentially closes #207 .

Features:

  • to_agen for wrapping synchronous generators;
  • wrap uses asyncio.to_thread under the hood because Python3.9 is minimum version;
  • the asynchronous versions of the os.walk and os.fwalk (Linux only) are added (to_agen made it possible)
  • tests updated and test execution order is randomised for each run by default
  • minor changes

@stankudrow
stankudrow marked this pull request as draft November 12, 2024 08:28
@stankudrow
stankudrow marked this pull request as ready for review November 12, 2024 09:14
@stankudrow stankudrow changed the title Add os.listdir and os.walk wrappers Add os.walk wrapper Nov 12, 2024
@Tinche

Tinche commented Dec 4, 2024

Copy link
Copy Markdown
Owner

Apologies. I have a small baby nowadays.

So the original sync os.walk is a generator, right? This has some very useful properties which we should keep. Can we make the async version an async generator? Each step can do a step of the sync one.

@stankudrow

Copy link
Copy Markdown
Contributor Author

Apologies. I have a small baby nowadays.

So the original sync os.walk is a generator, right? This has some very useful properties which we should keep. Can we make the async version an async generator? Each step can do a step of the sync one.

Hello and no need for apologies, au contraire, congratulations.

Comment thread src/aiofiles/ospath.py Resolved
Comment thread src/aiofiles/os.py Resolved
@stankudrow
stankudrow requested a review from Tinche December 28, 2024 19:15
@stankudrow
stankudrow marked this pull request as draft February 1, 2025 17:22
@stankudrow
stankudrow marked this pull request as ready for review February 8, 2025 15:55
@stankudrow
stankudrow marked this pull request as draft February 8, 2025 16:27
@stankudrow
stankudrow marked this pull request as ready for review February 8, 2025 17:44
@Martin005

Copy link
Copy Markdown

Waits for the #199 to be merged.

That PR has been merged now, can we please merge this one as well? 🙂

@stankudrow
stankudrow marked this pull request as draft April 14, 2025 07:41
@stankudrow
stankudrow marked this pull request as ready for review April 25, 2025 06:03
@stankudrow
stankudrow marked this pull request as draft April 30, 2025 02:36
stankudrow pushed a commit to stankudrow/aiofiles that referenced this pull request Jun 20, 2025
@stankudrow stankudrow changed the title Add os.walk wrapper Add to_a(sync)gen wrapper able to turn a sync generator function into its async variant and introduce an async version of os.walk. Aug 12, 2025
@stankudrow stankudrow changed the title Add to_a(sync)gen wrapper able to turn a sync generator function into its async variant and introduce an async version of os.walk. Add to_agen wrapper and aiofiles.os.walk. Mar 6, 2026
@stankudrow
stankudrow marked this pull request as ready for review March 6, 2026 22:17
@Tinche

Tinche commented Mar 9, 2026

Copy link
Copy Markdown
Owner

Thanks for your patience. I'm going to review this in the next couple of days.

@Tinche

Tinche commented Mar 9, 2026

Copy link
Copy Markdown
Owner

Prior to Python 3.12 this fails because inspect.isgeneratorfunction(os.walk) is False and since Py3.12 it is True. Any ideas how to have goodies from both realms -> having one wrap which can handle routines and generator functions (or functions that return an iterator object) and are universal enough to work for Py3.9+ versions?

I wouldn't go through gymnastics to make this work - we know when we're wrapping a generator and when an ordinary function. Two functions are perfectly fine IMO.

@stankudrow stankudrow changed the title Add to_agen wrapper and aiofiles.os.walk. Add to_agen wrapper and aiofiles.os.walk and aiofiles.os.fwalk. Mar 10, 2026
@Tinche

Tinche commented Mar 12, 2026

Copy link
Copy Markdown
Owner

I'm very unconvinced by the implementation of to_agen. To me it looks like it's essentially performing a spin-lock while waiting for the (sync) queue to return anything. There's a while True: with a continue in a branch.

Let me think about doing this in a better way. Bridging async and sync is going to be a little tricky here.

@stankudrow
stankudrow marked this pull request as draft August 18, 2026 07:06

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Async os.walk attempted solution add "aiofiles.path.walk" or "aiofiles.path.listdir" Python 3.9: to_thread or not to_thread?

3 participants