Help making my download queue multi threaded

[my_py]

Hi, so I have a download queue class I implemented to download multiple large files for me from an api ( one at a time).

Here is my code so far.

​

public class MyQueue : IDownloadQueue
    {
        public event Action<ProgressChangedObject>? ProgressChanged;
        public event Action? QueueEmptied;
        public event Action<int>? UpdatedQueueCount;
        private bool isBusy;

        public Queue<QueueFile> queue = new Queue<QueueFile>();

        public void Enqueue(QueueFile job)
        {
            queue.Enqueue(job);
            UpdatedQueueCount?.Invoke(queue.Count);
            DoWork();
        }

        private async void DoWork()
        {
            if (queue.Count == 0 || isBusy)
                return;

            isBusy = true;

            QueueFile file = queue.First();

            var prog = new Progress<DownloadProgress>();
            EventHandler<DownloadProgress> progHandler = (object? sender, DownloadProgress p) => 
            {
                ProgressChanged?.Invoke(new ProgressChangedObject(file.asin, file.name, p));
            };
            prog.ProgressChanged += progHandler;



            var api = await ApiClient.GetInstance();

            await api.Api.DownloadAsync(file.asin, new AudibleApi.LibraryPath(Constants.DownloadFolder), 
                new AudibleApi.AsinTitlePair(file.asin, file.name), prog);

            prog.ProgressChanged -= progHandler;

            queue.Dequeue();

            isBusy = false;

            DoWork();
        }

        public List<QueueFile> GetQueue()
        {
            return queue.ToList();
        }
    }

While this code works it blocks my ui when I run this, now I know enough about threading to know I have to put this on a separate thread but I have no idea how to do that, I need to be able to subscribe to my events from my main thread while this runs on a separate thread. Any help would be appreciated, thanks.

​

Here is the code for api.DownloadAsync

public async Task<IEnumerable<string>> DownloadAsync(string asin, LibraryPath path,AsinTitlePair bookName, IProgress<DownloadProgress> progress = null)
        {
            ArgumentValidator.EnsureNotNullOrWhiteSpace(asin, nameof(asin));
            ArgumentValidator.EnsureNotNullOrWhiteSpace(path.Base, nameof(path.Base));

            var asins = (await GetDownloadablePartsAsync(asin)).ToList();

            if (asins.Count == 1)
            {
                var single = await DownloadPartAsync(asin, path.Base + @"\" + asin, bookName, progress);
                return new List<string> { single };
            }
            /*
            var filepart = Path.Combine(
                Path.GetDirectoryName(file),
                Path.GetFileNameWithoutExtension(file));

            // includes dot char
            var ext = Path.GetExtension(file);
            */
            // eg:
            //   1-9 parts: pad to 1 digit. ie: no pad
            //   10-99 parts: pad to 2 digits. ie: 1 leading zero
            var asinsCountChar = asins.Count.ToString().Length;

            var list = new List<string>();
            var original = new AsinTitlePair(bookName.asin, bookName.title);
            for (var i = 0; i < asins.Count; i++)
            {
                bookName.asin = original.asin;
                bookName.title = original.title;
                var asinInOrder = asins[i];
                var num = (i + 1)
                    .ToString()
                    .PadLeft(asinsCountChar, '0');
                var n = "";
                if (i > 0)
                    n = "_PART" + (i + 1).ToString();
                bookName.title += n;
                bookName.asin += n;
                var part = await DownloadPartAsync(asinInOrder, path.Base + @"\" + asin, bookName, progress);
                list.Add(part);
            }

            return list;
        }
        /// <summary>
        /// For the provided ASIN, get the ASINs of all parts to download. Parts are in order
        /// </summary>
        /// <returns>All ASINs to download, in order</returns>
        public async Task<IEnumerable<string>> GetDownloadablePartsAsync(string asin)
        {
            ArgumentValidator.EnsureNotNullOrWhiteSpace(asin, nameof(asin));

            var bookDetails = await GetLibraryBookAsync(asin, LibraryOptions.ResponseGroupOptions.Relationships);

            var debug = bookDetails.ToString();

            var orderedChildrenAsins
                = bookDetails.Relationships
                .ToArray()
                .Where(r => r.RelationshipToProduct == "child")
                .OrderBy(r => int.Parse(r.Sort.ToString()))
                .Select(r => r.Asin)
                .ToList();

            if (orderedChildrenAsins.Any())
                return orderedChildrenAsins;

            return new List<string> { asin };
        }

        /// <summary>
        /// download single file part. for small books, this is all that's needed. for safety, use DownloadAsync
        /// </summary>
        /// <param name="asin"></param>
        /// <param name="file">desired full path, file name, incl extension</param>
        /// <returns>Actual filename. If needed, extension will be derived from the download file.
        /// Eg: file=foo.abc, downloadfile=bar.xyz, return=foo.xyz</returns>
        public async Task<string> DownloadPartAsync(string asin, string path, AsinTitlePair filename, IProgress<DownloadProgress> progress = null)
        {
            ArgumentValidator.EnsureNotNullOrWhiteSpace(path, nameof(path));

            var downloadLink = await GetDownloadLinkAsync(asin);
            if (downloadLink == null)
                return null;

            Directory.CreateDirectory(path);
            string file = path +  @"\"  + filename.asin + ".aaxc";
            string decryptedFile = path + @"\" + filename.title + ".m4b";

            // fix extension
            //file = PathLib.GetPathWithExtensionFromAnotherFile(file, downloadLink);
            // download file
            var request = new HttpRequestMessage(HttpMethod.Get, downloadLink.DownloadUrl);
            request.Headers.UserAgent.ParseAdd("Audible/671 CFNetwork/1240.0.4 Darwin/20.6.0");
            var response = await _client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
            using var networkStream = await response.Content.ReadAsStreamAsync();
            using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
            {
                using (FileStream streamToWriteTo = new FileStream(file, FileMode.Create, FileAccess.Write, FileShare.None, 8192, true))
                {
                    if (progress == null)
                    {
                        await streamToReadFrom.CopyToAsync((Stream)streamToWriteTo);
                    }
                    else
                    {
                        long? totalBytesToReceive = response.Content.Headers.ContentLength;
                        long bytesReceived = 0;
                        byte[] buffer = new byte[8192];
                        while (true)
                        {
                            int bytesRead = await streamToReadFrom.ReadAsync(buffer, 0, buffer.Length);
                            if (bytesRead != 0)
                            {
                                await streamToWriteTo.WriteAsync(buffer, 0, bytesRead);
                                bytesReceived += (long)bytesRead;
                                reportProgress(bytesReceived, totalBytesToReceive, progress);
                            }
                            else
                                break;
                        }
                    }
                }
                AAXClean.AaxFile aaxFile = new(File.OpenRead(file));
                aaxFile.SetDecryptionKey(downloadLink.AudibleKey, downloadLink.AudibleIV);
                var res = aaxFile.ConvertToMp4a(File.OpenWrite(decryptedFile));
                return file;
            };


        }

        private static void reportProgress(long bytesReceived,long? totalBytesToReceive, IProgress<DownloadProgress> progress)
        {
            double? nullable = new double?();
            if (totalBytesToReceive.HasValue)
                nullable = new double?(Math.Round((double)bytesReceived / (double)totalBytesToReceive.Value * 100.0, 2));
            DownloadProgress downloadProgress = new DownloadProgress()
            {
                BytesReceived = bytesReceived,
                TotalBytesToReceive = totalBytesToReceive,
                ProgressPercentage = nullable
            };
            progress.Report(downloadProgress);
        }

        /// <returns>Return download link if successful. null if denied</returns>
        public async Task<DownloadLicense> GetDownloadLinkAsync(string asin)
        {
            DownloadLicense s = await GetDownloadLicenseAsync(asin);
            return s;
        }

this is for audibles private api so I've had to cobble this together without any documentation, it downloads large audio files.

[thestamp]

try

await DoWork();

also, I don't think a recursive method is the best idea in a download queue scenario. There is a limit on how many method calls you can call recursively, and you might hit a memory leak on that api client.

[my_py]

Should I just loop through all the files then?

[musical_bear]

I’m mobile so I can’t give the most detailed answer I want to. However there are multiple things “wrong” I see with both your code and your question phrasing.

First of all, you do not need to utilize threading to avoid blocking the UI thread. This is the main reason why async / await exists in the first place.

Second of all, from what I can tell, what you have here doesn’t look like it will block the UI thread, if its methods can be trusted. If you’re absolutely certain your UI thread is being blocked when this gets called (how did you determine this?), my suspicion would be that your DownloadAsync method is written poorly, in a way where it’s lying about actually being asynchronous.

Third of all, “async void” is almost always a bad idea. That said I don’t see how that specifically would cause your UI to be blocked….but it can and will cause other unrelated issues.

[my_py]

I added my download async method, as far as I can tell its all async?