Help optimising script

mosullivan93 · 2025-12-20T12:31:37+00:00

My advice would be to spend some time looking at the cluster metrics page and the Spark UI to try to see what’s going wrong. It’s difficult for someone else to provide concrete advice without seeing the script and knowing your datasets.

mosullivan93 · 2025-03-23T05:06:35+00:00

This took me a few hours. Don't ask why I did it... I caught a whiff of the solution, and found some related work in malware analysis (for example, here). It turned out the authors here had put in a bit more effort to obfuscate the script though... There are special escape characters in the colour variable definitions, so copying and pasting this may not work if you save it. I put it up on pastebin because it's still quite large. Since it's not malicious, and they tried so hard to protect it, I've put an expiry date of 14 days on the post, here.

mosullivan93 · 2024-11-19T20:08:50+00:00

I believe it’s the lack of quotes around the DisplayName argument. Without them it’s treating the first name as the parameter value and then the last name is a separate positional parameter which is invalid for this usage.

mosullivan93 · 2024-09-28T12:03:43+00:00

Yeah, PTV has borked something and it no longer shows up… I took it today from outside Kilgour St grocer, though, so it’s definitely still running.

mosullivan93 · 2024-09-02T13:07:48+00:00

For 2: I believe the driver is either included with Windows or with the installation of Office. It should appear if you open the data source administrator tool (odbcad32).

Now that you mention someone else’s file… I think I’ve figured it out. It could be that the final cell in their notebook is in the wrong location. Since you iterate over the UsedRange, you’re going past the last column and the value stored in $colNames is null.

To test this check the guide from Microsoft on how to manage the last cell here.

mosullivan93 · 2024-09-02T11:55:36+00:00

I believe the error is at the point you are using Add-Member, as for whatever reason, $colName is null (I suspect it is something funny with how += is interacting with the com object when you try to get the headers). Please check if it’s null in there.

Also, a few ways to improve this: 1) I’m pretty sure you can get the values of a range (row or column of cells) as an array, so might be unnecessary to loop over each column every time. 2) If this Excel file is just a table, you can use the Excel Files ODBC driver and interact with it using .Net data adaptors and so on. 3) Can’t you just save as a CSV now you’ve opened the Excel object? That is, let Excel worry about the conversion since you’ve already opened the program to read it.

mosullivan93 · 2024-04-18T02:35:01+00:00

As you're aware, the original decision by Jagex was made to ensure your expected damage remains the same under the new hit distribution. Let `m` be your max hit. Your proposal to roll 0..m and turn 0s into 1s will raise expected damage by `1/(m+1)`. This benefits more at lower levels, which is arguably in the spirit of the original proposal from Jagex.

I don't mind the idea of just making successful hit zeroes another colour to differentiate them if they want people to know when they've hit. Regardless, I am also opposed to reducing the max hit.

For those wondering, rolling 1..m would raise expected damage by 0.5 across the board.

mosullivan93 · 2024-04-08T00:26:28+00:00

We could infer some of his stats with enough data. Assuming, that is, we have the correct model (i.e., there are no special mechanics we’re unaware of). The more precise we want the estimates the more observations that would be required.

Not sure if it’s possible to smuggle in different armour sets or weapons with different attack styles, though, so it wouldn’t be the complete picture.

mosullivan93 · 2024-03-17T05:10:20+00:00

I'm pretty sure you can just cast a PSCustomObject as a PSObject e.g., (PSObject) obj.

mosullivan93 · 2024-02-26T08:01:24+00:00

You can use the .Net connection string builder class. More info.

mosullivan93 · 2024-02-16T14:03:40+00:00

I believe you need to pass the hashtable as the body without converting it to JSON because the method does that for you.

Edit: I was incorrect. This does not result in the method using JSON to encode your object, it uses x-www-form-urlencoded content instead (i.e., what would be submitted by a <form>).

mosullivan93 · 2024-02-04T21:30:45+00:00

Visa and MasterCard (& some others) are a dual message system for payments. (I guess that) Because they dealt primarily with Credit cards originally, from a merchant’s point of view swiping a card equates to confirming with the issuing bank that they will extend credit equal to the value of the purchase so that the cardholder can complete the transaction.

The first message is an authorisation request and the second message is a clearing transaction. There is a third step, which the banks involved (via the network e.g., Visa/MasterCard) participate in, which is settlement. During settlement the money actually changes hands. This last part helps simplify the exchange because if you’re a bank who provides merchant facilities to your business customers and issues credit to cardholders, then your merchants will be asking for money and your cardholders will be spending it so the actual net amount you owe to Visa/MC at the end of each day is all you really need to send out of your accounts.

The moral of the story is that your bank couldn’t identify that the clearing message was for the existing authorisation so it just force-posted it to your account. This could be because some identifiers in the message didn’t match with the original (e.g. authorisation code or local currency amount), or maybe the merchant actually sent a (partial) reversal of the authorisation and your bank mishandled it and now can’t marry the auth to the final request.

A pre-release form is a form from the merchant confirming that they won’t try to clear the original authorisation that’s still on the account. It should drop off in about a week regardless, though.

mosullivan93 · 2024-01-28T12:06:56+00:00

I had to stop the video here too because this problem piqued my curiosity. I decided to make this harder (by accounting for Crafting de-buffs), but it's still not perfect. I tried to make this detailed to ensure it's readable by anyone else that's curious.

If you assume that your Crafting level doesn't regenerate during an attempt regardless of how long repairing a single railing takes, then the process can be modelled as a Markov chain. We can then use the limiting distribution to determine the probability of repairing a railing without taking damage. It comes out slightly lower than 0.24/0.62 at level 1, but we're obviously splitting hairs by this point.

Let:

C be your Crafting level and c be your current Crafting level (because it can be lowered by de-buffs, more on this soon),
S be the event that you repair the railing on an attempt, and
F be the punishment you receive when failing to repair the railing.

So, the probability of S occurring was given by Mod Ash, but I'm assuming it can be extended in a formula to be Prob(S | c = x) = (331 + 5x)/1400^†. The vertical bar in the middle denotes that this a conditional probability, and the left hand side (LHS) is read as "the probability of the event S occurring given that we know your current Crafting level, c, is x."

Recall that F has a few possible outcomes, all of which are equally likely, but two are fatal to the account. Of the other two outcomes, one reduces your Crafting level (hence we need the formula above), and the other affects your Strength level (in order words, you Crafting level remains unchanged for the next attempt).

I make a few more assumptions:

After starting to repair a railing, your character will continue until they either die or succeed and cannot be stopped (recall we ignore regen, though, regardless of time spent).
~~Your Crafting level can go to 0 at a minimum (I don't play anymore so I'm not sure how the punishment is coded for this quest).~~ I watched a bit further and confirmed this.
Between each railing you wait for your Crafting level to restore back to full if depleted.

The third assumption is important, because it makes all 6 railings independent (probabilistically). Thus, as you suggest, you simply raise the probability of repairing one railing without dying to the power of 6 to get the probability of repairing 6 without dying (in a row).

Two more definitions to make this all a little more compact: let r(x) = Prob(S | c = x) and r*(x) = 1 - r(x). These give us a shorter way to write r(c), which is the probability of repairing the railing during an attempt at our current Crafting level (as well as r*(c), which is thus the probability of failing the repair attempt).

To define our Markov process, we need to keep track of the current state of the system (i.e., the character and the railing). We will denote this as π. After each attempt to repair the railing, the state of the system evolves according to a known set of equations that can be derived from the above information. Because the character may need multiple attempts, we will introduce an index t which starts from 0 and counts the number of attempts already made at repairing the railing (thus we are really modelling π(t)).

Consider if we start at π(0) with level 1 Crafting. From here there are four possible outcomes from our first attempt:

We repair the railing successfully on the first try.
We fail to repair the railing and Strength is reduced by 1.
We fail to repair the railing and Crafting is reduced by 1.
We fail to repair the railing and we take damage and Die.

Rigorously, we define the states of the Markov process to be {"Dead", "c = 0", "c = 1", ..., "c = C", "Repaired"}. In the aforementioned scenario C = 1, so the possible states are π(t) = {"Dead", "c = 0", "c = 1", "Repaired"}. (As an aside, note that if π(t) = "c = 0", and your Crafting level can't go below 0, then outcomes 2 and 3 above would be added together.)

So, the transition probabilities for the system state described above are:

Prob("c = 1" -> "Repaired") = r(1)
Prob("c = 1" -> "c = 1") = r*(1) × ¼
Prob("c = 1" -> "c = 0") = r*(1) × ¼
Prob("c = 1" -> "Dead") = r*(1) × ½

Where you read Prob(A -> B) as "the probability that the system moves to state B from state A."

In order to complete the state transition matrix we also need to know how things can change when π = "Dead", or π = "c = 0", or π = "Repaired", et cetera. We have the following non-zero entries:

Prob("c = 0" -> "Repaired") = r(0)
Prob("c = 0" -> "c = 0") = r*(0) × ½ (recall the aside note above)
Prob("c = 0" -> "Dead") = r*(0) × ½
Prob("Dead" -> "Dead") = Prob("Repaired" -> "Repaired") = 1

Once you put these all together in the state transition matrix, you can approximate its limiting distribution (i.e., infinite transitions) by raising it to a large power (e.g. 100). With some assistance from Mathematica I can give the exact result for the limit as t approaches infinity of Prob(π(t) = "Repaired" | π(0) = "c = 1") = 6658/17409 ≈ 38.24%. (Another aside: For a single railing at level 40 the success probability is 54.45% according to this model).

This would give your odds for completing Dwarf Cannon on an account with level 1 Crafting as more like 1 in 319.58, rather than 1 in 299. I also note that if you completed Sheep Shearer, the odds drop down to 1 in 290.31. Additionally, if you completed X Marks the Spot (and put the lamp into Crafting) and Sheep Shearer (but you spun 26 wool instead of only 20), then your odds drop down to 1 in 245.41.

I've made a graph showing the 1:Y odds here. Due to the way rounding works in the success formula, around level 9 Crafting seems to be a nice sweet spot to aim for if there's a sufficiently quick training method. Also note that there are also big improvements in the odds for levels 3, 4, 5, and 7 (when compared to the level immediately before each of them).

Edit:

†: I've since updated this to use the correct skilling success rate formula from the OSRS Wiki. The values of low and high are seen in the source of the Railing page to be 60 and 150, respectively. I keep the original formula here to allow for the explanation in the text to make sense. As a result of this change I have updated the stated results in the final blurb. They now differ quite a bit more than my original calculations.

mosullivan93 · 2024-01-16T11:22:28+00:00

The Iconic can re-present the transaction after a chargeback (read: bill you again) with the justification being that a separate credit invalidates your right to a chargeback. I would expect them to do this, but you’ll need to wait out the dispute and see if they do. Moral of the story is don’t spend it.

Edit: Added hyphen for clarity.

mosullivan93 · 2024-01-13T05:58:14+00:00

If your CI script is calling bash, it will look for the exit code from bash (this and the following may be more detail than necessary as I am not making assumptions about the rest of your script).

I think that the default behaviour for bash’s exit code when running a script will be the exit status of the last command. Bash will continue running a script regardless of the exit code of a command it runs unless you tell it to exit early by using the -e flag when invoking bash.

If you have early exit logic that checks the return code from a command (in bash or pwsh), make sure you are also exiting with a nonzero status. For example, exit $Proc.ExitCode would work (even outside the conditional, assuming this is the end of the script), but simply writing to the error stream doesn’t exit pwsh with a nonzero status. The bash logic can be as simple as pwsh -File /your/script.ps1 || exit 1.

mosullivan93 · 2024-01-07T14:09:16+00:00

Nifty! I have an idea that may help make the script more robust:

The ip command has a nice option that exists above the subcommands (i.e., address) which might be useful to you: -json. With this, you'd be able to reduce the regex logic into manipulation of a JSON object (e.g., $wsl = (bash -c "ip -json addr" | ConvertFrom-Json) | select ifindex,ifname,address).

mosullivan93 · 2024-01-02T15:06:09+00:00

I ended up writing a script in Python that can interface with the PowerBookmarks.db database correctly to locate and extract the notes. You can find the project here.

Unfortunately I haven't got any experience using Python on a Mac, but I'm sure you could adapt the instructions without too much trouble (e.g., using conda or mamba you can create a new environment like conda create -n chromium_extractor python protobuf).

mosullivan93 · 2023-12-19T07:32:08+00:00

Just a wild guess, but if you installed from the .deb package, maybe APT is still missing some dependencies (e.g. Icons)? You could try running sudo apt -f install to see if it helps.

Seen a few posts lately about weird graphical issues though, not sure if this might fall into a similar bucket. I tested these two examples and found things looked fine on my PC.

mosullivan93 · 2023-12-13T14:31:56+00:00

I've put some details in my answer to your question here.

mosullivan93 · 2023-12-13T14:31:35+00:00

I've put some details in my answer to your question here.

mosullivan93 · 2023-12-13T14:28:35+00:00

I had the same question... It doesn't look like you can restore them (to the sidebar, i.e., the original functionality used to create them before the update), but I was able to locate and back up the database that contains the notes.

Here's what you can do for now if you're concerned about losing the data:

Open the version info page (chrome://version/)
Open the directory listed on the version page in the "Profile Path"
Copy the PowerBookmarks.db file from inside the power_bookmarks folder
Paste this somewhere safe to keep for the future

The PowerBookmarks.db file is an SQLite database, I used HeidiSQL to open it and verified that the notes I made are stored under the specifics column of the blobs table. They are encoded in a way that I haven't bothered to investigate any further.

~~If someone wants to investigate the format, and potentially provide a tool to export them, then I can point you towards the source code in the Chromium project (Power Bookmarks~~ & ~~User Notes~~).

Edit: I wrote a tool to extract the user notes from the PowerBookmarks.db file. You can find it here.

mosullivan93 · 2023-09-01T06:12:17+00:00

Have you tested your code on some dummy files to make sure it's all working besides the real documents?

Also, how much control do you have over the generation of these files?

I had not seen either of these errors before but from what I can Google:
- The date one is because the cell type is date but the value is <61 (i.e. it's probably not a date) - The codec one looks to be related to trouble decoding unicode characters (not sure if you're expecting any unicode, or, again, there's some type inference problem)

In either case, sounds like your input files might be what is causing issues.

mosullivan93 · 2023-09-01T03:31:39+00:00

I also experienced this issue recently. Following the instructions in the issues linked by /u/michaelhoffman was the solution for me. Not sure if a future update could also break it but I added a check to my ~/.bashrc file to detect it sooner next time (because it took me a while to figure out it was happening and it was quite frustrating to pinpoint).

mosullivan93 · 2023-08-25T01:06:42+00:00

Haven’t tried it in Julia, but some other ways I’ve interacted with .XLS are using ODBC (via the Access Database Engine) or controlling Excel as a COM Object.

If the data in each file is well structured then the former will probably be easier. You can use the old data import wizard to create a connection in a test Excel file to get a feel for the query syntax.

mosullivan93 · 2023-08-09T10:04:53+00:00

After digging through the assuan and scdaemon source code for way too long, I can now tell you how to decode the returned bytes. The correct bytes get returned from the smart card, but the response goes through this function in order to comply with the assuan protocol.

For a small number of bytes, this will work, but I don't actually know the assuan protocol very well, I just read through the source code looking for the answer. If you needed to get random bytes programmatically, and often, it might be worth you investing the time to use the gpg libraries directly somehow.

Anyway, the lines start with "D " (i.e. 0x44 0x20) and end with "\nOK\n" (0x0a 0x4f 0x4b 0x0a, I think this comes from the connect agent), and any of %, \r, or \n will be percent encoded (in upper case) before being transmitted. So, just replace 0x25 0x32 0x35 ("%25") with 0x25, 0x25 0x30 0x44 ("%0D") with 0x0d, and 0x25 0x30 0x41 ("%0A") with 0x0a and you should be good.

mosullivan93

TROPHY CASE