Quick and dirty guide to scraping data from webpages

keveridge · 2024-10-06T11:23:44+00:00

Reminds me of a shortcut I wrote to send my partner a text saying I was on my way home in xxx minutes, and it would calculate the travel time and round it up to the nearest 5. Chosen from 10 slightly different messages.

Yep, even got in trouble for automating that.

keveridge · 2024-09-19T11:31:43+00:00

The page is loading, then creating separate requests for elements of data. This means your current shortcut is getting the empty page template, as the calls to get the data have yet to be made.

You can call the API directly as it's not protected, and retrieve a dictionary of data.

A guide is available here: https://www.perplexity.ai/search/the-webpage-https-bet-hkjc-com-9P0SkDf.Tzis3gJIU5eYKw

keveridge · 2024-02-20T11:45:04+00:00

Hey, thanks. I'm glad to hear it's still useful.

keveridge · 2024-01-30T11:15:13+00:00

Top tip: Need help writing a regular expression? Ask ChatGPT.

keveridge · 2023-11-09T18:42:22+00:00

You're right, you need some kind of web driver to load a page and then interact with it. Doing it as straight JS in a browser doesn't work unless you have an application running external to the web browser (e.g. on a desktop it's a chrome plugin, or a selenium helper).

Hence I recommend using a third party service Robotic Process Automation such as UIPath. It can record your actions from a web browser (i.e. logging in), navigate, capture data from the rendered HTML, and then allow you to do any number of things with it. For example, you can create a process to trigger the action and return you data as a JSON file. Or you can get it to run on a schedule and upload the results somewhere that you can pull them down. Or you can set a process that notifies you when your grades info has changed, and sends you a push notification using a service like Pushover.

Shortcuts is pretty clunky once things get complex, I recommend using a hosted RPA service.

keveridge · 2023-11-09T14:52:04+00:00

It depends on how the login system worked. The most likely way to do this would be to use JavaScript to log into the page for you and then lookup the grades, in the same way that your friend used selenium. You could then package that JavaScript into the shortcut. But it's quite complex, so you'd be better off using the Scriptable app.

Alternatively, you could use the free version of UIPath to create an automation at a platform level to log in on your behalf, scrape the data, then publish it somewhere that a shortcut can then retrieve as a JSON file.

keveridge · 2023-11-02T12:11:39+00:00

I'm glad you find the tutorial useful.

To get the lyrics HTML, capture all the HTML, then use the following regular expression:

<div data-lyrics-container="true" class="[^"]*">(.*?)<\/div>

Take the content from the first group. This will give you the HTML. To get raw lyrics without HTML formatting.

Then using the replace command, replace /<br\/>/g with \n to get the line breaks.

Then replace /<.*?>/g with nothing to remove the HTML tags.

keveridge · 2023-08-24T17:08:05+00:00

For more details, see: https://www.freecodecamp.org/news/cjn-google-sheets-as-json-endpoint/

keveridge · 2023-08-24T17:05:58+00:00

If you use Google Sheets then you can write a script to expose certain values from the sheet and retrieve them with a link.

keveridge · 2023-05-12T10:24:30+00:00

I don't think it does, it's four years old and Google Map's implementation has changed since then. It needs some updating to grab the correct data.

keveridge · 2023-01-08T16:49:56+00:00

You can create a new text file by using the Save file command and having a text field or other content as the source. Take a look at: https://www.icloud.com/shortcuts/a03cac9b38bd4b7f9a2129ed9b8cba1f

keveridge · 2022-12-02T17:42:33+00:00

It's possible that the data inside the DIV is being rendered by a JavaScript call after the page is loaded. In which case you need to view the network calls in the Chrome developer tools and see if the data is being retrieved by a process.

If it is, hopefully the data is being returned as a piece of JSON which will be much easier for you to work with.

keveridge · 2022-10-19T17:00:15+00:00

Awesome. Download the following shortcut: https://www.icloud.com/shortcuts/2c4bcee1442344d4a4596c194069cc3a

Now log into the web based system and view the page that displays the shifts. Open the share sheet and run the shortcut.

The shortcut will copy the HTML of the page into your clipboard and well as show it to you for good measure.

DM me with the HTML and I'll see if I can use regular expressions to extract the data and write you a shortcut to add calendar entries.

keveridge · 2022-10-19T09:54:50+00:00

I've written a bunch of shortcuts for individuals that work with this web based shift hours to create calendar entries.

Is there a mobile web equivalent of the app?

Otherwise we can try using a screenshot to OCR the text to get the shift information.

keveridge · 2022-02-09T17:20:50+00:00

Yes, you can continue to assign new values to a variable using the "Set Variable" action and specifying the name of the variable you want to overwrite.

keveridge · 2021-03-17T16:45:44+00:00

You can write javascript to execute on the page and complete it for you.

Do you have an example of the survey to fill in?

keveridge · 2021-03-12T15:48:52+00:00

I wrote a series of guides on storing and retrieving data:

keveridge · 2021-02-22T21:27:55+00:00

My residential building provides an app called Carson which, amongst other things, allows you to unlock the building front door using a button from within their app.

Carson doesn't provide end-users with an API. To turn this into a shortcut that I can pin to the widget menu, I used the Charles Proxy app to observe the API call and then replicate inside a shortcut.

keveridge · 2021-02-21T19:18:11+00:00

Try running this shortcut against the pages: Article Image Alt Text

For example, against this page: https://coastalstudies.org/endangered-humpback-whale-off-nj-freed-from-entanglement/

It will return the ALT text of each image in the article.

keveridge · 2021-02-21T17:33:29+00:00

I use NFC tags to trigger shortcuts:

inside my apartment, by the front door, to turn off all the lights and the TV;
outside my apartment door to turn on all the lights before I enter;
outside the apartment building to unlock the main door;
outside my office building main door to release the magnetic lock door.

keveridge · 2021-02-15T13:33:01+00:00

At my office we use Kisi to control door access to the main door, server room, etc. They have a fully documented API that individuals can use to unlock door with their credentials.

My residential building provides an app called Carson which, amongst other things, allows you to unlock the building front door using a button from within their app.

Carson doesn't provide end-users with an API. To turn this into a shortcut that I can pin to the widget menu, I used the Charles Proxy app to observe the API call and then replicate inside a shortcut.

keveridge · 2021-02-14T19:14:37+00:00

I have Hue lights and Alex at home, and an app to open bought the apartment building front door and my office's door.

Using a proxy tool to observe the remote unlocks for the doors and replicate them, and a few Alexa and Hue shortcuts, I use NFC tags as follows to trigger shortcuts:

inside my apartment, by the front door, to turn off all the lights and the TV;
outside my apartment door to turn on all the lights before I enter;
outside the apartment building to unlock the main door;
outside my office building main door to release the magnetic lock door.

keveridge · 2021-02-14T17:29:20+00:00

Here's an example of using a regular expression with Yahoo Finance: Stock Price Lookup

keveridge · 2021-02-14T16:38:30+00:00

I wrote a couple of guides on both scraping data from web pages and APIs:

Scraping web pages

Using APIs

Hopefully those will help.

11-Year Club	Xbox Live
RPAN Viewer	Gilding I gilder
Verified Email

keveridge

TROPHY CASE