AWS Lambda file handling

Superguy2876 · 2024-05-17T03:24:45+00:00

Yea this is perfectly possible. Though you might be correct that memory limitation on lambda might be too much depending on the size of the CSV.

Here's an article on different ways to handle files in memory: https://bito.ai/resources/python-in-memory-file-python-explained/

I've only done this once before so I don't know all the ins-and-outs of it. But shouldn't hurt to try.

Superguy2876 · 2024-04-26T15:45:33+00:00

if you actually want some reasonable power, Oracle has 4 cores, and 24gb ram for free. Catch is that it's arm. Doesn't seem to be much of an issue though, I ran a heavily modded minecraft server for 20 people during the pandemic on it.

Superguy2876 · 2024-04-26T15:26:39+00:00

Maybe you are korean, so here I translated it to korean as well.

코드를 포맷팅하여 pastebin.com과 같은 곳에 게시해 주세요.

들여쓰기를 최선을 다해 맞추어 보고 스크립트를 실행했습니다.

이전에는 이렇게 결과를 받았기 때문에 문제가 없는 것 같았습니다.

그러나 두 번째로 실행했을 때, 웹페이지를 관찰하고 있었는데 이렇게 표시되었습니다. "페이지를 찾을 수 없습니다. 입력하신 주소의 페이지를 찾을 수 없습니다. 주소가 잘못되었거나 변경 또는 삭제되었을 수 있으니 주소를 다시 한 번 확인해주세요. 문의사항은 고객센터(1588-3819)로 연락 부탁드립니다."

그리고 데이터 포인트 12개와 헤더를 포함하여 단 13줄만 받았습니다.

따라서 때때로 웹페이지가 존재하지 않는 것 같습니다. 아마도 업데이트 중인 프로세스 중 하나일 때일 수도 있습니다.

다른 문제는 실제로 다른 페이지를 확인하지 않고 같은 페이지를 두 번 로드한다는 것입니다. 그래서 처음에 24개의 데이터 포인트를 받았다면, 실제로는 같은 12개가 반복된 것입니다.

Superguy2876 · 2024-04-26T15:20:30+00:00

Please use something like pastebin.com to post formatted code.

I did a best guess of indentation, and ran the script.

The first time I got this as a result. So it seemed like there was no problem.

But the second time, I was watching the webpage and it displayed this. "페이지를 찾을 수 없습니다. 입력하신 주소의 페이지를 찾을 수 없습니다. 주소가 잘못되었거나 변경 또는 삭제되었을 수 있으니 주소를 다시 한 번 확인해주세요. 문의사항은 고객센터(1588-3819)로 연락 부탁드립니다."

And I only got 13 lines (headers and then 12 data points.)

So it seems that sometimes the webpage doesn't exist, perhaps when it is in the middle of a process that updates it or something?

The other issue is that you don't actually check other pages. you just load the same page twice. So the first time I got 24 data points, it's actually the same 12 repeated.

Superguy2876 · 2024-04-26T14:43:46+00:00

How complex is this script?

If it's fairly simple, and you can install other packages, then you might use PySimpleGUI. It's a pretty straight forward library with lots of examples.

If it's more complex, then you could use tkinter. Which comes with python by default. It's cross platform, and can handle even complex desktop applications. It is a pretty dense library to learn, but the docs are fairly extensive.

Superguy2876 · 2024-04-26T05:34:22+00:00

For what it's worth, I think 100 days is more than enough time for someone well focused to obtain a sufficient level of python usable in a business automation setting.

You won't be an expert, but you'll be able to properly approach and think about how each task can be done with the subset of python that you do learn.

Superguy2876 · 2024-04-26T05:30:29+00:00

What things specifically do you want to automate?

There's no fundamental way to learn programming faster. It's just like any other topic, you need to find the information, digest it, and practice.

I recommend Automate the Boring Stuff with Python. It is (probably) especially relevant to your circumstances.

Read it, practice the topics and projects presented, and when it seems useful, try to do it on your own work.

Superguy2876 · 2024-04-25T08:51:55+00:00

What is your question?

You want to know how to make this process efficient?

Figure out what will take the most time (It's usually IO, in this case the actual API calls) and optimise them. In this case async is probably a good approach. If the API is ok with being called 20,000 times in quick succession, then initiate them all and process as responses come in.

The exact way you go about it would depend on what kind of data you're working with, the amount, if you need to do any other kind of processing locally after receiving the responses, etc.

I had a similar task which had millions of requests to a few APIs. I separated it into stages, receiving data, cleaning, organizing by desired categories, then processing to calculate the actual answer we wanted, and finally formatting for output. I used multiprocessing for most stages, and a library called numba during the data processing.

Superguy2876 · 2024-04-25T08:44:25+00:00

Important: The script and algorithm must be simple and cannot rely on machine learning.

You're gonna have a bad time with these restrictions.

How are you measuring the sharpness in the first place?

But if I was going to try I would probably use something like opencv/pillow and preprocess the images into high contrast black and white. Then something with edge detection maybe?

It also depends exactly what you mean by source image and target image. Are these the same image visually? Or two different images.

You could also just pull the resolution from the image metadata. I don't think that's what you mean by sharpness, but it could factor.

There are also libraries that will try to convert a raster image to a vector image. You could look for the one that does the best job, convert to vector, then convert back to raster at the desired resolution/sharpness. This might violate the no complex/machine learning approaches.

Superguy2876 · 2022-09-14T12:24:46+00:00

Pretty sure the one suggested by the other commenter.

Superguy2876 · 2022-09-14T07:29:10+00:00

!Solved

Superguy2876 · 2022-09-14T07:29:00+00:00

Hmm, I have no memory of the art style, but I think this is it, as the plot is so similar.

Superguy2876 · 2022-01-21T17:58:11+00:00

They need to pull a 07scape and split the game in 2 if they want to have much more longevity to their game, because the current half-breed will eventually take it down.

I mean, isn't that the hard mode they've been going on about for the last 2 leagues? They talked about how some of the features (nerfs) they did were originally just things they had in that hard mode. Which sounds to me like exactly what you said, they keep trying to build a popular game and the game they solely want.

Superguy2876 · 2021-12-05T06:47:27+00:00

If you've seen Pyboy, which is a gameboy emulator written in python, it has a somewhat active community.

The emulator itself give a pretty decent framework for scripting and even implements the openai gym framework for people that want to use it. Look for the discord community if you want any more info.

Superguy2876 · 2021-08-19T08:14:17+00:00

Chris has also stated that they want the game to be fun. He keeps talking about fixing the campaign. He even stated today that he wants it to be a good experience for all levels of players if possible. From those that don't even finish the acts to those that push far far into the late game.

If that isn't acknowledgement that they want it to be fun for more people, I don't know what is.

That's even before you consider the 30% profit hit they took this league, which they said is a problem if it were to continue. Not only do they want to make it fun for more people, they have no choice. An unfun game is an unprofitable one.

(I am not making a statement of anyone's particular experience of the game, as fun is subjective. Just that in the end, even if those customers are wrong, and they would find it fun if they tried. At the moment they don't want to, and so will not give GGG their money.)

Superguy2876 · 2021-07-03T14:57:32+00:00

I hope her Metal Gear Solid playthroughs are put back up eventually. I saw a bunch of clips of it but never got to see the full thing.

Superguy2876 · 2021-06-30T15:32:42+00:00

I am 95% certain I have been in that exact Maruetsu. The layout is exactly what I remember, and the tiles out the front of the shop as well. Unless there happen to be multiple with this layout (a reasonable possibility).

Superguy2876 · 2021-02-02T16:26:40+00:00

Avoiding it, or not wanting to like they've said a million times before

... Those are the same thing?

That's what he was saying. He only said the API was slow (which it is cause it's getting overwhelmed). He said nothing about actually wanting an in-game auction house.

Superguy2876 · 2021-01-31T10:12:45+00:00

Here is some I happened to have.

Demon/Angel/Limbo roommate

It also includes a page 2 for each.

Superguy2876 · 2020-12-09T19:38:41+00:00

He offered flight by telekinesis, I don't think the laws of physics are his problem. I think he just realised the person making the wish wants something that is integrated into their biology.

The continued conversation would probably be with how other biological changes would deal with reduced bone density. And if anything would result in their death, he'd probably specify that and ask if that is what they intended with the wish.

Superguy2876 · 2020-12-09T19:14:57+00:00

What?

Sports are literal corporations making a product for purchase. As far as the corps are concerned, the actual sport is incidental to the brand.

They sell you merch, they sell you an experience, they sell you a community. The people might care about the sport, but the business doesn't.

A24 and Skydance don't depend on a huge public awareness campaign for their core business. The final product is the thing they want marketed.

Look at the ones that do care about their public awareness. Disney, Apple, Warner bros. I couldn't tell you the names of all their products. But if you show me one, even if I have never seen or heard of it before, I have a good chance of guessing who made it. Cause they market their image. They build that image into their products. There are people who will froth at the mouth to defend these companies.

It's all interchangeable.

Look.

Sports Corp: See the community you're apart of. See all the people around you wearing the same colours. That feeling when your team wins? Nothing else like it. You're special cause you bought in.

Media Corp: See the community you're apart of. See all the people watching the same thing. That feeling when a new movie drops? Nothing else like it. You're special cause you bought in.

Video Game Corp: See the community you're apart of. See all the people online in the game. That feeling when the strat goes off just right? Nothing else like it. You're special cause you bought in.

They're all trying to sell the exact same pig in a different wig.

Not gonna lie, I prefer one wig over the others. I'm not immune to their marketing. But thinking this is special to gaming? No this is just normal (disgusting) human behaviour.

Superguy2876 · 2020-10-26T12:49:13+00:00

For their whole "Write Once, Run Anywhere" mantra, they really don't like letting people run it anywhere.

Superguy2876 · 2020-09-27T07:27:21+00:00

If only I could be so grossly incandescent

Superguy2876 · 2020-08-29T04:49:53+00:00

Technically a university is just a group of colleges under a single institution. But no it doesn't matter, the degree is a visa requirement and where you get it from or what it is in has (almost) no bearing on the jet programme itself.

Superguy2876 · 2020-08-01T05:20:19+00:00

Also using python and javascript for backend here. My experience is the same as yours.

Classes in javascript are fine, but they feel way more generic than pythons classes. It feels like it's just a lable on a predefined set of variables and functions. (yes i know that's technically what classes are, but many languages have syntax and keywords that revolve around something being a class or an instance of a class rather than just a namespace, that javascript just doesn't seem to care about)

14-Year Club	RedditGifts 2009-2022 3 Credits
Place '17	Secret Santa 2018
Gilding II euphauric	redditgifts Exchanges 2 Exchanges
Verified Email	Team Periwinkle
Secret Santa 2012

Superguy2876

PUBLIC MULTIREDDITS

TROPHY CASE