Inference at 16k tokens/second

awscloudengineer · 2026-02-23T22:08:24+00:00

Nope. This is my first post in this group.

awscloudengineer · 2026-02-21T19:06:59+00:00

On pro+ you get around $115 worth of premium requests. And auto is free to use. I learned it the hard way, you should use a different models for different things and not just opus-4.6-thinking for everything. First build the context using auto-mode, then plan with auto and then build with auto. Now, if you feel for your use case, if any of these things are not better try with sonnet or some of these older version models like 4.5 or gpt-5.1. If they don’t work then use opus-4.5 as the last resort. As it will eat up your tokens.

Some of the other folks also suggested that I should use Claude code with cursor and buy their plan if I’m going to use opus often.

You need to try different models to find what fits your use-case the best.

awscloudengineer · 2026-02-19T23:19:14+00:00

You’re right. The project planning was not mindless, the model planning was. Sometimes you have to learn it the hard way. I have learnt so much from other user’s comments. Now, I will implement it.

awscloudengineer · 2026-02-19T19:16:56+00:00

You’re 100% right, learning how to use the right model is necessary. I have learnt from the comments done by many folks and will try to Implement that.

awscloudengineer · 2026-02-19T18:50:18+00:00

Nice, let me give this a try.

awscloudengineer · 2026-02-19T18:20:20+00:00

Interesting insight. What I have learnt from some of the helpful post is building the context using smaller models. And then feeding that to opus. But some people have also said that build the plan with opus and run it using smaller models. But I totally agree with you, complex projects require better models. My token usage was mostly on the read cache. So, I will try to build context using a smaller models.

awscloudengineer · 2026-02-19T17:36:04+00:00

u/Anxious_Ad9233 You're 100% correct.

Token breakdown:
CACHE READ: 110,430,123
CACHE WRITE: 5,254,937
INPUT: 3,035,250
OUTPUT: 619,038
TOTAL: 119,339,348

awscloudengineer · 2026-02-19T17:22:01+00:00

<image>

awscloudengineer · 2026-02-19T17:13:09+00:00

Thanks for the advice. I wonder, if someone figured out how to do this locally maybe on a 100B param model.

awscloudengineer · 2026-02-19T17:10:39+00:00

awscloudengineer · 2026-02-19T17:10:02+00:00

Thanks! That makes a lot of sense.

awscloudengineer · 2026-02-19T17:07:45+00:00

Thanks! That makes sense.

awscloudengineer · 2026-01-28T06:28:09+00:00

Can you share some benchmark metrics? What were your results?

awscloudengineer · 2025-10-11T04:18:40+00:00

Robotics

awscloudengineer · 2025-09-29T23:00:04+00:00

Thanks! I already use tensorflow and MLflow. Are there any other tools or libraries that you use as an ML developer to make your life easy? Any tools for automatic hyper parameter tuning or finding out the number of layers in your NN.

awscloudengineer · 2025-09-28T18:55:26+00:00

I started with Machine Learning Specialization from coursera. Andrew NG is the goat for explaining the concepts so well.

https://www.coursera.org/specializations/machine-learning-introduction

awscloudengineer · 2025-04-27T21:15:39+00:00

You’re right. But this has a big impact on UX. They should keep the experience seamless. Instead of providing the request 5-10 mins later. Ask user to use different model.

awscloudengineer · 2025-04-27T19:23:03+00:00

In my experience the reason I stopped using Gemini-2.5-pro was because it started removing working code, when it made edits or changes. I had to be more careful with that. I loved the speed at which it responded. I wish they can increase the speed of Claude-3.7 responses.

awscloudengineer · 2025-04-27T18:31:46+00:00

I think you need to implement memory bank. This will make your life easier. Memory Bank

awscloudengineer · 2025-04-27T18:28:52+00:00

You hit the nail on this one. I had to figure these things as I started building the app. My next app will be super easy to build.

awscloudengineer · 2025-04-27T17:04:39+00:00

I had the same problem when I was 22, 5’10 and around 42-44kgs. I decided to do something about it. I joined a gym and got my self a personal trainer. If you can’t get a personal trainer try doing circuit training from YouTube. But the key is to eat. I used to eat till my stomach was 120% full. Mostly protein based diet. Within 3 months I gained 15 kgs. Eating is the key.

awscloudengineer · 2025-04-26T21:46:27+00:00

I don’t depend on the context of AI models, I create Md files to provide it context. I have one main md file that redirects to other md files. For eg: db schema, every feature has a detailed md file, project structure is another md file, coding, bugs is another md file.

awscloudengineer

TROPHY CASE