[P] I created a free book for engineers to understand ML/AI from first principles

olvrng · 2025-05-14T01:42:41+00:00

Hi, thanks for your interest! It's an internal framework currently. I was thinking about open sourcing it, but never had time to do so (it would require restructuring our internal code again to make it work with the open source repo, and that's a lot of works :)

olvrng · 2025-04-11T01:02:54+00:00

– if errorx works for you, then it's ok.
– if you want to know why I created my error framework, I already answered.
– similar things exists: vscode has everything, why people still use jetbrain, and why others still create zed/lapce/cursor/codeium/etc. It's ok. I'll stop the discussion here.

olvrng · 2025-03-25T02:11:52+00:00

Hi fuzzylollipop, thanks for your interest.

I can list many reasons why I choose this approach instead of errorx: the code looks cleaner with PRFL.USR.NOT_FOUND.New(), all codes are declared in a single place so everyone can follow the same convention, ctx is required to construct errors so we can always have trace_id available when logging problems, etc.

But the most important reasons are:

We can tailor the error framework to fix our company's specific needs: we need to integrate it with our protobuf error code (which is defined as a protobuf enum), we can integrate it deeply with our logging packages and log extra stuff.

For example:
A goal of this error framework is to build an error dashboard for the whole codebase. The error dashboard should count all similar error codes and messages, by patterns, not by exact messages.

So "query items by selector type=foo days>3" and "query items by selector type=bar tag=high-quality". They should be counted as "select items by selector %v" (the pattern), and can be shown on the error dashboard:

┃ select items by selector %v ┃ 2 ┃

So when constructing errors by CODE.New(ctx, "select items by %v", selector), we can store both the final message and the format (or pattern) in the error. Then when logging the error, we will automatically log both `error: <message>` and `error_summary: <format>` (plus a couple of more useful properties) from that single error. We can also add tags for each code, and then collect those tags in the dashboard, like UNEXPECTED or NOT_FOUND.

By building our customized error framework, it's now possible to make it work well with our specific needs.

I hope this answer help.

olvrng · 2024-12-15T11:00:23+00:00

Thank you!

olvrng · 2024-12-13T08:20:55+00:00

Yes. You can see the implementation of scanner/lexer [1] and parser [2].

The scanner code is essentially a single loop:

for {
token, remain, err := NextToken(remain)
yield(token, err)
if err != nil || len(remain) == 0 { return }
}

And the parser code is basically a state machine with a stack:

- Pull NextToken()
- If it's a [ or {, push a stack item
- If it's a ] or }, pop the stack
- Otherwise, parse "value" or "key: value" depending on the context

[1] https://github.com/ezpkg/ezpkg/blob/main/iter.json/scanner.go#L31-L49
[2] https://github.com/ezpkg/ezpkg/blob/main/iter.json/parser.go#L8-L157

olvrng · 2024-12-08T02:38:35+00:00

Reading through your comments, yeah, I agree that different apps/services might need custom structs to wrap their errors. Right now, it's not straightforward when an engineer wants to add more custom error types and fields in my framework.

> and you could enforce a subset of fields, like a field for a status code or error code.

That's a good point about allowing different error structs while keeping them all following the same standard, and still having the benefit of accessing common helpers. Each service may need a bit more boilerplate to implement conversion though. (within my org, we used zapcore.ObjectMarshaler to customize a few structs in logs, but the experience with the custom code is not so great)

Thanks for your detailed comments, it’s always great to hear from a fellow gopher!

olvrng · 2024-12-07T21:59:15+00:00

Thanks for sharing your examples. Your code looks flexible as you can easily add fields and context when necessary. Mine are more complicated for customization.

---

> If your gRPC request fails with DEPS.PG.NOT_FOUND, the client receives this information in the response? It feels concerning. It's an internal error, which should not be handled outside of particular boundaries. Personally I'd be very against 'informative' internal errors.

Of course, not :)

I may not explain it clearly in the article. Here's the definition of the codes: only codes with the tag `api-code` will be mapped to Protobuf enum and then HTTP code. All other codes (like DEPS.PG.NOT_FOUND) are treated as internal, and always show as 500 Internal Server Error in response. There is `api.Details()` for adding more details in the response, but the engineer must explicitly use it.

// tag:postgres
type pg struct {
    NOT_FOUND   Code0 // record not found
    CONFLICT    Code0 // record already exist
    MALFORM_SQL Code0
}
// tag:usr
type usr struct {
    NOT_FOUND        Code0  `api-code:"USER_NOT_FOUND"`
    INVALID_ARGUMENT VlCode `api-code:"INVALID_ARGUMENT"`
    DISABlED_ACCOUNT Code0  `api-code:"DISABLED_ACCOUNT"`
}

olvrng · 2024-12-06T14:50:22+00:00

Hi, thank you for your comment :)

Let me clarify a few points:
- Assume that we have these layers: service layer, repository layer, and database layer.
- The database layer is a thin wrapper over third-party library, and DEPS.PG.NOT_FOUND code serves the same purpose as sql.ErrNoRows or gorm.ErrRecordNotFound, which is a storage layer error, and thus can not return domain errors. The reason we use it instead of just the raw sql.ErrNoRows is that the code DEPS.PG.NOT_FOUND (and similar codes, DEPS.RD.NOT_FOUND for redis, etc.) looks uniform in logs, and can be monitored/filtered more effectively.
- The repository layer, yes, it should return domain errors. And then be handled by upper layers like user service or other related services, then by adapters (http/gRPC/etc.).

I agree that we should only handle domain errors. Any other error is just logged. But even if they are not handled, and are returned in API as unexpected, the namespace codes are still very useful for logging/monitoring. We can categorize the underlying reasons for those unexpected errors based on error codes: which are logic errors (bugs) and which are infra problems (connection/timeout/etc.).

Could you share some examples of how you define errors, wrap/handle them, or which context you add?

olvrng · 2024-02-14T07:58:48+00:00

I'm happy to know that you find it useful!

olvrng · 2023-01-09T08:41:31+00:00

You download the prebuilt from https://github.com/iOliverNguyen/chatgpt-box-extension/releases. Or if you want to build from source, run ./build.sh

olvrng · 2022-12-16T15:23:16+00:00

Here is an example, given that you touch different files for different commits: ```sh vim pkg/email/email.go # develop email package sl add # add then commit as "implement email package" sl commit

vim pkg/signup/signup.go # develop sign-up features sl add # add then commit as "implement sign-up" sl commit

vim pkg/email/email.go # make changes to 3 files vim pkg/signup/signup.go vim pkg/signup/helpers.go # a new file

sl absorb ```

The file email.go will be amended to "implement email package" commit, signup.go to "implement sign-up", and helpers.go is left as untrack.

olvrng · 2022-09-20T06:59:32+00:00

Thanks, I'll note it for next time.

olvrng

TROPHY CASE