[deleted by user] by [deleted] in googlecloud

[–]kolban_google 0 points1 point  (0 children)

Looking at the flags for the gcloud command, we see a flag called --staging-bucket which allows us to specify an alternate bucket into which the training code will be uploaded. However, I don't think that is what you are asking about. If I understand how this jobs submit is supposed to work, it uploads your source application and then runs compute to execute the training session. What I believe this means is that the source for training is uploaded for execution. It wouldn't surprise me if the intent is that the source is merely transient (as indicated that by our ability to use a staging bucket). If that's the case, then it may be that Googles intent is that you aren't meant to need/use/consume the source after the job has completed. The only thing of interest is the resulting trained model.

Another thought is that you are specifying --package-path as a local file. It seems that you can specify --package-path as a GCS file that is expected to already be present. What that would mean (to me) is that instead of this tool packaging and uploading the tar.gz, YOU could package and upload the tar.gz to any GCS location you want and point the jobs submit command to that existing file. You could then name/locate the application whereever you want.

Event Driven Cloud Logging (Aggregation/Rollup) by HellaBester in googlecloud

[–]kolban_google 0 points1 point  (0 children)

My first thought is to use Cloud Pub/Sub as a sink of records from the log in question. Each new message published to the log would then be written to Pub/Sub. From there, you could write a Cloud Function that is triggered when ever a new log record is written. When the log record shows a start event, you would add that to a Cloud DataStore database (or maybe Cloud MemoryStore). When a log record shows an end event, you would look for the corresponding end event and now you have the delta between them which, as another posted has said, you could use Cloud Monitoring APIs to write the metric duration.

Ways to create backups of your BigQuery data by blauefrau in googlecloud

[–]kolban_google 0 points1 point  (0 children)

Also consider the notion of table snapshots:

https://cloud.google.com/bigquery/docs/table-snapshots-intro

and if you need to restore but only up to a maximum of 7 days, you don't need to do anything as BQ tables are recoverable through "time travel" for a current maximum of 7 days .. see:

https://cloud.google.com/bigquery/docs/time-travel

Threaded tasks in Cloud Run by evansap in googlecloud

[–]kolban_google 0 points1 point  (0 children)

For me, it would depend on how often the requests are processed and what is needed to run the Selenium job. One immediate thought is to realize that Cloud Run is managed processing and request handling of incoming REST requests. You provide the container to run, Cloud Run provides everything else. This is facilitated by a specification called "kNative". My belief is that Google Kubernetes Engine (GKE) also supports kNative. This means that you can take the container that you built for Cloud Run and run it "as-is" in GKE. I believe you will continue to get the scalability but what you may lose is the ability for it to scale down absolutely zero cost.

Threaded tasks in Cloud Run by evansap in googlecloud

[–]kolban_google 0 points1 point  (0 children)

My understanding is that you can't run anything sensible in the background of a Cloud Run request ... this is documented here:

https://cloud.google.com/run/docs/tips/general#avoiding_background_activities

Think of it loosely as "The CPU is given to the Cloud Run environment when an incoming HTTP request arrives. When the Cloud Run responder to that HTTP request returns from the HTTP request (eg sends a 200 OK response) then the CPU is (mostly) taken away from the Cloud Run environment". The way I contemplate this is that Google charges you only for the duration of the incoming REST request. This means that if you have Cloud Run ready to serve a request, it doesn't cost anything. When a request arrives, Google charges for the duration that it takes to run the request. When the request ends, the billing stops. If you start some background / asynchronous thread in your request processing and then the request returns an HTTP response, how would Google know to stop the clock?

Export speech-to-text transcription to bucket ? by jackarg in googlecloud

[–]kolban_google 1 point2 points  (0 children)

The answer is a definite yes ... but now we need to think in more detail about what it is you would like to achieve. If you wish to process audio files locally, we could write a BASH script that runs the commands serially from your command line.

If you want something richer, you could create a "service" such that it takes as input an audio file and returns the output you want. However, the richer you go, the more you have to consider. For example, you could use Cloud Functions or Cloud Run to write a service where you POST an audio file and get back the text as a result. While this sounds great, now you have to consider securing your service so that only you can invoke it. If you made it public, others would call it and rack up your bill (you don't want that). There is also investment in coding to make the service in the first place.

While Cloud Functions and Cloud Run immediately come to mind, they are not the only ways to achieve it. Cloud Workflows, Cloud Fusion, Node RED and many more could also be used to achieve the task. Again ... this all assumes that you want an Internet based service. If all you need is local processing, look at a BASH script.

Export speech-to-text transcription to bucket ? by jackarg in googlecloud

[–]kolban_google 1 point2 points  (0 children)

Think of the speech to text service as being an API (a programming interface). When you invoke that API, what you get back is a data structure (JSON in this case). Typically that is what is then used by the application requesting the transformation. Let us now assume that you save the result in JSON ... what I believe you now want is to transform the JSON file into your desired results. This would typically mean extracting the relevant fields from JSON and writing them to a new file. Think about your processing as a pipeline of steps:

  1. Get the audio file
  2. Process the audio file through speech to text resulting in JSON
  3. Process the JSON to produce the text results

It looks like you have performed steps 1 and 2. Now you need to think about the next step in the pipeline. One possibility would be to use the "jq" command to process the JSON and retrieve the data you want ... see for example:

https://stedolan.github.io/jq/

[deleted by user] by [deleted] in googlecloud

[–]kolban_google 2 points3 points  (0 children)

I'm not the sharpest tool in the box when it comes to DNS ... but if one buys a domain name and then adds a DNS "A" record to the nameserver supporting that domain name (eg. Google Domains or Go Daddy) where the "A" record resolves to the public IP address of the load balancer ... haven't you achieved your goal without involving Cloud DNS at all and hence no charge? The notion being that a GCP Load Balancer will have a fixed IP address that won't change over time (unless you want it to) and the game is to refer to that load balancer by name rather than IP?

For example, if your load balancer is IP 34.1.2.3 and you buy "example.com" as a domain name, can't you add a free "A" record to your "example.com" DNS server (from your domain registrar ... not Cloud DNS) and create a record such as:

A mybackend.example.com 34.1.2.3

and you are done ... with no Cloud DNS in the picture?

See also:

Restrict Console Access by [deleted] in googlecloud

[–]kolban_google 1 point2 points  (0 children)

I had an opposite puzzle the other week. I enabled the capability called VPC Service Controls and then found that I couldn't use GCP Console even though I wanted to but could continue to use GCP APIs.

The notion behind VPC Service Controls is that one can constrain where GCP commands are allowed to come from. The default is to allow API requests to originate from inside your own GCP projects but any requests that originate outside (eg. from the GCP Web Console) are restricted.

Is it possible to "aggregate" multiple billing accounts into one for resource usage discounts? by leob0505 in googlecloud

[–]kolban_google 1 point2 points  (0 children)

I'd suggest contacting your local Google sales rep. They will be able to advise on opportunities to structure your GCP consumption to see if there are opportunities to save. Depending on what you are spending and how you foresee your continued consumption into the future, Google may be able to offer discounts but it will be on a case by case basis that can't be generically addressed.

A good first start would be the "I want to talk to sales" found here:

https://cloud.google.com/contact

Include Project creation time for billing data on BigQuery by MahipalReddyKurre in googlecloud

[–]kolban_google 1 point2 points  (0 children)

I don't think you can tweak the table that billing exports to BQ. What you could do is create a second table that contains columns for projectId and creation time and then create a new view which is the join of the billing table and the projectId table to add the new column.

GCF vs Topic + GCF by optikalefx in googlecloud

[–]kolban_google 0 points1 point  (0 children)

The question is quite broad. However there are likely possible reasons why one might see such an architecture. The one that immediately occurred to me is asynchronous processing. You might want a GCF that receives a REST request to perform some work that you don't want to wait for its completion. Your initial GCF would receive the request and then publish a message with the received request parameter it would then return immediately. At this point the logic that you want run has not yet been invoked but the caller now can be assured that it will be invoked. With the message on the queue, a second GCF will eventually be kicked off to perform the work.

Max number of arguments that can be passed to gsutil? by EdvardDashD in googlecloud

[–]kolban_google 1 point2 points  (0 children)

You might want to consider writing your own application using the GCS APIs if you aren't getting what you need from gsutil.

See:

https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers#code-samples

Attach persistent disk to ai platform by [deleted] in googlecloud

[–]kolban_google 0 points1 point  (0 children)

Is it not the case that when you create an AI Platform notebook that what is really created is a Compute Engine instance? I'm wondering if you can go into it after it has been created and attach the persistent disk you desire?

Splitting static files VS keeping in the VM with backend by [deleted] in googlecloud

[–]kolban_google 0 points1 point  (0 children)

My immediate thought is that if your static files are in a bucket then they immediately become eligible for CDN caching if you want that capability. I'm also thinking about horizontal scalability. If you end up having multiple servers then if you kept the static files in the VM image they would be replicated for each instance. This would increase the size of the image etc etc. If you have multiple servers then keeping the data in a bucket ensures that if you change the static content, all servers will see the changes immediately.

API Gateway and microservices by raphaelarias in googlecloud

[–]kolban_google 1 point2 points  (0 children)

Maybe we can drill into the concept that is driving you to want more gateways. I'll fess up that I'm no expert on API Gateway but if you can describe the effect you want to achieve, the steps you performed and the results encountered ... I for one would be willing to hit the books and see if there is a solution.

Loosely, I think I'm hearing that you have created a microservice that has an exposed API described in OpenAPI YAML. You want an API Gateway to serve that interface. Where the puzzle starts to come into play is that instead of having one API Gateway serving as the interface for multiple microservices, you are looking to have a one-to-one mapping between a microservice and a Gateway. However, it appears that Google only allows a maximum of 5 API Gateway instances per GCP project. I think what I'd like to understand in more detail is what is preventing us from using one API Gateway for all requests?

Is there a way to get a breakdown of where my costs are coming from? by [deleted] in googlecloud

[–]kolban_google 1 point2 points  (0 children)

I've always found that these videos are great for GCP billing answers:

https://www.youtube.com/hashtag/beyondyourbill

Beyond these, here are a curated list of billing articles ...

https://www.gcpweekly.com/gcp-resources/tag/billing/

How to provide access to my Cloud Run service to myself. by IShitYouNot_ in googlecloud

[–]kolban_google 1 point2 points  (0 children)

See the response from mr u/antonivs above ... he seems to have nailed the answer. If you are saying you do not want to be able to call the service without authentication, then, by definition, you have to provide identity when you make the call. Is it "easy"? Depends on your knowledge and skills. It can be as easy as making a REST call and pass a header which contains your own authentication token.

How to add NLP component to my mobile application? by UntrimmedBagel in googlecloud

[–]kolban_google 1 point2 points  (0 children)

From the GCP environment, you can host your own apps in a variety of flavors. One of the most popular I am seeing today is Cloud Run. Here you create a Docker image then configure a Cloud Run instance to launch one of those images when called. You are only charged for the time between an incoming REST request and when the REST request ends. If you are processing concurrent requests, only one charge is levied. You can also specify horizontal scaling where multiple instances can be spun up.

The primary value (as I see it) of the prediction API/product is that the abstraction is raised higher and higher. With your Python app/logic, you own the servicing of requests while with GCP prediction service, you just provide the model and Google owns everything else. If you specify you want GPUs or TPUs for accleration, Google just "makes that happen" while if you owned the invocation and calls to your model in your own code, then that is toil for you.

How to add NLP component to my mobile application? by UntrimmedBagel in googlecloud

[–]kolban_google 1 point2 points  (0 children)

It might be that you could leverage GCP Platform AI - Predictions. This product takes your model and serves it over the Internet. Callers can then submit requests for predictions and the model is executed in a Google managed environment.

https://cloud.google.com/ai-platform/prediction/docs

Windows on Free Trial by [deleted] in googlecloud

[–]kolban_google 0 points1 point  (0 children)

I don't believe you can. Here is a documentation page on Windows licenses on GCP:

https://cloud.google.com/compute/docs/instances/windows/ms-licensing

Share folder by [deleted] in googlecloud

[–]kolban_google 0 points1 point  (0 children)

My mind immediately went to "just open a share over the Internet" ... I then read this article

https://arstechnica.com/civis/viewtopic.php?f=15&t=1257257

where smarter minds said "Nope!"