Logging Python and Google Cloud

Question

I am creating an application which can be run locally or on Google Cloud. To set up google cloud logging I've used Google Cloud Logging, made a cloud logger and basically log using the class below

class CloudLogger():

    def __init__(self, instance_id: str = LOGGER_INSTANCE_ID, instance_zone: str = LOGGER_INSTANCE_ZONE) -> None:
        self.instance_id = instance_id
        self.instance_zone = instance_zone
        self.cred = service_account.Credentials.from_service_account_file(CREDENTIAL_FILE)
        self.client = gcp_logging.Client(project = PROJECT, credentials=self.cred)
        self.res = Resource(type="gce_instance", 
                    labels={
                        "instance_id": self.instance_id, 
                        "zone": self.instance_zone
                        })
        self.hdlr = CloudLoggingHandler(self.client, resource = self.res)
        self.logger = logging.getLogger('my_gcp_logger')
        self.hdlr.setFormatter(logging.Formatter('%(message)s'))
        if not self.logger.handlers:
            self.logger.addHandler(self.hdlr)
            self.logger.setLevel(logging.INFO)

    def info(self, log_this):    
        self.logger.setLevel(logging.INFO)
        self.logger.info(log_this)

I want to have this so that if it is running on the cloud, it uses the GCP logger, and if run locally, it uses python logging. I can either pass in as an argument ("Cloud", "Local") or make it intelligent enough to understand on its own. But I want the underlying logic to be the same so that I can log to cloud/local seamlessly. How would I go about doing this?

Wondering if (maybe) theres some way to create a local logger. And have those local logs parsed to GCP if running on the cloud.

By GCP Logger do you mean Stackdriver? What problem are you having? Is the issue that you want to detect that you are running in the cloud? Running in Google Cloud? Or do you want to support both local log files and Stackdriver? Stackdriver logging is available from your desktop, from inside Google Cloud and from almost anywhere. What is your criteria to determine log locally versus to Stackdriver? Edit your question with details. — John Hanley
– John Hanley, Commented Nov 18, 2019 at 15:06
@JohnHanley I want to use the python logger for all logging. And if I'm running on the cloud, pass those logs to Stackdriver. So Primary logger: Python, if cloud: Stackdriver. — DUDANF
– DUDANF, Commented Nov 18, 2019 at 15:17
@daudnadeem, I know the problem and I'm in discussion with Google about it. @JohnHanley, when you log thing in Cloud Run (or AppEngine) for example, like logging.debug, there is no level detected in stackdriver logging, the log are in grey because there are printed in plain text. If you use the Cloud Logging formater, it works well, but in local, your logs are in FluentD format (JSON, hardly readable). If you use Cloud Function, it works well, because Logger is well formatted in the Function Build step. — guillaume blaquiere
– guillaume blaquiere, Commented Nov 18, 2019 at 20:00

guillaume blaquiere · Accepted Answer · 2019-11-18 20:09:25Z

2

I coped with this issue and Google is aware of it (and also for Go). My helper do this:

I perform a request to metadata server. A get to http://metadata.google.internal/computeMetadata/v1/project/ with this header Metadata-Flavor: Google
- If I have a 404, I'm in local, and I set up my logger as local
- If I have a 2XX, I'm on GCP and I set up the logger to use FluentD format (GCP Cloud Logging)

Not perfect, but enough for me!

answered Nov 18, 2019 at 20:09

guillaume blaquiere

76.5k3 gold badges65 silver badges102 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

DUDANF Over a year ago

Is there any way to automate this process? I recently learnt it's bad practice for my code to know whether it's on GCP or not. Seems like my writing to python logger, the logs get passed to Stackdriver -> Global -> Python directly. I'm trying to figure out a way to write all logs to python. If on gcp, pass to stackdriver. Else keep in python.

guillaume blaquiere Over a year ago

I don't know where you read this, I'm interesting to dig into this topic. From my understanding, you never have to adapt your process according with the underlying infrastructure. However, for technical data, like log format, you can change the format without changing the business logic, which is the case here.

DUDANF Over a year ago

I actually attended google cloud next '19, and over there they had an "ask an expert" area. So I asked an expert about my logging issue and he said all you should do is: 1. Install stack-driver-logging on the machine 2. configure it. 3. it will pick up local logs and push them to stackdriver. Also, when you see your logs, do you look at them thru global>python or under the resource name e.g. <VMInstance-name>

guillaume blaquiere Over a year ago

Very interesting feedback. I was also in London last week. I didn't have this feedback from Google when I submitted them my implementation example. I dug into the Google Cloud Stackdriver Logging library and the code of the get_default_handler do something close to my proposal. I will loop again with Google to be sure

dain Over a year ago

Just to state the obvious: don't do this on every log event, but make sure to cache the results on startup. Also, outside of GCP this won't be a 404 but a DNS lookup fail so you'll need to handle an exception instead of an HTTP error code.

|

Collectives™ on Stack Overflow

Logging Python and Google Cloud

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related