Logs uploader (near real-time)

Logs Uploader is a feature that enables an automatic export of CDN resource logs to your storage in near real-time.

Exported logs contain information about user requests sent to cache servers and pre-cache servers (if you have the Origin shielding feature enabled).

Info

This is a paid feature. To activate Logs Uploader, contact the Gcore support team.

Logs uploader settings

In this section, you can find general information about log settings, statuses, and how to configure logs exporting for different storage types.

Log format example

"$remote_addr" "-" "$remote_user" "[$time_local]" "$request" "$status"  
"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent"  
"$edgename" "$scheme" "$host" "$request_time"  
"$upstream_response_time" "$request_length" "$http_range" "[$responding_node]"  
"$upstream_cache_status" "$upstream_response_length" "$upstream_addr"  
"$gcdn_api_client_id" "$gcdn_api_resource_id" "$uid_got" "$uid_set"  
"$geoip_country_code" "$geoip_city" "$shield_type" "$server_addr" "$server_port"  
"$upstream_status" "-" "$upstream_connect_time" "$upstream_header_time"  
"$shard_addr" "$geoip2_data_asnumber" "$connection" "$connection_requests"  
"$http_traceparent" "$http_x_forwarded_proto" "$gcdn_internal_status_code" "$ssl_cipher"  
"$ssl_session_id" "$ssl_session_reused" "$sent_http_content_type" "$tcpinfo_rtt" 
"$server_country_code" "$gcdn_tcpinfo_snd_cwnd" "$gcdn_tcpinfo_total_retrans" "$gcdn_rule_id" 

It’s OK if you find a field that’s not listed in the example. We occasionally add new fields to the end of the line. If some fields are added to logs, you’ll receive an email about the update.

Log fields

The following table contains a complete list of available log fields. Fields formatted in italics relate to our internal CDN system, so you can ignore them.

You can check other fields—they can be helpful for traffic analysis or statistics.

View log fields

Field	Log value example	Description
$remote_addr	0.0.0.0	User’s IP address
$remote_user (internal system variable)	-	Username used in Basic authentication
[$time_local]	[26/Apr/2019:09:47:40 +0000]	Local time in Common Log Format
$request	GET /ContentCommon/images/image.png HTTP/1.1	HTTP method, requested file path, and HTTP version
$status	200	Response status code from a CDN server
$body_bytes_sent	1514283	Number of bytes sent to a user, excluding the response header size
$http_referer	https://example.com/videos/10	Referrer – a URL requested by a user
$http_user_agent	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36	User agent that was used to send a request (browser or other application)
$bytes_sent	1514848	Number of bytes sent to a user
$edgename	[dh-up-gc18]	CDN server that forwarded the requested file
$scheme	https	Protocol (HTTP or HTTPS) of a request
$host	cdn.example.com	Requested hostname of a CDN resource
$request_time	1.500	Request processing time in seconds (accurate to milliseconds); time elapsed between the first bytes of a request being processed and logging after the last bytes were sent to a user
$upstream_response_time	0.445	Number of seconds (accurate to milliseconds) it took to receive a response from an origin. In case of multiple responses, commas and colons are used
$request_length	157	Request length (including request line, header, and request body)
$http_range	bytes=0-1901653	File fragment size in a Range request
[$responding_node]	dh	Responding data center
$upstream_cache_status	MISS	Status of a requested file in CDN cache: HIT: response served from the CDN cache. STALE: outdated response that failed to update (origin not responding or responding incorrectly). UPDATING: outdated response still updating from a previous request. REVALIDATED: response matching one on an origin (based on the proxy_cache_revalidate directive). EXPIRED: response that has expired in cache but still matches one on an origin; a request was sent to re-cache it. MISS: response served directly from an origin rather than from cache. BYPASS: response for the first file request after clearing the cache (the first request from each CDN server results in BYPASS; subsequent requests on that server result in HIT).
$upstream_response_length	10485760	Response length from an origin in bytes. In case of multiple responses, commas and colons are used
$upstream_addr	0.0.0.0:80	Origin’s IP address and port
$gcdn_api_client_id (internal system variable)	123	Your ID in our system
$gcdn_api_resource_id (internal system variable)	01	Your CDN-resource ID in our system
$uid_got (internal system variable)	-	Cookie name and received user ID
$uid_set (internal system variable)	-	Cookie name and provided user ID
$geoip_country_code	KZ	User’s country code according to the ISO 3166 standard (Alpha-2 code)
$geoip_city	-	User’s city code
$shield_type (internal system variable)	shield_no	Indicates whether Origin Shielding is enabled: shield_old – enabled shield_no – disabled
$server_addr (internal system variable)	0.0.0.0	IP address of an Anycast zone or CDN server
$server_port (internal system variable)	80	Requested port
$upstream_status	206	Origin response code
$upstream_connect_time	0.000	Number of seconds (accurate to milliseconds) it took to access an origin server
$upstream_header_time	0.200	Number of seconds (accurate to milliseconds) it took to receive a response header from an origin server
$shard_addr (internal system variable)	0.0.0.0	IP address of a CDN server that was first to accept a request if the Cache Sharding feature is enabled
$geoip2_data_asnumber	asnumber	Number of an autonomous system that sent a request
$connection (internal system variable)	2897494295	Connection serial number
$connection_requests (internal system variable)	1	Current number of requests made through a connection
$http_traceparent (internal system variable)	00-d5fe1dc9035165ce36952daf29686b6c-14330be33197dd1a-01	Unique request identifier. More info in the Traceparent guide
$http_x_forwarded_proto	-	Initial protocol of an incoming request (HTTP or HTTPS)
$gcdn_internal_status_code (internal system variables)	-	Initial status code. Possible values are: -, or 1000-1200
$ssl_cipher (internal system variable)	ECDHE-RSA-AES256-GCM-SHA384	Cipher name used for an established SSL connection
$ssl_session_id (internal system variable)	28a4184139cb43cdc79006cf2d1a4ac93bdc****	Session ID of an established SSL connection
$ssl_session_reused (internal system variable)	r	Shows whether a session was reused (r) or not (.)
$sent_http_content_type	application/json	Value of the Content-Type HTTP header, indicating the MIME type of a transmitted file
$tcpinfo_rtt	21	Average time (latency) it takes to transfer a packet to/from a server. The unit of time is microseconds.
$server_country_code	PL	Server’s country code according to the ISO 3166 standard (Alpha-2 code).
$gcdn_tcpinfo_snd_cwnd	45	Size of the TCP Congestion window, i.e., the maximum number of TCP segments that the connection can send before an acknowledgment is required.
$tcpi_total_retrans	10	Total number of retransmitted packets over the life of the connection.
$gcdn_rule_id	100700	Initial rule ID (beta). Possible values are: -, or 100700

Internal status codes table

Reason	HTTP Code	Internal Code	Comment
Country ACL	403	1001
Refferer ACL	403	1002
IP ACL	403	1003
User-Agent ACL	403	1004
Secure Token	403	1005	If the requested link passes the authenticity check, the $secure_link variable is set to the link extracted from the request URI. Otherwise, $secure_link is set to an empty string.
Secure Token	410	1006	If the link has a limited lifetime and the time has expired, the $secure_link variable is set to "0".
WAF	403	1007	Request blocked by WAF
Bot challenge / Testcookie	307	1008	Redirection sent by bot challenge (testcookie)
Blocklist	403	1009	Request blocked by bot protection blocklist
HTTP method	405	1200	AllowedHttpMethods enabled AND HTTP request method not in AllowedHttpMethods list
Streaming disabled .(ts|m3u8)	402	1201	Streaming feature disabled for the client
LE Validation /.well-known/acme-challenge/	404	1202	We return 404 if http_user_agent is not in the list `“cert-manager-challenges	acme.zerossl.com	Cpanel-HTTP-Client	Buypass validation client	Google-Certificates-Bridge	vercel-fetch	win-acme	Typhoeus	Go-http-client”`.
check $http_x_cdn_requestor not empty	403	1203	Non-authorized request to shield
Force Return	ANY	1204	Status code option, this internal code indicates that the response was generated by this feature

Configure logs for export

You can enable and set up the Logs uploader feature in the Gcore Customer Portal on the Gcore CDN page.

To access log settings, navigate to Logs > Logs uploader and configure the feature as described in the following steps.

Step 1 (optional). Include empty logs

Keep the Do not send empty logs option selected if you don’t want to receive empty logs. Otherwise, uncheck it.

Step 2 (optional). Enable origin shielding

If you are using the origin shielding feature, you’ll see the Add logs from origin shielding checkbox when configuring Logs uploader.

We recommend that you select this option as it ensures that the logs report will include both requests to cache services and requests to the pre-cache server. Thus, you’ll receive more detailed information on resource usage. To include these logs, select the checkbox to enable Include logs from origin shielding.

Tip

If you don’t see the origin shielding option on the Logs uploader page, this feature is not activated for your account. For details on how to activate origin shielding, check our dedicated guide.

Step 3. Select log fields

Choose which log fields you want to include in the exported report. By default, all fields are selected.

Step 4. Configure a storage provider

Amazon

Follow these instructions to export logs to AWS storage:

1. In the Storage provider, select Amazon.

2. Provide your access key ID and secret access key, which together form long-term AWS credentials.

3. (Optional). Choose a storage region. While the region is often determined automatically, we recommend specifying it to ensure that your logs are exported successfully.

4. Specify the name of a bucket where you want to export CDN logs.

5. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket.

6. Click Save changes to apply the updates.

Follow these instructions to export logs to AWS storage:

1. In the Storage provider, select Amazon.

2. Provide your access key ID and secret access key, which together form long-term AWS credentials.

3. (Optional). Choose a storage region. While the region is often determined automatically, we recommend specifying it to ensure that your logs are exported successfully.

4. Specify the name of a bucket where you want to export CDN logs.

5. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket.

6. Click Save changes to apply the updates.

Follow these instructions to export logs to configure logs for Alibaba Cloud Object Storage Service (OSS):

1. In the Storage provider, select OSS.

2. Provide your access key ID and secret access key. Check the official OSS documentation for details.

3. (Optional). Choose a storage region. While the region is often determined automatically, we recommend specifying it to ensure that your logs are exported successfully.

4. Specify the name of a bucket where you want to export CDN logs.

5. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket.

6. Click Save changes to apply the updates.

Follow these instructions to export logs to Gcore Object Storage or any S3 storage of your choice:

1.In the Storage provider, select Gcore or Other.

2. Specify a hostname—a name that’s assigned to a storage server within a network and is used instead of an IP address. In you’re using Gcore Storage, you can find your hostname in the storage details section.

3. Provide your access key ID and secret access key. You can find this information in the Details of the storage.

4. Specify the name of a bucket where you want to export CDN logs.

5. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket.

6. Click Save changes to apply the updates.

How near real-time log exporting works

Logs are achived and exported to your endpoint every five minutes. If CDN servers are not requested and you didn’t select the Do not send empty logs checkbox when configuring Logs uploader, an empty log file (± 20 bytes) will be sent to your storage.