Tutorial: Archiving Your API Logs

This tutorial shows an example of using a script to archive API logs, either locally or to the OUTSCALE Object Storage (OOS) service.

You can now use the osc-logs tool to archive your API logs.

Creating a Script for API Log Pagination

The following is an example of a script which runs ReadApiLogs with pagination to retrieve all logs starting from a given date, and then saves them in a CSV file named logs.csv.

By default, the script works with Linux. For it to work with macOS, you need to comment out the two lines below the # Linux line and uncomment the two lines below the # macOS line.
apilogs.sh
#!/bin/bash

# Exit when any command fails
set -e

# Usage: apilogs.sh [PROFILE] [YYYY-MM-DD]

profile=${1:-default}
# Linux: use $(date --date '2 weeks ago' +%F)
query_date_after=${2:-$(date --date '2 weeks ago' +%F)}
query_date_before=${2:-$(date +%F)}
# macOS/BSD: use $(date -v [+|-]val[ymwdHMS] +%F)
#query_date_after=${2:-$(date -v -2w +%F)}
#query_date_before=${2:-$(date +%F)}
next_page="null"

retrieve_apilogs(){
    # Perform ReadApiLogs calls with/without page token
    if [ "$next_page" = "null" ]; then
        osc-cli --profile $profile api ReadApiLogs --Filters '{"QueryDateAfter":"'$query_date_after'","QueryDateBefore":"'$query_date_before'"}' > tmp_logs.json;
    else
        echo ">>> Retrieving new page" $next_page
        osc-cli --profile $profile api ReadApiLogs --Filters '{"QueryDateAfter":"'$query_date_after'","QueryDateBefore":"'$query_date_before'"}' --NextPageToken $next_page > tmp_logs.json;
    fi
    # Concatenate selected fields from logs and save to CSV file
    cat tmp_logs.json | jq -r '.Logs[] | [.RequestId, .AccountId, .QueryAccessKey, .ResponseStatusCode, .QueryCallName, .QueryIpAddress, .QueryDate, .QueryUserAgent] | @csv' >> logs.csv
    # Check whether another page needs to be retrieved
    next_page=$(cat tmp_logs.json | jq -r '.NextPageToken // "null"')
}

retrieve_apilogs
while [ "$next_page" != "null" ]; do
    retrieve_apilogs
done

# Nicely display logs as columns with selected fields in shell
cat logs.csv | column -t -s ',' | tr -d '"'

# Cleanup
rm tmp_logs.json

Archiving Your API Logs Locally

Before you begin:

You can run the previous script as follows:

$ bash apilogs.sh PROFILE YYYY-MM-DD

where:

  • PROFILE is the name of your OSC CLI profile (by default, default).

  • YYYY-MM-DD is the start date from which you want to retrieve logs (by default, the script sets this date to two weeks ago). The script retrieves logs from this date up to today by using this date for the QueryDateAfter parameter of ReadApiLogs, and the date of today for the QueryDateBefore parameter.

Archiving Your API Logs to OOS

Before you begin:

The OUTSCALE Object Storage (OOS) service enables you to store sensitive data on cold storage, in a secure way. It also enables you to share the access to this data.

To export your API logs to OOS, you can combine the previous script with cron, a utility which automates the execution of tasks at set intervals.

Linux

  1. Open your cron task file using the following command:

    $ crontab -e

  2. In the file, insert one of the following lines, and then save the file:

    For archiving everyday
    0 0 * * * MDATE=$(date --date '1 day ago' +%F) ; bash readapilogs.sh default $MDATE ; aws s3api put-object --body logs.csv --bucket archives --key logs-$MDATE.csv ; rm logs.csv
    For archiving every 30 days
    0 0 */30 * * MDATE=$(date --date '30 days ago' +%F) ; bash readapilogs.sh default $MDATE ; aws s3api put-object --body logs.csv --bucket archives --key logs-$MDATE.csv ; rm logs.csv

The task thus created uses the script to create a local CSV file, exports it to OOS, and then deletes the local version.

macOS

  1. Open your cron task file using the following command:

    $ crontab -e

  2. In the file, insert one of the following lines, and then save the file:

    For archiving everyday
    0 0 * * * MDATE=$(date -v -1d +%F) ; bash readapilogs.sh default $MDATE ; aws s3api put-object --body logs.csv --bucket archives --key logs-$MDATE.csv ; rm logs.csv
    For archiving every 30 days
    0 0 */30 * * MDATE=$(date -v -30d +%F) ; bash readapilogs.sh default $MDATE ; aws s3api put-object --body logs.csv --bucket archives --key logs-$MDATE.csv ; rm logs.csv

The task thus created uses the script to create a local CSV file, exports it to OOS, and then deletes the local version.

Related Pages

Corresponding API Method

AWS™ and Amazon Web Services™ are trademarks of Amazon Technologies, Inc or its affiliates in the United States and/or other countries.

Export to PDF