Overview
One of the most important considerations in working with an DSP is designing the data pipeline by which auction, bid, and win events are processed and analyzed.
With AIDEM technology you can access the full, unfiltered stream of RTB events, in much the same way you would if you were building a bidder from scratch.
Streaming vs Batch
The first consideration for designing your data pipeline is whether you prefer to get the data in batch form, or as a continuous stream. For the very high-volume data, like auction logs, AIDEM only supports batch data. For win logs (impressions) AIDEM supports both methods, and there are pros and cons to each:
| Pipeline Method | Description | Pros | Cons |
|---|---|---|---|
| Batch | Hourly or daily files of data placed in an S3 bucket. | Fairly easy to ingest, fault tolerant | Delay in utilizing data. Also may include many files written per hour. |
| Stream | Near real-time data in JSON or protobuf format sent over http or to AWS Kinesis | Use data as fast as you can process it | Higher cost and complexity to support data ingestion |
Data Definitions
Column definitions, protobuf mapping, field lists and a data dictionary can be found below.
AIDEM makes multiple types of data available as described in the chart below. Based on your use case you may need some, or all of this data. Because some of this data can be quite large, additional fees may apply (contact your Account Manager for more information).
| Log Type | Description | Expected Delivery Timing | Delivered Via | S3 Region Requirements | Supported Delivery Formats | Log Headers | More Information |
|---|---|---|---|---|---|---|---|
| Batch Hourly Wins | The winning auctions impressions, clicks and video activities | Up to 7.5 hours after event occurrence. We wait 6 hours to join late arriving clicks and video activities, and delivery occurs with 90 mins of the 6 hour window. | S3 | Single bucket across regions | Gzipped CSV or Parquet (Snappy-compression) | Link | Batch files are joined, which means bid, click and activity data matched against impression data will be included |
| Batch Daily Wins | See above | Between 6-7am ET (6am during daylight savings time, 7am during non-DST periods) | S3 | Single bucket across regions | Gzipped CSV or Parquet (Snappy-compression) | Link | See above |
| Streaming Unjoined Win Logs | Stream of events associated with an impression with each event as a separate message | Realtime | HTTP or Kinesis | N/A | Protobuf or JSON (HTTP Only) | TBD | Does not include the RequestLogMessage. |
| Streaming Joined Win Logs | Stream of events associated with an impression, where all events are joined in a single message. | Realtime | HTTP or Kinesis | N/A | Protobuf or JSON | TBD | RequestLogMessage is included when available. ConversionEventMessages will not be included. Unavailable for DSP customers. |
| Attributed Conversions | The conversions events recorded by AIDEM. Conversions will be attributed to an auction. | Approximately 8 hours | S3 | Single bucket across reigons | Gzipped CSV or Parquet | Link | Logs are unjoined, and may need to be joined to win logs based on use case. |
| IP Attributed Conversions | The IP conversion events recorded by AIDEM. IP Conversions will be attributed to an auction. | Approximately 8 hours | S3 | Single bucket across reigons | Gzipped CSV or Parquet | Link | Logs are unjoined, and may need to be joined to win logs based on use case. |
| Conversions | The conversion events recorded by AIDEM, regardless of attribution. | Near realtime | S3 | Buckets must be co-located in same region as Bidder | Gzipped CSV | Link | Logs are unjoined, and may need to be joined to win logs based on use case. |
| Bid Response Feedback | Loss logs provided by a limited set of exchanges | 1 Hour Delay | S3 | Single bucket across regions | Gzipped CSV | Link | |
| Bids | Bids returned from AIDEM to the Exchange | Near real-time | S3 | Buckets must be co-located in same region as Bidder | Gzipped CSV | Link | Bid logs are mainly used to calculate win rates on any dimension that comes through on the bid & win logs, which you can then use to improve your bidding strategies. Often, clients that use a custom bidding agent will log the bids themselves. |
| Auctions | The auctions request from the exchange. Logs are typically sampled unless otherwise requested. | Near real-time | S3 | Buckets must be co-located in same region as Bidder | Gzipped CSV | Link | Auction Logs can be used for inventory analytics. You can use the data to inform future bidding strategies. Importantly, per your agreement with AIDEM, these logs can never be used to create derivative segments for your use or resale. For example, you may not record the User IDs that come through on the bid requests and retarget those users in a different auction. Unavailable for DSP Customers. |
| Segments | All 1st Party Segments and Auction ID associations | Near real-time | S3 | Buckets must be co-located in same region as Bidder | Gzipped CSV | Link | Unavailable for DSP Customers |