Overview

One of the most important considerations in working with an DSP is designing the data pipeline by which auction, bid, and win events are processed and analyzed.

With AIDEM technology you can access the full, unfiltered stream of RTB events, in much the same way you would if you were building a bidder from scratch.

Streaming vs Batch

The first consideration for designing your data pipeline is whether you prefer to get the data in batch form, or as a continuous stream. For the very high-volume data, like auction logs, AIDEM only supports batch data. For win logs (impressions) AIDEM supports both methods, and there are pros and cons to each:

Pipeline Method	Description	Pros	Cons
Batch	Hourly or daily files of data placed in an S3 bucket.	Fairly easy to ingest, fault tolerant	Delay in utilizing data. Also may include many files written per hour.
Stream	Near real-time data in JSON or protobuf format sent over http or to AWS Kinesis	Use data as fast as you can process it	Higher cost and complexity to support data ingestion

Data Definitions

Column definitions, protobuf mapping, field lists and a data dictionary can be found below.

AIDEM makes multiple types of data available as described in the chart below. Based on your use case you may need some, or all of this data. Because some of this data can be quite large, additional fees may apply (contact your Account Manager for more information).

Log Type	Description	Expected Delivery Timing	Delivered Via	S3 Region Requirements	Supported Delivery Formats	Log Headers	More Information
Batch Hourly Wins	The winning auctions impressions, clicks and video activities	Up to 7.5 hours after event occurrence. We wait 6 hours to join late arriving clicks and video activities, and delivery occurs with 90 mins of the 6 hour window.	S3	Single bucket across regions	Gzipped CSV or Parquet (Snappy-compression)	Link	Batch files are joined, which means bid, click and activity data matched against impression data will be included
Batch Daily Wins	See above	Between 6-7am ET (6am during daylight savings time, 7am during non-DST periods)	S3	Single bucket across regions	Gzipped CSV or Parquet (Snappy-compression)	Link	See above
Streaming Unjoined Win Logs	Stream of events associated with an impression with each event as a separate message	Realtime	HTTP or Kinesis	N/A	Protobuf or JSON (HTTP Only)	TBD	Does not include the RequestLogMessage.
Streaming Joined Win Logs	Stream of events associated with an impression, where all events are joined in a single message.	Realtime	HTTP or Kinesis	N/A	Protobuf or JSON	TBD	RequestLogMessage is included when available. ConversionEventMessages will not be included. Unavailable for DSP customers.
Attributed Conversions	The conversions events recorded by AIDEM. Conversions will be attributed to an auction.	Approximately 8 hours	S3	Single bucket across reigons	Gzipped CSV or Parquet	Link	Logs are unjoined, and may need to be joined to win logs based on use case.
IP Attributed Conversions	The IP conversion events recorded by AIDEM. IP Conversions will be attributed to an auction.	Approximately 8 hours	S3	Single bucket across reigons	Gzipped CSV or Parquet	Link	Logs are unjoined, and may need to be joined to win logs based on use case.
Conversions	The conversion events recorded by AIDEM, regardless of attribution.	Near realtime	S3	Buckets must be co-located in same region as Bidder	Gzipped CSV	Link	Logs are unjoined, and may need to be joined to win logs based on use case.
Bid Response Feedback	Loss logs provided by a limited set of exchanges	1 Hour Delay	S3	Single bucket across regions	Gzipped CSV	Link
Bids	Bids returned from AIDEM to the Exchange	Near real-time	S3	Buckets must be co-located in same region as Bidder	Gzipped CSV	Link	Bid logs are mainly used to calculate win rates on any dimension that comes through on the bid & win logs, which you can then use to improve your bidding strategies. Often, clients that use a custom bidding agent will log the bids themselves.
Auctions	The auctions request from the exchange. Logs are typically sampled unless otherwise requested.	Near real-time	S3	Buckets must be co-located in same region as Bidder	Gzipped CSV	Link	Auction Logs can be used for inventory analytics. You can use the data to inform future bidding strategies. Importantly, per your agreement with AIDEM, these logs can never be used to create derivative segments for your use or resale. For example, you may not record the User IDs that come through on the bid requests and retarget those users in a different auction. Unavailable for DSP Customers.
Segments	All 1st Party Segments and Auction ID associations	Near real-time	S3	Buckets must be co-located in same region as Bidder	Gzipped CSV	Link	Unavailable for DSP Customers