About a month ago we announced that we were conducting a clean-up with regard to DoubleClick FTP login accounts, files and folders. We're happy to report that work was completed without any noticeable affect to you.
We'll be completing an additional phase of this clean-up on Tuesday, July 19th. On that date we'll be updating select user accounts which have 'delete' or 'modify' privileges on files and folders.
The reason for this change is to protect your files from inadvertently being deleted by, well, you. We do our best to re-post files in such cases, but this is not always possible. We believe this measure ultimately protects your valuable business data and is, on the balance, the best course of action.
If you have concerns with this action, please contact your Support rep through the usual channels:
dfa-support@google.com
dfp-support@google.com
Please include your FTP username (but not password) in that email.
-- Matthew Trojanovich, Data Transfer Team
NOTICE - DoubleClick FTP audit and clean-up, part 2
Friday, July 15, 2011 | 3:09 PM
Labels: FTP, notification, security
NOTICE - DoubleClick FTP audit and clean-up
Friday, June 10, 2011 | 1:57 PM
Labels: FTP, notification, security
Do you pick up files (DT files, Match Tables, Custom Reports, etc) from the DoubleClick FTP server?
If so, then this notice is for you.
If so, then this notice is for you.
During the week of June 13th - 17th we'll be doing a bit of clean-up with regard to files and folders. Specifically, we've completed a review of read/write/delete privileges and will be taking action on those user accounts which do not have approval for write and/or delete. If you have write and/or delete privileges but are not sure whether you have approval for them get in touch with your local support rep through the usual channels:
Please include your account/username (but not your password) in that email.
And, as a general note, we strongly recommend that you not share your DoubleClick FTP account information with any outside party. (Refer to our previous Secure Your Login post).
-- Matthew Trojanovich, the DoubleClick Data Transfer Team
Tools to help you manipulate text data
Thursday, April 21, 2011 | 4:45 PM
Labels: best practices, crush, data manipulation
You may be wondering what tools we recommend for working with Data Transfer and Match Tables. While there are many options, both commercial and otherwise, the following are some of our recommendations. What they all have in common is that they are tools designed for manipulation of large text files. These tools also share something else: they are all either free or open-source, so we hope that makes them even more useful for you. Some of these same tools are used by the DoubleClick Customization team - the DoubleClick team that builds the occasional DT-based custom report or application for clients.
So if you're looking for some alternatives to try out, we hope you find the following tools useful:
Crush-Tools - CRUSH (Custom Reporting Utilities for SHell) is a collection of tools for processing delimited-text data from the command line or in shell scripts. The tools are especially "Data Transfer aware" in that they understand DT headers, delimiters and column types. At the site find a tutorial and usergroup. A simple example of CRUSH is the following command line statement where familiar shell commands are enhanced:
gzdog <dt_filename> | grepfield -F Event-ID [12] | aggregate2 -K Site-ID,Activity-Sub-Type -S Revenue -L | pivot -P Site-ID,Activity-Sub-Type -A Revenue-Sum
"unzip and cat the DT file, select only post-click and post-impression activities, aggregate the revenue by Site and Floodlight tag, and pivot the results for each site/activity combination."
Google Refine - a power tool for working with messy data, cleaning it up, transforming it from one format into another, extending it with web services, and linking it to databases like Freebase. The tool can help make merging disparate data a breeze. Check out the engaging demo videos.
Google Fusion Tables (beta) - Upload small or large data sets from spreadsheets or CSV files. Visualize your data on maps, timelines and charts. Pick who can access your data; hide parts of your data if needed. Merge data from multiple tables. Discuss your data with others. Track changes and discussions. Drop a DT file into Fusion Tables and in short order you'll be able to graph and sift through the data through a spreadsheet like UI.
Google Visualization API - lets you access multiple sources of structured data that you can display, choosing from a large selection of visualizations. Thus you can create reports and dashboards as well as analyze and display your data through the wealth of available visualization applications. In terms of DT, once you've processed your files, make it visually useful!
-- Matthew Trojanovich, the DoubleClick Data Transfer Team
So if you're looking for some alternatives to try out, we hope you find the following tools useful:
Crush-Tools - CRUSH (Custom Reporting Utilities for SHell) is a collection of tools for processing delimited-text data from the command line or in shell scripts. The tools are especially "Data Transfer aware" in that they understand DT headers, delimiters and column types. At the site find a tutorial and usergroup. A simple example of CRUSH is the following command line statement where familiar shell commands are enhanced:
gzdog <dt_filename> | grepfield -F Event-ID [12] | aggregate2 -K Site-ID,Activity-Sub-Type -S Revenue -L | pivot -P Site-ID,Activity-Sub-Type -A Revenue-Sum
"unzip and cat the DT file, select only post-click and post-impression activities, aggregate the revenue by Site and Floodlight tag, and pivot the results for each site/activity combination."
Google Refine - a power tool for working with messy data, cleaning it up, transforming it from one format into another, extending it with web services, and linking it to databases like Freebase. The tool can help make merging disparate data a breeze. Check out the engaging demo videos.
Google Fusion Tables (beta) - Upload small or large data sets from spreadsheets or CSV files. Visualize your data on maps, timelines and charts. Pick who can access your data; hide parts of your data if needed. Merge data from multiple tables. Discuss your data with others. Track changes and discussions. Drop a DT file into Fusion Tables and in short order you'll be able to graph and sift through the data through a spreadsheet like UI.
Google Visualization API - lets you access multiple sources of structured data that you can display, choosing from a large selection of visualizations. Thus you can create reports and dashboards as well as analyze and display your data through the wealth of available visualization applications. In terms of DT, once you've processed your files, make it visually useful!
-- Matthew Trojanovich, the DoubleClick Data Transfer Team
Aggregated performance reporting for DFA
Tuesday, March 15, 2011 | 1:49 PM
Labels: aggregate reports, API, DFA
Today we're happy to announce version 1.13 of the DoubleClick for Advertisers (DFA) API.
Why are we mentioning it here in the Data Transfer blog? Well, if you use our DFA product then it's likely that you use some combination of event level data mixed with aggregated data to generate your digital advertising insights. We'll discuss why v1.13 is notable as well as how you can make greater use of the aggregated reports from the API.
The DFA report service
The release marks the first time that a web services API interface is available for DFA's Report Central. Previously, if you wanted to programmatically extract reports you had to either pick up file attachments from scheduled email reports or use the Java-only DFA API reporting interface. For automated systems and ETLs, the email attachments approach is only really suitable as a fall-back mechanism. And while the Java interface offers a full-featured programming interface (including gets, sets, creates and deletes), it can result in more effort on your part having to develop both in Java and whatever other language you use to interact with the DFA trafficking web services interface.
To help solve these problems the DFA API v1.13 release includes a new report service that allows you to generate and retrieve reports from predefined queries. Check on them periodically to see when they have finished being compiled and retrieve a URL where you can download the completed report.
A report produced by the API works like any other in the DFA reporting interface; like your Data Transfer files, it can be published every morning, local time. Typically you'll be able to retrieve your report at the same time that your DT Match Tables are done.
Using aggregated performance data to compliment DT event data
So when can it be useful or even required to use aggregated reporting data? There are two general cases: (1) letting DoubleClick do the simple aggs for you, (2) the metric you're interested in isn't offered within Data Transfer. For the first, we see customers using aggregations to provide ready-made reports for their users. This can free-up your time to focus on higher value reports - typically those that are user (User-ID/cookie) based. We also see customers using aggregations to test and validate against event level DT data. This involves examining a previous day's aggregated click counts or a previous month's aggregated matched conversions (referred to as "post-click" and "post-impression" in DFA reporting parlance). And in perhaps the most interesting case related to the API, some customers use the web services interface to expand DFA IDs that either haven't come through in Match Tables or to gather dimensions not delivered in Match Tables. Let's discuss this last case a little more...
We strive to provide ID lookups in your daily Match Tables but sometimes a dimension is missing. This can happen when, for example, a dynamic Click Ad is reactivated after having been dormant for more than 30 days. A small number (perhaps the first 0.05%) of requests for that Ad may lack a Buy-ID. Since we generate most of your Match Tables daily you might otherwise have to wait a full day to resolve the name value. Interrogating the API (trafficking/UI in this case) can be a better way to resolve the name value in near real time.
Regarding dimensions not included in your Match Tables, there are two ways to resolve this: request a newer Match Table or a custom Match Table, or expand your IDs by querying the API. To request new/custom Match Tables (have you checked if you are receiving the "latest and greatest" in a while? We don't change your Match Tables until you ask us to.), you can ask your Account Manager to start a DT audit. You can also expand your IDs directly by making batch requests to the API (either reporting or trafficking services, as necessary). For example, some clients may want to periodically pull the Placement-Comments for one particular advertiser. Doing this with the API is a snap. Another useful application, if you're a DoubleClick Search customer, is to use the reporting service to capture search engine click stats since only the DFA equivalent click stats appear in Data Transfer.
-- Matthew Trojanovich, the DoubleClick Data Transfer Team
Why are we mentioning it here in the Data Transfer blog? Well, if you use our DFA product then it's likely that you use some combination of event level data mixed with aggregated data to generate your digital advertising insights. We'll discuss why v1.13 is notable as well as how you can make greater use of the aggregated reports from the API.
The DFA report service
The release marks the first time that a web services API interface is available for DFA's Report Central. Previously, if you wanted to programmatically extract reports you had to either pick up file attachments from scheduled email reports or use the Java-only DFA API reporting interface. For automated systems and ETLs, the email attachments approach is only really suitable as a fall-back mechanism. And while the Java interface offers a full-featured programming interface (including gets, sets, creates and deletes), it can result in more effort on your part having to develop both in Java and whatever other language you use to interact with the DFA trafficking web services interface.
To help solve these problems the DFA API v1.13 release includes a new report service that allows you to generate and retrieve reports from predefined queries. Check on them periodically to see when they have finished being compiled and retrieve a URL where you can download the completed report.
A report produced by the API works like any other in the DFA reporting interface; like your Data Transfer files, it can be published every morning, local time. Typically you'll be able to retrieve your report at the same time that your DT Match Tables are done.
Using aggregated performance data to compliment DT event data
So when can it be useful or even required to use aggregated reporting data? There are two general cases: (1) letting DoubleClick do the simple aggs for you, (2) the metric you're interested in isn't offered within Data Transfer. For the first, we see customers using aggregations to provide ready-made reports for their users. This can free-up your time to focus on higher value reports - typically those that are user (User-ID/cookie) based. We also see customers using aggregations to test and validate against event level DT data. This involves examining a previous day's aggregated click counts or a previous month's aggregated matched conversions (referred to as "post-click" and "post-impression" in DFA reporting parlance). And in perhaps the most interesting case related to the API, some customers use the web services interface to expand DFA IDs that either haven't come through in Match Tables or to gather dimensions not delivered in Match Tables. Let's discuss this last case a little more...
We strive to provide ID lookups in your daily Match Tables but sometimes a dimension is missing. This can happen when, for example, a dynamic Click Ad is reactivated after having been dormant for more than 30 days. A small number (perhaps the first 0.05%) of requests for that Ad may lack a Buy-ID. Since we generate most of your Match Tables daily you might otherwise have to wait a full day to resolve the name value. Interrogating the API (trafficking/UI in this case) can be a better way to resolve the name value in near real time.
Regarding dimensions not included in your Match Tables, there are two ways to resolve this: request a newer Match Table or a custom Match Table, or expand your IDs by querying the API. To request new/custom Match Tables (have you checked if you are receiving the "latest and greatest" in a while? We don't change your Match Tables until you ask us to.), you can ask your Account Manager to start a DT audit. You can also expand your IDs directly by making batch requests to the API (either reporting or trafficking services, as necessary). For example, some clients may want to periodically pull the Placement-Comments for one particular advertiser. Doing this with the API is a snap. Another useful application, if you're a DoubleClick Search customer, is to use the reporting service to capture search engine click stats since only the DFA equivalent click stats appear in Data Transfer.
-- Matthew Trojanovich, the DoubleClick Data Transfer Team
5 tips for consuming Data Transfer files
Tuesday, October 26, 2010 | 1:03 PM
Labels: .done files, best practices, crush, security
Every day we receive various support tickets asking for help with existing Data Transfer configurations. Sometimes the answer to a question can be found within the DT FAQs in the Help Center (DFA | DFP | Rich Media) and other times it requires personal assistance. We regularly review both sources of information in an effort to improve your Data Transfer experience. With that in mind, we've identified a number of tips that will help improve your work with DT.
1) Download Daily
DT files are published daily* according to your account's timezone. Correspondingly, we suggest that you download your new DT files every day. While you can download files every other day, only on weekdays, or even once a week, downloading every day ensures that you see your data when it is fresh. It also means that you can identify issues while they are still recent. DT files stay on the server, by default, for 30 days from the date of post - but don't wait until the files are aged to pick them up! Purged DT files cannot be recovered.
2) Secure your login
If your files are used by different parties, it's best to limit access to your FTP account. For instance, say you have a reporting team, an analytics team and a 3rd party consulting team all consuming your DT files. Suppose that each is located in a different office or is part of a different company. Sharing your login means that you cannot control which group sees what data. If one of these parties should need to have access restricted (filtered) or to revoked, it's difficult to do with a shared login. Some DT (and Match Table) data can be quite sensitive. The best approach is to download all of your files locally and then to distribute as needed. While this requires more effort on your part, it's almost always well worth it. We get requests to build custom scripts to help clients distribute their DT files, but such scripts are "black box", limited in their business rules, and are charged at an additional fee.
If you do need to reset your FTP login, please contact your support team and they'll be happy to assist.
3) Save for a rainy day
We recommend that you save 60 days of DT files in their raw, un-aggregated format. Storage these days is relatively cheap. Storing your files in their raw un-aggregated format means that you can re-load them into your data store easily if needed. Once files have aged off of the DoubleClick FTP server, they cannot be regenerated. DT files are special since, among other reasons, they contain your non-sampled User-ID ad events. We sometimes hear about clients who discover a clever (but legal!) new use for their event files - and frequently that new use requires granular values like User-ID. Moreover, depending upon your system, it's possible to query flat files just like querying a database. There are a number of open-source options to achieve this.
4) Identifying DT fields
DT file types (eg, impression, click and activity) change formats very infrequently. In fact, if there is a change in format, it's likely because you have asked us to add or remove a field. In short, you can depend on both the file format (gzipped text) and the structure (column A, column B and so on). From time to time, however, you may want to change the order of your columns. To ensure that your ETL system is flexible, identify DT columns based on column name rather than on column position. Identifying a field by name means that changes to DT format require no further programming on your part. Indexing columns by their name is dependable; fields like Time, User-ID, Ad-ID, Site-ID, etc, have never changed. When we build custom reports based on DT, we always index on column name.
5) Use .done files
.done files are a somewhat under-used free feature of the Match Tables (MT) system. .done files are produced daily and provide you with information that can be very useful in ensuring smooth ETL processing. A .done file is produced once all dependencies have been met for that day. By default the dependencies are your newly updated MT files. Thus, if a .done file is posted for a given day, it means that you can start downloading your MT files and begin identifying event dimensions (names). .done files also provide you with SHA checksums as well as a filename list and filesize list. Note: if you have network level Activities, you'll also receive a .done file. This file however is different than the MT .done file.
To request .done files for your account, contact your account manager or support team.
6) Use open source tools
Processing your DT and MT files doesn't have to involve expensive ETL resources. There are a number of open-source ETL tools (as well as open-source databases such as MySQL) which can do a fine job while at the same time helping you manage operation costs. One particular tool is Crush-Tools. We designed Crush-Tools internally and released it to the open-source community a few years back. Crush-Tools is a unix-based library of utilities that are specifically designed to consume, manipulate and operate on DoubleClick Data Transfer files. For instance, via pipes and redirects you can filter, aggregate, calculate and produce reports all directly from DT flat files. We even use Crush-Tools extensively ourselves! Read more about Crush-Tools here.
We hope these tips will prove helpful to you.
--Matthew Trojanovich, the Data Transfer Service Team
* Some clients have asked to receive their DT files more frequently. Our product team is indeed working on this. Dates cannot be given yet, but stay tuned to this channel for more.
1) Download Daily
DT files are published daily* according to your account's timezone. Correspondingly, we suggest that you download your new DT files every day. While you can download files every other day, only on weekdays, or even once a week, downloading every day ensures that you see your data when it is fresh. It also means that you can identify issues while they are still recent. DT files stay on the server, by default, for 30 days from the date of post - but don't wait until the files are aged to pick them up! Purged DT files cannot be recovered.
2) Secure your login
If your files are used by different parties, it's best to limit access to your FTP account. For instance, say you have a reporting team, an analytics team and a 3rd party consulting team all consuming your DT files. Suppose that each is located in a different office or is part of a different company. Sharing your login means that you cannot control which group sees what data. If one of these parties should need to have access restricted (filtered) or to revoked, it's difficult to do with a shared login. Some DT (and Match Table) data can be quite sensitive. The best approach is to download all of your files locally and then to distribute as needed. While this requires more effort on your part, it's almost always well worth it. We get requests to build custom scripts to help clients distribute their DT files, but such scripts are "black box", limited in their business rules, and are charged at an additional fee.
If you do need to reset your FTP login, please contact your support team and they'll be happy to assist.
3) Save for a rainy day
We recommend that you save 60 days of DT files in their raw, un-aggregated format. Storage these days is relatively cheap. Storing your files in their raw un-aggregated format means that you can re-load them into your data store easily if needed. Once files have aged off of the DoubleClick FTP server, they cannot be regenerated. DT files are special since, among other reasons, they contain your non-sampled User-ID ad events. We sometimes hear about clients who discover a clever (but legal!) new use for their event files - and frequently that new use requires granular values like User-ID. Moreover, depending upon your system, it's possible to query flat files just like querying a database. There are a number of open-source options to achieve this.
4) Identifying DT fields
DT file types (eg, impression, click and activity) change formats very infrequently. In fact, if there is a change in format, it's likely because you have asked us to add or remove a field. In short, you can depend on both the file format (gzipped text) and the structure (column A, column B and so on). From time to time, however, you may want to change the order of your columns. To ensure that your ETL system is flexible, identify DT columns based on column name rather than on column position. Identifying a field by name means that changes to DT format require no further programming on your part. Indexing columns by their name is dependable; fields like Time, User-ID, Ad-ID, Site-ID, etc, have never changed. When we build custom reports based on DT, we always index on column name.
5) Use .done files
.done files are a somewhat under-used free feature of the Match Tables (MT) system. .done files are produced daily and provide you with information that can be very useful in ensuring smooth ETL processing. A .done file is produced once all dependencies have been met for that day. By default the dependencies are your newly updated MT files. Thus, if a .done file is posted for a given day, it means that you can start downloading your MT files and begin identifying event dimensions (names). .done files also provide you with SHA checksums as well as a filename list and filesize list. Note: if you have network level Activities, you'll also receive a .done file. This file however is different than the MT .done file.
To request .done files for your account, contact your account manager or support team.
6) Use open source tools
Processing your DT and MT files doesn't have to involve expensive ETL resources. There are a number of open-source ETL tools (as well as open-source databases such as MySQL) which can do a fine job while at the same time helping you manage operation costs. One particular tool is Crush-Tools. We designed Crush-Tools internally and released it to the open-source community a few years back. Crush-Tools is a unix-based library of utilities that are specifically designed to consume, manipulate and operate on DoubleClick Data Transfer files. For instance, via pipes and redirects you can filter, aggregate, calculate and produce reports all directly from DT flat files. We even use Crush-Tools extensively ourselves! Read more about Crush-Tools here.
We hope these tips will prove helpful to you.
--Matthew Trojanovich, the Data Transfer Service Team
* Some clients have asked to receive their DT files more frequently. Our product team is indeed working on this. Dates cannot be given yet, but stay tuned to this channel for more.
Subscribe to:
Posts (Atom)