Analyze Your Custom Data Extract Online

The IPUMS online analysis system uses high-speed tabulation software developed at UC-Berkeley's Computer-assisted Survey Methods Program. All data are provided by participating National Statistical Offices.

Until recently, online analysis was only available for the pre-built datasets listed on the Analyze Data Online page. However, we are currently beta-testing a new feature that allows users to analyze custom data extracts built with our online data extract system. This will allow users to analyze any arbitrary combination of samples and variables (subject to size limits) online, and to take advantage of special extract system features like Attach Characteristics.

To gain access to this feature by becoming a beta tester, please email us at ipums+analyzeonline@umn.edu.

NOTE: The statistics produced by the online analysis system are derived from sample data processed by IPUMS. They may not match official published statistics.

Extracts Eligible to Analyze Online


There are a few limits on the kinds of extracts that can be analyzed online:

Extracts must be no larger than 10 GB in size

Extract size for fixed-width text extracts depends on the number of records, the number of variables, and number of digits used by each variable, and is thus hard to predict based solely on the number of samples and variables you have selected. The best way to make sure your extract is not too large to analyze online is to build an extract with the samples and variables you're interested in, then click on Create Data Extract in your Data Cart. About halfway down the page you will see the "Estimated Size" of your extract in MB. If the estimated size is close to or more than 10,000 MB, you might want to remove some samples or variables or use the Customize Sample Sizes feature to avoid exceeding the 10 GB size limit.

To give you a rough idea of how big a 10 GB fixed-width text extract is, if we assume an average of two digits per variable, and use the average IPUMSI census sample size of 3 million records, an extract with 33 samples (~100 million records) and 50 variables would be roughly 10 GB.

Extracts must have a rectangular data structure

IPUMS International extracts can have a rectangular or hierarchical data structure, but only rectangular extracts are eligible for online analysis. In a rectangular extract, every record is a person record and household-level variables are repeated on each person in a household. By contrast, in a hierarchical extract, each record can be either a household or a person record, and the records for people in the same household are listed consecutively immediately after the corresponding household record.

If you're planning to analyze your data extract online, make sure the "Structure" field on the extract submission page is set to "Rectangular (person)". If that field says "Hierarchical", click the "Change" link to change to a rectangular structure.

Extracts must have a fixed-width text data format

IPUMS International extracts can be created in one of five data formats: fixed-width text (.dat), Stata (.dta), SPSS (.sav), SAS (.sas7bdat), and comma-delimited (.csv).

If you're planning to analyze your data extract online, make sure the "Data Format" field on the extract submission page is set to ".dat (fixed-width text)". If that field says anything other than ".dat (fixed-width text)", click the "Change" link to change to a fixed-width text data format.

Building Your Online Analysis Workspace

After receiving beta access to this feature, the first step is to create a new custom data extract that meets the eligibility criteria listed above. If you're new to our data extract system, check out this video tutorial on creating an IPUMS International data extract. Extracts you created before gaining access to this feature won't be eligible for online analysis.

When you submit your extract, your browser will navigate to the "Download or Revise Extracts" page as usual, but you will see a new column with the heading "Analyze Online" in the table that lists your extracts. While the Status of your extract is still "Processing", the Analyze Online cell will contain a message indicating that the extract is not available. If your extract does not meet the data structure or data eligibility criteria described above, the Analyze Online column will contain a message indicating that the extract is not eligible to be analyzed online, along with the reason it is not eligible.

Assuming the extract is eligible, once it finishes processing, a button labeled "Build SDA Workspace" will appear in the Analyze Online cell. Click on this button to start building your workspace. While the workspace is being built, the Analyze Online cell for your extract will have a value of "Building". The time required to build a workspace depends on your extract size. A workspace for a smaller extract may be ready in a few minutes, while a workspace for an extract close to the 10 GB size limit may take 15-20 minutes to build.

Once the workspace is ready, two new buttons will appear with labels "Open in SDA" and "Deactivate". Click the Open in SDA button to launch your online analysis workspace in a new tab. Click the "Deactivate" button when you have finished using your workspace.

See the instructions page or the video tutorial for information on using the SDA online analysis system.

Other Limitations


In addition to the extract eligibility limits described above, there are a few other limits we've put in place to allocate our computing resources as effectively as possible:

Back to Top