The homepage for the ACLED data project can be found here, but to use this dataset for a DSAN 6000 project, you should use the full, extracted data available in the S3 bucket at the following URI:
s3://dsan6000-data/acled_events.parquetFor this pathway, imagine you have been hired by ACLED and asked to apply your DSAN 6000 skills to a research project aimed towards improving their event-processing pipeline. Their needs are:
As part of this task, you have been given access to ACLED’s OLTP database (described below), and you have been told to “take it from there”: your job, on Questions 1-6 on the Midterm, is to plan out what each step in an OLTP \(\leadsto\) OLAP \(\leadsto\) analytic insights pipeline for this task might look like.
For the purposes of the midterm, you should think of the data you worked with in HW4 and HW5 as the form of this data already extracted to OLAP. Meaning, for the midterm you are going a few steps back to the OLTP records that are originally received by ACLED. An example of the OLTP records that might show up in a given period of time is as follows, where you can imagine this as a sample of the most-recently-added rows in an events table (think of the streaming-events OLTP demo from the Week 1 slides!)
import pandas as pd
from great_tables import GT
acled_oltp_df = pd.read_csv("acled_oltp.csv")
(
GT(acled_oltp_df)
.fmt_url(columns='report_url')
.fmt_datetime(columns='received')
.cols_width(cases={
# 'received': '50px',
'report_title': '50%',
# 'report_url': '50px',
})
)Note the complexities that may be involved in turning this OLTP-format stream of events into an analytics-friendly OLAP infrastructure meeting ACLED’s needs (as listed above). For example:
received timestamps (between February 28 and March 29), these reports actually describe two separate incidents from US bombings in Iran earlier this year, which occurred on the same day (February 28)