Data Science on Kailas Venkitasubramanian

Announcing nationalparkscolors - Color palettes inspired by America's National Parks

Sun, 15 Feb 2026 00:00:00 +0000

“In every walk with nature, one receives far more than he seeks.” - John Muir

Deploying AI across the Research Life Cycle

Fri, 06 Feb 2026 00:00:00 +0000

Deploying AI across the research life cycle

Last summer, I gave a workshop to the staff on augmenting the research proposal process with AI. Thought I’ll sketch out what an AI-assisted research workflow might actually look like for both quantitative and qualitative work. One caveat before we get into it. This is a map of possibilities, not a prescription. What you can actually do depends on your data, your project, and what UNC Charlotte currently permits. Before trying any of this, check what the university’s Office of OneIT has published on approved tools and data classification. That guidance sets the real boundaries. This is more of a documentation of experiments that hopefully will be part of how our AI strategy takes shape.

Some Thoughts on AI-Augmented Community Research at the Charlotte Urban Institute

Tue, 11 Nov 2025 00:00:00 +0000

I thought of compiling a few thoughts on using AI at the institute or broadly in organizations similar to ours. I’ve been using AI tools in my research for over a year and I still can’t decide if I’m more excited or more unsettled but I feel it’s more of the former these days. But this ambivalence feels like the right starting point for a conversation about where we, as an institute built on community trust and research, actually want to go with this. When I say AI, I mean the generative AI kind.

Beautiful Chord Diagrams with Flourish

Tue, 08 Jul 2025 00:00:00 +0000

There’s a moment in data visualization where a chart stops being a chart and becomes a conversation starter. Chord diagrams do that reliably. Whenever I’ve dropped one into a presentation or a report, someone in the room leans forward. They want to trace the lines. They want to understand what connects to what. That reaction is exactly why I keep coming back to them.

What Is a Chord Diagram?

A chord diagram is a circular visualization that shows relationships between entities. Imagine a circle divided into arcs — each arc represents a category or group. Ribbons stretch between arcs, connecting them. The width of each ribbon encodes the magnitude of the relationship: a thick ribbon means a strong or frequent connection; a thin one means the opposite.

Embracing multilingualism in data science

Thu, 10 Apr 2025 00:00:00 +0000

Both of those efforts — reproducibility and pipelines — rest on a more basic question: which programming languages should a small research team actually use? In the previous posts of this series, I covered why reproducibility matters and how we are designing reproducible data pipelines at the UNC Charlotte Urban Institute. This post is about the layer underneath both.

Specifically, I want to argue that embracing multilingualism—fluency in both R and Python, rather than loyalty to one—has quietly done more for our team’s output than almost any other choice we’ve made.

Switching from ArcGIS to QGIS (and a bit of R too)

Tue, 18 Feb 2025 00:00:00 +0000

I have been using ArcGIS for longer than I care to admit. I started with it during my postgraduate years, and for a long time it was just the GIS software, the one everyone used, the one you learned if you wanted to do spatial analysis seriously. Our university has an enterprise license, so it has always been available, and old habits die hard.

But lately, I find myself opening it less and less. And when I do, there is usually a nagging feeling that I could be doing this in QGIS instead.

Bayesian Improved Surname Geocoding: How It Works and Where We Use It

Sun, 15 Sep 2024 00:00:00 +0000

If you work with administrative data long enough, you run into the same wall eventually: the dataset has everything you need except race and ethnicity. Hospital discharge records, voter files, tax records, benefits enrollment data — these are often rich with information about where people live, what services they use, and what outcomes they experience. But ask whether they capture race or ethnicity, and the answer is usually no, or inconsistently, or only in ways that aren’t usable.

Using tidycensus to Analyze ACS PUMS Data

Sun, 12 May 2024 00:00:00 +0000

If you’ve spent any time working with Census data, you know the drill: pull a pre-aggregated table, get median household income by county, move on. It works, and for a lot of questions, it’s exactly what you need. But sometimes the published tables just don’t cut it. What if you want to look at wage distributions for workers with specific educational credentials? Or model individual-level outcomes rather than tract-level averages? That’s where PUMS comes in — and once you start using it, it’s hard to go back.

Changing CRDT operations under a Cloud

Fri, 09 Jun 2023 00:00:00 +0000

The promise and peril of a large contract

Much of the previous challenges in managing the technical operations at the data trust stemmed from a lack of understanding of the scope and extent of effort for a given piece of work and having no barometer to measure productivity (or the lack of it). This meant that everyone knew that a given piece of work took 1 month to complete, everyone agreed that this delay was not acceptable,but no one really could pinpoint where the bottlenecks were and why they existed.

Revamping the Charlotte Regional Data Trust - The story so far

Mon, 22 May 2023 00:00:00 +0000

Plunging into the Data Trust black box, and Deep Cleaning the System

Sat, 20 May 2023 00:00:00 +0000

Diving into the world of administrative data and CRDT

Administrative data is messy is not much of an adage as much as it is a reality. When I took reins of managing the data infrastructure and analytical operations of Institute for Social Capital or ISC (now called the Charlotte Regional Data Trust) in the middle of 2021, messiness extended beyond data. The dysfunction was deep in how data was collected and organized, the way data operations and analyses were conducted, how information was collected from stakeholders, and how data was disseminated.

Releasing v1.0 of the Charlotte-Mecklenburg Quality of Life Explorer Data Pipeline

Thu, 16 Mar 2023 00:00:00 +0000

A successful wrap

It’s a great day today.

We completed the version 1.0 of the Charlotte-Mecklenburg Quality of Life Explorer automation project. It’s hard to put in words how exciting and rewarding this feels but I’ll try anyways.

We managed to automate most of the data and computational processes needed to generate the 80-odd quality of life indicators featured in the explorer and create functional data pipelines to serve the application. Through this work, we’ve accomplished a significant reduction of our project workload and fundamentally transformed the nature of our engagement in this project. The completion of this work also revitalizes our team’s vision to building a reproducible data science framework at the Urban Institute and a unified data ecosystem. Let me tell you how all this worked out.

Towards reproducible data science for community and policy research - An experiential roadmap

Mon, 06 Mar 2023 00:00:00 +0000

On developing a reproducible data science framework and practice at the Charlotte Urban Insitute

Charlotte Regional Data Trust - Technical Operations Manual

Sat, 06 Aug 2022 00:00:00 +0000

On how we developed the technical operations manual at the Charlotte Regional Data Trust

Reproducible Research Framework at the Charlotte Urban Institute: Why It Matters Now

Fri, 08 Jul 2022 00:00:00 +0000

In recent years, conversations about reproducibility have moved from academic journals into policy circles, foundations, and government agencies. What was once framed as a “replication crisis” in psychology has broadened into a wider concern about the credibility, transparency, and cumulative nature of scientific work across disciplines (Open Science Collaboration, 2015).

For those of us engaged in quantitative community research—especially in dynamic regional contexts like the Charlotte metropolitan area—reproducibility is not merely a philosophical concern. It is an operational one.

Make your code purr - A short intro to iterating using PURRR in R

Tue, 10 May 2022 00:00:00 +0000

Introducing ‘Purrr’

What’s programming without iterating? Purrr is a package that provides a set of tools for working with functions and vectors in R.

Built on top of the map functions (e.g., lapply(), sapply(), etc.) in base R Purrr provides a more consistent and modern syntax. Purrr is particularly useful for performing repeated operations on lists and data frames. It provides a collection of functions that make it easier to apply functions to data, iterate over lists and vectors, and manipulate data in a flexible way.