Building a data pipeline to defend New York from cyber threats


Defending New York City’s IT infrastructure is a daunting task. With 8.6 million residents just within the city’s five boroughs, the city hosts hundreds of web applications so residents can track and use services like street plowing, as well as the popular site. With more than 330,000 employees within the city and 400,000 endpoints to keep track of — all within several, federated agencies such as the NYPD and Immigrant Affairs — the attack surface is huge.

It’s a responsibility that falls to New York City Cyber Command, an 18-month-old agency charged with defending the city from cyber threats and enabling New Yorkers to lead safe digital lives.

Given the scale and federation of New York CIty’s IT infrastructure, the agency decided to build its own data pipeline. The agency wanted to build a secure, cloud-based security log aggregation platform for city systems — one that enabled alerting, visualization and analysis for cybersecurity professionals. The pipeline also had to allow the agency to scale non-linearly as the demand on services grows and cybersecurity threats grow.

Read more…