10x Genomics
January 2021 to July 2025
We had a smaller team at 10x, and as a result I performed a wide variety of duties as needed.
EDR Telemetry Storage/Querying: I designed and implemented a solution to query EDR telemetry. At the time we used Carbon Black which had a relatively short
retention time. This negatively impacted a number of our investigations which required EDR telemetry.
Essentially Carbon Black offered a container which could forward EDR telemetry in jsonl format to an S3 bucket, but the volume was exceedingly large (on the order of 2-3TB/month in which all machines were comingled).
I was able to use a combination of scripts running on AWS Lambda and AWS Glue/Athena to index all the telemetry on a daily basis into Parquet format. This allowed us to query telemetry for an individual host out of Athena on a date range of 6 months in a few seconds as opposed to the query simply timing out for more than a day.
This functionality proved indispensible during a series of incidents and investigations over a few years.
HRIS-IT Data Sync: we had persistent issues with the HR-IT data sync frequently resulting in offboarding failures which led to frequent investigations and necessitated frequent manual review. I was tasked with fixing this.
The first step was to reverse engineer the existing system: it definitely met the criteria for a "rube goldberg" machine, passing through 5 discrete implementation technologies, in order:
Workday report, Boomi jobs, ServiceNow sys_user table, SN business rules and schedule jobs, a ServiceNow MID server and relevant wretched powershell scripts (including a "# sleep 10 seconds so that the badge system can pick up the deactivation"), active directory, and then an Okta-AD agent.
Data Platform Team: I was asked to help pivot the data team to incorporate software engineering and modern tooling. I performed technical evaluations for three data/software engineers and set up software development practices.
We built a large variety of internal tooling to facilitate migration of data pipelines off of Boomi, and evaluated and implemented several platforms, including Fivetran, Astronomer (ha), and SplashBI. I also set up Okta SCIM integration for our apps and worked on simplifying the Snowflake RBAC configuration to achieve SoX compliance though this project was still ongoing when I left. I also took the time to underscore to my friends at Snowflake the absolute joy of trying to use their RBAC model to answer any substantive compliance question.
The largest challenge was funnily enough getting raw usable data out of Oracle Fusion Cloud ERP as Oracle seems to have "mailed in" the completion of their data export solution for Fusion Cloud versions called BICC. We were able to work with SplashBI to "bit bang" paginate the raw data tables out over their BIP reporting interface, which then allowed access to full copies incrementally synchronized to our Snowflake data warehouse.
Google
January 2017 to January 2021
At Google I work on the Cloud Security team, specifically Security Health Analytics. We are doing vulnerability detection for various benchmarks like CIS.
Microsoft Corporation
Summer 2015 Intern & Full Time August 2016 to December 2017
Authentication
I spent a lot of time working on improving the reliability of authentication. There were a lot of user complaints surrounding authentication problems -- particularly being prompted to re-enter account information daily but also not being able to sign in at all.
I was able to trace the issue back to an error caching behavior in the Office code. Errors were remembered and the presence of any error would set an "Invalid" flag on the account. Many routines would first check against the "Invalid" flag and explicitly fail if it was set, therefore any mishandled error would cause many routines in the future to break. There was no well-defined way in which this error would be cleared except for sign-in, so the clients would end up displaying "Please re-enter your credentials" and after a user re-entered them the error state would be cleared. The underlying OAuth account object never experienced a problem.
The investigation was fairly lengthy and involved the analysis of a "black box" system. We had the advantage of being able to read the code (not that the code was readable...) but the large number of modules and the high level of overengineering made this a bit difficult. We also spent a while interfacing with "politically distant" teams to do pair programming and additional discussion of best practices.
To accomplish this, I spent a good deal of time instrumenting a telemetry system. To make matters worse, there was no good set of hooks to keep track of the authentication subsystem. Each state (unauthenticated or authenticated) was valid and we cared primarily about "invalid" transitions between state. I ended up doing feature work to:
- Map identity to resource at time of opening
- Develop a modularized authentication system to track state in a thread-safe way
- From the state, produce telemetry about the system on a per-account level
- Use an activity telemetry system to track progress and behavior of user sign in to understand behavior of authenticaiton modules. I had to optimize this telemetry to limit data volume given the scale of our deployment (installed on every Windows 10 device by default).
This also gave me hooks to retry things like clearing the underlying error state after an initial authentication error (which ended up clearing up 90%+ of the "Please sign in" bars).
I also did other bug fixing work, particularly refining the errors that we interpreted as "authentication-related." I also ironed out a problem with incomplete module initialization which I later discovered resulted in some 4 million spurious prompts per month.
Support & Customer Engagement
I'm very passionate about strong customer engagement and providing an exceptional support experience. Upon seeing some of the difficulties users were facing working with our application, I helped develop a support tool for agents to resolve common problems within OneNote and refined it based upon feedback.
I also got a chance to fly down to Dallas and engage with our support team there, particularly to provide support and escalation procedures and to help identify communication barriers that were hampering their progress. The result has been greatly improved cooperation between support and engineering and the support was recently called out at a conference as being some of the best in Office.
In the spirit of listening to customers, I also assisted in the development and implementation of a customer engagement procedure to leverage our pre-production deployments for early discovery of any build or service problems. We wanted to maximize the value of our investigations by setting individual procedures for the value of the data, giving greater priority to internal "dogfood" users as well as customers running pre-production versions of our product.
I've also worked with internal teams to optimize external messaging for any ongoing incidents, and monitored social media platforms like Reddit myself.
Other
Architected and implemented a collaboration telemetry set and dashboard to monitor usage and performance of collaboration scenarios.
Implemented a telemetry dashboard for error reporting on the health of OneNote files in the cloud. Included work in C# and ScopeScript.
Boston AV Productions, LLC
Founder and President (April 2013-December 2016)
I founded an Audio Visual Event Services company to compete with my University's own AV Services in an effort to provide better service and better value to on-campus groups. My roll is obviously flexible, but I went through LLC registration, accounting, strategic planning, insurance, payroll, taxes, etc. Fantastic learning experience and I'm glad I had the opportunity to do it!
General Electric Intelligent Platforms
Software Development Intern (Summer 2014)
Worked an the Proficy Vision platform team, up and down the stack from Database work to Java coding to interfacing multiple components. Spearheaded a project to enable SSL termination through a NodeJS-based proxy server that integrated all APIs during the transitional development phase and enabled future support for web sockets.
Editshare, LLC
Software Development Intern (Summer 2013)
Software development including some fun reverse engineering work. Architecture help with a Linux kernel cgroup-based QoS system for disk access. Reverse engineering work on OSX platform and a bit of dynamic library shimming to "overcome some technical restrictions on Apple's platform."
Cartera Commerce, Inc.
Operations Intern (Summer 2012)
Implemented scripts to integrate company and third party services. Also worked on provisioning and managing servers through Puppet for company web services.
