Charlie Allom - Curriculum Vitae

London, SE5 8BS, UK   charlie@evilforbeginners.com   https://github.com/yeled
                      07432154285                    https://spodder.com

Profile

Network engineering leader with over 20 years in production networks and more than a decade leading, and hiring for, lean network teams at GitHub and Yelp: hundreds of CVs screened, tens of engineers hired, some through people I already knew in the industry. At GitHub I took the network from a flat, hand-administered topology to a segmented, IS-IS SR, fully automated and telemetry-driven one while outbound traffic grew from Gigabits to Terabits. At Yelp I turned a startup network into an enterprise one spanning the USA, the Atlantic and five AWS regions.

The small teams I have led meant that I was a majority player in: setting the roadmap, negotiating with carriers and vendors, looking after capacity and procurement, designing the on-call, and moving network changes into the same code-reviewed, CI-deployed workflow the developers use. I am gregarious, get on well with people, and am an autodidact.

Work Experience

GitHub 2017 (7 years)

Led the network team (always lean, 2-4 people) for seven years, taking an unmanaged flat topology that the previous engineers administered over telnet to a segmented, IS-IS SR, telemetry-driven and fully automated network. I brought the whole network into GitHub’s internal developer method: deployed and treated like a group of applications, rather than a tended datacentre with siloed experts.

• Built and headed a 2-4 person team through Gigabit to Terabit growth in outbound streaming: set the roadmap and design standards, ran the on-call, and moved every network change into code review and CI so a lean team could run a network of that size.
• At the front of the team’s hiring pipeline for seven years: writing the engineering tests, screening, interviewing and making the hiring call, and bringing engineers in through my own contacts as well as the open pipeline.
• Owned capacity strategy and procurement. Replaced a process I first ran by hand with automated growth alerts and metrics that drove purchasing of long-lead-time capacity.
• Directed and managed the commercial negotiations for transit, backbone, PNI and all DCI interconnections.
• Designed the DCI ring topology on unlit dark fibre using Coriant (later Infinera, now Nokia) G30 DWDM to provide Petabits of connectivity.
• Built the deployment workflows in GitHub Actions and an autonomous reactive system needing little human input, including DDoS mitigation with S/RTBH and external signalling to Akamai. Weathered the largest DDoS recorded at the time, 2 Tbps inbound in February 2018, with that automation (covered by Wired).
• Transformed a static RSVP-TE and BGP network into a dynamic IS-IS SR and BGP design, using BGP for signalling between applications, so the network reacts faster to events.
• Removed A10 and Juniper NAT devices, replacing them with a horizontally scalable nftables and GRE tunnelled SNAT service in Linux, with health monitoring.
• 100% automated config generation for every device in the network.

Yelp 2013 (4 years)

Led the production network team, taking Yelp’s network from a startup mentality to an enterprise built on modern practices, transparent design and documentation.

• Migrated from static routing, OpenVPN and Linux routers to a failover-tolerant network of Juniper MX routers: redundant 10G lambdas spanning the USA and the Atlantic, and five AWS regions connected to datacentres over MPLS for multi-tenant growth.
• Ran the network team’s hiring from writing job specs, through CV screening to decision.
• DDoS mitigation planning and rollout. Identified the risk and went from zero to soaking up 99% of volumetric attacks.
• Led the team to Dockerise and Ansible every moving part (flow and SNMP monitoring, configuration management), with all network services HA.
• Designed internal and external anycast for HA services (DNS, code repositories, Puppet) with exabgp and BIRD.
• Instigated peering, and the culture around it, for a better mobile experience, backed by ongoing metrics and reporting.
• Monitoring with RIPE Atlas, Catchpoint, SNMP traps, in-house systems and live BGP updates fed into Sensu, Nagios and IRC bots for visibility of moving parts.
• Managed and tracked IRR resources in the ARIN and RIPE regions.
• Shared the operations on-call rota with the SREs, who described me as their network-SRE. Owned the US and EU office networks and their corporate on-call for 18 months.

Media Service Provider 2007 (6 years)

Moved to the UK. Launching an independent music service with restricted capital and strong performance and scalability requirements taught me to work inside a budget and prioritise business-critical issues. A small team meant being a jack of all trades and finding innovative ways to get the job done with minimum fuss.

• Designed and ran a fully redundant white-label music platform for ISPs serving hundreds of thousands of users, including day-to-day operations, hiring and training the junior sysadmins, runbooks for them and the developers, and complete DR plans.
• Owned the edge BGP architecture through each stage of growth while minimising capex and opex. Instigated RIPE LIR membership and PA/PI resourcing.
• Built a residential ISP with DPI for tracking licensed P2P content (a working proof of concept later used by Virgin Media).
• Built a 10GE ring for storage and applications, and grew a multi-petabyte MogileFS storage network across multiple London datacentres.
• Provisioned remote branches with Juniper SRXs and routing policies, and migrated all offices from a legacy PBX to hosted SIP with BCP for security and QoS.
• Enabled full IPv6 to all desktops and partially to infrastructure by 2009. Documented the network at L2 and L3 with runbooks for staff.
• Designed production and development virtualisation environments (first Xen hypervisor with an NFS backend in 2007) and helped the development team adopt release cycles, semantic versioning and code review.

Earlier roles

• Editure (formerly MyInternet) 2006 (6 months) - High-level support for UK educational authorities’ network and application infrastructure from Melbourne, modelling previously unknown complex systems from minimal information.
• Freshtel R&D Holdings 2006 (6 months) - Guided the sysadmin team on process and management. Deployed Checkpoint firewalls, VPNs for SIP trunks and Foundry load balancing for a heavily patched Asterisk environment.
• MyInternet 2002 (4 years) - Grew from one of eight sysadmins to managing the team and acting as technical account manager for the company’s two largest clients. Designed and supported scalable failover UNIX farms (Debian and Solaris) of up to 80 machines per POP, serving 3 million students and teachers 24x7.
• RMIT 2001 (2 years) - First networking and UNIX role. Supported over 300 students and staff, introduced Mac and PC cloning, remote management and ARP network audits, and deployed Squid proxies for student VLANs. Accredited Apple repairer.

Education

• RMIT 1998 (3 years) - Graphic design (typography). Moved to Melbourne to study design and soon learnt I was a natural problem solver.